Umm. What's this inability to see that data is data is data?
Why do you think Unicode has anything in particular to do with filenames?
Those same unicode strings are often part of the file data itself, and

then that encoding damn well is visible in "ls -l".
Doing
	echo ä > file

	ls -l file
sure shows that "underlying octet" thing that you wanted to avoid so much.

My point was that those underlying octets are always there, and they do

matter. The fact that the differences may not be visible when you compare

the normalized forms doesn't make it any less true.
You can choose to put blinders on and try to claim that normalization is

invisible, but it's only invisible TO THOSE THINGS THAT DON'T WANT TO SEE

IT.
But that doesn't change the fact that a lot of things *do* see it. There

are very few things that are "Unicode specific", and a *lot* of tools that

are just "general data tools".
The problem is that the UTF-8 form is different, so if you save things in

UTF-8 (which we hopefully agree is a sane thing to do), then you should

try to use a representation that people agree on.
And NFC is the more common normalization form by far, so by normalizing to

something else, you actually de-normalize as far as those other people are

concerned.
I blame them for encouraging normalization at all.
It's stupid.
You don't need it.
The people who care about "are these strings equivalent" shouldn't do a

"memcmp()" on them in the first place. And if you don't do a memcmp() on

things, then you don't need to normalize. 
So you have two cases:

 (a) the cases that care about *identity*. They don't want normalization

 (b) the cases that care about *equivalence*. And they shouldn't do

      octet-by-octet comparison.
See? Either you want to see equivalence, or you don't. And in neither case

is normalization the right thing to do (except as *possibly* an internal

part of the comparison, but there are actually better ways to check for

equivalence...
[ message continues ]

I'm not sure what you mean. I stated a fact - at least on OS X, the =20

filename does not contribute to the listed filesize, so changing the =20

encoding of the filename doesn't change the filesize. This isn't a =20
I don't, but I do think this discussion revolves around filenames, =20
Yes, I am well aware that the encoding of the *file contents* affects =20=
filesize. But when did I suggest changing the encoding of filenames =20

inside file contents? If you treat filenames as strings, there's no =20

requirement to change the encoding of filenames inside file contents. =20=
I'm talking specifically about the filenames, not about file contents, =20=
Don't want to, or don't need to? It's not a matter of ignoring =20

encoding because I don't want to deal with it, it's ignoring encoding =20=
Yes, I realize that. See my previous message about discussing ideal vs =20=
Was NFC the common normalization form back in 1998? My understanding =20

is Unicode was still in the process of being adopted back then, so =20
I could argue against this, but frankly, I'm really tired of arguing =20

this same point. I suggest we simply agree to disagree, and move on to =20=
actually fixing the problem.
-Kevin Ballard
--=20

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Changing the encoding of the file name most certainly changes the

file size of the directory.
--

David Kastrup, Kriemhildstr. 15, 44793 Bochum

-

And my point was that your *whole* argument boils down to "normalization

is invisible".
When it isn't. It's not invisible for filenames, it's not invisible for

file contents.
You're trying to claim that normalization cannot matter. I'm just pointing

out that it sure as hell can. Exactly because lots of things don't

actually look at data other than as just a Unicode string. They do look at

the raw format.
I'm surprised that you make generalized sweeping statements about how it's

ok to normalize because normalization is "invisible", and then when I

point out that that isn't true, you try to limit it.
And no, that normalization is not invisible EVEN IN FILENAMES. If it was, 
I don't know which argument you're talking about. Git (and, btw, Linux)

does the "ideal" thing (don't screw up peoples data), and it turns out to

be the "practical" thing too (it can handle a wider range of cases than OS

X can).
So no, this is not "ideal" vs "practical". They aren't in any conflict 
.. and people have even suggested how. Hide the idiotic OS X choices by

making a OS X-specific wrapper around readdir() that turns it into NFC.
That's just about the best we can do. We can't *fix* the thing that OS X

loses information, but a least we can then show the lost information in

the same form it _probably_ was in originally.
But no, it won't "fix" git on OS X. 
			Linus

-

I'm really surprised that, after all of this, you're still horribly

misunderstanding my argument. I never said it was invisible. NEVER.
I'm also surprised that you seem to care more about this argument then  
You misunderstand my point. In a previous email I specifically used

the words "ideal" and "practical" to describe arguments, which is what  
And I've responded to that suggestion, multiple times, saying that  
Quite a while ago it was suggested that git uses a table that maps the

original byte sequence as seen in the index to the form returned by

readdir(). So far this has sounded like the best solution, but as I've

said before I don't know git's internals enough (or, really, at all)

to be able to work on this myself.
This solution should only "lose" information in the case where the

index has 2 filenames that HFS+ treats as a single filename.
Is there some reason this won't work?
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

You said it was invisible when you treat things "as text". Here's the

quote:
	.. when you treat filenames as text, it DOESN'T MATTER if the

 	string gets normalized ..
Without ever apparently realizing that "as text" is part of the problem in

itself. What is "text" to one person is gibberish to another.
In particular, the biggest reason to not normalize is that you don't know

it's text or Unicode in the first place. Which is why git doesn't do it.
And no, even with filenames you don't know that they are "text". People

encode stuff in them. And people don't always use UTF-8. 
Of course, you could ask everybody to create OS X-only programs that know

that under OS X, you only have a subset of filenames. If so, you're

complaining about the wrong tool. Especially when the whole point of the

tool was to be distributed (not to mention coming from an environment that

simply doesn't have the same silly limitations OS X has).
So here's a few clues:
 - "as text" isn't "as unicode": it may well be Latin1 or EUC-JP or

   something. Yes, it's still used. Git doesn't care, and very consciously

   has avoided forcing character sets, even if the *default* (and notice

   how it's overridable) commit message encoding may be utf-8.
 - In fact, even in unicode, the difference between "identical" and

   "equivalent" strings exists, and even in the standard, unicode

   strings are very much defined to be arbitrary codepoint sequences, not

   normalized.
So even for the very specific case of unicode text, it's simply not true

that "it doesn't matter if the string gets normalized". The unicode spec

itself talks about cases where even canonical normalization makes a

difference.
Search for this quote:
  "Not all processes are required to respect canonical equivalence. For

   example:
    * A function that collects a set of the General_Category values

      present in a string will and should produce a different value for

      <angstrom sign, semicolon&gt...
[ message continues ]

Which is actually a good argument as to why filenames should be  
Sure, I understand why git doesn't do it. I'm saying in a system which

uses unicode top-to-bottom, which you can create if you're using HFS+  
Again, I was talking about a system that used unicode top-to-bottom.  
I find it amusing that you keep arguing against having git treat

filenames as unicode when, if you had actually taken my advice and

read my previous email talking about "ideal" vs "practical", you'd

realize that I was not suggesting git should. I was simply describing

why having the filesystem specifically treat filenames as utf-8 isn't

a problem when the entire system is unicode-aware, and thus showing

how the problems that are cropping up in git aren't because the

filesystem treats filenames as unicode, but rather because the

filesystem treats filenames differently than other filesystems. In

other words, I was trying to illustrate that HFS+ isn't wrong, it's

just different, and the difference is causing the problem.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

NO I DO NOT!
Dammit, stop this idiocy.
I think it's fine having git treat filenames "as unicode", as long as you

don't do any munging on it.
Why? Because if it's utf-8, then treating them "as unicode" means exactly

the same as treating them "as a user-specified string".
So stop lying about this whole thing. I have never *ever* argued against

unicode per se.
All my complaints - every single one of them - comes down to making the

idiotic choice of trying to munge those strings (not even strictly

"normalize") into something they are not.
And what you don't seem to understand is that once you accept _unmodified_

raw UTF-8 as a good unicode transport mechanism, suddenly other encodings

are possible. I'm not out to force my world-view on users. If they are

using legacy encodings (whether in filenames *or* in commit texts or in

their file contents), that's *their* choice.
I actually personally happen to use UTF-8-encoded unicode.
I'm just not stupid enough to think that (a) corrupting it is a good idea,

*or* (b) that I should force every Asian installation of git to also force

people to use unicode (or even having all the conversion libraries and

overheads!)
So stop this idiotic "unicode == normalization" crap. 
I'm a huge fan of UTF-8. But that does not mean that I think normalization

is a good idea.
		Linus

-

Please read to the bottom of this email. As near as I can figure out,

you haven't done that on any of my previous emails.
When I say "treat filenames as unicode" I'm implying the equivalence  
If that's what "as unicode" meant, then the phrase "as unicode" has  
No, you've argued against unicode equivalency in filenames. Can't you

figure out, when the entire time I've been talking about equivalency,  
You're not using raw UTF-8, you're just using raw bytes. Calling it

UTF-8 doesn't mean anything, since you don't actually know that's what  
How many times must I say the same thing over and over? I'm not

arguing that forced normalization is a good thing. I'm arguing that,

in a system which is unicode-aware top to bottom, forced normalization

is irrelevant to the user, since they don't care about the exact byte

sequence. And I'm also arguing that git should have some solution to

this problem. I find it interesting that you're perfectly happy to

rant and rail against your misperception of my argument, and yet you

consistently and repeatedly ignore my offers to stop this argument and

work towards a solution, as well as my comments on existing proposed

solutions.
Are you even reading to the end of my emails?
- Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Yes, because you're an idiot.
I've told you over and over again that equivalence is stupid.
It's stupid when it's "equivalent except for case", and it's stupid when 
I agree: normalization and equivalency is idiotic.
You don't seem to.
The thing is, the two are inexorably intertwined. Any filename equivalence

(except for the trivial "identity" equivalence) INVARIABLY means that

filenames get munged.
Why?
Think about the file name "Abc", and think about what happens when you

create it.
Now, think about what happens if that filename is considered equivalent in

case..
See? The filesystem has to *corrupt* the filename.
Can you not UNDERSTAND this? Equivalence and normalization is STUPID. It's

just two sides of the exact same coin. They both INVARIABLY cause the

filename to be munged.
And changing user data is not acceptable.
Do you get it now?
			Linus "probably not" Torvalds

-

Let me make this really clear, because I'm afraid that you won't get it

when I leave out any steps of the way.
Let us say that there is a filename "xyz" that is equivalent to a filename

"abc" in *any* way. It does not matter if xyz/abc is Hello/hello, or

whether it's two canonically equivalent strings.
So now, do
	close(open(xyz, O_WRONLY | O_CREAT, 0666));

	close(open(abc, O_WRONLY | O_CREAT, 0666));
and then look at the directory contents afterwards.
There are two, and only two, choices here (*):

 - the filesystem created both files, and they show up as created

 - the filesystem decided they were equivalent, and munged one (or both)

   of them
Now, let's go back to my claim:

 - munging user data is unacceptable

and realize that equivalence BY DEFINITION must do it.
So no, you do *not* get to have your cake and eat it too. You simply

fundamentally *cannot* have both filename equivalence and a non-munging

filesystem. See above why.
		Linus
(*) Actually, there is third choice above, which is:
 - the filesystem created the first file, and errored out on the second

   because it noticed it was equivalent - but not identical - to one it

   already had
   This one is actually a perfectly fine choice, but it's not "your" kind

   of equivalence, since it actually makes a difference between two

   equivalent but non-identical names. So the filenames aren't actually

   interchangable, and this case is really more of a "the filesystem has

   some very specific limitations on what it allows".

-

Linus, have you even bothered to read my arguments, or do you just get

a kick out of building these straw man arguments? You have

consistently failed to actually address what I'm talking about, and

instead persist in explaining stuff I already know, as if that was the

answer to anything I've been talking about. You are clearly incapable

of understanding my basic point, no matter how simple I break it down.

I suspect it's because you've been working low-level so long you can't

think high-level, and so you manage to misinterpret my high-level

arguments as boneheaded low-level mistakes.
Anyway, please see my countless former emails where I ask to work

towards a solution instead of just arguing.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

We know what the solution is:
 - The OS X filesystem _is_ crap (and you seem to have almost admitted as

   much by your comment that the HFS+ designers did it back in the dark

   ages and didn't mean for it to ever be a server filesystem anyway)
 - But we can at least make a wrapper around readdir() return the NFC form

   on OS X, and effectively hide much of the fallout from the crap.
There is no way around it. Your "solutions" all seem to boil down to

asking git to do the same idiotic crap that OS X does, taking all the

same performance hits, and just generally doing crap just to work around

crap in your favourite OS.
And no, making git be stupid just to suit a stupid filesystem simply isn't

going to happen.
So how about you see _my_ point instead: OS X may have an inferior

filesystem, but we don't have to make git inferior just for that. The fact

that OS X does case independence is *its* problem, not git's.
		Linus

-

I agree that HFS+ isn't well suited for tasks which it is being asked

to do. I was never arguing that it was the perfect filesystem. But

that hardly matters now, I know nobody's going to bother understanding  
Again, I don't think that's the correct solution. What about the

translation table that was suggested back at the beginning of the

thread? That would solve the case insensitivity issue as well, whereas  
No, I am not asking git to do the same thing HFS+ does. You just

persist in misinterpreting my arguments, no matter how many times I  
So, what, you're saying git shouldn't do any work at all to try and

behave nicer on OS X? Because OS X sure as hell can't change to suit

git.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Kevin,
you seem to know the problem fairly well. Could you write up a set of

testcases that show the bug? See the "t" directory in the git sources

-- you don't need to learn much about git internals, they are just

shell scripts (mostly, I think there's some perl there too). That

could lead to a good contribution to the project.
... and keep you from telling everyone else that you know better how

to hack a project that you know nothing about ;-)
Kevin - for your edification, that question is usually referred to as

"trolling" in this place we call the internet. Linus outlined what his

technical plan is, so git will probably do something designed by

someone who knows a thing or two about git's internals. So when you

pretend that he is saying the opposite of what he is saying... well,

people do get upset.
cheers,
m

-

See now this is actually a very good suggestion. I probably should

have done this long ago. Thank you very much for actually responding

about the problem. You are the first person to do so.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Sure you do. You continue to say that unicode is the only choice, and you

continue to say that unicode requires that equivalent names be considered

the same.
Umm. Git works perfectly fine on OS X, and it's not like we can do a whole

lot more about it, exactly because we cannot fix the real problem. We can

hide some of the fallout (idiotic choice of normalization), but the bigger

issues we can hardly even do anything about (case independence).
And quite frankly, you've also made sure that I have absolutely zero

interest in even trying to help people with it.
			Linus

-

If by "ideal" you mean a world where 100% of all computers were

designed by Steve Jobs, you might have a point.  But trying to argue

for such a state of idealism seems to be stupid, and certainly a

complete waste of everyone's time on the git mailing list.  It's

simply not reality.  It's like with the infamous resource forks, which

would have worked fine if all the world were MacOS, but which had a

tendency to get stripped off whenver you used a program that wasn't

resource fork aware, like zip, or a protocol that wasn't resource fork

aware, like FTP.  And so people had to put in all sorts of kludges

like BinHex to work around MacOS's "if only the entire world was like

*me*, no one would get hurt" attitude.  In some ways, the MacOS

designers are even worse than Microsoft in terms of having the "the
And if you want to interoperate with the rest of the world, where at

least count over 92% of computers are NOT running HFS+, then "Thinking

Different" is indeed causing the problem, yes.  And whose fault is that?
The whole point of interoperability is that when we communicate, we

have to do so in a uniform and predictable way.  If we can't, the next

best thing is to have protocol translators; but in order to do that,

we must avoid lossy transformations, such as HFS+'s

pseudo-normalization.  (Why, by the way, will not result in a "normal"

form for any glyph which can be encoded with and without a combining

character if said glyph was introduced into Unicode after 1988.  So

you can't even call it a "normalization" algorithm, but just a

pseudo-normalization transformation which is lossy and which DESTROYS

filename information in an irrecoverable way.)
	      	     	     	 	  	    - Ted

-

NO NO NO NO NO. READ MY EMAIL. STOP MAKING ASSUMPTIONS ABOUT WHAT I'M

TALKING ABOUT.
The most frustrating thing about this thread is everybody keeps

arguing about what they *assume* I'm talking about without actually  
And if you want to interoperate with the rest of the world, where at

least count over 92% of computers are running Windows, then using

another OS is stupid, right? Right? I mean, if everyone else is doing  
Sure it's normalization, it's just not using one of the standard

forms. But the form is well-defined.
And yes, protocol translators are a good idea. That's why I thought

the original suggestion of using a table to map index filenames <-> HFS

+ filenames sounded like it could work. The only time that should fail

is if the index contains multiple filenames that HFS+ will treat as a

single filename. Is there a problem with this approach?
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Hmmm. I m pretty sure HFS+ has a lot of problems if you run OSX as an

NFS server with clients in different encodings. It would never work in

real life. The "envelope" OSs have to work in is hugely varied -- much
Did you spot the rather nasty issues that Ted mentioned earlier in the

thread? I would say HFS+ is a bit "special" rather than "different".
cheers,
m

-

Kevin,
as you might know, Linus' "other hobby" is to write kernels ;-) From

taht POV, a filename is as much data as the data in the file. Doing

odd things like sorting it, searching through it, etc, is all work for

code higher in the stack that is free to mangle the data in any way it

wants, including creating nice case-insensitive indexes, and

who-knows-what for ideogram-based languages. In contrast, the core OS

treats user data a sacred stuff, and I'm thankful it does.
And from a kernel/filesystem POV, a directory is also a file. So if a

filename has a different number of octets, the directory will be

different.
For all the searching and matching, it really makes sense to have

something like locate or SpotLight or whatever to index user files

that should be easy to find and match, because all the locale rules

for matching are hideously expensive to apply. Even today, most UTF-8

aware (and supposedly collation-smart) applications have trouble

matching MARTÍN when asked for martín in a case-insensitive search.

That pesky latin í trips them up everytime.
cheers,
martin

-

That's certainly a reasonable POV. However, it's not the only one. As =20=
evidenced by the Mac, treating filenames as strings rather than bytes =20=
is a viable alternative POV - you can't argue that it doesn't work, =20

because OS X proves it does.
Sure, that makes sense. That's why, if you are going to mangle =20

filenames, you need to pick a stable form to always use, which HFS+ =20
Perhaps you should try OS X. Every single Cocoa app should do the =20

search properly. In fact, I just checked using 3 different text =20

engines (WebKit, Cocoa's text engine, and ATSUI) and all 3 did the =20

case-insensitive search properly. That said, this isn't particularly =20

relevant.
-Kevin Ballard
--=20

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

With its own slew of bugs. See Ted's reply earlier for a mouthful of
OSX has given me enough grief with other filesystem and general OS

problems that I have definitely abandoned it after 2 years trying to

use it part-time. It has been back to linux for me ;-)
cheers,
martin

-

Linus' approach is _FAST_.
Why do you think Git has now acquired a reputation of kicking asses all

around the SCM scene?
The HFS+ approach might be fine if you think of it in terms of "the user

will be awfully confused if two file names are shown identically in the

File Open dialog box".  But it otherwise sucks big time when it comes to

high performance applications needing to deal with a huge amount of file

names at once.
Normalization will always hurt performances.  This is an overhead.

Sometimes that overhead might be insignificant and not be perceptible,

but sometimes it is.  And Git is clearly in the later case. Performances

will be hurt big time the day it is made aware of that normalization.

This is why there is so much resistance about it, especially when the

benefits of normalizing file names are not shown to be worth their cost

in performance and complexity, as other systems do rather fine without

it.
Nicolas

-

Normalization is cheap if you normalize user input.  The user will

always be quite slower than any reasonable normalization algorithm.  But

in the filesystem, one is normalizing the same stuff over and over.
--

David Kastrup, Kriemhildstr. 15, 44793 Bochum

-

I agree, Linus's approach is indeed fast. And if speed is more

important than treating filenames as text instead of octets, then so

be it. This is a trade-off. But a trade-off doesn't mean one approach

is "wrong", it just means the authors of HFS+ thought it was an

acceptable trade-off. HFS+ wasn't designed to be a high-performance

filesystem that deals with lots of files, it was designed to be a

filesystem used by regular people on the Mac, and I believe treating

filenames as text is a good choice in this scenario. Unfortunately,

this does mean git has to do extra work to behave correctly on this

system.
Now, to move on to actually coming up with a solution. Unfortunately I

don't know enough about the internals of git to really evaluate the

proposed ideas myself, or to write a patch. Hopefully I'll come up

with the time to acquire the necessary knowledge, but until then I can

only participate in these higher-level discussions.
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Regular people have brains, not filesystems.  HFS+ is employed by

computers, and computers can produce or query or process lots of data in

very short time spans, in their own pace.  And if Mac users did not want

to make use of that, they would still be using Mac classics.
--

David Kastrup, Kriemhildstr. 15, 44793 Bochum

-

Git's data model relies on SHA-1 hashing of data, including filenames.

So at some level, git _has_ to treat data as octets, and "equivalent"

strings must be the same at the octet level (or else you lose all of the

useful properties that the hashing data model provides). You can argue

about where in the program conversion and normalization occur, but I

don't think you can get around the fact that you're going to need

to think of the "filename as octets at some point."
-Peff

-

Could people _please_ stop this already?
I think the sane people see the difference between equivalence

and sameness, and we established that a filesystem that mangles

the filenames behind user's back is a bad design.  Anybody who

followed the thread and still does not agree with you is, eh,

"ugly-and-stupid", as you might say ;-).  You cannot educate

them all.
The thing is, even if you mange to educate them all, that broken

filesystem, and other filesystems with similar brokenness, do

not go away.  If your ultimate objective is to declare that it

is the right thing for git not to support such broken

filesystems, and to make everybody agree to it, that is fine.

Please keep pouring fuel to the fire.  But if that is not the

case, we would need to devise a way to help lives easier for the

unfortunate people who are stuck on such filesystems.  They may

not even realize that they are unfortunate now, and I agree that

some education is justified, but this thread has raged on long

enough to salvage any salvageable lost souls (the remaining ones

may be beyond salvation but let's not waste time on them).
I'd rather see our mental bandwidth spent on coming up with a

workable workaround for such broken filesystems, while not

hurting use of git on sane platforms.
I fear it might have to end up to be very messy and slow,

though.

-

Random thought.  Would it make sense to implement a git paranoid

mode to autodetect name mangling.
I.e.  After opening or creating a file by name we do a readdir in the

same directory to make certain we can find that same name/inode

combination.  Then on name-mangling systems we can autodetect they

exist and limit ourselves to just what they don't mangle with no

prior knowledge.  By refusing to process names that actively

get mangled.   For small directories that you frequently see in

development it shouldn't even be that slow.
Eric

-

Inside init-db where we already check how the filesystem

behaves, we could have an autodetection. A rough equivalent of

what I had in mind is:
	mkdir -p "Märchen/Märchen"

	if test "$(cd Märchen && echo M*)" = "Märchen"

        then

        	: not mangling

	else

        	git config core.namemangle true

	fi
(of course we do that in C not in shell).

-

I wonder if that is good enough.  Git repositories can be copied over to

different filesystems.
Nicolas

-

Do you mean "cp -a"?  If I am not mistaken we already have that

issue, due to core.filemode, when user does that across

filesystems with different behaviours.
There is not much we can do against "cp -a" other than telling

users that some configurations need to be adjusted.

-

Hi,
Actually, I see some value in calling them names, see 
I was almost starting with hacking on this, but then the discussion

annoyed me too much, and I asked myself for who I think I'd do this.
IMHO those people should ask "how could I begin to work on this".
Instead, they started a useless flamewar.
Now, back to the issue: Robin posted a link to his UTF-8 work.  While it

is way too intrusive, and not limited to filenames at all, I think it has

a few good pointers.
Ciao,

Dscho "who needs to calm down now"
-

As far as I can tell, the only time you ever run into the problems

you've described on a filesystem which treats filenames as unicode

strings (and therefore is free to normalize), are when you're trying

to interact with a filesystem that treats filenames as sequences of

bytes.
This doesn't mean treating filenames as unicode strings is wrong, it

just means that the world would be much better if every filesystem had

the same behaviour here. It's kinda like the endian issue, except

there's no simple solution here.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Hi,
[please do not top post. Just delete everything you do not reply to]
If you read the first message in this thread, you would probably know

that the problem exists on Mac even without any other filesystem being

involved. My understanding of it is that is caused by HFS+ converting

one sequence of Unicode characters (generated by Mac keyboard driver)

to another sequence using "fast decomposed" conversion.
[And please stop calling by normalization what is not. Mac does NOT

normalize Unicode strings, it uses some sub-standard conversion,

which neither produce a normalized string nor is guaranteed to be
Actually, there is, if you care to do something. You can write a wrapper

around readdir(3) that will recodes filenames in Unicode Normal Forms C.

This does not require much knowledge of Git -- what it requires the

desire to do something to solve the problem. Of course, this step alone

is not a complete solution (it does not solve case-insensitive issue),

but the first step in the right direction...
BTW, Git is far from being only software that ran into this problem with

Mac. But not being first, we can benefit from other people experiences:

http://osdir.com/ml/network.gnutella.limewire.core.devel/2003-01/msg0000...
Dmitry

-

If somebody wants to do this, then readdir() isn't the only place, but

yes, readdir() is one of the places.
I suspect that if we were to just do the "turn into NFC on readdir() on OS

X", that might actually be good enough to hide most of the problems. The

issue isn't just that OS X mangles the filenames, it's that it picks a

particularly *stupid* way to mangle them (the decomposed forms), which

means that OS X will actually not just corrupt "odd cases" of Unicode, but

will corrupt the obvious and *common* Latin1 translations of Unicode.
I don't know if NFC is better for other locales, but I doubt it. Usually

people want to do the *composite* forms, not the *de*composed forms.
A trivial example of this for some cross-OS issue:
 - let's say that you have a file "Märchen" on just about *any* other OS

   than OS X. It could be Latin1 or it could be Unicode, but even if it is

   Unicode, I can almost guarantee that the 'ä' is going to be the

   *single* Unicode character U+00e4 (utf-8: "\xc3\xa4", latin1: "\xe4")
   So from a cross-OS standpoint, that's the *common* representation, and

   yes, you can create the file that way (I don't know what happens if you

   actually create it with the Latin1 encoding, but I would not be

   surprised if OS X notices that it's not a valid UTF sequence and

   assumes it's Latin1 and converts it to Unicode)
 - But on OS X, because of Apples *insane* choice of normal form, it will

   then be turned into "a¨". I doubt *anybody* else does that. If you have

   to normalize it, NFD is just about the *worst* choice.
So yeah, even just re-coding it as NFC on readdir() would at least mean

that any OS X git client would be MORE LIKELY to pick the same

representation as git clients on other OS's.
It wouldn't solve all problems (and it would almost certainly create a few

new ones), but it would likely at least increase compatibility between

systems.
So doing the NFC conversion on readdir() on OS X is probably a good ide...
[ message continues ]

For what it's worth, their choice wasn't entirely "insane" ie. it did

have an element of rationality: that decomposed forms are a little bit

simpler to sort.
Of course, this doesn't excuse them for creating a file system that

interacts so horridly with basically everything else out there.
Cheers,

Wincent
-

No they are *not*.
In many languages, 'ä' does *not* sort like 'a' at all, and if you think

it does, you'll sort at least Finnish and Swedish totally wrong (åäö are

real letters, and they sort at the *end* of the alphabet, they have

nothing what-so-ever to do with the letters 'a' or 'o').
The fact that in *some* languages the decomposed forms sort as the base

letter is immaterial. It's only true in some cases.
So no, sort order is not it. To sort right, you need to use the a real

Unicode sort (and the decomposed form is *not* going to help you one bit,

quite the reverse).
It may be that a case compare is easier in NFD (ie you basically only do

the case-compare on the base letter).
		Linus

-

That's what I get for believing Wikipedia ("This makes sorting far

simpler"):
http://en.wikipedia.org/wiki/UTF-8#Mac_OS_X
Cheers,

Wincent
-

But there is no way to know whether 'ä' in a document is the Finnish 'ä'
Unicode sort is not enough, there is no language indicator in an Unicode

document, which is why Unicode, while solving a bunch of problems, has

its very own, cf. the infamous CJK problem.
But that's all very OT.
Mike

-

... without knowing the locale. Correct.
That's why sorting is locale-dependent, even in Unicode. And why you

should always sort using the *combined* character, not think that you can

sort by decompsed sequence.
That said, even then you get the wrong thing. Some things cannot be sorted

character by character at all, and have semantical sorting at a higher

level entirely. I think most European family names are traditionally

sorted by effectively using the prefixes (ie d', von, etc) as a secondary

sort key (so even though they are in front, they sort as if they were

at the _end_ of the name).
So unicode doesn't help with sorting, and you shouldn't even try to find

sort rules in the Unicode spec or tech reports. But in general,

decomposing the characters just makes things worse, not better. To sort

well, you tend to need the bigger picture, not the details.
Of course, for something like git, we sort by binary value, because we

also require the sort to be not just well-defined, but *stable*. A sort

based on any kind of unicode rule is rather likely to change over time.
			Linus

-

That said, the locale doesn't necessarily express the language in which

the document is written. It's easy enough to read documents that are not

written in your native language on the net. That's already what we are both

doing right now. Fortunately, HTTP and HTML have ways to indicate the

language in which a document is written in, but that leaves out plain

mail, for instance. 
That said, the "decomposed" version of UTF-8 has nice side effects on

OSX, with UTF-8 encoded RockRidge ISO-9660 volumes (with or without

Joliet ; OSX will use RockRidge by default when it's there), for instance.
Mike

-

AFAIK, the RockRidge standard prescribes to use the portable character

set, and it has nothing to do with Unicode. Basically, it is a subset of

ASCII.
http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap06.html
So, I don't think UTF-8 encoded filenames are valid regardless whether

they are decomposed or not.
Dmitry

-

Actually, it prescribes to use the portable *filename* character set,

which is even more restrictive than just portable character set.
http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html#t...
Anyway, there is no place for UTF-8 in it.
Dmitry

-

.. and quite commonly, there are multiple languages per document.
The good news is that sorting is almost never relevant or done over

general documents. You sort almost only well-behaved data, and quite often

the exact order is less than important: and when it is, you have very

specific rules (which probably seldom have anything what-so-ever to do 
I think Unicode in general (and UTF-8 in particular) is a great thing. I

do not argue against Unicode at all.  It's what I use myself.
The thing I argue against is that they force normalization (and then, as a

secondary complaint, their insane choice of target format).
Linux is generally UTF-8 too, and does all of this much better. No forced

normalization, and it uses UTF-8 everywhere as the encoding model. Joliet

and RR works beautifully.
(I don't think RR is NFD, btw. It's the standard microsoft UTF-16 without

normalization, afaik. I think you can happily generate a Rock Ridge disk

that has two _different_ filenames that OS X cannot tell apart, but that

both Linux and Windows can see peoperly)
		Linus
-

Hi,
I think a better approach would be to try to match the name to what we

have in the index.  Then we could implement case-insensitivity and MacOSX

workaround at the same time.
Ciao,

Dscho
-

I thought about that, but the problem is that HFS+ _already_ mangled

names from what the user entered (and what is used by anyone else)

to some sub-standard form, which no one outside of Mac likes or uses.

Thus, bringing filenames back to the NFC form (which is what almost

anyone uses) is the only sane thing do, because no one outside of Mac

really needs to know about this HFS+ specific craziness.
So I really dislike the idea that due to some HFS+ specific conversion,

we may end up having some strangely encoded names in a Git repository.

Sane people enter names only in NFC, so why should they suffer because

of some insane conversation made by filesystem behind everyone's back?

And I am not entertaining the idea of having this Mac OS/X specific

workaround outside of Mac OS/X.
Besides, writing a wrapper around readdir() is not difficult. We

already have git-compat-util.h, which redefines some functions for

some platforms, so I don't see any problem with writing a wrapper

around readdir().
Dmitry

-

Hi,
So?  That's why I said "match", not "compare for identity".
To be a little bit more precise: I think a viable plan would be to
- have a config switch which determines what type of filename mangling we

  allow the host OS to perform (Unicode "normalisation", case mongering),

  and leave _everybody_ alone who left that switch unset,
- "overload" readdir() (by the famous git_X(); #define X git_X trick),
- have the overloaded readdir() _know_ which is the current prefix, and

  load the index if it has not yet been loaded (but probably into a static

  variable to avoid reloading, and to avoid interfering with the global

  "cache" instance).
It _could_ be wise to store the "normalised" forms at one stage (instead

of the index) to speed up comparison -- the prefix has to be normalised

for readdir()s purposes, too, then.
This is possible with the HFS+ problem, since we know exactly how HFS+

tries to "help", and for case insensitivity too, I think.  But it may be

restricting ourselves for other filename "equivalences" we might want to

handle one day.
BTW: I cannot think of anything else than readdir() which should have the

"problem" of reading back a name that the user did not specify.  What am I 
No.  I think that would be a serious mistake.  If you add a file on MacOSX

(with a _mangled_ filename, think of "git add ."), git should not try to 
It _is_ UTF-8, so what's the problem?
As for the HFS+ specfic conversion: like the CRLF issue, I am opposed to

have a "solution" affecting other people than those on broken system.  So 
Exactly.
Ciao,

Dscho
-

Well, more importantly, most of the important cases actually don't have an

index entry yet.
For example, what about "git add"? That's when it really matters that you

add things in a sane format, and by definition, you don't have an index

entry to try to match to. 
So once you aim for NFC in "git add", now the index will generally be in

NFC anyway (since I agree that that's what you'd normally get on non-OSX

systems), so there is little point in then matching the index.
But no, it won't fix all problems. I do suspect it would make them less

obvious in practice, though.
		Linus

-

FWIW: I just made a test and it seems that MacOS X refuses the creation 
Maybe I'll try it.
Regards,

Mark

-

In this case, git's index counts as a filesystem that treats filenames

as sequences of bytes. But yes, it is possible, though somewhat

difficult, to produce this problem on just HFS+. It's far more common  
 From what the HFS+ technote says, it produces a variant of Normal

Form D. This variant, while not guaranteed to be stable across

versions of HFS+, but in practice it is stable.
I'm not sure how that would solve anything. Sure, it would provide a

stable, known encoding for git to compare filenames against, but that

would only work if the filename is known to be Unicode, and as it has

been pointed out on other filesystems the filename can be whatever  
It looks like their problem was binary compatibility with strings from

other clients that were using Normal Form C instead of Normal Form D.

git's problem is that it's only even using a known encoding on HFS+.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

There is no such thing in the standard as a variant of NFD. Moreover,

even if this conversion were described in the standard, it would never

called as normalization, because normalization means conversion that

makes all equivalent strings having identical binary representations.
Apple calls it as decomposition, which is correct even if it is not full
I believe that Git internally should use only UTF-8 for encoding file

names, commit messages, etc. The problem with some other filesystems

should be addressed separately (by those who work on those systems or

at least have access to them). Regardless interoperability with other

systems, this change alone should solve the issue that was described

in the first message of this thread.
Dmitry

-

There are canonically equivalent, but they are different sequences

of characters as Unicode is concerned. In one case, we have one

character in the other case, we have two characters that canonically
By defition, sequences of characters that are canonically equivalent
I am afraid it is you who confuses "characters" with "abstract

characters", there is no place in the standard saying that

"characters" are "abstract characters" only. On contrary, the

term "characters" is used to refer non abstract characters.
Dmitry

-

Perhaps it's just a case of confusion about naming conventions. I tend

to use "character" as a "grapheme cluster", i.e a "user character" (to

the end user, "ä" and "a"+diaeresis is the same character, no matter if

they would display as different glyphs), whereas some people use

"character" as a "code point", which would be more of a "programmer

character". And then there are some people that still use "character"

interchangibly for "bytes" or "code units" (for UTF-16; a pair of

surrogate code units is still only one "code point").
--

\\// Peter - http://www.softwolves.pp.se/

-

I just don't understand why you insist that the filename is data, when

it is clearly metadata. The filename has two purposes: the identify

the file to the user, and to provide a handle with which to reference

the file contents. The specific byte sequence is in no way sacred.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Uhh. And exactly how do you know the difference, and why should it matter?
A lot of data is metadata. Look at the git index file. It's *all*

metadata. Does that mean that the OS has the right to corrupt it?
IOW, why do you seem to argue that metadata something you can corrupt, but

not then "regular" data?
Why is it ok to change a filename, when that same filename may *also* be

encoded by the user in a regular data file (think about MD5SUM files, for

example, that include the pathname, but now the pathname is part of the

file data, not on a filesystem). 
So filenames are data, they're metadata, they're whatever. None of that

means that it's acceptable to corrupt them, or gives the OS any reason to

say that it "knows better" than the user in how users use them. It's still

the *users* metadata, not the filesystems own metadata!
In many cases, users use filenames *as* data, ie the filename actually has

a meaning in itself, not just as a handle to get the file contents.
If this was truly metadata that isn't visible to the user, and not under

the users control (ie indirect block numbers etc), then you'd have a good

point. At that point, it's obviously entirely up to the filesystem how the

heck it encodes it.
But that's not what filenames are. Filenames are an index specified by the

user, not by the computer. 
		Linus

-

It's UTF-16 (when needed). I think it's all in the Linux kernel for you
It uses the local 8-bit codepage, which is not UTF-8, often some latin-inspired

thingy, but in Asia multi-byte encodings are used. In western Europe it is

Windows-1252, which is almost, but not exactly iso-8859-1. Oh, and then we

have the cmd prompt which has another encoding in 8-bit mode.
I think there is a cygwin patch that converts to and from UTF-8. An application

can choose to use the "A" or "W" interfaces. The W-API's are the real ones and

the others' are just wrappers that convert to and from UTF-16 before anything

happens (i.e. CreateFileA is slower than CreateFileW and so on). 
-- robin

-

.. well, FAT certainly wasn't. But yes, VFAT probably is.  Not that I want 
Well, if it uses a 8-bit codepage, then that means that as far as the

POSIX filename interface is concerned, it has nothing what-so-ever to do

with Unicode (ie unicode is just a totally invisible internal encoding

issue, not externally visible).
I assume you have to use some insane Windows-only UCS-2 filename function

to actually see any Unicode behaviour.
Sad. Because there really is no reason to use a local 8-bit codepage when 
So the CreateFileW() is the "native UTF-16 interface", and CreateFileA()

is the 8-bit codepage one that has nothing to do with Unicode and is

purely some local thing.
But for a UNIX interface layer, the most logical thing would probably be

to map "open()" and friends not to CreateFileA(), but to

CreateFileW(utf8_to_utf16(filename)). 
Once you do that, then it sounds like Windows would basically be Unicode,

and hopefully without any crazy normalization (but presumably all the

crazy case-insensitivity cannot be fixed ;^).
So it probably really only depends on whether you choose to use the insane

8-bit code page translation or whether you just use a sane and trivial

UTF8<->UTF16 conversion.
Anybody know which one cygwin/mingw does?
			Linus

-

I just had to investigate this a bit, so on a Vista machine I started a cmd

prompt and typed mode con: cp select=65001, selected the lucida font and then

echo å >x.txt and opened it in notepad and it was UTF-8 encoded. So there might

be some hope after all. I don't know how to change the encoding for non-console

apps. I leave that as an excercise for the list.
-- robin

-

Yes, but have you tried to run any batch file? At least, on WinXP

all batch files silently stopped working after choosing 65001, and

I don't know what else gets broken, because Microsoft C library

does not work with encoding that requires more than two bytes per
It is not difficult to change the current encoding in any Windows

application, the real issue is that neither Microsoft C library nor

Cygwin library does not work correctly with UTF-8. There is a patch

for Cygwin though...
Dmitry

-

Indeed. On Windows, you should avoid using UTF-8 and instead use UTF-16

everywhere. That usually works better, and if you run on an NT-based

system it will convert all the data to WinAPI to UTF-16 anyway.
--

\\// Peter - http://www.softwolves.pp.se/

-

Yes, the default code page for the command prompt uses so-called OEM

encoding, and GUI programs uses another one, which MS calls as "ANSI"

encoding. However, if you use Cygwin, then you have ANSI encoding in

the command prompt. So, in the same command prompt window, you can have

Cygwin programs using one encoding and other window console programs
Some people tried to set the current code page to 65001, which is

the Microsoft code page for UTF-8. However, it seems that does not

work very well.
http://support.microsoft.com/kb/175392

http://blogs.msdn.com/michkap/archive/2006/03/13/550191.aspx
It seems to me that Win32 API functions work correctly with

UTF-8 (after all, they are just wrappers over UTF-16 functions),

but Microsoft's C library cannot handle UTF-8 (or any other
There is a patch for Cygwin that adds UTF-8 support for it, however,

Cygwin maintainers do not like it, so it is not integrated. I think

Cygwin 1.7 will support UTF-8, but I have no idea how soon it will be

released.
I don't know much about mingw, but if I am not mistaken, mingw relies

on Microsoft's C library, so I suppose it uses an "OEM" code page for

console programs by default.
Dmitry

-

That's the easist thing to do if you want to assure that things will

mostly work across multiple different OS's, with different levels of

sanity.  You might also want to include that it's a bad idea to create

two filenames that are identical on case-insensitive filesystems,

i.e., "makefile" and "Makefile", or "foo.H" and "foo.h" which even

though it works Just Fine on Linux, will likely cause problems on

Windows and MacOS filesystems, and other systems that are insane with

respect to case insensitivity.
							- Ted

-

Ahhhh ... now I understand.
Regards,

Mark
-

Hi,
+1.
And I would suggest the use of RFC 3454 as the guidelines for UTF-8

normalization.
Best regards,

--

Pedro Melo

Blog: http://www.simplicidade.org/notes/

XMPP ID: melo@simplicidade.org

Use XMPP!
-

Hi,
-1.
It's just too arrogant to force your particular preferences down the

throat of every git user.
Ciao,

Dscho

-

It's not arrogant to make a suggestion. Where is your alternative solution?
However, what about storing an additional information like the file

system encoding (for every file)? This would result in the same

behaviour (and speed) as today as long as the file system encoding is

the same. Conversion will only be done when the targets file system

encoding is different.
BTW: This reminds me of the code page switching stuff back in the times

of MS-DOS 4/5. This really wasn't funny.
Regards,

Mark

-

Hi,
Do you agree that you need to store or at least calculate a

normalized version of each filename to see if you are already

tracking the file, to take in account all the the filesystems out

there who are not case-preserving, case-sensitive?
If so, do you think those rules should be an option? Or a preference?
Should I specify in my config file that I want my filenames to be

normalized?
Ignoring encoding, and case-sensitive issues in the git index creates

problems for those people who want/need to use non-ascii chars in

their filenames, and have some change of being able to collaborate

with other users on different operating systems.
Best regards,

--

Pedro Melo

Blog: http://www.simplicidade.org/notes/

XMPP ID: melo@simplicidade.org

Use XMPP!
-

The problem is that there is no way to know what the "target system

encoding" is.
And it wouldn't actually solve the bigger problem on OS X anyway: as long

as you are case-insensitive, you'll have all the same problems (ie the

insane OS X filesystem presumably thinks that "MÄRCHEN" and "Märchen" are

also identical, because they are "equivalent" names).
			Linus

-

Hi,
Correct. Storing or using a normalized version of the filename is

only part of the problem.
The full problem is:
User A <-> filesystem A <-#-> git < ...... > git <-#-> filesystem B <-

 > user B.
You have to encode/decode/normalize on all the <-#-> and there is no

magic bullet. Each user would have to tell git "Hey I'm using utf-8"

or "Hey, I'm a masochist using HFS+".
But I think its important for git to store the filenames in something

that at least permits this kind of scenario.
All encoding/decoding/normalization is of course optional, and for  
Correct. HFS+ has bigger problems. I'm not sure if this is enough to

solve it.
But it would solve two linux users using different encodings.
And given that the filtering layers are optional, you have to

configure them, it wont bite nobody.
Best regards,

--

Pedro Melo

Blog: http://www.simplicidade.org/notes/

XMPP ID: melo@simplicidade.org

Use XMPP!
-

No, the encoding was the same -- UTF-8. MacOSX converts one sequence of

Unicode characters to *another* sequence, which are canonical equivalent,

but being canonical equivalent does not mean they are the same characters.

In the same way, as being compatible equivalent does not mean being the

same. As well as, being case-insensitive equivalent does not mean being

the same... Do you remember DOS? It stored all filenames in upper-case,

so they original and stored names are case-insensitive equivalent, but

they are not the same!
Dmitry

-

Those of us who grew up on a case-insensitive filesystem don't find

there to be any problem with it. I can count on one hand the number of

times I've run into a problem caused by a case-insensitive filesystem.

That number is 1. And that 1 time is when git screwed up trying to  
That's only true if you don't know what type of filesystem you're on.

And, in the vast majority of cases (in fact, a content tracker is the

only exception I can think of), it doesn't matter. If the user said

'xyz' and you can stat() it, great, that's what the user wanted! Just

because it's really called 'Xyz' on the filesystem doesn't make any  
But git is a content tracker, so even if it's really a different

hardlink that shouldn't matter, it's still referencing the same

content. Go ahead and track whatever name the user specified

originally, as long as it maps to a file on disk with the expected

content you're set. If the file is really called 'foo' and I told git

to track 'Foo', I'm perfectly happy with it continuing to think 'foo'  
I don't see that as being a problem. Think of it, if you will, as if

every single file simply had an implicit hardlink for every possible

case or normalization variant. The whole point of the filename is that

it is meta-information, used as an identifier and not as actual

content, and thus it is perfectly fine for it to be a real string,

subject to interpretation, rather than treated as a sacred binary blob

like content is. The whole purpose of the name is to identify the

inode in question, and case and normalization aren't particularly  
Again, as someone who grew up in a case-insensitive world, there's no

problems here. I wish I could tell you that it causes problems, I wish

I could agree with you, but I can't.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

I guess you haven't used unix tools much. The ever-popular HEAD perl

utility (which does an HTTP HEAD against a URL), when installed,

silently overwrites the head shell utility, which is used for all

sorts of things, some even in startup scripts. Ooops! I've been hit by

this more than once - and if you google for it, it hurt a lot of
Hmmm. Many important tools - that I wouldn't want to ever fail! - have

similar needs to git. Backup/restore and file replication tools for
Ok - but how do you track the directory then (in git's terms, the

tree). There's no way to tell what the user wants. Does the user want

a copy of the file with different capitalization, or is the OS playing
Probably because you have been surrounded by tools that have a lot of

extra code to cope with the case insensitive way of life, and learned

to not do things that are completely valid, just to avoid trouble.

Which is ok, but I don't think it makes the OS design decision

defensible.
cheers,
m

-

I can imagine. However, I've never been hit by such a situation. This =20=
doesn't mean a case-insensitive filesystem is a problem per se, it =20

means interactions between a case-insensitive and a case-sensitive =20

filesystem can be a problem. That doesn't mean either way is "correct" =20=
it just means both don't work well together.
I like ice cream, and I like steak, but I sure don't think a mixture =20
Both of which would be replicating the directory contents, not a =20

listing of files specified by the user. If, as a user, I were to say =20

"please replicate file FOO" and the file was really called "foo", I =20

wouldn't be in the least surprised to see the tool take me at my word =20=
and produce a file called "FOO" with the contents of "foo". But in =20

general, things like this operate on the filesystem, not on the user =20
If I say "track FOO", I probably mean it. So go ahead and track "FOO", =20=
even if you end up tracking the contents of file "foo". I certainly =20
Sure I do. I find it  very convenient, for example, to say "cd =20

documents/school" when I really want to go to "Documents/School". =20

Similarly, if I'm trying to reference gitweb/tests/M=C3=A4rchen, I'm =

quite =20

happy to not have to figure out what normalization the filename is =20

using and attempt to replicate that (especially as I have no idea =20

which normalization my input mechanism uses - unlike Linus, I don't =20

have a key dedicated to =C3=A4, and even if I did I wouldn't necessarily =

=20

expect it to use precomposed vs decomposed). I can't think of a single =20=
reason why I'd want to be able to have 2 different files named =20

"M=C3=A4rchen" on my disk. On the other hand, treating unicode =20

normalization as significant can pose security risks - how am I to =20

know that the file that is named "foo.txt" is really the same file =20

"foo.txt" that I last saw? Someone I know on IRC sent me this =20

image[1], which shows 6 files all apparently named "foo.txt" on a disk =20=
...
[ message continues ]

For those on Mac OS X: it is possible to create a case-sensitive HFS+

partition and

use it with git. You even can just create a disk image and mount it.

However,

I wouldn't quite try to use it as startup filesystem...
   -Geert
PS. I'm working on a proposal/patch for addressing the UFS/case

sensitivity issues.

     Will try to mail something later this week.

-

I was going to post this earlier, but wanted to search the archives

first. Here are the commands assuming you don't want to or can't

partition a drive and format as ufs (I don't care for HFS+ much). I

can't believe I didn't find the command in the git list archives, so

voilà:
$ hdiutil create -size 300m -fs UFS foo.dmg

...............................................................................

created: /Users/mitch/foo.dmg

$ hdiutil attach foo.dmg

/dev/disk2          	GUID_partition_scheme

/dev/disk2s1        	Apple_UFS                      	/Volumes/untitled

$ cd /Volumes/untitled && git clone git://git.kernel.org/pub/scm/git/

git.git

... snipped ...

$ cd git && git status

# On branch master

nothing to commit (working directory clean)
After git clone in HFS+ land...

$ git status

# On branch master

# Untracked files:

#   (use "git add <file>..." to include in what will be committed)

#

#	gitweb/test/Märchen

nothing added to commit but untracked files present (use "git add" to

track)
Should I just add this to the wiki? Then we can all go back to

ignoring the insane filesystems.
Mitch
-

While it's a nice workaround, it really is just that (a workaround)

because performance will be suboptimal in a repository running on a

disk image (and many of switched to Git because of its speed).
Cheers,

Wincent
-

..........................................................................=
Not only is it suboptimal, it's also not acceptable, plain and simple. =20=
If an individual wants to do that, sure, but it's simply not an =20

appropriate solution in general for this problem. I certainly don't =20

want to have to attach a disk image every time I want access to =20

anything I keep in a git repo, nor do I want to be restricted to =20

keeping everything within a certain filesystem on disk. Additionally, =20=
while I'm not certain it's impossible, it's certainly very difficult =20

to attach a disk image without anybody logged into the system at the =20

GUI, as diskarbitrationd won't be running.
-Kevin Ballard
--=20

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

Hi,
If it's not acceptable, do something about it (and I don't mean writing 50

emails). If you don't want to do something about it, I have to assume that

you accept it as-is.
Ciao,

Dscho
-

I never said I don't want to do anything about it. However, I do

believe that it will take a significant investment of time and energy

to learn all the gooey details of how git handles filenames and how

the index works and all that jazz, which is knowledge that other

people already have. I believe that, for me to solve this problem

independently, it may require so much time that it never gets done

(after all, I am fairly busy). However, if other people who already

have this knowledge are willing to help, that would make this task far

easier, especially given that if nobody else even acknowledges that

this is a problem I don't have much hope of getting a patch accepted.
So again, I'm certainly going to try, but working by myself it simply

may never get done.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

(This is only for those that think the problem should be solved somehow. The

rest can move on - nothing to see here)
You may look at http://rosenberg.homelinux.net/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=log...

for inspiration. It's pretty obsolete by now and only a "proof of concept", i.e.

it can be done, not that it necessarily should be done exactly this way.
Basically it intercepts the user's access to git, i.e. certain commands

and how files are named (since those names represent a user interface). Then

it assumes the internal encoding is UTF-8 (or garbage) converting to and

from the user's local encoding. The heuristics is based on the assumption that

a string (even random onesthat looks like UTF-8, with a very high probablity

actually is UTF-8 encoded.
The test cases might be usable almost as is.
-- robin

-

This is starting to stray far afield, but the first thing I did when I

got a Macbook was to reinstall it with case-sensitive HFS as the boot

file system.  Works fine, including with git.  The only problem I have

had is that FileVault does not work.  There are rumored to be some

third-part apps that do not work but I do not use that many of those

anyway.
andrew

-

The main problem with this approach is you know for certain that using

HFSX as the boot partition is barely tested by Apple, and certainly

untested by third-party apps. This means the potential for breakage is

extremely high.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

No, actually, HFSX boot partitions are fairly well tested by Apple and

most 3rd-party programs.  I had one for a while and the only problems

I encountered were with programs ported from Windows without Mac

versions, such as "Microsoft Office for Mac" and "World of Warcraft".

"Quake 4" has a few quirks which are easily worked around.
Cheers,

Kyle Moffett
-

Perhaps the big name companies might do some testing on HFSX, but I

can guarantee most third-party programs will not be tested under HFSX.
Also, World of Warcraft isn't a ported program. It was developed for

the Mac concurrently with the Windows version. Same with MS Office -

it's an entirely different team (the Mac BU) developing MS Office for

Mac independently of the Windows version, not a porting job. However,

if you're saying these two big-name programs had problems, I wouldn't

be surprised to see many more problems on various other third-party

apps from smaller companies.
In any case, "just use HFSX" is still not an appropriate solution to

the problem, especially since that will only take care of case

sensitivity and not the utf-8 stuff.
-Kevin Ballard
--

Kevin Ballard

http://kevin.sb.org

kevin@sb.org

http://www.tildesoft.com

THAT IS NOT TRUE!
How the hell does the computer know what the string means?
Hint: it does not.
The fact is, the user may use a non-normalized string on purpose. It's not

your place to say that the user is wrong. Your "undestanding" is simply

wrong. Two strings are *different* if they are [un]normalized differently.
Really.
The exact same way the word Polish and polish are different, just because 
You do not understand.
In *order* to do case-insensitivity, you generally need to normalize (and

do other things too - normalization is just *one* of the things you need

to do).
So if you are a case-insensitive filesystem, then normalization is sane.
You define "string" to be something totally made-up.
In your world "string" means "normalized". BUT IT'S NOT TRUE!
You define normalization to be a property of strings, without any actual

backing for why that would be.
The fact is, *looks the same* is very very different from *is the same*.
But you seem to be too stupid to undestand the differce.
		Linus

-

You're right. The normalization only really needs to happen as part of the

name comparison itself.
		Linus

-

Hi,
For the record, HFS+ is case-insensitive but case-preserving so I

believe they keep the original filename around. I don't have the spec

in front of me, but from memory I believe that this is what they do.
But I think that focusing on HFS+ is loosing sight of the real

problem. It's not about encoding at the filesystem, but encoding

inside the git structures.
Best regards,

--

Pedro Melo

Blog: http://www.simplicidade.org/notes/

XMPP ID: melo@simplicidade.org

Use XMPP!
-

Hi,
For the record, that's only the default setting.  AFAIK you can configure

it to care about case, too.
Also for the record, the whole thread was about HFS+ _not_ keeping the 
So far I have not seen anyone talking _seriously_ about this issue.  Only

a few shouts "you should support", and a few shouts back "I don't care

about insane filesystems".
Therefore, I fully agree with you that we're losing sight of the real

problem.
Ciao,

Dscho
-

Actually, there is no good reason for non-normalized forms (deficient

software not able to deal with some of the normalized forms is not a

good reason: such software should be fixed).
It is just that the file system is a rather quirky place for enforcing

the normalization.  One should not be able to get unnormalized forms
No.  Input methods are not the same as their resulting string.  I can

even produce some ASCII characters on my keyboard in more than one way
Yup.  But that does not mean that normalization is a bad idea.  It is

just that the filesystem is not the right place for it.
--

David Kastrup, Kriemhildstr. 15, 44793 Bochum

-

I'd actually agree, and it then boils down to the second sane choice I

gave earlier:
 - don't accept data you don't like
if you don't like non-normalized names, don't create them. That's fine.
Oh, absolutely. You can - and often should - normalize in the application

(or have libraries to do it for you). 
Not silently and behind peoples backs.
		Linus

-

"FUD" is a bit strong, don't you think?  HFS+ is the way it is and it

would be nice if Git could deal with it.
The problem is that HFS+ normalizes filenames to avoid multiple files

that appear to have the same name (eg "M<A WITH UMLAUT>rchen" vs

"Ma<UMLAUT MODIFIER>rchen", in gitweb/test).  This is sort of like

case sensitivity, but filenames are normalized when a file is

_created_.  Git, not unreasonably, expects a file to keep the name it

was created with.
As far as I can tell, as long as you add all your internationally

becharactered files to git from an HFS+ file system using a gui or

command-line completion, you'll be okay; trouble starts when you check

in a file with the composed form of a character, by typing the name on

the command line (I'm not sure about this one) or committing on

another OS.  Git will store the filename in composed form, but the

Mac's filesystem will decompose the filename when you check the file

out.
The result looks like this:
vredefort:[git]% git status

# On branch master

# Untracked files:

#   (use "git add <file>..." to include in what will be committed)

#

#	gitweb/test/Märchen

nothing added to commit but untracked files present (use "git add" to

track)
(this is directly after checking out git.git @ v1.5.4-rc3)
There are two things to note here.  One is that Git thinks that there

is a new file called "gitweb/test/Märchen" (decomposed) when it's

"really" just the same "gitweb/test/Märchen" (precomposed) that's in

the repository.  The other is that git _thinks_ that the "gitweb/test/

Märchen" (precomposed) it's expecting is still there, because the

filesystem, when asked for "gitweb/test/Märchen" in any form will

return the file "gitweb/test/Märchen" (decomposed).
Trying to check out the "next" branch at this point is a pain since

next's "Märchen" would overwrite the untracked "Märchen".
I can't provide links to any previous discussions about this, but

here's Ap...
[ message continues ]

FWIW, here's Sun's take on the issue of filesystems and i18n:
http://developers.sun.com/global/products_platforms/solaris/reference/pr...
j.

-

Pretty sane, from a quick read-through, although most of it seems to not

be about general issue, as about "let's emulate others correctly on their

filesystems" (ie the rules are different for NTFS and HFS+, little enough

discussion about "native" preferred logic).
However, while they don't consider normalization on file creates to be the

"preferred solution", they *do* consider filename comparison with

canonical equivalence to be that. Which means that you can get the same

odd problems:
	fd = open(filename, O_CREAT);

	+

	readdir()
can actually return a *different* filename than the one we just created,

if it already existed in the directory under the different normalization.
So it's basically "normalization-preserving, but normalization-ignoring"

(the same way many filesystems are case-preserving, but case-ignoring). I

don't much like it either, but as with case, the "preserving" behaviour is

probably the nicer one.
I'd guess the problems are harder to trigger in practice, but you can

still get some pretty hairy cases. It's just painful when readdir() and

your own file creation doesn't have any obvious 1:1 relationship.
			Linus

-

So here's what I can see as being useful additions to git:
* Allowing a repo to be *optionally* configured to disallow two files

in a directory that can cause aliasing problems, with options for

unicode normalization aliasing and/or case-insensitivity aliasing. Can

this already be done via hooks and someone just needs to write the

appropriate hooks?
* Having git warn during checkout if there are files which alias in

the working copy filesystem. I guess it might be interesting if there

were a mechanism in this situation for telling git which of the

aliases you want checked out, though that doesn't seem like a very

good feature.
Thoughts (besides "patches welcomed")?
j.

-

I think we already discussed a plan to store normalization

mapping in the index extension section and use it to avoid

getting confused by readdir(3) that lies to us.  Is there any

more thing that need to be discussed?
I would presume that we would still add _new_ paths using the

pathname we receive from the user (there is no need for us to be

similarly insane as broken "normalizing" filesystems), but when

deciding if a path is new or we already have it in the index

would be done by seeing if an entry already exists in the index

whose "normalized" form is the same as the "normalized" form of

the given path --- that way we would not add two paths to the

index that would "normalize" to the same string.

-

And what do we do when asked to check out a tree which has two

different files in it whose normalized forms are the same (ie. a clone

of a repo created on a non-HFS+ filesystem)?
We either have to fail catastrophically, preventing the user from

working with that tree on HFS+, or arbitrarily pick one of the files

as the "winner" which gets written out into the work tree. None of the

options is particularly attractive, although luckily this exact

situation is unlikely to come up in practice.
Cheers,

Wincent
-

Hi,
[Jay, don't cull Cc: lists on vger.kernel.org.  I consider it rude.]
On Thu, 17 Jan 2008, Wincent Colaiuta wrote:
> El 17/1/2008, a las 6:15, Junio C Hamano escribi

Hi,
I searched the archives for the posts about normalization and I could

not find them, sorry.
Is stringprep (RFC 3454) being proposed as an optional normalization

step before lookups in the index?
Best regards,

--

Pedro Melo

Blog: http://www.simplicidade.org/notes/

XMPP ID: melo@simplicidade.org

Use XMPP!
-

Hi,
> > > El 17/1/2008, a las 6:15, Junio C Hamano escribi

If this is really just a platform-specific hack, can we use platform-

specific code to do the normalization?
On Mac OS X we have (unfortunately only 10.4 and up):
CFStringCreateWithFileSystemRepresentation()

CFStringGetFileSystemRepresentation()

CFStringGetMaximumSizeOfFileSystemRepresentation()
If we were to use those you'd at least know that you're getting the  
http://developer.apple.com/qa/qa2001/qa1173.html
Cheers,

Wincent
-

Hi,
That remark about the version raises my eyebrows.  Where I live, 10.2.8 is

_still_ quite common.
Ciao,

Dscho
-

There may be alternatives that I don't know about.
All the way back to 10.0 you have -[NSString

fileSystemRepresentation], which does the same thing but that's

Objective-C. I wouldn't be surprised if that's just a wrapper for the

CF functions; that's often the way it is on Mac OS X. And often, the

CF functions *are* present on older systems, but they're just not

declared in public headers. I wouldn't actually recommend using a

private SPI, but they are often there.
Cheers,

Wincent
-

Such a special mode would be mostly useless in most contexts, where

Git is used to track source code. It would enable you to check out the

tree for inspection, but you probably couldn't build anything from it

seeing as at least one of the filenames specified in your Makefile

wouldn't be present in the work tree.
As such, in that kind of situation I'd rather see a big red warning

printed out that the checkout failed because a particular file

couldn't be written out, and perhaps an instruction to the user that

they can use "git show" if they want to see the blob/s which wasn't/

weren't written.
Cheers,

Wincent
-