On Jan 23, 2008, at 11:16 AM, Linus Torvalds wrote:

quoted text
> On Wed, 23 Jan 2008, Theodore Tso wrote:
>>
>> So this demonstrates that on my MacOS 10.4.11 system, on NFS, MacOS =20=

quoted text
>> is
>> doing no normalization, as it is creating two files.  On HFS+, MacOS
>> is mapping both filenames to the same decomposed name.
>
> Well, it demonstrates that (a) the OS and (b) _perl_ don't mangle
> filenames on non-HFS+ filesystems.
>
> The problem is that since most native applications *expect* that name
> mangling, they'll probably do name mangling of their own =20
> (internally) just
> to compare the names!

Well yes, any context in which a string is treated as Unicode instead =20=

of an opaque sequence of bytes will probably lead to normalization at =20=

some point (e.g. when searching text, I'm going to want M=E4rchen and =20=

M=E4rchen to be treated as the same string). The Mac OS X APIs use NFD, =20=

and everybody else uses NFC, but either way it's still normalization.

quoted text
> So I would not be surprised if the globbing libraries, for example, =20=

quoted text
> will
> do NFD-mangling in order to glob "correctly", so even programs =20
> ported from
> real Unix might end up getting pathnames subtly changed into NFD as =20=

quoted text
> part
> of some hot library-on-library action with UTF hackery inside.

Why would the globbing libraries have to do anything special to =20
understand NFD? In fact, I prefer that they don't - it's very handy to =20=

be able to type Ma* and have that match M=E4rchen, as the globbing =20
library sees Ma??rchen and is happy to match the ??rchen against *. =20
Were the filename in NFC, I couldn't do that. Similarly, Ma<tab> =20
autocompletes the name M=E4rchen for me. But the convenience is beside =20=

the point - what I'm trying to show here is that if the globbing =20
library were NFD-aware, it probably would decide Ma* shouldn't match =20
M=E4rchen, right?

I assume globbing libraries et al don't do UTF-8 hackery in Linux, =20
right? And yet using NFC-encoded filenames is fairly common? So why =20
should it be any different on OS X, especially since HFS+ isn't the =20
only option here (and thus doing NFD conversion in the library would =20
mess up other filesystems)?

In fact, probably the biggest reason the NFD-encoding was done at the =20=

HFS+ level is because they simply couldn't trust user-level libraries =20=

to always do the NFD conversion for pathnames. And I quote:

"I would prefer that case sensitivity and unicode normalization were =20
not the responsibility of the file system -- but I realize that we =20
cannot just ignore the problem and let the other layers sort it all =20
out."

quoted text
> Things like the finder etc, which must be very aware of the fact that
> filenames get corrupted, would presumably internally always convert
> everything they get into NFD in order to compare names from different
> sources. And as part of that, programs may well corrupt the name =20
> before
> they then use it to create a pathname.

I don't get why you're still calling it corruption when, on an HFS+ =20
system, NFD-encoding is correct. It would be corruption for HFS+ to =20
write anything else but NFD.

quoted text
> The fact that your perl program works under NFS, but creates NFD on =20=

quoted text
> a VFAT
> volume, does imply that they probably used at least some of the same
> routines they use in HFS+ for VFAT. Not entirely surprising: doing =20
> case
> insensitive stuff with Unicode is nasty code, so why not share it =20
> (even if
> it's then incorrect for FAT)..
>
> Piece of crap it is, though. Apple has painted themselves into a nasty
> corner there.

There's no reason to assume that OS X is actually storing the NFD on =20
the volume. In fact, it's quite explicitly not:

"As far as storing exactly what was passed in,  its not just HFS =20
that's involved her.  In Mac OS X,  SMB, MSDOS, UDF, ISO 9660 =20
(Joliet), NTFS and ZFS file systems all store in one form -- NFC.  We =20=

store in NFC since that what is expected for these files systems.  If =20=

we were to allow KFD to pass through, it would cause problems when =20
these names were accessed outside of Mac OS X.  So this is not just an =20=

HFS issue but an interchange issue for Mac OS X.  We have the legacy =20
NFD use/expectation in our applications and we chose not to ignore the =20=

problem but make a conscience effort to have the appropriate form used =20=

(NFD in Mac OS X APIs, NFC elsewhere).  Its not perfect but neither is =20=

the agnostic approach where both forms can be used and you can have =20
duplicate filenames in your file system."

-Kevin Ballard

--=20
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com