On Mon, 21 Jan 2008, Kevin Ballard wrote:
quoted text
> On Jan 21, 2008, at 9:14 AM, Peter Karlsson wrote:
> > 
> > I happen to prefer the text-as-string-of-characters (or code points,
> > since you use the other meaning of characters in your posts), since I
> > come from the text world, having worked a lot on Unicode text
> > processing.
> > 
> > You apparently prefer the text-as-sequence-of-octets, which I tend to
> > dislike because I would have thought computer engineers would have
> > evolved beyond this when we left the 1900s.
> 
> I agree. Every single problem that I can recall Linus bringing up as a
> consequence of HFS+ treating filenames as strings [..]

You say "I agree", BUT YOU DON'T EVEN SEEM TO UNDERSTAND WHAT IS GOING ON.

The fact is, text-as-string-of-codepoints (let's make the "codepoints" 
obvious, so that there is no ambiguity, but I'd also like to make it clear 
that a codepoint *is* how a Unicode character is defined, and a Unicode 
"string" is actually *defined* to be a sequence of codepoints, and totally 
independent of normalization!) is fine.

That was never the issue at all. Unicode codepoints are wonderful.

Now, git _also_ heavily depends on the actual encoding of those 
codepoints, since we create hashes etc, so in fact, as far ass git is 
concerned, names have to be in some particular encoding to be hashed, and 
UTF-8 is the only sane encoding for Unicode. People can blather about 
UCS-2 and UTF-16 and UTF-32 all they want, but the fact is, UTF-8 is 
simply technically superior in so many ways that I don't even understand 
why anybody ever uses anything else.

So I would not disagree with using UTF-8 at all.

But that is *entirely* a separate issue from "normalization". 

Kevin, you seem to think that normalization is somehow forced on you by 
the "text-as-codepoints" decision, and that is SIMPLY NOT TRUE. 
Normalization is a totally separate decision, and it's a STUPID one, 
because it breaks so many of the _nice_ properties of using UTF-8.

And THAT is where we differ. It has nothing to do with "octets". It has 
nothing to do with not liking Unicode. It has nothing to do with 
"strings". 

In short:

 - normalization is by no means required or even a good feature. It's 
   something you do when you want to know if two strings are equivalent, 
   but that doesn't actually mean that you should keep the strings 
   normalized all the time!

 - normalization has *nothing* to do with "treating text as octets". 
   That's entirely an encoding issue.

 - of *course* git has to treat things as a binary stream at some point, 
   since you need that to even compute a SHA1 in the first place, but that 
   has *nothing* to do with normalization or the lack of it.

Got it? Forced normalization is stupid, because it changes the data and 
removes information, and unless you know that change is safe, it's the 
wrong thing to do.

One reason _not_ to do normalization is that if you don't, you can still 
interact with no ambiguity with other non-Unicode locales. You can do the 
1:1 Latin1<->Unicode translation, and you *never* get into trouble. In 
cotnrast, if you normalize, it's no longer a 1:1 translation any more, and 
you can get into a situation where the translation from Latin1 to Unicode 
and back results in a *different* filename than the one you started with!

See? That's a *serious*problem*. A system that forces normalization BY 
DEFINITION cannot work with people who use a Latin1 filesystem, because it 
will corrupt the filenames!

But you are apparently too damn stupid to understand that "data 
corruption" == "bad", and too damn stupid to see that "Unicode" does not 
mean "Forced normalization".

But I'll try one more time. Let's say that I work on a project where there 
are some people who use Latin1, and some people who use UTF-8, and we use 
special characters. It should all work, as long as we use only the common 
subset, and we teach git to convert to UTF-8 as a common base. Right?

In your *idiotic* world, where you have to normalize and corrupting 
filenames is ok, that doesn't work! It works wonderfully well if you do 
the obvious 1:1 translation and you do *not* normalize, but the moment you 
start normalizing, you actually corrupt the filenames!

And yes, the character sequence 'a¨' is exactly one such sequence. It's 
perfectly representable in both Latin1 and in UTF-8: in latin1 it is a 
two-character '\x61\xa8', and when doing a Latin1->UTF-8 conversion, it 
becomes '\x61\xc2\xa8', and you can convert back and forth between those 
two forms an infinite amount of times, and you never corrupt it.

But the moment you add normalization to the mix, you start screwing up. 
Suddenly, the sequence '\x61\xa8' in Latin1 becomes (assuming NFD) 
'\xc3\xa4' in UTF-8, and when converted back to Latin1, it is now '\xe4', 
ie that filename hass been corrupted!

See? Normalization in the face of working together with others is a total 
and utter mistake, and yes, it really *does* corrupt data. It makes it 
fundamentally impossible to reliably work together with other encodings - 
even when you do converstion between the two!

[ And that's the really sad part. Non-normalized Unicode can pretty much 
  be used as a "generic encoding" for just about all locales - if you know 
  the locale you convert from and to, you can generally use UTF-8 as an 
  internal format, knowing that you can always get the same result back in 
  the original encoding. Normalization literally breaks that wonderful 
  generic capability of Unicode.

  And the fact that Unicode is such a "generic replacement" for any locale 
  is exactly what makes it so wonderful, and allows you to fairly 
  seamlessly convert piece-meal from some particular locale to Unicode: 
  even if you have some programs that still work in the original locale, 
  you know that you can convert back to it without loss of information.

  Except if you normalize. In that case, you *do* lose information, and 
  suddenly one of the best things about Unicode simply disappears.

  As a result, people who force-normalize are idiots. But they seem to 
  also be stupid enough that they don't understand that they are idiots.
  Sad. 

  It's a bit like whitespace. Whitespace "doesn't matter" in text (== is 
  equivalent), but an email client that force-normalizes whitespace in 
  text is a really *broken* email client, because it turns out that 
  sometimes even the "equivalent" forms simply do matter. Patches are 
  text, but whitespace is meaningful there. 

  Same exact deal: it's good to have the *ability* to normalize 
  whitespace (in email, we call this "text=flowed" or similar), and in 
  some ceses you might even want to make it the default action, but 
  *forcing* normalization is total idiocy and actually makes the system 
  less useful! ]

		Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html