A thread on the lkml started innocently enough about proper spacing in source code, then grew and grew into a somewhat philosophical debate about evolution and code design. The subject of the thread was "Coding style - a non-issue", the general consesus that 'scripts/Lindent' solved the problem, a one line bourne shell script that executes indent with the "proper" options, as defined in the Documentation/CodingStyle document. The script:

indent -kr -i8 -ts8 -sob -l80 -ss -bs -psl "$@"

However, before long a debate was sparked, leading to some interesting comments by Linus and some others, perhaps best summarized by Alan's comment:

"Engineering does not require science. Science helps a lot but people built perfectly good brick walls long before they knew why cement works."

What follows is a reprint of many of the more insteresting posts by some well known Linux kernel hackers:

---

Peter's posting was the beginning of this thread. He was replying to comments made earlier in another thread, regarding an attempt to cleanup some white space issues:

From: "Peter Waltenberg"
To: linux-kernel-mailing-list
Subject: Coding style - a non-issue
Date: Thu, 29 Nov 2001 09:29:26 +1000

The problem was solved years ago.

"man indent"

Someone who cares, come up with an indentrc for the kernel
code, and get it into Documentation/CodingStyle If the
maintainers run all new code through indent with that indentrc
before checkin, the problem goes away. The only one who'll
incur any pain then is a code submitter who didn't follow the
rules. (Exactly the person we want to be in pain ;)).

Then we can all get on with doing useful things.

Cheers
Peter

---

Several people replied:

From: Robert Love
Subject: Re: Coding style - a non-issue
Date: 	28 Nov 2001 18:48:59 -0500

See scripts/lindent in your source tree ... does just this.

	Robert Love




From: Russell King
Subject: Re: Coding style - a non-issue
Date: 	Wed, 28 Nov 2001 23:40:29 +0000

See: linux/scripts/Lindent

--
Russell King 




From: Alan Cox
Subject: Re: Coding style - a non-issue
Date: 	Wed, 28 Nov 2001 23:48:07 +0000 (GMT)

The problem was solved years ago.

scripts/Lindent

---

Somehow (perhaps through the process of evolution) this led into an interesting string of comments by Linux creator, Linus Torvalds, the first in response to Rik van Riel:

From: Linus Torvalds
Subject: Re: Coding style - a non-issue
Date: 	Fri, 30 Nov 2001 16:50:34 -0800 (PST)

On Fri, 30 Nov 2001, Rik van Riel wrote:
>
> I'm very interested too, though I'll have to agree with Larry
> that Linux really isn't going anywhere in particular and seems
> to be making progress through sheer luck.

Hey, that's not a bug, that's a FEATURE!

You know what the most complex piece of engineering known to man in the
whole solar system is?

Guess what - it's not Linux, it's not Solaris, and it's not your car.

It's you. And me.

And think about how you and me actually came about - not through any
complex design.

Right. "sheer luck".

Well, sheer luck, AND:
 - free availability and _crosspollination_ through sharing of "source
   code", although biologists call it DNA.
 - a rather unforgiving user environment, that happily replaces bad
   versions of us with better working versions and thus culls the herd
   (biologists often call this "survival of the fittest")
 - massive undirected parallel development ("trial and error")

I'm deadly serious: we humans have _never_ been able to replicate
something more complicated than what we ourselves are, yet natural
selection did it without even thinking.

Don't underestimate the power of survival of the fittest.

And don't EVER make the mistake that you can design something better than
what you get from ruthless massively parallel trial-and-error with a
feedback cycle. That's giving your intelligence _much_ too much credit.

Quite frankly, Sun is doomed. And it has nothing to do with their
engineering practices or their coding style.

		Linus

---

From: Linus Torvalds
Subject: Re: Coding style - a non-issue
Date: 	Fri, 30 Nov 2001 18:57:44 -0800 (PST)
Sender: linux-kernel-owner@vger.kernel.org

On Fri, 30 Nov 2001, Tim Hockin wrote:
>
> > I'm deadly serious: we humans have _never_ been able to replicate
> > something more complicated than what we ourselves are, yet natural
> > selection did it without even thinking.
>
> a very interesting argument, but not very pertinent - we don't have 10's of
> thousands of year or even really 10's of years.  We have to use intellect
> to root out the obviously bad ideas, and even more importantly the
> bad-but-not-obviously-bad ideas.

Directed evolution - ie evolution that has more specific goals, and faster
penalties for perceived failure, works on the scale of tens or hundreds of
years, not tens of thousands. Look at dog breeding, but look even more at
livestock breeding, where just a few decades have made a big difference.

The belief that evolution is necessarily slow is totally unfounded.

HOWEVER, the belief that _too_ much direction is bad is certainly not
unfounded: it tends to show up major design problems that do not show up
with less control. Again, see overly aggressive breeding of some dogs
causing bad characteristics, and especially the poultry industry.

And you have to realize that the above is with entities that are much more
complex than your random software project, and where historically you have
not been able to actually influence anything but selection itself.

Being able to influence not just selection, but actually influencing the
_mutations_ that happen directly obviously cuts down the time by another
large piece.

In short, your comment about "not pertinent" only shows that you are
either not very well informed about biological changes, or, more likely,
it's just a gut reaction without actually _thinking_ about it.

Biological evolution is alive and well, and does not take millions of
years to make changes. In fact, most paleolontologists consider some of
the changes due to natural disasters to have happened susprisingly fast,
even in the _absense_ of "intelligent direction".

Of course, at the same time evolution _does_ heavily tend to favour
"stable" life-forms (sharks and many amphibians have been around for
millions of years). That's not because evolution is slow, but simply
because good lifeforms work well in their environments, and if the
environment doesn't change radically they have very few pressures to
change.

There is no inherent "goodness" in change. In fact, there are a lot of
reasons _against_ change, something we often forget in technology. The
fact that evolution is slow when there is no big reason to evolve is a
_goodness_, not a strike against it.

> > Quite frankly, Sun is doomed. And it has nothing to do with their
> > engineering practices or their coding style.
>
> I'd love to hear your thoughts on why.

You heard them above. Sun is basically inbreeding. That tends to be good
to bring out specific characteristics of a breed, and tends to be good for
_specialization_. But it's horrible for actual survival, and generates a
very one-sided system that does not adapt well to change.

Microsoft, for all the arguments against them, is better off simply
because of the size of its population - they have a much wider consumer
base, which in turn has caused them largely to avoid specialization. As a
result, Microsoft has a much wider appeal - and suddenly most of the
niches that Sun used to have are all gone, and its fighting for its life
in many of its remaining ones.

Why do you think Linux ends up being the most widely deployed Unix? It's
avoided niches, it's avoided inbreeding, and not being too directed means
that it doesn't get the problems you see with unbalanced systems.

Face it, being one-sided is a BAD THING. Unix was dying because it was
becoming much too one-sided.

Try to prove me wrong.

		Linus

---

From: Linus Torvalds
Subject: Re: Coding style - a non-issue
Date: 	Fri, 30 Nov 2001 19:15:55 -0800 (PST)

On Fri, 30 Nov 2001, Victor Yodaiken wrote:
>
> > And don't EVER make the mistake that you can design something better than
> > what you get from ruthless massively parallel trial-and-error with a
> > feedback cycle. That's giving your intelligence _much_ too much credit.
>
> Linux is what it is because of design, not accident. And you know
> that better than anyone.

Let's just be honest, and admit that it wasn't designed.

Sure, there's design too - the design of UNIX made a scaffolding for the
system, and more importantly it made it easier for people to communicate
because people had a mental _model_ for what the system was like, which
means that it's much easier to discuss changes.

But that's like saying that you know that you're going to build a car with
four wheels and headlights - it's true, but the real bitch is in the
details.

And I know better than most that what I envisioned 10 years ago has
_nothing_ in common with what Linux is today. There was certainly no
premeditated design there.

And I will claim that nobody else "designed" Linux any more than I did,
and I doubt I'll have many people disagreeing. It grew. It grew with a lot
of mutations - and because the mutations were less than random, they were
faster and more directed than alpha-particles in DNA.

> The question is whether Linux can still be designed at
> current scale.

Trust me, it never was.

And I will go further and claim that _no_ major software project that has
been successful in a general marketplace (as opposed to niches) has ever
gone through those nice lifecycles they tell you about in CompSci classes.
Have you _ever_ heard of a project that actually started off with trying
to figure out what it should do, a rigorous design phase, and a
implementation phase?

Dream on.

Software evolves. It isn't designed. The only question is how strictly you
_control_ the evolution, and how open you are to external sources of
mutations.

And too much control of the evolution will kill you. Inevitably, and
without fail. Always. In biology, and in software.

Amen.

		Linus

---

From: Linus Torvalds
Subject: Re: Coding style - a non-issue
Date: 	Fri, 30 Nov 2001 19:34:30 -0800 (PST)

On Fri, 30 Nov 2001, Larry McVoy wrote:
>
> I can't believe the crap you are spewing on this one and I don't think you
> do either.  If you do, you need a break.  I'm all for letting people explore,
> let software evolve, that's all good.  But somebody needs to keep an eye on
> it.

Like somebody had to keep an eye on our evolution so that you had a chance
to be around?

Who's naive?

> If that's not true, Linus, then bow out.   You aren't needed and *you*
> just proved it.

Oh, absolutely.

I wish more people realized it. Some people realize it only when they get
really pissed off at me and say "Go screw yourself, I can do this on my
own". And you know what? They are right too, even if they come to that
conclusion for what I consider the wrong reasons.

The reason I'm doing Linux is not because I think I'm "needed". It's
because I enjoy it, and because I happen to believe that I'm better than
most at it. Not necessarily better than everybody else around there, but
good enough, and with the social ties to make me unbeatable right now.

But "indispensable"? Grow up, Larry. You give me too much credit.

And why should I bow out just because I'm not indispenable? Are you
indispensable for the continued well-being of humanity? I believe not,
although you are of course free to disagree. Should you thus "bow out" of
your life just because you're strictly speaking not really needed?

Do I direct some stuff? Yes. But, quite frankly, so do many others. Alan
Cox, Al Viro, David Miller, even you. And a lot of companies, which are
part of the evolution whether they realize it or not. And all the users,
who end up being part of the "fitness testing".

And yes, I actually do believe in what I'm saying.

		Linus

---

From: Linus Torvalds
Subject: Re: Coding style - a non-issue
Date: 	Fri, 30 Nov 2001 21:15:50 -0800 (PST)

On Fri, 30 Nov 2001, Victor Yodaiken wrote:
>
> Ok. There was no design, just "less than random mutations".
> Deep.

I'm not claiming to be deep, I'm claiming to do it for fun.

I _am_ claiming that the people who think you "design" software are
seriously simplifying the issue, and don't actually realize how they
themselves work.

> There was a overall architecture, from Dennis and Ken.

Ask them. I'll bet you five bucks they'll agree with me, not with you.
I've talked to both, but not really about this particular issue, so I
might lose, but I think I've got the much better odds.

If you want to see a system that was more thoroughly _designed_, you
should probably point not to Dennis and Ken, but to systems like L4 and
Plan-9, and people like Jochen Liedtk and Rob Pike.

And notice how they aren't all that popular or well known? "Design" is
like a religion - too much of it makes you inflexibly and unpopular.

The very architecture of UNIX has very much been an evolution. Sure, there
are some basic ideas, but basic ideas do not make a system.

When they say that the devil is in the details, they are trying to tell
you that the details MATTER. In fact, the details matter quite a lot more
than the design ever does.

> Here's a characteristic good Linux design method ,( or call it "less than random
> mutation method" if that makes you feel happy): read the literature,
> think hard, try something, implement

Hah.

I don't think I've seen very many examples of that particular design
methodology.

It's impressive that you think this is how stuff gets done, but from
personal experience I would say that it's certainly not true to any
noticeable degree. The impressive part is that Linux development could
_look_ to anybody like it is that organized.

Yes, people read literature too, but that tends to be quite spotty. t's
done mainly for details like TCP congestion control timeouts etc - they
are _important_ details, but at the same time we're talking about a few
hundred lines out of 20 _million_.

And no, I'm no tclaiming that the rest is "random". But I _am_ claiming
that there is no common goal, and that most development ends up being done
for fairly random reasons - one persons particular interest or similar.

It's "directed mutation" on a microscopic level, but there is very little
macroscopic direction. There are lots of individuals with some generic
feeling about where they want to take the system (and I'm obviously one of
them), but in the end we're all a bunch of people with not very good
vision.

And that is GOOD.

A strong vision and a sure hand sound like good things on paper. It's just
that I have never _ever_ met a technical person (including me) whom I
would trust to know what is really the right thing to do in the long run.

Too strong a strong vision can kill you - you'll walk right over the edge,
firm in the knowledge of the path in front of you.

I'd much rather have "brownian motion", where a lot of microscopic
directed improvements end up pushing the system slowly in a direction that
none of the individual developers really had the vision to see on their
own.

And I'm a firm believer that in order for this to work _well_, you have to
have a development group that is fairly strange and random.

To get back to the original claim - where Larry idolizes the Sun
engineering team for their singlemindedness and strict control - and the
claim that Linux seems ot get better "by luck": I really believe this is
important.

The problem with "singlemindedness and strict control" (or "design") is
that it sure gets you from point A to point B in a much straighter line,
and with less expenditure of energy, but how the HELL are you going to
consistently know where you actually want to end up? It's not like we know
that B is our final destination.

In fact, most developers don't know even what the right _intermediate_
destinations are, much less the final one. And having somebody who shows
you the "one true path" may be very nice for getting a project done, but I
have this strong belief that while the "one true path" sometimes ends up
being the right one (and with an intelligent leader it may _mostly_ be the
right one), every once in a while it's definitely the wrong thing to do.

And if you only walk in single file, and in the same direction, you only
need to make one mistake to die.

In contrast, if you walk in all directions at once, and kind of feel your
way around, you may not get to the point you _thought_ you wanted, but you
never make really bad mistakes, because you always ended up having to
satisfy a lot of _different_ opinions. You get a more balanced system.

This is what I meant by inbreeding, and the problem that UNIX
traditionally had with companies going for one niche.

(Linux companies also tend to aim for a niche, but they tend to aim for
_different_ niches, so the end result is the very "many different
directions" that I think is what you _want_ to have to survive).

> > And I will go further and claim that _no_ major software project that has
> > been successful in a general marketplace (as opposed to niches) has ever
> > gone through those nice lifecycles they tell you about in CompSci classes.
>
> That's classic:
> A) "trust me"
> B) now here's a monster bit of misdirection for you to choke on.
>
> Does anyone believe in those stupid software lifcycles?
> No.
> So does it follow that this has anything to do with design?
> No.

Hey, the whole _notion_ of "design" is that you know what you're going to
write, and you design for it.

Or do you have some other meaning of the word "design" that I haven't
heard about.

		Linus

---

Rik van Riel was evidently in disagreement to this theory:

From: Rik van Riel
Subject: Re: Coding style - a non-issue
Date: 	Sat, 1 Dec 2001 09:18:54 -0200 (BRST)

On Fri, 30 Nov 2001, Linus Torvalds wrote:

> And don't EVER make the mistake that you can design something better than
> what you get from ruthless massively parallel trial-and-error with a
> feedback cycle. That's giving your intelligence _much_ too much credit.

So, are we going to take out the appendix in 2.5 or will
we continue hoping our kernel doesn't catch an illness
without actually doing anything preventive ?

Biological selection does nothing except removing the weak
ones, it cannot automatically create systems which work well.

In short, I believe the biological selection is just that,
selection. The creation of stuff will need some direction.

regards,

Rik

---

Alan Cox, on the other hand, seems to agree with Linus' earlier statements:

From: Alan Cox
Subject: Re: Coding style - a non-issue
Date: 	Sat, 1 Dec 2001 08:57:17 +0000 (GMT)

> Here's a characteristic good Linux design method ,( or call it "less than random
> mutation method" if that makes you feel happy): read the literature,
> think hard, try something, implement it

That assumes computer science is a functional engineering discipline. Its
not, at best we are at the alchemy stage of progression. You put two things
together it goes bang and you try to work out why.

In many of these fields there is no formal literature. The scientific paper
system in computer science is based on publishing things people already
believe. Much of the rest of the knowledge is unwritten or locked away in
labs as a trade secret, and wil probably never be reused.

Take TCP for example. The TCP protocol is specified in a series of
documents. If you make a formally correct implementation of the base TCP RFC
you won't even make connections. Much of the flow control behaviour, the
queueing and the detail is learned only by being directly part of the
TCP implementing community. You can read  all the scientific papers you
like, it will not make you a good TCP implementor. 

Alan




From: Alan Cox
Subject: Re: Coding style - a non-issue
Date: 	Sat, 1 Dec 2001 13:38:11 +0000 (GMT)

> Recently, our correspondent from Wales wrote:
> ... the changes have been done and
> tested one at a time as they are merged. Real engineering process is the
> only way to get this sort of thing working well.

Which doesn't conflict. Engineering does not require science. Science helps
a lot but people built perfectly good brick walls long before they knew why
cement works.

> All the alchemists ever managed to create were cases of mercury
> poisoning.

and chemistry, eventually. You take it as far more demeaning than its meant.

But right now given two chunks of code, I find out what happens by putting
them together not by formal methods. In the case of alchemy v chemistry the
chemists know whether it will probably go bang before they try it (and the
chemical engineers still duck anyway)

Alan