OpenBSD creator Theo de Raadt began developing OpenBSD in October of 1995. KernelTrap first spoke with Theo back in November of 2001 [interview], around the time that OpenBSD 3.0 was released, discussing much of the early history of the project. The project has continued to offer regular releases of their "free, functional & secure" operating system every six months, with OpenBSD 3.9 made available yesterday, May 1, 2006.
In this latest interview, Theo examines the past five years of OpenBSD development. He also discusses the OpenBSD 3.9 theme song, "Blob!", detailing what blobs are, why OpenBSD avoids them, and how OpenBSD developers work to reverse engineer them. Looking to the development process, Theo talks about recent and future "mini-hackathons", small and focused OpenBSD development gatherings. Finally, Theo also discusses the OpenBSD project's funding issues, and the response to requests for funding from users of the project's OpenSSH software.
Background:
Jeremy Andrews: Your last interview with KernelTrap was back in November of 2001, shortly before OpenBSD 3.0 was released. Can you summarize some of the major improvements to OpenBSD between 3.0 and the current stable OpenBSD 3.8 release?
Theo de Raadt: That is almost 5 years, since we add "0.1" to our release number every 6 months. Our pf packet filter was born in 2001, sparc64 support showed up around 3.1, our W^X and other address space randomization changes around 3.4, amd64 support around 3.5, and after that all of our bgp/ospf routing daemons, much more recently we added many wireless drivers (staying ahead of Linux even) and now we are attempting to do more RAID management, temperature and other types of sensors... too much stuff.
Jeremy Andrews: How has the number of active OpenBSD developers changed over the years?
Theo de Raadt: The developer count increases and decreases, but mostly it hovers around 80 semi-active or active people. The hackathons and tree-unlocks (ie. when in the release cycle we branch the tree) create bursts of activity and then it seem like a lot of people. Sometimes development can seem a bit slow because of various reasons, but it always recovers.
Jeremy Andrews: What are some of the major challenges that the OpenBSD project has faced since OpenBSD 3.0?
Theo de Raadt: Well, internal to our project the biggest problem always is -- and likely will be -- that we do not have enough developers to keep up with the constant supply of new devices which vendors produce. But we try. At least, it appears our user community is fairly happy with our efforts. We try to focus our effort on the most important devices.
That brings us to the large external problem we have -- the most important devices to support sometimes are completely undocumented. It is quite a strange situation, to see the most common devices undocumented, while vendors of more fringe devices normally are very willing to provide documentation. In certain fields of operation, such as ethernet or i2c, devices which lack documentation are rare. Unfortunately a few vendors like Marvell and Broadcom are trying to lock ethernet up again. Some Linux (and recently FreeBSD too) developers are willing to sign NDAs so that a few people get the documentation, and I believe that this is the largest problem facing the kernel side of the open source community today.
Jeremy Andrews: Has the OpenBSD project succesfully convinced any vendors to release documentation that was previously refused?
Theo de Raadt: Yes, this has happened a few times. But normally this has been because the earlier refusal came from the wrong place inside the company. When we succeed later, it is because we have found a better way into the company.
Jeremy Andrews: How does OpenBSD convince vendors that they're better off releasing documentation?
Theo de Raadt: Well we tell them various angles on the truth, to make them understand. The problem is that the message you must give to a vendor depends largely on which particular person in the company you are talking to. Is he management? A decision maker? Someone who gains from making a risky move? Or someone who is afraid? Is he an engineer who needs ammunition for politics inside the company? Or have you fallen into the pit of PR despair, where the person is just supposed to shut you down? Each of these people needs to hear a different message. In the end of course, we firmly believe these sub-messages are all part of the same message -- giving us documentation allows us to support your devices so that some more consumers know that your devices are supported.
The problem is that normal business practices in the industry, these days, do not consider the end-user the consumer. Instead, the chipset vendors consider companies like Dell or IBM or HP to be their customer. You, the end user, are the product that these large VARs sell to the chipset vendors, because you become locked to the chipset vendor's product..
And I suppose it is therefore clear why the small vendors are willing to give us documentation for their chips.
Jeremy Andrews: Can you offer some specific success stories of the OpenBSD project regarding documentation?
Theo de Raadt: I think the biggest success is that most Taiwanese vendors give us documentation almost immediately. A little while ago we even had one (JMicron) contact us completely out of the blue, with full documentation and hardware.
Jeremy Andrews: Which specific vendors do you find to be the most difficult to work with?
Theo de Raadt: Well, TI has never returned a mail regarding their wireless chipset. Broadcom has never replied either, and even though driver support for their chip is now possible because of reverse engineering by some Linux folk, there is still the issue of no free re-distribution of the microcode, so this is a lot like the Intel situation. Intel talked to us, but they still refuse to make the microcode re-distributable; I really think it is such a simple thing we ask for. Mostly, it is American companies who I think feel entitled to not change their behaviour.
Jeremy Andrews: Which vendors stand out as being the most helpful to projects like OpenBSD?
Theo de Raadt: The little vendors.
Jeremy Andrews: OpenBSD 3.9 is scheduled for release on May 1, 2006. What are some of the new features and overall improvements that we can look forward to in this release?
Theo de Raadt: The big improvements are the thousands of little changes that are mostly not big :) Our project really believes in making small progressive changes instead of dangerous redesigns. This also allows us to lock our release schedule to 6 months, and I think that this in turn encourages the developers to think of how to develop carefully.
Jeremy Andrews: Each OpenBSD release includes a theme song that goes along with the release's artwork. OpenBSD 3.9's theme song is titled "Blob!". Can you explain what binary blobs are and why they're a problem?
Theo de Raadt: Vendors often try to hand off two kinds of binary code to us, which they expect we will happily incorporate into our system (and then, hopefully, we will shut up).
The first kind to mention is firmwares. Firmwares (like for instance on a Intel wireless card, or a such) are binary pieces of code that will run on the little processor that is on the wireless card. As an operating system, we need to load the code out to the card. To include a firmware in OpenBSD, we simply need a nice copyright statement from the vendor that lets us distribute the firmware. Some vendors won't even go that far, though.
The second kind of binary data vendors feed us are blobs. This is code that is expected to be linked against the operating system and run on the host processor. There are many problems with this. First off, can we trust the code to do what it should do? I don't think so. If there is a bug, can we fix it? No, as developers our hands are tied, and if our user community runs into bugs it just makes us look bad. Therefore when faced with the choice of supporting a device very poorly (as the blob would force us to) we instead choose to wait until we (or someone else) can reverse engineer it or.
Jeremy Andrews: What is it about binary firmware that you're willing to ship it, versus binary blobs? How can you trust the firmware binary to do what it should do? And what if the firmware has a bug?
Theo de Raadt: Quite honestly I prefer chips which have no firmware, and instead use correctly designed hardware logic, which our driver must then drive. Note that most ethernet chipsets do not use a processor, but many scsi chipsets do. Most IDE chipsets do not, but for wireless devices ... it is about half and half. This clearly has to do with the complexity of the data flow problem being dealt with.
But in the end, if we wish to support any such devices, we must be practical. We must accept the risk that there is a flaw in the firmware. (Is that not what many of us have been coping with for years now with Prism wireless chipsets and their firmware update tools?) But the legal climate is a real problem for us -- that is why we must get copyright permission to distribute the firmware images. Once they are distributed... at least the device works.
Of course, also note that we don't want to become Hermes (the architecture of the Lucent/Prism/Symbol chip) assembly language programmers... we have more than enough to do. Just a specific example. Please, people, don't load us up with more tasks ;)
Jeremy Andrews: Blobs seem especially common with wireless ethernet cards and graphics cards, why is that?
Theo de Raadt: Graphics cards have gotten to this point because of their complexity. But these blobs also cope with lots of bugs in the devices. These bugs are because graphics cards are devloped very quickly now, and the hope is that software would work around the hardware bugs.
I don't know why any wireless cards use blobs. In fact, very few do. They should just document their chips. There's a lot of hogwash flying around about FCC rules, but if that was a concern of theirs they should just design their chips to lock the channels in hardware. But of course, noone in Taiwan does. So did the US vendors just tie the own hands? Certainly, we don't feel like we are constricted. The minute we are able to support another wireless device, we immediately ship the driver.
Jeremy Andrews: Can you explain more about these FCC rules that wireless manufacturers refer to?
Theo de Raadt: Basically the rules say that the devices must be closed boxes; that people should not be able to tune them to transmit outside of the specified frequencies. Now the vendors have gone and created devices which can sometimes do that. But I don't see how it is our fault, as an operating system group that makes free code that interoperates with a user's device, if these vendors went and created flawed devices. Let the FCC go after the vendors who made the flawed devices. In any case, most our users and developers do not live in the country where the FCC operates.
Jeremy Andrews: How is OpenBSD able to ship without any binary blobs?
Theo de Raadt: In most cases we have reverse engineered the code. In other cases we have actually gotten Linux source code that implements the functionality. In at least one case we have even gotten a glimpse at some vendor code which turned out to be BSD licensed, and this was code found lying around on the net.
Jeremy Andrews: What blob-based wireless cards remain unsupported by OpenBSD?
Theo de Raadt: Just a few of the newest Atheros devices. The problem in the wireless domain for us is more about firmwares without distribution rights. If Intel and Broadcom would give their firmware out so that we could include it, the users would benefit. But again, these chipset vendors do not feel that there is any relationship between them and the users.
Jeremy Andrews: Why would a chipset vendor be unwilling to distribute their firmware? What do they have to gain by not distributing it?
Theo de Raadt: I just don't understand. I mean, they let Microsoft and IBM and the various VARs distribute their firmware. So why not us? Why not the Linux vendors? Oh, but a few Linux vendors will distribute the firmwares, under some rather restrictive rules.
Jeremy Andrews: Why do you suppose binary blobs have become so common?
Theo de Raadt: Well, I think they really have not become that common at all. All the time we see more vendors that are open. It is just that right now the most common devices are made by a very small set of vendors who are very strictly trying to lock down their market in this way. But it could go very backwards for them quickly, too. For instance, the (newish) Realtek Gigabit Ethernet chips are not too bad at all, and there's lots of documentation. So maybe the Taiwanese products are a little bit later to the market, but they are simpler and robust once they do make it to the market.
Jeremy Andrews: Are these documented Taiwanese products that you refer to becoming more common?
Theo de Raadt: Ralink thinks that they shipped 14 million wireless chips in 2005. That's a very significant market presence. Of course, most of those do not go into PC's these days. Again this comes down to what the VARs decide should be put into consumer products.
One must also remember that Realtek continues to sell more ethernet chips every year than everyone else. And their gigabit chip isn't as bad as their old stuff used to be. But what if Dell, HP, Asus and such continue to push only chips which lack documentation? It is not just dire for us -- it is also bad for Linux.
Jeremy Andrews: What recommendations would you offer to user's of operating systems that include binary blobs? That is, what can they do to help discourage the spread of binary blobs while still using their favorite operating system?
Theo de Raadt: I don't know what to tell such people. Their operating system supplier is deciding this for them. Even in the Linux community there are distributions that refuse to include binary blobs, or at least, don't encourage it.
That said, almost all these systems are setup so that these blobs can be loaded. Quite often they even just say it is some LKM that supports some feature. The users never ask for source.
Jeremy Andrews: What are some examples of drivers that have been reverse engineered by the OpenBSD project?
Theo de Raadt: Oh there's too many to mention. The problem is that we define reverse engineering to apply to almost any situation where we lack vendor documentation. If we have to build our driver by reading mysterious code from somewhere else, say for example a Linux driver that someone wrote based on documentation they had under a non-disclosure agreement, we would still consider it reverse engineering -- because typically such code totally blows! Typically such code is full of unexplained magic numbers, and it is a serious effort to learn enough, and then create a driver for our use.
Jeremy Andrews: Have OpenBSD developers ever been threatened by companies whose products are being reverse engineered?
Theo de Raadt: No.
Jeremy Andrews: How do you motivate your developers to reverse engineer drivers?
Theo de Raadt: Once in a while I mention a project to some person. But most of of the time I become aware of it when someone like Jonathan Gray or Reyk Floeter or Damien Bergamini says "I have started working on this"....
Jeremy Andrews: What is the state of graphics card support in OpenBSD? Are graphics card drivers being reverse engineered like ethernet drivers?
Theo de Raadt: Well, that is more a job for the X developers. Though I have helped them get some documentation for some chips as well. I don't know how things are going there. It is sad to me though that everyone has ignored the work of Loic Duflot who basically showed how horrid the X server security model is.
Jeremy Andrews: Each year OpenBSD sponsors a "hackathon", flying developers in from all over the world. How does the project pay for these events, and what is gained?
Theo de Raadt: The hotel is paid for either by the project directly out of our CD sales money. Twice, DARPA money paid for the event. Another time, NLnet gave us a donation that helped a fair bit. And finally, another time a company made a very large contribution to thank us for a feature pf had just gained (the overload keyword).
Jeremy Andrews: Why are these events so important to OpenBSD?
Theo de Raadt: I think it is clear that putting people together in a room makes a lot of things move smoother.
Jeremy Andrews: You've also made mention of "mini-hackathons". How do these work, and are any currently planned?
Theo de Raadt: A few years ago we held a mini-hackathon for pf specifically, after a cansecwest conference in Vancouver which a few developers were attending. The mini-hackathon was held in a cabin near the town of Sechelt, in the woods, up the Sunshine coast; it even took a ferry ride to get there. I think there were about 15 people, staying in giant hunting tents that our hunting developer Bob brought from Edmonton. The cabin had power, but no net! There was a DSL link just under 1k away, up a hill, but it was at the maximum distance. We cut with a machete and rolled fiber through the bramble between the two buildings, and then we had net and could do commits! We were afraid that the deer would break the fiber, but the next day we saw them avoiding it, very carefully stepping over it...
The pf hackathon was very successful. It took a few years, but last November before the OpenCon conference in Venice we held a "ports" hackathon on San Servelo island in the Venician bay. That was absolutely incredible, and very productive, with the remaining package upgrade stuff being completed at the event.
Having learned what works (and kind of what doesn't) we are now planning about 4 mini-hackathons for the coming year :)
Jeremy Andrews: What is the focus of each of these upcoming mini-hackathons?
Theo de Raadt: There will be one focused on our new ipsecctl tool, another just for OpenSSH. The first one to be held in June will focus on finishing the multipath routing support in the kernel, and I have a feeling that quite a few more routing daemon improvements will happen as well.
Jeremy Andrews: What is ipsecctl, and what is in store for it at the mini-hackathon?
Theo de Raadt: Hans-Joerg Hoexer and Mathieu Sauve-Frankel have designed and built a new configuration tool for IPSEC, that is modeled a bit on how pf and /etc/pf.conf work. It was already in 3.8, was further improved for 3.9, but there is still a lot of work to do, in the end it will make IPSEC much easier for regular mortal to use.
Jeremy Andrews: OpenSSH seems to do what it was designed for quite well. What more do you have in mind for it?
Theo de Raadt: Well, OpenSSH is very important. So the big changes happening there are to improve performance and increase the security. As we become aware of more problems in the C language, we are trying to be very agressive to make the code cleaner. Just the standard OpenBSD proactive auditing process.
Jeremy Andrews: Once OpenBSD 3.9 is out the door, what are the plans for OpenBSD 4.0?
Theo de Raadt: Well it is still early in the 4.0 cycle. There are some crazy projects being worked on, but we will have to see what is ready for release. I believe that some level of ACPI support (maybe battery support, maybe at least better interrupt routing) will make it. And who knows what else :)
Jeremy Andrews: The OpenBSD project recently had some media attention due to being low on funding. What are project funds used on, and where does the project get its funding from?
Theo de Raadt: First, I must clear up some misconceptions. We received some funding for a 2 year period when DARPA provided us with some funding -- essentially they paid the salaries of 5 people to work completely fulltime, bought about $30k in hardware, and paid for 3 hackathons). The original grant amount might have sounded positively gigantic at $2 million, but I swear I will never get over how incredibly much money a University acting as a middle man between DARPA and us can bleed the flow of financing. It was just astounding.
So, traditionally all our funding has come from user donations and users buying our CDs (our other products don't really make us much money). Obviously, that has not been a lot of money. We have however continued to make ends meet, and only cried for help when we started hitting problems.
Jeremy Andrews: I use OpenSSH every single day. I use it at home, and have used it at every computer job I've ever worked. I imagine this is true for a lot of people. What was the reaction to your recent request for donations from people and companies that benefit from OpenSSH?
Theo de Raadt: There were many reactions.
I don't understand why the Unix/Linux vendors (and Cisco) who ship our product are avoiding us. Maybe they are afraid to give to the first project, because then other projects will come asking too.
I think that contributions should have come first from the vendors, secondly from the corporate users, and thirdly from individual users. But the response has been almost entirely the opposite, with almost a 15 to 1 dollar ratio in favor of the little people. Thanks a lot, little people!
Jeremy Andrews: You refer to OpenSSH-p, the portable version of OpenSSH that runs on other operating systems besides OpenBSD. How much effort is involved in maintaining OpenSSH-p?
Theo de Raadt: A fair bit of effort, though it is mostly handled by 3 people. It is not getting easier, though, because many of the newer things we want to do (described above) will be rather agressive code re-organizations.
Jeremy Andrews: You have a reputation among non-OpenBSD projects as being difficult to get along with. Why do you suppose this is, and how does it affect you?
Theo de Raadt: I think lots of other projects have difficult people. Why is strlcpy still not in glibc? Because one person decided to be difficult. I am very easy to get along with, but I don't have time to waste being nice to people who are being stupid.
Jeremy Andrews: You have been known to occasionally make anti-Linux statements. What is it about Linux and the Linux community that draws your ire?
Theo de Raadt: I don't have a problem with Linux; I just don't use it. Nor do I think it is a newer and better or brighter or has less calories; everything we build is turds, we just move them around or shine them or have a different view on which way they should be rolled. I'm just tired of the various evangelical approach being taken by so many of the Linux users and developers. "Can't we all just get along?" If they keep throwing their ire at me, I will continue to call things as they are.
Jeremy Andrews: Why haven't we seen an OpenCC, or OpenX?
Theo de Raadt: To a certain extent we already have an OpenX, since Matthieu Herrb works in the OpenBSD X tree to develop new things. This is why our xterm has been privilege revoking for years now, and our X server is privilege separating.
I would love a new C compiler, but mostly because I feel that the gcc model is unmaintainable in the long term. What the gcc people have created is a compiler that is heavy on optimization. The result of their imperfect efforts thus is a compiler that generates incorrect code from time to time, and this affects us all. That also makes it very slow. I would love to see a new C compiler that was fully compliant, did minimal optimization, was small and fast, and high quality. But there is nothing in the open source sphere for that today. Nothing, nothing at all. We have investigated about 5 choices.
Jeremy Andrews: A high quality fully compliant C compiler that's small and fast, that sounds exactly like something we'd expect from the OpenBSD project. Is there any chance we'll hear about a C compiler mini-hackathon some day?
Theo de Raadt: I don't think we have found the right people to do this yet. Maybe we are picking up some skills with the effort we are putting into lint right now. It sure is teaching us how little people actually know about modern C.
Jeremy Andrews: You invest a tremendous amount of time and energy into creating a free, functional and secure operating system. Why is this so important to you?
Theo de Raadt: Whenever I work in a piece of code, I always get frustrated when I see very complicated API's designed by people... who are inexperienced, or so I used to think. Actually it turns out that designing correct (translation: SIMPLE) APIs is one of the most difficult problems in software development. The Unix kernel:userland API is such a great example of clever interface, and early on I became addicted to learning how to design things so simple, yet so powerful, especially compared to the Amiga, which I was experienced with beforehands. Don't get me wrong -- there are little flaws and mistakes, or at least, corner cases that one can never get just right. Over the last 20 years this has led me to where I am. In our group, I always stick my nose into any new API which someone builds, to try to see if I can help make sure it is just right.
That's the technical side. The other side is that I've got this great team who I get to play with every day. Why wouldn't that be important?
Jeremy Andrews: You live an adventurous life outside of leading the OpenBSD project, doing a lot of hiking and biking. What is it that attracts you to this lifestyle?
Theo de Raadt: Well, less biking these days. Eventually the eyes get tired of the screen, the wrists of the keyboard, the brain of the semantic arguments, and it is time to go explore some mountains!
Theo de Raadt: I think lots o
Theo de Raadt: I think lots of other projects have difficult people. Why is strlcpy still not in glibc? Because one person decided to be difficult. I am very easy to get along with, but I don't have time to waste being nice to people who are being stupid.
That's exactly why I don't waste time being nice to you, Theo.
What else should have Theo do
What else should have Theo done, exactly?
How about this...
Theos problem isn't that he doesn't suffer fools, it's that he considers anyone who doesn't agree with him to be a fool. Not wrong, not of a different opinion...a fool. He then proceeds to treat all fools with appalling contempt. What should he have done? How about behave like an adult.
exactly
I don't know this person but it fits the profile of someone we all know. The benefit of considering anyone who doesn't agree with us "stupid" is that it allows us to avoid thinking. It also allows us to avoid the kind of humility that allows us to learn something . We've all done that but for some people is a lifestyle.
--Frank VanderSloot
"That's exactly why I don't w
"That's exactly why I don't waste time being nice to you, Theo.".
That's funny, I doubt anyone in the BSD world cares.
In the long run...
> That's exactly why I don't waste time being nice to you, Theo.
I'm sure Theo cares what Mr. Anonymous thinks about him.
I have been monitoring (and participating in a small way) FOSS for 15 years now and what stands out is that people like Theo or RMS who stick to their guns and never ever waver from their convictions are the ones who are right in the long run. Yeah, maybe Theo is arrogant. RMS is annoyingly repeating the same thing every single day. But their line of uncompromising freedom is the only way to make computing last past today.
Examples in point: blobs. bitkeeper.
zealots, you mean?
... who stick to their guns and never ever waver from their convictions are the ones who are right in the long run.
That must make a lot of religious loonies all over the planet very happy. Are you writing this from a mosque or baptist church by any chance? ;)
People on the extreme side of things are never right. They are just needed to create the necessary force fields, creating boundaries the rest of the minions can play between.
The same thing could be said about Bill Gates, and he's surely not right. :)
Sorry, I know it's a rather off-topic rant, but I can't stand people who celebrate the lack of doubt or inquisitive attitude. There are enough brainless believers as is.
I'm sure Theo has his doubts from time to time. And as much as I agree with RMS -- I understand his 'prophetical extremism' is needed to front the free software movement -- I doubt if anybody would think it wise to follow him to the letter, all the way into Utopia.
Right is an extreme.
No. Right is an extreme. If you are not extreme, you are always wrong. A compromize can not be right, as it will always contain something wrong.
Say two kids that are told to share candy. One wants it all, the other wants half of it. Will then the compromize (one get 75% and the other 25%) be right, or will the idea about sharing equally be the right one?
(I am not claimning that all extremist are right. Extreme do not right or wrong make. The question is what are you extreme about.)
"And Elijah came unto all the people, and said, How long halt ye between two opinions? if the LORD be God, follow him: but if Baal, then follow him. And the people answered him not a word." The Bible
"People on the extreme side
"People on the extreme side of things are never right."
They may not be right, but more often than not, they are victorious.
History is made by extremists. Moderates fall by the wayside, if they even survive at all.
It wasn't by being reasonable or playing nice or showing a willingness to compromise that Christianity and Islam became the world's dominant religions.
A more relevant example: isn't it the zealotry of the Linux crowd, rather than the technical merits of their OS, which has made Linux more successful than the technically-superior BSD?
Why?
While I agree that Theo is probably too harsh against newbies, being harsh against stupid people such as the glibc maintainer which refused to include strlcpy in glibc is normal IMHO:
it's really hard to understand why in 2006, there is still not an easy to use, safe strings library as part of the C common libraries, and strlcpy/strlcat provide this.
strlcpy/strlcat makes things worse
I'm far from an Ulrich Drepper fanboy, but I agree fully with his reasons for excluding strlcpy/strlcat from glibc. He's not being a lazy ass. He has a coherent argument that you should read.
Boiling down that argument from memory:
By using strlcat/strlcpy you continue to perpetuate the notion that you can write secure software with little awareness of string size. These functions mangle your strings. That may be less dreadful than mangling the stack, depending on the circumstances. Ever wonder why the mem* functions like memcpy are in string.h and memory.h does not exist? Proper string handling in C uses the mem* functions, not the str* functions.
If you don't know how big your data is, you have big problems. If you do know how big your data is, then you don't need a function (from the str* group) which stops on a NUL character. You use memcpy, which is faster anyway and, by virtue of simple operation, easier to use/read/debug.
strlcpy/strlcat make things better
Your argument ignores two simple and completely obvious facts:
First, there are many, many existing usage instances of unsafe string handling functions that can be greatly improved through a trivial conversion to strlcpy/strlcat. It is silly to expect them to all be completely rewritten to use unbounded strings, which can have their own problems that fixed length strings do not (potential memory consumption DoS, integer overflow, etc.)
Second, there are many, many cases where you want or need a fixed length string. Lots of network protocols, file formats and system interfaces require them, so why not provide good functions for working with them?
glibc is the only recalcitrant left (Solaris, the Linux kernel, Samba and hundreds of other packages already have these functions). Drepper is being absurd and egomaniacal in refusing these functions in.
pointless argument
Even if glibc had strlcat/strlcpy, using them would make my code more difficult to port to platforms that don't have them. If the C standard committe adopts these functions, then I'll consider using them, but until then, I just don't see significant gain.
And strlcat/strlcpy are completely broken for data transmitted across networks, since they don't zero-fill the remaining space in the buffer. strncpy may not properly null-terminate in all cases, but at least it gets the information leakage issue right.
First, this is a chicken/egg
First, this is a chicken/egg problem: standardisation comitee tend to standard what is widely used, witness gcc extension: many of them were incorporated into C99 and if gcc developpers hadn't push for them we would probably have a less useful C language now.
Then, who really cares about information leak??? Buffer overflow is a much much bigger problem, this is a very bad excuse.
Please stop making stupid excuse against inclusion such as 'programmers must manage perfectly their strings' from somebodytookmyname, in the real world programmers makes mistakes, security mistakes are very hard to spot as normal use case don't see any bad behaviour, providing an API which is safe and simple to use is very useful so glibc should include it, Ulrich Drepper is just being stupid..
Hopefuly, the next C standardisation will include such kind of function, but what a waste of time..
Like, which ones?
Even if glibc had strlcat/strlcpy, using them would make my code more difficult to port to platforms that don't have them.
You mean, like other UNIXes such as the BSDs? Oh wait, those have them. Maybe that pesky commercial UNIX, Solaris? Oh wait, it has them also. Ok, so you might not be able to port as easily to those widely popular FLOSS platforms such as Windows or UnixWare.
Sorry, the "less portable" argument doesn't fly with me. Ever hear of autoconf and #ifdef? I've been using snprintf() for years, long before Solaris ever got a copy, despite the fact my code sometimes needs to run on Solaris.
As the other respondant said, it's something of a chicken-and-egg problem. It's not as if you can get all OS vendors to declare a "flag day" wherein all of them will simultaneously start supporting some new "Feature X." And it's unclear whether something like this would ever make it into the ANSI C standard without widespread pre-existing support. Heck, strdup() is not in the ANSI C standard library, but you'll be hard pressed to find an implementation that lacks it. (I know of only one off-hand: TI's ANSI C compiler and RTS for their DSPs etc. lacks it.)
The "broken for networks" doesn't fly either. If you need zero fill, then keep using strncpy()/strncat(), and just add "buf[SZ-1]=0" or whatever to unconditionally NUL-terminate. It's really that simple. Just because you have strlcpy()/strncat() doesn't mean you blindly search/replace all instances of strncpy()/strncat() with them.
In other words, strlcpy()/strlcat() are tools to help you fix problems caused by using strncpy()/strncat(). Just because they don't solve all of the problems caused by the strn functions doesn't mean they're not useful.
portability issue or stupidity ? its four lines and a headerfile
if you want an easy and simple example of how to make stuff portable
with those functions and use the native ones where avialable,
take a look at the VIrus project on freshmeat.net - there may be complexer source with more elegant fixed, but this is what it boils
down to in about 100k of traffic. the author included strlcpy et al
in a simple and portable way (altough it could have been more elegant).
telling me that you are not able to include
is just a hilarious argument and clean sign of you ignorance of
how C works. sorry to be so blunt, the argument is stupid.
back to the books for you.
and regarding your zero-fill issue:
you should be perfectly capable of defining a macro to
zerofill from the position of the null character to end of string
with memchr and memset with ease.
security and integrity of data first. then speed.
you cannot be sure your data is reliable when the box is hacked due to a buffer overflow - after all if there is a faint chance, people
like me will find the overflow.
Newbies
I personally don't find Theo to be rude to newbies. I myself am green behind the ears, but have been replied to by Theo in a more than courteous manner. In short, if you put yourself across as an idiot, you'll be treated as one, newbie or not.
I get the impression that most of it comes down to egos. Those with the bruised egos tend to become Theo's critics. Be respectful and you will find the respect returned. Attack Theo or OpenBSD and you'll be retaliated against. It's not Theo's problem that he counter punches like he does.
"Those with the bruised egos
"Those with the bruised egos tend to become Theo's critics."
Bullshit. Wrong generalization.
Theo loves to stir trouble.
Thats the fact. See the most recent (and wrong) "discussion"
about Linux developer hijacking BSD code into Linux kernel.
Theo is exactly right in
Theo is exactly right in that case also.
I have a great deal of
I have a great deal of admiration for Theo. He occasionally misses the mark, I think - but then, I never completely agree with anyone but myself.
Trying to defend him against everyone that attacks him is pointless. There are some with reasonable arguments, some that call him all sorts of names (and show themselves to be worse then he is), but all are childishly attacking somebody with little real reason to. I really don't understand why a small collection of GNU users insist on attacking Theo and OpenBSD every time the topic comes up. It's childish.
With amusement, I would also like to take the opportunity to point out Linus. Theo's temper flares sometimes, but he's no Linus. Linus openly admits to loving flame wars. He's overly critical against things he disagrees with. He happens to love "stirring the pot" and causing flamewars. Don't take my word for it, check the lists or read an interview with him that mentions it.
I don't have time to waste b
I don't have time to waste being nice to people who are being stupid.
That's exactly why I don't waste time being nice to you, Theo.
Oh okay. So you freely admit that you are stupid. Since you are admitting this publicly, I guess it only stands to reason that you have come to believe and accept that you are stupid.
So if this is the case, why don't you just shut up and listen to people who are not stupid? Theo has clearly made some huge achievements due to his forward thinking ways, achievements which have benefited OSS at large, not just OpenBSD. So he is clearly not stupid.
So what have you achieved?
nice
You sound pretty nice still, though.
C Compilers
I assume you investigated TCC. What was lacking in it?
What other compilers did you review?
tcc ... tried it, liked it.
tcc ... tried it, liked it.
but it's rather broken for quite many projects who heavily heavily depend on misc. features of gcc.
there's little chance of hundreds/thousands ports to be backported to that level.
i hope it will evolve a bit and can finally kick the $%&#^@ out of the old and sturdy gcc.
That's a great suggestion!
From the TCC docs:
From the OpenBSD Hardware platforms page:
Admittedly, not all of those targets use distinct CPUs.
It doesn't support all the ar
It doesn't support all the archs OpenBSD does.
Platforms, i386 only. tcc ne
Platforms, i386 only. tcc needs to not suck for it to be acceptable.
kencc has been investigated as well - but it's under Lucent restrictions.
Watcom
One wonders if they've looked into OpenWatcom... it has a fairly long history of not sucking (except for the char defaunting to unsigned of course)
Char defaulting to unsigned i
Char defaulting to unsigned is perfectly allowed by the standard.
char default
... and unsigned chars is the sane choice anyway! Who has ever heard of
negative letters? About as insane a concept as purple
double-precision numbers... It is one of the fundamental design flaws
of C that the signedness of plain char is implementation-specific.
But OpenWatcom has the same problem with platforms as tcc. If you
need a free compiler targeting multiple platforms, gcc is really the only
game in town.
TCC has quite a few bugs. It
TCC has quite a few bugs. It chokes on the stdlib.h shipped with Fedora Core 5, for example.
let's start rating vendors and products ...
publish lists of vendors and product who are friendly to opensource projects and rate their products. start giving the power of choice to prospective buyers of new hardware.
Well put.
Well put.
Exactly!
I completely agree with this idea. A site with some nice and clear good/bad icons would let everybody know the culprits of slowing development and usefulness down.
There should also be a case for "law" and "egoism", "enslavement of customers" and such.
It should also be non-distribution-specific (combining all free OSes).
Rating vendors and funding OpenBSD
Actually not a bad idea at all, perhaps this should be proposed on a mailing list or such?
It's a shame OpenBSD doesn't have any more funding behind it. Even though the OS itself is often lacking a lot of newer stuff and testing, a lot of the most practical and generally useful code is still being written by the OpenBSD folk, such as pf and OpenSSH.
Perhaps Theo could be prescribed some sedatives, to help him ignore morons instead of insulting them ;-)
theo and seditatives ?
c`mon, more than one of us crowd are getting used to the verbal slappings. we might miss something :D
http://www.vendorwatch.org/
http://www.vendorwatch.org/
OpenBSD has no support for advanced gfx cards
OpenBSD was supposed to be advanced operating system for serious work. Why the hell can't you debug Matrox Parhelia display drivers? I'm doing scientific stuff with my computer and I need dual 2D DVI display.
LOL What a profoundly stupid
LOL What a profoundly stupid statement.
Did you read the article ? It told you where and why such comments should be directed.
FCC
I think it's a little silly to build chips that can only operate within the bounds of FCC requirements. For one thing, there are markets outside the US that the same chip can serve. (Ever hear of "world mode" on a card?) For another thing, the FCC's own regulations can change. Making the enforcement logic soft only makes sense.
And while I'm not one to diss my employer in a public forum, I do find it disappointing TI hasn't responded to OpenBSD's requests. It also doesn't surprise me too much.
One might also wonder if stri
One might also wonder if strict FCC rules are still a thing to keep. Some coordination surely is necessary in order to keep things functioning, but if it leads to these problems, isn't it time to get over FCC cathedralic dictatorship, too?
Theo's home page is reworked
I see Theo's updated his personal home page, looks great, and up to date.
Blobs are bad, m-kay?
Theo: First off, can we trust the code to do what it should do? I don't think so.
The companies are preparing us for the day when the NSA makes a deal with a company to insert undocumented back doors into our operating systems through our video cards and netwrok adapters. A Redmond company will be complicit, but it maintains plausible deniability because it's not their code.
Ok, ok, that's just a wild conspiracy theory, but without source code for people to review, how will we really know?
you have the object code, che
you have the object code, check the instructions....
I'm glad you have the time to
I'm glad you have the time to run every branch of every piece of the 30 gigs or more of code on a modern desktop through the debugger. I don't.
Especially...
Especially if you do sneaky things like overwriting return addresses on the stack or function pointers in data structures in order to get to your secret functions.
Not good enough.
The back door could already be in the compiler and you wouldn't know it. Done correctly, a source review of the compiler wouldn't be able to reveal it.
http://www.acm.org/classics/sep95/
Object code, not source code, you ninny. :-)
That's why he said you have to check the object code, branch by branch.
SMP and Threads
Darn, no questions regarding further SMP developments and user/kernel threading. I wanted to know more about rthreads, especially how they compare to the threading mechanisms in other Open Source Unix variants (namely Linux and FreeBSD). I would really like to know how they expect rthreads to perform against the competition. It would be nice to see OpenBSD become a real performer in SMP and muli-threading situations (i.e. for running databases and other threaded applications).
I wouldn't hold your breath w
I wouldn't hold your breath waiting for OpenBSD to compete for performance. I'm sure, for the most part, performance will only get better. But, I doubt anybody on the OpenBSD team cares to edge out Linux or FreeBSD across the board. Moving to things like O(1) algorithms not-withstanding, OpenBSD will always strive for correctness and maintainability first. And since correctness is a difficult quality, performance is something that will sit in the background for a long, long time for any particular chunk of the system.
Also note that SMP and threads are extremely difficult to get right.
I suspect OpenBSD will always just stick with picking the low hanging fruit as far as SMP goes, and I'm just fine with that. I don't mind using Linux for high performance, single use applications, and OpenBSD as general purpose Unix machines (shell, e-mail, et al).
OTOH, OpenBSD written daemons, if you haven't noticed, are asynchronous I/O based, not multi-threaded. OpenBSD is somewhat competitive in this space, given the kqueue(2) API. So most of the developers really have no use for threading. If you code for OpenBSD, expect to code for OpenBSD. That's what I do.