I arrived in Calgary this afternoon, and headed straight downtown to the hotel in which the OpenBSD hackathon is taking place. Walking through the fancy hotel's front door, the concierge steered me upstairs to the hackathon rooms. A few minutes later, I entered into a dimly lit room crowded with tables, people, wires, switches, laptops and servers. Each table is covered with a white cloth, with ethernet cables dropping down from the ceiling, suspended by brown tape.
The hackathon is taking place in two rooms, attended by around 60 OpenBSD developers. Obtaining an accurate count of the attendees is difficult, as people have arrived from all over the world and are still working in different timezones, populating the room around the clock. I was warmly greeted, and given free access to wander around asking questions and taking notes. At first glance, it seemed there was very little order to the event, but speaking with OpenBSD creator Theo de Raadt [interview [1]] he was able to point from table to table explaining what each person or group of people were doing.
Theo explained that he's personally not working on any specific hacking himself, but instead is involved in everything. "I'm involved with lots of ideas," he explained, "shooting them down, changing them, approving them, pointing them to someone they should talk to..." He described his primary role as, "accelerating communication." Indeed, the one week hackathon is designed for just that, accelerating communication between OpenBSD developers who have gathered from all over the world. Ideas are flowing, and code is being written.
I walked around among the tables and over the course of the day interrupted over a dozen developers, asking questions to get a better feel as to what they're working on. We covered a wide range of topics, with each developer excited by what they were working on, a benefit of doing what you want to do. Below are just a few samples from the conversations, with additional articles to follow based on the rest of the conversations in the upcoming days.
Improved threading:
Ted Unangst has been intent on his laptop for much of the day, working on a new and improved threading library for OpenBSD. "Currently we're using a userland threading libary," he began. "We can get better performance by switching to kernel threads." And that's what he's been working on, a simple 1:1 threading module based on the rfork() system call, a forking function adapted from Plan 9. The new threading implementation is currently called rthreads, "so far it's incomplete, but it seems to work a lot better for some programs."
Ted explained that the main problem with the current threading library is that only one thread within a process can run at a time. When, for example, one of the threads reads from the disk, the whole process is blocked because the kernel has no way of knowing that there are other threads needing to run. In the new implementation, this is no longer the case because the userland threading library no longer performs any scheduling, instead letting the kernel do this work. Each thread is now treated like a normal process, and so for example on a quad processor server it's possible to run four different threads on four different CPUs.
Effort was began on this 2 years ago after a hackathon, when Ted first hacked up a prototype in about two days. It then sat for two years, until now when he finally sat down to finish it. "We've been struggling with the current threading library for years," Ted said. "We never just sat down and started over, which turned out being a lot easier than expected. Of course, the real test comes when people start using it with all the different programs that use thread in different ways." Both the new and the old libraries will coexist for a time, and are meant to be binary compatible. The new library should be available in OpenBSD 3.8, but whether or not it is the default threading library by then is yet to be seen.
One of the main advantages of the new library is its simplicity. Currently comprised of only 750 lines of code, Ted estimates that when finished it will be around 2,000 lines long. Compare that to the old library which weighs in at 16,000 lines. "Because it's a lot simpler," he noted, "we can fix it easier and it's easier to understand."
Currently the only program that Ted has gotten fully working with the new library is xmms. Comparing the new library to the old, Ted commented, "when xmms was playing, you could scroll the playlist which would cause the program to read mp3 headers from 100 different files and your music would start skipping. It doesn't do that anymore." He noted that's he's also close to getting Mozilla running. Then the next big program he'll focus on will be MySQL. "MySQL performance is currently not the best on OpenBSD, depending on the workload. On high workloads it doesn't perform well because threads can't keep up." The new library should solve for this, and should be finished enough tonight for other people to start working on it.
PCI Devices On AMD 64
Jason Wright was focused on getting PCI devices so they can access all physical memory on an AMD 64 when I interrupted him. Currently, 32-bit devices have a 4GB DMA limitation in that they can only directly access 4 GB of memory. In order for a PCI device to directly access more than 4 GB of memory, Jason is working on remapping memory using the AGP GART. GART is the Graphics Aperture Relocation Table, and is essentially being used to remap memory in such a way that the PCI device believes it is only accessing 4 GB of memory when it is actually accessing all available memory.
Explaining why this works, Jason said, "all IO devices can only see 4 GB of memory at a time, but no device needs to see all of memory at any given time." He likened the idea to virtual memory, paging in what you need. "Most processes typically only need a few megs of memory," he added, "but for DMA that small piece needs to be anywhere in physical memory." DMA stand for Direct Memory Access, and is used in all modern devices. "It's the ability for a device to read from main memory or write to main memory without assistance from the processor. The card is given a real physical address of memory, it accesses that memory, then it notifies the CPU when it's done." With the work Jason is doing, the "real physical address" is actually a virtual address that is re-mapped to the physical address.
The effort is currently being implemented on a server with less than 4 GB of RAM, tested by forcing DMA to map through the GART. "If it didn't work," Jason explained, "the machine would have crashed by now because the addresses I picked aren't actually there. If it actually tried to write there, it would crash." New hardware is currently en-route to the hackathon for testing tomorrow to verify that it indeed works on a machine with more than 4 GB of RAM.
The implementation is based on documentation provided by AMD, "the BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD Opteron Processors". "The coolest thing about AMD," Jason noted, "as just a regular user I went to their website and downloaded everything I needed. I didn't have to sign in, I didn't have to register, I didn't need to get a special login, I just went to the website and downloaded." He added that AMD has always been very open, something highly valued by the OpenBSD project. In contrast, he noted, "certain parts of Intel have also been very open, but not all parts. For example, networking." He laughed, "they're keeping secrets of 15 year old technology."
Jason noted that support for this is currently in Linux, and that occasionally he refers to that code to compare their implementation to his own. "I check for specific things," he explained, "checking how it matches with the documentation, or when running into a bug in my code. It can be helpful as a sanity check where someone has done something similar." He noted that obviously this is only useful when someone else has done something similar, not when he's breaking new ground.
There's still more work to be done. For example, currently the implementation only works for one CPU, though support for multiple CPUs is nearly complete. Additionally, at this time it only works when the BIOS hasn't initialized the GART. He still plans to add code to deal with when the BIOS has already done the initialization work.
Supporting the Zaurus C3000
Support for Sharp's Zaurus C3000 [Array] was first added in OpenBSD 3.7 by Dale Rahn. A tiny little $800 PDA, the ARM-based Zaurus C3000 has a 4GB hard drive and 64 MB of RAM. It currently uses Linux as a boot loader because there currently isn't a BSD licensed open source flash filesystem. This does allow the device to be easily dual booted into either OpenBSD or Linux.
Initially the port began on the Zaurus C860, an earlier model of the PDA. "Theo wanted something he could use at the local bar to just be online," Dale noted. "'Here, make it work', he told me." The C860 didn't have a hard drive, but Dale got it working when Theo offered him the newer C3000 with a hard drive as a reward for making it work. Following this offer Dale noted, "about two weeks later I was running single user, and committed it after Christmas of last year." Support for the C860 only lasted about a week, as once Dale received his new C3000 that was the device of choice.
Uwe Stuehler is also involved in supporting the C3000. "I had one I bought myself," he noted, but after working to improve support he added, "now I have a second one Theo bought me." Uwe pointed out that Dale is still the official maintainer, to which Dale retorted, "until I manage to give it to you." While I was sitting there, Uwe plugged in a blue tooth connector for the first time and was excited to find that it recognized it.
Since the initial Zaurus release in 3.7, support for audio has been improved, support for infrared has been added, and the console has been rotated so it scrolls top down rather than left to right, allowing you to type on it much like a normal laptop. Serial support was also fixed to allow for debugging. Future plans include possibly adding Secure Digital for SD IO cards or SDRAM, and USB slave mode allowing you to plug it into a PC and show up as a network device or mass storage, though neither of these features is currently being worked on. Otherwise, the port is considered to be feature complete.
The ARM processor itself contains nearly all IO devices and is very well documented. Uwe brought up a 12 MB document which he described as, "accurate and helpful as far as it goes." However, outside the processor are other devices such as a GPIO and a power controller for the PCMIA sockets. "Documents for those features are not easily available," Uwe noted. Instead, the Linux kernel source is read as documentation. Referring to the GPL, Dale explained, "we can't copy the code, but we can read the code and try to determine how it works. Then we can write code based on that interpretation."
Dale explained that the Zaurus distribution provides an identical working environment as any other OpenBSD distribution, "OpenBGPD works, IPSEC works, you could use it as a firewall, you could build an access point out of it. There's nothing special about it compared to other distributions." The base OpenBSD install only takes up 128 MB of the 4 GB hard drive. He noted that suspend works much like a laptop, however requiring that you hold down the power button down for a couple of seconds. Dale explained that some use this feature to leave the Zaurus running while the top is closed, "people walk around with kernels compiling in their pocket. It takes about an hour to compile a kernel."
David Gwynne worked on USB support for the Zaurus. The device has a mini USB port which provides very little power itself, preferring to be plugged into a powered USB hub. "We have way better USB device support on the Zaurus than Linux," David said.
The discussion drifted to comparing Linux adoption with OpenBSD adoption. "Linux has business mindshare," Dale said, "Linux has done a better job of selling itself to businesses." In contrast, he continued, "we don't sell ourselves to customers. We're happy if they use our code, but we don't court them. And sometimes when they dangle money in front of us we say no."
Dale continued explaining, "we like to feel like we're in control. If they are giving you money, they're in control and can tell you what to do. Most of the developers are doing this all in their spare time, and are not getting paid for it. We want to play, we don't want to have to work."
Timezones
Having come from Florida via Alaska, I'm not sure what time it currently is. From the feel of it, quite late. Though many have drifted off to their rooms for the night, the room is still full of OpenBSD hackers working on various projects. Music is playing, a few conversations are going on with people solving problems together, and others are focused quietly on their own computers. From the looks of it, the activity will continue through the night.