In a moment of weakness/insanity/community spirit [cross out the least credible] I have promised the Dæmon News crew some articles about the FreeBSD kernel.

Back when Dæmon News started, a couple of us came up with the idea of a series of "blueprint" articles, intended to shine some light on the actual construction details of the kernel. This is my attempt to make good on that idea.

I will put a big disclaimer up here from the start: For close to a decade I have been running FreeBSD-current on my laptop, and everything I talk about is relative to FreeBSD-current.

FreeBSD's Robert Watson says that effective today, newfs(8) and sysinstall(8) will create UFS2 file systems by default, unless explicitly specified. Users wanting to create UFS1 file systems for whatever reason (interoperability with earlier versions, etc) should be sure to employ the -O1 flag to newfs(8), or hit '1' in the label editor in sysinstall(8) to select UFS1.

Jake Burkholder announced today that he has committed changes to support PAE and more than 4GB of RAM in the FreeBSD -current tree. He explains:

"Basically what this does is allows physical memory above 4G to be used normally by the kernel and userland. Except in certain circumstances no distinction is made between memory above and below 4G, it all just becomes part of the general page pool. This does not increase the amount of virtual address space, just the amount of physical memory you can use."

Jake went on to say that the code has been extensively tested on a system with 6GB of RAM, and that he is quite interested to hear from anyone who is able to test it on larger systems. Additionally, he noted that at this time there are a few caveats to be aware of: not all device drivers work properly, kernel modules must not be loaded with PAE enabled, and auto-tuning within the kernel "starts to fall apart pretty fast with lots of memory" (prevented by manually adjusting kern.maxvnodes). Read on for Jake's announcement email.

Murray Stokely announced today the availability of FreeBSD 4.8, following October's 4.7 release [story] by 6 months. Included in 4.8 are "conservative updates" of a number of software programs from the base system, several known security fixes, initial Firewire support, HyperThreading support, and support of "other new hardware technologies".

Murray notes, "This release does not include all of the new technologies that were introduced with FreeBSD 5.0 in January. FreeBSD 4.X releases offer a more conservative platform than FreeBSD 5.0 at this time." In other words, 4.8 is currently considered to be the -stable production release, whereas 5.0 [story] remains the development or New Technology release, as reflected here. More information about 4.8 can be found in the release notes and known errata. Murray's full announcement follows.

Jeff Roberson recently announced a new 1:1 threading implementation that has been merged into FreeBSD -current. The effort builds upon the work done so far on KSE [story], offering SMP scalability and working proof of KSE's design. Jeff explains, "This code works in parallel with KSE and does not break it in any way. It actually helps bring M:N threading closer by testing out shared bits."

In his announcement, Jeff notes that he has succesfully run Mozilla and Open Office with this new threading library. With the March 31'st merge annoucement, Jeff noted that the code is beta quailty, and he included a small list of known errata .

Following the announcement of this new threading library, it was asked how it fits together with the overall SMPng effort. Terry Lambert provided an interesting summary of each of these projects, including pthreads, libc_r, KSE, libthr, and SMPng. He provides a few related links, however explaining, "most of the documentation lives in mailing list archives, and is not terribly formal (Software Engineers, not English Majors, and all that...)" Read on for the full details.

Scott Long recently posted this year's first FreeBSD status report. The document begins with a quick look at the recently released FreeBSD 5.0 [story], then looks to the future roadmap [story] with the 5.1 maintenance release coming within a couple of months, and the stable 5.2 release by the end of the summer. Also mentioned is the upcoming 4-STABLE release, 4.8, which includes XFree86 4.3 and support for HyperThreading. The document continues on to talk about the status of a number of current FreeBSD projects and upcoming events.

"Another busy two months have passed in the FreeBSD project. With 5.0 released, attention is focusing on making it faster via more fine-grained locking, adding more high-end features like large memory (PAE) support for i386, and further progress on many other projects."

Subscribers of the FreeBSD -current mailing list receive regular emails regarding "tinderbox failure's". A tinderbox being a test system that automatically builds the -current tree, reporting breakage repeatedly until it gets fixed. Lists receiving these tinderbox failure reports include -alpha, -current, -ia64 and -sparc64.

Occasionally the subject comes up of moving these errors to a separate mailing list giving subscribers the choice of receiving them. David Schultz replied to this suggestion, "I think most people who track -CURRENT are subscribed to current@ precisely because they want to know when things break."

Scott Long posted a lengthy email to the current FreeBSD mailing list, laying out the roadmap for FreeBSD 5.0:

"After nearly three years of work, FreeBSD 5.0 was released in January of 2003. Features like the GEOM block layer, Mandatory Access Controls, ACPI, sparc64 and ia64 platform support, and UFS snapshots, background filesystem checks, and 64-bit inode sizes make it an exciting operating system for both desktop and production users. However, some important features are not complete. The foundations for fine-grained locking and preemption in the kernel exist, but much more work is left to be done. Work on Kernel Schedulable Entities, also known as Scheduler Activations, has been ongoing but needs a push to realize its benefit. Performance compared to FreeBSD 4.x has declined and must be restored and surpassed."

Scott goes on to draw a parallel between the recent 5.0 release [story] and the much earlier 3.0 release, which didn't become truly stable until it reached version 3.2. His goal with this document is to prevent the same from happening to the 5.x line. Read on for a very informative look into the current status of 5.0.

A recent conversation on the FreeBSD hackers mailing list discussed the possibility of implementing random expiration of the disk cache. The example given refers to multiple large sequential reads of the same data, that instead of throwing away the oldest blocks as newer ones are read, scatter this cache eviction randomly with an appropriate algorithm. After some interesting discussion, the theory was shot down mathematically by Terry Lambert, describing the idea as "Bogus. What you are effectively hoping for is that the pages will not be evicted before they are hit again, because your shotgun pellets will not shoot them down." He pointed out that as the size of the data set grows compared to the cache size, random eviction won't help.

Matt Dillon [interview] reiterated:

"What Terry is saying is that if you have a dataset that is 2x the size of your cache, the cache hit rate on that data with random page replacement is NOT going to be 50%. This is because with random page replacement the likelihood of a piece of data being found in the cache depends on how long the data has been sitting in the cache. The longer the data has been sitting in the cache, the less likely you will find it when you need it (because it is more likely to have been replaced by the random replacement algorithm over time)."

FreeBSD 5.0 has been officially released. For an in depth view of what's changed since 4.x, take a look at the release notes. Known bugs and security advisories can be found here. Finally, an excellent overview to help you get started with 5.0 can be found in the Early Adopter's Guide. Download a copy of this new FreeBSD release from a local mirror.

FreeBSD 5.0 currently supports 5 architectures: alpha, ia64 (new), i386, pc98, and sparc64 (new). New features boasted by this release [story] include UFS2 ("shattering the current 1TB filesystem barrier"), background filesystem checking, filesystem snapshots, experimental support of Mandatory Access Controls, Kernel Scheduled Entities [story], fine-grained SMP support, the GEOM extensible and flexible storage framework [story], and support for GCC 3.2.1 [story]. Read on for Scott Long's announcement email.

Scott Long announced Release Candidate 3 of FreeBSD 5.0 today, asking people to test it as much as possible. The final release of 5.0 is scheduled to be compiled in a few short days, on January 15'th, widely available by the 19'th. According to the early adopter's guide:

"FreeBSD 5.0 marks the first new major version of FreeBSD in over two years. Besides a number of new features, it also contains a number of major developments in the underlying system architecture. Along with these advances, however, comes a system that incorporates a tremendous amount of new and not-widely-tested code. Compared to the existing line of 4.X releases, 5.0 may have regressions in areas of stability, performance, and occasionally functionality."

Further information as to what's been added in FreeBSD 5.0 can be found in the "What's New" section of the release notes [story]. The remaining 5.0 open issues are quickly being fixed.

There was an interesting exchange recently on the FreeBSD hackers mailing list. A user with a FreeBSD firewall was suffering a DDoS (Distributed Denial of Service) attack and was asking for suggestions on how to lessen the effect. The resulting thread offers some good advice.

An older but also interesting story about an unrelated DDoS type attack is found in this article by Steve Gibson, making for an addicting read. Another example being the recent DDoS attacks against the DNS root servers which with ping flooding successfully crippled 7 of the 13 root servers for nearly an hour.

Each kernel and networking stack certainly has its own strengths and weaknesses against different types of attacks. What sorts of war stories do KernelTrap readers have regarding DDoS attacks, and what sorts of advice can be offered?

FreeBSD 5.0-RC2 is ready and waiting for your feedback. The plan going forward is to cut an RC3 in early January, followed by a 5.0-RELEASE a week later. The full release announcement follows.

The FreeBSD project announces the availability of the second preview for their upcoming 5.0 release. This major release will be based on the GCC 3.2 branch, and bring us an extensible Mandatory Access Control framework (named TrustedBSD), the new UFS2 format with support for larger filesystems, support for Firewire devices, on-disk encryption, SMPng (next generation SMP support), KSE for multiple kernel-level threads, and much more. The Early Adopter's Guide might be a good read for those interested in getting started with the new 5.0 codebase.

The release announcement follows.

Marko Zec provided patches to the 4.7-Release FreeBSD kernel tree which offer "the functionality of maintaining multiple independent network stack images within a single operating system kernel." Marko's implementation is designed so that the virtualization modifications within the kernel do not modify any ABI's, so FreeBSD binaries should run on the modified kernel without modifications themselves. Marko adds, "Furthermore, as there are no address translation hacks, library replacements/hooks etc., the overall performance penalty of introduction of virtualization layer is mostly [negligible]." He went on to state that in preliminary TCP flow tests he saw a 1-2% performance penalty in maximum throughput, however noting that further testing was required, the results of which will be posted to his web page.

Marko's patch divides the kernel into a number of virtual images, each providing entirely independent sets of network interfaces, addresses, routing tables, raw protocol control blocks, net.inet tuneables, and traffic counters / statistics. It is even possible to configure unique ipfw firewalls and run separate dummynet tests within each of these virtual images. Download the source from the BSD network stack virtualization home page. If unwilling to boot a patched FreeBSD kernel on your server, Marko also provides 320MB disk image allowing you to easily test a patched kernel from inside VMWare. Read on for Marko's full announcement and the resulting discussion thread.