Constantine Murenin offered a history of the OpenBSD hardware sensors framework during his talk at BSDCan 2008, describing how it was originally based on a port from NetBSD, then evolved and was eventually ported to all the BSDs. He also discussed his own involvement with the framework, having ported it from OpenBSD to FreeBSD as a Summer of Code project, and how his port was merged into DragonFly BSD. At the end of the talk, there were some interesting ecxhanges between Constantine and Poul-Henning Kamp, the latter explaining why he'd had the code backed out of FreeBSD and why he continues to oppose it being merged back in.

Leslie Hawthorn, a Program Manager in Google's Open Source team, gave a talk at BSDCAN 2008 on Google's ongoing Summer of Code project. She started by explaining what the open source team does, including enforcing license compliance, hosting over 700,000 open source projects with Google Code, academic research, funding open source development, and community outreach including the sponsorship of conferences such as BSDCan. She went on to discuss how she got started running the project after its initial launch in 2005.

Having sponsored four summer of code's now, Leslie noted that Google has had over 1,500 "graduates" and over 2,000 mentors involved, coming from over 98 countries and working with over 175 open source projects. By the end of the currently in progress 2008 Summer of Code, the project will have provided over 10 million US dollars in funding, generating over 6 million lines of code.

Randall Stewart of Cisco Systems gave a talk titled SCTP, what it is and how to use it, discussing the Stream Control Transmission Protocol (SCTP). A paper that was displayed on the overhead projecter before the talk began summarized:

"Integrated into FreeBSD 7.0 -- first standardized by the Internet Engineering Task force (IETF) in October of 2000, in RFC 2960 and later updated by RFC 4960. SCTP is a message oriented protocol providing reliable end to end communication between two peers in an IP network."

Randall explained that SCTP is an alternative protocol to TCP, UDP. To describe SCTP, he suggested you start with TCP features, including: reliable retransmission, congestion control, flow control, connection oriented, and selective acknowledgements. You then add to it more features, including: "association" 4-way handshake, framing and ordered service, multistreaming, multihoming, and reachability.

Pawel Dawidek first ported ZFS to FreeBSD from OpenSolaris in April of 2007. He continues to actively port new ZFS features from OpenSolaris, and focuses on improving overall ZFS stability. During the introduction to his talk at BSDCan, he explained that his goal was to offer an accessible view of ZFS internals. His discussion was broken into three sections, a review of the layers ZFS is built from and how they work together, a look at unique features found in ZFS and how they work internally, and a report on the current status of ZFS in FreeBSD.

The BSDCan website notes that Pawel is a FreeBSD committer, adding:

"In the FreeBSD project, he works mostly in the storage subsystems area (GEOM, file systems), security (disk encryption, opencrypto framework, IPsec, jails), but his code is also in many other parts of the system. Pawel currently lives in Warsaw, Poland, running his small company."

BSDCan 2008 officially started this morning at 9AM with an opening talk by the event's organizer, Dan Langille. However, in reality the event has already been running for two days, with the FreeBSD tutorials having started on the 14'th. After arriving in Ottawa yesterday afternoon and finding my room in a 20 story University of Ottawa residence, I wandered down to the Royal Oak Pub for early registration, meeting several dozen BSD hackers from all over the world.

This morning's opening talk was well attended, filling up first with clusters of laptop users around the power outlets along both walls. By 15 minutes after the hour, the room was completely full, and Dan started with a humorous slideshow of example letters he's been receiving ever since posting the words "letter of invitation" somewhere on the BSDCan website two year back. Coming primarily from Nigeria, the letter's authors often claim to represent large groups of developers, yet always coming from "disposable" email addresses. After some laughs, he launched into his opening keynote.

KernelTrap is excited to be able to offer live coverage of this year's BSDCan 2008 in Ottawa, Canada on May 16th and 17th. The two day conference takes place at the University of Ottawa, and was organized for the fifth consecutive year by Dan Langille who has also made it possible for me to attend and cover the event on KernelTrap. I spoke with Dan to get some background information on the conference, and learn about some of the upcoming highlights.

The event's webpage explains:

"BSDCan, a BSD conference held in Ottawa, Canada, has quickly established itself as the technical conference for people working on and with 4.4BSD based operating systems and related projects. The organizers have found a fantastic formula that appeals to a wide range of people from extreme novices to advanced developers."

"I and the other Kernel.org admins would like to announce downtime for ALL kernel.org machines (this includes all of the mirror machines, the public machines and the backend master). The downtime is scheduled to start on or around April 2nd, 2008 on or around 0001 UTC," began a GPG signed message on the Linux kernel mailing list from John 'Warthog9' Hawley, one of the kernel.org admins. Referencing a recent Slashdot discussion that compared Linux and FreeBSD performance, he continued:

"After much deliberation, research and argument in #korg (along with screaming matches between HPA and I over dinner) we are upgrading the kernel.org machines from Fedora Core 5 to FreeBSD 7.0. This decision does not come lightly to the Kernel.org admins, and we would like to point out several key things that helped us form our decision:"

John concluded, "we feel that we can better serve our mirrors, our users and the community by making the switch, and we hope to have the transition done very shortly."

"Google Summer of Code 2008 is on! Over the past three years, the program has brought together over 1500 students and 2000 mentors from 90 countries worldwide, all for the love of code. We look forward to welcoming more new contributors and projects this year," begins a page listing all the projects planning to participate in this year's GSoC. Among the numerous planned participtants there are many kernel projects, including DragonFly BSD, FreeBSD, Git, GNU/Hurd, Linux, Minix, and NetBSD.

Student applications for GSoC projects begin today, running through the end of the month. Read on for many of the participation announcements from the above projects. For more information about the GSoC, the program's FAQ explains:

"Google Summer of Code (GSoC) is a program that offers student developers stipends to write code for various open source projects. Google will be working with a several open source, free software, and technology-related groups to identify and fund several projects over a three month period. Historically, the program has brought together over 1,500 students with over 130 open source projects to create millions of lines of code. The program, which kicked off in 2005, is now in its fourth year."

"I'd like to send a small update on my progress on the Performance Tracker project," noted Erik Cederstrand on the FreeBSD -current mailing list. He continued, "I now have a small setup of a server and a slave chugging along, currently collecting data. I'm following CURRENT and collecting results from super-smack and unixbench." The project performs regular benchmarks of the FreeBSD -current source tree using Unixbench and Super Smack, allowing you to chart the results over time. Erik highlighted an example of a visible change in performance when the generic kernel moved from the 4BSD scheduler to the ULE scheduler on October 19th, 2007.

Kris Kennaway responded favorably, then noted, "one suggestion I have is that as more metrics are added it becomes important for an 'at a glance; overview of changes so we can monitor for performance improvements and regressions among many workloads." He went on to suggest, "at some point the ability to annotate the data will become important (e.g. 'We understand the cause of this, it was r1.123 of foo.c, which was corrected in r1.124. The developer responsible has been shot.")" Erik agreed with both recommendations, and noted that he would continue to work in that direction.

A recent thread on the FreeBSD -current mailing list discussed the stability of ZFS on FreeBSD. Scott Long noted that ZFS requires proper tuning to be stable:

"I guess what makes me mad about ZFS is that it's all-or-nothing; either it works, or it crashes. It doesn't automatically recognize limits and make adjustments or sacrifices when it reaches those limits, it just crashes. Wanting multiple gigabytes of RAM for caching in order to optimize performance is great, but crashing when it doesn't get those multiple gigabytes of RAM is not so great, and it leaves a bad taste in my mouth about ZFS in general."

ZFS was committed in April of 2007 by Pawel Dawidek who notes that he is using ZFS quite successfully on all of his systems. He then cautioned, "of course all this doesn't mean ZFS works great on FreeBSD. No. It is still an experimental feature." In response to some negative comments about ZFS on FreeBSD, Pawel noted, "in my opinion people are panicing in this thread much more than ZFS:) Let try to think how we can warn people clearly about proper tunning and what proper tunning actually means. I think we should advise increasing KVA_PAGES on i386 and not only vm.kmem_size. We could also warn that running ZFS on 32bit systems is not generally recommended."

"This report covers FreeBSD related projects between July and October 2007," began the latest FreeBSD Quarterly Status Report, posted by Brad Davis. He included a summary of the recent Google Summer of Code projects noting, "lots of participants are working getting their code merged back into FreeBSD." Regarding the upcoming FreeBSD 7.0 release he noted, "the bugs in the FreeBSD HEAD branch are being shaked out and it is being prepared for the FreeBSD 7 branching. If your are curious about what's new in FreeBSD 7.0 we suggest reading Ivan Voras' excellent summary."

Among the many projects discussed in the status report was work by Marko Zec on network stack virtualization, "the network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This allows for networking independence between jail-like environmens, each maintaining its private network interface set, IPv4 and IPv6 network and port address space, routing tables, IPSec configuration, firewalls, and more." Another project discussed was the porting of Linux KVM, "a software package that can be used to create virtual machines fully emulating x86 hardware on top of machines supporting Intel VT-x or AMD-V virtualization extensions." The report noted, "Linux KVM has been ported to FreeBSD as a loadable kernel module, using the linux-kmod-compat port (in /usr/ports/devel/) to reuse as much as possible of the original source code, plus an userspace client consisting in a modified version of qemu, that uses KVM for the execution of its guests."

Andrew Doran posted some threading benchmark results to NetBSD's tech-kern mailing list, following up to some benchmarks he'd posted earlier. The results compared NetBSD -current with FreeBSD -current, and the Linux 2.6.21 kernel. Kris Kennaway was surprised by the results, and ran his own benchmarks with minimal configuration changes, summarizing, "this measurement shows that FreeBSD is performing 70-80% better than NetBSD in this 4 CPU configuration. This is in contrast to Andrew's findings which seem to show NetBSD performing 10% better than FreeBSD on a 4 CPU system (a very old one though)." He added, "the drop-off above 8 threads on FreeBSD is due to non-scalability of mysql itself. i.e. it comes from pthread mutex contention in userland."

Kris ran additional benchmarks with PostgreSQL instead of MySQL, showing much improved scalability above 8 threads, "postgresql is much more scalable than mysql on this workload and doesn't have silly scaling bottlenecks inside the application (cf the tail of the FreeBSD curve for mysql which is where pthread mutex contention kicked in)." He continued his testing, and found that on older 4CPU P3 hardware NetBSD did outperform FreeBSD, "but only by 3-4% (in particular I am not seeing the ~10% difference that Andrew observes on his 4*p3 700MHz). Given the age of the hardware and the fact that I am not seeing it on other workloads or on modern hardware it might just be due to a small scheduling difference on this configuration."

A potential bug reported against the Completely Fair Scheduler suggested that it was causing a network slowdown, measured with the 'Iperf' bandwidth performance benchmarking tool. The performance hit was quickly tracked to the previously discussed changes in how CFS handles sched_yield(). When it was suggested that this was a bug in the new process scheduler, Ingo explained:

"I had a quick look at the source code, and the reason for that weird yield usage was that there's a locking bug in iperf's 'Reporter thread' abstraction and apparently instead of fixing the bug it was worked around via a horrible yield() based user-space lock."

He then submit a small patch to fix the bug and remove the call to sched_yield() resulting in, "iperf uses _much_ less CPU time. On my Core2Duo test system, before the patch it used up 100% CPU time to saturate 1 gigabit of network traffic to another box. With the patch applied it now uses 9% of CPU time." He added playfully, "sched_yield() is almost always the symptom of broken locking or other bug. In that sense CFS does the right thing by exposing such bugs =B-)" Stephen Hemminger pointed out that a similar patch had been submitted to the Iperf project last month as it caused an identical problem with FreeBSD's scheduler.