FreeBSD Status Report

Introduction

   This report covers FreeBSD related projects between June and October
   2006. This includes the conclusion of this year's Google Summer of
   Code with 13 successful students. Some of last year's and the current
   SoC participants have meanwhile joined the committer ranks, kept
   working on their projects, and improving FreeBSD in general.

   This year's EuroBSDCon in Milan, Italy has meanwhile published an
   exciting program. Many developers will be there to discuss these
   current and future projects at the Developer Summit prior the
   conference. Next year's conference calendar has a new entry - in
   addition to the now well established BSDCan in Ottawa - AsiaBSDCon
   will take place in Tokyo at the begining of March.

   As we are closing in on FreeBSD 6.2 release many bugs are being fixed
   and new features have been MFCed. On the other hand a lot of the
   projects below already are focusing on FreeBSD 7.0 and promise a lot
   of exciting news and features to come.

   Thanks to all the reporters for the excellent work! We hope you enjoy
   reading.
     _________________________________________________________________

Google Summer of Code

     * Analyze and Improve the Interrupt Handling Infrastructure
     * Bundled PXE Installer
     * Gvirstor
     * IPv6 Stack Vulnerabilities
     * Jail Resource Limits
     * Nss-LDAP importing and nsswitch subsystem improvement
     * Porting the seref policy and setools to SEBSD
     * Porting Xen to FreeBSD
     * SNMP monitoring (BSNMP)
     * Summer of Code Summary
     * Update of the Linux compatibility environment in the kernel

Projects

     * CScout on the FreeBSD Source Code Base
     * DTrace
     * Embedded FreeBSD
     * FreeSBIE
     * GJournal
     * iSCSI Initiator
     * Porting ZFS to FreeBSD
     * Summer of FreeBSD security development
     * TrustedBSD Audit
     * USB

FreeBSD Team Reports

     * FreeBSD Security Officer and Security Team
     * Ports Collection
     * Release Engineering
     * The FreeBSD Foundation

Network Infrastructure

     * Bridge Spanning Tree Protocol Improvements
     * FAST_IPSEC Upgrade
     * Highly improved implementations of sendfile(2), sosend_*() and
       soreceive_stream()
     * SCTP Integration
     * TSO - TCP Segmentation Offload committed

Kernel

     * Gvinum improvements
     * MMC/SD Support
     * Sound Subsystem Improvements

Documentation

     * Chinese (Simplified) Project
     * Hungarian translation of the webpages

Userland Programs

     * Libelf
     * OpenBSD dhclient

Architectures

     * CPU Microcode Update Software
     * FreeBSD/arm on Atmel AT91RM9200
     * Sun Niagara port
     * Xen Port

Ports

     * Enlightenment DR17 support in the ports tree
     * FreshPorts
     * Improving FreeBSD Ports Collection Infrastructure
     * OCaml language support in ports

Miscellaneous

     * AsiaBSDCon 2007
     * BSDCan 2007
     * EuroBSDCon 2006
     * FreeBSD Multimedia Resources List
     _________________________________________________________________

Analyze and Improve the Interrupt Handling Infrastructure

   URL: http://wikitest.freebsd.org/Interrupts

   Contact: Paolo Pisati <pisati@FreeBSD.org>
   Contact: John Baldwin <jhb@FreeBSD.org>

   This project consisted in the improvement of the Interrupt Handling
   System in FreeBSD: while retaining backward compatibility with the
   previous models (FAST and ITHREAD), a new method called 'Interrupt
   filtering' was added. With interrupt filtering, the interrupt handler
   is divided into 2 parts: the filter (that checks if the actual
   interrupt belong to this device) and the ithread (that is scheduled in
   case some blocking work has to be done). The main benefits of
   interrupt filtering are:
     * Feedback from filters (the system finally knows if any handler has
       serviced an interrupt or not, and can react consequently).
     * Lower latency/overhead for shared interrupt line.
     * Previous experiments with interrupt filtering showed an increase
       in performance against the plain ithread model

   Moreover, during the development of interrupt filtering, some MD
   dependent code was converted into MI code, PPC was fixed to support
   multiple FAST handlers per line and an interrupt stray storm detection
   logic was added. While the framework is done, there are still machine
   dependent bits to be written (the support for ppc, sparc64, arm and
   itanium has to be written/reviewed) and a serious analysis of the
   performance of this model against the previous one is a
   work-in-progress
     _________________________________________________________________

AsiaBSDCon 2007

   URL: http://www.asiabsdcon.org/

   Contact: Hiroki Sato <hrs@freebsd.org>
   Contact: George Neville-Neil <gnn@freebsd.org>
   Contact: <secretary@asiabsdcon.org>

   Web site is up and we're soliciting papers and presentations. Some
   tutorials are already scheduled. Email secretary@asibsdcon.org if you
   have questions or submissions.

Open tasks:

    1. Send in more papers!
     _________________________________________________________________

Bridge Spanning Tree Protocol Improvements

   Contact: Andrew Thompson <thompsa@FreeBSD.org>

   Work is almost finished to implement the Rapid Spanning Tree Protocol
   (RSTP) which supersedes Spanning Tree Protocol (STP). RSTP has a much
   faster link failover time of around one second compared to 30-60
   seconds for STP, this is very important on modern networks. The code
   will be posted shortly for testing and feedback.
     _________________________________________________________________

BSDCan 2007

   URL: http://www.bsdcan.org/

   Contact: Dan Langille <dan@langille.org>

   The dates for BSDCan 2007 has been set: 11-12 May 2007. As is usual,
   BSDCan will be held at University of Ottawa, with two days of
   tutorials prior to the conference starting.

   The call for papers will go out in mid December. Start thinking about
   your submissions now!
     _________________________________________________________________

Bundled PXE Installer

   URL: http://wikitest.freebsd.org/MarkusBoelter

   Contact: Markus Boelter <m@FreeBSD.org>
   Contact: Paul Saab <ps@FreeBSD.org>

   For me, the Google Summer of Code was a new and very exciting
   experience. I got actively involved in doing Open Source Software and
   giving something back to the community. Facing some challenges within
   the project forced me to look behind the scenery of FreeBSD. The
   result was a better understanding of the overall project. Working with
   a lot of developers directly also gave a very special spirit to the
   Google Summer of Code.

   I really enjoyed the time and will continue to work on the project
   after the deadline. For me, it was a great chance to get involved in
   active development and not just some scripts and hacks at home.
   Getting paid for the work was just a small part of the overall
   feeling.

   Thanks to the people at the FreeBSD Project and Google for the really,
   really great time!
     _________________________________________________________________

Chinese (Simplified) Project

   URL: http://cnsnap.cn.FreeBSD.org/zh_CN/
   URL: http://cnsnap.cn.FreeBSD.org/doc/zh_CN.GB2312/

   Contact: Xin LI <delphij@FreeBSD.org>

   In the previous quarter we primarily focused on overall quality of the
   translation rather than just increasing the number of translations,
   and we have strived to make sure that these translated stuff are
   up-to-date with their English revisions. Also, we have merged the
   translated website into the central repository.

   In the next quarter we will focus on developing documentation that
   will help to attract more developers.

Open tasks:

    1. Translate more development related documentation.
    2. Review more of the currently translated documentation.
     _________________________________________________________________

CPU Microcode Update Software

   Contact: Stanislav Sedov <stas@FreeBSD.org>

   Last month I was working on a driver/module to update the microcode of
   Intel or AMD CPUs that support having their microcode updated. As you
   might know these processors are microcode-driven and this firmware can
   be updated. Intel(R) often releases microcode updates, and AMD(R)
   updates can be found in BIOS programs. The work is almost finished
   now, I just need to find a bit of time to test it on AMD64 systems and
   perform some code cleanup. The driver also provide a way for userland
   programs to access the Machine Specific Registers (MSR) and CPUID info
   for a certain cpu. This will allow some programs like x86info to
   provide more accurate information about cpus in SMP systems and make
   assumptions based on the contents of the MSR.

   Thanks to John Baldwin, Kostik Belousov, John-Mark Gurney and Divacky
   Roman for helping during development.

Open tasks:

    1. Perform testing on the AMD64-based systems.
    2. Write manpage.
    3. Code cleanup/checks.
     _________________________________________________________________

CScout on the FreeBSD Source Code Base

   URL: http://wikitest.freebsd.org/CScout

   Contact: Diomidis Spinellis <dds@FreeBSD.org>

   CScout is a refactoring editor and source code browser for collections
   of C code. The aim of the project is to make it easy for FreeBSD
   developers to use CScout and to improve the FreeBSD source code
   quality through CScout-based queries and refactorings.

   CScout was first applied to the FreeBSD kernel in 2003. Its
   application at that point involved substantial tinkering with the
   build system. The version released in October 2006 makes the running
   of CScout on the three Tier-1 architectures a fairly straightforward
   procedure. The current version can also draw a number of call graphs;
   this might help developers better understand foreign code.

Open tasks:

    1. Use CScout to locate problematic code areas (for example unused or
       too liberaly visible objects).
    2. Use CScout to globaly rename identifiers in a more consistent
       fashion.
    3. Apply CScout to the userland code.
    4. Identify CScout extensions that would help us improve the quality
       of our code.
    5. Arrange for the continous availability of a live CScout kernel
       session on the current version of the source code.
     _________________________________________________________________

DTrace

   Contact: John Birrell <jb@freebsd.org>

   Progress this month has been limited due to my sea-change, moving
   house to the country.

   Sun's OpenSolaris developers have followed through and released the
   DTrace test suite as part of the OpenSolaris distribution.

   jkoshy@'s work on libbsdelf is nearing feature completion for DTrace
   and will make life easier in FreeBSD for DTrace, given that we have
   more architectures to support than Sun has.

   The FreeBSD project has made available a dual processor AMD64 machine
   for DTrace porting.

   I am currently working through the diffs between the DTrace project in
   P4 and -current, committing files to -current if they are ready,
     _________________________________________________________________

Embedded FreeBSD

   URL: http://www.embeddedfreebsd.org/

   Contact: George Neville-Neil <gnn@freebsd.org>

   Moved the HTML pages into the project CVS tree.

Open tasks:

    1. Setup the web site to be served from projects CVS so that it can
       be updated by others.
    2. Complete the ARM port.
    3. Work on the MIPS port.
    4. Update the documentation to include common tasks for embedded
       engineers.
     _________________________________________________________________

Enlightenment DR17 support in the ports tree

   Contact: Stanislav Sedov <stas@FreeBSD.org>

   Integration of the new innovative e17 window manager into the ports
   tree is almost completed. A lot of new e17-related applications was
   ported, all old ports were updated to the latest stable cvs snapshot.
   The special framework (bsd.efl.mk) was created to support the whole
   thing and simplify the creation of dependent ports. I'll commit the
   changes in the days before the ports freeze.

   Thanks to Sergey Matveychuk (sem@) for providing a machine to place
   CVS snapshots on. Without his help it will be impossible.

Open tasks:

    1. Port Entrance (xdm-like app, but very appealing).
    2. Port Net and Wlan e17 module.
    3. Develop FreeBSD-specific e17 apps/modules to use The Ports
       Collection, system configs, etc.
     _________________________________________________________________

EuroBSDCon 2006

   URL: http://www.eurobsdcon.org/
   URL: http://www.eurobsdcon.org/register/

   Contact: EuroBSDCon Organizing Committee <info@eurobsdcon.org>

   EuroBSDCon 2006 is taking place in Milan (Italy), from the 10th to the
   12th of November.

   EuroBSDCon represents the biggest gathering for BSD developers from
   the old continent, as well as users and passionates from around the
   World. It is also a chance to share experiences, know-how, and
   cultures.

   The program is rich in talks about FreeBSD, with topics ranging from
   "How the FreeBSD ports collection works" to "Interrupt Filtering in
   FreeBSD". This means that both the novice and the hacker can enjoy the
   conference.

   Registration is open. The EuroBSDCon Organizing Committee hopes to see
   you in Milan.
     _________________________________________________________________

FAST_IPSEC Upgrade

   URL: www.freebsd.org/~gnn/fast_ipv6.patch

   Contact: George Neville-Neil <gnn@freebsd.org>
   Contact: Bjoern Zeeb <bz@freebsd.org>

   First working version of code. Does not pass all TAHI tests, but does
   pass packets correctly and does not panic.

Open tasks:

    1. More testing of the patch needed.
     _________________________________________________________________

FreeBSD Multimedia Resources List

   URL: http://www.mavetju.org/unix/multimedia.php
   URL: http://www.mavetju.org/unix/multimedia-rss.php

   Contact: Edwin Groothuis <edwin@FreeBSD.org>

   I have setup the FreeBSD Multimedia Resources List, a one-stop-shop
   for FreeBSD related podcasts, vodcasts and audio/video resources.
   Hopefully this list will make it easier for people to find and keep up
   to date with these recordings. The overview is available as a normal
   HTML page and as an XML/RSS feed.

   The ultimate goal is to have this list to reside under the
   www.FreeBSD.org umbrella.
     _________________________________________________________________

FreeBSD Security Officer and Security Team

   URL: http://www.freebsd.org/security/
   URL:
   http://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributors/staff
   -listing.html#STAFF-SECTEAM
   URL: http://vuxml.freebsd.org/

   Contact: Security Officer <security-officer@FreeBSD.org>
   Contact: Security Team <security-team@FreeBSD.org>

   In the time since the last status report, six security advisories have
   been issued concerning problems in the base system of FreeBSD; of
   these, five problems were in "contributed" code, while one was in code
   maintained within FreeBSD. The Vulnerabilities and Exposures Markup
   Language (VuXML) document has continued to be updated by the Security
   Team and Ports Committers documenting new vulnerabilities in the
   FreeBSD Ports Collection; since the last status report, 57 new entries
   have been added, bringing the total up to 814.

   The following FreeBSD releases are supported by the FreeBSD Security
   Team: FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, FreeBSD 5.5, FreeBSD
   6.0, and FreeBSD 6.1. The respective End of Life dates of supported
   releases are listed on the web site; of particular note, FreeBSD 5.3
   and FreeBSD 5.4 will cease to be supported at the end of October 2006,
   while FreeBSD 6.0 will cease to be supported at the end of November
   2006 (or possibly a short time thereafter in order to allow time for
   upgrades to the upcoming FreeBSD 6.2).
     _________________________________________________________________

FreeBSD/arm on Atmel AT91RM9200

   Contact: Warner Losh <imp@freebsd.org>
   Contact: Olivier Houchard <cognet@freebsd.org>

   The FreeBSD/arm port has grown support for the Atmel AT91RM9200.
   Boards based on this machine are booting to multiuser off either NFS
   or an SD card. The onboard serial ports, PIO, ethernet and SD/MMC card
   controllers are well supported. Support for the SSC, IIC and SPI flash
   parts in the kernel will be forthcoming shortly.

   In addition to normal kernel support, the port includes a boot loader
   that can initialize memory and boot off IIC eeprom, SPI DataFlash,
   BOOTP/TFTP and SD memory cards.

   The port will be included in forth coming commercial products.

Open tasks:

    1. Add support for other members of the AT91 family of arm9
       processors.
    2. Finish support for AT45D* flash parts.
    3. Finish support for USB ports
    4. Write support for USB Device functionality
     _________________________________________________________________

FreeSBIE

   URL: http://www.FreeSBIE.org
   URL: http://liste.gufi.org/mailman/listinfo/freesbie
   URL: http://people.freebsd.org/~matteo/GMV/GMVAnnounce.txt

   Contact: FreeSBIE Staff <staff@FreeSBIE.org>
   Contact: Matteo Riondato <matteo@FreeBSD.org>

   FreeSBIE is a FreeBSD based LiveCD.

   On August 19th, Matteo Riondato, a member of the FreeSBIE staff,
   released an unofficial ISO, codename FreeSBIE GMV, based on FreeBSD
   -CURRENT (read the Announcement to download it). This is supposed to
   be the first in a series of four ISOs that will end up with the
   release of FreeSBIE 2.0. Matteo is now working on another ISO,
   codename FreeSBIE LVC, which is scheduled to be released October 12th.

   FreeSBIE 2.0 will be based on FreeBSD 6.2-RELEASE and will hopefully
   be released at EuroBSDCon 2006 in Milan. It will be available for the
   i386 and AMD64 platforms.

Open tasks:

    1. Test the released ISO in preparation for the release.
    2. Suggest software to include in the ISO.
    3. Submit a simple and clear but complete fluxbox configuration.
     _________________________________________________________________

FreshPorts

   URL: http://www.freshports.org/

   Contact: Dan Langille <dan@langille.org>

   The new 2U server mentioned in the last report now has a collection of
   Raptor drives in a RAID-10 configuration. Thanks to very generous
   donations from the community, I purchased eight of these drives at
   very good prices. The server will be deployed in the next few weeks.

   There has been quite a bit of work since the last report in June. Some
   highlights include:
     * New news feed formats, including newsfeeds for your watch list.
     * Better pages caching for faster response.
     * Sanity Test Failures now available online.
     * Ability to search for all commits (ports, doc, src, etc) under a
       given point in the tree.

   For more detail, please review the FreshPorts Blog .
     _________________________________________________________________

GJournal

   URL: http://people.freebsd.org/~pjd/patches/gjournal_20060930.patch
   URL: http://people.freebsd.org/~pjd/patches/gjournal6_20060930.patch

   Contact: Pawel Jakub Dawidek <pjd@FreeBSD.org>

   GJournal seems to be finished. I fixed the last serious bug and it is
   now stable and reliable in our tests. I'm planning to commit it really
   soon now.

   The work was sponsored by home.pl
     _________________________________________________________________

Gvinum improvements

   URL:
   http://folk.ntnu.no/lulf/patches/freebsd/gvinum/gvinum_all_current.dif
   f

   Contact: Ulf Lilleengen <lulf@pvv.ntnu.no>

   I thought that since I sent a status report the last time, I might as
   well send one now.

   Since the last status report I have done work on several of the
   remaining commands as attach, detach, and finally the concat command
   to be able to create concatenated volumes with one easy command. The
   mirror and stripe commands are the next step after this.

   The most important thing I've been working on is maybe the
   implementation of drivegroups. I have posted a bit information on this
   mailinglists, but basically, it's a way to group drives with the same
   configuration. This way, you can make many commands operate on groups
   instead of drives, and the group-abstraction will handle how the
   underlying subdisks are created on the drives. In the future one will
   be able to move groups to different machines, etc.

   I've created a patch of all my work that is not in HEAD yet here (this
   is a snapshot of my developement branch, so how thing's are done might
   be changed quite fast):
   http://folk.ntnu.no/lulf/patches/freebsd/gvinum/gvinum_all_current.dif
   f

   Be aware that a there will probably be bugs in the code, so don't use
   it in production yet!

   Thanks to Greg Lehey for offering to help me on getting this into CVS,
   and all feedback on this has been good.

Open tasks:

    1. Remaining components, mirror, stripe and some info commands.
     _________________________________________________________________

Gvirstor

   URL: http://wiki.freebsd.org/gvirstor

   Contact: Ivan Voras <ivoras@freebsd.org>

   Gvirstor is a GEOM class providing virtual ("overcommit") storage
   devices larger than physical available storage, with possibility to
   add physical storage on-line when the need arises. Current status is
   that it's done and waiting commit to HEAD, scheduled for some time
   after 6.2 is released.

Open tasks:

    1. The project is in need of testing! If you have the equipment and
       time, please give it a try so possible bugs can be fixed before it
       goes into -CURRENT.
     _________________________________________________________________

Highly improved implementations of sendfile(2), sosend_*() and
soreceive_stream()

   URL:
   http://lists.freebsd.org/pipermail/freebsd-current/2006-September/0659
   97.html
   URL:
   http://lists.freebsd.org/pipermail/freebsd-current/2006-September/0661
   99.html
   URL:
   http://people.freebsd.org/~andre/sendfile+sosend+soreceive-20061006.di
   ff

   Contact: Andre Oppermann <andre@freebsd.org>

   The addition of TSO (TCP Segmentation Offload) has highlighted some
   shortcomings in the sendfile(2) and sosend_*() kernel implementations.

   The current sendfile(2) code simply loops over the file, turns each 4K
   page into an mbuf and sends it off. This has the effect that TSO can
   only generate 2 packets per send instead of up to 44 at its maximum of
   64K. kern_sendfile() has been rewritten to work in two loops, the
   inner which turns as many pages into mbufs as it can -- up to the free
   send socket buffer space. The outer loop then drops the whole mbuf
   chain into the send socket buffer, calls tcp_output() on it and then
   waits until 50% of the socket buffer are free again to repeat the
   cycle. This way tcp_output() gets the full amount of data to work with
   and can issue up to 64K sends for TSO to chop up in the network
   adapter without using any CPU cycles. Thus it gets very efficient
   especially with the readahead the VM and I/O system do.

   Looking at the benchmarks we see some very nice improvements: 181%
   faster with new sendfile vs. old sendfile (non-TSO), 570% faster with
   new sendfile vs. old sendfile (TSO).

   The current sosend_*() code uses a sosend_copyin() function that loops
   over the supplied struct uio and does interleaved mbuf allocations and
   uiomove() calls. m_getm() has been rewritten to be simpler and to
   allocate PAGE_SIZE sized jumbo mbuf clusters (4k on most
   architectures). m_uiotombuf() has been rewritten to use the new
   m_getm() to obtain all mbuf space in one go. It then loops over it and
   copies the data into the mbufs by using uiomove(). sosend_dgram() and
   sosend_generic() have been changed to use m_uiotombuf() instead of
   sosend_copyin().

   Looking at the benchmarks we see some very nice improvements: 290%
   faster with new sosend vs. old sosend (non-TSO), 280% faster with new
   sosend vs. old sosend (TSO).

   Newly written is a specific soreceive_stream() function for stream
   protocols (primarily TCP) that does only one socket buffer lock per
   socket read instead of one per data mbuf copied to userland. When
   doing netperf tests with WITNESS (full lock tracking and validation
   enabled) the receive performance increases from ~360Mbit/s to
   ~520Mbit/s. Without WITNESS I could not measure any statistically
   significant improvement on a otherwise unloaded machine. The reason is
   two-fold: 1) per packet we do a wakeup and readv() is pretty much as
   many times as packets come it, thus the general overhead dominates; 2)
   the packet input path has a pretty high overhead too. On heavily
   loaded machines which do a lot of high speed receives a performance
   increase should be measureable.

   The patches are scheduled to be committed to FreeBSD-current at end of
   October or early November 2006.

   This work was sponsored by the TCP/IP Optimization Fundraiser 2005.
     _________________________________________________________________

Hungarian translation of the webpages

   URL: http://gabor.t-hosting.hu/data/hu/

   Contact: G