BitKeeper was first utilized by a Linux project in December of 1999, when it was employed by the Linux PowerPC project. Then in February of 2002, Linux creator Linus Torvalds decided that BitKeeper was "the best tool for the job" and started using it to manage the mainline kernel, an event that received much attention in the free and open source communities [story], and beyond. BitMover, the company behind BitKeeper, was founded by its current CEO, Larry McVoy [interview], who originally conceived of BitKeeper as a tool to keep Linus from getting burnt out by the growing task of managing the Linux Kernel. Since Linus began using the tool three years ago, the pace of Linux kernel development has doubled [story].

There are two definitions for the word "free" that are commonly used to describe software. The first is "Free as in Freedom", and the other is "Free as in Free Beer". BitKeeper was made available freely under the latter definition, allowing free and open source software developers to use the tool without having to pay any money. It was provided under the agreement that anyone actively using the free tool would not develop a competing product at the same time. In other words, the aim was to provide a tool that could be freely used, but not freely cloned. At the same time, a more advanced version of BitKeeper has been sold commercially, and both products remain the intellectual property of BitMover.

A vocal group has long protested Linus' use of BitKeeper, considering Linux the free and open source flagship product. GNU Project founder Richard Stallman [interview] is among the protestors, harshly criticizing Linus' decision to use a non-free (as in freedom) tool [story]. However, most acknowledge that no free tool currently exists that is as powerful as BitKeeper, offering the ability to perform truly distributed development. Attempts to reverse engineer some of BitKeeper's features have lead to repeated cautions by BitMover. Most recently two such reverse engineering attempts have contributed to BitMover's decision to end the development and availability of the free BitKeeper product.

"Best Tool For The Job"
Shortly after Linus began using BitKeeper, he described it as the "Best Tool For The Job". Unlike most tools that require a central repository to house the master copy of whatever data is under version control, BitKeeper allows a truly distributed system in which everybody owns their own master copy. Furthermore, most version control systems look at individual files, and even individual sections of the same files, as indepent entities. If a change is made in file A, and another change is then made in file B, the version control software will allow someone to only merge the changes in file B. With BitKeeper, this is not possible. Instead, to get the changes in file B, you must first get the changes in file A. This design ensures that the two repositories are exact duplicates.

Larry McVoy explained that when Linus first began using BitKeeper, he struggled with and complained about this feature, "suddenly Linus couldn't cherry pick individual patches out of a larger set of changes." Jeff Garzik details this conceptual difference in his bk-kernel-howto document included with the kernel. Prior to using BitKeeper, Linus would review every single patch detail by detail, and he would pull out just the pieces that he wanted. But with BitKeeper, this became more difficult. Larry explained that this resulted in more trust being placed on some of the various subsystem maintainers, with Linus beginning to look at some patches category by category rather than line by line. Those maintainers that Linus felt he could trust more required less review, and at the same time those maintainers that Linus felt he could trust less received more review. Effectively, much of the effort involved in assuring high quality was delegated, and ultimately this led to doubling the pace of kernel development.

Free Versus Free
In spite of the increase in productivity, a vocal group continued to protest Linus' use of a tool that was not open source, nor "free as in freedom". Efforts to provide BitKeeper-like functionality in open source products began in earnest. This lead to more flame wars on the Linux Kernel Mailing List with Larry trying to explain that he was happy to let Linus and other developers use the product for kernel development, but not for reverse engineering. Most recently, two such reverse engineering efforts led BitMover to question the free portion of their business model. Worse, one of these efforts was inadvertandly funded by the OSDL, Linus Torvalds' place of employement.

Larry explained that a contracter still under pay from OSDL for an unrelated project was also actively working on reverse engineering the BitKeeper protocol. Discussion began about five weeks ago to try and resolve the situation, getting so far as to obtain a verbal agreement that the individual would stop his efforts. After that time, however, it turned out that the reverse engineering effort had continued. Although OSDL wasn't directly paying for the reverse engineering effort, they were still employing someone who was actively developing a competing product, something the free BitKeeper license doesn't allow. Larry added, "OSDL had a chance to resolve this issue, but instead shrugged their shoulders and said 'it's not my problem'". It became the proverbial straw that broke the camel's back. Weighing this and earlier transgressions against the $500,000 dollars a year BitMover spends developing and supporting the free BitKeeper tool, the tough decision was made to phase out support and availability of their free product.

What Went Wrong
Discussing what led to this decision, Larry explained, "this is really an open source community problem and I have to say that the open source community couldn't have failed more than they have." He pointed out that as a long-time open source fanatic and the CEO of BitMover, "we represent as open-source friendly a commercial organization as you are *ever* going to see", cautioning that the events that have caused BitMover to phase out its free product could also result in other companies never even bothering to make products available on Linux. "back at Sun they had a saying 'it's the apps, stupid'. Which meant that all the 'my OS is better than your OS' rhetoric was nothing compared to having more applications on your platform than the other guy's platform. That's true for Linux as well and the point is that no company is going to port their applications to a platform who's stated goal and track record is to reverse engineer everything they find useful. At some point the open source world needs to either decide they'll tolerate commercial software and respect the fact that commercial companies are entitled to make money off their efforts or step up and take on the challenge of providing open source versions of *every* application."

He continued, "our position is that we don't think we have any chance of changing how the 'open source community' behaves. Unlike the Marine corp, the open source community is more than willing to ignore their bad apples as 'not my problem' (the Marine corp punishes the group for the behavior of the bad apples, pretty soon there are no bad apples). Maybe that will get fixed some day but until then we have to do what makes sense for our business and letting the open source guys put us out of business doesn't make sense."

Larry also noted that some people could view the end of a free BitKeeper product as a failure, but that this wasn't the case. Linus pointed out that, "three years of using BitKeeper has made some profound improvements to the workflow", describing how instead of focusing on patch by patch changes he's now able to work with other trusted developers at a higher and more efficient level.

The decision was not an easy one. "I was trying to help and it has been heartbreaking to watch parts of the community lash back at me because I couldn't figure out a politically correct way to help," Larry explained. "You have no idea how miserable that has been."

What Next
Over the next three months, BitMover intends to phase out the free BitKeeper product. Some money has been set aside to provide commercial licenses for certain kernel developers, however Linus Torvalds is not one of them. Larry suggested, "if Linus and Andrew and the others moved elsewhere, we'd glady comp them licenses", referring to their current employment with OSDL.

Beyond this, it has become obvious to all involved that BitMover no longer gains back what it spends supporting the free product. In the beginning the exchange of the service and product for marketing gains and bug reports was enough. Now that this is no longer true, Linus, not interested in a one sided deal even when he's on the winning side, decided for this and other reasons to migrate away from BitKeeper and to begin looking for an alternative.

Larry noted that the kernel tree will continue to be tracked by BitKeeper, as many kernel developers have been commercially licensing the product for that purpose. This includes employees of many large companies who actively contribute to Linux development such as IBM, Intel, HP, Nokia and Sun as well as many smaller companies.

As for what tool will replace BitKeeper to handle Linux kernel development, no decision has been made yet. Several tools have been tested, but nothing currently has been selected, as so far nothing offers sufficient performance and functionality. Larry noted that Linus tried monotone, but a simple import of flat files without versioning information took 2 hours, far too slow. The same was true for Darcs. Eventually however, some tool will be selected and further developed to meet the needs of the kernel developers.

Larry pointed out that such a tool would only need to run on Linux, having no need to support other platforms such as Solaris or Windows, and thus could take advantage of certain Linux-specific features allowing it to perform "blindingly fast". In fact, Linus has already begun working on a simplistic tool that takes advantage of Linux performance features. This effort has been aided by Larry, who added that it was also good for BitMover, as a Linux-only tool would not be competition to BitKeeper, whose largest market is on Windows.

BitKeeper versus the Open Source Replacement
By phasing out the free BitKeeper product, it is guaranteed that some other project is going to step forward to take its place as the version control system used by the Linux kernel. When asked if he was concerned about this resulting in the creation of a project that ultimately competes with BitKeeper, Larry replied, "yes, of course. We'd be idiots to not be." However, he then went on to point out some reasons that this was unlikely. In maintaining two products, he was suprised to learn that the needs of the open source community was much different than the needs of the commercial community. Certainly there was some overlap, but they found that the two communities were pushing them in different directions.

A specific example is in binary managment. There are not many binary files in the Linux kernel distribution, and what few there are don't get frequently modified. "For example", Larry said, "there may be a jpeg image which is a logo, and it may be updated once every year or so, but there's not a great need for complicated binary managment." Conversely, in the commercial community binary managment is critical. For example, someone may be tracking a 1 MB word document that goes through hundreds of revisions resulting in consuming 1 GB of space. BitKeeper is focused on improving this functionality as it is common for commercial uses, whereas an open source solution concerned with the Linux kernel wouldn't have such a need.

Furthermore, Larry pointed out that there is no "killer feature" that makes BitKeeper such a great tool. Instead, it is a collection of thousands of little features altogether which make it so powerful. He added that he hopes an open source solution will become viable soon, as until that happens features will inadvertantly be passed from commercial BitKeeper users to the open source community. Already, in the discussions about what will be needed to replace BitKeeper, the wishlists come straight out of BitKeeper's feature set.

Last Free BitKeeper Release
In a post to the Linux Kernel mailing list in February of 2005, Larry discussed a 16 bit limitation of the existing free product [story]. With nearly 64,000 changesets in the mainline kernel tree, future development will quickly exceed this limitation. For that reason, it is likely that BitMover will provide one final release of its free BitKeeper product, allowing kernel developers a graceful transition. By the end of July, the goal is to have completed the migration, thereby terminating the free product and focusing fully on the commercial product.

As the two versions of the product tended to have greatly divergent requirements, the decision to drop the free product will benefit users of the commercial product. "On a scale of one to ten", Larry explained, "I think we're a four. Of course all the other products out there are a three or less, but they don't realize it. We have a roadmap of features that will take us to maybe eight out of ten. Those will keep us busy for the next three to five years and we're very excited to be working on those problems."

Summary:
The three years that Linux was under BitKeeper source control were largely beneficial to the Linux kernel development process. It taught good source control habits, and offered kernel developers a more efficient way to manage distributed development.

The debate regarding "Free as in Freedom" versus "Free as in Free Beer" is far from over, and we are certain to see lengthy debates on the Linux Kernel Mailing List as kernel developers migrate away from BitKeeper. There is no obvious replacement today, but it is only a matter of time until a project steps forward to satisfy the needs of kernel developers. Now that the kernel has been maintained under a quality source control product, it is quite likely that it will remain so, inventing a solution if necessary, for as Plato so aptly said, "necessity is the mother of invention."

Translations:

• Russian (OSRC.info)