KernelTrap - ext4

Linux: Properly Creating And Testing Patches

Jeremy — Fri, 09 Apr 2010 20:18:27 +0000

"If you're wondering why I'm taking a long time to respond to your patches,", began Theodore Ts'o on the linux-ext4 mailing list, in a thread that offered much insight into how and why to properly submit and test patches. "Patches that are accepted into mainline should do one and only one thing," Ted continued, "so if someone suggests that you make changes to your submitted patch, ideally what you should do is to resubmit the patch with the fixes --- and not submit a patch which is a delta to the previous one." He also noted that patch submitters often greatly outnumber maintainers dictating a higher standard of quality, "consider that for some maintainers, there may be 10 or 20 or 30 or more patch submitters in their subsystem. With that kind of submitter-to-maintainer ratio, the patch submitter simply has to do much more of the work, since otherwise the subsystem maintainer simply can't keep up."

Ted went on to acknowledge, "I happen to believe that we need to encourage newcomers to the kernel developer community, and so I spend more time mentoring people who are new to the process." He noted that his time was finite however, and that patches are accepted more quickly when they are easy to review and integrate. Regarding the filesystem for which the patches had been submitted, he added, "Ext4 is actually quite stable at this point. Very large numbers of people are using it, and most users are quite happy." For this reason, he pointed out that it is even more critical that the patches merged be of high quality. That said, he continued, "there is no such thing as code which is not buggy. For any non-trivial program, it's almost certain there are bugs. [...] Ext4 is not exempt from these fundamental laws of software engineering. 'Code is always buggy until the last user of the program dies'." He tied this back to the importance of testing patches before submitting, "keep in mind that the maxim that code which is not buggy also applies to your patches."

ext4 2.6.25 Merge Plans

Jeremy — Tue, 22 Jan 2008 20:54:53 +0000

"The following patches have been in the -mm tree for a while, and I plan to push them to Linus when the 2.6.25 merge window opens," began Theodore Ts'o, offering the patches for review before they are merged. He explained that the patches introduce some of the final changes to the ext4 on-disk format, "ext4, shouldn't be deployed to production systems yet, although we do salute those who are willing to be guinea pigs and play with this code!" He continued:

"With this patch series, it is expected that [the] ext4 format should be settling down. We still have delayed allocation and online defrag which aren't quite ready to merge, but those shouldn't affect the on-disk format. I don't expect any other on-disk format changes to show up after this point, but I've been wrong before.... any such changes would have to have a Really Good Reason, though."

ext4 2.6.24 Merge Plans

Jeremy — Thu, 04 Oct 2007 08:53:36 +0000

"I've just released the 2.6.23-rc9-ext4-1. It collapses some patches in preparation for pushing them to Linus, and adds some of the cleanup patches that had been incorporated into Andrew's broken-out-2007-10-01-04-09 series," announced Theodore Ts'o. He also noted of the current ext4 git tree, "it also has some new development patches in the unstable (not yet ready to push to mainline) portion of the patch series." In an earlier thread Theodore posted a series of patches specifically intended for inclusion in the upcoming 2.6.24 kernel. Included in the patch series was a patch for improving fsck performance, "in performance tests testing e2fsck time, we have seen that e2fsck time on ext3 grows linearly with the total number of inodes in the filesytem. In ext4 with the uninitialized block groups feature, the e2fsck time is constant, based solely on the number of used inodes rather than the total inode count." The patch included an explanation of how the feature works, enabled through a mkfs option:

"With this feature, there is a a high water mark of used inodes for each block group. Block and inode bitmaps can be uninitialized on disk via a flag in the group descriptor to avoid reading or scanning them at e2fsck time. A checksum of each group descriptor is used to ensure that corruption in the group descriptor's bit flags does not cause incorrect operation."

UBIFS Writeback

Jeremy — Sat, 29 Sep 2007 11:23:31 +0000

UBIFS is described as, "a new flash file system which is designed to work on top of UBI." It has replaced the JFFS3 project, a choice explained on the project webpage, "we have realized that creating a scalable flash file system on top of bare flash is a difficult task, just because the flash media is so problematic (wear-leveling, bad eraseblocks). We have tried this way, and it turned out to be that we solved media problems, instead of concentrating on file system issues. So we decided to split one big and complex tasks into 2 sub-tasks: UBI solves the media problems, like bad eraseblocks and wear-leveling, and UBIFS implements the file system on top. And now finally, we may concentrate on file-system issues: implementing write-back caching, multi-headed journal, garbage collector, indexing information management and so on. There are a lot of FS problems to solve - orphaned files, deletions, recoverability after unclean reboots and so on."

In a recent posting to the lkml, Artem Bityutskiy noted that UBIFS has to take into account that there is a small amount of unused block space at the ends of eraseblocks, and the size of pages written to disk are smaller than they are in memory as the filesystem performs compression. "So, if our current liability is X, we do not know exactly how much flash space (Y) it will take. All we can do is to introduce some pessimistic, worst-case function Y = F(X). This pessimistic function assumes that pages won't be compressible, and it assumes worst-case wastage." The calculation is necessary as even though data is not written immdiately to the flash device, it's important to be able to inform the application writing data if there's not enough space left. "So my question is: how can we flush _few_ oldest dirty pages/inodes while we are inside UBIFS (e.g., in ->prepare_write(), ->mkdir(), ->link(), etc)?"

Improving fsck Speeds in ext4

Jeremy — Wed, 19 Sep 2007 06:40:27 +0000

"In [the first pass] of e2fsck, every inode table in the fileystem is scanned and checked, regardless of whether it is in use," Avantika Mathur began. "This is the most time consuming part of the filesystem check. The unintialized block group feature can greatly reduce e2fsck time by eliminating checking of uninitialized inodes." She went on to explain how it works, "with this feature, there is a a high water mark of used inodes for each block group. Block and inode bitmaps can be uninitialized on disk via a flag in the group descriptor to avoid reading or scanning them at e2fsck time. A checksum of each group descriptor is used to ensure that corruption in the group descriptor's bit flags does not cause incorrect operation." Avantika attached a graph illustrating the advantage of the patch which she summarized as follows:

"The patches have been stress tested with fsstress and fsx. In performance tests testing e2fsck time, we have seen that e2fsck time on ext3 grows linearly with the total number of inodes in the filesytem. In ext4 with the uninitialized block groups feature, the e2fsck time is constant, based solely on the number of used inodes rather than the total inode count. Since typical ext4 filesystems only use 1-10% of their inodes, this feature can greatly reduce e2fsck time for users. With performance improvement of 2-20 times, depending on how full the filesystem is."

Linux: ext4 Development Status

Jeremy — Mon, 30 Apr 2007 23:35:56 +0000

Theodore Ts'o posted an update on the ext4 filesystem [story], "I've respun the ext4 development patchset, with Amit's updated fallocate patches. I've added Dave's patch to add ia64 support to the fallocate system call, but *not* the XFS fallocate support patches. (Probably better for them to live in an xfs tree, where they can more easily tested and updated.) Yes, we haven't reached complete closure on the fallocate system call calling convention, but it's enough for us to get more testing in -mm." Jeff Garzik noted that none of this development was happening in the kernel as originally planned, "why isn't this stuff going upstream rapidly? AFAICT nothing much at all has happened upstream besides a mass renaming? The whole point of having ext4 in the kernel is to do development upstream, in the public view, getting new stuff in ASAP (even if that means changing or pulling some stuff later)."

Theodore acknowledged, "in general, yes, ext4 development has been a little slow; part of the problem is that we have a lot of people, but a number of folks are new and their patches need review before they are ready for upstream acceptance, and a number of other folks who should be doing the review have been overloaded with multiple other projects and have been time-sharing." He went on to note, "but we also get flamed when the patches don't meet various criteria, up to and including breaking on ia64. We are in the process of setting up automated testing to help address that problem, but it's a taken a little while to get that going. I'm also trying to schedule more time so I can do the needed review of the patches so they meet basic upstream standards so we *can* push them. If other folks would like to help with the review process, that would be more than welcome. But yes, we will try to get more of the patches pushed sooner rather than later."

Linux: 2.6.19 Kernel Released

Jeremy — Wed, 29 Nov 2006 23:58:44 +0000

Linus Torvalds announced the release of the 2.6.19 Linux kernel, following the previous stable kernel release by two months [story]. "It's one of those rare 'perfect' kernels," Linus joked, "so if it doesn't happen to compile with your config (or it does compile, but then does unspeakable acts of perversion with your pet dachshund), you can rest easy knowing that it's all your own d*mn fault, and you should just fix your evil ways." He went on to add, "you could send me and the kernel mailing list a note about it anyway, of course. (And perhaps pictures, if your dachshund is involved. Not that we'd be interested, of course. No. Just so that we'd know to avoid it next time)."

The latest kernel source can be downloaded from your nearest Linux Kernel archive mirror [story]. You can browse through all the changes using the gitweb interface. Kernel Newbiews also maintains a useful summary of all the changes that went into this new version of the Linux kernel, including the inclusion of three new filesystems, GFS2, ext4 [story], and eCryptfs.

Linux: ext4 Merged Into -mm

Jeremy — Mon, 16 Oct 2006 15:50:25 +0000

With the release of the 2.6.19-rc1-mm1 kernel, the ext4 filesystem [story] was merged into Andrew Morton [interview]'s -mm tree for further testing. In the announcement Andrew notes that the new filesystem is compatible with ext3 until you add a file that has extents. He also notes, "when comparing performance with other filesystems, remember that ext3/4 by default offers higher data integrity guarantees than most. So when comparing with a metadata-only journalling filesystem, use `mount -o data=writeback'. (Although this doesn't seem to make much difference with ext3)" The goal is to stabilize the new filesystem within the next six to nine months, and ultimately to replace the ext3 filesystem.