login
Search
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey Everyone, So as the subject states this is more a centralized discussion on migration plans to using and providing xz for content on kernel.org. Currently we provide gz and bz2, with gz acting as the original content and kernel.org itself generating the resulting bz2 files. There are a couple of possible proposals and wanted to toss them out there, and get feedback from everyone: the kernel community, the mirrors of kernel.org and the direct users of kernel.org. ======================================================================== Option 1) Leave gz as the master, and migrate bz2 to xz. This will happen in stages obviously. with bz2 ultimately being phased out. Migration option 1) All new content would be provided in .bz2 and .xz with an ultimate date set that the .bz2 files would stop being generated with new content. This would leave all existing content alone and it would not be a migration of the current .bz2 files to xz Migration option 2) At some point there would be a mass conversion of all existing content to include .bz2 and .xz. These would be run in parallel for a time period until it was determined that .bz2 was no longer needed and it would be removed from the servers leaving .gz and .xz Option 2) Convert the master data from gz to bz2 and use xz as the new file format. This has the downside of causing more tool churn as it means the kernel developers will have to eventually convert from gz to bz2, which means for a time there will be nag e-mails if you upload gz instead of bz2 and such. It would also mean that we (kernel.org) would need to be able to support .gz and .bz2 as master data for a time. Migration options are identical to Option 1 more or less, with either just new content getting converted, or all content getting converted. ======================================================================== I'm personally leaning towards option 1, though ...
I thought that the result of the poll that Linus did a few days ago was that there were advantages in having .gz, but that .xz was both smaller and faster than .bz2 and therefor the best thing to do would be to end up with .gz and .xz. As such your option 2 doesn't seem like a reasonable path to go through. David Lang --
My personal recommendation would be for Option 1, Migration option 2. I think the idea of having two "best" file formats (Migration Option 1) in use indefinitely is rather pointless. As for the motivations for Option 1: a) Currently, .gz contents is original, whereas .bz2 contents is automatically generated. Flushing the .gz files would mean flushing original content. b) .gz is one of the most widely supported compression formats ever created, plus it is very fast, especially on the decompression side. .xz is reasonably fast but very memory intensive; .bz2 is moderately fast and still fairly memory intensive (but not as much as .xz). In other words, .gz provides an option for small, slow or old systems, or systems running inferior operating systems. .xz provides the best compression. .bz2 is in between, but it doesn't serve either purpose as well as the other two. Realistically speaking, kernel.org itself will carry all three formats for an extended transition time (at least a year); we will probably discontinue the victim format only when we start running shy on disk space. However, we obviously don't want to push this burden onto all the mirrors. Therefore, I would really appreciate feedback from mirror admins as to how you would prefer to see the transition -- either transition -- happen. -hpa --
H. Peter Anvin (hpa@zytor.com) wrote on 11 February 2010 11:48: >On 02/11/2010 10:36 AM, J.H. wrote: >> >> Option 1) >> >> Leave gz as the master, and migrate bz2 to xz. This will happen in >> stages obviously. with bz2 ultimately being phased out. >> >> Migration option 1) >> >> All new content would be provided in .bz2 and .xz with >> an ultimate date set that the .bz2 files would stop >> being generated with new content. This would leave all >> existing content alone and it would not be a migration >> of the current .bz2 files to xz >> >> Migration option 2) >> >> At some point there would be a mass conversion of all >> existing content to include .bz2 and .xz. These would >> be run in parallel for a time period until it was >> determined that .bz2 was no longer needed and it would >> be removed from the servers leaving .gz and .xz >> Option 2) >> >> Convert the master data from gz to bz2 and use xz as the new file >> format. This has the downside of causing more tool churn as it means >> the kernel developers will have to eventually convert from gz to bz2, >> which means for a time there will be nag e-mails if you upload gz >> instead of bz2 and such. It would also mean that we (kernel.org) would >> need to be able to support .gz and .bz2 as master data for a time. >> >> Migration options are identical to Option 1 more or less, with either >> just new content getting converted, or all content getting converted. > >My personal recommendation would be for Option 1, Migration option 2. Agreed (speaking for the ftp.br.kernel.org mirror). --
Hi John, On Thu, Feb 11, 2010 at 10:36:03AM -0800, J.H. wrote: I too think option 1 is better for various reasons, one of them being that gzip is present on every platform and OS right now, while bzip2 is orders of magnitude less common. Of course on Linux we all have both of them, but I find it important to continue to provide the sources in a standard format. Also, on the platforms where bzip2 is not installed by default, switching to xz should not be that hard, because either bz2 is currently not used and that changes nothing, or the users managed to install it themselves, and they'll simply have to do the same with xz. I don't have much preference for any of the two migration options though. Just my 2 cents, Willy --
I believe this is cleanest. gzip has performance advantages, and old files suddenly disappearing would be just weird. -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
It would also be "just weird" to: a) require that the end user knows the particular compression format used by a particular legacy file. b) having to have the mirror system deal with a mix of .bz2 and .xz files. -hpa --
While we're here... I think it is weird that the testing files for the latest kernel are in testing/ directly, while the older ones have their own subdirectory. I have a script which is greatly confused by this, and I can imagine that other tools (for example ketchup) appreciate this inconsistency moderately too. It is probably not ideal for mirrors either... I don't know if the mirroring system is smart enough to deal with file moves? If not then it generates useless traffic. So I would like to propose that testing files are placed directly in their final location. This would be both more consistent and easier for everyone. Thanks, -- Jean Delvare --
Hi Pavel, all, I have been investigating this issue and would like to share my findings as an additional data point for this discussion. For my testing, I have been using the slowest machine I still have available here: a Pentium 166 MMX, with 64 MB of memory and a slow hard disk drive. I've been writing down the duration of each task it took to boot kernel 2.6.27.45 on this machine. I did this for both .gz and .bz2 formats. Raw results are as follow (format=min:s): downloading linux-2.6.27.tar.bz2 5:01 downloading patch-2.6.27.45.bz2 0:02 unpacking linux-2.6.27.tar.bz2 7:28 applying patch-2.6.27.45.bz2 1:21 ---------------------------------------------- total for bz2 13:52 downloading linux-2.6.27.tar.gz 6:23 downloading patch-2.6.27.45.gz 0:02 unpacking linux-2.6.27.tar.gz 3:20 applying patch-2.6.27.45.gz 1:10 ---------------------------------------------- total for gz 10:55 So the gz option is unsurprisingly faster, setting up the source tree takes almost 3 minutes less (-21%). Then the (common) build and installation times: building 117:26 installing modules 0:12 ---------------------------------------------- total 117:38 This is a customized kernel, as small as I could do, with almost no features and the minimal set of drivers. As you can see, the build time is one order of magnitude greater than the tree setup time. Comparing the total times from download to install between bz2 and gz: bz2: 13:52 + 117:38 = 131:30 gz: 10:55 + 117:38 = 128:33 Compared to bz2, gz saves... 2% on the overall time. As a conclusion, I think we can plain discard the argument "I need .gz because my machine is slow" from now on. It simply doesn't hold. -- Jean Delvare --
166 MHz and 64 MB of RAM is still an order of magnitude more than some other
machines Linux is capable of running on.
Of course we no longer build kernels on those machines natively, we
cross-compile
on much faster machines (and use git to fetch the kernel sources, FWIW ;-).
BTW, who still downloads full kernel tarballs? From my experience, that's mostly
distro people who have scripts to build kernels from tarballs + a
bunch of patches?
I guess actual developers use git nowadays, even if they're stuck on
an old 2.6.x kernel?
Backporting critical fixes is sooo much easier using git cherry-pick...
Do we have statistics about tarball downloads vs. git clone / git pull?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
--
All of my daily build & boot testing downloads tarballs. They use full linux-x.y.z for "releases" (like 2.6.32) and they use patch-x.y.z.gz/bz2 for intermediate versions. I don't mind updating the scripts to use .xz tarballs, but I would That would be good to see. -- ~Randy --
When I download to develop, I use git. When I just want to install the latest kernel on a box (not used for development) I use ketchup (downloads tarballs). For those that just want the latest kernel release, git is too bloated. -- Steve --
If the download link had been slower than about 75 kB/s, the bz2 option would have been faster even on this old machine. With xz, download would be faster than bz2 and decompression would be somewhere between bz2 and gz --- at least on machines without notable memory constraints. xz's decompressor is more memory hungry than bzip2's one as far as I understand their manual pages. But at the default xz compressor setting of -6, the decompressor will still use just 10 MB and should therefore not cause even your 64 MB machine to Yep, whether the target machine is meant to compile the kernel or to be used to browse the source code (with a bare minimum of comfort), the hardware resources required for either task mean that there isn't such a great difference between gz, bz2, xz WRT resource requirements at the receivers, except that xz goes easiest on network bandwidth and disk utilization. -- Stefan Richter -=====-==-=- --=- -==-= http://arcgraph.de/sr/ --
Note that, if memory consumption is really a concern on either end, we could use xz -5, which still achieves much better compression than bz2 but doesn't require more memory for decompression. -- Jean Delvare --
I agree, but, IMHO the main argument for keeping .gz is cross-platform availability and wide language support, not hardware limitations. Doing a quick google brings up .gz interfaces for every language you can think of (C, Java, Perl, Python, TCL etc.), not to mention complete separate implementations in Java and Pascal (not just wrappers on top of the zlib library), and probably more. With xz you have just one C/C++ implementation with a single library with an undocumented API for C/C++ programmers. It may be a slight stretch of the imagination, but with with .gz you can conceive programmers writing programs to download a .gz from kernel.org and decompressing/searching it, in almost any language of choice. With the JAVA implementation .gz is genuinely cross platform and you don't need glibc/ C++ compilers, just a Java VM. Contrast with xz, where if the xz utility isn't available, or doesn't do what you want, you're stuck with programming in C/C++ with all the baggage that entails. Phillip --
This can probably be easily explained. gz is very fast decompressing so it is a very good choice for transparent decompression of files which must be accessible fast but aren't used frequently. Manual pages or printer drivers come to mind. bz2 and lzma, OTOH, are meant for longer term archiving. Their compression ratio benefit is only worth it for larger files that you don't access that frequently. I am not claiming that gzip is dead. It is very useful and it is there to stay for the years to come, no doubt about that. What I'm saying is that it isn't the best choice for large files to be downloaded from a Honestly, I don't think we care at all when it comes to the kernel.org files. Accessing individual files inside a compressed kernel tarball without first expanding it entirely would be horribly slow and unpractical, no matter which compression format was used. I can't think of any case where you won't unpack the tarball first, and for this task an external tool will do just fine. And, once again, there are several public instances of gitweb and LXR available if you only want to browse the code. -- Jean Delvare --
just out of curiosity what would happen if by say I take a file and turn it into .gz then turn the .gz into .xz or vice versa? so at the end of the day you have a list of .gz's(or whatever), then expending on the type(.gz,.bz2,etc..) unpackage and voila either a tree or some other compressed file(.bz2,xz, or .gz). just thinking out loud(so don't shoot me please). Justin P. Mattock --
Hi Jean, Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" and have it transparently uncompressed and dumped on my terminal. It doesn't do that on bz2. We could find multiple examples. Another thing comes to mind, because I've been beaten by this in the past. People working in enterprises where the internet access passes via mandatory proxies coupled with anti-virus can often download many things but not binaries that can't be analysed. At this time, I could only download tar.gz but not .bz2. And please don't tell me I have to go to the admin to tell him to change his proxy's configuration, you can't do that when you work as a consultant for a 20000 persons group where products are selected after 6-12 months of testing and managed by different people from those who qualify them. In my opinion, .xz is a very good option to replace .bz2 as it will bring advantages without downsides. But .gz should stay as it has been available from day 1, at least for all people who may have trouble with .xz for whatever reason. If it has not been a problem for the last 16 years, I don't see why it would suddenly become one. Regards, Willy --
Salut Willy, This only underlines what I just wrote: the purpose of gzip is different from the purpose of bzip2 or lzma. Transparent decompression makes sense for the former but little for the latter two. But this doesn't mean everyone must make all their files available in .gz format. Not every file, and not everyone, needs transparent decompression. It makes sense on patch files, but not on tarballs. You have the need, but I don't. Furthermore, the fact that your local usage of patch files is more convenient if they come in .gz format doesn't imply that this is the format in which you have to download them. You are free to repack files after downloading them. This can even be easily automated. hpa seems to be very reluctant to the idea, but one thing I had in mind was that patches could be in .gz format and tarballs in .xz, for example. Having to stick to a common strategy for all the files seems suboptimal, given their different natures and uses. (And for completeness: on my distribution, the bzip2 package comes with a little helper script named bzless, which does exactly what you want I've been working in such environments in the past, so I see what you mean. And I do not disagree that the format in which projects make their files available can be more or less convenient in such situations. For example, I remember being deeply annoyed by projects not releasing tarballs, because I simply couldn't access CVS repositories. That being said, I also don't think that you can put all the burden of bad company policies on the shoulder of open source software providers. If someone worked in a company where even .gz compressed files can't be downloaded, we wouldn't ask kernel.org to provide uncompressed files on top of .gz and .bz2, would we? There is a trade-off to be found between flexibility of access and disk and network usage. It should be possible to retrieve the kernel sources using git over HTTP these days anyway, right? Or are firewalls frequently ...
>>>>> "W" == Willy Tarreau <w@1wt.eu> writes: W> Well, I personally like to be able to simply run "less patch-2.6.27.45.gz" W> and have it transparently uncompressed and dumped on my terminal. It W> doesn't do that on bz2. We could find multiple examples. It does here, and lzma & xz, too. And has since just days after Lasse annouced that the new name would be xz. Your LESSOPEN controls that, and can be easily coded to support any archive. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 1024D/ED7DAEA6 --
Just checked and I found it funny to see that patch-2.6.1.bz2 is not correctly opened while 2.6.27.45.bz2 is. Maybe some things have slightly changed in the tools by that time and the output differs slightly. However bzcat opens them both so the file is not corrupted. This raises the point of the maturity of the tools BTW. GZ is mature, BZ2 has become mature over the years, XZ is very recent and may still be buggy at times. We'll only know that when people will complain that they cannot open one file from time to time. Willy --
This isn't a huge problem. Since it is generated content we can, if necessary, run a robot over the archive and verify and regenerate files that have problems. -hpa --
There is a bug in lesspipe.sh, at least here. It confuses it with compresses
man pages.
--- lesspipe.sh~ 2009-11-06 02:14:58.000000000 +0200
+++ lesspipe.sh 2010-02-17 12:19:09.000000000 +0200
@@ -50,6 +50,7 @@
*.1.bz2|*.2.bz2|*.3.bz2|*.4.bz2|*.5.bz2|*.6.bz2|*.7.bz2|*.8.bz2|*.9.bz2|*.n.bz2|*.man.bz2)
# compressed *roff src?
if bzip2 -dc "$1" | file - | grep roff 1> /dev/null ; then
bzip2 -dc "$1" | nroff -S -mandoc -
+ else bzip2 -dc "$1" 2>/dev/null
fi ;;
*.gz) gzip -dc "$1" 2>/dev/null ;;
*.bz2) bzip2 -dc "$1" 2>/dev/null ;;
--
... which is not even widely deployed. I did a quick survey and none of my systems (none of which terrible old and not particularly embedded) have it installed and for most them there's only "lzma-utils" in the distribution package repositories which I understand is not compatible. I would basically need to download the source by hand and install it like back in the bad old "unix with all useful commands missing" HP-UX/Solaris/etc. days. Please keep .gz and better .bz2 -Andi -- ak@linux.intel.com -- Speaking for myself only. --
When was the last time you compiled a Linux kernel on HP-UX or Solaris? I think others did that first (me being along the party), and quite frankly, the plain Solaris without CSW has quickly-reachable limits even for non-kernels. --
I completely agree that language support is bad compared to the .gz format, but I think the above sentence makes it sound a bit too bad. There are three C (no C++ needed) libraries that support the .xz format: - LZMA SDK (7-zip.org) - XZ Utils has liblzma (tukaani.org) - XZ Embedded (tukaani.org, limited support only) The latter two are more or less based on LZMA SDK, although the code looks very different (different coding style, different APIs etc.). The liblzma API has reference documentation as Doxygen tags in the API headers. They aren't the best docs and there's no tutorial or example programs yet, but the API certainly isn't undocumented. It has similarities to the zlib API, so those used to zlib shouldn't have trouble learning the basic features of liblzma, which are enough for most people. There are liblzma bindings for Perl and Python, but I don't know how good they are. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode --
Don't you have download statistics available? If we knew which compression format is preferred, an by which margin, it would help make Maybe that's just me, but my main concern is neither download times nor decompression times. My main concern is the access time to directory indexes when browsing the kernel archive, because there are 5 entries for every patch or tarball: .bz2, .bz2.sign, .gz, .gz.sign and .sign. This is horribly slow. The main directory for 2.6 kernels has an index weighting over 300 kB raw, turning into a ~600 kB document when HTML-ized. Just fetching it takes 3 seconds and then my browser takes a long time to format it. There are 3881 entries in that directory today, and it keeps growing! So, once we have settled for a compression strategy, I think it would be the right time to discuss the directory structure. With the advent of the stable branches and the new development model - which pretty much implies that we'll live with main version 2.6 forever - the file count is much higher than it used to me. I can think of several ways to improve the situation here, some of which could be combined. 1* Keep a single compression format. This saves almost 40% of the files. 2* Move one of the compression formats somewhere else, so that it doesn't get in the way but is still available if needed. 3* Create a new subdirectory for every 2.6.x kernel, and move all the related files there. This would shrink the main index drastically, and each subdirectory would have a reasonable size (except maybe 2.6.16 and 2.6.27.) Oddly enough this has been done for the files under testing/ already, so I am curious why we don't do it for the release files (and the testing/incr/ files, while we're at it.) 4* Get rid of the LATEST-IS-* files. This is a small count, won't save much, but these files seem totally useless to me these days. Depending on what you want exactly, there are many versions which can be considered the latest, and there are better ways to know which they ...
As the new maintainer of ketchup: git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ketchup.git I could write a fix for the new format real quick. My concern is that users of ketchup may not see this update for a long time. -- Steve --
What would you consider a reasonable turnaround time? 3 month? 6 month? More? We could make a decision now, you update ketchup immediately, and the actual change happens at the specified date in the future. -- Jean Delvare --
Usually users don't update ketchup until it breaks ;) But sure, 3 months probably would be fine. Then Matt Mackall will start getting emails about it not working, and he will forward them to me. Where I can then point them to the new repository. -- Steve --
You can probably anticipate this a bit by warning the maintainers of the ketchup package in openSUSE, Fedora etc. -- Jean Delvare --
Then the archive location can change immediately after you issue an update ;-) Would it be an option for a newer version of ketchup to have a fallback codepath to read a machine-readable description of how the archive is structured from a known location? E.g. instead of appending a hard-coded string '/people/akpm/patches/2.6/' when computing latest_mm(), the script would read that "relative path to mm" from such a description file. The logic of doing 'url = "%s/v%s/%s" % (kernel_url, t, f)' in install_nearest would similarly be customizable, without too much hassle, I think. --
Are you suggesting that the path be stored at a known location on kernel.org? Such that it can be read and the tools like ketchup can find where a particular version resides? -- Steve --
I'd really like to fix ketchup to behave sanely in late -rc stages:
going from -rc6 to -rc8 is possible using two downloads through --rc7
(like 100KB?) instead of two big ones through full release (like
10MB).
But first step is --only-dl option -- when you want to cache the
needed patches but not apply anything yet.
Please apply,
Pavel
diff --git a/ketchup b/ketchup
index 0728aec..3249cbc 100755
--- a/ketchup
+++ b/ketchup
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
r = " -R"
qprint("Applying %s%s" % (os.path.basename(p), r))
- if options["dry-run"]:
+ if options["dry-run"] or options["only-dl"]:
return ver
def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
ver = list[0][2]
qprint("Unpacking %s" % os.path.basename(f))
- if options["dry-run"]: return ver
+ if options["dry-run"] or options["only-dl"]: return ver
untar(f)
return ver
@@ -658,6 +658,7 @@ opts = [
('l', 'list-trees', None, 'list supported trees'),
('m', 'show-makefile', None, 'output version in makefile <arg>'),
('n', 'dry-run', None, 'don\'t download or apply patches'),
+ ('o', 'only-dl', None, 'don\'t apply patches'),
('p', 'show-previous', None, 'output version previous to <arg>'),
('q', 'quiet', None, 'reduce output'),
('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
b = find_ver(args[0])
qprint("%s -> %s" % (a, b))
transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
rename_dir(b)
if postcommand and os.system(postcommand):
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
Can you send this to me as an official patch. With SoB and all. Thanks, -- Steve --
Add --only-dl option -- when you want to cache the needed patches but
not apply anything yet.
Signed-off-by: Pavel Machek <pavel@ucw.cz>
diff --git a/ketchup b/ketchup
index 0728aec..3249cbc 100755
--- a/ketchup
+++ b/ketchup
@@ -405,7 +405,7 @@ def apply_patch(ver, reverse = 0):
r = " -R"
qprint("Applying %s%s" % (os.path.basename(p), r))
- if options["dry-run"]:
+ if options["dry-run"] or options["only-dl"]:
return ver
def cmd(patch, reverse, dry):
@@ -484,7 +484,7 @@ def install_nearest(ver):
ver = list[0][2]
qprint("Unpacking %s" % os.path.basename(f))
- if options["dry-run"]: return ver
+ if options["dry-run"] or options["only-dl"]: return ver
untar(f)
return ver
@@ -658,6 +658,7 @@ opts = [
('l', 'list-trees', None, 'list supported trees'),
('m', 'show-makefile', None, 'output version in makefile <arg>'),
('n', 'dry-run', None, 'don\'t download or apply patches'),
+ ('o', 'only-dl', None, 'don\'t apply patches'),
('p', 'show-previous', None, 'output version previous to <arg>'),
('q', 'quiet', None, 'reduce output'),
('r', 'rename-directory', None, 'rename updated directory to %s<v>'
@@ -750,7 +751,7 @@ if not a and os.listdir("."):
b = find_ver(args[0])
qprint("%s -> %s" % (a, b))
transform(a, b)
-if options["rename-directory"] and not options["dry-run"]:
+if options["rename-directory"] and not options["dry-run"] and not options["only-dl"] :
rename_dir(b)
if postcommand and os.system(postcommand):
diff --git a/ketchup.1 b/ketchup.1
index 0a313ee..9e5a385 100644
--- a/ketchup.1
+++ b/ketchup.1
@@ -1,5 +1,5 @@
.\" Hey, EMACS: -*- nroff -*-
-.TH KETCHUP 1 "April 12, 2006"
+.TH KETCHUP 1 "February 16, 2010"
.\" Please adjust this date whenever revising the manpage.
.\"
.\" Some roff macros, for reference:
@@ -74,6 +74,11 @@ output version in makefile <arg>
.IP
don't download or apply ...Actually, you'll probably get a better patch if you relace 'only-dl' -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
Works for me.
And here's quick patch to show what I'm doing: it teaches ketchup
about 2.6.33-rc1-rc2 patches, thus saving lot of bandwidth. Of course,
I'll need to get it to work for more than 2.6.A->2.6.B-rcC case,
but...
If there are some comments, or maybe better approach, let me know...
diff --git a/ketchup b/ketchup
index 3249cbc..dc1bbf8 100755
--- a/ketchup
+++ b/ketchup
@@ -107,19 +107,28 @@ local_trees = {}
# Functions to parse version strings
def tree(ver):
+ """returns 2.6"""
return float(re.match(r'(\d+\.\d+)', ver).group(1))
def rev(ver):
+ """given 2.6.31 or 2.6.32-rc1 returns 31"""
p = pre(ver)
r = int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
if p: r = r - 1
return r
def pre(ver):
+ """returns rc5"""
try: return re.match(r'\d+\.\d+\.\d+(\.\d+)?-((rc|pre)\d+)', ver).group(2)
except: return None
+def next_pre(ver):
+ s = pre(ver)
+ i = int(re.match(r'rc(\d+)', s).group(1))
+ return "rc%d" % (i+1)
+
def post(ver):
+ """given 2.6.27.1 returns 1"""
try: return re.match(r'\d+\.\d+\.\d+\.(\d+)', ver).group(1)
except: return None
@@ -132,9 +141,15 @@ def prenum(ver):
except: return None
def prebase(ver):
+ """returns 2.6.13-rc1"""
return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.)\d+)?)', ver).group(1)
+def preincr(ver):
+ """only use when incremental patches are requested"""
+ return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.).+)?)', ver).group(1)
+
def revbase(ver):
+ """returns 2.6.23 for 2.6.23.15 or 2.6.24-rc5"""
return "%s.%s" % (tree(ver), rev(ver))
def base(ver):
@@ -283,8 +298,12 @@ def find_info(ver):
f = forkname(ver)
p = pre(ver)
+ print "find_info (ver) ", ver, "f=", f
+
s = b
- if f:
+ if re.match(".*rc.*rc.*", ver):
+ s = "%s-incrc" %b
+ elif f:
s = "%s-%s" % (b, f)
elif p:
s = "%s-pre" % b
@@ -297,11 +316,14 @@ def version_urls(ver):
if ...This should work; save time&bandwidth by using incremental patches
between -rcX.
Is there testsuite for ketchup? It would be useful at this point...
Pavel
--- ketchup.orig 2010-02-16 16:36:51.000000000 +0100
+++ ketchup 2010-02-22 19:53:56.000000000 +0100
@@ -107,19 +107,32 @@
# Functions to parse version strings
def tree(ver):
+ """returns 2.6"""
return float(re.match(r'(\d+\.\d+)', ver).group(1))
+def rawrev(ver):
+ """given 2.6.31 or 2.6.31-rc1 returns 31"""
+ return int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
+
def rev(ver):
+ """given 2.6.31 or 2.6.32-rc1 returns 31"""
p = pre(ver)
- r = int(re.match(r'\d+\.\d+\.(\d+)', ver).group(1))
+ r = rawrev(ver)
if p: r = r - 1
return r
def pre(ver):
+ """returns rc5"""
try: return re.match(r'\d+\.\d+\.\d+(\.\d+)?-((rc|pre)\d+)', ver).group(2)
except: return None
+def next_pre(ver, inc = 1):
+ s = pre(ver)
+ i = int(re.match(r'rc(\d+)', s).group(1))
+ return "rc%d" % (i+inc)
+
def post(ver):
+ """given 2.6.27.1 returns 1"""
try: return re.match(r'\d+\.\d+\.\d+\.(\d+)', ver).group(1)
except: return None
@@ -132,9 +145,15 @@
except: return None
def prebase(ver):
+ """returns 2.6.13-rc1"""
return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.)\d+)?)', ver).group(1)
+def preincr(ver):
+ """only use when incremental patches are requested"""
+ return re.match(r'(\d+\.\d+\.\d+((-(rc|pre)|\.).+)?)', ver).group(1)
+
def revbase(ver):
+ """returns 2.6.23 for 2.6.23.15 or 2.6.24-rc5"""
return "%s.%s" % (tree(ver), rev(ver))
def base(ver):
@@ -283,8 +302,12 @@
f = forkname(ver)
p = pre(ver)
+ print "find_info (ver) ", ver, "f=", f
+
s = b
- if f:
+ if re.match(".*rc.*rc.*", ver):
+ s = "%s-incrc" %b
+ elif f:
s = "%s-%s" % (b, f)
elif p:
s = "%s-pre" % b
@@ -297,11 +320,14 @@
if ...I don't have one, but that would be helpful. Would you like to take over maintaining ketchup? I took it over because Matt did not want it anymore and I was the last to send him patches. I've been quite busy working on Ftrace, trace-cmd and kconfig stuff, that I'm willing to hand this over to you, if you want it. My last release of ketchup is my git tree on kernel.org. -- Steve --
I'd really prefer not to maintain it -- I'm using it on my zaurus, because git is just too slow there -- but I don't really have time just now. (Plus, I'm not good enough with git, I'm afraid). But, if you think the patches are good enough, I can just prepare nice, easy to apply series? (But I'd really like someone to check that the basic idea is right...) Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
This was actually the main reason for me personally to ask about just dropping support for .gz files - not because I care deeply about how much disk space kernel.org wastes, but because the long directory listings make I did 3* for the testing kernels (exactly because the directory listing got to be unreadable), and you just complained about it ;) Of course, your complaint was that the subdirectory wasn't done immediately, and that the old files get moved to their own subdirectory later as a "archival" thing. Yeah, they also end up continually being stale. Linus --
It's probably worth keeping things like the .gz files around, if nothing else for older distros, systems, etc that don't have xz yet (since it's still relatively new) Breaking things out into directories or such might be the easiest way with that v2.6/ v2.6/2.6.23 v2.6/2.6.32.6 etc Only thoughts there are that there seem to be a lot of automated processes that rely on LATEST-IS-*. Personally I'd rather see them snag the RSS feed and figure out what they want from there, but that may not be completely feasible. It also gives a quick indication as to what's latest in the directory and a quick search on the page usually gets me what I'm looking for when I do it. - John 'Warthog9' Hawley --
Hardly a good reason IMHO. xz can be installed on these systems. When I'd rather group all 2.6.32.* files together, so that the main index is as small as possible. We're adding one indirection step, so it should Care to give details? Given how often the old files get stuck, I am surprised any process can really rely on them. And I also can't really There's RSS, there's a mailing list and there's a web page. Certainly What's your workflow? I normally go to the download directory after either reading the kernel.org main page or some post on the announce mailing list. So I already know which version I am looking for. Having to search for "LATEST-IS" and then again for the version doesn't look terribly efficient. If we really want a helper to locate the latest version, I'd rather have a "latest" symbolic link pointing to the most recent v2.6.x subdirectory. Or maybe two, "latest-stable" and "latest-devel". Can be discussed... -- Jean Delvare --
I don't agree, it's different. Git is only used by developers, and even not all of them. Sources are a reference. Anyone can download them to look for anything. Switching to a specific format which really is not common at all on older distros nor on any system looks a bit like proprietary encoding eventhough it's not the case. But it's a way to tell people that if they want the sources in clear text form, they first have to find a tool capable of decompressing them. Gzip is well defined as a standard, it's even described in an RFC and is present on almost any system (unix or not) now. Any student who wants to take a look at the kernel will have access to gunzip, even from an old Solaris 8 workstation or a Windows XP desktop PC. XZ if far from being there, and the student will not necessarily be able to install it. And Peter raised some valid points about the hardware requirements to run such tools ; I'm not sure the guys running Linux on their old Sparc-2 would like XZ only a lot. Regards, Willy --
It is not just student on old workstation. I'm trying to keep Linux working on spitz PDA and kohjinsha subnotebook. Especially zaurus has about power of old sparc two... otoh it runs 4 hours on cellphone battery. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
Do you actually require to download and unpack (and configure, build) the kernel sources from kernel.org to the PDA directly? As Jean wrote, how does unpack time matter compared to build time? -- Stefan Richter -=====-==-=- --=- -==-= http://arcgraph.de/sr/ --
PS: It boils down to CPU time requirements. For small memory systems, there is the XZ Embedded decompressor. -- Stefan Richter -=====-==-=- --=- -==-= http://arcgraph.de/sr/ --
XZ Embedded saves only code size. If the file was compressed with settings that make the decompressor require e.g. 65 MiB RAM (xz -9), using XZ Embedded won't help you. XZ Embedded is also very limited. It cannot decompress all .xz files. It should still be quite useful, since on some architectures it can compress the kernel image better than the current LZMA support in the kernel, and the API should be nice to use e.g. by Squashfs. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode --
Salut Willy, Just like switching to git was a way to tell people that if they wanted to contribute to the kernel, they first had to install the right, at the time uncommon tool. Of course the audience isn't the same, but the principle is similar. And the audience is still fairly limited in both cases. My parents aren't downloading kernel tarballs. I would assume that anyone willing and able to download, read and understand the linux kernel source code wouldn't be frightened by having to install a small tool to extract these sources. And they may not even have to do this: sources can be read with just a web browser: we have the gitweb interface, and several public LXR installations are deployed as well. These days, web browsers are much more popular than ftp clients and tarballs. As a matter of fact, I am advocating the use of xz while I don't have it installed on most of my machines. I really don't see this as a Really? I have a Windows XP laptop at hand and it can't read .gz files. If I ask it to try, it tells me I should install WinZip. I also seem to recall that I had to install GNU gzip myself back when I was working on I don't quite buy this argument either. I suspect this is a very limited count of users, and these users have access to other, more powerful machines where they can easily achieve any format conversion they need. I have an old, slow machine here which I am going to use to perform some real world testing, and I'll post the results when I'm done. But I suspect that building a kernel on this machine, even a small one with just the drivers it needs, will take much longer than unpacking the sources. So anyone worrying about performance would rather rely on cross-compilation, and in turn can afford whatever decompression tool is needed. -- Jean Delvare --
As a side note: 7zip, a very popular and Free archiving tool for Windows, supports xz since its version 9 which is currently available as a beta. So, WRT the need to get an extra unarchiver, xz is just as accessible to Windows users as gz is. -- Stefan Richter -=====-==-=- --=- -==-= http://arcgraph.de/sr/ --
Can you even unpack a kernel tree on Windows? There are some files
(e.g. net/ipv4/netfilter/ipt_{ecn,ECN}.c) which conflict on a
case-preserving filesystem.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
--
What filesystem version are you talking about ? ntfs as of recent windows is not case insensitive . What do you mean by , "a case-preserving filesystem" ? Now whether or not the tools available are case insensitive is another matter . Hth , JimL -- +------------------------------------------------------------------+ | James W. Laferriere | System Techniques | Give me VMS | | Network&System Engineer | 3237 Holden Road | Give me Linux | | babydr@baby-dragons.com | Fairbanks, AK. 99709 | only on AXP | +------------------------------------------------------------------+ --
Given a sane filesystem, yes. If (the) native filesystem(s) are sane, is All of NTFS (and basically FAT*). Are there are different versions? If It *is* case insensitive (as it compares filesystem names Apart from "please google that, thank you": The filesystem (driver)) preserves the case of a filename if you create it (TTBOMK). But it ignores the case of the filename for other file operations like open(), ACK. But if you have a case-insensitive or case-preserving filesystem, the tools can´t really do that much. And given the "understanding" of the monopolist (as such) in that area, most of the "tools" are actually inherent part of the OS (and not just another 3rd party app). SCNR .... Bernd -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at --
On Sat, Feb 13, 2010 at 4:39 PM, Bernd Petrovitsch There are case sensitive Windows subsystems (e.g. SFU/SUA at least when running over NTFS), and the default behavior for Win32 apps even can be changed to be case sensitive via a registry key: ObCaseInsensitive). More important is how easy it is to install - since XZ is not even available via apt-get install on recent distros (e.g. April 2009 Ubuntu 9.04), this discussion seems about a year premature. -- Thanks, Steve --
It's somewhat - ähemm - strange IMHO that the casing is an app-specific feature (and not filesystem specific). And it is really that implemented that the app can choose in what way the filesystem below - given that it supports that feature - compares It's in recent Fedoras so Ubuntu is perhaps just late. And FWIW, it' 2 packages too as the second one is the .lzma support (so probably Debian should be able to fix the clash with whatever the current lzma package is). Bernd -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at --
There's a package on Fedora called 'xz' that provides it on my Fedora 11 & 12 boxes. - John 'Warthog9' Hawley --
Wow, what a stretch. There's about milion tools supporting gz for Windows, and many of them are out of beta... xz is not even supported on my zaurus, and I'll need to write some emails to get it installed on school servers... Anyway, gz+xz should be reasonable combination. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
And roughly half of this million archiving/ compression tools use 7zip's backend library, which is just one way of getting xz support in a Windows application. Besides, what one project calls beta is called dot-zero release elsewhere. :-) Seriously, getting xz support on Windows is just as easy or as difficult as getting gz support. -- Stefan Richter -=====-==-=- --=- -===- http://arcgraph.de/sr/ --
Sorry, but I just don't think that's true. No more windows systems here. On android, gzip is supported, xzip is not. Debian machine supports both, but gzip was preinstalled and I had to pull xzip. (This does not help either: root@amd:~# apt-cache search xzip xzip - Interpreter of Infocom-format story-files ) Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
Considering that xz support is available even on niche systems like Amiga OS and BeOS (via p7zip if not by other means) and xz-utils proper build even on DOS, OpenVMS and other systems, how hard can it be to obtain an xz decompressor on Android, Debian, or Ubuntu¹? (¹About which there was a note somewhere else in this thread that there is a conflict between xz-utils and lzma-utils... That's basically because the former supersede the latter.) The name confusion between xz-utils and xzip can be avoided if you search for the package in a package manager which shows package categories (archivers vs. games). -- Stefan Richter -=====-==-=- --=- -==== http://arcgraph.de/sr/ --
This, I think gets to one of the problems. You're telling me that the
p7zip thing I installed for work is this .xz thing? And it's really all
this LZMA algo? That's part of the transition problem, folks quite
likely have access, easily, to a decompressor but don't know what it is
(.gz->gzip, .bz2 -> bzip2, .xz -> {p7zip, p7zip-full, xz-utils, ???}).
In fact, why not .lzma? I'm assuming we're talking about the format,
and xz-utils nad p7zip and others all implement the same format and it's
all compatible and it's just a "how do we get there quickest / fastest"
Actually, no, there's no 'xz-utils' in Debian/Lenny or Ubuntu 9.04, but
there is in 9.10. But in 9.10, searching on xzip (a good, but
incorrect guess) only gives that. Searching on xz shows xz-utils too.
At least with apt-cache, and since the desc doesn't list xzip (since
it's not xzip, it's an incorrect but not illogical guess), that
wouldn't help.
--
Tom Rini
--
Naturally. Information on it is quickly found online though. -- Stefan Richter -=====-==-=- --=- =---- http://arcgraph.de/sr/ --
Eh? Making many people around the world install uncommon tool is not On zaurus, kernel compilation takes 4 hours. (I.e. "one night"). So that one is ... well ... done overnight. Untar is something I normally wait for, since you need to run (interactive) oldconfig after that. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
Pavel Machek wrote: If the download takes m minutes and unpacking and unarchiving tar.gz takes another n minutes, what difference does it make for this workflow when instead m + p minutes are spent for download + unpacking and unarchiving tar.bz2 or tar.xz? -- Stefan Richter -=====-==-=- --=- -===- http://arcgraph.de/sr/ --
The difference is that the user is waiting for the n or m minutes, but goes off and does something else for the p minutes. So it doesn't really matter if the compile takes 1 hour or 6 hours. As Pavel noted, this is done overnight and it doesn't matter if it gets done at 2am or 6am. But the uncompression has a user waiting for it (to do the config step), so here it makes a big difference if it takes 6 minutes ot 8 minutes. David Lang --
You totally missed Stefan's point. Read again. -- Jean Delvare --
Indeed. Let me make this straight before we waste more breath on this: we're not going to go to xz only at this time. Heck, we were getting pushback on the idea of going to bz2 only *ten years* after it was deployed. -hpa --
It's pretty obvious that xz will become popular quickly, at least on Linux and BSD systems, much like bz2 is today. I'm not asking people to start using ClearCase ;) xz will supersede bz2, it's only a matter of Out of curiosity, if it takes that long, why don't you use a You'll have to wait, no matter what compression format you use (and even if you don't compress the tarball). Judging by the duration of the build on your machine, I'd estimate the decompression time to 7 minutes for gz vs. 15 minutes for bz2 maybe? I doubt you sit in front of the machine for 7 minutes waiting for tar.gz to decompress, right? So I fail to see what difference it makes. You'll just do something else for 15 minutes instead of doing something else for 7 minutes. Anyway, as I have been saying several times already, nothing prevents you from repacking tarballs to gz before uploading it to your slow system if such is your desire. I can understand the portability argument, but the decompression time, no way. -- Jean Delvare --
If by "quickly" you mean 'ten years', sure, maybe. Keep in mind that there are people where who are still using RHEL 3, and some of them might want to download from ftp.kernel.org. So those people who are suggesting that we replace .gz files with .xz on kernel.org are *really* smoking something good. People who think xz are good should be working to get it installed by default into the community and then enterprise distro's, first.... - Ted --
As a note xz is available via EPEL for Redhat Enterprice Linux 5 and anything that's derived from that. Doesn't seem to be available, directly, for 4 or lower. Not sure on Suse, and Debian's already been mentioned. Just trying to do a quick survey of what's out there already. - John 'Warthog9' Hawley --
xz is available since openSUSE 11.2, not before. -- Jean Delvare --
Something I noticed, too. Hopefully going to work on getting set up as a packager and get xz pushed out to EPEL EL-4 over the next couple weeks unless someone else beats me to it as I still have a handful of EL-4 machines grinding away. :-) -Dave --
Because I hack it on the go. First build is long and ugly, but subsequent builds only take 5 minutes or so, so development is possible. (If I crosscompiled it, I'd have to crosscompile even the subsequent Actually I do sit in front of the machine, reading mails while it decompresses. [I'll get some numbers.] sh-3.2$ time bzip2 -d < ~/.ketchup/l^Ginux-2.6.31.tar.bz2 > delme.tar 485.73user 137.35system 683.32 (11m23.320s) elapsed 91.18%CPU sh-3.2$ df -h^H^H^H^H^Htime cat delme.tar > /usr/src/delme.tar 0.57user 109.03system 381.13 (6m21.133s) elapsed 28.75%CPU sh-3.2$ time zca^G^Gt delme.tar > /u ^H^H ^H^H ^H^H ^H^Hdelme.tar.gz ^H^H ^H^H ^H^H +^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H.gz > delme.tar 43.97user 106.22system 223.26 (3m43.261s) elapsed 67.27%CPU So... gzip is actually _faster_ than uncompressed data, while bzip is twice slower. Don't know about xzip. Anyway, gzip just makes sense. It is both smaller and faster than Ok, so lets go by the portability argument. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html --
Can I feature-request that someone reduces the git.kernel.org frontpage? It's almost twice as large as the v2.6 dir index in http-delivered form, so you can already experience what it's like when v2.6/ grows bigger. --
That's a topic for a completely different thread / audience than this, lets try to keep this a bit more focused since we have the mirrors on this discussion. - John 'Warthog9' Hawley --
5* Archive all the older 2.6.x files and move them into a separate
directory (e.g. v2.6-pre20). Moving all the pre 2.6.20 files
saves 42% of the file listing.
This seems an obvious solution, what am I missing?
Thanks
Phillip
--
This is confusing, inconsistent and unstable. Confusing because 2.6-pre referred so far to the releases immediately preceding 2.6.0. Inconsistent because it requires the downloader to have preliminary knowledge about what the break point is. Unstable because, while you consider pre20 to qualify as "old" today, in 5 years you will want pre30 to qualify as "old" instead, meaning that tools such as ketchup would have to be updated once again. I think we want to come up with a directory structure which won't change in the future. -- Jean Delvare --
I didn't say "2.6-pre", anyway it could be called something different, You yourself said "I wouldn't worry too much about breaking the current locations. Just give some time for software authors (ketchup comes to mind) to update their code and it shouldn't be a big problem." The major advantage with my suggestion is for the majority of users/tools interested in "recent" kernels, nothing changes at all. Your suggestions I think trying to do that is utterly futile. Phillip --
Yes, I did say that, and I can repeat it if needed. There's a big difference between breaking an old scheme which used to be valid and happens to no longer be, and designing a new scheme where breakages are bound to happen by design. Even if technically that's the same breakage, its reception by the affected users will be very different, Prove it. Many people out there are still working on older trees. I am working on 2.6.5 and 2.6.16 kernels on a weekly basis. If ketchup or other tools break for these trees only and not more recent ones, that This doesn't make me happy, but at least it is consistent and durable. When you change something for everyone, it has the advantage that the solutions are general, come quickly and are widely documented. Using quirks to limit the effects is a burden for the future, and may not You didn't have to join this discussion in the first place. -- Jean Delvare --
And who are you to tell me I shouldn't have? I was trying to be constructive, and it seems as if I've been insulted just because I dared to disagree with you. Grow up. My comment regarding futility is that things rarely turn out as planned even with the best intentions, and it's rarely worth getting hung up about designing the perfect system for something you can't predict. Phillip --
Why are you still working on 2.6.5 and 2.6.16 kernels on a weekly basis? This could quite useful to know. I could be pedantic and simply turn your "prove it" around, and ask you to prove you're not an isolated case, but that wouldn't be constructive, but irritating, like you. Phillip --
Because I am doing support for enterprise customers who are using distributions based on these kernel versions. These are SLE 9 and SLE 10, to name them, but RHEL supporters are in the same situation. And I've heard embedded developers report many times that they had to stick to older 2.6.x kernels too for various reasons. Not everyone is using a recent 2.6.x kernel, which makes it hard to draw a line between what should be considered old ones and what should be considered new ones. -- Jean Delvare "It is not necessary to hope in order to undertake, nor to succeed in order to persevere" -- Charles The Bold --
Embedded and enterprise distro users are usually stuck on ancient kernels that were downloaded from kernel.org and patched *years ago*. The reason they're stuck on them is due to local modifications, and so they're not going to be downloading ancient vanilla kernels from kernel.org now. Phillip --
Hi Phillip, They perfectly could. This is exactly what we're doing at Suse and I can easily imagine other companies follow the same model. We store our local changes as patches on top of the old kernel version. When a new developer joins the team and needs to setup a working tree, our setup script gets the patches from our internal repository, fetches the relevant kernel tarball from kernel.org, unpacks it and applies all the patches. This is one of the reasons why others have been claiming in this discussion: it would be weird if files which were previously available would suddenly disappear. We can discuss the cost and benefits of any change done to the tree structure, compression formats etc. but please do not assume that nobody is downloading the old files from kernel.org. Personally I wouldn't mind at all if old files would disappear and our tools have to be adjusted accordingly, as long as it happens only once in a long while and not on a regular basis by (broken) design. -- Jean Delvare --
not trying to cut in, but the best example I can see for this (hopefully), or a good example of just changing everything (cut the middle man per say)is libc there is no libc-2.11.90.so .tar.gz(etc..)only through git(but could be wrong). My system that I built is only handling everything(meaning every package as much as possible)through git. Justin P. Mattock --
Maybe try getting glasses instead? ;) -- Jean Delvare --
Well, part of the reason why is that we're functionally "stuck" on 2.6; a prefix which really has lost all meaning. It might open up the question if we shouldn't just do a Solaris and drop the leading 2 (so the next kernel would be 6.33) or call the kernel after that 3.0 instead of 2.6.34, and then 3.1 instead of 2.6.35. -hpa --
Damn, we forgot to have that fight at Kernel Summit last year. I'm in favour of the 3.0 / 3.1 / 3.2 with stable@ being responsible for 3.0.1, 3.0.2, 3.1.1, etc. -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." --
I'm in favor of almost _anything_ new, the current numbering scheme drives me crazy, but then, I'm the one having to deal with it more than anyone these days... thanks, greg k-h --
At last years LinuxCon conference, I was practically bouncing out of my chair wanting to ask Linus about 3.0 in front of everyone while he was on the panel discussion. But James Bottomley refused to take my question. -- Steve --
Like I suggested in October 2008, but it would have been more natural at that time: <http://marc.info/?l=linux-kernel&m=122418454113793&w=2> -- Hilsen Harald. --
You and several other people. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. --
I remember the whole LKML discussion about this a few years back: http://lkml.org/lkml/2008/10/15/377 The whole year.version or year/month versioning Greg HK proposed made a lot of sense to me. It would also solve our problem with the 2.6 directory just growing and growing as the year versioning would make a natural hierarchy which keeps going no matter what. -- Jeffry Sleddens Rotterdam University --
Note also that every time this conversation happens it starts to pull away in different directions, and as a result nothing happens. I'm going to stick my foot in it and state the following: I think incremental numbers work well, and everyone are used to them. It doesn't seem to be the major issue with the current scheme; the issue with the current scheme is that we have one or two levels too much. -hpa --
Exactly. And for having worked with projects using dates, it's a hell complicated to handle delayed releases... I'd like 3.0, 3.1, ... too, and am using that scheme for my projects, which people easily understand very well, as opposed to x.y.z.t. But now I think that Linus has a "grep -v 3.0" filter on his inbox, I never got a reply from him on the subject and I know I'm not the only one :-( Willy --
This sounds like a good plan (and since we have so far failed to come up with some new feature in Linux that is so awesome that it warrants bumping the version number to 3.0 ... we might as well manufacture one, and "The HTML for the archive directory on kernel.org has gotten too big" seems a pretty good reason to me). So the plan could be: 1) Declare 2.6.35 (or so) to be the last in the 2.6 series. 2) Define a better archive directory structure for the 3.x releases (scripts will have to be changed anyway to handle the 3.x change) 3) Keep all the .gz and .bz2 in the old 2.x hierarchy 4) Only have .xz in the new 3.x directories. This should give time for people to update scripts and for xz compression tools to become widely available. People on the bleeding edge versions are the most likely ones to update their tools promptly. People still using 2.x series kernels can keep using their old tools forever. -Tony --
I really dislike this for reasons already stated. Keep in mind that the compression management is global to the entire archive, not just to the kernel, too. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. --
How hard a requirement is this? It might be the right time to reconsider... Tony's plan may not be perfect but overall I think I like it. -- Jean Delvare --
It's pretty darn hard. It makes it particularly messy for the mirrors to ignore it. -hpa --
There are a couple of reasons to not do that. a) the moment we do that, some mirrors will start purging those areas. We have tried to keep our mirror system universally usable by keeping a "whole mirrors only, please" policy, and it has worked really well so far. b) it would prevent us from *ever* reorganizing those areas, as many be required. c) it complicates a lot of the machinery, including the bits that allow mirrors to only carry some of the compression formats. That includes mirror-side configuration, which I'd really loathe to make anything more than very simple; mirror admins want things to work as hands-off and automatic as possible, with a minimum of setup. Having to learn the intricacies of each mirror target is just plain hell. -hpa --
My would go for Option 1. With respect to migration Option 2. The last time we had a succsessful change of the default compression format from compress to gzip, it required a change from a encumbered format to an unencombered format, and gunzip was able to decompress files created with compress. Neither xz nor bzip2 can decompress gzip'd files, and appear to be heavy enough that for modest file sizes they give little gain. bzip2 fills a niche but does not act as a generally available, general purpose compressor today. It appears that xz is coming up fast in bzip2's niche, and provides better compression, so would appear to be a good replacement for bzip2. The role of gzip is to be compatible with the everything, and I would be very sad to see it gzip disappear as that would make files on kernel.org much less accessible. Migration option 2 sounds like I would could just replace bzip2 support in scripts with xz support so it sounds handy, but I will leave the details to those who have to run the servers. Right now xz suffers from limited availability. I just checked the systems I have handy and of the 4 distro's I have installed on various machines only 1 provides xz, so I expect it will be a while before xz achieves ubiquity. At the same time the fact that xz reduces the linux-2.6.32 tarball by 10Megs is a welcome benefit, and seems to make the replacement of bzip2 with xz worth doing. Eric --
On Sun, 14 Feb 2010 10:03:31 -0800 xv is not standard on Ubuntu. Installing xv-utils on Ubuntu 9.10 produces big scary warning about package conflict with lzma. This seems like a premature optimization at this point -- --
That would be a packaging bug, xz should transparently supersede lzma. -- Jean Delvare --
