login
Search
Alex Adriaanse reported an issue with making incremental backups using GNU tar's "--listed incremental" option on a reiserfs file system. The issue involved renaming the file between backups causing it to disappear upon restoration. Alex tracked the problem down to reiserfs not updating the file's ctime when renaming a file, providing a two line patch to theoritcally solve the problem.
This led into a very interesting discussion as to whether or not a file's ctime actually should be updated upon a rename. J.W. Schultz used several man pages to examine both sides of the issue, suggesting that "reiserfs is on this point definitely standards conformant already."
The upcoming 2.6 kernel [forum] maintainer Andrew Morton [interview] acknowledged that regardless of correctness, "minix, ext2 and ext3 do set ctime, so it is 'the Linux standard'." In response to the question of why not "fix" these filesystems, Andrew replied, "Because that's the way we've always done it and there have been zero complaints about it and changing it risks breaking things." Alex's patch, or a variant, will likely be merged into reseirfs to resolve this issue. And going forward, efforts will also be made to look into fixing GNU tar.
From: Alex Adriaanse [email blocked]
To: [email blocked]
Subject: ReiserFS patch for updating ctimes of renamed files
Date: Sun, 12 Oct 2003 01:05:19 -0500
Hi,
I ran into some trouble trying to do incremental backups with GNU tar
(using --listed-incremental) where renaming a file in between backups would
cause the file to disappear upon restoration. When investigating the issue
I discovered that this doesn't happen on ext2, ext3, and tmpfs filesystems
but only on ReiserFS filesystems. I also noticed that for example ext3
updates the affected file's ctime upon rename whereas ReiserFS doesn't, so
I'm thinking this causes tar to believe that the file existed before the
first backup was taking under the new name, and as a result it doesn't back
it up during the second backup. So I believe ReiserFS needs to update
ctimes for renamed files in order for incremental GNU tar backups to work
reliably.
I made some changes to the reiserfs_rename function that I *think* should
fix the problem. However, I don't know much about ReiserFS's internals, and
I haven't been able to test them out to see if things work now since I can't
afford to deal with potential FS corruption with my current Linux box.
I included a patch below against the 2.4.22 kernel with my changes. Would
somebody mind taking a look at this to see if I did things right here (and
perhaps wouldn't mind testing it out either)? If it works then I (and I'm
sure others who've experienced the same problem) would like to see the
changes applied to the next 2.4.x (and 2.6.x?) release.
Thanks a lot.
Alex
--- fs/reiserfs/namei.c.orig Mon Aug 25 06:44:43 2003
+++ fs/reiserfs/namei.c Sun Oct 12 00:39:05 2003
@@ -1207,6 +1207,8 @@
journal_mark_dirty (&th, old_dir->i_sb, old_de.de_bh);
old_dir->i_ctime = old_dir->i_mtime = CURRENT_TIME;
new_dir->i_ctime = new_dir->i_mtime = CURRENT_TIME;
+ old_inode->i_ctime = CURRENT_TIME;
+ reiserfs_update_sd (&th, old_inode);
if (new_dentry_inode) {
// adjust link number of the victim
From: jw schultz [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Sun, 12 Oct 2003 00:14:48 -0700
On Sun, Oct 12, 2003 at 01:05:19AM -0500, Alex Adriaanse wrote:
>
> I included a patch below against the 2.4.22 kernel with my changes. Would
> somebody mind taking a look at this to see if I did things right here (and
> perhaps wouldn't mind testing it out either)? If it works then I (and I'm
> sure others who've experienced the same problem) would like to see the
> changes applied to the next 2.4.x (and 2.6.x?) release.
Hmm. I'm conflicted.
rename(2) manpage:
Any other hard links to the file (as created using
link(2)) are unaffected.
A change to ctime would affect the other links.
stat(2) manpage:
The field st_ctime is changed by writing or by
setting inode information (i.e., owner, group, link
count, mode, etc.).
I am not aware of any field in the inode structure that must
be changed by an atomic rename. Per documentation the only
reason rename should update st_ctime is if it does a
link+unlink sequence which would alter st_nlink briefly.
On the other hand it does seem to me there ought to be some
record that something about the inode changed. st_ctime would
be the only appropriate indicator.
rename() SUSv3:
Some implementations mark for update the st_ctime
field of renamed files and some do not. Applications
which make use of the st_ctime field may behave
differently with respect to renamed files unless
they are designed to allow for either behavior.
So reiserfs is on this point definitely standards conformant
already. A change could at best be seen as an enhancement.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: [email blocked]
Remember Cernan and Schmitt
From: Hans Reiser [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Mon, 13 Oct 2003 09:49:20 +0400
thanks Mr. Schultz, you saved us a lot of work in reviewing this issue.
In theory it is cleaner and purer to do it the way we did. In practice,
Alex's problem seems like a real one, and I don't know how hard it is to
change tar to do the right thing. We'll discuss it in a small seminar
today.
--
Hans
From: Andrew Morton [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Mon, 13 Oct 2003 03:24:31 -0700
Hans Reiser [email blocked] wrote:
>
> In theory it is cleaner and purer to do it the way we did. In practice,
> Alex's problem seems like a real one, and I don't know how hard it is to
> change tar to do the right thing. We'll discuss it in a small seminar
> today.
It would be best to make this change. minix, ext2 and ext3 do set ctime,
so it is "the Linux standard".
btw, this code:
old_dir->i_ctime = old_dir->i_mtime = CURRENT_TIME;
new_dir->i_ctime = new_dir->i_mtime = CURRENT_TIME;
old_inode->i_ctime = CURRENT_TIME;
should avoid evaluating CURRENT_TIME three times: is has some computational
cost and if an interrupt happens at the wrong time you end up with
differing times in the inode(s).
From: Hans Reiser [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Tue, 14 Oct 2003 10:13:11 +0400
Andrew Morton wrote:
>
>It would be best to make this change. minix, ext2 and ext3 do set ctime,
>so it is "the Linux standard".
do you think schultz's arguments about why it is wrong are correct?
They seem well thought out to me.
--
Hans
From: Andrew Morton [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Mon, 13 Oct 2003 23:25:27 -0700
Hans Reiser [email blocked] wrote:
>
> do you think schultz's arguments about why it is wrong are correct?
> They seem well thought out to me.
Well given that you've renamed the file, you do want the backup program to
pick up the "new" file. But it'd be a pretty lame backup program which was
fooled by a missing ctime update.
Yes, John has a point but we're not going to go and change all the other
filesystems (are we?).
From: Hans Reiser [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Tue, 14 Oct 2003 10:30:56 +0400
Andrew Morton wrote:
>
>Yes, John has a point but we're not going to go and change all the other
>filesystems (are we?).
>
why not? It is correct therefor....
--
Hans
From: Andrew Morton [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Mon, 13 Oct 2003 23:44:38 -0700
Hans Reiser [email blocked] wrote:
>
> why not? It is correct therefor....
Because that's the way we've always done it and there have been zero
complaints about it and changing it risks breaking things.
I should think that's pretty obvious?
From: jw schultz [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Tue, 14 Oct 2003 00:09:33 -0700
On Mon, Oct 13, 2003 at 11:25:27PM -0700, Andrew Morton wrote:
> Hans Reiser [email blocked] wrote:
> >
> > do you think schultz's arguments about why it is wrong are correct?
> > They seem well thought out to me.
>
> Well given that you've renamed the file, you do want the backup program to
> pick up the "new" file. But it'd be a pretty lame backup program which was
> fooled by a missing ctime update.
>
> Yes, John has a point but we're not going to go and change all the other
> filesystems (are we?).
I don't know who John is but i sure hope we are not going to
go changing how working filesystems function. It may be
technically correct to not update ctime but that doesn't
mean that it is incorrect to update it either. It all
depends on the filesystem. They aren't all the same. We
have some that don't support symlinks or hardlinks or that
have or lack other features.
<OT>
There are change detections through timestamp (mtime) i am
concerned about. As an rsync maintainer i worry about be
extended attributes or ACLs changing with no modifiable
timestamps being updated. And, by the way, because you
cannot set ctime it doesn't qualify. Then there is jfs
which i found did not update mtime when directories change
unless the block count changes, but that might have been
fixed already.
</OT>
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: [email blocked]
Remember Cernan and Schmitt
From: Hans Reiser [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Mon, 13 Oct 2003 12:45:58 +0400
Alex, are you convinced by jw? (I think I am.) Would you be willing to
submit a patch for tar instead?
Hans
From: Alex Adriaanse [email blocked]
Subject: RE: ReiserFS patch for updating ctimes of renamed files
Date: Mon, 13 Oct 2003 21:37:24 -0500
Hans,
Yes, I agree with J.W. However, I also think that Andrew has a good point
in that the behavior across Linux filesystems (ReiserFS, ext2, ext3, minix,
etc.) should be consistent. Either they should all update ctime during
renames, or none of them should.
Anyway, I'll try to work with the GNU tar maintainer to get this problem in
tar fixed. It'll probably be a lot harder to fix in tar than to have
ReiserFS update ctimes since it'll require major changes in
the --listed-incremental snapshot files. However, if you don't think it's a
good idea to make these changes to ReiserFS then we'll just work on fixing
up tar.
Thanks,
Alex
From: Hans Reiser [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Tue, 14 Oct 2003 10:09:40 +0400
Let's see what Andrew says after he reads J.W.'s reasoning. I agree
that reiserfs should do the same thing as the other filesystems in
Linux, but J.W. seems to be right that they are doing the wrong thing.
Hans
Alex Adriaanse wrote:
From: jw schultz [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Mon, 13 Oct 2003 23:49:24 -0700
On Tue, Oct 14, 2003 at 10:09:40AM +0400, Hans Reiser wrote:
> Let's see what Andrew says after he reads J.W.'s reasoning. I agree
> that reiserfs should do the same thing as the other filesystems in
> Linux, but J.W. seems to be right that they are doing the wrong thing.
Whoa there. I am not saying that the other filesystems
are wrong. All i am saying as i can see no reason why, as
defined, the ctime of a file should be updated when nothing
else in the inode structure as been updated.
Of course if i had designed it in the first place with the
filesystem semantics that we have now there might be no
rename system call. Renames would be done by link(oldpath,
newpath); unlink(oldpath); A sequence that would cause
ctime to change as a result of nlink changes. A sequence
that might be appropriate in some cases even inside the
filesystem code.
If you read my first posting on this thread you will see
that i do see some, albeit questionable, value to the ctime
update as a means of spotting the fact something relating to
the inode has changed. While not technically required it is
not necessarily wrong to update ctime. SUSv3 even allows
for it and for some filesystems it would positively be the
right thing to do.
PS. The name is J.W. or JW and has never been John.
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: [email blocked]
Remember Cernan and Schmitt
From: Jamie Lokier [email blocked]
Subject: Re: ReiserFS patch for updating ctimes of renamed files
Date: Tue, 14 Oct 2003 10:29:09 +0100
jw schultz wrote:
> Of course if i had designed it in the first place with the
> filesystem semantics that we have now there might be no
> rename system call. Renames would be done by link(oldpath,
> newpath); unlink(oldpath); A sequence that would cause
> ctime to change as a result of nlink changes. A sequence
> that might be appropriate in some cases even inside the
> filesystem code.
Once upon a time, that's how renames were always done.
The rename() system call was added because (a) it provides the
additional atomicity semantic, which link+unlink does not; (b) it is
useful to allow directory renames, but directory hard links are
dangerous so not allowed any more.
-- Jamie
