On Fri, Nov 09, 2007 at 04:27:49AM -0500, Daniel Ouellet wrote:

quoted text
> >>Any clue as to how to tackle this problem, or any trick around it?
> >
> >I really do not understand the problem here. But you might be able to
> >detect sparse files compartaring the size vs the number of blocks it uses.
> 
> Without making a bit writing out of it. Let say that the problem is for 
> now a storage capacity problem on the destinations servers, a timing one 
> in the extended transfer process and the additional bandwidth required 
> at some of the destination point and the volumes of files. Let just say 
> that if it was syncing 100K files, it would be a piece of cake, but it's 
> much bigger.
> 
> Just for example, a source file that is sparse badly, don't really have 
> allocated disk block yet, but when copy over, via scp, or rsync will 
> actually use that space on the destination servers. All the servers are 
> identical (or suppose to be anyway) but what is happening is the copy of 
> them are running out of space at time in the copy process. Like when it 
> is copying them, it may easy use twice the amount of space in the 
> process and sadly filling up the destinations then then the sync process 
> stop making the distribution of the load unusable. I need to increase 
> the capacity yes, except that it will take me times to do so.
> 
> Sparse file for database example is a very good thing, but not for 
> everything however.
> 
> The problem is not the sparse file at the source. It sure can stay as 
> is. It's just offset pointers anyway.
> 
> The problem is in the sync process between multiple servers using the 
> Internet to sync them and the bandwidth waisted as well as the lack of 
> space available at the destination. Plus because the copy is different 
> in size, then the sync process see it as different files and as such 
> will copy them again.

The size will not be different, just the disk space used.

quoted text
> 
> Or it can be copy using -S with rsync, however this process will inflate 
> the file at the destination and run out of space during the process and 
> make them smaller at the end. Plus this obviously take a lots more time 
> and as such, the timely sync process that was good for a long time now, 
> well... Let say, not reliable. Let say, sync without concern for sparse 
> is done just in a few minutes, but then use lots more space on the 
> destination. Doing it with -S to address the capacity issue fix that, 
> but then it takes a HUGE amount of time more and sadly there is useless 
> transfer of null data cause from the sparse source empty space.

So your problem seems to be that rsync -S is inefficient to the point
where it is not useable.  I do not use rsync a lot, so I do not know
if there's a solution to that problem. It does seem strange that a
feature to solve a problem actually make the problem worse. 

quoted text
> I can manage, I find ways to use ls -laR, or du -k and do diff's between 
> them and fine the files that are getting out of wack, replace them and 
> then continue, but this really is painful.

stat -s gives the raw info in one go. Some shell script hacking should
make it easy to detect sparse files.

	-Otto

quoted text
> Obviously when the capacity will be there, it will be a none issue, 
> however I am sadly not at that point yet and it will take me some time.
> 
> Not sure if that explain it any better, I hope so.
> 
> But I was looking if it was possible to identify these files in a more 
> efficient way.
> 
> If not, I will just deal with it.
> 
> It's just going to be painful for sometime that's all.
> 
> The issue is really in the transfer process and at the final 
> destination. Not at the source.
> 
> I hope it make more sense explaining it this way, if not I apologists 
> for the lack of better thinking at the moment in explaining it.
> 
> Best,
> 
> Daniel