> On Thu, Nov 18, 2010 at 10:30:47AM -0500, Chris Mason wrote:
> > Excerpts from Bron Gondwana's message of 2010-11-16 23:11:48 -0500:
> > > > > a) program creates piles of small temporary files, hard
> > > > > links them out to different directories, unlinks the
> > > > > originals.
> > > > >
> > > > > b) filesystem size: ~ 300Gb (backed by hardware RAID5)
> > > > >
> > > > > c) as the filesystem grows (currently about 30% full)
> > > > > the unlink performance becomes horrible. Watching
> > > > > iostat, there's a lot of reading going on as well.
> > > >
> > > > It sounds like the unlink speed is limited by the reading, and the reads
> > > > are coming from one of two places. We're either reading to cache cold
> > > > block groups or we're reading to find the directory entries.
> > >
> > > All the unlinks for a single process will be happening in the same
> > > directory (though the hard linked copies will be all over)
> > >
> > > > Could you sysrq-w while the performance is bad? That would narrow it
> > > > down.
> > >
> > > Here's one:
> > >
> > >
http://pastebin.com/Tg7agv42
> >
> > Ok, we're mixing unlinks and fsyncs. If it fsyncing directories too?
>
> Nup. I'm pretty sure it doesn't, just files. Yes - there will certainly
> be fsyncs going on as well - Cyrus is very careful to fsync everything it
> cares about at the file level, but all it does with directories is mkdir
> them if they don't exist.
