[gentoo-user] Sparse files and df

[gentoo-user] Sparse files and df

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

Hi,

Here's a bit of a puzzle for me...

I've got an 40G LVM partition with empty reiserfs on it.
Then I've started rtorrent using this fs as a storage and added two 50G
torrents to it.
Rtorrent had no problems with the fact that partition is smaller than
either of them and created all the downloaded files as sparse, so that
"du -s --apparent-size" showed 100G.
df reported that fs is still empty.

Few days passed and some data actually hit the file system.
Knowing that it can't handle that much of data I've downloaded files
selectively, pushing completed ones to another fs, leaving a symlink in
their place.

df now shows that only 5G is free but "du -s" says that files occupy
15G and apparent size is 65G.
I've moved 5G worth of them and du now reports 10G/60G, while df still
shows that only 5G left (and decreasing).

Now I know that df can actually show weird results sometimes but I
wonder why (re)moving files from a file system doesn't affect it's
output at all.
Do I really have 5G there which will be depleted soon and there's
nothing I can do to help it?
If not, how come anyone trusts df output at all when it can report
almost-empty fs to be almost-full?

Wonder if anyone can make things a bit more clear for me here.
Thanks.

-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Sparse files and df

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 763 bytes --]

On Sun, 24 May 2009 12:42:07 +0600
Mike Kazantsev <mk.fraggod@gmail.com> wrote:

> Now I know that df can actually show weird results sometimes but I
> wonder why (re)moving files from a file system doesn't affect it's
> output at all.
> Do I really have 5G there which will be depleted soon and there's
> nothing I can do to help it?
> If not, how come anyone trusts df output at all when it can report
> almost-empty fs to be almost-full?

The idea that rtorrent should re-open all the files hit me a bit too
late - just restarting it fixed df output.
Now I wonder how can a file descriptor remain valid even after file was
moved to another fs, but I guess it might be some os feature.
Sorry for the noise.

-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Sparse files and df

[Nikos Chantziaras]

Mike Kazantsev wrote:
> Hi,
> 
> Here's a bit of a puzzle for me...
> 
> I've got an 40G LVM partition with empty reiserfs on it.
> Then I've started rtorrent using this fs as a storage and added two 50G
> torrents to it.
> Rtorrent had no problems with the fact that partition is smaller than
> either of them and created all the downloaded files as sparse, so that
> "du -s --apparent-size" showed 100G.
> df reported that fs is still empty.
> 
> Few days passed and some data actually hit the file system.
> Knowing that it can't handle that much of data I've downloaded files
> selectively, pushing completed ones to another fs, leaving a symlink in
> their place.
> 
> df now shows that only 5G is free but "du -s" says that files occupy
> 15G and apparent size is 65G.

I might be wrong, but here's my take on it:

The files you moved won't be deleted until they are closed.  That means 
quiting the torrent client.  "du" shows space occupied by files that 
actually have a filename.  The files you deleted are still there if the 
torrent client still has handles on them; they just lack an entry in the 
directory and therefore "du" doesn't pick them up, but of course "df" 
does since it's not looking at files individually but asks the 
filesystem directly.

So I suppose simply quitting the torrent client will result in no more 
handles pointing at those files and therefore they will finally be 
deleted by the filesystem.  If you don't quit the client, new data will 
be stored in the "invisible" deleted files rather than in the ones 
pointed to by the symlinks, resulting in "df" showing less and less free 
space even though "du" won't agree (and losing the downloaded data too 
since its stored in the deleted files.)

Re: [gentoo-user] Re: Sparse files and df

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

On Sun, 24 May 2009 10:13:09 +0300
Nikos Chantziaras <realnc@arcor.de> wrote:

> I might be wrong, but here's my take on it:
> 
> The files you moved won't be deleted until they are closed.  That means 
> quiting the torrent client.  "du" shows space occupied by files that 
> actually have a filename.  The files you deleted are still there if the 
> torrent client still has handles on them; they just lack an entry in the 
> directory and therefore "du" doesn't pick them up, but of course "df" 
> does since it's not looking at files individually but asks the 
> filesystem directly.

That indeed explains what happened and why re-opening files helped.
Somehow, I haven't considered that the files might not be removed if
something held the handles to them.

Thanks.

-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Sparse files and df

[Alan McKinnon]

On Sunday 24 May 2009 08:42:07 Mike Kazantsev wrote:
> Now I know that df can actually show weird results sometimes but I
> wonder why (re)moving files from a file system doesn't affect it's
> output at all.

I had this on an Oracle machine a while ago - huge amounts of space being 
consumed by files that are not there. It was deleted files to which a running 
process still had an open handle.

lsof | grep 'deleted' reveals these buggers
 
> Do I really have 5G there which will be depleted soon and there's
> nothing I can do to help it?
> If not, how come anyone trusts df output at all when it can report
> almost-empty fs to be almost-full?


-- 
alan dot mckinnon at gmail dot com