[gentoo-user] can't create file but disk isn't full

[gentoo-user] can't create file but disk isn't full

[Crístian Viana]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

hi everyone,

something weird is happening on my system. I can't create new files, it says
"No space left on device", but the disk has several gigabytes of free space!
the output of "df -h" is:

Filesystem            Size  Used Avail Use% Mounted on
rootfs                 35G   23G   11G  69% /
/dev/root              35G   23G   11G  69% /
rc-svcdir             1.0M  120K  904K  12% /lib64/rc/init.d
udev                   10M  240K  9.8M   3% /dev
shm                   974M  1.1M  973M   1% /dev/shm
/dev/sda6             157G  133G   17G  89% /home
/dev/sda1              35G   28G  7.6G  79% /mnt/windows

I'm running out of space on the home partition. it says it has 17 GB of free
space, but right now there's only 330 MB available (I wrote a small program
to create the largest file it can). this was happening before, when I had a
few gigabytes free, but now the [fake] available space is growing and I'm
losing space on my disk each day! where should I start looking for to solve
this problem? the filesystem of the home partition is ext4.

I don't even know if this is related to Gentoo, but that's the OS I'm
running :-) I'm running ~amd64, by the way.

regards.

[-- Attachment #2: Type: text/html, Size: 1527 bytes --]

Re: [gentoo-user] can't create file but disk isn't full

[Volker Armin Hemmann]

On Samstag 08 Mai 2010, Crístian Viana wrote:
> hi everyone,
> 
> something weird is happening on my system. I can't create new files, it
> says "No space left on device", but the disk has several gigabytes of free
> space! the output of "df -h" is:
> 
> Filesystem            Size  Used Avail Use% Mounted on
> rootfs                 35G   23G   11G  69% /
> /dev/root              35G   23G   11G  69% /
> rc-svcdir             1.0M  120K  904K  12% /lib64/rc/init.d
> udev                   10M  240K  9.8M   3% /dev
> shm                   974M  1.1M  973M   1% /dev/shm
> /dev/sda6             157G  133G   17G  89% /home
> /dev/sda1              35G   28G  7.6G  79% /mnt/windows
> 
> I'm running out of space on the home partition. it says it has 17 GB of
> free space, but right now there's only 330 MB available (I wrote a small
> program to create the largest file it can). this was happening before,
> when I had a few gigabytes free, but now the [fake] available space is
> growing and I'm losing space on my disk each day! where should I start
> looking for to solve this problem? the filesystem of the home partition is
> ext4.
> 
> I don't even know if this is related to Gentoo, but that's the OS I'm
> running :-) I'm running ~amd64, by the way.
> 
> regards.

run out of inodes?

[gentoo-user] Re: can't create file but disk isn't full

[Nikos Chantziaras]

On 05/08/2010 09:21 PM, Crístian Viana wrote:
> hi everyone,
>
> something weird is happening on my system. I can't create new files, it
> says "No space left on device", but the disk has several gigabytes of
> free space!

The filesystem probably ran out of inodes.  "df -i /home" will show 
inode usage.  This can happen when you have many small files; they eat 
inodes but not storage space.

Re: [gentoo-user] Re: can't create file but disk isn't full

[Crístian Viana]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

it doesn't seem so :-(

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda6            20856832  108698 20748134    1% /home

I didn't know that the filesystem could run out of inodes before the disk
space itself! thanks for the information :-)

On Sat, May 8, 2010 at 4:07 PM, Nikos Chantziaras <realnc@arcor.de> wrote:

> On 05/08/2010 09:21 PM, Crístian Viana wrote:
>
>> hi everyone,
>>
>> something weird is happening on my system. I can't create new files, it
>> says "No space left on device", but the disk has several gigabytes of
>> free space!
>>
>
> The filesystem probably ran out of inodes.  "df -i /home" will show inode
> usage.  This can happen when you have many small files; they eat inodes but
> not storage space.
>
>
>

[-- Attachment #2: Type: text/html, Size: 1271 bytes --]

Re: [gentoo-user] Re: can't create file but disk isn't full

[Alan McKinnon]

You probably have files opened that have since been deleted. du doesn't report 
them as the names are no longer in the directory and df doesn't report them as 
they are pending deletion once the last handle to them is closed.

It's a nasty thing to find. Run this:

lsof | grep deleted

You should find a ton of junk temp files (they will go away when you log out). 
Look for big numbers in column 8




On Sunday 09 May 2010 00:46:28 Crístian Viana wrote:
> it doesn't seem so :-(
> 
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/sda6            20856832  108698 20748134    1% /home
> 
> I didn't know that the filesystem could run out of inodes before the disk
> space itself! thanks for the information :-)
> 
> On Sat, May 8, 2010 at 4:07 PM, Nikos Chantziaras <realnc@arcor.de> wrote:
> > On 05/08/2010 09:21 PM, Crístian Viana wrote:
> >> hi everyone,
> >> 
> >> something weird is happening on my system. I can't create new files, it
> >> says "No space left on device", but the disk has several gigabytes of
> >> free space!
> > 
> > The filesystem probably ran out of inodes.  "df -i /home" will show inode
> > usage.  This can happen when you have many small files; they eat inodes
> > but not storage space.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: can't create file but disk isn't full

[Crístian Viana]

[-- Attachment #1: Type: text/plain, Size: 1990 bytes --]

I shutdown this computer everyday, those temp files shouldn't be alive for
months.

I ran lsof | grep deleted and it returned 132 lines, the biggest number
being 2032226 (2 MB?), belonging to the Chromium browser process. even if
every line had that value (which is not), that would sum up 264 MB, but the
difference of reported/real free space is way bigger than that.

changing the filesystem back to ext3 can solve this problem? it was ext3
before I've changed it to ext4 some months ago.

On Sat, May 8, 2010 at 8:00 PM, Alan McKinnon <alan.mckinnon@gmail.com>wrote:

> You probably have files opened that have since been deleted. du doesn't
> report
> them as the names are no longer in the directory and df doesn't report them
> as
> they are pending deletion once the last handle to them is closed.
>
> It's a nasty thing to find. Run this:
>
> lsof | grep deleted
>
> You should find a ton of junk temp files (they will go away when you log
> out).
> Look for big numbers in column 8
>
>
>
>
> On Sunday 09 May 2010 00:46:28 Crístian Viana wrote:
> > it doesn't seem so :-(
> >
> > Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> > /dev/sda6            20856832  108698 20748134    1% /home
> >
> > I didn't know that the filesystem could run out of inodes before the disk
> > space itself! thanks for the information :-)
> >
> > On Sat, May 8, 2010 at 4:07 PM, Nikos Chantziaras <realnc@arcor.de>
> wrote:
> > > On 05/08/2010 09:21 PM, Crístian Viana wrote:
> > >> hi everyone,
> > >>
> > >> something weird is happening on my system. I can't create new files,
> it
> > >> says "No space left on device", but the disk has several gigabytes of
> > >> free space!
> > >
> > > The filesystem probably ran out of inodes.  "df -i /home" will show
> inode
> > > usage.  This can happen when you have many small files; they eat inodes
> > > but not storage space.
>
> --
> alan dot mckinnon at gmail dot com
>
>

[-- Attachment #2: Type: text/html, Size: 2623 bytes --]

Re: [gentoo-user] Re: can't create file but disk isn't full

[Alan McKinnon]

On Sunday 09 May 2010 01:39:54 Crístian Viana wrote:
> I shutdown this computer everyday, those temp files shouldn't be alive for
> months.
> 
> I ran lsof | grep deleted and it returned 132 lines, the biggest number
> being 2032226 (2 MB?), belonging to the Chromium browser process. even if
> every line had that value (which is not), that would sum up 264 MB, but the
> difference of reported/real free space is way bigger than that.
> 
> changing the filesystem back to ext3 can solve this problem? it was ext3
> before I've changed it to ext4 some months ago.

I'm fresh out of ideas on this one.

As I understand it, downgrading from ext4 to ext3 normally doesn't work out. 
There are features in ext4 that make it very attractive and most folk enable 
them, but they are incompatible with ext3. Or so I have read.

I would boot into a rescue system and run an fsck on that volume if you have 
not already done so.





> 
> On Sat, May 8, 2010 at 8:00 PM, Alan McKinnon 
<alan.mckinnon@gmail.com>wrote:
> > You probably have files opened that have since been deleted. du doesn't
> > report
> > them as the names are no longer in the directory and df doesn't report
> > them as
> > they are pending deletion once the last handle to them is closed.
> > 
> > It's a nasty thing to find. Run this:
> > 
> > lsof | grep deleted
> > 
> > You should find a ton of junk temp files (they will go away when you log
> > out).
> > Look for big numbers in column 8
> > 
> > On Sunday 09 May 2010 00:46:28 Crístian Viana wrote:
> > > it doesn't seem so :-(
> > > 
> > > Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> > > /dev/sda6            20856832  108698 20748134    1% /home
> > > 
> > > I didn't know that the filesystem could run out of inodes before the
> > > disk space itself! thanks for the information :-)
> > > 
> > > On Sat, May 8, 2010 at 4:07 PM, Nikos Chantziaras <realnc@arcor.de>
> > 
> > wrote:
> > > > On 05/08/2010 09:21 PM, Crístian Viana wrote:
> > > >> hi everyone,
> > > >> 
> > > >> something weird is happening on my system. I can't create new files,
> > 
> > it
> > 
> > > >> says "No space left on device", but the disk has several gigabytes
> > > >> of free space!
> > > > 
> > > > The filesystem probably ran out of inodes.  "df -i /home" will show
> > 
> > inode
> > 
> > > > usage.  This can happen when you have many small files; they eat
> > > > inodes but not storage space.
> > 
> > --
> > alan dot mckinnon at gmail dot com

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: can't create file but disk isn't full

[Johannes Kimmel]

On 05/09/2010 01:39 AM, Crístian Viana wrote:
> I shutdown this computer everyday, those temp files shouldn't be alive
> for months.
>
> I ran lsof | grep deleted and it returned 132 lines, the biggest number
> being 2032226 (2 MB?), belonging to the Chromium browser process. even
> if every line had that value (which is not), that would sum up 264 MB,
> but the difference of reported/real free space is way bigger than that.
>
> changing the filesystem back to ext3 can solve this problem? it was ext3
> before I've changed it to ext4 some months ago.
>
> On Sat, May 8, 2010 at 8:00 PM, Alan McKinnon <alan.mckinnon@gmail.com
> <mailto:alan.mckinnon@gmail.com>> wrote:
>
>     You probably have files opened that have since been deleted. du
>     doesn't report
>     them as the names are no longer in the directory and df doesn't
>     report them as
>     they are pending deletion once the last handle to them is closed.
>
>     It's a nasty thing to find. Run this:
>
>     lsof | grep deleted
>
>     You should find a ton of junk temp files (they will go away when you
>     log out).
>     Look for big numbers in column 8
>
>
>
>
>     On Sunday 09 May 2010 00:46:28 Crístian Viana wrote:
>      > it doesn't seem so :-(
>      >
>      > Filesystem            Inodes   IUsed   IFree IUse% Mounted on
>      > /dev/sda6            20856832  108698 20748134    1% /home
>      >
>      > I didn't know that the filesystem could run out of inodes before
>     the disk
>      > space itself! thanks for the information :-)
>      >
>      > On Sat, May 8, 2010 at 4:07 PM, Nikos Chantziaras
>     <realnc@arcor.de <mailto:realnc@arcor.de>> wrote:
>      > > On 05/08/2010 09:21 PM, Crístian Viana wrote:
>      > >> hi everyone,
>      > >>
>      > >> something weird is happening on my system. I can't create new
>     files, it
>      > >> says "No space left on device", but the disk has several
>     gigabytes of
>      > >> free space!
>      > >
>      > > The filesystem probably ran out of inodes. "df -i /home" will
>     show inode
>      > > usage.  This can happen when you have many small files; they
>     eat inodes
>      > > but not storage space.
>
>     --
>     alan dot mckinnon at gmail dot com
>
>

this is unlikely, but can you create files as root? ext filesystems 
reserve a certain amount of space for root use only. you can change this 
with tune2fs if necessary.

[gentoo-user] Re: can't create file but disk isn't full

[Nikos Chantziaras]

On 05/09/2010 01:46 AM, Crístian Viana wrote:
> it doesn't seem so :-(
>
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/sda6            20856832  108698 20748134    1% /home
>
> I didn't know that the filesystem could run out of inodes before the
> disk space itself! thanks for the information :-)

Long shot, but check if root can write files.  If yes, it probably means 
your reserved block count is a bit high (default is 5% I believe).  The 
reserved block count is a mechanism that disallows further writes to the 
filesystem if it gets too full, and only root can keep writing.

If that's your problem, the reserved block count can be changed with the 
tune2fs tool.  To set it to, say 2%, you would run:

   tune2fs -m 2 /dev/sda6

I don't know if it's safe to do this while the filesystem is mounted. 
To play it safe, go to single user mode, umount /home, and only then run 
the above command.

Re: [gentoo-user] Re: can't create file but disk isn't full

[Crístian Viana]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

root can create new files! I created a big file with the remaining 17 GB
logged in with root. I'll run this tune2fs later, before shutting down the
machine.

what exactly is this reserved block count? is it about the number of inodes?
does that mean that, by default, regular users can only use 95% of the
inodes? and why did I use all these inodes? I don't think I have that many
small files on this partition...

On Sat, May 8, 2010 at 10:48 PM, Nikos Chantziaras <realnc@arcor.de> wrote:

> On 05/09/2010 01:46 AM, Crístian Viana wrote:
>
>> it doesn't seem so :-(
>>
>> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
>> /dev/sda6            20856832  108698 20748134    1% /home
>>
>> I didn't know that the filesystem could run out of inodes before the
>> disk space itself! thanks for the information :-)
>>
>
> Long shot, but check if root can write files.  If yes, it probably means
> your reserved block count is a bit high (default is 5% I believe).  The
> reserved block count is a mechanism that disallows further writes to the
> filesystem if it gets too full, and only root can keep writing.
>
> If that's your problem, the reserved block count can be changed with the
> tune2fs tool.  To set it to, say 2%, you would run:
>
>  tune2fs -m 2 /dev/sda6
>
> I don't know if it's safe to do this while the filesystem is mounted. To
> play it safe, go to single user mode, umount /home, and only then run the
> above command.
>
>
>

[-- Attachment #2: Type: text/html, Size: 1947 bytes --]

Re: [gentoo-user] Re: can't create file but disk isn't full

[Willie Wong]

On Sun, May 09, 2010 at 07:39:01PM -0300, Crístian Viana wrote:
> what exactly is this reserved block count? is it about the number of inodes?
> does that mean that, by default, regular users can only use 95% of the
> inodes? and why did I use all these inodes? I don't think I have that many
> small files on this partition...

When the filesystem fills up, services can start failing left and
right because they cannot write logs, cannot write temp files, etc. At
this point human intervention is necessary: root has to log in and
clear out the disk. But if the $ROOT filesystem is completely full,
one may not even be able to log in and/or that one cannot do any sort
of maintenance that is needed. So you have some sort of circularity. 
(In which case you have to reboot, perhaps using another medium...)

The way out is to reserve some breathing room for root so that when
everybody else is having problems he can still get in and fix the
problem. 

The 5% is historical from days when disks are much smaller. If you
have a sensible partition scheme you only really need to reserve the
blocks on the $ROOT filesystem. If the partition in question (IIRC) is
only for /home, then you can just turn off the reserved blocks all
together. 

Cheers, 

W
-- 
Willie W. Wong                                     wwong@math.princeton.edu
Data aequatione quotcunque fluentes quantitae involvente fluxiones invenire 
         et vice versa   ~~~  I. Newton

Re: [gentoo-user] Re: can't create file but disk isn't full

[Crístian Viana]

[-- Attachment #1: Type: text/plain, Size: 1628 bytes --]

thanks! I'll set it to 0% then.

On Mon, May 10, 2010 at 12:48 PM, Willie Wong <wwong@math.princeton.edu>wrote:

> On Sun, May 09, 2010 at 07:39:01PM -0300, Crístian Viana wrote:
> > what exactly is this reserved block count? is it about the number of
> inodes?
> > does that mean that, by default, regular users can only use 95% of the
> > inodes? and why did I use all these inodes? I don't think I have that
> many
> > small files on this partition...
>
> When the filesystem fills up, services can start failing left and
> right because they cannot write logs, cannot write temp files, etc. At
> this point human intervention is necessary: root has to log in and
> clear out the disk. But if the $ROOT filesystem is completely full,
> one may not even be able to log in and/or that one cannot do any sort
> of maintenance that is needed. So you have some sort of circularity.
> (In which case you have to reboot, perhaps using another medium...)
>
> The way out is to reserve some breathing room for root so that when
> everybody else is having problems he can still get in and fix the
> problem.
>
> The 5% is historical from days when disks are much smaller. If you
> have a sensible partition scheme you only really need to reserve the
> blocks on the $ROOT filesystem. If the partition in question (IIRC) is
> only for /home, then you can just turn off the reserved blocks all
> together.
>
> Cheers,
>
> W
> --
> Willie W. Wong
> wwong@math.princeton.edu
> Data aequatione quotcunque fluentes quantitae involvente fluxiones invenire
>         et vice versa   ~~~  I. Newton
>
>

[-- Attachment #2: Type: text/html, Size: 2101 bytes --]

Re: [gentoo-user] Re: can't create file but disk isn't full

[Alex Schuster]

Willie Wong writes:

> When the filesystem fills up, services can start failing left and
> right because they cannot write logs, cannot write temp files, etc. At
> this point human intervention is necessary: root has to log in and
> clear out the disk. But if the $ROOT filesystem is completely full,
> one may not even be able to log in and/or that one cannot do any sort
> of maintenance that is needed. So you have some sort of circularity.
> (In which case you have to reboot, perhaps using another medium...)
> 
> The way out is to reserve some breathing room for root so that when
> everybody else is having problems he can still get in and fix the
> problem.
> 
> The 5% is historical from days when disks are much smaller. If you
> have a sensible partition scheme you only really need to reserve the
> blocks on the $ROOT filesystem. If the partition in question (IIRC) is
> only for /home, then you can just turn off the reserved blocks all
> together.

Isn't another purpose of those 5% the reduction of fragmentation that 
occurs more when there is few free space left? Although I also reduce ift 
on very large partitions. But I never set it to exactly zero.

	Wonko

Re: [gentoo-user] Re: can't create file but disk isn't full

[Willie Wong]

On Thu, May 13, 2010 at 12:25:08AM +0200, Alex Schuster wrote:
> > The 5% is historical from days when disks are much smaller. If you
> > have a sensible partition scheme you only really need to reserve the
> > blocks on the $ROOT filesystem. If the partition in question (IIRC) is
> > only for /home, then you can just turn off the reserved blocks all
> > together.
> 
> Isn't another purpose of those 5% the reduction of fragmentation that 
> occurs more when there is few free space left? Although I also reduce ift 
> on very large partitions. But I never set it to exactly zero.

Perhaps? I don't know. My ext3 partitions with 0% are all for large
files (videos and music) that are more or less static, so I can't say
anything about fragmentation on them. My other partitions are all
reiser, so can't say anything about fragmentation on them either :)

W
-- 
Willie W. Wong                                     wwong@math.princeton.edu
Data aequatione quotcunque fluentes quantitae involvente fluxiones invenire 
         et vice versa   ~~~  I. Newton

[gentoo-user] Re: can't create file but disk isn't full

[Nikos Chantziaras]

On 05/13/2010 01:56 AM, Willie Wong wrote:
> On Thu, May 13, 2010 at 12:25:08AM +0200, Alex Schuster wrote:
>>> The 5% is historical from days when disks are much smaller. If you
>>> have a sensible partition scheme you only really need to reserve the
>>> blocks on the $ROOT filesystem. If the partition in question (IIRC) is
>>> only for /home, then you can just turn off the reserved blocks all
>>> together.
>>
>> Isn't another purpose of those 5% the reduction of fragmentation that
>> occurs more when there is few free space left? Although I also reduce ift
>> on very large partitions. But I never set it to exactly zero.
>
> Perhaps? I don't know. My ext3 partitions with 0% are all for large
> files (videos and music) that are more or less static, so I can't say
> anything about fragmentation on them. My other partitions are all
> reiser, so can't say anything about fragmentation on them either :)

The tune2fs man page mentions that fragmentation is also a reason:

  -m reserved-blocks-percentage
     Set the percentage of the filesystem which may only be allocated
     by privileged processes. Reserving some number of filesystem
     blocks for use by privileged processes is done to avoid
     filesystem fragmentation, and to allow system daemons, such as
     syslogd(8), to continue to function correctly after non-
     privileged processes are prevented from writing to the
     filesystem. Normally, the default percentage of reserved blocks
     is 5%.