[gentoo-user] Ext3 FS File Size Limits

[gentoo-user] Ext3 FS File Size Limits

[Fast Turtle]

I've been going around with this little problem for a while.

I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped.

One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there?

I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems

[defaults]
	base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
	default_mntopts = acl,user_xattr
	enable_periodic_fsck = 0
	blocksize = 4096
	inode_size = 256
	inode_ratio = 16384

Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc

I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. 

Re: [gentoo-user] Ext3 FS File Size Limits

[Dale]

Fast Turtle wrote:
> I've been going around with this little problem for a while.
>
> I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped.
>
> One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there?
>
> I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems
>
> [defaults]
> 	base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
> 	default_mntopts = acl,user_xattr
> 	enable_periodic_fsck = 0
> 	blocksize = 4096
> 	inode_size = 256
> 	inode_ratio = 16384
>
> Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc
>
> I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. 
>
>


I can offer this:  df -i shows inodes. 

root@fireball / # df -i
Filesystem                   Inodes  IUsed     IFree IUse% Mounted on
rootfs                      1525920  22728   1503192    2% /
/dev/sda6                   1525920  22728   1503192    2% /
devtmpfs                    2049540    593   2048947    1% /dev
tmpfs                       2058249    654   2057595    1% /run
shm                         2058249      2   2058247    1% /dev/shm
/dev/sda1                     98392    794     97598    1% /boot
/dev/mapper/OS-usr          1638400 462712   1175688   29% /usr
/dev/mapper/OS-var          1703936 259049   1444887   16% /var
/dev/mapper/home-home     183148544 316215 182832329    1% /home
/dev/mapper/backup-backup  61046784   5818  61040966    1% /backup
tmpfs                       2058249 122993   1935256    6% /var/tmp/portage
root@fireball / #


Hope that helps on that part at least. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] Ext3 FS File Size Limits

[Wang Xuerui]

2013/6/2 Fast Turtle <fturtle@gmail.com>:
> One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there?

You can try `touch`ing a file on that partition and see if it fails.
If so, you've run out of inodes; no need to peek at fs internals :)

Re: [gentoo-user] Ext3 FS File Size Limits

[Alan Mackenzie]

On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote:
> I've been going around with this little problem for a while.

> I have several 30GB files I'm trying to restore from an NTFS formatted
> external backup to an ext3 partition, yet every attempt has failed
> right after 16GB of copying without fail or error message. They
> silently failing and I'm stumped.

> One of the possible causes I've thought of was running out of innodes
> but don't know how to check that or any of the other options used to
> create the file system on - anyone want to help there?

> I've also decided to look at the mke2f.conf file in /etc and see some
> default options being passed that may be causing the problems

> [defaults]
> 	base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
> 	default_mntopts = acl,user_xattr
> 	enable_periodic_fsck = 0
> 	blocksize = 4096
> 	inode_size = 256
> 	inode_ratio = 16384

> Normally I use either a 1024 for most everything due to the many small
> files though for the partition I'm attempting to restore the files to,
> I've used 2048 as a compromise due to the number of larger files
> (music/videos) and critical backups from /etc

The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum
file size is 16GB, but with a 2kB blocksize it's 256GB.  Could it be
you've somehow actually got a 1kB blocksize on the partition?

> I've also tried it with a default 4096 size on a 32GB ext2 formatted
> flash drive but even then, it's failing at 16GB w/o any error message. 

Sounds like laziness on the part of the author of the copy program you're
using.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [gentoo-user] Ext3 FS File Size Limits

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]

On Sun, 2 Jun 2013 04:55:40 -0700, Fast Turtle wrote:

> I have several 30GB files I'm trying to restore from an NTFS formatted
> external backup to an ext3 partition, yet every attempt has failed
> right after 16GB of copying without fail or error message. They
> silently failing and I'm stumped.

Are you sure its the destination causing the problem? What happens
if you try to copy to another directory on the NTFS filesystem?


-- 
Neil Bothwick

Mosquito - designed to make houseflies look better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Ext3 FS File Size Limits

[Bruce Hill]

On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote:
> I've been going around with this little problem for a while.
> 
> I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped.
> 
> One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there?
> 
> I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems
> 
> [defaults]
> 	base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
> 	default_mntopts = acl,user_xattr
> 	enable_periodic_fsck = 0
> 	blocksize = 4096
> 	inode_size = 256
> 	inode_ratio = 16384
> 
> Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc
> 
> I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. 

What command are you using to copy this data? Try:

rsync -av /source/ /destination/

and see if you get any output when it fails. There have been other guesses,
but with the information you've provided, that's all we can do. My "guess" is
you're using cp and it gets to a large, corrupt file and fails.

Post back with your command and it's output, then the rsync command above and
it's output; as well as the inode command Dale gave you, and any other
applicable commands you've had suggested.
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.                                                                                                                                                          
Q: Why is top-posting such a bad thing?                                                                                                                                                                                        
A: Top-posting.                                                                                                                                                                                                                
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] Ext3 FS File Size Limits

[Joerg Schilling]

Alan Mackenzie <acm@muc.de> wrote:

> The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum
> file size is 16GB, but with a 2kB blocksize it's 256GB.  Could it be
> you've somehow actually got a 1kB blocksize on the partition?

Where does such a strange limitation come from?

Ext* started as a UFS "clone" and UFS filesize is limited to 2**63 while
UFS filesystem size is limited to 1 TB.

That is much more than you claim for Ext3

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] Ext3 FS File Size Limits

[Alan Mackenzie]

Hello, Jörg.

On Sun, Jun 02, 2013 at 04:06:11PM +0200, Joerg Schilling wrote:
> Alan Mackenzie <acm@muc.de> wrote:

> > The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum
> > file size is 16GB, but with a 2kB blocksize it's 256GB.  Could it be
> > you've somehow actually got a 1kB blocksize on the partition?

> Where does such a strange limitation come from?

Haven't a clue.  I would have expected the maximum file size to be a
number of blocks, which makes it seem strange that doubling the block
size multiplies max file size by 16.

> Ext* started as a UFS "clone" and UFS filesize is limited to 2**63 while
> UFS filesystem size is limited to 1 TB.

Just for ease of comparison, 16GB = 2**34 bytes = 2**24 1k blocks.  1TB =
2**40 bytes.

> That is much more than you claim for Ext3

I'm not doing any claiming, since I'm not an expert on the subject.  I
was just drawing the OP's attention to something which might be useful.

> Jörg

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [gentoo-user] Ext3 FS File Size Limits

[gottlieb]

On Sun, Jun 02 2013, Alan Mackenzie wrote:

> Hello, Jörg.
>
> On Sun, Jun 02, 2013 at 04:06:11PM +0200, Joerg Schilling wrote:
>> Alan Mackenzie <acm@muc.de> wrote:
>
>> > The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum
>> > file size is 16GB, but with a 2kB blocksize it's 256GB.  Could it be
>> > you've somehow actually got a 1kB blocksize on the partition?
>
>> Where does such a strange limitation come from?
>
> Haven't a clue.  I would have expected the maximum file size to be a
> number of blocks, which makes it seem strange that doubling the block
> size multiplies max file size by 16.


The wonders of indirect blocks.  When the blocksize doubles, a single
indirect block points to twice as many direct blocks, each of which is
double the size.  For double indirect you get twice as many single
indirect, etc.

There are plenty of places to read about this.  One is my lecture notes
http://cs.nyu.edu/~gottlieb/courses/os/class-notes.html#inodes

allan

Re: [gentoo-user] Ext3 FS File Size Limits

[James Cloos]

>>>>> "AM" == Alan Mackenzie <acm@muc.de> writes:

AM> Haven't a clue.  I would have expected the maximum file size to be a
AM> number of blocks, which makes it seem strange that doubling the block
AM> size multiplies max file size by 16.

Doubling the block size means that the structure specifying which blocks
are in use by a given inode is twice as long.  And the structure
supports indirect blocks, which contain references to the actual blocks
used by the file data.  So you get one doubling for the size of the
blocks, and another three for the number of blocks each file can use.

Most ext3 and ext4 filesystems use 4096 octet blocks.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: [gentoo-user] Ext3 FS File Size Limits

[Fast Turtle]

On Sun, 2 Jun 2013 13:01:22 +0000
Alan Mackenzie <acm@muc.de> wrote:

> On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote:
> > I've been going around with this little problem for a while.
> 
> > I have several 30GB files I'm trying to restore from an NTFS formatted
> > external backup to an ext3 partition, yet every attempt has failed
> > right after 16GB of copying without fail or error message. They
> > silently failing and I'm stumped.
> 
> > One of the possible causes I've thought of was running out of innodes
> > but don't know how to check that or any of the other options used to
> > create the file system on - anyone want to help there?
> 
> > I've also decided to look at the mke2f.conf file in /etc and see some
> > default options being passed that may be causing the problems
> 
> > [defaults]
> > 	base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
> > 	default_mntopts = acl,user_xattr
> > 	enable_periodic_fsck = 0
> > 	blocksize = 4096
> > 	inode_size = 256
> > 	inode_ratio = 16384
> 
> > Normally I use either a 1024 for most everything due to the many small
> > files though for the partition I'm attempting to restore the files to,
> > I've used 2048 as a compromise due to the number of larger files
> > (music/videos) and critical backups from /etc
> 
> The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum
> file size is 16GB, but with a 2kB blocksize it's 256GB.  Could it be
> you've somehow actually got a 1kB blocksize on the partition?
> 
> > I've also tried it with a default 4096 size on a 32GB ext2 formatted
> > flash drive but even then, it's failing at 16GB w/o any error message. 
> 
> Sounds like laziness on the part of the author of the copy program you're
> using.
> 
> -- 
> Alan Mackenzie (Nuremberg, Germany).
> 
Thanks for the wikipedia info (could very well be the problem) Don't recall if I used 1k block sizes on the partition but it could be. Would certainly explain the failure right on 16GB - filesize was to large (never had problems copying between ntfs formatted drives under Windows).

Re: [gentoo-user] Ext3 FS File Size Limits

[Fast Turtle]

On Sun, 02 Jun 2013 07:11:11 -0500
Dale <rdalek1967@gmail.com> wrote:

> Fast Turtle wrote:
> > I've been going around with this little problem for a while.
> >
> > I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped.
> >
> > One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there?
> >
> > I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems
> >
> > [defaults]
> > 	base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
> > 	default_mntopts = acl,user_xattr
> > 	enable_periodic_fsck = 0
> > 	blocksize = 4096
> > 	inode_size = 256
> > 	inode_ratio = 16384
> >
> > Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc
> >
> > I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. 
> >
> >
> 
> 
> I can offer this:  df -i shows inodes. 
> 
> root@fireball / # df -i
> Filesystem                   Inodes  IUsed     IFree IUse% Mounted on
> rootfs                      1525920  22728   1503192    2% /
> /dev/sda6                   1525920  22728   1503192    2% /
> devtmpfs                    2049540    593   2048947    1% /dev
> tmpfs                       2058249    654   2057595    1% /run
> shm                         2058249      2   2058247    1% /dev/shm
> /dev/sda1                     98392    794     97598    1% /boot
> /dev/mapper/OS-usr          1638400 462712   1175688   29% /usr
> /dev/mapper/OS-var          1703936 259049   1444887   16% /var
> /dev/mapper/home-home     183148544 316215 182832329    1% /home
> /dev/mapper/backup-backup  61046784   5818  61040966    1% /backup
> tmpfs                       2058249 122993   1935256    6% /var/tmp/portage
> root@fireball / #
> 
> 
> Hope that helps on that part at least. 
> 
> Dale
> 
> :-)  :-) 
> 
> -- 
> I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
> 
> 
Thanks Dale - Interesting what info you can get

All: Accidently discovered that tune2fs -l /dev/sda# lists all of the config info so I shouldn't have any problems 

Filesystem volume name:   home
Last mounted on:          <not available>
Filesystem UUID:          d4102f68-defd-497b-84e4-f165fd171ed7
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              39075840
Block count:              156282705
Reserved block count:     7814135
Free blocks:              104269552
Free inodes:              38399824
First block:              0
Block size:               4096
Fragment size:            4096

Had KDE lockup the other day and it forced a clean install w/o it. Seems I inadvertently formatted /home using the standard 4k block size this time around. 

Would explain why the 30GB files copied over w/o issue this time

I'll repost this: Thanks for the Wiki Entry Alan (keep forgetting to check it) as that certainly explains the issue and I'm going to put that into my System Log book (important notes section).

When I started playing with Linux back in 2000, I never thought much about file size limits as ext2 could handle files larger then Win98 could. Now it's come full circle as you do have to be aware of the formatting because it can affect file size limits as I discovered.

Re: [gentoo-user] Ext3 FS File Size Limits

[Fast Turtle]

On Sun, 2 Jun 2013 13:01:22 +0000
Alan Mackenzie <acm@muc.de> wrote:

> On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote:
> > I've been going around with this little problem for a while.
> 
> > I have several 30GB files I'm trying to restore from an NTFS formatted
> > external backup to an ext3 partition, yet every attempt has failed
> > right after 16GB of copying without fail or error message. They
> > silently failing and I'm stumped.
> 
> > One of the possible causes I've thought of was running out of innodes
> > but don't know how to check that or any of the other options used to
> > create the file system on - anyone want to help there?
> 
> > I've also decided to look at the mke2f.conf file in /etc and see some
> > default options being passed that may be causing the problems
> 
> > [defaults]
> > 	base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
> > 	default_mntopts = acl,user_xattr
> > 	enable_periodic_fsck = 0
> > 	blocksize = 4096
> > 	inode_size = 256
> > 	inode_ratio = 16384
> 
> > Normally I use either a 1024 for most everything due to the many small
> > files though for the partition I'm attempting to restore the files to,
> > I've used 2048 as a compromise due to the number of larger files
> > (music/videos) and critical backups from /etc
> 
> The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum
> file size is 16GB, but with a 2kB blocksize it's 256GB.  Could it be
> you've somehow actually got a 1kB blocksize on the partition?
> 
> > I've also tried it with a default 4096 size on a 32GB ext2 formatted
> > flash drive but even then, it's failing at 16GB w/o any error message. 
> 
> Sounds like laziness on the part of the author of the copy program you're
> using.

if it is, then everyone is in trouble as it's the cp command - been part of *nix for many years

> 
> -- 
> Alan Mackenzie (Nuremberg, Germany).
>