[gentoo-dev] coreutils patch for 'ls -l' of GB files

[gentoo-dev] coreutils patch for 'ls -l' of GB files

[Jason Cooper]

Well, the directions said to send problems to the gentoo developers
before sending them upstream to the package owners, so I'll toss this
your way :)

I've been doing a lot of work with multi-GB files, and I use the command
'watch -n 3 ls -lR workdir/' while my script is ripping the dvd.  I do
_not_ want human-readable output, eg 'ls -lh' because the files are
different scales, some bytes, some kb, some gb, etc.  

Here is the problem: file sizes with more than eight digits make the
output not line up.  This is because 8 characters minimum is hard coded 
into the ls source.  So I changed the hard coded width to 10.  Here is 
the output before:

################old output##################
total 10504120
-rw-r--r--    1 root     root         1163 Apr 24 10:39 dvd9to5.log
-rw-r--r--    1 root     root     4200529896 Apr 24 10:39 movie.m2v
-rw-r--r--    1 root     root     2040659968 Apr 24 10:42 movie.mpeg
-rw-r--r--    1 root     root     295174144 Apr 24 10:34 ofile.ac3
-rw-r--r--    1 root     root     4219849312 Apr 24 10:34 ofile.m2v


And the output after:

################new output##################
total 12989797
drwxr-xr-x    4 root     root             96 Apr 24 10:47 dvd
-rw-r--r--    1 root     root           1903 Apr 24 10:56 dvd9to5.log
-rw-r--r--    1 root     root     4200529896 Apr 24 10:39 movie.m2v
-rw-r--r--    1 root     root     4585990144 Apr 24 10:47 movie.mpeg
-rw-r--r--    1 root     root      295174144 Apr 24 10:34 ofile.ac3
-rw-r--r--    1 root     root     4219849312 Apr 24 10:34 ofile.m2v


So my question for you guys is this:  Should I submit this to the
coreutils owner?  It's stupid simple, literally a one-line patch.  I'd
make it a command line option, but I think they left it hard-coded for a
reason...  

TIA,

Cooper.

Patch attached.

diff -Nurd coreutils-5.0.91/src/ls.c coreutils-5.0.91.lsgb/src/ls.c
--- coreutils-5.0.91/src/ls.c 2003-07-27 02:33:36.000000000 -0400
+++ coreutils-5.0.91.lsgb/src/ls.c  2004-04-24 11:10:55.313158520 -0400
@@ -3008,7 +3008,7 @@
    are actually positive values that have wrapped around.  */
       size += (f->stat.st_size < 0) * ((uintmax_t) OFF_T_MAX -
OFF_T_MIN + 1);

-      sprintf (p, "%8s ",
+      sprintf (p, "%10s ",
          human_readable (size, hbuf, human_output_opts,
                1, file_output_block_size));
     }


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Daniel Drake]

Hi,

Jason Cooper wrote:

> So my question for you guys is this:  Should I submit this to the
> coreutils owner?  It's stupid simple, literally a one-line patch.  I'd
> make it a command line option, but I think they left it hard-coded for a
> reason...  

If you make your new behaviour default, you may break many peoples scripts 
that expect the size column to be 8 characters long.

My opinion is that your modification should become accessible only through a 
non-default commandline parameter.

Daniel

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Daniel Drake]

Jason Cooper wrote:
> Agreed, would an "auto-adjusting width" be a worthwhile endevour?  It
> seems kind of pointless to add a commandline option just to increase the
> column width two characters, not to mention temporary (tera/peta-byte size
> files?).  A more robust solution would be for the option to allow the
> column width to shrink and grow based on the largest file in the list.

Take a look at the NameWidth option here:
http://httpd.apache.org/docs-2.0/mod/mod_autoindex.html#indexoptions
(ok, its for file names, but the same principle applies)

So we could have something like:

ls -l
  - gives sizes as 8 characters as normal

ls -l --sizewidth=10
  - gives size column as 10 characters

ls -l --sizewidth=*
  - size column auto-expands to fit


Daniel

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Jason Cooper]

Daniel Drake (dsd@gentoo.org) scribbled:
> Hi,
> 
> Jason Cooper wrote:
> 
> >So my question for you guys is this:  Should I submit this to the
> >coreutils owner?  It's stupid simple, literally a one-line patch.  I'd
> >make it a command line option, but I think they left it hard-coded for a
> >reason...  
> 
> If you make your new behaviour default, you may break many peoples scripts 
> that expect the size column to be 8 characters long.

You're right, I hadn't thought of that.  I'll blame a lack of coffee on
a Saturday morning :)

> 
> My opinion is that your modification should become accessible only through 
> a non-default commandline parameter.
> 

Agreed, would an "auto-adjusting width" be a worthwhile endevour?  It
seems kind of pointless to add a commandline option just to increase the
column width two characters, not to mention temporary (tera/peta-byte size
files?).  A more robust solution would be for the option to allow the
column width to shrink and grow based on the largest file in the list.

Thanks for your patience, 

Cooper.


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Christian Birchinger]

On Sat, Apr 24, 2004 at 11:50:06AM -0400, Jason Cooper wrote:
> I've been doing a lot of work with multi-GB files, and I use the command
> 'watch -n 3 ls -lR workdir/' while my script is ripping the dvd.  I do
> _not_ want human-readable output, eg 'ls -lh' because the files are
> different scales, some bytes, some kb, some gb, etc.  

You could use "ls -lk" to make it display everything in
kilobytes (same scale for everything).

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Drake Wyrm]

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

On Sat, 2004-04-24, 17:55:03 -0400, in <20040424215503.GC14678@lakedaemon.net>, Jason Cooper <gentoo@lakedaemon.net> wrote:
> Daniel Drake (dsd@gentoo.org) scribbled:
> > Jason Cooper wrote:
> > 
> > >So my question for you guys is this:  Should I submit this to the
> > >coreutils owner?  It's stupid simple, literally a one-line patch.  I'd
> > >make it a command line option, but I think they left it hard-coded for a
> > >reason...  

Probably the same reason programmers used two-digit years for so long.
"This will work for now. I'll fix it later, before it becomes a
problem." What was the largest hard drive available in 1985? Files
larger than 95 GB were never going to happen...

> > If you make your new behaviour default, you may break many peoples scripts 
> > that expect the size column to be 8 characters long.

Files larger than 99999999 bytes will also break scripts which depend
on an 8-character size column.

> > My opinion is that your modification should become accessible only through 
> > a non-default commandline parameter.
> 
> Agreed, would an "auto-adjusting width" be a worthwhile endevour?  It
> seems kind of pointless to add a commandline option just to increase the
> column width two characters, not to mention temporary (tera/peta-byte size
> files?).  A more robust solution would be for the option to allow the
> column width to shrink and grow based on the largest file in the list.

Frankly, an "auto-adjusting width" should probably be the default. It
should not break anything which grabs the fifth space-separated field,
and anything which grabs characters 35 through 42 is broken, anyway.

Thinking from another point of view, the output from `ls -l` is intended
for human consumption. Robust scripts which need to determine the size
of a file (should) use `stat -c %s`.

-- 
Batou: Hey, Major... You ever hear of "human rights"?
Kusanagi: I understand the concept, but I've never seen it in action.
  --Ghost in the Shell

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Mike Frysinger]

On Saturday 24 April 2004 01:09 pm, Daniel Drake wrote:
> If you make your new behaviour default, you may break many peoples scripts
> that expect the size column to be 8 characters long.
>
> My opinion is that your modification should become accessible only through
> a non-default commandline parameter.

actually i dont think anyone does that since 8 char is the *max* not the size 
*always* used

i was just programming something when i noticed that i had a dir of files that 
all were under 10000bytes ... then i made a file (by accident) that had more 
than that ... the output of `ls -l` changed to accomodate the larger file ... 
when i deleted it, the # of chars used to output the size shrunk from 
before ...

so i dont think you'd break anything if the new patch became default 
behavior ... 'proper' parsing would have done it via awk or cut or something 
that operated on existence of whitespace (and not the size) as the 
delimiter ...

as such, please file a bug with your patch Jason and assign it to 
base-system@gentoo.org
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Jason Cooper]

Mike Frysinger (vapier@gentoo.org) scribbled:
> On Saturday 24 April 2004 01:09 pm, Daniel Drake wrote:
> > If you make your new behaviour default, you may break many peoples scripts
> > that expect the size column to be 8 characters long.
> >
> > My opinion is that your modification should become accessible only through
> > a non-default commandline parameter.
> 
> actually i dont think anyone does that since 8 char is the *max* not the size 
> *always* used

?  Here is the output of my `ls -l` after creating 15 5k files w/ dd.

total 120
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp01
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp02
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp03
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp04
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp05
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp06
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp07
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp08
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp09
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp10
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp11
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp12
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp13
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp14
-rw-r--r--    1 jcooper  users        5120 Apr 29 07:39 temp15
                |        |        |        |   <---8 chars and a space.

This is with the unmodified version.  It appears to be staying with
eight chars width.  What version of coreutils are you using?  I'm using:

*  sys-apps/coreutils
      Latest version available: 5.0.91-r4
      Latest version installed: 5.0.91-r4
> 
> i was just programming something when i noticed that i had a dir of files that 
> all were under 10000bytes ... then i made a file (by accident) that had more 
> than that ... the output of `ls -l` changed to accomodate the larger file ... 
> when i deleted it, the # of chars used to output the size shrunk from 
> before ...
> 
> so i dont think you'd break anything if the new patch became default 
> behavior ... 'proper' parsing would have done it via awk or cut or something 
> that operated on existence of whitespace (and not the size) as the 
> delimiter ...
> 
> as such, please file a bug with your patch Jason and assign it to 
> base-system@gentoo.org
> -mike

Working on it. :)


Cooper.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Mike Frysinger]

On Thursday 29 April 2004 08:32 am, Jason Cooper wrote:
> ?  Here is the output of my `ls -l` after creating 15 5k files w/ dd.
>
> This is with the unmodified version.  It appears to be staying with
> eight chars width.  What version of coreutils are you using?  I'm using:

5.2.0
and i just checked it and it seems that gig-sized files are shown correctly 
with 5.2.0
root@vapier 0 / # ls -lh | tail
drwxr-xr-x    2 root root 4.0K Feb  5 01:58 service
drwxr-xr-x    8 root root    0 Apr 27 22:10 sys
drwxrwxrwt   29 root root  12K May  3 14:56 tmp
drwxr-xr-x   18 root root 4.0K May  2 15:50 usr
drwxr-xr-x   19 root root 4.0K May  2 02:59 var
-rw-r--r--    2 root root 2.2M May  2 04:30 vmlinuz
-rw-r--r--    1 root root 1.3M Jul  2  2003 vmlinuz-2.4.21
-rw-r--r--    2 root root 2.2M May  2 04:30 vmlinuz-2.6
-rw-r--r--    1 root root 2.2M Apr 27 22:01 vmlinuz-2.6-work
-rw-r--r--    1 root root 2.1M Apr  5 17:04 vmlinuz-2.6.3
root@vapier 0 / # ls -l | tail
drwxr-xr-x    2 root root    4096 Feb  5 01:58 service
drwxr-xr-x    8 root root       0 Apr 27 22:10 sys
drwxrwxrwt   29 root root   12288 May  3 14:56 tmp
drwxr-xr-x   18 root root    4096 May  2 15:50 usr
drwxr-xr-x   19 root root    4096 May  2 02:59 var
-rw-r--r--    2 root root 2280614 May  2 04:30 vmlinuz
-rw-r--r--    1 root root 1264091 Jul  2  2003 vmlinuz-2.4.21
-rw-r--r--    2 root root 2280614 May  2 04:30 vmlinuz-2.6
-rw-r--r--    1 root root 2285561 Apr 27 22:01 vmlinuz-2.6-work
-rw-r--r--    1 root root 2139784 Apr  5 17:04 vmlinuz-2.6.3
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] coreutils patch for 'ls -l' of GB files

[Peter Ruskin]

On Monday 03 May 2004 19:56, Mike Frysinger wrote:
> On Thursday 29 April 2004 08:32 am, Jason Cooper wrote:
> > ?  Here is the output of my `ls -l` after creating 15 5k files w/
> > dd.
> >
> > This is with the unmodified version.  It appears to be staying with
> > eight chars width.  What version of coreutils are you using?  I'm
> > using:
>
> 5.2.0
> and i just checked it and it seems that gig-sized files are shown
> correctly with 5.2.0
> root@vapier 0 / # ls -lh | tail
> drwxr-xr-x    2 root root 4.0K Feb  5 01:58 service
> drwxr-xr-x    8 root root    0 Apr 27 22:10 sys
> drwxrwxrwt   29 root root  12K May  3 14:56 tmp
> drwxr-xr-x   18 root root 4.0K May  2 15:50 usr
> drwxr-xr-x   19 root root 4.0K May  2 02:59 var
> -rw-r--r--    2 root root 2.2M May  2 04:30 vmlinuz
> -rw-r--r--    1 root root 1.3M Jul  2  2003 vmlinuz-2.4.21
> -rw-r--r--    2 root root 2.2M May  2 04:30 vmlinuz-2.6
> -rw-r--r--    1 root root 2.2M Apr 27 22:01 vmlinuz-2.6-work
> -rw-r--r--    1 root root 2.1M Apr  5 17:04 vmlinuz-2.6.3
> root@vapier 0 / # ls -l | tail
> drwxr-xr-x    2 root root    4096 Feb  5 01:58 service
> drwxr-xr-x    8 root root       0 Apr 27 22:10 sys
> drwxrwxrwt   29 root root   12288 May  3 14:56 tmp
> drwxr-xr-x   18 root root    4096 May  2 15:50 usr
> drwxr-xr-x   19 root root    4096 May  2 02:59 var
> -rw-r--r--    2 root root 2280614 May  2 04:30 vmlinuz
> -rw-r--r--    1 root root 1264091 Jul  2  2003 vmlinuz-2.4.21
> -rw-r--r--    2 root root 2280614 May  2 04:30 vmlinuz-2.6
> -rw-r--r--    1 root root 2285561 Apr 27 22:01 vmlinuz-2.6-work
> -rw-r--r--    1 root root 2139784 Apr  5 17:04 vmlinuz-2.6.3

Hmmm, that doesn't prove anything because your largest file is only 
2.2M.  However...

[21:45 root@peter /mnt/win/n/Pan/Men.In.Black/video_ts]
# ls -l
total 3877248
-rwxrwxrwx  1 root root       8192 Apr 27 13:24 video_ts.bup
-rwxrwxrwx  1 root root       8192 Apr 27 13:24 video_ts.ifo
-rwxrwxrwx  1 root root      12288 Apr 27 13:24 vts_01_0.bup
-rwxrwxrwx  1 root root      12288 Apr 27 13:24 vts_01_0.ifo
-rwxrwxrwx  1 root root   14145536 Apr 27 13:24 vts_01_1.vob
-rwxrwxrwx  1 root root      12288 Apr 27 13:24 vts_02_0.bup
-rwxrwxrwx  1 root root      12288 Apr 27 13:24 vts_02_0.ifo
-rwxrwxrwx  1 root root    7811072 Apr 27 13:24 vts_02_1.vob
-rwxrwxrwx  1 root root      63488 Apr 27 13:24 vts_03_0.bup
-rwxrwxrwx  1 root root      63488 Apr 27 13:24 vts_03_0.ifo
-rwxrwxrwx  1 root root 1073739776 Apr 27 13:39 vts_03_1.vob
-rwxrwxrwx  1 root root 1073739776 Apr 27 13:42 vts_03_2.vob
-rwxrwxrwx  1 root root 1073739776 Apr 27 13:46 vts_03_3.vob
-rwxrwxrwx  1 root root  726738944 Apr 27 13:52 vts_03_4.vob

...so with coreutils-5.2.0-r1 ls does display correctly.

-- 
Peter
========================================================================
Gentoo Linux: Portage 2.0.50-r6.	kernel-2.6.5-gentoo-r1.
i686 AMD Athlon(tm) XP 3200+.		gcc(GCC): 3.3.2.
KDE: 3.2.2.				Qt: 3.3.2.
========================================================================

--
gentoo-dev@gentoo.org mailing list