[gentoo-user] Fragmentation of my drives. Curious mostly

[gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Hi folks,

I have a pretty old install tho I have moved it from one drive to
another once, using cp -av.  It got to big for the old hard drive. 
Anyway, the install is about 5 years old or so and about 3 years since
it got moved.  I have Gentoo on it naturally, a ship load of pictures
and a few other standard documents and such.  I found something called
frag check on the forums a long time ago and I got this from it:

> 9.46189733169307% non contiguous files, 1.33812703640673 average
> fragments.

I emerged shake and ran it on each partition then got this:

> 9.83100532514448% non contiguous files, 1.37302298534689 average
> fragments.

I'm not sure but it looks like it is more fragmented now than it was. 
It may just be that given file sizes and such that that is as good as it
gets.  Here is some additional info:

> root@smoker / # mount
> /dev/hda6 on / type reiserfs (rw)
> /dev/hda1 on /boot type ext2 (rw)
> /dev/hda7 on /usr/portage type ext3 (rw)
> /dev/hda8 on /home type reiserfs (rw)
> /dev/hda9 on /data type reiserfs (rw)
> root@smoker / # 

> root@smoker / # df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/hda6             19530340   5938156  13592184  31% /
> /dev/hda1               189339     14901    164662   9% /boot
> /dev/hda7              4806048   2692300   1869612  60% /usr/portage
> /dev/hda8              4882532   1758376   3124156  37% /home
> /dev/hda9             48596656  22332356  26264300  46% /data
> root@smoker / # 

Is a little under 10% normal and acceptable?  As you may notice, I use
reiserfs.  It has never been more than 10% that I can recall, even when
it does its checking during bootup.  I'm not sure what is more accurate,
frag check, shake or the check during boot up.  Shake is new to me so it
does not make much sense yet.  Give me a while on it.  LOL 

If this is a little high, what would be the best way to defrag it?  Is
there a best way?  I do have a second hard drive that I back up too. 
Both Drives are 80Gbs and I do have a set of DVD back ups as well.  I
can update those pretty quick.

Thoughts??  Thanks !!!

Dale

:-)  :-) 

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Friday 28 November 2008 13:14:42 Dale wrote:
> If this is a little high, what would be the best way to defrag it? 

By not defragging it.

It's not Windows. Windows boxes needs defragging not because fragmentation is 
a huge problem in itself, but because windows filesystems are a steaming mess 
of cr@p that do little right and most things wrong. Defrag treats the 
symptom, not the cause :-)

Reiser tends to self-balance itself out. What is especially noteworthy is that 
none of the general purpose Linux filesystems provide a defrag utility. 
Theodore 'Tso and Hans Reiser are both exceptional programmers, if there was 
a need for such a tool they would assuredly have written one. They did not, 
so there probably isn't.

Any Linux defrag tool you encounter will have been written by a third party 
separate from the developers. It will move blocks around and update 
superblocks, the drive will have to be unmounted for that to work and a 
slight misunderstanding of how to do it will ruin data.

Are you willing to take the very real risk of data corruption?

> Is 
> there a best way?  I do have a second hard drive that I back up too.
> Both Drives are 80Gbs and I do have a set of DVD back ups as well.  I
> can update those pretty quick.



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Florian Philipp]

Alan McKinnon schrieb:
[...]
> Reiser tends to self-balance itself out. What is especially noteworthy is that 
> none of the general purpose Linux filesystems provide a defrag utility. 
> Theodore 'Tso and Hans Reiser are both exceptional programmers, if there was 
> a need for such a tool they would assuredly have written one. They did not, 
> so there probably isn't.
> 
[...]

Well, as far as I know, ext4 will get an online defrag tool.

I once experienced performance losses on a reiserfs-volume used for 
ccache (tailpacking enabled).

But otherwise you are right, fragmentation is usually caused by a bad 
filesystem (FAT*), a filesystem that is mostly filled (when 99% is full, 
the allocator has no choice but scatter any new writes over the whole 
volume) or unusual usage patterns (write-delete-write-delete-... like my 
ccache issue).

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Stroller]

On 28 Nov 2008, at 11:46, Alan McKinnon wrote:

> On Friday 28 November 2008 13:14:42 Dale wrote:
>> If this is a little high, what would be the best way to defrag it?
>
> By not defragging it.
>
> It's not Windows. Windows boxes needs defragging not because  
> fragmentation is
> a huge problem in itself, but because windows filesystems are a  
> steaming mess
> of cr@p that do little right and most things wrong. Defrag treats the
> symptom, not the cause :-)

I have occasionally heard this said about the Mac's HFS filing system,  
too - that "it's not windows", it doesn't need defragging, Apple  
provides no tools for defragging (true) and that the benefits of  
defragging are mythical.

Whilst I have to agree that many of the system repair & maintenance  
"techniques" I've read used by well-established Mac "technicians" are  
clearly smoke & mirrors, I have myself noticed REMARKABLE performance  
improvements by running iDefrag.

> Reiser tends to self-balance itself out. What is especially  
> noteworthy is that
> none of the general purpose Linux filesystems provide a defrag  
> utility.
> Theodore 'Tso and Hans Reiser are both exceptional programmers, if  
> there was
> a need for such a tool they would assuredly have written one. They  
> did not,
> so there probably isn't.

I understood that ReiserFS's trees could become out-of-balance,  
resulting in performance loss, and that the way to deal with this was  
to tar the contents of the drive to another file-system and then untar  
them back.

More information on the subject - beyond the sort of idle chitchat one  
finds on lists like this one - is not terribly obvious, but Wikipedia  
suggests I may possibly be correct:

    Also, ReiserFS had a problem with very fast filesystem aging when
    compared to other filesystems - in several usage scenarios  
filesystem
    performance lowered dramatically with time.
    ...
    There are no programs to specifically defragment a ReiserFS file
    system, although tools have been written to automatically copy the
    contents of fragmented files hoping that more contiguous blocks of
    free space can be found. However, Reiser4 will have a repacker that
    optimizes file fragmentation.

(It also seems to suggest that tarring the files elsewhere &  
reformatting the drive is not the best solution. Using `rm` to remove  
everything from the filesystem may do the trick, but you might want to  
dd zeros onto it before reformatting to be sure).

Stroller.

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Florian Philipp wrote:
> Alan McKinnon schrieb:
> [...]
>> Reiser tends to self-balance itself out. What is especially
>> noteworthy is that none of the general purpose Linux filesystems
>> provide a defrag utility. Theodore 'Tso and Hans Reiser are both
>> exceptional programmers, if there was a need for such a tool they
>> would assuredly have written one. They did not, so there probably isn't.
>>
> [...]
>
> Well, as far as I know, ext4 will get an online defrag tool.
>
> I once experienced performance losses on a reiserfs-volume used for
> ccache (tailpacking enabled).
>
> But otherwise you are right, fragmentation is usually caused by a bad
> filesystem (FAT*), a filesystem that is mostly filled (when 99% is
> full, the allocator has no choice but scatter any new writes over the
> whole volume) or unusual usage patterns (write-delete-write-delete-...
> like my ccache issue).
>
>


I have read the same as what you are saying but I did want to "test"
that shake thing.  What is funny to me, it was higher AFTER running a
defrag tool than it was BEFORE running it.  Sort of makes you think.

I have said myself that Linux does not generally need to be defraged.  I
have never seen a Linux file system get anything near as bad as
windoze.  While I don't run windoze I do have family and friends that do
so I know how bad it can be.  I have seen a lot of windoze be at 40 and
50%.  Looked like about every file on the thing was all over the place
like bird shot from a shot gun.  Sorry, I'm a southern country boy.  lol

So I assume 10% or so is not so bad?  I didn't think it was but wanted
to ask a couple gurus for their opinions.

Thanks for the replies.

Dale

:-)  :-)

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Joshua Murphy]

On Fri, Nov 28, 2008 at 6:46 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On Friday 28 November 2008 13:14:42 Dale wrote:
>> If this is a little high, what would be the best way to defrag it?
>
> By not defragging it.
>
> It's not Windows. Windows boxes needs defragging not because fragmentation is
> a huge problem in itself, but because windows filesystems are a steaming mess
> of cr@p that do little right and most things wrong. Defrag treats the
> symptom, not the cause :-)
>
> Reiser tends to self-balance itself out. What is especially noteworthy is that
> none of the general purpose Linux filesystems provide a defrag utility.
> Theodore 'Tso and Hans Reiser are both exceptional programmers, if there was
> a need for such a tool they would assuredly have written one. They did not,
> so there probably isn't.
>
> Any Linux defrag tool you encounter will have been written by a third party
> separate from the developers. It will move blocks around and update
> superblocks, the drive will have to be unmounted for that to work and a
> slight misunderstanding of how to do it will ruin data.
>
> Are you willing to take the very real risk of data corruption?
>
>> Is
>> there a best way?  I do have a second hard drive that I back up too.
>> Both Drives are 80Gbs and I do have a set of DVD back ups as well.  I
>> can update those pretty quick.
>
>
>
> --
> alan dot mckinnon at gmail dot com
>
>

While not trying to incite flames here... xfs isn't general purpose?
xfs_fsr defrags xfs partitions while they're mounted and is designed
to be used from cron (it's in xfsdump, not xfsprogs). File
fragmentation, while a fact of life on any filesystem that sees any
real use, does slow access times, as the drive head has to jump from
one place to another, so a lot of fragmentation is a bad thing... but
as you say, we're not dealing with FAT based FS's here, so severe
fragmentation only shows itself on very full filesystems.  I very
rarely see over 80% usage of my filesystems and have never
consistently checked fragmentation levels, though, so I can't say
whether xfs's being the exception on having a tool for the job means
it particularly needed one...

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Florian Philipp]

Dale schrieb:

> I have said myself that Linux does not generally need to be defraged.  I
> have never seen a Linux file system get anything near as bad as
> windoze.  While I don't run windoze I do have family and friends that do
> so I know how bad it can be.  I have seen a lot of windoze be at 40 and
> 50%.  Looked like about every file on the thing was all over the place
> like bird shot from a shot gun.  Sorry, I'm a southern country boy.  lol

I'm wondering, why is Windows that bad in this regard? Of course, FAT* 
is bad, but what's about NTFS? It is at least as modern as most Linux FS 
and has some nice features. Surely MS should be capable of implementing 
the same allocation algorithms we use. Or is NTFS really not bad in this 
regard and it's all just that people mix experiences with FAT* with NTFS?

> 
> So I assume 10% or so is not so bad?  I didn't think it was but wanted
> to ask a couple gurus for their opinions.
> 

As far as I know, Windows and e2fsck calculate fragmentation differently 
(don't know about this tool for reiser you mentioned). So you can not 
expect these values to be comparable. AFAIK 10% reported by e2fsck are 
worse than 10% reported by Windows.

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Florian Philipp wrote:
> Dale schrieb:
>
>> I have said myself that Linux does not generally need to be defraged.  I
>> have never seen a Linux file system get anything near as bad as
>> windoze.  While I don't run windoze I do have family and friends that do
>> so I know how bad it can be.  I have seen a lot of windoze be at 40 and
>> 50%.  Looked like about every file on the thing was all over the place
>> like bird shot from a shot gun.  Sorry, I'm a southern country boy.  lol
>
> I'm wondering, why is Windows that bad in this regard? Of course, FAT*
> is bad, but what's about NTFS? It is at least as modern as most Linux
> FS and has some nice features. Surely MS should be capable of
> implementing the same allocation algorithms we use. Or is NTFS really
> not bad in this regard and it's all just that people mix experiences
> with FAT* with NTFS?

My brother has windoze XP and NTFS.  It gets very fragmented.  He
defrags that thing pretty regular.  I would think that NTFS may be
better than *FAT but it doesn't seem to be much.

>
>>
>> So I assume 10% or so is not so bad?  I didn't think it was but wanted
>> to ask a couple gurus for their opinions.
>>
>
> As far as I know, Windows and e2fsck calculate fragmentation
> differently (don't know about this tool for reiser you mentioned). So
> you can not expect these values to be comparable. AFAIK 10% reported
> by e2fsck are worse than 10% reported by Windows.
>
>

I have to say, my system is pretty snappy.  I have a AMD 2500+ with 2Gbs
of ram.  My drives are the bottle neck but I can't say that it is
because of it being fragmented.  It is a ATA133 system but one drive
runs ATA100 if I recall correctly.  May be that one drive is older than
the other.

Frag check is a script that someone wrote.  It is not part of reiserfs
tools or anything.  I'm not even sure how accurate it is with reiserfs
or if the file system matters.

Dale

:-)  :-) 

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Joshua Murphy wrote:
> On Fri, Nov 28, 2008 at 6:46 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>   
>> On Friday 28 November 2008 13:14:42 Dale wrote:
>>     
>>> If this is a little high, what would be the best way to defrag it?
>>>       
>> By not defragging it.
>>
>> It's not Windows. Windows boxes needs defragging not because fragmentation is
>> a huge problem in itself, but because windows filesystems are a steaming mess
>> of cr@p that do little right and most things wrong. Defrag treats the
>> symptom, not the cause :-)
>>
>> Reiser tends to self-balance itself out. What is especially noteworthy is that
>> none of the general purpose Linux filesystems provide a defrag utility.
>> Theodore 'Tso and Hans Reiser are both exceptional programmers, if there was
>> a need for such a tool they would assuredly have written one. They did not,
>> so there probably isn't.
>>
>> Any Linux defrag tool you encounter will have been written by a third party
>> separate from the developers. It will move blocks around and update
>> superblocks, the drive will have to be unmounted for that to work and a
>> slight misunderstanding of how to do it will ruin data.
>>
>> Are you willing to take the very real risk of data corruption?
>>
>>     
>>> Is
>>> there a best way?  I do have a second hard drive that I back up too.
>>> Both Drives are 80Gbs and I do have a set of DVD back ups as well.  I
>>> can update those pretty quick.
>>>       
>>
>> --
>> alan dot mckinnon at gmail dot com
>>
>>
>>     
>
> While not trying to incite flames here... xfs isn't general purpose?
> xfs_fsr defrags xfs partitions while they're mounted and is designed
> to be used from cron (it's in xfsdump, not xfsprogs). File
> fragmentation, while a fact of life on any filesystem that sees any
> real use, does slow access times, as the drive head has to jump from
> one place to another, so a lot of fragmentation is a bad thing... but
> as you say, we're not dealing with FAT based FS's here, so severe
> fragmentation only shows itself on very full filesystems.  I very
> rarely see over 80% usage of my filesystems and have never
> consistently checked fragmentation levels, though, so I can't say
> whether xfs's being the exception on having a tool for the job means
> it particularly needed one...
>
>   

Given my experience with XFS, I won't be switching anytime soon.  I used
that once on a in-laws system.  After each crash, power failure, I had
to reinstall.  Let's just say it left a bad taste in my mouth.  ;-)  I'm
not saying it is a bad file system for someone but certainly not for me. 

You are right tho, every file system has some fragmentation.  It just
can't be otherwise.  I guess I could always make my back ups, then redo
my partitions, and copy them back.  I have done that once before. 
Worked very well then but not real sure about how udev would like that. 
I would think it would work OK but call me chicken.

Dale

:-)  :-) 

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Joshua Murphy]

On Fri, Nov 28, 2008 at 12:24 PM, Dale <rdalek1967@gmail.com> wrote:
> Joshua Murphy wrote:
>> While not trying to incite flames here... xfs isn't general purpose?
>> xfs_fsr defrags xfs partitions while they're mounted and is designed
>> to be used from cron (it's in xfsdump, not xfsprogs). File
>> fragmentation, while a fact of life on any filesystem that sees any
>> real use, does slow access times, as the drive head has to jump from
>> one place to another, so a lot of fragmentation is a bad thing... but
>> as you say, we're not dealing with FAT based FS's here, so severe
>> fragmentation only shows itself on very full filesystems.  I very
>> rarely see over 80% usage of my filesystems and have never
>> consistently checked fragmentation levels, though, so I can't say
>> whether xfs's being the exception on having a tool for the job means
>> it particularly needed one...
>>
>>
>
> Given my experience with XFS, I won't be switching anytime soon.  I used
> that once on a in-laws system.  After each crash, power failure, I had
> to reinstall.  Let's just say it left a bad taste in my mouth.  ;-)  I'm
> not saying it is a bad file system for someone but certainly not for me.
>
> You are right tho, every file system has some fragmentation.  It just
> can't be otherwise.  I guess I could always make my back ups, then redo
> my partitions, and copy them back.  I have done that once before.
> Worked very well then but not real sure about how udev would like that.
> I would think it would work OK but call me chicken.
>
> Dale
>
> :-)  :-)

Not trying to convert anyone, just correcting the "no linux fs has a
native tool for defrag" statement... as for udev's robustness in being
moved about... unless you're doing some very strange things with your
system, I'm certain that it'd take it quite well.

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Joshua Murphy wrote:
> On Fri, Nov 28, 2008 at 12:24 PM, Dale <rdalek1967@gmail.com> wrote:
>   
>> Given my experience with XFS, I won't be switching anytime soon.  I used
>> that once on a in-laws system.  After each crash, power failure, I had
>> to reinstall.  Let's just say it left a bad taste in my mouth.  ;-)  I'm
>> not saying it is a bad file system for someone but certainly not for me.
>>
>> You are right tho, every file system has some fragmentation.  It just
>> can't be otherwise.  I guess I could always make my back ups, then redo
>> my partitions, and copy them back.  I have done that once before.
>> Worked very well then but not real sure about how udev would like that.
>> I would think it would work OK but call me chicken.
>>
>> Dale
>>
>> :-)  :-)
>>     
>
> Not trying to convert anyone, just correcting the "no linux fs has a
> native tool for defrag" statement... as for udev's robustness in being
> moved about... unless you're doing some very strange things with your
> system, I'm certain that it'd take it quite well.
>
>   

You may be right about udev.  I know at one point udev was having
trouble if /dev/console and some other file was not there when it was at
the first part of booting up.  It was talked about so I suspect that
little glitch has been fixed but one can never tell about these things. 

I'm not a real big fan of defrag on Linux but was just sort of nosing
around a bit.  I just like some things to be in their place and I was
bored at the time. 

Dale

:-)  :-)

[gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Nikos Chantziaras]

Alan McKinnon wrote:
> On Friday 28 November 2008 13:14:42 Dale wrote:
>> If this is a little high, what would be the best way to defrag it? 
> 
> By not defragging it.
> 
> It's not Windows. Windows boxes needs defragging not because fragmentation is 
> a huge problem in itself, but because windows filesystems are a steaming mess 
> of cr@p that do little right and most things wrong. Defrag treats the 
> symptom, not the cause :-)

I don't buy into that argument and never did.  Every few months I copy 
the whole HD to another one and then back to counter fragmentation 
(ext3) and the system becomes noticeably faster after doing it (speed 
increase in emerge --sync for example.)  Maybe it's not fragmentation 
but rather related files being more closely together after I do this.

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Jorge Peixoto de Morais Neto]

>>> [...] what would be the best way to defrag it?
>>
>> By not defragging it.
>>
>> It's not Windows. Windows boxes needs defragging not because fragmentation
>> is a huge problem in itself, but because windows filesystems are a steaming
>> mess of cr@p that do little right and most things wrong. Defrag treats the
>> symptom, not the cause :-)
>
> I don't buy into that argument and never did.  Every few months I copy the
> whole HD to another one and then back to counter fragmentation (ext3) and
> the system becomes noticeably faster after doing it (speed increase in
> emerge --sync for example.)  Maybe it's not fragmentation but rather related
> files being more closely together after I do this.

How exactly do you copy the files? Be careful not to lose some file
property. How about sparse files, for example?
AFAIK, you can make a complete backup of a filesytem with (as root,
running from another system - such as a liveCD)
$ cd /path/to/mountpoint
$ tar -cSv -f /path/to/tarball.tar .

But I am not sure.

[gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Nikos Chantziaras]

Jorge Peixoto de Morais Neto wrote:
>>>> [...] what would be the best way to defrag it?
>>> By not defragging it.
>>>
>>> It's not Windows. Windows boxes needs defragging not because fragmentation
>>> is a huge problem in itself, but because windows filesystems are a steaming
>>> mess of cr@p that do little right and most things wrong. Defrag treats the
>>> symptom, not the cause :-)
>> I don't buy into that argument and never did.  Every few months I copy the
>> whole HD to another one and then back to counter fragmentation (ext3) and
>> the system becomes noticeably faster after doing it (speed increase in
>> emerge --sync for example.)  Maybe it's not fragmentation but rather related
>> files being more closely together after I do this.
> 
> How exactly do you copy the files? Be careful not to lose some file
> property. How about sparse files, for example?
> AFAIK, you can make a complete backup of a filesytem with (as root,
> running from another system - such as a liveCD)
> $ cd /path/to/mountpoint
> $ tar -cSv -f /path/to/tarball.tar .
> 
> But I am not sure.

I simply boot from the Gentoo DVD and rsync to another ext3 partition, 
wipe the current filesystem and then rsync back.

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Alan Mackenzie]

Hi, Dale!

On Fri, Nov 28, 2008 at 05:14:42AM -0600, Dale wrote:
> Hi folks,

> I have a pretty old install tho I have moved it from one drive to
> another once, using cp -av.  It got to big for the old hard drive. 
> Anyway, the install is about 5 years old or so and about 3 years since
> it got moved.  I have Gentoo on it naturally, a ship load of pictures
> and a few other standard documents and such.  I found something called
> frag check on the forums a long time ago and I got this from it:

> > 9.46189733169307% non contiguous files, 1.33812703640673 average
> > fragments.

That would be, what, 24683 non contiguous files from 251073 in total?

Just curious.  :-)

> Dale

> :-)  :-) 

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Stroller]

On 28 Nov 2008, at 16:31, Florian Philipp wrote:
> ...
> I'm wondering, why is Windows that bad in this regard? Of course,  
> FAT* is bad, but what's about NTFS? It is at least as modern as most  
> Linux FS and has some nice features. Surely MS should be capable of  
> implementing the same allocation algorithms we use. Or is NTFS  
> really not bad in this regard and it's all just that people mix  
> experiences with FAT* with NTFS?

No, NTFS *is* really bad in that regard. I've seen it HORRIBLY  
fragmented, and defragging it make a REMARKABLE difference. At least  
the nice thing is that Defrag not only fixes you the problem, but also  
shows it before you run it.

Stroller.

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Stroller]

On 28 Nov 2008, at 19:27, Jorge Peixoto de Morais Neto wrote:
>> ...
>> I don't buy into that argument and never did.  Every few months I  
>> copy the
>> whole HD to another one and then back to counter fragmentation  
>> (ext3) and
>> the system becomes noticeably faster after doing it (speed increase  
>> in
>> emerge --sync for example.)  Maybe it's not fragmentation but  
>> rather related
>> files being more closely together after I do this.
>
> How exactly do you copy the files? Be careful not to lose some file
> property. How about sparse files, for example?
> AFAIK, you can make a complete backup of a filesytem with (as root,
> running from another system - such as a liveCD)
> $ cd /path/to/mountpoint
> $ tar -cSv -f /path/to/tarball.tar .

Shouldn't creating a stage4 be safe for this?

There are a number of scripts on the forums (or the wiki?) that are  
supposed to be safe for creating stage4s from a live system. I think  
this would result in minimal downtime.(??)

Stroller.

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Jorge Peixoto de Morais Neto wrote:
>>>> [...] what would be the best way to defrag it?
>>>>         
>>> By not defragging it.
>>>
>>> It's not Windows. Windows boxes needs defragging not because fragmentation
>>> is a huge problem in itself, but because windows filesystems are a steaming
>>> mess of cr@p that do little right and most things wrong. Defrag treats the
>>> symptom, not the cause :-)
>>>       
>> I don't buy into that argument and never did.  Every few months I copy the
>> whole HD to another one and then back to counter fragmentation (ext3) and
>> the system becomes noticeably faster after doing it (speed increase in
>> emerge --sync for example.)  Maybe it's not fragmentation but rather related
>> files being more closely together after I do this.
>>     
>
> How exactly do you copy the files? Be careful not to lose some file
> property. How about sparse files, for example?
> AFAIK, you can make a complete backup of a filesytem with (as root,
> running from another system - such as a liveCD)
> $ cd /path/to/mountpoint
> $ tar -cSv -f /path/to/tarball.tar .
>
> But I am not sure.
>
>
>   


I use cp -av to copy mine.  From what I have read it keeps permission,
links and everything.  I have done it before and it worked fine but that
was before udev came along.  Also, I do that booted from a CD, either
Knoppix or Gentoo CD.

I would think that since everything is wiped out in /dev/ when I
shutdown, at least that is how it is set anyway, that udev has to
recreate all the files in /dev/ during boot up.  Of course, I have never
checked that to make sure.

Dale

:-)  :-) 

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Alan Mackenzie wrote:
> Hi, Dale!
>
> On Fri, Nov 28, 2008 at 05:14:42AM -0600, Dale wrote:
>   
>> Hi folks,
>>     
>
>   
>> I have a pretty old install tho I have moved it from one drive to
>> another once, using cp -av.  It got to big for the old hard drive. 
>> Anyway, the install is about 5 years old or so and about 3 years since
>> it got moved.  I have Gentoo on it naturally, a ship load of pictures
>> and a few other standard documents and such.  I found something called
>> frag check on the forums a long time ago and I got this from it:
>>     
>
>   
>>> 9.46189733169307% non contiguous files, 1.33812703640673 average
>>> fragments.
>>>       
>
> That would be, what, 24683 non contiguous files from 251073 in total?
>
> Just curious.  :-)
>
>   
>> Dale
>>     
>
>   
>> :-)  :-) 
>>     
>
>   

Oh, I got about 86,000 pictures alone.  Some I got from the net and some
with my camera.  I'm a pack rat so if I take a pic, I keep it even if it
sucks.  I don't have but a few mp3 files tho. 

I checked with Konqueror and there are 491,319 files and 52,074
directories.  Since some of those may be big files, OOo comes to mind,
10% may not be to bad.  Not sure how it would treat huge files that are
spread out just because of its size.

Still curious?  Ask away.

Dale

:-)  :-)

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Friday 28 November 2008 20:24:38 Nikos Chantziaras wrote:
> Alan McKinnon wrote:
> > On Friday 28 November 2008 13:14:42 Dale wrote:
> >> If this is a little high, what would be the best way to defrag it?
> >
> > By not defragging it.
> >
> > It's not Windows. Windows boxes needs defragging not because
> > fragmentation is a huge problem in itself, but because windows
> > filesystems are a steaming mess of cr@p that do little right and most
> > things wrong. Defrag treats the symptom, not the cause :-)
>
> I don't buy into that argument and never did.  Every few months I copy
> the whole HD to another one and then back to counter fragmentation
> (ext3) and the system becomes noticeably faster after doing it (speed
> increase in emerge --sync for example.)  Maybe it's not fragmentation
> but rather related files being more closely together after I do this.

Only a proper analysis of your files will tell you this. It's easy enough to 
check for individual file fragmentation and get stats on that before you do 
the copy-off/copy-back.



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Friday 28 November 2008 18:09:37 Joshua Murphy wrote:
> On Fri, Nov 28, 2008 at 6:46 AM, Alan McKinnon <alan.mckinnon@gmail.com> 
wrote:
> > On Friday 28 November 2008 13:14:42 Dale wrote:
> >> If this is a little high, what would be the best way to defrag it?
> >
> > By not defragging it.
> >
> > It's not Windows. Windows boxes needs defragging not because
> > fragmentation is a huge problem in itself, but because windows
> > filesystems are a steaming mess of cr@p that do little right and most
> > things wrong. Defrag treats the symptom, not the cause :-)
> >
> > Reiser tends to self-balance itself out. What is especially noteworthy is
> > that none of the general purpose Linux filesystems provide a defrag
> > utility. Theodore 'Tso and Hans Reiser are both exceptional programmers,
> > if there was a need for such a tool they would assuredly have written
> > one. They did not, so there probably isn't.
> >
> > Any Linux defrag tool you encounter will have been written by a third
> > party separate from the developers. It will move blocks around and update
> > superblocks, the drive will have to be unmounted for that to work and a
> > slight misunderstanding of how to do it will ruin data.
> >
> > Are you willing to take the very real risk of data corruption?
> >
> >> Is
> >> there a best way?  I do have a second hard drive that I back up too.
> >> Both Drives are 80Gbs and I do have a set of DVD back ups as well.  I
> >> can update those pretty quick.
> >
> > --
> > alan dot mckinnon at gmail dot com
>
> While not trying to incite flames here... xfs isn't general purpose?
> xfs_fsr defrags xfs partitions while they're mounted and is designed
> to be used from cron (it's in xfsdump, not xfsprogs). File
> fragmentation, while a fact of life on any filesystem that sees any
> real use, does slow access times, as the drive head has to jump from
> one place to another, so a lot of fragmentation is a bad thing...

On a proper multi-user multitasking OS like Unixes, the heads are going to be 
moving around all over the disk partition anyway just in general usage, even 
with zero file fragmentation. How much extra movement does fragmentation 
introduce?

I've been waiting for a proper statistical analysis of this question for 
years. I'm still waiting :-) Besides, modern storage presents an extra 
wrinkle. Defrag as most of the world knows it originated in DOS, where disk 
sectors were guaranteed to be laid out on disk in the order of their sector 
number. These days we have no such guarantee, and you cannot really be sure 
if blocks are laid out contiguously on-disk just by looking at the blocks 
numbers. I don't know of any filesystem tool that knows how to interrogate a 
drive's firmware and get it right for every storage type out there.


> but 
> as you say, we're not dealing with FAT based FS's here, so severe
> fragmentation only shows itself on very full filesystems.  I very
> rarely see over 80% usage of my filesystems and have never
> consistently checked fragmentation levels, though, so I can't say
> whether xfs's being the exception on having a tool for the job means
> it particularly needed one...



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Friday 28 November 2008 17:08:03 Stroller wrote:
> I understood that ReiserFS's trees could become out-of-balance,  
> resulting in performance loss, and that the way to deal with this was  
> to tar the contents of the drive to another file-system and then untar  
> them back.

That's what I tell people to do.

My purely anecdotal evidence: I observe that to get a Windows disk to reliably 
defrag, it should be taken off-line (otherwise it constantly restarts the 
process). A huge NTFS disk takes ages to defrag, the time involved seems 
about the same order as simply move the files off, copy them back. One day I 
may even test this with a stop watch :-)

As someone else said elsewhere, it's the user's problem to do the right thing 
with sparse files, symlinks and to remember which files are actually hard 
links.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Alan McKinnon wrote:
>
> I've been waiting for a proper statistical analysis of this question for 
> years. I'm still waiting :-) Besides, modern storage presents an extra 
> wrinkle. Defrag as most of the world knows it originated in DOS, where disk 
> sectors were guaranteed to be laid out on disk in the order of their sector 
> number. These days we have no such guarantee, and you cannot really be sure 
> if blocks are laid out contiguously on-disk just by looking at the blocks 
> numbers. I don't know of any filesystem tool that knows how to interrogate a 
> drive's firmware and get it right for every storage type out there.
>
>
>   

I never thought about the blocks and how they are laid out on the
drive.  If you think about it, they can have those things anywhere. 
Even if they are laid out by a defrag tool in proper sequence, they may
not be laid out that way on the platter(s).  Sort of makes one wonder if
some file systems can really even be defragged below a certain amount.

< Dale scratches his chin and thinks >

Dale

:-)  :-) 

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Daniel Iliev]

On Fri, 28 Nov 2008 13:46:01 +0200
Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> On Friday 28 November 2008 13:14:42 Dale wrote:
> > If this is a little high, what would be the best way to defrag it?
>
> By not defragging it.

I beg to defer. The simplest way to defrag a partition is to make
backup and restore. If it's worth the effort is another story.

>
> It's not Windows. Windows boxes needs defragging not because
> fragmentation is a huge problem in itself, but because windows
> filesystems are a steaming mess of cr@p that do little right and most
> things wrong. Defrag treats the symptom, not the cause :-)
>

Personally I think NTFS is one of the things MS have done right. It is
fast, stable and has the features of the Linux FSes and even more. It
has journal, quotas, permissions, mount points, symbolic links. Does
any of ext, reiserfs or xfs have compression and/or encryption
capabilities? I don't think so.
I have some experience with MS Windows and I've never seen data
corruption after a system crash or power loss, a thing I can't say
about ReiserFS or ext3 (when not mounted with data=journal).


> Reiser tends to self-balance itself out. What is especially
> noteworthy is that none of the general purpose Linux filesystems
> provide a defrag utility. Theodore 'Tso and Hans Reiser are both
> exceptional programmers, if there was a need for such a tool they
> would assuredly have written one. They did not, so there probably
> isn't.


It would be just as easy to pull the exactly opposite argument out:
since they like to experiment and "to boldly go where no man has gone
before", they won't bother to write a defrag tool, because it would
be too trivial and no fun. (just an example speculation on my part).


Now let's put the assumptions aside and do a test.

root@localhost # test $ cat /usr/portage/packages/All/* > test1
root@localhost # test $ cp test1 test2
root@localhost # test $ ls -lah
total 2.3G
drwxr-xr-x  2 root users 4.0K 2008-11-29 01:38 .
drwxr-xr-x 44 root users 4.0K 2008-11-29 01:36 ..
-rw-r--r--  1 root users 1.2G 2008-11-29 01:38 test1
-rw-r--r--  1 root users 1.2G 2008-11-29 01:40 test2
localhost test # filefrag *
test1: 1125 extents found, perfection would be 10 extents
test2: 1923 extents found, perfection would be 10 extents
localhost test # time cat test1 > /dev/null

real    0m26.747s
user    0m2.110s
sys     0m1.450s
localhost test # time cat test2 > /dev/null

real    0m29.825s
user    0m1.780s
sys     0m1.690s


All this with ext3 (rw,noatime,nodiratime,data=journal,commit=1) on
a partition with 84% free space.

It took 29.825-26.747=3.078 seconds more to read the same data, when
the file has
1923-1125=798 additional fragments. So, the fragmentation led to ~10%
performance
decrease in this case and it appears the dogma "Linux FS-es are smart
and don't need
to be de-fragmented" is not quite true, right? Unfortunately I have no
Windows at hand
to make a similar test for comparison, but I believe the results won't
be quite different.
BTW fiflefrag is written by Mr T'so.


> Any Linux defrag tool you encounter will have been written by a third
> party separate from the developers. It will move blocks around and
> update superblocks, the drive will have to be unmounted for that to
> work and a slight misunderstanding of how to do it will ruin data.
>
> Are you willing to take the very real risk of data corruption?


Who says it has to work that way? :)
I have seen on the Net a defrag tool written by Mr Con Kolivas [1]. It
was just a bash script which basically only copies the file into a new
one, checks the number of fragments of both files and deletes the one
that has more extents. :)

[1] http://en.wikipedia.org/wiki/Con_Kolivas


-- 
Best regards,
Daniel

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

On Sat, 29 Nov 2008 11:19:47 +0200, Daniel Iliev wrote:

> Personally I think NTFS is one of the things MS have done right. It is
> fast, stable and has the features of the Linux FSes and even more. It
> has journal, quotas, permissions, mount points, symbolic links. Does
> any of ext, reiserfs or xfs have compression and/or encryption
> capabilities? I don't think so.

No, and nor should they. Each filesystem implementing its own encryption
layer would be a nightmare. A separate encryption layer on which you
create any filesystem you like is much more sensible.


-- 
Neil Bothwick

COBOL: (n.) an old computer language, designed to be read and not
       run. Unfortunately, it is often run anyway.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Fri, 28 Nov 2008 11:24:08 -0600, Dale wrote:

> Given my experience with XFS, I won't be switching anytime soon.  I used
> that once on a in-laws system.  After each crash, power failure, I had
> to reinstall.  Let's just say it left a bad taste in my mouth.  ;-)  I'm
> not saying it is a bad file system for someone but certainly not for
> me. 

You're absolutely right with the last comment, XFS is intended for use in
environments that have stable power supplies. I use it on my desktop,
because it has a UPS, and my laptop.


-- 
Neil Bothwick

The gene pool could use a little chlorine.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Saturday 29 November 2008 11:19:47 Daniel Iliev wrote:
> Now let's put the assumptions aside and do a test.
>
> root@localhost # test $ cat /usr/portage/packages/All/* > test1
> root@localhost # test $ cp test1 test2
> root@localhost # test $ ls -lah
> total 2.3G
> drwxr-xr-x  2 root users 4.0K 2008-11-29 01:38 .
> drwxr-xr-x 44 root users 4.0K 2008-11-29 01:36 ..
> -rw-r--r--  1 root users 1.2G 2008-11-29 01:38 test1
> -rw-r--r--  1 root users 1.2G 2008-11-29 01:40 test2
> localhost test # filefrag *
> test1: 1125 extents found, perfection would be 10 extents
> test2: 1923 extents found, perfection would be 10 extents
> localhost test # time cat test1 > /dev/null
>
> real    0m26.747s
> user    0m2.110s
> sys     0m1.450s
> localhost test # time cat test2 > /dev/null
>
> real    0m29.825s
> user    0m1.780s
> sys     0m1.690s

This is not a test unfortunately. You did one run on one file and one run on 
another file. We do not know what else the machine was doing at that time, 
and that unknown is a considerable one.

Repeat your test eliminating this factor. Preferably, remount the filesystems 
after each run and repeat 1000 times. Then analyze the statistical 
distribution of your results. This should eliminate most random factors and 
give a more realistic real-world view.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Daniel Iliev]

On Sat, 29 Nov 2008 12:23:23 +0200
Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> On Saturday 29 November 2008 11:19:47 Daniel Iliev wrote:
> > Now let's put the assumptions aside and do a test.
> >
> > root@localhost # test $ cat /usr/portage/packages/All/* > test1
> > root@localhost # test $ cp test1 test2
> > root@localhost # test $ ls -lah
> > total 2.3G
> > drwxr-xr-x  2 root users 4.0K 2008-11-29 01:38 .
> > drwxr-xr-x 44 root users 4.0K 2008-11-29 01:36 ..
> > -rw-r--r--  1 root users 1.2G 2008-11-29 01:38 test1
> > -rw-r--r--  1 root users 1.2G 2008-11-29 01:40 test2
> > localhost test # filefrag *
> > test1: 1125 extents found, perfection would be 10 extents
> > test2: 1923 extents found, perfection would be 10 extents
> > localhost test # time cat test1 > /dev/null
> >
> > real    0m26.747s
> > user    0m2.110s
> > sys     0m1.450s
> > localhost test # time cat test2 > /dev/null
> >
> > real    0m29.825s
> > user    0m1.780s
> > sys     0m1.690s
>
> This is not a test unfortunately. You did one run on one file and one
> run on another file. We do not know what else the machine was doing
> at that time, and that unknown is a considerable one.
>


This result is from the last of three repetitions. Its values were in
the middle (not average). The deviations were in the range 1 to 1.5
seconds. Every time reading the more fragmented was slower.

The system was idle in level 3, no X running. I used iostat before
the tests and it registered several tens of KBs written for 1 min before
the test. Compared to the size of the test files and the speed of the
disk it is insignificant.


> Repeat your test eliminating this factor. Preferably, remount the
> filesystems after each run and repeat 1000 times. Then analyze the
> statistical distribution of your results. This should eliminate most
> random factors and give a more realistic real-world view.
>

I'm not willing to waste time for 1000 repetitions, but why don't you
do the test yourself just a couple of times and see if there will be a
case when the more fragmented file gets read faster?

Actually my results are a little lower than what I expected but enough
for me to say that fragmentation matters. At least until proved
otherwise. Fragmentation leads to seeks. The average seek time on
modern disks is several milliseconds. Yes, there are algorithms
reordering the I/O requests to minimize the seek penalty, but still
seeks are there and they hurt performance.


-- 
Best regards,
Daniel

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Daniel Iliev]

On Sat, 29 Nov 2008 09:45:40 +0000
Neil Bothwick <neil@digimed.co.uk> wrote:

> No, and nor should they. Each filesystem implementing its own
> encryption layer would be a nightmare. A separate encryption layer on
> which you create any filesystem you like is much more sensible.

It's true for Linux and any other kernel designed to work with many
implementations of "the same" thing (FS in this case). MS doesn't care
about many file systems, but just their own, therefore they don't
face this problem.

-- 
Best regards,
Daniel

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Sunday 30 November 2008 11:53:10 Daniel Iliev wrote:
> On Sat, 29 Nov 2008 09:45:40 +0000
>
> Neil Bothwick <neil@digimed.co.uk> wrote:
> > No, and nor should they. Each filesystem implementing its own
> > encryption layer would be a nightmare. A separate encryption layer on
> > which you create any filesystem you like is much more sensible.
>
> It's true for Linux and any other kernel designed to work with many
> implementations of "the same" thing (FS in this case). MS doesn't care
> about many file systems, but just their own, therefore they don't
> face this problem.

True, MS will not face this problem. We will. In common with what we have to 
do with everything else MS makes. Please don't give them any more clever 
ideas...

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Sunday 30 November 2008 11:50:18 Daniel Iliev wrote:

> > > real    0m26.747s
> > > user    0m2.110s
> > > sys     0m1.450s
> > > localhost test # time cat test2 > /dev/null
> > >
> > > real    0m29.825s
> > > user    0m1.780s
> > > sys     0m1.690s
> >
> > This is not a test unfortunately. You did one run on one file and one
> > run on another file. We do not know what else the machine was doing
> > at that time, and that unknown is a considerable one.
>
> This result is from the last of three repetitions. Its values were in
> the middle (not average). The deviations were in the range 1 to 1.5
> seconds. Every time reading the more fragmented was slower.

That's a HUGE deviation, I assume it's real time. What's the difference in 
user and sys time? Those are much more important here.

> The system was idle in level 3, no X running. I used iostat before
> the tests and it registered several tens of KBs written for 1 min before
> the test. Compared to the size of the test files and the speed of the
> disk it is insignificant.

You didn't remove the effect of buffers and caches. The only way I know of to 
test this correctly is to reboot the machine after every run, start no 
services and run the test with a cold cache.


Otherwise you don't really know what you are measuring.

> > Repeat your test eliminating this factor. Preferably, remount the
> > filesystems after each run and repeat 1000 times. Then analyze the
> > statistical distribution of your results. This should eliminate most
> > random factors and give a more realistic real-world view.
>
> I'm not willing to waste time for 1000 repetitions, but why don't you
> do the test yourself just a couple of times and see if there will be a
> case when the more fragmented file gets read faster?

You are the one making them claims! You do the test!

> Actually my results are a little lower than what I expected but enough
> for me to say that fragmentation matters. At least until proved
> otherwise. Fragmentation leads to seeks. The average seek time on
> modern disks is several milliseconds. Yes, there are algorithms
> reordering the I/O requests to minimize the seek penalty, but still
> seeks are there and they hurt performance.

Buffers and cache? You have two files of 1.2G each. I have more free RAM than 
that for buffers and cache easily 90% of the time my notebook is on.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Daniel Iliev]

On Sun, 30 Nov 2008 14:27:22 +0200
Alan McKinnon <alan.mckinnon@gmail.com> wrote:

[-snip-]

>
> You are the one making them claims! You do the test!


Me? No, my dear. It is you who claims that fragmentation only matters
on MS file systems while Linux FS remain unaffected w/o providing
any proof. The common sense says otherwise: fragmentation leads to
seeks, seeks hurt performance.

Anyways, I'm going to do the tests because I want to see the numbers.
Should the outcome prove you were right I'd be the first to
congratulate and thank you for opening my eyes.

Since I'm going to use my workstation at work to do the job at night
I'll limit the repetitions to 400 which would take about 13 hours with
two files, 2GB each. The machine has 1GB RAM installed and I'll run the
test in Gentoo's "boot" rc level. I've closed everything except sshd
and screen (see "ps" outpput below) and have taken care of the cache
by clearing it before each run via /proc (see the script). The file
system is ext3 and during the test it will be used in about 30% of its
capacity:

localhost test # df -h | grep home
/dev/sdc1   15G  718M   14G   5% /home

localhost test # mount -v | grep home
/dev/sdc1 on /home/ type ext3
(rw,noatime,nodiratime,data=journal,commit=1)


localhost test # rc-status
Runlevel: boot
bootmisc	[ started  ]
checkfs		[ started  ]
checkroot	[ started  ]
clock		[ started  ]
hostname	[ started  ]
iptables	[ started  ]
localmount	[ started  ]
modules		[ started  ]
net.eth0	[ started  ]
net.lo		[ started  ]
sshd		[ started  ]
udev-postmount	[ started  ]


localhost test # ps af
  PID TTY      STAT   TIME COMMAND
12191 pts/0    Ss     0:00 -bash
12227 pts/0    S      0:00  \_ su -
12231 pts/0    S      0:00      \_ -su
12266 pts/0    S+     0:00          \_ screen -x
 8575 pts/3    Ss+    0:00 -/bin/bash
 4967 pts/2    Ss     0:00 -/bin/bash
12830 pts/2    R+     0:00  \_ ps af
 4895 pts/1    Ss+    0:03 -/bin/bash
 4755 tty6     Ss+    0:00 /sbin/agetty 38400 tty6 linux
 4754 tty5     Ss+    0:00 /sbin/agetty 38400 tty5 linux
 4753 tty4     Ss+    0:00 /sbin/agetty 38400 tty4 linux
 4752 tty3     Ss+    0:00 /sbin/agetty 38400 tty3 linux
 4751 tty2     Ss+    0:00 /sbin/agetty 38400 tty2 linux
 4750 tty1     Ss+    0:00 /sbin/agetty 38400 tty1 linux


The script:

#!/bin/bash

cat /usr/portage/distfiles/* > test1
cp test1 test2

filefrag test*

sync

for (( i=0 ; i<=400 ; i++ ))
do

  sync
  echo 3 > /proc/sys/vm/drop_caches

  echo -n "try_no=$i "

  /usr/bin/time -f "\
command=%C|\
real_t=%e|\
kernel_t=%S|\
user_t=%U|\
major_faults=%F|\
minor_faults=%R|\
context_sw=%c|\
io_waits=%w|\
fs_reads=%I|\
fs_writes=%O" cp test1 /dev/null

  sync
  echo 3 > /proc/sys/vm/drop_caches

  echo -n "try_no=$i "

  /usr/bin/time -f "\
command=%C|\
real_t=%e|\
kernel_t=%S|\
user_t=%U|\
major_faults=%F|\
minor_faults=%R|\
context_sw=%c|\
io_waits=%w|\
fs_reads=%I|\
fs_writes=%O" cp test2 /dev/null


done

###########EOF



I hope all this will be enough for you to accept the results.



-- 
Best regards,
Daniel

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Joshua Murphy]

On Mon, Dec 1, 2008 at 3:11 PM, Daniel Iliev <daniel.iliev@gmail.com> wrote:
> On Sun, 30 Nov 2008 14:27:22 +0200
> Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>
> [-snip-]
>
>>
>> You are the one making them claims! You do the test!
>
>
> Me? No, my dear. It is you who claims that fragmentation only matters
> on MS file systems while Linux FS remain unaffected w/o providing
> any proof. The common sense says otherwise: fragmentation leads to
> seeks, seeks hurt performance.
>
> Anyways, I'm going to do the tests because I want to see the numbers.
> Should the outcome prove you were right I'd be the first to
> congratulate and thank you for opening my eyes.
>
> Since I'm going to use my workstation at work to do the job at night
> I'll limit the repetitions to 400 which would take about 13 hours with
> two files, 2GB each. The machine has 1GB RAM installed and I'll run the
> test in Gentoo's "boot" rc level. I've closed everything except sshd
> and screen (see "ps" outpput below) and have taken care of the cache
> by clearing it before each run via /proc (see the script). The file
> system is ext3 and during the test it will be used in about 30% of its
> capacity:
>
> localhost test # df -h | grep home
> /dev/sdc1   15G  718M   14G   5% /home
>
> localhost test # mount -v | grep home
> /dev/sdc1 on /home/ type ext3
> (rw,noatime,nodiratime,data=journal,commit=1)
>
>
> localhost test # rc-status
> Runlevel: boot
> bootmisc        [ started  ]
> checkfs         [ started  ]
> checkroot       [ started  ]
> clock           [ started  ]
> hostname        [ started  ]
> iptables        [ started  ]
> localmount      [ started  ]
> modules         [ started  ]
> net.eth0        [ started  ]
> net.lo          [ started  ]
> sshd            [ started  ]
> udev-postmount  [ started  ]
>
>
> localhost test # ps af
>  PID TTY      STAT   TIME COMMAND
> 12191 pts/0    Ss     0:00 -bash
> 12227 pts/0    S      0:00  \_ su -
> 12231 pts/0    S      0:00      \_ -su
> 12266 pts/0    S+     0:00          \_ screen -x
>  8575 pts/3    Ss+    0:00 -/bin/bash
>  4967 pts/2    Ss     0:00 -/bin/bash
> 12830 pts/2    R+     0:00  \_ ps af
>  4895 pts/1    Ss+    0:03 -/bin/bash
>  4755 tty6     Ss+    0:00 /sbin/agetty 38400 tty6 linux
>  4754 tty5     Ss+    0:00 /sbin/agetty 38400 tty5 linux
>  4753 tty4     Ss+    0:00 /sbin/agetty 38400 tty4 linux
>  4752 tty3     Ss+    0:00 /sbin/agetty 38400 tty3 linux
>  4751 tty2     Ss+    0:00 /sbin/agetty 38400 tty2 linux
>  4750 tty1     Ss+    0:00 /sbin/agetty 38400 tty1 linux
>
>
> The script:
>
> #!/bin/bash
>
> cat /usr/portage/distfiles/* > test1
> cp test1 test2
>
> filefrag test*
>
> sync
>
> for (( i=0 ; i<=400 ; i++ ))
> do
>
>  sync
>  echo 3 > /proc/sys/vm/drop_caches
>
>  echo -n "try_no=$i "
>
>  /usr/bin/time -f "\
> command=%C|\
> real_t=%e|\
> kernel_t=%S|\
> user_t=%U|\
> major_faults=%F|\
> minor_faults=%R|\
> context_sw=%c|\
> io_waits=%w|\
> fs_reads=%I|\
> fs_writes=%O" cp test1 /dev/null
>
>  sync
>  echo 3 > /proc/sys/vm/drop_caches
>
>  echo -n "try_no=$i "
>
>  /usr/bin/time -f "\
> command=%C|\
> real_t=%e|\
> kernel_t=%S|\
> user_t=%U|\
> major_faults=%F|\
> minor_faults=%R|\
> context_sw=%c|\
> io_waits=%w|\
> fs_reads=%I|\
> fs_writes=%O" cp test2 /dev/null
>
>
> done
>
> ###########EOF
>
>
>
> I hope all this will be enough for you to accept the results.
>
>
>
> --
> Best regards,
> Daniel
>
>

One thing that comes to mind... if one of those files is primarily
closer to the start of the disk than the other, and the drive tapers
down on speed as bad as some... well, that's one variable you can't
easily resolve. Whether that would influence enough to matter,
compared to seeks, is questionable though.

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] [Even More OT] Fragmentation of my drives. Curious mostly

[Anthony Metcalf]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

Ignoring the MS/*nix argument for a moment.....

How does fragmentation work on hardware RAID?

The normal argument of more fragmentation = more seeks = worse
performance would seem to be eliminated by a controller with a large
cache and lots of drives (say a Dell/Equallogic PS5000E in RAID 10.....).

A


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Daniel Iliev]

On Mon, 1 Dec 2008 15:40:11 -0500
"Joshua Murphy" <poisonbl@gmail.com> wrote:

=
> One thing that comes to mind... if one of those files is primarily
> closer to the start of the disk than the other, and the drive tapers
> down on speed as bad as some... well, that's one variable you can't
> easily resolve. Whether that would influence enough to matter,
> compared to seeks, is questionable though.
>


Yes. That has crossed my mind too, but I can't figure out if there's
anything I alone can do about it. It would taint the results in a very bad
way, because it is impossible to catch by increasing the number of
repetitions. If my memory serves well its called "systematic error" in
statistics.

I'm open to suggestions.


P.S.

The way to eliminate the influence of this factor is to find
many other people to make the test and share the results.



-- 
Best regards,
Daniel

Re: [gentoo-user] [Even More OT] Fragmentation of my drives. Curious mostly

[Joshua Murphy]

On Mon, Dec 1, 2008 at 5:07 PM, Anthony Metcalf <nevyn@anferny.me.uk> wrote:
> Ignoring the MS/*nix argument for a moment.....
>
> How does fragmentation work on hardware RAID?
>
> The normal argument of more fragmentation = more seeks = worse
> performance would seem to be eliminated by a controller with a large
> cache and lots of drives (say a Dell/Equallogic PS5000E in RAID 10.....).
>
> A

Assuming a level that performs striping, it would be alleviated, yes,
but even striping is at its fastest when performing sequential reads,
it just reads sequentially from more than one disk at a time... but
the fact that one disk will likely be reading while another performs a
seek would probably reduce the impact of the seeks considerably.

(all of this is, of course, purely pulling from my understanding of
raid and how it operates and could very well be 100% incorrect)

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Joshua Murphy]

On Mon, Dec 1, 2008 at 5:19 PM, Daniel Iliev <daniel.iliev@gmail.com> wrote:
> Yes. That has crossed my mind too, but I can't figure out if there's
> anything I alone can do about it. It would taint the results in a very bad
> way, because it is impossible to catch by increasing the number of
> repetitions. If my memory serves well its called "systematic error" in
> statistics.
>
> I'm open to suggestions.
>
>
> P.S.
>
> The way to eliminate the influence of this factor is to find
> many other people to make the test and share the results.
>
>
>
> --
> Best regards,
> Daniel

I'm tempted to donate my system for a short bit to try it.. but at
least in the case of my desktop... I pre-cache all my major software
at boot time, so I don't really want to dump caches there ;) ... maybe
it'll be a good use for my old amd64... but, umm... it'll need an OS
and even the slightest potential of fragmented files first... maybe
I'll make that happen tonight if I get bored.

As a side note, SSDs are the quickest way to remove all worries where
fragmentation is concerned, having negligible seek times as they do...
and physically smaller platters (2.5in SAS drives, the Velociraptor,
etc) at least make a small dent on worst case seek times... reducing
the impact that way.

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Peter Humphrey]

On Friday 28 November 2008 21:31:51 Stroller wrote:

> NTFS *is* really bad in that regard. I've seen it HORRIBLY fragmented, and
> defragging it make a REMARKABLE difference.

I remember also that when M$ introduced NTFS they made a big thing of not 
needing to defrag it. Only later, when others were making a living selling 
NTFS defraggers, did they bow to the inevitable.

-- 
Rgds
Peter

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Dale]

Stroller wrote:
>
>
> No, NTFS *is* really bad in that regard. I've seen it HORRIBLY
> fragmented, and defragging it make a REMARKABLE difference. At least
> the nice thing is that Defrag not only fixes you the problem, but also
> shows it before you run it.
>
> Stroller.
>
>
>

I agree.  I defrag my brothers sometimes when I am up there and it has
data spread all over the place and takes a long time to get it all back
to somewhat reasonable.  I usually have to do it twice to get it back
together.  NTFS may be better than FAT but not by much.

Dale

:-)  :-) 

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Wolfgang Liebich]

Daniel Iliev schrieb:
> On Fri, 28 Nov 2008 13:46:01 +0200
> Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>
>   
>> On Friday 28 November 2008 13:14:42 Dale wrote:
>>     
>>> If this is a little high, what would be the best way to defrag it?
>>>       
>> By not defragging it.
>>     
>
> I beg to defer. The simplest way to defrag a partition is to make
> backup and restore. If it's worth the effort is another story.
>
>   
>> It's not Windows. Windows boxes needs defragging not because
>> fragmentation is a huge problem in itself, but because windows
>> filesystems are a steaming mess of cr@p that do little right and most
>> things wrong. Defrag treats the symptom, not the cause :-)
>>
>>     
>
> Personally I think NTFS is one of the things MS have done right. It is
> fast, stable and has the features of the Linux FSes and even more. It
> has journal, quotas, permissions, mount points, symbolic links. Does
> any of ext, reiserfs or xfs have compression and/or encryption
> capabilities? I don't think so.
> I have some experience with MS Windows and I've never seen data
> corruption after a system crash or power loss, a thing I can't say
> about ReiserFS or ext3 (when not mounted with data=journal).
>
>
>   
<SNIP>

My experience with NTFS is somewhat more balanced (maybe). In about 12
years I experienced one damaged NTFS instance. This was caused by a
crash during an installation (don't remember what we installed - it's
been about 9 years ago :-)
BUT this was an example of total destruction and mayhem -- absolutely
irreparable. After about 120 errors (filenames with very much foreign
sounding names - high bit turned on) we gave up and reinstalled everything.
Probably the MFT was damaged beyond repair.

So my conclusion --- NTFS is not so easy to damage, but if you manage
it, you're toast :-/

Ciao,
Wolfgang Liebich

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Stroller]

On 2 Dec 2008, at 13:13, Wolfgang Liebich wrote:
> ...
> My experience with NTFS is somewhat more balanced (maybe). In about 12
> years I experienced one damaged NTFS instance. This was caused by a
> crash during an installation ...
>
> So my conclusion --- NTFS is not so easy to damage, but if you manage
> it, you're toast :-/

I'm not sure that your experience with NTFS _is_ more balanced - I've  
seen a number of PCs this year which fail booting into Windows  
(displaying the XP splashscreen before rebooting again in an unending  
cycle) which have been repaired using only chkdsk.

Because the user liked to power his PC down at the wall - for energy- 
saving reasons or peace-of-mind over house-fires, I'm knackered if I  
know - and because "it was taking too long to shutdown" when he wanted  
to go to bed - I know one of these was unplugged whilst still shutting  
down, but surely not all of them were.

Stroller.

Re: [gentoo-user] [OT] Fragmentation of my drives. Curious mostly

[Daniel Iliev]

On Wed, 3 Dec 2008 00:28:54 +0000
Stroller <stroller@stellar.eclipse.co.uk> wrote:

>
> On 2 Dec 2008, at 13:13, Wolfgang Liebich wrote:
> > ...
> > My experience with NTFS is somewhat more balanced (maybe). In about
> > 12 years I experienced one damaged NTFS instance. This was caused
> > by a crash during an installation ...
> >
> > So my conclusion --- NTFS is not so easy to damage, but if you
> > manage it, you're toast :-/
>
> I'm not sure that your experience with NTFS _is_ more balanced -
> I've seen a number of PCs this year which fail booting into Windows
> (displaying the XP splashscreen before rebooting again in an
> unending cycle) which have been repaired using only chkdsk.
>
> Because the user liked to power his PC down at the wall - for energy-
> saving reasons or peace-of-mind over house-fires, I'm knackered if I
> know - and because "it was taking too long to shutdown" when he
> wanted to go to bed - I know one of these was unplugged whilst still
> shutting down, but surely not all of them were.
>
> Stroller.
>


Very much the same happens with Linux FSs when abused like this.
I live in area with frequent power outages and have my share of
damaged FSes and destroyed HDDs. (Un)fortunately I can't put NTFS in
the same chart for comparison, because I have "no Windows, no Gates -
only Apache inside".

(sorry for the old joke, I couldn't help it) :)


P.S.

It's not Windows vs Linux as somebody implied. My choice is
clear. I'm just trying to stay objective.


-- 
Best regards,
Daniel

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Paul Hartman]

On Fri, Nov 28, 2008 at 10:09 AM, Joshua Murphy <poisonbl@gmail.com> wrote:
> On Fri, Nov 28, 2008 at 6:46 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> On Friday 28 November 2008 13:14:42 Dale wrote:
>>> If this is a little high, what would be the best way to defrag it?
>>
>> By not defragging it.
>>
>> It's not Windows. Windows boxes needs defragging not because fragmentation is
>> a huge problem in itself, but because windows filesystems are a steaming mess
>> of cr@p that do little right and most things wrong. Defrag treats the
>> symptom, not the cause :-)
>>
>> Reiser tends to self-balance itself out. What is especially noteworthy is that
>> none of the general purpose Linux filesystems provide a defrag utility.
>> Theodore 'Tso and Hans Reiser are both exceptional programmers, if there was
>> a need for such a tool they would assuredly have written one. They did not,
>> so there probably isn't.
>>
>> Any Linux defrag tool you encounter will have been written by a third party
>> separate from the developers. It will move blocks around and update
>> superblocks, the drive will have to be unmounted for that to work and a
>> slight misunderstanding of how to do it will ruin data.
>>
>> Are you willing to take the very real risk of data corruption?
>>
>>> Is
>>> there a best way?  I do have a second hard drive that I back up too.
>>> Both Drives are 80Gbs and I do have a set of DVD back ups as well.  I
>>> can update those pretty quick.
>>
>>
>>
>> --
>> alan dot mckinnon at gmail dot com
>>
>>
>
> While not trying to incite flames here... xfs isn't general purpose?
> xfs_fsr defrags xfs partitions while they're mounted and is designed
> to be used from cron (it's in xfsdump, not xfsprogs). File
> fragmentation, while a fact of life on any filesystem that sees any
> real use, does slow access times, as the drive head has to jump from
> one place to another, so a lot of fragmentation is a bad thing... but
> as you say, we're not dealing with FAT based FS's here, so severe
> fragmentation only shows itself on very full filesystems.  I very
> rarely see over 80% usage of my filesystems and have never
> consistently checked fragmentation levels, though, so I can't say
> whether xfs's being the exception on having a tool for the job means
> it particularly needed one...

I believe JFS has a filesystem defrag tool as well, but I don't think
it has a file defrag tool.

My favorite way to defrag individual files is to mv to /dev/shm, sync,
mv back to hard drive.

I've found fragmention to be noticable (as far as slowing disk read
speeds) on large files that were downloaded over the internet. Large
ISO images, TV shows etc that are hundreds of megabytes downloaded
over a long period of time (especially if multiple downloads are
streaming at once). On my old "slow" computer (P4 2.8ghz) the file
fragmentation on ext3 would get so bad that I could not burn DVD
backups of the files at full speed without first defragmenting them.

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Alan McKinnon wrote:
>
> Only a proper analysis of your files will tell you this. It's easy enough to 
> check for individual file fragmentation and get stats on that before you do 
> the copy-off/copy-back.
>
>
>
>   

This is interesting.  I am starting a new install on my backup drive. 
I'm part way through the install, fetching all the KDE stuff right now. 
This is what I got from the little frag script:

root@smoker / # /root/fragck.pl /backup/
0.953336175120985% non contiguous files, 1.02414182192021 average fragments.
root@smoker / #

Less than 1% is my starting point I guess.  This currently has ext3 on
it.  I did start out with a freshly formatted file system.  Also, this
is all on one big partition at the moment. 

I'll post later what it says after compiling a few packages.  I figure
KDE should stir up something.  LOL

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Dale wrote:
>
> This is interesting.  I am starting a new install on my backup drive. 
> I'm part way through the install, fetching all the KDE stuff right now. 
> This is what I got from the little frag script:
>
> root@smoker / # /root/fragck.pl /backup/
> 0.953336175120985% non contiguous files, 1.02414182192021 average fragments.
> root@smoker / #
>
> Less than 1% is my starting point I guess.  This currently has ext3 on
> it.  I did start out with a freshly formatted file system.  Also, this
> is all on one big partition at the moment. 
>
> I'll post later what it says after compiling a few packages.  I figure
> KDE should stir up something.  LOL
>
> Dale
>
> :-)  :-) 
>
>   

This is after a almost complete install.  About to start OOo. 

root@smoker / # /root/fragck.pl /backup/
2.00854614717917% non contiguous files, 1.04611358582092 average fragments.
root@smoker / #

root@smoker / # du -shc /backup/
5.6G    /backup/
5.6G    total
root@smoker / #

I would assume that would be something like it was when I started my
current install years ago.  Which is at 10% or so now. 

Thoughts anyone?

Dale

:-)  :-)

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Dale wrote:
> Dale wrote:
>   
>> This is interesting.  I am starting a new install on my backup drive. 
>> I'm part way through the install, fetching all the KDE stuff right now. 
>> This is what I got from the little frag script:
>>
>> root@smoker / # /root/fragck.pl /backup/
>> 0.953336175120985% non contiguous files, 1.02414182192021 average fragments.
>> root@smoker / #
>>
>> Less than 1% is my starting point I guess.  This currently has ext3 on
>> it.  I did start out with a freshly formatted file system.  Also, this
>> is all on one big partition at the moment. 
>>
>> I'll post later what it says after compiling a few packages.  I figure
>> KDE should stir up something.  LOL
>>
>> Dale
>>
>> :-)  :-) 
>>
>>   
>>     
>
> This is after a almost complete install.  About to start OOo. 
>
> root@smoker / # /root/fragck.pl /backup/
> 2.00854614717917% non contiguous files, 1.04611358582092 average fragments.
> root@smoker / #
>
> root@smoker / # du -shc /backup/
> 5.6G    /backup/
> 5.6G    total
> root@smoker / #
>
> I would assume that would be something like it was when I started my
> current install years ago.  Which is at 10% or so now. 
>
> Thoughts anyone?
>
> Dale
>
> :-)  :-)
>
>   

OK.  I completed my install and got everything working.  This is what I
got after that:

2.24954051453251% non contiguous files, 1.06439409487064 average fragments.

I then ran shake just to see if it changed for the better or worse.  I
got this surprising answer:

25.2668178520421% non contiguous files, 1.41060290111655 average fragments.

You may want to look twice at the decimal point.  It appears that shake
makes things much worse or the fragck script has some serious issues
one.  I have no clue which.

I'm not to worried about this since I will be moving this over to the
other drive anyway.  I would like to know what command I should use to
tar up everything, transfer it over and untar it all on one line if
possible?  I plan to do this while booted from a Gentoo CD.  I just want
to try this so that it will be compressed then transfered and untared
once on the way.  Does this make since?  I have used cp -av in the past.

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Daniel Troeder]

[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]

Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale:
> Dale wrote:
> > Dale wrote:
> >   
> >> This is interesting.  I am starting a new install on my backup drive. 
> >> I'm part way through the install, fetching all the KDE stuff right now. 
> >> This is what I got from the little frag script:
> >>
> >> root@smoker / # /root/fragck.pl /backup/
> >> 0.953336175120985% non contiguous files, 1.02414182192021 average fragments.
> >> root@smoker / #
> >>
> >> Less than 1% is my starting point I guess.  This currently has ext3 on
> >> it.  I did start out with a freshly formatted file system.  Also, this
> >> is all on one big partition at the moment. 
> >>
> >> I'll post later what it says after compiling a few packages.  I figure
> >> KDE should stir up something.  LOL
> >>
> >> Dale
> >>
> >> :-)  :-) 
> >>
> >>   
> >>     
> >
> > This is after a almost complete install.  About to start OOo. 
> >
> > root@smoker / # /root/fragck.pl /backup/
> > 2.00854614717917% non contiguous files, 1.04611358582092 average fragments.
> > root@smoker / #
> >
> > root@smoker / # du -shc /backup/
> > 5.6G    /backup/
> > 5.6G    total
> > root@smoker / #
> >
> > I would assume that would be something like it was when I started my
> > current install years ago.  Which is at 10% or so now. 
> >
> > Thoughts anyone?
> >
> > Dale
> >
> > :-)  :-)
> >
> >   
> 
> OK.  I completed my install and got everything working.  This is what I
> got after that:
> 
> 2.24954051453251% non contiguous files, 1.06439409487064 average fragments.
> 
> I then ran shake just to see if it changed for the better or worse.  I
> got this surprising answer:
> 
> 25.2668178520421% non contiguous files, 1.41060290111655 average fragments.
> 
> You may want to look twice at the decimal point.  It appears that shake
> makes things much worse or the fragck script has some serious issues
> one.  I have no clue which.
> 
> I'm not to worried about this since I will be moving this over to the
> other drive anyway.  I would like to know what command I should use to
> tar up everything, transfer it over and untar it all on one line if
> possible?  I plan to do this while booted from a Gentoo CD.  I just want
> to try this so that it will be compressed then transfered and untared
> once on the way.  Does this make since?  I have used cp -av in the past.
> 
> Thanks.
> 
> Dale
> 
> :-)  :-) 
With "transfer" do you mean over a network, or to another local drive?

You can of course use something like
# tar czpf - | ssh remote - tar xzpf -C /dir
(above probably not syntactically correct), but there are faster and
easier options:

"cp -a" costs little resources locally and maintains POSIX permissions,
while "rsync -aASH --numeric-ids" is perfect for remote copy.

You can use rsync also locally. It will (with the "-A" switch) also
transfer POSIX-ACLs, if that is of any concern. It is also useful, if a
transfer breaks at some moment, because it will kind of continue it :)

Omiting the "-v" switch can significantly speed up things - depends on
your terminal. In every case it helps to only see the errors, and not
let them scroll away by everything that went well.

Bye,
Daniel

-- 
PGP key: http://pgpkeys.pca.dfn.de/pks/lookup?search=0xBB9D4887&op=get
# gpg --recv-keys --keyserver hkp://subkeys.pgp.net 0xBB9D4887


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Daniel Troeder wrote:
> Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale:
>   
>>
>> I'm not to worried about this since I will be moving this over to the
>> other drive anyway.  I would like to know what command I should use to
>> tar up everything, transfer it over and untar it all on one line if
>> possible?  I plan to do this while booted from a Gentoo CD.  I just want
>> to try this so that it will be compressed then transfered and untared
>> once on the way.  Does this make since?  I have used cp -av in the past.
>>
>> Thanks.
>>
>> Dale
>>
>> :-)  :-) 
>>     
> With "transfer" do you mean over a network, or to another local drive?
>
> You can of course use something like
> # tar czpf - | ssh remote - tar xzpf -C /dir
> (above probably not syntactically correct), but there are faster and
> easier options:
>
> "cp -a" costs little resources locally and maintains POSIX permissions,
> while "rsync -aASH --numeric-ids" is perfect for remote copy.
>
> You can use rsync also locally. It will (with the "-A" switch) also
> transfer POSIX-ACLs, if that is of any concern. It is also useful, if a
> transfer breaks at some moment, because it will kind of continue it :)
>
> Omiting the "-v" switch can significantly speed up things - depends on
> your terminal. In every case it helps to only see the errors, and not
> let them scroll away by everything that went well.
>
> Bye,
> Daniel
>
>   

The drive is in the same machine so there is no network involved. 
Should help make it a little more simple.  Would this work?

tar czpf - | tar xzpf -C /dir

Basically, I want as clean a file system as I can get to start off with
at least.  Goal is very little fragmentation.

Thanks

Dale

:-)  :-) 

Re: [gentoo-user] Fragmentation of my drives. Curious mostly

[Daniel Troeder]

[-- Attachment #1: Type: text/plain, Size: 1138 bytes --]

Am Freitag, den 28.11.2008, 13:46 +0200 schrieb Alan McKinnon:
> Any Linux defrag tool you encounter will have been written by a third party 
> separate from the developers. It will move blocks around and update 
> superblocks, the drive will have to be unmounted for that to work and a 
> slight misunderstanding of how to do it will ruin data.
With one exception:
xfs_fsr ("filesystem reorganizer for XFS") which is hidden (on most
distros) in the "xfsdump"-package.

It is a tool written by the xfs-developers, to defrag xfs-partitions.
Most importantly it works (only) on mounted partitions!

It can be run from cron, to walk all your xfs-partitions at night and
continue where it left of yesterday.
It seems to be designed for very huge and hyperactive installations.

In the man page it notes: "In general we do not foresee the need to run
xfs_fsr on system partitions such as /, /boot and /usr as in general
these will not suffer from fragmentation."

Bye,
Daniel

-- 
PGP key: http://pgpkeys.pca.dfn.de/pks/lookup?search=0xBB9D4887&op=get
# gpg --recv-keys --keyserver hkp://subkeys.pgp.net 0xBB9D4887


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Daniel Troeder]

[-- Attachment #1: Type: text/plain, Size: 2414 bytes --]

Am Dienstag, den 16.12.2008, 03:15 -0600 schrieb Dale:
> Daniel Troeder wrote:
> > Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale:
> >   
> >>
> >> I'm not to worried about this since I will be moving this over to the
> >> other drive anyway.  I would like to know what command I should use to
> >> tar up everything, transfer it over and untar it all on one line if
> >> possible?  I plan to do this while booted from a Gentoo CD.  I just want
> >> to try this so that it will be compressed then transfered and untared
> >> once on the way.  Does this make since?  I have used cp -av in the past.
> >>
> >> Thanks.
> >>
> >> Dale
> >>
> >> :-)  :-) 
> >>     
> > With "transfer" do you mean over a network, or to another local drive?
> >
> > You can of course use something like
> > # tar czpf - | ssh remote - tar xzpf -C /dir
> > (above probably not syntactically correct), but there are faster and
> > easier options:
> >
> > "cp -a" costs little resources locally and maintains POSIX permissions,
> > while "rsync -aASH --numeric-ids" is perfect for remote copy.
> >
> > You can use rsync also locally. It will (with the "-A" switch) also
> > transfer POSIX-ACLs, if that is of any concern. It is also useful, if a
> > transfer breaks at some moment, because it will kind of continue it :)
> >
> > Omiting the "-v" switch can significantly speed up things - depends on
> > your terminal. In every case it helps to only see the errors, and not
> > let them scroll away by everything that went well.
> >
> > Bye,
> > Daniel
> >
> >   
> 
> The drive is in the same machine so there is no network involved. 
> Should help make it a little more simple.  Would this work?
> 
> tar czpf - | tar xzpf -C /dir
> 
> Basically, I want as clean a file system as I can get to start off with
> at least.  Goal is very little fragmentation.
> 
> Thanks
> 
> Dale
While this will work perfectly well, this command is a waste of
resources. The compression ("-z") makes locally no sense, and there is
no need to tar the data (which will basically just concat files). You
will get the exact same result with
# cp -a /source /dest

If the FS has been formatted before, no fragmentation should occur in
every scenario, as long as no parallelism is used while copying, because
each file will be created and filled with data one after another.

Bye,
Daniel


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Daniel Troeder wrote:
> Am Dienstag, den 16.12.2008, 03:15 -0600 schrieb Dale:
>   
>> Daniel Troeder wrote:
>>     
>>> Am Dienstag, den 16.12.2008, 01:59 -0600 schrieb Dale:
>>>   
>>>       
>>>> I'm not to worried about this since I will be moving this over to the
>>>> other drive anyway.  I would like to know what command I should use to
>>>> tar up everything, transfer it over and untar it all on one line if
>>>> possible?  I plan to do this while booted from a Gentoo CD.  I just want
>>>> to try this so that it will be compressed then transfered and untared
>>>> once on the way.  Does this make since?  I have used cp -av in the past.
>>>>
>>>> Thanks.
>>>>
>>>> Dale
>>>>
>>>> :-)  :-) 
>>>>     
>>>>         
>>> With "transfer" do you mean over a network, or to another local drive?
>>>
>>> You can of course use something like
>>> # tar czpf - | ssh remote - tar xzpf -C /dir
>>> (above probably not syntactically correct), but there are faster and
>>> easier options:
>>>
>>> "cp -a" costs little resources locally and maintains POSIX permissions,
>>> while "rsync -aASH --numeric-ids" is perfect for remote copy.
>>>
>>> You can use rsync also locally. It will (with the "-A" switch) also
>>> transfer POSIX-ACLs, if that is of any concern. It is also useful, if a
>>> transfer breaks at some moment, because it will kind of continue it :)
>>>
>>> Omiting the "-v" switch can significantly speed up things - depends on
>>> your terminal. In every case it helps to only see the errors, and not
>>> let them scroll away by everything that went well.
>>>
>>> Bye,
>>> Daniel
>>>
>>>   
>>>       
>> The drive is in the same machine so there is no network involved. 
>> Should help make it a little more simple.  Would this work?
>>
>> tar czpf - | tar xzpf -C /dir
>>
>> Basically, I want as clean a file system as I can get to start off with
>> at least.  Goal is very little fragmentation.
>>
>> Thanks
>>
>> Dale
>>     
> While this will work perfectly well, this command is a waste of
> resources. The compression ("-z") makes locally no sense, and there is
> no need to tar the data (which will basically just concat files). You
> will get the exact same result with
> # cp -a /source /dest
>
> If the FS has been formatted before, no fragmentation should occur in
> every scenario, as long as no parallelism is used while copying, because
> each file will be created and filled with data one after another.
>
> Bye,
> Daniel
>
>   

Cool.  Then I can just use cp -a and let her rip.  I plan to redo my
partitions so I will have to reformat the partitions too.  I guess this
will be as good as it gets.  I'll also report the results of fragck when
I get this done.  Just curious myself.  I think I will skip shake this
time tho.  ;-)

Thanks much.

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Miguel Ramos]

Another argument in favour of cp in Linux: holes in sparse files are
kept correctly, whereas using tar they are not.

It is curious that this is very OS dependent.
In FreeBSD, with cp, holes always go away, using tar, or better
dump/restore is a way to keep all file attributes.
In Linux, cp -a seems to be better for archives than tar, because it
preserves these properties better, even across devices.

2008/12/16 Dale <rdalek1967@gmail.com>:
> Daniel Troeder wrote:
>> Am Dienstag, den 16.12.2008, 03:15 -0600 schrieb Dale:
[...]
>> While this will work perfectly well, this command is a waste of
>> resources. The compression ("-z") makes locally no sense, and there is
>> no need to tar the data (which will basically just concat files). You
>> will get the exact same result with
>> # cp -a /source /dest
>>
>> If the FS has been formatted before, no fragmentation should occur in
>> every scenario, as long as no parallelism is used while copying, because
>> each file will be created and filled with data one after another.
>>
>> Bye,
>> Daniel
>>
>>
>
> Cool.  Then I can just use cp -a and let her rip.  I plan to redo my
> partitions so I will have to reformat the partitions too.  I guess this
> will be as good as it gets.  I'll also report the results of fragck when
> I get this done.  Just curious myself.  I think I will skip shake this
> time tho.  ;-)
>
> Thanks much.
>
> Dale



-- 
Miguel Ramos <2008@miguel.ramos.name>
GnuPG ID 0xA006A14C

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On Tue, 16 Dec 2008 10:32:00 +0100, Daniel Troeder wrote:

> While this will work perfectly well, this command is a waste of
> resources. The compression ("-z") makes locally no sense, and there is
> no need to tar the data (which will basically just concat files). You
> will get the exact same result with
> # cp -a /source /dest

There is one slight disadvantage to cp in that it changed the modified
time of directories to the current time, which rsync does not. I'd use

rsync -ax /source/ /dest/


-- 
Neil Bothwick

If ignorance is bliss, you must be orgasmic.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Neil Bothwick wrote:
> On Tue, 16 Dec 2008 10:32:00 +0100, Daniel Troeder wrote:
>
>   
>> While this will work perfectly well, this command is a waste of
>> resources. The compression ("-z") makes locally no sense, and there is
>> no need to tar the data (which will basically just concat files). You
>> will get the exact same result with
>> # cp -a /source /dest
>>     
>
> There is one slight disadvantage to cp in that it changed the modified
> time of directories to the current time, which rsync does not. I'd use
>
> rsync -ax /source/ /dest/
>
>
>   

I made a note of that command and will give that a try.  I'll also read
the man page to see how to get it to skip /dev /sys /proc etc etc. 

Thanks

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

On Tue, 16 Dec 2008 15:34:28 -0600, Dale wrote:

> > rsync -ax /source/ /dest/

> I made a note of that command and will give that a try.  I'll also read
> the man page to see how to get it to skip /dev /sys /proc etc etc. 

That's what the -x is for.


-- 
Neil Bothwick

Be nice to moderators. They HATE that!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Neil Bothwick wrote:
> On Tue, 16 Dec 2008 15:34:28 -0600, Dale wrote:
>
>   
>>> rsync -ax /source/ /dest/
>>>       
>
>   
>> I made a note of that command and will give that a try.  I'll also read
>> the man page to see how to get it to skip /dev /sys /proc etc etc. 
>>     
>
> That's what the -x is for.
>
>
>   

Thanks for the info.  As you may can tell, I have never used rsync
before.  :/  Then again, since I will be booted from the CD, shouldn't
they be empty anyway?  Except maybe for /dev/null and /dev/console I guess?

I got to go visit a friend so it may be tomorrow before I get to
transfer now. 

Thanks again.

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

On Tue, 16 Dec 2008 17:49:11 -0600, Dale wrote:

> >> I made a note of that command and will give that a try.  I'll also
> >> read the man page to see how to get it to skip /dev /sys /proc etc
> >> etc. 

> > That's what the -x is for.

> Thanks for the info.  As you may can tell, I have never used rsync
> before.  :/

Your portage tree must be very outdated by now :P

> Then again, since I will be booted from the CD, shouldn't
> they be empty anyway?  Except maybe for /dev/null and /dev/console I
> guess?


Yes, and that is a better way of doing it as you will copy the two files
in the underlying /dev. -x tells rsync to not cross filesystem
boundaries, so it will make no difference if you are working from a live
CD.


-- 
Neil Bothwick

Exercise daily. Eat wisely. Die anyway.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Neil Bothwick wrote:
> On Tue, 16 Dec 2008 17:49:11 -0600, Dale wrote:
>
>   
>>>> I made a note of that command and will give that a try.  I'll also
>>>> read the man page to see how to get it to skip /dev /sys /proc etc
>>>> etc. 
>>>>         
>
>   
>>> That's what the -x is for.
>>>       
>
>   
>> Thanks for the info.  As you may can tell, I have never used rsync
>> before.  :/
>>     
>
> Your portage tree must be very outdated by now :P
>
>   
>> Then again, since I will be booted from the CD, shouldn't
>> they be empty anyway?  Except maybe for /dev/null and /dev/console I
>> guess?
>>     
>
>
> Yes, and that is a better way of doing it as you will copy the two files
> in the underlying /dev. -x tells rsync to not cross filesystem
> boundaries, so it will make no difference if you are working from a live
> CD.
>
>
>   

Well, I have used eix-sync but not rsync directly as a command.  I sync
once a week, give or take.

Light bulb warning.  So null and console are on the drive for it to
start up but once it mounts /dev then it uses that "virtual" thing? 
Cool, if I understand that correctly. 


I like it when my light bulb comes on.  It's so . . . . . pretty.  LOL

Dale

:-)  :-) 

P. S.  My friend got called in to work.  May be done tonight after all. 
:-( 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On Tue, 16 Dec 2008 19:06:06 -0600, Dale wrote:

> Light bulb warning.  So null and console are on the drive for it to
> start up but once it mounts /dev then it uses that "virtual" thing? 
> Cool, if I understand that correctly. 

Yes, those two devices are needed before udev starts,so they have to be on
the root filesystem. If you have anything else in dev on the root
filesystem, you are only wasting space.


-- 
Neil Bothwick

WinErr 007: System price error - Inadequate money spent on hardware

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Neil Bothwick wrote:
> On Tue, 16 Dec 2008 19:06:06 -0600, Dale wrote:
>
>   
>> Light bulb warning.  So null and console are on the drive for it to
>> start up but once it mounts /dev then it uses that "virtual" thing? 
>> Cool, if I understand that correctly. 
>>     
>
> Yes, those two devices are needed before udev starts,so they have to be on
> the root filesystem. If you have anything else in dev on the root
> filesystem, you are only wasting space.
>
>
>   

I got it transfered over.  I noticed something weird tho.  I was booted
from the CD.  When I was checking the permissions to make sure things
were going well, it kept showing gentoo:users instead of dale:users for
example.  The ones that were root were fine but the ones that should be
dale:users was gentoo:users.  I stopped and reformatted the drives and
it always did the same thing.  I finally gave in and let it copy anyway.

After it was copied, I chroot'ed in and all the permissions were like
they should be including dale:users.  Any idea why it did that?  It did
the same thing with both rsync -ax and cp -av.  Just thought it was
weird is all.

I did copy null and console over after deleting the rest.  I also
checked /sys and /proc to make sure.

I also ran the fragck on it again.  I got this:

3.26229678132721% non contiguous files, 1.08394810041292 average fragments.

Not to bad I guess.  There is about 500,000 files or so. 

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

On Wed, 17 Dec 2008 06:20:38 -0600, Dale wrote:

> I got it transfered over.  I noticed something weird tho.  I was booted
> from the CD.  When I was checking the permissions to make sure things
> were going well, it kept showing gentoo:users instead of dale:users for
> example.  The ones that were root were fine but the ones that should be
> dale:users was gentoo:users.  I stopped and reformatted the drives and
> it always did the same thing.  I finally gave in and let it copy anyway.
> 
> After it was copied, I chroot'ed in and all the permissions were like
> they should be including dale:users.  Any idea why it did that?  It did
> the same thing with both rsync -ax and cp -av.  Just thought it was
> weird is all.

Filesystems store numeric values for UID/GID, commands like ls translate
these to actual names. Gentoo normally makes the first user 1000, which
is probably the UID of dale on your installation and gentoo on the live
CD.Root is always UID 0, which is why that was shown correctly.


-- 
Neil Bothwick

I doubt therefore I might be.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Neil Bothwick wrote:
> On Wed, 17 Dec 2008 06:20:38 -0600, Dale wrote:
>
>   
>> I got it transfered over.  I noticed something weird tho.  I was booted
>> from the CD.  When I was checking the permissions to make sure things
>> were going well, it kept showing gentoo:users instead of dale:users for
>> example.  The ones that were root were fine but the ones that should be
>> dale:users was gentoo:users.  I stopped and reformatted the drives and
>> it always did the same thing.  I finally gave in and let it copy anyway.
>>
>> After it was copied, I chroot'ed in and all the permissions were like
>> they should be including dale:users.  Any idea why it did that?  It did
>> the same thing with both rsync -ax and cp -av.  Just thought it was
>> weird is all.
>>     
>
> Filesystems store numeric values for UID/GID, commands like ls translate
> these to actual names. Gentoo normally makes the first user 1000, which
> is probably the UID of dale on your installation and gentoo on the live
> CD.Root is always UID 0, which is why that was shown correctly.
>
>
>   

Oh, makes sense.  Should have known that computers reduce everything to
numbers.  ROFLMAO  At least now I know why it did that.  It had me
freaked out for a bit there.

New transfer is working very well.  Pretty swift but not much difference
from the old one.  At least I got some of the cruft cleaned out.

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Mick]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

On Tuesday 16 December 2008, Miguel Ramos wrote:
> Another argument in favour of cp in Linux: holes in sparse files are
> kept correctly, whereas using tar they are not.
>
> It is curious that this is very OS dependent.
> In FreeBSD, with cp, holes always go away, using tar, or better
> dump/restore is a way to keep all file attributes.
> In Linux, cp -a seems to be better for archives than tar, because it
> preserves these properties better, even across devices.

Hmm..., with tar, -p will preserve permissions and -S will handle sparce files 
efficiently.  -W will additionally verify that that data was archived without 
corruption.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Nikos Chantziaras]

Nikos Chantziaras wrote:
> Jorge Peixoto de Morais Neto wrote:
>>>>> [...] what would be the best way to defrag it?
>>>> By not defragging it.
>>>> [...]
>>> I don't buy into that argument and never did.  Every few months I 
>>> copy the
>>> whole HD to another one and then back to counter fragmentation (ext3) 
>>> and
>>> the system becomes noticeably faster after doing it (speed increase in
>>> emerge --sync for example.)  Maybe it's not fragmentation but rather 
>>> related
>>> files being more closely together after I do this.
>>
>> How exactly do you copy the files?  [...]
> 
> I simply boot from the Gentoo DVD and rsync to another ext3 partition, 
> wipe the current filesystem and then rsync back.

OK, I once again verified that fragmentation seems to be a big issue 
even on Linux.  I just migrated to ext4, and in order to do that I had 
to rsync, format and rsync back.  The result is similar to the last time 
I did this (over 8 months ago):

emerge --sync takes 15 seconds (at least 3 minutes yesterday)
update-eix takes 2 seconds (20 seconds yesterday)

And I don't believe it's due to ext4.  It's a nice speed-up from ext3, 
but not THAT nice.

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Nikos Chantziaras wrote:
> Nikos Chantziaras wrote:
>> Jorge Peixoto de Morais Neto wrote:
>>>>>> [...] what would be the best way to defrag it?
>>>>> By not defragging it.
>>>>> [...]
>>>> I don't buy into that argument and never did.  Every few months I
>>>> copy the
>>>> whole HD to another one and then back to counter fragmentation
>>>> (ext3) and
>>>> the system becomes noticeably faster after doing it (speed increase in
>>>> emerge --sync for example.)  Maybe it's not fragmentation but
>>>> rather related
>>>> files being more closely together after I do this.
>>>
>>> How exactly do you copy the files?  [...]
>>
>> I simply boot from the Gentoo DVD and rsync to another ext3
>> partition, wipe the current filesystem and then rsync back.
>
> OK, I once again verified that fragmentation seems to be a big issue
> even on Linux.  I just migrated to ext4, and in order to do that I had
> to rsync, format and rsync back.  The result is similar to the last
> time I did this (over 8 months ago):
>
> emerge --sync takes 15 seconds (at least 3 minutes yesterday)
> update-eix takes 2 seconds (20 seconds yesterday)
>
> And I don't believe it's due to ext4.  It's a nice speed-up from ext3,
> but not THAT nice.
>
>
>

Well, try as I may, I could not get mine past 10% on resiserfs. 
Fragmentation happens on any file system but I think the point is that
Linux doesn't get as bad as the windoze file system.  10% or so is not
to bad depending on the size of the files.  Files that are large will
have to be fragmented no matter what file system you use. 

I posted in another the reply right after a copy to another drive.  I
think that was before I even booted into the OS and was still on the
CD.  It is around 2% or so.  I doubt given that condition that it could
get any better. 

Dale

:-)   :-) 

[gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Nikos Chantziaras]

Dale wrote:
> Nikos Chantziaras wrote:
>> OK, I once again verified that fragmentation seems to be a big issue
>> even on Linux.  I just migrated to ext4, and in order to do that I had
>> to rsync, format and rsync back.  The result is similar to the last
>> time I did this (over 8 months ago):
>>
>> emerge --sync takes 15 seconds (at least 3 minutes yesterday)
>> update-eix takes 2 seconds (20 seconds yesterday)
>>
>> And I don't believe it's due to ext4.  It's a nice speed-up from ext3,
>> but not THAT nice.
> 
> Well, try as I may, I could not get mine past 10% on resiserfs. 
> Fragmentation happens on any file system but I think the point is that
> Linux doesn't get as bad as the windoze file system.  10% or so is not
> to bad depending on the size of the files.  Files that are large will
> have to be fragmented no matter what file system you use. 
> 
> I posted in another the reply right after a copy to another drive.  I
> think that was before I even booted into the OS and was still on the
> CD.  It is around 2% or so.  I doubt given that condition that it could
> get any better. 

I think the main problem may not be so much fragmentation of files, but 
rather their position on disk.  Even if files are not fragmented, if 
they are located too far from each other even though they're related 
(same directory for example) or there's simply too much empty space 
between files (I think this is intentional in order to reduce 
fragmentation) then seek times get really bad.  After I rsync the data 
back, it's nicely and sequentially laid out on disk.  I guess over time 
it starts to get further apart again (to combat fragmentation) and 
emerge --sync goes up from 15 seconds to 2 minutes again.  Even though 
the files aren't fragmented at all.

Some defrag apps for Windoze actually offer to put the files back closer 
together without trying to defragment at all.  I guess this is why :P

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Alan McKinnon]

On Friday 26 December 2008 21:49:02 Nikos Chantziaras wrote:

> OK, I once again verified that fragmentation seems to be a big issue
> even on Linux.  I just migrated to ext4, and in order to do that I had
> to rsync, format and rsync back.  The result is similar to the last time
> I did this (over 8 months ago):
>
> emerge --sync takes 15 seconds (at least 3 minutes yesterday)
> update-eix takes 2 seconds (20 seconds yesterday)
>
> And I don't believe it's due to ext4.  It's a nice speed-up from ext3,
> but not THAT nice.

Um, did it occur to you that after you emerge --sync'ed yesterday and ran 
update-eix that your portage tree is now very up to date and your eix cache 
is hot?

Therefore successive runs will naturally be much quicker? And that yesterday 
was xmas day, a day most likely to involve very few if any portage updates? 
Or that emerge --sync could easily speed up simply because you had more 
bandwidth?

Your speed-ups likely have very little to do with your filesystem. 

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Nikos Chantziaras]

Alan McKinnon wrote:
> On Friday 26 December 2008 21:49:02 Nikos Chantziaras wrote:
> 
>> OK, I once again verified that fragmentation seems to be a big issue
>> even on Linux.  I just migrated to ext4, and in order to do that I had
>> to rsync, format and rsync back.  The result is similar to the last time
>> I did this (over 8 months ago):
>>
>> emerge --sync takes 15 seconds (at least 3 minutes yesterday)
>> update-eix takes 2 seconds (20 seconds yesterday)
>>
>> And I don't believe it's due to ext4.  It's a nice speed-up from ext3,
>> but not THAT nice.
> 
> Um, did it occur to you that after you emerge --sync'ed yesterday and ran 
> update-eix that your portage tree is now very up to date and your eix cache 
> is hot?
> 
> Therefore successive runs will naturally be much quicker? And that yesterday 
> was xmas day, a day most likely to involve very few if any portage updates? 
> Or that emerge --sync could easily speed up simply because you had more 
> bandwidth?
> 
> Your speed-ups likely have very little to do with your filesystem. 

Well, instead of "yesterday" let's just say "the past 5 months".  I 
already did the rsync/format thing a few times over the last years, and 
the results are always the same: very fast filesystem for about a month, 
then it starts getting slower over time.

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Nikos Chantziaras wrote:
> Dale wrote:
>> Nikos Chantziaras wrote:
>>> OK, I once again verified that fragmentation seems to be a big issue
>>> even on Linux.  I just migrated to ext4, and in order to do that I had
>>> to rsync, format and rsync back.  The result is similar to the last
>>> time I did this (over 8 months ago):
>>>
>>> emerge --sync takes 15 seconds (at least 3 minutes yesterday)
>>> update-eix takes 2 seconds (20 seconds yesterday)
>>>
>>> And I don't believe it's due to ext4.  It's a nice speed-up from ext3,
>>> but not THAT nice.
>>
>> Well, try as I may, I could not get mine past 10% on resiserfs.
>> Fragmentation happens on any file system but I think the point is that
>> Linux doesn't get as bad as the windoze file system.  10% or so is not
>> to bad depending on the size of the files.  Files that are large will
>> have to be fragmented no matter what file system you use.
>> I posted in another the reply right after a copy to another drive.  I
>> think that was before I even booted into the OS and was still on the
>> CD.  It is around 2% or so.  I doubt given that condition that it could
>> get any better. 
>
> I think the main problem may not be so much fragmentation of files,
> but rather their position on disk.  Even if files are not fragmented,
> if they are located too far from each other even though they're
> related (same directory for example) or there's simply too much empty
> space between files (I think this is intentional in order to reduce
> fragmentation) then seek times get really bad.  After I rsync the data
> back, it's nicely and sequentially laid out on disk.  I guess over
> time it starts to get further apart again (to combat fragmentation)
> and emerge --sync goes up from 15 seconds to 2 minutes again.  Even
> though the files aren't fragmented at all.
>
> Some defrag apps for Windoze actually offer to put the files back
> closer together without trying to defragment at all.  I guess this is
> why :P
>
>
>


Well, this is what I got on my rig.  Sort of interesting in a way.

root@smoker / # mount
/dev/hda6 on / type reiserfs (rw)
/proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec)
udev on /dev type tmpfs (rw,nosuid)
devpts on /dev/pts type devpts (rw,nosuid,noexec)
/dev/hda1 on /boot type ext2 (rw)
/dev/hda7 on /home type reiserfs (rw)
/dev/hda8 on /usr/portage type ext2 (rw)
/dev/hda9 on /data type reiserfs (rw)
none on /dev/shm type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc
(rw,noexec,nosuid,nodev)
/dev/hdd on /media/hdd type udf
(rw,noexec,nosuid,nodev,uid=0,gid=0,umask=007)
root@smoker / #

root@smoker / # /root/fragck.pl /
3.10978840211776% non contiguous files, 1.08156705459019 average fragments.
root@smoker / # /root/fragck.pl /usr/portage/
0.0276657266269232% non contiguous files, 1.00029450612216 average
fragments.
root@smoker / # /root/fragck.pl /boot/
6.25% non contiguous files, 1.0625 average fragments.
root@smoker / # /root/fragck.pl /home/
3.2440588457186% non contiguous files, 1.16408902301018 average fragments.
root@smoker / # /root/fragck.pl /data/
5.56267766568196% non contiguous files, 1.06797837355777 average fragments.
root@smoker / #

Now keep in mind that the first one includes all the others.  I'm logged
into a GUI so I can't umount /home at least.  May do that in single mode
someday.

I think sometimes the files are just to big to fit on one section.  I
know I have some files that are pretty big.  I got a couple videos that
are big that came off my camera and one video that is a hour or so
long.  I think there are a lot of variables that without a microscope we
can never see and know for sure.

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Nikos Chantziaras wrote:
> Alan McKinnon wrote:
>> On Friday 26 December 2008 21:49:02 Nikos Chantziaras wrote:
>>
>>> OK, I once again verified that fragmentation seems to be a big issue
>>> even on Linux.  I just migrated to ext4, and in order to do that I had
>>> to rsync, format and rsync back.  The result is similar to the last
>>> time
>>> I did this (over 8 months ago):
>>>
>>> emerge --sync takes 15 seconds (at least 3 minutes yesterday)
>>> update-eix takes 2 seconds (20 seconds yesterday)
>>>
>>> And I don't believe it's due to ext4.  It's a nice speed-up from ext3,
>>> but not THAT nice.
>>
>> Um, did it occur to you that after you emerge --sync'ed yesterday and
>> ran update-eix that your portage tree is now very up to date and your
>> eix cache is hot?
>>
>> Therefore successive runs will naturally be much quicker? And that
>> yesterday was xmas day, a day most likely to involve very few if any
>> portage updates? Or that emerge --sync could easily speed up simply
>> because you had more bandwidth?
>>
>> Your speed-ups likely have very little to do with your filesystem. 
>
> Well, instead of "yesterday" let's just say "the past 5 months".  I
> already did the rsync/format thing a few times over the last years,
> and the results are always the same: very fast filesystem for about a
> month, then it starts getting slower over time.
>
>
>

I have to say that after my recent transfer, my login got a whole one
second faster.  I can't tell any difference anywhere else.  Of course,
portage has always been on its own partition and used ext3.

We need a hard drive engineer on here.  :/

Dale

:-)  :-) 

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Matt Harrison]

Dale wrote:
> I have to say that after my recent transfer, my login got a whole one
> second faster.  I can't tell any difference anywhere else.  Of course,
> portage has always been on its own partition and used ext3.
> 
> We need a hard drive engineer on here.  :/
> 
> Dale
> 
> :-)  :-) 
> 

Hey, I've been following this thread with some interest. I just wanted 
to note that you guys might like to subscribe to Sun's ZFS-discuss list 
and possibly Storage-discuss. The guys on there really are hard-disk 
gurus and some of the things they talk about are miles over my head.

It's just interesting as ZFS is supposedly (and I believe it) THE 
filesystem when it comes to combating fragmentation. Maybe reading over 
what those guys chat about would be interesting to some folks from this 
thread.

In fact, the guys over at Sun are so hot on fighting fragmentation, 
they're already looking at some really advanced things like low level 
algorithms for deduplication and some other things that scare me and 
make me want to take a hot shower :P

Happy holidays

Matt

[gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Nikos Chantziaras]

Dale wrote:
> Nikos Chantziaras wrote:
>> I already did the rsync/format thing a few times over the last years,
>> and the results are always the same: very fast filesystem for about a
>> month, then it starts getting slower over time.
> 
> I have to say that after my recent transfer, my login got a whole one
> second faster.  I can't tell any difference anywhere else.  Of course,
> portage has always been on its own partition and used ext3.
> 
> We need a hard drive engineer on here.  :/

Well, I have everything in /.  Except for /boot.  Maybe I should 
reconsider my setup :P

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Matt Harrison wrote:
> Dale wrote:
>> I have to say that after my recent transfer, my login got a whole one
>> second faster.  I can't tell any difference anywhere else.  Of course,
>> portage has always been on its own partition and used ext3.
>>
>> We need a hard drive engineer on here.  :/
>>
>> Dale
>>
>> :-)  :-)
>
> Hey, I've been following this thread with some interest. I just wanted
> to note that you guys might like to subscribe to Sun's ZFS-discuss
> list and possibly Storage-discuss. The guys on there really are
> hard-disk gurus and some of the things they talk about are miles over
> my head.
>
> It's just interesting as ZFS is supposedly (and I believe it) THE
> filesystem when it comes to combating fragmentation. Maybe reading
> over what those guys chat about would be interesting to some folks
> from this thread.
>
> In fact, the guys over at Sun are so hot on fighting fragmentation,
> they're already looking at some really advanced things like low level
> algorithms for deduplication and some other things that scare me and
> make me want to take a hot shower :P
>
> Happy holidays
>
> Matt
>
>

But if we learned to much, we may be dangerous or something.  Sometimes
to much knowledge can be bad.  lol

I !think! I tried XFS once.  If it was XFS, you need to have a UPS for
sure.  Every time the system crashed I had to re-install.  I never got
it to recover even once.  I have heard the same thing about its defrag
efficiency tho.  Just don't trust it to much with my data.

Dale

:-)  :-)

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Dale]

Nikos Chantziaras wrote:
> Dale wrote:
>> Nikos Chantziaras wrote:
>>> I already did the rsync/format thing a few times over the last years,
>>> and the results are always the same: very fast filesystem for about a
>>> month, then it starts getting slower over time.
>>
>> I have to say that after my recent transfer, my login got a whole one
>> second faster.  I can't tell any difference anywhere else.  Of course,
>> portage has always been on its own partition and used ext3.
>>
>> We need a hard drive engineer on here.  :/
>
> Well, I have everything in /.  Except for /boot.  Maybe I should
> reconsider my setup :P
>
>
>

How you partition depends on what you are doing.  I just have a desktop
myself.  I keep /home and /data separate in case I need to switch over
for some reason and portage just to keep it from getting to fragmented. 
I guess it helps some at least.

Dale

:-)  :-)

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Matt Harrison]

Dale wrote:
> But if we learned to much, we may be dangerous or something.  Sometimes
> to much knowledge can be bad.  lol
> 
> I !think! I tried XFS once.  If it was XFS, you need to have a UPS for
> sure.  Every time the system crashed I had to re-install.  I never got
> it to recover even once.  I have heard the same thing about its defrag
> efficiency tho.  Just don't trust it to much with my data.
> 
> Dale
> 
> :-)  :-)
> 

Not sure if you're talking about something else but I was talking about 
ZFS[1], not XFS. ZFS is the latest filesystem from Sun which ships with 
the later versions of Solaris/OpenSolaris. I don't want to be seen to 
advertise it loads here, but it really is good. I recently moved my 
fileserver to a solaris/ZFS box instead of raid on gentoo. Since then my 
data hasn't been inaccessible once, and I haven't had the scary problems 
like when gentoo decides to reboot and not bring my arrays back online :)

[1]http://en.wikipedia.org/wiki/ZFS

Matt

Re: [gentoo-user] Re: Fragmentation of my drives. Curious mostly

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Fri, 26 Dec 2008 23:02:38 +0200, Nikos Chantziaras wrote:

> Well, instead of "yesterday" let's just say "the past 5 months".  I 
> already did the rsync/format thing a few times over the last years, and 
> the results are always the same: very fast filesystem for about a
> month, then it starts getting slower over time.

I keep /usr/portage on a separate, small filesystem, using ext2. If
it starts to slow down,I can simply reformat it and sync again.


-- 
Neil Bothwick

Some people are born mediocre, some people achieve mediocrity, and some
people have mediocrity thrust upon them.  - Joseph Heller, "Catch-22"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]