[gentoo-user] Anybody tried shake defragmenter?

[gentoo-user] Anybody tried shake defragmenter?

[Grant]

My HD is getting noisier during access and I wonder if it's a
fragmentation issue.  I have:

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3            960872076 754795944 157266648  83% /

I know Linux systems aren't supposed to become fragmented, but I've
also read that it can happen eventually.  I'm on ext3.  I've read that
ext4 will have a defragmenter but that it doesn't have one yet.  Has
anyone tried the shake defragmenter?

- Grant

Re: [gentoo-user] Anybody tried shake defragmenter?

[Albert Hopkins]

On Mon, 2009-08-03 at 13:22 -0700, Grant wrote:
> My HD is getting noisier during access and I wonder if it's a
> fragmentation issue. 

Are you sure it's not a HD-about-to-die issue?

-a

[gentoo-user] Re: Anybody tried shake defragmenter?

[Grant Edwards]

On 2009-08-03, Grant <emailgrant@gmail.com> wrote:
> My HD is getting noisier during access and I wonder if it's a
> fragmentation issue.  I have:
>
> # df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sda3            960872076 754795944 157266648  83% /
>
> I know Linux systems aren't supposed to become fragmented,

Not true.  They become fragmented.  However, it's not supposed
to matter.

-- 
Grant Edwards                   grante             Yow! The FALAFEL SANDWICH
                                  at               lands on my HEAD and I
                               visi.com            become a VEGETARIAN ...

Re: [gentoo-user] Anybody tried shake defragmenter?

[Thierry de Coulon]

On Monday 03 August 2009, Grant wrote:
> # df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sda3            960872076 754795944 157266648  83% /

The partition is fairly full, probably the system has a hard time finding a 
spot to create an unfragmented file. I remember I read a partition should not 
be more than 50% used, maybe I'm wrong.

Anyway, I would not use such a full partition for / or /home. When it happend 
I moved /usr to another partition.

Thierry

Re: [gentoo-user] Anybody tried shake defragmenter?

[Alan McKinnon]

On Monday 03 August 2009 22:51:58 Thierry de Coulon wrote:
> On Monday 03 August 2009, Grant wrote:
> > # df
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > /dev/sda3            960872076 754795944 157266648  83% /
>
> The partition is fairly full, probably the system has a hard time finding a
> spot to create an unfragmented file. I remember I read a partition should
> not be more than 50% used, maybe I'm wrong.

Well, that is just flat out wrong and simple logic tells you why.

If it were true, you could never use more than half your disk space. So you 
buy a 1T disk to get 500G. Doesn't make sense right?

The world is full of people who talk through holes in their arses. You seem to 
have read one of their missives.

> Anyway, I would not use such a full partition for / or /home. When it
> happend I moved /usr to another partition.

You do want some breathing space, at least as big as the largest chunk of data 
the fs layer is going to move around in one operation. This of course is a 
highly variable amount. About 5% is a reasonable rule of thumb, modified by 
benchmarks you do on your own data.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Anybody tried shake defragmenter?

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 236 bytes --]

Am Montag 03 August 2009 22:51:58 schrieb Thierry de Coulon:

> Anyway, I would not use such a full partition for / or /home. When it
> happend I moved /usr to another partition.

Hmm, I simply extend the logical volume.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Anybody tried shake defragmenter?

[Paul Hartman]

On Mon, Aug 3, 2009 at 3:22 PM, Grant<emailgrant@gmail.com> wrote:
> I know Linux systems aren't supposed to become fragmented, but I've
> also read that it can happen eventually.  I'm on ext3.  I've read that
> ext4 will have a defragmenter but that it doesn't have one yet.

It's not that they aren't supposed to become fragmented, it is that
they try to avoid it. There is a big difference, and things like
streaming writes (downloads, bittorrents, etc) can cause extreme
fragmentation.

The time-honored way of fixing this is "backup, delete, restore". In
my case my simple defragmenter is to move a file to tmpfs and then
move it back to the hard drive. I always do this to files I'm about to
burn to a CD/DVD to ensure the read speed is optimal.

> Has anyone tried the shake defragmenter?

Yes, nothing has blown up yet. :)

Re: [gentoo-user] Anybody tried shake defragmenter?

[Alan McKinnon]

On Monday 03 August 2009 23:05:02 Paul Hartman wrote:
> On Mon, Aug 3, 2009 at 3:22 PM, Grant<emailgrant@gmail.com> wrote:
> > I know Linux systems aren't supposed to become fragmented, but I've
> > also read that it can happen eventually.  I'm on ext3.  I've read that
> > ext4 will have a defragmenter but that it doesn't have one yet.
>
> It's not that they aren't supposed to become fragmented, it is that
> they try to avoid it. There is a big difference, and things like
> streaming writes (downloads, bittorrents, etc) can cause extreme
> fragmentation.
>
> The time-honored way of fixing this is "backup, delete, restore". In
> my case my simple defragmenter is to move a file to tmpfs and then
> move it back to the hard drive. I always do this to files I'm about to
> burn to a CD/DVD to ensure the read speed is optimal.

Until one day someone write a super-duper disk cache algorithm that delays 
writes safely, notices that you are putting back unmodified something you just 
deleted, then reverts "to be deleted" flag on the block pointers. meaning that 
nothing has changed.

Lucky for us, I do not believe that such a driver has been written yet.
Unlucky for us, I believe that such a driver is entirely possible.

:-)

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: Anybody tried shake defragmenter?

[Grant Edwards]

On 2009-08-03, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On Monday 03 August 2009 23:05:02 Paul Hartman wrote:
>
>> The time-honored way of fixing this is "backup, delete,
>> restore". In my case my simple defragmenter is to move a file
>> to tmpfs and then move it back to the hard drive. I always do
>> this to files I'm about to burn to a CD/DVD to ensure the read
>> speed is optimal.
>
> Until one day someone write a super-duper disk cache algorithm
> that delays writes safely, notices that you are putting back
> unmodified something you just deleted, then reverts "to be
> deleted" flag on the block pointers. meaning that nothing has
> changed.
>
> Lucky for us, I do not believe that such a driver has been
> written yet. Unlucky for us, I believe that such a driver is
> entirely possible.

And actually quite simple once the
content-addressable-disk-drive is invented.

-- 
Grant Edwards                   grante             Yow! Didn't I buy a 1951
                                  at               Packard from you last March
                               visi.com            in Cairo?

Re: [gentoo-user] Re: Anybody tried shake defragmenter?

[Alan McKinnon]

On Monday 03 August 2009 23:16:05 Grant Edwards wrote:
> On 2009-08-03, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> > On Monday 03 August 2009 23:05:02 Paul Hartman wrote:
> >> The time-honored way of fixing this is "backup, delete,
> >> restore". In my case my simple defragmenter is to move a file
> >> to tmpfs and then move it back to the hard drive. I always do
> >> this to files I'm about to burn to a CD/DVD to ensure the read
> >> speed is optimal.
> >
> > Until one day someone write a super-duper disk cache algorithm
> > that delays writes safely, notices that you are putting back
> > unmodified something you just deleted, then reverts "to be
> > deleted" flag on the block pointers. meaning that nothing has
> > changed.
> >
> > Lucky for us, I do not believe that such a driver has been
> > written yet. Unlucky for us, I believe that such a driver is
> > entirely possible.
>
> And actually quite simple once the
> content-addressable-disk-drive is invented.

We tried that already, it was called WinFS.

Unfortunately, it was an idea ahead of it's time and technology was not quite 
ready for it yet :-)

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Anybody tried shake defragmenter?

[Volker Armin Hemmann]

On Montag 03 August 2009, Alan McKinnon wrote:
> On Monday 03 August 2009 23:16:05 Grant Edwards wrote:
> > On 2009-08-03, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> > > On Monday 03 August 2009 23:05:02 Paul Hartman wrote:
> > >> The time-honored way of fixing this is "backup, delete,
> > >> restore". In my case my simple defragmenter is to move a file
> > >> to tmpfs and then move it back to the hard drive. I always do
> > >> this to files I'm about to burn to a CD/DVD to ensure the read
> > >> speed is optimal.
> > >
> > > Until one day someone write a super-duper disk cache algorithm
> > > that delays writes safely, notices that you are putting back
> > > unmodified something you just deleted, then reverts "to be
> > > deleted" flag on the block pointers. meaning that nothing has
> > > changed.
> > >
> > > Lucky for us, I do not believe that such a driver has been
> > > written yet. Unlucky for us, I believe that such a driver is
> > > entirely possible.
> >
> > And actually quite simple once the
> > content-addressable-disk-drive is invented.
>
> We tried that already, it was called WinFS.
>
> Unfortunately, it was an idea ahead of it's time and technology was not
> quite ready for it yet :-)


http://en.wikipedia.org/wiki/Be_File_System

was first and did it.

Re: [gentoo-user] Re: Anybody tried shake defragmenter?

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 301 bytes --]

Am Dienstag 04 August 2009 11:21:49 schrieb Volker Armin Hemmann:

> http://en.wikipedia.org/wiki/Be_File_System
>
> was first and did it.

Yeah, I liked the database characteristics of Old BFS, but unfortunately its 
performance was suboptimal, even on the faster BeBoxen.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Anybody tried shake defragmenter?

[Dale]

Paul Hartman wrote:
> On Mon, Aug 3, 2009 at 3:22 PM, Grant<emailgrant@gmail.com> wrote:
>   
>> Has anyone tried the shake defragmenter?
>>     
>
> Yes, nothing has blown up yet. :)
>
>
>   

I used it a while back but couldn't really see a whole lot of
difference.  The numbers said it helped but not much else changed.  I
think logging into KDE was a little faster is about all.  I'm with Alan
on this one.  It just doesn't get fragmented like windoze does. 

Dale

:-)  :-) 

Re: [gentoo-user] Anybody tried shake defragmenter?

[Paul Hartman]

On Mon, Aug 3, 2009 at 4:33 PM, Dale<rdalek1967@gmail.com> wrote:
> Paul Hartman wrote:
>> On Mon, Aug 3, 2009 at 3:22 PM, Grant<emailgrant@gmail.com> wrote:
>>
>>> Has anyone tried the shake defragmenter?
>>>
>>
>> Yes, nothing has blown up yet. :)
>>
>>
>>
>
> I used it a while back but couldn't really see a whole lot of
> difference.  The numbers said it helped but not much else changed.  I
> think logging into KDE was a little faster is about all.  I'm with Alan
> on this one.  It just doesn't get fragmented like windoze does.

I think it really depends on the situation. For example I have a fast
connection (20 megabit) so to maximize it I will often have several
downloads in parallel, which causes files to be very fragmented. I
have experienced a noticeable slowdown reading really fragmented files
(2 or 3Mbyte/sec, when normal reads are around 45Mbyte/sec). At speeds
that slow it can be slower than the burn speed of a DVD, which is not
good, and it just slows everything down in gernal.

Small files (less than 1 megabyte) are rarely fragmented and even when
they are, it isn't going to have any significant effect on
performance.

I would defrag large files or files that are downloaded/appended, such
as /usr/portage/distfiles and /var/log. If you're dealing with large
digital camera pictures, audio or video then I would definitely defrag
those files. Everything else in /usr/bin and so on are probably not
fragmented to begin with since the files are are written at-once and
whole when you emerge packages.

Re: [gentoo-user] Anybody tried shake defragmenter?

[Grant]

>> I know Linux systems aren't supposed to become fragmented, but I've
>> also read that it can happen eventually.  I'm on ext3.  I've read that
>> ext4 will have a defragmenter but that it doesn't have one yet.
>
> It's not that they aren't supposed to become fragmented, it is that
> they try to avoid it. There is a big difference, and things like
> streaming writes (downloads, bittorrents, etc) can cause extreme
> fragmentation.

Yeah, that's when I'm hearing the HD access I didn't hear before.  I
run miro and it's downloading several torrents all the time.  It never
made a sound before, but now there's a rhythmic grinding sound when
miro is running, maybe because the HD is more full now.  Could shake
help with this?  To find out, should I be running it on the partially
downloaded torrents?

- Grant


> The time-honored way of fixing this is "backup, delete, restore". In
> my case my simple defragmenter is to move a file to tmpfs and then
> move it back to the hard drive. I always do this to files I'm about to
> burn to a CD/DVD to ensure the read speed is optimal.
>
>> Has anyone tried the shake defragmenter?
>
> Yes, nothing has blown up yet. :)

Re: [gentoo-user] Anybody tried shake defragmenter?

[Paul Hartman]

On Mon, Aug 3, 2009 at 6:48 PM, Grant<emailgrant@gmail.com> wrote:
>>> I know Linux systems aren't supposed to become fragmented, but I've
>>> also read that it can happen eventually.  I'm on ext3.  I've read that
>>> ext4 will have a defragmenter but that it doesn't have one yet.
>>
>> It's not that they aren't supposed to become fragmented, it is that
>> they try to avoid it. There is a big difference, and things like
>> streaming writes (downloads, bittorrents, etc) can cause extreme
>> fragmentation.
>
> Yeah, that's when I'm hearing the HD access I didn't hear before.  I
> run miro and it's downloading several torrents all the time.  It never
> made a sound before, but now there's a rhythmic grinding sound when
> miro is running, maybe because the HD is more full now.  Could shake
> help with this?  To find out, should I be running it on the partially
> downloaded torrents?

Well, bittorent does not download in sequential order, so it is
constantly doing random reads and writes. You may not be able to avoid
the HD grinding during this kind of activity. Download to a RAM drive
or SSD or something perhaps.

Fragmentation definitely gets worse the nearer you are to full (which
for me is always). I have seen very small files with hundreds of
fragments as I live at 99% of my space used. They say a hard drive has
2 states: new and full :)

It certainly wouldn't hurt to defrag the partial files, though you may
want to pause your download before doing it (I don't know how much
locking/blocking may occur on in-use files). Some bittorrent clients
have an option to write a placeholder file; this is supposed to
prevent fragmentation since it's allocating the space for the whole
file immediately. Vuze is what I use, it calls this option "allocate
and zero new files on creation". The down-side is it could take a
while to initialize if you're downloading something huge, especially
if you're saving to a network or USB hard drive that's not very fast.

Re: [gentoo-user] Anybody tried shake defragmenter?

[Grant]

>>>> I know Linux systems aren't supposed to become fragmented, but I've
>>>> also read that it can happen eventually.  I'm on ext3.  I've read that
>>>> ext4 will have a defragmenter but that it doesn't have one yet.
>>>
>>> It's not that they aren't supposed to become fragmented, it is that
>>> they try to avoid it. There is a big difference, and things like
>>> streaming writes (downloads, bittorrents, etc) can cause extreme
>>> fragmentation.
>>
>> Yeah, that's when I'm hearing the HD access I didn't hear before.  I
>> run miro and it's downloading several torrents all the time.  It never
>> made a sound before, but now there's a rhythmic grinding sound when
>> miro is running, maybe because the HD is more full now.  Could shake
>> help with this?  To find out, should I be running it on the partially
>> downloaded torrents?
>
> Well, bittorent does not download in sequential order, so it is
> constantly doing random reads and writes. You may not be able to avoid
> the HD grinding during this kind of activity. Download to a RAM drive
> or SSD or something perhaps.
>
> Fragmentation definitely gets worse the nearer you are to full (which
> for me is always). I have seen very small files with hundreds of
> fragments as I live at 99% of my space used. They say a hard drive has
> 2 states: new and full :)
>
> It certainly wouldn't hurt to defrag the partial files, though you may
> want to pause your download before doing it (I don't know how much
> locking/blocking may occur on in-use files). Some bittorrent clients
> have an option to write a placeholder file; this is supposed to
> prevent fragmentation since it's allocating the space for the whole
> file immediately. Vuze is what I use, it calls this option "allocate
> and zero new files on creation". The down-side is it could take a
> while to initialize if you're downloading something huge, especially
> if you're saving to a network or USB hard drive that's not very fast.

Is there any tool available to show which files are being written to
any any given time?  iotop is great for watching the I/O rate and
which process is responsible, but sometimes I wonder which files are
being written.  For example, miro is showing a constant 3.5Mbps write
in iotop, and I only have 50kbps downloading and 30kbps uploading.
I'd really like to know what is being written to.

Here's how I'm running shake, please let me know if you would modify
this to work on my noisy drive problem:

shake -vX --new 0 --old 0 --bigsize 0 folder

Does anyone know what these headers indicate (FRAGC and SHOCKED for
example)?  There is no info in man or on the homepage:

IDEAL	START	END	FRAGC	CRUMBC	AGE	SHOCKED	NAME

- Grant

Re: [gentoo-user] Anybody tried shake defragmenter?

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 1719 bytes --]

On Tue, 4 Aug 2009 11:26:26 -0700
Grant <emailgrant@gmail.com> wrote:

> >> Yeah, that's when I'm hearing the HD access I didn't hear before.
> >>  I run miro and it's downloading several torrents all the time.
> >>  It never made a sound before, but now there's a rhythmic grinding
> >> sound when miro is running, maybe because the HD is more full now.
> >>  Could shake help with this?  To find out, should I be running it
> >> on the partially downloaded torrents?
> >
> > Well, bittorent does not download in sequential order, so it is
> > constantly doing random reads and writes. You may not be able to
> > avoid the HD grinding during this kind of activity. Download to a
> > RAM drive or SSD or something perhaps.

Note that this problem can also be (easily?) solved on software level by
pre-allocating files (like "dd if=/dev/zero of=file").

Sure, that won't make writes sequential, but that should guarantee that
resulting file would be as non-fragmented as fs allows at a time of
it's creation.

In fact, rtorrent (and libtorrent) seem to have such a feature, prehaps
other clients should have it somewhere, as well.

http://libtorrent.rakshasa.no/ticket/460


> Is there any tool available to show which files are being written to
> any any given time?  iotop is great for watching the I/O rate and
> which process is responsible, but sometimes I wonder which files are
> being written.  For example, miro is showing a constant 3.5Mbps write
> in iotop, and I only have 50kbps downloading and 30kbps uploading.
> I'd really like to know what is being written to.

Check out sys-fs/inotify-tools (need inotify enabled in kernel).


-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Anybody tried shake defragmenter?

[Stroller]

On 7 Aug 2009, at 04:56, Mike Kazantsev wrote:
> ...
> Note that this problem can also be (easily?) solved on software  
> level by
> pre-allocating files (like "dd if=/dev/zero of=file").
>
> Sure, that won't make writes sequential, but that should guarantee  
> that
> resulting file would be as non-fragmented as fs allows at a time of
> it's creation.
>
> In fact, rtorrent (and libtorrent) seem to have such a feature,  
> prehaps
> other clients should have it somewhere, as well.
>
> http://libtorrent.rakshasa.no/ticket/460

I think this went out of fashion after BitTorrent clients became  
clever / advanced enough to download single files.

I'm old enough to remember the days when opening a torrent would  
download the only the whole thing. If the torrent contained several  
files (mp3s, for instance), of which you wanted only one, then tough  
luck - the client would download random chunks of all the files until  
it had 100% of all of them, and the chances were that the one single  
file you wanted would be incomplete until the whole torrent was at  
least 99% finished (and there was no easy way to tell, anyway; you  
just had to download the whole lot).

Once BitTorrent clients added the feature to select individual files  
for download out of the "compilation", this became quite a popular use  
of them amongst the general public (who are not, as a rule,  
downloading Linux CDs) and led to complaints about all the space being  
"wasted" by preallocation in this way. I gather that many BitTorrent  
users may be interested in only 5% of a typical complete torrent.

I don't use BitTorrent as actively as I used to, but my recollection  
is that NOT pre-allocating the space was a "feature" that was ADDED to  
the more sophisticated clients. Ideally it should indeed be an option,  
but it may not be ubiquitous.

Stroller.

Re: [gentoo-user] Anybody tried shake defragmenter?

[Dan Farrell]

On Mon, 3 Aug 2009 16:48:06 -0700
Grant <emailgrant@gmail.com> wrote:

>  It never
> made a sound before, but now there's a rhythmic grinding sound when
> miro is running, maybe because the HD is more full now. 

In my experience, the rate of change of hard drive access volume is
inversely proportional with the drive's lifetime.  The faster it gets
louder, the sooner it's going to die.  

Time to start planning for replacement. 

Oh, and you're utilizing SMART, right?

Re: [gentoo-user] Anybody tried shake defragmenter?

[Grant]

>>  It never
>> made a sound before, but now there's a rhythmic grinding sound when
>> miro is running, maybe because the HD is more full now.
>
> In my experience, the rate of change of hard drive access volume is
> inversely proportional with the drive's lifetime.  The faster it gets
> louder, the sooner it's going to die.
>
> Time to start planning for replacement.
>
> Oh, and you're utilizing SMART, right?

Should I be doing more than running this test:

smartctl -t long /dev/sda

?

- Grant

Re: [gentoo-user] Anybody tried shake defragmenter?

[Dan Farrell]

On Wed, 5 Aug 2009 07:42:24 -0700
Grant <emailgrant@gmail.com> wrote:

> >>  It never
> >> made a sound before, but now there's a rhythmic grinding sound when
> >> miro is running, maybe because the HD is more full now.
> >
> > In my experience, the rate of change of hard drive access volume is
> > inversely proportional with the drive's lifetime.  The faster it
> > gets louder, the sooner it's going to die.
> >
> > Time to start planning for replacement.
> >
> > Oh, and you're utilizing SMART, right?
> 
> Should I be doing more than running this test:
> 
> smartctl -t long /dev/sda
> 
> ?
> 
> - Grant
> 

If the host can send mail you might want to look into the option of it
mailing you when problems are found.  

Things can go bad quickly....

Other than that though, no, I think you're in good shape.  

Re: [gentoo-user] Anybody tried shake defragmenter?

[Grant]

>> >>  It never
>> >> made a sound before, but now there's a rhythmic grinding sound when
>> >> miro is running, maybe because the HD is more full now.
>> >
>> > In my experience, the rate of change of hard drive access volume is
>> > inversely proportional with the drive's lifetime.  The faster it
>> > gets louder, the sooner it's going to die.
>> >
>> > Time to start planning for replacement.
>> >
>> > Oh, and you're utilizing SMART, right?
>>
>> Should I be doing more than running this test:
>>
>> smartctl -t long /dev/sda
>>
>> ?
>>
>> - Grant
>>
>
> If the host can send mail you might want to look into the option of it
> mailing you when problems are found.
>
> Things can go bad quickly....
>
> Other than that though, no, I think you're in good shape.

Does this indicate everything is OK as far as SMART can tell?

Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     14109         -

- Grant

Re: [gentoo-user] Anybody tried shake defragmenter?

[Alex Schuster]

Grant writes:

> >> > Oh, and you're utilizing SMART, right?
> >>
> >> Should I be doing more than running this test:
> >>
> >> smartctl -t long /dev/sda
[...]
> Does this indicate everything is OK as far as SMART can tell?
>
> Num  Test_Description    Status                  Remaining
> LifeTime(hours)  LBA_of_first_error
> # 1  Extended offline    Completed without error     00%   14109 -    

Looks good. Have a look at the output of 'smartctl -H /dev/sda', too. And 
also of 'smartctl -A /dev/sda', there you may spot things that are wearing 
down, but not failing imminently. The output is a little hard to interpret, 
though.

Here's an article about smartmontools:
http://www.linuxjournal.com:80/article/6983

	Wonko

Re: [gentoo-user] Anybody tried shake defragmenter?

[Grant]

>> >> > Oh, and you're utilizing SMART, right?
>> >>
>> >> Should I be doing more than running this test:
>> >>
>> >> smartctl -t long /dev/sda
> [...]
>> Does this indicate everything is OK as far as SMART can tell?
>>
>> Num  Test_Description    Status                  Remaining
>> LifeTime(hours)  LBA_of_first_error
>> # 1  Extended offline    Completed without error     00%   14109 -
>
> Looks good. Have a look at the output of 'smartctl -H /dev/sda', too. And
> also of 'smartctl -A /dev/sda', there you may spot things that are wearing
> down, but not failing imminently. The output is a little hard to interpret,
> though.
>
> Here's an article about smartmontools:
> http://www.linuxjournal.com:80/article/6983
>
>        Wonko

Thank you for that.  I do get this on one HDD:

SMART overall-health self-assessment test result: PASSED
Please note the following marginal Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
190 Airflow_Temperature_Cel 0x0022   066   035   045    Old_age
Always   In_the_past 34 (Lifetime Min/Max 20/38)

But based on the info here:

http://forum.synology.com/enu/viewtopic.php?f=117&t=9806&start=15

it doesn't sound like a big deal.  That HDD was previously in another
system which I think had a temperature problem.

- Grant

Re: [gentoo-user] Anybody tried shake defragmenter?

[meino.cramer]

Paul Hartman <paul.hartman+gentoo@gmail.com> [09-08-03 23:09]:
> On Mon, Aug 3, 2009 at 3:22 PM, Grant<emailgrant@gmail.com> wrote:
> > I know Linux systems aren't supposed to become fragmented, but I've
> > also read that it can happen eventually.  I'm on ext3.  I've read that
> > ext4 will have a defragmenter but that it doesn't have one yet.
> 
> It's not that they aren't supposed to become fragmented, it is that
> they try to avoid it. There is a big difference, and things like
> streaming writes (downloads, bittorrents, etc) can cause extreme
> fragmentation.
> 
> The time-honored way of fixing this is "backup, delete, restore". In
> my case my simple defragmenter is to move a file to tmpfs and then
> move it back to the hard drive. I always do this to files I'm about to
> burn to a CD/DVD to ensure the read speed is optimal.
> 
> > Has anyone tried the shake defragmenter?
> 
> Yes, nothing has blown up yet. :)

Hi,

 I have several encfs-encrypted partions. As fas as I had understood
 encfs, only the contents of the data file and not their
 organisational data are encrypted (?).
 But I may be wrong...

 So, do I any harm to shake those partions without mounting them in
 beforehand?

 Kind regards,
 Meino Cramer

 PS: How can I make a mount -o remount,user_xattr work?
     Do I have to re-mkfs the partions (please not..) ?



-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] Anybody tried shake defragmenter?

[Grant]

>> > I know Linux systems aren't supposed to become fragmented, but I've
>> > also read that it can happen eventually.  I'm on ext3.  I've read that
>> > ext4 will have a defragmenter but that it doesn't have one yet.
>>
>> It's not that they aren't supposed to become fragmented, it is that
>> they try to avoid it. There is a big difference, and things like
>> streaming writes (downloads, bittorrents, etc) can cause extreme
>> fragmentation.
>>
>> The time-honored way of fixing this is "backup, delete, restore". In
>> my case my simple defragmenter is to move a file to tmpfs and then
>> move it back to the hard drive. I always do this to files I'm about to
>> burn to a CD/DVD to ensure the read speed is optimal.
>>
>> > Has anyone tried the shake defragmenter?
>>
>> Yes, nothing has blown up yet. :)
>
> Hi,
>
>  I have several encfs-encrypted partions. As fas as I had understood
>  encfs, only the contents of the data file and not their
>  organisational data are encrypted (?).
>  But I may be wrong...
>
>  So, do I any harm to shake those partions without mounting them in
>  beforehand?
>
>  Kind regards,
>  Meino Cramer
>
>  PS: How can I make a mount -o remount,user_xattr work?
>     Do I have to re-mkfs the partions (please not..) ?

You can use -X with shake to skip the xattr stuff.

- Grant

Re: [gentoo-user] Anybody tried shake defragmenter?

[meino.cramer]

Grant <emailgrant@gmail.com> [09-08-07 17:40]:
> >> > I know Linux systems aren't supposed to become fragmented, but I've
> >> > also read that it can happen eventually.  I'm on ext3.  I've read that
> >> > ext4 will have a defragmenter but that it doesn't have one yet.
> >>
> >> It's not that they aren't supposed to become fragmented, it is that
> >> they try to avoid it. There is a big difference, and things like
> >> streaming writes (downloads, bittorrents, etc) can cause extreme
> >> fragmentation.
> >>
> >> The time-honored way of fixing this is "backup, delete, restore". In
> >> my case my simple defragmenter is to move a file to tmpfs and then
> >> move it back to the hard drive. I always do this to files I'm about to
> >> burn to a CD/DVD to ensure the read speed is optimal.
> >>
> >> > Has anyone tried the shake defragmenter?
> >>
> >> Yes, nothing has blown up yet. :)
> >
> > Hi,
> >
> >  I have several encfs-encrypted partions. As fas as I had understood
> >  encfs, only the contents of the data file and not their
> >  organisational data are encrypted (?).
> >  But I may be wrong...
> >
> >  So, do I any harm to shake those partions without mounting them in
> >  beforehand?
> >
> >  Kind regards,
> >  Meino Cramer
> >
> >  PS: How can I make a mount -o remount,user_xattr work?
> >     Do I have to re-mkfs the partions (please not..) ?
> 
> You can use -X with shake to skip the xattr stuff.
> 
> - Grant

Hi Grant,

 thank you very much for your. 

  I have several encfs-encrypted partions. As fas as I had understood
  encfs, only the contents of the data file and not their
  organisational data are encrypted (?).
  But I may be wrong...

  So, do I any harm to shake those partions without mounting them in
  beforehand?

  How can I make a mount -o remount,user_xattr work?
  Do I have to re-mkfs the partions (please not..) ?

  Thank your very much in advance for your help!

  Kind regards,
  Meino Cramer


-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] Anybody tried shake defragmenter?

[meino.cramer]

Paul Hartman <paul.hartman+gentoo@gmail.com> [09-08-03 23:09]:
> On Mon, Aug 3, 2009 at 3:22 PM, Grant<emailgrant@gmail.com> wrote:
> > I know Linux systems aren't supposed to become fragmented, but I've
> > also read that it can happen eventually.  I'm on ext3.  I've read that
> > ext4 will have a defragmenter but that it doesn't have one yet.
> 
> It's not that they aren't supposed to become fragmented, it is that
> they try to avoid it. There is a big difference, and things like
> streaming writes (downloads, bittorrents, etc) can cause extreme
> fragmentation.
> 
> The time-honored way of fixing this is "backup, delete, restore". In
> my case my simple defragmenter is to move a file to tmpfs and then
> move it back to the hard drive. I always do this to files I'm about to
> burn to a CD/DVD to ensure the read speed is optimal.
> 
> > Has anyone tried the shake defragmenter?
> 
> Yes, nothing has blown up yet. :)

Hi,

does anyone know a source of information -- except reading the C-source
of shake itsself -- what the meaning of the different columns of 

    shake -pvv <dir>

are ?

Thank you very much in advance for any help!

Have a nice weekend!
mcc




-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] Anybody tried shake defragmenter?

[Paul Hartman]

On Sat, Aug 8, 2009 at 12:40 AM, <meino.cramer@gmx.de> wrote:
> Paul Hartman <paul.hartman+gentoo@gmail.com> [09-08-03 23:09]:
>> On Mon, Aug 3, 2009 at 3:22 PM, Grant<emailgrant@gmail.com> wrote:
>> > I know Linux systems aren't supposed to become fragmented, but I've
>> > also read that it can happen eventually.  I'm on ext3.  I've read that
>> > ext4 will have a defragmenter but that it doesn't have one yet.
>>
>> It's not that they aren't supposed to become fragmented, it is that
>> they try to avoid it. There is a big difference, and things like
>> streaming writes (downloads, bittorrents, etc) can cause extreme
>> fragmentation.
>>
>> The time-honored way of fixing this is "backup, delete, restore". In
>> my case my simple defragmenter is to move a file to tmpfs and then
>> move it back to the hard drive. I always do this to files I'm about to
>> burn to a CD/DVD to ensure the read speed is optimal.
>>
>> > Has anyone tried the shake defragmenter?
>>
>> Yes, nothing has blown up yet. :)
>
> Hi,
>
> does anyone know a source of information -- except reading the C-source
> of shake itsself -- what the meaning of the different columns of
>
>    shake -pvv <dir>
>
> are ?

No. :) There's no documentation really. The source cide is funny,
everything is named after law, investigations, accused, trials and
judgments. :)  There best I can do is reproduce the part of the source
that shows this info and hope you can infer from the names what they
are showing:

/* Show statistics about an accused */
void
show_reg (struct accused *a, struct law *l)
{
  /* Show file status */
  printf ("%lli\t%lli\t%lli\t%i\t%i\t%i\t%i\t%s",
          a->ideal, a->start / 1024, a->end / 1024, a->fragc, a->crumbc,
          (int) (a->age / 3600 / 24), a->guilty, a->name);
  /* And, eventualy, list of frags and crumbs */
  if (l->verbosity > 2 && a->poslog && a->poslog[0] != -1)
    {
      uint n;
      putchar ('\t');
      for (n = 0; a->sizelog[n + 1] != -1; n++)
        printf ("%lli:%lli,", a->poslog[n] / 1024, a->sizelog[n] / 1024);
      printf ("%lli:%lli\n", a->poslog[n] / 1024, a->sizelog[n] / 1024);
    }
  else
    putchar ('\n');
}

Re: [gentoo-user] Anybody tried shake defragmenter?

[Dale]

Paul Hartman wrote:
>
> No. :) There's no documentation really. The source cide is funny,
> everything is named after law, investigations, accused, trials and
> judgments. :)  There best I can do is reproduce the part of the source
> that shows this info and hope you can infer from the names what they
> are showing:
>
> /* Show statistics about an accused */
> void
> show_reg (struct accused *a, struct law *l)
> {
>   /* Show file status */
>   printf ("%lli\t%lli\t%lli\t%i\t%i\t%i\t%i\t%s",
>           a->ideal, a->start / 1024, a->end / 1024, a->fragc, a->crumbc,
>           (int) (a->age / 3600 / 24), a->guilty, a->name);
>   /* And, eventualy, list of frags and crumbs */
>   if (l->verbosity > 2 && a->poslog && a->poslog[0] != -1)
>     {
>       uint n;
>       putchar ('\t');
>       for (n = 0; a->sizelog[n + 1] != -1; n++)
>         printf ("%lli:%lli,", a->poslog[n] / 1024, a->sizelog[n] / 1024);
>       printf ("%lli:%lli\n", a->poslog[n] / 1024, a->sizelog[n] / 1024);
>     }
>   else
>     putchar ('\n');
> }
>
>
>   

Does this make sense to anyone?  It looks like it got worse instead of
better.

root@smoker / # /root/fragck.pl /home/
9.8476210220794% non contiguous files, 1.989530423966 average fragments.
root@smoker / # shake --old=0 -X /home/
root@smoker / # /root/fragck.pl /home/
14.6129132552596% non contiguous files, 1.65074100943103 average fragments.
root@smoker / # /root/fragck.pl /usr/portage/distfiles/
24.3989314336598% non contiguous files, 9.93054318788958 average fragments.
root@smoker / # shake --old=0 -X /usr/portage/distfiles/
root@smoker / # /root/fragck.pl /usr/portage/distfiles/
38.646482635797% non contiguous files, 10.9777382012467 average fragments.
root@smoker / #

Am I reading this wrong or something?

Dale

:-)  :-)