[gentoo-user] OT:Choosing a filesystem

[gentoo-user] OT:Choosing a filesystem

[meino.cramer]

Hi,

I googled down some - often fairly outdated - texts about "the 
best filesystem" fpr a Linux box. Other texts focussed on 
uses, which do not aplly to me: Fileservers, webservers, database
machines  etc.

Wnat I want is a fast and stable (!) filesystem for a desktop PC 
with one 1TByte harddisk. Since using Gentoo and a lot of sources
I do compile very often "bigger things" (blender-2.50 for example).
Another thing: Due to my experimenting it is possible that I have to
reboot "hard", which means, the filesystem will be unmounted not
cleanly ("dirty" do to say...;) The choosen filesystem should be
good in recovering such thing.

I am currently using a vanilla 2.6.32.10 kernel.

The question, what remains is: What choose should I make?

I thank you very much in advance for any help! 
Best regards,
mcc



-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] OT:Choosing a filesystem

[Michele Alzetta]

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

I have been using reiserfs ( 3, not 4 ) for several years and have found it
to recover without any problems from "dirty" shutdowns, at most I've had to
use reiserfstools to fix the filesystem. I had no such luck with ext3,
although as a journalled filesystem in theory it should do the same. I have
never tried other journalled filesystems, so I can't give you my opinion.

HTH

-- 
Michele

[-- Attachment #2: Type: text/html, Size: 431 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[Bartosz Szatkowski]

Its kind of tricky question :) and if You look closely You could find
som flames about it :P

Iam using ReiserFS for my root and xfs for the rest and testing btrfs.
I never gets any problem with "broken partition table" etc. (and i
experienced several "quick power downs").

But reiserfs have some problems with bkl etc and i am not sure if its
still "main line active" because of developing raiser4. In some reviews
You can find that its eats more cpu time too.

btrfs could be very nice but its not stable yet (or maybe not for real
use :P)

Maybe someone could share experience about ext4 - iam using it on my g1
but its hard to say anything ...

-- 
Pozdrawiam,
Bartosz Szatkowski

You must exorcise any evil proprietary operating systems that possess
any of the computers under your control, and then install a wholly/holy
free operating system, and then only install Free Software on top of
that.

Re: [gentoo-user] OT:Choosing a filesystem

[Dale]

meino.cramer@gmx.de wrote:
> Hi,
>
> I googled down some - often fairly outdated - texts about "the
> best filesystem" fpr a Linux box. Other texts focussed on
> uses, which do not aplly to me: Fileservers, webservers, database
> machines  etc.
>
> Wnat I want is a fast and stable (!) filesystem for a desktop PC
> with one 1TByte harddisk. Since using Gentoo and a lot of sources
> I do compile very often "bigger things" (blender-2.50 for example).
> Another thing: Due to my experimenting it is possible that I have to
> reboot "hard", which means, the filesystem will be unmounted not
> cleanly ("dirty" do to say...;) The choosen filesystem should be
> good in recovering such thing.
>
> I am currently using a vanilla 2.6.32.10 kernel.
>
> The question, what remains is: What choose should I make?
>
> I thank you very much in advance for any help!
> Best regards,
> mcc
>
>    

I notice you have a fairly large drive.  You may want to indicate 
whether or not you will be putting large files on it or what not.  If 
you are, some files systems work better with large files.  That said, if 
you plan to have a lot of small files, then another file system may work 
better.  If you plan to have a mix, then it could get interesting.  ;-)

I use reiserfs myself and have had no problems, even with a hard 
shutdown or some other failure.  Thing is, most file systems are good 
but it depends on what you will be putting on it.  From my experience, 
don't use XFS unless you have a UPS and will not be having to pull the 
plug.  I tried XFS a while back and each time there was a hard shutdown, 
I had to reinstall.  It was running Mandriva so I didn't know how to 
recover with it and no other bootable CD either.   XFS has its good 
points but surviving a power plug pull is not one of them.

I will also say this, it is a good idea to ask first on this.  There are 
a lot of good file systems out there and each one has its strong 
points.  It's best to get the right one first rather than to have to 
redo things later on.

Dale

:-)  :-)

Re: [gentoo-user] OT:Choosing a filesystem

[meino.cramer]

Dale <rdalek1967@gmail.com> [10-04-01 20:36]:
> meino.cramer@gmx.de wrote:
> >Hi,
> >
> >I googled down some - often fairly outdated - texts about "the
> >best filesystem" fpr a Linux box. Other texts focussed on
> >uses, which do not aplly to me: Fileservers, webservers, database
> >machines  etc.
> >
> >Wnat I want is a fast and stable (!) filesystem for a desktop PC
> >with one 1TByte harddisk. Since using Gentoo and a lot of sources
> >I do compile very often "bigger things" (blender-2.50 for example).
> >Another thing: Due to my experimenting it is possible that I have to
> >reboot "hard", which means, the filesystem will be unmounted not
> >cleanly ("dirty" do to say...;) The choosen filesystem should be
> >good in recovering such thing.
> >
> >I am currently using a vanilla 2.6.32.10 kernel.
> >
> >The question, what remains is: What choose should I make?
> >
> >I thank you very much in advance for any help!
> >Best regards,
> >mcc
> >
> >   
> 
> I notice you have a fairly large drive.  You may want to indicate 
> whether or not you will be putting large files on it or what not.  If 
> you are, some files systems work better with large files.  That said, 
> if you plan to have a lot of small files, then another file system may 
> work better.  If you plan to have a mix, then it could get interesting. 
>  ;-)
> 
> I use reiserfs myself and have had no problems, even with a hard 
> shutdown or some other failure.  Thing is, most file systems are good 
> but it depends on what you will be putting on it.  From my experience, 
> don't use XFS unless you have a UPS and will not be having to pull the 
> plug.  I tried XFS a while back and each time there was a hard 
> shutdown, I had to reinstall.  It was running Mandriva so I didn't know 
> how to recover with it and no other bootable CD either.   XFS has its 
> good points but surviving a power plug pull is not one of them.
> 
> I will also say this, it is a good idea to ask first on this.  There 
> are a lot of good file systems out there and each one has its strong 
> points.  It's best to get the right one first rather than to have to 
> redo things later on.
> 
> Dale
> 
> :-)  :-)
> 

Hi,

thanks for all the input to all who have answered ! :)

I will try to characterize ("characterise" ?) what I plan to do with
my TByte disk.
My current drive is 200GByte and it becomes too small...
I DONT LIKE (read: hate) to put CDs or DVDs into my drive, to mount
it only to get access to documentations. CDs and DVDs as storage media
in the sense of "backup" is ok, but (at least for me) as a replacement
or extension to the harddisk it is much to slow (at least for me).

So I have a lot of docs (specs of microcontrollers, howtos, programm
and source code docs...etc) on my disk.
This one part.

Then: I often transer videos from my DVB-T-receiver/recorder to my
harddisk to cut out the advertising and to transcode the videos to
somethings better than "ts" (transport streams),
This is another part.

Then I plan to have two roots this time: One to experiment with and
one "good and stable"-version which is used/updated/... "strictly as
recommended". Filesizes and usage do vary here...take a look at your
own roots ;)))

Then I want something encrypted, either as a partition or as a files
(carrying a encrypted fs), which I can copy to dvd and will be able 
to mount this dvd and use it without to have to copy the whole dvd
first to harddisk before using it...
Currently I am using encfs...(outdated?). What can I do use instead?
This is for personal things like letters, photos, texts ... etc.
Files vary from some kb up to about 2GByte (guessed). Most of them 
smaller than 200MByte

Last thing: I have a lot iof copies of code from svn repositories because
I like to have the "bleeding edge" of some projects (do you know the
new Blender 2.50??? :O)

This implies a lot of compile work. This will be the only case where
files are created as often as read.

Most files will be far more read than written...

I have not planned a webserver, fileserve, extensive database usage (ok
emerge and helpers a little of database usage...), experimental file
creation and deletion...etcpp

I would say...maximium file size will be around 4GB for all of that
above, since everything above that I cannot backup onto DVDRWs....

May be this will give you a little "look inside my harddisk" ;)

Any recommendations?

keep hacking! ;)
mcc



-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] OT:Choosing a filesystem

[Dale]

meino.cramer@gmx.de wrote:
> Hi,
>
> thanks for all the input to all who have answered ! :)
>
> I will try to characterize ("characterise" ?) what I plan to do with
> my TByte disk.
> My current drive is 200GByte and it becomes too small...
> I DONT LIKE (read: hate) to put CDs or DVDs into my drive, to mount
> it only to get access to documentations. CDs and DVDs as storage media
> in the sense of "backup" is ok, but (at least for me) as a replacement
> or extension to the harddisk it is much to slow (at least for me).
>
> So I have a lot of docs (specs of microcontrollers, howtos, programm
> and source code docs...etc) on my disk.
> This one part.
>
> Then: I often transer videos from my DVB-T-receiver/recorder to my
> harddisk to cut out the advertising and to transcode the videos to
> somethings better than "ts" (transport streams),
> This is another part.
>
> Then I plan to have two roots this time: One to experiment with and
> one "good and stable"-version which is used/updated/... "strictly as
> recommended". Filesizes and usage do vary here...take a look at your
> own roots ;)))
>
> Then I want something encrypted, either as a partition or as a files
> (carrying a encrypted fs), which I can copy to dvd and will be able
> to mount this dvd and use it without to have to copy the whole dvd
> first to harddisk before using it...
> Currently I am using encfs...(outdated?). What can I do use instead?
> This is for personal things like letters, photos, texts ... etc.
> Files vary from some kb up to about 2GByte (guessed). Most of them
> smaller than 200MByte
>
> Last thing: I have a lot iof copies of code from svn repositories because
> I like to have the "bleeding edge" of some projects (do you know the
> new Blender 2.50??? :O)
>
> This implies a lot of compile work. This will be the only case where
> files are created as often as read.
>
> Most files will be far more read than written...
>
> I have not planned a webserver, fileserve, extensive database usage (ok
> emerge and helpers a little of database usage...), experimental file
> creation and deletion...etcpp
>
> I would say...maximium file size will be around 4GB for all of that
> above, since everything above that I cannot backup onto DVDRWs....
>
> May be this will give you a little "look inside my harddisk" ;)
>
> Any recommendations?
>
> keep hacking! ;)
> mcc
>
>    

I'm no file system guru but that will help inform people on what you 
will be using it for.  The people on this list that do use all sorts of 
different file systems can now offer better advice on what might best 
suite you.  Someone a good while back had a huge video or something that 
was causing trouble and if I recall correctly it was because of some 
file system limitation or something to that effect.

Give the thread a day or so so that others can chime in with advice.  
Some people are in different time zones, some answer at home, some at 
work etc etc so it takes a bit to let the gurus catch up.

Dale

:-)  :-)

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:

> So I have a lot of docs (specs of microcontrollers, howtos, programm
> and source code docs...etc) on my disk.
> This one part.

Those are fairly normal files.

> Then: I often transer videos from my DVB-T-receiver/recorder to my
> harddisk to cut out the advertising and to transcode the videos to
> somethings better than "ts" (transport streams),

These tend to be bigger, often in the GB range, so I'd use a separate
filesystem for them with XFS, which handles large files better in my
experience.

> Then I want something encrypted, either as a partition or as a files
> (carrying a encrypted fs), which I can copy to dvd and will be able 
> to mount this dvd and use it without to have to copy the whole dvd
> first to harddisk before using it...
> Currently I am using encfs...(outdated?). What can I do use instead?

ecryptfs does much the same job as encfs but is in the kernel.

I'd say something like reiser3 for most areas and an XFS filesystem for
the videos would be a good starting point. I would strongly recommend you
use LVM and only set up volumes for what you need. That gives you extra
space to play with and even experiment with different filesystems to see
which work for you.


-- 
Neil Bothwick

The facts, although interesting, are irrelevant.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[meino.cramer]

Neil Bothwick <neil@digimed.co.uk> [10-04-02 10:52]:
> On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:
> 
> > So I have a lot of docs (specs of microcontrollers, howtos, programm
> > and source code docs...etc) on my disk.
> > This one part.
> 
> Those are fairly normal files.
> 
> > Then: I often transer videos from my DVB-T-receiver/recorder to my
> > harddisk to cut out the advertising and to transcode the videos to
> > somethings better than "ts" (transport streams),
> 
> These tend to be bigger, often in the GB range, so I'd use a separate
> filesystem for them with XFS, which handles large files better in my
> experience.
> 
> > Then I want something encrypted, either as a partition or as a files
> > (carrying a encrypted fs), which I can copy to dvd and will be able 
> > to mount this dvd and use it without to have to copy the whole dvd
> > first to harddisk before using it...
> > Currently I am using encfs...(outdated?). What can I do use instead?
> 
> ecryptfs does much the same job as encfs but is in the kernel.
> 
> I'd say something like reiser3 for most areas and an XFS filesystem for
> the videos would be a good starting point. I would strongly recommend you
> use LVM and only set up volumes for what you need. That gives you extra
> space to play with and even experiment with different filesystems to see
> which work for you.
> 
> 
> -- 
> Neil Bothwick
> 
> The facts, although interesting, are irrelevant.

Hi Neil,

Thank you for your help! :)

A question to LVM: As much as I know, LVM combines several partition
to one big partition, and if one partition fails, at least other
others of that volume are damaged, too.
What is the advantage of using LVM and several small partitions
instead of one in the size of the sum of the others and not using
LVM?

Best regards,
mcc




-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] OT:Choosing a filesystem

[William Kenworthy]

On Fri, 2010-04-02 at 11:11 +0200, meino.cramer@gmx.de wrote:
> Neil Bothwick <neil@digimed.co.uk> [10-04-02 10:52]:
> > On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:
> > 
> > > So I have a lot of docs (specs of microcontrollers, howtos, programm
> > > and source code docs...etc) on my disk.
> > > This one part.
> > 
> > Those are fairly normal files.
> > 
> > > Then: I often transer videos from my DVB-T-receiver/recorder to my
> > > harddisk to cut out the advertising and to transcode the videos to
> > > somethings better than "ts" (transport streams),
> > 
> > These tend to be bigger, often in the GB range, so I'd use a separate
> > filesystem for them with XFS, which handles large files better in my
> > experience.
> > 
> > > Then I want something encrypted, either as a partition or as a files
> > > (carrying a encrypted fs), which I can copy to dvd and will be able 
> > > to mount this dvd and use it without to have to copy the whole dvd
> > > first to harddisk before using it...
> > > Currently I am using encfs...(outdated?). What can I do use instead?
> > 
> > ecryptfs does much the same job as encfs but is in the kernel.
> > 
> > I'd say something like reiser3 for most areas and an XFS filesystem for
> > the videos would be a good starting point. I would strongly recommend you
> > use LVM and only set up volumes for what you need. That gives you extra
> > space to play with and even experiment with different filesystems to see
> > which work for you.
> > 
> > 
> > -- 
> > Neil Bothwick
> > 
> > The facts, although interesting, are irrelevant.
> 
> Hi Neil,
> 
> Thank you for your help! :)
> 
> A question to LVM: As much as I know, LVM combines several partition
> to one big partition, and if one partition fails, at least other
> others of that volume are damaged, too.
> What is the advantage of using LVM and several small partitions
> instead of one in the size of the sum of the others and not using
> LVM?
> 
> Best regards,
> mcc
> 
The advantage is flexibility - you absolutely love LVM when you discover
you have made a file system too small!  Shrinking/enlarging/adding more
storage etc is a real bonus.

Downside as you mention is lose one disk and you may lose all - however
I believe that sometimes the remaining data can be recovered.

Also keep in mind that while small partitions can be a pain and waste
space, normal corruption is limited to one partition, and physical data
protection is better (i.e., when one partition fills up, others are
safe)

BillK

Re: [gentoo-user] OT:Choosing a filesystem

[meino.cramer]

William Kenworthy <billk@iinet.net.au> [10-04-02 11:32]:
> On Fri, 2010-04-02 at 11:11 +0200, meino.cramer@gmx.de wrote:
> > Neil Bothwick <neil@digimed.co.uk> [10-04-02 10:52]:
> > > On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:
> > > 
> > > > So I have a lot of docs (specs of microcontrollers, howtos, programm
> > > > and source code docs...etc) on my disk.
> > > > This one part.
> > > 
> > > Those are fairly normal files.
> > > 
> > > > Then: I often transer videos from my DVB-T-receiver/recorder to my
> > > > harddisk to cut out the advertising and to transcode the videos to
> > > > somethings better than "ts" (transport streams),
> > > 
> > > These tend to be bigger, often in the GB range, so I'd use a separate
> > > filesystem for them with XFS, which handles large files better in my
> > > experience.
> > > 
> > > > Then I want something encrypted, either as a partition or as a files
> > > > (carrying a encrypted fs), which I can copy to dvd and will be able 
> > > > to mount this dvd and use it without to have to copy the whole dvd
> > > > first to harddisk before using it...
> > > > Currently I am using encfs...(outdated?). What can I do use instead?
> > > 
> > > ecryptfs does much the same job as encfs but is in the kernel.
> > > 
> > > I'd say something like reiser3 for most areas and an XFS filesystem for
> > > the videos would be a good starting point. I would strongly recommend you
> > > use LVM and only set up volumes for what you need. That gives you extra
> > > space to play with and even experiment with different filesystems to see
> > > which work for you.
> > > 
> > > 
> > > -- 
> > > Neil Bothwick
> > > 
> > > The facts, although interesting, are irrelevant.
> > 
> > Hi Neil,
> > 
> > Thank you for your help! :)
> > 
> > A question to LVM: As much as I know, LVM combines several partition
> > to one big partition, and if one partition fails, at least other
> > others of that volume are damaged, too.
> > What is the advantage of using LVM and several small partitions
> > instead of one in the size of the sum of the others and not using
> > LVM?
> > 
> > Best regards,
> > mcc
> > 
> The advantage is flexibility - you absolutely love LVM when you discover
> you have made a file system too small!  Shrinking/enlarging/adding more
> storage etc is a real bonus.
> 
> Downside as you mention is lose one disk and you may lose all - however
> I believe that sometimes the remaining data can be recovered.
> 
> Also keep in mind that while small partitions can be a pain and waste
> space, normal corruption is limited to one partition, and physical data
> protection is better (i.e., when one partition fills up, others are
> safe)
> 
> BillK
> 
> 
> 

Hi Bill,

tahnks for your reply! :)
Seems that that, what I thought to have remembered of LVM seems to be
still correct. 

mcc

-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

On Fri, 2 Apr 2010 11:11:30 +0200, meino.cramer@gmx.de wrote:

> A question to LVM: As much as I know, LVM combines several partition
> to one big partition, and if one partition fails, at least other
> others of that volume are damaged, too.

It can be used that way, but you have only one disk, so you would create
a single physical volume from a large partition on that disk and then use
LVM to create individual logical volumes within it.

> What is the advantage of using LVM and several small partitions
> instead of one in the size of the sum of the others and not using
> LVM?

Flexibility and convenience. No single filesystem is right for all of
your needs, with LVM you can use XFS where it is best suited and
something else elsewhere, and you can resize and reorganise your volumes
without needing to repartition the drive. I have a few hundred GB unused
on my volume group, so I can add volumes or resize existing ones in
seconds with minimal effort and no downtime.

Just one note of caution, XFS filesystems cannot be shrunk, although they
are easy to grow, so make any XFS volumes no larger than your current
needs. That advice applies to all your volumes, because growing is easier
and faster than shrinking, but doubly so to XFS.


-- 
Neil Bothwick

Better to understand a little than to misunderstand a lot.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[meino.cramer]

Neil Bothwick <neil@digimed.co.uk> [10-04-02 12:48]:
> On Fri, 2 Apr 2010 11:11:30 +0200, meino.cramer@gmx.de wrote:
> 
> > A question to LVM: As much as I know, LVM combines several partition
> > to one big partition, and if one partition fails, at least other
> > others of that volume are damaged, too.
> 
> It can be used that way, but you have only one disk, so you would create
> a single physical volume from a large partition on that disk and then use
> LVM to create individual logical volumes within it.
> 
> > What is the advantage of using LVM and several small partitions
> > instead of one in the size of the sum of the others and not using
> > LVM?
> 
> Flexibility and convenience. No single filesystem is right for all of
> your needs, with LVM you can use XFS where it is best suited and
> something else elsewhere, and you can resize and reorganise your volumes
> without needing to repartition the drive. I have a few hundred GB unused
> on my volume group, so I can add volumes or resize existing ones in
> seconds with minimal effort and no downtime.
> 
> Just one note of caution, XFS filesystems cannot be shrunk, although they
> are easy to grow, so make any XFS volumes no larger than your current
> needs. That advice applies to all your volumes, because growing is easier
> and faster than shrinking, but doubly so to XFS.
> 
> 
> -- 
> Neil Bothwick
> 
> Better to understand a little than to misunderstand a lot.

Hi Neil,

only to be sure to have understood everything correctly:
Suggestion is to create for example one root partition and a swap
partion. And I will create on big "rest of the disk"-partition.
The last one will be subdivided with LVM into portions as needed.

Since the last big partition is big due to physical reasons (not for logical one):
What will happen, if -- for example -- one portion will be not unmounted cleanly
and while booting/checking fails to recover? Are all others
damaged/lost?

Best regards,
mcc



-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

On Fri, 2 Apr 2010 13:04:53 +0200, meino.cramer@gmx.de wrote:

> only to be sure to have understood everything correctly:
> Suggestion is to create for example one root partition and a swap
> partion. And I will create on big "rest of the disk"-partition.
> The last one will be subdivided with LVM into portions as needed.

Yes.

> Since the last big partition is big due to physical reasons (not for
> logical one): What will happen, if -- for example -- one portion will
> be not unmounted cleanly and while booting/checking fails to recover?
> Are all others damaged/lost?

No, because the failure you describe is at the filesystem level. Even the
volume containing that filesystem will retain integrity, only the
filesystem itself will be corrupted. As you have left free space on the
volume group, you can just create a new volume, format it and copy over
everything you can recover from the broken filesystem before deleting it.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[meino.cramer]

Neil Bothwick <neil@digimed.co.uk> [10-04-02 14:08]:
> On Fri, 2 Apr 2010 13:04:53 +0200, meino.cramer@gmx.de wrote:
> 
> > only to be sure to have understood everything correctly:
> > Suggestion is to create for example one root partition and a swap
> > partion. And I will create on big "rest of the disk"-partition.
> > The last one will be subdivided with LVM into portions as needed.
> 
> Yes.
> 
> > Since the last big partition is big due to physical reasons (not for
> > logical one): What will happen, if -- for example -- one portion will
> > be not unmounted cleanly and while booting/checking fails to recover?
> > Are all others damaged/lost?
> 
> No, because the failure you describe is at the filesystem level. Even the
> volume containing that filesystem will retain integrity, only the
> filesystem itself will be corrupted. As you have left free space on the
> volume group, you can just create a new volume, format it and copy over
> everything you can recover from the broken filesystem before deleting it.
> 

Hi Neil,

yes, sounds good, very good.
Last question: How heavy is the performance impact of such a setup ?





-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] OT:Choosing a filesystem

[Volker Armin Hemmann]

On Freitag 02 April 2010, meino.cramer@gmx.de wrote:
> Neil Bothwick <neil@digimed.co.uk> [10-04-02 14:08]:
> > On Fri, 2 Apr 2010 13:04:53 +0200, meino.cramer@gmx.de wrote:
> > > only to be sure to have understood everything correctly:
> > > Suggestion is to create for example one root partition and a swap
> > > partion. And I will create on big "rest of the disk"-partition.
> > > The last one will be subdivided with LVM into portions as needed.
> > 
> > Yes.
> > 
> > > Since the last big partition is big due to physical reasons (not for
> > > logical one): What will happen, if -- for example -- one portion will
> > > be not unmounted cleanly and while booting/checking fails to recover?
> > > Are all others damaged/lost?
> > 
> > No, because the failure you describe is at the filesystem level. Even the
> > volume containing that filesystem will retain integrity, only the
> > filesystem itself will be corrupted. As you have left free space on the
> > volume group, you can just create a new volume, format it and copy over
> > everything you can recover from the broken filesystem before deleting it.
> 
> Hi Neil,
> 
> yes, sounds good, very good.
> Last question: How heavy is the performance impact of such a setup ?

seriously lvm sounds nice. But it isn't. It easily breaks.

You want a save setup? Go raid5 or raid6. As a bonus - you can get more space 
if you need it by just adding another disk. And you are not depending on some 
complex stuff to get it working.

Re: [gentoo-user] OT:Choosing a filesystem

[William Kenworthy]

On Fri, 2010-04-02 at 14:45 +0200, Volker Armin Hemmann wrote:
> On Freitag 02 April 2010, meino.cramer@gmx.de wrote:
> > Neil Bothwick <neil@digimed.co.uk> [10-04-02 14:08]:
> > > On Fri, 2 Apr 2010 13:04:53 +0200, meino.cramer@gmx.de wrote:
> > > > only to be sure to have understood everything correctly:
> > > > Suggestion is to create for example one root partition and a swap
> > > > partion. And I will create on big "rest of the disk"-partition.
> > > > The last one will be subdivided with LVM into portions as needed.
> > > 
> > > Yes.
> > > 
> > > > Since the last big partition is big due to physical reasons (not for
> > > > logical one): What will happen, if -- for example -- one portion will
> > > > be not unmounted cleanly and while booting/checking fails to recover?
> > > > Are all others damaged/lost?
> > > 
> > > No, because the failure you describe is at the filesystem level. Even the
> > > volume containing that filesystem will retain integrity, only the
> > > filesystem itself will be corrupted. As you have left free space on the
> > > volume group, you can just create a new volume, format it and copy over
> > > everything you can recover from the broken filesystem before deleting it.
> > 
> > Hi Neil,
> > 
> > yes, sounds good, very good.
> > Last question: How heavy is the performance impact of such a setup ?
> 
> seriously lvm sounds nice. But it isn't. It easily breaks.
> 
> You want a save setup? Go raid5 or raid6. As a bonus - you can get more space 
> if you need it by just adding another disk. And you are not depending on some 
> complex stuff to get it working.
> 

My experience is lvm itself is quite robust and very low impact on
performance.  More reliable than linux software raid at least (well the
raid 0 that I was using: ) - never had a problem I could trace to lvm.

The only thing thats affected lvm for me were hardware errors (disk
died).

My experience was with raid 0, while the higher raid redundancy will
shift the reliability figures back the other way.

Its really down to space and management or losing space to redundancy.
Yes its an extra layer on top of the raw hardware (but so is raid
really) so its the flexibility thats important.

BillK

Re: [gentoo-user] OT:Choosing a filesystem

[Volker Armin Hemmann]

On Freitag 02 April 2010, William Kenworthy wrote:

> My experience was with raid 0, while the higher raid redundancy will
> shift the reliability figures back the other way.

wrong. Raid0 is meant for 0 redudancy and reduced reliability for more 
performance.

Before you start talking about Raid and redundandy you should read about raid 
levels and what they mean first.

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Fri, 2 Apr 2010 14:45:29 +0200, Volker Armin Hemmann wrote:

> seriously lvm sounds nice. But it isn't. It easily breaks.

Do you have something to back that up?

> You want a save setup? Go raid5 or raid6. As a bonus - you can get more
> space if you need it by just adding another disk. And you are not
> depending on some complex stuff to get it working.

LVM and RAID are completely different animals. No one suggested using it
for any reasons of data security, running LVM on a RAID array gives both
security and flexibility. As for being able to add space to RAID, you
can't temporarily add a new volume whenever you want, you have to go out
and buy another drive, then power down the computer to fit it, assuming
there is room in the case for an extra drive.

Remember this thread started with a question about a single large disk.


-- 
Neil Bothwick

Sacred cows make great hamburgers.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[Volker Armin Hemmann]

On Freitag 02 April 2010, Neil Bothwick wrote:
> On Fri, 2 Apr 2010 14:45:29 +0200, Volker Armin Hemmann wrote:
> > seriously lvm sounds nice. But it isn't. It easily breaks.
> 
> Do you have something to back that up?
> 
> > You want a save setup? Go raid5 or raid6. As a bonus - you can get more
> > space if you need it by just adding another disk. And you are not
> > depending on some complex stuff to get it working.
> 
> LVM and RAID are completely different animals. No one suggested using it
> for any reasons of data security, running LVM on a RAID array gives both
> security and flexibility. As for being able to add space to RAID, you
> can't temporarily add a new volume whenever you want, you have to go out
> and buy another drive, then power down the computer to fit it, assuming
> there is room in the case for an extra drive.
> 

no need to power down - and you can add and remove drives. Read man mdadm.

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On Fri, 2 Apr 2010 20:40:54 +0200, Volker Armin Hemmann wrote:

> > LVM and RAID are completely different animals. No one suggested using
> > it for any reasons of data security, running LVM on a RAID array
> > gives both security and flexibility. As for being able to add space
> > to RAID, you can't temporarily add a new volume whenever you want,
> > you have to go out and buy another drive, then power down the
> > computer to fit it, assuming there is room in the case for an extra
> > drive. 
> 
> no need to power down - and you can add and remove drives. Read man
> mdadm.

Assuming your controller supports hotplugging, assuming you have a drive
available to plug in, assuming you are able to physically add a drive.


-- 
Neil Bothwick

Pound for pound, the amoeba is the most vicious animal on the earth.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[Volker Armin Hemmann]

On Freitag 02 April 2010, Neil Bothwick wrote:
> On Fri, 2 Apr 2010 20:40:54 +0200, Volker Armin Hemmann wrote:
> > > LVM and RAID are completely different animals. No one suggested using
> > > it for any reasons of data security, running LVM on a RAID array
> > > gives both security and flexibility. As for being able to add space
> > > to RAID, you can't temporarily add a new volume whenever you want,
> > > you have to go out and buy another drive, then power down the
> > > computer to fit it, assuming there is room in the case for an extra
> > > drive.
> > 
> > no need to power down - and you can add and remove drives. Read man
> > mdadm.
> 
> Assuming your controller supports hotplugging, assuming you have a drive
> available to plug in, assuming you are able to physically add a drive.

sata can hotplug. all ahci controlers can hotplug and all sata drives can 
hotplug. If you insist on technology straight from the stone ages that is your 
problem.

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

On Fri, 2 Apr 2010 21:50:09 +0200, Volker Armin Hemmann wrote:

> > Assuming your controller supports hotplugging, assuming you have a
> > drive available to plug in, assuming you are able to physically add a
> > drive.  
> 
> sata can hotplug. all ahci controlers can hotplug and all sata drives
> can hotplug. If you insist on technology straight from the stone ages
> that is your problem.

I'd like to see you hotplug another SATA drive into this netbook, whereas
I can add another volume in seconds.


-- 
Neil Bothwick

Dream as if you'll live forever. Live as if you'll die today.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[Alan McKinnon]

On Friday 02 April 2010 23:28:26 Neil Bothwick wrote:
> On Fri, 2 Apr 2010 21:50:09 +0200, Volker Armin Hemmann wrote:
> > > Assuming your controller supports hotplugging, assuming you have a
> > > drive available to plug in, assuming you are able to physically add a
> > > drive.
> > 
> > sata can hotplug. all ahci controlers can hotplug and all sata drives
> > can hotplug. If you insist on technology straight from the stone ages
> > that is your problem.
> 
> I'd like to see you hotplug another SATA drive into this netbook, whereas
> I can add another volume in seconds.

I'd like to see him add another SATA drive to my nameservers sitting in New 
York or the vmhost in Nairobi. I'm in Johannesburg.

Taking down that NewYork nameserver on a whim to add disks is not an option. 
It's an old machine, but a critical one and serves DNS to our entire European 
and US markets. Taking it down on a whim gets me fired.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] OT:Choosing a filesystem

[Peter Humphrey]

On Saturday 03 April 2010 09:55:39 Alan McKinnon wrote:
> On Friday 02 April 2010 23:28:26 Neil Bothwick wrote:
> > On Fri, 2 Apr 2010 21:50:09 +0200, Volker Armin Hemmann wrote:
> > > > Assuming your controller supports hotplugging, assuming you
> > > > have a drive available to plug in, assuming you are able to
> > > > physically add a drive.
> > > 
> > > sata can hotplug. all ahci controlers can hotplug and all sata
> > > drives can hotplug. If you insist on technology straight from
> > > the stone ages that is your problem.
> > 
> > I'd like to see you hotplug another SATA drive into this netbook,
> > whereas I can add another volume in seconds.
> 
> I'd like to see him add another SATA drive to my nameservers sitting
> in New York or the vmhost in Nairobi. I'm in Johannesburg.
> 
> Taking down that NewYork nameserver on a whim to add disks is not an
> option. It's an old machine, but a critical one and serves DNS to
> our entire European and US markets. Taking it down on a whim gets me
> fired.

Reminds me of that old chestnut, "all generalisations are false".

-- 
Rgds
Peter.

Re: [gentoo-user] OT:Choosing a filesystem

[Paul Hartman]

On Fri, Apr 2, 2010 at 2:50 PM, Volker Armin Hemmann
<volkerarmin@googlemail.com> wrote:
> On Freitag 02 April 2010, Neil Bothwick wrote:
>> On Fri, 2 Apr 2010 20:40:54 +0200, Volker Armin Hemmann wrote:
>> > > LVM and RAID are completely different animals. No one suggested using
>> > > it for any reasons of data security, running LVM on a RAID array
>> > > gives both security and flexibility. As for being able to add space
>> > > to RAID, you can't temporarily add a new volume whenever you want,
>> > > you have to go out and buy another drive, then power down the
>> > > computer to fit it, assuming there is room in the case for an extra
>> > > drive.
>> >
>> > no need to power down - and you can add and remove drives. Read man
>> > mdadm.
>>
>> Assuming your controller supports hotplugging, assuming you have a drive
>> available to plug in, assuming you are able to physically add a drive.
>
> sata can hotplug. all ahci controlers can hotplug and all sata drives can
> hotplug. If you insist on technology straight from the stone ages that is your
> problem.

Do you know if it's necessary to signal to the system (like /proc/scsi
something) that I'm about to unplug the drive, and in which order the
power/data need to be disconnected to prevent a problem? I'm curious
in case of future need. :)

Thanks.

Re: [gentoo-user] OT:Choosing a filesystem

[Mark Knecht]

On Fri, Apr 2, 2010 at 2:49 PM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Fri, Apr 2, 2010 at 2:50 PM, Volker Armin Hemmann
> <volkerarmin@googlemail.com> wrote:
>> On Freitag 02 April 2010, Neil Bothwick wrote:
>>> On Fri, 2 Apr 2010 20:40:54 +0200, Volker Armin Hemmann wrote:
>>> > > LVM and RAID are completely different animals. No one suggested using
>>> > > it for any reasons of data security, running LVM on a RAID array
>>> > > gives both security and flexibility. As for being able to add space
>>> > > to RAID, you can't temporarily add a new volume whenever you want,
>>> > > you have to go out and buy another drive, then power down the
>>> > > computer to fit it, assuming there is room in the case for an extra
>>> > > drive.
>>> >
>>> > no need to power down - and you can add and remove drives. Read man
>>> > mdadm.
>>>
>>> Assuming your controller supports hotplugging, assuming you have a drive
>>> available to plug in, assuming you are able to physically add a drive.
>>
>> sata can hotplug. all ahci controlers can hotplug and all sata drives can
>> hotplug. If you insist on technology straight from the stone ages that is your
>> problem.
>
> Do you know if it's necessary to signal to the system (like /proc/scsi
> something) that I'm about to unplug the drive, and in which order the
> power/data need to be disconnected to prevent a problem? I'm curious
> in case of future need. :)
>
> Thanks.

If it's part of a RAID the new one gets rebuilt.

If it's not part of a RAID then I think, as per Neil's example, the
computer is pretty much dead, right? However if you wanted to try it
(and I'm not brave enough so don't listen to me) then you might want
to do something like

grep -A 1 dirty /proc/vmstat

and wait until nothing is dirty.

Just an idea,
Mark

Re: [gentoo-user] OT:Choosing a filesystem

[Paul Hartman]

On Fri, Apr 2, 2010 at 4:55 PM, Mark Knecht <markknecht@gmail.com> wrote:
> On Fri, Apr 2, 2010 at 2:49 PM, Paul Hartman
> <paul.hartman+gentoo@gmail.com> wrote:
>> On Fri, Apr 2, 2010 at 2:50 PM, Volker Armin Hemmann
>> <volkerarmin@googlemail.com> wrote:
>>> On Freitag 02 April 2010, Neil Bothwick wrote:
>>>> On Fri, 2 Apr 2010 20:40:54 +0200, Volker Armin Hemmann wrote:
>>>> > > LVM and RAID are completely different animals. No one suggested using
>>>> > > it for any reasons of data security, running LVM on a RAID array
>>>> > > gives both security and flexibility. As for being able to add space
>>>> > > to RAID, you can't temporarily add a new volume whenever you want,
>>>> > > you have to go out and buy another drive, then power down the
>>>> > > computer to fit it, assuming there is room in the case for an extra
>>>> > > drive.
>>>> >
>>>> > no need to power down - and you can add and remove drives. Read man
>>>> > mdadm.
>>>>
>>>> Assuming your controller supports hotplugging, assuming you have a drive
>>>> available to plug in, assuming you are able to physically add a drive.
>>>
>>> sata can hotplug. all ahci controlers can hotplug and all sata drives can
>>> hotplug. If you insist on technology straight from the stone ages that is your
>>> problem.
>>
>> Do you know if it's necessary to signal to the system (like /proc/scsi
>> something) that I'm about to unplug the drive, and in which order the
>> power/data need to be disconnected to prevent a problem? I'm curious
>> in case of future need. :)
>>
>> Thanks.
>
> If it's part of a RAID the new one gets rebuilt.
>
> If it's not part of a RAID then I think, as per Neil's example, the
> computer is pretty much dead, right? However if you wanted to try it
> (and I'm not brave enough so don't listen to me) then you might want
> to do something like
>
> grep -A 1 dirty /proc/vmstat
>
> and wait until nothing is dirty.
>
> Just an idea,

Well, forgetting about RAID and bad drives, I should be able to
unmount a normal, working SATA drive and unplug it safely, just like
with a USB hard drive. I just don't know if you have to signal to
SATA/AHCI that you're going to unplug (like with old hot-swappable
SCSI drives), or if you need to unplug data cable before unplugging
the power cable, for example.

Re: [gentoo-user] OT:Choosing a filesystem

[Mark Knecht]

On Fri, Apr 2, 2010 at 3:03 PM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Fri, Apr 2, 2010 at 4:55 PM, Mark Knecht <markknecht@gmail.com> wrote:
>> On Fri, Apr 2, 2010 at 2:49 PM, Paul Hartman
>> <paul.hartman+gentoo@gmail.com> wrote:
>>> On Fri, Apr 2, 2010 at 2:50 PM, Volker Armin Hemmann
>>> <volkerarmin@googlemail.com> wrote:
>>>> On Freitag 02 April 2010, Neil Bothwick wrote:
>>>>> On Fri, 2 Apr 2010 20:40:54 +0200, Volker Armin Hemmann wrote:
>>>>> > > LVM and RAID are completely different animals. No one suggested using
>>>>> > > it for any reasons of data security, running LVM on a RAID array
>>>>> > > gives both security and flexibility. As for being able to add space
>>>>> > > to RAID, you can't temporarily add a new volume whenever you want,
>>>>> > > you have to go out and buy another drive, then power down the
>>>>> > > computer to fit it, assuming there is room in the case for an extra
>>>>> > > drive.
>>>>> >
>>>>> > no need to power down - and you can add and remove drives. Read man
>>>>> > mdadm.
>>>>>
>>>>> Assuming your controller supports hotplugging, assuming you have a drive
>>>>> available to plug in, assuming you are able to physically add a drive.
>>>>
>>>> sata can hotplug. all ahci controlers can hotplug and all sata drives can
>>>> hotplug. If you insist on technology straight from the stone ages that is your
>>>> problem.
>>>
>>> Do you know if it's necessary to signal to the system (like /proc/scsi
>>> something) that I'm about to unplug the drive, and in which order the
>>> power/data need to be disconnected to prevent a problem? I'm curious
>>> in case of future need. :)
>>>
>>> Thanks.
>>
>> If it's part of a RAID the new one gets rebuilt.
>>
>> If it's not part of a RAID then I think, as per Neil's example, the
>> computer is pretty much dead, right? However if you wanted to try it
>> (and I'm not brave enough so don't listen to me) then you might want
>> to do something like
>>
>> grep -A 1 dirty /proc/vmstat
>>
>> and wait until nothing is dirty.
>>
>> Just an idea,
>
> Well, forgetting about RAID and bad drives, I should be able to
> unmount a normal, working SATA drive and unplug it safely, just like
> with a USB hard drive. I just don't know if you have to signal to
> SATA/AHCI that you're going to unplug (like with old hot-swappable
> SCSI drives), or if you need to unplug data cable before unplugging
> the power cable, for example.
>
>

I've never done it but according to the SATA spec yes. As with all
drive umount first. Nothing I've read says it's truly safe to do it
too many times. It's easy to damage or wear out the connectors or the
drive.

It's the #1 'end-user benefit' according to the SATA spec web pages:

http://www.serialata.org/technology/why_sata.asp

- Mark

Re: [gentoo-user] OT:Choosing a filesystem

[Alan McKinnon]

On Friday 02 April 2010 14:45:29 Volker Armin Hemmann wrote:
> On Freitag 02 April 2010, meino.cramer@gmx.de wrote:
> > Neil Bothwick <neil@digimed.co.uk> [10-04-02 14:08]:
> > > On Fri, 2 Apr 2010 13:04:53 +0200, meino.cramer@gmx.de wrote:
> > > > only to be sure to have understood everything correctly:
> > > > Suggestion is to create for example one root partition and a swap
> > > > partion. And I will create on big "rest of the disk"-partition.
> > > > The last one will be subdivided with LVM into portions as needed.
> > > 
> > > Yes.
> > > 
> > > > Since the last big partition is big due to physical reasons (not for
> > > > logical one): What will happen, if -- for example -- one portion will
> > > > be not unmounted cleanly and while booting/checking fails to recover?
> > > > Are all others damaged/lost?
> > > 
> > > No, because the failure you describe is at the filesystem level. Even
> > > the volume containing that filesystem will retain integrity, only the
> > > filesystem itself will be corrupted. As you have left free space on
> > > the volume group, you can just create a new volume, format it and copy
> > > over everything you can recover from the broken filesystem before
> > > deleting it.
> > 
> > Hi Neil,
> > 
> > yes, sounds good, very good.
> > Last question: How heavy is the performance impact of such a setup ?
> 
> seriously lvm sounds nice. But it isn't. It easily breaks.

Can you back that up with some facts? I use LVM on many machines and have 
never had it breaks. I'm also quite ruthless on some machines with how I use 
it - manipulating volumes with apparently gay abandon.

I attribute this lack of failure to me understanding how LVm works and using 
it as designed, without trying to be cute and/or clever.

> You want a save setup? Go raid5 or raid6. As a bonus - you can get more
> space if you need it by just adding another disk. And you are not
> depending on some complex stuff to get it working.

The various raid levels do not address the problem that LVM solves - how to 
rapidly create and manipulate sub-volumes. If your /var/log fills up, how 
would you add an extra 10G to it to gain breathing space without using 
something LVM-like (evms is for example LVM-like. So are the native HP-UX 
tools)?

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] OT:Choosing a filesystem

[Dale]

Neil Bothwick wrote:
> On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:
>
>    
>> Then: I often transer videos from my DVB-T-receiver/recorder to my
>> harddisk to cut out the advertising and to transcode the videos to
>> somethings better than "ts" (transport streams),
>>      
> These tend to be bigger, often in the GB range, so I'd use a separate
> filesystem for them with XFS, which handles large files better in my
> experience.
>
>    

He mentioned in one of the first few posts that he regularly has hard 
shutdowns.  I took that as pulling the plug.  The last bit of experience 
I had with XFS, it does not like that sort of thing to happen.   Each 
time I had a hard shutdown, I had to reinstall the OS.  Has XFS changed 
so that power loss is not s problem or should he not use this after all?

Would hate for the OP to use XFS if it has not improved in that area.

Dale

:-)  :-)

Re: [gentoo-user] OT:Choosing a filesystem

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1779 bytes --]

On Friday 02 April 2010 16:28:43 Dale wrote:
> Neil Bothwick wrote:
> > On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:
> >> Then: I often transer videos from my DVB-T-receiver/recorder 
to my
> >> harddisk to cut out the advertising and to transcode the 
videos to
> >> somethings better than "ts" (transport streams),
> >
> > These tend to be bigger, often in the GB range, so I'd use a 
separate
> > filesystem for them with XFS, which handles large files 
better in my
> > experience.
> 
> He mentioned in one of the first few posts that he regularly 
has hard
> shutdowns.  I took that as pulling the plug.  The last bit of 
experience
> I had with XFS, it does not like that sort of thing to happen.   
Each
> time I had a hard shutdown, I had to reinstall the OS.  Has XFS 
changed
> so that power loss is not s problem or should he not use this 
after all?
> 
> Would hate for the OP to use XFS if it has not improved in that 
area.

XFS was ropey in its early days.  I had to re-install a partition 
once too (on a laptop!).  It is much more stable now (have not 
had a problem in the last 4+ years).

reiserfs is absolutely bullet proof here, with hundreds of 
crashes on a machine that had bad memory (like twice or three 
times a day I would have to pull the plug, for months on end 
until I isolated the error on a memory module).

reiser4 seems to be on a class of its own in terms of 
performance.  Perhaps not as forgiving on hard crashes as the 
reiserfs?  Not sure.  It's early days yet on this machine, but I 
have only praises for it so far.  I just hope they incorporate it 
in the kernel so that I don't have to manually patch it every 
time.

This is just my 2c's - so YMMV.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[Dale]

Mick wrote:
> On Friday 02 April 2010 16:28:43 Dale wrote:
>    
>> Neil Bothwick wrote:
>>      
>>> On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:
>>>        
>>>> Then: I often transer videos from my DVB-T-receiver/recorder
>>>>          
> to my
>    
>>>> harddisk to cut out the advertising and to transcode the
>>>>          
> videos to
>    
>>>> somethings better than "ts" (transport streams),
>>>>          
>>> These tend to be bigger, often in the GB range, so I'd use a
>>>        
> separate
>    
>>> filesystem for them with XFS, which handles large files
>>>        
> better in my
>    
>>> experience.
>>>        
>> He mentioned in one of the first few posts that he regularly
>>      
> has hard
>    
>> shutdowns.  I took that as pulling the plug.  The last bit of
>>      
> experience
>    
>> I had with XFS, it does not like that sort of thing to happen.
>>      
> Each
>    
>> time I had a hard shutdown, I had to reinstall the OS.  Has XFS
>>      
> changed
>    
>> so that power loss is not s problem or should he not use this
>>      
> after all?
>    
>> Would hate for the OP to use XFS if it has not improved in that
>>      
> area.
>
> XFS was ropey in its early days.  I had to re-install a partition
> once too (on a laptop!).  It is much more stable now (have not
> had a problem in the last 4+ years).
>
> reiserfs is absolutely bullet proof here, with hundreds of
> crashes on a machine that had bad memory (like twice or three
> times a day I would have to pull the plug, for months on end
> until I isolated the error on a memory module).
>
> reiser4 seems to be on a class of its own in terms of
> performance.  Perhaps not as forgiving on hard crashes as the
> reiserfs?  Not sure.  It's early days yet on this machine, but I
> have only praises for it so far.  I just hope they incorporate it
> in the kernel so that I don't have to manually patch it every
> time.
>
> This is just my 2c's - so YMMV.
>    

I haven't used XFS in several years.  I was hoping that it had 
improved.  I just wanted to make sure that it had improved and that it 
would be safe considering the OP has hard shutdowns.  I wouldn't want 
the OP to use it if he would lose data the first time he had a hard 
shutdown.  That would pretty much suck.

I agree on reiserfs tho.  I use it a lot here as well.  It works very 
well for me.

Dale

:-)  :-)

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Fri, 02 Apr 2010 10:28:43 -0500, Dale wrote:

> >> Then: I often transer videos from my DVB-T-receiver/recorder to my
> >> harddisk to cut out the advertising and to transcode the videos to
> >> somethings better than "ts" (transport streams),
> >>        
> > These tend to be bigger, often in the GB range, so I'd use a separate
> > filesystem for them with XFS, which handles large files better in my

> He mentioned in one of the first few posts that he regularly has hard 
> shutdowns.  I took that as pulling the plug.  The last bit of
> experience I had with XFS, it does not like that sort of thing to
> happen.   Each time I had a hard shutdown, I had to reinstall the OS.
> Has XFS changed so that power loss is not s problem or should he not
> use this after all?

If the system crashes so hard that even Magic SysRq can't help, he should
be fixing that first, rather than trying to find a filesystem that likes
such shutdowns. Having said that XFS is much better now and I was
recommending using it for video files, which are hardly life and death.


-- 
Neil Bothwick

I am ready to meet my Maker. Whether my Maker is prepared for the great
ordeal of meeting me is another matter. - Sir Winston Churchill
(1874-1965)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: OT:Choosing a filesystem

[Jörg Schaible]

Dale wrote:

> Neil Bothwick wrote:
>> On Thu, 1 Apr 2010 21:09:30 +0200, meino.cramer@gmx.de wrote:
>>
>>    
>>> Then: I often transer videos from my DVB-T-receiver/recorder to my
>>> harddisk to cut out the advertising and to transcode the videos to
>>> somethings better than "ts" (transport streams),
>>>      
>> These tend to be bigger, often in the GB range, so I'd use a separate
>> filesystem for them with XFS, which handles large files better in my
>> experience.
>>
>>    
> 
> He mentioned in one of the first few posts that he regularly has hard
> shutdowns.  I took that as pulling the plug.  The last bit of experience
> I had with XFS, it does not like that sort of thing to happen.   Each
> time I had a hard shutdown, I had to reinstall the OS.  Has XFS changed
> so that power loss is not s problem or should he not use this after all?
> 
> Would hate for the OP to use XFS if it has not improved in that area.

I am using XFS now for several years and I had only once an issue with it 
where I hit a known bug and I could restore the integrity with its own 
tools. And yes, I had all over the time the necessity to hard reset my box, 
because new nvidia drivers typically tend to freeze it from time to time.

- Jörg

Re: [gentoo-user] OT:Choosing a filesystem

[Ciprian Dorin, Craciun]

On Thu, Apr 1, 2010 at 10:09 PM,  <meino.cramer@gmx.de> wrote:
> [ ... snip ... ]
>
> So I have a lot of docs (specs of microcontrollers, howtos, programm
> and source code docs...etc) on my disk.
> This one part.


    I've seen that nobody mentioned JFS yet... :)

    In some benchmarks the best FS for most tasks is either XFS and
JFS, but it seems that JFS has less CPU and memory usage. So for small
and medium files I would say it's best. (I think it was on Tom's
Hardware site?)

    I'll also describe my history on the issue: initially I've only
used ReiserFS until something (not the hard drive) just snapped and
I've almost lost all my data. At that moment I've migrated to Ext3.

    But Ext3 has the problem of needing constant (usually once a moth)
checking (I know this is optional or tunable but it seems it is
recommended) which for large file systems takes incredibly long (60GB
HDD takes about 2 or 3 minutes... So imagine what would to to 1TB...)

    So I got angry again and moved to JFS... And I'm using JFS for
about two years without major incidents... (Only once I've lost the
contents of a configuration file due to a power interruption but this
is because of the editor.)

    So as a conclusion for this task I would recommend JFS (I also
have 200GB of documentation which covers about 100 thousand files I
guess.)

    Also see at the end for my notes on journaled file systems.


> Then: I often transer videos from my DVB-T-receiver/recorder to my
> harddisk to cut out the advertising and to transcode the videos to
> somethings better than "ts" (transport streams),
> This is another part.


    Although JFS could handle this, maybe a file system specially
designed for this would do best: Ext4 with it's extent feature. (But
be aware that by just using a file system is not enough... The
software also has to be specially crafted if you want high
performance. Just see the `fallocate` and `fadvise` system calls.)


> Then I plan to have two roots this time: One to experiment with and
> one "good and stable"-version which is used/updated/... "strictly as
> recommended". Filesizes and usage do vary here...take a look at your
> own roots ;)))


    :) This sounds like my setup: 160GB HDD from my laptop has the
following layout:
    * GPT partition table (not MBR) -- this gives me more partitions
without needing the "extended" partition feature of MBR;
    * 2 boot partitions of 512MB (maybe 1GB would have been better) --
one for current usage (Grub 0.97 with GPT patches) and one for
experimentation; these are Ext2 for safety and compatibility;
    * 3 root partitions of 4GB (I should have made them 8GB) -- one
for the current operating system, and two for future upgrades /
experimentation; currently JFS and maybe also so in the future;
    * 1 swap of 8GB (encrypted with random password with the help of dm-crypt);
    * rest of the HDD as one big partition with LVM; (large extents 256MB);
    * from the LVM I have partitions for personal data (/home) and
other things -- everything is JFS;


> Then I want something encrypted, either as a partition or as a files
> (carrying a encrypted fs), which I can copy to dvd and will be able
> to mount this dvd and use it without to have to copy the whole dvd
> first to harddisk before using it...
> Currently I am using encfs...(outdated?). What can I do use instead?
> This is for personal things like letters, photos, texts ... etc.
> Files vary from some kb up to about 2GByte (guessed). Most of them
> smaller than 200MByte


    As someone noted maybe EncryptFS (in kernel one) would be
better... (It's an install option in Ubuntu so I would say it's mature
enough.)

    But for this encrypted purpose I would use dm-crypt with
`aes-xts-essiv:sha256` encryption. (In the past I've used LoopAES but
I had some minor issues with kernel building as it's not in the
vanilla kernel...)


> Last thing: I have a lot iof copies of code from svn repositories because
> I like to have the "bleeding edge" of some projects (do you know the
> new Blender 2.50??? :O)


    I also have a lot of repositories on JFS and everything works nice.


> This implies a lot of compile work. This will be the only case where
> files are created as often as read.


    For temporary folders while compiling I would recommend to
instruct your build scripts to build inside /tmp where you have tmpfs
mounted... It's blazingly fast...


    And some notes about journaled file systems: they journal
meta-data (that is file creation, deletion, rename, etc), not data
(that is the contents)... (Of course there are a few (Ext3 maybe?)
file systems that have the option to also journal data...)

    What does this mean: well when you edit a file and save it and
then cut the power, the file still exists (the meta-data), but the
contents could (and usually is) wrong: either no content (like I've
encountered once with JFS), either mixed content (old and new)...

    So the fineprint here is: no journaled file system is safe... They
are all safe if you also use fsync (which forces everything to go on
disk)... This is why I say that the fault for the file content lost is
from the editor:
    * it opened the file by truncating it => 0 length;
    * it wrote to it and closed it;
    * it DIDN'T `fsync`-ed it which means that the data still remained
in the buffer cache;
    * and when the power was lost so was the data in the buffer cache;

    Another fine-print about file system performance -- memory helps a
lot... I've upgraded my laptop from 2GB to 4GB of RAM and with some
fine-tuning the file operations are more snappier... The fine-tuning
included delaying the write-back to about 3 minutes or until 1GB of
data is dirty... Which means that if my laptop loses power or I need
to hard power-off it I'll lose a great deal of data...

    Hope I was of some help,
    Ciprian.

    P.S.: The following link could give you some insight on the
journaled file system problems (not only about ReiserFS)
    http://zork.net/~nick/mail/why-reiserfs-is-teh-sukc

Re: [gentoo-user] OT:Choosing a filesystem, OT: Blender

[Frank Steinmetzger]

[-- Attachment #1: Type: Text/Plain, Size: 976 bytes --]

Am Donnerstag, 1. April 2010 schrieb meino.cramer@gmx.de:

> thanks for all the input to all who have answered ! :)
> 
> I will try to characterize ("characterise" ?) what I plan to do with
> my TByte disk.

Characterise if you’re in British domains, characterize if you are in the US.

> Last thing: I have a lot iof copies of code from svn repositories because
> I like to have the "bleeding edge" of some projects (do you know the
> new Blender 2.50??? :O)

I’ve tried 2.50 yesterday, but something’s not right here. Does yours run 
normally? When I didn’t get it to compile by hand (mkdir build; cd build; 
cmake ../; make), I tried an ebuild (also to make sure I have all 
dependencies). That compiled through, but the GUI is incomplete and buggy. I 
get lots of "missing module bpy_types" on startup, but the file is there. Do 
you have some hints?
-- 
Gruß | Greetings | Qapla'
I haven’t lost my mind; it's backed up on tape somewhere!

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[Mark Knecht]

On Thu, Apr 1, 2010 at 10:47 AM,  <meino.cramer@gmx.de> wrote:
>
> Hi,
>
> I googled down some - often fairly outdated - texts about "the
> best filesystem" fpr a Linux box. Other texts focussed on
> uses, which do not aplly to me: Fileservers, webservers, database
> machines  etc.
>
> Wnat I want is a fast and stable (!) filesystem for a desktop PC
> with one 1TByte harddisk. Since using Gentoo and a lot of sources
> I do compile very often "bigger things" (blender-2.50 for example).
> Another thing: Due to my experimenting it is possible that I have to
> reboot "hard", which means, the filesystem will be unmounted not
> cleanly ("dirty" do to say...;) The choosen filesystem should be
> good in recovering such thing.
>
> I am currently using a vanilla 2.6.32.10 kernel.
>
> The question, what remains is: What choose should I make?
>
> I thank you very much in advance for any help!
> Best regards,
> mcc
>

This doesn't address why you would choose one over another but it was
a recent view of Reiser4 vs a couple of others.

http://www.phoronix.com/scan.php?page=article&item=reiser4_benchmarks&num=1

I'm way behind. I haven't even tried ext4 yet!

Good luck,
Mark

Re: [gentoo-user] OT:Choosing a filesystem

[stosss]

On Thu, Apr 1, 2010 at 1:47 PM,  <meino.cramer@gmx.de> wrote:
>
> Hi,
>
> I googled down some - often fairly outdated - texts about "the
> best filesystem" fpr a Linux box. Other texts focussed on
> uses, which do not aplly to me: Fileservers, webservers, database
> machines  etc.
>
> Wnat I want is a fast and stable (!) filesystem for a desktop PC
> with one 1TByte harddisk. Since using Gentoo and a lot of sources
> I do compile very often "bigger things" (blender-2.50 for example).
> Another thing: Due to my experimenting it is possible that I have to
> reboot "hard", which means, the filesystem will be unmounted not
> cleanly ("dirty" do to say...;) The choosen filesystem should be
> good in recovering such thing.
>
> I am currently using a vanilla 2.6.32.10 kernel.
>
> The question, what remains is: What choose should I make?

I have been following this thread. I decided to research to do my own
comparisons of ext3, ext4, JFS and XFS.

ext3 has 3 journaling levels:

Journal (lowest risk)
Ordered (medium risk) most Linux distributions are using this one
Writeback (highest risk)

XFS uses Ordered (medium risk)
JFS uses Writeback (highest risk)

It appears from the documentation that ext4 takes the best of ext3, XFS and JFS.

My research also showed that ext2/3 is the most widely used on Linux
and has the greatest community support coverage.

ext4 falls into the same category as XFS and JFS in this respect.

It appears that ext4, XFS or JFS or some combination of them would be
the best choice.

If you want to know where I got my information use Google like I did.

-- 
If we can but prevent the government from wasting the labours of the
people, under the pretence of taking care of them, they must become
happy. - Thomas Jefferson

Re: [gentoo-user] OT:Choosing a filesystem

[meino.cramer]

stosss <stosss@gmail.com> [10-04-03 05:31]:
> On Thu, Apr 1, 2010 at 1:47 PM,  <meino.cramer@gmx.de> wrote:
> >
> > Hi,
> >
> > I googled down some - often fairly outdated - texts about "the
> > best filesystem" fpr a Linux box. Other texts focussed on
> > uses, which do not aplly to me: Fileservers, webservers, database
> > machines  etc.
> >
> > Wnat I want is a fast and stable (!) filesystem for a desktop PC
> > with one 1TByte harddisk. Since using Gentoo and a lot of sources
> > I do compile very often "bigger things" (blender-2.50 for example).
> > Another thing: Due to my experimenting it is possible that I have to
> > reboot "hard", which means, the filesystem will be unmounted not
> > cleanly ("dirty" do to say...;) The choosen filesystem should be
> > good in recovering such thing.
> >
> > I am currently using a vanilla 2.6.32.10 kernel.
> >
> > The question, what remains is: What choose should I make?
> 
> I have been following this thread. I decided to research to do my own
> comparisons of ext3, ext4, JFS and XFS.
> 
> ext3 has 3 journaling levels:
> 
> Journal (lowest risk)
> Ordered (medium risk) most Linux distributions are using this one
> Writeback (highest risk)
> 
> XFS uses Ordered (medium risk)
> JFS uses Writeback (highest risk)
> 
> It appears from the documentation that ext4 takes the best of ext3, XFS and JFS.
> 
> My research also showed that ext2/3 is the most widely used on Linux
> and has the greatest community support coverage.
> 
> ext4 falls into the same category as XFS and JFS in this respect.
> 
> It appears that ext4, XFS or JFS or some combination of them would be
> the best choice.
> 
> If you want to know where I got my information use Google like I did.
> 
> -- 
> If we can but prevent the government from wasting the labours of the
> people, under the pretence of taking care of them, they must become
> happy. - Thomas Jefferson
> 

Hi Stoss,

thanks for your effort ! :)

As I wrote, I did googling before starting this thread and found
mostly outdated informations or informations not applying to my
situation. Often it is best -- regardless what papers of 2008 or
before are stateing -- to ask people for their current and 
uptodate experiences. Additionally your informations are all pure
technical based...they are missing exactly what I was searching
for: Experiences of people using different setups. And as you can
see: This thread reports many of that.

Best reagrds.
mcc



-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] OT:Choosing a filesystem

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 342 bytes --]

On Fri, 2 Apr 2010 18:01:50 -0400, stosss wrote:

> I have been following this thread. I decided to research to do my own
> comparisons of ext3, ext4, JFS and XFS.

Why have you ignored reiser3 and reiser4? The former in particular is
widely used.


-- 
Neil Bothwick

Pepperami. Its a bit of an animal.
What animal & what bit?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] OT:Choosing a filesystem

[David W Noon]

[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]

On Fri, 02 Apr 2010 11:20:02 +0200, meino.cramer@gmx.de wrote about Re:
[gentoo-user] OT:Choosing a filesystem:

[snip]
>A question to LVM: As much as I know, LVM combines several partition
>to one big partition, and if one partition fails, at least other
>others of that volume are damaged, too.

Disks fail.  Sectors fail.  Partitions do not fail.  Logical volumes do
not fail.

If your disk fails you lose *all* partitions on it.

If some sectors fail, the file(s) backed by those sectors will be
corrupted -- regardless of filesystem type.  If the defective sectors
back a filesystem's superblock or other infrastructure, you could well
lose the filesystem; but most modern filesystems keep redundant copies
of their infrastructure, and fsck can sometimes recover.

>What is the advantage of using LVM and several small partitions
>instead of one in the size of the sum of the others and not using
>LVM?

LVM provides immense flexibility in creating, deleting and expanding
filesystems.  Once you get used to using LVM, which is not difficult,
you will never go back to partitions.
-- 
Regards,

Dave  [RLU #314465]
======================================================================
dwnoon@ntlworld.com (David W Noon)
======================================================================

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]