[gentoo-user] Good/better/best filesystem for large, static video library?

[gentoo-user] Good/better/best filesystem for large, static video library?

[Mark Knecht]

Hi,
   Merry Christmas to all.

   Upgrading an external USB2 drive at home this Christmas morning to
1TB for more video storage space. One large partition, non-raid, files
are around 1GB. The drive holds only static video files that get
written once and don't change or get erased. No MythTV stuff or
anything like that.

   This disk reside on my main desktop machine and gets backed up
every couple of days to another USB2 drive (FAT formatted
unfortunately) which attaches to the TV.

   With the previous local drive I used ext3 and have had no problems.
I'm just wondering if there's a better choice & why.

Cheers,
Mark

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

On Dec 25, 2012 10:44 PM, "Mark Knecht" <markknecht@gmail.com> wrote:
>
> Hi,
>    Merry Christmas to all.
>
>    Upgrading an external USB2 drive at home this Christmas morning to
> 1TB for more video storage space. One large partition, non-raid, files
> are around 1GB. The drive holds only static video files that get
> written once and don't change or get erased. No MythTV stuff or
> anything like that.
>
>    This disk reside on my main desktop machine and gets backed up
> every couple of days to another USB2 drive (FAT formatted
> unfortunately) which attaches to the TV.
>
>    With the previous local drive I used ext3 and have had no problems.
> I'm just wondering if there's a better choice & why.
>
> Cheers,
> Mark
>

For your usage, I think ext3 is the most suitable.

Do you have another fs in mind?

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 1119 bytes --]

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Mark Knecht]

On Tue, Dec 25, 2012 at 8:33 AM, Pandu Poluan <pandu@poluan.info> wrote:
>
> On Dec 25, 2012 10:44 PM, "Mark Knecht" <markknecht@gmail.com> wrote:
<SNIP>
>>    With the previous local drive I used ext3 and have had no problems.
>> I'm just wondering if there's a better choice & why.
<SNIP>
>
> For your usage, I think ext3 is the most suitable.
>
> Do you have another fs in mind?

Really, no. ext3 has been fine. I didn't see any real advantage to
ext4 myself. Florian offers the removal argument but I've never
removed files from this database. It's just movies so the systems just
grows over time.

I suppose I wondered whether some other filesystem might get through
an fsck _much_ faster. This machine gets shut down in the evening so
fsck operations happen roughly once a month. At times I need to get up
and running in the morning and get held up behind an fsck after so
many days.

Other than that I don't have any real issues, and presupposing that
ext3 would be my final choice I put it on and started rsyncing the
files, but if another answer is really better I have no problems with
blowing that away and starting again.

Thanks,
Mark

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Michael Orlitzky]

On 12/25/2012 12:07 PM, Mark Knecht wrote:
> On Tue, Dec 25, 2012 at 8:33 AM, Pandu Poluan <pandu@poluan.info> wrote:
>>
>> On Dec 25, 2012 10:44 PM, "Mark Knecht" <markknecht@gmail.com> wrote:
> <SNIP>
>>>    With the previous local drive I used ext3 and have had no problems.
>>> I'm just wondering if there's a better choice & why.
> <SNIP>
>>
>> For your usage, I think ext3 is the most suitable.
>>
>> Do you have another fs in mind?
> 
> Really, no. ext3 has been fine. I didn't see any real advantage to
> ext4 myself. Florian offers the removal argument but I've never
> removed files from this database. It's just movies so the systems just
> grows over time.
> 
> I suppose I wondered whether some other filesystem might get through
> an fsck _much_ faster.
> 

There's really no reason to use ext3 over ext4. Ext4 does have a faster
fsck.

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Dale]

Michael Orlitzky wrote:
> On 12/25/2012 12:07 PM, Mark Knecht wrote:
>> On Tue, Dec 25, 2012 at 8:33 AM, Pandu Poluan <pandu@poluan.info> wrote:
>>> On Dec 25, 2012 10:44 PM, "Mark Knecht" <markknecht@gmail.com> wrote:
>> <SNIP>
>>>>    With the previous local drive I used ext3 and have had no problems.
>>>> I'm just wondering if there's a better choice & why.
>> <SNIP>
>>> For your usage, I think ext3 is the most suitable.
>>>
>>> Do you have another fs in mind?
>> Really, no. ext3 has been fine. I didn't see any real advantage to
>> ext4 myself. Florian offers the removal argument but I've never
>> removed files from this database. It's just movies so the systems just
>> grows over time.
>>
>> I suppose I wondered whether some other filesystem might get through
>> an fsck _much_ faster.
>>
> There's really no reason to use ext3 over ext4. Ext4 does have a faster
> fsck.
>
>
>

I have noticed the fsck is fast here too, faster than reiserfs anyway. 
It seems ext4 is pretty fast with everything, at least in my eye.  I
also found that ext4 has a defrag tool.  It rarely finds any fragments
but at least it is available. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Mark Knecht]

On Tue, Dec 25, 2012 at 9:18 AM, Michael Orlitzky <michael@orlitzky.com> wrote:
> On 12/25/2012 12:07 PM, Mark Knecht wrote:
>> On Tue, Dec 25, 2012 at 8:33 AM, Pandu Poluan <pandu@poluan.info> wrote:
>>>
>>> On Dec 25, 2012 10:44 PM, "Mark Knecht" <markknecht@gmail.com> wrote:
>> <SNIP>
>>>>    With the previous local drive I used ext3 and have had no problems.
>>>> I'm just wondering if there's a better choice & why.
>> <SNIP>
>>>
>>> For your usage, I think ext3 is the most suitable.
>>>
>>> Do you have another fs in mind?
>>
>> Really, no. ext3 has been fine. I didn't see any real advantage to
>> ext4 myself. Florian offers the removal argument but I've never
>> removed files from this database. It's just movies so the systems just
>> grows over time.
>>
>> I suppose I wondered whether some other filesystem might get through
>> an fsck _much_ faster.
>>
>
> There's really no reason to use ext3 over ext4. Ext4 does have a faster
> fsck.
>
>

If the graph here

http://kerneltrap.org/Linux/Improving_fsck_Speeds_in_Ext4

represents real speed improvement then it's likely worth it to me. The
drive is doubling in size but initially the data isn't. (500GB getting
rsync'ed to 1TB)

As I have no immediate needs for the older drive I will give ext4 a
try and just hang on to the old drive&data for a few months and see
how it goes. I'd do that anyway in case the new drive has an infant
mortality issue show up, but really the backup is the 1TB on the TV
which is stable and in use for over a year with no smartctl issues.
(yet) However that drive is FAT formatted so I don't really want to
depend on it for anything long term.

I wonder if there's anything to be said for changing block sizes, etc.
away from whatever the defaults are? All of the files are currently
between 350MB & 1.2GB so there's never going to be many more than 2K
files on the drive and I'm assuming the rsync operation if file by
file so fragmentation in the beginning, and probably over time, is
going to be pretty low I think.

Thanks for the info.

Cheers,
Mark

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]

Am 25.12.2012 19:26, schrieb Mark Knecht:
> On Tue, Dec 25, 2012 at 9:18 AM, Michael Orlitzky <michael@orlitzky.com> wrote:
>> On 12/25/2012 12:07 PM, Mark Knecht wrote:
>>> On Tue, Dec 25, 2012 at 8:33 AM, Pandu Poluan <pandu@poluan.info> wrote:
>>>>
>>>> On Dec 25, 2012 10:44 PM, "Mark Knecht" <markknecht@gmail.com> wrote:
>>> <SNIP>
>>>>>    With the previous local drive I used ext3 and have had no problems.
>>>>> I'm just wondering if there's a better choice & why.
>>> <SNIP>
>>>>
>>>> For your usage, I think ext3 is the most suitable.
>>>>
>>>> Do you have another fs in mind?
>>>
>>> Really, no. ext3 has been fine. I didn't see any real advantage to
>>> ext4 myself. Florian offers the removal argument but I've never
>>> removed files from this database. It's just movies so the systems just
>>> grows over time.
>>>
[...]
> 
> I wonder if there's anything to be said for changing block sizes, etc.
> away from whatever the defaults are? All of the files are currently
> between 350MB & 1.2GB so there's never going to be many more than 2K
> files on the drive and I'm assuming the rsync operation if file by
> file so fragmentation in the beginning, and probably over time, is
> going to be pretty low I think.
> 

The default of 4k blocks is the largest possible and with big files
there is no advantage in using smaller blocks. In fact, it could come
back to bite you as it limits file sizes. I guess it also increases fsck
times.

Another issue I noticed is that it can cause terrible performance if the
block size is smaller than the physical block size of the device. mke2fs
warns when it detects this.

Regards,
Florian Philipp



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

Am 25.12.2012 16:41, schrieb Mark Knecht:
> Hi,
>    Merry Christmas to all.
> 
>    Upgrading an external USB2 drive at home this Christmas morning to
> 1TB for more video storage space. One large partition, non-raid, files
> are around 1GB. The drive holds only static video files that get
> written once and don't change or get erased. No MythTV stuff or
> anything like that.
> 
>    This disk reside on my main desktop machine and gets backed up
> every couple of days to another USB2 drive (FAT formatted
> unfortunately) which attaches to the TV.
> 
>    With the previous local drive I used ext3 and have had no problems.
> I'm just wondering if there's a better choice & why.
> 
> Cheers,
> Mark
> 

Ext4 offers better performance for large files. This is especially
notable when you remove them but other operations are faster, too. XFS
would be the traditional large-file choice but since the arrival of
Ext4, I don't see a point in putting up with its quirks anymore.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Dale]

Florian Philipp wrote:
> Am 25.12.2012 16:41, schrieb Mark Knecht:
>> Hi,
>>    Merry Christmas to all.
>>
>>    Upgrading an external USB2 drive at home this Christmas morning to
>> 1TB for more video storage space. One large partition, non-raid, files
>> are around 1GB. The drive holds only static video files that get
>> written once and don't change or get erased. No MythTV stuff or
>> anything like that.
>>
>>    This disk reside on my main desktop machine and gets backed up
>> every couple of days to another USB2 drive (FAT formatted
>> unfortunately) which attaches to the TV.
>>
>>    With the previous local drive I used ext3 and have had no problems.
>> I'm just wondering if there's a better choice & why.
>>
>> Cheers,
>> Mark
>>
> Ext4 offers better performance for large files. This is especially
> notable when you remove them but other operations are faster, too. XFS
> would be the traditional large-file choice but since the arrival of
> Ext4, I don't see a point in putting up with its quirks anymore.
>
> Regards,
> Florian Philipp
>


For those who keep up with my adventures, I use ext4 for my home
directory which has a LOT of videos.  Some videos are small and some are
large but I can say this, it is really fast.  No fragmentation either. 

I also use ext4 for my backup drive.  I accidentally deleted some stuff
one day and I can say this, it is VERY fast.  I'm just glad it was only
a backup. 

I would second the idea for ext4.  It works great for me.

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 2074 bytes --]

Am 25.12.2012 18:15, schrieb Dale:
> Florian Philipp wrote:
>> Am 25.12.2012 16:41, schrieb Mark Knecht:
>>> Hi,
>>>    Merry Christmas to all.
>>>
>>>    Upgrading an external USB2 drive at home this Christmas morning to
>>> 1TB for more video storage space. One large partition, non-raid, files
>>> are around 1GB. The drive holds only static video files that get
>>> written once and don't change or get erased. No MythTV stuff or
>>> anything like that.
>>>
>>>    This disk reside on my main desktop machine and gets backed up
>>> every couple of days to another USB2 drive (FAT formatted
>>> unfortunately) which attaches to the TV.
>>>
>>>    With the previous local drive I used ext3 and have had no problems.
>>> I'm just wondering if there's a better choice & why.
>>>
>>> Cheers,
>>> Mark
>>>
>> Ext4 offers better performance for large files. This is especially
>> notable when you remove them but other operations are faster, too. XFS
>> would be the traditional large-file choice but since the arrival of
>> Ext4, I don't see a point in putting up with its quirks anymore.
>>
>> Regards,
>> Florian Philipp
>>
> 
> 
> For those who keep up with my adventures, I use ext4 for my home
> directory which has a LOT of videos.  Some videos are small and some are
> large but I can say this, it is really fast.  No fragmentation either. 
> 
> I also use ext4 for my backup drive.  I accidentally deleted some stuff
> one day and I can say this, it is VERY fast.  I'm just glad it was only
> a backup. 
> 
> I would second the idea for ext4.  It works great for me.
> 
> Dale
> 

On an amusing side note: I once was very happy that I happened to be
running ext3 at work. I accidentally started `rm -rf ~/*` and after more
than a minute I wondered why my disk was so busy ...

Anyway, thanks to its slow speed, rm only had time to eat away my rather
unimportant "archive" and "bin" directories and hasn't reached "doc"
yet. Thankfully bash sorts its glob expansions alphabetically. :D

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Dale]

Florian Philipp wrote:
> Am 25.12.2012 18:15, schrieb Dale:
>> Florian Philipp wrote:
>>> Am 25.12.2012 16:41, schrieb Mark Knecht:
>>>> Hi,
>>>>    Merry Christmas to all.
>>>>
>>>>    Upgrading an external USB2 drive at home this Christmas morning to
>>>> 1TB for more video storage space. One large partition, non-raid, files
>>>> are around 1GB. The drive holds only static video files that get
>>>> written once and don't change or get erased. No MythTV stuff or
>>>> anything like that.
>>>>
>>>>    This disk reside on my main desktop machine and gets backed up
>>>> every couple of days to another USB2 drive (FAT formatted
>>>> unfortunately) which attaches to the TV.
>>>>
>>>>    With the previous local drive I used ext3 and have had no problems.
>>>> I'm just wondering if there's a better choice & why.
>>>>
>>>> Cheers,
>>>> Mark
>>>>
>>> Ext4 offers better performance for large files. This is especially
>>> notable when you remove them but other operations are faster, too. XFS
>>> would be the traditional large-file choice but since the arrival of
>>> Ext4, I don't see a point in putting up with its quirks anymore.
>>>
>>> Regards,
>>> Florian Philipp
>>>
>>
>> For those who keep up with my adventures, I use ext4 for my home
>> directory which has a LOT of videos.  Some videos are small and some are
>> large but I can say this, it is really fast.  No fragmentation either. 
>>
>> I also use ext4 for my backup drive.  I accidentally deleted some stuff
>> one day and I can say this, it is VERY fast.  I'm just glad it was only
>> a backup. 
>>
>> I would second the idea for ext4.  It works great for me.
>>
>> Dale
>>
> On an amusing side note: I once was very happy that I happened to be
> running ext3 at work. I accidentally started `rm -rf ~/*` and after more
> than a minute I wondered why my disk was so busy ...
>
> Anyway, thanks to its slow speed, rm only had time to eat away my rather
> unimportant "archive" and "bin" directories and hasn't reached "doc"
> yet. Thankfully bash sorts its glob expansions alphabetically. :D
>
> Regards,
> Florian Philipp
>

Mine only had a few seconds, it was ALL gone.  I hit the ctrl C just
about the time the prompt came back.  Well, I got to make new backups. 
lol   Almost 1Tb of stuff too. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Marc Stürmer]

2012/12/25 Mark Knecht <markknecht@gmail.com>:

>    Upgrading an external USB2 drive at home this Christmas morning to
> 1TB for more video storage space. One large partition, non-raid, files
> are around 1GB. The drive holds only static video files that get
> written once and don't change or get erased. No MythTV stuff or
> anything like that.
>
>    This disk reside on my main desktop machine and gets backed up
> every couple of days to another USB2 drive (FAT formatted
> unfortunately) which attaches to the TV.

Well in your usage case ext3 is still well suited enough.

Ext4 though is the superior filesystem, since it is more advanced in
technical terms. Maximum file system size in ext3 is around 16
Terabyte, something not so far ahead now even more in homes. Ext4 has
the maximum file system size of 1024 Petabyte.

ext4 also uses extents, which ext3 has not - meaning file system
checks are able to run faster. Erasing big files on ext4 works
therefore faster than on ext3. It also tends to fragment less than
ext3.

ext4 has persistent preallocation when writing large files, meaning
space is being guaranteed and most probably contiguous.

tl;dr: ext3 should be well suited enough for your computer, but if you
can reformat your hard disk drive, using ext4 will not hurt either and
you gain some faster speed.

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Alan McKinnon]

On Tue, 25 Dec 2012 07:41:01 -0800
Mark Knecht <markknecht@gmail.com> wrote:

> Hi,
>    Merry Christmas to all.
> 
>    Upgrading an external USB2 drive at home this Christmas morning to
> 1TB for more video storage space. One large partition, non-raid, files
> are around 1GB. The drive holds only static video files that get
> written once and don't change or get erased. No MythTV stuff or
> anything like that.
> 
>    This disk reside on my main desktop machine and gets backed up
> every couple of days to another USB2 drive (FAT formatted
> unfortunately) which attaches to the TV.
> 
>    With the previous local drive I used ext3 and have had no problems.
> I'm just wondering if there's a better choice & why.


I am *very* impressed with ZFS for this. Yes, I know, it's not really
there on Linux - I use it on FreeBSD (FreeNAS).

It has everything I've wanted in a filesystem for a long time, and all
the crap I've stuffed into my head over many years related to storage
just goes away. It doesn't go to some place I don't have to deal with
it, it just ceases to exist. Very nice.

There is no more weird partitions from the days of DOS, no PV/VG/LV to
remember the details of. There is only storage and ZFS knows what I
want to happen with each "chunk" of it. A "chunk" (my term) in this
context is a directory and everything below it.

ZFS doesn't have partitions and filesystems. It has volumes. A volume
is sort of a cross between a filesystem (you mount it and can assign
quotas to it) and a directory (you assign permissions and ownerships to
it). You can overcommit storage space and quotas - you do not get "disk
full" errors and three days of nightmares while you figure out how to
deal with this. the FS just tells you it used more than the allocated
space and keeps telling you till you get it under the limit.

mv'ing a few TB of video to a different FS to free up space is not fun
at all, but with ZFS it's like an mv on the same FS (that volume thing
again). It checksums every write and lets you know if things fail. It
has proper snapshots built in - that's proper as in copy-on-write so
they don't really take up space until you start modifying files. Your
media collection is like mine - I only add to it and seldom delete, so
I have months of snapshots that consume about 1% extras space. Dale's
rm problem cannot happen to me anymore hehehehe ;-)

In summary, it does everything I want and does it well. It can also do
other things I don't want but others might (eg de-dupe).


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Stefan G. Weichinger]

Am 27.12.2012 01:18, schrieb Alan McKinnon:

> I am *very* impressed with ZFS for this. Yes, I know, it's not really
> there on Linux - I use it on FreeBSD (FreeNAS).
> 
> It has everything I've wanted in a filesystem for a long time, and all
> the crap I've stuffed into my head over many years related to storage
> just goes away. It doesn't go to some place I don't have to deal with
> it, it just ceases to exist. Very nice.
> 
> There is no more weird partitions from the days of DOS, no PV/VG/LV to
> remember the details of. There is only storage and ZFS knows what I
> want to happen with each "chunk" of it. A "chunk" (my term) in this
> context is a directory and everything below it.
> 
> ZFS doesn't have partitions and filesystems. It has volumes. A volume
> is sort of a cross between a filesystem (you mount it and can assign
> quotas to it) and a directory (you assign permissions and ownerships to
> it). You can overcommit storage space and quotas - you do not get "disk
> full" errors and three days of nightmares while you figure out how to
> deal with this. the FS just tells you it used more than the allocated
> space and keeps telling you till you get it under the limit.
> 
> mv'ing a few TB of video to a different FS to free up space is not fun
> at all, but with ZFS it's like an mv on the same FS (that volume thing
> again). It checksums every write and lets you know if things fail. It
> has proper snapshots built in - that's proper as in copy-on-write so
> they don't really take up space until you start modifying files. Your
> media collection is like mine - I only add to it and seldom delete, so
> I have months of snapshots that consume about 1% extras space. Dale's
> rm problem cannot happen to me anymore hehehehe ;-)
> 
> In summary, it does everything I want and does it well. It can also do
> other things I don't want but others might (eg de-dupe).

I also used ZFS for that kind of storage and it was very pleasant to
work with. Performance is an issue if you use zfs-fuse for example,
although there are people running zfs-on-linux on LUKS for their main
working machine (using an SSD, that helps!):

https://mthode.org/

describes such a setup (although slightly off-topic in context of media
libraries).

I had the pleasure to write a feature about that and provide a demo-vm:

http://www.oops.co.at/en/publications/beitrag-fuer-linux-magazin-012013-zfs-und-luks

(german feature, sorry ...)

-

Using snapshots with ZFS really is fun and let's you rethink stuff. It
makes doing backups easier and you can send and receive them via pipes
(think ssh here).

ZFS thinks of storage in the way we think of RAM: plug in some more and
everything (as in "every filesystem") is able to use it, just a pool of
ressources (actually it uses that term all over, "a zfs pool called tank").

I'd love to use it as root-fs sometimes, but I still hesitate. Might be
better to stay with the filesystems most linux-users use, just to
benefit of the huge tester-group :-)

For media-storage I wouldn't hesitate to run zfs-on-linux on a mirror of
2 disks or so. With that you benefit of the so-called self-healing:

If the checksum for block X on disk sda isn't correct, very likely the
checksum for block X on disk sdb is still valid. ZFS sees that, creates
a new block X on sda, with the correct content&checksum and drops the
corrupted block (and doesn't reuse it, AFAIK).

That is a great feature ... maybe not too important for video where the
occasional bit error isn't that much of a problem. But good to have for
other stuff ...

Greets, Stefan

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Michael Hampicke]

Am 27.12.2012 01:18, schrieb Alan McKinnon:
> On Tue, 25 Dec 2012 07:41:01 -0800
> Mark Knecht <markknecht@gmail.com> wrote:
> 
>> Hi,
>>    Merry Christmas to all.
>>
>>    Upgrading an external USB2 drive at home this Christmas morning to
>> 1TB for more video storage space. One large partition, non-raid, files
>> are around 1GB. The drive holds only static video files that get
>> written once and don't change or get erased. No MythTV stuff or
>> anything like that.
>>
>>    This disk reside on my main desktop machine and gets backed up
>> every couple of days to another USB2 drive (FAT formatted
>> unfortunately) which attaches to the TV.
>>
>>    With the previous local drive I used ext3 and have had no problems.
>> I'm just wondering if there's a better choice & why.
> 
> 
> I am *very* impressed with ZFS for this. Yes, I know, it's not really
> there on Linux - I use it on FreeBSD (FreeNAS).
> 
> It has everything I've wanted in a filesystem for a long time, and all
> the crap I've stuffed into my head over many years related to storage
> just goes away. It doesn't go to some place I don't have to deal with
> it, it just ceases to exist. Very nice.
> 
> There is no more weird partitions from the days of DOS, no PV/VG/LV to
> remember the details of. There is only storage and ZFS knows what I
> want to happen with each "chunk" of it. A "chunk" (my term) in this
> context is a directory and everything below it.
> 
> ZFS doesn't have partitions and filesystems. It has volumes. A volume
> is sort of a cross between a filesystem (you mount it and can assign
> quotas to it) and a directory (you assign permissions and ownerships to
> it). You can overcommit storage space and quotas - you do not get "disk
> full" errors and three days of nightmares while you figure out how to
> deal with this. the FS just tells you it used more than the allocated
> space and keeps telling you till you get it under the limit.
> 
> mv'ing a few TB of video to a different FS to free up space is not fun
> at all, but with ZFS it's like an mv on the same FS (that volume thing
> again). It checksums every write and lets you know if things fail. It
> has proper snapshots built in - that's proper as in copy-on-write so
> they don't really take up space until you start modifying files. Your
> media collection is like mine - I only add to it and seldom delete, so
> I have months of snapshots that consume about 1% extras space. Dale's
> rm problem cannot happen to me anymore hehehehe ;-)
> 
> In summary, it does everything I want and does it well. It can also do
> other things I don't want but others might (eg de-dupe).
> 
> 

I am a big fan of zfs myself. I use zfsonlinx on my workstation (only
/usr/portage and /usr/src atm, but with on-the-fly compression, very
nice on my small SSD). Unfortunately the zfs implementation is a few
large steps behind zfs on *bsd

Migrating my NAS to ZFS is something that has been floating around my
head for a longer time. But I am not really sure if I want to switch
from gentoo to FreeBSD on my NAS. zfsonlinux is there, but it's first
release was early 2011, so it's still pretty young. I guess for the time
being I stick to an old credo: never touch a running system

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Alan McKinnon]

On Thu, 27 Dec 2012 14:57:09 +0100
Michael Hampicke <gentoo-user@hadt.biz> wrote:

> Migrating my NAS to ZFS is something that has been floating around my
> head for a longer time. But I am not really sure if I want to switch
> from gentoo to FreeBSD on my NAS. zfsonlinux is there, but it's first
> release was early 2011, so it's still pretty young. I guess for the
> time being I stick to an old credo: never touch a running system

My main requirement was simple: the NAS must be no-touch.

I'm perfectly happy fiddling till 3am with every other system I have,
but not that one :-)

I needed something that family could use too with all reasonable
share types active (the family can pick how they want to connect), and
that must stay the same for long periods. I figured the only way to get
that was to rig it so I couldn't fiddle too much. FreeBSD is familiar
territory so FreeNAS was the logical choice.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1134 bytes --]

On Thu, 27 Dec 2012 02:18:00 +0200, Alan McKinnon wrote:

> There is no more weird partitions from the days of DOS, no PV/VG/LV to
> remember the details of. There is only storage and ZFS knows what I
> want to happen with each "chunk" of it. A "chunk" (my term) in this
> context is a directory and everything below it.
> 
> ZFS doesn't have partitions and filesystems. It has volumes. A volume
> is sort of a cross between a filesystem (you mount it and can assign
> quotas to it) and a directory (you assign permissions and ownerships to
> it). You can overcommit storage space and quotas - you do not get "disk
> full" errors and three days of nightmares while you figure out how to
> deal with this. the FS just tells you it used more than the allocated
> space and keeps telling you till you get it under the limit.

I've been looking at zfsonlinux and it looks a lot simpler than the
layers of RAID and LVM, but what about encryption. Can I encrypt
directories within ZFS or do I have to use something like ecryptfs on top
of it?


-- 
Neil Bothwick

A clean desk is a sign of a cluttered desk drawer.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Alan McKinnon]

On Wed, 2 Jan 2013 13:04:50 +0000
Neil Bothwick <neil@digimed.co.uk> wrote:

> On Thu, 27 Dec 2012 02:18:00 +0200, Alan McKinnon wrote:
> 
> > There is no more weird partitions from the days of DOS, no PV/VG/LV
> > to remember the details of. There is only storage and ZFS knows
> > what I want to happen with each "chunk" of it. A "chunk" (my term)
> > in this context is a directory and everything below it.
> > 
> > ZFS doesn't have partitions and filesystems. It has volumes. A
> > volume is sort of a cross between a filesystem (you mount it and
> > can assign quotas to it) and a directory (you assign permissions
> > and ownerships to it). You can overcommit storage space and quotas
> > - you do not get "disk full" errors and three days of nightmares
> > while you figure out how to deal with this. the FS just tells you
> > it used more than the allocated space and keeps telling you till
> > you get it under the limit.
> 
> I've been looking at zfsonlinux and it looks a lot simpler than the
> layers of RAID and LVM, but what about encryption. Can I encrypt
> directories within ZFS or do I have to use something like ecryptfs on
> top of it?


AFAIK, Oracle included encryption in ZFS v30 but this has not been
released as opensource. The last OSS version released was 28.

What this means to me is that devs could include disk-encryption but
they probably won't have a standard to code to, and that implies a
whole lotta YMMV. You'd have to use ecryptfs or friends for now.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Volker Armin Hemmann]

Am Mittwoch, 2. Januar 2013, 15:38:01 schrieb Alan McKinnon:

> AFAIK, Oracle included encryption in ZFS v30 but this has not been
> released as opensource. The last OSS version released was 28.
> 
> What this means to me is that devs could include disk-encryption but
> they probably won't have a standard to code to, and that implies a
> whole lotta YMMV. You'd have to use ecryptfs or friends for now.

ecryptfs looks easy to use with zfs:
http://pthree.org/2012/08/21/encrypted-zfs-filesystems-on-linux/

-- 
#163933

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

On Wed, 02 Jan 2013 15:08:57 +0100, Volker Armin Hemmann wrote:

> ecryptfs looks easy to use with zfs:
> http://pthree.org/2012/08/21/encrypted-zfs-filesystems-on-linux/

I found that one on subsequent googling. I had hoped that it would be
included in ZFS, meaning I could replace four layers; RAID, LVM,
encryption and filesystem, with one. Still, 4->2 is still good.


-- 
Neil Bothwick

Top Oxymorons Number 5: Twelve-ounce pound cake

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Good/better/best filesystem for large, static video library?

[Alecks Gates]

On Tue, Dec 25, 2012 at 9:41 AM, Mark Knecht <markknecht@gmail.com> wrote:
> Hi,
>    Merry Christmas to all.
>
>    Upgrading an external USB2 drive at home this Christmas morning to
> 1TB for more video storage space. One large partition, non-raid, files
> are around 1GB. The drive holds only static video files that get
> written once and don't change or get erased. No MythTV stuff or
> anything like that.
>
>    This disk reside on my main desktop machine and gets backed up
> every couple of days to another USB2 drive (FAT formatted
> unfortunately) which attaches to the TV.
>
>    With the previous local drive I used ext3 and have had no problems.
> I'm just wondering if there's a better choice & why.
>
> Cheers,
> Mark
>

I use ext4 for my videos, ranging from ~150 MB to ~20 GB.  It's very
painless and is included directly in the kernel.  I went on a very
long search for the "best" filesystem for videos and thought to myself
the only problem I was having was imaginary, ext4 is fine.


Alecks