[gentoo-user] Hard drive storage questions

[gentoo-user] Hard drive storage questions

[Dale]

Howdy,

I have a 3TB hard drive that I use for my /home partition.  I'm going to
be having to expand this before to long, lots of videos on there.  The
4TB is a bit pricey and I would end up having to expand that to before
to long.  So, I got to thinking, why not buy another 3TB drive and just
add that which would double my space.  I use LVM by the way.  I may try
BTFS, (sp?).   Either way, adding a drive shouldn't be to much of a
problem. 

On one hand, adding a drive would double my space.  It would also spread
out my stuff in the event a drive failed.  On the other hand, one more
drive to have spinning that could fail too.  These large drives makes me
wonder sometimes. 

What do you guys, gals too, think about this?  Just add a drive or buy a
larger drive and move things over?  Or is this a six of one and half
dozen of the other thing? 

Dale

:-)  :-) 

P. S.


Filesystem                                Size      Used     Avail    
Use%     Mounted on
/dev/mapper/Home2-Home2    2.7T     1.8T      945G      66%     /home

[gentoo-user] Hard drive storage questions

[Dale]

Howdy,

I have a 3TB hard drive that I use for my /home partition.  I'm going to
be having to expand this before to long, lots of videos on there.  The
4TB is a bit pricey and I would end up having to expand that to before
to long.  So, I got to thinking, why not buy another 3TB drive and just
add that which would double my space.  I use LVM by the way.  I may try
BTFS, (sp?).   Either way, adding a drive shouldn't be to much of a
problem.

On one hand, adding a drive would double my space.  It would also spread
out my stuff in the event a drive failed.  On the other hand, one more
drive to have spinning that could fail too.  These large drives makes me
wonder sometimes.

What do you guys, gals too, think about this?  Just add a drive or buy a
larger drive and move things over?  Or is this a six of one and half
dozen of the other thing?

Dale



P. S.


Filesystem      	 Size      Used     Avail    Use%     Mounted on
/dev/mapper/Home2-Home2  2.7T     1.8T      945G      66%     /home

Re: [gentoo-user] Hard drive storage questions

[Francisco Ares]

[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]

2015-04-28 5:39 GMT-03:00 Dale <rdalek1967@gmail.com>:

> Howdy,
>
> I have a 3TB hard drive that I use for my /home partition.  I'm going to
> be having to expand this before to long, lots of videos on there.  The
> 4TB is a bit pricey and I would end up having to expand that to before
> to long.  So, I got to thinking, why not buy another 3TB drive and just
> add that which would double my space.  I use LVM by the way.  I may try
> BTFS, (sp?).   Either way, adding a drive shouldn't be to much of a
> problem.
>
> On one hand, adding a drive would double my space.  It would also spread
> out my stuff in the event a drive failed.  On the other hand, one more
> drive to have spinning that could fail too.  These large drives makes me
> wonder sometimes.
>
> What do you guys, gals too, think about this?  Just add a drive or buy a
> larger drive and move things over?  Or is this a six of one and half
> dozen of the other thing?
>
> Dale
>
>
>
> P. S.
>
>
> Filesystem               Size      Used     Avail    Use%     Mounted on
> /dev/mapper/Home2-Home2  2.7T     1.8T      945G      66%     /home
>
>
>
>
I've already had my /home partition filled up, and it was just one of the
partitions on the only hard drive inside this machine.

Nowadays I have 3 more hard drives on this same machine, which makes 4
drives in total. None of them so big, though, the newest one is 1TB.

Now, talking about redundancy, if you worry about many - or all - of your
files, perhaps you should consider a RAID. Although expensive, it makes
one's mind more relaxed ;-) .  Basically, what I have seen is that many
people worry about backing up files when at work, but completely forget
about it on personal computers.

Best regards, and good luck!
Francisco

[-- Attachment #2: Type: text/html, Size: 2419 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Alan McKinnon]

On 28/04/2015 10:39, Dale wrote:
> Howdy,
> 
> I have a 3TB hard drive that I use for my /home partition.  I'm going to
> be having to expand this before to long, lots of videos on there.  The
> 4TB is a bit pricey and I would end up having to expand that to before
> to long.  So, I got to thinking, why not buy another 3TB drive and just
> add that which would double my space.  I use LVM by the way.  I may try
> BTFS, (sp?).   Either way, adding a drive shouldn't be to much of a
> problem.
> 
> On one hand, adding a drive would double my space.  It would also spread
> out my stuff in the event a drive failed.  On the other hand, one more
> drive to have spinning that could fail too.  These large drives makes me
> wonder sometimes.
> 
> What do you guys, gals too, think about this?  Just add a drive or buy a
> larger drive and move things over?  Or is this a six of one and half
> dozen of the other thing?
> 
> Dale
> 
> 
> 
> P. S.
> 
> 
> Filesystem      	 Size      Used     Avail    Use%     Mounted on
> /dev/mapper/Home2-Home2  2.7T     1.8T      945G      66%     /home
> 
> 
> 


When you're up into the TB range you run a higher risk of losing data
than with disks of a few 100 GB simply because it's bigger and there are
more bits that can flip [1].

When you use only LVM for this and nothing else, you have a high risk of
losing everything if one disk fails. Why? Because LVM decides itself
which extent it will put data on. Maybe a whole file is on one disk,
maybe it's spread across two, because the software is designed so that
you don't have to be concerned with that. The only thing that LVM does
is expand your storage space as a single volume and make it easier to
shuffle things around without having to backup/repartition/restore.

The best solution for you depends on what you need and what you have. If
your disks are full of YouTube videos that you can easily download again
(or stream), maybe you don't care too much. Precious family photos that
can't be replaced? You need to care a lot.

Personally, I like the ZFS approach and do it all in software, catching
errors that RAID misses.
RAID is also an option - 1:1 mirroring works great if you are much more
concerned about data than about cost.

There is no general advice in this area[2], the trick is to understand
the various technologies, fully understand your own needs and budget,
then plan accordingly.



[1] All things being equal that is. A 3TB disk is probably not really
the same as a 500G disk, just bigger. It's safe to assume that disk
manufacturers pat attention to error rates etc and improve their
products over time to make them more reliable. As to by how much - I
don't know.

[2] There is however a vendor's desire to maximize their profit while
still leaving you with warm and fuzzies </sarcasm>

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Tue, Apr 28, 2015 at 4:39 AM, Dale <rdalek1967@gmail.com> wrote:
>
> What do you guys, gals too, think about this?  Just add a drive or buy a
> larger drive and move things over?  Or is this a six of one and half
> dozen of the other thing?
>

Generally I buy drives at the sweet spot in cost/capacity, so that is
about 3TB last time I checked (for spinning disks).

I ALWAYS use RAID or full backups of some kind.  RAID isn't really a
substitute for backups, but I use it as such for low-priority data
such as mythtv recordings or re-generatable data.  Right now I'm
running on mirrored btrfs with a full backup to ext4 (since btrfs is
living dangerously).  I'm actually getting tight on space and debating
dropping the full backups for lower-priority data, which would free up
a 3TB drive to add to the btrfs array.  Long-term I'd prefer to move
to raid5 which is much more efficient, but I wouldn't recommend doing
that on btrfs yet - it is very immature.

raid5 on mdadm and lvm with ext4 is very mature, and is probably your
most space-efficient option with some level of redundancy.  With large
arrays having raid6 isn't a bad idea these days - it takes a lot of
time to recover a failure.  However, if you have a single drive today
there is no way to add only a single disk and get both more space and
redundancy at the same time.  If you want more space and only want to
buy one drive, then you're stuck with just simple lvm and if a drive
fails you're going to lose a lot of stuff.

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]

On Tue, 28 Apr 2015 17:01:49 +0200, Alan McKinnon wrote:

> When you use only LVM for this and nothing else, you have a high risk of
> losing everything if one disk fails. Why? Because LVM decides itself
> which extent it will put data on. Maybe a whole file is on one disk,
> maybe it's spread across two, because the software is designed so that
> you don't have to be concerned with that. The only thing that LVM does
> is expand your storage space as a single volume and make it easier to
> shuffle things around without having to backup/repartition/restore.

An alternative is to create a new volume group on the new disk and mounts
PVs at various points in your home directory. That way you get the extra
space and much of the flexibility without the risk of a failure on a
single drive taking out data on both. However, if you are concerned about
data loss, you should be using RAID t a minimum, preferably with an error
detecting filesystem.

> Personally, I like the ZFS approach and do it all in software, catching
> errors that RAID misses.

The same is also  possible with BTRFS, including built in RAID. RAID5 in
btrfs is expermiental, but its RAID1 is like RAID5 in some ways, such as
giving the capacity of n-1 disks and tolerating a single disk failure.


-- 
Neil Bothwick

PC DOS Error #04: Out of disk space. Delete Windows? (Y)es (H)ell yes!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Tue, Apr 28, 2015 at 11:24 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> The same is also  possible with BTRFS, including built in RAID. RAID5 in
> btrfs is expermiental, but its RAID1 is like RAID5 in some ways, such as
> giving the capacity of n-1 disks and tolerating a single disk failure.
>

btrfs raid5 is still fairly experimental (though now it supports
recovery) and works more-or-less how you'd expect raid5 to work.
Raid1 on btrfs gives you the capacity of n/2 and not n-1 disks, as you
would expect.  It does allow disks to be of different size, in which
case it gives you up to n/2 capacity (and usually more than you'd get
with traditional raid1 - it tries to fill the largest disks first so
3+1+1+1 TB will give you 3TB of storage, not 2TB as you'd get with
mdadm raid1).

Here is a btrfs raid1:
df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/sdh2                    3.2T  2.7T  477G  86% /data

btrfs fi df /data
Data, RAID1: total=2.93TiB, used=2.65TiB
System, RAID1: total=32.00MiB, used=472.00KiB
Metadata, RAID1: total=16.00GiB, used=14.08GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

btrfs fi sho /data
Label: 'datafs'  uuid: cd074207-9bc3-402d-bee8-6a8c77d56959
        Total devices 5 FS bytes used 2.67TiB
        devid    1 size 2.73TiB used 2.63TiB path /dev/sdh2
        devid    2 size 931.32GiB used 832.03GiB path /dev/sda2
        devid    3 size 931.32GiB used 834.00GiB path /dev/sde2
        devid    4 size 931.32GiB used 832.00GiB path /dev/sdd2
        devid    5 size 931.32GiB used 833.00GiB path /dev/sdb2


2.7TiB of data is stored on the array, which has nearly exhausted the
space of 6.4TiB of drives (or 7TB).  There are ~500GiB free, which
would let me store ~250GiB of data.

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

On Tue, 28 Apr 2015 13:38:55 -0400, Rich Freeman wrote:

> > The same is also  possible with BTRFS, including built in RAID. RAID5
> > in btrfs is expermiental, but its RAID1 is like RAID5 in some ways,
> > such as giving the capacity of n-1 disks and tolerating a single disk
> > failure. 
> 
> btrfs raid5 is still fairly experimental (though now it supports
> recovery) and works more-or-less how you'd expect raid5 to work.
> Raid1 on btrfs gives you the capacity of n/2 and not n-1 disks,

You're right, I was clearly confused (an oxymoron?) when I wrote that.

So RAID1 gives less capacity than RAID5 on BTRFS, but it is stable (in
btrfs terms).


-- 
Neil Bothwick

This universe is sold by mass, not by volume.
Some expansion may have occurred during shipment

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Daniel Frey]

On 04/27/2015 12:41 AM, Dale wrote:
> What do you guys, gals too, think about this?  Just add a drive or buy a
> larger drive and move things over?  Or is this a six of one and half
> dozen of the other thing? 

I just went through this myself, and I found a NAS with four drives in
it. I actually got it as a part of some special, I couldn't even buy the
four 2TB disks included in it individually for the price of the whole NAS.

That reminds me, I should set it up. :-)

Dan

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Tue, Apr 28, 2015 at 2:11 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Tue, 28 Apr 2015 13:38:55 -0400, Rich Freeman wrote:
>
>> > The same is also  possible with BTRFS, including built in RAID. RAID5
>> > in btrfs is expermiental, but its RAID1 is like RAID5 in some ways,
>> > such as giving the capacity of n-1 disks and tolerating a single disk
>> > failure.
>>
>> btrfs raid5 is still fairly experimental (though now it supports
>> recovery) and works more-or-less how you'd expect raid5 to work.
>> Raid1 on btrfs gives you the capacity of n/2 and not n-1 disks,
>
> You're right, I was clearly confused (an oxymoron?) when I wrote that.
>
> So RAID1 gives less capacity than RAID5 on BTRFS, but it is stable (in
> btrfs terms).
>

Correct.

For drives of identical size and not using compression, I'd expect
space use on btrfs to be equivalent to the same raid level on
mdadm+lvm+ext4.  With mixed drives you will potentially get more space
on btrfs, and compression will of course get you more space.

As far as data security goes there is a tradeoff.  Btrfs is still
immature and I seem to have issues with it 1-2 times per year (but
I've yet to have unrecoverable data loss).  On the other hand, btrfs
does do full data checksumming which means you're less likely to lose
data due to issues with the physical storage than with mdadm - as with
zfs it always checks the checksum and will recover from another disk
if possible, and in the event of raid disparity it always knows which
(if any) of the copies is right.

I'm hopeful that at some point I'll be able to recommend it without
reservation.  Right now, that isn't entirely the case.  I'm still
patching the 3.18 kernel series so that it actually mounts my root
partition when whatever is causing it to panic (probably also btrfs)
does so (the patch is in the queue, but hasn't made it to 3.18 yet for
some reason - I believe it has been in the other stable series for a
release or two now).

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Tue, 28 Apr 2015 14:31:23 -0400, Rich Freeman wrote:

> >> btrfs raid5 is still fairly experimental (though now it supports
> >> recovery) and works more-or-less how you'd expect raid5 to work.
> >> Raid1 on btrfs gives you the capacity of n/2 and not n-1 disks,  
> >
> > You're right, I was clearly confused (an oxymoron?) when I wrote that.
> >
> > So RAID1 gives less capacity than RAID5 on BTRFS, but it is stable (in
> > btrfs terms).
> >  
> 
> For drives of identical size and not using compression, I'd expect
> space use on btrfs to be equivalent to the same raid level on
> mdadm+lvm+ext4.

That's only true for RAID1 when you have 2 drives, with more drives btrfs
gives more space.

3 x 2TB drives give 2TB on MD RAID1
3 x 2TB drives give 3TB on btrfs RAID1

Although you will get slightly less usable space with btrfs because of
the space used for metadata, but I'm not sure how significant that is.


-- 
Neil Bothwick

QOTD:
	The only easy way to tell a hamster from a gerbil is that the
	gerbil has more dark meat.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Dale]

Daniel Frey wrote:
> On 04/27/2015 12:41 AM, Dale wrote:
>> What do you guys, gals too, think about this?  Just add a drive or buy a
>> larger drive and move things over?  Or is this a six of one and half
>> dozen of the other thing? 
> I just went through this myself, and I found a NAS with four drives in
> it. I actually got it as a part of some special, I couldn't even buy the
> four 2TB disks included in it individually for the price of the whole NAS.
>
> That reminds me, I should set it up. :-)
>
> Dan
>
>

I've noticed the external drives sometimes have better prices than
internal ones at times.  Makes me wonder.  :/  I thought about taking
the drives out and putting them in my rig and use the external case for
something else.

For some reason, I got a message back that other message didn't make it
through so I resent it.  I guess it did make it after all.  ;-)

Dale

:-)  :-) 

[gentoo-user] Re: Hard drive storage questions

[walt]

On 04/28/2015 08:24 AM, Neil Bothwick wrote:
> On Tue, 28 Apr 2015 17:01:49 +0200, Alan McKinnon wrote:
 
>> Personally, I like the ZFS approach and do it all in software, catching
>> errors that RAID misses.
> 
> The same is also  possible with BTRFS,

I have the impression (without knowing what I'm talking about) that BTRFS
was created to be just like ZFS, minus the software licensing problems.

Is my impression right or wrong?

Re: [gentoo-user] Re: Hard drive storage questions

[Rich Freeman]

On Tue, Apr 28, 2015 at 6:02 PM, walt <w41ter@gmail.com> wrote:
> On 04/28/2015 08:24 AM, Neil Bothwick wrote:
>> On Tue, 28 Apr 2015 17:01:49 +0200, Alan McKinnon wrote:
>
>>> Personally, I like the ZFS approach and do it all in software, catching
>>> errors that RAID misses.
>>
>> The same is also  possible with BTRFS,
>
> I have the impression (without knowing what I'm talking about) that BTRFS
> was created to be just like ZFS, minus the software licensing problems.
>
> Is my impression right or wrong?

Kinda.  Same sort of idea, and the licensing obviously has a big part in it.

The underlying designs are different, which means that when fully
mature each will likely have different pros/cons, which is great since
we can all pick/choose what we need.

The other difference is that ZFS is targeted more at enterprise /
large-scale use, and btrfs is targeted more as a general-purpose
filesystem that you might use on a single-disk PC.  That isn't to say
that either can't be used in either situation, but you can definitely
see where there has been more focus in feature development.  For
example, with zfs you can not only have large pools of drives, but you
can also bind them into smaller redundancy pools.  So, you can have 10
"raid6" arrays bound together which ensures that the scale of rebuilds
is limited while giving you a common pool of space.  On the other
hand, with btrfs you can have a 3-disk raid5 and turn it into a 4-disk
raid5 without having to copy/restore all your data (or you could turn
a 3-disk raid1 into a 4-disk raid5, and even switch halfway so that
half your data is in raid1 mode and half in raid5).  That is the sort
of thing that is handy in a small PC where you don't just have stacks
of disks lying around to build fresh new arrays from, but less
important for a big enterprise SAN where you don't need to add one
disk at a time to a 40-disk storage unit.

I'm sure many features exclusive to either btrfs or zfs will
eventually make their way to the other.  However, their differing
focuses make it likely that some features will mature faster than
others.

And of course btrfs has been taking a fairly long time to mature - it
just doesn't seem like as serious of an enterprise-y project.  But,
neither is Gentoo.  :)

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Alan McKinnon]

On 28/04/2015 17:24, Neil Bothwick wrote:
> On Tue, 28 Apr 2015 17:01:49 +0200, Alan McKinnon wrote:
> 
>> When you use only LVM for this and nothing else, you have a high risk of
>> losing everything if one disk fails. Why? Because LVM decides itself
>> which extent it will put data on. Maybe a whole file is on one disk,
>> maybe it's spread across two, because the software is designed so that
>> you don't have to be concerned with that. The only thing that LVM does
>> is expand your storage space as a single volume and make it easier to
>> shuffle things around without having to backup/repartition/restore.
> 
> An alternative is to create a new volume group on the new disk and mounts
> PVs at various points in your home directory. That way you get the extra
> space and much of the flexibility without the risk of a failure on a
> single drive taking out data on both. However, if you are concerned about
> data loss, you should be using RAID t a minimum, preferably with an error
> detecting filesystem.


I've used that scheme myself in the past. You do get the increased space
but you don't get much in the way of flexibility. And it get COMPLICATED
really quickly.

To get around the situation of one drive almost full and the other
having lots of space, folks often use symlinked directories, which you
forget about and no-one else can figure out what you did...

It all smacks of the old saw:

For any non-trivial problem, there is always at least one solution that
is simple, elegant, and wrong.



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: Hard drive storage questions

[Alan McKinnon]

On 29/04/2015 00:02, walt wrote:
> On 04/28/2015 08:24 AM, Neil Bothwick wrote:
>> On Tue, 28 Apr 2015 17:01:49 +0200, Alan McKinnon wrote:
>  
>>> Personally, I like the ZFS approach and do it all in software, catching
>>> errors that RAID misses.
>>
>> The same is also  possible with BTRFS,
> 
> I have the impression (without knowing what I'm talking about) that BTRFS
> was created to be just like ZFS, minus the software licensing problems.
> 
> Is my impression right or wrong?


As with all things, it's probably more complicated than that :-)

I personally think that ZFS (from Sun Microsystems) and BTRFS (from
Oracle) were originally convergent solutions to the same problem, much
like Gnome and KDE both try solve the desktop problem. ZFS started out
in the Solaris world, and BTRFS in the Oracle-cloned-Red-Hat world, so
there is that difference.

Then Oracle bought Sun and now Oracle "owns" both codebases, so who
knows what's going in internally at that corporation wrt modern filesystems.

ZFS licensing is a problem that should not exist. AFAIK, Sun owned the
entire codebase and used their own license. Oracle owns it now, so there
doesn't seem to be anything stopping Oracle from releasing the whole
thing under multiple licenses, making the problem go away.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]

On Wed, 29 Apr 2015 08:13:41 +0200, Alan McKinnon wrote:

> > An alternative is to create a new volume group on the new disk and
> > mounts PVs at various points in your home directory. That way you get
> > the extra space and much of the flexibility without the risk of a
> > failure on a single drive taking out data on both. However, if you
> > are concerned about data loss, you should be using RAID at a minimum,
> > preferably with an error detecting filesystem.  
> 
> I've used that scheme myself in the past. You do get the increased space
> but you don't get much in the way of flexibility. And it get COMPLICATED
> really quickly.

It certainly can, but for a simple two drive home system it shouldn't get
out of hand. However, it does avoid the "one disk errors kills two
disks' data" problem.

> To get around the situation of one drive almost full and the other
> having lots of space, folks often use symlinked directories, which you
> forget about and no-one else can figure out what you did...

I wasn't suggesting symlinks, just LVs mounted at appropriate points. It
rather depends on the spread of Dale's data. If he just needs extra space
for his videos, he could get a new drive and mount it at ~/videos.

> It all smacks of the old saw:
> 
> For any non-trivial problem, there is always at least one solution that
> is simple, elegant, and wrong.

:-)

I consider what I suggested somewhat simple but far from elegant. Often
though, it's a lot less work in the long run to go for the initially more
complex solution. If Dale is worried about the likelihood of disk
failure, he really should be using RAID - either MDRAID under LVM or one
of the next-gen filesystems.


-- 
Neil Bothwick

... Taglines: and How They Affect Women. Next On Oprah.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[gentoo-user] Re: Hard drive storage questions

[Grant Edwards]

On 2015-04-29, Alan McKinnon <alan.mckinnon@gmail.com> wrote:

Regarding ZFS licensing problems:

> [...] there doesn't seem to be anything stopping Oracle from [...]
> making the problem go away.

In my rather limited experience with Oracle, "making the problem go
away" never seemed to be high on their agenda for anything they were
involved in.

-- 
Grant Edwards               grant.b.edwards        Yow! Now we can become
                                  at               alcoholics!
                              gmail.com            

Re: [gentoo-user] Hard drive storage questions

[Dale]

Dale wrote:
> Howdy,
>
> <<  SNIP >>
>
> Dale
>
>
>
> P. S.
>
>
> Filesystem      	 Size      Used     Avail    Use%     Mounted on
> /dev/mapper/Home2-Home2  2.7T     1.8T      945G      66%     /home
>
>
>


Well, I read replies a few times and I think it is best to just add a
new drive.  Heck, I've already had a 3TB drive to fail.  Anyway, I also
need to look into some sort of backup system.  I used to do this with
DVDs but with this much "stuff", that just isn't a good idea, not to
mention that DVDs have their own issues.   I may take a peek into a RAID
setup since really, that is about the best if not only way to do it. 

Thanks all for the replies.

Dale

:-)  :-) 

Re: [gentoo-user] Hard drive storage questions

[Dale]

[-- Attachment #1: Type: text/plain, Size: 2687 bytes --]

Neil Bothwick wrote:
> On Wed, 29 Apr 2015 08:13:41 +0200, Alan McKinnon wrote:
>
>>> An alternative is to create a new volume group on the new disk and
>>> mounts PVs at various points in your home directory. That way you get
>>> the extra space and much of the flexibility without the risk of a
>>> failure on a single drive taking out data on both. However, if you
>>> are concerned about data loss, you should be using RAID at a minimum,
>>> preferably with an error detecting filesystem. 
>>
>> I've used that scheme myself in the past. You do get the increased space
>> but you don't get much in the way of flexibility. And it get COMPLICATED
>> really quickly.
>
> It certainly can, but for a simple two drive home system it shouldn't get
> out of hand. However, it does avoid the "one disk errors kills two
> disks' data" problem.

Yea, right now, I'm only using two drives.  One for the OS and one for
/home.  I have a third drive but it isn't in use.  I'm thinking about
moving everything but the videos to that drive, 750GB, and leave just
the videos on the large 3TB drive.  It'll free up a *little* space too.


>
>
>> To get around the situation of one drive almost full and the other
>> having lots of space, folks often use symlinked directories, which you
>> forget about and no-one else can figure out what you did...
>
> I wasn't suggesting symlinks, just LVs mounted at appropriate points. It
> rather depends on the spread of Dale's data. If he just needs extra space
> for his videos, he could get a new drive and mount it at ~/videos.

The bulk of the space is used by the videos.  It's everything from TV
shows to movies to youtube howtos.  I'm using roughly 1.8TB on the drive
and the videos take up roughly 1.7TB of that space.  My camera pics only
use 21GBs of space.  Rest is basically a rounding error.  :/

>
>> It all smacks of the old saw:
>>
>> For any non-trivial problem, there is always at least one solution that
>> is simple, elegant, and wrong.
>
> :-)
>
> I consider what I suggested somewhat simple but far from elegant. Often
> though, it's a lot less work in the long run to go for the initially more
> complex solution. If Dale is worried about the likelihood of disk
> failure, he really should be using RAID - either MDRAID under LVM or one
> of the next-gen filesystems.
>
>

I really do need to set up RAID at least for some stuff that I may not
be able to get back.  Some videos I have are no longer available.   What
I wish, I had a second puter in a outbuilding that I could copy to over
ethernet or something.  May help in the event of a house fire etc.

Dale

:-)  :-)


[-- Attachment #2: Type: text/html, Size: 3790 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]

On Mon, 04 May 2015 02:39:10 -0500, Dale wrote:

> > I wasn't suggesting symlinks, just LVs mounted at appropriate points.
> > It rather depends on the spread of Dale's data. If he just needs
> > extra space for his videos, he could get a new drive and mount it at
> > ~/videos.
> 
> The bulk of the space is used by the videos.  It's everything from TV
> shows to movies to youtube howtos.  I'm using roughly 1.8TB on the drive
> and the videos take up roughly 1.7TB of that space.  My camera pics only
> use 21GBs of space.  Rest is basically a rounding error.  :/

You need to separate those anyway, for backup purposes. Anything you
downloaded, you can usually download again, so you only need a list of
the files to be able to find them again.

On the other hand, you photos are irreplaceable and need to be backed up.
 
> I really do need to set up RAID at least for some stuff that I may not
> be able to get back.  Some videos I have are no longer available.

RAID is not a backup solution.

> What
> I wish, I had a second puter in a outbuilding that I could copy to over
> ethernet or something.  May help in the event of a house fire etc.

You have, it's called Amazon S3 :) It's a lot cheaper than a second
computer, and a lot more reliable.
 

-- 
Neil Bothwick

Two rights don't make a wrong, they make an airplane.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 799 bytes --]

On Monday 04 May 2015 08:46:26 Neil Bothwick wrote:
> On Mon, 04 May 2015 02:39:10 -0500, Dale wrote:

> > I really do need to set up RAID at least for some stuff that I may not
> > be able to get back.  Some videos I have are no longer available.
> 
> RAID is not a backup solution.

Not only RAID 1 isn't a back up solution, because it offers temporary 
redundancy rather than diverse protection, but under certain scenarios you 
have a much higher chance of losing your data when the first drive fails.  If 
you bought two (or more) drives at the same time and built a RAID from them, 
their failure performance due to same construction and age could be quite 
similar.  On many occasions your last healthy drive fails, just as you try to 
rebuild the RAID.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Dale]

[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]

Neil Bothwick wrote:
> On Mon, 04 May 2015 02:39:10 -0500, Dale wrote:
>
>>> I wasn't suggesting symlinks, just LVs mounted at appropriate points.
>>> It rather depends on the spread of Dale's data. If he just needs
>>> extra space for his videos, he could get a new drive and mount it at
>>> ~/videos.
>>
>> The bulk of the space is used by the videos.  It's everything from TV
>> shows to movies to youtube howtos.  I'm using roughly 1.8TB on the drive
>> and the videos take up roughly 1.7TB of that space.  My camera pics only
>> use 21GBs of space.  Rest is basically a rounding error.  :/
>
> You need to separate those anyway, for backup purposes. Anything you
> downloaded, you can usually download again, so you only need a list of
> the files to be able to find them again.
>
> On the other hand, you photos are irreplaceable and need to be backed up.

Well, some videos aren't available either.  I'd hate to know I had to
find some of the ones that are available.  Some take some diggin IF I
can even remember some of them.

My pics I backup to DVDs, two sets just in case.  I keep those in a
outbuilding.  If everything here burns, I'm likely gone anyway.


>
> 
>> I really do need to set up RAID at least for some stuff that I may not
>> be able to get back.  Some videos I have are no longer available.
>
> RAID is not a backup solution.

True but at least it would help if a drive fails.  I've been there a
couple times.

>
>
>> What
>> I wish, I had a second puter in a outbuilding that I could copy to over
>> ethernet or something.  May help in the event of a house fire etc.
>
> You have, it's called Amazon S3 :) It's a lot cheaper than a second
> computer, and a lot more reliable.
> 
>


My internet is way to slow for that.  It would take weeks maybe a month
to upload all this stuff.  I have DSL but it is the basic package.  If I
were on cable or had a real fast DSL, maybe.  Thing is, I really don't
want some of my stuff on the internet anyway.  ;-)

I'll come up with something tho.

Dale

:-)  :-)


[-- Attachment #2: Type: text/html, Size: 3087 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Dale]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

Mick wrote:
> On Monday 04 May 2015 08:46:26 Neil Bothwick wrote:
>> On Mon, 04 May 2015 02:39:10 -0500, Dale wrote:
>
>>> I really do need to set up RAID at least for some stuff that I may not
>>> be able to get back.  Some videos I have are no longer available.
>>
>> RAID is not a backup solution.
>
> Not only RAID 1 isn't a back up solution, because it offers temporary
> redundancy rather than diverse protection, but under certain scenarios
you
> have a much higher chance of losing your data when the first drive
fails.  If
> you bought two (or more) drives at the same time and built a RAID from
them,
> their failure performance due to same construction and age could be quite
> similar.  On many occasions your last healthy drive fails, just as you
try to
> rebuild the RAID.
>


I think this has happened to folks on this list.  I've read about this
somewhere before.  It makes sense too.  I'd like to have two different
brands of drives if I could.  That should spread things out, maybe.

Dale

:-)  :-)


[-- Attachment #2: Type: text/html, Size: 1596 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

On Mon, 04 May 2015 03:23:48 -0500, Dale wrote:

> >> What
> >> I wish, I had a second puter in a outbuilding that I could copy to
> >> over ethernet or something.  May help in the event of a house fire
> >> etc.  
> >
> > You have, it's called Amazon S3 :) It's a lot cheaper than a second
> > computer, and a lot more reliable.

> My internet is way to slow for that.  It would take weeks maybe a month
> to upload all this stuff.  I have DSL but it is the basic package.  If I
> were on cable or had a real fast DSL, maybe.  Thing is, I really don't
> want some of my stuff on the internet anyway.  ;-)

You only need to upload it once, so it doesn't really matter how long it
takes. After that you do incremental backups. I use app-backup/duplicity
which not only takes care of incremental backups and communicating with
S3, but also encrypts everything with GPG. No one would know you were
uploading goat porn :)


-- 
Neil Bothwick

When there's a will, I want to be in it.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Dale]

Neil Bothwick wrote:
> On Mon, 04 May 2015 03:23:48 -0500, Dale wrote:
>
>>>> What
>>>> I wish, I had a second puter in a outbuilding that I could copy to
>>>> over ethernet or something.  May help in the event of a house fire
>>>> etc.  
>>> You have, it's called Amazon S3 :) It's a lot cheaper than a second
>>> computer, and a lot more reliable.
>> My internet is way to slow for that.  It would take weeks maybe a month
>> to upload all this stuff.  I have DSL but it is the basic package.  If I
>> were on cable or had a real fast DSL, maybe.  Thing is, I really don't
>> want some of my stuff on the internet anyway.  ;-)
> You only need to upload it once, so it doesn't really matter how long it
> takes. After that you do incremental backups. I use app-backup/duplicity
> which not only takes care of incremental backups and communicating with
> S3, but also encrypts everything with GPG. No one would know you were
> uploading goat porn :)
>
>


It may be only once but it would be a very large once plus I'm on my
puter a lot.  Uploading slows my surfing to almost a dead stop.  Newegg
is a nightmare for me to surf on.  Slowest thing I ever seen. Newegg
isn't alone tho.

Dale

:-)  :-) 

Re: [gentoo-user] Hard drive storage questions

[Alan Mackenzie]

Hello, Dale.

On Mon, May 04, 2015 at 03:23:48AM -0500, Dale wrote:
> Neil Bothwick wrote:

> >> What I wish, I had a second puter in a outbuilding that I could copy
> >> to over ethernet or something.  May help in the event of a house
> >> fire etc.

> > You have, it's called Amazon S3 :) It's a lot cheaper than a second
> > computer, and a lot more reliable.

> My internet is way to slow for that.  It would take weeks maybe a month
> to upload all this stuff.  I have DSL but it is the basic package.  If I
> were on cable or had a real fast DSL, maybe.  Thing is, I really don't
> want some of my stuff on the internet anyway.  ;-)

For the stuff you don't want on the internet, encrypt it!  I've recently
started using ccrypt.  It takes MUCH less time to encrypt things than it
does to transmit them over the net to a server - for my ~4.6 Gb backup,
it takes about 3 minutes to encrypt.  Sending it to my backup server then
takes the best par of an hour (at 10 Mbit/s upload speed).

I suspect your upload speed is way less, but if you had a few hundred
megabytes of really special stuff, this route might be useful.

> I'll come up with something tho.

> Dale

> :-)  :-)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

On Mon, 04 May 2015 05:40:25 -0500, Dale wrote:

> > You only need to upload it once, so it doesn't really matter how long
> > it takes. After that you do incremental backups. I use
> > app-backup/duplicity which not only takes care of incremental backups
> > and communicating with S3, but also encrypts everything with GPG. No
> > one would know you were uploading goat porn :)

> It may be only once but it would be a very large once plus I'm on my
> puter a lot. 

You have to sleep some time, your computer doesn't :)

> Uploading slows my surfing to almost a dead stop.  Newegg
> is a nightmare for me to surf on.  Slowest thing I ever seen. Newegg
> isn't alone tho.

As long as you restrict the upload speed to around 80-80% of your
available upstream bandwidth, it shouldn't affect downloading
significantly. It's when you saturate the upstream that your downloads
are affected.


-- 
Neil Bothwick

Computer apathy error: don't bother striking any key.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Mon, May 4, 2015 at 6:31 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Mon, 04 May 2015 03:23:48 -0500, Dale wrote:
>
>> >> What
>> >> I wish, I had a second puter in a outbuilding that I could copy to
>> >> over ethernet or something.  May help in the event of a house fire
>> >> etc.
>> >
>> > You have, it's called Amazon S3 :) It's a lot cheaper than a second
>> > computer, and a lot more reliable.
>
>> My internet is way to slow for that.  It would take weeks maybe a month
>> to upload all this stuff.  I have DSL but it is the basic package.  If I
>> were on cable or had a real fast DSL, maybe.  Thing is, I really don't
>> want some of my stuff on the internet anyway.  ;-)
>
> You only need to upload it once, so it doesn't really matter how long it
> takes. After that you do incremental backups. I use app-backup/duplicity
> which not only takes care of incremental backups and communicating with
> S3, but also encrypts everything with GPG. No one would know you were
> uploading goat porn :)

I tend to use a few strategies.

Typical stuff in /home, /etc: duplicity daily backups to S3.  It is
small, and safe.  Oh, and it is all on RAID too, which reduces the
risk of needing to actually restore it (RAID is primarily about
downtime, not backup).  Encryption keys are burned to multiple CDs and
stored in multiple safe places.

Photos and other valuable media:  Also gets the duplicity S3
treatment, but after every few GB I do a one-time upload to Glacier
and then remove it from my daily backups.  This stuff is write-once,
so backing it up daily is overkill.  When S3 was more expensive I
would burn two copies to DVD and store offsite, but that became a PITA
and Amazon is a lot cheaper now.  If I ever need to restore it it is
unlikely I'd need it all at once, so I can do so slowly and not get
killed by fees.

MythTV recordings, random video from internet, etc:  btrfs raid plus a
second daily rsync to ext4 (still local).  The rsync is only because
I'm still in playing-around mode with btrfs.  Once I trust it fully
I'll drop it and just rely on the RAID.  I'd be annoyed if I lost all
this stuff, but only for a week or two.  Trying to properly back up
multiple TB of media is just way too expensive and this stuff just
isn't valuable enough to care about.

I structure my filesystem around my backup strategy.  All the stuff I
really care about is in /home.  Stuff I don't care so much about goes
outside of /home and is symlinked back in where necessary.  So, I
don't need to play around with too many exclusion rules.

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Nuno Magalhães]

Greetings gents.

I may have missed it, but i haven't seen this suggested yet: RAID+LVM.
If you already have a 3TB drive, buy another (or two more) and build a
RAID1 or 5 array on them. Then build your LVM on top of /dev/md0 (or
whatever device your raid is).

Another approach is ZFS with RAID-Z or similar. I don't know how/if
ZFS splits data among the drives, but i assume it's wise enough to do
so in a way similar to a RAID+LVM combo.

If going RAID, make sure the rpm and cache are the same for
performance's sake, and you can mix and match drives from different
vendors (perhaps you should, to add to the redundancy).

I don't know about btrfs, seems like it's still in a testing-phase so
i'm not touching it yet.

Just my 2¢

Cheers,
Nuno

Re: [gentoo-user] Hard drive storage questions

[Walter Dnes]

On Mon, May 04, 2015 at 02:23:55AM -0500, Dale wrote
> Dale wrote:
> 
> Well, I read replies a few times and I think it is best to just add a
> new drive.  Heck, I've already had a 3TB drive to fail.  Anyway, I also
> need to look into some sort of backup system.  I used to do this with
> DVDs but with this much "stuff", that just isn't a good idea, not to
> mention that DVDs have their own issues.   I may take a peek into a RAID
> setup since really, that is about the best if not only way to do it. 

  How often do you need to refresh your backups?  And how much does a
medium-size safety-deposit box cost in your area?  Would the bank object
to you going into your safety-deposit box once a month?  Here's a plan...

1 computer with a main drive, and 2 backup drives.  The backup drives
are either removable internals, or standalone externals.  In either
case, they would have to fit inside the safety deposit box.

  Month #

1) make duplicate backups of your machine to both backup drives, and
stick 1 into the bank safety-deposit box, and keep the other at home

2) - update your home backup
   - take it to the bank
   - swap it with the other drive
   - bring the other drive home and update that backup immediately

3) all succeeding months... GOTO 2 (rinse; lather; repeat)

  No worry about uploading terabytes of data over a slow ADSL link.
This is basically a reprise of Andrew Tanenbaum's quote...

> Never underestimate the bandwidth of a station wagon full of tapes
> hurtling down the highway.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Hard drive storage questions

[Alan McKinnon]

On 04/05/2015 20:42, Nuno Magalhães wrote:
> Greetings gents.
> 
> I may have missed it, but i haven't seen this suggested yet: RAID+LVM.
> If you already have a 3TB drive, buy another (or two more) and build a
> RAID1 or 5 array on them. Then build your LVM on top of /dev/md0 (or
> whatever device your raid is).
> 
> Another approach is ZFS with RAID-Z or similar. I don't know how/if
> ZFS splits data among the drives, but i assume it's wise enough to do
> so in a way similar to a RAID+LVM combo.

That's a good analogy. ZFS spreads it's data out across multiple drives
using an n-drives for data plus m-drives for parity type setup. And the
whole lot forms a storage pool.

The main difference is that ZFS does all it's checksumming in software,
and it's one monolithic system. You can think of it being sort of a
combination of RAID and LVM features, but it's not implemented that way.

> If going RAID, make sure the rpm and cache are the same for
> performance's sake, and you can mix and match drives from different
> vendors (perhaps you should, to add to the redundancy).
> 
> I don't know about btrfs, seems like it's still in a testing-phase so
> i'm not touching it yet.
> 
> Just my 2¢
> 
> Cheers,
> Nuno
> 


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Mon, May 4, 2015 at 2:42 PM, Nuno Magalhães <nunomagalhaes@eu.ipp.pt> wrote:
>
> Another approach is ZFS with RAID-Z or similar. I don't know how/if
> ZFS splits data among the drives, but i assume it's wise enough to do
> so in a way similar to a RAID+LVM combo.
>...
>
> I don't know about btrfs, seems like it's still in a testing-phase so
> i'm not touching it yet.

My understanding is that both zfs and btrfs on linux are fairly
experimental.  The codebase for zfs is much more mature in general,
though its integration on Linux is recent.  The codebase for btrfs
changes rapidly, with quite a few regressions.  I've never
irrecoverably lost data on btrfs, but it wouldn't be my first choice
for a production environment unless I basically did my own QC on the
kernel.  However, all my important data is on btrfs nonetheless (with
a full backup to ext4 daily right now).

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1273 bytes --]

On Tue, 5 May 2015 06:56:20 -0400, Rich Freeman wrote:

> > I don't know about btrfs, seems like it's still in a testing-phase so
> > i'm not touching it yet.  
> 
> My understanding is that both zfs and btrfs on linux are fairly
> experimental.  The codebase for zfs is much more mature in general,
> though its integration on Linux is recent.

It's also based on an older version of ZFS, so we can expect stability to
improve where necessary, but little in the way of new features
(unless that has changed since I last used it and Sun have open sourced a
later release).

> The codebase for btrfs
> changes rapidly, with quite a few regressions.  I've never
> irrecoverably lost data on btrfs, but it wouldn't be my first choice
> for a production environment unless I basically did my own QC on the
> kernel.  However, all my important data is on btrfs nonetheless (with
> a full backup to ext4 daily right now).

I have a similar approach, although with duplicity backups to a file
server. I have had a couple of problems with btrfs on my laptop,
connected with unclean shutdowns. I didn't lose any data but the repair
process took a *long* time.


-- 
Neil Bothwick

The considered application of terror is also a form of communication.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1751 bytes --]

On Tuesday 05 May 2015 12:33:38 Neil Bothwick wrote:
> On Tue, 5 May 2015 06:56:20 -0400, Rich Freeman wrote:
> > > I don't know about btrfs, seems like it's still in a testing-phase so
> > > i'm not touching it yet.
> > 
> > My understanding is that both zfs and btrfs on linux are fairly
> > experimental.  The codebase for zfs is much more mature in general,
> > though its integration on Linux is recent.
> 
> It's also based on an older version of ZFS, so we can expect stability to
> improve where necessary, but little in the way of new features
> (unless that has changed since I last used it and Sun have open sourced a
> later release).
> 
> > The codebase for btrfs
> > changes rapidly, with quite a few regressions.  I've never
> > irrecoverably lost data on btrfs, but it wouldn't be my first choice
> > for a production environment unless I basically did my own QC on the
> > kernel.  However, all my important data is on btrfs nonetheless (with
> > a full backup to ext4 daily right now).
> 
> I have a similar approach, although with duplicity backups to a file
> server. I have had a couple of problems with btrfs on my laptop,
> connected with unclean shutdowns. I didn't lose any data but the repair
> process took a *long* time.

During a backup of a home directory I noticed loads of Chromium and Firefox 
crash/recovery files being copied over.  However, I don't know if this is a 
btrfs problem, or the fact that I had to forcefully shut down KDE once or 
twice recently, because the desktop would not logout/shutdown normally.

The fsck which ran when the machine rebooted did not revealed any problems.  
Is there a different recommended way for checking for fs errors?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

On Tue, 5 May 2015 13:05:55 +0100, Mick wrote:

> During a backup of a home directory I noticed loads of Chromium and
> Firefox crash/recovery files being copied over.  However, I don't know
> if this is a btrfs problem, or the fact that I had to forcefully shut
> down KDE once or twice recently, because the desktop would not
> logout/shutdown normally.

Chromium saves its open tabs and reopens them after a reboot, even if it
is shut down forcibly.
 
> The fsck which ran when the machine rebooted did not revealed any
> problems. Is there a different recommended way for checking for fs
> errors?

btrfs check - it needs the filesystem to be unmounted and has a repair
option.


-- 
Neil Bothwick

Do you steal taglines too?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1084 bytes --]

On Tuesday 05 May 2015 13:21:47 Neil Bothwick wrote:
> On Tue, 5 May 2015 13:05:55 +0100, Mick wrote:
> > During a backup of a home directory I noticed loads of Chromium and
> > Firefox crash/recovery files being copied over.  However, I don't know
> > if this is a btrfs problem, or the fact that I had to forcefully shut
> > down KDE once or twice recently, because the desktop would not
> > logout/shutdown normally.
> 
> Chromium saves its open tabs and reopens them after a reboot, even if it
> is shut down forcibly.
> 
> > The fsck which ran when the machine rebooted did not revealed any
> > problems. Is there a different recommended way for checking for fs
> > errors?
> 
> btrfs check - it needs the filesystem to be unmounted and has a repair
> option.

Right, I didn't run the repair option, thinking that this is only needed if 
one receives error reports during a checking cycle.  Admittedly I am also 
being fearful of breaking filesystems (I was traumatised repeatedly when 
trying to repair reiser4 back when ... LOL! ).

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Tue, May 5, 2015 at 8:21 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Tue, 5 May 2015 13:05:55 +0100, Mick wrote:
>
>> During a backup of a home directory I noticed loads of Chromium and
>> Firefox crash/recovery files being copied over.  However, I don't know
>> if this is a btrfs problem, or the fact that I had to forcefully shut
>> down KDE once or twice recently, because the desktop would not
>> logout/shutdown normally.
>
> Chromium saves its open tabs and reopens them after a reboot, even if it
> is shut down forcibly.
>
>> The fsck which ran when the machine rebooted did not revealed any
>> problems. Is there a different recommended way for checking for fs
>> errors?
>
> btrfs check - it needs the filesystem to be unmounted and has a repair
> option.

I don't think the chromium/firefox issues are in any way a sign of a
filesystem problem.  Application crashing and filesystem errors are
completely different matters.  If atop dumps core 14 times a day (as
it seems to love to do) btrfs just happily stores them in case I ever
want to look at them.

In general btrfs tends to do most of its fixing online.  I'd only run
btrfs check if the filesystem is unmountable.  I wouldn't trust it not
to do more harm than good.  For a very long time it didn't even exist,
and btrfs is a bit different from most other filesystems in this
regard.  btrfs doesn't complain if you mount it unclean - almost all
the recovery code is in the kernel and it will generally tidy up as it
goes.  This is in contrast to many other filesystems that force you to
run fsck if they were not cleanly unmounted.

I'm not saying it is broken.  I haven't really used it much.  However,
for the most part btrfs was designed around doing most of its
operations online and these are probably the more mature code paths.

That said, btrfs check without the --repair option should be
read-only, so you can always try it.  However, I wouldn't be surprised
at all if there are no problems with your filesystem (assuming you run
it after a clean shutdown).  If there were any problems, btrfs should
have cleaned them up on your last mount.  btrfs does not overwrite
data in-place in any case, so it is a bit like ext4 with data=journal
in normal operation.  And that is what I like about btrfs (and the
same applies to zfs) - the basic design of the filesystem tends to
prioritize data integrity, and thus even with all my panics and
mounting problems with btrfs, I've always been able to recover, and at
every point I could at least mount the filesystems read-only and read
everything off of them.  And a successful read on btrfs/zfs means that
the checksum matched, so the risk of corruption is fairly low.

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

On Tue, 5 May 2015 08:53:29 -0400, Rich Freeman wrote:

> In general btrfs tends to do most of its fixing online.  I'd only run
> btrfs check if the filesystem is unmountable.

That's the only time I've had to use it. This was on a laptop with a
single SSD, so there was no where to sync good data from. It worked
without data loss but was very slow, around 12 hours for a 250GB
filesystem on one occasion.


-- 
Neil Bothwick

Top Oxymorons Number 30: Business ethics

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Bill Kenworthy]

On 06/05/15 05:50, Neil Bothwick wrote:
> On Tue, 5 May 2015 08:53:29 -0400, Rich Freeman wrote:
> 
>> In general btrfs tends to do most of its fixing online.  I'd only run
>> btrfs check if the filesystem is unmountable.
> 
> That's the only time I've had to use it. This was on a laptop with a
> single SSD, so there was no where to sync good data from. It worked
> without data loss but was very slow, around 12 hours for a 250GB
> filesystem on one occasion.
> 
> 

I have two btrfs raid1 multidisk arrays (3x and 5x 2T drives) that take
"forever" to fsck and  after a power crash (had two in 2 days :( appear
to hang at the fsck stage on reboot (no disk activity and some hours
later still none.)  Ive taken to booting from a usb key and editing
fstab to get a full start and manually fsck - awkward.

Its been quite awhile now since I've found problems either with an fsck
or a scrub after abusing the arrays so I would like to force a degraded
mount on reboot after a crash. I am using genkernel and openrc - where
can I to modify the boot scripts? (looks like
/usr/share/genkernel/defaults/linuxrc but the actual place to add the
btrfs options isn't obvious)

BillK

Re: [gentoo-user] Hard drive storage questions

[Bill Kenworthy]

On 06/05/15 06:21, Bill Kenworthy wrote:
> On 06/05/15 05:50, Neil Bothwick wrote:
>> On Tue, 5 May 2015 08:53:29 -0400, Rich Freeman wrote:
>>
>>> In general btrfs tends to do most of its fixing online.  I'd only run
>>> btrfs check if the filesystem is unmountable.
>>
>> That's the only time I've had to use it. This was on a laptop with a
>> single SSD, so there was no where to sync good data from. It worked
>> without data loss but was very slow, around 12 hours for a 250GB
>> filesystem on one occasion.
>>
>>
> 
> I have two btrfs raid1 multidisk arrays (3x and 5x 2T drives) that take
> "forever" to fsck and  after a power crash (had two in 2 days :( appear
> to hang at the fsck stage on reboot (no disk activity and some hours
> later still none.)  Ive taken to booting from a usb key and editing
> fstab to get a full start and manually fsck - awkward.
> 
> Its been quite awhile now since I've found problems either with an fsck
> or a scrub after abusing the arrays so I would like to force a degraded
> mount on reboot after a crash. I am using genkernel and openrc - where
> can I to modify the boot scripts? (looks like
> /usr/share/genkernel/defaults/linuxrc but the actual place to add the
> btrfs options isn't obvious)
> 
> BillK
> 
> 

Hmm ... looks like its actually /etc/init.d/localmount ...

BillK

Re: [gentoo-user] Hard drive storage questions

[Dale]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Neil Bothwick wrote:
> On Mon, 04 May 2015 05:40:25 -0500, Dale wrote:
>
>>> You only need to upload it once, so it doesn't really matter how long
>>> it takes. After that you do incremental backups. I use
>>> app-backup/duplicity which not only takes care of incremental backups
>>> and communicating with S3, but also encrypts everything with GPG. No
>>> one would know you were uploading goat porn :)
>
>> It may be only once but it would be a very large once plus I'm on my
>> puter a lot.
>
> You have to sleep some time, your computer doesn't :)

A lot of the time, I'm downloading a list of movies while I am
sleeping.  That's when I do most of my downloading.  I use download
helper.  Sometimes it can download for several hours.  There are times
when I nap and when I wake up, it is still downloading.  ;-)  I do the
same when I leave to go to town too.


>
>
>> Uploading slows my surfing to almost a dead stop.  Newegg
>> is a nightmare for me to surf on.  Slowest thing I ever seen. Newegg
>> isn't alone tho.
>
> As long as you restrict the upload speed to around 80-80% of your
> available upstream bandwidth, it shouldn't affect downloading
> significantly. It's when you saturate the upstream that your downloads
> are affected.
>
>

I don't know how to limit that.  Still, I have a really slow upload
speed.  While I wouldn't want to lose some of it, it also would be a lot
of trouble given the large volume of data.  I'd much prefer something
local and much faster.  Now for my camera pics, that could be a option. 
Much less data and lots more important too.  I'm assuming this is what
you are talking about?

https://aws.amazon.com/s3/

I'm trying to figure out just how much this would cost here.  o_O  Just
for my pics tho.

Dale

:-)  :-)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlVN58wACgkQiBoxVpK2GMAqfwCeMSL9IrGPJl5gJrI4crhk0SZn
K0MAnig1nRYvR3lB9fY8b/+ou+K3NjCh
=0DMG
-----END PGP SIGNATURE-----

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Sat, May 9, 2015 at 6:56 AM, Dale <rdalek1967@gmail.com> wrote:
>
> https://aws.amazon.com/s3/
>
> I'm trying to figure out just how much this would cost here.  o_O  Just
> for my pics tho.
>

It works out to 1-3 cents/GB/month, depending on storage tier.
Glacier is cheapest and very secure (or so they claim), but you will
pay more to retrieve the data if you need it.  If you aren't using
RAID then I probably wouldn't use glacier since it is very likely that
you'll be doing retrievals on occasion.  The most expensive figure
costs you 10c/GB to retrieve, and should be secure (again, their
claims).  The in-between figure is for reduced redundancy - it also
costs 10c/GB to retrieve, but is less secure.


I typically use the mid-cost reduced-redundancy option, since this is
intended solely as a backup.  If I were archiving data and not keeping
a copy locally I would not use reduced-redundancy.  As a backup, it is
already redundant - what are the odds of my house and the Amazon
datacenter having a disaster on the same day?  Otherwise, if their
datacenter burns down and the data disappears, then on the next day
duplicity will simply do another full backup and I'm protected again.

One thing you can't cheaply do with Amazon is verify your backups.
Duplicity will happily check the data files against the manifest
hashes with a simple command, but it will cost you 10c/GB for whatever
you verify, since it will need to be transferred out.  I guess another
option is to launch an EC2 instance with duplicity on it and have it
do the verify.  That would be an internal Amazon transfer which is
both free and much faster, but it will cost you a few cents per hour
for the CPU time.  I also don't know if duplicity can verify a backup
without the encryption keys - if it can't then you'll have to upload
your keys to EC2 which means Amazon could read your backups if they
wanted to.  Otherwise duplicity is encrypting locally and all Amazon
does is store a bunch of encrypted data and regurgitate it on demand.

--
Rich

Re: [gentoo-user] Hard drive storage questions

[Todd Goodman]

* Rich Freeman <rich0@gentoo.org> [150509 09:00]:
[..SNIP..]
> One thing you can't cheaply do with Amazon is verify your backups.
> Duplicity will happily check the data files against the manifest
> hashes with a simple command, but it will cost you 10c/GB for whatever
> you verify, since it will need to be transferred out.  I guess another
> option is to launch an EC2 instance with duplicity on it and have it
> do the verify.  That would be an internal Amazon transfer which is
> both free and much faster, but it will cost you a few cents per hour
> for the CPU time.  I also don't know if duplicity can verify a backup
> without the encryption keys - if it can't then you'll have to upload
> your keys to EC2 which means Amazon could read your backups if they
> wanted to.  Otherwise duplicity is encrypting locally and all Amazon
> does is store a bunch of encrypted data and regurgitate it on demand.
> 
> --
> Rich

Thanks for the great post Rich.

As for keys, you could use Amazon's AWS Key Management Service.
Of course they could be sitting there gathering keys, but at some point
you either have to trust they'll do what they say or simply decide not
to use them at all (IMNHO.)

You could also use AWS Key Management for backup data you want
"reasonably" secured and then your own keys for data you want more
highly secured (hopefully much smaller so the verify costs are more
reasonable.)

Todd

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Sat, May 9, 2015 at 10:46 AM, Todd Goodman <tsg@bonedaddy.net> wrote:
>
> As for keys, you could use Amazon's AWS Key Management Service.
> Of course they could be sitting there gathering keys, but at some point
> you either have to trust they'll do what they say or simply decide not
> to use them at all (IMNHO.)

That is really intended more for credentials used for hosted systems
to communicate with other services/each other/etc.  If you have to
have your credentials in the cloud, then you might as well have a
somewhat secure way to manage them.  However, that is clearly inferior
to not putting credentials in the cloud in the first place.

>
> You could also use AWS Key Management for backup data you want
> "reasonably" secured and then your own keys for data you want more
> highly secured (hopefully much smaller so the verify costs are more
> reasonable.)
>

I just don't frequently verify my backups.  I'm willing to trust
Amazon to have my data when I ask for it.  That is their entire
business model with S3 and they're probably one of the stronger links
in the data security chain.  If I'm going to be paranoid about that,
I'm going to probably have other things I'd prefer to improve first.

I keep copies of my backup keys in a few places.  My thread model is
somebody hacking my account looking for personal data
(finances/keys/whatever).  If they hack into Amazon they won't have
the necessary keys.  If somebody manages to steal one of my keys in
safekeeping elsewhere, they won't have access to any of the data
encrypted using the key.  If the NSA or whoever is going to access my
Amazon data and also ask my bank to open my safe deposit box or
whatever, then more power to them.  I run a tor node, so they've
probably rooted my box anyway.


-- 
Rich

[gentoo-user] Hard drive storage questions

[Dale]

Howdy to all,

I have a interesting problem coming up.  Currently, I have two 3TB
drives for my /home mount point.  A lot of this is videos but some pdf
files and other documents as well plus a photo collection of family
stuff etc. 

Filesystem                                   Size  Used Avail Use%
Mounted on
/dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home

I've got a little over 25% or so of usable space left.  At that point or
shortly thereafter, it could start causing some issues according to what
I've read anyway.  Either way, shortly after that, being full will
certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
if I buy a larger drive or drives, I'd have to unplug one to move things
over and likely repeat that a few times.  I could do that and likely
will have to anyway but I'm trying to think a little farther ahead. 
Currently I have the more important stuff backed up to a external single
6TB USB drive, previous thread on that.  I'm trying to come up with a
plan that allows me to grow easier and without having to worry about
running out of motherboard based ports. 

I thought about a store bought enclosure with more than one hard drive
that connects by ethernet.  The ones I've found are fairly expensive. 
Doing it over USB concerns me for other reasons, USB isn't always that
stable.  So, internal isn't working out to well long term.  Ethernet
based is expensive, what I could find anyway.  USB isn't that stable. 

I'm planning to upgrade my current system.  Upgrade the CPU, memory and
maybe even the video card as well.  I thought about using a older spare
motherboard, those removed components and building a mini system sort of
thing.  I could have one small drive for a OS and then add large drives
for storage.  Then I can access those from my main system, ethernet I
would guess.  Even then, I'd still be limited to the SATA ports on the
MOBO at some point but it would be a ways into the future.  So far tho,
this is one of the better ideas.  So far.  Does anyone else have other
ideas on how to do this?  Some method that I've never heard of but
doesn't cost a lot of money to do? 

Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
If you need more info, let me know. 

Dale

:-)  :-) 

Re: [gentoo-user] Hard drive storage questions

[Jack]

On 2018.11.08 20:16, Dale wrote:
> Howdy to all,
> 
> I have a interesting problem coming up.  Currently, I have two 3TB
> drives for my /home mount point.  A lot of this is videos but some pdf
> files and other documents as well plus a photo collection of family
> stuff etc. 
> 
> Filesystem                                   Size  Used Avail Use%
> Mounted on
> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
> 
> I've got a little over 25% or so of usable space left.  At that point  
> or
> shortly thereafter, it could start causing some issues according to  
> what
> I've read anyway.  Either way, shortly after that, being full will
> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
> if I buy a larger drive or drives, I'd have to unplug one to move  
> things
> over and likely repeat that a few times.  I could do that and likely
> will have to anyway but I'm trying to think a little farther ahead. 
> Currently I have the more important stuff backed up to a external  
> single
> 6TB USB drive, previous thread on that.  I'm trying to come up with a
> plan that allows me to grow easier and without having to worry about
> running out of motherboard based ports. 
> 
> I thought about a store bought enclosure with more than one hard drive
> that connects by ethernet.  The ones I've found are fairly expensive. 
> Doing it over USB concerns me for other reasons, USB isn't always that
> stable.  So, internal isn't working out to well long term.  Ethernet
> based is expensive, what I could find anyway.  USB isn't that stable. 
> 
> I'm planning to upgrade my current system.  Upgrade the CPU, memory  
> and
> maybe even the video card as well.  I thought about using a older  
> spare
> motherboard, those removed components and building a mini system sort  
> of
> thing.  I could have one small drive for a OS and then add large  
> drives
> for storage.  Then I can access those from my main system, ethernet I
> would guess.  Even then, I'd still be limited to the SATA ports on the
> MOBO at some point but it would be a ways into the future.  So far  
> tho,
> this is one of the better ideas.  So far.  Does anyone else have other
> ideas on how to do this?  Some method that I've never heard of but
> doesn't cost a lot of money to do? 
> 
> Thanks in advance for any ideas.  I can't think of anything else.   
> ;-) 
> If you need more info, let me know. 
> 
> Dale
> 
If you have space on the mobo for another card, you should be able to  
get an additional SATA card.  I have no idea on prices, but I'd be  
surprised if it's prohibitive.

Jack

Re: [gentoo-user] Hard drive storage questions

[Dale]

Jack wrote:
> On 2018.11.08 20:16, Dale wrote:
>> Howdy to all,
>>
>> I have a interesting problem coming up.  Currently, I have two 3TB
>> drives for my /home mount point.  A lot of this is videos but some pdf
>> files and other documents as well plus a photo collection of family
>> stuff etc. 
>>
>> Filesystem                                   Size  Used Avail Use%
>> Mounted on
>> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>>
>> I've got a little over 25% or so of usable space left.  At that point or
>> shortly thereafter, it could start causing some issues according to what
>> I've read anyway.  Either way, shortly after that, being full will
>> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
>> if I buy a larger drive or drives, I'd have to unplug one to move things
>> over and likely repeat that a few times.  I could do that and likely
>> will have to anyway but I'm trying to think a little farther ahead. 
>> Currently I have the more important stuff backed up to a external single
>> 6TB USB drive, previous thread on that.  I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports. 
>>
>> I thought about a store bought enclosure with more than one hard drive
>> that connects by ethernet.  The ones I've found are fairly expensive. 
>> Doing it over USB concerns me for other reasons, USB isn't always that
>> stable.  So, internal isn't working out to well long term.  Ethernet
>> based is expensive, what I could find anyway.  USB isn't that stable. 
>>
>> I'm planning to upgrade my current system.  Upgrade the CPU, memory and
>> maybe even the video card as well.  I thought about using a older spare
>> motherboard, those removed components and building a mini system sort of
>> thing.  I could have one small drive for a OS and then add large drives
>> for storage.  Then I can access those from my main system, ethernet I
>> would guess.  Even then, I'd still be limited to the SATA ports on the
>> MOBO at some point but it would be a ways into the future.  So far tho,
>> this is one of the better ideas.  So far.  Does anyone else have other
>> ideas on how to do this?  Some method that I've never heard of but
>> doesn't cost a lot of money to do? 
>>
>> Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
>> If you need more info, let me know. 
>>
>> Dale
>>
> If you have space on the mobo for another card, you should be able to
> get an additional SATA card.  I have no idea on prices, but I'd be
> surprised if it's prohibitive.
>
> Jack
>


I hadn't thought of adding a SATA card.  I have a ethernet card and the
video card and I don't think there are any others.  I should have some
open slots there.  Well, that is one idea that I hadn't thought of. 
lol  Since I have a LARGE case, Cooler Master HAF-932, I have space for
more drives.  I think this thing holds like nine or ten pretty easy. 

Thanks.  Another option to look into. 

Dale

:-)  :-) 

Re: [gentoo-user] Hard drive storage questions

[Andrew Lowe]

[-- Attachment #1: Type: text/plain, Size: 4054 bytes --]

    Firstly sorry about the top post, on the phone.

    I've had the same sort of thing happen to me. I was lucky to have available sata ports so bought two WD 8TB video archive drives and attached them to the MB. The card idea from the previous post is basically the same thing. 

    In turn the archive drives will spin down after about 10 minutes, you will hear this happen and is a bit disconcerting the first couple of times, and use hardly any power. In turn if you need anything from the drives it takes just a few seconds to spin up again.

    In comparison to a small NAS thingy, I'm way ahead. No purchase of bare NAS, and much lower power consumption. Much higher data transfer rates as well. Having said that, I may have to buy a NAS in the long run as a backup for other purposes.

      Andrew

Sent from my phone

-----Original Message-----
From: Dale <rdalek1967@gmail.com>
To: gentoo-user@lists.gentoo.org
Sent: Fri., 09 Nov. 2018 9:43
Subject: Re: [gentoo-user] Hard drive storage questions

Jack wrote:
> On 2018.11.08 20:16, Dale wrote:
>> Howdy to all,
>>
>> I have a interesting problem coming up.  Currently, I have two 3TB
>> drives for my /home mount point.  A lot of this is videos but some pdf
>> files and other documents as well plus a photo collection of family
>> stuff etc. 
>>
>> Filesystem                                   Size  Used Avail Use%
>> Mounted on
>> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>>
>> I've got a little over 25% or so of usable space left.  At that point or
>> shortly thereafter, it could start causing some issues according to what
>> I've read anyway.  Either way, shortly after that, being full will
>> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
>> if I buy a larger drive or drives, I'd have to unplug one to move things
>> over and likely repeat that a few times.  I could do that and likely
>> will have to anyway but I'm trying to think a little farther ahead. 
>> Currently I have the more important stuff backed up to a external single
>> 6TB USB drive, previous thread on that.  I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports. 
>>
>> I thought about a store bought enclosure with more than one hard drive
>> that connects by ethernet.  The ones I've found are fairly expensive. 
>> Doing it over USB concerns me for other reasons, USB isn't always that
>> stable.  So, internal isn't working out to well long term.  Ethernet
>> based is expensive, what I could find anyway.  USB isn't that stable. 
>>
>> I'm planning to upgrade my current system.  Upgrade the CPU, memory and
>> maybe even the video card as well.  I thought about using a older spare
>> motherboard, those removed components and building a mini system sort of
>> thing.  I could have one small drive for a OS and then add large drives
>> for storage.  Then I can access those from my main system, ethernet I
>> would guess.  Even then, I'd still be limited to the SATA ports on the
>> MOBO at some point but it would be a ways into the future.  So far tho,
>> this is one of the better ideas.  So far.  Does anyone else have other
>> ideas on how to do this?  Some method that I've never heard of but
>> doesn't cost a lot of money to do? 
>>
>> Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
>> If you need more info, let me know. 
>>
>> Dale
>>
> If you have space on the mobo for another card, you should be able to
> get an additional SATA card.  I have no idea on prices, but I'd be
> surprised if it's prohibitive.
>
> Jack
>


I hadn't thought of adding a SATA card.  I have a ethernet card and the
video card and I don't think there are any others.  I should have some
open slots there.  Well, that is one idea that I hadn't thought of. 
lol  Since I have a LARGE case, Cooler Master HAF-932, I have space for
more drives.  I think this thing holds like nine or ten pretty easy. 

Thanks.  Another option to look into. 

Dale

:-)  :-) 


[-- Attachment #2: Type: text/html, Size: 7591 bytes --]

Re: [gentoo-user] Hard drive storage questions

[Bill Kenworthy]

On 09/11/18 09:43, Dale wrote:
> Jack wrote:
>> On 2018.11.08 20:16, Dale wrote:
>>> Howdy to all,
>>>
>>> I have a interesting problem coming up.  Currently, I have two 3TB
>>> drives for my /home mount point.  A lot of this is videos but some pdf
>>> files and other documents as well plus a photo collection of family
>>> stuff etc. 
>>>
>>> Filesystem                                   Size  Used Avail Use%
>>> Mounted on
>>> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
>>>
>>> I've got a little over 25% or so of usable space left.  At that point or
>>> shortly thereafter, it could start causing some issues according to what
>>> I've read anyway.  Either way, shortly after that, being full will
>>> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
>>> if I buy a larger drive or drives, I'd have to unplug one to move things
>>> over and likely repeat that a few times.  I could do that and likely
>>> will have to anyway but I'm trying to think a little farther ahead. 
>>> Currently I have the more important stuff backed up to a external single
>>> 6TB USB drive, previous thread on that.  I'm trying to come up with a
>>> plan that allows me to grow easier and without having to worry about
>>> running out of motherboard based ports. 
>>>
>>> I thought about a store bought enclosure with more than one hard drive
>>> that connects by ethernet.  The ones I've found are fairly expensive. 
>>> Doing it over USB concerns me for other reasons, USB isn't always that
>>> stable.  So, internal isn't working out to well long term.  Ethernet
>>> based is expensive, what I could find anyway.  USB isn't that stable. 
>>>
>>> I'm planning to upgrade my current system.  Upgrade the CPU, memory and
>>> maybe even the video card as well.  I thought about using a older spare
>>> motherboard, those removed components and building a mini system sort of
>>> thing.  I could have one small drive for a OS and then add large drives
>>> for storage.  Then I can access those from my main system, ethernet I
>>> would guess.  Even then, I'd still be limited to the SATA ports on the
>>> MOBO at some point but it would be a ways into the future.  So far tho,
>>> this is one of the better ideas.  So far.  Does anyone else have other
>>> ideas on how to do this?  Some method that I've never heard of but
>>> doesn't cost a lot of money to do? 
>>>
>>> Thanks in advance for any ideas.  I can't think of anything else.  ;-) 
>>> If you need more info, let me know. 
>>>
>>> Dale
>>>
>> If you have space on the mobo for another card, you should be able to
>> get an additional SATA card.  I have no idea on prices, but I'd be
>> surprised if it's prohibitive.
>>
>> Jack
>>
>
> I hadn't thought of adding a SATA card.  I have a ethernet card and the
> video card and I don't think there are any others.  I should have some
> open slots there.  Well, that is one idea that I hadn't thought of. 
> lol  Since I have a LARGE case, Cooler Master HAF-932, I have space for
> more drives.  I think this thing holds like nine or ten pretty easy. 
>
> Thanks.  Another option to look into. 
>
> Dale
>
> :-)  :-) 
>
I have used a mini-pcie board from ebay (takes two sata connections)
alongside a number of other connection types in a btrfs raid 10 for some
months as a temporary expansion - worked fine, but make sure to check
Linux compatibility first.


BillK

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Thu, Nov 8, 2018 at 8:16 PM Dale <rdalek1967@gmail.com> wrote:
>
> I'm trying to come up with a
> plan that allows me to grow easier and without having to worry about
> running out of motherboard based ports.
>

So, this is an issue I've been changing my mind on over the years.
There are a few common approaches:

* Find ways to cram a lot of drives on one host
* Use a patchwork of NAS devices or improvised hosts sharing over
samba/nfs/etc and end up with a mess of mount points.
* Use a distributed FS

Right now I'm mainly using the first approach, and I'm trying to move
to the last.  The middle option has never appealed to me.

So, to do more of what you're doing in the most efficient way
possible, I recommend finding used LSI HBA cards.  These have mini-SAS
ports on them, and one of these can be attached to a breakout cable
that gets you 4 SATA ports.  I just picked up two of these for $20
each on ebay (used) and they have 4 mini-SAS ports each, which is
capacity for 16 SATA drives per card.  Typically these have 4x or
larger PCIe interfaces, so you'll need a large slot, or one with a
cutout.  You'd have to do the math but I suspect that if the card+MB
supports PCIe 3.0 you're not losing much if you cram it into a smaller
slot.  If most of the drives are idle most of the time then that also
demands less bandwidth.  16 fully busy hard drives obviously can put
out a lot of data if reading sequentially.

You can of course get more consumer-oriented SATA cards, but you're
lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
you main limiting factor assuming you have power and case space.

Oh, and those HBA cards need to be flashed into "IT" mode - they're
often sold this way, but if they support RAID you want to flash the IT
firmware that just makes them into a bunch of standalone SATA slots.
This is usually a PITA that involves DOS or whatever, but I have
noticed some of the software needed in the Gentoo repo.

If you go that route it is just like having a ton of SATA ports in
your system - they just show up as sda...sdz and so on (no idea where
it goes after that).  Software-wise you just keep doing what you're
already doing (though you should be seriously considering
mdadm/zfs/btrfs/whatever at that point).

That is the more traditional route.

Now let me talk about distributed filesystems, which is the more
scalable approach.  I'm getting tired of being limited by SATA ports,
and cases, and such.  I'm also frustrated with some of zfs's
inflexibility around removing drives.  These are constraints that make
upgrading painful, and often inefficient.  Distributed filesystems
offer a different solution.

A distributed filesystem spreads its storage across many hosts, with
an arbitrary number of drives per host (more or less).  So, you can
add more hosts, add more drives to a host, and so on.  That means
you're never forced to try to find a way to cram a few more drives in
one host.  The resulting filesystem appears as one gigantic filesystem
(unless you want to split it up), which means no mess of nfs
mountpoints and so on, and all the other headaches of nfs.  Just as
with RAID these support redundancy, except now you can lose entire
hosts without issue.  With many you can even tell it which
PDU/rack/whatever each host is plugged into, and it will make sure you
can lose all the hosts in one rack.  You can also mount the filesystem
on as many hosts as you want at the same time.

They do tend to be a bit more complex.  The big players can scale VERY
large - thousands of drives easily.  Everything seems to be moving
towards Ceph/CephFS.  If you were hosting a datacenter full of
VMs/containers/etc I'd be telling you to host it on Ceph.  However,
for small scale (which you definitely are right now), I'm not thrilled
with it.  Due to the way it allocates data (hash-based) anytime
anything changes you end up having to move all the data around in the
cluster, and all the reports I've read suggests it doesn't perform all
that great if you only have a few nodes.  Ceph storage nodes are also
RAM-hungry, and I want to run these on ARM to save power, and few ARM
boards have that kind of RAM, and they're very expensive.

Personally I'm working on deploying a cluster of a few nodes running
LizardFS, which is basically a fork/derivative of MooseFS.  While it
won't scale nearly as well, below 100 nodes should be fine, and in
particular it sounds like it works fairly well with only a few nodes.
It has its pros and cons, but for my needs it should be sufficient.
It also isn't RAM-hungry.  I'm going to be testing it on some
RockPro64s, with the LSI HBAs.

I did note that Gentoo lacks a LizardFS client.  I suspect I'll be
looking to fix that - I'm sure the moosefs ebuild would be a good
starting point.  I'm probably going to be a whimp and run the storage
nodes on Ubuntu or whatever upstream targets - they're basically
appliances as far as I'm concerned.

So, those are the two routes I'd recommend.  Just get yourself an HBA
if you only want a few more drives.  If you see your needs expanding
then consider a distributed filesystem.  The advantage of the latter
is that you can keep expanding it however you want with additional
drives/nodes/whatever.  If you're going over 20 nodes I'd use Ceph for
sure - IMO that seems to be the future of this space.

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Bill Kenworthy]

On 09/11/18 10:29, Rich Freeman wrote:
> On Thu, Nov 8, 2018 at 8:16 PM Dale <rdalek1967@gmail.com> wrote:
>> I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports.
>>
> So, this is an issue I've been changing my mind on over the years.
> There are a few common approaches:
>
> * Find ways to cram a lot of drives on one host
> * Use a patchwork of NAS devices or improvised hosts sharing over
> samba/nfs/etc and end up with a mess of mount points.
> * Use a distributed FS
>
> Right now I'm mainly using the first approach, and I'm trying to move
> to the last.  The middle option has never appealed to me.
>
> So, to do more of what you're doing in the most efficient way
> possible, I recommend finding used LSI HBA cards.  These have mini-SAS
> ports on them, and one of these can be attached to a breakout cable
> that gets you 4 SATA ports.  I just picked up two of these for $20
> each on ebay (used) and they have 4 mini-SAS ports each, which is
> capacity for 16 SATA drives per card.  Typically these have 4x or
> larger PCIe interfaces, so you'll need a large slot, or one with a
> cutout.  You'd have to do the math but I suspect that if the card+MB
> supports PCIe 3.0 you're not losing much if you cram it into a smaller
> slot.  If most of the drives are idle most of the time then that also
> demands less bandwidth.  16 fully busy hard drives obviously can put
> out a lot of data if reading sequentially.
>
> You can of course get more consumer-oriented SATA cards, but you're
> lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
> HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
> you main limiting factor assuming you have power and case space.
>
> Oh, and those HBA cards need to be flashed into "IT" mode - they're
> often sold this way, but if they support RAID you want to flash the IT
> firmware that just makes them into a bunch of standalone SATA slots.
> This is usually a PITA that involves DOS or whatever, but I have
> noticed some of the software needed in the Gentoo repo.
>
> If you go that route it is just like having a ton of SATA ports in
> your system - they just show up as sda...sdz and so on (no idea where
> it goes after that).  Software-wise you just keep doing what you're
> already doing (though you should be seriously considering
> mdadm/zfs/btrfs/whatever at that point).
>
> That is the more traditional route.
>
> Now let me talk about distributed filesystems, which is the more
> scalable approach.  I'm getting tired of being limited by SATA ports,
> and cases, and such.  I'm also frustrated with some of zfs's
> inflexibility around removing drives.  These are constraints that make
> upgrading painful, and often inefficient.  Distributed filesystems
> offer a different solution.
>
> A distributed filesystem spreads its storage across many hosts, with
> an arbitrary number of drives per host (more or less).  So, you can
> add more hosts, add more drives to a host, and so on.  That means
> you're never forced to try to find a way to cram a few more drives in
> one host.  The resulting filesystem appears as one gigantic filesystem
> (unless you want to split it up), which means no mess of nfs
> mountpoints and so on, and all the other headaches of nfs.  Just as
> with RAID these support redundancy, except now you can lose entire
> hosts without issue.  With many you can even tell it which
> PDU/rack/whatever each host is plugged into, and it will make sure you
> can lose all the hosts in one rack.  You can also mount the filesystem
> on as many hosts as you want at the same time.
>
> They do tend to be a bit more complex.  The big players can scale VERY
> large - thousands of drives easily.  Everything seems to be moving
> towards Ceph/CephFS.  If you were hosting a datacenter full of
> VMs/containers/etc I'd be telling you to host it on Ceph.  However,
> for small scale (which you definitely are right now), I'm not thrilled
> with it.  Due to the way it allocates data (hash-based) anytime
> anything changes you end up having to move all the data around in the
> cluster, and all the reports I've read suggests it doesn't perform all
> that great if you only have a few nodes.  Ceph storage nodes are also
> RAM-hungry, and I want to run these on ARM to save power, and few ARM
> boards have that kind of RAM, and they're very expensive.
>
> Personally I'm working on deploying a cluster of a few nodes running
> LizardFS, which is basically a fork/derivative of MooseFS.  While it
> won't scale nearly as well, below 100 nodes should be fine, and in
> particular it sounds like it works fairly well with only a few nodes.
> It has its pros and cons, but for my needs it should be sufficient.
> It also isn't RAM-hungry.  I'm going to be testing it on some
> RockPro64s, with the LSI HBAs.
>
> I did note that Gentoo lacks a LizardFS client.  I suspect I'll be
> looking to fix that - I'm sure the moosefs ebuild would be a good
> starting point.  I'm probably going to be a whimp and run the storage
> nodes on Ubuntu or whatever upstream targets - they're basically
> appliances as far as I'm concerned.
>
> So, those are the two routes I'd recommend.  Just get yourself an HBA
> if you only want a few more drives.  If you see your needs expanding
> then consider a distributed filesystem.  The advantage of the latter
> is that you can keep expanding it however you want with additional
> drives/nodes/whatever.  If you're going over 20 nodes I'd use Ceph for
> sure - IMO that seems to be the future of this space.
>
I'll second your comments on ceph after my experience - great idea for
large scale systems, otherwise performance is quite poor on small
systems. Needs at least GB connections with two networks as well as only
one or two drives per host to work properly.

I think I'll give lizardfs a go - an interesting read.


BillK

Re: [gentoo-user] Hard drive storage questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

On Fri, 9 Nov 2018 10:07:58 +0800, Bill Kenworthy wrote:

> I have used a mini-pcie board from ebay (takes two sata connections)
> alongside a number of other connection types in a btrfs raid 10 for some
> months as a temporary expansion - worked fine, but make sure to check
> Linux compatibility first.

I have both of these in use, and they "just worked"

https://www.amazon.co.uk/Syba-SI-PEX40062-Profile-Bracket-Green/dp/B00AZ9T41M/ref=sr_1_1?ie=UTF8&qid=1541752572&sr=8-1&keywords=Syba+SI-PEX40062

https://www.amazon.co.uk/gp/product/B00UVJI3IQ/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

Neither is in a particularly demanding application so I can't comment on
outright performance.


-- 
Neil Bothwick

For security reasons, all text in this mail is double-rot13 encrypted.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Hard drive storage questions

[J. Roeleveld]

On Friday, November 9, 2018 3:29:52 AM CET Rich Freeman wrote:
> On Thu, Nov 8, 2018 at 8:16 PM Dale <rdalek1967@gmail.com> wrote:
> > I'm trying to come up with a
> > plan that allows me to grow easier and without having to worry about
> > running out of motherboard based ports.
> 
> So, this is an issue I've been changing my mind on over the years.
> There are a few common approaches:
> 
> * Find ways to cram a lot of drives on one host
> * Use a patchwork of NAS devices or improvised hosts sharing over
> samba/nfs/etc and end up with a mess of mount points.
> * Use a distributed FS
> 
> Right now I'm mainly using the first approach, and I'm trying to move
> to the last.  The middle option has never appealed to me.

I'm actually in the middle, but have a single large NAS.

> So, to do more of what you're doing in the most efficient way
> possible, I recommend finding used LSI HBA cards.  These have mini-SAS
> ports on them, and one of these can be attached to a breakout cable
> that gets you 4 SATA ports.  I just picked up two of these for $20
> each on ebay (used) and they have 4 mini-SAS ports each, which is
> capacity for 16 SATA drives per card.  Typically these have 4x or
> larger PCIe interfaces, so you'll need a large slot, or one with a
> cutout.  You'd have to do the math but I suspect that if the card+MB
> supports PCIe 3.0 you're not losing much if you cram it into a smaller
> slot.  If most of the drives are idle most of the time then that also
> demands less bandwidth.  16 fully busy hard drives obviously can put
> out a lot of data if reading sequentially.

I also recommend LSI HBA cards, they work really well and are really well 
supported by Linux.

> You can of course get more consumer-oriented SATA cards, but you're
> lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
> HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
> you main limiting factor assuming you have power and case space.
>
> Oh, and those HBA cards need to be flashed into "IT" mode - they're
> often sold this way, but if they support RAID you want to flash the IT
> firmware that just makes them into a bunch of standalone SATA slots.
> This is usually a PITA that involves DOS or whatever, but I have
> noticed some of the software needed in the Gentoo repo.

Even with Raid-firmware, they can be configured for JBOD.

> If you go that route it is just like having a ton of SATA ports in
> your system - they just show up as sda...sdz and so on (no idea where
> it goes after that).

I tested this once, ended up getting sdaa, sdab,...

> Software-wise you just keep doing what you're
> already doing (though you should be seriously considering
> mdadm/zfs/btrfs/whatever at that point).

I would suggest ZFS or BTRFS over mdadm. Gives you more flexibility and is a 
logical follow-up to LVM.

> That is the more traditional route.
> 
> Now let me talk about distributed filesystems, which is the more
> scalable approach.  I'm getting tired of being limited by SATA ports,
> and cases, and such.  I'm also frustrated with some of zfs's
> inflexibility around removing drives.

IMHO, ZFS is nice for large storage devices, not so much for regular desktops. 
This is why I am hoping BTRFS will solve the resilver issues. (not kept up, is 
this still not working?)

--
Joost

Re: [gentoo-user] Hard drive storage questions

[Wols Lists]

On 09/11/18 01:16, Dale wrote:
> Howdy to all,
> 
> I have a interesting problem coming up.  Currently, I have two 3TB
> drives for my /home mount point.  A lot of this is videos but some pdf
> files and other documents as well plus a photo collection of family
> stuff etc. 
> 
> Filesystem                                   Size  Used Avail Use%
> Mounted on
> /dev/mapper/Home2-Home2        5.4T  3.7T  1.8T  68% /home
> 
> I've got a little over 25% or so of usable space left.  At that point or
> shortly thereafter, it could start causing some issues according to what
> I've read anyway.  Either way, shortly after that, being full will
> certainly be a issue.  I'm full up on my motherboard SATA ports.  Even
> if I buy a larger drive or drives, I'd have to unplug one to move things
> over and likely repeat that a few times.  I could do that and likely
> will have to anyway but I'm trying to think a little farther ahead. 
> Currently I have the more important stuff backed up to a external single
> 6TB USB drive, previous thread on that.  I'm trying to come up with a
> plan that allows me to grow easier and without having to worry about
> running out of motherboard based ports. 
> 
Rich's ideas sound good. I'm in a similar situation to you - I'm about
to outgrow my existing 3TB setup, but I've got a new 4TB lined up
waiting to go (if I can get it to POST).

Small consumer grade SATA expansions are about £35, you should be able
to get a card that does port multiplier for that money, so that's 8 SATA
drives on that card. Watch out - while I would recommend them, some
cards are advertised as "2 SATA, 2 eSATA". That's only two ports, that
can be jumper-switched between SATA or eSATA. Great for backup or
disaster recovery as well as expansion.

Then think about how you're going to lay your filesystem out. My system
currently has three partitions (/, /var and /home). Mirrored (md raid 1)
onto a second 3TB drive, giving me 6TB total. Actually, not very
conducive to future expansion.

The new system is going to have the bare-metal drives set up as one huge
partition (actually not quite true - they'll be a 3TB and a 1TB for
practical reasons). The 3TB partitions will be raided into a raid 5 with
one of the old drives. The 1TB partitions will be raid 1 (system and
home, basically).

I'll then put lvm on the two raids before actually partitioning that to
give me /, /home, and anything else I want.

At this point, I can now replace any of the drives, grow my raid, my
lvm, or my partitions, at any time and underneath a running system
(okay, I don't have hotplug so I'd have to shut down the system to
physically swap a drive over, but ...).

And I'm thinking about btrfs, but the one thing you have to be really
careful with that is snapshots running out of space.

Cheers,
Wol

Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Fri, Nov 9, 2018 at 3:17 AM Bill Kenworthy <billk@iinet.net.au> wrote:
>
> I'll second your comments on ceph after my experience - great idea for
> large scale systems, otherwise performance is quite poor on small
> systems. Needs at least GB connections with two networks as well as only
> one or two drives per host to work properly.
>
> I think I'll give lizardfs a go - an interesting read.
>

So, ANY distributed/NAS solution is going to want a good network
(gigabit or better), if you care about performance.  With Ceph and the
rebuilds/etc it probably makes an even bigger difference, but lizardfs
still shuttles data around.  With replication any kind of write is
multiplied so even moderate use is going to use a lot of network
bandwidth.  If you're talking about hosting OS images for VMs it is a
big deal.  If you're talking about hosting TV shows for your Myth
server or whatever, it probably isn't as big a deal unless you have 14
tuners and 12 clients.

Lizardfs isn't without its issues.  For my purposes it is fine, but it
is NOT as robust as Ceph.  Finding direct comparisons online is
difficult, but here are some of my observations (having not actually
used either, but having read up on both):

* Ceph (esp for obj store) is designed to avoid bottlenecks.  Lizardfs
has a single master server that ALL metadata requests have to go
through.  When you start getting into dozens of nodes that will start
to be a bottleneck, but it also eliminates some of the rigidity of
Ceph since clients don't have to know where all the data is.  I
imagine it adds a bit of latency to reads.

* Lizardfs defaults to acking writes after the first node receives
them, then replicates them.  Ceph defaults to acking after all
replicas are made.  For any application that takes transactions
seriously there is a HUGE data security difference, but it of course
will lower write latency for lizardfs.

* Lizardfs makes it a lot easier to tweak storage policy at the
directory/file level.  Cephfs basically does this more at the
mountpoint level.

* Ceph CRUSH maps are much more configurable than Lizardfs goals.
With Ceph you could easily say that you want 2 copies, and they have
to be on hard drives with different vendors, and in different
datacenters.  With Lizardfs combining tags like this is less
convenient, and while you could say that you want one copy in rack A
and one in rack B, you can't say that you don't care which 2 as long
as they are different.

* The lizardfs high-availability stuff (equiv of Ceph monitors) only
recently went FOSS, and probably isn't stabilized on most distros.
You can have backup masters that are ready to go, but you need your
own solution for promoting them.

* Lizardfs security seems to be non-existent.  Don't stick it on your
intranet if you are a business.  Fine for home, or for a segregated
SAN, maybe, or you could stick it all behind some kind of VPN and roll
your own security layer.  Ceph security seems pretty robust, but
watching what the ansible playbook did to set it up makes me shudder
at the thought of doing it myself.  Lots of keys that all need to be
in sync so that everything can talk to each other.  I'm not sure if
for clients whether it can outsource authentication to kerberos/etc -
not a need for me but I wouldn't be surprised if this is supported.
The key syncing makes a lot more sense within the cluster itself.

* Lizardfs is MUCH simpler to set up.  For Ceph I recommend the
ansible playbook, though if I were using it in production I'd want to
do some serious config management as it seems rather complex and it
seems like the sort of thing that could take out half a datacenter if
it had a bug.  For Lizardfs if you're willing to use the suggested
hostnames about 95% of it is auto-configuring as storage nodes just
reach out to the default master DNS name and report in, and everything
trusts everything (not just by default - I don't think you even can
lock it down unless you stick every node behind a VPN to limit who can
talk to who).

-- 
Rich

Re: [gentoo-user] Hard drive storage questions

[Dale]

Rich Freeman wrote:
> On Thu, Nov 8, 2018 at 8:16 PM Dale <rdalek1967@gmail.com> wrote:
>> I'm trying to come up with a
>> plan that allows me to grow easier and without having to worry about
>> running out of motherboard based ports.
>>
> So, this is an issue I've been changing my mind on over the years.
> There are a few common approaches:
>
> * Find ways to cram a lot of drives on one host
> * Use a patchwork of NAS devices or improvised hosts sharing over
> samba/nfs/etc and end up with a mess of mount points.
> * Use a distributed FS
>
> Right now I'm mainly using the first approach, and I'm trying to move
> to the last.  The middle option has never appealed to me.

And this is what I'm trying to avoid.  Doing one thing, realizing I
should have done it different and then having to spend even more money
to do it the right way.  I'm trying to get advice on what is the best
way forward that I can afford.  Obviously I don't need a setup like
facebook, google or something but I don't want to spend a few hundred
dollars doing something only to realize, it needs to be sold to the next
idiot on Ebay.   ROFL  You're giving me some good options to think on
here.  ;-) 

>
> So, to do more of what you're doing in the most efficient way
> possible, I recommend finding used LSI HBA cards.  These have mini-SAS
> ports on them, and one of these can be attached to a breakout cable
> that gets you 4 SATA ports.  I just picked up two of these for $20
> each on ebay (used) and they have 4 mini-SAS ports each, which is
> capacity for 16 SATA drives per card.  Typically these have 4x or
> larger PCIe interfaces, so you'll need a large slot, or one with a
> cutout.  You'd have to do the math but I suspect that if the card+MB
> supports PCIe 3.0 you're not losing much if you cram it into a smaller
> slot.  If most of the drives are idle most of the time then that also
> demands less bandwidth.  16 fully busy hard drives obviously can put
> out a lot of data if reading sequentially.
>
> You can of course get more consumer-oriented SATA cards, but you're
> lucky to get 2-4 SATA ports on a card that runs you $30.  The mini-SAS
> HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
> you main limiting factor assuming you have power and case space.
>
> Oh, and those HBA cards need to be flashed into "IT" mode - they're
> often sold this way, but if they support RAID you want to flash the IT
> firmware that just makes them into a bunch of standalone SATA slots.
> This is usually a PITA that involves DOS or whatever, but I have
> noticed some of the software needed in the Gentoo repo.
>
> If you go that route it is just like having a ton of SATA ports in
> your system - they just show up as sda...sdz and so on (no idea where
> it goes after that).  Software-wise you just keep doing what you're
> already doing (though you should be seriously considering
> mdadm/zfs/btrfs/whatever at that point).
>
> That is the more traditional route.
>
> Now let me talk about distributed filesystems, which is the more
> scalable approach.  I'm getting tired of being limited by SATA ports,
> and cases, and such.  I'm also frustrated with some of zfs's
> inflexibility around removing drives.  These are constraints that make
> upgrading painful, and often inefficient.  Distributed filesystems
> offer a different solution.
>
> A distributed filesystem spreads its storage across many hosts, with
> an arbitrary number of drives per host (more or less).  So, you can
> add more hosts, add more drives to a host, and so on.  That means
> you're never forced to try to find a way to cram a few more drives in
> one host.  The resulting filesystem appears as one gigantic filesystem
> (unless you want to split it up), which means no mess of nfs
> mountpoints and so on, and all the other headaches of nfs.  Just as
> with RAID these support redundancy, except now you can lose entire
> hosts without issue.  With many you can even tell it which
> PDU/rack/whatever each host is plugged into, and it will make sure you
> can lose all the hosts in one rack.  You can also mount the filesystem
> on as many hosts as you want at the same time.
>
> They do tend to be a bit more complex.  The big players can scale VERY
> large - thousands of drives easily.  Everything seems to be moving
> towards Ceph/CephFS.  If you were hosting a datacenter full of
> VMs/containers/etc I'd be telling you to host it on Ceph.  However,
> for small scale (which you definitely are right now), I'm not thrilled
> with it.  Due to the way it allocates data (hash-based) anytime
> anything changes you end up having to move all the data around in the
> cluster, and all the reports I've read suggests it doesn't perform all
> that great if you only have a few nodes.  Ceph storage nodes are also
> RAM-hungry, and I want to run these on ARM to save power, and few ARM
> boards have that kind of RAM, and they're very expensive.
>
> Personally I'm working on deploying a cluster of a few nodes running
> LizardFS, which is basically a fork/derivative of MooseFS.  While it
> won't scale nearly as well, below 100 nodes should be fine, and in
> particular it sounds like it works fairly well with only a few nodes.
> It has its pros and cons, but for my needs it should be sufficient.
> It also isn't RAM-hungry.  I'm going to be testing it on some
> RockPro64s, with the LSI HBAs.
>
> I did note that Gentoo lacks a LizardFS client.  I suspect I'll be
> looking to fix that - I'm sure the moosefs ebuild would be a good
> starting point.  I'm probably going to be a whimp and run the storage
> nodes on Ubuntu or whatever upstream targets - they're basically
> appliances as far as I'm concerned.
>
> So, those are the two routes I'd recommend.  Just get yourself an HBA
> if you only want a few more drives.  If you see your needs expanding
> then consider a distributed filesystem.  The advantage of the latter
> is that you can keep expanding it however you want with additional
> drives/nodes/whatever.  If you're going over 20 nodes I'd use Ceph for
> sure - IMO that seems to be the future of this space.
>

This is a lot to think on.  Money wise, and maybe even expansion wise, I
may go with the PCI SATA cards and add drives inside my case.  I have
plenty of power supply since it pulls at most 200 watts and I think my
P/S is like 700 or 800 watts.  I can also add a external SATA card or
another USB drive to do backups with as well.  At some point tho, I may
have to build one of those little tiny systems that is basically nothing
but SATA drive controllers and ethernet enabled.  Have that sitting in a
closet somewhere running some small OS.  I can always just move the
drives from my system to it if needed.

One thing is for sure, you gave a lot of info and different ways to
think on this. 

Thanks to Rich and everyone else to for their thoughts.  It's certainly
helped give me ideas I haven't thought of. 

Dale

:-)  :-)

P. S.  For those who may wonder, my Mom is home and doing pretty well. 
:-D 

Re: [gentoo-user] Hard drive storage questions

[Wol's lists]

On 11/11/2018 00:45, Dale wrote:
> This is a lot to think on.  Money wise, and maybe even expansion wise, I
> may go with the PCI SATA cards and add drives inside my case.  I have
> plenty of power supply since it pulls at most 200 watts and I think my
> P/S is like 700 or 800 watts.  I can also add a external SATA card or
> another USB drive to do backups with as well.  At some point tho, I may
> have to build one of those little tiny systems that is basically nothing
> but SATA drive controllers and ethernet enabled.  Have that sitting in a
> closet somewhere running some small OS.  I can always just move the
> drives from my system to it if needed.

https://raid.wiki.kernel.org/index.php/What_is_RAID_and_why_should_you_want_it%3F

(disclaimer - I wrote it :-)

You've got a bunch of questions to ask yourself. Is this an amateur 
setup (sounds a bit like it in that it appears to be a home server) or 
is it a professional "money no object" setup.

Either way, if you spend good money on good disks (WD Red, Seagate 
Ironwolf, etc) then most of your investment will be good to re-purpose. 
My current 3TB drives are Barracudas - not a good idea for a 
fault-tolerant system - which is why the replacements are Ironwolves.

Then, as that web-page makes clear, do you want your raid/volume 
management to be separate from your filesystem - mdraid/lvm under ext4 - 
or do you want a filesystem that is hardware-aware like zfs or xfs, or 
do you want something like btrfs which tries to be the latter, but is 
better used as the former.

One thing to seriously watch out for - many filesystems are aware of the 
underlying layer even when you don't expect it. Not sure which 
filesystem it is but I remember an email discussion where the filesystem 
was aware it was running over mdraid and balanced itself for the 
underlying disks. The filesystem developer didn't realise that mdraid 
can add and remove disks so the underlying structure can change, and the 
recommendation was "once you've set up the raid, if you want to grow 
your space move it to a new raid".

At the end of the day, there is no perfect answer, and you need to ask 
yourself what you are trying to achieve, and what you can afford.

Cheers,
Wol

Re: [gentoo-user] Hard drive storage questions

[Dale]

Wol's lists wrote:
> On 11/11/2018 00:45, Dale wrote:
>> This is a lot to think on.  Money wise, and maybe even expansion wise, I
>> may go with the PCI SATA cards and add drives inside my case.  I have
>> plenty of power supply since it pulls at most 200 watts and I think my
>> P/S is like 700 or 800 watts.  I can also add a external SATA card or
>> another USB drive to do backups with as well.  At some point tho, I may
>> have to build one of those little tiny systems that is basically nothing
>> but SATA drive controllers and ethernet enabled.  Have that sitting in a
>> closet somewhere running some small OS.  I can always just move the
>> drives from my system to it if needed.
>
> https://raid.wiki.kernel.org/index.php/What_is_RAID_and_why_should_you_want_it%3F
>
>
> (disclaimer - I wrote it :-)
>
> You've got a bunch of questions to ask yourself. Is this an amateur
> setup (sounds a bit like it in that it appears to be a home server) or
> is it a professional "money no object" setup.
>
> Either way, if you spend good money on good disks (WD Red, Seagate
> Ironwolf, etc) then most of your investment will be good to
> re-purpose. My current 3TB drives are Barracudas - not a good idea for
> a fault-tolerant system - which is why the replacements are Ironwolves.
>
> Then, as that web-page makes clear, do you want your raid/volume
> management to be separate from your filesystem - mdraid/lvm under ext4
> - or do you want a filesystem that is hardware-aware like zfs or xfs,
> or do you want something like btrfs which tries to be the latter, but
> is better used as the former.
>
> One thing to seriously watch out for - many filesystems are aware of
> the underlying layer even when you don't expect it. Not sure which
> filesystem it is but I remember an email discussion where the
> filesystem was aware it was running over mdraid and balanced itself
> for the underlying disks. The filesystem developer didn't realise that
> mdraid can add and remove disks so the underlying structure can
> change, and the recommendation was "once you've set up the raid, if
> you want to grow your space move it to a new raid".
>
> At the end of the day, there is no perfect answer, and you need to ask
> yourself what you are trying to achieve, and what you can afford.
>
> Cheers,
> Wol
>


I've considered RAID before.  For what I have here, doing regular
backups is enough, I hope.  Right now, I backup family pics from my
camera, documents and all the videos, which is a LOT.  If a drive were
to fail, at least I have the backups to go to and would lose little if
anything.  Since I backup pretty regular, I could still recover most
everything that may not have made it to backup. 

I'm on a limited income and this is just a home system.  One thing I try
to do is to find a good way forward first, then do something.  That way
I don't do something that costs money, realize I did it in a bad way and
then have to spend more money doing it again and have parts that I will
likely never need or use again.  I try to skip the worthless part when I
can.  ;-) 

That said, when I do get some drives and they are installed, I'll likely
be asking questions about btrfs, zfs, xfs and anything else that may
apply.  Currently I use LVM and ext4.  I like it pretty well but that
doesn't mean there can't be something better out there. 

Dale

:-)  :-)