[gentoo-user] Backup questions

[gentoo-user] Backup questions

[Dale]

Howdy,

Long story short that leads up to my questions, I paid off some debt. 
Finally I'm getting around to doing some things I been wanting to do. 
One of them, backups.  I bought a hard drive enclosure that has a fan to
keep things cool.  Figured I would get a decent one that hopefully will
keep my drives working.  My relevant setup. 

I started with a 3TB hard drive for my /home directory, one partition,
years ago.  I stuck LVM on it so that I could add to or move things
around a bit with some ease.  Once that was starting to fill up, I
bought a second 3TB drive and added it to the LVM volume.  So, in
effect, I have a 6TB drive with 3.6TB of data on it at the moment.  The
biggest user of that space is videos. 

I just bought two external drive enclosures.  One is sort of a spare but
I do plan to do some backups on it, mostly pictures from my camera.  In
one of the enclosures I put a single 6TB drive that I found on ebay.  It
has about 7,000 hours on it so it should have some life left yet and it
passed the smartctl tests.  It is USB but it transfers fast.  Now to
some questions.  I use rsync.  Command looks something like rsync -auv
/source/ /destination/.  If I backup the config files in my home
directory, should I also include the --delete option?  If after a
upgrade for example a config file is deleted, because it is no longer
needed, or renamed, should the old file be removed or is there a reason
to keep them on the backups?  Adding the --delete option isn't a problem
command wise BUT I wonder if it can cause a problem at some point. 
Thoughts on that.  I plan to use the --delete option for videos since if
I deleted one, it is likely broken or something.  Biggest question is
about config files.

On the second enclosure I currently have a 160GB drive.  It's big enough
for my camera pictures.  I would like to backup up my pics to it and
then put the drive somewhere besides in the house.  I have a couple
external buildings that would be safe as far as rain etc but they are
not cooled, even tho it gets close to 100F and humid, real humid, here. 
My question is this.  Is it safe to store a drive in that sort of
environment?  I could see the building getting close to outside temps
during the day.  I do put a heater in it to prevent freezing during the
winter.  I usually set the heat to 40F.  I'm hoping someone has some
real world experience on storing in this sort of environment and not
just a text book theory.  One reason I want to put them elsewhere, house
fire.  Even a huge power strike could cause problems if plugged in.  We
do get lightening strikes here.  Maybe not as many as some but our fair
share.  The 6TB and 3TB drive may join this one as well.

I have a 3TB drive on the way, got a good deal on it.  I plan to remove
the 160GB and put the 3TB drive in the enclosure when it gets here. 
Later, I will get another enclosure for the 160GB drive.  In the end, I
will have 6TB, 3TB and a 160GB external drive.  Main backups on the 6TB,
possible second backup on 3TB of some important files and camera pics on
the 160GB. 

Anyone have some real world experience on these sorts of questions? 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Backup questions

[Bill Kenworthy]

On 08/08/18 11:43, Dale wrote:
> Howdy,
>
> Long story short that leads up to my questions, I paid off some debt. 
>
Hi Dale,

    what you are talking about is not a real backup but a single copy of
your data that may or may not be complete (the delete option you
mention) at a single point in time - not quite as useful as a proper
versioned backup.  Whatever your choice, also look at the restore
procedure - very important.


Have a look at Dirvish or borgbackup (both in portage) for what they can
do.  Having a space efficient store at regular points of time is a
lifesaver at times. To restore from dirvish its a copy from the selected
tree.  With borg its either restore with a command, or mount it and copy
the data out of the mount.

http://dirvish.org/

https://www.borgbackup.org/

I moved from dirvish to borg 12 months ago and they are both excellent.

BillK

Re: [gentoo-user] Backup questions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

On Thu, 9 Aug 2018 16:18:43 +0800, Bill Kenworthy wrote:

> Have a look at Dirvish or borgbackup (both in portage) for what they can
> do.  Having a space efficient store at regular points of time is a
> lifesaver at times. To restore from dirvish its a copy from the selected
> tree.  With borg its either restore with a command, or mount it and copy
> the data out of the mount.

I agree with all of this and I would also add Duplicity as a possible
candidate, although not quite as simple to use as BorgBackup (I haven't
tried Dirvish) I usually end up putting a wrapper script around such tasks
anyway.

If you have your offsite backups in another building, how will you get
the backups there? If they are networked, great, but if it relies on you
physically moving hard drives between locations, it won't be done when
you need it. Cloud storage is cheap these days and most backup programs
have an option to store encrypted data on an external service. I'm
currently using a combination of Duplicity and Hubic for this.


-- 
Neil Bothwick

<Linuxgeek> How do i find the model of my card?
<Serena[T]> your nick is misleading, seriously

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Backup questions

[zless]

În ziua de joi, 9 august 2018, la 11:37:38 EEST, Neil Bothwick a scris:
> I agree with all of this and I would also add Duplicity as a possible
> candidate, although not quite as simple to use as BorgBackup (I haven't
> tried Dirvish) I usually end up putting a wrapper script around such tasks
> anyway.

I've been using Duplicity in production for a few years.
I find its features quite better than borgbackup.

However, it has one very annoying design problem: the local cache.
In my case, doing daily backups for two years for ~200GB of data
requires a growing and growing local cache. It's 50GB as of now.

You delete the cache and it comes back when doing the next 
incremental backup.

Re: [gentoo-user] Backup questions

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2643 bytes --]

On Thursday, 9 August 2018 09:18:43 BST Bill Kenworthy wrote:
> On 08/08/18 11:43, Dale wrote:
> > Howdy,
> > 
> > Long story short that leads up to my questions, I paid off some debt. 
> 
> Hi Dale,
> 
>     what you are talking about is not a real backup but a single copy of
> your data that may or may not be complete (the delete option you
> mention) at a single point in time - not quite as useful as a proper
> versioned backup.  Whatever your choice, also look at the restore
> procedure - very important.

Well, a static mirror is a full backup at that point in time.  If the backed 
up data changes little over time, it is a valid backup, which can prove its 
worth if/when the original drive dies, or files are deleted accidentally on 
the original.

On the points Dale raised:

The --delete option will remove from the destination any files which no longer 
exist on the source.  So if you delete photo-1 on the source and then run 
rsync, photo-1 *will* be deleted from the full back up, to mirror what is 
currently available on the source directory.

Here is where incremental/differential backup strategies can be of use, in 
case some time in the relatively near future you change your mind and wish you 
never had deleted that old photo-1.  The same may apply to user config files, 
if you stop using an application, manually clean/delete its config files from 
your home and rsync --delete thereafter.  If in the near future you review 
your position and decide you wanted that application after all and the 2 weeks 
you had spent configuring it would be of use again, with the --delete option 
your config files will be gone from the backup.  So, use --delete judiciously.

rsync can on its own provide you with incremental and differential backups, 
using hard links to the full backup directory, so as to avoid duplication and 
minimise storage space usage.  This means that incremental backups take only a 
fraction of the space and additional disks or enclosures may be redundant.  
Take a look at the --backup, --backup-dir, and --link-dest, options.

As others have posted there are a number of applications which use rsync as a 
back end and have scripted with config files its options.  There's also quite 
a number of bash scripts on the interwebs offering a starting point if you 
prefer to hack your own.

With regards to heat and humidity I suggest you take a look at the 
manufacturer's specifications, both for the enclosure and for the drives.  
Invariably environmental thresholds are printed on labels on the devices 
themselves, or you could google using the part numbers off them.

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Backup questions

[Wols Lists]

On 08/08/18 04:43, Dale wrote:
> Howdy,
> 

> 
> I just bought two external drive enclosures.  One is sort of a spare but
> I do plan to do some backups on it, mostly pictures from my camera.  In
> one of the enclosures I put a single 6TB drive that I found on ebay.  It
> has about 7,000 hours on it so it should have some life left yet and it
> passed the smartctl tests.  It is USB but it transfers fast.  Now to
> some questions.  I use rsync.  Command looks something like rsync -auv
> /source/ /destination/.  If I backup the config files in my home
> directory, should I also include the --delete option?  If after a
> upgrade for example a config file is deleted, because it is no longer
> needed, or renamed, should the old file be removed or is there a reason
> to keep them on the backups?  Adding the --delete option isn't a problem
> command wise BUT I wonder if it can cause a problem at some point. 
> Thoughts on that.  I plan to use the --delete option for videos since if
> I deleted one, it is likely broken or something.  Biggest question is
> about config files.
> 
May I suggest using btrfs for your backup drive? One MAJOR caveat - DO
NOT let the drive fill up - a combination of snapshots and drive-full
has been known (quite often) to trash the file system. But provided you
make sure it doesn't go above about 80% you should be fine.

You can add an option to rsync such that it will back up "in place". In
other words, if only 1K is changed in a 1M file, it will overwrite that
1K. So when you back up, the procedure is to take a snapshot, then run
rsync with both "in place" and "delete".

This will give you the space economy of incremental backups, combined
with the utility of full backups - each snapshot is a full backup as of
that date, but each new snapshot only increases disk usage by the
changes since the last. And you reclaim space by deleting old snapshots.

> On the second enclosure I currently have a 160GB drive.  It's big enough
> for my camera pictures.  I would like to backup up my pics to it and
> then put the drive somewhere besides in the house.  I have a couple
> external buildings that would be safe as far as rain etc but they are
> not cooled, even tho it gets close to 100F and humid, real humid, here. 
> My question is this.  Is it safe to store a drive in that sort of
> environment?  I could see the building getting close to outside temps
> during the day.  I do put a heater in it to prevent freezing during the
> winter.  I usually set the heat to 40F.  I'm hoping someone has some
> real world experience on storing in this sort of environment and not
> just a text book theory.  One reason I want to put them elsewhere, house
> fire.  Even a huge power strike could cause problems if plugged in.  We
> do get lightening strikes here.  Maybe not as many as some but our fair
> share.  The 6TB and 3TB drive may join this one as well.
> 
A drive that's shut down will take more mistreatment than one that is
running. So no worries on that score. Plus heat causes far less problems
than people think, although yes it's best avoided.

Do your outbuildings have power? Do you have a fridge (or possibly
freezer) out there, or could you find an excuse for one - a wine-store
maybe :-) What you really want is some form of insulation that will
prevent rapid fluctuations in temperature, and sticking your drives (in
sealed bags) in a wine fridge would probably be near ideal. I had a
cellar for my wine, and daily fluctuations were near nil even though
there was a maybe 20C variation between summer and winter. That's what
you want to aim for. Or maybe if you can dig a mini-cellar in one of
your outbuildings :-)

Cheers,
Wol

Re: [gentoo-user] Backup questions

[Dale]

Bill Kenworthy wrote:
> On 08/08/18 11:43, Dale wrote:
>> Howdy,
>>
>> Long story short that leads up to my questions, I paid off some debt. 
>>
> Hi Dale,
>
>     what you are talking about is not a real backup but a single copy of
> your data that may or may not be complete (the delete option you
> mention) at a single point in time - not quite as useful as a proper
> versioned backup.  Whatever your choice, also look at the restore
> procedure - very important.
>
>
> Have a look at Dirvish or borgbackup (both in portage) for what they can
> do.  Having a space efficient store at regular points of time is a
> lifesaver at times. To restore from dirvish its a copy from the selected
> tree.  With borg its either restore with a command, or mount it and copy
> the data out of the mount.
>
> http://dirvish.org/
>
> https://www.borgbackup.org/
>
> I moved from dirvish to borg 12 months ago and they are both excellent.
>
> BillK
>

It's a backup to me.  I may not be using backup software but if I lose
the original file, I have another copy that I can back up from.  Given
that I have two drives that can currently hold the files I don't want to
lose for sure, I have two backup copies.  Whether it is called a backup
or called a copy doesn't matter.  All that matters is that if my drive
should fail, my computer blows up, my house burns down or any number of
other possibilities, I can restore the files if needed.  Whether it is a
technical backup or a copy ends the same way.  Maybe calling it a copy
is better.  :-)  Maybe I'm to old school. lol 

I will look into those software options tho.  Right now I have the rsync
commands to backup a few directories in a script.  It's not fancy but
basically one copies my camera pics, one copies my videos and the last
one copies my email directory.  In all honesty, if I have those three
things, everything else can be reinstalled or be reconfigured.  I'm not
trying or even planning to copy/backup the OS itself.  If something
happened and I had to rebuild or redo my system, I'd do a fresh install
anyway.  Having the config files would be nice but only IF it wouldn't
cause more problems than it solves.  That was the reason for my question
about using --delete on config files.  I tend to backup/copy the files
in /etc until I reboot then I start a new set.  That way if I run into a
problem, I can either use the old file in whole or take parts of it
until I get whatever working again.  I haven't ran into that problem in
a really long time tho.  I can't recall the last time I do to be
honest.  It's been years, many years.  I'm not sure on the config files
in my home directory tho.  I know KDE does some weird things during some
major upgrades. 

As for restore, easy, rsync the files back over.  Even if the
permissions are messed up, I can fix that easy enough.  Other than that,
I'm not sure what other problem I could run into.  The biggest thing,
having a copy I can use if I lose the originals.  Also, with them being
plain copies, I can take the drive to a friend or family member and plug
the drive in to get to the videos, documents etc.  No special software
really needed.  Heck, for the videos, I could watch them straight from
the USB drive. 

Now to go check into those backup programs.  Borg.  Sounds Star Trekish
to me, or was that Star Wars.  ROFL 

Thanks much for the info. 

Dale

:-)  :-) 

Re: [gentoo-user] Backup questions

[Dale]

Wols Lists wrote:
> On 08/08/18 04:43, Dale wrote:
>> Howdy,
>>
>> I just bought two external drive enclosures.  One is sort of a spare but
>> I do plan to do some backups on it, mostly pictures from my camera.  In
>> one of the enclosures I put a single 6TB drive that I found on ebay.  It
>> has about 7,000 hours on it so it should have some life left yet and it
>> passed the smartctl tests.  It is USB but it transfers fast.  Now to
>> some questions.  I use rsync.  Command looks something like rsync -auv
>> /source/ /destination/.  If I backup the config files in my home
>> directory, should I also include the --delete option?  If after a
>> upgrade for example a config file is deleted, because it is no longer
>> needed, or renamed, should the old file be removed or is there a reason
>> to keep them on the backups?  Adding the --delete option isn't a problem
>> command wise BUT I wonder if it can cause a problem at some point. 
>> Thoughts on that.  I plan to use the --delete option for videos since if
>> I deleted one, it is likely broken or something.  Biggest question is
>> about config files.
>>
> May I suggest using btrfs for your backup drive? One MAJOR caveat - DO
> NOT let the drive fill up - a combination of snapshots and drive-full
> has been known (quite often) to trash the file system. But provided you
> make sure it doesn't go above about 80% you should be fine.
>
> You can add an option to rsync such that it will back up "in place". In
> other words, if only 1K is changed in a 1M file, it will overwrite that
> 1K. So when you back up, the procedure is to take a snapshot, then run
> rsync with both "in place" and "delete".
>
> This will give you the space economy of incremental backups, combined
> with the utility of full backups - each snapshot is a full backup as of
> that date, but each new snapshot only increases disk usage by the
> changes since the last. And you reclaim space by deleting old snapshots.
>

I did think about btrfs.  I've read a lot of threads on here about
people using it and it seems to have come a long ways and be pretty
stable.  Right now, I've got a lot going on and really don't have the
time to sit down and read up on it and how it works or what all it can
do.  In all honesty, if my system were to crash later when I don't have
so much going on, I'd like to move to btrfs for as much as possible of
my system.  I suspect /boot would still have to be ext2 or something
because of grub. 


>> On the second enclosure I currently have a 160GB drive.  It's big enough
>> for my camera pictures.  I would like to backup up my pics to it and
>> then put the drive somewhere besides in the house.  I have a couple
>> external buildings that would be safe as far as rain etc but they are
>> not cooled, even tho it gets close to 100F and humid, real humid, here. 
>> My question is this.  Is it safe to store a drive in that sort of
>> environment?  I could see the building getting close to outside temps
>> during the day.  I do put a heater in it to prevent freezing during the
>> winter.  I usually set the heat to 40F.  I'm hoping someone has some
>> real world experience on storing in this sort of environment and not
>> just a text book theory.  One reason I want to put them elsewhere, house
>> fire.  Even a huge power strike could cause problems if plugged in.  We
>> do get lightening strikes here.  Maybe not as many as some but our fair
>> share.  The 6TB and 3TB drive may join this one as well.
>>
> A drive that's shut down will take more mistreatment than one that is
> running. So no worries on that score. Plus heat causes far less problems
> than people think, although yes it's best avoided.
>
> Do your outbuildings have power? Do you have a fridge (or possibly
> freezer) out there, or could you find an excuse for one - a wine-store
> maybe :-) What you really want is some form of insulation that will
> prevent rapid fluctuations in temperature, and sticking your drives (in
> sealed bags) in a wine fridge would probably be near ideal. I had a
> cellar for my wine, and daily fluctuations were near nil even though
> there was a maybe 20C variation between summer and winter. That's what
> you want to aim for. Or maybe if you can dig a mini-cellar in one of
> your outbuildings :-)
>
> Cheers,
> Wol
>

I've read the labels in the past.  They can handle more when not powered
up so you are right there.  I was just curious if someone had real world
experience with such conditions.  The past several days it has been
pretty hot and humid here.  If I think about it, I'll try to go to the
building and take some temps and check the humidity.  I think I have a
humidity meter out there, it's with the temp meter I think.  I know in
the past I've seen it be over 90F in there.  For what is usually stored
in there, I don't worry about heat as much as I do cold.  Some things in
there can't freeze.  The drives are what makes me consider the heat.

I do have some foam coolers I could put them in.  I was also planning to
buy some of those thick bags to put them in while in the house then take
the sealed bag drives to the building.  Then later on when I need the
drives, I could bring them in, let them sit overnight or something if
needed and then remove them from the bag and plug in to use.  I could
also add a couple desiccant bags or something to those bags.  The
biggest thing is avoiding condensation if it is cold out there and warm
in here.  The bags I'm talking about is those vacuum bags like people
put clothes in.  They are pretty thick and tend to work pretty well.

It has power.  I'm not sure where I'd put a fridge, even a tiny one.  I
wish it was twice as big as it is.  Of course, I'd fill that up in no
time too.  Isn't that the way it works?  ROFL 

I'm getting interesting ideas tho.  Pondering that backup software
option too.  It has its pluses.  ;-)

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Backup questions

[Bill Kenworthy]

On 10/08/18 10:46, Dale wrote:
> Wols Lists wrote:
>> On 08/08/18 04:43, Dale wrote:
>>> Howdy,
>>>
>>> I just bought two external drive enclosures.  One is sort of a spare but
>>> ....

>>> It has power.  I'm not sure where I'd put a fridge, even a tiny one.  I
>>> wish it was twice as big as it is.  Of course, I'd fill that up in no
>>> time too.  Isn't that the way it works?  ROFL 
>>>
>>> I'm getting interesting ideas tho.  Pondering that backup software
>>> option too.  It has its pluses.  ;-)
>>>
>>> Thanks.
>>>
>>> Dale
>>>
>>> :-)  :-) 
>>>

I would think the cost of setting it up and running a fridge would add
up to hard drive cost over a couple of years anyway (at least for what
we pay in Western Australia :)  In a humid environment you would need to
be very careful of condensation, and a sealed system will still need a
way to transfer the heat as if the cooling fails, it will cook itself
very fast

I have the fans set to spin up faster at 35c and above.  Without the
fans they sit around 45c on a typical day and use.  I have found that in
an enclosure its just as important to have good conduction of heat
(disks mounted to the metal frame) and clear the heated air out of the
enclosure as flowing air over the disks.

Here it gets to 40c+ (>100F) and sometimes humid (not far from the
ocean).  Cooling is fans only, and I have 4 WD Green, 2 WD red and two
seagates (all 2G, in 2x btrfs raid 10) and a few intel and samsung SSD's
(for bcache and system disks) that run ~16 hrs a day with no failures
for the last few years (~10).


Sometimes its better to play the odds.

BillK

Re: [gentoo-user] Backup questions

[Dale]

Mick wrote:
> On Thursday, 9 August 2018 09:18:43 BST Bill Kenworthy wrote:
>> On 08/08/18 11:43, Dale wrote:
>>> Howdy,
>>>
>>> Long story short that leads up to my questions, I paid off some debt. 
>> Hi Dale,
>>
>>     what you are talking about is not a real backup but a single copy of
>> your data that may or may not be complete (the delete option you
>> mention) at a single point in time - not quite as useful as a proper
>> versioned backup.  Whatever your choice, also look at the restore
>> procedure - very important.
> Well, a static mirror is a full backup at that point in time.  If the backed 
> up data changes little over time, it is a valid backup, which can prove its 
> worth if/when the original drive dies, or files are deleted accidentally on 
> the original.

And that is the result I was wanting.  The added benefit of a plain file
copy, portability.  If I wanted to, I could take the drive to a friend,
plug it in and watch videos directly from the drive.  I could also copy
them to the friends computer if I wanted to without any special
software.  I just need a basic, no frills copy that I can go to if I
lose the originals.  Most of this is video and pictures where once saved
will never change again.  I'm not needing anything fancy for sure.  I
might add, I used to do similar when doing system backups.  I'd boot
from a DVD tool such as sysrescue, mount the partitions and then copy
everything over to a second drive. 


>
> On the points Dale raised:
>
> The --delete option will remove from the destination any files which no longer 
> exist on the source.  So if you delete photo-1 on the source and then run 
> rsync, photo-1 *will* be deleted from the full back up, to mirror what is 
> currently available on the source directory.
>
> Here is where incremental/differential backup strategies can be of use, in 
> case some time in the relatively near future you change your mind and wish you 
> never had deleted that old photo-1.  The same may apply to user config files, 
> if you stop using an application, manually clean/delete its config files from 
> your home and rsync --delete thereafter.  If in the near future you review 
> your position and decide you wanted that application after all and the 2 weeks 
> you had spent configuring it would be of use again, with the --delete option 
> your config files will be gone from the backup.  So, use --delete judiciously.
>
> rsync can on its own provide you with incremental and differential backups, 
> using hard links to the full backup directory, so as to avoid duplication and 
> minimise storage space usage.  This means that incremental backups take only a 
> fraction of the space and additional disks or enclosures may be redundant.  
> Take a look at the --backup, --backup-dir, and --link-dest, options.
>
> As others have posted there are a number of applications which use rsync as a 
> back end and have scripted with config files its options.  There's also quite 
> a number of bash scripts on the interwebs offering a starting point if you 
> prefer to hack your own.
>
> With regards to heat and humidity I suggest you take a look at the 
> manufacturer's specifications, both for the enclosure and for the drives.  
> Invariably environmental thresholds are printed on labels on the devices 
> themselves, or you could google using the part numbers off them.
>
> HTH.


I'll look into that option.  I've got a lot going on, Mom in hospital,
me trying to do some cleaning that is otherwise difficult to do when Mom
is here, plus a few other things that are routine as well.  Trying to
keep my head above water here.  lol  Eventually, I'll find the best way
since I've had some good ideas mentioned in these replies.  If I do go
the software backup route, I've got some good recommendations here on
what to look into.  If I continue the rsync route, I've got a couple new
options to look into.  The idea of putting the drives in a cooler was
also a good one that I hadn't thought of.  That would help control the
temperature fluctuations for sure.  I'm not sure about the fridge part
tho.  Space is limited since I have a lot of there. 

Thanks for the info. 

Dale

:-)  :-) 

Re: [gentoo-user] Backup questions

[Dale]

Bill Kenworthy wrote:
> On 10/08/18 10:46, Dale wrote:
>> Wols Lists wrote:
>>> On 08/08/18 04:43, Dale wrote:
>>>> Howdy,
>>>>
>>>> I just bought two external drive enclosures.  One is sort of a spare but
>>>> ....
>>>> It has power.  I'm not sure where I'd put a fridge, even a tiny one.  I
>>>> wish it was twice as big as it is.  Of course, I'd fill that up in no
>>>> time too.  Isn't that the way it works?  ROFL 
>>>>
>>>> I'm getting interesting ideas tho.  Pondering that backup software
>>>> option too.  It has its pluses.  ;-)
>>>>
>>>> Thanks.
>>>>
>>>> Dale
>>>>
>>>> :-)  :-) 
>>>>
> I would think the cost of setting it up and running a fridge would add
> up to hard drive cost over a couple of years anyway (at least for what
> we pay in Western Australia :)  In a humid environment you would need to
> be very careful of condensation, and a sealed system will still need a
> way to transfer the heat as if the cooling fails, it will cook itself
> very fast

I actually have a couple of those really small fridges that I could
use.  I have one in the house that I store seeds in.  The problem is
space.  The little out building is pretty full.  There is no shelf space
and very little floor space.  I'm just not sure where I could put it
since it requires some room for the coils to shed their heat.  I'll have
to check into it more. 


>
> I have the fans set to spin up faster at 35c and above.  Without the
> fans they sit around 45c on a typical day and use.  I have found that in
> an enclosure its just as important to have good conduction of heat
> (disks mounted to the metal frame) and clear the heated air out of the
> enclosure as flowing air over the disks.
>
> Here it gets to 40c+ (>100F) and sometimes humid (not far from the
> ocean).  Cooling is fans only, and I have 4 WD Green, 2 WD red and two
> seagates (all 2G, in 2x btrfs raid 10) and a few intel and samsung SSD's
> (for bcache and system disks) that run ~16 hrs a day with no failures
> for the last few years (~10).
>
>
> Sometimes its better to play the odds.
>
> BillK
>

If you have those drives in use with those temps, I should be fine
here.  When I put the drives in the outbuilding, they won't be powered
up or anything.  They can handle more when powered off than they can
when powered up and in use.  While I still want to avoid heat, that real
world info does help a lot.  If I recall correctly, we have similar
weather as well.  I don't live close to the ocean but when the winds
come from the south, the moisture gets here soon enough.  I live about
250 miles from the gulf coast.  For weather tho, it doesn't take long
for heat or moisture to go that far.  The biggest thing, the salt part
is gone by the time it gets here.  ;-)

One thing about the enclosure I got, it has a fan on it.  It moves a
fair amount of air.  Drives don't use much power so it doesn't take a
whole lot of air flow to get the job done.  Here is a link to one like I
have.

https://www.ebay.com/itm/132706512662

One reason I got it, someone else using Linux said it worked for them. 
Since I only use Linux, good deal. 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Backup questions

[Stefan G. Weichinger]

Am 09.08.18 um 18:52 schrieb Wols Lists:

> May I suggest using btrfs for your backup drive? 

I like btrbk:

https://github.com/digint/btrbk

I run btrfs as main filesystem on 3 systems (2 laptops, 1 main desktop)
and so far I am not looking back.

-> btrbk snapshots to local subvolumes ("time machine") + pulling btrbk
backups to external disks + btrbk backups to another server.

Works fine for me.

-

Another solution I use at customers:

amanda backups to so-called vtapes, virtual tapes = directories on 2 or
more usb-disks. As soon as you have the setup done, you get rotated
backups to those directories, incrementals and fulls mixed via amanda's
algorithm. OK, a bit of a learning curve, but you won't avoid that anywhere.

Re: [gentoo-user] Backup questions

[John Covici]

On Tue, 14 Aug 2018 06:55:05 -0400,
Stefan G. Weichinger wrote:
> 
> Am 09.08.18 um 18:52 schrieb Wols Lists:
> 
> > May I suggest using btrfs for your backup drive? 
> 
> I like btrbk:
> 
> https://github.com/digint/btrbk
> 
> I run btrfs as main filesystem on 3 systems (2 laptops, 1 main desktop)
> and so far I am not looking back.
> 
> -> btrbk snapshots to local subvolumes ("time machine") + pulling btrbk
> backups to external disks + btrbk backups to another server.
> 
> Works fine for me.
> 
> -
> 
> Another solution I use at customers:
> 
> amanda backups to so-called vtapes, virtual tapes = directories on 2 or
> more usb-disks. As soon as you have the setup done, you get rotated
> backups to those directories, incrementals and fulls mixed via amanda's
> algorithm. OK, a bit of a learning curve, but you won't avoid that anywhere.
> 
> 

I use sanoid/syncoid to back up using zfs.  Its great, keeps snapshots
for as long as I want them (I use 80 days for now).  And it keeps
hourlies for the last couple of days as well, so I could roll back in
case of a problem.  Very nice if you use zfs.  I don't think btrfs is
ready for prime time -- its been under development for a while, but I
am scared to use it -- I did try once, but got nowhere.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici wb2una
         covici@ccs.covici.com

Re: [gentoo-user] sanoid (was Backup questions)

[J. Roeleveld]

On August 14, 2018 11:42:18 AM UTC, John Covici <covici@ccs.covici.com> wrote:

>I use sanoid/syncoid to back up using zfs.  Its great, keeps snapshots
>for as long as I want them (I use 80 days for now).  And it keeps
>hourlies for the last couple of days as well, so I could roll back in
>case of a problem.  Very nice if you use zfs.

I tried sanoid, but it has a few problems which really become annoying when you have a lot of datasets:
1) every dataset is handled seperately, no use of recursive snapshots when datasets are inside the same tree
2) it keeps seperate hourly, daily,.... snapshots, which means it will happily create multiple snapshots with only a few seconds difference for every dataset around midnight.
3) when rolling back several snapshots, there are multiple errors reported because the cache (where does it store that?) does not match reality.

Have these been resolved yet?

I ended up writing my own system for this, got some extra intelligence in there to work around any possible error condition I have encountered.

--
Joost


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] Backup questions

[Wols Lists]

On 14/08/18 12:42, John Covici wrote:
> I use sanoid/syncoid to back up using zfs.  Its great, keeps snapshots
> for as long as I want them (I use 80 days for now).  And it keeps
> hourlies for the last couple of days as well, so I could roll back in
> case of a problem.  Very nice if you use zfs.  I don't think btrfs is
> ready for prime time -- its been under development for a while, but I
> am scared to use it -- I did try once, but got nowhere.

btrfs is pretty reliable apparently unless  you stress it in ways where
it is known to be flaky ... and it's now SUSE's default root filesystem ...

Basically, DO NOT use raid, unless it's raid 1. DO NOT let the disk fill
up if you're using snapshots.

I think one of the reasons it's still got  a bad rap, is a lot of SUSE
installs fell over pretty badly. And the reason they fell over is that
the developers forgot a lot of users don't have  powerful machines ...

The SUSE defaults were to enable snapshotting for upgrades and updates.
And not to check the size of disk allocated to / ... :-( so it was not
unusual for an update to snapshot, install, run out of disk and BOOOOM :-(

Made worse by the fact that when they realised this,  they didn't try
to fix it retrospectively, leaving a lot of time-bombs out there just
waiting to go off.

Fortunately, when I hit this, I had spare disk I could add to the volume
and it promptly sorted itself out, but a simple dedicated root should
not fill up a 10GB partition (or was it 80? I'm not sure ...). But no, a
lot of people have trashed their disks trying to recover from a disk
full scenario :-(

Cheers,
Wol

Re: [gentoo-user] sanoid (was Backup questions)

[John Covici]

On Tue, 14 Aug 2018 16:06:21 -0400,
J. Roeleveld wrote:
> 
> On August 14, 2018 11:42:18 AM UTC, John Covici <covici@ccs.covici.com> wrote:
> 
> >I use sanoid/syncoid to back up using zfs.  Its great, keeps snapshots
> >for as long as I want them (I use 80 days for now).  And it keeps
> >hourlies for the last couple of days as well, so I could roll back in
> >case of a problem.  Very nice if you use zfs.
> 
> I tried sanoid, but it has a few problems which really become annoying when you have a lot of datasets:
> 1) every dataset is handled seperately, no use of recursive snapshots when datasets are inside the same tree
> 2) it keeps seperate hourly, daily,.... snapshots, which means it will happily create multiple snapshots with only a few seconds difference for every dataset around midnight.
> 3) when rolling back several snapshots, there are multiple errors reported because the cache (where does it store that?) does not match reality.
> 
> Have these been resolved yet?
> 
> I ended up writing my own system for this, got some extra intelligence in there to work around any possible error condition I have encountered.
> 

Well, I got around your second point by having a special job at 11:59
pm to create the dailies and the one at midnight works well.  I only
do the cron jobs hourly, not every minute like they wanted.

If your script is not special for you, I would like to see it, maybe I
would use it instead.  Things seem to work for now, however with those
modifications.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici wb2una
         covici@ccs.covici.com

Re: [gentoo-user] sanoid (was Backup questions)

[J. Roeleveld]

On Tuesday, August 14, 2018 10:52:30 PM CEST John Covici wrote:
> On Tue, 14 Aug 2018 16:06:21 -0400,
> 
> J. Roeleveld wrote:
> > On August 14, 2018 11:42:18 AM UTC, John Covici <covici@ccs.covici.com> 
wrote:
> > >I use sanoid/syncoid to back up using zfs.  Its great, keeps snapshots
> > >for as long as I want them (I use 80 days for now).  And it keeps
> > >hourlies for the last couple of days as well, so I could roll back in
> > >case of a problem.  Very nice if you use zfs.
> > 
> > I tried sanoid, but it has a few problems which really become annoying
> > when you have a lot of datasets: 1) every dataset is handled seperately,
> > no use of recursive snapshots when datasets are inside the same tree 2)
> > it keeps seperate hourly, daily,.... snapshots, which means it will
> > happily create multiple snapshots with only a few seconds difference for
> > every dataset around midnight. 3) when rolling back several snapshots,
> > there are multiple errors reported because the cache (where does it store
> > that?) does not match reality.
> > 
> > Have these been resolved yet?
> > 
> > I ended up writing my own system for this, got some extra intelligence in
> > there to work around any possible error condition I have encountered.
> Well, I got around your second point by having a special job at 11:59
> pm to create the dailies and the one at midnight works well.  I only
> do the cron jobs hourly, not every minute like they wanted.
> 
> If your script is not special for you, I would like to see it, maybe I
> would use it instead.  Things seem to work for now, however with those
> modifications.

Current code is too specific for my situation. I think you would be quicker to 
write something yourself instead of modifying my code.

The steps are:
- Check if an snapshot needs to be done (incl. type: hourly, daily, weekly, 
monthly)
- If yes:
1) create a recursive snapshot on the entire pool
2) remove unnecessary snapshots (temp, swap, 
placeholders,not_for_current_type)
3) register snapshots into database
4) clean up old snapshots

Because I register actual snapshots and point snapshot-type entries to these, 
I can quickly determine which snapshots are really unecessary. This also 
drastically reduces the amount of snapshots on the system. (My SAN currently 
has 2611 ZFS snapshots).

I also found it is far quicker to create a recursive snapshot on the entire 
pool and then removing all the unecessarily created ones.

--
Joost

Re: [gentoo-user] sanoid (was Backup questions)

[John Covici]

On Wed, 15 Aug 2018 02:45:20 -0400,
J. Roeleveld wrote:
> 
> On Tuesday, August 14, 2018 10:52:30 PM CEST John Covici wrote:
> > On Tue, 14 Aug 2018 16:06:21 -0400,
> > 
> > J. Roeleveld wrote:
> > > On August 14, 2018 11:42:18 AM UTC, John Covici <covici@ccs.covici.com> 
> wrote:
> > > >I use sanoid/syncoid to back up using zfs.  Its great, keeps snapshots
> > > >for as long as I want them (I use 80 days for now).  And it keeps
> > > >hourlies for the last couple of days as well, so I could roll back in
> > > >case of a problem.  Very nice if you use zfs.
> > > 
> > > I tried sanoid, but it has a few problems which really become annoying
> > > when you have a lot of datasets: 1) every dataset is handled seperately,
> > > no use of recursive snapshots when datasets are inside the same tree 2)
> > > it keeps seperate hourly, daily,.... snapshots, which means it will
> > > happily create multiple snapshots with only a few seconds difference for
> > > every dataset around midnight. 3) when rolling back several snapshots,
> > > there are multiple errors reported because the cache (where does it store
> > > that?) does not match reality.
> > > 
> > > Have these been resolved yet?
> > > 
> > > I ended up writing my own system for this, got some extra intelligence in
> > > there to work around any possible error condition I have encountered.
> > Well, I got around your second point by having a special job at 11:59
> > pm to create the dailies and the one at midnight works well.  I only
> > do the cron jobs hourly, not every minute like they wanted.
> > 
> > If your script is not special for you, I would like to see it, maybe I
> > would use it instead.  Things seem to work for now, however with those
> > modifications.
> 
> Current code is too specific for my situation. I think you would be quicker to 
> write something yourself instead of modifying my code.
> 
> The steps are:
> - Check if an snapshot needs to be done (incl. type: hourly, daily, weekly, 
> monthly)
> - If yes:
> 1) create a recursive snapshot on the entire pool
> 2) remove unnecessary snapshots (temp, swap, 
> placeholders,not_for_current_type)
> 3) register snapshots into database
> 4) clean up old snapshots
> 
> Because I register actual snapshots and point snapshot-type entries to these, 
> I can quickly determine which snapshots are really unecessary. This also 
> drastically reduces the amount of snapshots on the system. (My SAN currently 
> has 2611 ZFS snapshots).
> 
> I also found it is far quicker to create a recursive snapshot on the entire 
> pool and then removing all the unecessarily created ones.

OK, I will look into that.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici wb2una
         covici@ccs.covici.com

Re: [gentoo-user] Backup questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 3311 bytes --]

Am Freitag, 10. August 2018, 04:46:17 CEST schrieb Dale:
> Wols Lists wrote:
> > On 08/08/18 04:43, Dale wrote:
> >> Howdy,
> >> 
> >> I just bought two external drive enclosures.  One is sort of a spare but
> >> I do plan to do some backups on it, mostly pictures from my camera.  In
> >> one of the enclosures I put a single 6TB drive that I found on ebay.  It
> >> has about 7,000 hours on it so it should have some life left yet and it
> >> passed the smartctl tests.  It is USB but it transfers fast.  Now to
> >> some questions.  I use rsync.  Command looks something like rsync -auv
> >> /source/ /destination/.  If I backup the config files in my home
> >> directory, should I also include the --delete option?  If after a
> >> upgrade for example a config file is deleted, because it is no longer
> >> needed, or renamed, should the old file be removed or is there a reason
> >> to keep them on the backups?  Adding the --delete option isn't a problem
> >> command wise BUT I wonder if it can cause a problem at some point.
> >> Thoughts on that.  I plan to use the --delete option for videos since if
> >> I deleted one, it is likely broken or something.  Biggest question is
> >> about config files.
> > 
> > May I suggest using btrfs for your backup drive? One MAJOR caveat - DO
> > NOT let the drive fill up - a combination of snapshots and drive-full
> > has been known (quite often) to trash the file system. But provided you
> > make sure it doesn't go above about 80% you should be fine.
> > 
> > You can add an option to rsync such that it will back up "in place". In
> > other words, if only 1K is changed in a 1M file, it will overwrite that
> > 1K. So when you back up, the procedure is to take a snapshot, then run
> > rsync with both "in place" and "delete".
> > 
> > This will give you the space economy of incremental backups, combined
> > with the utility of full backups - each snapshot is a full backup as of
> > that date, but each new snapshot only increases disk usage by the
> > changes since the last. And you reclaim space by deleting old snapshots.
> 
> I did think about btrfs.  I've read a lot of threads on here about
> people using it and it seems to have come a long ways and be pretty
> stable.  Right now, I've got a lot going on and really don't have the
> time to sit down and read up on it and how it works or what all it can
> do.  In all honesty, if my system were to crash later when I don't have
> so much going on, I'd like to move to btrfs for as much as possible of
> my system.

Yeah, it's a good idea to wait until you have time :) .  And then migrate 
piecemeal, not all at once.  Following up on Wol's suggestion, I would start 
with the backup drive, since you can exploit most of the features there (well, 
snapshots and compression, at least).  Personally, I've had mostly good 
experience with btrfs and enjoy its send/receive feature for full-system 
incremental backups.

> I suspect /boot would still have to be ext2 or something
> because of grub.

GRUB actually supports btrfs.  However, on a UEFI system you will need a FAT32 
file system for /boot, so I would argue that on a relatively recent system the 
issue is moot.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Backup questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

Am Dienstag, 14. August 2018, 12:55:05 CEST schrieb Stefan G. Weichinger:
> Am 09.08.18 um 18:52 schrieb Wols Lists:
> > May I suggest using btrfs for your backup drive?
> 
> I like btrbk:
> 
> https://github.com/digint/btrbk
> 
> I run btrfs as main filesystem on 3 systems (2 laptops, 1 main desktop)
> and so far I am not looking back.
> 
> -> btrbk snapshots to local subvolumes ("time machine") + pulling btrbk
> backups to external disks + btrbk backups to another server.
> 
> Works fine for me.

Works for me, too :) .

In all seriousness, I also ended up at btrbk for driving a btrfs snapshot 
based backup system (btrbk is basically a fancy UI on top of the btrfs send/
receive machinery).  I've been using it for, wow, for almost 3 years now 
(first merged on 7th September 2015).

Greetings
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Backup questions

[Dale]

Marc Joliet wrote:
> Am Freitag, 10. August 2018, 04:46:17 CEST schrieb Dale:
>> Wols Lists wrote:
>>> On 08/08/18 04:43, Dale wrote:
>>>> Howdy,
>>>>
>>>> I just bought two external drive enclosures.  One is sort of a spare but
>>>> I do plan to do some backups on it, mostly pictures from my camera.  In
>>>> one of the enclosures I put a single 6TB drive that I found on ebay.  It
>>>> has about 7,000 hours on it so it should have some life left yet and it
>>>> passed the smartctl tests.  It is USB but it transfers fast.  Now to
>>>> some questions.  I use rsync.  Command looks something like rsync -auv
>>>> /source/ /destination/.  If I backup the config files in my home
>>>> directory, should I also include the --delete option?  If after a
>>>> upgrade for example a config file is deleted, because it is no longer
>>>> needed, or renamed, should the old file be removed or is there a reason
>>>> to keep them on the backups?  Adding the --delete option isn't a problem
>>>> command wise BUT I wonder if it can cause a problem at some point.
>>>> Thoughts on that.  I plan to use the --delete option for videos since if
>>>> I deleted one, it is likely broken or something.  Biggest question is
>>>> about config files.
>>> May I suggest using btrfs for your backup drive? One MAJOR caveat - DO
>>> NOT let the drive fill up - a combination of snapshots and drive-full
>>> has been known (quite often) to trash the file system. But provided you
>>> make sure it doesn't go above about 80% you should be fine.
>>>
>>> You can add an option to rsync such that it will back up "in place". In
>>> other words, if only 1K is changed in a 1M file, it will overwrite that
>>> 1K. So when you back up, the procedure is to take a snapshot, then run
>>> rsync with both "in place" and "delete".
>>>
>>> This will give you the space economy of incremental backups, combined
>>> with the utility of full backups - each snapshot is a full backup as of
>>> that date, but each new snapshot only increases disk usage by the
>>> changes since the last. And you reclaim space by deleting old snapshots.
>> I did think about btrfs.  I've read a lot of threads on here about
>> people using it and it seems to have come a long ways and be pretty
>> stable.  Right now, I've got a lot going on and really don't have the
>> time to sit down and read up on it and how it works or what all it can
>> do.  In all honesty, if my system were to crash later when I don't have
>> so much going on, I'd like to move to btrfs for as much as possible of
>> my system.
> Yeah, it's a good idea to wait until you have time :) .  And then migrate 
> piecemeal, not all at once.  Following up on Wol's suggestion, I would start 
> with the backup drive, since you can exploit most of the features there (well, 
> snapshots and compression, at least).  Personally, I've had mostly good 
> experience with btrfs and enjoy its send/receive feature for full-system 
> incremental backups.
>
>> I suspect /boot would still have to be ext2 or something
>> because of grub.
> GRUB actually supports btrfs.  However, on a UEFI system you will need a FAT32 
> file system for /boot, so I would argue that on a relatively recent system the 
> issue is moot.
>


Yea, time is even more limited now.  My Mom is still in the hospital
about a hour away.  I'm not supposed to visit people there, and other
places where a lot of sick people can be, but it's my Mom.  I went
twice.  The morning after the second visit, I was at the Doctors
office.  Now I'm sick.  Luckily, the meds are working.  Thing is, I
don't feel like messing with computer stuff right now.  Even cooking a
meal is not so interesting.  I can't taste or smell anyway.  So, it may
be a while before I get the time to deal with any of this.  I would
likely not learn anything about a new file system even if I read it more
than once.  I'm even avoiding updates just to prevent anything from
going sideways.  I wonder, how many times will I proof and edit this
email???? 

I thought I read Grub, the new version, supported more file systems. 
Still, just for safety, I'd likely still use ext2.  There's a lot of new
stuff out there.  Just tons of options. 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Backup questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 2586 bytes --]

Am Freitag, 17. August 2018, 07:55:57 CEST schrieb Dale:
> Marc Joliet wrote:
> > Am Freitag, 10. August 2018, 04:46:17 CEST schrieb Dale:
[...]
> >> I did think about btrfs.  I've read a lot of threads on here about
> >> people using it and it seems to have come a long ways and be pretty
> >> stable.  Right now, I've got a lot going on and really don't have the
> >> time to sit down and read up on it and how it works or what all it can
> >> do.  In all honesty, if my system were to crash later when I don't have
> >> so much going on, I'd like to move to btrfs for as much as possible of
> >> my system.
> > 
> > Yeah, it's a good idea to wait until you have time :) .  And then migrate
> > piecemeal, not all at once.  Following up on Wol's suggestion, I would
> > start with the backup drive, since you can exploit most of the features
> > there (well, snapshots and compression, at least).  Personally, I've had
> > mostly good experience with btrfs and enjoy its send/receive feature for
> > full-system incremental backups.
> > 
> >> I suspect /boot would still have to be ext2 or something
> >> because of grub.
> > 
> > GRUB actually supports btrfs.  However, on a UEFI system you will need a
> > FAT32 file system for /boot, so I would argue that on a relatively recent
> > system the issue is moot.
> 
> Yea, time is even more limited now.  My Mom is still in the hospital
> about a hour away.  I'm not supposed to visit people there, and other
> places where a lot of sick people can be, but it's my Mom.  I went
> twice.  The morning after the second visit, I was at the Doctors
> office.  Now I'm sick.  Luckily, the meds are working.  Thing is, I
> don't feel like messing with computer stuff right now.  Even cooking a
> meal is not so interesting.  I can't taste or smell anyway.  So, it may
> be a while before I get the time to deal with any of this.  I would
> likely not learn anything about a new file system even if I read it more
> than once.  I'm even avoiding updates just to prevent anything from
> going sideways.  I wonder, how many times will I proof and edit this
> email???? 

In that case I wish you both a speedy recovery!  It's not like the technology 
will run away from you, anyway.

> I thought I read Grub, the new version, supported more file systems. 
> Still, just for safety, I'd likely still use ext2.  There's a lot of new
> stuff out there.  Just tons of options. 
> 
> Thanks.
> 
> Dale
> 
> :-)  :-) 

Greetings
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]