[gentoo-user] How to build a time machine on Gentoo

[gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 1868 bytes --]

Hello all,

My first post on the list. I thought I would start with something that I
started
to think of as 'essential' after losing 90GB of data. Now I have two main
questions in mind: what to and how to back up on gentoo most efficiently.

1. Apart from users' home directories and the followings, what should be
backed
up on a gentoo machine?
/etc/portage/
/root
/var/lib/portage
...?

2. Erm..okay, I am gonna say, what magic I want and then ask your way.
I first started making gzipped tar balls as follows:

tar czpf /media/backups/userA-`date +%Y.%m.%d`.tgz -X userA-excludelist /etc

But these can get huge especially for home dirs. I also want safe dvd
copies.
Though I can find enough space on the external drives, I don't trust them
any more. See above..sigh..(No I recovered about one third of it with
testdisk/photorec
which names them as file000001 file00002.. and half them are zero sized..
which
quite justifies my agony)

Here is what I wanna do. I want to have only one big backup for, say,
userA-2010.08.07.tgz and other small backup tars containing only the
files/folders that were modified since last update, 2010.08.07, as
userA-diff-2010.08.14.tgz, userA-diff-2010.08.21.tgz,
 userA-diff-2010.08.28.tgz
etc. Now if I want to take the userA back to the future, 2010.08.21,  I want
to
do it by first extracting the huge tar userA-2010.08.07.tgz and then the
tiny
backup userA-diff-2010-08-21.tgz.

But the thing is I don't know how to do this. I am hoping maybe you can tell
me
how to of it as well as a better way of doing backups.

By the way, since I want dvd backups as well, and I want to use +rw dvds so
I
can overwrite old backup after a while, what is best way of ensuring the
integrity and safety of them. Is it a good idea to use truecrypt containers?
Or
nothing tops signing and encrypting with gpg?

Thanks for any comment in advance.

[-- Attachment #2: Type: text/html, Size: 2410 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Alex Schuster]

Nganon <nganon+gentoo@gmail.com> writes:

> Hello all,
> 
> My first post on the list. I thought I would start with something
> that I started
> to think of as 'essential' after losing 90GB of data. Now I have two
> main questions in mind: what to and how to back up on gentoo most
> efficiently.
> 
> 1. Apart from users' home directories and the followings, what should
> be backed
> up on a gentoo machine?
> /etc/portage/
> /root
> /var/lib/portage
> ...?

Wouldn't it be easier to just backup the whole / directory?
Excluding /home, /usr/portage and /var/tmp/portage?


> 2. Erm..okay, I am gonna say, what magic I want and then ask your way.
> I first started making gzipped tar balls as follows:
> 
> tar czpf /media/backups/userA-`date +%Y.%m.%d`.tgz -X
> userA-excludelist /etc
> 
> But these can get huge especially for home dirs. I also want safe dvd
> copies.
> Though I can find enough space on the external drives, I don't trust
> them any more. See above..sigh..(No I recovered about one third of it
> with testdisk/photorec
> which names them as file000001 file00002.. and half them are zero
> sized.. which
> quite justifies my agony)

Okay, but I don't trust DVDs. Although DVD-RAM is quite safe I heard.
But external disks are flexible, offer more space, and if you want more
security, just use yet another drive, so you are safe even if your main
drive and a backup drive fails.

I suggest you have a look at rdiff-backup. It gives you a 1:1 copy of
the source directory, but also does incremental backups, which are
stored (in compressed form) in an additional folder in the destination
directory. I would use this at least for things like /etc, where I
sometimes might want to retrieve an old version of a file. Similar to
your approach with big tar files and small ones containing the
increments.

I use a script for my backups, which I mentioned here on 2010-05-07,
subject 'Snackup'. It optionally creates LVM snapshots so I can make
backups from the running system, even if the source directory is
altered during the backup. This works on LVM only, though, and also
allows the volume to be LUKS-encrypted. It does backups by rdiff-backup,
rsync, cp, tar or dd. It may be overkill when not using the LVM
features, but still I suggest to use some script for backups, so one
does not always have to remember the backup commands. When I want to
update my backup, I enter something like 'snackup boot root home src',
and the script backs up my boot, root and home partition in the
background, and creates tar files each directory in /usr/src.


> By the way, since I want dvd backups as well, and I want to use +rw
> dvds so I can overwrite old backup after a while, what is best way of
> ensuring the integrity and safety of them. Is it a good idea to use
> truecrypt containers? Or nothing tops signing and encrypting with gpg?

I'd use DVD-RAM. The media is a little more expensive, but AFAIK they
were made with long-time backups in mind. And access is much easier,
you just copy the files as to an external drive, no need to burn ISOs.

	Wonko

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 3916 bytes --]

On 16 August 2010 03:15, Alex Schuster <wonko@wonkology.org> wrote:

> Nganon <nganon+gentoo@gmail.com <nganon%2Bgentoo@gmail.com>> writes:
>
> > Hello all,
> >
> > My first post on the list. I thought I would start with something
> > that I started
> > to think of as 'essential' after losing 90GB of data. Now I have two
> > main questions in mind: what to and how to back up on gentoo most
> > efficiently.
> >
> > 1. Apart from users' home directories and the followings, what should
> > be backed
> > up on a gentoo machine?
> > /etc/portage/
> > /root
> > /var/lib/portage
> > ...?
>
> Wouldn't it be easier to just backup the whole / directory?
> Excluding /home, /usr/portage and /var/tmp/portage?
>
> Yes but that would not solve my huge backup file problem, would it?


>  > 2. Erm..okay, I am gonna say, what magic I want and then ask your way.
> > I first started making gzipped tar balls as follows:
> >
> > tar czpf /media/backups/userA-`date +%Y.%m.%d`.tgz -X
> > userA-excludelist /etc
> >
> > But these can get huge especially for home dirs. I also want safe dvd
> > copies.
> > Though I can find enough space on the external drives, I don't trust
> > them any more. See above..sigh..(No I recovered about one third of it
> > with testdisk/photorec
> > which names them as file000001 file00002.. and half them are zero
> > sized.. which
> > quite justifies my agony)
>
> Okay, but I don't trust DVDs. Although DVD-RAM is quite safe I heard.
> But external disks are flexible, offer more space, and if you want more
> security, just use yet another drive, so you are safe even if your main
> drive and a backup drive fails.
>
> I did not give a thought to DVD-RAM before. Will give it a try. Thanks.

I suggest you have a look at rdiff-backup. It gives you a 1:1 copy of
> the source directory, but also does incremental backups, which are
> stored (in compressed form) in an additional folder in the destination
> directory. I would use this at least for things like /etc, where I
> sometimes might want to retrieve an old version of a file. Similar to
> your approach with big tar files and small ones containing the
> increments.
>
> I just emerged and tried it. Seems like incremental backups was what I
was looking for. But from what I see, it is mirroring the src to dist and
storing the metadata/stats of increments as archived. Maybe there is an
option to archive increments along side their metadata as well. I will keep
playing with it.

I use a script for my backups, which I mentioned here on 2010-05-07,
> subject 'Snackup'. It optionally creates LVM snapshots so I can make
> backups from the running system, even if the source directory is
> altered during the backup. This works on LVM only, though, and also
> allows the volume to be LUKS-encrypted. It does backups by rdiff-backup,
> rsync, cp, tar or dd. It may be overkill when not using the LVM
> features, but still I suggest to use some script for backups, so one
> does not always have to remember the backup commands. When I want to
> update my backup, I enter something like 'snackup boot root home src',
> and the script backs up my boot, root and home partition in the
> background, and creates tar files each directory in /usr/src.
>
> I found the thread and script. I am gonna take a look at it. Once I decide
on
how to backup, I am gonna cron a script for it.


>  > By the way, since I want dvd backups as well, and I want to use +rw
> > dvds so I can overwrite old backup after a while, what is best way of
> > ensuring the integrity and safety of them. Is it a good idea to use
> > truecrypt containers? Or nothing tops signing and encrypting with gpg?
>
> I'd use DVD-RAM. The media is a little more expensive, but AFAIK they
> were made with long-time backups in mind. And access is much easier,
> you just copy the files as to an external drive, no need to burn ISOs.
>
>        Wonko
>
>
Thanks for the advises.

[-- Attachment #2: Type: text/html, Size: 5263 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Marco]

On Mon, Aug 16, 2010 at 1:11 AM, Nganon <nganon+gentoo@gmail.com> wrote:
> [...]
> Here is what I wanna do. I want to have only one big backup for, say,
> userA-2010.08.07.tgz and other small backup tars containing only the
> files/folders that were modified since last update, 2010.08.07, as
> userA-diff-2010.08.14.tgz, userA-diff-2010.08.21.tgz,
>  userA-diff-2010.08.28.tgz
> etc. Now if I want to take the userA back to the future, 2010.08.21,  I want
> to
> do it by first extracting the huge tar userA-2010.08.07.tgz and then the
> tiny
> backup userA-diff-2010-08-21.tgz.

backup2l can do exactly what you want:

http://backup2l.sourceforge.net/

HTH!

--
Marco

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

On 16 August 2010 11:36, Marco <listworks@gmail.com> wrote:

> On Mon, Aug 16, 2010 at 1:11 AM, Nganon <nganon+gentoo@gmail.com<nganon%2Bgentoo@gmail.com>>
> wrote:
> > [...]
> > Here is what I wanna do. I want to have only one big backup for, say,
> > userA-2010.08.07.tgz and other small backup tars containing only the
> > files/folders that were modified since last update, 2010.08.07, as
> > userA-diff-2010.08.14.tgz, userA-diff-2010.08.21.tgz,
> >  userA-diff-2010.08.28.tgz
> > etc. Now if I want to take the userA back to the future, 2010.08.21,  I
> want
> > to
> > do it by first extracting the huge tar userA-2010.08.07.tgz and then the
> > tiny
> > backup userA-diff-2010-08-21.tgz.
>
> backup2l can do exactly what you want:
>
> http://backup2l.sourceforge.net/
>
>
Nice one indeed. Exactly does what I want. It is also good that the backups
can be use without the program itself.

It does not seem to be updated since 2009 but I will give it a try.

Thanks

[-- Attachment #2: Type: text/html, Size: 1499 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1491 bytes --]

On Monday 16 August 2010 11:30:36 Nganon wrote:
> On 16 August 2010 11:36, Marco <listworks@gmail.com> wrote:
> > On Mon, Aug 16, 2010 at 1:11 AM, Nganon
> > <nganon+gentoo@gmail.com<nganon%2Bgentoo@gmail.com>>
> > 
> > wrote:
> > > [...]
> > > Here is what I wanna do. I want to have only one big backup for, say,
> > > userA-2010.08.07.tgz and other small backup tars containing only the
> > > files/folders that were modified since last update, 2010.08.07, as
> > > userA-diff-2010.08.14.tgz, userA-diff-2010.08.21.tgz,
> > > 
> > >  userA-diff-2010.08.28.tgz
> > > 
> > > etc. Now if I want to take the userA back to the future, 2010.08.21,  I
> > 
> > want
> > 
> > > to
> > > do it by first extracting the huge tar userA-2010.08.07.tgz and then
> > > the tiny
> > > backup userA-diff-2010-08-21.tgz.
> > 
> > backup2l can do exactly what you want:
> > 
> > http://backup2l.sourceforge.net/
> 
> Nice one indeed. Exactly does what I want. It is also good that the backups
> can be use without the program itself.
> 
> It does not seem to be updated since 2009 but I will give it a try.

Not sure if it's in an overlay, but I don't think it's in portage.

Run eix -l backup and see how many back up tools and scripts pop up.

I have been using tar, star and rsync.  They all work and they can all make 
incremental back ups.  You'll find that a lot of the other 'smart' back up 
applications are based on these anyway.

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Thomas Yao]

On Tue, Aug 17, 2010 at 5:37 AM, Mick <michaelkintzios@gmail.com> wrote:
> Not sure if it's in an overlay, but I don't think it's in portage.
>
> Run eix -l backup and see how many back up tools and scripts pop up.
>
> I have been using tar, star and rsync.  They all work and they can all make
> incremental back ups.  You'll find that a lot of the other 'smart' back up
> applications are based on these anyway.

A disk archive utility called dar is a great staff to backup dir trees and files

-- 
@ghosTM55
Mechanism, not policy

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

On 17 August 2010 02:53, Thomas Yao <t.yao426@gmail.com> wrote:

> On Tue, Aug 17, 2010 at 5:37 AM, Mick <michaelkintzios@gmail.com> wrote:
> > Not sure if it's in an overlay, but I don't think it's in portage.
> >
> > Run eix -l backup and see how many back up tools and scripts pop up.
> >
> > I have been using tar, star and rsync.  They all work and they can all
> make
> > incremental back ups.  You'll find that a lot of the other 'smart' back
> up
> > applications are based on these anyway.
>
> A disk archive utility called dar is a great staff to backup dir trees and
> files
>
>
Looks like the -A switch is exactly what I am looking for:

    This way you can make a first full backup, then many differential
backup,
    each taking as reference the last backup made.

Thanks!

--
> @ghosTM55
> Mechanism, not policy
>
>

[-- Attachment #2: Type: text/html, Size: 1528 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

On 17 August 2010 00:37, Mick <michaelkintzios@gmail.com> wrote:

> On Monday 16 August 2010 11:30:36 Nganon wrote:
> > On 16 August 2010 11:36, Marco <listworks@gmail.com> wrote:
> > > On Mon, Aug 16, 2010 at 1:11 AM, Nganon
> > > <nganon+gentoo@gmail.com <nganon%2Bgentoo@gmail.com><
> nganon%2Bgentoo@gmail.com <nganon%252Bgentoo@gmail.com>>>
> > >
> > > wrote:
> > > > [...]
> > > > Here is what I wanna do. I want to have only one big backup for, say,
> > > > userA-2010.08.07.tgz and other small backup tars containing only the
> > > > files/folders that were modified since last update, 2010.08.07, as
> > > > userA-diff-2010.08.14.tgz, userA-diff-2010.08.21.tgz,
> > > >
> > > >  userA-diff-2010.08.28.tgz
> > > >
> > > > etc. Now if I want to take the userA back to the future, 2010.08.21,
>  I
> > >
> > > want
> > >
> > > > to
> > > > do it by first extracting the huge tar userA-2010.08.07.tgz and then
> > > > the tiny
> > > > backup userA-diff-2010-08-21.tgz.
> > >
> > > backup2l can do exactly what you want:
> > >
> > > http://backup2l.sourceforge.net/
> >
> > Nice one indeed. Exactly does what I want. It is also good that the
> backups
> > can be use without the program itself.
> >
> > It does not seem to be updated since 2009 but I will give it a try.
>
> Not sure if it's in an overlay, but I don't think it's in portage.
>
> Its not in the protage, I found a ebuild proposal at bugs.gentoo but it
didnt even make it to sunrise.

Run eix -l backup and see how many back up tools and scripts pop up.
>
> I know there are dozens of them in the portage. searching for backup
does not answer me 'you can do that by these packages'. Perhaps if NLP
is one day implemented in portage, usage of gentoo user lists will reduce
by half.

I have been using tar, star and rsync.  They all work and they can all make
> incremental back ups.  You'll find that a lot of the other 'smart' back up
> applications are based on these anyway.
>
> I will take a look at star, thanks.


HTH.
> --
> Regards,
> Mick
>

[-- Attachment #2: Type: text/html, Size: 3339 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Maximilian Bräutigam]

Hi

Am 16.08.2010 01:11, schrieb Nganon:
> Hello all,
> 
> My first post on the list. I thought I would start with something that I
> started
> to think of as 'essential' after losing 90GB of data. Now I have two main
> questions in mind: what to and how to back up on gentoo most efficiently.
> 
> 1. Apart from users' home directories and the followings, what should be
> backed
> up on a gentoo machine? 
> /etc/portage/
> /root
> /var/lib/portage
> ...?

You should backup all in / except
/tmp/*
/sys/*
/proc/*
/lost+found/*
/dev/*

I have no solution how to bzip or gzip your backups or how to make a dvd
backup, but I use "app-backup/rsnapshot" which uses rsync but implements
an intelligent rotating system that is done daily, weekly, monthly,
yearly according to your config. Of course you should store the backup
on another physical hdd.

By the way, since a new hdd of one TB is pretty cheap, think about
running your gentoo in a software RAID. Guides:

http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml
http://en.gentoo-wiki.com/wiki/Software_RAID_Install

Kind regards,
der Max

> 2. Erm..okay, I am gonna say, what magic I want and then ask your way.
> I first started making gzipped tar balls as follows:
> 
> tar czpf /media/backups/userA-`date +%Y.%m.%d`.tgz -X userA-excludelist /etc
> 
> But these can get huge especially for home dirs. I also want safe dvd
> copies. 
> Though I can find enough space on the external drives, I don't trust them
> any more. See above..sigh..(No I recovered about one third of it with
> testdisk/photorec  
> which names them as file000001 file00002.. and half them are zero
> sized.. which 
> quite justifies my agony)
> 
> Here is what I wanna do. I want to have only one big backup for, say,
> userA-2010.08.07.tgz and other small backup tars containing only the
> files/folders that were modified since last update, 2010.08.07, as
> userA-diff-2010.08.14.tgz, userA-diff-2010.08.21.tgz,
>  userA-diff-2010.08.28.tgz
> etc. Now if I want to take the userA back to the future, 2010.08.21,  I
> want to
> do it by first extracting the huge tar userA-2010.08.07.tgz and then the
> tiny
> backup userA-diff-2010-08-21.tgz. 
> 
> But the thing is I don't know how to do this. I am hoping maybe you can
> tell me 
> how to of it as well as a better way of doing backups. 
> 
> By the way, since I want dvd backups as well, and I want to use +rw dvds
> so I 
> can overwrite old backup after a while, what is best way of ensuring the 
> integrity and safety of them. Is it a good idea to use truecrypt
> containers? Or 
> nothing tops signing and encrypting with gpg?
> 
> Thanks for any comment in advance. 
> 

Re: [gentoo-user] How to build a time machine on Gentoo

[Alex Schuster]

Maximilian Bräutigam writes:

> You should backup all in / except
> /tmp/*
> /sys/*
> /proc/*
> /lost+found/*
> /dev/*

Attention here, you need at least the null and console entries in /dev, or 
the system will not come up. I also have tty and tty1 in there, I think 
those were neede for tuxonice.

	Wonko

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

2010/8/17 Maximilian Bräutigam <max-braeu@gmx.de>

> You should backup all in / except
> /tmp/*
> /sys/*
> /proc/*
> /lost+found/*
> /dev/*
>
> Distfiles are saved outside the root  and I can afford to rebuild world.
My
main concern was losing (gentoo) config files, speaking of which, I
remembered to back up /usr/src/linux/.config, and user folders.

I have no solution how to bzip or gzip your backups or how to make a dvd
> backup, but I use "app-backup/rsnapshot" which uses rsync but implements
> an intelligent rotating system that is done daily, weekly, monthly,
> yearly according to your config. Of course you should store the backup
> on another physical hdd.
>
> By the way, since a new hdd of one TB is pretty cheap, think about
> running your gentoo in a software RAID. Guides:
>
> http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml
> http://en.gentoo-wiki.com/wiki/Software_RAID_Install
>

Thanks for the advices. I am adding rsnapshot to my list.

[-- Attachment #2: Type: text/html, Size: 1792 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On Tue, 17 Aug 2010 14:14:27 +0200, Maximilian Bräutigam wrote:

> You should backup all in / except
> /tmp/*
> /sys/*
> /proc/*
> /lost+found/*
> /dev/*

That backs up a lot of stuff that isn't needed. As long as you have /etc
and /var/lib you can recreate the system. Depending on space vs. time,
you may prefer not to backup the gigabytes in /usr that can be recreated
by portage (although saving /usr/local is a good idea).

> By the way, since a new hdd of one TB is pretty cheap, think about
> running your gentoo in a software RAID. Guides:

RAID is not an alternative to backups, a corrupted filesystem on a RAID is
just as corrupted as if it were on a single disk, you just get extra
copies of the corruption.


-- 
Neil Bothwick

Despite the cost of living, have you noticed how it remains so popular?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

On 18 August 2010 14:34, Neil Bothwick <neil@digimed.co.uk> wrote:

> On Tue, 17 Aug 2010 14:14:27 +0200, Maximilian Bräutigam wrote:
>
> > You should backup all in / except
> > /tmp/*
> > /sys/*
> > /proc/*
> > /lost+found/*
> > /dev/*
>
> That backs up a lot of stuff that isn't needed. As long as you have /etc
> and /var/lib you can recreate the system. Depending on space vs. time,
> you may prefer not to backup the gigabytes in /usr that can be recreated
> by portage (although saving /usr/local is a good idea).
>
>
Thanks a lot for the valuable advice. I have a dozen of scripts in
/usr/local/bin
that I forgot about.


>  > By the way, since a new hdd of one TB is pretty cheap, think about
> > running your gentoo in a software RAID. Guides:
>
> RAID is not an alternative to backups, a corrupted filesystem on a RAID is
> just as corrupted as if it were on a single disk, you just get extra
> copies of the corruption.
>
>
I did not know that. I was thinking of, in couple of months, buying a
notebook
with two HDDs with RAID1 installed and using the usb drive as a backup
destination. So if RAID got corruped, the backups, made since then, would
be
useless? How would you resolve it?

 --
> Neil Bothwick
>
> Despite the cost of living, have you noticed how it remains so popular?
>

[-- Attachment #2: Type: text/html, Size: 2120 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Bill Longman]

On 08/18/2010 04:53 AM, Nganon wrote:
> I did not know that. I was thinking of, in couple of months, buying a
> notebook 
> with two HDDs with RAID1 installed and using the usb drive as a backup 
> destination. So if RAID got corruped, the backups, made since then,
> would be 
> useless? How would you resolve it? 

The ONLY thing RAID will save you from is hardware failure.

Copying data from one disk to another is not RAID. If you use dd to copy
from one corrupt filesystem to another, you have two corrupt filesystems.

If you want a robust filesystem, look into ZFS/BTRFS.

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

On 18 August 2010 17:53, Bill Longman <bill.longman@gmail.com> wrote:

> On 08/18/2010 04:53 AM, Nganon wrote:
> > I did not know that. I was thinking of, in couple of months, buying a
> > notebook
> > with two HDDs with RAID1 installed and using the usb drive as a backup
> > destination. So if RAID got corruped, the backups, made since then,
> > would be
> > useless? How would you resolve it?
>
> The ONLY thing RAID will save you from is hardware failure.
>
> Copying data from one disk to another is not RAID. If you use dd to copy
> from one corrupt filesystem to another, you have two corrupt filesystems.
>
> Clear now, thanks.


> If you want a robust filesystem, look into ZFS/BTRFS.
>
>
AFAIK ZFS is unmaintained and BTRFS is not stable, am I wrong?

[-- Attachment #2: Type: text/html, Size: 1249 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Bill Longman]

On 08/18/2010 11:03 AM, Nganon wrote:
> Clear now, thanks.
>  
> 
>     If you want a robust filesystem, look into ZFS/BTRFS.
> 
> 
> AFAIK ZFS is unmaintained and BTRFS is not stable, am I wrong? 

Not really. ZFS is only available on Solaris right now. I seem to
remember it was running on one of the BSD's, too, since it's a matter of
licensing that is the hurdle of greatest height. I've only played with
BTRFS on my dev box and the simple workout I gave it did not tax it in
any way--it worked okay.

Re: [gentoo-user] How to build a time machine on Gentoo

[Joerg Schilling]

Bill Longman <bill.longman@gmail.com> wrote:

> On 08/18/2010 11:03 AM, Nganon wrote:
> > Clear now, thanks.
> >  
> > 
> >     If you want a robust filesystem, look into ZFS/BTRFS.
> > 
> > 
> > AFAIK ZFS is unmaintained and BTRFS is not stable, am I wrong? 

Why do you believe ZFS is unmaintained?

> Not really. ZFS is only available on Solaris right now. I seem to
> remember it was running on one of the BSD's, too, since it's a matter of
> licensing that is the hurdle of greatest height. I've only played with
> BTRFS on my dev box and the simple workout I gave it did not tax it in
> any way--it worked okay.

ZFS has a very free license. This was the reason, why it could be ported to the 
BSDs. So why do you believe there is a "license hurdle"?

Also note: btrfs now is three years old. ZFS was started aprox. 10 years ago.
For this reason, btrfs is expected to need another 7 years to readh the level 
of stability currently seen with ZFS.


Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]

On 18 August 2010 21:49, Joerg Schilling <
Joerg.Schilling@fokus.fraunhofer.de> wrote:

> Bill Longman <bill.longman@gmail.com> wrote:
>
> > On 08/18/2010 11:03 AM, Nganon wrote:
> > > Clear now, thanks.
> > >
> > >
> > >     If you want a robust filesystem, look into ZFS/BTRFS.
> > >
> > >
> > > AFAIK ZFS is unmaintained and BTRFS is not stable, am I wrong?
>
> Why do you believe ZFS is unmaintained?
>
>
I was unsure, thats why I asked if I was wrong. My bad.

 > Not really. ZFS is only available on Solaris right now. I seem to
> > remember it was running on one of the BSD's, too, since it's a matter of
> > licensing that is the hurdle of greatest height. I've only played with
> > BTRFS on my dev box and the simple workout I gave it did not tax it in
> > any way--it worked okay.
>
> ZFS has a very free license. This was the reason, why it could be ported to
> the
> BSDs. So why do you believe there is a "license hurdle"?
>
>
Because ZFS is licensed with Sun CDDL, which is incompatible with GNU GPL,
so it cant be distributed with Linux kernel. That's why it is ported to
FUSE.

 Also note: btrfs now is three years old. ZFS was started aprox. 10 years
> ago.
> For this reason, btrfs is expected to need another 7 years to readh the
> level
> of stability currently seen with ZFS.
>
>
ZFS was announced on 2004. So approximately six year, not ten. Besides,
things in
computer world do not always work that linearly, you know.

[-- Attachment #2: Type: text/html, Size: 2366 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Joerg Schilling]

Nganon <nganon+gentoo@gmail.com> wrote:

>  > Not really. ZFS is only available on Solaris right now. I seem to
> > > remember it was running on one of the BSD's, too, since it's a matter of
> > > licensing that is the hurdle of greatest height. I've only played with
> > > BTRFS on my dev box and the simple workout I gave it did not tax it in
> > > any way--it worked okay.
> >
> > ZFS has a very free license. This was the reason, why it could be ported to
> > the
> > BSDs. So why do you believe there is a "license hurdle"?
> >
> >
> Because ZFS is licensed with Sun CDDL, which is incompatible with GNU GPL,
> so it cant be distributed with Linux kernel. That's why it is ported to
> FUSE.

The CDDL is as incompatible to the GPL as the BSD license.

Both do not allow to change the license of the code and for this reason, a code 
combination cannot happen as a "derivative work". If you like to create a 
derivative work from a GPL program and BSD code from another person, you would 
need to declare the code from another person to be _your_ "modification". This
is something that would violate the Copyright law.

The GPL however permits code combinations as "collective work". As the GPL 
then is only valir for the GPL part of the whole, this does not cause 
"incompatibilities".


>  Also note: btrfs now is three years old. ZFS was started aprox. 10 years
> > ago.
> > For this reason, btrfs is expected to need another 7 years to readh the
> > level
> > of stability currently seen with ZFS.
> >
> >
> ZFS was announced on 2004. So approximately six year, not ten. Besides,
> things in
> computer world do not always work that linearly, you know.

In 2002, the ZFS developers have already been forced to have their 
homedirectories on ZFS. In 2004, ZFS was already under test in universities.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] How to build a time machine on Gentoo

[Bill Longman]

On 08/18/2010 11:49 AM, Joerg Schilling wrote:
> Bill Longman <bill.longman@gmail.com> wrote:
> 
>> On 08/18/2010 11:03 AM, Nganon wrote:
>>> Clear now, thanks.
>>>  
>>>
>>>     If you want a robust filesystem, look into ZFS/BTRFS.
>>>
>>>
>>> AFAIK ZFS is unmaintained and BTRFS is not stable, am I wrong? 
> 
> Why do you believe ZFS is unmaintained?

That's Nganon's comment. I'll let him answer.

> 
>> Not really. ZFS is only available on Solaris right now. I seem to
>> remember it was running on one of the BSD's, too, since it's a matter of
>> licensing that is the hurdle of greatest height. I've only played with
>> BTRFS on my dev box and the simple workout I gave it did not tax it in
>> any way--it worked okay.
> 
> ZFS has a very free license. This was the reason, why it could be ported to the 
> BSDs. So why do you believe there is a "license hurdle"?

Only for getting it to run on Linux. The CDDL doesn't play well with GPL.

> Also note: btrfs now is three years old. ZFS was started aprox. 10 years ago.
> For this reason, btrfs is expected to need another 7 years to readh the level 
> of stability currently seen with ZFS.

Might take even less!

Re: [gentoo-user] How to build a time machine on Gentoo

[Alan McKinnon]

Apparently, though unproven, at 20:49 on Wednesday 18 August 2010, Joerg 
Schilling did opine thusly:

> > remember it was running on one of the BSD's, too, since it's a matter of
> > licensing that is the hurdle of greatest height. I've only played with
> > BTRFS on my dev box and the simple workout I gave it did not tax it in
> > any way--it worked okay.
> 
> ZFS has a very free license. This was the reason, why it could be ported to
> the  BSDs. So why do you believe there is a "license hurdle"?

You appear to not fully understand the licenses.

Remember that the Linux kernel is GPL-2 and it's modules are considered 
derivative works. The GPL-2 license demands that all derivative works be 
either GPL-2 licensed or 100% compatible with the GPL-2.

ZFS is licensed CCDL which although free and liberal, is not GPL-2 compatible. 
It is BSD-compatible which is why the BSDs can (and some do) ship it.

The ZFS license is thus not a mere hurdle, it is an un-overcomeable barrier in 
it's current form. If Oracle were to re-license it then the problem could be 
solved, but few in this game hold any hope of that ever happening.

But all of this has been hashed to death many many many times here and in 
other places - to the point where it is now conclusive. Google will reveal the 
entire discussion in all it's painful detail. Start with lkml.

Let's not rehash it here again. Please, I beg of you. Let us not do that.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] How to build a time machine on Gentoo

[Joerg Schilling]

Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> > ZFS has a very free license. This was the reason, why it could be ported to
> > the  BSDs. So why do you believe there is a "license hurdle"?
>
> You appear to not fully understand the licenses.

Well, I of course fully understand the licenses. It may however be that you 
are missinformed because you have been listening to the wrong people. 

> Remember that the Linux kernel is GPL-2 and it's modules are considered 
> derivative works. The GPL-2 license demands that all derivative works be 
> either GPL-2 licensed or 100% compatible with the GPL-2.

This is a claim that in conflict with the US copyright law.

Check out:

http://www.osscc.net/en/gpl.html

There are verious statements from various lawyers that explain this and even 
give evidence for their claims.

The GPL tries to redefine the definition for the term "derivative work" but this 
is forbidden by US Copyright law title 17 section 106. See my other posting for 
more information.


> ZFS is licensed CCDL which although free and liberal, is not GPL-2 compatible. 
> It is BSD-compatible which is why the BSDs can (and some do) ship it.
>
> The ZFS license is thus not a mere hurdle, it is an un-overcomeable barrier in 

The only hurdle is in the brain of some Linux developers.

I know of not a single lawyer that could claim such incompatibility and gives 
evidence for his statements.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] How to build a time machine on Gentoo

[Joerg Schilling]

Let me add some text I wrote for another place but that explains how things 
would work:

1) The OpenSource definition http://www.opensource.org/docs/definition.php 
section 9 makes it very clear that an OSS license must not restrict other 
software and must not prevent to bundle different works under different 
licenses on one medium.

2) given the fact that the GPL is an aproved OSS licensse, it obviously 
complies with the OSS definition.

3) as a result, any GPL interpretation that is based on the assumption that a 
separate distribution would fix problems is wrong.

There is a simple rule:

-	If you modify a GPLd work by your own, so that all you add was written
	by you for this modification only, then you created a "derivative work"
	and you need to put your modifications under GPL.

-	If you add another independent work to a GPLd work, you create a so 
	called "collective work". This is permitted by the GPL. In this case,
	the GPL only applies to the GPLd part and the license for the other 
	work applies to the other work. You need to respect the sum of all
	claims from all licenses in this case. Such a collective work can only
	be distributed if the claims from the licenses are not contradicting.
	If one license e.g. permits redistribution on Mondays only and the
	other permits redistribution on Wednesdays only, you cannot publish
	the collective work.

	ZFS is an independent work with respect to the Linx kernel. It was not
	written for or with the Linux kernel.

-	If like to you add ZFS to the Linux kernel, you first need to create a
	derivative work from ZFS and another derivative work from the Linux 
	kernel in order to allow both to work together. You later create a
	collective work from the combination of the derivative works mentioned
	before. The modification in the ZFS code (in case they appear in 
	files that come with ZFS) need to be put under CDDL, the modifications
	in the Linux kernel need to be put under GPL.

Note: the GPLv3 tries to disallow most collective works, so be careful with 
GPLv3.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

On 18 August 2010 21:37, Bill Longman <bill.longman@gmail.com> wrote:

> On 08/18/2010 11:03 AM, Nganon wrote:
> > Clear now, thanks.
> >
> >
> >     If you want a robust filesystem, look into ZFS/BTRFS.
> >
> >
> > AFAIK ZFS is unmaintained and BTRFS is not stable, am I wrong?
>
> Not really. ZFS is only available on Solaris right now. I seem to
> remember it was running on one of the BSD's, too, since it's a matter of
> licensing that is the hurdle of greatest height. I've only played with
> BTRFS on my dev box and the simple workout I gave it did not tax it in
> any way--it worked okay.
>
>
Right, remembered after reading the 'licensing', it was only available
through FUSE.

I just checked on BTRFS. wiki.kernel.org says BTRFS is stable as of 2.6.29
but still
masked on Gentoo. I will read more about it later and decide whether to
convert.

[-- Attachment #2: Type: text/html, Size: 1328 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[Enrico Weigelt]

* Nganon <nganon+gentoo@gmail.com> wrote:

Hi,

> My first post on the list. I thought I would start with something that I
> started
> to think of as 'essential' after losing 90GB of data. Now I have two main
> questions in mind: what to and how to back up on gentoo most efficiently.

I'm using a little script like that:

    #!/bin/bash
    cd /var/backup/blackwidow || exit 1

    DATE=`date +%F-%H-%M-%S`

    rsync --exclude-from /scr/etc/backup-blackwidow.exclude     \
	  -avz blackwidow:/ /var/backup/blackwidow/ROOT         \
        --backup                                              \
        "--backup-dir=/var/backup/blackwidow/BACKUP-$DATE"    \
        --delete                                              \
        --delete-excluded                                     \
        --delay-updates                                       \
        --progress                                            \


(of course with a carefully maintained exclude file ;-p)
This doesnt make a real rotating backup, but stores all files that
get overwritten into their own directory (named by the current date).
And from time to time, I'm cleaning up the backup volume and look
for things that I still might need.

For things I'd like to keep an history (eg. /etc) I'm using git, and
pushing the repo to a remote server (denying non-fastfoward updates
there, so an theorectical highjacker cannot destroy my history)

For really long-term backups with bigger content (that might be
too big for git), you could also give venti+vac (from plan9port)
a try. I'm using it for storing larger media files (videos, etc)
in my MediaCloud platform.
 
> 1. Apart from users' home directories and the followings, what should be
> backed
> up on a gentoo machine?
> /etc/portage/
> /root
> /var/lib/portage
> ...?

Depends on how long the recovery may take. For example, if you can
afford recompiling world (or have another compatible image somewhere
else), you can exclude everything where packages are installed
(bindirs, libdirs, /usr/share, etc, etc) - assuming everything's
installed by portage.

> Though I can find enough space on the external drives, I don't trust them
> any more. See above..sigh..(No I recovered about one third of it with
> testdisk/photorec
> which names them as file000001 file00002.. and half them are zero sized..
> which
> quite justifies my agony)

Make multiple copies on different media (eg. different servers).
 

cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 321 bytes --]

On 17 August 2010 22:34, Enrico Weigelt <weigelt@metux.de> wrote:

> For things I'd like to keep an history (eg. /etc) I'm using git, and
> pushing the repo to a remote server (denying non-fastfoward updates
> there, so an theorectical highjacker cannot destroy my history)
>

Using git for /etc is a great idea.
Thanks.

[-- Attachment #2: Type: text/html, Size: 609 bytes --]

Re: [gentoo-user] How to build a time machine on Gentoo

[William Kenworthy]

On Wed, 2010-08-18 at 14:09 +0300, Nganon wrote:
> 
> 
> On 17 August 2010 22:34, Enrico Weigelt <weigelt@metux.de> wrote:
>         For things I'd like to keep an history (eg. /etc) I'm using
>         git, and
>         pushing the repo to a remote server (denying non-fastfoward
>         updates
>         there, so an theorectical highjacker cannot destroy my
>         history)
>         
> 
> Using git for /etc is a great idea. 
> Thanks. 
> 
Another option is:
*  app-backup/dirvish
      Latest version available: 1.2.1
      Latest version installed: 1.2.1
      Size of downloaded files: 47 kB
      Homepage:    http://www.dirvish.org/
      Description: Dirvish is a fast, disk based, rotating network
backup system.
      License:     OSL-2.0


Works by first creating a copy (--init) and then hard-linking subsequent
versions of files/directories back to the original original if its
identical.  If a file is changed/new, it is copied instead of linked so
actual space usage quickly stabilises even with a varying number of
versions.  Backup over the network (this is how I have configured mine)
uses rsync over ssh with keys and is "pull" from a cron job on the
backup server or manual on demand (i.e., server initiated).

Version management is by a reasonably sophisticated date of version
scheme where by running "dirvish-expire" deletes out of date versions
(runs in a cron job).  The smart part is that once the last hard link to
file is deleted, its gone, otherwise its kept in the remaining
versions :)

Restore is a simple matter of identifying the version you want and
copying it back - Ive restored individual files through to complete
systems after total disk failure.

Can do includes/excludes, whole systems or just directories such as /etc
and can be easily automated.

Doesnt use compression, but most backup regimes (every day for a weekly
rota + a Sunday kept for 6 months) stabilise at about 2x the original
(gross) copy size, no matter how many copies with average changes
between versions.  Though large scale changes such as an "emerge -e
world" will take more as it will generate new copies of most files.

Downside is it will hammer the destination file system - reiserfs3 works
well, ext2/ext3 have been hopeless everytime I've tried - mass
corruption.  The file system will need a large number of inodes (for
links) if there are an excessive number of files x versions - again
reiserfs3 scores well here.

Highly recommended!

BillK

Re: [gentoo-user] How to build a time machine on Gentoo

[Nganon]

[-- Attachment #1: Type: text/plain, Size: 2899 bytes --]

On 18 August 2010 14:59, William Kenworthy <billk@iinet.net.au> wrote:

> On Wed, 2010-08-18 at 14:09 +0300, Nganon wrote:
> >
> >
> > On 17 August 2010 22:34, Enrico Weigelt <weigelt@metux.de> wrote:
> >         For things I'd like to keep an history (eg. /etc) I'm using
> >         git, and
> >         pushing the repo to a remote server (denying non-fastfoward
> >         updates
> >         there, so an theorectical highjacker cannot destroy my
> >         history)
> >
> >
> > Using git for /etc is a great idea.
> > Thanks.
> >
> Another option is:
> *  app-backup/dirvish
>      Latest version available: 1.2.1
>      Latest version installed: 1.2.1
>      Size of downloaded files: 47 kB
>      Homepage:    http://www.dirvish.org/
>      Description: Dirvish is a fast, disk based, rotating network
> backup system.
>      License:     OSL-2.0
>
>
> Works by first creating a copy (--init) and then hard-linking subsequent
> versions of files/directories back to the original original if its
> identical.  If a file is changed/new, it is copied instead of linked so
> actual space usage quickly stabilises even with a varying number of
> versions.  Backup over the network (this is how I have configured mine)
> uses rsync over ssh with keys and is "pull" from a cron job on the
> backup server or manual on demand (i.e., server initiated).
>
> Version management is by a reasonably sophisticated date of version
> scheme where by running "dirvish-expire" deletes out of date versions
> (runs in a cron job).  The smart part is that once the last hard link to
> file is deleted, its gone, otherwise its kept in the remaining
> versions :)
>
> Restore is a simple matter of identifying the version you want and
> copying it back - Ive restored individual files through to complete
> systems after total disk failure.
>
> Can do includes/excludes, whole systems or just directories such as /etc
> and can be easily automated.
>
> Doesnt use compression, but most backup regimes (every day for a weekly
> rota + a Sunday kept for 6 months) stabilise at about 2x the original
> (gross) copy size, no matter how many copies with average changes
> between versions.  Though large scale changes such as an "emerge -e
> world" will take more as it will generate new copies of most files.
>
> Downside is it will hammer the destination file system - reiserfs3 works
> well, ext2/ext3 have been hopeless everytime I've tried - mass
> corruption.  The file system will need a large number of inodes (for
> links) if there are an excessive number of files x versions - again
> reiserfs3 scores well here.
>
> Highly recommended!
>
> BillK
>
>
>
>
Thanks. It sound just it is made just for this. It even call itself 'time
machine'.
Obviously compression is left out by using links but it sounds kind of
overwhelming to me. I don't have a reiserfs partition and cannot afford to
have one at the mo..

[-- Attachment #2: Type: text/html, Size: 3657 bytes --]