[gentoo-server] Git as backup tool

[gentoo-server] Git as backup tool

[Andrey Utkin]

Hi all! Long live the gentoo masters!
I'd like to hear from anybody who uses (or tried) git on production
servers for saving the points of possible restore. Please, share your
practices, like commit patterns, .gitignore contents, etc. I've begun
to use it a couple of days ago for that, and pointed out some issues.
I control the whole root fs with git.
The problematic part is bunch of files that update frequently, but i
am not familiar with them and i'm not sure if system will load without
them.
Namely, these are files in /usr/lib64/portage/pym/
Also wtmp, utmp files hurt - likely without them box won't boot, but
they shouldn't be in git control, too, coz they update often.
Thus, backup restoring requires not git repo only, but also some tar of base?

-- 
Andrey Utkin

Re: [gentoo-server] Git as backup tool

[Dmitriy Petrov]

Hi!
I also thinking about using git as backup tool :)
And also would like to find out if someone used them in production

On 11/02/2011 01:16 AM, Andrey Utkin wrote:
> Hi all! Long live the gentoo masters!
> I'd like to hear from anybody who uses (or tried) git on production
> servers for saving the points of possible restore. Please, share your
> practices, like commit patterns, .gitignore contents, etc. I've begun
> to use it a couple of days ago for that, and pointed out some issues.
> I control the whole root fs with git.
> The problematic part is bunch of files that update frequently, but i
> am not familiar with them and i'm not sure if system will load without
> them.
> Namely, these are files in /usr/lib64/portage/pym/
> Also wtmp, utmp files hurt - likely without them box won't boot, but
> they shouldn't be in git control, too, coz they update often.
> Thus, backup restoring requires not git repo only, but also some tar of base?
>

Re: [gentoo-server] Git as backup tool

[Tanner Danzey]

Generally, using git is a bad idea for backups (from what I've read)

git stores it's data uncompressed and inefficiently. If you are backing
up things like configuration files or web pages that can change a lot,
sure, but for storing binary files with git, I'd recommend against it,
since binaries vary greatly from version to version (unlike text files)
and you'd just accumulate tons of useless binaries. programs like
duplicity and rsync are great for backups, though.

in all, the drawbacks outweigh the benefits of using a code management
tool to back up entire systems...

On Tue, 2011-11-01 at 23:16 +0200, Andrey Utkin wrote:
> Hi all! Long live the gentoo masters!
> I'd like to hear from anybody who uses (or tried) git on production
> servers for saving the points of possible restore. Please, share your
> practices, like commit patterns, .gitignore contents, etc. I've begun
> to use it a couple of days ago for that, and pointed out some issues.
> I control the whole root fs with git.
> The problematic part is bunch of files that update frequently, but i
> am not familiar with them and i'm not sure if system will load without
> them.
> Namely, these are files in /usr/lib64/portage/pym/
> Also wtmp, utmp files hurt - likely without them box won't boot, but
> they shouldn't be in git control, too, coz they update often.
> Thus, backup restoring requires not git repo only, but also some tar of base?
> 

Re: [gentoo-server] Git as backup tool

[Brian Kroth]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

Tanner Danzey <arkaniad@gmail.com> 2011-11-01 18:50:
> Generally, using git is a bad idea for backups (from what I've read)
>
> git stores it's data uncompressed and inefficiently. If you are backing
> up things like configuration files or web pages that can change a lot,
> sure, but for storing binary files with git, I'd recommend against it,
> since binaries vary greatly from version to version (unlike text files)
> and you'd just accumulate tons of useless binaries. programs like
> duplicity and rsync are great for backups, though.

Agreed.  There are lots of other spin offs, each with their own pros and 
cons: rsnapshot, rdiff, etc.  I personally use some homegrown perl, 
rsync, and zfs snapshots (transparent compression, dedup, each snapshot 
looks like a full backup, etc.).  I'm sure you could use something like 
btrfs in that scheme as well.

However, using git, hg, svn, whatever, for storing your config file 
repositories for something like cfengine, puppet, whatever is a good 
idea, but that's a different issue than backups.

> in all, the drawbacks outweigh the benefits of using a code management
> tool to back up entire systems...
>
> On Tue, 2011-11-01 at 23:16 +0200, Andrey Utkin wrote:
>> Hi all! Long live the gentoo masters!
>> I'd like to hear from anybody who uses (or tried) git on production
>> servers for saving the points of possible restore. Please, share your
>> practices, like commit patterns, .gitignore contents, etc. I've begun
>> to use it a couple of days ago for that, and pointed out some issues.
>> I control the whole root fs with git.
>> The problematic part is bunch of files that update frequently, but i
>> am not familiar with them and i'm not sure if system will load without
>> them.
>> Namely, these are files in /usr/lib64/portage/pym/
>> Also wtmp, utmp files hurt - likely without them box won't boot, but
>> they shouldn't be in git control, too, coz they update often.
>> Thus, backup restoring requires not git repo only, but also some tar of base?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-server] Git as backup tool

[Tanner Danzey]

On Tue, 2011-11-01 at 19:19 -0500, Brian Kroth wrote:
Not to mention finding a previous snapshot.
"deffee46b0ef8c504498a002443ab23019ee0cc9" isn't really a very good
indicator of when said backup was taken. :P
> Tanner Danzey <arkaniad@gmail.com> 2011-11-01 18:50:
> > Generally, using git is a bad idea for backups (from what I've read)
> >
> > git stores it's data uncompressed and inefficiently. If you are backing
> > up things like configuration files or web pages that can change a lot,
> > sure, but for storing binary files with git, I'd recommend against it,
> > since binaries vary greatly from version to version (unlike text files)
> > and you'd just accumulate tons of useless binaries. programs like
> > duplicity and rsync are great for backups, though.
> 
> Agreed.  There are lots of other spin offs, each with their own pros and 
> cons: rsnapshot, rdiff, etc.  I personally use some homegrown perl, 
> rsync, and zfs snapshots (transparent compression, dedup, each snapshot 
> looks like a full backup, etc.).  I'm sure you could use something like 
> btrfs in that scheme as well.
> 
> However, using git, hg, svn, whatever, for storing your config file 
> repositories for something like cfengine, puppet, whatever is a good 
> idea, but that's a different issue than backups.
> 
> > in all, the drawbacks outweigh the benefits of using a code management
> > tool to back up entire systems...
> >
> > On Tue, 2011-11-01 at 23:16 +0200, Andrey Utkin wrote:
> >> Hi all! Long live the gentoo masters!
> >> I'd like to hear from anybody who uses (or tried) git on production
> >> servers for saving the points of possible restore. Please, share your
> >> practices, like commit patterns, .gitignore contents, etc. I've begun
> >> to use it a couple of days ago for that, and pointed out some issues.
> >> I control the whole root fs with git.
> >> The problematic part is bunch of files that update frequently, but i
> >> am not familiar with them and i'm not sure if system will load without
> >> them.
> >> Namely, these are files in /usr/lib64/portage/pym/
> >> Also wtmp, utmp files hurt - likely without them box won't boot, but
> >> they shouldn't be in git control, too, coz they update often.
> >> Thus, backup restoring requires not git repo only, but also some tar of base?

Re: [gentoo-server] Git as backup tool

[Stefan Behte]

Hi,

Am 01.11.2011 22:16, schrieb Andrey Utkin:
> I'd like to hear from anybody who uses (or tried) git on production
> servers for saving the points of possible restore. Please, share your

Yes, I'm doing it in production on some systems. It's ok for small
directories like /etc and /root, but don't try it on binaries. The repo
will grow and grow, it might need large amounts of RAM if you have big
files and you won't have a possiblility to use a good Backup rotation
scheme (http://en.wikipedia.org/wiki/Backup_rotation_scheme).

Go Bacula!

Greetings, Craig

Re: [gentoo-server] Git as backup tool

[Dmitriy Petrov]

Yeah. But `git log` is pretty good :)

On 11/02/2011 04:39 AM, Tanner Danzey wrote:
> On Tue, 2011-11-01 at 19:19 -0500, Brian Kroth wrote:
> Not to mention finding a previous snapshot.
> "deffee46b0ef8c504498a002443ab23019ee0cc9" isn't really a very good
> indicator of when said backup was taken. :P
>> Tanner Danzey<arkaniad@gmail.com>  2011-11-01 18:50:
>>> Generally, using git is a bad idea for backups (from what I've read)
>>>
>>> git stores it's data uncompressed and inefficiently. If you are backing
>>> up things like configuration files or web pages that can change a lot,
>>> sure, but for storing binary files with git, I'd recommend against it,
>>> since binaries vary greatly from version to version (unlike text files)
>>> and you'd just accumulate tons of useless binaries. programs like
>>> duplicity and rsync are great for backups, though.
>> Agreed.  There are lots of other spin offs, each with their own pros and
>> cons: rsnapshot, rdiff, etc.  I personally use some homegrown perl,
>> rsync, and zfs snapshots (transparent compression, dedup, each snapshot
>> looks like a full backup, etc.).  I'm sure you could use something like
>> btrfs in that scheme as well.
>>
>> However, using git, hg, svn, whatever, for storing your config file
>> repositories for something like cfengine, puppet, whatever is a good
>> idea, but that's a different issue than backups.
>>
>>> in all, the drawbacks outweigh the benefits of using a code management
>>> tool to back up entire systems...
>>>
>>> On Tue, 2011-11-01 at 23:16 +0200, Andrey Utkin wrote:
>>>> Hi all! Long live the gentoo masters!
>>>> I'd like to hear from anybody who uses (or tried) git on production
>>>> servers for saving the points of possible restore. Please, share your
>>>> practices, like commit patterns, .gitignore contents, etc. I've begun
>>>> to use it a couple of days ago for that, and pointed out some issues.
>>>> I control the whole root fs with git.
>>>> The problematic part is bunch of files that update frequently, but i
>>>> am not familiar with them and i'm not sure if system will load without
>>>> them.
>>>> Namely, these are files in /usr/lib64/portage/pym/
>>>> Also wtmp, utmp files hurt - likely without them box won't boot, but
>>>> they shouldn't be in git control, too, coz they update often.
>>>> Thus, backup restoring requires not git repo only, but also some tar of base?
>
>

Re: [gentoo-server] Git as backup tool

[Tanner Danzey]

Yeah, its fine with /etc and directories of that sort because those
files changes are usually subtle :P

On Wed, 2011-11-02 at 02:21 +0100, Stefan Behte wrote:
> Hi,
> 
> Am 01.11.2011 22:16, schrieb Andrey Utkin:
> > I'd like to hear from anybody who uses (or tried) git on production
> > servers for saving the points of possible restore. Please, share your
> 
> Yes, I'm doing it in production on some systems. It's ok for small
> directories like /etc and /root, but don't try it on binaries. The repo
> will grow and grow, it might need large amounts of RAM if you have big
> files and you won't have a possiblility to use a good Backup rotation
> scheme (http://en.wikipedia.org/wiki/Backup_rotation_scheme).
> 
> Go Bacula!
> 
> Greetings, Craig
> 
> 

Re: [gentoo-server] Git as backup tool

[Tanner Danzey]

As long as you document your changelogs well! 

On Wed, 2011-11-02 at 06:06 +0400, Dmitriy Petrov wrote:
> Yeah. But `git log` is pretty good :)
> 
> On 11/02/2011 04:39 AM, Tanner Danzey wrote:
> > On Tue, 2011-11-01 at 19:19 -0500, Brian Kroth wrote:
> > Not to mention finding a previous snapshot.
> > "deffee46b0ef8c504498a002443ab23019ee0cc9" isn't really a very good
> > indicator of when said backup was taken. :P
> >> Tanner Danzey<arkaniad@gmail.com>  2011-11-01 18:50:
> >>> Generally, using git is a bad idea for backups (from what I've read)
> >>>
> >>> git stores it's data uncompressed and inefficiently. If you are backing
> >>> up things like configuration files or web pages that can change a lot,
> >>> sure, but for storing binary files with git, I'd recommend against it,
> >>> since binaries vary greatly from version to version (unlike text files)
> >>> and you'd just accumulate tons of useless binaries. programs like
> >>> duplicity and rsync are great for backups, though.
> >> Agreed.  There are lots of other spin offs, each with their own pros and
> >> cons: rsnapshot, rdiff, etc.  I personally use some homegrown perl,
> >> rsync, and zfs snapshots (transparent compression, dedup, each snapshot
> >> looks like a full backup, etc.).  I'm sure you could use something like
> >> btrfs in that scheme as well.
> >>
> >> However, using git, hg, svn, whatever, for storing your config file
> >> repositories for something like cfengine, puppet, whatever is a good
> >> idea, but that's a different issue than backups.
> >>
> >>> in all, the drawbacks outweigh the benefits of using a code management
> >>> tool to back up entire systems...
> >>>
> >>> On Tue, 2011-11-01 at 23:16 +0200, Andrey Utkin wrote:
> >>>> Hi all! Long live the gentoo masters!
> >>>> I'd like to hear from anybody who uses (or tried) git on production
> >>>> servers for saving the points of possible restore. Please, share your
> >>>> practices, like commit patterns, .gitignore contents, etc. I've begun
> >>>> to use it a couple of days ago for that, and pointed out some issues.
> >>>> I control the whole root fs with git.
> >>>> The problematic part is bunch of files that update frequently, but i
> >>>> am not familiar with them and i'm not sure if system will load without
> >>>> them.
> >>>> Namely, these are files in /usr/lib64/portage/pym/
> >>>> Also wtmp, utmp files hurt - likely without them box won't boot, but
> >>>> they shouldn't be in git control, too, coz they update often.
> >>>> Thus, backup restoring requires not git repo only, but also some tar of base?
> >
> >
> 
> 

Re: [gentoo-server] Git as backup tool

[Andrey Utkin]

2011/11/2 Stefan Behte <craig@gentoo.org>:
> Yes, I'm doing it in production on some systems. It's ok for small
> directories like /etc and /root, but don't try it on binaries. The repo
> will grow and grow

Git allows removing old revisions.

>, it might need large amounts of RAM if you have big
> files

possibly

> and you won't have a possiblility to use a good Backup rotation
> scheme (http://en.wikipedia.org/wiki/Backup_rotation_scheme).

I didn't get your point.
Did you mean that this way i won't have ability to choose from
multiple recovery points?
Or that i can't store the backups in several different places?
Please, elaborate.

> Go Bacula!

Thanks, I'll try.

Thanks 2 all for replies.

-- 
Andrey Utkin

Re: [gentoo-server] Git as backup tool

[Ciprian Dorin Craciun]

On Tue, Nov 1, 2011 at 23:16, Andrey Utkin
<andrey.krieger.utkin@gmail.com> wrote:
> Hi all! Long live the gentoo masters!
> I'd like to hear from anybody who uses (or tried) git on production
> servers for saving the points of possible restore. Please, share your
> practices, like commit patterns, .gitignore contents, etc. I've begun
> to use it a couple of days ago for that, and pointed out some issues.
> I control the whole root fs with git.
> The problematic part is bunch of files that update frequently, but i
> am not familiar with them and i'm not sure if system will load without
> them.
> Namely, these are files in /usr/lib64/portage/pym/
> Also wtmp, utmp files hurt - likely without them box won't boot, but
> they shouldn't be in git control, too, coz they update often.
> Thus, backup restoring requires not git repo only, but also some tar of base?
>
> --
> Andrey Utkin


    There is an interesting project targeting exactly this: using Git
as a backend for a backup tool:
        https://github.com/apenwarr/bup

    Ciprian.

    P.S.: I haven't tried it myself, but I've subscribed to their
mailing list and I keep a close eye... And when I have some time I
want to try it...