[gentoo-user] {OT} rdiff-backup: push or pull?

[gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

I'm setting up an automated rdiff-backup system and I'm stuck between
pushing the backups to the backup server, and pulling the backups to
the backup server.  If I push, I have to allow read/write access of my
backups via SSH keys.  If I pull, I have to enable root logins on each
system to be backed-up, allow root read access of each system via SSH
keys, and I have to deal with openvpn or ssh -R so my laptop can back
up from behind foreign routers.  The conventional wisdom online seems
to indicate pulling is better, but pushing seems like it might be
better to me.  Do you push or pull?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Monday, August 15, 2011 09:58:00 PM Grant wrote:
> I'm setting up an automated rdiff-backup system and I'm stuck between
> pushing the backups to the backup server, and pulling the backups to
> the backup server.  If I push, I have to allow read/write access of my
> backups via SSH keys.  If I pull, I have to enable root logins on each
> system to be backed-up, allow root read access of each system via SSH
> keys, and I have to deal with openvpn or ssh -R so my laptop can back
> up from behind foreign routers.  The conventional wisdom online seems
> to indicate pulling is better, but pushing seems like it might be
> better to me.  Do you push or pull?

I would push, to be honest.

You can seperate the backups by giving each system a different account where 
to store the backups.
This way you can also have better control over when to do the backup. If your 
laptop hooks up via VPN just to quickly check email over an expensive or slow 
link, you might not want the backup to start downloading all the pictures you 
took during the holiday or that 300-page manuscript you wrote for your book.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> I'm setting up an automated rdiff-backup system and I'm stuck between
>> pushing the backups to the backup server, and pulling the backups to
>> the backup server.  If I push, I have to allow read/write access of my
>> backups via SSH keys.  If I pull, I have to enable root logins on each
>> system to be backed-up, allow root read access of each system via SSH
>> keys, and I have to deal with openvpn or ssh -R so my laptop can back
>> up from behind foreign routers.  The conventional wisdom online seems
>> to indicate pulling is better, but pushing seems like it might be
>> better to me.  Do you push or pull?
>
> I would push, to be honest.

Me too.  The rdiff-backup "UnattendedRdiff" wiki page only has
instructions for pulling but that doesn't seem like the way to go:

http://wiki.rdiff-backup.org/wiki/index.php/UnattendedRdiff

> You can seperate the backups by giving each system a different account where
> to store the backups.

I'm not sure what you mean.  The backups are all stored on the backup server.

> This way you can also have better control over when to do the backup. If your
> laptop hooks up via VPN just to quickly check email over an expensive or slow
> link, you might not want the backup to start downloading all the pictures you
> took during the holiday or that 300-page manuscript you wrote for your book.
>
> --
> Joost

Here's what I'm doing.  root on 3 machines pushes to non-root on a 4th
machine via rdiff-backup and SSH keys.  The SSH keys are restricted
like so (although there is no from= for the laptop's key since it
could be behind any IP):

command="rdiff-backup
--server",from="12.34.56.78",no-port-forwarding,no-X11-forwarding,no-pty
ssh-rsa ... root@machine1

Is this a good arrangement?  I think the worst-case scenario
(compromised SSH keys) is read/write access of the non-root user on
the backup server via rdiff-backup.

Additionally, the backups on the 4th machine are pushed to another
machine by root to non-root via rsync and SSH keys.  Is there a way to
restrict SSH keys to the rsync command?

Should the non-root backup user have any special configuration?

Can I reserve 0% for root on my USB hard drive which is only used for
backups and does not contain an OS?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Tuesday, August 16, 2011 04:50:40 PM Grant wrote:
> > You can seperate the backups by giving each system a different account
> > where to store the backups.
> 
> I'm not sure what you mean.  The backups are all stored on the backup
> server.

Each machine to be backed up has a different account on the backup server. 
This will prevent machine A from accessing the backups of machine B.

This way, if one machine is compromised, only this machines backups can be 
accessed using the access-keys for the backup. And this machines keys can then 
be revoked without affecting other backups.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> > You can seperate the backups by giving each system a different account
>> > where to store the backups.
>>
>> I'm not sure what you mean.  The backups are all stored on the backup
>> server.
>
> Each machine to be backed up has a different account on the backup server.
> This will prevent machine A from accessing the backups of machine B.
>
> This way, if one machine is compromised, only this machines backups can be
> accessed using the access-keys for the backup. And this machines keys can then
> be revoked without affecting other backups.

That's a great idea.  I will do that.  Should that backup account have
any special configuration, or just a standard new user?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Wednesday, August 17, 2011 10:18:25 AM Grant wrote:
> >> > You can seperate the backups by giving each system a different
> >> > account
> >> > where to store the backups.
> >> 
> >> I'm not sure what you mean.  The backups are all stored on the backup
> >> server.
> > 
> > Each machine to be backed up has a different account on the backup
> > server. This will prevent machine A from accessing the backups of
> > machine B.
> > 
> > This way, if one machine is compromised, only this machines backups can
> > be accessed using the access-keys for the backup. And this machines
> > keys can then be revoked without affecting other backups.
> 
> That's a great idea.  I will do that.  Should that backup account have
> any special configuration, or just a standard new user?

I would suspect just a standard new user with default permissions.
Eg. only write-access to his/her own files.

And I'd prevent that user account from being able to get a shell-account.

A ".bashrc" with "exit" as the last or first entry is a nice touch. Especially 
if you set the permissions such that it works for the user but the user can 
never change that file.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> >> > You can seperate the backups by giving each system a different
>> >> > account
>> >> > where to store the backups.
>> >>
>> >> I'm not sure what you mean.  The backups are all stored on the backup
>> >> server.
>> >
>> > Each machine to be backed up has a different account on the backup
>> > server. This will prevent machine A from accessing the backups of
>> > machine B.
>> >
>> > This way, if one machine is compromised, only this machines backups can
>> > be accessed using the access-keys for the backup. And this machines
>> > keys can then be revoked without affecting other backups.
>>
>> That's a great idea.  I will do that.  Should that backup account have
>> any special configuration, or just a standard new user?
>
> I would suspect just a standard new user with default permissions.
> Eg. only write-access to his/her own files.
>
> And I'd prevent that user account from being able to get a shell-account.

I created the backup users and everything works as long as the backup
users have shells on the backup server and are listed in AllowUsers in
/etc/ssh/sshd_config on the backup server.  Did I do something wrong
or should the backup users need shells and to be listed in AllowUsers?

Should I set up any extra restrictions for them in sshd_config?
Should I set passwords for them?

- Grant


> A ".bashrc" with "exit" as the last or first entry is a nice touch. Especially
> if you set the permissions such that it works for the user but the user can
> never change that file.
>
> --
> Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Thursday, August 18, 2011 06:01:08 PM Grant wrote:
> >> >> > You can seperate the backups by giving each system a
> >> >> > different
> >> >> > account
> >> >> > where to store the backups.
> >> >> 
> >> >> I'm not sure what you mean.  The backups are all stored on the
> >> >> backup
> >> >> server.
> >> > 
> >> > Each machine to be backed up has a different account on the backup
> >> > server. This will prevent machine A from accessing the backups of
> >> > machine B.
> >> > 
> >> > This way, if one machine is compromised, only this machines
> >> > backups can be accessed using the access-keys for the backup. And
> >> > this machines keys can then be revoked without affecting other
> >> > backups.
> >> 
> >> That's a great idea.  I will do that.  Should that backup account have
> >> any special configuration, or just a standard new user?
> > 
> > I would suspect just a standard new user with default permissions.
> > Eg. only write-access to his/her own files.
> > 
> > And I'd prevent that user account from being able to get a
> > shell-account.
> 
> I created the backup users and everything works as long as the backup
> users have shells on the backup server and are listed in AllowUsers in
> /etc/ssh/sshd_config on the backup server.  Did I do something wrong
> or should the backup users need shells and to be listed in AllowUsers?

I'm not too familiar with rsync backups. A shell might be required, but if you 
set the command run on the server-side in the "authorized_keys" it should 
prevent any other command from being run.

> Should I set up any extra restrictions for them in sshd_config?

I have disabled all password-logins and only allow shared-key logins.

> Should I set passwords for them?

I don't set passwords for these type of users. By default, they can not login 
with any password that way. Setting a password will leave the possibility open 
someone might randomly guess the password.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> I created the backup users and everything works as long as the backup
>> users have shells on the backup server and are listed in AllowUsers in
>> /etc/ssh/sshd_config on the backup server.  Did I do something wrong
>> or should the backup users need shells and to be listed in AllowUsers?
>
> I'm not too familiar with rsync backups. A shell might be required, but if you
> set the command run on the server-side in the "authorized_keys" it should
> prevent any other command from being run.

I'm actually talking about rdiff-backup.  I'm prompted for a password
if the backup user doesn't have a shell.  Are you able to rdiff-backup
without a shell on the backup server?

>> Should I set up any extra restrictions for them in sshd_config?
>
> I have disabled all password-logins and only allow shared-key logins.

I want to be prompted for a password with my normal user but I want
the backup users to be restricted.  I tried
'ChallengeResponseAuthentication no' within a Match block for a backup
user but ChallengeResponseAuthentication isn't allowed in a Match
block.  Are my options to restrict all users or none?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Tuesday, August 16, 2011 04:50:40 PM Grant wrote:
>  Is there a way to
> restrict SSH keys to the rsync command?

Yes, via the "authorized_keys" file. you can add a "command" directive. this 
will always force that command to be executed whenever a connection is made 
using this key.

See the "sshd" manpage for details.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>>  Is there a way to
>> restrict SSH keys to the rsync command?
>
> Yes, via the "authorized_keys" file. you can add a "command" directive. this
> will always force that command to be executed whenever a connection is made
> using this key.

I'm using the command directive with rdiff-backup like
command="rdiff-backup --server" but I can't figure out the rsync
command to specify.  Is anyone restricting an SSK key to rsync with
the command directive?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Michael Orlitzky]

On 08/17/11 13:35, Grant wrote:
>>>  Is there a way to
>>> restrict SSH keys to the rsync command?
>>
>> Yes, via the "authorized_keys" file. you can add a "command" directive. this
>> will always force that command to be executed whenever a connection is made
>> using this key.
> 
> I'm using the command directive with rdiff-backup like
> command="rdiff-backup --server" but I can't figure out the rsync
> command to specify.  Is anyone restricting an SSK key to rsync with
> the command directive?
> 

We're doing the same thing for our backups. Here's that chunk of our
documentation, if it's helpful.


=== rdiff-backup Client ===

==== Creating the Remote User ====

First, create a new system user on the backup server. Log in (as root),
and run,

  useradd -d /home/<username> -m <username>

The ''-d'' parameter sets the home directory, and ''-m'' creates it
automatically.

The rdiff-backup program uses SSH to synchronize the local and remote
filesystems. As a result, non-interactive operation requires a
server/client certificate pair. Furthermore, we cannot prevent shell
logins for our new user account.

Give it a reasonably-complex password. You'll only need to type it twice:

  passwd <username>

==== Installing rdiff-backup ====

First things first; install rdiff-backup on the client. In Gentoo, all
this requires is the following,

  emerge rdiff-backup

If that works, go ahead and continue.

==== Setting up SSH Authentication ====

For now, we're done on the backup server. Log in to the client server
(the one to be backed up) as root. We need to generate an SSH key pair:

  ssh-keygen

Name the file something informative when asked. '''Do not create a
password for the key file.''' For example, your private key for
<backup_server> might be named ~/.ssh/<backup_server>_rsa. Now, copy the
public key, e.g. ~/.ssh/<backup_server>_rsa.pub to the backup server
using the user that we created earlier.

  scp ~/.ssh/<public_key_file> <remote_user>@<backup_server>:~/


And add a section to the local ~/.ssh/config file which corresponds to
the backup server. This forces the local machine to authenticate to the
backup server using its key rather than a password.

<pre>
Host <backup_server_hostname>
   Hostname <backup_server_hostname>
   IdentityFile ~/.ssh/<private_key_file>
   IdentitiesOnly yes
</pre>


Now, ssh into the backup server as your new user. Our goal is to add
this key as "trusted," allowing anyone with the corresponding key to
connect as this user. On the backup server (as our new user), execute,

 cat <public_key_file> >> ~/.ssh/authorized_keys
 rm <public_key_file>

and add the following to the authorized_keys file manually. Add it at
the beginning of the line for the new public key.

  command="/usr/bin/rdiff-backup --server",no-pty,no-port-forwarding

This will restrict the user with this public key to executing only the
rdiff-server command.

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

> We're doing the same thing for our backups. Here's that chunk of our
> documentation, if it's helpful.

Thanks Michael.  You've found that a shell account is required on the
backup server in order to push backups to it?

Is the purpose of the Host block in .ssh/config to store the hostname
of the backup server so it doesn't need to be used directly in the
rdiff-backup command?

Why create a password for the backup user?  Doesn't that open up the
possibility of someone logging in as that user, when otherwise the
account would only be used for backing up files?

- Grant


> === rdiff-backup Client ===
>
> ==== Creating the Remote User ====
>
> First, create a new system user on the backup server. Log in (as root),
> and run,
>
>  useradd -d /home/<username> -m <username>
>
> The ''-d'' parameter sets the home directory, and ''-m'' creates it
> automatically.
>
> The rdiff-backup program uses SSH to synchronize the local and remote
> filesystems. As a result, non-interactive operation requires a
> server/client certificate pair. Furthermore, we cannot prevent shell
> logins for our new user account.
>
> Give it a reasonably-complex password. You'll only need to type it twice:
>
>  passwd <username>
>
> ==== Installing rdiff-backup ====
>
> First things first; install rdiff-backup on the client. In Gentoo, all
> this requires is the following,
>
>  emerge rdiff-backup
>
> If that works, go ahead and continue.
>
> ==== Setting up SSH Authentication ====
>
> For now, we're done on the backup server. Log in to the client server
> (the one to be backed up) as root. We need to generate an SSH key pair:
>
>  ssh-keygen
>
> Name the file something informative when asked. '''Do not create a
> password for the key file.''' For example, your private key for
> <backup_server> might be named ~/.ssh/<backup_server>_rsa. Now, copy the
> public key, e.g. ~/.ssh/<backup_server>_rsa.pub to the backup server
> using the user that we created earlier.
>
>  scp ~/.ssh/<public_key_file> <remote_user>@<backup_server>:~/
>
>
> And add a section to the local ~/.ssh/config file which corresponds to
> the backup server. This forces the local machine to authenticate to the
> backup server using its key rather than a password.
>
> <pre>
> Host <backup_server_hostname>
>   Hostname <backup_server_hostname>
>   IdentityFile ~/.ssh/<private_key_file>
>   IdentitiesOnly yes
> </pre>
>
>
> Now, ssh into the backup server as your new user. Our goal is to add
> this key as "trusted," allowing anyone with the corresponding key to
> connect as this user. On the backup server (as our new user), execute,
>
>  cat <public_key_file> >> ~/.ssh/authorized_keys
>  rm <public_key_file>
>
> and add the following to the authorized_keys file manually. Add it at
> the beginning of the line for the new public key.
>
>  command="/usr/bin/rdiff-backup --server",no-pty,no-port-forwarding
>
> This will restrict the user with this public key to executing only the
> rdiff-server command.

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Michael Orlitzky]

On 08/19/11 14:00, Grant wrote:
>> We're doing the same thing for our backups. Here's that chunk of our
>> documentation, if it's helpful.
> 
> Thanks Michael.  You've found that a shell account is required on the
> backup server in order to push backups to it?

Yes, you have to be able to run a command (rdiff-backup --server...) and
that requires a shell. I tried to do it without a shell, but couldn't
figure out how to do it sensibly. I do `chmod 700` all home directories.


> Is the purpose of the Host block in .ssh/config to store the hostname
> of the backup server so it doesn't need to be used directly in the
> rdiff-backup command?

It forces key-based authentication when connecting to the backup server.
The default is password-based, which obviously won't work in a cron job.


> Why create a password for the backup user?  Doesn't that open up the
> possibility of someone logging in as that user, when otherwise the
> account would only be used for backing up files?

It might work without one; in these instructions the
machine-to-be-backed-up never connects to the backup server as root, and
so you need a way to SCP stuff to the backup server. I usually use a
`pwgen 16` password for these accounts and then immediately forget it,
so nobody will log in to them for a few billion years at least.

Does key-based authentication work with no password? I've never tried.

I am emotionally troubled by the existence of local shell accounts, but
rationally, I know that no one can ever log in to them.

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> Is the purpose of the Host block in .ssh/config to store the hostname
>> of the backup server so it doesn't need to be used directly in the
>> rdiff-backup command?
>
> It forces key-based authentication when connecting to the backup server.
> The default is password-based, which obviously won't work in a cron job.

I don't use an .ssh/config at all and I'm not prompted for a password
if the keys are in place.  My sshd_config is pretty much default and
my normal user is prompted for a password.

>> Why create a password for the backup user?  Doesn't that open up the
>> possibility of someone logging in as that user, when otherwise the
>> account would only be used for backing up files?
>
> It might work without one; in these instructions the
> machine-to-be-backed-up never connects to the backup server as root, and
> so you need a way to SCP stuff to the backup server. I usually use a
> `pwgen 16` password for these accounts and then immediately forget it,
> so nobody will log in to them for a few billion years at least.
>
> Does key-based authentication work with no password? I've never tried.

It does! :)

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Alan McKinnon]

On Fri 19 August 2011 12:58:10 Grant did opine thusly:
> >> Is the purpose of the Host block in .ssh/config to store the
> >> hostname of the backup server so it doesn't need to be used
> >> directly in the rdiff-backup command?
> > 
> > It forces key-based authentication when connecting to the backup
> > server. The default is password-based, which obviously won't
> > work in a cron job.
> I don't use an .ssh/config at all and I'm not prompted for a
> password if the keys are in place.  My sshd_config is pretty much
> default and my normal user is prompted for a password.


sshd can use various schemes for user authentication. The overall 
process is:

user connects
user is authenticated somehow
user's shell is launched

The middle step is highly variable. sshd can do all of it itself using 
only keys, or it could be happy with password authentication, it can 
even use PAM and obey whatever yes/no result PAM comes back with. 

sshd runs as root (therefore with access to /etc/shadow) so it could 
even validate passwords itself if it wanted, bypassing login and PAM 
entirely. This is of course a silly idea, but still technically 
feasible.
.
.ssh/config is only useful when the user desires options different 
from the global defaults in /etc/ssh/sshd_config, or wants to do extra 
actions for specific destination hosts



> 
> >> Why create a password for the backup user?  Doesn't that open
> >> up the possibility of someone logging in as that user, when
> >> otherwise the account would only be used for backing up
> >> files?
> > 
> > It might work without one; in these instructions the
> > machine-to-be-backed-up never connects to the backup server as
> > root, and so you need a way to SCP stuff to the backup server.
> > I usually use a `pwgen 16` password for these accounts and then
> > immediately forget it, so nobody will log in to them for a few
> > billion years at least.
> > 
> > Does key-based authentication work with no password? I've never
> > tried.
> It does! :)
> 
> - Grant
-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Tuesday, August 16, 2011 04:50:40 PM Grant wrote:
> Can I reserve 0% for root on my USB hard drive which is only used for
> backups and does not contain an OS?

Yes:

mke2fs -m 0 /dev/usb-drive

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> Can I reserve 0% for root on my USB hard drive which is only used for
>> backups and does not contain an OS?
>
> Yes:
>
> mke2fs -m 0 /dev/usb-drive

Thank you, is it safe to do so on a disk like that?  If I run out of
space on the USB hard drive, I'll only need space on the OS disk to
fix it, correct?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Alex Schuster]

Joost Roeleveld writes:

> On Tuesday, August 16, 2011 04:50:40 PM Grant wrote:
> > Can I reserve 0% for root on my USB hard drive which is only used for
> > backups and does not contain an OS?
> 
> Yes:
> 
> mke2fs -m 0 /dev/usb-drive

Although a value > 0 helps against fragmentation. And when rdiff-backup has 
failed because it ran out of space, regressing to the previous sane state 
will need a little free space.

	Wonko

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> > Can I reserve 0% for root on my USB hard drive which is only used for
>> > backups and does not contain an OS?
>>
>> Yes:
>>
>> mke2fs -m 0 /dev/usb-drive
>
> Although a value > 0 helps against fragmentation. And when rdiff-backup has
> failed because it ran out of space, regressing to the previous sane state
> will need a little free space.

Good points.  Should 10GB (1% of 1TB) do it?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Alex Schuster]

Grant writes:

> >> > Can I reserve 0% for root on my USB hard drive which is only
> >> > used for backups and does not contain an OS?
> >> 
> >> Yes:
> >> 
> >> mke2fs -m 0 /dev/usb-drive
> > 
> > Although a value > 0 helps against fragmentation. And when
> > rdiff-backup has failed because it ran out of space, regressing to
> > the previous sane state will need a little free space.
> 
> Good points.  Should 10GB (1% of 1TB) do it?

This I don't know. I use this value for large partitions of multimedia data, 
because I do not want to waste space (no matter how big the drives are, mine 
are always quite full), and performance should not be a big issue here. I 
keep the 5% default other partitions, like /home. BTW, you can also specify 
fractions like 0.5% if you like.

Another thing: Be sure to have enough inodes on the file system, I have run 
out of them in the past. Not only once.
Other than the percentage of reserved blocks, which can be changed with 
tune2fs -m, this value is fixed. If you have too few, you need to re-create 
the file system.

	Wonko

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Alan McKinnon]

On Wed 17 August 2011 23:49:16 Alex Schuster did opine thusly:
> Grant writes:
> > >> > Can I reserve 0% for root on my USB hard drive which
> > >> > is only
> > >> > used for backups and does not contain an OS?
> > >> 
> > >> Yes:
> > >> 
> > >> mke2fs -m 0 /dev/usb-drive
> > > 
> > > Although a value > 0 helps against fragmentation. And when
> > > rdiff-backup has failed because it ran out of space,
> > > regressing to the previous sane state will need a little
> > > free space.> 
> > Good points.  Should 10GB (1% of 1TB) do it?
> 
> This I don't know. I use this value for large partitions of
> multimedia data, because I do not want to waste space (no matter
> how big the drives are, mine are always quite full), and
> performance should not be a big issue here. I keep the 5% default
> other partitions, like /home. BTW, you can also specify fractions
> like 0.5% if you like.

I prefer to keep reminding myself where 5% comes from. Way back when 
ext2 was being developed, 500M drives were big. 5% reserved is 12.5M 
or about 12,500 blocks.

Why that amount? Is it really 5% or is it the number of blocks and 5% 
just happens to round that out nicely?

Median file sizes haven't changed much in 15 years. The number of 
files on an average machine has increased hugely, and the upper size 
limit is orders of magnitude bigger, but that doesn't affect the 
median size much.

I take the view that 5% is excessive these days and usually reserve 
only a tiny amount - something like 10 x the biggest file I expect to 
have to deal with when the disk is full. 

And besides, as Murphy would have it, it's usually a root process that 
fills drives anyway (syslog cough cough) rendering the reserved amount 
useless :-)


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Peter Humphrey]

On Wednesday 17 August 2011 23:03:32 Alan McKinnon wrote:

> Why that amount? Is it really 5% or is it the number of blocks and 5%
> just happens to round that out nicely?

No, it's just somebody licking his finger, sticking it up into the wind to 
see which way it's blowing. Think-of-a-number, in other words.

-- 
Rgds
Peter		Linux Counter 5290, 1994-04-23

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Wednesday, August 17, 2011 11:49:16 PM Alex Schuster wrote:
> Grant writes:
> > >> > Can I reserve 0% for root on my USB hard drive which is only
> > >> > used for backups and does not contain an OS?
> > >> 
> > >> Yes:
> > >> 
> > >> mke2fs -m 0 /dev/usb-drive
> > > 
> > > Although a value > 0 helps against fragmentation. And when
> > > rdiff-backup has failed because it ran out of space, regressing to
> > > the previous sane state will need a little free space.
> > 
> > Good points.  Should 10GB (1% of 1TB) do it?
> 
> This I don't know. I use this value for large partitions of multimedia data,
> because I do not want to waste space (no matter how big the drives are,
> mine are always quite full), and performance should not be a big issue
> here. I keep the 5% default other partitions, like /home. BTW, you can also
> specify fractions like 0.5% if you like.

I tend to leave it at default for most partitions. Only the ones serving the 
fileshare have it set to 0%. But these are on LVM partitions and I can 
increase their size when needed.

> Another thing: Be sure to have enough inodes on the file system, I have run
> out of them in the past. Not only once.

I usually run out of these during installation time when I use ext2/3 
filesystems for the portage tree.
That's why I tend to use reiserfs for that.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> I'm setting up an automated rdiff-backup system and I'm stuck between
>> pushing the backups to the backup server, and pulling the backups to
>> the backup server.  If I push, I have to allow read/write access of my
>> backups via SSH keys.  If I pull, I have to enable root logins on each
>> system to be backed-up, allow root read access of each system via SSH
>> keys, and I have to deal with openvpn or ssh -R so my laptop can back
>> up from behind foreign routers.  The conventional wisdom online seems
>> to indicate pulling is better, but pushing seems like it might be
>> better to me.  Do you push or pull?
>
> I would push, to be honest.

What can be done about the fact that any attacker who can break into a
system and wipe it out can also wipe out its backups?  That negates
one of the reasons for making the backups in the first place.

Should private SSH keys be excluded from the backup?  Should anything
else be excluded?

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Thursday, August 18, 2011 06:51:32 PM Grant wrote:
> >> I'm setting up an automated rdiff-backup system and I'm stuck between
> >> pushing the backups to the backup server, and pulling the backups to
> >> the backup server.  If I push, I have to allow read/write access of my
> >> backups via SSH keys.  If I pull, I have to enable root logins on each
> >> system to be backed-up, allow root read access of each system via SSH
> >> keys, and I have to deal with openvpn or ssh -R so my laptop can back
> >> up from behind foreign routers.  The conventional wisdom online seems
> >> to indicate pulling is better, but pushing seems like it might be
> >> better to me.  Do you push or pull?
> > 
> > I would push, to be honest.
> 
> What can be done about the fact that any attacker who can break into a
> system and wipe it out can also wipe out its backups?  That negates
> one of the reasons for making the backups in the first place.

True, except if, after a backup is finished, you move the actual backup to a 
different location. (Or you backup the backup server)

I store all important files on my server and the backups there can not be 
accessed from the fileserver itself. (That backup is done in "pull" mode every 
night.)

> Should private SSH keys be excluded from the backup?  Should anything
> else be excluded?

When a host is compromised, the corresponding entries in the "authorized_keys" 
should be removed from all other servers/hosts. This will make those private 
keys useless.

If you protect them with a passphrase, the private keys are not usable in any 
case. But this will require the backups to be started manually to allow you to 
enter the passphrase.
Or you unlock the passphrase in memory and use ssh-agent for that.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Grant]

>> >> I'm setting up an automated rdiff-backup system and I'm stuck between
>> >> pushing the backups to the backup server, and pulling the backups to
>> >> the backup server.  If I push, I have to allow read/write access of my
>> >> backups via SSH keys.  If I pull, I have to enable root logins on each
>> >> system to be backed-up, allow root read access of each system via SSH
>> >> keys, and I have to deal with openvpn or ssh -R so my laptop can back
>> >> up from behind foreign routers.  The conventional wisdom online seems
>> >> to indicate pulling is better, but pushing seems like it might be
>> >> better to me.  Do you push or pull?
>> >
>> > I would push, to be honest.
>>
>> What can be done about the fact that any attacker who can break into a
>> system and wipe it out can also wipe out its backups?  That negates
>> one of the reasons for making the backups in the first place.
>
> True, except if, after a backup is finished, you move the actual backup to a
> different location. (Or you backup the backup server)

I do back up the backup server to another system via rsync, but if the
backups on the backup server are wiped out, rsync will wipe them out
on the other system too.

> I store all important files on my server and the backups there can not be
> accessed from the fileserver itself. (That backup is done in "pull" mode every
> night.)

I thought you were in favor of "pushing"?  How do you back up to a
system that can't access the backups?

>> Should private SSH keys be excluded from the backup?  Should anything
>> else be excluded?
>
> When a host is compromised, the corresponding entries in the "authorized_keys"
> should be removed from all other servers/hosts. This will make those private
> keys useless.

So it's OK to back up a private key to another system?  I just want to
make sure I'm not breaking a "good admin" rule by doing this.

- Grant

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Joost Roeleveld]

On Friday, August 19, 2011 10:35:10 AM Grant wrote:
> >> >> I'm setting up an automated rdiff-backup system and I'm stuck
> >> >> between
> >> >> pushing the backups to the backup server, and pulling the
> >> >> backups to
> >> >> the backup server.  If I push, I have to allow read/write access
> >> >> of my backups via SSH keys.  If I pull, I have to enable root
> >> >> logins on each system to be backed-up, allow root read access
> >> >> of each system via SSH keys, and I have to deal with openvpn or
> >> >> ssh -R so my laptop can back up from behind foreign routers.
> >> >>  The conventional wisdom online seems to indicate pulling is
> >> >> better, but pushing seems like it might be better to me.  Do
> >> >> you push or pull?
> >> > 
> >> > I would push, to be honest.
> >> 
> >> What can be done about the fact that any attacker who can break into a
> >> system and wipe it out can also wipe out its backups?  That negates
> >> one of the reasons for making the backups in the first place.
> > 
> > True, except if, after a backup is finished, you move the actual backup
> > to a different location. (Or you backup the backup server)
> 
> I do back up the backup server to another system via rsync, but if the
> backups on the backup server are wiped out, rsync will wipe them out
> on the other system too.

Why not use a different backup tool that doesn't have this possible problem?
Rsync will clear the target is the source is emptied as well. Not sure if this 
can be prevented.

> > I store all important files on my server and the backups there can not
> > be
> > accessed from the fileserver itself. (That backup is done in "pull" mode
> > every night.)
> 
> I thought you were in favor of "pushing"?  How do you back up to a
> system that can't access the backups?

I am, when it comes to backing up desktops. The server is actually a xen-host 
with multiple xen-domains running on it.

I found it easier to have the host determine the backups. The sequence of 
steps is basically as follows:
1) host tells domain to stop service(s)
2) host tells domain to unmount filesystem
3) hosts disconnects filesystem from domain
4) host creates snapshot (LVM)
5) host reconnect filesystem to domain
6) host tells domain to remount filesystem
7) host tells domain to stop service(s)
8) host backs up snapshot
9) host deletes snapshot

I couldn't do a push-system for the virtual machines as I didn't want to 
expose the host.

> >> Should private SSH keys be excluded from the backup?  Should anything
> >> else be excluded?
> > 
> > When a host is compromised, the corresponding entries in the
> > "authorized_keys" should be removed from all other servers/hosts. This
> > will make those private keys useless.
> 
> So it's OK to back up a private key to another system?  I just want to
> make sure I'm not breaking a "good admin" rule by doing this.

Yes, I don't see any problem with that.
If the backup-server is compromised via a compromised other system, the keys 
on that system are compromised already anyway.

I don't have private keys without passphrase apart from the one the host uses 
to send commands to the virtual machines. And the host can't be accessed by 
remote.

--
Joost

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Bill Longman]

On 08/15/2011 09:58 PM, Grant wrote:
> the backup server.  If I push, I have to allow read/write access of my
> backups via SSH keys.  If I pull, I have to enable root logins on each
> system to be backed-up, allow root read access of each system via SSH

+1 push.

But my question is, "Why do you assert that you must allow root access?"
Surely each machine can do its own backups and plop them into a
directory accessible to a "backup" login.

Re: [gentoo-user] {OT} rdiff-backup: push or pull?

[Alan McKinnon]

On Tue 16 August 2011 06:39:39 Bill Longman did opine thusly:
> On 08/15/2011 09:58 PM, Grant wrote:
> > the backup server.  If I push, I have to allow read/write access
> > of my backups via SSH keys.  If I pull, I have to enable root
> > logins on each system to be backed-up, allow root read access
> > of each system via SSH
> +1 push.
> 
> But my question is, "Why do you assert that you must allow root
> access?" Surely each machine can do its own backups and plop them
> into a directory accessible to a "backup" login.

More often than not that results in you needing twice as much disk 
space as what you actually use, few people are willing to sacrifice 
that much.

Consider:

/usr   8G
/home  100G+
everything else - much less space

That's not unusual for people's personal machines. At some point you 
will need 100G free for a backup copy. Less if you pipe tar to gzip, 
but the actual amount is always unknown till you do it. The only 
amount certain to work is 100G if the data is binary with no benefit 
from compression.

Push backups are indeed the better route for the OP with a simple 
setup.


-- 
alan dot mckinnon at gmail dot com