Re: [gentoo-user] Best way to copy /* ?

Re: [gentoo-user] Best way to copy /* ?

[Etaoin Shrdlu]

On Mon, 27 Dec 2010 16:20:55 +0100
Marc Blumentritt <marc.blumentritt@arcor.de> wrote:

> Hi,
> 
> I have bought myself a Christmas present, a new shiny hard disk. Now I
> want to copy my old Gentoo system to my new disk like this:
> 
> 1.) boot with gentoo boot cd
> 2.) mount my old system ind /old ( / in one partition, /home, /usr,
> /var, /tmp and /opt in lvm2 volumes and /boot on it's own partition)
> 3.) mount my new disk ind /new (just 2 partitions, 1 for / and 1
> for /boot) 4.) copy from /old to /new
> 5.) modify fstab and prepare grub
> 6.) reboot
> 
> Concerning step 4: what is the best copy command?
> I tried with
> 
> cp -a /old/* /new
> 
> but got some problems in /home. My user dir got the wrong permissions (I
> d'ont know, if this is in some way connected with /home being a mount
> point). Of course this could be the same in other dirs.
> 
> Is there a better method? I read years ago on this list about using tar
> with this (piping the tar output into a second tar command, which
> extracts the files to their final destination).
> 
> Are there other tools? Or did I use cp in a wrong way?

I'd just use rsync.

[gentoo-user] Best way to copy /* ?

[Marc Blumentritt]

Hi,

I have bought myself a Christmas present, a new shiny hard disk. Now I
want to copy my old Gentoo system to my new disk like this:

1.) boot with gentoo boot cd
2.) mount my old system ind /old ( / in one partition, /home, /usr,
/var, /tmp and /opt in lvm2 volumes and /boot on it's own partition)
3.) mount my new disk ind /new (just 2 partitions, 1 for / and 1 for /boot)
4.) copy from /old to /new
5.) modify fstab and prepare grub
6.) reboot

Concerning step 4: what is the best copy command?
I tried with

cp -a /old/* /new

but got some problems in /home. My user dir got the wrong permissions (I
d'ont know, if this is in some way connected with /home being a mount
point). Of course this could be the same in other dirs.

Is there a better method? I read years ago on this list about using tar
with this (piping the tar output into a second tar command, which
extracts the files to their final destination).

Are there other tools? Or did I use cp in a wrong way?

Regards
Marc

Re: [gentoo-user] Best way to copy /* ?

[ich bins]

Am 27.12.2010 16:20, schrieb Marc Blumentritt:
> Hi,
> 
> I have bought myself a Christmas present, a new shiny hard disk. Now I
> want to copy my old Gentoo system to my new disk like this:
> 
> 1.) boot with gentoo boot cd
> 2.) mount my old system ind /old ( / in one partition, /home, /usr,
> /var, /tmp and /opt in lvm2 volumes and /boot on it's own partition)
> 3.) mount my new disk ind /new (just 2 partitions, 1 for / and 1 for /boot)
> 4.) copy from /old to /new
> 5.) modify fstab and prepare grub
> 6.) reboot
> 
> Concerning step 4: what is the best copy command?
> I tried with
> 
> cp -a /old/* /new
> 
> but got some problems in /home. My user dir got the wrong permissions (I
> d'ont know, if this is in some way connected with /home being a mount
> point). Of course this could be the same in other dirs.
> 
> Is there a better method? I read years ago on this list about using tar
> with this (piping the tar output into a second tar command, which
> extracts the files to their final destination).
> 
> Are there other tools? Or did I use cp in a wrong way?
> 
> Regards
> Marc
> 
> 
> 


http://en.gentoo-wiki.com/wiki/Custom_Stage4

[gentoo-user] Re: Best way to copy /* ?

[Lubos Kolouch]

Marc Blumentritt, Mon, 27 Dec 2010 16:20:55 +0100:

> Hi,
> 
> I have bought myself a Christmas present, a new shiny hard disk. Now I
> want to copy my old Gentoo system to my new disk like this:
> 
> 1.) boot with gentoo boot cd
> 2.) mount my old system ind /old ( / in one partition, /home, /usr,
> /var, /tmp and /opt in lvm2 volumes and /boot on it's own partition) 3.)
> mount my new disk ind /new (just 2 partitions, 1 for / and 1 for /boot)
> 4.) copy from /old to /new
> 5.) modify fstab and prepare grub
> 6.) reboot
> 
> Concerning step 4: what is the best copy command? I tried with
> 
> cp -a /old/* /new
> 
> but got some problems in /home. My user dir got the wrong permissions (I
> d'ont know, if this is in some way connected with /home being a mount
> point). Of course this could be the same in other dirs.
> 
> Is there a better method? I read years ago on this list about using tar
> with this (piping the tar output into a second tar command, which
> extracts the files to their final destination).
> 
> Are there other tools? Or did I use cp in a wrong way?
> 
> Regards
> Marc

Try
rsync -auD --verbose --progress --exclude="/proc" --exclude="/sys" --
exclude="/dev" /old/ /new/ 

and then copy over /dev/console, /dev/tty and/or any other /devices

Lubos

Re: [gentoo-user] Best way to copy /* ?

[Jarry]

On 27. 12. 2010 16:20, Marc Blumentritt wrote:
> want to copy my old Gentoo system to my new disk like this:
>...
> cp -a /old/* /new

Personally, I would use dump/restore (if you use ext3),
and you do not need to worry about permissions, links, etc.

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] Best way to copy /* ?

[Dale]

Marc Blumentritt wrote:
> Hi,
>
> I have bought myself a Christmas present, a new shiny hard disk. Now I
> want to copy my old Gentoo system to my new disk like this:
>
> 1.) boot with gentoo boot cd
> 2.) mount my old system ind /old ( / in one partition, /home, /usr,
> /var, /tmp and /opt in lvm2 volumes and /boot on it's own partition)
> 3.) mount my new disk ind /new (just 2 partitions, 1 for / and 1 for /boot)
> 4.) copy from /old to /new
> 5.) modify fstab and prepare grub
> 6.) reboot
>
> Concerning step 4: what is the best copy command?
> I tried with
>
> cp -a /old/* /new
>
> but got some problems in /home. My user dir got the wrong permissions (I
> d'ont know, if this is in some way connected with /home being a mount
> point). Of course this could be the same in other dirs.
>
> Is there a better method? I read years ago on this list about using tar
> with this (piping the tar output into a second tar command, which
> extracts the files to their final destination).
>
> Are there other tools? Or did I use cp in a wrong way?
>
> Regards
> Marc
>
>    

I always used cp -av myself and it always worked.  I never had 
permission problems either.

Some people do use tar especially if it is over a network or something 
like that.  I don't have the command tho since I never used it.

Dale

:-)  :-)

Re: [gentoo-user] Best way to copy /* ?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 175 bytes --]

On Mon, 27 Dec 2010 15:20:30 +0000, Etaoin Shrdlu wrote:

> I'd just use rsync.

+1


-- 
Neil Bothwick

Better to understand a little than to misunderstand a lot.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Mark Knecht]

On Mon, Dec 27, 2010 at 8:37 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Mon, 27 Dec 2010 15:20:30 +0000, Etaoin Shrdlu wrote:
>
>> I'd just use rsync.
>
> +1
>

What flags?

- Mark

[gentoo-user] Re: Best way to copy /* ?

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 67 bytes --]

Mark Knecht wrote:
> What flags?

At least -avHAX

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Volker Armin Hemmann]

if you don't use ACL (and if you don't know what I am talking about, you don't 
use them):
cp -auv

Re: [gentoo-user] Best way to copy /* ?

[Mark Knecht]

On Mon, Dec 27, 2010 at 9:27 AM, Volker Armin Hemmann
<volkerarmin@googlemail.com> wrote:
> if you don't use ACL (and if you don't know what I am talking about, you don't
> use them):
> cp -auv

Maybe

cp -auvx

?

Just asking,
Mark

Re: [gentoo-user] Best way to copy /* ?

[Dale]

Volker Armin Hemmann wrote:
> if you don't use ACL (and if you don't know what I am talking about, you don't
> use them):
> cp -auv
>
>    

The -u part can save time but I did run into trouble with that once.  I 
never did figure out why but rm -rfv everything and then doing a cp -av 
fixed it.  Maybe it was a fluke but who knows.  Also, if he is copying 
to a freshly partitioned drive, there shouldn't be anything there to 
update anyway so it will still copy everything.

Your mileage may vary tho.

Dale

:-)  :-)

Re: [gentoo-user] Best way to copy /* ?

[Volker Armin Hemmann]

On Monday 27 December 2010 09:38:42 Mark Knecht wrote:
> On Mon, Dec 27, 2010 at 9:27 AM, Volker Armin Hemmann
> 
> <volkerarmin@googlemail.com> wrote:
> > if you don't use ACL (and if you don't know what I am talking about, you
> > don't use them):
> > cp -auv
> 
> Maybe
> 
> cp -auvx
> 
> ?
> 
> Just asking,
> Mark

if you are copying a life-sytem -x is needed. Indeed.
But if you boot from a cd and mount source and destination on different mount 
points x is not needed.

Re: [gentoo-user] Best way to copy /* ?

[Mark Knecht]

On Mon, Dec 27, 2010 at 9:51 AM, Volker Armin Hemmann
<volkerarmin@googlemail.com> wrote:
> On Monday 27 December 2010 09:38:42 Mark Knecht wrote:
>> On Mon, Dec 27, 2010 at 9:27 AM, Volker Armin Hemmann
>>
>> <volkerarmin@googlemail.com> wrote:
>> > if you don't use ACL (and if you don't know what I am talking about, you
>> > don't use them):
>> > cp -auv
>>
>> Maybe
>>
>> cp -auvx
>>
>> ?
>>
>> Just asking,
>> Mark
>
> if you are copying a life-sytem -x is needed. Indeed.
> But if you boot from a cd and mount source and destination on different mount
> points x is not needed.
>
>
Ah, good point.

I wasn't thinking about 'live' so much as just a complicated system
with multiple hard drives/partitions. For instance, copying /home
where here are links to other partitions for /home/users1,
/home/user2, etc. I wouldn't want to copy those other users along with
/home/user3 that resides on the /home partition physically.

Thanks for the clarification,
Mark

Re: [gentoo-user] Best way to copy /* ?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Mon, 27 Dec 2010 11:45:01 -0600, Dale wrote:

> > cp -auv

> The -u part can save time but I did run into trouble with that once.  I 
> never did figure out why but rm -rfv everything and then doing a cp -av 
> fixed it.  Maybe it was a fluke but who knows.  Also, if he is copying 
> to a freshly partitioned drive, there shouldn't be anything there to 
> update anyway so it will still copy everything.

You've just explained why rsync is a better choice.


-- 
Neil Bothwick

... "I just forgot to increment the counter," Tom said, nonplussed.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Best way to copy /* ?

[Marc Blumentritt]

Am 27.12.2010 18:05, schrieb Remy Blank:
> Mark Knecht wrote:
>> What flags?
> 
> At least -avHAX

I just run this as root (as a test while running my old system):

rsync -avHAX --progress /home/ /new/

After a few seconds I canceled this with ctrl-c and got this result:

hive new # ls -lah /home/
total 44K
drwxr-xr-x  6 root root  4.0K May  2  2009 .
drwxr-xr-x 20 root root  4.0K Dec 27 17:59 ..
-rw-r--r--  1 root root     0 Apr 20  2007 .keep
drwx------  2 root root   16K Aug 18  2007 lost+found
drwxr-xr-x 81 marc users 4.0K Dec 27 21:07 marc
drwx------  6 root root  4.0K Dec 11 20:33 root_dev
drwxr-xr-x  4 marc root  4.0K Sep  7  2007 virtualbox
hive new # ls -lah /new/
total 24K
drwxr-xr-x  6 root root  4.0K May  2  2009 .
drwxr-xr-x 20 root root  4.0K Dec 27 17:59 ..
-rw-r--r--  1 root root     0 Apr 20  2007 .keep
drwx------  2 root root  4.0K Aug 18  2007 lost+found
drwxr-xr-x 81 marc users 4.0K Dec 27 21:13 marc
drwx------  2 root root  4.0K Dec 27 21:13 root_dev
drwx------  2 root root  4.0K Dec 27 21:13 virtualbox


Have a look at ownership an permissions of virtualbox!
Another problem:


hive new # ls -lah /new/marc/
total 584K
drwxr-xr-x 81 marc users 4.0K Dec 27 21:13 .
drwxr-xr-x  6 root root  4.0K May  2  2009 ..
[...]
drwx------  2 root root  4.0K Dec 27 21:13 Bilder
[...]
drwx------  2 root root  4.0K Dec 27 21:13 Dateien
[...]
drwx------  2 root root  4.0K Dec 27 21:13 Desktop
[...]


Originally all this dirs are owned by marc:users or marc:marc.

I have no clue at all what is happening here!

My home partition is ext3 mounted with these options (from fstab):
noatime,user_xattr

My new partition is ext4 mounted with no special options.

Any ideas?

Marc

[gentoo-user] Re: Best way to copy /* ?

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

Marc Blumentritt wrote:
> Have a look at ownership an permissions of virtualbox!

rsync sets the mtime and permissions of directories *after* recursing
into them. So if you interrupted the process while it was copying the
contents of virtualbox, the permissions of that directory haven't been
set yet.

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Dale]

Neil Bothwick wrote:
> On Mon, 27 Dec 2010 11:45:01 -0600, Dale wrote:
>
>    
>>> cp -auv
>>>        
>    
>> The -u part can save time but I did run into trouble with that once.  I
>> never did figure out why but rm -rfv everything and then doing a cp -av
>> fixed it.  Maybe it was a fluke but who knows.  Also, if he is copying
>> to a freshly partitioned drive, there shouldn't be anything there to
>> update anyway so it will still copy everything.
>>      
> You've just explained why rsync is a better choice.
>
>    

It may be better but my point was that I once had problems with the -u 
option.   It was a freshly formatted drive so no idea why the -u would 
have even mattered since it has to copy everything over anyway.

Dale

:-)  :-)

Re: [gentoo-user] Best way to copy /* ?

[Mike Edenfield]

On 12/27/2010 10:20 AM, Marc Blumentritt wrote:
> Hi,
>
> I have bought myself a Christmas present, a new shiny hard disk. Now I
> want to copy my old Gentoo system to my new disk like this:
>
> 1.) boot with gentoo boot cd
> 2.) mount my old system ind /old ( / in one partition, /home, /usr,
> /var, /tmp and /opt in lvm2 volumes and /boot on it's own partition)
> 3.) mount my new disk ind /new (just 2 partitions, 1 for / and 1 for /boot)
> 4.) copy from /old to /new
> 5.) modify fstab and prepare grub
> 6.) reboot
>
> Concerning step 4: what is the best copy command?
> I tried with
>
> cp -a /old/* /new

This should have gotten the permissions right; -a implies 
--preserve=all.  Not sure what happened there.

The tar method you're looking for is:

tar -C /old cpf - | tar -C /new xvpf -

You'll probably not want to do the entire / in a single go, 
since /proc, /sys, and /dev (at least) should be skipped. 
Copy /old/sbin -> /new/sbin, etc. for all of the root 
folders that aren't their own partitions.  The rest you can 
do the entire mount point at once, though I'm not sure you 
really need to copy /tmp either.

You can also use rsync, dump/restore, and probably a dozen 
other tools to make this work.  Google for "backup entire 
hard disk" and start reading :)

--Mike

[gentoo-user] Re: Best way to copy /* ?

[Marc Blumentritt]

Am 27.12.2010 22:23, schrieb Remy Blank:
> Marc Blumentritt wrote:
>> Have a look at ownership an permissions of virtualbox!
> 
> rsync sets the mtime and permissions of directories *after* recursing
> into them. So if you interrupted the process while it was copying the
> contents of virtualbox, the permissions of that directory haven't been
> set yet.

Ah, OK. Didn't know this. Do you know, why it does work like this?

And does cp -a work in the same way?

Regards,
Marc

[gentoo-user] Re: Best way to copy /* ?

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

Marc Blumentritt wrote:
> Ah, OK. Didn't know this. Do you know, why it does work like this?

It initially sets the permissions to 600 presumably to limit access to
an unfinished copy from other users.

The mtime must be set after copying the contents, because adding a file
or directory changes the mtime of the parent.

> And does cp -a work in the same way?

Probably yes, for the same reasons, but I haven't checked.

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Best way to copy /* ?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

On Tue, 28 Dec 2010 10:18:46 +0100, Remy Blank wrote:

> > Ah, OK. Didn't know this. Do you know, why it does work like this?  
> 
> It initially sets the permissions to 600 presumably to limit access to
> an unfinished copy from other users.
> 
> The mtime must be set after copying the contents, because adding a file
> or directory changes the mtime of the parent.
> 
> > And does cp -a work in the same way?  
> 
> Probably yes, for the same reasons, but I haven't checked.

No it doesn't. cp leaves the mtime of the directory at the time t copied
the files into it. If you want to preserve directory timestamps, use
rsync.


-- 
Neil Bothwick

The number one cause of computer problems is computer solutions.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Alex Schuster]

Mike Edenfield writes:

> The tar method you're looking for is:
> 
> tar -C /old cpf - | tar -C /new xvpf -
> 
> You'll probably not want to do the entire / in a single go,
> since /proc, /sys, and /dev (at least) should be skipped.
> Copy /old/sbin -> /new/sbin, etc. for all of the root
> folders that aren't their own partitions.  The rest you can
> do the entire mount point at once, though I'm not sure you
> really need to copy /tmp either.

Or bind mount root to somewhere else:
mount -o bind / /new
/new then contains all of / , but without the filesystems mounted at root. 
It also has has the original /dev  with its necessary entries console and 
null, but without the stuff udev added.

It also takes care of /tmp. Its contents are unimportant, but the 
permissions of the directory itself matter, the sticky bit needs to be set.

	Wonko

Re: [gentoo-user] Best way to copy /* ?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

On Tue, 28 Dec 2010 00:02:31 -0500, Mike Edenfield wrote:

> tar -C /old cpf - | tar -C /new xvpf -
> 
> You'll probably not want to do the entire / in a single go, 
> since /proc, /sys, and /dev (at least) should be skipped. 

Which can be done with the -l option.


-- 
Neil Bothwick

You can't teach a new mouse old clicks.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Joerg Schilling]

Neil Bothwick <neil@digimed.co.uk> wrote:

> On Tue, 28 Dec 2010 00:02:31 -0500, Mike Edenfield wrote:
>
> > tar -C /old cpf - | tar -C /new xvpf -
> > 
> > You'll probably not want to do the entire / in a single go, 
> > since /proc, /sys, and /dev (at least) should be skipped. 
>
> Which can be done with the -l option.

You are mistaken: The -l option causes tar to warn if not all hars links to a 
file could not be resolved.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] Best way to copy /* ?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 924 bytes --]

On Tuesday 28 December 2010 16:26:16 Joerg Schilling wrote:
> Neil Bothwick <neil@digimed.co.uk> wrote:
> > On Tue, 28 Dec 2010 00:02:31 -0500, Mike Edenfield wrote:
> > > tar -C /old cpf - | tar -C /new xvpf -
> > > 
> > > You'll probably not want to do the entire / in a single go,
> > > since /proc, /sys, and /dev (at least) should be skipped.
> > 
> > Which can be done with the -l option.
> 
> You are mistaken: The -l option causes tar to warn if not all hars links to
> a file could not be resolved.

I often use this with a LiveCD:

cd /home
tar --exclude File1 --exclude Dir1 -lcpvSf - . | (cd /mnt/new_partition; tar -
xpvf - )

or something like this with star:

star -copy -p -xdot -xattr -H=exustar -sparse -M -C /home . /mnt/new_partition

(You can use -V -pat=File1 to exclude files or directories with star, use the 
-M option to avoid following mount points).
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Tue, 28 Dec 2010 17:26:16 +0100, Joerg Schilling wrote:

> > Which can be done with the -l option.  
> 
> You are mistaken: The -l option causes tar to warn if not all hars
> links to a file could not be resolved.

You are right. I was referring to the --one-file-system, which I thought
had -l as the short option, but apparently not. Or has that changed in
the last couple of years?


-- 
Neil Bothwick

I'm Bugs Bunny of Borg.  What's up Collective?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Alex Schuster]

Joerg Schilling writes:

> Neil Bothwick <neil@digimed.co.uk> wrote:
> > On Tue, 28 Dec 2010 00:02:31 -0500, Mike Edenfield wrote:
> > > tar -C /old cpf - | tar -C /new xvpf -
> > > 
> > > You'll probably not want to do the entire / in a single go,
> > > since /proc, /sys, and /dev (at least) should be skipped.
> > 
> > Which can be done with the -l option.
> 
> You are mistaken: The -l option causes tar to warn if not all hars links
> to a file could not be resolved.

We're one the Gentoo Linux list here and so we're talking about GNU tar, 
where -l is a synonym for --one-file-system, Neil is right. Original Unix 
tar works like you say, but we don't use it here.

	Wonko

Re: [gentoo-user] Best way to copy /* ?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 608 bytes --]

On Tuesday 28 December 2010 22:33:53 Neil Bothwick wrote:
> On Tue, 28 Dec 2010 17:26:16 +0100, Joerg Schilling wrote:
> > > Which can be done with the -l option.
> > 
> > You are mistaken: The -l option causes tar to warn if not all hars
> > links to a file could not be resolved.
> 
> You are right. I was referring to the --one-file-system, which I thought
> had -l as the short option, but apparently not. Or has that changed in
> the last couple of years?

Yes, there used to be a warning on the man page saying that --one-file-system 
should be be used instead of -l.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Joerg Schilling]

Neil Bothwick <neil@digimed.co.uk> wrote:

> On Tue, 28 Dec 2010 17:26:16 +0100, Joerg Schilling wrote:
>
> > > Which can be done with the -l option.  
> > 
> > You are mistaken: The -l option causes tar to warn if not all hars
> > links to a file could not be resolved.
>
> You are right. I was referring to the --one-file-system, which I thought
> had -l as the short option, but apparently not. Or has that changed in
> the last couple of years?

Tar did not change it's behavior with respect to -l during the past 31 years.

On Linux, there is frequently gtar installed as tar and gtar is not respecting 
standards. Gtar in previous times was e.g. in conflict with the standard 
regarding to -l. Aprox. 10 years ago, I files a bug report against gtar for 
this standard deviation and it seems that they fixed this bug.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] Best way to copy /* ?

[Alex Schuster]

Joerg Schilling writes:

> On Linux, there is frequently gtar installed as tar and gtar is not
> respecting standards. Gtar in previous times was e.g. in conflict with
> the standard regarding to -l. Aprox. 10 years ago, I files a bug report
> against gtar for this standard deviation and it seems that they fixed
> this bug.

Oh, I stand corrected, you're right. I looked at the (german) man page, 
which has not been updated yet. I sent a mail to bug-tar@gnu.org about this.

	Wonko

[gentoo-user] Re: Best way to copy /* ? [SOLVED]

[Marc Blumentritt]

Am 27.12.2010 16:20, schrieb Marc Blumentritt:
> Hi,
> 
> I have bought myself a Christmas present, a new shiny hard disk. Now I
> want to copy my old Gentoo system to my new disk like this:
> 
> 1.) boot with gentoo boot cd
> 2.) mount my old system ind /old ( / in one partition, /home, /usr,
> /var, /tmp and /opt in lvm2 volumes and /boot on it's own partition)
> 3.) mount my new disk ind /new (just 2 partitions, 1 for / and 1 for /boot)
> 4.) copy from /old to /new
> 5.) modify fstab and prepare grub
> 6.) reboot
> 
> Concerning step 4: what is the best copy command?
> I tried with
> 
> cp -a /old/* /new
> 
> but got some problems in /home. My user dir got the wrong permissions (I
> d'ont know, if this is in some way connected with /home being a mount
> point). Of course this could be the same in other dirs.
> 
> Is there a better method? I read years ago on this list about using tar
> with this (piping the tar output into a second tar command, which
> extracts the files to their final destination).
> 
> Are there other tools? Or did I use cp in a wrong way?

I write this while booted from my new disk. I used rsync after booting
from a live CD. Everything worked perfect.

Thanks to everyone.

Regards,
Marc

Re: [gentoo-user] Best way to copy /* ?

[Joerg Schilling]

Mick <michaelkintzios@gmail.com> wrote:

> or something like this with star:
>
> star -copy -p -xdot -xattr -H=exustar -sparse -M -C /home . /mnt/new_partition
>
> (You can use -V -pat=File1 to exclude files or directories with star, use the 
> -M option to avoid following mount points).

star -copy by default uses the star "-dump" format which is the "exustar" 
format + extended dump metadata. There is no need to specify the archive format 
with star -copy.

Also note that star by default uses a safe extract method that calls fsync(2) 
at the end of each single file extract. This is the only way for star for being 
able to detect all possible extract problems. On Linux, the file system buffer 
cache is implemented in a very inefficient way and with some COW filesystems 
(like ZFS), a fsync(2) is an expensive instruction. In such cases, you may call 
star with the -no-fsync option and switch star to the same level of "safeness" 
as other software to speed up the extract or copy operation.

So if you are on Linux and use star -no-fsync, you will not be less secure than 
other software but get aprox. 20% better performance than with other copy 
methods.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] Best way to copy /* ?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1485 bytes --]

On Wednesday 29 December 2010 15:38:22 Joerg Schilling wrote:
> Mick <michaelkintzios@gmail.com> wrote:
> > or something like this with star:
> > 
> > star -copy -p -xdot -xattr -H=exustar -sparse -M -C /home .
> > /mnt/new_partition
> > 
> > (You can use -V -pat=File1 to exclude files or directories with star, use
> > the -M option to avoid following mount points).
> 
> star -copy by default uses the star "-dump" format which is the "exustar"
> format + extended dump metadata. There is no need to specify the archive
> format with star -copy.
> 
> Also note that star by default uses a safe extract method that calls
> fsync(2) at the end of each single file extract. This is the only way for
> star for being able to detect all possible extract problems. On Linux, the
> file system buffer cache is implemented in a very inefficient way and with
> some COW filesystems (like ZFS), a fsync(2) is an expensive instruction.
> In such cases, you may call star with the -no-fsync option and switch star
> to the same level of "safeness" as other software to speed up the extract
> or copy operation.
> 
> So if you are on Linux and use star -no-fsync, you will not be less secure
> than other software but get aprox. 20% better performance than with other
> copy methods.

Thanks for this Jörg,

I had noticed a small overhead compared to tar and guessed that this may be 
because start undertakes a more thorough check of data.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Peter Humphrey]

On Monday 27 December 2010 15:47:19 Dale wrote:

> Some people do use tar especially if it is over a network or
> something like that.  I don't have the command tho since I never
> used it.

Just for completeness:

(cd [source] && tar cpf - . | (cd [dest] && tar xpf - ) )

(I think).

Would someone here please tell me what the rhs should be if the 
destination is at the other end of a network link?

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] Best way to copy /* ?

[Maciej Grela]

2010/12/29 Peter Humphrey <peter@humphrey.ukfsn.org>:
> On Monday 27 December 2010 15:47:19 Dale wrote:
>
>> Some people do use tar especially if it is over a network or
>> something like that.  I don't have the command tho since I never
>> used it.
>
> Just for completeness:
>
> (cd [source] && tar cpf - . | (cd [dest] && tar xpf - ) )
>
> (I think).
>
> Would someone here please tell me what the rhs should be if the
> destination is at the other end of a network link?

On the destination

root@richese:~# nc -l 5555 | tar -xjv
tar: Rozmiar rekordu = 8 bloków
Pobrane/
Pobrane/Vol 28_16.pdf
Pobrane/EPWA-APP2.flac
Pobrane/EPWA-APP1.flac
root@richese:~#

On the source:

thermal@richese:~$ tar -cjv Pobrane/ | nc localhost 5555
Pobrane/
Pobrane/Vol 28_16.pdf
Pobrane/EPWA-APP2.flac
Pobrane/EPWA-APP1.flac
thermal@richese:~$

Voila !

root@richese:~# ls -l Pobrane/
razem 5580
-rw-r--r-- 1 thermal thermal 1623054 2010-12-26 23:57 EPWA-APP1.flac
-rw-r--r-- 1 thermal thermal 3657627 2010-12-26 23:57 EPWA-APP2.flac
-rw-r--r-- 1 thermal thermal  429580 2010-08-18 21:38 Vol 28_16.pdf
root@richese:~#

Br,
Maciej Grela

Re: [gentoo-user] Best way to copy /* ?

[Paul Hartman]

On Wed, Dec 29, 2010 at 11:14 AM, Maciej Grela <maciej.grela@gmail.com> wrote:
> 2010/12/29 Peter Humphrey <peter@humphrey.ukfsn.org>:
>> On Monday 27 December 2010 15:47:19 Dale wrote:
>>
>>> Some people do use tar especially if it is over a network or
>>> something like that.  I don't have the command tho since I never
>>> used it.
>>
>> Just for completeness:
>>
>> (cd [source] && tar cpf - . | (cd [dest] && tar xpf - ) )
>>
>> (I think).
>>
>> Would someone here please tell me what the rhs should be if the
>> destination is at the other end of a network link?
>
> On the destination
>
> root@richese:~# nc -l 5555 | tar -xjv
> tar: Rozmiar rekordu = 8 bloków
> Pobrane/
> Pobrane/Vol 28_16.pdf
> Pobrane/EPWA-APP2.flac
> Pobrane/EPWA-APP1.flac
> root@richese:~#
>
> On the source:
>
> thermal@richese:~$ tar -cjv Pobrane/ | nc localhost 5555
> Pobrane/
> Pobrane/Vol 28_16.pdf
> Pobrane/EPWA-APP2.flac
> Pobrane/EPWA-APP1.flac
> thermal@richese:~$
>
> Voila !

I copied several TB over network this way (netcat + tar), when I had
no possibility of physically copying data to disks (or removing disks
to copy to target machine). It worked perfectly and with a gigabit
network link between the two machines it was not so long of a job. :)

Re: [gentoo-user] Best way to copy /* ?

[Alex Schuster]

Peter Humphrey writes:

> On Monday 27 December 2010 15:47:19 Dale wrote:
> > Some people do use tar especially if it is over a network or
> > something like that.  I don't have the command tho since I never
> > used it.
> 
> Just for completeness:
> 
> (cd [source] && tar cpf - . | (cd [dest] && tar xpf - ) )
> 
> (I think).
> 
> Would someone here please tell me what the rhs should be if the
> destination is at the other end of a network link?

What Maciej said. Or, for greater security when the destination is outside 
the LAN:

cd [source] & tar xpf - . | ssh [user]@[host] 'cd [dest] && tar xpf -'

There's also the -C option for tar to change the directory, but it's not 
really needed.

	Wonko

Re: [gentoo-user] Best way to copy /* ?

[Peter Humphrey]

On Wednesday 29 December 2010 17:50:08 Alex Schuster wrote:

> What Maciej said. Or, for greater security when the destination is
> outside the LAN:
> 
> cd [source] & tar xpf - . | ssh [user]@[host] 'cd [dest] && tar xpf
> -'

That's what I was looking for - a single command I can run on the source 
machine. Thanks Alex.

Just one more thing - what if I only want to store the tar of the source 
directory as an archive on the remote machine? In that case I'd want to 
stream the incoming data into a file instead of untarring it.

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] Best way to copy /* ?

[Alex Schuster]

Peter Humphrey writes:

> On Wednesday 29 December 2010 17:50:08 Alex Schuster wrote:
> > What Maciej said. Or, for greater security when the destination is
> > outside the LAN:
> > 
> > cd [source] & tar xpf - . | ssh [user]@[host] 'cd [dest] && tar xpf
> > -'
> 
> That's what I was looking for - a single command I can run on the source
> machine. Thanks Alex.
> 
> Just one more thing - what if I only want to store the tar of the source
> directory as an archive on the remote machine? In that case I'd want to
> stream the incoming data into a file instead of untarring it.

Replace the tar by cat, and redirect into a file:

cd [source] & tar xpf - . | ssh [user]@[host] 'cat > [dest]/[name].tar'

	Wonko

Re: [gentoo-user] Best way to copy /* ?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1495 bytes --]

On Wednesday 29 December 2010 18:41:00 Alex Schuster wrote:
> Peter Humphrey writes:
> > On Wednesday 29 December 2010 17:50:08 Alex Schuster wrote:
> > > What Maciej said. Or, for greater security when the destination is
> > > outside the LAN:
> > > 
> > > cd [source] & tar xpf - . | ssh [user]@[host] 'cd [dest] && tar xpf
> > > -'
> > 
> > That's what I was looking for - a single command I can run on the source
> > machine. Thanks Alex.
> > 
> > Just one more thing - what if I only want to store the tar of the source
> > directory as an archive on the remote machine? In that case I'd want to
> > stream the incoming data into a file instead of untarring it.
> 
> Replace the tar by cat, and redirect into a file:
> 
> cd [source] & tar xpf - . | ssh [user]@[host] 'cat > [dest]/[name].tar'

The front part should be tar -cpf not -xpf?

Also, option -S manages sparse files more efficiently.

Finally, if it is a large archive and is going to travel over a slow network 
it would make sense to compress it first locally into a tar file (e.g. using -
j) then verify it (if it is important data that you rely on just add -W) and 
finally ssh the compressed tar file over.

If you are going to use pipes, then dd will also work instead of cat; i.e.

 cd [source] & tar cpvSf - . | ssh [user]@[host] "dd 
of=/backup_storage/mydata.tar.bz2"

I don't think that dd is any different to cat in performance terms (but 
haven't tested it).
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Best way to copy /* ?

[Stefan G. Weichinger]

Am 29.12.2010 20:16, schrieb Mick:

>>>> cd [source] & tar xpf - . | ssh [user]@[host] 'cd [dest] && tar xpf
>>>> -'
>>>
>>> That's what I was looking for - a single command I can run on the source
>>> machine. Thanks Alex.
>>>
>>> Just one more thing - what if I only want to store the tar of the source
>>> directory as an archive on the remote machine? 

I can't resist:

my bookmark for these issues is old, but working for me:

http://ultra.ap.krakow.pl/~bar/DOC/ssh_backup.html