[gentoo-user] Merging separate /usr back into / - one last time...

[gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

Hi all,

This was discussed within a couple of threads in the last few months, 
but I wanted to ask for final clarification before I go ahead with this 
(yeah, I know, 'paranoia will destroy ya')...

I'm not afraid of an initramfs any more, but I've decided that I still 
just really don't want one, so I'm looking for the safest & simplest way 
to merge my separate /usr back into /.

The only thing I'm really unsure of is the optimal method and exact 
syntax (ie, trailing slashes? arguments?) for the actual copy of /usr.

Is rsync -a enough for my relatively simple system setup, or would using 
any or all of the other options suggested in those threads be 
safer/better? Specifically:

-a, or -axAHX, or -apogXx, or -PvasHAX

or should I go with a combined -apogsvxAHPX ?

So, here's the plan, please check me...

1. Boot off of the latest gentoo LiveDVD

2. Mount / and create new /usr directory

mount /dev/sda3 /mnt/gentoo/
mkdir /mnt/gentoo/usr

3. Mount old /usr to be moved/merged

vgscan
vgchange -a y
mount /dev/vg/usr /mnt/gentoo/oldusr

4. Copy /oldusr to /usr

rsync -a? /mnt/gentoo/oldusr/ /mnt/gentoo/usr/

Are the trailing slashes required/important/necessary?

Which arguments should I use?

5. Edit /etc/fstab and comment/remove the /usr line

nano -wc /mnt/gentoo/etc/fstab

#/dev/vg/usr       /usr        reiserfs        noatime         0 0

6. Unmount mounted filesystems

umount /mnt/gentoo/oldusr
umount /mnt/gentoo

7. Reboot into new system

Done?

I'm pretty sure that:

1. There is no need to chroot into the real system during this process,

and

2. I only need to mount / and the old /usr, no need to mount anything
    else (/proc, /sys, /var, /home, activating swap, etc)

Correct?

Thanks

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[gottlieb]

On Mon, Dec 02 2013, tanstaafl@libertytrek.org wrote:

> Is rsync -a enough for my relatively simple system setup, or would
> using any or all of the other options suggested in those threads be
> safer/better? Specifically:
>
> -a, or -axAHX, or -apogXx, or -PvasHAX

I am not an expert but here goes.
-x would not hurt but should not be needed since i believe that your
current /dev/vg/usr is just one partition.

I didn't need -X -A because I don't have acls or extended attributes

> or should I go with a combined -apogsvxAHPX ?
>
> So, here's the plan, please check me...
>
> 1. Boot off of the latest gentoo LiveDVD
>
> 2. Mount / and create new /usr directory

I am missing something.  I would have thought your old / (dev/sda3)
already has an empty /usr directory where you previously mounted
/dev/vg/usr

> mount /dev/sda3 /mnt/gentoo/
> mkdir /mnt/gentoo/usr
>
> 3. Mount old /usr to be moved/merged
>
> vgscan
> vgchange -a y
> mount /dev/vg/usr /mnt/gentoo/oldusr
>
> 4. Copy /oldusr to /usr

This suggests that your current root (dev/sda3) is big enough to
include the previous /usr (dev/vg/usr).  That is indeed a simple case.
Many of us had to move partitions around to get a big enough partition
for / + /usr.

> rsync -a? /mnt/gentoo/oldusr/ /mnt/gentoo/usr/
>
> Are the trailing slashes required/important/necessary?

The first trailing slash (oldusr/) is important.  Without it, you would
be creating the directory /mnt/gentoo/usr/oldusr.  With it (as you
wrote) just the contents of /oldusr are copied not the directory itself.
So yes you want that slash.

I don't believe the 2nd trailing / (usr/) is needed, but doesn't hurt.
The rsync man page shows both uses and I don't see any words saying
anything about the difference.  I must say I never noticed the two
different uses in the man page can't remember what I used.  But again, I
believe the results are the same.

> Which arguments should I use?

Discussed above

> 5. Edit /etc/fstab and comment/remove the /usr line
>
> nano -wc /mnt/gentoo/etc/fstab
>
> #/dev/vg/usr       /usr        reiserfs        noatime         0 0
>
> 6. Unmount mounted filesystems
>
> umount /mnt/gentoo/oldusr
> umount /mnt/gentoo
>
> 7. Reboot into new system
>
> Done?
>
> I'm pretty sure that:
>
> 1. There is no need to chroot into the real system during this process,
>
> and
>
> 2. I only need to mount / and the old /usr, no need to mount anything
>    else (/proc, /sys, /var, /home, activating swap, etc)
>
> Correct?

Both of these seem correct to me.

Good luck!

allan

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-02 11:26 AM, gottlieb@nyu.edu <gottlieb@nyu.edu> wrote:
> On Mon, Dec 02 2013, tanstaafl@libertytrek.org wrote:
>> So, here's the plan, please check me...
>>
>> 1. Boot off of the latest gentoo LiveDVD
>>
>> 2. Mount / and create new /usr directory

> I am missing something.  I would have thought your old / (dev/sda3)
> already has an empty /usr directory where you previously mounted
> /dev/vg/usr

Hmmm... I guess you're right, although I guess I'd have realized that as 
soon as I mounted / to /mnt/gentoo and did an ls...

>> 4. Copy /oldusr to /usr

> This suggests that your current root (dev/sda3) is big enough to
> include the previous /usr (dev/vg/usr).

Yep, plenty of room...

Are you saying you went through this too?

Hopefully a few others will chime in with more on the exact rsync 
arguments I should use...

Thanks Allan... :)

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[gottlieb]

On Mon, Dec 02 2013, tanstaafl@libertytrek.org wrote:

> On 2013-12-02 11:26 AM, gottlieb@nyu.edu <gottlieb@nyu.edu> wrote:
>> On Mon, Dec 02 2013, tanstaafl@libertytrek.org wrote:
>>> So, here's the plan, please check me...
>>>
>>> 1. Boot off of the latest gentoo LiveDVD
>>>
>>> 2. Mount / and create new /usr directory
>
>> I am missing something.  I would have thought your old / (dev/sda3)
>> already has an empty /usr directory where you previously mounted
>> /dev/vg/usr
>
> Hmmm... I guess you're right, although I guess I'd have realized that
> as soon as I mounted / to /mnt/gentoo and did an ls...
>
>>> 4. Copy /oldusr to /usr
>
>> This suggests that your current root (dev/sda3) is big enough to
>> include the previous /usr (dev/vg/usr).
>
> Yep, plenty of room...
>
> Are you saying you went through this too?

Yes I did, but did not have the room to simply stick /usr into /.
Like you I had lvm.

> Hopefully a few others will chime in with more on the exact rsync
> arguments I should use...
>
> Thanks Allan... :)

My pleasure.
allan

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Thanasis]

on 12/02/2013 04:02 PM Tanstaafl wrote the following:
> 
> So, here's the plan, please check me...
> 
> 1. Boot off of the latest gentoo LiveDVD

If you boot a different system to do the rsync, or, if you do it over
ssh, add the option --numeric-ids

I usually do
rsync -aHvxW --numeric-ids --delete <sourcedit>/ <targetdir>/

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-02 1:47 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
> on 12/02/2013 04:02 PM Tanstaafl wrote the following:
>>
>> So, here's the plan, please check me...
>>
>> 1. Boot off of the latest gentoo LiveDVD
>
> If you boot a different system to do the rsync, or, if you do it over
> ssh, add the option --numeric-ids

Thanks, but no, like I said, I'll just boot that system to a LiveDVD and 
do it from there...

So, I guess my main question is...

Would it hurt anything to use all of the args:

ie... -apogsvxAHPX

I guess I may be overthinking this (again)...

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Thanasis]

on 12/02/2013 08:58 PM Tanstaafl wrote the following:
> On 2013-12-02 1:47 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
>> on 12/02/2013 04:02 PM Tanstaafl wrote the following:
>>>
>>> So, here's the plan, please check me...
>>>
>>> 1. Boot off of the latest gentoo LiveDVD
>>
>> If you boot a different system to do the rsync, or, if you do it over
>> ssh, add the option --numeric-ids
> 
> Thanks, but no, like I said, I'll just boot that system to a LiveDVD and
> do it from there...

Are you sure the user IDs of the LiveDVD are the same as the other
system's users' IDs?
That is why I recommend using the option --numeric-ids.
And using it would not hurt anyway.

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-02 2:41 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
> on 12/02/2013 08:58 PM Tanstaafl wrote the following:
>> On 2013-12-02 1:47 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
>>> on 12/02/2013 04:02 PM Tanstaafl wrote the following:
>>>>
>>>> So, here's the plan, please check me...
>>>>
>>>> 1. Boot off of the latest gentoo LiveDVD
>>>
>>> If you boot a different system to do the rsync, or, if you do it over
>>> ssh, add the option --numeric-ids
>>
>> Thanks, but no, like I said, I'll just boot that system to a LiveDVD and
>> do it from there...
>
> Are you sure the user IDs of the LiveDVD are the same as the other
> system's users' IDs?
> That is why I recommend using the option --numeric-ids.
> And using it would not hurt anyway.

Right... poison pointed this out...

This is why I asked for help about the arguments.

I honestly don't care about superflous/unnecessary arguments, I just 
want to make sure I use at least the ones needed for this to work.

Thanks...

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1713 bytes --]

On Monday 02 Dec 2013 20:40:28 Tanstaafl wrote:
> On 2013-12-02 2:41 PM, Thanasis <thanasis@asyr.hopto.org> wrote:

> > That is why I recommend using the option --numeric-ids.
> > And using it would not hurt anyway.
> 
> Right... poison pointed this out...
> 
> This is why I asked for help about the arguments.
> 
> I honestly don't care about superflous/unnecessary arguments, I just
> want to make sure I use at least the ones needed for this to work.
> 
> Thanks...

The comment about --numeric-ids that Thanasis made is valid.  I messed up some 
fs of mine last time I used rsync, when I wasn't paying much attention!  I 
made a mental note to always use it in the future.  On the other hand, if 
you're not that comfortable with it, a quick trial run with a test filesystem 
will offer some assurance that your chosen command and options will work as 
you intended.  BTW, you do not *have* to use rsync:

  cp -a

will do the same.

  su -
  cd /old_usr
  tar --one-file-system -cf . | (cd /new_usr ; tar -xvpf - )

will also do the same.

Finally, star -copy is my favourite faster alternative to copying directories, 
inc. respecting any acl's and the like if you specify it in the options:

  su -
  star -copy <options> -C /old_usr . /new_usr

Then you can also add -diff to see if any file was not copied correctly (use 
star diffopts=!<option> to exclude things like ctime, or you'll drown in the 
noise of the output).


Speaking from experience I suggest that you do not blast your old /usr away 
until you have booted with /usr mounted in the new location and have verified 
that ownership and access rights are as you expected.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[William Kenworthy]

You are looking far too deep ....

just rsync -avP to /newusr
reboot to livecd
rsync again with --delete to update ... takes a only few seconds this
time - minimal downtime :)
mv /usr /oldusr
mv /newusr /usr
reboot

The --numeric-ids is a good idea but I've made my systems consistent
with the standard gentoo id's so that's no longer a problem.

Ive done this many times over the years, and to the system I am writing
this on many times (moving to lvm2, restoring from backups after disk
failures, restoring from backups after user failure - rm -rf /usr !)

If you need to practice, run up a vm and test/destroy :)

You have got the disk space, so if you have a backup its reversible so
don't be a wimp :)

BillK




On 03/12/13 05:36, Mick wrote:
> On Monday 02 Dec 2013 20:40:28 Tanstaafl wrote:
>> On 2013-12-02 2:41 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
> 
>>> That is why I recommend using the option --numeric-ids.
>>> And using it would not hurt anyway.
>>
>> Right... poison pointed this out...
>>
>> This is why I asked for help about the arguments.
>>
>> I honestly don't care about superflous/unnecessary arguments, I just
>> want to make sure I use at least the ones needed for this to work.
>>
>> Thanks...
> 
> The comment about --numeric-ids that Thanasis made is valid.  I messed up some 
> fs of mine last time I used rsync, when I wasn't paying much attention!  I 
> made a mental note to always use it in the future.  On the other hand, if 
> you're not that comfortable with it, a quick trial run with a test filesystem 
> will offer some assurance that your chosen command and options will work as 
> you intended.  BTW, you do not *have* to use rsync:
> 
>   cp -a
> 
> will do the same.
> 
>   su -
>   cd /old_usr
>   tar --one-file-system -cf . | (cd /new_usr ; tar -xvpf - )
> 
> will also do the same.
> 
> Finally, star -copy is my favourite faster alternative to copying directories, 
> inc. respecting any acl's and the like if you specify it in the options:
> 
>   su -
>   star -copy <options> -C /old_usr . /new_usr
> 
> Then you can also add -diff to see if any file was not copied correctly (use 
> star diffopts=!<option> to exclude things like ctime, or you'll drown in the 
> noise of the output).
> 
> 
> Speaking from experience I suggest that you do not blast your old /usr away 
> until you have booted with /usr mounted in the new location and have verified 
> that ownership and access rights are as you expected.
> 

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On Tue, 03 Dec 2013 06:24:43 +0800, William Kenworthy wrote:

> You have got the disk space, so if you have a backup its reversible so
> don't be a wimp :)

It's reversible even if there is no backup, because data it copied
from /usr to /, not moved. If the new /usr doesn't work for any reason,
just mount the old one on it.


-- 
Neil Bothwick

"Unthinking respect for authority is the greatest enemy of truth."
 (Albert Einstein)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-02 8:02 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Tue, 03 Dec 2013 06:24:43 +0800, William Kenworthy wrote:
>
>> You have got the disk space, so if you have a backup its reversible so
>> don't be a wimp :)
>
> It's reversible even if there is no backup, because data it copied
> from /usr to /, not moved. If the new /usr doesn't work for any reason,
> just mount the old one on it.

True, I had forgotten about that... thanks, that actually does make it 
much less risky, so definitely will be trying this soon (maybe even this 
weekend if I have time)... :)

Yeah, when it comes to servers, I'm more of a wimp than not... but being 
careful and conservative on my servers has saved me more times than I 
can count, so I'm ok with it... ;)

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Bruce Hill]

On Tue, Dec 03, 2013 at 06:47:01AM -0500, Tanstaafl wrote:
> 
> Yeah, when it comes to servers, I'm more of a wimp than not... but being 
> careful and conservative on my servers has saved me more times than I 
> can count, so I'm ok with it... ;)

I have one server with separate /usr that's in LVM. It only gets rebooted for
kernel changes, or power loss greater than USB. The latter condition happened
two days ago, but everything worked when it was booted again.

I've got an initramfs on my laptop with everything on one LVM, so writing one
for that server wouldn't be impossible. My only issue is having initramfs
forced on us because of other people's bad ideas. And my server is setup the
way it is for a lot of reasons, security being one. It has:

o@server ~ $ df -hT                                                                    │    link/ether a0:88:b4:54:33:04 brd ff:ff:ff:ff:ff:ff
Filesystem                   Type      Size  Used Avail Use% Mounted on                      │    inet 192.168.11.7/24 brd 192.168.11.255 scope global wlan0
rootfs                       rootfs    2.0G  116M  1.9G   6% /                               │       valid_lft forever preferred_lft forever
/dev/root                    xfs       2.0G  116M  1.9G   6% /                               │baruch ~ # ip addr
devtmpfs                     devtmpfs  3.0G     0  3.0G   0% /dev                            │1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
tmpfs                        tmpfs     603M  464K  603M   1% /run                            │    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
shm                          tmpfs     3.0G     0  3.0G   0% /dev/shm                        │    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
/dev/mapper/system-var       xfs        10G  721M  9.3G   8% /var                            │       valid_lft forever preferred_lft forever
/dev/mapper/system-usr       xfs        10G  4.8G  5.2G  49% /usr                            │2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN 
/dev/mapper/system-home      xfs       6.0G  5.5G  580M  91% /home                           │    link/ether 6e:83:0f:ef:52:15 brd ff:ff:ff:ff:ff:ff
/dev/mapper/storage-photos   xfs       500G   19G  482G   4% /photos                         │3: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
/dev/mapper/storage-backups  xfs       500G  166G  335G  34% /backups                        │    link/ether 00:21:cc:5e:c3:12 brd ff:ff:ff:ff:ff:ff
/dev/mapper/storage-offload  fuseblk   300G  341M  300G   1% /offload                        │4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
/dev/mapper/storage-peter    xfs        25G  1.7G   24G   7% /peter                          │    link/ether a0:88:b4:54:33:04 brd ff:ff:ff:ff:ff:ff
/dev/mapper/storage-jeremiah xfs        10G  3.4G  6.7G  34% /jeremiah

so moving /usr into / isn't even an option.

So perhaps one day it will get an initramfs ... or go back to devfs. :D
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Bruce Hill]

On Tue, Dec 03, 2013 at 11:06:16AM -0600, Bruce Hill wrote:
> 
> mingdao@server ~ $ df -hT                                                                    │    link/ether a0:88:b4:54:33:04 brd ff:ff:ff:ff:ff:ff
> Filesystem                   Type      Size  Used Avail Use% Mounted on                      │    inet 192.168.11.7/24 brd 192.168.11.255 scope global wlan0
> rootfs                       rootfs    2.0G  116M  1.9G   6% /                               │       valid_lft forever preferred_lft forever
> /dev/root                    xfs       2.0G  116M  1.9G   6% /                               │baruch ~ # ip addr
> devtmpfs                     devtmpfs  3.0G     0  3.0G   0% /dev                            │1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
> tmpfs                        tmpfs     603M  464K  603M   1% /run                            │    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> shm                          tmpfs     3.0G     0  3.0G   0% /dev/shm                        │    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
> /dev/mapper/system-var       xfs        10G  721M  9.3G   8% /var                            │       valid_lft forever preferred_lft forever
> /dev/mapper/system-usr       xfs        10G  4.8G  5.2G  49% /usr                            │2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN 
> /dev/mapper/system-home      xfs       6.0G  5.5G  580M  91% /home                           │    link/ether 6e:83:0f:ef:52:15 brd ff:ff:ff:ff:ff:ff
> /dev/mapper/storage-photos   xfs       500G   19G  482G   4% /photos                         │3: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
> /dev/mapper/storage-backups  xfs       500G  166G  335G  34% /backups                        │    link/ether 00:21:cc:5e:c3:12 brd ff:ff:ff:ff:ff:ff
> /dev/mapper/storage-offload  fuseblk   300G  341M  300G   1% /offload                        │4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
> /dev/mapper/storage-peter    xfs        25G  1.7G   24G   7% /peter                          │    link/ether a0:88:b4:54:33:04 brd ff:ff:ff:ff:ff:ff
> /dev/mapper/storage-jeremiah xfs        10G  3.4G  6.7G  34% /jeremiah

Note to self: cp/paste from tmux isn't working as expected ... read moar docs
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Jc García]

2013/12/2 William Kenworthy <billk@iinet.net.au>
>
> You are looking far too deep ....
>
>
> just rsync -avP to /newusr

+1
I have done this more or less the same way
>
> reboot to livecd
>
> rsync again with --delete to update ... takes a only few seconds this
> time - minimal downtime :)
> mv /usr /oldusr
> mv /newusr /usr
> reboot


Let's make this thread more interesting, would it be possible to do
this without a reboot? ie: going single user mode, kill anything that
might still be running from usr,  umount /usr, mount it to /mnt, rsync
-avP to usr, going again into runlevel 3 or 5.
Obviously not possible if running systemd.
>
>
> The --numeric-ids is a good idea but I've made my systems consistent
> with the standard gentoo id's so that's no longer a problem.
>
> Ive done this many times over the years, and to the system I am writing
> this on many times (moving to lvm2, restoring from backups after disk
> failures, restoring from backups after user failure - rm -rf /usr !)
>
> If you need to practice, run up a vm and test/destroy :)
>
> You have got the disk space, so if you have a backup its reversible so
> don't be a wimp :)
>
> BillK
>
>
>
>
> On 03/12/13 05:36, Mick wrote:
> > On Monday 02 Dec 2013 20:40:28 Tanstaafl wrote:
> >> On 2013-12-02 2:41 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
> >
> >>> That is why I recommend using the option --numeric-ids.
> >>> And using it would not hurt anyway.
> >>
> >> Right... poison pointed this out...
> >>
> >> This is why I asked for help about the arguments.
> >>
> >> I honestly don't care about superflous/unnecessary arguments, I just
> >> want to make sure I use at least the ones needed for this to work.
> >>
> >> Thanks...
> >
> > The comment about --numeric-ids that Thanasis made is valid.  I messed up some
> > fs of mine last time I used rsync, when I wasn't paying much attention!  I
> > made a mental note to always use it in the future.  On the other hand, if
> > you're not that comfortable with it, a quick trial run with a test filesystem
> > will offer some assurance that your chosen command and options will work as
> > you intended.  BTW, you do not *have* to use rsync:
> >
> >   cp -a
> >
> > will do the same.
> >
> >   su -
> >   cd /old_usr
> >   tar --one-file-system -cf . | (cd /new_usr ; tar -xvpf - )
> >
> > will also do the same.
> >
> > Finally, star -copy is my favourite faster alternative to copying directories,
> > inc. respecting any acl's and the like if you specify it in the options:
> >
> >   su -
> >   star -copy <options> -C /old_usr . /new_usr
> >
> > Then you can also add -diff to see if any file was not copied correctly (use
> > star diffopts=!<option> to exclude things like ctime, or you'll drown in the
> > noise of the output).
> >
> >
> > Speaking from experience I suggest that you do not blast your old /usr away
> > until you have booted with /usr mounted in the new location and have verified
> > that ownership and access rights are as you expected.
> >
>
>

PD: sorry if my english is not so good

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[William Kenworthy]

On 03/12/13 12:34, Jc García wrote:
> 2013/12/2 William Kenworthy <billk@iinet.net.au>
>> You are looking far too deep ....
>>
>>
>> just rsync -avP to /newusr
> +1
> I have done this more or less the same way
>> reboot to livecd
>>
>> rsync again with --delete to update ... takes a only few seconds this
>> time - minimal downtime :)
>> mv /usr /oldusr
>> mv /newusr /usr
>> reboot
>
> Let's make this thread more interesting, would it be possible to do
> this without a reboot? ie: going single user mode, kill anything that
> might still be running from usr,  umount /usr, mount it to /mnt, rsync
> -avP to usr, going again into runlevel 3 or 5.
> Obviously not possible if running systemd.
>
I did try it single user mode - it works but I rebooted when the
pressure came off just in case.  Wanted minimal downtime.

BillK

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Canek Peláez Valdés]

On Mon, Dec 2, 2013 at 10:34 PM, Jc García <jyo.garcia@gmail.com> wrote:
> 2013/12/2 William Kenworthy <billk@iinet.net.au>
>>
>> You are looking far too deep ....
>>
>>
>> just rsync -avP to /newusr
>
> +1
> I have done this more or less the same way
>>
>> reboot to livecd
>>
>> rsync again with --delete to update ... takes a only few seconds this
>> time - minimal downtime :)
>> mv /usr /oldusr
>> mv /newusr /usr
>> reboot
>
>
> Let's make this thread more interesting, would it be possible to do
> this without a reboot? ie: going single user mode, kill anything that
> might still be running from usr,  umount /usr, mount it to /mnt, rsync
> -avP to usr, going again into runlevel 3 or 5.
> Obviously not possible if running systemd.

I'm not so sure it's not possible. Perhaps it's even easier.

If you do "systemctl isolate emergency.target" then remount /
read/write, do the move, and then again isolate multi-user.target or
graphical.target, I think is possible. I will try on a virtual
machine; is an interesting question. You would need to use absolute
pathnames when actually performing the move, but I think is possible.

Regards.
-- 
Canek Peláez Valdés
Posgrado en Ciencia e Ingeniería de la Computación
Universidad Nacional Autónoma de México

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Canek Peláez Valdés]

On Tue, Dec 3, 2013 at 12:06 AM, Canek Peláez Valdés <caneko@gmail.com> wrote:
> On Mon, Dec 2, 2013 at 10:34 PM, Jc García <jyo.garcia@gmail.com> wrote:
>> 2013/12/2 William Kenworthy <billk@iinet.net.au>
>>>
>>> You are looking far too deep ....
>>>
>>>
>>> just rsync -avP to /newusr
>>
>> +1
>> I have done this more or less the same way
>>>
>>> reboot to livecd
>>>
>>> rsync again with --delete to update ... takes a only few seconds this
>>> time - minimal downtime :)
>>> mv /usr /oldusr
>>> mv /newusr /usr
>>> reboot
>>
>>
>> Let's make this thread more interesting, would it be possible to do
>> this without a reboot? ie: going single user mode, kill anything that
>> might still be running from usr,  umount /usr, mount it to /mnt, rsync
>> -avP to usr, going again into runlevel 3 or 5.
>> Obviously not possible if running systemd.
>
> I'm not so sure it's not possible. Perhaps it's even easier.

So, yeah, I think it's easier with systemd. You just:

1. systemctl isolate emergency.target
2. log in again (all the normal gettys are killed with the above command)
3. rsync -PvasHA /usr/ /newusr/
4. mv /usr /oldusr # mv is on /bin, so no problems here
5. mv /newusr /usr
6. rm -rf /oldusr (to make sure nothing uses it anymore)
7. systemctl isolate multi-user.target
8. You have your system again.

I did this on a minimal QEMU virtual machine. However I think it
should work with even the most complex setups, as long as your
initramfs has the necessary tools, which is really easy with dracut.

Regards.
-- 
Canek Peláez Valdés
Posgrado en Ciencia e Ingeniería de la Computación
Universidad Nacional Autónoma de México

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Jc García]

2013/12/3 Canek Peláez Valdés <caneko@gmail.com>:
> On Tue, Dec 3, 2013 at 12:06 AM, Canek Peláez Valdés <caneko@gmail.com> wrote:
>> On Mon, Dec 2, 2013 at 10:34 PM, Jc García <jyo.garcia@gmail.com> wrote:
>>> 2013/12/2 William Kenworthy <billk@iinet.net.au>
>>>>
>>>> You are looking far too deep ....
>>>>
>>>>
>>>> just rsync -avP to /newusr
>>>
>>> +1
>>> I have done this more or less the same way
>>>>
>>>> reboot to livecd
>>>>
>>>> rsync again with --delete to update ... takes a only few seconds this
>>>> time - minimal downtime :)
>>>> mv /usr /oldusr
>>>> mv /newusr /usr
>>>> reboot
>>>
>>>
>>> Let's make this thread more interesting, would it be possible to do
>>> this without a reboot? ie: going single user mode, kill anything that
>>> might still be running from usr,  umount /usr, mount it to /mnt, rsync
>>> -avP to usr, going again into runlevel 3 or 5.
>>> Obviously not possible if running systemd.
>>
>> I'm not so sure it's not possible. Perhaps it's even easier.
>
> So, yeah, I think it's easier with systemd. You just:
>
> 1. systemctl isolate emergency.target
> 2. log in again (all the normal gettys are killed with the above command)
> 3. rsync -PvasHA /usr/ /newusr/
> 4. mv /usr /oldusr # mv is on /bin, so no problems here
> 5. mv /newusr /usr
> 6. rm -rf /oldusr (to make sure nothing uses it anymore)
> 7. systemctl isolate multi-user.target
> 8. You have your system again.
>
Nice, I thought systemd residing within /usr would be the limitation,
i haven't used systemd very much , so i don't really know it's options,
but later after thinking about it, i thought that using the initramfs
would be a way to go,
but as i understand here systemd already has an option to make use of it.

> I did this on a minimal QEMU virtual machine. However I think it
> should work with even the most complex setups, as long as your
> initramfs has the necessary tools, which is really easy with dracut.
>
> Regards.
> --
> Canek Peláez Valdés
> Posgrado en Ciencia e Ingeniería de la Computación
> Universidad Nacional Autónoma de México
>

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Canek Peláez Valdés]

On Tue, Dec 3, 2013 at 5:22 PM, Jc García <jyo.garcia@gmail.com> wrote:
> 2013/12/3 Canek Peláez Valdés <caneko@gmail.com>:
>> On Tue, Dec 3, 2013 at 12:06 AM, Canek Peláez Valdés <caneko@gmail.com> wrote:
>>> On Mon, Dec 2, 2013 at 10:34 PM, Jc García <jyo.garcia@gmail.com> wrote:
>>>> 2013/12/2 William Kenworthy <billk@iinet.net.au>
>>>>>
>>>>> You are looking far too deep ....
>>>>>
>>>>>
>>>>> just rsync -avP to /newusr
>>>>
>>>> +1
>>>> I have done this more or less the same way
>>>>>
>>>>> reboot to livecd
>>>>>
>>>>> rsync again with --delete to update ... takes a only few seconds this
>>>>> time - minimal downtime :)
>>>>> mv /usr /oldusr
>>>>> mv /newusr /usr
>>>>> reboot
>>>>
>>>>
>>>> Let's make this thread more interesting, would it be possible to do
>>>> this without a reboot? ie: going single user mode, kill anything that
>>>> might still be running from usr,  umount /usr, mount it to /mnt, rsync
>>>> -avP to usr, going again into runlevel 3 or 5.
>>>> Obviously not possible if running systemd.
>>>
>>> I'm not so sure it's not possible. Perhaps it's even easier.
>>
>> So, yeah, I think it's easier with systemd. You just:
>>
>> 1. systemctl isolate emergency.target
>> 2. log in again (all the normal gettys are killed with the above command)
>> 3. rsync -PvasHA /usr/ /newusr/
>> 4. mv /usr /oldusr # mv is on /bin, so no problems here
>> 5. mv /newusr /usr
>> 6. rm -rf /oldusr (to make sure nothing uses it anymore)
>> 7. systemctl isolate multi-user.target
>> 8. You have your system again.
>>
> Nice, I thought systemd residing within /usr would be the limitation,
> i haven't used systemd very much , so i don't really know it's options,
> but later after thinking about it, i thought that using the initramfs
> would be a way to go,
> but as i understand here systemd already has an option to make use of it.

Yeah, you can create an initramfs with dracut that uses systemd; the
initramfs mounts /usr, and then handles back control to the systemd
instance in it. When shutting down, the inverse occurs; the systemd in
/usr handles back control to the systemd in the initramfs, which in
turns shutdowns the machine.

I don't know exactly how problematic would be for a complex setup
(/usr using LVM+cryptfs+mdraid, for example), but if the necessary
tools are available in the initramfs, then I think  it could be done.

Of course, the simplest and easiest thing to do is to use a live CD.

Regards.
-- 
Canek Peláez Valdés
Posgrado en Ciencia e Ingeniería de la Computación
Universidad Nacional Autónoma de México

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-02 5:24 PM, William Kenworthy <billk@iinet.net.au> wrote:
> You are looking far too deep ....
>
> just rsync -avP to /newusr

But this would not copy the hardlinks... and there are a bunch on /usr, 
so...

> reboot to livecd
> rsync again with --delete to update ... takes a only few seconds this
> time - minimal downtime:)
> mv /usr /oldusr
> mv /newusr /usr
> reboot
>
> The --numeric-ids is a good idea but I've made my systems consistent
> with the standard gentoo id's so that's no longer a problem.

Ok, so, last questions...

When mounting /, do I need to specify the fstype and mount option 
showing in fstab, ie:

mount -t ext3 -o noatime /dev/sda3 /mnt/gentoo/

Then when mounting the old /usr, since it is reiserfs:

mount -t reiserfs -o noatime /dev/vg/usr /mnt/gentoo/oldusr

Last - are there any concerns about the fact that /usr is currently on a 
reiserfs file system, moving to an ext3 filesystem?

Current command I'll be using:

rsync -avHP --numeric-ids /mnt/gentoo/oldusr/ /mnt/gentoo/usr/

Thanks again,

Charles

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-03 8:19 AM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
> Current command I'll be using:
>
> rsync -avHP --numeric-ids /mnt/gentoo/oldusr/ /mnt/gentoo/usr/

I had initially been planning on just using cp, trying now to remember 
why I decided on using rsync - I think it was someone here who said it 
would be better.

Anyway, not to start a holy war about which is better, rsync or cp, but 
someone on the rsync list just posted that using rsync for this was 
really not the best use of rsync, and that cp was much better suited 
(would be much faster for one thing, and would not suffer the same 
potential problem of running out of memory because of too many hard 
links for another), and I think I agree with them...

<sigh>

So, can someone confirm that this command:

cp -ax /usr/. /usr.tmp/.

would accomplish the same goal?

I've never used the -x option with cp... what exactly is meant by 'stay 
on same filesystem'? Should I use this seeing as current /usr is 
reiserfs on LVM, and / is ext3 on simple partition - ie, *not* the 'same 
filesystem'?

Is the only real advantage of using rsync to do this it's ability to 
pick up where it left off if there is a problem?

If so, then I imagine I could use the cp command first on the live 
system to 'prime' it, then use the rsync command after booting to the 
liveCD to quickly confirm it - but if there were no issues during the 
initial cp, and nothing changes in between, there shouldn't really be 
any difference to copy anyway?

Sorry for all the questions, I promise this will be the last one on this 
subject...

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Peter Humphrey]

On Wednesday 04 Dec 2013 07:28:18 Tanstaafl wrote:

> I've never used the -x option with cp... what exactly is meant by 'stay
> on same filesystem’?Should 

"Stay on same filesystem" is for the case in which you have another partition 
mounted somewhere in the tree below the current working directory. It means 
that you want to omit everything in that second file system. If you haven’t any 
such complication you don’t need to specify -x.

For instance, I have separate partitions for /usr/portage and 
/usr/portage/packages. If I wanted to cp everything in portage but not in 
packages I’d specify -x.

> I use this seeing as current /usr is reiserfs on LVM, and / is ext3 on
> simple partition - ie, *not* the 'same filesystem’?

Doesn’t matter. The type of file system is not visible to the copying program: 
to it, a file is a file is a file. Well, for present purposes anyway. I think you 
can interpret file-system as identical to partition here.

> ... I imagine I could use the cp command first on the live system to ‘prime’ 
> it, then use the rsync command after booting to the liveCD to quickly confirm
> it - but if there were no issues during the initial cp, and nothing changes
> in between, there shouldn't really be any difference to copy anyway?

Indeed. I hope you don’t have experience of cp failing to copy what it should.

> Sorry for all the questions, I promise this will be the last one on this
> subject...

It’s how we learn, so don’t worry about it.   :-)

-- 
Regards
Peter

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-04 8:07 AM, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
> On Wednesday 04 Dec 2013 07:28:18 Tanstaafl wrote:
>> I've never used the -x option with cp... what exactly is meant by 'stay
>> on same filesystem’?Should

> "Stay on same filesystem" is for the case in which you have another partition
> mounted somewhere in the tree below the current working directory. It means
> that you want to omit everything in that second file system. If you haven’t any
> such complication you don’t need to specify -x.

Ah, ok that makes perfect sense, thanks.

And for the record (you didn't specifically say so), are you in 
agreement that

cp -a /usr/. /usr.tmp/.

will accomplish the exact same thing as the rsync command I was planning 
on using?

>> ... I imagine I could use the cp command first on the live system to ‘prime’
>> it, then use the rsync command after booting to the liveCD to quickly confirm
>> it - but if there were no issues during the initial cp, and nothing changes
>> in between, there shouldn't really be any difference to copy anyway?

> Indeed. I hope you don’t have experience of cp failing to copy what it should.

Right... if it did it would most likely indicate some kind of filesystem 
corruption...

Thanks again Peter,

Charles

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[godzil]

Le 2013-12-06 11:07, Tanstaafl wrote :
> And for the record (you didn't specifically say so), are you in 
> agreement that
> 
> cp -a /usr/. /usr.tmp/.
> 
> will accomplish the exact same thing as the rsync command I was
> planning on using?
> 

For me, it's best to use rsync, because rsync will not copy file if they 
are already existing, and are the same. It's quite usefull when using a 
copy over a network, or even locally when spurious error can occur and 
especially when the filesystem is live and file may be modified during 
the copy.

To copy large file tree rsync is, for me, always the best choice, and 
I'm pretty sure that there are some cases that rsync will behave better 
than a simple recursive cp

Cheers,
Godzil

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Bruce Hill]

On Fri, Dec 06, 2013 at 06:07:01AM -0500, Tanstaafl wrote:
> 
> And for the record (you didn't specifically say so), are you in 
> agreement that
> 
> cp -a /usr/. /usr.tmp/.
> 
> will accomplish the exact same thing as the rsync command I was planning 
> on using?

There are reasons why rsync is better than cp. The best one IMO is that rysnc
will not copy a corrupt file, while cp will. And rsync will tell you about the
corrupt file.

You have so many questions about rsync, and different opinions thrown at you.
Why do you not read "man rsync" and learn about it? This man page is thorough,
easy to read, and explains it's usage by examples.

Cheers,
Bruce
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-06 8:13 AM, Bruce Hill <daddy@happypenguincomputers.com> wrote:
> There are reasons why rsync is better than cp. The best one IMO is that rysnc
> will not copy a corrupt file, while cp will. And rsync will tell you about the
> corrupt file.

Interesting and a good reason to use rsync over cp... if true... are you 
certain of this?

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Bruce Hill]

On Fri, Dec 06, 2013 at 11:35:29AM -0500, Tanstaafl wrote:
> On 2013-12-06 8:13 AM, Bruce Hill <daddy@happypenguincomputers.com> wrote:
> > There are reasons why rsync is better than cp. The best one IMO is that rysnc
> > will not copy a corrupt file, while cp will. And rsync will tell you about the
> > corrupt file.
> 
> Interesting and a good reason to use rsync over cp... if true... are you 
> certain of this?

Yes, absolutely. Which is one reason I suggested you rean "man rsync". I don't
mean to be unkind, but there have been so many conflicting opinions in your
thread.
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

SOLVED - Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-03 8:19 AM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
> Current command I'll be using:
>
> rsync -avHP --numeric-ids /mnt/gentoo/oldusr/ /mnt/gentoo/usr/

Well, that was about as uneventful as it gets...

Took all of 6 minutes (and almost all of that was rsyncing /usr)...

Made a forum post in case anyone else wants to do this and may be a 
little hesitant like I was...

http://forums.gentoo.org/viewtopic-p-7457324.html#7457324

Thanks to all who patiently answered my questions...

I now have /usr merged back into / and no longer have to choose between 
using an intramfs (which I vehemently did not want to do) and updating 
my system without fear of breakage.

<whew>

Re: SOLVED - Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Peter Humphrey]

On Sunday 08 Dec 2013 14:41:44 Tanstaafl wrote:
> On 2013-12-03 8:19 AM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
> > Current command I'll be using:
> > 
> > rsync -avHP --numeric-ids /mnt/gentoo/oldusr/ /mnt/gentoo/usr/
> 
> Well, that was about as uneventful as it gets...
> 
> Took all of 6 minutes (and almost all of that was rsyncing /usr)...

Glad it went well. Now for the next hurdle, eh?  :-)

-- 
Regards
Peter

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Poison BL.]

An alternative to booting to external media, etc, would be a bind
mount of / and /usr on separate temporary mount points, then dumping
the data between them, leaving the existing system chugging along. A
re-mount of the current /usr in -o ro mode might not be a terrible
idea in that case. I had a good bit of luck going that route. A simple
"cp -a" did the trick on the one system I've bothered with it on so
far. I have a couple laptops that're rushing headlong into unsupported
land right now though, so I'll be revisiting this soon enough.

As for specifics:

# Make the temporary working areas
mkdir /tmp/a; mkdir /tmp/b
# Make sure nothing changes in /usr while the copy is done
mount -o remount,ro /usr
# Mount a mirror of the source and destination filesystems
mount --bind /usr/ /tmp/a
mount --bind / /tmp/b
# And now, copy.
cd /tmp/a
cp -a ./ /tmp/b/usr/

The one big point of what not to do would be "mount --rbind". Very
important (recursive bind would have the current /usr still visible in
/tmp/b/usr/).

After all that, comment out /usr in fstab and reboot. You *could* even
just drop to a minimal runlevel that doesn't require /usr, unmount the
old one and then jump back to your standard runlevel, but due to the
reasons this is required now, I'm not entirely sure that option exists
anymore (i.e. too much is dependent on /usr).

That said, if you are booting to a LiveDVD --

On Mon, Dec 2, 2013 at 1:58 PM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
> On 2013-12-02 1:47 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
>>
>> on 12/02/2013 04:02 PM Tanstaafl wrote the following:
>>>
>>>
>>> So, here's the plan, please check me...
>>>
>>> 1. Boot off of the latest gentoo LiveDVD
>>
>>
>> If you boot a different system to do the rsync, or, if you do it over
>> ssh, add the option --numeric-ids
>
>
> Thanks, but no, like I said, I'll just boot that system to a LiveDVD and do
> it from there...

That actually does fall under "boot a different system" since the
users won't line up between a LiveDVD and your actual system.

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] Merging separate /usr back into / - one last time...

[Tanstaafl]

On 2013-12-02 2:25 PM, Poison BL. <poisonbl@gmail.com> wrote:
> An alternative to booting to external media, etc, would be a bind
> mount of / and /usr on separate temporary mount points, then dumping
> the data between them, leaving the existing system chugging along. A
> re-mount of the current /usr in -o ro mode might not be a terrible
> idea in that case.

<snip>

Not comfortable doing that on a productions server at all... but thanks 
anyway... :)

> That said, if you are booting to a LiveDVD --
>
> On Mon, Dec 2, 2013 at 1:58 PM, Tanstaafl <tanstaafl@libertytrek.org> wrote:
>> Thanks, but no, like I said, I'll just boot that system to a LiveDVD and do
>> it from there...

> That actually does fall under "boot a different system" since the
> users won't line up between a LiveDVD and your actual system.

Hmmm, ok, but if I chroot'd into the system (ie, like I might do if I 
had an interrupted install, and wanted to pick it back up), that would 
solve that?

Or, I could just figure out the right rsync arguments to use... ;)

I just asked on the rsync list too, maybe someone there will chime in 
with a little more confidence...