[gentoo-user] How to move ext4 partition

[gentoo-user] How to move ext4 partition

[Grant Edwards]

I've got a Gentoo install using a GPT partition table and Legacy boot
using Grub2. There is a single /root parition and a single swap
partition on the drive.

I did not create a bios-boot partition at the start of the disk, so I
had to force grub2 to install using block-lists.  I'd like to fix that
now. This requires that I move the ext4 root partition towards the end
of the drive to create 2MB of free space at the start of the drive for
a new bios-boot partition.

I see that Gnu parted no longer has a move command.

However, GParted apparently does. Can GParted move an ext4 filesystem
to a destination location that overlaps its starting location?

For example, I have a 500GB partition containing an ext4 filesystem
starting at sector 2048 (1MiB). I want to move that filesystem so that
it starts at sector 3*2048 (3MiB).

Can that be done in-place?

Or should I just back up the filesystem to a second drive and start
from scratch?

Re: [gentoo-user] How to move ext4 partition

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

On Wed, 20 Sep 2023 20:24:17 -0000 (UTC), Grant Edwards wrote:

> However, GParted apparently does. Can GParted move an ext4 filesystem
> to a destination location that overlaps its starting location?
> 
> For example, I have a 500GB partition containing an ext4 filesystem
> starting at sector 2048 (1MiB). I want to move that filesystem so that
> it starts at sector 3*2048 (3MiB).
> 
> Can that be done in-place?
> 
> Or should I just back up the filesystem to a second drive and start
> from scratch?

Given that you'd want to backup before such an operation anyway, you may
as well then restore from that backup. I'm sure it will be a lot quicker
than GParted's moving all the data around.


-- 
Neil Bothwick

He who laughs last thinks slowest!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: How to move ext4 partition

[Grant Edwards]

On 2023-09-20, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 20 Sep 2023 20:24:17 -0000 (UTC), Grant Edwards wrote:
>
>> For example, I have a 500GB partition containing an ext4 filesystem
>> starting at sector 2048 (1MiB). I want to move that filesystem so that
>> it starts at sector 3*2048 (3MiB).
>> 
>> Can that be done in-place?
>> 
>> Or should I just back up the filesystem to a second drive and start
>> from scratch?
>
> Given that you'd want to backup before such an operation anyway,

It's a machine with very limited uses, so I'd probably only back up
/etc and /root. Reinstalling probably wouldn't take too much longer
than backing up and restoring /.

> you may as well then restore from that backup. I'm sure it will be a
> lot quicker than GParted's moving all the data around.

That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are. After 40 years using Unix, you'd
think I'd know that (or have it written down somewhere). :)

That said, I think I will go with the backup, repartition, restore
method. It's been many, many years since I used GParted, and I can
probably have the whole job done from the command-line before I can
figure out how the GParted GUI works.

Re: [gentoo-user] Re: How to move ext4 partition

[Victor Ivanov]

On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
> That depends on how long it takes me to decide on tar vs. rsync and
> what the appropriate options are.

I've done this a number of times for various reasons over the last 1-2
years, most recently a few months ago due to hard drive swap, and I
find tar works just fine:

$ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Likewise to extract, but make sure "--xattrs" is present

Provided backup space isn't an issue, I wouldn't bother with
compression. It could be a lot quicker too depending on the size of
your root partition.

> That said, I think I will go with the backup, repartition, restore
> method.
Sounds like a sound course of action to me, reinstalling is just too
much faff and can often be avoided. I have a long standing distrust of
partition management utilities that claim to move filesystem data too.
I'm sure they work juuuuust fine :)

Just make sure you update /etc/fstab and bootloader config file with
the new filesystem UUID or partition indices.

Regards,
V

Re: [gentoo-user] Re: How to move ext4 partition

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

Am Wed, Sep 20, 2023 at 10:57:00PM +0100 schrieb Victor Ivanov:

> On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> >
> > That depends on how long it takes me to decide on tar vs. rsync and
> > what the appropriate options are.
> 
> I've done this a number of times for various reasons over the last 1-2
> years, most recently a few months ago due to hard drive swap, and I
> find tar works just fine:
> 
> $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Does that stop at file system boundaries (because you tar up '/')? I think 
it must be, otherwise you wouldn’t use it that way.
But when copying a root file system, out of habit I first bind-mount it in a 
subdirectory and tar/rsync from there instead. This will also make files 
visible which might be hidden under an active mount.

This is not necessary if you do it from a live system, but then you wouldn’t 
tar up / in the first place.

> Likewise to extract, but make sure "--xattrs" is present
> 
> Provided backup space isn't an issue, I wouldn't bother with
> compression. It could be a lot quicker too depending on the size of
> your root partition.

Or not, depending on the speed of the backup device. ;-)
LZO compression (or zstd with a low setting) has negligible CPU cost, but 
can lower the file size quite nicely, specially with large binaries or debug 
files.

-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

Do you steel taglines, too?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: How to move ext4 partition

[Grant Edwards]

On 2023-09-20, Frank Steinmetzger <Warp_7@gmx.de> wrote:
> Am Wed, Sep 20, 2023 at 10:57:00PM +0100 schrieb Victor Ivanov:
>
>> On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>> >
>> > That depends on how long it takes me to decide on tar vs. rsync and
>> > what the appropriate options are.
>> 
>> I've done this a number of times for various reasons over the last 1-2
>> years, most recently a few months ago due to hard drive swap, and I
>> find tar works just fine:
>> 
>> $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .
>
> Does that stop at file system boundaries (because you tar up '/')? I think 
> it must be, otherwise you wouldn’t use it that way.
> But when copying a root file system, out of habit I first bind-mount it in a 
> subdirectory and tar/rsync from there instead. This will also make files 
> visible which might be hidden under an active mount.

The partition/fs being backed up isn't live (it's mounted, but it's
not the root partition of the host doing the backup), so nothing is
mounted within it and there aren't any /proc or /sys entries in
it. So, in this case there's no need to worry about crossing
filesystem boundaries.

--
Grant

Re: [gentoo-user] Re: How to move ext4 partition

[Victor Ivanov]

On Thu, 21 Sept 2023 at 02:01, Frank Steinmetzger <Warp_7@gmx.de> wrote:
> > $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .
>
> Does that stop at file system boundaries (because you tar up '/')? I think
> it must be, otherwise you wouldn’t use it that way.
No, it doesn't. It will archive everything recursively including
mounted directories, so fair point for raising this. It's something
that I do not normally consider, as I tend not to do full root backup
on the running system itself. But how cool is using bind mount for
this situation as you suggest? Simple and effective, I like it.

Actually, the presence of -C in the above example is purely out of
habit. I use it when pointing to a full path, e.g. "/path/to/dir" so
it doesn't end up creating the "path/to/dir" path prefix inside the
archive and only archives the contents of the path.

It's effectively changing to that directory and archiving everything
there, but saves you from doing the "cd".

Naturally, for "/" this is superfluous and "-C / ." can be replaced
with just "/".

> > Provided backup space isn't an issue, I wouldn't bother with
> > compression. It could be a lot quicker too depending on the size of
> > your root partition.
>
> Or not, depending on the speed of the backup device. ;-)
> LZO compression (or zstd with a low setting) has negligible CPU cost, but
> can lower the file size quite nicely, specially with large binaries or debug
> files.
>
That's true :) I had somehow forgotten of tar's support for LZO and
zstd as my default finger memory approach is to use -J for xz. Good
memory nudge here!

Regards,
V

[gentoo-user] Re: How to move ext4 partition

[Grant Edwards]

On 2023-09-20, Victor Ivanov <vic.m.ivanov@gmail.com> wrote:
> On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>>
>> That depends on how long it takes me to decide on tar vs. rsync and
>> what the appropriate options are.
>
> I've done this a number of times for various reasons over the last 1-2
> years, most recently a few months ago due to hard drive swap, and I
> find tar works just fine:
>
> $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .
>
> Likewise to extract, but make sure "--xattrs" is present

Yep, that's pretty much what I decided on based on the tar command
shown at

   https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage

Interestingly, the Arch Linux Wiki recommends using bsdtar because
"GNU tar with --xattrs will not preserve extended attributes".

> Provided backup space isn't an issue, I wouldn't bother with
> compression. It could be a lot quicker too depending on the size of
> your root partition.

Both the drive being "fixed" and the backup drive are in a USB3
attached dual slot drive dock, so I'm thinking compression might be
worthwhile.

> Just make sure you update /etc/fstab and bootloader config file with
> the new filesystem UUID or partition indices.

I always forget one or the other until after I try to boot the first
time.  That's why I keep systemrescuecd and Gentoo minimal install
USB drives on hand.

--
Grant

Re: [gentoo-user] Re: How to move ext4 partition

[Victor Ivanov]

On Wed, 20 Sept 2023 at 23:58, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> Yep, that's pretty much what I decided on based on the tar command
> shown at
>
>    https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage
>
> Interestingly, the Arch Linux Wiki recommends using bsdtar because
> "GNU tar with --xattrs will not preserve extended attributes".
I remember coming across this too as I've previously had issues
preserving some extended attributes, notably on files under my home
directory. However, I found that using "--xattrs-include='*.*'" in
addition to "--xattrs" works pretty well and does record attributes
that would otherwise be excluded with just "--xattrs".

I cannot comment, however, if it truly includes "everything" in every
possible scenario.

>
> Both the drive being "fixed" and the backup drive are in a USB3
> attached dual slot drive dock, so I'm thinking compression might be
> worthwhile.
>
Then LZO or zstd might indeed be a better approach as suggested by Frank.

> > Just make sure you update /etc/fstab and bootloader config file with
> > the new filesystem UUID or partition indices.
>
> I always forget one or the other until after I try to boot the first
> time.  That's why I keep systemrescuecd and Gentoo minimal install
> USB drives on hand.

Me too, even just recently when I migrated my OS to another build I
decided to do a few partition touch ups and fell once more into this
trap. I updated fstab but not the bootloader. Luckily, Gentoo minimal
install image is so tiny a bootable medium can literally be created in
minutes.

Good luck!

Regards,
V

[gentoo-user] Re: How to move ext4 partition

[Grant Edwards]

On 2023-09-21, Victor Ivanov <vic.m.ivanov@gmail.com> wrote:
> On Wed, 20 Sept 2023 at 23:58, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
>>> Just make sure you update /etc/fstab and bootloader config file
>>> with the new filesystem UUID or partition indices.
>>
>> I always forget one or the other until after I try to boot the
>> first time.  That's why I keep systemrescuecd and Gentoo minimal
>> install USB drives on hand.
>
> Me too, even just recently when I migrated my OS to another build I
> decided to do a few partition touch ups and fell once more into this
> trap. I updated fstab but not the bootloader. Luckily, Gentoo
> minimal install image is so tiny a bootable medium can literally be
> created in minutes.

The tar backup restore worked just fine (and didn't take long, even
though both drives were connected via USB).  I've since fixed a second
machine by adding a bios-boot partition. I should have started using
them when I switched from MBR to GPT, but I think I got bios-boot
partitions confused with UEFI boot partitions. :/

I'm also working on switching to using either labels or uuids in fstab
and grub configs so that changes in partition numbers don't cause
problems.  Of course I've discovered for the Nth time in the past
10-15 years, that for the root= command line argument, the kernel
doesn't grok LABEL or UUID values -- it only understands device names
and PARTUUID.

--
Grant

Re: [gentoo-user] Re: How to move ext4 partition

[Jack]

On 9/21/23 16:23, Grant Edwards wrote:
> On 2023-09-21, Victor Ivanov <vic.m.ivanov@gmail.com> wrote:
>> On Wed, 20 Sept 2023 at 23:58, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>>>> Just make sure you update /etc/fstab and bootloader config file
>>>> with the new filesystem UUID or partition indices.
>>> I always forget one or the other until after I try to boot the
>>> first time.  That's why I keep systemrescuecd and Gentoo minimal
>>> install USB drives on hand.
>> Me too, even just recently when I migrated my OS to another build I
>> decided to do a few partition touch ups and fell once more into this
>> trap. I updated fstab but not the bootloader. Luckily, Gentoo
>> minimal install image is so tiny a bootable medium can literally be
>> created in minutes.
> The tar backup restore worked just fine (and didn't take long, even
> though both drives were connected via USB).  I've since fixed a second
> machine by adding a bios-boot partition. I should have started using
> them when I switched from MBR to GPT, but I think I got bios-boot
> partitions confused with UEFI boot partitions. :/
>
> I'm also working on switching to using either labels or uuids in fstab
> and grub configs so that changes in partition numbers don't cause
> problems.  Of course I've discovered for the Nth time in the past
> 10-15 years, that for the root= command line argument, the kernel
> doesn't grok LABEL or UUID values -- it only understands device names
> and PARTUUID.
while my Gentoo grub.cfg has root=PARTUUID=, my Artix Linux install 
(using openrc) has root=UUID=.  I wasn't aware they had mucked with grub 
(2.12-rc1) nor do I know if it's a recent change in grub.

[gentoo-user] Re: How to move ext4 partition

[Grant Edwards]

On 2023-09-21, Jack <ostroffjh@users.sourceforge.net> wrote:
>
>> [...] Of course I've discovered for the Nth time in the past 10-15
>> years, that for the root= command line argument, the kernel doesn't
>> grok LABEL or UUID values -- it only understands device names and
>> PARTUUID.
>
> while my Gentoo grub.cfg has root=PARTUUID=, my Artix Linux install
> (using openrc) has root=UUID=.  I wasn't aware they had mucked with
> grub (2.12-rc1) nor do I know if it's a recent change in grub.

AFAIK, it's not grub (grub does know how to handle LABEL and UUID when
setting grub's root).  For the kernel, it's something in the initrd's
'init' executable that parses the root=UUID= or root=LABEL=, searches
the available filesystems to find a match, then mounts the matching
filesystem and does a chroot to it (or someting like that).

If you don't have an initrd, then the kernel itself has to handle the
root=<whatever> and that code only knows about device names and
partition UUIDs. It doesn't know anything about filesystems (which
doesn't make much sense, since the next step is to mount the specified
partition, and it obviously knows about filesystems at that point).

At least that's what I've read...

Re: [gentoo-user] Re: How to move ext4 partition

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 881 bytes --]

On Wed, 20 Sep 2023 21:28:29 -0000 (UTC), Grant Edwards wrote:

> > you may as well then restore from that backup. I'm sure it will be a
> > lot quicker than GParted's moving all the data around.  
> 
> That depends on how long it takes me to decide on tar vs. rsync and
> what the appropriate options are. After 40 years using Unix, you'd
> think I'd know that (or have it written down somewhere). :)

If you written everything down, after 40 years you'd have so many bits of
paper you wouldn't be able to find anything ;-)
 
> That said, I think I will go with the backup, repartition, restore
> method. It's been many, many years since I used GParted, and I can
> probably have the whole job done from the command-line before I can
> figure out how the GParted GUI works.

:)


-- 
Neil Bothwick

The facts, although interesting, are usually irrelevant.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] How to move ext4 partition

[Wol]

On 20/09/2023 22:16, Neil Bothwick wrote:
> On Wed, 20 Sep 2023 20:24:17 -0000 (UTC), Grant Edwards wrote:
> 
>> However, GParted apparently does. Can GParted move an ext4 filesystem
>> to a destination location that overlaps its starting location?
>>
>> For example, I have a 500GB partition containing an ext4 filesystem
>> starting at sector 2048 (1MiB). I want to move that filesystem so that
>> it starts at sector 3*2048 (3MiB).
>>
>> Can that be done in-place?
>>
>> Or should I just back up the filesystem to a second drive and start
>> from scratch?
> 
> Given that you'd want to backup before such an operation anyway, you may
> as well then restore from that backup. I'm sure it will be a lot quicker
> than GParted's moving all the data around.
> 
Or, assuming the people who wrote gparted have two brain cells to rub 
together, I'm pretty sure they use the same technique as memmove.

"If the regions overlap, make sure you start from whichever end won't 
overwrite the source, otherwise start at whichever end you like".

Barring screw-ups (a very unsafe assumption :-), I'm pretty certain you 
don't even need a backup!

I suspect the man-page even confirms this behaviour.

Cheers,
Wol

[gentoo-user] Re: How to move ext4 partition

[Grant Edwards]

On 2023-09-20, Wol <antlists@youngman.org.uk> wrote:

> Or, assuming the people who wrote gparted have two brain cells to rub 
> together, I'm pretty sure they use the same technique as memmove.
>
> "If the regions overlap, make sure you start from whichever end won't 
> overwrite the source, otherwise start at whichever end you like".
>
> Barring screw-ups (a very unsafe assumption :-), I'm pretty certain you 
> don't even need a backup!
>
> I suspect the man-page even confirms this behaviour.

Not that I could see.  The only mention of "move" on the man page is
in this list of features.

    With gparted you can accomplish the following tasks:
      - Create a partition table on a disk device.
      - Enable and disable partition flags such as boot and hidden.
      - Perform actions with partitions such as create, delete,
        resize, move, check, label, copy, and paste.

Assuming GParted is smart enough to do overlapping moves, is it smart
enough to only copy filesystem data and not copy "empty" sectors?
According to various forum posts, it is not: moving a partion copies
every sector. [That's certainly the obvious, safe thing to do.]

The partition in question is 200GB, but only 7GB is used, so I think
backup/restore is the way to go...

--
Grant

Re: [gentoo-user] Re: How to move ext4 partition

[Wols Lists]

On 20/09/2023 23:39, Grant Edwards wrote:
> Assuming GParted is smart enough to do overlapping moves, is it smart
> enough to only copy filesystem data and not copy "empty" sectors?
> According to various forum posts, it is not: moving a partion copies
> every sector. [That's certainly the obvious, safe thing to do.]

Seeing as it knows nothing about filesystems, and everything about 
partitions, it will treat the partition as an opaque blob and move it as 
a single object ...
> 
> The partition in question is 200GB, but only 7GB is used, so I think
> backup/restore is the way to go...

You would think so :-)

I use ext4, and make heavy use of hard links. Last time I tried a 
straight copy (not backup/restore) I think the copied partition would 
have been three times the size of the original - that is if it hadn't 
run out of space first :-)

But it sounds like that would work well for you.

Cheers,
Wol

Re: [gentoo-user] Re: How to move ext4 partition

[Håkon Alstadheim]

Den 22.09.2023 08:48, skrev Wols Lists:
> On 20/09/2023 23:39, Grant Edwards wrote:
>> Assuming GParted is smart enough to do overlapping moves, is it smart
>> enough to only copy filesystem data and not copy "empty" sectors?
>> According to various forum posts, it is not: moving a partion copies
>> every sector. [That's certainly the obvious, safe thing to do.]
>
> Seeing as it knows nothing about filesystems, and everything about 
> partitions, it will treat the partition as an opaque blob and move it 
> as a single object ...
>>
>> The partition in question is 200GB, but only 7GB is used, so I think
>> backup/restore is the way to go...
>
> You would think so :-)
>
> I use ext4, and make heavy use of hard links. Last time I tried a 
> straight copy (not backup/restore) I think the copied partition would 
> have been three times the size of the original - that is if it hadn't 
> run out of space first :-)
>
Just for completeness, you should check out the e2image command, though 
NOT for direct in-place overlapping move. Something like "e2image -ra 
$sourcevol $targetvol" has worked very well for me. That is obviously 
NOT for overlapping move, but for a backup it is quite fast. If you are 
moving to a bigger volume, I'd do resize as a separate step after the move.