[gentoo-dev] rfc: usrmerge script

[gentoo-dev] rfc: usrmerge script

[William Hubbs]

[-- Attachment #1.1: Type: text/plain, Size: 330 bytes --]

All,

the following is a script which will migrate a Gentoo system to the usr
merge layout. This is similar to the unsymlink-lib tool used to migrate
a system  from the 17.0 to the 17.1 profiles.

I'm attaching it here to get some comments before I package it, so
please let me know if I have missed something.

Thanks,

William


[-- Attachment #1.2: usrmerge --]
[-- Type: text/plain, Size: 3885 bytes --]

#!/bin/bb
# shellcheck disable=SC2039 shell=sh

rollback_path=/var/usrmerge-rollback

check_internal_commands() {
	for cmd in cp ln; do
		[ "$(command -v "${cmd}")" = ${cmd} ] && continue
		printf 'busybox does not include the %s command internally\n' "${cmd}"
		return 1
	done
	return 0
}

check_root() {
 [ "$(id -u)" = 0 ] && return 0
 printf 'This script must be run by root\n'
 return 1
}

run_command() {
	local cmd
	[ -n "${dryrun}" ] && cmd="echo"
	${cmd} "$@" || exit
	return 0
}

setup_rollback() {
	if [ -e "${rollback_path}" ]; then
		printf '%s already exists\n' "${rollback_path}"
		printf 'please remove it before continuing\n'
		return 1
	fi
	printf 'Creating %s to allow rollbacks\n' "${rollback_path}"
	run_command mkdir "${rollback_path}"
	run_command mkdir "${rollback_path}/usr"
	local dir
	for dir in /bin /lib* /sbin; do
		run_command cp -a "${dir}" "${rollback_path}"
	done
	for dir in /usr/bin /usr/lib* /usr/sbin; do
		[ "${dir}" = /usr/libexec ] && continue
		run_command cp -a "${dir}" "${rollback_path}/usr"
	done
	return 0
}

action_finish() {
	run_command rm -fr "${rollback_path}"
	return 0
}

action_help() {
	local cmd
	cmd="$(basename "${0}")"
	printf 'usage: %s -h\n' "${cmd}"
	printf '       %s [-d] -f|-m|-r\n' "${cmd}"
	printf '\n'
	printf '-h | --help      - displays this message\n'
	printf '-d | --dry-run   - show what would be done\n'
	printf '-f | --finish    - remove the rollback data\n'
	printf '-m | --merge     - perform the usr merge\n'
	printf '-r | --roll-back - attempt to undo the usr merge\n'
	return 0
}

action_merge() {
	if [ -L /bin ] || [ -L /sbin ]; then
		printf 'The /usr merge has been completed on this system\n'
		return 0
	fi

	local dir
	for dir in /lib*; do
		[ -L "${dir}" ] || continue
		printf '%s is a symbolic link.\n' "${dir}"
		printf 'This means you have not migrated to the 17.1 profiles yet\n'
		printf 'please do so before running this script\n'
		return 1
	done

	setup_rollback || return

	# copy root directories to /usr counterparts and create
	# the /usr merge compatibility symlinks
	for dir in /bin /lib* /sbin; do
		run_command cp -a -i "${dir}"/* /usr/"${dir}"
		run_command rm -rf "${dir}"
		run_command ln -snf usr/"${dir}" "${dir}"
	done

	# merge /usr/sbin into /usr/bin
	run_cmd cp -a -i /usr/sbin/* /usr/bin
	run_cmd rm -fr /usr/sbin
	run_cmd ln -snf bin /usr/sbin
	return 0
}

action_rollback() {
	if [ ! -d "${rollback_path}" ]; then
		printf '%s does not exist, unable to roll back\n' "${rollback_path}"
		return 1
	fi
	local dir rollback_dir
	for dir in /bin /lib* /sbin /usr/bin /usr/lib* /usr/sbin; do
		[ "${dir}" = /usr/libexec ] && continue
		rollback_dir="${rollback_path}/${dir}"
		[ -d "${rollback_dir}" ] && continue
		printf 'Unable to perform rollback, %s is missing\n' "${rollback_dir}"
		return 1
	done
	for dir in /bin /lib* /sbin ; do
		rollback_dir="${rollback_path}/${dir}"
		run_cmd  rm -fr "${dir}"
		run_cmd cp -a "${rollback_dir}" /
	done
	for dir in /usr/bin /usr/lib* /usr/sbin; do
		[ "${dir}" = /usr/libexec ] && continue
		rollback_dir="${rollback_path}/${dir}"
		run_cmd  rm -f "${dir}"
		run_cmd cp -a "${rollback_dir}" /usr
	done
	return 0
}

main() {
	local dryrun finish merge rollback
	while [ $# -gt 0 ]; do
		case $1 in
			-d|--dry-run)
				dryrun=1
				;;
			-h|--help)
				action_help
				return 0
				;;
			-f|--finish)
				finish=1
				;;
			-m|--merge)
				merge=1
				;;
			-r|--roll-back)
				rollback=1
				;;
		esac
		shift
	done
	check_internal_commands || return
	check_root || return
	if [ "${finish}" = "${merge}" ] && [ "${merge}" = "${rollback}" ]; then
		printf 'You must select -f, -m or -r\n'
		action_help
		return 1
	elif [ -n "${finish}" ]; then
		action_finish || return
	elif [ -n "${merge}" ]; then
		action_merge || return
	elif [ -n "${rollback}" ]; then
		action_rollback || return
	fi
	return 0
}

main "$@" || exit
exit 0

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: usrmerge script

[Matthias Maier]

Hi William,

I have migrated my system to a merged /usr a while ago.

In addition to moving everything to /usr and setting up symlinks, the
main thing I had to do was to set up a /etc/portage/bashrc hook for
post_src_install() that would move everything into /usr.

Do we have native support in portage for this now?

Best,
Matthias


On Sun, Mar 21, 2021, at 12:39 CDT, William Hubbs <williamh@gentoo.org> wrote:

> All,
>
> the following is a script which will migrate a Gentoo system to the usr
> merge layout. This is similar to the unsymlink-lib tool used to migrate
> a system  from the 17.0 to the 17.1 profiles.
>
> I'm attaching it here to get some comments before I package it, so
> please let me know if I have missed something.
>
> Thanks,
>
> William

Re: [gentoo-dev] rfc: usrmerge script

[Luigi Mantellini]

there are some typos at lines 93, 94 and 95 (run_cmd instead run_command).

ciao

luigi

On Sun, Mar 21, 2021 at 7:00 PM Matthias Maier <tamiko@gentoo.org> wrote:
>
> Hi William,
>
> I have migrated my system to a merged /usr a while ago.
>
> In addition to moving everything to /usr and setting up symlinks, the
> main thing I had to do was to set up a /etc/portage/bashrc hook for
> post_src_install() that would move everything into /usr.
>
> Do we have native support in portage for this now?
>
> Best,
> Matthias
>
>
> On Sun, Mar 21, 2021, at 12:39 CDT, William Hubbs <williamh@gentoo.org> wrote:
>
> > All,
> >
> > the following is a script which will migrate a Gentoo system to the usr
> > merge layout. This is similar to the unsymlink-lib tool used to migrate
> > a system  from the 17.0 to the 17.1 profiles.
> >
> > I'm attaching it here to get some comments before I package it, so
> > please let me know if I have missed something.
> >
> > Thanks,
> >
> > William
>


-- 
Luigi 'Comio' Mantellini
R&D - Software
Industrie Dial Face S.p.A.
Via Canzo, 4
20068 Peschiera Borromeo (MI), Italy

Tel.: +39 02 5167 2813
Fax: +39 02 5167 2459
web: www.idf-hit.com
mail: luigi.mantellini@idf-hit.com

Re: [gentoo-dev] rfc: usrmerge script

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

On Sun, Mar 21, 2021 at 01:00:01PM -0500, Matthias Maier wrote:
> Hi William,
> 
> I have migrated my system to a merged /usr a while ago.
> 
> In addition to moving everything to /usr and setting up symlinks, the
> main thing I had to do was to set up a /etc/portage/bashrc hook for
> post_src_install() that would move everything into /usr.
> 
> Do we have native support in portage for this now?

From what I know, you don't need that hook. The ebuilds handle this now
by using the split-usr use flag, so if you add this to
/etc/portage/make.conf you should be ok.

USE="${USE} -split-usr"

Let me know if that isn't the case.

I'm not exactly sure what portage will do without your hook though. What
happens if you remove your hook then some ebuild tries to install
something in /bin for example? I would argue that, if /bin resolves to a
path, portage should just follow the symlink and install where the
symlink points.  I base this argument on the return of `test -d` for
this situation.

```
$ ln -s /usr/bin foo
$ [ -d foo ] && echo "foo is a path"
```

Thanks,

William


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: usrmerge script

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

On Sun, Mar 21, 2021 at 08:08:00PM +0100, Luigi Mantellini wrote:
> there are some typos at lines 93, 94 and 95 (run_cmd instead run_command).
> 
> ciao
> 
> luigi
 
 Hi Luigi,

Thanks for catching these, all instances of run_cmd are now changed to
run_command.

Thanks,

William

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: usrmerge script

[Michał Górny]

On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote:
> All,
> 
> the following is a script which will migrate a Gentoo system to the usr
> merge layout. This is similar to the unsymlink-lib tool used to migrate
> a system  from the 17.0 to the 17.1 profiles.
> 
> I'm attaching it here to get some comments before I package it, so
> please let me know if I have missed something.

To be honest, I don't think critical system modifications should be done
in shell script, and especially not via a fringe non-standards complaint
shell implementation in busybox.  Even if you can assume you make no
mistakes, shell is unreliable by design.  For example, your script may
start behaving in unexpected ways if you run out of space.

You don't seem to be handling file collisions at all.  Even today we
have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared
libraries.  Silently ignoring the problem or requiring the users to
manually ensure their system is clean is not going to solve it.

You don't seem to provide any helpful messages.  When things fail, user
will be left in the blue with an error message from some system tool (or
rather, cheap-ass busybox rewrite, I guess).

Also, have you verified that busybox's cp(1) actually preserves all file
properties (including xattrs, ACLs, caps...)?

Please don't forget to include tests with it.  Docker's good for testing
stuff like this.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] rfc: usrmerge script

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 4274 bytes --]

On Tue, Mar 23, 2021 at 10:23:11AM +0100, Michał Górny wrote:
> On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote:
> > All,
> > 
> > the following is a script which will migrate a Gentoo system to the usr
> > merge layout. This is similar to the unsymlink-lib tool used to migrate
> > a system  from the 17.0 to the 17.1 profiles.
> > 
> > I'm attaching it here to get some comments before I package it, so
> > please let me know if I have missed something.
> 
> To be honest, I don't think critical system modifications should be done
> in shell script, and especially not via a fringe non-standards complaint
> shell implementation in busybox.  Even if you can assume you make no
> mistakes, shell is unreliable by design.  For example, your script may
> start behaving in unexpected ways if you run out of space.

I went with busybox shell in this case because busybox is a static
binary and I figured with everything being moved around on the system it
would be a good choice.

I could pretty easily go with straight sh/bash, I just was focusing on
bb since the commands are built in.

You are right about shell behaving in weird ways if you run out of
space, but that's true for anything that happens to be running when a
system runs out of space.

That is why I put the rollback directory in /var by
default figuring there is more space there than in /, especially on
systems with separate /var.

The run_command wrapper in the script causes an immediate exit when the
command it runs fails so things don't go any
further than the failing command, so I'm not sure what else I can do
with this situation. One option is to always echo the command we are
about to run then just not run it if -d is specified. That means you
would always see the commands the script is running.

> You don't seem to be handling file collisions at all.  Even today we
> have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared
> libraries.  Silently ignoring the problem or requiring the users to
> manually ensure their system is clean is not going to solve it.
 
I have added -i to the cp commands in my latest version of this so I'll
see when we have duplicate names, and yes that is an issue, even without
the /usr merge.
I'm curious why we are duplicating all of these names in the first
place honestly. 

One example of this is in coreutils, and we even say in the ebuild
comment that we need to figure out why we are doing it.

There are a couple of ways forward for this.

1) attempt to open bugs on the packages and remove the duplicate names
by dropping symlinks and putting the files in their cannonical
locations.  this could lead  to breakages if one of the alternate names is
expected by something until the /usr merge is done.

2) try to guess at resolving the duplicate names as part of the
   usr merge process.

a. symmlinks in /bin or /sbin that point to the same name in /usr/bin or
/usr/sbin should be removed.
b. symlinks in /usr/bin or /usr/sbin that point to the same name in /
should be removed.
c. Other symlinks that I can think of should be preserved.

Which path for handling this is best? Do you have any thoughts?

> You don't seem to provide any helpful messages.  When things fail, user
> will be left in the blue with an error message from some system tool (or
> rather, cheap-ass busybox rewrite, I guess).
 
If the rollback directory is populated, you can use -r to recover,
and the script will not let you perform the /usr merge if the rollback
directory is not populated.

> Also, have you verified that busybox's cp(1) actually preserves all file
> properties (including xattrs, ACLs, caps...)?
 
The documentation for busybox states that -p preserves file attributes
if possible,  and by doing ls -l in the chroot I can see that
ownership/permissions are preserved. So, logically I would guess it
preserves the others.

If switching back to non-busybox is fine for this operation, I have no
problem doing that either.

> Please don't forget to include tests with it.  Docker's good for testing
> stuff like this.

Docker can't be used during src_test that I'm aware of, so I'm not sure
how Docker could be used to test this.

William

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: usrmerge script

[Michał Górny]

On Wed, 2021-03-24 at 00:20 -0500, William Hubbs wrote:
> On Tue, Mar 23, 2021 at 10:23:11AM +0100, Michał Górny wrote:
> > On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote:
> > > All,
> > > 
> > > the following is a script which will migrate a Gentoo system to the usr
> > > merge layout. This is similar to the unsymlink-lib tool used to migrate
> > > a system  from the 17.0 to the 17.1 profiles.
> > > 
> > > I'm attaching it here to get some comments before I package it, so
> > > please let me know if I have missed something.
> > 
> > To be honest, I don't think critical system modifications should be done
> > in shell script, and especially not via a fringe non-standards complaint
> > shell implementation in busybox.  Even if you can assume you make no
> > mistakes, shell is unreliable by design.  For example, your script may
> > start behaving in unexpected ways if you run out of space.
> 
> I went with busybox shell in this case because busybox is a static
> binary and I figured with everything being moved around on the system it
> would be a good choice.
> 
> I could pretty easily go with straight sh/bash, I just was focusing on
> bb since the commands are built in.
> 
> You are right about shell behaving in weird ways if you run out of
> space, but that's true for anything that happens to be running when a
> system runs out of space.

This is not true.  Most of the programming languages can behave sanely
in out-of-space conditions.  Sure, I/O fails one way or another but if
it's just a matter of proper error handling.  Shells on the other hand,
tend to use temporary files under the hood and can misbehave
in unexpected ways.

> That is why I put the rollback directory in /var by
> default figuring there is more space there than in /, especially on
> systems with separate /var.

What really can help is reflinking on filesystems supporting that.

> The run_command wrapper in the script causes an immediate exit when the
> command it runs fails so things don't go any
> further than the failing command, so I'm not sure what else I can do
> with this situation. One option is to always echo the command we are
> about to run then just not run it if -d is specified. That means you
> would always see the commands the script is running.

Did you see the verbose explanatory messages unsymlink-lib produces
on every error condition?  Compared to this, your script is printing
Windows-level 'error figure-it-out-yourself'.

> > You don't seem to be handling file collisions at all.  Even today we
> > have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared
> > libraries.  Silently ignoring the problem or requiring the users to
> > manually ensure their system is clean is not going to solve it.

> I have added -i to the cp commands in my latest version of this so I'll
> see when we have duplicate names, and yes that is an issue, even without
> the /usr merge.

I don't think that's the best UI for this possible.  I doubt that
a random user running the script will figure out why it is asking about
file replacements.

> I'm curious why we are duplicating all of these names in the first
> place honestly. 
> 
> One example of this is in coreutils, and we even say in the ebuild
> comment that we need to figure out why we are doing it.
> 
> There are a couple of ways forward for this.
> 
> 1) attempt to open bugs on the packages and remove the duplicate names
> by dropping symlinks and putting the files in their cannonical
> locations.  this could lead  to breakages if one of the alternate names is
> expected by something until the /usr merge is done.
> 
> 2) try to guess at resolving the duplicate names as part of the
>    usr merge process.
> 
> a. symmlinks in /bin or /sbin that point to the same name in /usr/bin or
> /usr/sbin should be removed.
> b. symlinks in /usr/bin or /usr/sbin that point to the same name in /
> should be removed.
> c. Other symlinks that I can think of should be preserved.
> 
> Which path for handling this is best? Do you have any thoughts?

This is a thing that should have been researched before writing
a script.

One case I'm aware of is using symlinks to replace tools with
alternatives, e.g. /usr/bin/bzip2 -> lbzip2 (/bin/bzip2 remains
as the original).

I don't know what's the best way forward.

> 
> > You don't seem to provide any helpful messages.  When things fail, user
> > will be left in the blue with an error message from some system tool (or
> > rather, cheap-ass busybox rewrite, I guess).
> 
> If the rollback directory is populated, you can use -r to recover,
> and the script will not let you perform the /usr merge if the rollback
> directory is not populated.
> 
> > Also, have you verified that busybox's cp(1) actually preserves all file
> > properties (including xattrs, ACLs, caps...)?
> 
> The documentation for busybox states that -p preserves file attributes
> if possible,  and by doing ls -l in the chroot I can see that
> ownership/permissions are preserved. So, logically I would guess it
> preserves the others.

I don't see how you would come to this conclusion.  Just because some
tool is doing the absolute minimum it doesn't mean it does anything
else, and especially operations that require additional syscalls.

> If switching back to non-busybox is fine for this operation, I have no
> problem doing that either.
> 
> > Please don't forget to include tests with it.  Docker's good for testing
> > stuff like this.
> 
> Docker can't be used during src_test that I'm aware of, so I'm not sure
> how Docker could be used to test this.

You can test it outside of ebuild.  Some automated testing is better
than no testing at all.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] rfc: usrmerge script

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 7686 bytes --]

On Wed, Mar 24, 2021 at 08:48:41AM +0100, Michał Górny wrote:
> On Wed, 2021-03-24 at 00:20 -0500, William Hubbs wrote:
> > On Tue, Mar 23, 2021 at 10:23:11AM +0100, Michał Górny wrote:
> > > On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote:
> > > > All,
> > > > 
> > > > the following is a script which will migrate a Gentoo system to the usr
> > > > merge layout. This is similar to the unsymlink-lib tool used to migrate
> > > > a system  from the 17.0 to the 17.1 profiles.
> > > > 
> > > > I'm attaching it here to get some comments before I package it, so
> > > > please let me know if I have missed something.
> > > 
> > > To be honest, I don't think critical system modifications should be done
> > > in shell script, and especially not via a fringe non-standards complaint
> > > shell implementation in busybox.  Even if you can assume you make no
> > > mistakes, shell is unreliable by design.  For example, your script may
> > > start behaving in unexpected ways if you run out of space.
> > 
> > I went with busybox shell in this case because busybox is a static
> > binary and I figured with everything being moved around on the system it
> > would be a good choice.
> > 
> > I could pretty easily go with straight sh/bash, I just was focusing on
> > bb since the commands are built in.
> > 
> > You are right about shell behaving in weird ways if you run out of
> > space, but that's true for anything that happens to be running when a
> > system runs out of space.
> 
> This is not true.  Most of the programming languages can behave sanely
> in out-of-space conditions.  Sure, I/O fails one way or another but if
> it's just a matter of proper error handling.  Shells on the other hand,
> tend to use temporary files under the hood and can misbehave
> in unexpected ways.
 
Another reason I went with bb is there are no deps outside of bb and the
script. Also bb is on every gentoo system.
In short, I went for something that I knew would be everywhere.
If I'm going to do the work, it isn't going to be in python because I'm
not a fan. I can write some in python, but I don't consider my skills
there to be the best, and the indentation requirements make it tedious
for me to debug.

> > That is why I put the rollback directory in /var by
> > default figuring there is more space there than in /, especially on
> > systems with separate /var.
> 
> What really can help is reflinking on filesystems supporting that.
 
What really can help is more info instead of being terse like this.
Which filesystems support it?

> > The run_command wrapper in the script causes an immediate exit when the
> > command it runs fails so things don't go any
> > further than the failing command, so I'm not sure what else I can do
> > with this situation. One option is to always echo the command we are
> > about to run then just not run it if -d is specified. That means you
> > would always see the commands the script is running.
> 
> Did you see the verbose explanatory messages unsymlink-lib produces
> on every error condition?  Compared to this, your script is printing
> Windows-level 'error figure-it-out-yourself'.
 
This is early in development. Ideally I would handle most of the error
conditions before people start using this on production systems; that is
why I put it out here.

> > > You don't seem to be handling file collisions at all.  Even today we
> > > have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared
> > > libraries.  Silently ignoring the problem or requiring the users to
> > > manually ensure their system is clean is not going to solve it.
> 
> > I have added -i to the cp commands in my latest version of this so I'll
> > see when we have duplicate names, and yes that is an issue, even without
> > the /usr merge.
> 
> I don't think that's the best UI for this possible.  I doubt that
> a random user running the script will figure out why it is asking about
> file replacements.
 
 You are probably right, that's why I put this out here, and
 listed my thoughts about it below to generate discussion.

> > I'm curious why we are duplicating all of these names in the first
> > place honestly. 
> > 
> > One example of this is in coreutils, and we even say in the ebuild
> > comment that we need to figure out why we are doing it.
> > 
> > There are a couple of ways forward for this.
> > 
> > 1) attempt to open bugs on the packages and remove the duplicate names
> > by dropping symlinks and putting the files in their cannonical
> > locations.  this could lead  to breakages if one of the alternate names is
> > expected by something until the /usr merge is done.
> > 
> > 2) try to guess at resolving the duplicate names as part of the
> >    usr merge process.
> > 
> > a. symmlinks in /bin or /sbin that point to the same name in /usr/bin or
> > /usr/sbin should be removed.
> > b. symlinks in /usr/bin or /usr/sbin that point to the same name in /
> > should be removed.
> > c. Other symlinks that I can think of should be preserved.
> > 
> > Which path for handling this is best? Do you have any thoughts?
> 
> This is a thing that should have been researched before writing
> a script.
 
With all respect, knock it off. I'm not going to sit and read through
all of the ebuilds in the tree looking for every little thing.

> One case I'm aware of is using symlinks to replace tools with
> alternatives, e.g. /usr/bin/bzip2 -> lbzip2 (/bin/bzip2 remains
> as the original).

Your example doesn't exist here, so I wouldn't have known about it if
you hadn't mentioned it.

It looks like that specific situation is controlled by the
symlink use flag in lbzip, so it should be ok.

> I don't know what's the best way forward.
> 
> > 
> > > You don't seem to provide any helpful messages.  When things fail, user
> > > will be left in the blue with an error message from some system tool (or
> > > rather, cheap-ass busybox rewrite, I guess).
> > 
> > If the rollback directory is populated, you can use -r to recover,
> > and the script will not let you perform the /usr merge if the rollback
> > directory is not populated.
> > 
> > > Also, have you verified that busybox's cp(1) actually preserves all file
> > > properties (including xattrs, ACLs, caps...)?
> > 
> > The documentation for busybox states that -p preserves file attributes
> > if possible,  and by doing ls -l in the chroot I can see that
> > ownership/permissions are preserved. So, logically I would guess it
> > preserves the others.
> 
> I don't see how you would come to this conclusion.  Just because some
> tool is doing the absolute minimum it doesn't mean it does anything
> else, and especially operations that require additional syscalls.

Hmm, I think a better response here would have been to point out how I
could look at the other attributes of a file you are talking about.

> > If switching back to non-busybox is fine for this operation, I have no
> > problem doing that either.
> > 
> > > Please don't forget to include tests with it.  Docker's good for testing
> > > stuff like this.
> > 
> > Docker can't be used during src_test that I'm aware of, so I'm not sure
> > how Docker could be used to test this.
> 
> You can test it outside of ebuild.  Some automated testing is better
> than no testing at all.

With all respect, I think most of this email was sarcasm, trolling etc
instead of the type of constructive feedback we look for when sharing
code on the list. I will do what I can to use parts of it, but I really
have issues with your tone.

William


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: usrmerge script

[Rich Freeman]

On Wed, Mar 24, 2021 at 11:09 AM William Hubbs <williamh@gentoo.org> wrote:
>
> On Wed, Mar 24, 2021 at 08:48:41AM +0100, Michał Górny wrote:
> >
> > What really can help is reflinking on filesystems supporting that.
>
> What really can help is more info instead of being terse like this.
> Which filesystems support it?
>

According to Google right now: Btrfs, CIFS, NFS 4.2, OCFS2, overlayfs, and XFS

Lizardfs ought to, but doesn't currently.  zfs does not because clones
only are supported at the dataset level.

In any case, if you're using coreutils cp to do the copy, just pass
--reflink=auto.  Honestly, I have no idea why this isn't the default
behavior.  Who wouldn't want instant copy operations that consume zero
space (aside from metdata)?  If you're doing this in C or some other
language you would need to see if they have a library call to do it
easily - see man ioctl_ficlone.

-- 
Rich

Re: [gentoo-dev] rfc: usrmerge script

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1298 bytes --]

On Wed, Mar 24, 2021 at 01:09:52PM -0400, Rich Freeman wrote:
> On Wed, Mar 24, 2021 at 11:09 AM William Hubbs <williamh@gentoo.org> wrote:
> >
> > On Wed, Mar 24, 2021 at 08:48:41AM +0100, Michał Górny wrote:
> > >
> > > What really can help is reflinking on filesystems supporting that.
> >
> > What really can help is more info instead of being terse like this.
> > Which filesystems support it?
> >
> 
> According to Google right now: Btrfs, CIFS, NFS 4.2, OCFS2, overlayfs, and XFS
> 
> Lizardfs ought to, but doesn't currently.  zfs does not because clones
> only are supported at the dataset level.
> 
> In any case, if you're using coreutils cp to do the copy, just pass
> --reflink=auto.  Honestly, I have no idea why this isn't the default
> behavior.  Who wouldn't want instant copy operations that consume zero
> space (aside from metdata)?  If you're doing this in C or some other
> language you would need to see if they have a library call to do it
> easily - see man ioctl_ficlone.

I'm using busybox, and I just checked and it also supports the
--reflink=auto switch.

I thought about coreutils, but with everything on the fs being moved
around, I think that would get messy.

Thanks a lot for the info Rich. :-)

William

> 
> -- 
> Rich
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]