* [gentoo-dev] rfc: usrmerge script @ 2021-03-21 17:39 William Hubbs 2021-03-21 18:00 ` Matthias Maier 2021-03-23 9:23 ` Michał Górny 0 siblings, 2 replies; 11+ messages in thread From: William Hubbs @ 2021-03-21 17:39 UTC (permalink / raw To: gentoo development; +Cc: sam, soap [-- Attachment #1.1: Type: text/plain, Size: 330 bytes --] All, the following is a script which will migrate a Gentoo system to the usr merge layout. This is similar to the unsymlink-lib tool used to migrate a system from the 17.0 to the 17.1 profiles. I'm attaching it here to get some comments before I package it, so please let me know if I have missed something. Thanks, William [-- Attachment #1.2: usrmerge --] [-- Type: text/plain, Size: 3885 bytes --] #!/bin/bb # shellcheck disable=SC2039 shell=sh rollback_path=/var/usrmerge-rollback check_internal_commands() { for cmd in cp ln; do [ "$(command -v "${cmd}")" = ${cmd} ] && continue printf 'busybox does not include the %s command internally\n' "${cmd}" return 1 done return 0 } check_root() { [ "$(id -u)" = 0 ] && return 0 printf 'This script must be run by root\n' return 1 } run_command() { local cmd [ -n "${dryrun}" ] && cmd="echo" ${cmd} "$@" || exit return 0 } setup_rollback() { if [ -e "${rollback_path}" ]; then printf '%s already exists\n' "${rollback_path}" printf 'please remove it before continuing\n' return 1 fi printf 'Creating %s to allow rollbacks\n' "${rollback_path}" run_command mkdir "${rollback_path}" run_command mkdir "${rollback_path}/usr" local dir for dir in /bin /lib* /sbin; do run_command cp -a "${dir}" "${rollback_path}" done for dir in /usr/bin /usr/lib* /usr/sbin; do [ "${dir}" = /usr/libexec ] && continue run_command cp -a "${dir}" "${rollback_path}/usr" done return 0 } action_finish() { run_command rm -fr "${rollback_path}" return 0 } action_help() { local cmd cmd="$(basename "${0}")" printf 'usage: %s -h\n' "${cmd}" printf ' %s [-d] -f|-m|-r\n' "${cmd}" printf '\n' printf '-h | --help - displays this message\n' printf '-d | --dry-run - show what would be done\n' printf '-f | --finish - remove the rollback data\n' printf '-m | --merge - perform the usr merge\n' printf '-r | --roll-back - attempt to undo the usr merge\n' return 0 } action_merge() { if [ -L /bin ] || [ -L /sbin ]; then printf 'The /usr merge has been completed on this system\n' return 0 fi local dir for dir in /lib*; do [ -L "${dir}" ] || continue printf '%s is a symbolic link.\n' "${dir}" printf 'This means you have not migrated to the 17.1 profiles yet\n' printf 'please do so before running this script\n' return 1 done setup_rollback || return # copy root directories to /usr counterparts and create # the /usr merge compatibility symlinks for dir in /bin /lib* /sbin; do run_command cp -a -i "${dir}"/* /usr/"${dir}" run_command rm -rf "${dir}" run_command ln -snf usr/"${dir}" "${dir}" done # merge /usr/sbin into /usr/bin run_cmd cp -a -i /usr/sbin/* /usr/bin run_cmd rm -fr /usr/sbin run_cmd ln -snf bin /usr/sbin return 0 } action_rollback() { if [ ! -d "${rollback_path}" ]; then printf '%s does not exist, unable to roll back\n' "${rollback_path}" return 1 fi local dir rollback_dir for dir in /bin /lib* /sbin /usr/bin /usr/lib* /usr/sbin; do [ "${dir}" = /usr/libexec ] && continue rollback_dir="${rollback_path}/${dir}" [ -d "${rollback_dir}" ] && continue printf 'Unable to perform rollback, %s is missing\n' "${rollback_dir}" return 1 done for dir in /bin /lib* /sbin ; do rollback_dir="${rollback_path}/${dir}" run_cmd rm -fr "${dir}" run_cmd cp -a "${rollback_dir}" / done for dir in /usr/bin /usr/lib* /usr/sbin; do [ "${dir}" = /usr/libexec ] && continue rollback_dir="${rollback_path}/${dir}" run_cmd rm -f "${dir}" run_cmd cp -a "${rollback_dir}" /usr done return 0 } main() { local dryrun finish merge rollback while [ $# -gt 0 ]; do case $1 in -d|--dry-run) dryrun=1 ;; -h|--help) action_help return 0 ;; -f|--finish) finish=1 ;; -m|--merge) merge=1 ;; -r|--roll-back) rollback=1 ;; esac shift done check_internal_commands || return check_root || return if [ "${finish}" = "${merge}" ] && [ "${merge}" = "${rollback}" ]; then printf 'You must select -f, -m or -r\n' action_help return 1 elif [ -n "${finish}" ]; then action_finish || return elif [ -n "${merge}" ]; then action_merge || return elif [ -n "${rollback}" ]; then action_rollback || return fi return 0 } main "$@" || exit exit 0 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-21 17:39 [gentoo-dev] rfc: usrmerge script William Hubbs @ 2021-03-21 18:00 ` Matthias Maier 2021-03-21 19:08 ` Luigi Mantellini 2021-03-21 19:22 ` William Hubbs 2021-03-23 9:23 ` Michał Górny 1 sibling, 2 replies; 11+ messages in thread From: Matthias Maier @ 2021-03-21 18:00 UTC (permalink / raw To: gentoo-dev Hi William, I have migrated my system to a merged /usr a while ago. In addition to moving everything to /usr and setting up symlinks, the main thing I had to do was to set up a /etc/portage/bashrc hook for post_src_install() that would move everything into /usr. Do we have native support in portage for this now? Best, Matthias On Sun, Mar 21, 2021, at 12:39 CDT, William Hubbs <williamh@gentoo.org> wrote: > All, > > the following is a script which will migrate a Gentoo system to the usr > merge layout. This is similar to the unsymlink-lib tool used to migrate > a system from the 17.0 to the 17.1 profiles. > > I'm attaching it here to get some comments before I package it, so > please let me know if I have missed something. > > Thanks, > > William ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-21 18:00 ` Matthias Maier @ 2021-03-21 19:08 ` Luigi Mantellini 2021-03-21 19:27 ` William Hubbs 2021-03-21 19:22 ` William Hubbs 1 sibling, 1 reply; 11+ messages in thread From: Luigi Mantellini @ 2021-03-21 19:08 UTC (permalink / raw To: gentoo-dev there are some typos at lines 93, 94 and 95 (run_cmd instead run_command). ciao luigi On Sun, Mar 21, 2021 at 7:00 PM Matthias Maier <tamiko@gentoo.org> wrote: > > Hi William, > > I have migrated my system to a merged /usr a while ago. > > In addition to moving everything to /usr and setting up symlinks, the > main thing I had to do was to set up a /etc/portage/bashrc hook for > post_src_install() that would move everything into /usr. > > Do we have native support in portage for this now? > > Best, > Matthias > > > On Sun, Mar 21, 2021, at 12:39 CDT, William Hubbs <williamh@gentoo.org> wrote: > > > All, > > > > the following is a script which will migrate a Gentoo system to the usr > > merge layout. This is similar to the unsymlink-lib tool used to migrate > > a system from the 17.0 to the 17.1 profiles. > > > > I'm attaching it here to get some comments before I package it, so > > please let me know if I have missed something. > > > > Thanks, > > > > William > -- Luigi 'Comio' Mantellini R&D - Software Industrie Dial Face S.p.A. Via Canzo, 4 20068 Peschiera Borromeo (MI), Italy Tel.: +39 02 5167 2813 Fax: +39 02 5167 2459 web: www.idf-hit.com mail: luigi.mantellini@idf-hit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-21 19:08 ` Luigi Mantellini @ 2021-03-21 19:27 ` William Hubbs 0 siblings, 0 replies; 11+ messages in thread From: William Hubbs @ 2021-03-21 19:27 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 295 bytes --] On Sun, Mar 21, 2021 at 08:08:00PM +0100, Luigi Mantellini wrote: > there are some typos at lines 93, 94 and 95 (run_cmd instead run_command). > > ciao > > luigi Hi Luigi, Thanks for catching these, all instances of run_cmd are now changed to run_command. Thanks, William [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-21 18:00 ` Matthias Maier 2021-03-21 19:08 ` Luigi Mantellini @ 2021-03-21 19:22 ` William Hubbs 1 sibling, 0 replies; 11+ messages in thread From: William Hubbs @ 2021-03-21 19:22 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1116 bytes --] On Sun, Mar 21, 2021 at 01:00:01PM -0500, Matthias Maier wrote: > Hi William, > > I have migrated my system to a merged /usr a while ago. > > In addition to moving everything to /usr and setting up symlinks, the > main thing I had to do was to set up a /etc/portage/bashrc hook for > post_src_install() that would move everything into /usr. > > Do we have native support in portage for this now? From what I know, you don't need that hook. The ebuilds handle this now by using the split-usr use flag, so if you add this to /etc/portage/make.conf you should be ok. USE="${USE} -split-usr" Let me know if that isn't the case. I'm not exactly sure what portage will do without your hook though. What happens if you remove your hook then some ebuild tries to install something in /bin for example? I would argue that, if /bin resolves to a path, portage should just follow the symlink and install where the symlink points. I base this argument on the return of `test -d` for this situation. ``` $ ln -s /usr/bin foo $ [ -d foo ] && echo "foo is a path" ``` Thanks, William [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-21 17:39 [gentoo-dev] rfc: usrmerge script William Hubbs 2021-03-21 18:00 ` Matthias Maier @ 2021-03-23 9:23 ` Michał Górny 2021-03-24 5:20 ` William Hubbs 1 sibling, 1 reply; 11+ messages in thread From: Michał Górny @ 2021-03-23 9:23 UTC (permalink / raw To: gentoo-dev; +Cc: sam, soap On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote: > All, > > the following is a script which will migrate a Gentoo system to the usr > merge layout. This is similar to the unsymlink-lib tool used to migrate > a system from the 17.0 to the 17.1 profiles. > > I'm attaching it here to get some comments before I package it, so > please let me know if I have missed something. To be honest, I don't think critical system modifications should be done in shell script, and especially not via a fringe non-standards complaint shell implementation in busybox. Even if you can assume you make no mistakes, shell is unreliable by design. For example, your script may start behaving in unexpected ways if you run out of space. You don't seem to be handling file collisions at all. Even today we have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared libraries. Silently ignoring the problem or requiring the users to manually ensure their system is clean is not going to solve it. You don't seem to provide any helpful messages. When things fail, user will be left in the blue with an error message from some system tool (or rather, cheap-ass busybox rewrite, I guess). Also, have you verified that busybox's cp(1) actually preserves all file properties (including xattrs, ACLs, caps...)? Please don't forget to include tests with it. Docker's good for testing stuff like this. -- Best regards, Michał Górny ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-23 9:23 ` Michał Górny @ 2021-03-24 5:20 ` William Hubbs 2021-03-24 7:48 ` Michał Górny 0 siblings, 1 reply; 11+ messages in thread From: William Hubbs @ 2021-03-24 5:20 UTC (permalink / raw To: gentoo-dev; +Cc: mgorny [-- Attachment #1: Type: text/plain, Size: 4274 bytes --] On Tue, Mar 23, 2021 at 10:23:11AM +0100, Michał Górny wrote: > On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote: > > All, > > > > the following is a script which will migrate a Gentoo system to the usr > > merge layout. This is similar to the unsymlink-lib tool used to migrate > > a system from the 17.0 to the 17.1 profiles. > > > > I'm attaching it here to get some comments before I package it, so > > please let me know if I have missed something. > > To be honest, I don't think critical system modifications should be done > in shell script, and especially not via a fringe non-standards complaint > shell implementation in busybox. Even if you can assume you make no > mistakes, shell is unreliable by design. For example, your script may > start behaving in unexpected ways if you run out of space. I went with busybox shell in this case because busybox is a static binary and I figured with everything being moved around on the system it would be a good choice. I could pretty easily go with straight sh/bash, I just was focusing on bb since the commands are built in. You are right about shell behaving in weird ways if you run out of space, but that's true for anything that happens to be running when a system runs out of space. That is why I put the rollback directory in /var by default figuring there is more space there than in /, especially on systems with separate /var. The run_command wrapper in the script causes an immediate exit when the command it runs fails so things don't go any further than the failing command, so I'm not sure what else I can do with this situation. One option is to always echo the command we are about to run then just not run it if -d is specified. That means you would always see the commands the script is running. > You don't seem to be handling file collisions at all. Even today we > have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared > libraries. Silently ignoring the problem or requiring the users to > manually ensure their system is clean is not going to solve it. I have added -i to the cp commands in my latest version of this so I'll see when we have duplicate names, and yes that is an issue, even without the /usr merge. I'm curious why we are duplicating all of these names in the first place honestly. One example of this is in coreutils, and we even say in the ebuild comment that we need to figure out why we are doing it. There are a couple of ways forward for this. 1) attempt to open bugs on the packages and remove the duplicate names by dropping symlinks and putting the files in their cannonical locations. this could lead to breakages if one of the alternate names is expected by something until the /usr merge is done. 2) try to guess at resolving the duplicate names as part of the usr merge process. a. symmlinks in /bin or /sbin that point to the same name in /usr/bin or /usr/sbin should be removed. b. symlinks in /usr/bin or /usr/sbin that point to the same name in / should be removed. c. Other symlinks that I can think of should be preserved. Which path for handling this is best? Do you have any thoughts? > You don't seem to provide any helpful messages. When things fail, user > will be left in the blue with an error message from some system tool (or > rather, cheap-ass busybox rewrite, I guess). If the rollback directory is populated, you can use -r to recover, and the script will not let you perform the /usr merge if the rollback directory is not populated. > Also, have you verified that busybox's cp(1) actually preserves all file > properties (including xattrs, ACLs, caps...)? The documentation for busybox states that -p preserves file attributes if possible, and by doing ls -l in the chroot I can see that ownership/permissions are preserved. So, logically I would guess it preserves the others. If switching back to non-busybox is fine for this operation, I have no problem doing that either. > Please don't forget to include tests with it. Docker's good for testing > stuff like this. Docker can't be used during src_test that I'm aware of, so I'm not sure how Docker could be used to test this. William [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-24 5:20 ` William Hubbs @ 2021-03-24 7:48 ` Michał Górny 2021-03-24 15:09 ` William Hubbs 0 siblings, 1 reply; 11+ messages in thread From: Michał Górny @ 2021-03-24 7:48 UTC (permalink / raw To: gentoo-dev On Wed, 2021-03-24 at 00:20 -0500, William Hubbs wrote: > On Tue, Mar 23, 2021 at 10:23:11AM +0100, Michał Górny wrote: > > On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote: > > > All, > > > > > > the following is a script which will migrate a Gentoo system to the usr > > > merge layout. This is similar to the unsymlink-lib tool used to migrate > > > a system from the 17.0 to the 17.1 profiles. > > > > > > I'm attaching it here to get some comments before I package it, so > > > please let me know if I have missed something. > > > > To be honest, I don't think critical system modifications should be done > > in shell script, and especially not via a fringe non-standards complaint > > shell implementation in busybox. Even if you can assume you make no > > mistakes, shell is unreliable by design. For example, your script may > > start behaving in unexpected ways if you run out of space. > > I went with busybox shell in this case because busybox is a static > binary and I figured with everything being moved around on the system it > would be a good choice. > > I could pretty easily go with straight sh/bash, I just was focusing on > bb since the commands are built in. > > You are right about shell behaving in weird ways if you run out of > space, but that's true for anything that happens to be running when a > system runs out of space. This is not true. Most of the programming languages can behave sanely in out-of-space conditions. Sure, I/O fails one way or another but if it's just a matter of proper error handling. Shells on the other hand, tend to use temporary files under the hood and can misbehave in unexpected ways. > That is why I put the rollback directory in /var by > default figuring there is more space there than in /, especially on > systems with separate /var. What really can help is reflinking on filesystems supporting that. > The run_command wrapper in the script causes an immediate exit when the > command it runs fails so things don't go any > further than the failing command, so I'm not sure what else I can do > with this situation. One option is to always echo the command we are > about to run then just not run it if -d is specified. That means you > would always see the commands the script is running. Did you see the verbose explanatory messages unsymlink-lib produces on every error condition? Compared to this, your script is printing Windows-level 'error figure-it-out-yourself'. > > You don't seem to be handling file collisions at all. Even today we > > have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared > > libraries. Silently ignoring the problem or requiring the users to > > manually ensure their system is clean is not going to solve it. > I have added -i to the cp commands in my latest version of this so I'll > see when we have duplicate names, and yes that is an issue, even without > the /usr merge. I don't think that's the best UI for this possible. I doubt that a random user running the script will figure out why it is asking about file replacements. > I'm curious why we are duplicating all of these names in the first > place honestly. > > One example of this is in coreutils, and we even say in the ebuild > comment that we need to figure out why we are doing it. > > There are a couple of ways forward for this. > > 1) attempt to open bugs on the packages and remove the duplicate names > by dropping symlinks and putting the files in their cannonical > locations. this could lead to breakages if one of the alternate names is > expected by something until the /usr merge is done. > > 2) try to guess at resolving the duplicate names as part of the > usr merge process. > > a. symmlinks in /bin or /sbin that point to the same name in /usr/bin or > /usr/sbin should be removed. > b. symlinks in /usr/bin or /usr/sbin that point to the same name in / > should be removed. > c. Other symlinks that I can think of should be preserved. > > Which path for handling this is best? Do you have any thoughts? This is a thing that should have been researched before writing a script. One case I'm aware of is using symlinks to replace tools with alternatives, e.g. /usr/bin/bzip2 -> lbzip2 (/bin/bzip2 remains as the original). I don't know what's the best way forward. > > > You don't seem to provide any helpful messages. When things fail, user > > will be left in the blue with an error message from some system tool (or > > rather, cheap-ass busybox rewrite, I guess). > > If the rollback directory is populated, you can use -r to recover, > and the script will not let you perform the /usr merge if the rollback > directory is not populated. > > > Also, have you verified that busybox's cp(1) actually preserves all file > > properties (including xattrs, ACLs, caps...)? > > The documentation for busybox states that -p preserves file attributes > if possible, and by doing ls -l in the chroot I can see that > ownership/permissions are preserved. So, logically I would guess it > preserves the others. I don't see how you would come to this conclusion. Just because some tool is doing the absolute minimum it doesn't mean it does anything else, and especially operations that require additional syscalls. > If switching back to non-busybox is fine for this operation, I have no > problem doing that either. > > > Please don't forget to include tests with it. Docker's good for testing > > stuff like this. > > Docker can't be used during src_test that I'm aware of, so I'm not sure > how Docker could be used to test this. You can test it outside of ebuild. Some automated testing is better than no testing at all. -- Best regards, Michał Górny ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-24 7:48 ` Michał Górny @ 2021-03-24 15:09 ` William Hubbs 2021-03-24 17:09 ` Rich Freeman 0 siblings, 1 reply; 11+ messages in thread From: William Hubbs @ 2021-03-24 15:09 UTC (permalink / raw To: gentoo-dev; +Cc: mgorny [-- Attachment #1: Type: text/plain, Size: 7686 bytes --] On Wed, Mar 24, 2021 at 08:48:41AM +0100, Michał Górny wrote: > On Wed, 2021-03-24 at 00:20 -0500, William Hubbs wrote: > > On Tue, Mar 23, 2021 at 10:23:11AM +0100, Michał Górny wrote: > > > On Sun, 2021-03-21 at 12:39 -0500, William Hubbs wrote: > > > > All, > > > > > > > > the following is a script which will migrate a Gentoo system to the usr > > > > merge layout. This is similar to the unsymlink-lib tool used to migrate > > > > a system from the 17.0 to the 17.1 profiles. > > > > > > > > I'm attaching it here to get some comments before I package it, so > > > > please let me know if I have missed something. > > > > > > To be honest, I don't think critical system modifications should be done > > > in shell script, and especially not via a fringe non-standards complaint > > > shell implementation in busybox. Even if you can assume you make no > > > mistakes, shell is unreliable by design. For example, your script may > > > start behaving in unexpected ways if you run out of space. > > > > I went with busybox shell in this case because busybox is a static > > binary and I figured with everything being moved around on the system it > > would be a good choice. > > > > I could pretty easily go with straight sh/bash, I just was focusing on > > bb since the commands are built in. > > > > You are right about shell behaving in weird ways if you run out of > > space, but that's true for anything that happens to be running when a > > system runs out of space. > > This is not true. Most of the programming languages can behave sanely > in out-of-space conditions. Sure, I/O fails one way or another but if > it's just a matter of proper error handling. Shells on the other hand, > tend to use temporary files under the hood and can misbehave > in unexpected ways. Another reason I went with bb is there are no deps outside of bb and the script. Also bb is on every gentoo system. In short, I went for something that I knew would be everywhere. If I'm going to do the work, it isn't going to be in python because I'm not a fan. I can write some in python, but I don't consider my skills there to be the best, and the indentation requirements make it tedious for me to debug. > > That is why I put the rollback directory in /var by > > default figuring there is more space there than in /, especially on > > systems with separate /var. > > What really can help is reflinking on filesystems supporting that. What really can help is more info instead of being terse like this. Which filesystems support it? > > The run_command wrapper in the script causes an immediate exit when the > > command it runs fails so things don't go any > > further than the failing command, so I'm not sure what else I can do > > with this situation. One option is to always echo the command we are > > about to run then just not run it if -d is specified. That means you > > would always see the commands the script is running. > > Did you see the verbose explanatory messages unsymlink-lib produces > on every error condition? Compared to this, your script is printing > Windows-level 'error figure-it-out-yourself'. This is early in development. Ideally I would handle most of the error conditions before people start using this on production systems; that is why I put it out here. > > > You don't seem to be handling file collisions at all. Even today we > > > have files like /bin/bzip2 and /usr/bin/bzip2, not to mention shared > > > libraries. Silently ignoring the problem or requiring the users to > > > manually ensure their system is clean is not going to solve it. > > > I have added -i to the cp commands in my latest version of this so I'll > > see when we have duplicate names, and yes that is an issue, even without > > the /usr merge. > > I don't think that's the best UI for this possible. I doubt that > a random user running the script will figure out why it is asking about > file replacements. You are probably right, that's why I put this out here, and listed my thoughts about it below to generate discussion. > > I'm curious why we are duplicating all of these names in the first > > place honestly. > > > > One example of this is in coreutils, and we even say in the ebuild > > comment that we need to figure out why we are doing it. > > > > There are a couple of ways forward for this. > > > > 1) attempt to open bugs on the packages and remove the duplicate names > > by dropping symlinks and putting the files in their cannonical > > locations. this could lead to breakages if one of the alternate names is > > expected by something until the /usr merge is done. > > > > 2) try to guess at resolving the duplicate names as part of the > > usr merge process. > > > > a. symmlinks in /bin or /sbin that point to the same name in /usr/bin or > > /usr/sbin should be removed. > > b. symlinks in /usr/bin or /usr/sbin that point to the same name in / > > should be removed. > > c. Other symlinks that I can think of should be preserved. > > > > Which path for handling this is best? Do you have any thoughts? > > This is a thing that should have been researched before writing > a script. With all respect, knock it off. I'm not going to sit and read through all of the ebuilds in the tree looking for every little thing. > One case I'm aware of is using symlinks to replace tools with > alternatives, e.g. /usr/bin/bzip2 -> lbzip2 (/bin/bzip2 remains > as the original). Your example doesn't exist here, so I wouldn't have known about it if you hadn't mentioned it. It looks like that specific situation is controlled by the symlink use flag in lbzip, so it should be ok. > I don't know what's the best way forward. > > > > > > You don't seem to provide any helpful messages. When things fail, user > > > will be left in the blue with an error message from some system tool (or > > > rather, cheap-ass busybox rewrite, I guess). > > > > If the rollback directory is populated, you can use -r to recover, > > and the script will not let you perform the /usr merge if the rollback > > directory is not populated. > > > > > Also, have you verified that busybox's cp(1) actually preserves all file > > > properties (including xattrs, ACLs, caps...)? > > > > The documentation for busybox states that -p preserves file attributes > > if possible, and by doing ls -l in the chroot I can see that > > ownership/permissions are preserved. So, logically I would guess it > > preserves the others. > > I don't see how you would come to this conclusion. Just because some > tool is doing the absolute minimum it doesn't mean it does anything > else, and especially operations that require additional syscalls. Hmm, I think a better response here would have been to point out how I could look at the other attributes of a file you are talking about. > > If switching back to non-busybox is fine for this operation, I have no > > problem doing that either. > > > > > Please don't forget to include tests with it. Docker's good for testing > > > stuff like this. > > > > Docker can't be used during src_test that I'm aware of, so I'm not sure > > how Docker could be used to test this. > > You can test it outside of ebuild. Some automated testing is better > than no testing at all. With all respect, I think most of this email was sarcasm, trolling etc instead of the type of constructive feedback we look for when sharing code on the list. I will do what I can to use parts of it, but I really have issues with your tone. William [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-24 15:09 ` William Hubbs @ 2021-03-24 17:09 ` Rich Freeman 2021-03-24 18:37 ` William Hubbs 0 siblings, 1 reply; 11+ messages in thread From: Rich Freeman @ 2021-03-24 17:09 UTC (permalink / raw To: gentoo-dev On Wed, Mar 24, 2021 at 11:09 AM William Hubbs <williamh@gentoo.org> wrote: > > On Wed, Mar 24, 2021 at 08:48:41AM +0100, Michał Górny wrote: > > > > What really can help is reflinking on filesystems supporting that. > > What really can help is more info instead of being terse like this. > Which filesystems support it? > According to Google right now: Btrfs, CIFS, NFS 4.2, OCFS2, overlayfs, and XFS Lizardfs ought to, but doesn't currently. zfs does not because clones only are supported at the dataset level. In any case, if you're using coreutils cp to do the copy, just pass --reflink=auto. Honestly, I have no idea why this isn't the default behavior. Who wouldn't want instant copy operations that consume zero space (aside from metdata)? If you're doing this in C or some other language you would need to see if they have a library call to do it easily - see man ioctl_ficlone. -- Rich ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] rfc: usrmerge script 2021-03-24 17:09 ` Rich Freeman @ 2021-03-24 18:37 ` William Hubbs 0 siblings, 0 replies; 11+ messages in thread From: William Hubbs @ 2021-03-24 18:37 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1298 bytes --] On Wed, Mar 24, 2021 at 01:09:52PM -0400, Rich Freeman wrote: > On Wed, Mar 24, 2021 at 11:09 AM William Hubbs <williamh@gentoo.org> wrote: > > > > On Wed, Mar 24, 2021 at 08:48:41AM +0100, Michał Górny wrote: > > > > > > What really can help is reflinking on filesystems supporting that. > > > > What really can help is more info instead of being terse like this. > > Which filesystems support it? > > > > According to Google right now: Btrfs, CIFS, NFS 4.2, OCFS2, overlayfs, and XFS > > Lizardfs ought to, but doesn't currently. zfs does not because clones > only are supported at the dataset level. > > In any case, if you're using coreutils cp to do the copy, just pass > --reflink=auto. Honestly, I have no idea why this isn't the default > behavior. Who wouldn't want instant copy operations that consume zero > space (aside from metdata)? If you're doing this in C or some other > language you would need to see if they have a library call to do it > easily - see man ioctl_ficlone. I'm using busybox, and I just checked and it also supports the --reflink=auto switch. I thought about coreutils, but with everything on the fs being moved around, I think that would get messy. Thanks a lot for the info Rich. :-) William > > -- > Rich > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-03-24 18:37 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-03-21 17:39 [gentoo-dev] rfc: usrmerge script William Hubbs 2021-03-21 18:00 ` Matthias Maier 2021-03-21 19:08 ` Luigi Mantellini 2021-03-21 19:27 ` William Hubbs 2021-03-21 19:22 ` William Hubbs 2021-03-23 9:23 ` Michał Górny 2021-03-24 5:20 ` William Hubbs 2021-03-24 7:48 ` Michał Górny 2021-03-24 15:09 ` William Hubbs 2021-03-24 17:09 ` Rich Freeman 2021-03-24 18:37 ` William Hubbs
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox