[gentoo-portage-dev] Is portage (/usr)/bin-merge safe?

[gentoo-portage-dev] Is portage (/usr)/bin-merge safe?

[Duncan]

As in subject, is portage bin/usr-bin merge safe?

It appears most of my clashing files are /usr/bin/* -> /bin/* symlinks.  
(That's just bin, I've not looked at sbin.)

Does portage "just do the right thing" if the dirs are linked to each 
other?

Meanwhile, a quick eyeball of the results says 50-60% of the hits are 
coreutils, so if portage doesn't handle it automatically, fixing it for 
just that one package, even just using a USE flag instead of trying to 
detect it automatically, would kill a majority of the birds with a single 
stone.

(Since I don't have a separate /usr anyway, I've been thinking about 
it...  If it's not easily doable anyway, that cuts short the internal 
debate.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Is portage (/usr)/bin-merge safe?

[Zac Medico]

On 05/31/2013 10:36 PM, Duncan wrote:
> As in subject, is portage bin/usr-bin merge safe?
>
> It appears most of my clashing files are /usr/bin/* -> /bin/* symlinks.
> (That's just bin, I've not looked at sbin.)

I haven't tried it, but it should work just fine. Portage has always 
supported directory symlinks like these. I haven't heard any recent 
complaints regarding them.
-- 
Thanks,
Zac

[gentoo-portage-dev] Re: Is portage (/usr)/bin-merge safe?

[Duncan]

Zac Medico posted on Fri, 31 May 2013 22:49:02 -0700 as excerpted:

> On 05/31/2013 10:36 PM, Duncan wrote:
>> As in subject, is portage bin/usr-bin merge safe?
>>
>> It appears most of my clashing files are /usr/bin/* -> /bin/* symlinks.
>> (That's just bin, I've not looked at sbin.)
> 
> I haven't tried it, but it should work just fine. Portage has always
> supported directory symlinks like these. I haven't heard any recent
> complaints regarding them.

As the attribution says, I'm resurrecting a thread from 2013...

I set up a merged /usr/bin -> /bin (and sbin -> bin, and /usr -> .) soon 
after that, with very few problems, usually ebuilds doing unconditional 
rms in postinst or the like, until recently...

[I'll likely file this as a bug as well, but thought I'd post a followup 
to the old thread here, first.  I'm kinda busy troubleshooting the 
unrelated bug that triggered the coreutils expression of this bug for me, 
ATM.]

Something recently changed, as now I'm having many more problems, so far 
with four packages, glibc (!!), coreutils (!!), nano, and shadow, 
installing symlinks that ultimately point to themselves.  The glibc one 
is of course critical as it breaks pretty much the entire system right 
away, the coreutils set is critical due to the number of frequently used 
binaries it breaks, and I'm lucky I discovered nano before needing it as 
a low-dep fallback editor.  Being a single-user system I don't so often 
use passwd, but like nano, it's one of those things that when it's 
needed, it's REALLY needed.

From my current installmask file:

# 2017.1112 glibc: libm-2.*.so due to /usr -> . symlink,
# symlink overwrites the lib it points to!
INSTALL_MASK="
        $INSTALL_MASK
        /usr/lib64/libm-2.*.so
"

# 2017.1207 coreutils symlinks that overwrite their binaries
INSTALL_MASK="
        $INSTALL_MASK
        /usr/bin/basename
        /usr/bin/chroot
        /usr/bin/cut
        /usr/bin/dir
        /usr/bin/dirname
        /usr/bin/du
        /usr/bin/env
        /usr/bin/expr
        /usr/bin/head
        /usr/bin/mkfifo
        /usr/bin/mktemp
        /usr/bin/readlink
        /usr/bin/seq
        /usr/bin/sleep
        /usr/bin/sort
        /usr/bin/tail
        /usr/bin/touch
        /usr/bin/tr
        /usr/bin/tty
        /usr/bin/uname
        /usr/bin/vdir
        /usr/bin/wc
        /usr/bin/yes
"
# 2017.1207 shadow, nano symlinks
INSTALL_MASK="
        $INSTALL_MASK
        /usr/bin/nano
        /usr/bin/passwd
"

So what changed in portage that previously prevented the /usr/* symlinks 
from overwriting the non-usr binaries, but now allows the overwrites to 
go ahead, breaking the system?

Note that I ran into the glibc library symlink issue first.  I ran into 
the coreutils issue after a bad upgrade (unrelated, I think) broke X, 
forcing me back to a backup and I started upgrading a few packages at a 
time from binpkg, to see which one broke X again.  When I got to 
coreutils, the qmerge phase broke half way thru as a binary was replaced 
by a symlink to itself.  I'm not sure why it triggered in binpkg install 
but not when I had originally installed it on the system, but it may be 
due to the fact that I normally run parallel merges so the system is 
heavily loaded during normal merge, while with the binpkg merge it 
wasn't, thus implying a race condition of some sort.  I discovered the 
nano and shadow/passwd issues, seeing their binaries were broken symlinks 
to themselves, when fixing the coreutils issue. I've no idea when they 
happened.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Re: Is portage (/usr)/bin-merge safe?

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 4085 bytes --]

On 12/07/2017 12:37 AM, Duncan wrote:
> Zac Medico posted on Fri, 31 May 2013 22:49:02 -0700 as excerpted:
> 
>> On 05/31/2013 10:36 PM, Duncan wrote:
>>> As in subject, is portage bin/usr-bin merge safe?
>>>
>>> It appears most of my clashing files are /usr/bin/* -> /bin/* symlinks.
>>> (That's just bin, I've not looked at sbin.)
>>
>> I haven't tried it, but it should work just fine. Portage has always
>> supported directory symlinks like these. I haven't heard any recent
>> complaints regarding them.
> 
> As the attribution says, I'm resurrecting a thread from 2013...
> 
> I set up a merged /usr/bin -> /bin (and sbin -> bin, and /usr -> .) soon 
> after that, with very few problems, usually ebuilds doing unconditional 
> rms in postinst or the like, until recently...
> 
> [I'll likely file this as a bug as well, but thought I'd post a followup 
> to the old thread here, first.  I'm kinda busy troubleshooting the 
> unrelated bug that triggered the coreutils expression of this bug for me, 
> ATM.]
> 
> Something recently changed, as now I'm having many more problems, so far 
> with four packages, glibc (!!), coreutils (!!), nano, and shadow, 
> installing symlinks that ultimately point to themselves.  The glibc one 
> is of course critical as it breaks pretty much the entire system right 
> away, the coreutils set is critical due to the number of frequently used 
> binaries it breaks, and I'm lucky I discovered nano before needing it as 
> a low-dep fallback editor.  Being a single-user system I don't so often 
> use passwd, but like nano, it's one of those things that when it's 
> needed, it's REALLY needed.
> 
> From my current installmask file:
> 
> # 2017.1112 glibc: libm-2.*.so due to /usr -> . symlink,
> # symlink overwrites the lib it points to!
> INSTALL_MASK="
>         $INSTALL_MASK
>         /usr/lib64/libm-2.*.so
> "
> 
> # 2017.1207 coreutils symlinks that overwrite their binaries
> INSTALL_MASK="
>         $INSTALL_MASK
>         /usr/bin/basename
>         /usr/bin/chroot
>         /usr/bin/cut
>         /usr/bin/dir
>         /usr/bin/dirname
>         /usr/bin/du
>         /usr/bin/env
>         /usr/bin/expr
>         /usr/bin/head
>         /usr/bin/mkfifo
>         /usr/bin/mktemp
>         /usr/bin/readlink
>         /usr/bin/seq
>         /usr/bin/sleep
>         /usr/bin/sort
>         /usr/bin/tail
>         /usr/bin/touch
>         /usr/bin/tr
>         /usr/bin/tty
>         /usr/bin/uname
>         /usr/bin/vdir
>         /usr/bin/wc
>         /usr/bin/yes
> "
> # 2017.1207 shadow, nano symlinks
> INSTALL_MASK="
>         $INSTALL_MASK
>         /usr/bin/nano
>         /usr/bin/passwd
> "
> 
> So what changed in portage that previously prevented the /usr/* symlinks 
> from overwriting the non-usr binaries, but now allows the overwrites to 
> go ahead, breaking the system?
> 
> Note that I ran into the glibc library symlink issue first.  I ran into 
> the coreutils issue after a bad upgrade (unrelated, I think) broke X, 
> forcing me back to a backup and I started upgrading a few packages at a 
> time from binpkg, to see which one broke X again.  When I got to 
> coreutils, the qmerge phase broke half way thru as a binary was replaced 
> by a symlink to itself.  I'm not sure why it triggered in binpkg install 
> but not when I had originally installed it on the system, but it may be 
> due to the fact that I normally run parallel merges so the system is 
> heavily loaded during normal merge, while with the binpkg merge it 
> wasn't, thus implying a race condition of some sort.  I discovered the 
> nano and shadow/passwd issues, seeing their binaries were broken symlinks 
> to themselves, when fixing the coreutils issue. I've no idea when they 
> happened.
> 

I think the sort order of your root directory changed for some reason.
The order that readdir returns filenames depends on the filesystem
implementation:

http://man7.org/linux/man-pages/man3/readdir.3.html
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

[gentoo-portage-dev] Re: Is portage (/usr)/bin-merge safe?

[Duncan]

Zac Medico posted on Thu, 07 Dec 2017 01:07:21 -0800 as excerpted:

> On 12/07/2017 12:37 AM, Duncan wrote:
>> Zac Medico posted on Fri, 31 May 2013 22:49:02 -0700 as excerpted:
>> 
>>> On 05/31/2013 10:36 PM, Duncan wrote:
>>>> As in subject, is portage bin/usr-bin merge safe?
>>>>
>>>> It appears most of my clashing files are /usr/bin/* -> /bin/*
>>>> symlinks.
>>>
>>> I haven't tried it, but it should work just fine. Portage has always
>>> supported directory symlinks like these. I haven't heard any recent
>>> complaints regarding them.
>> 
>> As the attribution says, I'm resurrecting a thread from 2013...
>> 
>> I set up a merged /usr/bin -> /bin (and sbin -> bin, and /usr -> .)
>> soon after that, with very few problems, usually ebuilds doing
>> unconditional rms in postinst or the like, until recently...
>> 
>> Something recently changed, as now I'm having many more problems, so
>> far with four packages, glibc (!!), coreutils (!!), nano, and shadow,
>> installing symlinks that ultimately point to themselves.
>> 
> I think the sort order of your root directory changed for some reason.
> The order that readdir returns filenames depends on the filesystem
> implementation:
> 
> http://man7.org/linux/man-pages/man3/readdir.3.html

That's... strange.  Back in 2013 might have still been on reiserfs, but 
I've been on btrfs for awhile now.  I wonder what might make it change 
order?

Tho I /did/ somewhat recently upgrade ssds, thus copying the /bin dir 
and /usr -> . symlink, among other root entries.  Obviously back when I 
first setup the /usr -> . symlink it was the newest entry.  Maybe if I 
delete and recreate it so it's definitely the newest entry again...

I have no idea how long it might have been before I came up with the idea 
to try that on my own.  Thanks!  I'll (gingerly, I don't like major 
system breakage!) see if it makes a difference.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Is portage (/usr)/bin-merge safe?

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 300 bytes --]

On Saturday 01 June 2013 01:36:28 Duncan wrote:
> As in subject, is portage bin/usr-bin merge safe?

portage should be merge safe for all files.  it specifically writes it to a temp 
file and then does a rename so it's an atomic update.  otherwise you'd get 
things like ETXTBSY errors.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-portage-dev] Is portage (/usr)/bin-merge safe?

[vivo75]

On 06/01/13 07:36, Duncan wrote:
> As in subject, is portage bin/usr-bin merge safe?
>
> It appears most of my clashing files are /usr/bin/* -> /bin/* symlinks.  
> (That's just bin, I've not looked at sbin.)
Most but not all, I'm also quite sure in the past I've encountered one
(and only one) files collision (not symlinks), cannot remember which though.
>
> Does portage "just do the right thing" if the dirs are linked to each 
> other?
Trust the answers of Zac and Mike on this ;-)
>
> Meanwhile, a quick eyeball of the results says 50-60% of the hits are 
> coreutils, so if portage doesn't handle it automatically, fixing it for 
> just that one package, even just using a USE flag instead of trying to 
> detect it automatically, would kill a majority of the birds with a single 
> stone.
>
> (Since I don't have a separate /usr anyway, I've been thinking about 
> it...  If it's not easily doable anyway, that cuts short the internal 
> debate.)
While portage can be safe, for various reason (including the resultant
pkg) I do prefer to do the move in post_src_install() #1
All my tests have been done against a manually converted filesystem,
i.e. manually moving stuff around, creating the symlinks, putting the
code in bashrc and only after that (re)building everything inside the
system.
So I've no idea if this approach can work in a stage1 or equivalent.
It has been used for quite a long time from a "build server" =>
"consumer machine" environment.


#1 excerpt from bashrc, this code is rough but work in the gentoo
ebuilds tree domain

move_root_to_usr() {

  pushd "${D}"

  echo "PHASE: ${EBUILD_PHASE} move something around";

  for d in lib* *bin; do

    if [[ -d ${d} ]] ; then
      # move directories
      for f in $(find ${d}/ -type d) ; do
        if [[ -e usr/${f} ]] ; then
          if [[ -e ${f} ]] ; then
            echo "usr/${f} : exist, cannot move"
          else
            : # parent has already been moved
          fi
        else
          echo "${f} : ok to move"
          mv ${f} usr/${f}
        fi
      done

      # move files
      for f in $(find ${d}/ -type f -or -type l) ; do
        if [[ ( -e usr/${f} ) || (-L usr/${f}) ]] ; then
          if [[ ( -L ${f} ) && ( -f usr/${f} ) ]] ; then
            echo "${f} : is a symlink, remove"
            rm -f ${f}
          else
            if [[ ( -f ${f} ) && ( -L usr/${f} ) ]] ; then
              echo "${f} : file in usr is a symlink remove e move"
              rm -f usr/${f}
              mv ${f} usr/${f}
            else
              ls -ld ${f} usr/${f}
              echo "${f} : UNHANDLED CASE prepare to breakage"
            fi
          fi
        else
          echo "${f} : ok to move"
          mv ${f} usr/${f}
        fi
      done

      rmdir -p ${d} && [[ -e usr/${d} ]] || ln -s usr/${d} ${d}

    fi # if [[ -d ${d} ]] ; then

  done # for d in lib64 lib32 bin sbin; do

  rm -f lib/lib

  popd
}

post_src_install() {
    move_root_to_usr
}

[gentoo-portage-dev] Re: Is portage (/usr)/bin-merge safe?

[Duncan]

vivo75@gmail.com posted on Sun, 02 Jun 2013 13:14:41 +0200 as excerpted:

> While portage can be safe, for various reason (including the resultant
> pkg) I do prefer to do the move in post_src_install() #1 All my tests
> have been done against a manually converted filesystem

That's what mine would be...

> #1 excerpt from bashrc, this code is rough but work in the gentoo
> ebuilds tree domain
> 
> move_root_to_usr() {

Thanks.  What I was thinking would actually reverse that (/bin being the 
real dir, /sbin being a symlink to it), given my (traditional sysadmin) 
pref for short paths, but I hadn't thought of a bashrc solution at all, 
so that gives me yet another way of doing it. =:^)

My first thought is that I prefer standard layout packages, however, 
easing interoperability should I decide to swap binpkgs with someone.  
(Yes, I'm aware of the security issues if the parties don't trust each 
other...)

But OTOH I think that solves issues such as path-based equery belongs, 
for instance.  Being amd64 for nearing a decade now (and no-multilib for 
several years of it), I'm used to worrying about that with the symlinked 
lib/lib64 thing, and that's the one thing I wasn't looking forward to 
with unified bins.  (I think I'll keep bin/sbin separate at first, see 
how bin/usr-bin go first, then think about bin/sbin.)

But if your bashrc solution /does/ solve the equery belongs path thing I 
might well use it on lib/lib64 as well...  (Either that or since I 
believe the libs are a profile thing and I'm already running a heavily 
modified profile, no @system for instance, I could probably simply modify 
that...  Actually, that's probably a better solution in any case, since 
it's just undoing mainline settings the same way mainline does them in 
the first place.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Re: Is portage (/usr)/bin-merge safe?

[vivo75]

On 06/02/13 13:54, Duncan wrote:
> vivo75@gmail.com posted on Sun, 02 Jun 2013 13:14:41 +0200 as excerpted:
>
>> While portage can be safe, for various reason (including the resultant
>> pkg) I do prefer to do the move in post_src_install() #1 All my tests
>> have been done against a manually converted filesystem
> That's what mine would be...
>
>> #1 excerpt from bashrc, this code is rough but work in the gentoo
>> ebuilds tree domain
>>
>> move_root_to_usr() {
> Thanks.  What I was thinking would actually reverse that (/bin being the 
> real dir, /sbin being a symlink to it), given my (traditional sysadmin) 
> pref for short paths, but I hadn't thought of a bashrc solution at all, 
> so that gives me yet another way of doing it. =:^)
>
> My first thought is that I prefer standard layout packages, however, 
> easing interoperability should I decide to swap binpkgs with someone.  
> (Yes, I'm aware of the security issues if the parties don't trust each 
> other...)
>
> But OTOH I think that solves issues such as path-based equery belongs, 
> for instance.  Being amd64 for nearing a decade now (and no-multilib for 
> several years of it), I'm used to worrying about that with the symlinked 
> lib/lib64 thing, and that's the one thing I wasn't looking forward to 
> with unified bins.  (I think I'll keep bin/sbin separate at first, see 
> how bin/usr-bin go first, then think about bin/sbin.)
>
> But if your bashrc solution /does/ solve the equery belongs path thing I 
> might well use it on lib/lib64 as well...  (Either that or since I 
> believe the libs are a profile thing and I'm already running a heavily 
> modified profile, no @system for instance, I could probably simply modify 
> that...  Actually, that's probably a better solution in any case, since 
> it's just undoing mainline settings the same way mainline does them in 
> the first place.)

I do generally leave profiles untouched but yes it could be a solution,
maybe some research in debian maillist could be beneficial too.
In the meantime these commands results should tell you about equery belongs:

>hom>vivo$ qlist coreutils | grep -c '^/bin/'
0
>hom>vivo$ qlist coreutils | grep -c '^/usr/bin/'
101

>hom>vivo$ equery belongs /usr/bin/sleep
 * Searching for /usr/bin/sleep ...
sys-apps/coreutils-8.21 (/usr/bin/sleep)

[gentoo-portage-dev] Re: Is portage (/usr)/bin-merge safe?

[Duncan]

vivo75@gmail.com posted on Sun, 02 Jun 2013 14:47:54 +0200 as excerpted:

> In the meantime these commands results should tell you about equery
> belongs:
> 
>>hom>vivo$ qlist coreutils | grep -c '^/bin/'
> 0
>>hom>vivo$ qlist coreutils | grep -c '^/usr/bin/'
> 101
> 
>>hom>vivo$ equery belongs /usr/bin/sleep
>  * Searching for /usr/bin/sleep ...
> sys-apps/coreutils-8.21 (/usr/bin/sleep)

Thanks. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman