[gentoo-dev] Modification proposal for user/group creation when ROOT!="/"

[gentoo-dev] Modification proposal for user/group creation when ROOT!="/"

[Farid BENAMROUCHE]

Hi all,

Currently there is an old known limitation when using ROOT= option to install a package in a folder: user/groups are created in the host filesystem, not the target root filesystem.

So I've pushed some modifications to the upstream shadow repo.
Basically, I've added a --prefix option to user{add,mod,del} and group{add,mod,del}
This option does the same as --root option, but whithout a chroot (so compatible when cross compiling)
You can see more details (and the limitation of my implementation) in the shadow github repo:
https://github.com/shadow-maint/shadow/issues/18


Now, for the gentoo part, I do have a working solution that I've pushed in the following bugzilla:
https://bugs.gentoo.org/show_bug.cgi?id=541406

A new user.eclass file with modified enewuser,enewgroup and egetent that all supports ${ROOT} option via --prefix in shadow utilities.
For now I've only added this option for linux.

However, I've encountered some unexpected issues: some ebuilds are using direct calls to chown and fowners. Both are not compatible with ${ROOT}...

To solve this, I've created 2 new calls in user.eclass: echown and efowners.
The only thing the new functions are doing is to get the uid/gid from the correct passwd/group files from ${ROOT} using the modified egetent function and pass that to the native chown/fowners...

For example, in sys-power/nut we can find:
chown nut:nut ${ROOT}/var/lib/nut

This should be changed to
echown nut:nut ${ROOT}/var/lib/nut

Same to fowners.
If the modification is not done, either the ebuild will fail because the nut user does not exists in the host, or the incorrect uid will be user in ${ROOT}

The solution is not perfect, but at least better than what we have today, and totally usable I believe.

I've uploaded the patches for lighttpd and nut, plus my patch for user.eclass for review in this bug... we do have time until upstream shadow team reviews and commits my modifications (at least).

Side note: it's a bit complicated to know when to add ${ROOT} and when not in a ebuild... For example, chown needs ${ROOT} but fowners must not!...
Side note 2: maybe I should add a verification to check if useradd/groupadd supports my new --prefix solution, and fallback to original behavior if not?

Tests: I've compiled a full working system with my above solution, and it works. (cross compilation in a dedicated target root path)

Regards,
Farid

[gentoo-dev] Re: Modification proposal for user/group creation when ROOT!="/"

[Benda Xu]

Hi Farid,

This is an excellent idea!  It is very helpful for Gentoo Prefix/libc,
where we maintain a set of nss databases (passwd, group, shadow, etc.)
inside a directory prefix.

Farid BENAMROUCHE <fariouche@yahoo.fr> writes:

> Currently there is an old known limitation when using ROOT= option to
> install a package in a folder: user/groups are created in the host
> filesystem, not the target root filesystem.

Exactly.

> So I've pushed some modifications to the upstream shadow repo.
> Basically, I've added a --prefix option to user{add,mod,del} and
> group{add,mod,del} This option does the same as --root option, but
> whithout a chroot (so compatible when cross compiling) 

Cool.

> You can see more details (and the limitation of my implementation) in
> the shadow github repo:
> https://github.com/shadow-maint/shadow/issues/18

Hope the upstream accept your patch soon.

> Now, for the gentoo part, I do have a working solution that I've
> pushed in the following bugzilla:
> https://bugs.gentoo.org/show_bug.cgi?id=541406
>
> A new user.eclass file with modified enewuser,enewgroup and egetent
> that all supports ${ROOT} option via --prefix in shadow utilities.
> For now I've only added this option for linux.

The new user.eclass requires the new shadow.  After the upstream makes a
new release, it will take a long time for Gentoo to use the feature.
Because we should carefully handle the compatibility with the old
systems.

> However, I've encountered some unexpected issues: some ebuilds are
> using direct calls to chown and fowners. Both are not compatible with
> ${ROOT}...

Those ebuilds are broken and should be fixed.

> To solve this, I've created 2 new calls in user.eclass: echown and
> efowners.  The only thing the new functions are doing is to get the
> uid/gid from the correct passwd/group files from ${ROOT} using the
> modified egetent function and pass that to the native chown/fowners...
>
> For example, in sys-power/nut we can find: chown nut:nut
> ${ROOT}/var/lib/nut
>
> This should be changed to echown nut:nut ${ROOT}/var/lib/nut

Brilliant.

> Same to fowners.  If the modification is not done, either the ebuild
> will fail because the nut user does not exists in the host, or the
> incorrect uid will be user in ${ROOT}
>
> The solution is not perfect, but at least better than what we have
> today, and totally usable I believe.
>
> I've uploaded the patches for lighttpd and nut, plus my patch for
> user.eclass for review in this bug... we do have time until upstream
> shadow team reviews and commits my modifications (at least).
>
> Side note: it's a bit complicated to know when to add ${ROOT} and when
> not in a ebuild... For example, chown needs ${ROOT} but fowners must
> not!...  

Why not?  Could you give more details?

> Side note 2: maybe I should add a verification to check if
> useradd/groupadd supports my new --prefix solution, and fallback to
> original behavior if not?

IMHO, useradd/groupadd supporting --prefix will be captured by a new
EAPI in the (far) future.  At present we don't need to worry about it.

> Tests: I've compiled a full working system with my above solution, and
> it works. (cross compilation in a dedicated target root path)

Do you have an overlay for me reproduce your result?  I am also
interested in hosting it in proj/android.git[1] if you do not have one
yet.

Keep on your good work.

Yours,
Benda   

1. https://gitweb.gentoo.org/proj/android.git/

[gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[Farid BENAMROUCHE]

Hi all,

Already two years....
Yes, two years ago I've posted here to notify you about the creattion of users and groups when using "ROOT=".
As a reminder, if you currently emerge a package to a specific rootfs folder, some packages will actually not create the user and groups correctly inside this rootfs.
It will also not check for the existance of the user/group inside of the rootfs.
Everytime, it will check "/".

This very old gentoo issue (I have to find again the GLEP talking about this issue).

The solution is not possible without changing the behaviour of the tools used by portage. For example, portage is using shadow in most systems (and shadow is using the glibc).

I've submited a patch first to shadow upstream team to avec this feature, using a new argument (--prefix) to tools like usemod,useradd etc
You can create users in any rootfs you want, no need to be root (except if you local pam configuration requires it)

This is unfortunately not enough for gentoo: some packages are relying on chaonw directly. So I've added some wrappers (same as the current enewuser etc functions)
This enable people to build embedded systems rootfs from scratch, using a complete gentoo.
Some packages must be updated in order to behave properly in that case (replace chown with echown for example)
As a failsafe, I'm only triggering the new behavior in case ROOT env var is not "/" (and I think (I have to check) that if the command fails, it will fallback to the old behavior in case the --prefix argument is not implemented (incorrect shadow version for example)

This requires in that case a new EAPI I think. Just tell me what to do, where, and I will submit the proposal

I'm using the modifications for 2 years now.


Limitations: 
- selinux is not supported (ie: you will still use the system's selinux.). 
- Also, this only works with gentoo systems using the shadow package (the exact version will depend on when the new release will be done).
- Last one: I only tested on a linux configuration


Cheers


--------------------------------------------
En date de : Mar 24.5.16, Farid BENAMROUCHE <fariouche@yahoo.fr> a écrit :

 Objet: Modification proposal for user/group creation when ROOT!="/"
 À: gentoo-dev@lists.gentoo.org
 Date: Mardi 24 mai 2016, 21h00
 
 Hi all,
 
 Currently there is an old known
 limitation when using ROOT= option to install a package in a
 folder: user/groups are created in the host filesystem, not
 the target root filesystem.
 
 So I've pushed some modifications to
 the upstream shadow repo.
 Basically, I've added a --prefix option
 to user{add,mod,del} and group{add,mod,del}
 This option does the same as --root
 option, but whithout a chroot (so compatible when cross
 compiling)
 You can see more details (and the
 limitation of my implementation) in the shadow github repo:
 https://github.com/shadow-maint/shadow/issues/18
 
 
 Now, for the gentoo part, I do have a
 working solution that I've pushed in the following
 bugzilla:
 https://bugs.gentoo.org/show_bug.cgi?id=541406
 
 A new user.eclass file with modified
 enewuser,enewgroup and egetent that all supports ${ROOT}
 option via --prefix in shadow utilities.
 For now I've only added this option for
 linux.
 
 However, I've encountered some
 unexpected issues: some ebuilds are using direct calls to
 chown and fowners. Both are not compatible with ${ROOT}...
 
 To solve this, I've created 2 new calls
 in user.eclass: echown and efowners.
 The only thing the new functions are
 doing is to get the uid/gid from the correct passwd/group
 files from ${ROOT} using the modified egetent function and
 pass that to the native chown/fowners...
 
 For example, in sys-power/nut we can
 find:
 chown nut:nut ${ROOT}/var/lib/nut
 
 This should be changed to
 echown nut:nut ${ROOT}/var/lib/nut
 
 Same to fowners.
 If the modification is not done, either
 the ebuild will fail because the nut user does not exists in
 the host, or the incorrect uid will be user in ${ROOT}
 
 The solution is not perfect, but at
 least better than what we have today, and totally usable I
 believe.
 
 I've uploaded the patches for lighttpd
 and nut, plus my patch for user.eclass for review in this
 bug... we do have time until upstream shadow team reviews
 and commits my modifications (at least).
 
 Side note: it's a bit complicated to
 know when to add ${ROOT} and when not in a ebuild... For
 example, chown needs ${ROOT} but fowners must not!...
 Side note 2: maybe I should add a
 verification to check if useradd/groupadd supports my new
 --prefix solution, and fallback to original behavior if
 not?
 
 Tests: I've compiled a full working
 system with my above solution, and it works. (cross
 compilation in a dedicated target root path)
 
 Regards,
 Farid
 

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 2427 bytes --]

On Fri, 30 Mar 2018 18:52:18 +0000 (UTC)
Farid BENAMROUCHE <fariouche@yahoo.fr> wrote:

> Yes, two years ago I've posted here to notify you about the creattion of users and groups when using "ROOT=".
> As a reminder, if you currently emerge a package to a specific rootfs folder, some packages will actually not create the user and groups correctly inside this rootfs.
> It will also not check for the existance of the user/group inside of the rootfs.
> Everytime, it will check "/".
> 
> This very old gentoo issue (I have to find again the GLEP talking about this issue).
> 
> The solution is not possible without changing the behaviour of the tools used by portage. For example, portage is using shadow in most systems (and shadow is using the glibc).

Hi,

I have an interest this and was one of the early commenters in bug
#541406. I made my own suggestions about how this might work. Your
solution is cleaner in that it doesn't require modifying the users in
the / system but it does require significant changes to tools,
eclasses, and ebuilds so I'm on the fence about it.

I did just have a lightbulb moment though. I've been playing with
unshare recently and I wondered if we could leverage it here. First I
tried this.

  $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc/group /etc/group && groupadd foo"
  groupadd: failure while writing changes to /etc/group

It is possible to bind mount individual files but it doesn't work here
because it tries to rename /etc/group to made a backup. Next I tried
the whole directory but it gives a strange error.

  $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc /etc && groupadd foo"
  groupadd: Cannot determine your user name.

This reveals more.

  $ sudo unshare -m /bin/sh -c "id && mount --bind /mnt/utilite/mnt/moi/etc /etc && id"                        
  uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
  uid=0 gid=0 groups=0,1,2,3,4,6,10,11,20,26,27

I'm not sure why the IDs break like this and strace doesn't make it any
clearer. This seems like a route worth pursuing though because you
could create a bunch of wrappers for useradd, groupadd, chown and so on
and it would then all work transparently, even when not using the
eclass functions.

Regards,
-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

On Fri, 30 Mar 2018 20:23:49 +0100
James Le Cuirot <chewi@gentoo.org> wrote:

> I did just have a lightbulb moment though. I've been playing with
> unshare recently and I wondered if we could leverage it here.
> 
>   $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc /etc && groupadd foo"
>   groupadd: Cannot determine your user name.

Aha! I was trying to do this against an NFS share for a system with a
different architecture. If I use a local mount with a compatible
architecture, it actually does work. I'll explore this some more.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

On Fri, 30 Mar 2018 20:47:20 +0100
James Le Cuirot <chewi@gentoo.org> wrote:

> On Fri, 30 Mar 2018 20:23:49 +0100
> James Le Cuirot <chewi@gentoo.org> wrote:
> 
> > I did just have a lightbulb moment though. I've been playing with
> > unshare recently and I wondered if we could leverage it here.
> > 
> >   $ sudo unshare -m /bin/sh -c "mount --bind /mnt/somewhere/etc /etc && groupadd foo"
> >   groupadd: Cannot determine your user name.  
> 
> Aha! I was trying to do this against an NFS share for a system with a
> different architecture. If I use a local mount with a compatible
> architecture, it actually does work. I'll explore this some more.

Figured it out! The system I was doing this against has an ancient
glibc (long story) with an old nsswitch.conf. I replaced this file with
a newer one and it all started working. Do you agree this could be the
way forwards?

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[Farid BENAMROUCHE]

interresting aproach.
this could work. however, i can see a few limitations:
- you must be root.
- this is specific to linux as of today.
- if you want to hide the mechanism, i don't see how without doing the same portage modifications as in my solution.


but this is maybe worth investigating. my solution isn't perfect too, I admit.



--------------------------------------------
En date de : Ven 30.3.18, James Le Cuirot <chewi@gentoo.org> a écrit :

 Objet: Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"
 À: gentoo-dev@lists.gentoo.org
 Date: Vendredi 30 mars 2018, 21h56
 
 On Fri, 30 Mar 2018 20:47:20
 +0100
 James Le Cuirot <chewi@gentoo.org>
 wrote:
 
 > On Fri, 30 Mar
 2018 20:23:49 +0100
 > James Le Cuirot
 <chewi@gentoo.org>
 wrote:
 > 
 > > I did
 just have a lightbulb moment though. I've been playing
 with
 > > unshare recently and I
 wondered if we could leverage it here.
 >
 > 
 > >   $ sudo unshare -m /bin/sh
 -c "mount --bind /mnt/somewhere/etc /etc &&
 groupadd foo"
 > >   groupadd:
 Cannot determine your user name.  
 > 
 > Aha! I was trying to do this against an
 NFS share for a system with a
 > different
 architecture. If I use a local mount with a compatible
 > architecture, it actually does work.
 I'll explore this some more.
 
 Figured it out! The system I was doing this
 against has an ancient
 glibc (long story)
 with an old nsswitch.conf. I replaced this file with
 a newer one and it all started working. Do you
 agree this could be the
 way forwards?
 
 -- 
 James Le Cuirot (chewi)
 Gentoo Linux
 Developer

Re: [gentoo-dev] Re : Modification proposal for user/group creation when ROOT!="/"

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 1285 bytes --]

On Sat, 31 Mar 2018 09:39:47 +0000 (UTC)
Farid BENAMROUCHE <fariouche@yahoo.fr> wrote:

> interresting aproach.
> this could work. however, i can see a few limitations:
> - you must be root.

Actually you don't if you add -r to unshare, which gives you what is
sometimes called fakeroot. Obviously you still can't modify the files
if they are really owned by root but that's true of any solution.

> - this is specific to linux as of today.

True and I am only interested in Linux but I like to play nice. Other
platforms could potentially still briefly bind mount but it wouldn't be
isolated from the other processes so it wouldn't be entirely safe. Safe
enough though? You'd need to weigh this up against how many people use
ROOT!=/ on other platforms. Not many at all, I imagine.

> - if you want to hide the mechanism, i don't see how without doing
> the same portage modifications as in my solution.

You could handle this in the eclass functions but as you pointed out,
many things call chown/chgrp directly. Usage by ebuilds themselves can
be addressed but if a build system calls these then eclass functions
will not help. What would work is adding some identically-named
wrappers to the PATH.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]