[gentoo-dev] Determenistic system group and user id

[gentoo-dev] Determenistic system group and user id

[Alexey Shvetsov]

Hi all!

We trying to use ldap for users @work, many of our workstations running 
binary gentoo based distro called Calculate linux. However if we wanna 
have wide use of ldap there is a need for determenistic system group 
gids names and user uids.

Many ebuilds in tree uses enewgroup and enewuser with -1 (aka next 
available parameter)[1]. However it will be much better to set distro 
wide deterministic uid and gid for system service name. So for example 
ldap users may have determenistic groups like video, audio, plugdev, 
etc..

[1] $ egrep '(enewgroup|enewuser)' * -R | awk -F '/' '{print $1 "/" $2}' 
| grep -v eclass | sort -u | wc -l
443
So there not so much gid uids needed

-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

[gentoo-dev] Use GLEP27!

[Robin H. Johnson]

On Sun, Dec 13, 2015 at 09:03:51PM +0300, Alexey Shvetsov wrote:
> Hi all!
> 
> We trying to use ldap for users @work, many of our workstations running 
> binary gentoo based distro called Calculate linux. However if we wanna 
> have wide use of ldap there is a need for determenistic system group 
> gids names and user uids.
> 
> Many ebuilds in tree uses enewgroup and enewuser with -1 (aka next 
> available parameter)[1]. However it will be much better to set distro 
> wide deterministic uid and gid for system service name. So for example 
> ldap users may have determenistic groups like video, audio, plugdev, 
> etc..
GLEP27 was approved for this, however it is barely used.

Convert the rest of the tree to use it, and then you'll be done, aside
from the existing mess on user systems.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Determenistic system group and user id

[Mike Gilbert]

On Sun, Dec 13, 2015 at 1:03 PM, Alexey Shvetsov <alexxy@gentoo.org> wrote:
> We trying to use ldap for users @work, many of our workstations running
> binary gentoo based distro called Calculate linux. However if we wanna have
> wide use of ldap there is a need for determenistic system group gids names
> and user uids.

I think you are looking for GLEP 27 and the related bug 53269.

Unfortunately, it seems there has been no movement on that bug in
several years. Maybe you could help implement something?

https://wiki.gentoo.org/wiki/GLEP:27

https://bugs.gentoo.org/show_bug.cgi?id=53269

Re: [gentoo-dev] Determenistic system group and user id

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2186 bytes --]

On Sun, Dec 13, 2015 at 10:03 AM, Alexey Shvetsov <alexxy@gentoo.org> wrote:

> Hi all!
>
> We trying to use ldap for users @work, many of our workstations running
> binary gentoo based distro called Calculate linux. However if we wanna have
> wide use of ldap there is a need for determenistic system group gids names
> and user uids.
>
> Many ebuilds in tree uses enewgroup and enewuser with -1 (aka next
> available parameter)[1]. However it will be much better to set distro wide
> deterministic uid and gid for system service name. So for example ldap
> users may have determenistic groups like video, audio, plugdev, etc..
>

So the first question I normally ask here is:

1) Why do you need deterministic uid / gid's?
2) If you do need deterministic uid / gid's, I would recommend storing them
all in the same place.

For example, you typically want a deterministic UID for a user. To
accomplish this, you add that user to LDAP, give them a UID in LDAP, and
then either add LDAP to nssswitch or use something like nsscache to sync
the ldap UID's into the local system.

3) If you need deterministic GID's I would recommend storing them all in
LDAP and syncing the group memberships locally.

I never understood why people would think the distro should handle unique
gid / uids. Plus you usually end up running:

1) More than one distro.
2) More than one 'flavor' of a single distro where for whatever reason, uid
and gid decisions differed (they renumbered, etc.)

So if you want a consistent GID for a group, store the group name and gid
in ldap and sync it; do not rely on your distro to do it. IMHO doing so is
a design error.

-A


>
> [1] $ egrep '(enewgroup|enewuser)' * -R | awk -F '/' '{print $1 "/" $2}' |
> grep -v eclass | sort -u | wc -l
> 443
> So there not so much gid uids needed
>
> --
> Best Regards,
> Alexey 'Alexxy' Shvetsov
> Best Regards,
> Alexey 'Alexxy' Shvetsov, PhD
> Department of Molecular and Radiation Biophysics
> FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
> Leningrad region, Gatchina, Russia
> Gentoo Team Ru
> Gentoo Linux Dev
> mailto:alexxyum@gmail.com
> mailto:alexxy@gentoo.org
> mailto:alexxy@omrb.pnpi.spb.ru
>
>

[-- Attachment #2: Type: text/html, Size: 3124 bytes --]

Re: [gentoo-dev] Determenistic system group and user id

[Marc Schiffbauer]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

* Alec Warner schrieb am 13.12.15 um 23:23 Uhr:
[...]
> I never understood why people would think the distro should handle unique
> gid / uids. Plus you usually end up running:
> 
> 1) More than one distro.

not in most places

> 2) More than one 'flavor' of a single distro where for whatever reason, uid
> and gid decisions differed (they renumbered, etc.)
> 
> So if you want a consistent GID for a group, store the group name and gid
> in ldap and sync it; do not rely on your distro to do it. IMHO doing so is
> a design error.

I disagree here. Most (enterprise) environments use just one distro. And 
its just very useful if you have sticky UIDs for daemon users for 
example.

One example: You build an apache two-node cluster using DRBD (and 
pacemaker...).  If you happen to install some daemons in random order on 
both nodes you might end up with apache having different UIDs which will 
break things. This is a PITA.

ANd you do not want another central LDAP-Cluster just to have apache 
UIDs in sync ;)

Red Hat for example has unique distro UIDs for many years now.

I would strongly vote for making GLEP27 reality. It makes life easier in 
many places.

-Marc

-- 
0x35A64134 - 8AAC 5F46 83B4 DB70 8317
             3723 296C 6CCA 35A6 4134

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Alexey Shvetsov]

Hi!

Ok. Since there is GLEP27 we should make it reality. To do so i think we 
should
1. Have some list of system uid/gid (on wiki for example). Also we need 
to agree on uid/gid numbers for services
2. Add uid/gid from list to existing ebuilds
3. Make a repoman (or may be eclass) check, that will no allow to commit 
ebuilds with enewuser enewgroup calls with undefined uids
4. Make some script or howto to migrate to determenistic uids/gids from 
1



Robin H. Johnson писал 13-12-2015 23:41:
> On Sun, Dec 13, 2015 at 09:03:51PM +0300, Alexey Shvetsov wrote:
>> Hi all!
>> 
>> We trying to use ldap for users @work, many of our workstations 
>> running
>> binary gentoo based distro called Calculate linux. However if we wanna
>> have wide use of ldap there is a need for determenistic system group
>> gids names and user uids.
>> 
>> Many ebuilds in tree uses enewgroup and enewuser with -1 (aka next
>> available parameter)[1]. However it will be much better to set distro
>> wide deterministic uid and gid for system service name. So for example
>> ldap users may have determenistic groups like video, audio, plugdev,
>> etc..
> GLEP27 was approved for this, however it is barely used.
> 
> Convert the rest of the tree to use it, and then you'll be done, aside
> from the existing mess on user systems.

-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-dev] Determenistic system group and user id

[Alexey Shvetsov]

Hi Alec!

Alec Warner писал 14-12-2015 01:23:
> On Sun, Dec 13, 2015 at 10:03 AM, Alexey Shvetsov <alexxy@gentoo.org>
> wrote:
> 
>> Hi all!
>> 
>> We trying to use ldap for users @work, many of our workstations
>> running binary gentoo based distro called Calculate linux. However
>> if we wanna have wide use of ldap there is a need for determenistic
>> system group gids names and user uids.
>> 
>> Many ebuilds in tree uses enewgroup and enewuser with -1 (aka next
>> available parameter)[1]. However it will be much better to set
>> distro wide deterministic uid and gid for system service name. So
>> for example ldap users may have determenistic groups like video,
>> audio, plugdev, etc..
> 
> So the first question I normally ask here is:
> 
> 1) Why do you need deterministic uid / gid's?

for exmaple plugdev group may have random gid from range 10-1000+ (i 
have some systems when it have gid >1000)
so if you're ldap user want to mount external drive on workstation you 
dont know what gid it should have..

> 2) If you do need deterministic uid / gid's, I would recommend storing
> them all in the same place.
> 
> For example, you typically want a deterministic UID for a user. To
> accomplish this, you add that user to LDAP, give them a UID in LDAP,
> and then either add LDAP to nssswitch or use something like nsscache
> to sync the ldap UID's into the local system.
> 
> 3) If you need deterministic GID's I would recommend storing them all
> in LDAP and syncing the group memberships locally.

Syncing groups localy is major design error if you have more then 10+ 
systems.

> 
> I never understood why people would think the distro should handle
> unique gid / uids. Plus you usually end up running:
> 
> 1) More than one distro.

Its not the case. Most time there only one 'supported' distro by local 
IT stuff.

> 2) More than one 'flavor' of a single distro where for whatever
> reason, uid and gid decisions differed (they renumbered, etc.)
> 
> So if you want a consistent GID for a group, store the group name and
> gid in ldap and sync it; do not rely on your distro to do it. IMHO
> doing so is a design error.
> 
> -A
> 
>> [1] $ egrep '(enewgroup|enewuser)' * -R | awk -F '/' '{print $1 "/"
>> $2}' | grep -v eclass | sort -u | wc -l
>> 443
>> So there not so much gid uids needed
>> 
>> --
>> Best Regards,
>> Alexey 'Alexxy' Shvetsov
>> Best Regards,
>> Alexey 'Alexxy' Shvetsov, PhD
>> Department of Molecular and Radiation Biophysics
>> FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
>> Leningrad region, Gatchina, Russia
>> Gentoo Team Ru
>> Gentoo Linux Dev
>> mailto:alexxyum@gmail.com
>> mailto:alexxy@gentoo.org
>> mailto:alexxy@omrb.pnpi.spb.ru

-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-dev] Use GLEP27!

[Robin H. Johnson]

On Mon, Dec 14, 2015 at 07:49:42AM +0300, Alexey Shvetsov wrote:
> Hi!
> 
> Ok. Since there is GLEP27 we should make it reality. To do so i think we 
> should
> 1. Have some list of system uid/gid (on wiki for example). Also we need 
> to agree on uid/gid numbers for services
This database was already started, prior to GLEP27.
In CVS, you want gentoo-src/eid_database/

> 2. Add uid/gid from list to existing ebuilds

> 3. Make a repoman (or may be eclass) check, that will no allow to commit 
> ebuilds with enewuser enewgroup calls with undefined uids
I think in the original discussion, there were concerns that there were
cases where this was going to be valid. I think this check needs to come
later, after we rule those out. It should however start to warn about
them ASAP.

> 4. Make some script or howto to migrate to determenistic uids/gids from 
Much of the work was implemented for GSOC2006, "Creandus" by
developer pioto.

Cardoe did more work on it later on.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Determenistic system group and user id

[Robin H. Johnson]

On Sun, Dec 13, 2015 at 02:23:29PM -0800, Alec Warner wrote:
> 1) Why do you need deterministic uid / gid's?
> 2) If you do need deterministic uid / gid's, I would recommend storing them
> all in the same place.
They are ALL for system users and groups.

TL;DR: if you're sharing data/config for system users/groups between
multiple systems based on UID/GID (not username), you need consistent
generation.

Data on NFSv[23], with a shared apache/nginx user was one of the
original examples. I agree since then, that the data should NOT be owned
by apache/nginx anymore (NFSv4 also solves the problem).

A much newer example, is let's consider the system group 'plugdev'. It's
one that is created dynamically at the moment.

If I want to put my user in that group LDAP-wide, and have an LDAP
environment, I need to make sure the plugdev GID is the same on all
systems (actually it also varies slightly depending which LDAP group
membership model you're using for NSS data).

> For example, you typically want a deterministic UID for a user. To
> accomplish this, you add that user to LDAP, give them a UID in LDAP, and
> then either add LDAP to nssswitch or use something like nsscache to sync
> the ldap UID's into the local system.
> 
> 3) If you need deterministic GID's I would recommend storing them all in
> LDAP and syncing the group memberships locally.
So you want to define the group twice? Both in LDAP and locally?

> I never understood why people would think the distro should handle unique
> gid / uids. Plus you usually end up running:
> 
> 1) More than one distro.
> 2) More than one 'flavor' of a single distro where for whatever reason, uid
> and gid decisions differed (they renumbered, etc.)
Here's the work LSB did on it, with further references to what more
distros do:
https://github.com/LinuxStandardBase/lsb/blob/master/documents/wip/userNaming.txt

Here's the debian central database for it:
https://anonscm.debian.org/cgit/users/cjwatson/base-passwd.git/tree/README


> So if you want a consistent GID for a group, store the group name and gid
> in ldap and sync it; do not rely on your distro to do it. IMHO doing so is
> a design error.
Which is incompatible with NFSv3.

> > [1] $ egrep '(enewgroup|enewuser)' * -R | awk -F '/' '{print $1 "/" $2}' |
> > grep -v eclass | sort -u | wc -l
> > 443
> > So there not so much gid uids needed
There are definitely entries like these, so be very careful in your counting.
enewgroup $PN
enewuser ${PN} -1 -1 /var/lib/${PN} ${PN}

Based on counting unique tuples of
($CAT/$PN, $ARGS, I count 410+ of each enewgroup and enewuser calls.

$ git grep -e 'enewuser ' -e 'enewgroup ' | \
  sed -r -e 's,/[^/]+:[[:space:]]*,/: ,g' -e 's,#.*,,g' | \
  grep -e ': enew' |sort |uniq

Also watch out for packages that create MULTIPLE users/groups for privilege
separation (qmail is notorious for this).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Use GLEP27!

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]



Dnia 13 grudnia 2015 21:41:02 CET, "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
>On Sun, Dec 13, 2015 at 09:03:51PM +0300, Alexey Shvetsov wrote:
>> Hi all!
>> 
>> We trying to use ldap for users @work, many of our workstations
>running 
>> binary gentoo based distro called Calculate linux. However if we
>wanna 
>> have wide use of ldap there is a need for determenistic system group 
>> gids names and user uids.
>> 
>> Many ebuilds in tree uses enewgroup and enewuser with -1 (aka next 
>> available parameter)[1]. However it will be much better to set distro
>
>> wide deterministic uid and gid for system service name. So for
>example 
>> ldap users may have determenistic groups like video, audio, plugdev, 
>> etc..
>GLEP27 was approved for this, however it is barely used.
>
>Convert the rest of the tree to use it, and then you'll be done, aside
>from the existing mess on user systems.

As far as I can see, this GLEP predates EAPI and does not meet modern standards. It needs to be updated or killed with fire.

For a start, relation to EAPI needs to be defined. This will likely require both profiles and ebuilds to use the new EAPI.

Also, the contents of 'backwards compatibility' section are unacceptable. But that's probably going to be covered by EAPI.

The spec itself is hard to follow, though the idea seems simple. It makes me wonder if we aren't missing something important there.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1753 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Doug Goldstein]

[-- Attachment #1: Type: text/plain, Size: 1497 bytes --]

On 12/14/15 12:06 AM, Robin H. Johnson wrote:
> On Mon, Dec 14, 2015 at 07:49:42AM +0300, Alexey Shvetsov wrote:
>> Hi!
>>
>> Ok. Since there is GLEP27 we should make it reality. To do so i think we 
>> should
>> 1. Have some list of system uid/gid (on wiki for example). Also we need 
>> to agree on uid/gid numbers for services
> This database was already started, prior to GLEP27.
> In CVS, you want gentoo-src/eid_database/
> 
>> 2. Add uid/gid from list to existing ebuilds
> 
>> 3. Make a repoman (or may be eclass) check, that will no allow to commit 
>> ebuilds with enewuser enewgroup calls with undefined uids
> I think in the original discussion, there were concerns that there were
> cases where this was going to be valid. I think this check needs to come
> later, after we rule those out. It should however start to warn about
> them ASAP.
> 
>> 4. Make some script or howto to migrate to determenistic uids/gids from 
> Much of the work was implemented for GSOC2006, "Creandus" by
> developer pioto.
> 
> Cardoe did more work on it later on.
> 

I'll try to find what I did but at one point I had the database of
uid/gid updated to include everything in the tree. I had some patches
for enewuser/enewgroup to not allow them to do anything unless the ids
were in the database.

Sadly, its been a long long time. But I still would love to see this
happen. There just wasn't much interest from everyone in making this happen.

-- 
Doug Goldstein


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Doug Goldstein]

[-- Attachment #1: Type: text/plain, Size: 1708 bytes --]

On 12/14/15 3:22 AM, Michał Górny wrote:
> 
> 
> Dnia 13 grudnia 2015 21:41:02 CET, "Robin H. Johnson"
> <robbat2@gentoo.org> napisał(a):
>>On Sun, Dec 13, 2015 at 09:03:51PM +0300, Alexey Shvetsov wrote:
>>> Hi all!
>>>
>>> We trying to use ldap for users @work, many of our workstations
>>running
>>> binary gentoo based distro called Calculate linux. However if we
>>wanna
>>> have wide use of ldap there is a need for determenistic system group
>>> gids names and user uids.
>>>
>>> Many ebuilds in tree uses enewgroup and enewuser with -1 (aka next
>>> available parameter)[1]. However it will be much better to set distro
>>
>>> wide deterministic uid and gid for system service name. So for
>>example
>>> ldap users may have determenistic groups like video, audio, plugdev,
>>> etc..
>>GLEP27 was approved for this, however it is barely used.
>>
>>Convert the rest of the tree to use it, and then you'll be done, aside
>>from the existing mess on user systems.
> 
> As far as I can see, this GLEP predates EAPI and does not meet modern
> standards. It needs to be updated or killed with fire.
> 
> For a start, relation to EAPI needs to be defined. This will likely
> require both profiles and ebuilds to use the new EAPI.
> 
> Also, the contents of 'backwards compatibility' section are
> unacceptable. But that's probably going to be covered by EAPI.
> 
> The spec itself is hard to follow, though the idea seems simple. It
> makes me wonder if we aren't missing something important there.
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.


Don't kill it. The best way forward would be to update it.

-- 
Doug Goldstein


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

Re: [gentoo-dev] Determenistic system group and user id

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 4281 bytes --]

On Sun, Dec 13, 2015 at 10:39 PM, Robin H. Johnson <robbat2@gentoo.org>
wrote:

> On Sun, Dec 13, 2015 at 02:23:29PM -0800, Alec Warner wrote:
> > 1) Why do you need deterministic uid / gid's?
> > 2) If you do need deterministic uid / gid's, I would recommend storing
> them
> > all in the same place.
> They are ALL for system users and groups.
>

What does that mean though? What is a System user or a System group?
Why should you not store them in the user / group directory?


>
> TL;DR: if you're sharing data/config for system users/groups between
> multiple systems based on UID/GID (not username), you need consistent
> generation.
>

> Data on NFSv[23], with a shared apache/nginx user was one of the
> original examples. I agree since then, that the data should NOT be owned
> by apache/nginx anymore (NFSv4 also solves the problem).
>

This is certainly one workaround (you generate a shard user / role user
that is the same uid / gid on every machine and you make that own the data
/ run the service.) But you might as well just add apache to LDAP and call
it a day?


>
> A much newer example, is let's consider the system group 'plugdev'. It's
> one that is created dynamically at the moment.
>
> If I want to put my user in that group LDAP-wide, and have an LDAP
> environment, I need to make sure the plugdev GID is the same on all
> systems (actually it also varies slightly depending which LDAP group
> membership model you're using for NSS data).
>
> > For example, you typically want a deterministic UID for a user. To
> > accomplish this, you add that user to LDAP, give them a UID in LDAP, and
> > then either add LDAP to nssswitch or use something like nsscache to sync
> > the ldap UID's into the local system.
> >
> > 3) If you need deterministic GID's I would recommend storing them all in
> > LDAP and syncing the group memberships locally.
> So you want to define the group twice? Both in LDAP and locally?
>

Sorry, when I said sync I meant nsscache or similar. I understand why that
would be confusing ;)


>
> > I never understood why people would think the distro should handle unique
> > gid / uids. Plus you usually end up running:
> >
> > 1) More than one distro.
> > 2) More than one 'flavor' of a single distro where for whatever reason,
> uid
> > and gid decisions differed (they renumbered, etc.)
> Here's the work LSB did on it, with further references to what more
> distros do:
>
> https://github.com/LinuxStandardBase/lsb/blob/master/documents/wip/userNaming.txt
>
> Here's the debian central database for it:
> https://anonscm.debian.org/cgit/users/cjwatson/base-passwd.git/tree/README
>
>
I'm not opposed to this effort, simply that getting everyone to conform is
tricky and in complex environments you will inevitably end up in situations
outside of the scope of the effort. I prefer practicality ;)


> > So if you want a consistent GID for a group, store the group name and gid
> > in ldap and sync it; do not rely on your distro to do it. IMHO doing so
> is
> > a design error.
> Which is incompatible with NFSv3.
>

How so? (to be clear, most of my experience is with sec=krb5 nfsv3, and I
haven't had to futz with it in a number of years.)
But IIRC nfsv3 sends the uid / gid numbers in the protocol; you can't send
names. Which is why you need consistent uid / gids. But I do not quite
grasp how using LDAP to achieve this breaks nfsv3.


>
> > > [1] $ egrep '(enewgroup|enewuser)' * -R | awk -F '/' '{print $1 "/"
> $2}' |
> > > grep -v eclass | sort -u | wc -l
> > > 443
> > > So there not so much gid uids needed
> There are definitely entries like these, so be very careful in your
> counting.
> enewgroup $PN
> enewuser ${PN} -1 -1 /var/lib/${PN} ${PN}
>
> Based on counting unique tuples of
> ($CAT/$PN, $ARGS, I count 410+ of each enewgroup and enewuser calls.
>
> $ git grep -e 'enewuser ' -e 'enewgroup ' | \
>   sed -r -e 's,/[^/]+:[[:space:]]*,/: ,g' -e 's,#.*,,g' | \
>   grep -e ': enew' |sort |uniq
>
> Also watch out for packages that create MULTIPLE users/groups for privilege
> separation (qmail is notorious for this).
>
> --
> Robin Hugh Johnson
> Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
>
>

[-- Attachment #2: Type: text/html, Size: 6806 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1670 bytes --]

>>>>> On Mon, 14 Dec 2015, Michał Górny wrote:

> As far as I can see, this GLEP predates EAPI and does not meet
> modern standards. It needs to be updated or killed with fire.

> For a start, relation to EAPI needs to be defined. This will likely
> require both profiles and ebuilds to use the new EAPI.

Certainly this is true for ebuilds. We could introduce EUSERS and
EGROUPS variables as well as FEATURES=noautoaccts in EAPI 7.

I'd rather avoid bumping all profiles to EAPI 7, though. From a
pragmatic point of view, I think nothing bad would happen if we would
add user and group definition files to an existing (EAPI 5) profile,
as old package managers would simple ignore these files.

> Also, the contents of 'backwards compatibility' section are
> unacceptable. But that's probably going to be covered by EAPI.

> The spec itself is hard to follow, though the idea seems simple. 
> It makes me wonder if we aren't missing something important there.

The spec seems incomplete. I cannot find a description of the user and
group files' format. (But in fact, there is a standard format which
suggests itself, namely that of the passwd(5) and group(5) files.)

Also having whole directory trees seems wasteful and doesn't fit so
well into the existing design of profiles. It might be simpler to put
"user" (or "passwd") and "group" files directly in the profile.
(If directories are really needed, we could use the scheme foreseen
in [1] for package.* and use.* files.)

Also a mechanism how a subprofile could undefine a user or group
defined in its parent seems to be missing.

Ulrich


[1] https://bugs.gentoo.org/282296

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

Ulrich Mueller wrote:
> (If directories are really needed, we could use the scheme foreseen
> in [1] for package.* and use.* files.)

So package.{users,group} ?


> Also a mechanism how a subprofile could undefine a user or group
> defined in its parent seems to be missing.

Maybe set the id to -1 ?


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

On 14 Dec 2015 21:22, Ulrich Mueller wrote:
> The spec seems incomplete. I cannot find a description of the user and
> group files' format. (But in fact, there is a standard format which
> suggests itself, namely that of the passwd(5) and group(5) files.)

i recall going with xml at the time, but i can't find reference to it.

> Also having whole directory trees seems wasteful and doesn't fit so
> well into the existing design of profiles. It might be simpler to put
> "user" (or "passwd") and "group" files directly in the profile.
> (If directories are really needed, we could use the scheme foreseen
> in [1] for package.* and use.* files.)

we implemented this GLEP in Chromium OS and have been using it for a while:
https://chromium.googlesource.com/chromiumos/overlays/eclass-overlay/+/master

having a directory of files is way more user friendly imo and allows for
a format that is easier to read.  /etc/passwd and /etc/group format are
not that easy to scan and aren't portable.

> Also a mechanism how a subprofile could undefine a user or group
> defined in its parent seems to be missing.

what exactly do you mean by that ?  you want to make it so attempts to
use the account yield an undefined error ?  or you want to have it so
a child can revert back to an earlier definition ?
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1662 bytes --]

>>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:

> On 14 Dec 2015 21:22, Ulrich Mueller wrote:
>> The spec seems incomplete. I cannot find a description of the user and
>> group files' format. (But in fact, there is a standard format which
>> suggests itself, namely that of the passwd(5) and group(5) files.)

> i recall going with xml at the time, but i can't find reference to it.

So the package manager would have to parse XML? We have tried to avoid
that, so far.

>> Also having whole directory trees seems wasteful and doesn't fit so
>> well into the existing design of profiles. It might be simpler to put
>> "user" (or "passwd") and "group" files directly in the profile.
>> (If directories are really needed, we could use the scheme foreseen
>> in [1] for package.* and use.* files.)

> we implemented this GLEP in Chromium OS and have been using it for a while:
> https://chromium.googlesource.com/chromiumos/overlays/eclass-overlay/+/master

> having a directory of files is way more user friendly imo and allows for
> a format that is easier to read.  /etc/passwd and /etc/group format are
> not that easy to scan and aren't portable.

As we most likely will introduce profile file directories in EAPI 7
(see bug 282296), I'd rather use the same scheme for the user and
group files, but not something different.

>> Also a mechanism how a subprofile could undefine a user or group
>> defined in its parent seems to be missing.

> what exactly do you mean by that ?  you want to make it so attempts to
> use the account yield an undefined error ?  or you want to have it so
> a child can revert back to an earlier definition ?

The former.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2788 bytes --]

On 15 Dec 2015 07:28, Ulrich Mueller wrote:
> >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:
> 
> > On 14 Dec 2015 21:22, Ulrich Mueller wrote:
> >> The spec seems incomplete. I cannot find a description of the user and
> >> group files' format. (But in fact, there is a standard format which
> >> suggests itself, namely that of the passwd(5) and group(5) files.)
> 
> > i recall going with xml at the time, but i can't find reference to it.
> 
> So the package manager would have to parse XML? We have tried to avoid
> that, so far.

not entirely accurate considering we have metadata.xml

> >> Also having whole directory trees seems wasteful and doesn't fit so
> >> well into the existing design of profiles. It might be simpler to put
> >> "user" (or "passwd") and "group" files directly in the profile.
> >> (If directories are really needed, we could use the scheme foreseen
> >> in [1] for package.* and use.* files.)
> 
> > we implemented this GLEP in Chromium OS and have been using it for a while:
> > https://chromium.googlesource.com/chromiumos/overlays/eclass-overlay/+/master
> 
> > having a directory of files is way more user friendly imo and allows for
> > a format that is easier to read.  /etc/passwd and /etc/group format are
> > not that easy to scan and aren't portable.
> 
> As we most likely will introduce profile file directories in EAPI 7
> (see bug 282296), I'd rather use the same scheme for the user and
> group files, but not something different.

a flat text file akin to /etc/passwd is not readable.  xml is readable.

a markdown like format would work -- easy to parse by machines & humans
and is a single stackable file.
user:ntp
<whitespace>uid:203
<whitespace>gid:203
user:man
<whitespace>uid:13
<whitespace>gid:13
(using : as delimiter since that's what *NIX uses in /etc/passwd)

the main one would grow probably to about 2000+ lines (~400 users in the
tree and each entry takes ~4 lines assuming we enforce eliding of defaults)
which isn't that terrible.  CrOS has a python script already for validating
the db consistency.

> >> Also a mechanism how a subprofile could undefine a user or group
> >> defined in its parent seems to be missing.
> 
> > what exactly do you mean by that ?  you want to make it so attempts to
> > use the account yield an undefined error ?  or you want to have it so
> > a child can revert back to an earlier definition ?
> 
> The former.

we already handle this in CrOS by explicitly including a flag in the file:
defunct:true

thus the PM need not care about the status when locating files or otherwise
include any special handling.  if the last file you hit is marked as defunct,
then enew{user,group} will throw an error when they're called.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Tue, 15 Dec 2015 09:19:36 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> On 15 Dec 2015 07:28, Ulrich Mueller wrote:
> > >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:  
> >   
> > > On 14 Dec 2015 21:22, Ulrich Mueller wrote:  
> > >> The spec seems incomplete. I cannot find a description of the user and
> > >> group files' format. (But in fact, there is a standard format which
> > >> suggests itself, namely that of the passwd(5) and group(5) files.)  
> >   
> > > i recall going with xml at the time, but i can't find reference to it.  
> > 
> > So the package manager would have to parse XML? We have tried to avoid
> > that, so far.  
> 
> not entirely accurate considering we have metadata.xml

Not entirely accurate considering the contents of metadata.xml are not
necessary to build and install packages.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

>>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:

> a flat text file akin to /etc/passwd is not readable.  xml is readable.

ESR's case study about the password file format seems to disagree:
http://www.catb.org/esr/writings/taoup/html/ch05s01.html#id2901332

I think the name:password:uid:gid:gecos:directory:shell format is
readable well enough for human eyes. Certainly it is machine readable;
even with standard tools like fgetpwent(3) (or its equivalent in other
programming languages).

> a markdown like format would work -- easy to parse by machines & humans
> and is a single stackable file.

Reinventing the wheel?

> user:ntp
> <whitespace>uid:203
> <whitespace>gid:203
> user:man
> <whitespace>uid:13
> <whitespace>gid:13
> (using : as delimiter since that's what *NIX uses in /etc/passwd)

> the main one would grow probably to about 2000+ lines (~400 users in
> the tree and each entry takes ~4 lines assuming we enforce eliding
> of defaults) which isn't that terrible.

To me this looks like a lot of added redundancy for little (if any)
benefit.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

On Tue, 15 Dec 2015 15:56:34 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:  
> 
> > a flat text file akin to /etc/passwd is not readable.  xml is readable.  
> 
> ESR's case study about the password file format seems to disagree:
> http://www.catb.org/esr/writings/taoup/html/ch05s01.html#id2901332
> 
> I think the name:password:uid:gid:gecos:directory:shell format is
> readable well enough for human eyes. Certainly it is machine readable;
> even with standard tools like fgetpwent(3) (or its equivalent in other
> programming languages).

Alternatively we could go for whitespace-separated, like fstab. Column
aligning even more.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

On 15 Dec 2015 15:33, Michał Górny wrote:
> On Tue, 15 Dec 2015 09:19:36 -0500 Mike Frysinger wrote:
> > On 15 Dec 2015 07:28, Ulrich Mueller wrote:
> > > >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:  
> > >   
> > > > On 14 Dec 2015 21:22, Ulrich Mueller wrote:  
> > > >> The spec seems incomplete. I cannot find a description of the user and
> > > >> group files' format. (But in fact, there is a standard format which
> > > >> suggests itself, namely that of the passwd(5) and group(5) files.)  
> > >   
> > > > i recall going with xml at the time, but i can't find reference to it.  
> > > 
> > > So the package manager would have to parse XML? We have tried to avoid
> > > that, so far.  
> > 
> > not entirely accurate considering we have metadata.xml
> 
> Not entirely accurate considering the contents of metadata.xml are not
> necessary to build and install packages.

read the context.  your statement is fairly obvious.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On 15 Dec 2015 16:00, Michał Górny wrote:
> On Tue, 15 Dec 2015 15:56:34 +0100 Ulrich Mueller wrote:
> > >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:  
> > 
> > > a flat text file akin to /etc/passwd is not readable.  xml is readable.  
> > 
> > ESR's case study about the password file format seems to disagree:
> > http://www.catb.org/esr/writings/taoup/html/ch05s01.html#id2901332
> > 
> > I think the name:password:uid:gid:gecos:directory:shell format is
> > readable well enough for human eyes. Certainly it is machine readable;
> > even with standard tools like fgetpwent(3) (or its equivalent in other
> > programming languages).
> 
> Alternatively we could go for whitespace-separated, like fstab. Column
> aligning even more.

we cannot because whitespace is valid, and we use it already in fields
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1002 bytes --]

On 15 Dec 2015 15:56, Ulrich Mueller wrote:
> >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:
> 
> > a flat text file akin to /etc/passwd is not readable.  xml is readable.
> 
> ESR's case study about the password file format seems to disagree:
> http://www.catb.org/esr/writings/taoup/html/ch05s01.html#id2901332

i don't generally find anything ESR has to say useful, and his code even
less so

> I think the name:password:uid:gid:gecos:directory:shell format is
> readable well enough for human eyes.

having looked at way too many of these, i disagree.  it's harder (imo) to
scan due to the lack of alignment, it's more of a pita to deal with defaults,
and it doesn't permit for additional metadata like i already quoted (the
defunct field).

> Certainly it is machine readable;
> even with standard tools like fgetpwent(3) (or its equivalent in other
> programming languages).

again, not portable.  the format proposed is also trivial to parse by
python and awk.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Anthony G. Basile]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/15/15 10:08 AM, Mike Frysinger wrote:
> On 15 Dec 2015 15:33, Michał Górny wrote:
>> On Tue, 15 Dec 2015 09:19:36 -0500 Mike Frysinger wrote:
>>> On 15 Dec 2015 07:28, Ulrich Mueller wrote:
>>>>>>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote: 
>>>>  
>>>>> On 14 Dec 2015 21:22, Ulrich Mueller wrote: 
>>>>>> The spec seems incomplete. I cannot find a description of the
user and
>>>>>> group files' format. (But in fact, there is a standard format which
>>>>>> suggests itself, namely that of the passwd(5) and group(5) files.) 
>>>>  
>>>>> i recall going with xml at the time, but i can't find reference to
it. 
>>>>
>>>> So the package manager would have to parse XML? We have tried to avoid
>>>> that, so far. 
>>>
>>> not entirely accurate considering we have metadata.xml
>>
>> Not entirely accurate considering the contents of metadata.xml are not
>> necessary to build and install packages.
>
> read the context.  your statement is fairly obvious.
> -mike
i looked through portage code and we do have xml parsing sprinkled
throughout, mostly in repoman for obvious reasons.  why are we trying to
avoid xml?  to be honest i don't have strong feelings about either the
flat file (a la /etc/passwd) or xml.

i'm interested in this because the hardened-sources kernel does make use
of some special uid/gids.  gengor pointed me to glep 27 and suggested
that i implement it but i wasn't that interested.  still, it would clean
up the hardened.

- -- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iEYEARECAAYFAlZwMJkACgkQl5yvQNBFVTV07gCdHOlrLeoLOdwbEDTeN9TlTM1m
to8AoItHHwBbBvZDp1JkJw9ZO6sm6Sr3
=hYwF
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Use GLEP27!

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2944 bytes --]

On Tue, Dec 15, 2015 at 6:19 AM, Mike Frysinger <vapier@gentoo.org> wrote:

> On 15 Dec 2015 07:28, Ulrich Mueller wrote:
> > >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:
> >
> > > On 14 Dec 2015 21:22, Ulrich Mueller wrote:
> > >> The spec seems incomplete. I cannot find a description of the user and
> > >> group files' format. (But in fact, there is a standard format which
> > >> suggests itself, namely that of the passwd(5) and group(5) files.)
> >
> > > i recall going with xml at the time, but i can't find reference to it.
> >
> > So the package manager would have to parse XML? We have tried to avoid
> > that, so far.
>
> not entirely accurate considering we have metadata.xml
>
> > >> Also having whole directory trees seems wasteful and doesn't fit so
> > >> well into the existing design of profiles. It might be simpler to put
> > >> "user" (or "passwd") and "group" files directly in the profile.
> > >> (If directories are really needed, we could use the scheme foreseen
> > >> in [1] for package.* and use.* files.)
> >
> > > we implemented this GLEP in Chromium OS and have been using it for a
> while:
> > >
> https://chromium.googlesource.com/chromiumos/overlays/eclass-overlay/+/master
> >
> > > having a directory of files is way more user friendly imo and allows
> for
> > > a format that is easier to read.  /etc/passwd and /etc/group format are
> > > not that easy to scan and aren't portable.
> >
> > As we most likely will introduce profile file directories in EAPI 7
> > (see bug 282296), I'd rather use the same scheme for the user and
> > group files, but not something different.
>
> a flat text file akin to /etc/passwd is not readable.  xml is readable.
>

u trollin' bro?


>
> a markdown like format would work -- easy to parse by machines & humans
> and is a single stackable file.
> user:ntp
> <whitespace>uid:203
> <whitespace>gid:203
> user:man
> <whitespace>uid:13
> <whitespace>gid:13
> (using : as delimiter since that's what *NIX uses in /etc/passwd)
>
> the main one would grow probably to about 2000+ lines (~400 users in the
> tree and each entry takes ~4 lines assuming we enforce eliding of defaults)
> which isn't that terrible.  CrOS has a python script already for validating
> the db consistency.
>
> > >> Also a mechanism how a subprofile could undefine a user or group
> > >> defined in its parent seems to be missing.
> >
> > > what exactly do you mean by that ?  you want to make it so attempts to
> > > use the account yield an undefined error ?  or you want to have it so
> > > a child can revert back to an earlier definition ?
> >
> > The former.
>
> we already handle this in CrOS by explicitly including a flag in the file:
> defunct:true
>
> thus the PM need not care about the status when locating files or otherwise
> include any special handling.  if the last file you hit is marked as
> defunct,
> then enew{user,group} will throw an error when they're called.
> -mike
>

[-- Attachment #2: Type: text/html, Size: 4141 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

On 15 Dec 2015 10:24, Anthony G. Basile wrote:
> i looked through portage code and we do have xml parsing sprinkled
> throughout, mostly in repoman for obvious reasons.  why are we trying to
> avoid xml?  to be honest i don't have strong feelings about either the
> flat file (a la /etc/passwd) or xml.
> 
> i'm interested in this because the hardened-sources kernel does make use
> of some special uid/gids.  gengor pointed me to glep 27 and suggested
> that i implement it but i wasn't that interested.  still, it would clean
> up the hardened.

i don't think XML adds that much over RST, and are trivial to parse, both
on an ad-hoc basis, as well as python has modules to enable it.  i think
most people would agree RST or flat text files are less verbose than XML
w/out really losing that much (if anything).  at the time i was thinking
XML purely because we were much more of an XML shop at the time (we had
guidexml and metadata.xml), but with the rise of wiki/github/glep and the
fall of guidexml, the project has moved on.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

On 15 Dec 2015 10:35, Alec Warner wrote:
> On Tue, Dec 15, 2015 at 6:19 AM, Mike Frysinger wrote:
> > a flat text file akin to /etc/passwd is not readable.  xml is readable.
> 
> u trollin' bro?

in this domain, it's harder to screw up xml and not notice (either human or
machine) whereas it's pretty easy to add/omit an extra : and not notice as
the file will be valid from a parsing pov.

quick, off the top of your head, which of these are correct ?
and tell me which field exactly each one is supposed to go to ?
and then reasonably expect new comers to read this and not choke ?
lp:x:4:7::/var/spool/lpd:/bin/false
lp:x:4:7:/var/spool/lpd:/bin/false
lp:x:4:7:/var/spool/lpd::/bin/false
lp:x:4:7::lpd:/bin/false
lp:x:4:7:lpd:/bin/false
lp:x:4:7:lpd::/bin/false

i consider this format simply a different form of hazing except we all lose ;)
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]

On 15 Dec 2015 15:56, Ulrich Mueller wrote:
> >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:
> 
> > a flat text file akin to /etc/passwd is not readable.  xml is readable.
> 
> ESR's case study about the password file format seems to disagree:
> http://www.catb.org/esr/writings/taoup/html/ch05s01.html#id2901332

because you cited it, i read it anyways.  that document is about how text
formats should be preferred over binary formats because they do not require
custom tools to modify/update, and because it's easier for binary formats
to screw themselves over from a portability/extensible pov.  it does not
champion the passwd format all by itself, and even says that it's a bit
rigid, and you should consider tagged formats if you want something more.
which we do.

see also the example i posted to Alec as why the format is hostile to devs
whereas my simple RST proposal has none of these issues.

we also know the format works because it's been in use by CrOS for two
years now, including:
(1) profile stacking/overrides
(2) marking users as "dead"
(3) tools to verify/check consistency at rest
(4) parallel installs
(5) correct handling of ROOT and SYSROOT
although 4/5 were written only with Linux/flat files in mind, so they do
not support other OS's or account formats (e.g. ldap/etc...).  i imagine
we'll need to merge with the existing user.eclass logic rather than drop
in replace.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

>>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:

> On 15 Dec 2015 15:56, Ulrich Mueller wrote:
>> ESR's case study about the password file format seems to disagree:
>> http://www.catb.org/esr/writings/taoup/html/ch05s01.html#id2901332

> because you cited it, i read it anyways.  that document is about how text
> formats should be preferred over binary formats because they do not require
> custom tools to modify/update, and because it's easier for binary formats
> to screw themselves over from a portability/extensible pov.  it does not
> champion the passwd format all by itself, and even says that it's a bit
> rigid, and you should consider tagged formats if you want something more.
> which we do.

> see also the example i posted to Alec as why the format is hostile to devs
> whereas my simple RST proposal has none of these issues.

Whatever the format will be, the more important question is where this
would be implemented:

- In the package manager, with user and group definition in profiles.
  This seems to be what GLEP 27 suggests, and as far as I can see, it
  would require an EAPI bump. Certainly doable, but last time we
  bumped profiles to a new EAPI we had a rather long transition
  period.

- In user.eclass, which could be extended to use the EUSERS and
  EGROUPS variables defined in ebuilds. The problem is, where would
  one store the user and group definitions then? Profiles cannot be
  accessed from an eclass.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1647 bytes --]

On 15 Dec 2015 22:35, Ulrich Mueller wrote:
> Whatever the format will be, the more important question is where this
> would be implemented:
> 
> - In the package manager, with user and group definition in profiles.
>   This seems to be what GLEP 27 suggests, and as far as I can see, it
>   would require an EAPI bump. Certainly doable, but last time we
>   bumped profiles to a new EAPI we had a rather long transition
>   period.
> 
> - In user.eclass, which could be extended to use the EUSERS and
>   EGROUPS variables defined in ebuilds. The problem is, where would
>   one store the user and group definitions then? Profiles cannot be
>   accessed from an eclass.

long term, i think profiles are better to hold the db as it provides for
clean stacking and is trivial for site-specific extension/control, as well
as image builders via something like catalyst.

short/mid term, i was thinking of writing a new package that holds the db
and tools to query/manage it.  user.eclass would DEPEND on it and ask it
for details, perhaps even doing the actual fs updates (the bash code here
is not pretty wrt locks and python would be much nicer).  that tool could
even search additional site paths (like /usr/local) to locate overrides.

the API to ebuilds/eclasses would be unchanged.  in CrOS, we only look at
the first argument (the user/group name) and load all other details from
the db.  we could seamlessly migrate over existing ebuilds by opting in to
this simpler form.

maybe the short/mid term solution is enough to not get into profile mess
even if i think it's the correct data storage location.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Anthony G. Basile]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/15/15 4:55 PM, Mike Frysinger wrote:
> On 15 Dec 2015 22:35, Ulrich Mueller wrote:
>> Whatever the format will be, the more important question is where this
>> would be implemented:
>>
>> - In the package manager, with user and group definition in profiles.
>>   This seems to be what GLEP 27 suggests, and as far as I can see, it
>>   would require an EAPI bump. Certainly doable, but last time we
>>   bumped profiles to a new EAPI we had a rather long transition
>>   period.
>>
>> - In user.eclass, which could be extended to use the EUSERS and
>>   EGROUPS variables defined in ebuilds. The problem is, where would
>>   one store the user and group definitions then? Profiles cannot be
>>   accessed from an eclass.
>
> long term, i think profiles are better to hold the db as it provides for
> clean stacking and is trivial for site-specific extension/control, as well
> as image builders via something like catalyst.
>
> short/mid term, i was thinking of writing a new package that holds the db
> and tools to query/manage it.  user.eclass would DEPEND on it and ask it
> for details, perhaps even doing the actual fs updates (the bash code here
> is not pretty wrt locks and python would be much nicer).  that tool could
> even search additional site paths (like /usr/local) to locate overrides.

how do we get our own uid/gid's in there for our packages?  just open a
bug against the new package?

>
>
> the API to ebuilds/eclasses would be unchanged.  in CrOS, we only look at
> the first argument (the user/group name) and load all other details from
> the db.  we could seamlessly migrate over existing ebuilds by opting in to
> this simpler form.
>
> maybe the short/mid term solution is enough to not get into profile mess
> even if i think it's the correct data storage location.
> -mike


- -- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iEYEARECAAYFAlZwvQcACgkQl5yvQNBFVTVEaACfT1nMMFXsPyqM0u4rGDHJP29/
pFkAn0XOcHTmVAAp9K9opvWT9isuMOxp
=xPUD
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2098 bytes --]

On 15 Dec 2015 20:23, Anthony G. Basile wrote:
> On 12/15/15 4:55 PM, Mike Frysinger wrote:
> > On 15 Dec 2015 22:35, Ulrich Mueller wrote:
> >> Whatever the format will be, the more important question is where this
> >> would be implemented:
> >>
> >> - In the package manager, with user and group definition in profiles.
> >>   This seems to be what GLEP 27 suggests, and as far as I can see, it
> >>   would require an EAPI bump. Certainly doable, but last time we
> >>   bumped profiles to a new EAPI we had a rather long transition
> >>   period.
> >>
> >> - In user.eclass, which could be extended to use the EUSERS and
> >>   EGROUPS variables defined in ebuilds. The problem is, where would
> >>   one store the user and group definitions then? Profiles cannot be
> >>   accessed from an eclass.
> >
> > long term, i think profiles are better to hold the db as it provides for
> > clean stacking and is trivial for site-specific extension/control, as well
> > as image builders via something like catalyst.
> >
> > short/mid term, i was thinking of writing a new package that holds the db
> > and tools to query/manage it.  user.eclass would DEPEND on it and ask it
> > for details, perhaps even doing the actual fs updates (the bash code here
> > is not pretty wrt locks and python would be much nicer).  that tool could
> > even search additional site paths (like /usr/local) to locate overrides.
> 
> how do we get our own uid/gid's in there for our packages?  just open a
> bug against the new package?

i would open the git repo to all devs and just let anyone push directly
and roll releases.  maybe just push a tag and that's what the ebuild would
hit.  no need to be gate keepers here since we don't today w/ebuilds and
calls to enew{user,group}.

the question is how to handle the tools.  i'm thinking we should have two
distinct packages so the db could be completely overridden by overlays.

the tools could be in the same repo or a sep one.  i don't feel strongly
either way as we'd just mark base-system@ as the maintaining herd.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

>>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:

> On 15 Dec 2015 20:23, Anthony G. Basile wrote:
>> > short/mid term, i was thinking of writing a new package that
>> > holds the db and tools to query/manage it. user.eclass would
>> > DEPEND on it and ask it for details, perhaps even doing the
>> > actual fs updates (the bash code here is not pretty wrt locks and
>> > python would be much nicer). that tool could even search
>> > additional site paths (like /usr/local) to locate overrides.
>> 
>> how do we get our own uid/gid's in there for our packages? just
>> open a bug against the new package?

> i would open the git repo to all devs and just let anyone push
> directly and roll releases. maybe just push a tag and that's what
> the ebuild would hit. no need to be gate keepers here since we don't
> today w/ebuilds and calls to enew{user,group}.

So if I want to add a new user or group, I would have to commit it to
the repo of that package, then roll a new release, create an ebuild
for that release, add that version to DEPEND of my own package's
ebuild, and only then my ebuild could refer to the new user? And
eventually, I'd have to take care of stabilising things, too? That
looks like a cumbersome procedure, even if it is only intended as a
stopgap solution until we can move things to a profile.

Couldn't we add the user/group db to a subdir of the eclass dir
instead, so that user.eclass could access it there?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Use GLEP27!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1857 bytes --]

On 17 Dec 2015 11:57, Ulrich Mueller wrote:
> >>>>> On Tue, 15 Dec 2015, Mike Frysinger wrote:
> > On 15 Dec 2015 20:23, Anthony G. Basile wrote:
> >> > short/mid term, i was thinking of writing a new package that
> >> > holds the db and tools to query/manage it. user.eclass would
> >> > DEPEND on it and ask it for details, perhaps even doing the
> >> > actual fs updates (the bash code here is not pretty wrt locks and
> >> > python would be much nicer). that tool could even search
> >> > additional site paths (like /usr/local) to locate overrides.
> >> 
> >> how do we get our own uid/gid's in there for our packages? just
> >> open a bug against the new package?
> 
> > i would open the git repo to all devs and just let anyone push
> > directly and roll releases. maybe just push a tag and that's what
> > the ebuild would hit. no need to be gate keepers here since we don't
> > today w/ebuilds and calls to enew{user,group}.
> 
> So if I want to add a new user or group, I would have to commit it to
> the repo of that package, then roll a new release, create an ebuild
> for that release, add that version to DEPEND of my own package's
> ebuild, and only then my ebuild could refer to the new user? And
> eventually, I'd have to take care of stabilising things, too? That
> looks like a cumbersome procedure, even if it is only intended as a
> stopgap solution until we can move things to a profile.

yes, that's what you'd have to do.  considering packages themselves have
to go through stabilization, that particular aspect isn't a big deal.  or
we just auto-stable it.

> Couldn't we add the user/group db to a subdir of the eclass dir
> instead, so that user.eclass could access it there?

people get whiny when we leverage eclass/ for storage.  but we do it
already, so it makes no difference to me.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]