[gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Mike Kelly]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello all,

As some of you may know, I spent this past summer working on an
implementation of GLEP 27[1]. While my scripts aren't quite ready for
production use yet, I wanna get the ball rolling by asking devs who
maintain packages that use the current enewuser() and enewgroup()
functions from eutils.eclass to document their desired account settings
in one central location.

I have a list of all packages in the tree that currently use either of
those functions[2]. If you maintain one of these packages, I'd
especially appreciate your feedback.

The proposed storage format[3] is what my scripts (creandus, formerly
known as dynusers) currently use. To me, the format seems simple and
sane enough, but I would definitely appreciate any and all feedback on
it, since it's much easier for me to change it now than to change it
later.

Summarized, the format is:

 For each profile dir (e.g. profiles/base, profiles/default-linux, etc),
 a new subdirectory, called accounts is created as necessary. Inside
 that is a file called defaults, containing default uid/gid ranges,
 shells, etc for the given profile. Also, there are two directories,
 user/ and group/, which contain files named after the users and groups
 to be added. Those files contain more specific uid/gid info, etc.
 
 All the files are handled like other files in cascading profiles. Each
 line in the file is either a shell-style comment, or of the form:
 "key: value". The keys are: uid, shell, home, groups, comment, and gid.

The main point of this thread is to get lots of feedback and, hopefully,
acceptance of this proposed format. As this is basically what is already
outlined in GLEP 27, I don't think a new GLEP is in order, but if others
do, I'll draft one.

The main webpage for my Summer of Code project[4] has links to more
information about the project in general, and there are some posts on my
blog[5] regarding it.

[1] http://www.gentoo.org/proj/en/glep/glep-0027.html
[2] http://dev.gentoo.org/~pioto/creandus/enewusergroup-pkgnames.txt
[3] http://dev.gentoo.org/~pioto/creandus/doc/datafiles.html
[4] http://soc.pioto.org/
[5] http://blog.pioto.org/category/dynusers/

- -- 
Mike Kelly
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFIdHzokMzJ47YCzoRAoBYAJ0drAJrxAMx3p9g5jrnTwQu9mePaQCggV6Y
SpIBoDGFUJ6J0xBNdWANqtc=
=2P1l
-----END PGP SIGNATURE-----

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Alec Warner]

Mike Kelly wrote:
> Summarized, the format is:
> 
>  For each profile dir (e.g. profiles/base, profiles/default-linux, etc),
>  a new subdirectory, called accounts is created as necessary. Inside
>  that is a file called defaults, containing default uid/gid ranges,
>  shells, etc for the given profile. Also, there are two directories,
>  user/ and group/, which contain files named after the users and groups
>  to be added. Those files contain more specific uid/gid info, etc.

I hope to god they cascade like everything else ?

Also I don't see why we would have this in the profiles as opposed to 
somewhere in /etc/?

Have people expressed an interest in per-profile mangling of uid/gid ?
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Mike Kelly]

On Tue, 03 Oct 2006 00:09:11 -0400
Alec Warner <antarus@gentoo.org> wrote:

> Mike Kelly wrote:
> > Summarized, the format is:
> > 
> >  For each profile dir (e.g. profiles/base, profiles/default-linux,
> > etc), a new subdirectory, called accounts is created as necessary.
> > Inside that is a file called defaults, containing default uid/gid
> > ranges, shells, etc for the given profile. Also, there are two
> > directories, user/ and group/, which contain files named after the
> > users and groups to be added. Those files contain more specific
> > uid/gid info, etc.
> 
> I hope to god they cascade like everything else ?

> > All the files are handled like other files in cascading profiles.

Yes, they do. Sorry, I guess I didn't word that clearly enough.

> Also I don't see why we would have this in the profiles as opposed to 
> somewhere in /etc/?

Because, first of all, this data must exist before a package is
installed (so, it can't be part of the package itself).

It shouldn't just be in some creandus-data package because all
this is closely linked with the tree, and should be maintained by the
individual package maintainers.

Also, some specifics of the settings for users and groups are somewhat
profile-specific (see below).

> Have people expressed an interest in per-profile mangling of uid/gid ?

That isn't as important as, say, properly setting the default shell on
a per-profile basis. I mainly see stuff being set in the base, hardened,
and default-* profiles, not in say the default-linux/x86/2006.1/server/
profile specifically.

Only time I can see wanting to mangle uid/gid per profile is if this
gets adopted for managing even the system users currently provided in
the default /etc/passwd and /etc/group files, where some variance has
to happen for each USERLAND. For example, there isn't a root group on
FreeBSD, just a wheel group.

-- 
Mike Kelly
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

Mike Kelly wrote:
>  All the files are handled like other files in cascading profiles. Each
>  line in the file is either a shell-style comment, or of the form:
>  "key: value". The keys are: uid, shell, home, groups, comment, and gid.

I'd prefer that the format be key=value for easier use by bash, as
pretty much everything else in profiles is bash.

Thanks,
Donnie


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Mike Kelly]

On Mon, 02 Oct 2006 21:28:21 -0700
Donnie Berkholz <dberkholz@gentoo.org> wrote:

> Mike Kelly wrote:
> >  All the files are handled like other files in cascading profiles.
> > Each line in the file is either a shell-style comment, or of the
> > form: "key: value". The keys are: uid, shell, home, groups,
> > comment, and gid.
> 
> I'd prefer that the format be key=value for easier use by bash, as
> pretty much everything else in profiles is bash.

I would as well, since my code is written written in bash and that
would make it easier on me. I had tried to do it that way before,
but some folks suggested changing it to what it is now, although I
really forget why.

Only issue I can think of is that we'd have to be sure to pick key
names that don't conflict with bash variables like UID and SHELL, which
shouldn't really be that hard, so I'm sure there had to be some
stronger reason to not make it shell variables... I'll dig through logs
and try to figure out what it was.

-- 
Mike Kelly
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Mike Kelly]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

On Tue, 3 Oct 2006 00:44:57 -0400
Mike Kelly <pioto@gentoo.org> wrote:

> On Mon, 02 Oct 2006 21:28:21 -0700
> Donnie Berkholz <dberkholz@gentoo.org> wrote:
> 
> > I'd prefer that the format be key=value for easier use by bash, as
> > pretty much everything else in profiles is bash.
>
> Only issue I can think of is that we'd have to be sure to pick key
> names that don't conflict with bash variables like UID and SHELL,
> which shouldn't really be that hard, so I'm sure there had to be some
> stronger reason to not make it shell variables... I'll dig through
> logs and try to figure out what it was.

Well, I couldn't find anything in my IRC logs, all I could find is that
I switched away from using the key="value" stuff at r10 in my svn repo,
on 5 July (it's now at r213). No one else I talked to remembered why
the switch was done in the first place, so I'm just going to change the
format back to key="value", like bash variables. Unlike most other
variables in profiles, though, I am keeping these with all lowercase
variable names, e.g. uid="5".

I'll update the documentation in my dev space tomorrow.

-- 
Mike Kelly

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2147 bytes --]

On Mon, 2006-10-02 at 22:58 -0400, Mike Kelly wrote:
> I have a list of all packages in the tree that currently use either of
> those functions[2]. If you maintain one of these packages, I'd
> especially appreciate your feedback.

You missed games-* (yes, all of them) via the games.eclass, but I'm sure
there's a couple more eclasses that do user/group modification.

> Summarized, the format is:
> 
>  For each profile dir (e.g. profiles/base, profiles/default-linux, etc),
>  a new subdirectory, called accounts is created as necessary. Inside
>  that is a file called defaults, containing default uid/gid ranges,
>  shells, etc for the given profile. Also, there are two directories,
>  user/ and group/, which contain files named after the users and groups
>  to be added. Those files contain more specific uid/gid info, etc.
>  
>  All the files are handled like other files in cascading profiles. Each
>  line in the file is either a shell-style comment, or of the form:
>  "key: value". The keys are: uid, shell, home, groups, comment, and gid.

What about applications that aren't tied to a profile?  How do they
work?  Doesn't this increase the size of the profiles pretty
dramatically?  Does it need to be tons and tons of small files, or can
we get away with a set of larger files with some sort of header?

eg.
$ cat defaults

[default]
uid: 1-999
shell: /bin/false
home: /dev/null
groups:
comment: user created by portage
gid: 1-999

$ cat accounts

[portage]
uid: 250
shell: /bin/false
home: /var/tmp/portage
groups: portage
comment: portage
gid: 250

[apache]
uid: 81
shell: /bin/false
home: /var/www/localhost
groups: apache
comment: apache
gid: 81

As you can see, this changes very little, but reduces the number of
small files in the portage tree.  Is this necessary?  Who knows?  Will
it makes syncs slightly faster? Not a clue.  I'm just throwing out an
idea.

Anyway, this looks really good.  =]

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Mike Kelly]

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

On Tue, 03 Oct 2006 08:09:08 -0400
Chris Gianelloni <wolf31o2@gentoo.org> wrote:

> > I have a list of all packages in the tree that currently use either
> > of those functions[2]. If you maintain one of these packages, I'd
> > especially appreciate your feedback.
> 
> You missed games-* (yes, all of them) via the games.eclass, but I'm
> sure there's a couple more eclasses that do user/group modification.

Oops, I forgot to account for eclasses. I'll redo my script and run it
again later to account for that.

> What about applications that aren't tied to a profile?  How do they
> work?

Well, user management is inherently tied to the userland which is being
used, which is in turn tied to the profiles (default-linux,
default-bsd, etc).

Settings which are userland-agnostic (like default uids, member groups,
GECOS comment fields), would be in the settings for the base/ profile.

> Doesn't this increase the size of the profiles pretty
> dramatically?

I don't think it makes that much of a size difference. Most of this
information is now duplicated over many enewuser/group lines in many
different ebuilds. With this system, ebuilds just need to have a EUSERS
and EGROUPS variable defined listing the users/groups needed.

> Does it need to be tons and tons of small files, or can
> we get away with a set of larger files with some sort of header?
> 
> As you can see, this changes very little, but reduces the number of
> small files in the portage tree.  Is this necessary?  Who knows?  Will
> it makes syncs slightly faster? Not a clue.  I'm just throwing out an
> idea.

Hmm, parsing that would be a more difficult task for my scripts (which
are just basic bash code doing some greps). Also, it seems like the many
small files are easier to maintain. I don't know enough about rsync to
know how it would affect efficiency, though. It's something I'll try
and look into further.

> Anyway, this looks really good.  =]

Thanks!

-- 
Mike Kelly

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: GLEP 27: Revisited (aka dynusers/creandus)

[Duncan]

Mike Kelly <pioto@gentoo.org> posted
20061003134603.2e6e97d1@mk65-desktop.pioto.org, excerpted below, on  Tue,
03 Oct 2006 13:46:03 -0400:

>> Does it need to be tons and tons of small files, or can
>> we get away with a set of larger files with some sort of header?
> 
> Hmm, parsing that would be a more difficult task for my scripts (which
> are just basic bash code doing some greps). Also, it seems like the many
> small files are easier to maintain. I don't know enough about rsync to
> know how it would affect efficiency, though. It's something I'll try
> and look into further.

Also keep in mind the effect of many small files on block oriented
filesystems like ext2/3/4.  Some of us use reiserfs or similar
"compacting" filesystems where it matters little for our local copy of
the tree, but not everybody does, and that file-per-block thing can make a
sizable difference on small files in quantity.

FWIW I prefer the small files for ease of adm. as well, but just sayin'.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Mike Kelly]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

On Tue, 3 Oct 2006 13:46:03 -0400
Mike Kelly <pioto@gentoo.org> wrote:

> On Tue, 03 Oct 2006 08:09:08 -0400
> Chris Gianelloni <wolf31o2@gentoo.org> wrote:
> 
> > You missed games-* (yes, all of them) via the games.eclass, but I'm
> > sure there's a couple more eclasses that do user/group modification.
> 
> Oops, I forgot to account for eclasses. I'll redo my script and run it
> again later to account for that.

The script has been re-written and run again, results are posted[1].
The script itself is in my svn repo[2].

This most recent run gives a total of 992 ebuilds, from a total of 511
different packages all currently using enewuser or enewgroup. Not
counting the games-* ebuilds (which all are using the same line from
games.eclass), that's 657 ebuilds, from a total of 245 packages.

The tree at the time of this run has 24854 ebuilds from 11578 packages.
So, while that looks like a lot of affected packages at first glance,
it's only about 4% of the tree.

[1] http://dev.gentoo.org/~pioto/creandus/enewusergroup-pkgnames.txt
[2] http://svn.pioto.org/viewvc/creandus/scripts/scantree-enewusergroup.bash

-- 
Mike Kelly

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GLEP 27: Revisited (aka dynusers/creandus)

[Mike Kelly]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

On Tue, 03 Oct 2006 10:20:53 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:

> beside the syntax, as pointed by Donnie, it looks ok.
> I guess could be possible extract automagically from the current tree
> the data and create the datafile from it, isn't it?

Yeah, it should be. I'm writing up a few scripts now to do that, along
with one to let me file bugs with maintainers of packages that need
porting (I won't run that one for a while still).

-- 
Mike Kelly

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]