[gentoo-catalyst] Too many problems with groupadd when creating stage3

[gentoo-catalyst] Too many problems with groupadd when creating stage3

[Juan Luis Pérez Pérez]

I'm having problems again with groupadd, this time is about dbus,
which needs to add a new group called messagebus. I think stage2 with
desktop profile at least should have sys-app/shadow installed.

Should I add a bug report in bugzilla about this?

And, any suggestion about how to install shadow in stage2? I think
that I could create a pkg-bin and then untar inside stage2.

Here you are my specs:

stage1.spec:
subarch: i486
target: stage1
version_stamp: 20091007
rel_type: default
profile: default/linux/x86/2008.0
snapshot: 20091007
source_subpath: default/stage3-i486-20090915

stage2.spec:
subarch: i486
target: stage2
version_stamp: 20091007
rel_type: default
profile: default/linux/x86/2008.0/desktop
snapshot: 20091007
source_subpath: default/stage1-i486-20091007

stage3.spec:
subarch: i486
target: stage3
version_stamp: 20091007
rel_type: default
profile: default/linux/x86/2008.0/desktop
snapshot: 20091007
source_subpath: default/stage2-i486-20091007

Here is what catalyst complains:

make[1]: Leaving directory `/var/tmp/portage/sys-apps/dbus-1.2.3-r1/work/dbus-1
.2.3'
 * Adding group 'messagebus' to your system ...
 *  - Groupid: next available
/var/tmp/portage/sys-apps/dbus-1.2.3-r1/temp/environment: line 956: groupadd: c
ommand not found
 *
 * ERROR: sys-apps/dbus-1.2.3-r1 failed.
 * Call stack:
 *               ebuild.sh, line   49:  Called pkg_preinst
 *             environment, line 2386:  Called enewgroup 'messagebus'
 *             environment, line  956:  Called die
 * The specific snippet of code:
 *               groupadd ${opts} ${egroup} || die "enewgroup failed"
 *  The die message:
 *   enewgroup failed
 *
 * If you need support, post the topmost build error, and the call stack if rel
evant.
 * A complete build log is located at '/var/tmp/portage/sys-apps/dbus-1.2.3-r1/
temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/sys-apps/dbus-1.
2.3-r1/temp/environment'.
 *
!!! FAILED preinst: 1

Juan Luis Pérez Pérez
Website: http://www.juanluperez.com
Registered GNU/Linux user #408294 according to counter.li.org
--
Por favor, no adjunte documentos en formatos "doc", "ppt", sino en
algún formato estándar como "html", "pdf", "rtf", "odt" o "txt".
Gracias.
Cf: http://www.gnu.org/philosophy/no-word-attachments.es.html

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Peter Stuge]

Juan Luis Pérez Pérez wrote:
> I'm having problems again with groupadd, this time is about dbus,
> which needs to add a new group called messagebus. I think stage2
> with desktop profile at least should have sys-app/shadow installed.

Maybe it's a problem with eutils.eclass? If it's inherited then it's
fair to expect that all it's functions will work. How to solve this
for user database changes that want to call programs in shadow?


//Peter

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 568 bytes --]

On Thursday 15 October 2009 11:07:45 Peter Stuge wrote:
> Juan Luis Pérez Pérez wrote:
> > I'm having problems again with groupadd, this time is about dbus,
> > which needs to add a new group called messagebus. I think stage2
> > with desktop profile at least should have sys-app/shadow installed.
> 
> Maybe it's a problem with eutils.eclass? If it's inherited then it's
> fair to expect that all it's functions will work. How to solve this
> for user database changes that want to call programs in shadow?

there is no problem with eutils.eclass
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

Mike Frysinger wrote:
> > > I'm having problems again with groupadd,
> > 
> > Maybe it's a problem with eutils.eclass? If it's inherited then it's
> > fair to expect that all it's functions will work. How to solve this
> > for user database changes that want to call programs in shadow?
> 
> there is no problem with eutils.eclass

Because it is not intended to function for stage2? Or because it is
intended to assume that groupadd and friends are available?

But where is the problem? One idea is to provide another
implementation of these commands than the one in shadow. If that's
a good idea, then where should that implementation go?


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 855 bytes --]

On Thursday 15 October 2009 12:48:50 Peter Stuge wrote:
> Mike Frysinger wrote:
> > > > I'm having problems again with groupadd,
> > >
> > > Maybe it's a problem with eutils.eclass? If it's inherited then it's
> > > fair to expect that all it's functions will work. How to solve this
> > > for user database changes that want to call programs in shadow?
> >
> > there is no problem with eutils.eclass
> 
> Because it is not intended to function for stage2? Or because it is
> intended to assume that groupadd and friends are available?

the release building tools have to make shadow available earlier for Linux 
targets

> But where is the problem? One idea is to provide another
> implementation of these commands than the one in shadow. If that's
> a good idea, then where should that implementation go?

this is a bad idea
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

Mike Frysinger wrote:
> > > there is no problem with eutils.eclass
> > 
> > Because it is not intended to function for stage2? Or because it is
> > intended to assume that groupadd and friends are available?
> 
> the release building tools

(catalyst? or more like the profiles or spec files?)


> have to make shadow available earlier for Linux targets

Ok! Thanks for clearing it up!


> > But where is the problem? One idea is to provide another
> > implementation of these commands than the one in shadow.
> 
> this is a bad idea

I don't like it either. There would eventually be compatibility
problems, and it doesn't consider NSS.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 575 bytes --]

On Thursday 15 October 2009 13:45:16 Peter Stuge wrote:
> Mike Frysinger wrote:
> > > But where is the problem? One idea is to provide another
> > > implementation of these commands than the one in shadow.
> >
> > this is a bad idea
> 
> I don't like it either. There would eventually be compatibility
> problems, and it doesn't consider NSS.

that's one example (out of the infinite possibilities that pam introduces)

although you could probably make a case for usage only under stage1/2/3 builds 
since these will all be local

or just fix catalyst
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 264 bytes --]

On Thursday 15 October 2009 13:50:40 Mike Frysinger wrote:
> or just fix catalyst

erm, that's probably wrong.  catalyst does what it's told as directed by the 
portage tree profiles.  so the profiles probably need tweaking to produce a 
sane stage2.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Thu, 2009-10-15 at 13:52 -0400, Mike Frysinger wrote:
> On Thursday 15 October 2009 13:50:40 Mike Frysinger wrote:
> > or just fix catalyst
> 
> erm, that's probably wrong.  catalyst does what it's told as directed by the 
> portage tree profiles.  so the profiles probably need tweaking to produce a 
> sane stage2.

Tried it.  If you'd prefer, I can pull up the list of bugs where we hit
this on nearly every release, but since a viable solution was never
reached, I don't see how it would help.

Our main "fix" was to not build stages for desktop profiles, or to use a
complete stage3 tarball as input for another stage3, but that's equally
as broken.

I'd *gladly* fix catalyst to work around this, if I had a clue how to do
so.  Hacking profiles ended up yielding nothing workable, just so
everyone knows.

-- 
Chris Gianelloni
Developer
wolf31o2.org

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1269 bytes --]

On Tuesday 20 October 2009 19:56:09 Chris Gianelloni wrote:
> On Thu, 2009-10-15 at 13:52 -0400, Mike Frysinger wrote:
> > On Thursday 15 October 2009 13:50:40 Mike Frysinger wrote:
> > > or just fix catalyst
> >
> > erm, that's probably wrong.  catalyst does what it's told as directed by
> > the portage tree profiles.  so the profiles probably need tweaking to
> > produce a sane stage2.
> 
> Tried it.  If you'd prefer, I can pull up the list of bugs where we hit
> this on nearly every release, but since a viable solution was never
> reached, I don't see how it would help.
> 
> Our main "fix" was to not build stages for desktop profiles, or to use a
> complete stage3 tarball as input for another stage3, but that's equally
> as broken.
> 
> I'd *gladly* fix catalyst to work around this, if I had a clue how to do
> so.  Hacking profiles ended up yielding nothing workable, just so
> everyone knows.

not sure why we're still even using this stage[123] system.  seems like 
catalyst could be implemented in all of three commands:
ROOT=/tmp/stage3_pre emerge system -e
chroot /tmp/stage3 env ROOT=/tmp/stage3 emerge system -e

then we wouldnt need these USE="build bootstrap" hacks nor would we need the 
packages.build cruft
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Andrew Gaffney]

On 10/28/2009 03:42 PM, Mike Frysinger wrote:
> not sure why we're still even using this stage[123] system.  seems like 
> catalyst could be implemented in all of three commands:
> ROOT=/tmp/stage3_pre emerge system -e
> chroot /tmp/stage3 env ROOT=/tmp/stage3 emerge system -e
> 
> then we wouldnt need these USE="build bootstrap" hacks nor would we need the 
> packages.build cruft

I added a target called 'system' over 6 months ago in the catalyst-3 branch in
git (now the master branch) that does exactly this. There are a few things
missing compared to a "real" stage3, but it's just stuff like .ph files and
other random things that can't be generated when $ROOT != "/".

I had considered backporting it to catalyst-2, but I'd rather not touch the
catalyst-2 code any more than necessary at this point.

If anyone wants to play with it, you can get catalyst-3 by using the
dev-util/catalyst-9999 ebuild in the tree. However, I cannot guarantee that it's
in a working state at any given time.

-- 
Andrew Gaffney                                 http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer            Catalyst/Genkernel + Release Engineering Lead

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1045 bytes --]

On Thursday 29 October 2009 08:34:48 Andrew Gaffney wrote:
> On 10/28/2009 03:42 PM, Mike Frysinger wrote:
> > not sure why we're still even using this stage[123] system.  seems like
> > catalyst could be implemented in all of three commands:
> > ROOT=/tmp/stage3_pre emerge system -e
> > chroot /tmp/stage3 env ROOT=/tmp/stage3 emerge system -e
> >
> > then we wouldnt need these USE="build bootstrap" hacks nor would we need
> > the packages.build cruft
> 
> I added a target called 'system' over 6 months ago in the catalyst-3 branch
>  in git (now the master branch) that does exactly this. There are a few
>  things missing compared to a "real" stage3, but it's just stuff like .ph
>  files and other random things that can't be generated when $ROOT != "/".

i'm not entirely sure the second ROOT= is necessary if a clean up script does 
a sufficient job of catching all accumulated cruft.  i was just throwing the 
idea out there again, but seems it isnt terribly necessary if it's been picked 
up at some point.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Andrew Gaffney]

On 10/29/2009 08:56 AM, Mike Frysinger wrote:
> On Thursday 29 October 2009 08:34:48 Andrew Gaffney wrote:
>> On 10/28/2009 03:42 PM, Mike Frysinger wrote:
>>> not sure why we're still even using this stage[123] system.  seems like
>>> catalyst could be implemented in all of three commands:
>>> ROOT=/tmp/stage3_pre emerge system -e
>>> chroot /tmp/stage3 env ROOT=/tmp/stage3 emerge system -e
>>>
>>> then we wouldnt need these USE="build bootstrap" hacks nor would we need
>>> the packages.build cruft
>>
>> I added a target called 'system' over 6 months ago in the catalyst-3 branch
>>  in git (now the master branch) that does exactly this. There are a few
>>  things missing compared to a "real" stage3, but it's just stuff like .ph
>>  files and other random things that can't be generated when $ROOT != "/".
> 
> i'm not entirely sure the second ROOT= is necessary if a clean up script does 
> a sufficient job of catching all accumulated cruft.  i was just throwing the 
> idea out there again, but seems it isnt terribly necessary if it's been picked 
> up at some point.
> -mike

I didn't pay too much attention to what you actually said (aside from ROOT and
'emerge -e system'). The catalyst-3 'system' target just does the first command.
It's basically the stage1 procedure with @system instead of packages.build.

-- 
Andrew Gaffney                                 http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer            Catalyst/Genkernel + Release Engineering Lead

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Peter Stuge]

Andrew Gaffney wrote:
> catalyst-3

Are there nice new features besides the system target? :)

Info? Samples? Will check it out.


//Peter

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Andrew Gaffney]

On 10/29/2009 08:36 PM, Peter Stuge wrote:
> Andrew Gaffney wrote:
>> catalyst-3
> 
> Are there nice new features besides the system target? :)

There are planned features that aren't fully implemented (such as specifying
multiple targets in a single spec file).

> Info? Samples? Will check it out.

No, not really :P

-- 
Andrew Gaffney                                 http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer            Catalyst/Genkernel + Release Engineering Lead

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

On Thu, 2009-10-15 at 13:50 -0400, Mike Frysinger wrote:
> or just fix catalyst

Feel free to suggest how to do so, seeing as how we have had this
problem since... well, as long as I had been doing Release Engineering
work in 2004.  Every release, we had to work around this broken
behavior.  We tried everything from trying to force shadow to build
sooner to hand-hacking group/passwd files.  As you can guess, none of
the "solutions" were ever good enough for global inclusion.

-- 
Chris Gianelloni
Developer
wolf31o2.org

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 109 bytes --]

Chris Gianelloni wrote:
> trying to force shadow to build sooner

Do you remember why this failed?


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Juan Luis Pérez Pérez]

> the release building tools have to make shadow available earlier for Linux
> targets

Yes, but it doesn't. I created some stage3 with no desktop profile, I
suppose that the other profile didn't need to install any package
which uses enewgroup (and friends) and it worked.

I'm thinking about you said, and... maybe I should create stage1 with
desktop profile too?? But I think that this wouldn't make any
difference in early stage like stage1.

-- 
Juan Luis Pérez Pérez
Website: http://www.juanluperez.com
Registered GNU/Linux user #408294 according to counter.li.org
--
Por favor, no adjunte documentos en formatos "doc", "ppt", sino en
algún formato estándar como "html", "pdf", "rtf", "odt" o "txt".
Gracias.
Cf: http://www.gnu.org/philosophy/no-word-attachments.es.html

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Juan Luis Pérez Pérez]

openldap ebuild calls enewgroup and fails.
dbus ebuild calls enewgroup and fails.

I could add this in passwd, group and shadow files in /etc but this
would be just a work around I'd need to repeat this action always I
create a new stage3.

Juan Luis Pérez Pérez
Website: http://www.juanluperez.com
Registered GNU/Linux user #408294 according to counter.li.org
--
Por favor, no adjunte documentos en formatos "doc", "ppt", sino en
algún formato estándar como "html", "pdf", "rtf", "odt" o "txt".
Gracias.
Cf: http://www.gnu.org/philosophy/no-word-attachments.es.html





On Thu, Oct 15, 2009 at 17:07, Peter Stuge <peter@stuge.se> wrote:
> Juan Luis Pérez Pérez wrote:
>> I'm having problems again with groupadd, this time is about dbus,
>> which needs to add a new group called messagebus. I think stage2
>> with desktop profile at least should have sys-app/shadow installed.
>
> Maybe it's a problem with eutils.eclass? If it's inherited then it's
> fair to expect that all it's functions will work. How to solve this
> for user database changes that want to call programs in shadow?
>
>
> //Peter
>
>

Re: [gentoo-catalyst] Too many problems with groupadd when creating stage3

[Peter Stuge]

Juan Luis Pérez Pérez wrote:
> openldap ebuild calls enewgroup and fails.
> dbus ebuild calls enewgroup and fails.

Yes, this is understood. Several other ebuilds also call enewgroup
and enewuser, and would also fail if emerged at the same point.

enewgroup fails because eutils.eclass implements that function with a
call to groupadd (not the problem!) but there is no groupadd command
available in the system at that point (the problem!).

So where is the problem exactly? I don't know.

Make eutils.eclass somehow ensure that the neccessary commands will
be available?

Make eutils.eclass depend on shadow? (I don't like that idea.)
Have an alternative implementation for these commands built into
eutils.class as fallback? (I don't like that either.)
Have an alternative implementation somewhere else? Where?

Make all ebuilds which call enewgroup or similar functions depend on
shadow? (Feels very wrong to me.)


//Peter

[gentoo-catalyst] Re: Too many problems with groupadd when creating stage3

[Juan Luis Pérez Pérez]

2009/10/15 Juan Luis Pérez Pérez <juanluperez@gmail.com>:
> Here you are my specs:
>
> stage1.spec:
> subarch: i486
> target: stage1
> version_stamp: 20091007
> rel_type: default
> profile: default/linux/x86/2008.0
> snapshot: 20091007
> source_subpath: default/stage3-i486-20090915
>
> stage2.spec:
> subarch: i486
> target: stage2
> version_stamp: 20091007
> rel_type: default
> profile: default/linux/x86/2008.0/desktop
> snapshot: 20091007
> source_subpath: default/stage1-i486-20091007
>
> stage3.spec:
> subarch: i486
> target: stage3
> version_stamp: 20091007
> rel_type: default
> profile: default/linux/x86/2008.0/desktop
> snapshot: 20091007
> source_subpath: default/stage2-i486-20091007

I deleted the whole /var/tmp/catalyst, and I started from "scratch"
using these spec files.

catalyst -f stage1.spec results in stage1-i486-20091007.tar.bz2 which
contains groupadd and friends.

catalyst -f stage2.spec results in stage2-i486-20091007.tar.bz2 which
does NOT contain groupadd and the others.

I can't understand why stage1 has groupadd and the others generated by
it do not. I'm willing to test anything you think to find out where
the problem is.

-- 
Juan Luis Pérez Pérez
Website: http://www.juanluperez.com
Registered GNU/Linux user #408294 according to counter.li.org
--
Por favor, no adjunte documentos en formatos "doc", "ppt", sino en
algún formato estándar como "html", "pdf", "rtf", "odt" o "txt".
Gracias.
Cf: http://www.gnu.org/philosophy/no-word-attachments.es.html

[gentoo-catalyst] Re: Too many problems with groupadd when creating stage3

[Juan Luis Pérez Pérez]

2009/10/16 Juan Luis Pérez Pérez <juanluperez@gmail.com>:
> catalyst -f stage1.spec results in stage1-i486-20091007.tar.bz2 which
> contains groupadd and friends.

stage1-i486-20091007 resulting does not include shadow at all.

What's going wrong with this?

-- 
Juan Luis Pérez Pérez
Website: http://www.juanluperez.com
Registered GNU/Linux user #408294 according to counter.li.org
--
Por favor, no adjunte documentos en formatos "doc", "ppt", sino en
algún formato estándar como "html", "pdf", "rtf", "odt" o "txt".
Gracias.
Cf: http://www.gnu.org/philosophy/no-word-attachments.es.html

Re: [gentoo-catalyst] Re: Too many problems with groupadd when creating stage3

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

On Fri, 2009-10-16 at 19:07 +0200, Juan Luis P?rez P?rez wrote:
> 2009/10/16 Juan Luis Pérez Pérez <juanluperez@gmail.com>:
> > catalyst -f stage1.spec results in stage1-i486-20091007.tar.bz2 which
> > contains groupadd and friends.
> 
> stage1-i486-20091007 resulting does not include shadow at all.
> 
> What's going wrong with this?

Nothing, actually.  The shadow package isn't built until stage3.

While it doesn't produce the results that you would expect, catalyst is
doing what it has been told to do by the profiles.

I've been working on a solution for this, but have never come up with
anything that actually worked.

For most releases, we hand-hacked the /etc/group file via a patch to
baselayout (yes, you can say "eww"... we did)

-- 
Chris Gianelloni
Developer
wolf31o2.org

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]