[gentoo-dev] avoiding urgent stabilizations

[gentoo-dev] avoiding urgent stabilizations

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 620 bytes --]

From time to time there are stabilization bugs where the current stable
is broken. For example, https://bugs.gentoo.org/show_bug.cgi?id=353487

However, in theory that should not happen, because presumably the
current stable has been tested in the past and considered not broken.

Of course that would be rather idealistic to assume such situation will
never happen, but can we do something more to avoid detecting important
problems in the stable tree too late? Are we missing something when
stabilizing some important packages that later causes the breakages and
need for urgent stabilizations?

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Samuli Suominen]

On 02/07/2011 06:19 PM, "Paweł Hajdan, Jr." wrote:
> From time to time there are stabilization bugs where the current stable
> is broken. For example, https://bugs.gentoo.org/show_bug.cgi?id=353487
> 
> However, in theory that should not happen, because presumably the
> current stable has been tested in the past and considered not broken.
> 

In this case, xdm has been broken for more than a year because it has
been ignoring pambase by not using system-local-login.

Only the problem came to light after ConsoleKit really started to
require this functionality from pambase.

Tricky...

But it was still tested, unfortunately the test environment that was
used leaked some environment variables that made XDM appear to work, so
this was catched too late.

Sorry.

- Samuli

Re: [gentoo-dev] avoiding urgent stabilizations

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

El lun, 07-02-2011 a las 18:43 +0200, Samuli Suominen escribió:
> On 02/07/2011 06:19 PM, "Paweł Hajdan, Jr." wrote:
> > From time to time there are stabilization bugs where the current stable
> > is broken. For example, https://bugs.gentoo.org/show_bug.cgi?id=353487
> > 
> > However, in theory that should not happen, because presumably the
> > current stable has been tested in the past and considered not broken.
> > 
> 
> In this case, xdm has been broken for more than a year because it has
> been ignoring pambase by not using system-local-login.
> 
> Only the problem came to light after ConsoleKit really started to
> require this functionality from pambase.
> 
> Tricky...
> 
> But it was still tested, unfortunately the test environment that was
> used leaked some environment variables that made XDM appear to work, so
> this was catched too late.
> 
> Sorry.
> 
> - Samuli
> 
> 

I think that maybe we could have an even higher "priority" field called,
for example, "Broken Stable" that should be *only* used for that cases
and that arch teams should try fix sooner. What do you think?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 932 bytes --]


We've been discussing this @FOSDEM too. My suggestion was that any bug that 
visibly hurts stable users should always be considered at least MAJOR in 
bugzilla. 

To expand on this a bit more
* a stable update that makes the computer nonfunctional is definitely BLOCKER 
(and should be reverted in CVS immediately when it becomes known, at latest 
when it is understood, by anyone who is around at the time and can do so)
* a non-functional stable package in the system set should be CRITICAL.

Just my 2ct, but it is really important not to hurt stable users. This is how 
we lose most people.


> I think that maybe we could have an even higher "priority" field called,
> for example, "Broken Stable" that should be *only* used for that cases
> and that arch teams should try fix sooner. What do you think?

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfridge@gentoo.org
http://www.akhuettel.de/


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Markos Chandras]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Mon, Feb 07, 2011 at 06:45:10PM +0100, Andreas K. Huettel wrote:
> 
> We've been discussing this @FOSDEM too. My suggestion was that any bug that 
> visibly hurts stable users should always be considered at least MAJOR in 
> bugzilla. 
> 
> To expand on this a bit more
> * a stable update that makes the computer nonfunctional is definitely BLOCKER 
> (and should be reverted in CVS immediately when it becomes known, at latest 
> when it is understood, by anyone who is around at the time and can do so)
> * a non-functional stable package in the system set should be CRITICAL.
> 
> Just my 2ct, but it is really important not to hurt stable users. This is how 
> we lose most people.
>
The rolling way we stabilize the packages makes the stable tree pretty
much fragile to breakages and stuff. This is because you cannot predict
what is going to happen to the rest of the tree if you stabilize a newer
package. It may have unpredictable consequences to the rest of the
packages. My suggestion, as I said to fosdem, is to freeze, or take a
snapshot if you like, of the current tree, stabilize what you need to
stabilize, test the whole tree ( at least compile wise ) for a couple of
weeks and then replace the existing stable tree. Of course this requires
automated script testing, hardware facilities etc etc that we don't have
so claiming that stable tree is "stable" is quite wrong.

Regards,
-- 
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On 2/7/11 9:50 PM, Markos Chandras wrote:
> My suggestion, as I said to fosdem, is to freeze, or take a
> snapshot if you like, of the current tree, stabilize what you need to
> stabilize, test the whole tree ( at least compile wise ) for a couple of
> weeks and then replace the existing stable tree. Of course this requires
> automated script testing, hardware facilities etc etc that we don't have
> so claiming that stable tree is "stable" is quite wrong.

This more thorough testing sounds really interesting. But do we really
lack hardware resources?

There are machines available for various arches at
<http://www.gentoo.org/proj/en/infrastructure/dev-machines.xml>. I have
at least a few chromium-related chroots on miranda, and I've never heard
complaints, so it seems a few more chroots for arch testing wouldn't hurt.

Of course for testing bootability and whether X11 starts up correctly,
etc we'd probably have to host some virtual machines, but better compile
testing (for example for toolchain updates) would be a good start.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

[gentoo-dev] Re: avoiding urgent stabilizations

[Duncan]

Paweł Hajdan, Jr. posted on Mon, 07 Feb 2011 22:02:36 +0100 as excerpted:

> On 2/7/11 9:50 PM, Markos Chandras wrote:
>> My suggestion, as I said to fosdem, is to freeze, or take a
>> snapshot if you like, of the current tree, stabilize what you need to
>> stabilize, test the whole tree ( at least compile wise ) for a couple
>> of weeks and then replace the existing stable tree. Of course this
>> requires automated script testing, hardware facilities etc etc that we
>> don't have so claiming that stable tree is "stable" is quite wrong.
> 
> This more thorough testing sounds really interesting. But do we really
> lack hardware resources?

Disclaimer:  I run ~arch (plus choice unmasks/overlays where ~arch is 
already unsuitably outdated for my tastes), so this doesn't affect me 
regardless.  That said...

The above suggestion sounds to me like increasing the bureaucracy and 
hassle of stabilizing packages even more.  We already have trouble with 
outdated stable, especially on some archs.  Do we /really/ want the 
reputation of competing with Debian-stal^hble for staleness?

Every few years someone comes up with the idea of creating a /truly/ 
stable, aka "enterprise" keyword/branch/whatever.  Every few years, it 
doesn't happen, for both resource and (arguably) philosophical reasons.

IMO, that's simply not suitable for or compatible with "mainline" Gentoo 
and its rolling updates, etc.  Yes, it's possible to do it.  A lot of 
things are "possible", but that doesn't mean they're practical.  "It's 
Gentoo, Jim, but not as we know it!"

As such, if someone/somegroup does decide to go that route, IMO the best 
approach would be a separate Gentoo-based distribution, where freezing and 
testing an entire tree for self-consistency and compatibility makes a bit 
more sense.  There are already a number of Gentoo-based distributions out 
there.  Certainly, talking to them about the problems they face and the 
solutions they use, if not using one of them directly, could be a good 
place to start.

Similarly, just as Gentoo has never made any bones about not being a hand-
holding distribution, in many cases, "you break, you get to keep the 
pieces" (tho users and devs do try to help and devs do try to prevent, 
where it's "sane" to do so), and it's not uncommon to see people saying 
that if the install speed of a binary distribution or the centrally 
controlled decisions of an Ubuntu are what one is after, Gentoo isn't 
really where you should be looking, I'd say the same applies, to some 
degree, here.  Yes, we can try to keep stable breakage to a minimum, but 
on a rolling release system, it's /going/ to happen, and I'd argue that 
the sort of wholesale freeze-and-test discussed above really /would/ be 
"Gentoo, Jim, but not as we know it", were it to be implemented as such in 
Gentoo.  That's not what Gentoo is /about/.  The rolling updates are so 
much a part of what makes Gentoo /Gentoo/, that take them out, as 
wholesale freeze and test would effectively do, and you no longer have 
Gentoo, at least as historically known.

That being the case, why confuse people about both the new product and 
Gentoo as it currently is, by calling the new product Gentoo at all?  If 
it's effectively a Gentoo-based distribution that isn't itself Gentoo, at 
least as historically considered, call it a Gentoo based distribution.  
Don't call it Gentoo, thus avoiding the confusion on both sides.

JMHO, but I've been around long enough to have seen this discussion cycle 
at least twice before...  Unfortunately, the result is often the loss of a 
few devs.  OTOH, if their goal is to make Gentoo a wholesale freeze and 
test distribution, perhaps it's better for both them and Gentoo that such 
efforts get applied somewhere more appropriate to that goal.

OTOH, if some big name comes up with suitable big resources to devote to 
such a project and they like the Gentoo name, perhaps a Gentoo-Enterprise 
might indeed be practical.  But the resource scaling that's going to 
require is enough beyond what we're doing today that I'd think seeing 
someone step up with an offer of such resources before we start seriously 
discussing it, is a worthwhile prerequisite.  And even then, making Gentoo 
that dependent on that sort of major sponsorship, regardless of who or 
what form, would change the historically "community distribution" aspect 
substantially enough that arguably, that would be "Gentoo, Jim, but not as 
we know it", as well.  But at that point I guess it'd be a question for 
the Gentoo foundation to decide, since they own the name and decide 
permissions for it in that regard.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: avoiding urgent stabilizations

[Enrico Weigelt]

* Duncan <1i5t5.duncan@cox.net> schrieb:

> The above suggestion sounds to me like increasing the bureaucracy and 
> hassle of stabilizing packages even more.  We already have trouble with 
> outdated stable, especially on some archs.  Do we /really/ want the 
> reputation of competing with Debian-stal^hble for staleness?

Well, I often have cases where the stable tree breaks something
or requires deeper manual intervention. That doesn't make fun when
maintaining dozens of systems. So a more-stable tree (hmm, perhaps
call it 'mature' ;-)) would be a big win.

I could also imagine doing that on per-package basis.
Lets say, somehow automatically export the last time of unresolved
bugs per ebuild to some sane place in the portage tree (eg. some
new file in the per-package subdirs) so people could script up
something that automatically maintains package.mask ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] avoiding urgent stabilizations

[Markos Chandras]

[-- Attachment #1: Type: text/plain, Size: 1500 bytes --]

On Mon, Feb 07, 2011 at 10:02:36PM +0100, "Paweł Hajdan, Jr." wrote:
> On 2/7/11 9:50 PM, Markos Chandras wrote:
> > My suggestion, as I said to fosdem, is to freeze, or take a
> > snapshot if you like, of the current tree, stabilize what you need to
> > stabilize, test the whole tree ( at least compile wise ) for a couple of
> > weeks and then replace the existing stable tree. Of course this requires
> > automated script testing, hardware facilities etc etc that we don't have
> > so claiming that stable tree is "stable" is quite wrong.
> 
> This more thorough testing sounds really interesting. But do we really
> lack hardware resources?
>
Yes!
> There are machines available for various arches at
> <http://www.gentoo.org/proj/en/infrastructure/dev-machines.xml>. I have
> at least a few chromium-related chroots on miranda, and I've never heard
> complaints, so it seems a few more chroots for arch testing wouldn't hurt.
> 
No. Miranda, in particular, can go down anytime soon. This is what infra
told me twice. 

> Of course for testing bootability and whether X11 starts up correctly,
> etc we'd probably have to host some virtual machines, but better compile
> testing (for example for toolchain updates) would be a good start.
> 
> Paweł
> 
Lets move this conversation on the thread that Tim started earlier
today. Seems like there is some hardware available for us on OSUOSL

Regards,
-- 
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

On 2/8/11 9:24 AM, Markos Chandras wrote:
> On Mon, Feb 07, 2011 at 10:02:36PM +0100, "Paweł Hajdan, Jr." wrote:
>> There are machines available for various arches at
>> <http://www.gentoo.org/proj/en/infrastructure/dev-machines.xml>. I have
>> at least a few chromium-related chroots on miranda, and I've never heard
>> complaints, so it seems a few more chroots for arch testing wouldn't hurt.
>>
> No. Miranda, in particular, can go down anytime soon. This is what infra
> told me twice. 

Oh, they didn't tell me that. What's the problem with miranda?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 1783 bytes --]

On 2011.02.07 20:50, Markos Chandras wrote:
[snip]

> My suggestion, as I said to fosdem, is to freeze, or take a
> snapshot if you like, of the current tree, stabilize what you need to
> stabilize, test the whole tree ( at least compile wise ) for a couple
> of weeks and then replace the existing stable tree. Of course this
> requires automated script testing, hardware facilities etc etc that 
> we don't have so claiming that stable tree is "stable" is quite 
> wrong.
> 
> Regards,
> -- 
> Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2
> 

Markos,

This is exactly what releng used to do for installer CDs.  This was 
last used for 2008.0 CD/DVD. A snapshot of the stable tree was taken in 
February 200.8 and the release hit the mirrors in September.

The seven month test/fix/retest that it took meant that the CD would 
not boot on new hardware as the kernel lacked drivers.

You will find similar lags when releng used this approach for other 
earlier releases.

We would need to move to a release cycle like the kernel. Calling a 
feature freeze at the start of the test cycle. As we can't everything, 
we might as well distribute binaries of what was tested - just as 
releng used to do. 

To me thats not Gentoo as we would loose the rolling updates.
 
There are degrees of stable.  I believe most Gentoo users 
realise this fairly early on and stick with Gentoo because they like 
the balance it strikes between Debian stable and bleeding edge.
There is a price to pay for being more up to date and it a trade off 
Gentoo users are aware off. Of course, that does not prevent them 
bringing breakages to our attention. 

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
gentoo-ops
forum-mods
trustees

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Markos Chandras]

[-- Attachment #1: Type: text/plain, Size: 4202 bytes --]

On Tue, Feb 08, 2011 at 11:43:33AM +0000, Roy Bamford wrote:
> On 2011.02.07 20:50, Markos Chandras wrote:
> [snip]
> 
> > My suggestion, as I said to fosdem, is to freeze, or take a
> > snapshot if you like, of the current tree, stabilize what you need to
> > stabilize, test the whole tree ( at least compile wise ) for a couple
> > of weeks and then replace the existing stable tree. Of course this
> > requires automated script testing, hardware facilities etc etc that 
> > we don't have so claiming that stable tree is "stable" is quite 
> > wrong.
> > 
> > Regards,
> > -- 
> > Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2
> > 
> 
> Markos,
> 
> This is exactly what releng used to do for installer CDs.  This was 
> last used for 2008.0 CD/DVD. A snapshot of the stable tree was taken in 
> February 200.8 and the release hit the mirrors in September.
> 
> The seven month test/fix/retest that it took meant that the CD would 
> not boot on new hardware as the kernel lacked drivers.
> 
> You will find similar lags when releng used this approach for other 
> earlier releases.
> 
> We would need to move to a release cycle like the kernel. Calling a 
> feature freeze at the start of the test cycle. As we can't everything, 
> we might as well distribute binaries of what was tested - just as 
> releng used to do. 
> 
> To me thats not Gentoo as we would loose the rolling updates.
>  
> There are degrees of stable.  I believe most Gentoo users 
> realise this fairly early on and stick with Gentoo because they like 
> the balance it strikes between Debian stable and bleeding edge.
> There is a price to pay for being more up to date and it a trade off 
> Gentoo users are aware off. Of course, that does not prevent them 
> bringing breakages to our attention. 
> 
> -- 
> Regards,
> 
> Roy Bamford
> (Neddyseagoon) a member of
> gentoo-ops
> forum-mods
> trustees

Roy, 

I see what you are saying. However, the 6 months testing is far from
what I have in mind. My only intention is to bring a more stable
experience to our users. Or, stop claiming that our stable tree rocks
and Gentoo is perfect for servers because it is not. Ye ye ye I know
that many many of you have Gentoo on servers but do not forget that you
are developers and you know your way around during breakages. Yes,
stable tree breaks FAR TOO often. I blame myself for my arch testing of
course however I can't do much about that. Arch teams cannot afford any
slacking (at least the two major arches) because we have 100 new bugs
every three days. This is extremely dangers to demotivate people who
will burn out sooner or later. You may think that this is an extreme
case but I can guarantee you that amd64 stable tree can become really
obsolete within one month. . 

Our stable tree is definitely not suitable for server usage unless 
you have plenty of free time to
deal with stupid upgrades because nobody, for example, cared to write a
proper elog or news item. You are probably not aware of that, since 99%
of you run testing tree however if you visit forums and stuff you will
see many many users complaining about stable tree. If we keep going down
this road we will end up sooner or later to be similar to Exherbo where
only really power users and developers will know their way around. 

Either you like it or not, arch teams are understaffed. All of them.
Therefore we cannot afford a updated stable tree with high QA around
it. We need to find a more efficient way to test packages on testing
tree so we can mark them stable with minimal time and cpu cost. We need
dedicated build boxes, like Diego's tinderbox, to test the testing tree
over and over against critical/common/trivial QA problems. If we manage
that, moving packages from testing->stable will be much more time
efficient and we can guarantee a high quality stable tree.

ps1: Personally I have stopped suggesting gentoo stable for server usage
and I always suggest testing to new users.

ps2: Roy, this is not a personal attack. Do not misinterpret my tone :)

Regards,
-- 
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Fabian Groffen]

On 08-02-2011 12:03:48 +0000, Markos Chandras wrote:
> I see what you are saying. However, the 6 months testing is far from
> what I have in mind. My only intention is to bring a more stable
> experience to our users. Or, stop claiming that our stable tree rocks
> and Gentoo is perfect for servers because it is not. Ye ye ye I know
> that many many of you have Gentoo on servers but do not forget that you
> are developers and you know your way around during breakages. Yes,
> stable tree breaks FAR TOO often. I blame myself for my arch testing of

Hmmm, odd.  I experience amd64 (stable) as being pretty stable on my
servers.  Last breakage which really got me upset was php, but that's
already some time ago.

> Our stable tree is definitely not suitable for server usage unless 
> you have plenty of free time to
> deal with stupid upgrades because nobody, for example, cared to write a
> proper elog or news item. You are probably not aware of that, since 99%
> of you run testing tree however if you visit forums and stuff you will
> see many many users complaining about stable tree. If we keep going down

With Gentoo you should update on fairly regular intervals, and have the
time inbetween as short as possible, but 2 or 3 weeks appears to be
fine.  I myself have a cronjob that syncs every night, and mails me the
output of emerge -Dupv world.  When this list gets too large, it's
typically about time to do some updating.

I think it is just regular sysadmin work to evaluate the list of
packages that's going in, and whether you deem that necessary.  I have
masked new major releases of PostgreSQL and MySQL for instance, and of
course Python 3.  I keep the rest updated, and in this way, I can only
say that I experience Gentoo to be definitely suitable for server usage,
with only little time caring about them.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] avoiding urgent stabilizations

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

tl;dr - can we add more automated tests to auto-generated stages?

On 2/8/11 1:22 PM, Fabian Groffen wrote:
> Hmmm, odd.  I experience amd64 (stable) as being pretty stable on my 
> servers.  Last breakage which really got me upset was php, but
> that's already some time ago.

Makes sense. Most of the breakages I had in mind when starting this
thread were related to consolekit/policykit/login manager/de breakages,
or some desktop software not compiling with an updated gcc. Those issues
generally don't affect servers.

By the way, to turn this thread into some action: what testing do we
currently perform for auto-generated stages? It'd be cool to at least
compile-test that the stage can "emerge -e world" itself, and emerge
some common packages (with FEATURES="test" so that we run at least some
of the produced code).

> With Gentoo you should update on fairly regular intervals, and have
> the time inbetween as short as possible, but 2 or 3 weeks appears to
> be fine.  I myself have a cronjob that syncs every night, and mails
> me the output of emerge -Dupv world.  When this list gets too large,
> it's typically about time to do some updating.

Sounds like a good idea. I think I'll start using it.

> I have masked new major releases of PostgreSQL and MySQL for
> instance, and of course Python 3.

Yeah, this ease of masking is a great strength of Gentoo.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Enrico Weigelt]

* "Pawe?? Hajdan, Jr." <phajdan.jr@gentoo.org> schrieb:

> By the way, to turn this thread into some action: what testing do we
> currently perform for auto-generated stages? It'd be cool to at least
> compile-test that the stage can "emerge -e world" itself, and emerge
> some common packages (with FEATURES="test" so that we run at least some
> of the produced code).

If we have enough VM's, we could also collect configs and perhaps
even example workloads from users which run their updates with
~arch automatically and look whether they still run properly.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] avoiding urgent stabilizations

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On 13:22 Tue 08 Feb     , Fabian Groffen wrote:
> With Gentoo you should update on fairly regular intervals, and have 
> the time inbetween as short as possible, but 2 or 3 weeks appears to 
> be fine.  I myself have a cronjob that syncs every night, and mails me 
> the output of emerge -Dupv world.  When this list gets too large, it's 
> typically about time to do some updating.

FWIW, in experience from my past two positions, updates to "critical" 
systems at least quarterly, and no more than monthly, seem to produce 
good results that balance effort and stability.

(With exceptions for security issues.)

-- 
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Rich Freeman]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

On Feb 8, 2011 11:44 AM, "Donnie Berkholz" <dberkholz@gentoo.org> wrote:
>
> (With exceptions for security issues.)

Other than monitoring bugzilla, how does a Gentoo user even know that they
have a package pending a security update?  It seems like glsa's lag
stabilization by a considerable timeframe.

I get the impression that half the reason for this is due to the difficulty
of generating the xml files.

[-- Attachment #2: Type: text/html, Size: 511 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 492 bytes --]

> Other than monitoring bugzilla, how does a Gentoo user even know that they
> have a package pending a security update?  It seems like glsa's lag
> stabilization by a considerable timeframe.

Yep. GLSA is something that seems to happen roughly one year after no affected package is in tree anymore. 

[Sorry, no offense intended. I know we're all understaffed...]

-- 
Andreas K. Huettel
Gentoo Linux developer - kde, sci, arm, tex
dilfridge@gentoo.org
http://www.akhuettel.de/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Fabian Groffen]

On 08-02-2011 18:46:32 +0100, Andreas K. Huettel wrote:
> > Other than monitoring bugzilla, how does a Gentoo user even know that they
> > have a package pending a security update?  It seems like glsa's lag
> > stabilization by a considerable timeframe.
> 
> Yep. GLSA is something that seems to happen roughly one year after no affected package is in tree anymore. 

Well, it's not too bad lately:
http://archives.gentoo.org/gentoo-announce/


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] avoiding urgent stabilizations

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 723 bytes --]

On Tuesday 08 February 2011 18:57:20 Fabian Groffen wrote:
> On 08-02-2011 18:46:32 +0100, Andreas K. Huettel wrote:
> > > Other than monitoring bugzilla, how does a Gentoo user even know that they
> > > have a package pending a security update?  It seems like glsa's lag
> > > stabilization by a considerable timeframe.
> > 
> > Yep. GLSA is something that seems to happen roughly one year after no affected package is in tree anymore. 
> 
> Well, it's not too bad lately:
> http://archives.gentoo.org/gentoo-announce/
> 

Hmm. Maybe I should just subscribe to gentoo-announce again... :]

-- 
Andreas K. Huettel
Gentoo Linux developer - kde, sci, arm, tex
dilfridge@gentoo.org
http://www.akhuettel.de/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Rich Freeman]

On Tue, Feb 8, 2011 at 12:57 PM, Fabian Groffen <grobian@gentoo.org> wrote:
> On 08-02-2011 18:46:32 +0100, Andreas K. Huettel wrote:
>> > Other than monitoring bugzilla, how does a Gentoo user even know that they
>> > have a package pending a security update?  It seems like glsa's lag
>> > stabilization by a considerable timeframe.
>>
>> Yep. GLSA is something that seems to happen roughly one year after no affected package is in tree anymore.
>
> Well, it's not too bad lately:
> http://archives.gentoo.org/gentoo-announce/

So I'll agree that it is better now in the sense that we're actually
publishing them at all.

However, it still seems non-ideal.  Take this bug for example:
http://bugs.gentoo.org/show_bug.cgi?id=351920

amd64/x86 were stable weeks ago, but the GLSA still isn't published
because we're waiting on one arch.  That means that anybody who does
updates once a quarter or whatever except for security updates will be
vulnerable, because they don't know they still have a vulnerability.

Even after the last arch is updated it often takes a little time to
get the GLSA published.

About the only thing glsa-checking tools do for me is bug me about
having libpng-1.2.44   installed (bug 340261 - most likely glsa is
incorrect).  I almost never catch vulnerabilities on my live system
that way since even if I'm slow I get the updates installed before the
glsa comes out anyway.  However, I do get noise sometimes.

Perhaps we should target having glsas published within a certain
amount of time after a vulnerability is disclosed, whether corrected
or not.  We could re-publish a final notice once all is well.  We
really shouldn't consider users safe from a security vulnerability
until the vulnerability is patched in the tree AND the notice to
update has been sent out.

Rich

Re: [gentoo-dev] GSLA improvements (WAS: avoiding urgent stabilisations)

[Fabian Groffen]

On 09-02-2011 08:57:25 -0500, Rich Freeman wrote:
> Perhaps we should target having glsas published within a certain
> amount of time after a vulnerability is disclosed, whether corrected
> or not.  We could re-publish a final notice once all is well.  We
> really shouldn't consider users safe from a security vulnerability
> until the vulnerability is patched in the tree AND the notice to
> update has been sent out.

Excellent, take this up with the security team.  Reevaluate which archs
are security supported, and see if you can get a timeout policy
implemented.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] avoiding urgent stabilizations

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

On 2/9/11 2:57 PM, Rich Freeman wrote:
> Perhaps we should target having glsas published within a certain
> amount of time after a vulnerability is disclosed, whether corrected
> or not.  We could re-publish a final notice once all is well.  We
> really shouldn't consider users safe from a security vulnerability
> until the vulnerability is patched in the tree AND the notice to
> update has been sent out.

I think http://www.gentoo.org/security/en/vulnerability-policy.xml
specifies the target delay, and also mentions temporary GLSAs.
Unfortunately, that process does not seem to be followed due to general
difficulty of drafting GLSAs (I don't even know what is the problem, as
GLSAmaker is only available to security team members).


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Rich Freeman]

On Wed, Feb 9, 2011 at 9:08 AM, "Paweł Hajdan, Jr."
<phajdan.jr@gentoo.org> wrote:
> I think http://www.gentoo.org/security/en/vulnerability-policy.xml
> specifies the target delay, and also mentions temporary GLSAs.
> Unfortunately, that process does not seem to be followed due to general
> difficulty of drafting GLSAs (I don't even know what is the problem, as
> GLSAmaker is only available to security team members).
>

I think the policy itself is completely appropriate, and of course
publishing it makes the process transparent to the users.

I think our problem is more with complying with that policy.

I have heard similar complaints about GLSAmaker.  I half-wonder if it
would make more sense to just edit the xml files directly and validate
them with a tool, and send out an email, if the tool really is that
bad.

Could the security team use a staff position of some kind that an
interested user could take on that handled some of the more
administrative aspects of security bugs?  Maybe we aren't that bad at
fixing our code, but nobody wants to sit around tinkering with
notices/etc.  Perhaps we might have interested users who wouldn't mind
sending out notices and closing bugs who otherwise might not want to
or be able to maintain ebuilds/etc?

Rich

Re: [gentoo-dev] avoiding urgent stabilizations

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On 10:26 Wed 09 Feb     , Rich Freeman wrote:
> I have heard similar complaints about GLSAmaker.  I half-wonder if it 
> would make more sense to just edit the xml files directly and validate 
> them with a tool, and send out an email, if the tool really is that 
> bad.

If this is really the problem, please propose a GSoC project:

http://en.gentoo-wiki.com/wiki/Google_Summer_of_Code_2011_ideas

Sure, it's a ways off, so maybe we'd need an interim workaround. But at 
least we'd be likely to end up with a fix in the end, and more GSoC 
ideas are always welcome (we had 19 students last year!).

-- 
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Robin H. Johnson]

On Wed, Feb 09, 2011 at 10:26:19AM -0500, Rich Freeman wrote:
> I have heard similar complaints about GLSAmaker.  I half-wonder if it
> would make more sense to just edit the xml files directly and validate
> them with a tool, and send out an email, if the tool really is that
> bad.
a3li has been working on GLSAmaker2. I don't know the status.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] avoiding urgent stabilizations

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On 12:37 Tue 08 Feb     , Rich Freeman wrote:
> On Feb 8, 2011 11:44 AM, "Donnie Berkholz" <dberkholz@gentoo.org> wrote:
> > (With exceptions for security issues.)
> 
> Other than monitoring bugzilla, how does a Gentoo user even know that 
> they have a package pending a security update?  It seems like glsa's 
> lag stabilization by a considerable timeframe.

I read LWN's security page and ensure updates happen, regardless of 
whether the ebuild exists yet.

But if I didn't, I would rely on GLSAs — either via the mailing list or 
with a daily sync and glsa-check.

-- 
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Enrico Weigelt]

* Fabian Groffen <grobian@gentoo.org> schrieb:

> Hmmm, odd.  I experience amd64 (stable) as being pretty stable on my
> servers.  Last breakage which really got me upset was php, but that's
> already some time ago.

the ini file issue ?

> With Gentoo you should update on fairly regular intervals, and have the
> time inbetween as short as possible, but 2 or 3 weeks appears to be
> fine.  I myself have a cronjob that syncs every night, and mails me the
> output of emerge -Dupv world.  When this list gets too large, it's
> typically about time to do some updating.

I've automatized even a bit more: my cron script also automatically
rebuilds the packages I have explicitly whitelisted in my
/etc/portage/package.autoupdate file.

Let me know if anyone likes to have it.

> I have masked new major releases of PostgreSQL and MySQL for instance,
> and of course Python 3. 

/me too.

Perhaps we can find a way to make the update safe (so eg. new
postmaster is installed along the old one) and provide some
migration tool ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] avoiding urgent stabilizations

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 3953 bytes --]

Markos,

A few thoughts inlined.

On 2011.02.08 12:03, Markos Chandras wrote:
 
My main point was that as you move from an old dated set of packages to 
newer packages which by definition are less well tested, stability 
decreases. Users pick somewhere between the two extremes that they are 
happy with. Gentoo stable lies somewhere between Debian stable and LFS 
built live from all the repositories.

> I see what you are saying. However, the 6 months testing is far from
> what I have in mind. 
Thats what releng used to take.

> My only intention is to bring a more stable
> experience to our users. Or, stop claiming that our stable tree rocks
> and Gentoo is perfect for servers because it is not. Ye ye ye I know
> that many many of you have Gentoo on servers but do not forget that
> you
> are developers and you know your way around during breakages. Yes,
> stable tree breaks FAR TOO often. I blame myself for my arch testing
> of
> course however I can't do much about that. 
[snip]

For servers I can point you at the stillborn Gentoo-LAMP project. I 
don't remember much more than its name. Google seems to have forgotten 
it too.

A big part of the problem comes from being a meta-distro. Everyones 
Gentoo is different and we we cannot test all combinations to ensure 
everyone is ok.

More testing will not eliminate the issue but would catch some 
problems. There would be less breakage but not zero.  There is a trade 
off to be made there by both the developers doing the testing and 
the users experiencing the breakage.

I agree that given more resources, the tree could be improved but 
before we move in that direction, I would like to ask is that the best 
use of resources?

As I said above, users are aware of the trade offs involved in choosing 
Gentoo. Are our users really unhappy, or are they just looking for help 
to fix issues when they occur?
Most users do not expect a zero issue upgrade path.

[snip]
> 
> Our stable tree is definitely not suitable for server usage unless 
> you have plenty of free time to
> deal with stupid upgrades because nobody, for example, cared to write
> a
> proper elog or news item. 
[snip]
> 
> Either you like it or not, arch teams are understaffed. All of them.
All of Gentoo is understaffed.

> Therefore we cannot afford a updated stable tree with high QA around
> it. We need to find a more efficient way to test packages on testing
> tree so we can mark them stable with minimal time and cpu cost. We
> need
> dedicated build boxes, like Diego's tinderbox, to test the testing
> tree
> over and over against critical/common/trivial QA problems. If we
> manage
> that, moving packages from testing->stable will be much more time
> efficient and we can guarantee a high quality stable tree.
If this means a more up to date stable tree, that has to be good as the 
stable tree will move closer to testing and there will be fewer 
packages to maintain.  (Counting different versions as packages)
> 
> ps1: Personally I have stopped suggesting gentoo stable for server
> usage
> and I always suggest testing to new users.

I don't quite agree about not recommending Gentoo for servers. Gentoo 
is fine on servers but you need to run a testing environment for your 
updates so you know when you do do an update, exactly what in involved 
and what will happen.  Without your own testing, your server will go 
down from time to time. If you cannot do your own testing, either 
tolerate the downtime or don't use Gentoo.

> 
> ps2: Roy, this is not a personal attack. Do not misinterpret my tone
> :)

I see no personal attack in your words. 
> 
> Regards,
> -- 
> Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2
> 
I'll buy you a <insert_refreshment_of_your_choice> next time we meet.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
gentoo-ops
forum-mods
trustees

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] avoiding urgent stabilizations

[Rich Freeman]

On Tue, Feb 8, 2011 at 7:03 AM, Markos Chandras <hwoarang@gentoo.org> wrote:
> I see what you are saying. However, the 6 months testing is far from
> what I have in mind.

I could see there being room for something in-between, but I share the
concerns of others that rolling releases are part of what makes
Gentoo, well, Gentoo.

The problem with snapshots is that there is almost always SOMETHING
wrong with them, and if you don't release until they're near-perfect
then you're pursing 99.999% quality and most devs don't care enough to
work hard towards that.  As a result you end up with very long release
cycles.

I could see room for a system where every week a portage snapshot is
created, and then run through automated testing.  The test results are
then posted, and the release tarball is made available for download.
Then people can update to it if they think it is good enough.  Serious
issues would of course be spotted and immediately fixed in-tree so
that the next weekly release is better, and the typical user
experience would still be to use the live tree so that they get an
experience similar to what they have.

Honestly, I don't even know that this would really work well.  It
might be better to just have a tinderbox that does automated full-tree
testing weekly and just post the results and let devs look at them and
fix things.

However, I don't think any system is likely to work (except on Debian
timelines) if it involves a release-when-its-ready approach unless
ready is something really minimal like "system set compiles and
boots."

Time vs quality vs cost - pick two.  Oh, for Gentoo we've pretty-much
picked cost as being about as close to zero as you can get, so make
that pick one.  Debian stable favors quality, and there are definitely
things I'd use debian for that I'd never use Gentoo for.  That isn't
knocking Gentoo - it is just a reflection of the fact that the distros
have different philosophies.

Re: [gentoo-dev] avoiding urgent stabilizations

[Enrico Weigelt]

* Markos Chandras <hwoarang@gentoo.org> schrieb:

> My suggestion, as I said to fosdem, is to freeze, or take a
> snapshot if you like, of the current tree, stabilize what you
> need to stabilize, test the whole tree ( at least compile wise )
> for a couple of weeks and then replace the existing stable tree. 

hmm, would it make sense to add a new masking for the testing
tree, so users could decide which stability grade vs they wish ?
or perhaps use overlays for that ?

For example, I'd like to have the critical packages (everything
that's needed to bootup and do basic remote maintenance) from
the new frozen-stable tree, but other things should stay as
they are.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] avoiding urgent stabilizations

[Ed W]

On 21/02/2011 00:11, Enrico Weigelt wrote:
> * Markos Chandras<hwoarang@gentoo.org>  schrieb:
>
>> My suggestion, as I said to fosdem, is to freeze, or take a
>> snapshot if you like, of the current tree, stabilize what you
>> need to stabilize, test the whole tree ( at least compile wise )
>> for a couple of weeks and then replace the existing stable tree.
> hmm, would it make sense to add a new masking for the testing
> tree, so users could decide which stability grade vs they wish ?
> or perhaps use overlays for that ?
>
> For example, I'd like to have the critical packages (everything
> that's needed to bootup and do basic remote maintenance) from
> the new frozen-stable tree, but other things should stay as
> they are.
>

Perhaps this is an argument for a git based portage tree?  Master can 
stay as the current status quo and anyone who wants to can maintain a 
branch or fork which points to a slightly different subset of the tree?

I doubt we actually have the capacity to make this work, but it would at 
least in theory be cool to have a (weekly/monthly) branch which gets 
cut, run through a tinderbox in various forms and then pushed?  Or if 
someone wants to maintain a redhat style antique set of packages where 
the tree is largely held back to 2005 state with only bug fixes and 
essential packages bumped?

Just thinking...

Ed W

Re: [gentoo-dev] avoiding urgent stabilizations

[Matthew Marlowe]

All,

> Perhaps this is an argument for a git based portage tree?  Master can
> stay as the current status quo and anyone who wants to can maintain a
> branch or fork which points to a slightly different subset of the tree?
> 

I'm starting to put together a portage/stable server configuration for a large 
number of gentoo VM's that will eventually be hosted on a VMware ESX 4.1U1 
cluster - with the goal of limiting major changes to once/year and otherwise 
only applying security/minimum necessary updates.  I doubt it will be easy but 
I'm doing my best at it :)

As part of that I'm maintaining on github several related repositories, 
including portage mask/use/config files, cluster management utilities, etc.

     https://github.com/deploylinux

I've also started to document work on my blog:

     http://www.deploylinux.net/matt

I'm not currently planning to utilize a separate overlay for packages/ebuilds, 
but it's not out of the question.

I'd be happy to work with any other devs w/ similiar interests or production 
networks to manage.....github makes group development relatively easy.

MattM
-- 
Matthew Marlowe    /  858-400-7430  /    DeployLinux Consulting, Inc
  Professional Linux Hosting and Systems Administration Services
              www.deploylinux.net   *   matt@deploylinux.net
                             'MattM' @ irc.freenode.net
       

Re: [gentoo-dev] avoiding urgent stabilizations

[Ed W]

Hi

> I'm starting to put together a portage/stable server configuration for a large
> number of gentoo VM's that will eventually be hosted on a VMware ESX 4.1U1
> cluster - with the goal of limiting major changes to once/year and otherwise
> only applying security/minimum necessary updates.  I doubt it will be easy but
> I'm doing my best at it :)

This sounds very interesting.  I haven't yet plugged through your blog, 
but just to chime in:

I maintain a, likely much smaller, number of VMs using linux vservers.  
The approach here is to almost cut each machine down to a chroot that 
runs only one (or thereabouts) interesting service.  To do this I have 
found customised portage profiles to be the under-plugged secret since 
they allow you to basically push a set of packages which should be 
installed and control "per type of vm" use flags and package keywords 
(eg I have www-nginx, www-apache, mail, proxy, mysql, ftp, etc 
profiles).  Additionally I have a small overlay of local ebuilds that 
sit in the same tree

Up until now I haven't really made any effort to sync this whole tree 
across multiple physical machines and it's a bit of an ad-hoc process.  
Using something like git would probably be perfect

The still missing step is configuration management across the machine 
types, eg I want to upgrade all my "Apache-WWW" class machines and merge 
in all changes in /etc in a certain way... At the moment I just run 
dispatch-conf across all machines, but it can be quite boring merging 20 
instances of sshd.conf...  Seems like Puppet/Chef could be a solution 
here, but the step up and investment to make it work seems pretty large?



It does appear like managing large numbers of virtual machines is one 
are that gentoo could score very well?  Interested to see any chatter on 
how others solve this problem, or any general advocacy?  Probably we 
should start a new thread though...

Regards

Ed W

[gentoo-dev] Community Development of Gentoo Server Mgmt Tools for VMware Clusters/etc - Puppet modules, github collaboration, etc - Fork of Dev Topic: Avoiding Urgent Stabilizations

[Matthew Marlowe]

Reader note -- Direct all respones to gentoo-server ML to avoid flooding 
gentoo-dev.

For those on server-ml who missed dev-ml:

I'm starting to put together a portage/stable server configuration for a large 
number of gentoo VM's that will eventually be hosted on a VMware ESX 4.1U1 
cluster - with the goal of limiting major changes to once/year and otherwise 
only applying security/minimum necessary updates.  I doubt it will be easy but 
I'm doing my best at it :)

As part of that I'm maintaining on github several related repositories, 
including portage mask/use/config files, cluster management utilities, etc.

     https://github.com/deploylinux

I've also started to document work on my blog:

     http://www.deploylinux.net/matt

I'm not currently planning to utilize a separate overlay for packages/ebuilds, 
but it's not out of the question.

I'd be happy to work with any other devs or users w/ similiar interests or 
production networks to manage.....github makes group development relatively 
easy.

On Friday, February 25, 2011 03:37:36 am Ed W wrote:
> I maintain a, likely much smaller, number of VMs using linux vservers.
> The approach here is to almost cut each machine down to a chroot that
> runs only one (or thereabouts) interesting service.  To do this I have
> found customised portage profiles to be the under-plugged secret since
> they allow you to basically push a set of packages which should be
> installed and control "per type of vm" use flags and package keywords
> (eg I have www-nginx, www-apache, mail, proxy, mysql, ftp, etc
> profiles).  Additionally I have a small overlay of local ebuilds that
> sit in the same tree
> 

I'd rather not maintain a separate profile for each server/node type.
Instead it would seem to be easier to use to a buildhost to create a combined 
repository of packages, and then use puppet to define nodetypes and tell each 
individual node which packages to install, which users to create, and which 
services to enable.

> Up until now I haven't really made any effort to sync this whole tree
> across multiple physical machines and it's a bit of an ad-hoc process.
> Using something like git would probably be perfect
> 

Agreed -- git *is* perfect and widely used in enterprise configuration 
management repositories.  The main advantage is that it allows individual 
users to maintain their own forks, and yet merge contributions from other 
trees or to request that other repositories merge their changes w/o much 
effort.  Github makes this even easier w/ a web interface, tools to visual 
changes and track contributions, and discussions/wiki's for each repository.

> The still missing step is configuration management across the machine
> types, eg I want to upgrade all my "Apache-WWW" class machines and merge
> in all changes in /etc in a certain way... At the moment I just run
> dispatch-conf across all machines, but it can be quite boring merging 20
> instances of sshd.conf...  Seems like Puppet/Chef could be a solution
> here, but the step up and investment to make it work seems pretty large?
> 

Yes, I'm currently planning that puppet will control installation of new 
packages and pushing all configuration files.  This works very well w/ RedHat 
Enterprise Linux/etc and if we are careful should also work for gentoo-- For 
updates, we may have to either extend puppet to be aware of revdep-rebuild and 
gentoo package versioning better, or have an external script for updating the 
relevant nodes after the buildhost has finished compiling them.

> 
> 
> It does appear like managing large numbers of virtual machines is one
> are that gentoo could score very well?  Interested to see any chatter on
> how others solve this problem, or any general advocacy?  Probably we
> should start a new thread though...
> 

Agreed - With careful management, the time investment in QA and package 
management/maintenance should be more than outweighed by the benefits for 
reasonably sized server clusters.  It would be even easier if we can have 
gentoo server admins work together to help build puppet modules and general 
mgmt utilities using a service like github.

> Regards
> 
> Ed W

Matt
-- 
Matthew Marlowe    /  858-400-7430  /    DeployLinux Consulting, Inc
  Professional Linux Hosting and Systems Administration Services
              www.deploylinux.net   *   matt@deploylinux.net
                             'MattM' @ irc.freenode.net
       

Re: [gentoo-dev] avoiding urgent stabilizations

[Enrico Weigelt]

* Ed W <lists@wildgooses.com> schrieb:

> I maintain a, likely much smaller, number of VMs using linux vservers.  
> The approach here is to almost cut each machine down to a chroot that 
> runs only one (or thereabouts) interesting service.

I'm working in a similar way: my dedicated boxes are VM hosts
(currently ovz, but later lguest or lxc), each of them running
only specific services (eg. one for nginx-based front proxy,
several other for backend webservers, RDBMs'es, mailservers, etc).

But, for me, even a trimmed-down Gentoo is still too large
(has to contain the whole base packages, from portage to
toolchain, includes, etc). I'd prefer having only the essential
runtime stuff within the containers.

For this we need a different approach (strictly separating build
and production environments). Binary distros (eg. Debian) might
be one option, but they're lacking the configurability and mostly
are still too large. So I'm going a different route using my own
buildsystem - called Briegel - which originally was designed for
embedded/small-device targets.

For now I didn't have the spare time to port all the packages
required for complete server systems (most of it is making
them all cleanly crosscompile'able, as this is a fundamental
concept of Briegel). But maybe you'd like to join in and try it :)

> Using something like git would probably be perfect

I'm using git'ed portage trees in production for several month now
(I sometimes had to touch some eclasses, which _IMHO_ doesn't via
overlays). The repos are configured to rebase on pull and everything
runs automatically :)

> The still missing step is configuration management across the machine 
> types, eg I want to upgrade all my "Apache-WWW" class machines and merge 
> in all changes in /etc in a certain way... At the moment I just run 
> dispatch-conf across all machines, but it can be quite boring merging 20 
> instances of sshd.conf...  Seems like Puppet/Chef could be a solution 
> here, but the step up and investment to make it work seems pretty large?

I haven't used puppet yet, but several collegues have made good
experiences with it. If you're maintaining dozens of very similar
systems (mine tend to be very different from each other), it likely
worth investigating.

> It does appear like managing large numbers of virtual machines is one 
> are that gentoo could score very well?  Interested to see any chatter on 
> how others solve this problem, or any general advocacy?  Probably we 
> should start a new thread though...

I'm not sure if Gentoo really is the right distro for that purpose,
as it's targeted to very different systems (i.g. Gentoo boxes are
expected to be quite unique, beginning with different per-package
useflags, even down to cflags, etc). But it might still be a good
basis for building specific system images (let's call them stage5 ;-))

An setup for 100 equal webserver vm's could look like this:

* run a normal Gentoo vm (tailored for the webserver appliance),
  where do you do regular updates (emerge, revdep-rebuild, etc, etc)
* from time to time take a snapshot, strip off the buildtime-only
  stuff (hmm, could turn out to be a bit tricky ;-o)
* this stripped snapshot now goes into testing vm's
* when approved, the individual production vm's are switched over
  to the new image (maybe using some mount magic, unionfs, etc)


At this point I've got a question for to the other folks here:

emerge has an --root option which allows to (un)merge in a separate
system image. So it should be possible to unmerge a lot of system
packages which are just required for updating/building (even
portage itself), but this still will be manual - what about 
dependency handling ?

Is there some way to drop at least parts of the standard system set,
so eg. portage, python, gcc, etc, etc get unmerged by --depclean
if nobody else (in world set) doesn't explicitly require them ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] avoiding urgent stabilizations

[Ed W]

Hi

> But, for me, even a trimmed-down Gentoo is still too large
> (has to contain the whole base packages, from portage to
> toolchain, includes, etc). I'd prefer having only the essential
> runtime stuff within the containers.

I'm just building some embedded devices on the side using gentoo and my 
minimal builds are only a few MB? Curious why you feel you need to move 
from Gentoo to get the size smaller?

Seems like your complaint is that you have gentoo installs which are 
full featured with a toolchain and portage, which you are comparing to 
an installation you built with a different tool that doesn't have a 
toolchain installed?  However, you can do the same using gentoo if you 
wish? (you just need a lightweight package installer to avoid installing 
portage)

I think your main options are:

1) Build your base images without a toolchain or portage and use a 
minimal package installer to install pre-built binary packages.  This 
seems fraught with issues long term though...

2) Build your base images without a toolchain, but with portage (and 
perhaps a very minimal python). This gives you full dependency tracking 
and obviously bind mount/nfs mount the actual portage tree to avoid 
space used there. This seems workable and minimal?

3) If we are talking virtual machines then who cares if your containers 
are individually quite large, if the files in them are duplicated across 
all containers?  Simply use an appropriate de-duplication strategy to 
coalesce the space and most of the disadvantages disappear?  eg 
linux-vserver you can simply hardlink all the common files across your 
installations and allow the COW patch to break hardlinks if anyone 
alters a file in a single instance. Or you could use aufs to mount a 
writeable layer over your common base VM instance?  Or you could use one 
of the filesystems which de-duplicates files in the background (some 
caveats apply here to avoid memory still being used multiple times in 
each VM).  Or under KVM there is the memory coalescing feature which 
merges similar code pages (forget it's name?)

Personally I think option 3) is quite interesting from the medium number 
of virtual machines, ie in the 10s to hundreds, ie simply don't worry 
about it, let the OS do the work.  In the hundreds to thousands plus 
level I guess you have unique challenges and I would be wrong to try and 
suggest a solution from the comfort of a laptop without having that 
responsibility, but I would have thought there was some advantage in a 
very rigidly deployed base OS generated and updated very precisely?


> For this we need a different approach (strictly separating build
> and production environments). Binary distros (eg. Debian) might
> be one option, but they're lacking the configurability and mostly
> are still too large. So I'm going a different route using my own
> buildsystem - called Briegel - which originally was designed for
> embedded/small-device targets.
>
> For now I didn't have the spare time to port all the packages
> required for complete server systems (most of it is making
> them all cleanly crosscompile'able, as this is a fundamental
> concept of Briegel). But maybe you'd like to join in and try it :)

Sounds like an interesting challenge, but I'm unconvinced you can't 
solve 90% of your problem within the constraints of Gentoo? This saves 
you a bunch of time that could be invested in the last 10% through more 
traditional means?


>> It does appear like managing large numbers of virtual machines is one
>> are that gentoo could score very well?  Interested to see any chatter on
>> how others solve this problem, or any general advocacy?  Probably we
>> should start a new thread though...
> I'm not sure if Gentoo really is the right distro for that purpose,
> as it's targeted to very different systems (i.g. Gentoo boxes are
> expected to be quite unique, beginning with different per-package
> useflags, even down to cflags, etc). But it might still be a good
> basis for building specific system images (let's call them stage5 ;-))

I won't disagree on your "where it's targeted", but just to re-iterate 
why I think Gentoo works well is that it does have a very workable 
binary package feature!

My way of working is to use (several) shared binary package repos and 
the guests largely pull from those shared package directories.  In fact 
what I do is have a minimal number of shared "/usr/portage/package" 
directories and I mount an appropriate one to the guest type at boot 
time.  At the moment my main two options are "32bit" and "64bit" for the 
package mounts, but I recently introduced a new machine type which is 
held back to perl 5.8 and that guest gets it's own package mount since 
it's obviously linking a lot of binaries differently

So, my process is to test an update on a small number of guests, either 
dedicated test guests or less important live guests.  If this looks good 
then I run the upgrade against all other Vms of the same type and they 
will update quickly from package binaries

Now, the icing is that this works extremely well even once you decide to 
lightly customise machine types.  So for example my binary packages are 
very high level (eg 32/64bit), my "profiles" would be fairly high level, 
eg I have www-apache and www-nginx base profiles.  However, a specific 
virtual machine running say nginx might itself need a specific PHP 
application installed, and that itself might need some dependencies, 
which in turn might require a specific set of customisation of use flags 
and versions.

Now, the neat thing is that the binary upgrade options are *either* to 
use *only* binary packages, OR to use binary packages *if* they were 
built with the correct USE flags. So for example I haven't bothered to 
split out my packages directory to be specific to the nginx/apache 
machines, however, this causes the PHP package to be regularly rebuilt 
depending on whether it was last used to upgrade an nginx or apache 
guest (different use flags needed for each guest).  I could fix this 
easily enough, but it's not a problem for me and it's automatically 
handled through the portage binary package updates

So the end result is that you can make efficient use of binary updates, 
but portage will still customise the odd package here or there where a 
local machine requires something which differs from the norm.  To my eye 
this keeps most of the benefits of an RPM/DEB style binary updater, with 
the flexibility of a per machine, customised USE flag gentoo installation?


> An setup for 100 equal webserver vm's could look like this:
>
> * run a normal Gentoo vm (tailored for the webserver appliance),
>    where do you do regular updates (emerge, revdep-rebuild, etc, etc)
> * from time to time take a snapshot, strip off the buildtime-only
>    stuff (hmm, could turn out to be a bit tricky ;-o)
> * this stripped snapshot now goes into testing vm's
> * when approved, the individual production vm's are switched over
>    to the new image (maybe using some mount magic, unionfs, etc)

This could work and perhaps for 100 identical Vms you have enough meat 
to work on something quite customised anyway?

Personally for 20-80 identical VMs running very limited variety of web 
software I would go for:
- Slightly cut down gentoo VM
- Hardlinked across all instances OR single installation which is read only
- Writeable data areas mounted to their own space (/var/www, /tmp, 
/home, etc)

By separating the data from the OS you have a lot of flexibility to 
upgrade the base webserver install and mount the data back on the new 
VM?  With linux-vservers or other container style, you will find that 
the OS shares code segments across all virtual machines (due to all 
files sharing the same inode) and the memory usage should be much lower 
and nearer to firing up an instance of the shared app and it then 
forking (ie data is duplicated, but the code segment is shared)


For 100+ Vms I guess I would be looking very strongly at a common 
read-only OS partition and container style virtualisation

For 20-80 near identical VMs, but running a wider variety of web 
software I would go for the hardlinked option with a straightforward 
"emerge" upgrade option across them.  Hardlinking keeps the memory usage 
sane where possible, without the pain of trying to keep the base install 
absolutely identical and read-only to make the common mount option work?


> At this point I've got a question for to the other folks here:
>
> emerge has an --root option which allows to (un)merge in a separate
> system image. So it should be possible to unmerge a lot of system
> packages which are just required for updating/building (even
> portage itself), but this still will be manual - what about
> dependency handling ?

This is correct.  In fact this is how you build a stage 1,2,3 etc and 
how catalyst works!

The information is a bit spread out over several out of date wiki 
articles, but perhaps start with:
     http://en.gentoo-wiki.com/wiki/Tiny_Gentoo

Roughly speaking you could "freshen" your current installation with 
(from memory):
     ROOT="/tmp/new_build" emerge -av world

This has minor gremlins when I test it, probably due to some symlinks 
being created differently if you follow the current catalyst build 
script through stage 1,2,3 etc, but roughly speaking it does the same 
thing only jumping straight to the end result and building a completely 
new identical install to your current OS...

Even more special is that you can set an alternative portage source, so 
if you want to build your new ROOT with alternative make.conf, 
/etc/portage/*, etc then just put your new files somewhere and set 
PORTAGE_CONFIGROOT to point to it.  Cross compiling is also done through 
an extension of this basic method

So, following your chain of thought - yes it's not too hard to quickly 
generate a customised base OS installation to use for your future VMs.  
Further, if you wish you can make those VMs have a reduced or missing 
toolchain etc.  In fact if you google a bit I think you will find some 
recipes for very minimal VMs using this method where the base VM is a 
very minimal install...

> Is there some way to drop at least parts of the standard system set,
> so eg. portage, python, gcc, etc, etc get unmerged by --depclean
> if nobody else (in world set) doesn't explicitly require them ?

You are almost thinking about it all wrong.  ("There is no spoon...")

This is gentoo, so at this more advanced level, stop thinking about 
"standard system set" and instead free your mind to start with 
"nothing".  Go read the old bootstrap from stage 1 instructions, plus 
the TinyGentoo pages and you can quickly see that Catalyst builds your 
working installation by starting from a working installation, creating 
an empty directory, adding some minimal packages to that directory and 
building up from there.

So absolutely nothing stops you from just starting with an empty 
directory and just emerging a few basic packages into it (couple MB) and 
then chrooting into it and having some fun...  There is *no* minimal 
package set, you can install whatever you want (as long as it boots). 
Largely the portage dependency tracker will help you pull in the minimal 
needed dependencies, but beware that system packages arent generally 
explicitly tracked so you may stumble across some deps when you are 
going really basic and omiting standard system packages (just use common 
sense: it should be fairly obvious if an application requires a compiler 
and you didn't install one then you have a conflict of interest...)


Have another look at gentoo!  I definitely believe that it's flexibility 
to build you highly customised packages, plus strong templating of those 
packages, plus decent ability to distribute binaries of the end result 
is a very strong combo!  Better binary support is really the only thing 
missing here, but it's pretty adequate as it stands!

Good luck

Ed W

Re: [gentoo-dev] avoiding urgent stabilizations

[Enrico Weigelt]

* Ed W <lists@wildgooses.com> schrieb:

> I'm just building some embedded devices on the side using
> gentoo and my minimal builds are only a few MB? 

How to do you get out all the buildtime stuff (portage, toolchain, etc) ?

> Seems like your complaint is that you have gentoo installs which are 
> full featured with a toolchain and portage, which you are comparing to 
> an installation you built with a different tool that doesn't have a 
> toolchain installed? 

Well, with Briegel, I'm strictly differenciating between the build
host and the target image. Everything's crosscompiled by design.
So I can explicitly define which packages need to be in the final
image (and only those are built). The build host can by any fairly
recent GNU/Linux system (happens to be Gentoo at my site)

> 1) Build your base images without a toolchain or portage and use a 
> minimal package installer to install pre-built binary packages.
> This seems fraught with issues long term though...

How can this be done w/ Gentoo ?
AFAIK it always has a system set containing things like portage,
toolchain, etc, etc.

> 2) Build your base images without a toolchain, but with portage (and 
> perhaps a very minimal python). This gives you full dependency tracking 
> and obviously bind mount/nfs mount the actual portage tree to avoid 
> space used there. This seems workable and minimal?

Would be still too much for my goals: I don't wanna have anything in 
the final system that's not necessary for runtime. Building and even
installation should be driven from outside (actually, the final system
could even have /bin, /usr, etc mounted ro and /var w/o noexec)

> 3) If we are talking virtual machines then who cares if your containers 
> are individually quite large, if the files in them are duplicated across 
> all containers?  Simply use an appropriate de-duplication strategy to 
> coalesce the space and most of the disadvantages disappear?

Well, that might be an option, if the *is* enough duplication
(within an storage enclousure). But storage space is not the only
reason. My personal background is from medical and industrial
appliances, so I have some more rigid rules ;-o

> Personally I think option 3) is quite interesting from the medium number 
> of virtual machines, ie in the 10s to hundreds, ie simply don't worry 
> about it, let the OS do the work.  In the hundreds to thousands plus 
> level I guess you have unique challenges and I would be wrong to try and 
> suggest a solution from the comfort of a laptop without having that 
> responsibility, but I would have thought there was some advantage in a 
> very rigidly deployed base OS generated and updated very precisely?

I'm thinking of several thousands of vm's for dedicated purposes,
which fall under similar requirements as industrial embedded
systems (eg. there will be practically no human operator).

> >For this we need a different approach (strictly separating build
> >and production environments). Binary distros (eg. Debian) might
> >be one option, but they're lacking the configurability and mostly
> >are still too large. So I'm going a different route using my own
> >buildsystem - called Briegel - which originally was designed for
> >embedded/small-device targets.
> >
> >For now I didn't have the spare time to port all the packages
> >required for complete server systems (most of it is making
> >them all cleanly crosscompile'able, as this is a fundamental
> >concept of Briegel). But maybe you'd like to join in and try it :)
> 
> Sounds like an interesting challenge, but I'm unconvinced you can't 
> solve 90% of your problem within the constraints of Gentoo? This saves 
> you a bunch of time that could be invested in the last 10% through more 
> traditional means?

Maybe. But I need 100%. (cant explain all the reasons here right now).
I'm well aware that this all is far out of Gentoo's scope.

> >I'm not sure if Gentoo really is the right distro for that purpose,
> >as it's targeted to very different systems (i.g. Gentoo boxes are
> >expected to be quite unique, beginning with different per-package
> >useflags, even down to cflags, etc). But it might still be a good
> >basis for building specific system images (let's call them stage5 ;-))
> 
> I won't disagree on your "where it's targeted", but just to re-iterate 
> why I think Gentoo works well is that it does have a very workable 
> binary package feature!

Note that Gentoo's binary packages (which are practically snapshots
of what will be copied in by merge stage) are alway bound to specific
domains, eg. the full configuration (make.conf, useflags, etc, etc),
and there may be a lot of interdependencies between installed packages
(and versions), so it takes some care to use it in stable ways.

This might be very fine for you (eg. if you have dozens of really
equal systems), but you'll have to be aware of the limitations.

> My way of working is to use (several) shared binary package repos and 
> the guests largely pull from those shared package directories.  In fact 
> what I do is have a minimal number of shared "/usr/portage/package" 
> directories and I mount an appropriate one to the guest type at boot 
> time.  At the moment my main two options are "32bit" and "64bit" for the 
> package mounts, but I recently introduced a new machine type which is 
> held back to perl 5.8 and that guest gets it's own package mount since 
> it's obviously linking a lot of binaries differently

As long as you have the same make.conf/useflag/etc settings for all
systems (within a guest type), this might work well.

> Now, the icing is that this works extremely well even once you decide to 
> lightly customise machine types.  So for example my binary packages are 
> very high level (eg 32/64bit), my "profiles" would be fairly high level, 
> eg I have www-apache and www-nginx base profiles.  However, a specific 
> virtual machine running say nginx might itself need a specific PHP 
> application installed, and that itself might need some dependencies, 
> which in turn might require a specific set of customisation of use flags 
> and versions.

At this point, Gentoo's binary packages could become tricky.
The big questions eg. are: can it handle multiple binpkgs of the
same (source) package with differing useflags ? How are version-
specific dependencies handled, etc, etc.

> >An setup for 100 equal webserver vm's could look like this:
> >
> >* run a normal Gentoo vm (tailored for the webserver appliance),
> >   where do you do regular updates (emerge, revdep-rebuild, etc, etc)
> >* from time to time take a snapshot, strip off the buildtime-only
> >   stuff (hmm, could turn out to be a bit tricky ;-o)
> >* this stripped snapshot now goes into testing vm's
> >* when approved, the individual production vm's are switched over
> >   to the new image (maybe using some mount magic, unionfs, etc)
> 
> This could work and perhaps for 100 identical Vms you have enough meat 
> to work on something quite customised anyway?

Yes, in this case, all the VMs have to have identical build configuration.
Once you need things like differing useflag, that approach wont fit anymore.
 
> >At this point I've got a question for to the other folks here:
> >
> >emerge has an --root option which allows to (un)merge in a separate
> >system image. So it should be possible to unmerge a lot of system
> >packages which are just required for updating/building (even
> >portage itself), but this still will be manual - what about
> >dependency handling ?
> 
> This is correct.  In fact this is how you build a stage 1,2,3 etc and 
> how catalyst works!
> 
> The information is a bit spread out over several out of date wiki 
> articles, but perhaps start with:
>     http://en.gentoo-wiki.com/wiki/Tiny_Gentoo
> 
> Roughly speaking you could "freshen" your current installation with 
> (from memory):
>     ROOT="/tmp/new_build" emerge -av world

Is this just the installation root or also the sysroot for the
toolchain ?

<snip>

> Further, if you wish you can make those VMs have a reduced or missing 
> toolchain etc. 

hmm, do I have to unmerge certain packages explicitly or can I
tell it to only install those packages required for runtime in
the first place ?

> >Is there some way to drop at least parts of the standard system set,
> >so eg. portage, python, gcc, etc, etc get unmerged by --depclean
> >if nobody else (in world set) doesn't explicitly require them ?
> 
> You are almost thinking about it all wrong.  ("There is no spoon...")
> 
> This is gentoo, so at this more advanced level, stop thinking about 
> "standard system set" and instead free your mind to start with 
> "nothing".

With "standard system set" I mean those packages which are in @system
on a particular arch. In my vision, @system would be simply empty,
instead @world tells which packages you want and the dependencies
are pulled in automatically.

No idea if that's valid anymore, but some (long?) time ago, there
was the rule that individual packages weren't allowed to explicitly
DEPEND= on system packages. Obviously this wouldn't work with an
empty system set (you'd have to find out the deps and add them
to world set manually).

> Largely the portage dependency tracker will help you pull in the minimal 
> needed dependencies, but beware that system packages arent generally 
> explicitly tracked so you may stumble across some deps when you are 
> going really basic and omiting standard system packages (just use common 
> sense: it should be fairly obvious if an application requires a compiler 
> and you didn't install one then you have a conflict of interest...)

Well, that's where it gets complicated: if some packages depend on some
library libfoo in @system (which I would empty in this case), portage
lacks information to compute the full dependency tree, so I'd have to
find them out and add them to @world manually. Exactly what I do not want.


OTOH, Briegel doenst have that issue, as all dependencies are stated
explicitly in each package (well, with one exception: libc).
(of course, Briegel yet lacks a large package repository as I'm
still working mostly alone on it)


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] avoiding urgent stabilizations

[Ed W]

On 26/02/2011 15:57, Enrico Weigelt wrote:
> * Ed W<lists@wildgooses.com>  schrieb:
>
>> I'm just building some embedded devices on the side using
>> gentoo and my minimal builds are only a few MB?
> How to do you get out all the buildtime stuff (portage, toolchain, etc) ?
>
>> Seems like your complaint is that you have gentoo installs which are
>> full featured with a toolchain and portage, which you are comparing to
>> an installation you built with a different tool that doesn't have a
>> toolchain installed?
> Well, with Briegel, I'm strictly differenciating between the build
> host and the target image. Everything's crosscompiled by design.
> So I can explicitly define which packages need to be in the final
> image (and only those are built). The build host can by any fairly
> recent GNU/Linux system (happens to be Gentoo at my site)

I think you are missing the point that the gentoo build system is quite 
happily building you a destination build with as little as one package 
in it? Whether that destination boots is a function of careful package 
choice, but the point is you are definitely not constrained from 
building a highly minimal target if you wish?

Just from the top of my head I should imagine this would likely boot:

mkdir -p /newroot/{proc,dev,sys}
cp -a /dev/{null,zero} /newroot/dev/
emerge —root /newroot glibc shadow baselayout openrc util-linux udev


If you want to go more minimal then I have a uclibc chroot that I use as 
a build environment, the first line of my build script for my minimal 
embedded device is simply:

emerge --root /embedded baselayout uclibc busybox

I haven't checked, but give or take creating /proc, /sys and some 
minimal devices in /dev, the above should boot fine.



If you were booting into a "container" based virtualisation then you 
don't need to manage /dev and mounting devices and so you can get even 
more minimal than the top example I posted (ie I don't think you need 
udev, etc)

Obviously such minimal builds will have minimal package management, but 
this may not be a problem if you have a super fixed configuration? 
Simply construct your environment such that the data is separate from 
the OS and also make sure any OS customisation is scriptable and now you 
would have a way to occasionally build a new image which you rolled out 
to all virtual machines


>> 1) Build your base images without a toolchain or portage and use a
>> minimal package installer to install pre-built binary packages.
>> This seems fraught with issues long term though...
> How can this be done w/ Gentoo ?
> AFAIK it always has a system set containing things like portage,
> toolchain, etc, etc.

"System" is just a statement of some defaults. At the level you are 
working at you can largely ignore it.

Build what you like into the target build....


> Would be still too much for my goals: I don't wanna have anything in
> the final system that's not necessary for runtime. Building and even
> installation should be driven from outside (actually, the final system
> could even have /bin, /usr, etc mounted ro and /var w/o noexec)

OK, so sounds like a simple build script like the one above would suit you?

For more advanced use consider learning to use catalyst or Funtoos build 
system? These are full on scriptable ways to repeatably build non 
trivial target builds. (remember they can build as little as single 
package targets or more complicated builds)


> Note that Gentoo's binary packages (which are practically snapshots
> of what will be copied in by merge stage) are alway bound to specific
> domains, eg. the full configuration (make.conf, useflags, etc, etc),
> and there may be a lot of interdependencies between installed packages
> (and versions), so it takes some care to use it in stable ways.
>
> This might be very fine for you (eg. if you have dozens of really
> equal systems), but you'll have to be aware of the limitations.

Well, your requirements suggest that you don't need a package manager 
and you will instead build a new read-only OS build every so often and 
deploy that. However, in the interests of exploring the idea further:

- By definition binary packages only work where you have equal systems? 
This is true of gentoo AND rpm/deb.
- If you need binary packages for non equal systems then you need some 
kind of namespacing method to separate the binaries. Gentoo has no 
automated way to do this, but it's reasonably workable to have binary 
package repos and simply tell your guest machine to use the appropriate 
binary repo
- Gentoo's binaries are slightly cleverer than some notice in that they 
have some awareness of when they don't match the target and the operator 
has the option to either ignore that and continue the install, OR to 
ignore the binary package and generate it's own binaries using the local 
compiler, etc.

So, I don't really see the problem, in fact Gentoo binaries can be seen 
as slightly more flexible than RPMs to some extent if you allow them to 
be "optional".


> As long as you have the same make.conf/useflag/etc settings for all
> systems (within a guest type), this might work well.

But that's true of Redhat/Debian/Gentoo?

RPMs don't allow you to have a single binary repo where some of the 
packages are linked against perl 5.8 and some against 5.12? I guess it 
would be possible to enhance the packaging format to cater for that, but 
largely the whole premise of binary packages is that you lock down your 
target options and build all the systems identically? There is no easy 
way around this? (In fact it's what you want)


> At this point, Gentoo's binary packages could become tricky.
> The big questions eg. are: can it handle multiple binpkgs of the
> same (source) package with differing useflags ? How are version-
> specific dependencies handled, etc, etc.

As it stands, no it can't. However, neither can RPM/DEB...

Remember the premise of binaries is that they are all the same... Pick a 
system configuration and build all your binaries with that 
configuration. If you need to support multiple configurations then setup 
multiple binary repositories

>> articles, but perhaps start with:
>>      http://en.gentoo-wiki.com/wiki/Tiny_Gentoo
>>
>> Roughly speaking you could "freshen" your current installation with
>> (from memory):
>>      ROOT="/tmp/new_build" emerge -av world
> Is this just the installation root or also the sysroot for the
> toolchain ?

I think you need to read through the tiny_gentoo type wiki pages. The 
general build process is:

1) Start with a working machine
2) Build a build environment
3) Jump into the build environment and repeat step 2) to build a target 
environment
4) Jump into the target environment

Now in the absence of cross compiling you can often skip step 2) and 
build the target environment directly, but it's normally useful to have 
that build environment so that you can customise things a bit.

Also this will help explain some of the catalyst recipes and why you see 
everything built twice, once to get a build environment, then again to 
build the target build


>> Further, if you wish you can make those VMs have a reduced or missing
>> toolchain etc.
> hmm, do I have to unmerge certain packages explicitly or can I
> tell it to only install those packages required for runtime in
> the first place ?

Well your really simplest "build" would be to take a stage3/4 and simply 
"unmerge" anything you don't need... bit crude though

However, from the email above you can see that it's more desirable to 
instead "build up", starting with an empty build and from there add the 
packages you actually want.


>> You are almost thinking about it all wrong.  ("There is no spoon...")
>>
>> This is gentoo, so at this more advanced level, stop thinking about
>> "standard system set" and instead free your mind to start with
>> "nothing".
> With "standard system set" I mean those packages which are in @system
> on a particular arch. In my vision, @system would be simply empty,
> instead @world tells which packages you want and the dependencies
> are pulled in automatically.

Seriously, "there is no spoon"...

If you want to make your own build script then there isn't even an 
@system...

> No idea if that's valid anymore, but some (long?) time ago, there
> was the rule that individual packages weren't allowed to explicitly
> DEPEND= on system packages. Obviously this wouldn't work with an
> empty system set (you'd have to find out the deps and add them
> to world set manually).

I believe this is correct. However, there aren't many packages in system 
and so whilst it's kind of buyer beware once you start chopping stuff 
out of system, the point is only if you don't install gzip or gcc then 
you need to beware stuff which depends on either.

I think in practice manually managing those dependencies is going to be 
very straightforward, ie you are more going to fix the applications to 
not require the dependencies. After all if you don't want gcc, then you 
really don't want gcc, simple really... eg, if some simple application 
uses make to update it's config files then you will likely look at some 
workaround rather than installing make...


> Well, that's where it gets complicated: if some packages depend on some
> library libfoo in @system (which I would empty in this case), portage
> lacks information to compute the full dependency tree, so I'd have to
> find them out and add them to @world manually. Exactly what I do not want.

Oh come on. You are working at the level of building your OS image from 
scratch and you don't know if any of your applications require (at 
runtime) gcc, bison, m4, wget, man, os-headers, portage or ssh?

It seems likely that you can quickly determine if you can live without 
most of the above (at runtime)...?!!

> OTOH, Briegel doenst have that issue, as all dependencies are stated
> explicitly in each package (well, with one exception: libc).

So, you can probably already use that information to figure out your 
minimal set of system packages that you likely need (for runtime).

Just checking that Briegel tracks both runtime and build time 
dependencies separately? eg you might need gzip to build something, but 
not to run it?


So far I think you could easily convert your briegel recipes to a simple 
ebuild script and run them side by side? It's actually such an easy 
thing to do, to build a new stage2/3/4 that it's a shame folks don't do 
it more!

Good luck

Ed W