[gentoo-doc] Portage per-package environment/behavior

[gentoo-doc] Portage per-package environment/behavior

[Sven Vermeulen]

Hi guys,

I noticed we don't describe in the handbook that Portage can have
per-package environment variables (like CFLAGS) through /etc/portage/env.
This can be even (ab?)used to execute steps before or after specific phases
(based on the EBUILD_PHASE information), something I use for updating IDS
systems (postinst/prerm phase).

But I'm not sure if and where in the handbook this can be positioned best.
The environment variable stuff could be placed in the section on
"Environment Variables", but is quite off from the rest of the content
(since the rest of that chapter has nothing really to do with Portage or
build environments).

"Configuring through Variables" is probably the best location (somewhere in
the beginning as we talk there about Build-specific Options), but I do feel
that this particular feature is already more targeting advanced users, where
the location in the handbook somewhat suggests this for more beginner-like
types.

Perhaps another section in "Working with Portage", called "Advanced Portage
Features" or so? This can then contain the per-package env information, but
also overriding profile information and perhaps others we don't talk about
yet.

Any ideas on this?

	Sven

Re: [gentoo-doc] Portage per-package environment/behavior

[wireless]

On 12/28/11 05:14, Sven Vermeulen wrote:
> Hi guys,
> 
> I noticed we don't describe in the handbook that Portage can have
> per-package environment variables (like CFLAGS) through /etc/portage/env.
> This can be even (ab?)used to execute steps before or after specific phases
> (based on the EBUILD_PHASE information), something I use for updating IDS
> systems (postinst/prerm phase).
> 
> But I'm not sure if and where in the handbook this can be positioned best.
> The environment variable stuff could be placed in the section on
> "Environment Variables", but is quite off from the rest of the content
> (since the rest of that chapter has nothing really to do with Portage or
> build environments).
> 
> "Configuring through Variables" is probably the best location (somewhere in
> the beginning as we talk there about Build-specific Options), but I do feel
> that this particular feature is already more targeting advanced users, where
> the location in the handbook somewhat suggests this for more beginner-like
> types.
> 
> Perhaps another section in "Working with Portage", called "Advanced Portage
> Features" or so? This can then contain the per-package env information, but
> also overriding profile information and perhaps others we don't talk about
> yet.
> 
> Any ideas on this?
> 
> 	Sven

Well, imho, a handbook installation is about basics and during the
initial build, particularly for one new to gentoo, extensive
flag settings (customization via /etc/portage) might cause more
problems than any gains might achieve. Remember we do that the
various profiles for the different desktop and server configs.

What would be nice is to created stripped down versions of the handbook
(on the WIKI) where the focus is more highly specialized configuration
for the use of the newly installed (gentoo) machine. Also on the gentoo
wiki, we can lower the bar, and let those accomplished individuals
create something cool, and a developing admin take it over and
maintain the document, or extend  it to different hardware platforms.
Flag customization could easily differ on different platforms.
Why, one could even use embedded gentoo with uClib as a basis
for a targeted, customized build of Gentoo on the new installation
(hardware).

(side note, if AMD follows through and starts offering
ARM (A15) machines, our current handbook will be ripped
at the seams, imho.

Some examples for the WIKI:
An Apache server:
 complete with either hardened or SElinux setup,
including packages such as perl, python, php (or whatever mix).
There, your customized flag settings would be focused and very
keenly appreciated for an intermediate level gentoo admin to
leverage.

Some other intermediate level ideas for the WIKI:

Light-weight workstation:
with gui for older hardware (not Gnome or KDE)

secure/encrypted mail server

firewall (with DMZ for servers).

transparent bridge.

transparent sniffer:
 with lots of ethernet interfaces set up in stealth mode
(taps to other secure ethernet segments for passive monitoring.


In these (and many others exist) customization as well as minimization
of flag settings would be of keen interest to the wider, intermediate
gentoo community.  God forbid one of the really sharp (gentoo) hacks
share a little magic with us commoners?

So keep the handbook the same but start experimenting (via the WIKI)
on derivatives that are more targeted and focused on customizations,
including but not limited to stringent flag settings

Just my thoughts, YMMV.

James

[gentoo-doc] Re: Portage per-package environment/behavior

[Duncan]

Sven Vermeulen posted on Wed, 28 Dec 2011 11:14:49 +0100 as excerpted:

> I noticed we don't describe in the handbook that Portage can have
> per-package environment variables (like CFLAGS) through
> /etc/portage/env.

IIRC this (and /etc/portage/patches) was originally deliberately kept an 
"undocumented feature", to avoid trouble with missing info in bug 
reports, since emerge info didn't report package specific information.

Now that portage has emerge --info <pkg> and it's what's requested in the 
usual output when a build aborts (tho AFAIK bugzilla still mentions 
emerge info output only), and the logs themselves are rather better at 
reporting user patches (for the patches subdir), that's not so much of a 
problem, and the portage documentation itself mentions both usages of the 
env subdir (tho not the patches subdir since that's still ebuild 
activated).

But the handbooks were rather stale before you came back and started the 
update push (thanks again =:^), and this is the first time I believe I've 
seen the issue raised.

I like the idea and believe it best fits in working with portage.  The 
new "advanced features" section you proposed sounds good.

Now that ccache has been deemphasized, perhaps move it there as well, 
getting it a bit further out of the the newbie path. (Tho I personally 
use it and have never had an issue... and don't necessarily agree with 
the deemphasis since it really helps speed up rebuilds in the case where 
users are likely to be most frustrated, a broken build that's later 
fixed, or that aborted due to random job-token issues that aren't 
repeatable, etc.  But I understand why it's being done, from the gentoo-
dev perspective.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-doc] Re: Portage per-package environment/behavior

[Sven Vermeulen]

On Wed, Dec 28, 2011 at 05:47:18PM +0000, Duncan wrote:
> IIRC this (and /etc/portage/patches) was originally deliberately kept an 
> "undocumented feature", to avoid trouble with missing info in bug 
> reports, since emerge info didn't report package specific information.

This "/etc/portage/patches" thingie is something that the ebuilds must
support themselves afaik (call epatch_user), although I can imagine a way to
have it triggered for all systems by fiddling with /etc/portage/env too ;-)

> Now that portage has emerge --info <pkg> and it's what's requested in the 
> usual output when a build aborts (tho AFAIK bugzilla still mentions 
> emerge info output only), and the logs themselves are rather better at 
> reporting user patches (for the patches subdir), that's not so much of a 
> problem, and the portage documentation itself mentions both usages of the 
> env subdir (tho not the patches subdir since that's still ebuild 
> activated).

We might want to catch some bugzie admins then and suggest to update their
page ;-)

Wkr,
	Sven Vermeulen
 

Re: [gentoo-doc] Portage per-package environment/behavior

[Matthew Summers]

On Wed, Dec 28, 2011 at 4:14 AM, Sven Vermeulen
<sven.vermeulen@siphos.be> wrote:
> Hi guys,
>
> I noticed we don't describe in the handbook that Portage can have
> per-package environment variables (like CFLAGS) through /etc/portage/env.
> This can be even (ab?)used to execute steps before or after specific phases
> (based on the EBUILD_PHASE information), something I use for updating IDS
> systems (postinst/prerm phase).
>
> But I'm not sure if and where in the handbook this can be positioned best.
> The environment variable stuff could be placed in the section on
> "Environment Variables", but is quite off from the rest of the content
> (since the rest of that chapter has nothing really to do with Portage or
> build environments).
>
> "Configuring through Variables" is probably the best location (somewhere in
> the beginning as we talk there about Build-specific Options), but I do feel
> that this particular feature is already more targeting advanced users, where
> the location in the handbook somewhat suggests this for more beginner-like
> types.
>
> Perhaps another section in "Working with Portage", called "Advanced Portage
> Features" or so? This can then contain the per-package env information, but
> also overriding profile information and perhaps others we don't talk about
> yet.
>
> Any ideas on this?
>
>        Sven
>

Hey Sven,

Thanks for bringing this up. Quite a long while ago I talked with Zac
about this very issue, that some of the advanced portage features were
not documented in an user friendly way. It seemed to me that these
features are outside the scope of the current handbook. However, your
idea about extending the chapter on "Working with Portage" brings me
around a little. In fact, I think it may well be an excellent place
for this sort of thing. My only concern would be that these advanced
features might be misused and create extra bug work for the wranglers.

In any case, I would enjoy working with you on this in some capacity,
as its one of the many things I have wanted to do myself for a long
time. Its always more enjoyable when collaborating anyway.

Thanks for the initiative!

Matt
-- 
Matthew W. Summers
Gentoo Foundation Inc.

Re: [gentoo-doc] Portage per-package environment/behavior

[Sven Vermeulen]

On Wed, Dec 28, 2011 at 11:14:49AM +0100, Sven Vermeulen wrote:
> I noticed we don't describe in the handbook that Portage can have
> per-package environment variables (like CFLAGS) through /etc/portage/env.
> This can be even (ab?)used to execute steps before or after specific phases
> (based on the EBUILD_PHASE information), something I use for updating IDS
> systems (postinst/prerm phase).

So, considering the idea of having a section added to "Working with Portage"
called "Advanced Portage Features", this is what it could look like:

http://dev.gentoo.org/~swift/docs/previews/hb-portage-advanced.xml

Ideas? Thoughts?

Wkr,
	Sven Vermeulen

[gentoo-doc] Re: Portage per-package environment/behavior

[Duncan]

Sven Vermeulen posted on Thu, 29 Dec 2011 18:18:07 +0000 as excerpted:

> On Wed, Dec 28, 2011 at 11:14:49AM +0100, Sven Vermeulen wrote:
>> I noticed we don't describe in the handbook that Portage can have
>> per-package environment variables (like CFLAGS) through
>> /etc/portage/env. This can be even (ab?)used to execute steps before or
>> after specific phases (based on the EBUILD_PHASE information),
>> something I use for updating IDS systems (postinst/prerm phase).
> 
> So, considering the idea of having a section added to "Working with
> Portage"
> called "Advanced Portage Features", this is what it could look like:
> 
> http://dev.gentoo.org/~swift/docs/previews/hb-portage-advanced.xml

Looks quite good to me so far. =:^)

But all sections were section 1.  Is that simply due to the prototype 
process or is it a mistake?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-doc] Re: Portage per-package environment/behavior

[Sven Vermeulen]

On Fri, Dec 30, 2011 at 01:56:12AM +0000, Duncan wrote:
> > http://dev.gentoo.org/~swift/docs/previews/hb-portage-advanced.xml
> 
> Looks quite good to me so far. =:^)
> 
> But all sections were section 1.  Is that simply due to the prototype 
> process or is it a mistake?

It's due to the prototype (actually, the numbering will only be correct the
moment the chapter is integrated in the handbook) because the URL points to
the file itself (rather than using the "handbook.xml?part=3&chap=5"
approach.

Wkr,
	Sven Vermeulen

Re: [gentoo-doc] Portage per-package environment/behavior

[Sven Vermeulen]

On Thu, Dec 29, 2011 at 06:18:07PM +0000, Sven Vermeulen wrote:
> So, considering the idea of having a section added to "Working with Portage"
> called "Advanced Portage Features", this is what it could look like:
> 
> http://dev.gentoo.org/~swift/docs/previews/hb-portage-advanced.xml

Bug #396549 is used for tracking. I also added a section in it about
/etc/portage/postsync.d.

Wkr,
	Sven Vermeulen

[gentoo-doc] Re: Portage per-package environment/behavior

[Duncan]

Sven Vermeulen posted on Sat, 31 Dec 2011 20:16:35 +0000 as excerpted:

> On Thu, Dec 29, 2011 at 06:18:07PM +0000, Sven Vermeulen wrote:
>> So, considering the idea of having a section added to "Working with
>> Portage"
>> called "Advanced Portage Features", this is what it could look like:
>> 
>> http://dev.gentoo.org/~swift/docs/previews/hb-portage-advanced.xml
> 
> Bug #396549 is used for tracking. I also added a section in it about
> /etc/portage/postsync.d.

Good.

I thought about suggesting a postsync.d mention too, but decided that it 
might be old/stale cruft related to the ed catmur scripts that I used to 
run that seem to be the basis of some of these recent portage extensions 
(like /etc/portage/patches, and this one too, IIRC) as I didn't see 
anything in the portage (5) manpage about it.

So unless the portage docs for postsync.d are there and I just missed 
them, you/I/we might want to ask the portage folks about including the 
documentation in the portage manpage as well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-doc] Portage per-package environment/behavior

[Joshua Saddler]

[-- Attachment #1: Type: text/plain, Size: 1714 bytes --]

On Wed, 28 Dec 2011 11:14:49 +0100
Sven Vermeulen <sven.vermeulen@siphos.be> wrote:

> Hi guys,
> 
> I noticed we don't describe in the handbook that Portage can have
> per-package environment variables (like CFLAGS)
> through /etc/portage/env. This can be even (ab?)used to execute
> steps before or after specific phases (based on the EBUILD_PHASE
> information), something I use for updating IDS systems
> (postinst/prerm phase).
> 
> But I'm not sure if and where in the handbook this can be
> positioned best. The environment variable stuff could be placed in
> the section on "Environment Variables", but is quite off from the
> rest of the content (since the rest of that chapter has nothing
> really to do with Portage or build environments).
> 
> "Configuring through Variables" is probably the best location
> (somewhere in the beginning as we talk there about Build-specific
> Options), but I do feel that this particular feature is already
> more targeting advanced users, where the location in the handbook
> somewhat suggests this for more beginner-like types.
> 
> Perhaps another section in "Working with Portage", called "Advanced
> Portage Features" or so? This can then contain the per-package env
> information, but also overriding profile information and perhaps
> others we don't talk about yet.
> 
> Any ideas on this?
> 
> 	Sven
> 

per-package cflags has never been an officially supported portage
feature. in fact, it's mostly been an env hack that users have been
actively discouraged from using. they can expect to see bugs closed
RESO WONTFIX when per-package cflag monkeying has been detected. as
such, we shouldn't be documenting how to do it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-doc] Portage per-package environment/behavior

[Sven Vermeulen]

On Sat, Dec 31, 2011 at 07:13:59PM -0800, Joshua Saddler wrote:
> per-package cflags has never been an officially supported portage
> feature. in fact, it's mostly been an env hack that users have been
> actively discouraged from using. they can expect to see bugs closed
> RESO WONTFIX when per-package cflag monkeying has been detected. as
> such, we shouldn't be documenting how to do it.

I disagree. It is actively being suggested on #gentoo, and we already
document features that could (or even will) result in RESO:WONTFIX, like
using ~arch (both for a small set of packages as well as the entire system).

I would rather have a <warn> in place then that sais it will hinder support
during bug reports and might result in RESO:WONTFIX unless the user removes
the specific settings and rebuilds with the system ones.

Not documenting is, in my opinion, a lesser solution than documenting and
saying why it is bad.

Wkr,
	Sven Vermeulen

Re: [gentoo-doc] Portage per-package environment/behavior

[Joshua Saddler]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

On Sun, 1 Jan 2012 08:59:25 +0000
Sven Vermeulen <swift@gentoo.org> wrote:

> On Sat, Dec 31, 2011 at 07:13:59PM -0800, Joshua Saddler wrote:
> > per-package cflags has never been an officially supported portage
> > feature. in fact, it's mostly been an env hack that users have
> > been actively discouraged from using. they can expect to see bugs
> > closed RESO WONTFIX when per-package cflag monkeying has been
> > detected. as such, we shouldn't be documenting how to do it.
> 
> I disagree. It is actively being suggested on #gentoo, and we
> already document features that could (or even will) result in
> RESO:WONTFIX, like using ~arch (both for a small set of packages as
> well as the entire system).

and who's suggesting it on #gentoo? users? devs? we need to talk to
the portage team and anyone else that would have to troubleshoot
build fixes caused by (ab)using it. if we get the okay from them,
then something like your proposed text should be added to the
handbook.

> I would rather have a <warn> in place then that sais it will hinder
> support during bug reports and might result in RESO:WONTFIX unless
> the user removes the specific settings and rebuilds with the system
> ones.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-doc] Portage per-package environment/behavior

[Sven Vermeulen]

On Mon, Jan 02, 2012 at 10:54:20AM -0800, Joshua Saddler wrote:
> > I disagree. It is actively being suggested on #gentoo, and we
> > already document features that could (or even will) result in
> > RESO:WONTFIX, like using ~arch (both for a small set of packages as
> > well as the entire system).
> 
> and who's suggesting it on #gentoo? users? devs? we need to talk to
> the portage team and anyone else that would have to troubleshoot
> build fixes caused by (ab)using it. if we get the okay from them,
> then something like your proposed text should be added to the
> handbook.

I'll bring it through -dev just to be sure.

Regarding #gentoo suggestions, a quick grep in my logs shows it being
mentinoed about twice a week, mostly by users (there aren't that many
developers on it) but also a few developers (amongst some in the qa team).

Of course, that's not considering the context of the discussion - it may be
very well a specific scenario there.

Wkr,
	Sven Vermeulen

[gentoo-doc] Re: Portage per-package environment/behavior

[Duncan]

Joshua Saddler posted on Sat, 31 Dec 2011 19:13:59 -0800 as excerpted:

> per-package cflags has never been an officially supported portage
> feature. in fact, it's mostly been an env hack that users have been
> actively discouraged from using. they can expect to see bugs closed RESO
> WONTFIX when per-package cflag monkeying has been detected. as such, we
> shouldn't be documenting how to do it.

You are correct that it used to be that way, but isn't that what emerge
--info <pkg>  was designed to fix and why portage's error messages are  
far more specific now about both that and various log files that should 
be posted than they used to be?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman