[gentoo-dev] stacking profile.bashrc?

[gentoo-dev] stacking profile.bashrc?

[Michael Haubenwallner]

Hi,

I'm facing this problem in Prefix on AIX, although it is a generic
problem IMO:

We do have post_src_install() hook in profiles/prefix/profile.bashrc to
drop charset.alias for packages != libiconv, required for non-glibc
platforms.

In profiles/prefix/aix/profiles.bashrc, there is another
post_src_install() hook with some aix specific hacks, required for
portage to allow for merging shared libraries there.

The problem now is that the latter post_src_install() overrides the
former, and I get collisions on charset.alias as the former is not
executed.

Is this a portage bug (each of them should be executed)?
Or is there some defined way to handle this I just was unable to find?

Thank you!
/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

Re: [gentoo-dev] stacking profile.bashrc?

[Zac Medico]

Michael Haubenwallner wrote:
> Hi,
> 
> I'm facing this problem in Prefix on AIX, although it is a generic
> problem IMO:
> 
> We do have post_src_install() hook in profiles/prefix/profile.bashrc to
> drop charset.alias for packages != libiconv, required for non-glibc
> platforms.
> 
> In profiles/prefix/aix/profiles.bashrc, there is another
> post_src_install() hook with some aix specific hacks, required for
> portage to allow for merging shared libraries there.
> 
> The problem now is that the latter post_src_install() overrides the
> former, and I get collisions on charset.alias as the former is not
> executed.
> 
> Is this a portage bug (each of them should be executed)?
> Or is there some defined way to handle this I just was unable to find?

The pre/post phase hooks are not designed for this, they are only
intended for users to put in /etc/portage/bashrc. For what you are
trying to do, it seems like a registration interface would be more
appropriate (something like register_die_hook). Maybe the usage
could go something like this:

  register_phase_hook install post my_post_src_install

> Thank you!
> /haubi/


-- 
Thanks,
Zac

Re: [gentoo-dev] stacking profile.bashrc?

[Michael Haubenwallner]

On Wed, 2009-07-15 at 11:39 -0700, Zac Medico wrote:
> Michael Haubenwallner wrote:
 
> > We do have post_src_install() hook in profiles/prefix/profile.bashrc ...
> > In profiles/prefix/aix/profiles.bashrc, there is another
> > post_src_install() hook ...
> > The problem now is that the latter post_src_install() overrides the
> > former, ...

> The pre/post phase hooks are not designed for this, they are only
> intended for users to put in /etc/portage/bashrc. For what you are
> trying to do, it seems like a registration interface would be more
> appropriate (something like register_die_hook).

Should this registration api be provided by portage, or by
base/profile.bashrc?
Hmm, will have to be portage, because /etc/portage/bashrc would override
base/profile.bashrc too. base/profile.bashrc just might jump in when not
provided by installed portage yet.

>  Maybe the usage
> could go something like this:
> 
>   register_phase_hook install post my_post_src_install

For readability, I'd more like to have 'post' in front of 'install'.

Or eventually something like:

        register_phase_hook \
          --before-src_install my_pre_src_install_1 my_pre_src_install_2 \
          --after-src_install my_post_src_install_1 my_post_src_install_2 \
          --before-<phase-function-name> <multiple hooks> 

Or with phase-names instead of phase-function-names:

        register_phase_hook \
          --before-{setup,nofetch,unpack,prepare,configure,compile,...} <hooks> \
          --after-{...,test,install,preinst,postinst,prerm,postrm} <hooks>

Suggesting "before&after" here as I'm always confused with
pre_pkg_preinst, post_pkg_preinst, pre_pkg_postinst, post_pkg_postinst,
pre_pkg_prerm, post_pkg_prerm, pre_pkg_postrm, post_pkg_prerm.

Anyway: besides implementation, what else does it need to get this done?

Thank you!

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

Re: [gentoo-dev] stacking profile.bashrc?

[Zac Medico]

Michael Haubenwallner wrote:
> On Wed, 2009-07-15 at 11:39 -0700, Zac Medico wrote:
>> Michael Haubenwallner wrote:
>  
>>> We do have post_src_install() hook in profiles/prefix/profile.bashrc ...
>>> In profiles/prefix/aix/profiles.bashrc, there is another
>>> post_src_install() hook ...
>>> The problem now is that the latter post_src_install() overrides the
>>> former, ...
> 
>> The pre/post phase hooks are not designed for this, they are only
>> intended for users to put in /etc/portage/bashrc. For what you are
>> trying to do, it seems like a registration interface would be more
>> appropriate (something like register_die_hook).
> 
> Should this registration api be provided by portage, or by
> base/profile.bashrc?
> Hmm, will have to be portage, because /etc/portage/bashrc would override
> base/profile.bashrc too. base/profile.bashrc just might jump in when not
> provided by installed portage yet.
> 
>>  Maybe the usage
>> could go something like this:
>>
>>   register_phase_hook install post my_post_src_install
> 
> For readability, I'd more like to have 'post' in front of 'install'.
> 
> Or eventually something like:
> 
>         register_phase_hook \
>           --before-src_install my_pre_src_install_1 my_pre_src_install_2 \
>           --after-src_install my_post_src_install_1 my_post_src_install_2 \
>           --before-<phase-function-name> <multiple hooks> 
> 
> Or with phase-names instead of phase-function-names:
> 
>         register_phase_hook \
>           --before-{setup,nofetch,unpack,prepare,configure,compile,...} <hooks> \
>           --after-{...,test,install,preinst,postinst,prerm,postrm} <hooks>
> 
> Suggesting "before&after" here as I'm always confused with
> pre_pkg_preinst, post_pkg_preinst, pre_pkg_postinst, post_pkg_postinst,
> pre_pkg_prerm, post_pkg_prerm, pre_pkg_postrm, post_pkg_prerm.
> 
> Anyway: besides implementation, what else does it need to get this done?

The specification is really the most important part, and you have to
give the -dev community an opportunity to participate in refining
the spec (via RFC email, GLEP, or whatnot).

It seems like this idea will probably serve for bug 179800, which is
about allowing eclasses to register phase hooks:

  http://bugs.gentoo.org/show_bug.cgi?id=179800

> 
> Thank you!
> 
> /haubi/


-- 
Thanks,
Zac

[gentoo-dev] Re: stacking profile.bashrc?

[Steven J Long]

Zac Medico wrote:

> The specification is really the most important part, and you have to
> give the -dev community an opportunity to participate in refining
> the spec (via RFC email, GLEP, or whatnot).
> 
> It seems like this idea will probably serve for bug 179800, which is
> about allowing eclasses to register phase hooks:
> 
>   http://bugs.gentoo.org/show_bug.cgi?id=179800
>
Hmm given that this relies on profile.bashrc, in specification terms one
would have to ensure that http://bugs.gentoo.org/202631 (which was recently
raised in #-council) were resolved. The sunrise people raised being able to
tweak bashrc per-overlay in #-portage recently, and the phase hooks were
also raised by javaJake wrt having directory-based hooks.

As outlined at:
http://www.mail-archive.com/gentoo-portage-dev%40lists.gentoo.org/msg00544.html
..the phase hooks can all be done via bashrc; pre- and post-pkg need mangler
support, if they cannot reasonably be implemented via existing hooks.
(There were others, that seemed more appropriately handled in a wrapper,
imo. Pre- and post- the whole process iirc.)
 
OFC for that to happen, the spec needs to acknowledge that bashrc's exist so
that downstream can use them in overlays, irrespective of the user's choice
of package-mangler.
-- 
#friendly-coders -- We're friendly but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: stacking profile.bashrc?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

On Mon, 20 Jul 2009 05:52:30 +0100
Steven J Long <slong@rathaus.eclipse.co.uk> wrote:
> Hmm given that this relies on profile.bashrc, in specification terms
> one would have to ensure that http://bugs.gentoo.org/202631 (which
> was recently raised in #-council) were resolved.

No, because as you've repeatedly been told, profile.bashrc lies outside
of the specification precisely to allow the kind of Portage-internals
trickery that's being discussed here.

> OFC for that to happen, the spec needs to acknowledge that bashrc's
> exist so that downstream can use them in overlays, irrespective of
> the user's choice of package-mangler.

If profile.bashrc were to be specified, none of this would be legal,
since it would have to restrict the contents to portable code.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: stacking profile.bashrc?

[Zac Medico]

Steven J Long wrote:
> Zac Medico wrote:
> 
>> The specification is really the most important part, and you have to
>> give the -dev community an opportunity to participate in refining
>> the spec (via RFC email, GLEP, or whatnot).
>>
>> It seems like this idea will probably serve for bug 179800, which is
>> about allowing eclasses to register phase hooks:
>>
>>   http://bugs.gentoo.org/show_bug.cgi?id=179800
>>
> Hmm given that this relies on profile.bashrc, in specification terms one
> would have to ensure that http://bugs.gentoo.org/202631 (which was recently
> raised in #-council) were resolved. The sunrise people raised being able to
> tweak bashrc per-overlay in #-portage recently, and the phase hooks were
> also raised by javaJake wrt having directory-based hooks.

For the overlay people, it seems like we need something that's a
little different from the existing profile support, since profile's
are user-selectable and it seems like you want something that's
global/mandatory at the repository level. For example, it could be
located at profiles/profile.bashrc, similar to
profiles.package.mask, which is global/mandatory rather than being
part of a specific user-selectable profile.
-- 
Thanks,
Zac

[gentoo-dev] Re: Re: stacking profile.bashrc?

[Steven J Long]

Zac Medico wrote:

> Steven J Long wrote:
>> Zac Medico wrote:
>> 
>>> The specification is really the most important part, and you have to
>>> give the -dev community an opportunity to participate in refining
>>> the spec (via RFC email, GLEP, or whatnot).
>>>
>>> It seems like this idea will probably serve for bug 179800, which is
>>> about allowing eclasses to register phase hooks:
>>>
>>>   http://bugs.gentoo.org/show_bug.cgi?id=179800
>>>
>> Hmm given that this relies on profile.bashrc, in specification terms one
>> would have to ensure that http://bugs.gentoo.org/202631 (which was
>> recently raised in #-council) were resolved. The sunrise people raised
>> being able to tweak bashrc per-overlay in #-portage recently, and the
>> phase hooks were also raised by javaJake wrt having directory-based
>> hooks.
> 
> For the overlay people, it seems like we need something that's a
> little different from the existing profile support, since profile's
> are user-selectable and it seems like you want something that's
> global/mandatory at the repository level. For example, it could be
> located at profiles/profile.bashrc, similar to
> profiles.package.mask, which is global/mandatory rather than being
> part of a specific user-selectable profile.

Yeah sounds right. Perhaps a per-category bashrc split (both for
usual /etc/portage case and for overlays) might also be useful?
(Overlay admin can always test PN should the need arise.)

You mentioned in #-portage that per-phase execution is no longer used, wrt
how overlays would only be executing bashrc at start. I take it we can
still test $EBUILD_PHASE? (Sorry if I've misunderstood what you were
saying.)
-- 
#friendly-coders -- We're friendly but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: Re: stacking profile.bashrc?

[Zac Medico]

Steven J Long wrote:
> Yeah sounds right. Perhaps a per-category bashrc split (both for
> usual /etc/portage case and for overlays) might also be useful?
> (Overlay admin can always test PN should the need arise.)

Maybe that's more in the domain of eclasses (or some sort of elibs
is they don't need to change metadata), in cases when it's easy
enough to inherit whichever ones are needed.

> You mentioned in #-portage that per-phase execution is no longer used, wrt
> how overlays would only be executing bashrc at start. I take it we can
> still test $EBUILD_PHASE? (Sorry if I've misunderstood what you were
> saying.)

Current portage bashrc support allows $EBUILD_PHASE conditionals,
but I think in the long term we may want to drop that in favor of
function hooks that are sourced only once. The $EBUILD_PHASE
conditional approach just seems somewhat clumsy in comparison
(sourcing the bashrc during every phase might also be considered
inefficient/ugly).
-- 
Thanks,
Zac

[gentoo-dev] Re: Re: Re: stacking profile.bashrc?

[Steven J Long]

Zac Medico wrote:

> Steven J Long wrote:
>> Yeah sounds right. Perhaps a per-category bashrc split (both for
>> usual /etc/portage case and for overlays) might also be useful?
>> (Overlay admin can always test PN should the need arise.)
> 
> Maybe that's more in the domain of eclasses (or some sort of elibs
> is they don't need to change metadata), in cases when it's easy
> enough to inherit whichever ones are needed.
>
Well the directory-based approach is for network/overlay admins or
downstream distros to tweak stuff without needing to edit and digest
ebuilds in a local overlay. JavaJake wanted to split the actual phases, so
we have a directory per-phase, which can ofc easily be done with a bit of
BASH or shell-script from the existing bashrc. This seems more useful for
end-user admins (whether local or network.)

For an overlay, from what I've seen in my limited exposure to the issue,
there is more of a need for influencing metadata, eg IUSE. I hesitate to
speak for the sunrise bods, and any other overlay admins, on this, though,
so if you're reading this, guys, please chip in. :) That ties in more with
the next point; although as you say it could be done by inheritance from an
eclass, again that potentially involves editing the ebuild.

With the existing bashrc capability end-users can do all this ourselves;
it'd just be nice to be able to do it in overlays, and for what we already
have to be specified since it applies to both pkgcore and portage, and has
done for several years.
 
>> You mentioned in #-portage that per-phase execution is no longer used,
>> wrt how overlays would only be executing bashrc at start. I take it we
>> can still test $EBUILD_PHASE? (Sorry if I've misunderstood what you were
>> saying.)
> 
> Current portage bashrc support allows $EBUILD_PHASE conditionals,
> but I think in the long term we may want to drop that in favor of
> function hooks that are sourced only once. The $EBUILD_PHASE
> conditional approach just seems somewhat clumsy in comparison
> (sourcing the bashrc during every phase might also be considered
> inefficient/ugly).

My only concern here is that changes the admin makes, eg in
post_pkg_postinst, won't be reflected in uninstalling a package which was
installed before the change. For the DEPEND phase, as in IUSE modification,
that's not so much a problem afaict, since a) it's not typically what
network admins want to tweak, and b) it's right at the start of the whole
process. Any clarity you want to add will be gratefully received ;)

Regards,
Steve.
-- 
#friendly-coders -- We're friendly but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: Re: Re: stacking profile.bashrc?

[Zac Medico]

Steven J Long wrote:
> Zac Medico wrote:
> 
>> Steven J Long wrote:
>>> Yeah sounds right. Perhaps a per-category bashrc split (both for
>>> usual /etc/portage case and for overlays) might also be useful?
>>> (Overlay admin can always test PN should the need arise.)
>> Maybe that's more in the domain of eclasses (or some sort of elibs
>> is they don't need to change metadata), in cases when it's easy
>> enough to inherit whichever ones are needed.
>>
> Well the directory-based approach is for network/overlay admins or
> downstream distros to tweak stuff without needing to edit and digest
> ebuilds in a local overlay. JavaJake wanted to split the actual phases, so
> we have a directory per-phase, which can ofc easily be done with a bit of
> BASH or shell-script from the existing bashrc. This seems more useful for
> end-user admins (whether local or network.)

That seems like a reasonable use-case for per-phase sourcing.

> For an overlay, from what I've seen in my limited exposure to the issue,
> there is more of a need for influencing metadata, eg IUSE. I hesitate to
> speak for the sunrise bods, and any other overlay admins, on this, though,
> so if you're reading this, guys, please chip in. :) That ties in more with
> the next point; although as you say it could be done by inheritance from an
> eclass, again that potentially involves editing the ebuild.

Note that any changes to ebuild metadata generation mean changes in
metadata cache validation. For example, if you want to modify ebuild
metadata from profile.bashrc, then you have to make sure to
invalidate metadata cache any time the profile.bashrc is modified.
If such a change affects all ebuilds in the repo, it can be time
consuming to regenerate all of the metadata cache. Also, if the
cache validation mechanism isn't robust enough, users will
experience annoying issues with stale cache. For example see this bug:

  http://bugs.gentoo.org/show_bug.cgi?id=139134

Also, see this related discussion on cache validation:


http://archives.gentoo.org/gentoo-dev/msg_cfa80e33ee5fa6f854120ddfb9b468b3.xml

> With the existing bashrc capability end-users can do all this ourselves;
> it'd just be nice to be able to do it in overlays, and for what we already
> have to be specified since it applies to both pkgcore and portage, and has
> done for several years.

We might want to have two separate bashrcs. One that's per-phase,
and another one that's only sourced once. If it's only sourced once
then it might allow you to make more radical changes that you
couldn't otherwise make without breaking uninstall phases in some way.

>>> You mentioned in #-portage that per-phase execution is no longer used,
>>> wrt how overlays would only be executing bashrc at start. I take it we
>>> can still test $EBUILD_PHASE? (Sorry if I've misunderstood what you were
>>> saying.)
>> Current portage bashrc support allows $EBUILD_PHASE conditionals,
>> but I think in the long term we may want to drop that in favor of
>> function hooks that are sourced only once. The $EBUILD_PHASE
>> conditional approach just seems somewhat clumsy in comparison
>> (sourcing the bashrc during every phase might also be considered
>> inefficient/ugly).
> 
> My only concern here is that changes the admin makes, eg in
> post_pkg_postinst, won't be reflected in uninstalling a package which was

I assume you mean post_pkg_postrm, since post_pkg_postinst is never
executed during uninstall. Well, it is for the replacing package, if
there is one, but that should have the latest environment anyway (at
least, assuming it's not a binary package).

> installed before the change. For the DEPEND phase, as in IUSE modification,
> that's not so much a problem afaict, since a) it's not typically what
> network admins want to tweak, and b) it's right at the start of the whole
> process. Any clarity you want to add will be gratefully received ;)

Again, metadata changes introduce cache validation issues that need
to be carefully considered.

> Regards,
> Steve.


-- 
Thanks,
Zac

[gentoo-dev] Re: Re: Re: Re: stacking profile.bashrc?

[Steven J Long]

Zac Medico wrote:

> Steven J Long wrote:
>> Zac Medico wrote:
>> 
>>> Steven J Long wrote:
>>>> Yeah sounds right. Perhaps a per-category bashrc split (both for
>>>> usual /etc/portage case and for overlays) might also be useful?
>>>> (Overlay admin can always test PN should the need arise.)
>>> Maybe that's more in the domain of eclasses (or some sort of elibs
>>> is they don't need to change metadata), in cases when it's easy
>>> enough to inherit whichever ones are needed.
>>>
>> Well the directory-based approach is for network/overlay admins or
>> downstream distros to tweak stuff without needing to edit and digest
>> ebuilds in a local overlay. JavaJake wanted to split the actual phases,
>> so we have a directory per-phase, which can ofc easily be done with a bit
>> of BASH or shell-script from the existing bashrc. This seems more useful
>> for end-user admins (whether local or network.)
> 
> That seems like a reasonable use-case for per-phase sourcing.
>
Yeah, not something I see myself using a lot but I can see the attraction
for an end-user. (Standard technique is to only run scripts/include config
if they are marked executable.) OFC the utility is limited, as ferringb
outlined many moons ago [1], since the only variables available for an
external as opposed to source'd scriptlet are those exposed in the process
environment.
I guess that argues for just sourcing those? Dunno, some input from javaJake
would be good here, eh? ;)

[1] recommended for anyone who wants more insight into why we have bashrc
the way we do:
http://www.mail-archive.com/gentoo-portage-dev%40lists.gentoo.org/msg00544.html

>> For an overlay, from what I've seen in my limited exposure to the issue,
>> there is more of a need for influencing metadata, eg IUSE. .. That ties
>> in more with the next point; although as you say it could be done by
>> inheritance from an eclass, again that potentially involves editing the
>> ebuild.
> 
> Note that any changes to ebuild metadata generation mean changes in
> metadata cache validation. For example, if you want to modify ebuild
> metadata from profile.bashrc, then you have to make sure to
> invalidate metadata cache any time the profile.bashrc is modified.

Is there some sort of issue with mtime checks that isn't dealt with via
git's technique of a config-flag for coarser-grained checks?
(I'm all for stricter digest based checking as well, but it seems odd we
can't use the mtime to key off, at least by default.)

> If such a change affects all ebuilds in the repo, it can be time
> consuming to regenerate all of the metadata cache. Also, if the 
> cache validation mechanism isn't robust enough, users will
> experience annoying issues with stale cache.
<snip>

Certainly seems to be room to play with cache generation, or better to allow
overlay to distribute a per-repo cache. That could be rsync'ed alongside
the standard vcs-managed files, to avoid the hit of storing it in the vcs.
Might be worth considering a git digest as one of the types?

>> With the existing bashrc capability end-users can do all this ourselves;
>> it'd just be nice to be able to do it in overlays, and for what we
>> already have to be specified since it applies to both pkgcore and
>> portage, and has done for several years.
> 
> We might want to have two separate bashrcs. One that's per-phase,
> and another one that's only sourced once. If it's only sourced once
> then it might allow you to make more radical changes that you
> couldn't otherwise make without breaking uninstall phases in some way.
>
That sounds nice; we have clear exemplars of use-case for each one. We're
really looking at two types: metadata-only and user (use latest version
whenever we're called, and don't save in binpkg) with policy enforced for
tree or overlay.

>>>> You mentioned in #-portage that per-phase execution is no longer used,
>>>> wrt how overlays would only be executing bashrc at start. I take it we
>>>> can still test $EBUILD_PHASE? (Sorry if I've misunderstood what you
>>>> were saying.)
>>> Current portage bashrc support allows $EBUILD_PHASE conditionals,
>>> but I think in the long term we may want to drop that in favor of
>>> function hooks that are sourced only once. The $EBUILD_PHASE
>>> conditional approach just seems somewhat clumsy in comparison
>>> (sourcing the bashrc during every phase might also be considered
>>> inefficient/ugly).
>> 
>> My only concern here is that changes the admin makes, eg in
>> post_pkg_postinst, won't be reflected in uninstalling a package which was
> 
> I assume you mean post_pkg_postrm, since post_pkg_postinst is never
> executed during uninstall.

Er yeah, sorry was tired and tidying up before holiday.

> Well, it is for the replacing package, if 
> there is one, but that should have the latest environment anyway (at
> least, assuming it's not a binary package).
>
Yeah binary packages were the other concern.
 
>> installed before the change. For the DEPEND phase, as in IUSE
>> modification, that's not so much a problem afaict, since a) it's not
>> typically what network admins want to tweak, and b) it's right at the
>> start of the whole process. Any clarity you want to add will be
>> gratefully received ;)
> 
> Again, metadata changes introduce cache validation issues that need
> to be carefully considered.
>
Hmm is this where we get into eclass digests? (again;)

-- 
#friendly-coders -- We're friendly but we're not /that/ friendly ;-)