[gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]

this is not a "implicit vs explicit" thread; if you want that discussion start 
your own

we've said the relationship of DEPEND atoms in ebuilds should be independent 
of the DEPEND atoms found in eclasses as logically ebuilds should not care 
what it takes for eclasses to work and vice versa ... for the most part, this 
is true ...

however, semi-recently, a change was made such that the implicit 
RDEPEND=$DEPEND was dropped from ebuilds if an inherited class set RDEPEND in 
any way ... that means if you have an ebuild at the moment that does:
DEPEND="foo"
and you dont inherit any eclasses, then you also get for free:
RDEPEND="foo"

if you decide to inherit eclasses though, you had better do some research as 
any eclass that does even RDEPEND="" will change that behavior ... or if you 
are an eclass writer and you decided to add RDEPEND to your eclass, you had 
better do a reverse check and make sure that any ebuild that inherits your 
eclass (directly or indirectly) does not utilize implicit RDEPEND behavior as 
you would have just broken it ... awesome ;)

i posted a patch to fix this regression, but since it took so long before 
anyone noticed, zmedico wants to see if anyone is opposed (i dont know why 
they would be, but i cant think of everything)

so to recap, the fix here changes it back to the historically documented 
behavior that the implicit RDEPEND happens in ebuilds regardless of what 
eclasses do
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 5525 bytes --]

On Fri, Nov 03, 2006 at 02:29:45AM -0500, Mike Frysinger wrote:
> this is not a "implicit vs explicit" thread; if you want that discussion start 
> your own

That discussion (dropping it to explicit, as has been discussed often 
enough) should be started off again since fixing it isn't exactly a 
matter of applying a patch and ignoring it.... so...

swegener?  Feel free to chime in, you were one of the main folks 
after killing implicit from what I recall.

Meanwhile...

> we've said the relationship of DEPEND atoms in ebuilds should be independent 
> of the DEPEND atoms found in eclasses as logically ebuilds should not care 
> what it takes for eclasses to work and vice versa ... for the most part, this 
> is true ...
> 
> however, semi-recently, a change was made such that the implicit 
> RDEPEND=$DEPEND was dropped from ebuilds if an inherited class set RDEPEND in 
> any way ... that means if you have an ebuild at the moment that does:
> DEPEND="foo"
> and you dont inherit any eclasses, then you also get for free:
> RDEPEND="foo"
> 
> if you decide to inherit eclasses though, you had better do some research as 
> any eclass that does even RDEPEND="" will change that behavior ... or if you 
> are an eclass writer and you decided to add RDEPEND to your eclass, you had 
> better do a reverse check and make sure that any ebuild that inherits your 
> eclass (directly or indirectly) does not utilize implicit RDEPEND behavior as 
> you would have just broken it ... awesome ;)
> 
> i posted a patch to fix this regression, but since it took so long before 
> anyone noticed, zmedico wants to see if anyone is opposed (i dont know why 
> they would be, but i cant think of everything)
> 
> so to recap, the fix here changes it back to the historically documented 
> behavior that the implicit RDEPEND happens in ebuilds regardless of what 
> eclasses do

Mildly screwed on this one from a cache standpoint; 

1) the change went out unversioned, meaning that while it was 
spreading, folks were getting a mix of it.  Likely their are still 
cache entries distributed with gentoo-x86 that are still old style 
deps.

Nice.

2) Same issue via reverting it; it's an unversioned change, so portage 
will *not* pick up on it and fix it.  Additionally, cause I was being 
anal about making --metadata as fast as possible, portage will *not* 
pick up the change since mtimes won't differ, thus will not write the 
new entry into the localized cache (cache.util.py, note the 
is_eclass_data_valid check controlling write_it boolean).

#1 was stupid, #2 pretty royally gets us up the backside.


So... my suggestion.  Decide what you're going to do long term now, no 
more (bluntly) fucking around with this.


Got a few courses of action from where I'm sitting-

1) EAPI bump for the restored behaviour.  This sucks since it forces 
 the entire tree to be bumped to force falling back to older behaviour 
 (this is why changes like this are supposed to be EAPI bumped in the 
 first place btw).

2) (several steps)
2.a) Revbump all afflicted portage versions with a patch 
 restoring original behaviour, leaving keywords the same (meaning 
 2.1-r3 goes in with stable keywords); pull all portage versions that 
 have the broken behaviour (>=2.1 basically).

2.b) Force mtime touches of _all_ eclasses to force both rsync 
 transfer within the mirror tier (can do this other ways), but more 
 importantly, to force the transfer of all eclass consumers to the 
 localized cache on folks machines.  This at least gets them the old 
 style deps in their cache.  It will *not* fix the rdeps for overlay 
 ebuilds that are afflicted, and does not fix it if the user triggers 
 regeneration locally.

3) ignore it, and laugh like nero till everyone has upgraded to a 
 fixed version of portage and enough churn in the tree has occured to 
 have forced the corrected cache entries locally.


#1 blows since either it requires changing EAPI in every ebuild ( 
easier, base/profile.bashrc, although I never intended profile bashrc 
to influence ebuild metadata), or auditing the rdeps and fixing them, 
bumping when after older behaviour.  Additionally, requires leaving a 
portage version behind at the older EAPI for upgrading.

#2 is fairly brutal initially, but should clear out the corruption 
pretty quickly for majority of users; for overlay users and devs 
however, it relies on folks upgrading to a fixed portage fairly 
quickly so that rdeps are proper.  Doubly so for devs, since their 
portage installation is the first line of dependency verification 
(folks running automated repoman/pcheck dependency scans being second 
line).

One upshot though, this method will push the proper deps into the 
localized cache for stage* users.

Downside is that it triggers a pretty heavy hit on rsync mirrors, 
although the eclass touching can be stretched out over a few days to 
incrementally push out the updated cache.

#3 means that broken rdeps linger for the next few months till folks 
have upgraded, and the mess has gradually cleaned itself out.  Will 
linger for a long while in older ebuilds, although a forced cache 
regen fixes it for folks who either wipe their localized cache, or 
somehow trigger a regen.

So... I'd go with #2.

Pardon the essay, but getting the effects of it fixed isn't trivial, 
and the long term lingering effects aren't either imo.

So... thoughts?

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 125 bytes --]

On Friday 03 November 2006 03:23, Brian Harring wrote:
> so...

so... start a new thread exactly like i told you so :P
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Zac Medico]

[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Mike Frysinger wrote:
> this is not a "implicit vs explicit" thread; if you want that discussion start 
> your own
> 
> we've said the relationship of DEPEND atoms in ebuilds should be independent 
> of the DEPEND atoms found in eclasses as logically ebuilds should not care 
> what it takes for eclasses to work and vice versa ... for the most part, this 
> is true ...
> 
> however, semi-recently, a change was made such that the implicit 
> RDEPEND=$DEPEND was dropped from ebuilds if an inherited class set RDEPEND in 
> any way ... that means if you have an ebuild at the moment that does:
> DEPEND="foo"
> and you dont inherit any eclasses, then you also get for free:
> RDEPEND="foo"

The change occurred sometime between portage-2.0.51 and portage-2.0.52, and the
first stable version to have it was portage-2.0.53 (released in December of
2005).  People didn't really start to notice the difference until after
portage-2.1 had been deployed on the master rsync mirror last June (it had been
running portage-2.0.51.22-r3 up until then).

> if you decide to inherit eclasses though, you had better do some research as 
> any eclass that does even RDEPEND="" will change that behavior ... or if you 
> are an eclass writer and you decided to add RDEPEND to your eclass, you had 
> better do a reverse check and make sure that any ebuild that inherits your 
> eclass (directly or indirectly) does not utilize implicit RDEPEND behavior as 
> you would have just broken it ... awesome ;)
> 
> i posted a patch to fix this regression, but since it took so long before 
> anyone noticed, zmedico wants to see if anyone is opposed (i dont know why 
> they would be, but i cant think of everything)
>
> so to recap, the fix here changes it back to the historically documented 
> behavior that the implicit RDEPEND happens in ebuilds regardless of what 
> eclasses do
> -mike

If we do this then I want to make sure everybody is prepared for the
consequences.  Basically, it restricts implicit RDEPEND to the ebuild level,
making it independent of whatever RDEPEND may or may not be defined in the
inherited eclasses.  That means that some ebuilds will get implicit RDEPEND
that they don't get currently.  Also, some ebuilds will loose some implicit
RDEPEND that they current get from eclasses.

I've attached the patch which reverts the behavior.  If we do this, I think that
we should do it in one big sweep, via a sys-apps/portage revbump and a
simultaneous application of the patch on the master rsync mirror (see Brian
Harring's email for more details).

Zac

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFSwFI/ejvha5XGaMRArjaAKDi8B++F71vt3D2QkG5vyRhZ0yWvgCghBVA
ZrVSROlN0E7X/xdkFbJ6NXo=
=yRsF
-----END PGP SIGNATURE-----

[-- Attachment #2: revert_to_2.0.51_rdepend.patch --]
[-- Type: text/plain, Size: 510 bytes --]

Index: bin/ebuild.sh
===================================================================
--- bin/ebuild.sh	(revision 4913)
+++ bin/ebuild.sh	(working copy)
@@ -1504,8 +1504,8 @@
 #syntax from getting expanded :)
 #check eclass rdepends also.
 set -f
-if [ "${RDEPEND-unset}" == "unset" ] && [ "${E_RDEPEND-unset}" == "unset" ] ; then
-	export RDEPEND="${DEPEND} ${E_DEPEND}"
+if [ "${RDEPEND-unset}" == "unset" ]; then
+	export RDEPEND=${DEPEND}
 	debug-print "RDEPEND: not set... Setting to: ${DEPEND}"
 fi
 

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Peter Volkov (pva)]

[-- Attachment #1: Type: text/plain, Size: 252 bytes --]

On 2006-11-03 at 00:43 -0800, Zac Medico wrote:
> Also, some ebuilds will loose some implicit RDEPEND that they current
> get from eclasses.

Why? I suppose more logical solution is to adjoin DEPEND from ebuild and
RDEPEND from eclass.

Peter.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Volkov (pva) wrote:
> On 2006-11-03 at 00:43 -0800, Zac Medico wrote:
>> Also, some ebuilds will loose some implicit RDEPEND that they current
>> get from eclasses.
> 
> Why? I suppose more logical solution is to adjoin DEPEND from ebuild and
> RDEPEND from eclass.
> 
> Peter.

You've misunderstood the meaning of "implicit RDEPEND" in my statement above (I
don't blame you, implicit RDEPEND can be a confusing topic).  When I say
"implicit RDEPEND", I am talking about DEPEND that has been implicitly converted
to RDEPEND.  Some ebuilds may currently have some implicit RDEPEND that
originated as DEPEND in an eclass.  If we use the patch to revert that behavior,
those specific implicit RDEPEND atoms will go away.  I hope this makes sense. :)

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFSw45/ejvha5XGaMRAntfAJ0X0K9U+CtyB4nhq73v8p5EBd5w8ACg8nc4
jN+Q4rWo+tfvoVL1YUY01E8=
=X/2/
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

On Friday 03 November 2006 04:32, Peter Volkov (pva) wrote:
> On 2006-11-03 at 00:43 -0800, Zac Medico wrote:
> > Also, some ebuilds will loose some implicit RDEPEND that they current
> > get from eclasses.
>
> I suppose more logical solution is to adjoin DEPEND from ebuild and
> RDEPEND from eclass.

that is the behavior that we'd be moving to
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Zac Medico wrote:
> If we do this then I want to make sure everybody is prepared for the
> consequences.  Basically, it restricts implicit RDEPEND to the ebuild level,
> making it independent of whatever RDEPEND may or may not be defined in the
> inherited eclasses.  That means that some ebuilds will get implicit RDEPEND
> that they don't get currently.  Also, some ebuilds will loose some implicit
> RDEPEND that they current get from eclasses.

I've generated some data to show what the differences in RDEPEND will be after the
patch has been applied.  The data includes a list of affected packages [1], and two
separate files (tab separated values) for those packages that will have an implicit
RDEPEND decrease [2] and those that will have an implicit RDEPEND increase [3].  The
format is:

${CATEGORY}/${PF} <tab> RDEPEND before patch <tab> RDEPEND after patch

Fortunately, I don't anticipate that this change will cause significant disruption.
My plan is to release a sys-apps/portage-2.1.1-r2 revbump masked by package.mask and
have all of the arch teams keyword it.  The ebuild will have an ewarn message in
pkg_postinst() that instructs gentoo-x86 cvs users to clean out their existing cache
and regenerate it from scratch.

After all of the teams have keyworded portage-2.1.1-r2, I'll unmask it and apply the
patch on the master rsync mirror.  The only extra load on the rsync mirrors will be
for the cache entries of approximately 2700 packages [1].  The combined total size of
these cache entries will be approximately 1.5 megabytes.  For comparison, cache entry
updates average approximately 425 per day, but in those cases there is additional
overhead for changes in ebuilds, eclasses, and auxiliary files.  For the ~2700 cache
entries that need to be regenerated after application of the patch, only the cache
entries alone will incur load on the rsync mirrors.  The new version of portage will
include a patch that will cause it to automatically recognize the changed RDEPEND
after a sync (unlike earlier versions of portage that relied on mtime alone).

Anyone concerned about the impact of this patch on any packages that they maintain
should check the list of packages [1] to see if they are affected.  Please do that
and let me know if you anticipate any problems.

Thanks,

Zac

[1] http://dev.gentoo.org/~zmedico/bug_153591/data/20061104/package_names.txt.bz2
[2] http://dev.gentoo.org/~zmedico/bug_153591/data/20061104/rdepend_decrease.csv.bz2
[3] http://dev.gentoo.org/~zmedico/bug_153591/data/20061104/rdepend_increase.csv.bz2

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFTV5Z/ejvha5XGaMRApbiAKCWDEU1gjK3NaH0xQmTt7CT8ITHsACgk56y
JdiNGWgX5BpX5cZ+sgxkEfo=
=g2g4
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

On Saturday 04 November 2006 22:45, Zac Medico wrote:
> Fortunately, I don't anticipate that this change will cause significant
> disruption. My plan is to release a sys-apps/portage-2.1.1-r2 revbump
> masked by package.mask and have all of the arch teams keyword it.  The
> ebuild will have an ewarn message in pkg_postinst() that instructs
> gentoo-x86 cvs users to clean out their existing cache and regenerate it
> from scratch.

thanks

> Anyone concerned about the impact of this patch on any packages that they
> maintain should check the list of packages [1] to see if they are affected.

i only see fixes for my packages ;)
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Friday 03 November 2006 08:29, Mike Frysinger wrote:
> so to recap, the fix here changes it back to the historically documented
> behavior that the implicit RDEPEND happens in ebuilds regardless of what
> eclasses do
Fine by me, that would probably solve quite a bit of problems (and although I 
tried, I can't think of a way how restoring this will break stuff.. the worse 
it can do is making pointless some extra DEPEND="${RDEPEND}" or vice-versa in 
ebuilds).

-- 
Diego "Flameeyes" Pettenò - http://farragut.flameeyes.is-a-geek.org/
Gentoo/Alt lead, Gentoo/FreeBSD, Video, Sound, ALSA, PAM, KDE, CJK, Ruby ...

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] [RFC] fixing up portage implicit RDEPEND behavior

[Stephen Bennett]

On Fri, 3 Nov 2006 02:29:45 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> so to recap, the fix here changes it back to the historically
> documented behavior that the implicit RDEPEND happens in ebuilds
> regardless of what eclasses do

Do it please. The current behaviour is retarded however you look at it.
-- 
gentoo-dev@gentoo.org mailing list