[gentoo-dev] Dynamic SLOTs

[gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

Hi all,

Working on SLOT upgrades and the like, I'm having trouble working around 
dynamic SLOTs - not so much how to do it, but how to get accurate information 
regarding the SLOT that an ebuild is in. For SLOT upgrades to work, portage 
needs to know what SLOT a package is in and why it is in that SLOT.

I'm sure that most of you know already, but I'll quickly go over the issue for 
those that don't. Portage caches ebuilds. Users get a prebuilt cache from an 
rsync server. Some ebuilds set variables based on the environment. When users 
run portage, it uses information based on the environment of the rsync 
server.

In terms of SLOTs, we currently have four types of ebuilds:

* Static in the context of the ebuild (SLOT="${PV}" et al)
* Mapped directly on to the kernel (SLOT="${KV}")
* Based on a user set variable (SLOT="${CCHOST}-3.2")
* Decided on the basis of other software (SLOT="${APACHEVER}")

The first is obviously not an issue. I added support for USE-based SLOTs to 
support the third and fourth. I'm not sure what to do about the second. Three 
possibilities:

* Find a solution to the specific ${KV} case.

Ugly. So ugly in fact, I've dropped USE-based SLOTs again.

* Find a general solution to dynamic SLOTs.

The general solution proposed thus far has been a RESTRICT="cache". Doing a 
quick test to compare the performance, I found that sourcing the ebuild is 
roughly 400 times slower than going to cache. Not really acceptable.

* Make SLOTs static and find an alternate solution.

My preferred solution as SLOT upgrades are ridiculously complex when there's a 
number of disparate factors affecting the SLOT of a package - still quite 
difficult when it's only USE affecting it.

Only five ebuilds in total have dynamic SLOTs that aren't ${KV} - gcc, 
libstdc++, mod_php, mod_scgi and mod_pcgi2. These could all be changed to 
install all the different versions in one merge. Using the mod_php ebuild as 
an example, it could build and install files for apache1, reconfigure and 
then build and install files for apache2 in one go if both USE flags are set.

As for ${KV}, a better solution for the short term would be RESTRICT="clean". 
It would at least describe what we actually have at the moment. For the long 
term, I think something like FEATURES="kernelmodule" or some such would be 
much more appropriate to the task.

Thoughts? Flames? Yells for a GLEP?

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Mike Frysinger]

On Saturday 31 July 2004 12:09 am, Jason Stubbs wrote:
> I added support for USE-based SLOTs to
> support the third and fourth.

i know this is off topic of your e-mail, but could you go more into depth on 
this ?
CHOST / CCHOST / cross compiling dynamic SLOTs interest embedded very much
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Saturday 31 July 2004 15:46, Mike Frysinger wrote:
> On Saturday 31 July 2004 12:09 am, Jason Stubbs wrote:
> > I added support for USE-based SLOTs to
> > support the third and fourth.
>
> i know this is off topic of your e-mail, but could you go more into depth
> on this ?
> CHOST / CCHOST / cross compiling dynamic SLOTs interest embedded very much

Well, the idea was that to create local USE flags of the form cc-${ARCH}, have 
the gcc ebuilds use them to configure the cross compiler and also have them 
control the the slot. The only real difference to what there is now would be 
that portage would be able to forecast the SLOT correctly.

A sideline issue to USE-based SLOTs is that it would be way to easy to end up 
with spaces inside the USE-reduced SLOT string. Working support for spaces 
within a single atom (for when SLOT-based deps come in), while not so 
difficult, would slow everything down - as would figuring out whether every 
non-SLOT satisfying ebuild can be configured to be satisfying actually.

<OT>
If you're asking about using portage itself to do the cross-compiling, well 
we're still a ways from that I think. Isn't the main issue there that 
packages link against / instead of ${ROOT}? I'd say portage can be changed to 
support it fairly easily, but every ebuild will need to be changed to 
configure/hack Makefiles/whatever to link against ${ROOT} instead.

To summarize the required portage features (as far as I can see):
* Define ROOT as "/" if it's not defined by the user
* Add support for an "alias" file to profiles

The "alias" file would simply hold aliases for commands. In this case, 
somebody building for an embedded device would create the file and have it 
specify which gcc they wanted and whatever else. ("alias" or something to 
that effect, btw, will probably be needed for darwin/bsd to alias things such 
as make to gmake.)
</OT>

Whichever your question was, I hope I answered it. :)

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Mike Frysinger]

On Saturday 31 July 2004 04:32 am, Jason Stubbs wrote:
> Well, the idea was that to create local USE flags of the form cc-${ARCH},
> have the gcc ebuilds use them to configure the cross compiler and also have
> them control the the slot. The only real difference to what there is now
> would be that portage would be able to forecast the SLOT correctly.

so the user would do USE="cc-arm" to get an arm cross compiler ?

and then in the ebuild we'd have to do something like:
SLOT="cc-arm? ( blah )" ?
how'd would we specify a default slot ?

> If you're asking about using portage itself to do the cross-compiling, well
> we're still a ways from that I think. Isn't the main issue there that
> packages link against / instead of ${ROOT}?

a few things / ideas ...
- you wouldnt merge the cross compiler to $ROOT, you would merge it to /
- it would be impossible to link against things in / since those libraries are 
of a different format
- i believe cross compilers have a diff linking / include / etc... path setup
- if we start supporting $LDFLAGS more, this will be easier :)
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Sunday 01 August 2004 01:29, Mike Frysinger wrote:
> On Saturday 31 July 2004 04:32 am, Jason Stubbs wrote:
> > Well, the idea was that to create local USE flags of the form cc-${ARCH},
> > have the gcc ebuilds use them to configure the cross compiler and also
> > have them control the the slot. The only real difference to what there is
> > now would be that portage would be able to forecast the SLOT correctly.
>
> so the user would do USE="cc-arm" to get an arm cross compiler ?
>
> and then in the ebuild we'd have to do something like:
> SLOT="cc-arm? ( blah )" ?
> how'd would we specify a default slot ?

SLOT="!cc-arm? ( !cc-foo? ( !cc-bar? ( !cc-baz ( defaultslot ) ) )"

Terrible, no? This is gone anyway.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1114 bytes --]

On Sun, 2004-08-01 at 04:01, Jason Stubbs wrote:
> On Sunday 01 August 2004 01:29, Mike Frysinger wrote:
> > On Saturday 31 July 2004 04:32 am, Jason Stubbs wrote:
> > > Well, the idea was that to create local USE flags of the form cc-${ARCH},
> > > have the gcc ebuilds use them to configure the cross compiler and also
> > > have them control the the slot. The only real difference to what there is
> > > now would be that portage would be able to forecast the SLOT correctly.
> >
> > so the user would do USE="cc-arm" to get an arm cross compiler ?
> >
> > and then in the ebuild we'd have to do something like:
> > SLOT="cc-arm? ( blah )" ?
> > how'd would we specify a default slot ?
> 
> SLOT="!cc-arm? ( !cc-foo? ( !cc-bar? ( !cc-baz ( defaultslot ) ) )"
> 

I really do not know why there hasn't been a '&&' and '||' logic
added yet?  Will it really be that difficult to implement?

> Terrible, no? This is gone anyway.
> 

What exactly do you mean by this?


Thanks,

-- 

Martin Schlemmer
Gentoo Linux Developer, Desktop/System Team Developer
Cape Town, South Africa



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Monday 02 August 2004 06:27, Martin Schlemmer wrote:
> On Sun, 2004-08-01 at 04:01, Jason Stubbs wrote:
> > On Sunday 01 August 2004 01:29, Mike Frysinger wrote:
> > > On Saturday 31 July 2004 04:32 am, Jason Stubbs wrote:
> > > > Well, the idea was that to create local USE flags of the form
> > > > cc-${ARCH}, have the gcc ebuilds use them to configure the cross
> > > > compiler and also have them control the the slot. The only real
> > > > difference to what there is now would be that portage would be able
> > > > to forecast the SLOT correctly.
> > >
> > > so the user would do USE="cc-arm" to get an arm cross compiler ?
> > >
> > > and then in the ebuild we'd have to do something like:
> > > SLOT="cc-arm? ( blah )" ?
> > > how'd would we specify a default slot ?
> >
> > SLOT="!cc-arm? ( !cc-foo? ( !cc-bar? ( !cc-baz ( defaultslot ) ) )"
>
> I really do not know why there hasn't been a '&&' and '||' logic
> added yet?  Will it really be that difficult to implement?

It would take some time to implement and then testing time and what not. The 
main reason it hasn't been done is that it doesn't add anything that can't be 
done already.

> > Terrible, no? This is gone anyway.
>
> What exactly do you mean by this?

There is no point adding complexity to SLOT (and making dep calculation 
intricate and very slow) for the benefit of four ebuilds that could be 
implemented without the use of dynamic SLOTs anyway.

As far as I know, allowing ${KV} as a SLOT was a quick hack to prevent 
previously installed kernel modules from being unmerged. It was/is a 
temporary solution until something better is found. That better solution is 
what I'm looking for here.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Mike Frysinger]

On Tuesday 03 August 2004 07:34 pm, Jason Stubbs wrote:
> That better solution is what I'm looking for here.

even though the rest of the e-mail suggested otherwise, the truth is a 
solution does not exist yet to satisfy the needs of cross compiling ebuilds ?
-mie

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Wednesday 04 August 2004 08:57, Mike Frysinger wrote:
> On Tuesday 03 August 2004 07:34 pm, Jason Stubbs wrote:
> > That better solution is what I'm looking for here.
>
> even though the rest of the e-mail suggested otherwise, the truth is a
> solution does not exist yet to satisfy the needs of cross compiling ebuilds

You may have read it in IRC today, but I'll repeat it for verbosity's sake.

I think we can get portage managed cross compilation happening with relatively 
few changes. Note that this doesn't extend to multilib, which I can't see as 
being solvable in the same way. Note this also assumes that cross-compilation 
will always be done into a different root.

1. Allow a profile to define aliases for arbitrary commands. This will allow a 
   profile (even customization of an official profile) to specify to use a 
   cross compiler.
2. Fix portage so that the make.conf, /etc/profile/* and /etc/portage/* found 
   in ${ROOT} are used and the ones found in / are not used at all for any 
   package being compiled for ${ROOT}.
3. Have portage define ${ROOT} as "/" when it is not set by the user.
4. Fix ebuilds so that they link against ${ROOT} rather than assuming "/".

#1 is available as of 2.0.51_pre15 (pre14?). #2 is a little difficult but is 
not beyond reason. #3 is dead simple. #4 is up to everyone else. Is anything 
else required for cross-compilation?

As for gcc slotting the cross-compiler, I don't think it's necessary nor has 
any bareing on cross-compilation. Having gcc respond to architecture USE 
flags (prepended with cc- if necessary) and building the cross-compilers in 
the same merge would work, no? No files get lost track of, all the compilers 
are upgraded on an emerge --update and, best yet, it's possible to do right 
now. Am I missing something here? Is there some reason that the gcc ebuilds 
couldn't work in that way? Same question re mod_php and mod_scgi.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Aron Griffis]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

Jason Stubbs wrote:	[Wed Aug 04 2004, 10:28:04AM EDT]
> 4. Fix ebuilds so that they link against ${ROOT} rather than assuming "/".

This sounds Really Hard.

How about devising a method to chroot to ${ROOT} for src_compile when
[[ ${ROOT} != / ]].  I know this would require some work, but it just
might be less work than fixing the ebuilds to link against ${ROOT},
which sounds extremely tedious and error-prone.

Regards,
Aron

--
Aron Griffis
Gentoo Linux Developer


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Thursday 05 August 2004 05:11, Aron Griffis wrote:
> Jason Stubbs wrote:	[Wed Aug 04 2004, 10:28:04AM EDT]
>
> > 4. Fix ebuilds so that they link against ${ROOT} rather than assuming
> > "/".
>
> This sounds Really Hard.
>
> How about devising a method to chroot to ${ROOT} for src_compile when
> [[ ${ROOT} != / ]].  I know this would require some work, but it just
> might be less work than fixing the ebuilds to link against ${ROOT},
> which sounds extremely tedious and error-prone.

That would require getting the immediate build deps and the deep runtime build 
deps copied into a temporary directory within $ROOT at the start of every 
emerge. It would also require every ebuild to be updated so that all deps, 
including those in system, are listed.

That sounds just as error-prone as fixing the ebuilds to me. However, fixing 
the ebuilds has another benefit. Down the line it will help in being able to 
install stuff into ${HOME}. 

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Lina Pezzella]

Jason Stubbs wrote:

>On Thursday 05 August 2004 05:11, Aron Griffis wrote:
>  
>
>>Jason Stubbs wrote:	[Wed Aug 04 2004, 10:28:04AM EDT]
>>
>>    
>>
>>>4. Fix ebuilds so that they link against ${ROOT} rather than assuming
>>>"/".
>>>      
>>>
>>This sounds Really Hard.
>>
>>How about devising a method to chroot to ${ROOT} for src_compile when
>>[[ ${ROOT} != / ]].  I know this would require some work, but it just
>>might be less work than fixing the ebuilds to link against ${ROOT},
>>which sounds extremely tedious and error-prone.
>>    
>>
>
>That would require getting the immediate build deps and the deep runtime build 
>deps copied into a temporary directory within $ROOT at the start of every 
>emerge. It would also require every ebuild to be updated so that all deps, 
>including those in system, are listed.
>
>That sounds just as error-prone as fixing the ebuilds to me. However, fixing 
>the ebuilds has another benefit. Down the line it will help in being able to 
>install stuff into ${HOME}. 
>
>Regards,
>Jason Stubbs
>
>--
>gentoo-dev@gentoo.org mailing list
>
>  
>
Yes, it does sound just as error-prone as fixing the ebuilds, however
chrooting would fix the macos .a library (needs ranlib run on the
livefs, or something pretending to be the livefs) problem.

--j4rg0n && gongloo

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Aron Griffis]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

Jason Stubbs wrote:	[Wed Aug 04 2004, 07:47:41PM EDT]
> On Thursday 05 August 2004 05:11, Aron Griffis wrote:
> > Jason Stubbs wrote:	[Wed Aug 04 2004, 10:28:04AM EDT]
> >
> > > 4. Fix ebuilds so that they link against ${ROOT} rather than assuming
> > > "/".
> >
> > This sounds Really Hard.
> >
> > How about devising a method to chroot to ${ROOT} for src_compile when
> > [[ ${ROOT} != / ]].  I know this would require some work, but it just
> > might be less work than fixing the ebuilds to link against ${ROOT},
> > which sounds extremely tedious and error-prone.
> 
> That would require getting the immediate build deps and the deep
> runtime build deps copied into a temporary directory within $ROOT at
> the start of every emerge. It would also require every ebuild to be
> updated so that all deps, including those in system, are listed.

I must really be missing something.  I thought that if you're "fixing
ebuilds so they link against ${ROOT}" then you must already have what
you need in ${ROOT}, right?

It seems like either method depends on that.  I'm sure I'm missing
something...

Regards,
Aron

--
Aron Griffis
Gentoo Linux Developer


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Ned Ludd]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

On Wed, 2004-08-04 at 16:11, Aron Griffis wrote:
> Jason Stubbs wrote:	[Wed Aug 04 2004, 10:28:04AM EDT]
> > 4. Fix ebuilds so that they link against ${ROOT} rather than assuming "/".
> 
> This sounds Really Hard.
> 
> How about devising a method to chroot to ${ROOT} for src_compile when
> [[ ${ROOT} != / ]].  I know this would require some work, but it just
> might be less work than fixing the ebuilds to link against ${ROOT},
> which sounds extremely tedious and error-prone.

A chroot is only possible when the build arch matches the target arch
and chrooting into a mipsel/arm/ppc target from a x86 build host wont
work unfortunately.

> 
> Regards,
> Aron
> 
> --
> Aron Griffis
> Gentoo Linux Developer
-- 
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Thursday 05 August 2004 12:00, Aron Griffis wrote:
> Jason Stubbs wrote:	[Wed Aug 04 2004, 07:47:41PM EDT]
>
> > On Thursday 05 August 2004 05:11, Aron Griffis wrote:
> > > Jason Stubbs wrote:	[Wed Aug 04 2004, 10:28:04AM EDT]
> > >
> > > > 4. Fix ebuilds so that they link against ${ROOT} rather than assuming
> > > > "/".
> > >
> > > This sounds Really Hard.
> > >
> > > How about devising a method to chroot to ${ROOT} for src_compile when
> > > [[ ${ROOT} != / ]].  I know this would require some work, but it just
> > > might be less work than fixing the ebuilds to link against ${ROOT},
> > > which sounds extremely tedious and error-prone.
> >
> > That would require getting the immediate build deps and the deep
> > runtime build deps copied into a temporary directory within $ROOT at
> > the start of every emerge. It would also require every ebuild to be
> > updated so that all deps, including those in system, are listed.
>
> I must really be missing something.  I thought that if you're "fixing
> ebuilds so they link against ${ROOT}" then you must already have what
> you need in ${ROOT}, right?
>
> It seems like either method depends on that.  I'm sure I'm missing
> something...

Only the runtime dependencies need to go into $ROOT. Build time dependencies 
don't need to be, which is especially important for embedded systems.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Thursday 05 August 2004 10:45, Lina Pezzella wrote:
> Jason Stubbs wrote:
> >On Thursday 05 August 2004 05:11, Aron Griffis wrote:
> >>Jason Stubbs wrote:	[Wed Aug 04 2004, 10:28:04AM EDT]
> >>
> >>>4. Fix ebuilds so that they link against ${ROOT} rather than assuming
> >>>"/".
> >>
> >>This sounds Really Hard.
> >>
> >>How about devising a method to chroot to ${ROOT} for src_compile when
> >>[[ ${ROOT} != / ]].  I know this would require some work, but it just
> >>might be less work than fixing the ebuilds to link against ${ROOT},
> >>which sounds extremely tedious and error-prone.
> >
> >That would require getting the immediate build deps and the deep runtime
> > build deps copied into a temporary directory within $ROOT at the start of
> > every emerge. It would also require every ebuild to be updated so that
> > all deps, including those in system, are listed.
> >
> >That sounds just as error-prone as fixing the ebuilds to me. However,
> > fixing the ebuilds has another benefit. Down the line it will help in
> > being able to install stuff into ${HOME}.
>
> Yes, it does sound just as error-prone as fixing the ebuilds, however
> chrooting would fix the macos .a library (needs ranlib run on the
> livefs, or something pretending to be the livefs) problem.

That's a relatively minor problem which can be dealt with in many ways.

I didn't seem to mention it elsewhere although I thought I did. chroot'ing 
can't be used as a general solution as the binaries in $ROOT won't necessary 
run on the architecture that is building them. Which of course means that 
where back to fixing ebuilds to build against $ROOT. Note this doesn't mean 
that every ebuild would need to be fixed at once.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Mike Frysinger]

On Wednesday 04 August 2004 10:28 am, Jason Stubbs wrote:
> As for gcc slotting the cross-compiler, I don't think it's necessary nor
> has any bareing on cross-compilation. Having gcc respond to architecture
> USE flags (prepended with cc- if necessary) and building the
> cross-compilers in the same merge would work, no? No files get lost track
> of, all the compilers are upgraded on an emerge --update and, best yet,
> it's possible to do right now. Am I missing something here? Is there some
> reason that the gcc ebuilds couldn't work in that way? Same question re
> mod_php and mod_scgi.

doesnt work that way ... portage goes 'uh oh, packages with the same $SLOT in 
the same $ROOT, lets clean out the old one !'

cross compiling toolchains need to be in ROOT=/ while the cross compiled 
packages need to go in ROOT=/some/where/else

in other words, you try to merge a cross compiling version and blam, you lost 
your native one
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Friday 06 August 2004 12:10, Mike Frysinger wrote:
> On Wednesday 04 August 2004 10:28 am, Jason Stubbs wrote:
> > As for gcc slotting the cross-compiler, I don't think it's necessary nor
> > has any bareing on cross-compilation. Having gcc respond to architecture
> > USE flags (prepended with cc- if necessary) and building the
> > cross-compilers in the same merge would work, no? No files get lost track
> > of, all the compilers are upgraded on an emerge --update and, best yet,
> > it's possible to do right now. Am I missing something here? Is there some
> > reason that the gcc ebuilds couldn't work in that way? Same question re
> > mod_php and mod_scgi.
>
> doesnt work that way ... portage goes 'uh oh, packages with the same $SLOT
> in the same $ROOT, lets clean out the old one !'

Well, duh.. That's what I am suggesting should be fixed. All this 
cross-compiling talk is a side-track. However, Travis Tilley (Lv) clued me in 
anyway. For cross compilers, mod_php, et al the user may want the different 
instances merged with different use flags.

> cross compiling toolchains need to be in ROOT=/ while the cross compiled
> packages need to go in ROOT=/some/where/else

I've also said this myself elsewhere in this thread.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Saturday 31 July 2004 08:46, Mike Frysinger wrote:
> On Saturday 31 July 2004 12:09 am, Jason Stubbs wrote:
> > I added support for USE-based SLOTs to
> > support the third and fourth.
>
> i know this is off topic of your e-mail, but could you go more into depth
> on this ?
> CHOST / CCHOST / cross compiling dynamic SLOTs interest embedded very much
> -mike

I'm working on my own portage_ng version. It includes a c++ parser that 
replaces the bash cruft, is a lot faster, and parses most existing ebuilds. 
(compilation etc. still happens in bash) (It does not allow at all dynamic 
behaviour on the toplevel, but it will include a much more flexible 
dependency structure. On top of that the parser itself does not (yet at 
least) perform the variable substitution, so the variables are still 
available for reading. Once I get to that point I will also include some kind 
of dynamic slot/ multislot feature based on some conditions. It will 
certainly allow multiple installation of the same version of the same package 
(think crosscompiling, php for apache1 and apache2) given that they have 
different MULTISLOT values

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Friday 06 August 2004 20:56, Paul de Vrieze wrote:
> On Saturday 31 July 2004 08:46, Mike Frysinger wrote:
> > On Saturday 31 July 2004 12:09 am, Jason Stubbs wrote:
> > > I added support for USE-based SLOTs to
> > > support the third and fourth.
> >
> > i know this is off topic of your e-mail, but could you go more into depth
> > on this ?
> > CHOST / CCHOST / cross compiling dynamic SLOTs interest embedded very
> > much -mike
>
> I'm working on my own portage_ng version. It includes a c++ parser that
> replaces the bash cruft, is a lot faster, and parses most existing ebuilds.
> (compilation etc. still happens in bash) (It does not allow at all dynamic
> behaviour on the toplevel, but it will include a much more flexible
> dependency structure. On top of that the parser itself does not (yet at
> least) perform the variable substitution, so the variables are still
> available for reading. Once I get to that point I will also include some
> kind of dynamic slot/ multislot feature based on some conditions. It will
> certainly allow multiple installation of the same version of the same
> package (think crosscompiling, php for apache1 and apache2) given that they
> have different MULTISLOT values

Are you going to share information or just offer another sales pitch? For 
example, what algorithm(s) are you going use to handle the following?

pkg-a-1.0:SLOT="foo? ( 2 ) !foo ( 1 )"
pkg-a-1.1:SLOT="foo? ( 1 ) !foo ( 2 )"
pkg-b-1.0:DEPEND="foo? ( pkg-a:2 )"
pkg-c-1.0:DEPEND="pkg-b[foo]"
USE="-foo" emerge pkg-c

How about if pkg-a is installed in both slot 1 and slot 2 and the user issues 
"emerge -u pkg-a" ?

How about if we add pkg-d-1.0:DEPEND="pkg-a[-foo]" and the user issues
"emerge pkg-c pkg-d" ?

All those have a definite solution, but doing it in code that isn't as slow as 
hell is another matter.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 2732 bytes --]

On Friday 06 August 2004 14:41, Jason Stubbs wrote:
> On Friday 06 August 2004 20:56, Paul de Vrieze wrote:
> > On Saturday 31 July 2004 08:46, Mike Frysinger wrote:
> > > On Saturday 31 July 2004 12:09 am, Jason Stubbs wrote:
> > > > I added support for USE-based SLOTs to
> > > > support the third and fourth.
> > >
> > > i know this is off topic of your e-mail, but could you go more into
> > > depth on this ?
> > > CHOST / CCHOST / cross compiling dynamic SLOTs interest embedded very
> > > much -mike
> >
> > I'm working on my own portage_ng version. It includes a c++ parser that
> > replaces the bash cruft, is a lot faster, and parses most existing
> > ebuilds. (compilation etc. still happens in bash) (It does not allow at
> > all dynamic behaviour on the toplevel, but it will include a much more
> > flexible dependency structure. On top of that the parser itself does not
> > (yet at least) perform the variable substitution, so the variables are
> > still available for reading. Once I get to that point I will also include
> > some kind of dynamic slot/ multislot feature based on some conditions. It
> > will certainly allow multiple installation of the same version of the
> > same package (think crosscompiling, php for apache1 and apache2) given
> > that they have different MULTISLOT values
>
> Are you going to share information or just offer another sales pitch? For
> example, what algorithm(s) are you going use to handle the following?
>
> pkg-a-1.0:SLOT="foo? ( 2 ) !foo ( 1 )"
> pkg-a-1.1:SLOT="foo? ( 1 ) !foo ( 2 )"
> pkg-b-1.0:DEPEND="foo? ( pkg-a:2 )"
> pkg-c-1.0:DEPEND="pkg-b[foo]"
> USE="-foo" emerge pkg-c

For one thing, I'm going to introduce a prefix based notation (similar to 
lisp) for dependencies, with the current syntax as a shortcut. I'm not yet 
very clear on how the MULTISLOT will work exactly, but variable based 
MULTISLOT should not be any problem

>
> How about if pkg-a is installed in both slot 1 and slot 2 and the user
> issues "emerge -u pkg-a" ?

I have not thought about this yet at all. Probably update both, or give a big 
fat warning/error message

>
> How about if we add pkg-d-1.0:DEPEND="pkg-a[-foo]" and the user issues
> "emerge pkg-c pkg-d" ?

Then we check whether pkg-a[-foo] can be satisfied in the current installed 
tree. If it can it will be (possibly by remerging it), else there is a 
conflict and an error message will be produced

>
> All those have a definite solution, but doing it in code that isn't as slow
> as hell is another matter.

Well, I'll try to work on it with fast code

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Mike Frysinger]

On Friday 06 August 2004 11:18 am, Paul de Vrieze wrote:
> On Friday 06 August 2004 14:41, Jason Stubbs wrote:
> > All those have a definite solution, but doing it in code that isn't as
> > slow as hell is another matter.
>
> Well, I'll try to work on it with fast code

i think what he was trying to get at is that he hasnt heard of your portage_ng 
work before ... the portage devs get kind of angry when they hear of people 
working on 'ng' projects with the idea of replacing the current portage ... 
not because the project will replace portage, but because it was worked on 
completely removed from the current development process

this 'ng' projects will be a *lot* better if everyone knows what everyone else 
is doing and people work together instead of different groups working in 
isolation ... why waste development time developing the same damn thing
-mike

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Saturday 07 August 2004 00:18, Paul de Vrieze wrote:
> On Friday 06 August 2004 14:41, Jason Stubbs wrote:
> > On Friday 06 August 2004 20:56, Paul de Vrieze wrote:
> > > On Saturday 31 July 2004 08:46, Mike Frysinger wrote:
> > > > On Saturday 31 July 2004 12:09 am, Jason Stubbs wrote:
> > > > > I added support for USE-based SLOTs to
> > > > > support the third and fourth.
> > > >
> > > > i know this is off topic of your e-mail, but could you go more into
> > > > depth on this ?
> > > > CHOST / CCHOST / cross compiling dynamic SLOTs interest embedded very
> > > > much -mike
> > >
> > > I'm working on my own portage_ng version. It includes a c++ parser that
> > > replaces the bash cruft, is a lot faster, and parses most existing
> > > ebuilds. (compilation etc. still happens in bash) (It does not allow at
> > > all dynamic behaviour on the toplevel, but it will include a much more
> > > flexible dependency structure. On top of that the parser itself does
> > > not (yet at least) perform the variable substitution, so the variables
> > > are still available for reading. Once I get to that point I will also
> > > include some kind of dynamic slot/ multislot feature based on some
> > > conditions. It will certainly allow multiple installation of the same
> > > version of the same package (think crosscompiling, php for apache1 and
> > > apache2) given that they have different MULTISLOT values
> >
> > Are you going to share information or just offer another sales pitch? For
> > example, what algorithm(s) are you going use to handle the following?
> >
> > pkg-a-1.0:SLOT="foo? ( 2 ) !foo ( 1 )"
> > pkg-a-1.1:SLOT="foo? ( 1 ) !foo ( 2 )"
> > pkg-b-1.0:DEPEND="foo? ( pkg-a:2 )"
> > pkg-c-1.0:DEPEND="pkg-b[foo]"
> > USE="-foo" emerge pkg-c
>
> For one thing, I'm going to introduce a prefix based notation (similar to
> lisp) for dependencies, with the current syntax as a shortcut. I'm not yet
> very clear on how the MULTISLOT will work exactly, but variable based
> MULTISLOT should not be any problem

The notation used shouldn't have any baring on what is actually being done.

> > How about if pkg-a is installed in both slot 1 and slot 2 and the user
> > issues "emerge -u pkg-a" ?
>
> I have not thought about this yet at all. Probably update both, or give a
> big fat warning/error message
>
> > How about if we add pkg-d-1.0:DEPEND="pkg-a[-foo]" and the user issues
> > "emerge pkg-c pkg-d" ?
>
> Then we check whether pkg-a[-foo] can be satisfied in the current installed
> tree. If it can it will be (possibly by remerging it), else there is a
> conflict and an error message will be produced

But pkg-c may change what's in the installed package tree...

> > All those have a definite solution, but doing it in code that isn't as
> > slow as hell is another matter.
>
> Well, I'll try to work on it with fast code

The code isn't what needs to be fast. It's the algorithm used that are 
important. I'm not so worried about ARCH-based or KV-based SLOTs any longer 
as I've figured out how they can be handled cleanly without slowing down 
everything else. On the other hand, introducing USE-based SLOTs makes 
everything about a package variable. When atoms can specify any of that 
variation, it becomes horrifically complex.

I, too, am working on a portage-ng. It's called portage. Presently, I'm near 
finishing a new dependency resolver that supports:
* Ranged deps
* USE-based atoms
* SLOT-based atoms
* SLOT upgrades
* Full circular dependency support

Even with all that, a full dependency tree is built with all circular 
dependencies resolved in about 6 seconds, where currently emerge takes about 
2 seconds. I can't see a way to get USE-based SLOTs in without at least 
doubling that figure (which is incidentally the time that 2.0.50 took). I'm 
open to suggestions on how to get around it.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

On Saturday 07 August 2004 01:57, Jason Stubbs wrote:
>
> I, too, am working on a portage-ng. It's called portage. Presently, I'm
> near finishing a new dependency resolver that supports:
> * Ranged deps
> * USE-based atoms
> * SLOT-based atoms
> * SLOT upgrades
> * Full circular dependency support
>
> Even with all that, a full dependency tree is built with all circular
> dependencies resolved in about 6 seconds, where currently emerge takes
> about 2 seconds. I can't see a way to get USE-based SLOTs in without at
> least doubling that figure (which is incidentally the time that 2.0.50
> took). I'm open to suggestions on how to get around it.

I don't have a dependency resolver at all yet (I've been working on the 
general architecture and parsing of ebuilds. I'll be happy to help out where 
I can.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Chris Bainbridge]

On Saturday 07 August 2004 00:57, Jason Stubbs wrote:
> Even with all that, a full dependency tree is built with all circular
> dependencies resolved in about 6 seconds, where currently emerge takes
> about 2 seconds. I can't see a way to get USE-based SLOTs in without at
> least doubling that figure (which is incidentally the time that 2.0.50
> took). I'm open to suggestions on how to get around it.

Maybe I'm not understanding this fully, but couldn't you just compute the 
dependency tree once and then cache it? And allow quick updates to the cached 
info by only editing dependency links for a single ebuild when it changes. 
You'd need to cache the dependencies for different sets of use flags as well.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Saturday 07 August 2004 20:07, Chris Bainbridge wrote:
> On Saturday 07 August 2004 00:57, Jason Stubbs wrote:
> > Even with all that, a full dependency tree is built with all circular
> > dependencies resolved in about 6 seconds, where currently emerge takes
> > about 2 seconds. I can't see a way to get USE-based SLOTs in without at
> > least doubling that figure (which is incidentally the time that 2.0.50
> > took). I'm open to suggestions on how to get around it.
>
> Maybe I'm not understanding this fully, but couldn't you just compute the
> dependency tree once and then cache it? And allow quick updates to the
> cached info by only editing dependency links for a single ebuild when it
> changes. You'd need to cache the dependencies for different sets of use
> flags as well.

The dependencies of a single package with a single set of conditions don't 
take long to calculate. The issue is that, real soon, packages are going to 
be able to specify some of those conditions for other packages. Figuring out 
the best combination of packages/conditions and whether it is indeed possible 
(ie. no conflicts) is what takes the time.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Matthieu Sozeau]

[-- Attachment #1: Type: text/plain, Size: 6958 bytes --]

On Wed, Aug 04, 2004 at 08:34:24AM +0900, Jason Stubbs wrote:
> On Monday 02 August 2004 06:27, Martin Schlemmer wrote:
> > On Sun, 2004-08-01 at 04:01, Jason Stubbs wrote:
> > > On Sunday 01 August 2004 01:29, Mike Frysinger wrote:
> > > > On Saturday 31 July 2004 04:32 am, Jason Stubbs wrote:
> > > > > Well, the idea was that to create local USE flags of the form
> > > > > cc-${ARCH}, have the gcc ebuilds use them to configure the cross
> > > > > compiler and also have them control the the slot. The only real
> > > > > difference to what there is now would be that portage would be able
> > > > > to forecast the SLOT correctly.
> > > >
> > > > so the user would do USE="cc-arm" to get an arm cross compiler ?
> > > >
> > > > and then in the ebuild we'd have to do something like:
> > > > SLOT="cc-arm? ( blah )" ?
> > > > how'd would we specify a default slot ?
> > >
> > > SLOT="!cc-arm? ( !cc-foo? ( !cc-bar? ( !cc-baz ( defaultslot ) ) )"
> >
> > I really do not know why there hasn't been a '&&' and '||' logic
> > added yet?  Will it really be that difficult to implement?
> 
> It would take some time to implement and then testing time and what not. The 
> main reason it hasn't been done is that it doesn't add anything that can't be 
> done already.
> 
> > > Terrible, no? This is gone anyway.
> >
> > What exactly do you mean by this?
> 
> There is no point adding complexity to SLOT (and making dep calculation 
> intricate and very slow) for the benefit of four ebuilds that could be 
> implemented without the use of dynamic SLOTs anyway.
> 
> As far as I know, allowing ${KV} as a SLOT was a quick hack to prevent 
> previously installed kernel modules from being unmerged. It was/is a 
> temporary solution until something better is found. That better solution is 
> what I'm looking for here.
> 
> Regards,
> Jason Stubbs
> 

I'm happy those issues are being discussed as i didn't have much incentive to
post a feature request like what follows until now. First, some context:
I maintain the ML packages and particularly the ocaml ones. As some of
you may know, ocaml is not binary compatible from version to version and
it is required to recompile all dependent libraries on compiler updates,
(in the right order of course). This situation is very similar with Perl
or Python actually. Managing more than one version of such packages is
currently impossible (using portage only) AFAIK.

What I would like is a proper framework to deal with this
compilers/interpreters, and hopefully one general enough to deal with
cross-compilation too. I have hacked a solution on my own to deal with
the problem of incompatible compilers, using dynamic SLOTS, an ocaml
eclass and an ocaml-config script (similar to gcc-config). All ocaml
packages use this eclass which sets the slot to the compiler version and
sets some env vars (pathes) to allow flawless cohabitation of the
different compiler versions. It works just like a charm. I can update
all my packages for new versions of ocaml, do backward-compatibility
tests (i'm an ocaml dev) and if i wanted to, i could manage the CVS
version of ocaml and all ocaml packages using only portage and my
ocaml-config script.

The problem is obviously that i'm using portage for something it was not
intended to do (cleanly, at least). I have some (clear?) ideas about how it
could be done though, and i'm willing to give help. Here's my wish:

	      *** Portage support for compilers ***

  I never heard anybody mention it, but I think it is required to get a
  clean implementation. It would introduce some changes:

  - Factorized, standard config files and utilities for all compilers/interpreters,
  gcc, python, perl, ocaml, you name it. Configurations would be simply a set
  of env vars, much like what gcc-config uses today.
  You could set different configurations for different users etc...
  A set of compiler policies (in the tree) would also be needed to 
  mention the fact that no ocaml compilers are compatible, but that 
  different gcc's are. A new flag could be added to emerge telling him
  to rebuild automatically all packages that depended on a previous
  compiler that is incompatible with a newer one.

  - An ebuild metadata variable mentioning the compiler configurations
  it depends on: a CDEPEND in fact, mentioning which compiler versions
  it is compatible with. Unset would mean any, but when set, it could 
  well be cached and maybe some pruning in the dep tree could be done too.

  - When invoking an ebuild for some package, portage would simply set 
  the compiler variables (for the compilers mentionned in CDEPEND).
  The process of dependency checking would introduce one more step on
  each depended-on pkg check : When a package dependent on the same language
  as one mentionned is found, try to find an installed one matching the
  dep requirements and the compiler policies, if none matches, use the
  same policies as the current one for dep-checks etc... Informative
  messages could be easily crafted in case of errors. 

Now for the critics, i will point out that my scheme doesn't extend
ebuild syntax, which is good as most compiler compatibility problems can
be handled at once and for all packages. It would also factorize some
code that's between the various *-config scripts and as an added benefit
it would make every compiler update easy, getting rid of rebuild
scripts. The main advantage is a finer, more robust dependency
checking. 

Here's a test case:
I've got ocaml-3.07, lablgl and lablgtk merged, i want the latest
mldonkey which requires lablgtk-1.2.7 which in turn works only with
ocaml-3.08. Portage will see that mldonkey has no particular preference
regarding compiler versions, it will check lablgtk which will tell that
it needs ocaml-3.08 and no other, so, as you have the opengl USE flag
set, portage will check ocaml compiler policies to see if lablgl in 3.07
would be compatible with lablgtk compiled with 3.08, the policy says no,
hence portage checks that lablgl can be compiled with 3.08 and finally
it sets its roadmap to: install 3.08, lablgl for 3.08, lablgtk for 3.08 and
finally merge mldonkey for 3.08.

Now for the cross-compiling, it would greatly help to have such support
built into portage. If i want to build ppc packages on my athlon,
i can tell portage to use a (more general than just compilers)
configuration that's declared incompatible with my current one 
(as it should be) that sets the root to some other place, have a
particular make.conf, cross-compiler config etc... I'm just prospecting here.

That was just my 2 euros, i'm more than open to criticism about my
proposal which i ruminate for some time now (see bug #39246), and 
i'll get it as far as I can, writing a GLEP and coding included (i'm in
holidays after all :).

Thanks for your patience !

-- mattam

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

Hi,

I couldn't really follow all that you said, but bug 39246 made a lot more 
sense. Let me see if I can summarise your requirements...

For portage's role,
* The compiler is slotted
* The libraries are slotted against the compiler that was used to compile them
* When the compiler is upgraded, all libraries are recompiled with the new 
compiler

Then there is a compiler-config proggy that'll set what compiler is currently 
in use. Does that about sum it up?

If the compiler wasn't slotted, the compiler-config program wouldn't be 
necessary and the other two requirements would be easy to satisfy. If it 
is... Anyway, I'll let you confirm that these are the actual requirements 
before bouncing around ideas.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Matthieu Sozeau]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

On Tuesday 10 August 2004 14:21, Jason Stubbs wrote:
> Hi,
>
> I couldn't really follow all that you said, but bug 39246 made a lot more
> sense. Let me see if I can summarise your requirements...
>
> For portage's role,
> * The compiler is slotted
> * The libraries are slotted against the compiler that was used to compile
> them 
> * When the compiler is upgraded, all libraries are recompiled with the 
> new compiler
>
> Then there is a compiler-config proggy that'll set what compiler is
> currently in use. Does that about sum it up?

That's what i implemented yes, my proposal just adds more flexibility and 
describe a correct (i hope) design (avoiding the use of dynamic SLOTs for the 
libraries, as the 'dynamicity' would be in the hands of portage only with 
compiler policies).

> If the compiler wasn't slotted, the compiler-config program wouldn't be
> necessary and the other two requirements would be easy to satisfy. If it
> is... 

We obviously have to use static SLOTs to support different compiler 
versions...

> Anyway, I'll let you confirm that these are the actual requirements 
> before bouncing around ideas.

I confirm those are the main ideas, go on !
-- 
BOFH Excuse #447: According to Microsoft, it's by design 

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

On Tuesday 10 August 2004 21:53, Matthieu Sozeau wrote:
> On Tuesday 10 August 2004 14:21, Jason Stubbs wrote:
> > Hi,
> >
> > I couldn't really follow all that you said, but bug 39246 made a lot more
> > sense. Let me see if I can summarise your requirements...
> >
> > For portage's role,
> > * The compiler is slotted
> > * The libraries are slotted against the compiler that was used to compile
> > them
> > * When the compiler is upgraded, all libraries are recompiled with the
> > new compiler
> >
> > Then there is a compiler-config proggy that'll set what compiler is
> > currently in use. Does that about sum it up?
>
> That's what i implemented yes,

Which means that it'll be mishandled once portage makes use of SLOTs as the 
SLOTting is wrong until it is installed.

> my proposal just adds more flexibility and describe a correct (i hope) 
> design (avoiding the use of dynamic SLOTs for the libraries, as the 
> 'dynamicity' would be in the hands of portage only with compiler policies).  

I don't understand how dynamic slots are avoided. Whether the information is 
in a variable named SLOT or is deduced from a combination of a package's deps 
and the compiler policy, I don't see a difference. Have I misunderstood 
something?

> > If the compiler wasn't slotted, the compiler-config program wouldn't be
> > necessary and the other two requirements would be easy to satisfy. If it
> > is...
>
> We obviously have to use static SLOTs to support different compiler
> versions...

Well.. if the compiler wasn't slotted (or portage only had to care about the 
most recent version installed [which is wrong imo]) everything could be 
handled by simply adding a new variable to state that any reverse deps 
compiled against a version of this package outside of said atom must be 
recompiled.

For example, foobar-2.3:xDEPEND="=foobar-2*". When upgrading from foobar-1.7, 
anything that was built with foobar-1.7 would be recompiled after foobar-2.3 
was installed.

> > Anyway, I'll let you confirm that these are the actual requirements
> > before bouncing around ideas.
>
> I confirm those are the main ideas, go on !

And the major issue that covers this entire thread...

emerge '~dev-lang/ocaml-3.06'
emerge lablgtk
emerge -u dev-lang/ocaml
emerge lablgl

The issue that's preventing this from happening is whether portage should also 
emerge lablgl against ocaml-3.06. I believe it should and I think it is a 
requirement for most of the people that want dynamically slotted packages.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Dynamic SLOTs

[Jason Stubbs]

Replying to my own so this doesn't get too deep.

No Grand Unified Theory of Everything yet, but I have an idea that may at 
least solve the multilib and crosscompilation support issues other than gcc 
itself.. very rudimentary, mind you.

Thoughts in random order:

Specifying a toolchain implies package atoms, binary names and any parameters.
Each profile specifies what toolchain should be used for its own arch.
Each profile specifies what toolchain would be required from a host arch.
A profile can specify another profile to use for compiling for a different 
arch destined for the same root.
Packages for a different arch cannot be merged into a root unless specified by 
the profile.
New atom type: cat/pkg&arch  <-- substitute whatever token feels good.
/var/db/pkg/${ARCH}/  <-- packages are always slotted by arch

No time to put these thoughts in to paragraphs but I hope you get the idea..

Still to come: Dynamic slots.. How did we get so off topic?

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Cross Compilation and Dynamic Slots

[Jason Stubbs]

On Tuesday 10 August 2004 23:55, Jason Stubbs wrote:
> No Grand Unified Theory of Everything yet, but ...
Almost there!

I'll summarise the list of requirements that I see and then go through how 
they can solve various problems with regards to ${SUBJECT}.

PROFILES
packages.cc
   Defines a set of atoms that must be on the host system to be able to
   compile for this profile.
bashrc - available as of portage-2.0.51_pre17
   ARCH, USERLAND, LIBC & KERNEL are defined as those required for the
   package that is about to be compiled when sourced. Bashrc should define
   aliases for the toolchain to define how to compile for the host
   specified with a system of this profile.

ATOMS
cat/pkg&keyword
   Defines that a package needs to be compiled as the specified keyword.
   Portage expands the keyword to ARCH, USERLAND, LIBC & KERNEL and passes
   those to the host profile's bashrc. If an atom does not specify a
   keyword, it is taken to be that of the target root's profile.
cat/pkg{atom [atom]}
   The braced atom specifies extra bounds for a matching cat/pkg's DSLOT.

EBUILDS
DSLOT="atom [atom ...]"
   Portage scans installed and to be installed packages against the
   specified atom(s) and builds the DSLOTted package once for each
   individual package match, passing the matched package name to the
   ebuild in the DSLOT variable.
pkg_select()
   This function configures the system to use this package by default. If
   it is successful or if there is no need to do so, it should return 0.
   It should also export any information needed to revert the system back
   to its original state. If it is not successful, it should do any
   cleanup required immediately and return 1.
pkg_revert()
   This function reverts the system to back to the state it was in before
   running pkg_select() based on information exported by pkg_select().

INSTALLED DB
/var/db/pkg/keyword/cat/pkg-ver[--pkg-ver]
   The keyword is taken or inferred from the cat/pkg&keyword specified for
   the installed package. The [-pkg-ver] component represents the DSLOTted
   version of the package.

Not too much I hope... Anyway, on to how I envision it working.


MOD_PHP

DEPEND would simply contain "net-www/apache" without respect to USE flags as 
well as any other DEPENDs it might have. DSLOT would also contain 
"net-www/apache". Apache doesn't need to do any special setup and so would 
simply return a 0 for both pkg_select() and pkg_revert(). In the case that 
both apache-1.3.31-r2 and apache-2.0.50 are installed, portage would elect to 
install mod_php twice, passing in the different versions of apache each time. 
Each installation would end up in a directories under /var/db/pkg labelled 
mod_php-4.3.8--apache-1.3.31-r2 and mod_php-4.3.8--apache-2.0.50.

As for differing use flags, users could specify them in package.use using the 
new atom syntax. For example, "dev-php/mod_php{=apache-2*} -mysql". Users 
could also restrict the combination by way of package.mask. If they wanted 
both apache-1 and apache-2 installed but only wanted mod_php for apache-2, 
for example, a user could specify "dev-php/mod_php{=apache-1*}" in 
package.mask and so on and so forth.

NVIDIA_KERNEL

The only required change would be DSLOT="virtual/linux-sources" as the ebuild 
already detects what kernel version is at /usr/src/linux and compiles against 
it. The kernels, on the other hand, would define pkg_select() to change the 
linux symlink to its own sources and export what was previously linked. Then 
in pkg_revert() it would restore the symlink to its original state.

OTHERS

The only other that I know of, ocaml, could use the pkg_select() function to 
run ocaml-config to set the compiler to the version required. Any package 
that needs to be slotted based on the version of another package should be 
able to be done so in pretty much the same way.

GCC

To build a cross compiler, the atom would be "sys-devel/gcc&keyword". Gcc 
would then receive the keyword in the form of ARCH, USERLAND, LIBC & KERNEL 
but would still be able to tell the host architecture by way of the keyword 
'USE flag'. Portage would then slot the compiler into /var/db/pkg/keyword/.
Gcc would also define pkg_select() and pkg_revert() to do the same as ocaml.

CROSS COMPILATION

A "chroot" would be created and the profile set. Any other files (such as 
make.conf, et al) would also be created before any emerging starts. The 
profile then dictates what tools are required from the host by way of 
packages.cc. In a sparc profile, for example, would be "sys-devel/gcc&sparc". 
Portage would then emerge gcc into the host system as specified above.

When packages are to be merged into the "chroot", portage will look at the 
packages.cc and run pkg_select() on each tool required before doing the 
compilation and merging and then run pkg_revert() after each.

MULTILIB

I'll use amd64 as the example. In the profile, bashrc would set aliases 
defining how to run gcc to create 32bit binaries when ARCH="x86". The 
profile's ACCEPT_KEYWORDS would be "amd64 x86". When keyword matching is 
done, they are search in order of ACCEPT_KEYWORDS and the first match is used 
when sourcing bashrc and when recording the installation into the installed 
package database.

Any packages that the to be installed package depends on will then be 
installed using the accepted keyword.


I'm sure there are a few issues remaining here. I think that most of them will 
be much easier to deal with, though. Feedback is greatly welcome.

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Cross Compilation and Dynamic Slots

[Ilya A. Volynets-Evenbakh]

Here are few of my comments/questions.

Let me try to understand multilib on MIPS example (which is probably 
most complex ABI screw-up case).
For starters, standrard mips toolchain, when compiled with multilibs 
enabled, is capable of producing
binaries for any ABI. At the same time, this imposes specific file 
system layout
(o32 libraries go into /lib /usr/lib /usr/lib/gcc-lib/lib, n32 - /lib32 
/usr/lib32 /usr/lib/gcc-lib/lib32, n64 -
/lib64 /usr/lib64 /usr/lib/gcc-lib/lib64).

On one hand this is good - different ABIs can coexist, and even mix in 
some cases. On the other hand
thus is bad - many programs do not expect to have libraried not in /lib 
/usr/lib places. As it is now,
I have special patch, that makes glibc and gcc use /lib /usr/lib layout 
when using N32 and not using
multilibs.

So, for one thing, we should probably add another variable - ABI - since 
ARCH will be mips in all three cases.

I'm not sure I understand what you mean by

 When keyword matching is 
done, they are search in order of ACCEPT_KEYWORDS and the first match is used 
when sourcing bashrc and when recording the installation into the installed 
package database.

How do we handle the case, where we have two different apps, depending 
on same lib, and
one is requested to be compiled as o32 and another as n32? And in 
general, how do we allow
user to specify which ABI he wants to use for this specific compile (Yes 
- it is possible some
will want to compile different apps with different ABIs - i.e. n32 is 
generally faster then n64
because it has lmaller pointer size, but for that same reason its 
addressing is limited. Thus
in order to utilize all 8G of ram in my origin, I will need certain apps 
to be n64).

I'll have more question later ;-)

    Ilya

>MULTILIB
>
>I'll use amd64 as the example. In the profile, bashrc would set aliases 
>defining how to run gcc to create 32bit binaries when ARCH="x86". The 
>profile's ACCEPT_KEYWORDS would be "amd64 x86". When keyword matching is 
>done, they are search in order of ACCEPT_KEYWORDS and the first match is used 
>when sourcing bashrc and when recording the installation into the installed 
>package database.
>
>Any packages that the to be installed package depends on will then be 
>installed using the accepted keyword.
>
>
>I'm sure there are a few issues remaining here. I think that most of them will 
>be much easier to deal with, though. Feedback is greatly welcome.
>
>Regards,
>Jason Stubbs
>
>--
>gentoo-dev@gentoo.org mailing list
>
>  
>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Cross Compilation and Dynamic Slots (ARCH_LIBDIR)

[Travis Tilley]

could we also get a LIBDIR into this spec? something where we could edit 
the profile to say:

[ "$ARCH" == "amd64" ] && ARCH_LIBDIR="lib64"
[ "$ARCH" == "x86" ] && ARCH_LIBDIR="lib"

or...

[ "$ARCH" == "mips-o32" ] && ARCH_LIBDIR="lib"
[ "$ARCH" == "mips-n32" ] && ARCH_LIBDIR="lib32"
[ "$ARCH" == "mips-n64" ] && ARCH_LIBDIR="lib64"

(only done in a way that makes more sense for a profile)

and then also have the following addition to econf:

[ "${ARCH_LIBDIR}" != "" ] && EXTRA_ECONF="--libdir=/usr/${ARCH_LIBDIR} 
${EXTRA_ECONF}"

that way, user and ebuild specified configuration will overwrite the 
default --libdir if specified, and the default behavior of not 
specifying --libdir isnt changed what this var isnt set. i would really 
appreciate if this edit found it's way into portage before 2.0.51, as it 
wont effect anything now and will help me to work on getting ebuilds 
lib64 aware locally (even without the multilib dependency changes).

also, with an ARCH_LIBDIR setting, we can fix any ebuild that doesnt use 
/usr/lib (like kde) by specifying --libdir=/usr/kde/3.3/${ARCH_LIBDIR} 
if ARCH_LIBDIR is set (and only if it's set)... all without changing the 
default behavior when it isnt.

my previous econf edit idea didnt allow for as much flexibility, and 
with this setup we get support for any arch that needs a specific libdir 
for free... like with mips needing lib, lib32, and lib64 depending on 
ABI. so the work i do getting amd64 multilib ready will also help mips 
get there too. it's a win/win/win/win situation :)


Travis Tilley <lv@gentoo.org>

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Cross Compilation and Dynamic Slots

[Jason Stubbs]

On Thursday 12 August 2004 01:17, Ilya A. Volynets-Evenbakh wrote:
> Let me try to understand multilib on MIPS example (which is probably
> most complex ABI screw-up case).
> For starters, standrard mips toolchain, when compiled with multilibs
> enabled, is capable of producing binaries for any ABI. At the same time, 
> this imposes specific file system layout
> (o32 libraries go into /lib /usr/lib /usr/lib/gcc-lib/lib,
> n32 - /lib32 /usr/lib32 /usr/lib/gcc-lib/lib32,
> n64 - /lib64 /usr/lib64 /usr/lib/gcc-lib/lib64).      
>
> On one hand this is good - different ABIs can coexist, and even mix in
> some cases.

Sounds pretty much the same as AMD64 so far...

> On the other hand thus is bad - many programs do not expect to have 
> libraried not in /lib /usr/lib places. 

This issue affects any user of multilib and perhaps any user wishing to cross 
compile. This point hasn't really been addressed yet.

> As it is now, I have special patch, that makes glibc and gcc 
> use /lib /usr/lib layout when using N32 and not using multilibs. 

So N32 is the "default", then?

> So, for one thing, we should probably add another variable - ABI - since
> ARCH will be mips in all three cases.

ARCH comes from the expansion of the keyword, as per GLEP 22. For portage to 
be able to pass the ABI, under this proposal, keywords would need to be 
extended to include it - something like mips-n64-gnu-linux-glibc. 
Architectures not supporting more than one ABI could use be called "std". 

However, to get this included you will need to convince the general developer 
community that it is a Good Thing. It'd help if you can find another 
supported architecture that could make use of it.

> How do we handle the case, where we have two different apps, depending
> on same lib, and one is requested to be compiled as o32 and another as n32?

I covered this in the proposal but I'll cover it again below.

> And in general, how do we allow user to specify which ABI he wants to use 
> for this specific compile (Yes - it is possible some will want to compile 
> different apps with different ABIs - i.e. n32 is generally faster then n64 
> because it has lmaller pointer size, but for that same reason its addressing 
> is limited. Thus in order to utilize all 8G of ram in my origin, I will need 
> certain apps to be n64).

I covered this too, but to sum it up "emerge mem-hungry-app&mips-n64"


I'll edit what I posted last time in order to clear up issues others have 
brought up (thanks solar & lv!)

On Wednesday 11 August 2004 23:21, Jason Stubbs wrote:
=== GENERAL ===

> EBUILDS
> pkg_select()
>    This function configures the system to use this package by default. If
>    it is successful or if there is no need to do so, it should return 0.
>    It should also export any information needed to revert the system back
>    to its original state. If it is not successful, it should do any
>    cleanup required immediately and return 1.
> pkg_revert()
>    This function reverts the system to back to the state it was in before
>    running pkg_select() based on information exported by pkg_select().

> PROFILE GLOBALS
keywords.desc
   Each "old" keyword gets two new fields. One field defines what the keyword
   is expanded to by default. The other field defines what class of keyword
   will match the "old" keyword. Using MIPS as an example, the three fields
   would look something like:
   mips mips-n32-gnu-linux-glibc mips-*-gnu-linux-glibc


=== DYNAMIC SLOTTING ===

> ATOMS
> cat/pkg{atom [atom]}
>    The braced atom specifies extra bounds for a matching cat/pkg's DSLOT. 
   This is used to specify a particular combination of a package and one or
   more of the packages it is dynamically slotted.

> EBUILDS
> DSLOT="atom [atom ...]"
>    Portage scans installed and to be installed packages against the
>    specified atom(s) and builds the DSLOTted package once for each
>    individual package match, passing the matched package name to the
>    ebuild in the DSLOT variable.

> INSTALLED DB
> /var/db/pkg/.../pkg-ver[--cat--pkg--slot]
   The --cat--pkg--slot represents the package that pkg-ver was DLOTted
   against.


=== CROSS COMPILATION & MULTILIB ===

> PROFILES
make.defaults
   KEYWORDS will be replace ARCH to allow multiple keywords. When portage
   checks the masking status of a set of packages, it checks all packages
   against each keyword from first to last and stops when it finds a match.
   Any ACCEPT_KEYWORDS settings will also be reordered to match the ordering
   of KEYWORDS so that the profile defined precedence is adhered to.
> packages.cc
>    Defines a set of atoms that must be on the host system to be able to
>    compile for this profile.
> bashrc - available as of portage-2.0.51_pre17
   ARCH, ABI, USERLAND, LIBC & KERNEL are defined as those required for the
   package that is about to be compiled when sourced. These will be defined
   as per the keyword matched or the keyword specified by the requesting
   atom. Bashrc should define aliases, set up CFLAGS or anything required to
   compile for the passed (expanded) keyword using the packages defined in
   packages.cc.

> ATOMS
> cat/pkg&keyword
>    Defines that a package needs to be compiled as the specified keyword.
   A package will only match if it has keyword (or a derivative) listed in its
   KEYWORDS. This overrides the ordering set by the profile's KEYWORDS. If the
   specified keyword (or a derivative) does not exist in the profile's
   KEYWORDS, the atom will not match any packages.

> GCC
>
> To build a cross compiler, the atom would be "sys-devel/gcc&keyword". Gcc

This is wrong and is the result of lack of sleep. "sys-devel/gcc&keyword" 
would result in the package not being installed if 'keyword' did not exist in 
the profile's KEYWORDS. The only workaround I can think of for this is a 
per-profile package.keywords and have the profile specify all keywords for 
sys-devel/gcc. Gcc would then still be able to contain the following logic.

> would then receive the keyword in the form of ARCH, USERLAND, LIBC & KERNEL
> but would still be able to tell the host architecture by way of the keyword
> 'USE flag'. Portage would then slot the compiler into /var/db/pkg/keyword/.
> Gcc would also define pkg_select() and pkg_revert() to do the same as
> ocaml.

If anybody can think of a better way to do this, I'd love to hear it. As it 
stands, gcc is the only sore thumb to this whole proposal.

> INSTALLED DB
> /var/db/pkg/keyword/cat/...
   The keyword field is the KEYWORD that was matched when selecting the
   package to be installed. The keyword is always in its full form as per
   GLEP 22 (with any approved ABI extension).

DEPENDENCY RESOLUTION
   When portage resolves a package, all dependencies and sub dependencies are
   only matched against the keyword that the top package was matched against.
   For example, if the profile's KEYWORDS is "foo bar" and a user does emerge
   somepkg&bar (or the package does not have KEYWORDS="foo") somepkg and any
   of its dependencies will be installed as "bar".


=== EXAMPLE ===

I'll only cover cross compilation and multilib as I think the others speak for 
themselves. For the example, I'll theoretically go through the cross 
compilation of a package for both mips-n32 and mips-n64 into the one altroot.


-- Start --
* The user sets up ${alt} root with the mips profile to be compiled for.
  - The profile has KEYWORDS="mips-n32 mips-o32 mips-n64"
* The user executes "ROOT=${alt} emerge some/pkg some/pkg&mips-n64"
  - The some/pkg ebuild has KEYWORDS="mips"
* Portage checks ${alt} profile's package.cc and ensures that all packages 
   are installed.
  - packages.cc would contain sys-devel/gcc&mips and whatever else
  - If they weren't installed, they would be in the same fashion as the
    following.

-- some/pkg --
* Portage scans the profile's KEYWORDS and finds the first to be mips
* Portage scans the global keywords.desc to find mips expands to
  mips-n32-gnu-linux-glibc.
* Portage scans the ebuild's KEYWORDS and when it finds mips,
* Portage scans the global keywords.desc to find the mips keyword will
  match mips-*-gnu-linux-glibc.
* Portage chooses mips-n32-gnu-linux-glibc for this package.
* Portage runs pkg_select() for each of packages.cc's installed packages on
  the host to prepare for cross compilation.
* Portage expands the keyword into variables and passes them to ${alt}'s 
  bashrc.
* Bashrc sets up for n32 compilation.
* The ebuild does it's regular thing.
* Portage records the package to 
  ${alt}/var/db/pkg/mips-n32-gnu-linux-glibc/some/pkg
* Portage runs pkg_revert() on each of the earlier successful pkg_select()s

-- some/pkg&mips-n64 --
* Portage scans the global keywords.desc to find mips-n64 expands to
  mips-n64-gnu-linux-glibc.
* Portage scans and expands each of the profile's KEYWORDS to find that
  mips matches.
* Portage scans the ebuild's KEYWORDS and when it finds mips,
* Portage scans the global keywords.desc to find the mips keyword will
  match mips-*-gnu-linux-glibc.
* Portage chooses mips-n64-gnu-linux-glibc for this package.
* Portage runs pkg_select() for each of packages.cc's installed packages on
  the host to prepare for cross compilation.
* Portage expands the keyword into variables and passes them to ${alt}'s 
  bashrc.
* Bashrc sets up for n64 compilation.
* The ebuild does it's regular thing.
* Portage records the package to 
  ${alt}/var/db/pkg/mips-n64-gnu-linux-glibc/some/pkg
* Portage runs pkg_revert() on each of the earlier successful pkg_select()s


Yes, I realize that mips-n32 and mips-n64 were almost the same - that's the 
beauty of it! ;)

Regards,
Jason Stubbs

--
gentoo-dev@gentoo.org mailing list