Re: [gentoo-dev] Stage1 and dependencies

[Jason Stubbs <jstubbs@gentoo.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 26 June 2004 18:52, Robin H. Johnson wrote:
> On Sat, Jun 26, 2004 at 06:27:30PM +0900, Jason Stubbs wrote:
> > On Saturday 26 June 2004 17:06, Robin H. Johnson wrote:
> > > On Sat, Jun 26, 2004 at 03:06:52PM +0900, Jason Stubbs wrote:
> > > > Issue #1: Ignoring dependencies
> > > > The "cleaning" of the vardb to trick portage is IMO a bad thing.
> > > > There is obviously enough in a stage 1 to be able to build up all of
> > > > system, but according to the data portage it is impossible. This then
> > > > requires a hack as indicated above to attempt the merge anyway.
> > > > However, this hack affects users of installed systems as well as
> > > > portage will go ahead and attempt a compilation that is guaranteed to
> > > > fail.
> > >
> > > What's your solution for this?
> >
> > Well, I don't actually understand why the var db needs to be cleaned. I
> > can't see why a combination of --nodeps and --onlydeps on package x and y
> > can't solve whatever the problem is, though...
>
> Try it and see :-), if you don't see why it shouldn't work, it should
> probably work... if not fix it ;-).

(The following was in regards to this "fix it" - but I just realized that you 
were probably talking about bootstrap.sh :)

I've rewritten the dependency checker as part of the API, but it would be a 
_real_ pain to port it back to portage. The main issue with the current dep 
resolver is that if A reqs B and C, and B reqs C, the dep resolution is:
* check A
* add A to the resolved order, add A to the graph
* check B
* add B to the resolved order, add B to the graph
* add dep from A to B
* check C (as dep from B)
* add C to the resolved order, add C to the graph
* add dep from B to C
* check C (as dep from A)
* C exists in graph, skip it.

The dep from A to C is not included. Emerging goes:
* First package added is A.. has deps? yes
* Next package added is B.. has deps? yes
* Next package added is C.. has deps? no.
* Install C and remove its deps.
* First package added is A.. has deps? yes
* Next package added is B.. has deps? no.
* Install B and remove its deps.
* First package added is A.. has deps? no.
* Install A and remove its deps.

Hence, the order is usually right but can go wrong when circular dependencies 
come into play.

> > > I've got no objections to putting the correct information into the
> > > tree, but portage needs to be able to handle circular dependancies much
> > > better first. One other glaring problem in your listing is
> > > sys-devel/make. One other thing that will bite at some points is the
> > > things that some upstream developers put into their configure scripts,
> > > that are decidedly not always present on a system (perl and rpm to name
> > > the worst offenders I've seen). If you can tell me I won't run into any
> > > problems by adding in the deps for virtual/libc virtual/cc
> > > sys-devel/make etc into my packages, I'll go around and correct them
> > > now.
> >
> > I don't think that the change would bring about any more bad dependency
> > ordering than already exists, but I can't be sure. Perhaps, the solution
> > is to fix the dependency resolver then fix the packages and then enable
> > parallel emerges. That sound acceptable?
>
> Yup, that should be fine. Before .51 comes out, could you maybe add a
> bit of code in repoman to detect circular dependancies (and give a
> non-fatal warning at the moment), so we can work on them from an early
> stage?

I think circular dependencies are inevitable. gcc requires glibc and glibc 
requires gcc. I don't think there's anything that can be done about that.

> I see two cases of circular dependancies that need to be worried about:
> - Directly:
>   pkgA DEPEND on pkgB, and pkgB DEPEND on pkgA - The only solution here
>   is to code something to work around the circular dependancy, via
>   bootstrapping one of the packages (similar to getting gcc onto a
>   machine without a compiler). I know at least one case of this existing
>   in the tree at a point, but I think it's been fixed at the moment.

Presently, I'm using the following logic:
1. Install all packages that have no dependencies and remove them from the
   graph.
2. Go back to 1, unless there are no leaf packages.
3. Find the smallest circular dependency set.
4. Find the total number of reverse dependencies for each package in the set.
   (A deps on B deps on C deps on D - D has 3 rather than 1)
5. Starting with the package with the most reverse deps, iterate through all 
   packages on each of the following steps:
   a. Install it and go back to 1 if installed packages can satisfy all of its
      dependencies.
6. Ignore dependencies for the package with the most reverse deps.

I want to add a 5c for cases where:
foo-1 is installed.
foo-3 is the latest and requires >=bar-3
bar-3 requires >=foo-2
foo-2 requires >= bar-2
bar-2 requires >= foo-1

> - Conditionally directly:
>   openldap:DEPEND="sasl? ( cyrus-sasl )"
>   cyrus-sasl:DEPEND="ldap? ( openldap )"
>   This one is much simpler, just build one of the two without the USE
>   flag that causes the circular trap. Then build the other package, and
>   rebuild the first. Hard part would be choosing which one to build
>   without the USE flag. A few bugs like this exact one exist, see bug
>   #33440 to track them.

I will be addding a 5b for this.

> > > virtual/cc is something I think should exist, as some people want to
> > > use dev-lang/icc or dev-lang/ccc instead of sys-devel/gcc.
> > >
> > > Also, where does the adding of dependancies stop? Eg I have something
> > > that uses kernel-headers and glibc, so do I just specify glibc and
> > > ignore kernel-headers as that's needed by glibc?
> >
> > That's a design question that's open to debate. Portage can do it either
> > way, but I would suggest that the package depend on both kernel-headers
> > and glibc for robustness in the tree. Unlikely in this case, but it
> > ensures that a change to glibc's dependencies don't break the packages
> > that depend on it.
>
> Ok, so listing as many dependancies as required (within reason) is
> suitable. One other question comes up in your mention of eclasses to
> make it easier, namely what's the speed hit going to be with so many
> eclasses being added to packages? Won't it be better to change ebuilds
> directly?

Well, caching should make it fine for end users in terms of eclasses. Brian 
Harring is currently looking at speeding up portage's bash usage in a way 
that should more than counter any speed loss, though. More on my mind is the 
memory and cpu requirements for the shear number of dependencies that there 
actually are... The dependency graph is much easier and quicker to process if 
it can be assumed that all packages explicitly state all dependencies, but 
the actual building of the graph would take more time.

Really, I can only offer an opinion on this question. The choice is should 
really be made by people who have been maintaining ebuilds for a long time. 
Which way would be easier to maintain?

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQN1mCFoikN4/5jfsAQLdKQP/fend4MJOZyn57jR/WqgG2wWJ39x/cMpV
hjc35ze6oNR2AnKx9L8kGDmDKPudOkVBX6YlOsaL7DFDDtQjXei/RJCfRpA2NiAi
TrtJlS4VVg9BAWgOvUw2m9L5bA/k6U9WaZgnrfKbp+yVPS3sbn332GaPVetM7INp
CQ9RtvarAtg=
=7Ysg
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list