[gentoo-user] dynamic deps, wtf are they exactly

[gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

Does anyone here know what dynamic deps are exactly, how they work and
can describe them?

There's lots of talk over on -dev about getting rid of them and the
closest description is from Ciaran, something like:

"ancient shit that never worked right, involving phases of the moon"

10 days ago I switched to --dynamic-deps=m in make.conf, had to emerge
-e world to get around all the resulting kde hard blocks, and all was
good till today, kio-extras was blocking plasma-workspace in impossible
ways, noticeably that the block wasn't in the ebuild at all :-)

kio-extras and dolphon were to be upgraded, plasma-workspace was not.
I've fixed it (remerge plasma-workspace to clean up cruft).

I would have thought emerge -e would have cleaned all that stuff out,
maybe it's related to removing the kde overlay.

So, my question: wtf are dynamic deps? How do I find the records they
must leave behind in portage?
-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Mike Gilbert]

On Sun, Sep 27, 2015 at 11:07 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> So, my question: wtf are dynamic deps? How do I find the records they
> must leave behind in portage?

For the latter question, you can rebuild affected packages like so:

emerge --with-bdeps=y @changed-deps.

You can also add --changed-deps to your emerge command line for world updates.

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

On 27/09/2015 17:12, Mike Gilbert wrote:
> On Sun, Sep 27, 2015 at 11:07 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> So, my question: wtf are dynamic deps? How do I find the records they
>> must leave behind in portage?
> 
> For the latter question, you can rebuild affected packages like so:
> 
> emerge --with-bdeps=y @changed-deps.
> 
> You can also add --changed-deps to your emerge command line for world updates.
> 


Thanks. Running that gives surprising and unexpected results. I'm now
curious what changed-deps really does, and the man page is terse on this.

I would have thought portage already does that automatically, but I see
a difference. If a package's deps change, but everything is still
satisfied, portage will do nothing. Does @changed-deps rebuild
everything that changed anyway, regardless is portage thinks it still OK?

Also, these two similar commands return different results (I have
bdeps=y in DEFAULT_OPTS btw):

emerge -uND --changed-deps=y world         (51 packages)
emerge @changed-deps                       (11 packages)

Do you know why those commands give different results?
The smaller list is a strict subset of the longer one.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Mike Gilbert]

On Sun, Sep 27, 2015 at 11:28 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 27/09/2015 17:12, Mike Gilbert wrote:
>> On Sun, Sep 27, 2015 at 11:07 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>>> So, my question: wtf are dynamic deps? How do I find the records they
>>> must leave behind in portage?
>>
>> For the latter question, you can rebuild affected packages like so:
>>
>> emerge --with-bdeps=y @changed-deps.
>>
>> You can also add --changed-deps to your emerge command line for world updates.
>>
>
>
> Thanks. Running that gives surprising and unexpected results. I'm now
> curious what changed-deps really does, and the man page is terse on this.
>
> I would have thought portage already does that automatically, but I see
> a difference. If a package's deps change, but everything is still
> satisfied, portage will do nothing. Does @changed-deps rebuild
> everything that changed anyway, regardless is portage thinks it still OK?

Basically, yes. If [R]DEPEND in /var/db/pkg is different from the
values in the ebuilds in the tree, @changed-deps will select it.

> Also, these two similar commands return different results (I have
> bdeps=y in DEFAULT_OPTS btw):
>
> emerge -uND --changed-deps=y world         (51 packages)
> emerge @changed-deps                       (11 packages)
>
> Do you know why those commands give different results?
> The smaller list is a strict subset of the longer one.

That difference is surprising; you would probably need to ask the
portage developers to get a real answer.

[gentoo-user] Re: dynamic deps, wtf are they exactly

[James]

Mike Gilbert <floppym <at> gentoo.org> writes:


> Basically, yes. If [R]DEPEND in /var/db/pkg is different from the
> values in the ebuilds in the tree,  <at> changed-deps will select it.

> > Also, these two similar commands return different results (I have
> > bdeps=y in DEFAULT_OPTS btw):
> >
> > emerge -uND --changed-deps=y world         (51 packages)
> > emerge  <at> changed-deps                       (11 packages)

> > Do you know why those commands give different results?
> > The smaller list is a strict subset of the longer one.

> That difference is surprising; you would probably need to ask the
> portage developers to get a real answer.


I do not "own' the code underneath these options, nor would I waste my
time on what is undoubtedly broken logic. Portage and associated tools do
not have a complete description of the problem with  the current VDB. A DAG
is the best way to solve the problem of these portage anomalies. 

Or just rebuild @world  and occasionally @system, and then cobble together
a script that hopefully finds all the other installed codes and packages....
Still, cruft (some of the old files that should have been removed at some
point) remains.

I certainly and not saying this is an easy problem to solve; all distros
suffer tremendously here. That's why many just 'reinstall' periodically.
But without a robust implementation of a DAG or something similar, deep
problems will remain unresolved and hopefully, not noticed. Many large data
centers just 'reload' entire rows of racked systems too.

These are big problems on clusters/clouds. The current approach is to build
many 'customized frameworks' for different classes of problems, on top of 
poorly implemented distros with a collection of packages. That approach
works well, if the packages are relegated to one complex language and
scripts. Once a collection of codes, a package if you like, requires several
complex programming languages to compile and execute, thus crossing
framework boundaries, then the problems exponentiate, and it becomes
an admin/security mess. I do believe that the current gentoo approach is
going to be robustly application as a pristine solution for this gargantuan
cluster problem; but a DAG sort of fundamental tooling is going to be
minimally sufficient if  stability is to be achieved.  Clusters are already
making their way to gentoo so these problem in only going to grow.


Oddly enough, I feel like and uneducated  admin in these matters. But, I do
now have several major corps that want me to be a temporary consultant
on cluster problems.   We shall see. We shall see what's what.


wwr,
James

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Michael Orlitzky]

On 09/27/2015 11:07 AM, Alan McKinnon wrote:
> Does anyone here know what dynamic deps are exactly, how they work and
> can describe them?
> 
> There's lots of talk over on -dev about getting rid of them and the
> closest description is from Ciaran, something like:
> 
> "ancient shit that never worked right, involving phases of the moon"
> 

"Dynamic dependencies" is a portage option. The abridged version is:
whenever you go to install/update something, some of its dependencies
may already exist on your machine. For example, if I go to install gimp
right now, gtk is already there.

With dynamic deps, portage will scan (that is, execute) all of the
ebuilds for installed packages that could affect the dependency graph.
If any of those ebuilds have changed, portage will use the new
information rather than the info present when you originally installed
the package. So if the gtk ebuild (that I installed a month ago) has
been altered, portage will re-read it on the fly, ignoring what was in
there a month ago.

too short; didn't explain?

When you install a package, its dependencies are recorded in the VDB:

  $ cat /var/db/pkg/app-editors/nano-2.3.6/RDEPEND
  >=sys-libs/ncurses-5.9-r1[unicode] sys-apps/file

That's nice, because now if you want to install or update something
else, portage doesn't have to re-execute every ebuild/eclass that could
possibly affect the new thing -- it only has to check the VDB.

But, if I'm the maintainer of nano and I change that ncurses dependency
(let's say I drop it), then I have to make a revision bump on nano.
Otherwise, the wrong dependency would stay recorded on everyone's
system, and portage would happily use the recorded deps.

I guess at some point there were a bunch of devs who were messing with
dependencies and not bothering to make revision bumps. This can cause
users pain, so portage added a new option to ignore the cache and rescan
every single relevant ebuild/eclass for sneaky dependency changes. This
ensures that portage has the correct dependencies in its head while it's
doing resolution, but is much slower.

And then that mode was made default, which is where we are today. It
doesn't sound that bad so far -- just a tradeoff between speed and the
risk of using an incorrect cached dependency.

Buuuuuuutttttt dynamic-deps mode doesn't always kick in. It interacts
weirdly with subslots and other things. If portage can't find the same
ebuild to scan, it will find one that "looks like it." If that doesn't
work, it's supposed to fall back to the cache. Nobody has a flow chart
of exactly how this works. It's all in the code and there's no
specification.

And, there are other package managers besides portage. When developers
rely on dynamic deps and skip revision bumps, they're screwing users of
paludis and pkgcore (and you, now that you have dynamic deps disabled).
None of the magic is mentioned in the PMS, so you really can't rely on
it and revbumps are needed regardless.

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

On 27/09/2015 19:26, Michael Orlitzky wrote:
> On 09/27/2015 11:07 AM, Alan McKinnon wrote:
>> Does anyone here know what dynamic deps are exactly, how they work and
>> can describe them?
>>
>> There's lots of talk over on -dev about getting rid of them and the
>> closest description is from Ciaran, something like:
>>
>> "ancient shit that never worked right, involving phases of the moon"
>>
> 
> "Dynamic dependencies" is a portage option. The abridged version is:
> whenever you go to install/update something, some of its dependencies
> may already exist on your machine. For example, if I go to install gimp
> right now, gtk is already there.
> 
> With dynamic deps, portage will scan (that is, execute) all of the
> ebuilds for installed packages that could affect the dependency graph.
> If any of those ebuilds have changed, portage will use the new
> information rather than the info present when you originally installed
> the package. So if the gtk ebuild (that I installed a month ago) has
> been altered, portage will re-read it on the fly, ignoring what was in
> there a month ago.
> 
> too short; didn't explain?

Nope, I see a problem already :-)

If portage can't reliably tell if the ebuild changed, the cache may or
may not be valid and portage wouldn't know.


> When you install a package, its dependencies are recorded in the VDB:
> 
>   $ cat /var/db/pkg/app-editors/nano-2.3.6/RDEPEND
>   >=sys-libs/ncurses-5.9-r1[unicode] sys-apps/file
> 
> That's nice, because now if you want to install or update something
> else, portage doesn't have to re-execute every ebuild/eclass that could
> possibly affect the new thing -- it only has to check the VDB.
> 
> But, if I'm the maintainer of nano and I change that ncurses dependency
> (let's say I drop it), then I have to make a revision bump on nano.
> Otherwise, the wrong dependency would stay recorded on everyone's
> system, and portage would happily use the recorded deps.
> 
> I guess at some point there were a bunch of devs who were messing with
> dependencies and not bothering to make revision bumps. This can cause
> users pain, so portage added a new option to ignore the cache and rescan
> every single relevant ebuild/eclass for sneaky dependency changes. This
> ensures that portage has the correct dependencies in its head while it's
> doing resolution, but is much slower.
> 
> And then that mode was made default, which is where we are today. It
> doesn't sound that bad so far -- just a tradeoff between speed and the
> risk of using an incorrect cached dependency.
> 
> Buuuuuuutttttt dynamic-deps mode doesn't always kick in. It interacts
> weirdly with subslots and other things. If portage can't find the same
> ebuild to scan, it will find one that "looks like it." If that doesn't
> work, it's supposed to fall back to the cache. Nobody has a flow chart
> of exactly how this works. It's all in the code and there's no
> specification.
> 
> And, there are other package managers besides portage. When developers
> rely on dynamic deps and skip revision bumps, they're screwing users of
> paludis and pkgcore (and you, now that you have dynamic deps disabled).
> None of the magic is mentioned in the PMS, so you really can't rely on
> it and revbumps are needed regardless.


Now it all makes sense, as a bonus I now see why why so many senior devs
are so pedantic about revbumps (they have reason).

Now that I know what the symptom will be, are bug reports useful?


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Michael Orlitzky]

On 09/27/2015 03:34 PM, Alan McKinnon wrote:
> 
> Now it all makes sense, as a bonus I now see why why so many senior devs
> are so pedantic about revbumps (they have reason).
> 
> Now that I know what the symptom will be, are bug reports useful?
> 

I don't think most developers are aware of the problem, so if you point
out that the lack of a revbump causes pain, many will be glad to know
and adjust their behavior.

Or, you might just get yelled at. It's one of /those/ issues.

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

On 28/09/2015 00:21, Michael Orlitzky wrote:
> On 09/27/2015 03:34 PM, Alan McKinnon wrote:
>>
>> Now it all makes sense, as a bonus I now see why why so many senior devs
>> are so pedantic about revbumps (they have reason).
>>
>> Now that I know what the symptom will be, are bug reports useful?
>>
> 
> I don't think most developers are aware of the problem, so if you point
> out that the lack of a revbump causes pain, many will be glad to know
> and adjust their behavior.
> 
> Or, you might just get yelled at. It's one of /those/ issues.



Oh, we're well used to deal with touchy issues around here. And we do
our fair share of yelling too :-)

Thanks for the feedback. Appreciated.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Rich Freeman]

On Sun, Sep 27, 2015 at 6:21 PM, Michael Orlitzky <mjo@gentoo.org> wrote:
> On 09/27/2015 03:34 PM, Alan McKinnon wrote:
>>
>> Now it all makes sense, as a bonus I now see why why so many senior devs
>> are so pedantic about revbumps (they have reason).
>>
>> Now that I know what the symptom will be, are bug reports useful?
>>
>
> I don't think most developers are aware of the problem, so if you point
> out that the lack of a revbump causes pain, many will be glad to know
> and adjust their behavior.
>
> Or, you might just get yelled at. It's one of /those/ issues.

While working out the right way to handle dep changes in eclasses
might take a little longer, I suspect that we'll have a policy for
revbumping on dep changes in ebuilds and perhaps virtuals fairly soon.
There really wasn't much loud objection when the proposal came up
again last week - not nearly as much as the last time this came up.
With the increasing issues with subslots I think devs are appreciating
that the current state isn't really great.

-- 
Rich

[gentoo-user] Re: dynamic deps, wtf are they exactly

[Martin Vaeth]

Rich Freeman <rich0@gentoo.org> wrote:
> There really wasn't much loud objection when the proposal came up
> again last week

This does not mean that everybody agreed.
However, all arguments had been exchanged before,
so repeating them would just have been pointless:
Eventually a decision had to be made, and I am confident
that it was made by the portage team in the full awareness
of the positive and negative consequences of that decision,
because all portage developers had participated in the
previous discussion.

Re: [gentoo-user] Re: dynamic deps, wtf are they exactly

[Rich Freeman]

On Sun, Sep 27, 2015 at 8:33 PM, Martin Vaeth <martin@mvath.de> wrote:
> Rich Freeman <rich0@gentoo.org> wrote:
>> There really wasn't much loud objection when the proposal came up
>> again last week
>
> This does not mean that everybody agreed.
> However, all arguments had been exchanged before,
> so repeating them would just have been pointless:
> Eventually a decision had to be made, and I am confident
> that it was made by the portage team in the full awareness
> of the positive and negative consequences of that decision,
> because all portage developers had participated in the
> previous discussion.
>
>

Sure, but the portage team can really only dictate the upstream
defaults of portage, not tree policy.  I don't disagree with them, but
it wouldn't hurt to have the Council or QA weigh in just so that we're
not endlessly bickering about whether it is official or not.

-- 
Rich

[gentoo-user] Re: dynamic deps, wtf are they exactly

[Martin Vaeth]

Rich Freeman <rich0@gentoo.org> wrote:
>
> Sure, but the portage team can really only dictate the upstream
> defaults of portage, not tree policy.

As I understand, they intend to remove non-dynamic deps
(if they agreed to not implement it properly for sub-slots,
this makes sense).

So we are not speaking about defaults but a fixed behaviour of
portage. Paludis had this fixed behaviour since ever.
Thus, esssentially, there is no other choice than to adopt the
necessary tree policy to the only existing implementations -
not council decision is needed for it unless there are package
managers which do it differently.

Note that there can really be no exceptions. Even apparently
trivial changes in DEPS like adding an alternative implementation
*necessarily* require a revbump (because otherwise users who
installed the previous ebuild could possibly never get rid
of the original implementation).
We will see a flood of "unnecessary" revbumps, but people
knew this since the previous discussions.

Re: [gentoo-user] Re: dynamic deps, wtf are they exactly

[Rich Freeman]

On Mon, Sep 28, 2015 at 3:57 AM, Martin Vaeth <martin@mvath.de> wrote:
> Rich Freeman <rich0@gentoo.org> wrote:
>>
>> Sure, but the portage team can really only dictate the upstream
>> defaults of portage, not tree policy.
>
> As I understand, they intend to remove non-dynamic deps
> (if they agreed to not implement it properly for sub-slots,
> this makes sense).
>
> So we are not speaking about defaults but a fixed behaviour of
> portage. Paludis had this fixed behaviour since ever.
> Thus, esssentially, there is no other choice than to adopt the
> necessary tree policy to the only existing implementations -
> not council decision is needed for it unless there are package
> managers which do it differently.

Like I said, we can either have a formal decision, or listen to
everybody fight WW3 over it for three years.  What is the value in
doing the latter?

The fact that none of the package managers work with a tree practice
won't stop developers from doing it anyway.  Plus, any of them can
just fork portage and put that in the tree - there is no policy that
states that there can be only one implementation of portage.  Heck,
they could just follow the same upstream and patch it in the ebuild.

People seem to think that just going and imposing a change on
everybody else without their input somehow makes things more
efficient, or less political.  The reality is that it just results in
more politics, since many will not accept the validity of their
actions.  It also isn't how we do things around here.  If you want to
change tree policy, propose it on a list, let everybody have their
say, and then if necessary let the council impose a decision.  People
might not like the decision, but most devs will at least respect the
legitimacy of it.  If they don't respect the decision they can be
booted, and the council will back that up.

This isn't about who is or isn't right, or whether the portage team
knows what they're doing.  This is about having a process (GLEP 39)
and following it.  The portage team can do whatever they want with
portage (after all, nobody has to use portage), but if they want to
change what everybody else is doing with their ebuilds they have to
follow the process.

-- 
Rich

[gentoo-user] Re: dynamic deps, wtf are they exactly

[James]

Michael Orlitzky <mjo <at> gentoo.org> writes:


> With dynamic deps, portage will scan (that is, execute) all of the
> ebuilds for installed packages that could affect the dependency graph.
> If any of those ebuilds have changed, portage will use the new
> information rather than the info present when you originally installed
> the package. So if the gtk ebuild (that I installed a month ago) has
> been altered, portage will re-read it on the fly, ignoring what was in
> there a month ago.

Seems reasonable from a  first glance point of view.


> That's nice, because now if you want to install or update something
> else, portage doesn't have to re-execute every ebuild/eclass that could
> possibly affect the new thing -- it only has to check the VDB.

I think that this is what GLEP-64 is all about. Enhancing the functional
utilityh of the VDB with a DAG (Directed Acyclic Graph)?


> But, if I'm the maintainer of nano and I change that ncurses dependency
> (let's say I drop it), then I have to make a revision bump on nano.
> Otherwise, the wrong dependency would stay recorded on everyone's
> system, and portage would happily use the recorded deps.

Basically from my point of view, something like TUP [1] is needed so
that at dependency check time you only list files that need
attention (linking, loading, compiling etc) thus speeding up the
update processes for the Package Manager (portage). I do not study
Palaudis or others.



> I guess at some point there were a bunch of devs who were messing with
> dependencies and not bothering to make revision bumps. This can cause
> users pain, so portage added a new option to ignore the cache and rescan
> every single relevant ebuild/eclass for sneaky dependency changes. This
> ensures that portage has the correct dependencies in its head while it's
> doing resolution, but is much slower.

There is no proper mechanism to accurately track all of these issue,
currently, or did I miss this point?


> And then that mode was made default, which is where we are today. It
> doesn't sound that bad so far -- just a tradeoff between speed and the
> risk of using an incorrect cached dependency.
> 
> Buuuuuuutttttt dynamic-deps mode doesn't always kick in. It interacts
> weirdly with subslots and other things. If portage can't find the same
> ebuild to scan, it will find one that "looks like it." If that doesn't
> work, it's supposed to fall back to the cache. Nobody has a flow chart
> of exactly how this works. It's all in the code and there's no
> specification.

It' a difficult subject so half baked measures abound, imho. A full DAG of
the things in the VDB that tracks and retains additional information, is
currently one possible solution. But the DAG only collects the data needed
to develop more robust tools, or at least that is my understanding. I have
read that Ninja, could implement a DAG, but the devs have not made that
commitment yet for Ninja. Neil posted a while back about "CheckInstall"
which addresses some of the problems that are related too. Thesre are not 
package managers but merely "tools" that could be employed to resolve some
of the related issues.


> And, there are other package managers besides portage. When developers
> rely on dynamic deps and skip revision bumps, they're screwing users of
> paludis and pkgcore (and you, now that you have dynamic deps disabled).
> None of the magic is mentioned in the PMS, so you really can't rely on
> it and revbumps are needed regardless.

The Package Management Specification [3] is also intertwined in this issue.
It's a very complicated issue and a problem everywhere you have complex
codes and large numbers of codes and packages that define a system. [[4] an
excellent read].  Please, folks with deeper understanding, do write as much
as you can. An updated gentoo dev blog post, at least framing the issues,
would be keen background for everyone, imho.


It becomes even more complicated  when you look at clusters, the resources
and the frameworks to solve problems, including Continuous Integration,
compiling across many languages and other goals of the cluster. I am far,
far away from possessing a sufficiently deep understanding on these issues.
I just peel back a bit of the onion most days and try to discern the issues
 I encounter:: designing a proper cluster for gentoo  does lead one down the
path of unravelling these aforementioned issues, and other issues, from the
myriad of codes, packages and strategies for distributed computing.


hth,
James

[1] http://gittup.org/tup/

[2] http://asic-linux.com.mx/~izto/checkinstall/

[3] https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification

[4] https://blogs.gentoo.org/blueness/2014/08/

Re: [gentoo-user] Re: dynamic deps, wtf are they exactly

[Michael Orlitzky]

On 09/27/2015 03:52 PM, James wrote:
> 
>> That's nice, because now if you want to install or update something
>> else, portage doesn't have to re-execute every ebuild/eclass that could
>> possibly affect the new thing -- it only has to check the VDB.
> 
> I think that this is what GLEP-64 is all about. Enhancing the functional
> utilityh of the VDB with a DAG (Directed Acyclic Graph)?
> 

AFAIK, that GLEP is just about standardizing what goes in the VDB. The
VDB isn't specified in the PMS either, but all package managers need to
record e.g. what files were installed by the package. The GLEP would
standardize some of that stuff, and in the future, the PMS would give
you a way to read it reliably.

The dynamic dependencies issue is more about the contents of the VDB
being wrong.


>> I guess at some point there were a bunch of devs who were messing with
>> dependencies and not bothering to make revision bumps. This can cause
>> users pain, so portage added a new option to ignore the cache and rescan
>> every single relevant ebuild/eclass for sneaky dependency changes. This
>> ensures that portage has the correct dependencies in its head while it's
>> doing resolution, but is much slower.
> 
> There is no proper mechanism to accurately track all of these issue,
> currently, or did I miss this point?

The proper mechanism is "don't do that." The rules for when (not) to
revbump could go on for pages, if there was anyone qualified to write
them -- and I'm not convinced there is. The only safe thing to do is
always revbump unless you're damned sure that you're an exception.

[gentoo-user] Re: dynamic deps, wtf are they exactly

[Martin Vaeth]

James <wireless@tampabay.rr.com> wrote:
>
> Basically from my point of view, something like TUP [1] is needed so
> that at dependency check time you only list files that need
> attention (linking, loading, compiling etc) thus speeding up the
> update processes for the Package Manager (portage).

This is a misunderstanding (originally already from Michael).
The issue is not at all about speed of portage - reading one or
the other file takes the same amount of time.
(And having a dependency graph of files would help concerning
speed only in the very rare situation that no file involving
your current package dependency tree does change.)

The whole issue is only about the policy: If the dependency
information of the installed package and of the current
package differs - what is the "correct" information?

For each choice, it is possible to give examples where
the policy leads to a bad situation, so both,
static deps as well as dynamic deps can break in
certain situations. It is only a question which breakage
you consider more severe and/or harder to recognize/fix
by the user.

Making more revbumps will increase the chance of
no breakage - in case of static deps only for users
who sync and update sufficiently frequently -
of course at the cost of redundant recompiles.

>> I guess at some point there were a bunch of devs who were messing with
>> dependencies and not bothering to make revision bumps. This can cause
>> users pain, so portage added a new option to ignore the cache and rescan
>> every single relevant ebuild/eclass for sneaky dependency changes. This
>> ensures that portage has the correct dependencies in its head while it's
>> doing resolution, but is much slower.

This is historically not correct. Dynamic deps had always been
the only way portage treated dependencies - static deps have only
been used as a fallback and (unfortunately) with the introduction
of subslots.
Once more: It is not about speed, but about: What *are* the
"correct" dependencies? The ones referring to an historical tree
which - especially if you did not update for a long time -
might have almost nothing in common with the current tree
(static deps), or the ones referring to current tree
(dynamic deps)?

With static deps, you will have a strange mixture of historical
dependencies and current ones for the updates.
With dynamic deps, the tree might not be appropriate for your
installed packages.

> There is no proper mechanism to accurately track all of these issue,
> currently, or did I miss this point?

There is no way to automatically decide correctly which of
two differing informations should be used...

[gentoo-user] Re: dynamic deps, wtf are they exactly

[James]

Martin Vaeth <martin <at> mvath.de> writes:

> 
> James <wireless <at> tampabay.rr.com> wrote:
> >
> > Basically from my point of view, something like TUP [1] is needed so
> > that at dependency check time you only list files that need
> > attention (linking, loading, compiling etc) thus speeding up the
> > update processes for the Package Manager (portage).
> 
> This is a misunderstanding (originally already from Michael).
> The issue is not at all about speed of portage - reading one or
> the other file takes the same amount of time.
> (And having a dependency graph of files would help concerning
> speed only in the very rare situation that no file involving
> your current package dependency tree does change.)

One aspect of the problem, but you are partially correct.

> The whole issue is only about the policy: If the dependency
> information of the installed package and of the current
> package differs - what is the "correct" information?

DAG's, from 'graph theory' are often used to form the basis of more
advanced tool, including dependencies between the files, inlcuding
date changes of last compile/touch times.


> For each choice, it is possible to give examples where
> the policy leads to a bad situation, so both,
> static deps as well as dynamic deps can break in
> certain situations. It is only a question which breakage
> you consider more severe and/or harder to recognize/fix
> by the user.

Exactly. The current tools are insufficient, because the data
necessary to build sufficient tools with ever evolving logic constraints 
form a myriad of system requirements, is insufficient; hence what Blueness
refers to as a "Direcent linkages" is a a subset of what a full blown DAG
can do. I use TUP et. al. as examples where a DAG is use to minimize the
item listing for re-compile. But that is by no means the only usage of a DAG
or similar structural enhancement to the tools that can be built.


> Making more revbumps will increase the chance of
> no breakage - in case of static deps only for users
> who sync and update sufficiently frequently -
> of course at the cost of redundant recompiles.

Yes, DAGs can track timestamps and many other forms of data, and organize it
to for easy parsing.



> >> I guess at some point there were a bunch of devs who were messing with
> >> dependencies and not bothering to make revision bumps. This can cause
> >> users pain, so portage added a new option to ignore the cache and rescan
> >> every single relevant ebuild/eclass for sneaky dependency changes. This
> >> ensures that portage has the correct dependencies in its head while it's
> >> doing resolution, but is much slower.

True, but you are looking antedotally, not with the lens of developing tool
that keep systems, robustly pristine; hence the struggles with package
management.


> This is historically not correct. Dynamic deps had always been
> the only way portage treated dependencies - static deps have only
> been used as a fallback and (unfortunately) with the introduction
> of subslots.
> Once more: It is not about speed, but about: What *are* the
> "correct" dependencies? The ones referring to an historical tree
> which - especially if you did not update for a long time -
> might have almost nothing in common with the current tree
> (static deps), or the ones referring to current tree
> (dynamic deps)?

Corrent. But with a DAG approach, you can develop tools that solve these
and newer, unforseen problems. A DAG or a collection of DAGS can lead to a
fully characterized system, where every file, and it's inter-related
relationships, is tracked. This approach will allow those interested the
ability to develop system tools for ensuring a system is and remains,
pristine. Object oriented paradyms result in lots of "slop" and cruft
on a system.


> With static deps, you will have a strange mixture of historical
> dependencies and current ones for the updates.
> With dynamic deps, the tree might not be appropriate for your
> installed packages.

Ever look closing, manually parsing up and down the directory structure of
an older gentoo install, say a few years?  Cruft abounds. Cruft leads to the
'dark side' of computing. *every file* should be fully accounted for, have
it's relationships mapped and documented, including timestamps.
That one reason why commercial, embedded product systems, just run for years
and years, without intervention.

> > There is no proper mechanism to accurately track all of these issue,
> > currently, or did I miss this point?
> 
> There is no way to automatically decide correctly which of
> two differing informations should be used...


Correct, so you build DAGs for data, then parse it, statistically or
manually and discern problems. DAGs are fundamental and probably the
singular most important thing you learn in one of your most important
computer science undergraduate course in graph theory. Computers are all
about repeatability and assurances to accuracy. DAGs are a fundamental tool
for these sorts of scenarios.



hth,
James

[gentoo-user] Re: dynamic deps, wtf are they exactly

[Martin Vaeth]

James <wireless@tampabay.rr.com> wrote:
>[cr
> DAG's

All this can work only if you reflect the complete history
in the DAG. Such approaches had been discussed and eliminated
as unrealistic: You do not want to keep the history forever;
the data will always grow and eventually be too much.
Moreover, there can be overlays which might be added,
perhaps eventually are abondened, replaced by other
overlays, etc. It is not realistic to expect a complete
history from them since you installed once from them.

One must face the fact that at one stage you have the tree you
installed and at another stage you have the current tree with
possibly dramatic changes and no complete history of all changes.

In the lack of such a history, simply there is information
missing to decide the correct proceeding.

You have to choose your poison which is either to take
the old tree or the new tree as a basis (and to fill the
gaps from the other tree).

> Exactly. The current tools are insufficient

The available information is *in principle* insufficient.

[gentoo-user] Re: dynamic deps, wtf are they exactly

[Martin Vaeth]

Michael Orlitzky <mjo@gentoo.org> wrote:
>
> With dynamic deps, portage will scan (that is, execute) all of the
> ebuilds for installed packages that could affect the dependency graph.

This is not correct. This data is already stored in metadata/
(or in /var/cache/edb, depending on the backend),
and just has to be read from there instead of /var/db

The difference between dynamic deps and static deps is
that in the latter case always the information from
/var/db is used, while in the former case the (usually more
up-to-date) version from metadata/ is preferred if it is available.

> Buuuuuuutttttt dynamic-deps mode doesn't always kick in. It interacts
> weirdly with subslots and other things.

That's the real problem: When an automatic (":=") subslot is
involved, portage did not check metadata/ but always took /var/db.
So it was necessary to either fix this inconsistent behaviour
of portage (which would not have been easy though possible),
or to switch to static deps completely which requires a
change of the dump policy.

Both approaches have advantages and disadvantages and
lead to different types of breakage in different situations -
see the lengthy and partly emotional discussion on the developers
list some years ago; I am not going to repeat the arguments.

The developers deciceded to go the way which is simplest for
them but perhaps less convenient for the user: For the user,
it means that in the future he has to rebuild *a lot* of packages
much more often than previously, for no other benefit than
updating /var/db (which would obviously not be necessary
at all for dynamic deps).