Re: [gentoo-user] dynamic deps, wtf are they exactly

Re: [gentoo-user] dynamic deps, wtf are they exactly

[brettrsears]

                                                                                                                      
------Original Message------
From: Michael Orlitzky
To: gentoo-user@lists.gentoo.org
ReplyTo: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] dynamic deps, wtf are they exactly
Sent: Sep 27, 2015 5:21 PM

On 09/27/2015 03:34 PM, Alan McKinnon wrote:
> 
> Now it all makes sense, as a bonus I now see why why so many senior devs
> are so pedantic about revbumps (they have reason).
> 
> Now that I know what the symptom will be, are bug reports useful?
> 

I don't think most developers are aware of the problem, so if you point
out that the lack of a revbump causes pain, many will be glad to know
and adjust their behavior.

Or, you might just get yelled at. It's one of /those/ issues.




Sent from my Verizon Wireless BlackBerry

[gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

Does anyone here know what dynamic deps are exactly, how they work and
can describe them?

There's lots of talk over on -dev about getting rid of them and the
closest description is from Ciaran, something like:

"ancient shit that never worked right, involving phases of the moon"

10 days ago I switched to --dynamic-deps=m in make.conf, had to emerge
-e world to get around all the resulting kde hard blocks, and all was
good till today, kio-extras was blocking plasma-workspace in impossible
ways, noticeably that the block wasn't in the ebuild at all :-)

kio-extras and dolphon were to be upgraded, plasma-workspace was not.
I've fixed it (remerge plasma-workspace to clean up cruft).

I would have thought emerge -e would have cleaned all that stuff out,
maybe it's related to removing the kde overlay.

So, my question: wtf are dynamic deps? How do I find the records they
must leave behind in portage?
-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Mike Gilbert]

On Sun, Sep 27, 2015 at 11:07 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> So, my question: wtf are dynamic deps? How do I find the records they
> must leave behind in portage?

For the latter question, you can rebuild affected packages like so:

emerge --with-bdeps=y @changed-deps.

You can also add --changed-deps to your emerge command line for world updates.

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

On 27/09/2015 17:12, Mike Gilbert wrote:
> On Sun, Sep 27, 2015 at 11:07 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> So, my question: wtf are dynamic deps? How do I find the records they
>> must leave behind in portage?
> 
> For the latter question, you can rebuild affected packages like so:
> 
> emerge --with-bdeps=y @changed-deps.
> 
> You can also add --changed-deps to your emerge command line for world updates.
> 


Thanks. Running that gives surprising and unexpected results. I'm now
curious what changed-deps really does, and the man page is terse on this.

I would have thought portage already does that automatically, but I see
a difference. If a package's deps change, but everything is still
satisfied, portage will do nothing. Does @changed-deps rebuild
everything that changed anyway, regardless is portage thinks it still OK?

Also, these two similar commands return different results (I have
bdeps=y in DEFAULT_OPTS btw):

emerge -uND --changed-deps=y world         (51 packages)
emerge @changed-deps                       (11 packages)

Do you know why those commands give different results?
The smaller list is a strict subset of the longer one.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Mike Gilbert]

On Sun, Sep 27, 2015 at 11:28 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 27/09/2015 17:12, Mike Gilbert wrote:
>> On Sun, Sep 27, 2015 at 11:07 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>>> So, my question: wtf are dynamic deps? How do I find the records they
>>> must leave behind in portage?
>>
>> For the latter question, you can rebuild affected packages like so:
>>
>> emerge --with-bdeps=y @changed-deps.
>>
>> You can also add --changed-deps to your emerge command line for world updates.
>>
>
>
> Thanks. Running that gives surprising and unexpected results. I'm now
> curious what changed-deps really does, and the man page is terse on this.
>
> I would have thought portage already does that automatically, but I see
> a difference. If a package's deps change, but everything is still
> satisfied, portage will do nothing. Does @changed-deps rebuild
> everything that changed anyway, regardless is portage thinks it still OK?

Basically, yes. If [R]DEPEND in /var/db/pkg is different from the
values in the ebuilds in the tree, @changed-deps will select it.

> Also, these two similar commands return different results (I have
> bdeps=y in DEFAULT_OPTS btw):
>
> emerge -uND --changed-deps=y world         (51 packages)
> emerge @changed-deps                       (11 packages)
>
> Do you know why those commands give different results?
> The smaller list is a strict subset of the longer one.

That difference is surprising; you would probably need to ask the
portage developers to get a real answer.

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Michael Orlitzky]

On 09/27/2015 11:07 AM, Alan McKinnon wrote:
> Does anyone here know what dynamic deps are exactly, how they work and
> can describe them?
> 
> There's lots of talk over on -dev about getting rid of them and the
> closest description is from Ciaran, something like:
> 
> "ancient shit that never worked right, involving phases of the moon"
> 

"Dynamic dependencies" is a portage option. The abridged version is:
whenever you go to install/update something, some of its dependencies
may already exist on your machine. For example, if I go to install gimp
right now, gtk is already there.

With dynamic deps, portage will scan (that is, execute) all of the
ebuilds for installed packages that could affect the dependency graph.
If any of those ebuilds have changed, portage will use the new
information rather than the info present when you originally installed
the package. So if the gtk ebuild (that I installed a month ago) has
been altered, portage will re-read it on the fly, ignoring what was in
there a month ago.

too short; didn't explain?

When you install a package, its dependencies are recorded in the VDB:

  $ cat /var/db/pkg/app-editors/nano-2.3.6/RDEPEND
  >=sys-libs/ncurses-5.9-r1[unicode] sys-apps/file

That's nice, because now if you want to install or update something
else, portage doesn't have to re-execute every ebuild/eclass that could
possibly affect the new thing -- it only has to check the VDB.

But, if I'm the maintainer of nano and I change that ncurses dependency
(let's say I drop it), then I have to make a revision bump on nano.
Otherwise, the wrong dependency would stay recorded on everyone's
system, and portage would happily use the recorded deps.

I guess at some point there were a bunch of devs who were messing with
dependencies and not bothering to make revision bumps. This can cause
users pain, so portage added a new option to ignore the cache and rescan
every single relevant ebuild/eclass for sneaky dependency changes. This
ensures that portage has the correct dependencies in its head while it's
doing resolution, but is much slower.

And then that mode was made default, which is where we are today. It
doesn't sound that bad so far -- just a tradeoff between speed and the
risk of using an incorrect cached dependency.

Buuuuuuutttttt dynamic-deps mode doesn't always kick in. It interacts
weirdly with subslots and other things. If portage can't find the same
ebuild to scan, it will find one that "looks like it." If that doesn't
work, it's supposed to fall back to the cache. Nobody has a flow chart
of exactly how this works. It's all in the code and there's no
specification.

And, there are other package managers besides portage. When developers
rely on dynamic deps and skip revision bumps, they're screwing users of
paludis and pkgcore (and you, now that you have dynamic deps disabled).
None of the magic is mentioned in the PMS, so you really can't rely on
it and revbumps are needed regardless.

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

On 27/09/2015 19:26, Michael Orlitzky wrote:
> On 09/27/2015 11:07 AM, Alan McKinnon wrote:
>> Does anyone here know what dynamic deps are exactly, how they work and
>> can describe them?
>>
>> There's lots of talk over on -dev about getting rid of them and the
>> closest description is from Ciaran, something like:
>>
>> "ancient shit that never worked right, involving phases of the moon"
>>
> 
> "Dynamic dependencies" is a portage option. The abridged version is:
> whenever you go to install/update something, some of its dependencies
> may already exist on your machine. For example, if I go to install gimp
> right now, gtk is already there.
> 
> With dynamic deps, portage will scan (that is, execute) all of the
> ebuilds for installed packages that could affect the dependency graph.
> If any of those ebuilds have changed, portage will use the new
> information rather than the info present when you originally installed
> the package. So if the gtk ebuild (that I installed a month ago) has
> been altered, portage will re-read it on the fly, ignoring what was in
> there a month ago.
> 
> too short; didn't explain?

Nope, I see a problem already :-)

If portage can't reliably tell if the ebuild changed, the cache may or
may not be valid and portage wouldn't know.


> When you install a package, its dependencies are recorded in the VDB:
> 
>   $ cat /var/db/pkg/app-editors/nano-2.3.6/RDEPEND
>   >=sys-libs/ncurses-5.9-r1[unicode] sys-apps/file
> 
> That's nice, because now if you want to install or update something
> else, portage doesn't have to re-execute every ebuild/eclass that could
> possibly affect the new thing -- it only has to check the VDB.
> 
> But, if I'm the maintainer of nano and I change that ncurses dependency
> (let's say I drop it), then I have to make a revision bump on nano.
> Otherwise, the wrong dependency would stay recorded on everyone's
> system, and portage would happily use the recorded deps.
> 
> I guess at some point there were a bunch of devs who were messing with
> dependencies and not bothering to make revision bumps. This can cause
> users pain, so portage added a new option to ignore the cache and rescan
> every single relevant ebuild/eclass for sneaky dependency changes. This
> ensures that portage has the correct dependencies in its head while it's
> doing resolution, but is much slower.
> 
> And then that mode was made default, which is where we are today. It
> doesn't sound that bad so far -- just a tradeoff between speed and the
> risk of using an incorrect cached dependency.
> 
> Buuuuuuutttttt dynamic-deps mode doesn't always kick in. It interacts
> weirdly with subslots and other things. If portage can't find the same
> ebuild to scan, it will find one that "looks like it." If that doesn't
> work, it's supposed to fall back to the cache. Nobody has a flow chart
> of exactly how this works. It's all in the code and there's no
> specification.
> 
> And, there are other package managers besides portage. When developers
> rely on dynamic deps and skip revision bumps, they're screwing users of
> paludis and pkgcore (and you, now that you have dynamic deps disabled).
> None of the magic is mentioned in the PMS, so you really can't rely on
> it and revbumps are needed regardless.


Now it all makes sense, as a bonus I now see why why so many senior devs
are so pedantic about revbumps (they have reason).

Now that I know what the symptom will be, are bug reports useful?


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Michael Orlitzky]

On 09/27/2015 03:34 PM, Alan McKinnon wrote:
> 
> Now it all makes sense, as a bonus I now see why why so many senior devs
> are so pedantic about revbumps (they have reason).
> 
> Now that I know what the symptom will be, are bug reports useful?
> 

I don't think most developers are aware of the problem, so if you point
out that the lack of a revbump causes pain, many will be glad to know
and adjust their behavior.

Or, you might just get yelled at. It's one of /those/ issues.

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Alan McKinnon]

On 28/09/2015 00:21, Michael Orlitzky wrote:
> On 09/27/2015 03:34 PM, Alan McKinnon wrote:
>>
>> Now it all makes sense, as a bonus I now see why why so many senior devs
>> are so pedantic about revbumps (they have reason).
>>
>> Now that I know what the symptom will be, are bug reports useful?
>>
> 
> I don't think most developers are aware of the problem, so if you point
> out that the lack of a revbump causes pain, many will be glad to know
> and adjust their behavior.
> 
> Or, you might just get yelled at. It's one of /those/ issues.



Oh, we're well used to deal with touchy issues around here. And we do
our fair share of yelling too :-)

Thanks for the feedback. Appreciated.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] dynamic deps, wtf are they exactly

[Rich Freeman]

On Sun, Sep 27, 2015 at 6:21 PM, Michael Orlitzky <mjo@gentoo.org> wrote:
> On 09/27/2015 03:34 PM, Alan McKinnon wrote:
>>
>> Now it all makes sense, as a bonus I now see why why so many senior devs
>> are so pedantic about revbumps (they have reason).
>>
>> Now that I know what the symptom will be, are bug reports useful?
>>
>
> I don't think most developers are aware of the problem, so if you point
> out that the lack of a revbump causes pain, many will be glad to know
> and adjust their behavior.
>
> Or, you might just get yelled at. It's one of /those/ issues.

While working out the right way to handle dep changes in eclasses
might take a little longer, I suspect that we'll have a policy for
revbumping on dep changes in ebuilds and perhaps virtuals fairly soon.
There really wasn't much loud objection when the proposal came up
again last week - not nearly as much as the last time this came up.
With the increasing issues with subslots I think devs are appreciating
that the current state isn't really great.

-- 
Rich