From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id E87E1138350 for ; Sat, 28 Mar 2020 06:48:29 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 2AF1EE0CBD; Sat, 28 Mar 2020 06:48:29 +0000 (UTC) Received: from mail-yb1-xb36.google.com (mail-yb1-xb36.google.com [IPv6:2607:f8b0:4864:20::b36]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id BA028E0CBD for ; Sat, 28 Mar 2020 06:48:28 +0000 (UTC) Received: by mail-yb1-xb36.google.com with SMTP id l84so5885229ybb.1 for ; Fri, 27 Mar 2020 23:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gentoo-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wqgN3DdOyYlzxonXfG2FBykpMOhOZNzbDQtA8Q9XOww=; b=CxA2z5/1yinophLs4RpzjgVKIO+370Sh7gtF6KR4pGmGZuBZFWa6g2Ks7V1PZZ13ZP Ur5rLbjT6iT3ATuF4O+LZWs1BTdmSg4z8FCHQd9PgfGsKGMnZpo79LXzvTmIi8ySqcNk b7WZzS8rQEjWJYTOdVMlNi+ruPEdHdgj02g8h1HucDWXcPTKWz78OKdO/KrgD8MmJvZT vbjnyGQpBFbWNiIRRj1cnk1kg1TlG8TgQYq+L3YtqPGwaTkMHFdD5nXpiSKY8GynPn5n oNViRldiqpbfKUxVz4h+oSPNoaGnqVEwITMaOd0Vyg5KfWJfybZCTg0C3wk+Y8C8VS15 mTLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wqgN3DdOyYlzxonXfG2FBykpMOhOZNzbDQtA8Q9XOww=; b=Kx+80r+ttwoyhm3PZw+oRbeoSrt67WBB6wGwiDl/TP6f9RL13oQiIuOyFSjVvnfS/b R3x8GsEE+wPNS/rPYs2ZEooPhlpN9YNf43M58kNjJbfbC/EKIv3B5c2PX3kC02K62Bku 5AYjD3VtvX8hX/sHumUvJ3Gt/nM5BrqbX3smhX/x6BAdGXx6GPbDF9kUsgWi4jHvpyuj QDuXcmAOeLYUcTOU2Zlfj93rUUAF146OzM2TF5Sjpa83Rw2FtD49j6IU9TQFUSsx52Va Y9IOdIDDMGQLBe6gjZ7k0EigqfPV1hsC/mV52hpjcIwCj4lLe7oXSBSuyaR+EivjE6aU 7dxg== X-Gm-Message-State: ANhLgQ3aOO3tau7tQGA8XlgwyPi/xiMArl7B9WUmORmLAOmLRNkfPORO jyndbODVP/rFjBDSFt60Q2NfQKy6U/oOGjri3FhRNIHtgSQErA== X-Google-Smtp-Source: ADFU+vtKrXzln80fqO8uSPigRkz2Lcb3sp5/v902+G7vCwNDW88TOv19C2qUJ9NKE+apImJFGFsrsFyvRSn/lpM2rPM= X-Received: by 2002:a25:84d1:: with SMTP id x17mr961664ybm.507.1585378107156; Fri, 27 Mar 2020 23:48:27 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <1928379103.2245185.1585313194988.JavaMail.zimbra@laposte.net> <324215029.2913035.1585317594992.JavaMail.zimbra@laposte.net> In-Reply-To: <324215029.2913035.1585317594992.JavaMail.zimbra@laposte.net> From: Alec Warner Date: Fri, 27 Mar 2020 23:48:16 -0700 Message-ID: Subject: Re: [gentoo-portage-dev] precisions on installed packages' dependencies To: gentoo-portage-dev@lists.gentoo.org Cc: Zac Medico Content-Type: multipart/alternative; boundary="000000000000e98e0605a1e49a2f" X-Archives-Salt: d4671493-b1d8-4cb8-b47f-7fde11da3b8c X-Archives-Hash: e5bb0c7a8432198140c43c1767290171 --000000000000e98e0605a1e49a2f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Mar 27, 2020 at 7:00 AM wrote: > > ----- Alec Warner a =C3=A9crit : > > On Tue, Mar 24, 2020 at 11:31 AM wrote: > > > However, I still doubt that only storing the soname dependencies is > enough. > > > Consider package A (that cannot be recompiled) that depends on packag= e > B > > > which provides lib L.so. > > > B is recompiled with different use flags, which put different > > > functionalities in L.so. > > > The dependencies of A are still satisfied (B is installed, L.so is > > > available), but since the content of L.so changed, A cannot execute > anymore. > > > Hypothetically, can this scenario occur? > > > Can this scenario occur in practice? > > > Is there a way in emerge/portage to avoid it? > > > > > You have far more experience than me on this, and it would be nice fo= r > me > > > to know what I'm up against. > > > > A lot of this has to do with the specifics of how package managers mana= ge > > system state, as well as various quirks of subsets of the tree. For > > example, a perl upgrade (X->Y) will often break perl modules who expect > > perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (= so > > we SLOT perl and have N perls installed.) Then you need to decide which > > version of perl to build things against (X or Y, or both?) We took this > > tactic in the python ecosystem; but perl is not slotted in Gentoo, and = so > > upgrading perl breaks all perl modules. There is a tool > > (gentoo-perl-cleaner) that will walk the deptree and fix all of these > > broken packages that you run after an upgrade. > > > > I'm not sure it's strictly avoidable. You could build perl-Y, then > rebuild > > all perl-modules against perl-Y, then merge the entire result to the > > livefs. This will reduce the breakage time but likely not eliminate it; > > plus it seems hard to implement in practice without modern filesystem > tools > > (overlayfs, btrfs, zfs or similar tech to make it atomic.) It also > doesn't > > account for executing code. What happens to perl-X code that is executi= ng > > when you unmerge perl-X? The short answer is that code might break and > > 'proper' management means you should restart services after an upgrade > > (something Gentoo doesn't typically do; but is common in Debian for > > example.) > > Many thanks for this answer. > To sum up what I understood, the problem is not really the dependencies, > but which recompilation (and service restart) are triggered with an updat= e. > Yes and no. Assume a spherical package manager that could detect everything, we basically need to do the following for a perl X -> Y upgrade= . 1 User triggers X - Y upgrade. 2 build perl-Y, but don't merge it to the livefs. 3 numerate all deps of perl-Y, build those (but don't merge them to the livefs, but they need to build against perl-Y, not perl-X) 4 with atomic_transaction: 4a Merge perl-Y to the livefs 4b Merge perl-Y's dependencies to the live fs 5 restart anything that is running perl-X, so that it runs with perl-Y 6 unmerge perl-X In practice we cannot always do 3 or 4 or 5. We will miss dependencies (due to missing depgraph information) both in the package depgraph as well as the service depgraph. So in practice we do: 1 user triggers X -Y upgrade. 2 build perl-Y, merge it to the livefs, unmerge perl-X 3 run gentoo-perl-cleaner to upgrade the dependencies broken by the X - Y upgrade (via slot deps, or whatever mechanism, it can vary.) 4 restart anything that is running perl-X And just accept that from 2-4..well stuff will be broken. We can minimize the time by building binpkgs ahead of time for example, and merging the binpkgs in a parallel. Note that 5 and 4 are the same problem in both lists. Note that per the guide linked below, sometimes 2-3 can be done 'at once', although again practically speaking "During Perl upgrade, packages that depend on Perl may become unavailable." even if they are all handled by emerge, because there are many race conditions in the order that packages get merged to the livefs. > In gentoo, there is the ":=3D" slot operator (and others similar) in > dependencies that trigger the recompilation when the dependency's slot > change, but this is the only existing mechanism. > And this is why every time perl changes, the compilation of its modules i= s > not triggered and they are most probably broken. > Correct? > They are supposed to be triggered: https://wiki.gentoo.org/wiki/Perl#Upgrading_.28major_version.29 for example says this. But this is up to how callers run the tool. For example gentoo infra executes emerge via puppet, and we never execute emerge -uDNav --with-bdeps=3Dy --backtrack=3D100 --autounmask-keep-masks=3Dy @world, we h= ave our own piece of code that handles perl upgrades. > And then, in this context, keeping the installed packages' dependencies > consistent is up to debate: packages will get broken in any case... > It is clearly impossible to have a tool that automatically detect all > implementation dependency breakage. > > Again, there's something I probably don't see: why was slot operators > chosen (among other possibilities) as a mechanism to trigger recompilatio= n? > I'm very grateful to you all for the time you take to read and answer my > questions. Sorry I'm being overly academic. My concern earlier is that you mentioned a goal of "never breaking installed packages' which I found to be a fairly audacious goal. The idea is that we should build tools that achieve this practically (e.g. via heuristics such as :=3D slot operators) while understanding that the complexities of application deploys are legion and the tool will never handle them all. So the goal of never breaking them is more an idealistic goal rather than a practical reality. -A > Best, > Michael > > --000000000000e98e0605a1e49a2f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Mar 27, 2020 at 7:00 AM <<= a href=3D"mailto:michael.lienhardt@laposte.net">michael.lienhardt@laposte.n= et> wrote:
antarus@gentoo.org> a =C3=A9crit :
> On Tue, Mar 24, 2020 at 11:31 AM <michael.lienhardt@laposte.net> wro= te:
> > However, I still doubt that only storing the soname dependencies = is enough.
> > Consider package A (that cannot be recompiled) that depends on pa= ckage B
> > which provides lib L.so.
> > B is recompiled with different use flags, which put different
> > functionalities in L.so.
> > The dependencies of A are still satisfied (B is installed, L.so i= s
> > available), but since the content of L.so changed, A cannot execu= te anymore.
> > Hypothetically, can this scenario occur?
> > Can this scenario occur in practice?
> > Is there a way in emerge/portage to avoid it?


> > You have far more experience than me on this, and it would be nic= e for me
> > to know what I'm up against.
>
> A lot of this has to do with the specifics of how package managers man= age
> system state, as well as various quirks of subsets of the tree. For > example, a perl upgrade (X->Y) will often break perl modules who ex= pect
> perl-X, but get perl-Y. So one fix is to try to keep perl-X installed = (so
> we SLOT perl and have N perls installed.) Then you need to decide whic= h
> version of perl to build things against (X or Y, or both?) We took thi= s
> tactic in the python ecosystem; but perl is not slotted in Gentoo, and= so
> upgrading perl breaks all perl modules. There is a tool
> (gentoo-perl-cleaner) that will walk the deptree and fix all of these<= br> > broken packages that you run after an upgrade.
>
> I'm not sure it's strictly avoidable. You could build perl-Y, = then rebuild
> all perl-modules against perl-Y, then merge the entire result to the > livefs. This will reduce the breakage time but likely not eliminate it= ;
> plus it seems hard to implement in practice without modern filesystem = tools
> (overlayfs, btrfs, zfs or similar tech to make it atomic.) It also doe= sn't
> account for executing code. What happens to perl-X code that is execut= ing
> when you unmerge perl-X? The short answer is that code might break and=
> 'proper' management means you should restart services after an= upgrade
> (something Gentoo doesn't typically do; but is common in Debian fo= r
> example.)

Many thanks for this answer.
To sum up what I understood, the problem is not really the dependencies, bu= t which recompilation (and service restart) are triggered with an update.

Yes and no. Assume a spherical package m= anager that could detect everything, we basically need to do the following = for a perl X -> Y upgrade.

1 User triggers X - = Y upgrade.
2 build perl-Y, but don't merge it to the livefs.<= /div>
3 numerate all deps of perl-Y, build those (but don't merge t= hem to the livefs, but they need to build against perl-Y, not perl-X)
=
4 with atomic_transaction:
=C2=A0 4a Merge perl-Y to the liv= efs
=C2=A0 4b Merge perl-Y's dependencies to the live fs
5 restart anything that is running perl-X, so that it runs with perl-= Y
6 unmerge perl-X

In practice we cannot= always do 3 or 4 or 5. We will miss dependencies (due to missing depgraph = information) both in the package depgraph as well as the service depgraph.<= /div>

So in practice we do:
1 user triggers X = -Y upgrade.
2 build perl-Y, merge it to the livefs, unmerge perl-= X
3 run gentoo-perl-cleaner to upgrade the dependencies broken by= the X - Y upgrade (via slot deps, or whatever mechanism, it can vary.)
4 restart anything that is running perl-X

A= nd just accept that from 2-4..well stuff will be broken. We can minimize th= e time by building binpkgs=C2=A0ahead of time for example, and merging the = binpkgs in a parallel. Note that 5 and 4 are=C2=A0the same problem in both = lists. Note that per the guide linked below, sometimes 2-3 can be done '= ;at once', although again practically speaking "During Perl upgrad= e, packages that depend on Perl may become unavailable." even if they = are all handled by emerge, because there are many race conditions in the or= der that packages get merged to the livefs.
=C2=A0
In gentoo, there is the ":=3D" slot operator (and others similar)= in dependencies that trigger the recompilation when the dependency's s= lot change, but this is the only existing mechanism.
And this is why every time perl changes, the compilation of its modules is = not triggered and they are most probably broken.
Correct?

And then, in this context, keeping the installed packages' dependencies= consistent is up to debate: packages will get broken in any case...
It is clearly impossible to have a tool that automatically detect all imple= mentation dependency breakage.

Again, there's something I probably don't see: why was slot operato= rs chosen (among other possibilities) as a mechanism to trigger recompilati= on?
I'm very grateful to you all for the time you take to read and answer m= y questions.




Best,
Michael

--000000000000e98e0605a1e49a2f--