From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 92F0B138350 for ; Thu, 26 Mar 2020 08:07:04 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id D33F2E0BF1; Thu, 26 Mar 2020 08:07:03 +0000 (UTC) Received: from mail-yb1-xb41.google.com (mail-yb1-xb41.google.com [IPv6:2607:f8b0:4864:20::b41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 502ACE0BF1 for ; Thu, 26 Mar 2020 08:07:03 +0000 (UTC) Received: by mail-yb1-xb41.google.com with SMTP id n2so1399743ybg.4 for ; Thu, 26 Mar 2020 01:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gentoo-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xcBCW3b3fezyV5BhJ4K/jHPozBsEO6VjQsU9jAQ/rWE=; b=f718+xc3FeIM37VEIqVsRJTQYzq6ST0N8W56SzMBXtuS0eBt9KL8Xq6TuuFxmjKL3i yzaB/WtDz6xP//vwjUutuixh+7J/DUWb49SWnjuWpmS/Vn+p2fWx/pD2FigE5qOBnOaK j/y6/pmgLpWxfWespCVJbT/HjH1nZd0L6MmQMauzZOQa/NdMZoJChST61ttubzGITxw5 NdNhAPnTTCX2x5V+tpLiCmLq+JgOD57XAaKtzhTkfwjknXP7FByWQPDmzMaxA6yAvWES FYSNGBEEmxJ1GO/utFR11tHvEeMpwlqqRk27SPJgXkPrkweELtOQ8NvEu/tAZvVIZ8/i GJOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xcBCW3b3fezyV5BhJ4K/jHPozBsEO6VjQsU9jAQ/rWE=; b=B9y6Cfe1/qZjS8ogz0ZW8JFH8L46qqvxkMxeDDYMKims8I/YdK2L9W95TlGALsSKV8 dxj6iEvukBNGHaTaFIWVbUnK4zi657myw9b6VV0SMovi8KSUaYsCQjejwLgC8w3GkTjj eq8O/1xxqp4duPwRg6u+ekmEGpOHzqGZqRL3CW+5RIim8YlNFruEwU1PzS75E99KCkU4 9dCKTnubMl3SqURhdzqzTT35ruZY2s7EeJcZSC95tXdsjhhfZcTCDliQVnKFDLgOnLTk p/8OIJ5k05LAa/iEruxUcoxeZbCMuYUiBHOPlxtzixyttukj65IX9YE8Nsr2sTRw7Gwm brRQ== X-Gm-Message-State: ANhLgQ0HHgjnqM+Ar/uMp1iGYVQNGXNfAOH/2eeCRwGtTW1TmI5Nog3s Cs5VSMdBb6VJhjTUD2OzYwK5e5CkUCSDaixuR+Ab8fniWc0= X-Google-Smtp-Source: ADFU+vsFyBVal1b8cwYBM4lBzhUk83FmPq61jERk4dX0IBQzPc0RHJxYTbc5lCPuSITGcwMAVRO6TESQDSyYnxuQ5sQ= X-Received: by 2002:a25:bac7:: with SMTP id a7mr11243378ybk.211.1585210022046; Thu, 26 Mar 2020 01:07:02 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <1914109345.7077150.1584923899601.JavaMail.zimbra@laposte.net> <2f3e382c-67d5-3bc5-e1c0-8df1c8b5f7ab@gentoo.org> <869515804.6231453.1585002075418.JavaMail.zimbra@laposte.net> <518e57d9-423c-99ee-91c1-c591dec902be@gentoo.org> <763997347.4854538.1585074666541.JavaMail.zimbra@laposte.net> In-Reply-To: <763997347.4854538.1585074666541.JavaMail.zimbra@laposte.net> From: Alec Warner Date: Thu, 26 Mar 2020 01:06:51 -0700 Message-ID: Subject: Re: [gentoo-portage-dev] precisions on installed packages' dependencies To: gentoo-portage-dev@lists.gentoo.org Cc: Zac Medico Content-Type: multipart/alternative; boundary="0000000000004242c505a1bd78df" X-Archives-Salt: 83cf5f4d-a22f-453d-abe1-3d23d5df2d41 X-Archives-Hash: f187045ec58c11b4f518a588a3647615 --0000000000004242c505a1bd78df Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Mar 24, 2020 at 11:31 AM wrote: > > ----- Zac Medico a =C3=A9crit : > > > > The goal of my tool is to have correct manipulation of package > dependencies, and in particular here, I focus on the packages that are > installed but not in the portage tree/a local overlay anymore (the proble= m > does not occur for other packages). > > > It seems that installed packages do not store which are the actual cp= v > they depend on. Correct? > > > > Right, because unfortunately that's something that changes over time. > > > > Also, we may not be able to pin it down at any given moment if we have > > inconsistent || preferences as described here: > > > > > https://archives.gentoo.org/gentoo-dev/message/550d3859dea6d0fb0b39064628= 992634 > > Hmm, I think I see what you mean. > Storing the cpvs that was used during solving the package's dependencies > would be too restrictive, since two different packages could provide the > exact same functionalities/libraries. > And so, during a system update, only looking at the cpv dependencies woul= d > trigger useless recompilation of the packages that depend on the updated > packages. > Correct? > > Btw, my tool's solver does not have the problem discussed in the thread > you're mentioning: atom order in lists has no influence in my solver. > Would fixing the inconsistent || preferences make storing cpvs for > installed packages more realistic? > > > > > Also, I wanted to use the ebuild tool to install/uninstall package, > which is not possible with this solution apparently. > > > > Why not? Would the preserve-libs feature solve your problem? > > ... I'm sorry, I wasn't aware of this feature. > It would definitively solve the issue (except, as described in the bug > 459038, when external tools remove libs). > > This discussion is very interesting! > If I take this double layer of dependencies, I have to check how this > influences the theory underlying my tool. > > However, I still doubt that only storing the soname dependencies is enoug= h. > Consider package A (that cannot be recompiled) that depends on package B > which provides lib L.so. > B is recompiled with different use flags, which put different > functionalities in L.so. > The dependencies of A are still satisfied (B is installed, L.so is > available), but since the content of L.so changed, A cannot execute anymo= re. > Hypothetically, can this scenario occur? > Can this scenario occur in practice? > Is there a way in emerge/portage to avoid it? > > > > Well, there are a lot of upgrades that can't be performed without > > temporarily breaking something, so the "forbid broken dependencies" ide= a > > doesn't sound feasible to me. > > Could you tell me about several instances of such needed dependency > breakage? > You have far more experience than me on this, and it would be nice for me > to know what I'm up against. > A lot of this has to do with the specifics of how package managers manage system state, as well as various quirks of subsets of the tree. For example, a perl upgrade (X->Y) will often break perl modules who expect perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (so we SLOT perl and have N perls installed.) Then you need to decide which version of perl to build things against (X or Y, or both?) We took this tactic in the python ecosystem; but perl is not slotted in Gentoo, and so upgrading perl breaks all perl modules. There is a tool (gentoo-perl-cleaner) that will walk the deptree and fix all of these broken packages that you run after an upgrade. I'm not sure it's strictly avoidable. You could build perl-Y, then rebuild all perl-modules against perl-Y, then merge the entire result to the livefs. This will reduce the breakage time but likely not eliminate it; plus it seems hard to implement in practice without modern filesystem tools (overlayfs, btrfs, zfs or similar tech to make it atomic.) It also doesn't account for executing code. What happens to perl-X code that is executing when you unmerge perl-X? The short answer is that code might break and 'proper' management means you should restart services after an upgrade (something Gentoo doesn't typically do; but is common in Debian for example.) -A > Many thanks! > Michael > > --0000000000004242c505a1bd78df Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Mar 24, 2020 at 11:31 AM <= michael.lienhardt@laposte.= net> wrote:

----- Zac Medico <zmedico@gentoo.org> a =C3=A9crit :

> > The goal of my tool is to have correct manipulation of package de= pendencies, and in particular here, I focus on the packages that are instal= led but not in the portage tree/a local overlay anymore (the problem does n= ot occur for other packages).
> > It seems that installed packages do not store which are the actua= l cpv they depend on. Correct?
>
> Right, because unfortunately that's something that changes over ti= me.
>
> Also, we may not be able to pin it down at any given moment if we have=
> inconsistent || preferences as described here:
>
> https://archives= .gentoo.org/gentoo-dev/message/550d3859dea6d0fb0b39064628992634

Hmm, I think I see what you mean.
Storing the cpvs that was used during solving the package's dependencie= s would be too restrictive, since two different packages could provide the = exact same functionalities/libraries.
And so, during a system update, only looking at the cpv dependencies would = trigger useless recompilation of the packages that depend on the updated pa= ckages.
Correct?

Btw, my tool's solver does not have the problem discussed in the thread= you're mentioning: atom order in lists has no influence in my solver.<= br> Would fixing the inconsistent || preferences make storing cpvs for installe= d packages more realistic?


> > Also, I wanted to use the ebuild tool to install/uninstall packag= e, which is not possible with this solution apparently.
>
> Why not? Would the preserve-libs feature solve your problem?

... I'm sorry, I wasn't aware of this feature.
It would definitively solve the issue (except, as described in the bug 4590= 38, when external tools remove libs).

This discussion is very interesting!
If I take this double layer of dependencies, I have to check how this influ= ences the theory underlying my tool.

However, I still doubt that only storing the soname dependencies is enough.=
Consider package A (that cannot be recompiled) that depends on package B wh= ich provides lib L.so.
B is recompiled with different use flags, which put different functionaliti= es in L.so.
The dependencies of A are still satisfied (B is installed, L.so is availabl= e), but since the content of L.so changed, A cannot execute anymore.
Hypothetically, can this scenario occur?
Can this scenario occur in practice?
Is there a way in emerge/portage to avoid it?


> Well, there are a lot of upgrades that can't be performed without<= br> > temporarily breaking something, so the "forbid broken dependencie= s" idea
> doesn't sound feasible to me.

Could you tell me about several instances of such needed dependency breakag= e?
You have far more experience than me on this, and it would be nice for me t= o know what I'm up against.

<= div>A lot of this has to do with the specifics of how package managers mana= ge system state, as well as various quirks of subsets of the tree. For exam= ple, a perl upgrade (X->Y) will often break perl modules who expect perl= -X, but get perl-Y. So one fix is to try to keep perl-X installed (so we SL= OT perl and have N perls installed.) Then you need to decide which version = of perl to build things against (X or Y, or both?) We took this tactic in t= he python ecosystem; but perl is not slotted in Gentoo, and so upgrading pe= rl breaks all perl modules. There is a tool (gentoo-perl-cleaner) that will= walk the deptree and fix all of these broken packages that you run after a= n upgrade.

I'm not sure it's strictly avoi= dable. You could build perl-Y, then rebuild all perl-modules against perl-Y= , then merge the entire result to the livefs. This will reduce the breakage= time but likely not eliminate it; plus it seems hard to implement in pract= ice without modern filesystem tools (overlayfs, btrfs, zfs or similar tech = to make it atomic.) It also doesn't account for executing code. What ha= ppens to perl-X code that is executing when you unmerge perl-X? The short a= nswer is that code might break and 'proper' management means you sh= ould restart services after an upgrade (something Gentoo doesn't typica= lly do; but is common in Debian for example.)

-A


Many thanks!
Michael

--0000000000004242c505a1bd78df--