[gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Sebastian Pipping]

Hi all,


I noticed that dev-util/cmake depends on dev-libs/expat and that
libexpat upstream (where I'm involved) is in the process of
dropping GNU Autotools altogether in favor of CMake in the near future,
potentially the next release (without any known target release date).

CMake bundles a (previously outdated and vulnerable) copy of expat so
I'm not sure if re-activating that bundle — say with a new use flag
"system-expat" — would be a good thing to resort to for breaking the
cycle, with regard to security in particular.

Do you have any ideas how to avoid a bad circular dependency issue for
our users in the future?  Are you aware of similar problems and
solutions from the past?

Thanks and best



Sebastian

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On Wed, 2019-12-18 at 22:02 +0100, Sebastian Pipping wrote:
> Hi all,
> 
> 
> I noticed that dev-util/cmake depends on dev-libs/expat and that
> libexpat upstream (where I'm involved) is in the process of
> dropping GNU Autotools altogether in favor of CMake in the near future,
> potentially the next release (without any known target release date).
> 
> CMake bundles a (previously outdated and vulnerable) copy of expat so
> I'm not sure if re-activating that bundle — say with a new use flag
> "system-expat" — would be a good thing to resort to for breaking the
> cycle, with regard to security in particular.
> 
> Do you have any ideas how to avoid a bad circular dependency issue for
> our users in the future?  Are you aware of similar problems and
> solutions from the past?
> 

I know that's an unhappy idea but maybe it's time to include CMake
in stage3.  Then it would be just a matter of temporarily enabling
bundled libs for stage builds, I guess.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Piotr Karbowski]

[-- Attachment #1.1: Type: text/plain, Size: 396 bytes --]

Hi,

On 18/12/2019 22.08, Michał Górny wrote:
> I know that's an unhappy idea but maybe it's time to include CMake
> in stage3.  Then it would be just a matter of temporarily enabling
> bundled libs for stage builds, I guess.

Not sure what's unhappy about it, but I like the idea, it will be
painless for new users with rather limited cost of ownership on Gentoo side.

-- Piotr.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

On Wed, 2019-12-18 at 22:10 +0100, Piotr Karbowski wrote:
> Hi,
> 
> On 18/12/2019 22.08, Michał Górny wrote:
> > I know that's an unhappy idea but maybe it's time to include CMake
> > in stage3.  Then it would be just a matter of temporarily enabling
> > bundled libs for stage builds, I guess.
> 
> Not sure what's unhappy about it, but I like the idea, it will be
> painless for new users with rather limited cost of ownership on Gentoo side.
> 

Increase of stage size may make people unhappy.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

Il giorno mer 18 dic 2019 alle ore 22:03 Sebastian Pipping <sping@gentoo.org>
ha scritto:

>
> CMake bundles a (previously outdated and vulnerable) copy of expat so
> I'm not sure if re-activating that bundle — say with a new use flag
> "system-expat" — would be a good thing to resort to for breaking the
> cycle, with regard to security in particular.
>
> Pushing gently upstream to upgrade bundled expat copy would (at least
temporarily) fix the issue and also benefit other use cases. Maybe they are
Gentoo friendly
they also release quite often, which would fix the problem soon

[-- Attachment #2: Type: text/html, Size: 932 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Sergei Trofimovich]

On Wed, 18 Dec 2019 22:02:47 +0100
Sebastian Pipping <sping@gentoo.org> wrote:

> Hi all,
> 
> 
> I noticed that dev-util/cmake depends on dev-libs/expat and that
> libexpat upstream (where I'm involved) is in the process of
> dropping GNU Autotools altogether in favor of CMake in the near future,
> potentially the next release (without any known target release date).
> 
> CMake bundles a (previously outdated and vulnerable) copy of expat so
> I'm not sure if re-activating that bundle — say with a new use flag
> "system-expat" — would be a good thing to resort to for breaking the
> cycle, with regard to security in particular.
> 
> Do you have any ideas how to avoid a bad circular dependency issue for
> our users in the future?  Are you aware of similar problems and
> solutions from the past?

Fun fact: cmake is not keyworded for riscv.

To think of solutions enumerating the arising problems explicitly might
help here. I see a few:
1. initial system bootstrap requires solving a circular dependency
2. recovery from broken state (expat soname bump without preserved
  libs or cmake broken by one of many depends it has)

[2.] effectively forbids a dependency circle.

[1.] is hard to solve without users' intervention to at least flip a default flag.

Some other distributions provide two packages to break the cycle.
Example Gentoo solution would be: "cmake.ebuild" depends on "expat.ebuild",
"expat.ebuild" depends on "cmake-with-bundled-expat.ebuild".

Some examples of circular dependencies are:
a) gcc/glibc/binutils toolchain. the solution is to provide prebuilt
  binaries as part of stage3.
b) self-hosted languages without an interpreter in C (rust, golang, ghc).
  The solution is to provide prebuilt binaries in ebuilds.
c) circular dependencies in tests. The solution is careful user's USE flags
  juggling: enable USE=test only for a package being tested.
d) glibc depends on libidn2 to implement modern DNS demangling.
    glibc fixes it by not depending on libidn at build time and does manual
    library loading with dlopen()/dlsym().
e) vast majority of others: dependency bundling (users of autotools, gnulib, zlib, lua).

All the above are not pretty. a) is simplest to maintain by ebuild maintainer
and gentoo user, but probably not by releng. c) moves the burden to user.

I personally would explore [e]: unconditional bundling of expat into
cmake and make sure it's kept up-to-date there.

[c] would be nice to be solved at portage level if portage could schedule
multiple merges for the same package with different USE flags.

-- 

  Sergei

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michael Orlitzky]

On 12/18/19 4:02 PM, Sebastian Pipping wrote:
> Hi all,
> 
> 
> I noticed that dev-util/cmake depends on dev-libs/expat and that
> libexpat upstream (where I'm involved) is in the process of
> dropping GNU Autotools altogether in favor of CMake in the near future,
> potentially the next release (without any known target release date).
>> ...
> 
> Do you have any ideas how to avoid a bad circular dependency issue for
> our users in the future?  Are you aware of similar problems and
> solutions from the past?

Keep autotools?

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

On Wed, 18 Dec 2019 23:58:22 +0000
Sergei Trofimovich <slyfox@gentoo.org> wrote:

> [c] would be nice to be solved at portage level if portage could schedule
> multiple merges for the same package with different USE flags.

Related bugs:

- Portage should be able to auto-flip USE="test" & FEATURES="test" 
  https://bugs.gentoo.org/697914#c0

- Portage needs an extension to package.use specification to provide a
  level of discretion to portage to flip flags on its own 
  https://bugs.gentoo.org/698920#c0


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2116 bytes --]

On Wed, 2019-12-18 at 23:58 +0000, Sergei Trofimovich wrote:
> On Wed, 18 Dec 2019 22:02:47 +0100
> Sebastian Pipping <sping@gentoo.org> wrote:
> 
> > Hi all,
> > 
> > 
> > I noticed that dev-util/cmake depends on dev-libs/expat and that
> > libexpat upstream (where I'm involved) is in the process of
> > dropping GNU Autotools altogether in favor of CMake in the near future,
> > potentially the next release (without any known target release date).
> > 
> > CMake bundles a (previously outdated and vulnerable) copy of expat so
> > I'm not sure if re-activating that bundle — say with a new use flag
> > "system-expat" — would be a good thing to resort to for breaking the
> > cycle, with regard to security in particular.
> > 
> > Do you have any ideas how to avoid a bad circular dependency issue for
> > our users in the future?  Are you aware of similar problems and
> > solutions from the past?
> 
> Some other distributions provide two packages to break the cycle.
> Example Gentoo solution would be: "cmake.ebuild" depends on "expat.ebuild",
> "expat.ebuild" depends on "cmake-with-bundled-expat.ebuild".
> 

I actually think this is the cleanest solution of all.  To be more
specific, create dev-util/cmake-bootstrap that either includes bundled
dependencies (let's not forget about jsoncpp here) and installs into
some dedicated prefix (e.g. /usr/lib/cmake-bootstrap).

Then you'd have expat and jsoncpp would BDEPEND:

  || ( dev-util/cmake-bootstrap dev-util/cmake )

and the ebuild would do something like, roughly:

  has_version -b dev-util/cmake ||
    local -x PATH=${BROOT}/usr/lib/cmake-bootstrap/bin:${PATH}

Since we don't need blockers there, Portage should be able to resolve
the depgraph peacefully and pull both packages in gracefully.  You
wouldn't have to do anything else in further revdeps.  The bootstrap
package would be safely isolated from the other revdeps, and it would be
depcleaned once other packages pull in regular cmake.

I can make a proof-of-concept based on jsoncpp if you like.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Rolf Eike Beer]

Am 2019-12-18 22:44, schrieb Francesco Riosa:
> Il giorno mer 18 dic 2019 alle ore 22:03 Sebastian Pipping 
> <sping@gentoo.org>
> ha scritto:
> 
>> 
>> CMake bundles a (previously outdated and vulnerable) copy of expat so
>> I'm not sure if re-activating that bundle — say with a new use flag
>> "system-expat" — would be a good thing to resort to for breaking the
>> cycle, with regard to security in particular.
>> 
> Pushing gently upstream to upgrade bundled expat copy would (at least
> temporarily) fix the issue and also benefit other use cases. Maybe they 
> are
> Gentoo friendly
> they also release quite often, which would fix the problem soon

This is in CMake 3.16.0:

commit 50bc359184472700e9776a0a9d6f7e06ea82b9ce
Author: Brad King <brad.king@kitware.com>
Date:   Mon Nov 11 10:44:17 2019 -0500

     expat: Update CMake build for 2.2.9

commit b63a5c88a2089494e53f22f83db1925435161934
Merge: 512fabaa9d 1712885b4f
Author: Brad King <brad.king@kitware.com>
Date:   Mon Nov 11 10:42:32 2019 -0500

     Merge branch 'upstream-expat' into update-expat

     * upstream-expat:
       expat 2019-09-25 (a7bc26b6)

These things _are_ updated regularly, but in case something is missed 
just file a bug at gitlab.kitware.com. All these bundled thing bumps are 
scripted as far as possible, so the actual overhead is quite small.

Eike

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Sebastian Pipping]

On 19.12.19 14:32, Rolf Eike Beer wrote:
> These things _are_ updated regularly

To be fair they update because I keep opening update requests:

https://gitlab.kitware.com/cmake/cmake/issues?scope=all&utf8=%E2%9C%93&state=closed&search=expat+update

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Sebastian Pipping]

Hey!


Thanks everyone for your thoughts so far!

From what I heard, these two options seem realistic to me:

A) Ask the KDE team for help with teaming up on a new package
   dev-util/cmake-bootstrap, keep it in sync with dev-util/cmake,
   make sure both packages co-exists with full disjoint operation,
   i.e. zero file conflicts + zero cross package file usage (tricky?).

B) Introduce USE flag "system-expat" to CMake similar to existing
   flag "system-jsoncpp", have it off by default, keep reminding
   CMake upstream to update their bundle

I favor (B) by more than just a bit.  Does anyone have strong concerns
against moving in the dev-util/cmake[-system-expat] (B) direction?  Is
it acceptable if I make those changes to the CMake ebuild myself?

Thanks again



Sebastian

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]

On Thu, 2019-12-19 at 15:39 +0100, Sebastian Pipping wrote:
> Hey!
> 
> 
> Thanks everyone for your thoughts so far!
> 
> From what I heard, these two options seem realistic to me:
> 
> A) Ask the KDE team for help with teaming up on a new package
>    dev-util/cmake-bootstrap, keep it in sync with dev-util/cmake,
>    make sure both packages co-exists with full disjoint operation,
>    i.e. zero file conflicts + zero cross package file usage (tricky?).
> 
> B) Introduce USE flag "system-expat" to CMake similar to existing
>    flag "system-jsoncpp", have it off by default, keep reminding
>    CMake upstream to update their bundle
> 
> I favor (B) by more than just a bit.  Does anyone have strong concerns
> against moving in the dev-util/cmake[-system-expat] (B) direction?  Is
> it acceptable if I make those changes to the CMake ebuild myself?
> 

It violates the policy on bundled libraries.  What's worse, the awful
USE flags solution means that most of the Gentoo devs end up using
bundled libraries just because people are manually required to figure
out what to do in order to disable them.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Sebastian Pipping]

Hey!


On 19.12.19 17:03, Michał Górny wrote:
>> B) Introduce USE flag "system-expat" to CMake similar to existing
>>    flag "system-jsoncpp", have it off by default, keep reminding
>>    CMake upstream to update their bundle
>>
>> [..]
> 
> It violates the policy on bundled libraries.

Same for the dev-util/cmake-bootstrap approach, right?


>  What's worse, the awful
> USE flags solution means that most of the Gentoo devs end up using
> bundled libraries just because people are manually required to figure
> out what to do in order to disable them.

I didn't say that it's perfect :)  It's the same approach that we have
have with the system-jsoncpp USE flag already so that was considered
good enough at some point in the past.  I guess we want the same for
Expat and jsoncpp?  Which alternative do you see as better than a new
flag system-expat?

Best



Sebastian

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

On Thu, 2019-12-19 at 18:28 +0100, Sebastian Pipping wrote:
> Hey!
> 
> 
> On 19.12.19 17:03, Michał Górny wrote:
> > > B) Introduce USE flag "system-expat" to CMake similar to existing
> > >    flag "system-jsoncpp", have it off by default, keep reminding
> > >    CMake upstream to update their bundle
> > > 
> > > [..]
> > 
> > It violates the policy on bundled libraries.
> 
> Same for the dev-util/cmake-bootstrap approach, right?
> 
> 
> >  What's worse, the awful
> > USE flags solution means that most of the Gentoo devs end up using
> > bundled libraries just because people are manually required to figure
> > out what to do in order to disable them.
> 
> I didn't say that it's perfect :)  It's the same approach that we have
> have with the system-jsoncpp USE flag already so that was considered
> good enough at some point in the past.  I guess we want the same for
> Expat and jsoncpp?  Which alternative do you see as better than a new
> flag system-expat?
> 

Just because someone did something crappy, it doesn't mean it was
considered 'good enough'.  It was just a cheap hack that someone once
did just to get it over with and stop caring.  Not a good solution we
should keep copying.

We have a better alternative that lets us limit the impact on the users.
Why not use it?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Sebastian Pipping]

On 19.12.19 18:37, Michał Górny wrote:
> We have a better alternative that lets us limit the impact on the users.
> Why not use it?

Which one?  The CMake bootstrap copy?  The adding to stage3 one?

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 489 bytes --]

On Thu, 2019-12-19 at 19:43 +0100, Sebastian Pipping wrote:
> On 19.12.19 18:37, Michał Górny wrote:
> > We have a better alternative that lets us limit the impact on the users.
> > Why not use it?
> 
> Which one?  The CMake bootstrap copy?  The adding to stage3 one?
> 

Bootstrap version.  It doesn't have to be a strict copy.  The whole
point is that even if it's buggy, unmaintained, vulnerable, it's impact
is going to be minimal.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Michael Orlitzky]

On 12/19/19 12:37 PM, Michał Górny wrote:
> 
> Just because someone did something crappy, it doesn't mean it was
> considered 'good enough'.  It was just a cheap hack that someone once
> did just to get it over with and stop caring.  Not a good solution we
> should keep copying.
> 

These should all be USE=bundled-foo, and off by default at the very least.

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Gerion Entrup]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

Am Donnerstag, 19. Dezember 2019, 19:43:37 CET schrieb Sebastian Pipping:
> On 19.12.19 18:37, Michał Górny wrote:
> > We have a better alternative that lets us limit the impact on the users.
> > Why not use it?
> 
> Which one?  The CMake bootstrap copy?  The adding to stage3 one?

Is it possible to show a specific error message when running in a
circular dependency?

When I get it right, the problem occurs only one time when solved
with a bundled use flag. As soon as expat and/or cmake are installed,
follow up versions can be merge without problems.

If an error message can be shown, maybe this is enough as a hint:
"expat and cmake have circular dependencies. Emerge it the first time with:
USE=bundled-cmake emerge -1 cmake expat
and then just don't care anymore about this use flag."

Of course, the same applies for other known circular dependencies, too.
At least for me as a user, this would be enough.

Best,
Gerion

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Rich Freeman]

On Fri, Dec 20, 2019 at 8:41 AM Gerion Entrup <gerion.entrup@flump.de> wrote:
>
> Am Donnerstag, 19. Dezember 2019, 19:43:37 CET schrieb Sebastian Pipping:
> > On 19.12.19 18:37, Michał Górny wrote:
> > > We have a better alternative that lets us limit the impact on the users.
> > > Why not use it?
> >
> > Which one?  The CMake bootstrap copy?  The adding to stage3 one?
>
> If an error message can be shown, maybe this is enough as a hint:
> "expat and cmake have circular dependencies. Emerge it the first time with:
> USE=bundled-cmake emerge -1 cmake expat
> and then just don't care anymore about this use flag."
>

Not sure about custom error messages but in any case when using that
bundled-cmake USE flag it would probably make sense to do an ewarn
that the bundling took place and is only intended for temporary use,
and that the package should be re-merged without the flag once expat
is working.  That would also cover users who inadvertently set the
flag.

I think that bundled-cmake also might make more sense than
system-cmake, even though the latter is more established.  We have a
lot of users who set USE=-* and that means they're going to end up
with lots of stuff bundled that doesn't need to be.  I'm no fan of
USE=-* in general but it seems like we should try to avoid making not
setting a USE flag the less secure option since it does happen a lot.

I'm just commenting on the USE-based solution.  The compat package
solution obviously bypasses this particular problem entirely.

-- 
Rich