[gentoo-dev] Integrating Portage with other package managers

[gentoo-dev] Integrating Portage with other package managers

[anoteros]

Having used Gentoo for a few years now, one thing that has been annoying
to me is the tremendous duplication of effort and uphill battle of
creating ebuilds (build recipes) for language-specific packages that
already have their own build systems.

For example, many languages such as Python (pip), Node (npm), Ruby
(gems), TeXLive (tlmgr), Haskell (cabal), Rust (cargo) have ways of
redistributing up-to-date dependencies and ensuring that they build
properly in most environments. Portage, it seems, does not take
advantage of this at all. If I want to install the package 'foo', which
has 'bar' as a dependency (which has been installed through cabal, for
instance). Why should portage download some outdated second copy of the
sources for 'bar', rebuild it, and scatter it around the file system
where it cannot be used by other programs installed by cabal?

It seems reasonable to me to 'hook' portage into these other package
managers, so that running 'emerge bar' would actually run 'cabal install
bar' rather than downloading sources and running 'ghc'. This would make
managing dependencies much easier as versions change, since most (all?)
these alternate package managers have ways of specifying specific
versions of packages or pinning. It would also put an end to the
breakage caused by, for example, running 'pip' as root and breaking all
the python libraries portage has pulled in.

I also notice that there is a GSoC 2018 project for integrating Rust
more closely within Gentoo. It seems to me that allowing ebuilds to be
written like this would help resolve this not only for Rust, but for all
languages with similar build systems (which is most modern languages,
frankly).

The only real issue I see with this is the potential loss of granularity
with USE flags for some of these packages, but having poked around the
sort of language-specific libraries and bindings available, hardly any
have any USE flags available, and those that do only offer debug symbols
or documentation, which many of these other build systems offer as well,
so I doubt this would be an issue for anyone. I could of course be wrong.

Is this a feature/improvement other Gentoo users/developers would be
interested in? If so, I would love to help discuss and potentially help
with its implementation.

Re: [gentoo-dev] Integrating Portage with other package managers

[Michael Orlitzky]

On 03/07/2018 11:06 AM, anoteros@teknik.io wrote:
> Why should portage download some outdated second copy of the
> sources for 'bar', rebuild it, and scatter it around the file system
> where it cannot be used by other programs installed by cabal?

These other package managers don't solve any hard problems -- they're
basically a fancy interface around wget and "git clone." Portage on the
other hand has ~20 years of good ideas and hard work on the hard
problems in package management. For example...


> It seems reasonable to me to 'hook' portage into these other package
> managers, so that running 'emerge bar' would actually run 'cabal install
> bar'

Can "cabal install" build or even identify the C libraries that your
Haskell package needs? No, because nobody ever thought of that, and it
seems kind of hard now that the cabal build system has no ability to
build non-Haskell packages, so no one is ever going to work on it.

Can "cabal install" rebuild your Haskell packages when the ABI of some
library changes? No, because "cabal install" doesn't have any idea
what's installed on your system.

Can "cabal install" uninstall a package? Nope, it has no idea what was
done during the installation, and thus no idea what to undo.

Can "cabal install" verify the integrity of your downloaded source code?
No, because by design it fetches and runs code from complete strangers.

Can "cabal install" use a local tarball to function without network
access? Etc. We're dead in the water.

Every other language-specific package manager has the same problems,
because they're all written by people who didn't know anything about
package management and then got bored when they realized that there's
more to it than parsing a json file of dependencies.

If you want to eliminate the duplication of effort, tell these people to
use Gentoo Prefix instead of writing the N+1st crippled PM. Doing it the
other way around won't work because we'd be replacing one good thing
with 75 shitty things.

Re: [gentoo-dev] Integrating Portage with other package managers

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 3747 bytes --]

On Wed, Mar 7, 2018 at 11:51 AM, Michael Orlitzky <mjo@gentoo.org> wrote:

> On 03/07/2018 11:06 AM, anoteros@teknik.io wrote:
> > Why should portage download some outdated second copy of the
> > sources for 'bar', rebuild it, and scatter it around the file system
> > where it cannot be used by other programs installed by cabal?
>
>
I'm really not happy with the tone of this email, so I'm going to comment
on it a bit.


> These other package managers don't solve any hard problems -- they're
> basically a fancy interface around wget and "git clone." Portage on the
> other hand has ~20 years of good ideas and hard work on the hard
> problems in package management. For example...
>

Portage is also full of not-good ideas; many of these we papered over with
PMS and EAPI to make
the actual API people use less horrific. Lets not preach from our ivory
tower here.


>
>
> > It seems reasonable to me to 'hook' portage into these other package
> > managers, so that running 'emerge bar' would actually run 'cabal install
> > bar'
>
> Can "cabal install" build or even identify the C libraries that your
> Haskell package needs? No, because nobody ever thought of that, and it
> seems kind of hard now that the cabal build system has no ability to
> build non-Haskell packages, so no one is ever going to work on it.
>
> Can "cabal install" rebuild your Haskell packages when the ABI of some
> library changes? No, because "cabal install" doesn't have any idea
> what's installed on your system.
>
> Can "cabal install" uninstall a package? Nope, it has no idea what was
> done during the installation, and thus no idea what to undo.
>
> Can "cabal install" verify the integrity of your downloaded source code?
> No, because by design it fetches and runs code from complete strangers.
>
> Can "cabal install" use a local tarball to function without network
> access? Etc. We're dead in the water.
>
> Every other language-specific package manager has the same problems,
> because they're all written by people who didn't know anything about
> package management and then got bored when they realized that there's
> more to it than parsing a json file of dependencies.
>
>
They are written by people who are not you, who have different problems
than you and often don't care about the above use cases.
It turns out this stuff exists because:

1) Upstream wants to push 1 single thing and have it work in all distros.
2) pip / virtualenv / cargo / whatever work reasonable well.
3) Rolling-based distros couldn't keep up with packaging.
4) Snapshot-based distros (debian stable / ubuntu) were not designed with
this in mind as much; because packages were developed with a high velocity.




> If you want to eliminate the duplication of effort, tell these people to
> use Gentoo Prefix instead of writing the N+1st crippled PM. Doing it the
> other way around won't work because we'd be replacing one good thing
> with 75 shitty things.


I agree that in theory they could have published ebuilds for Gentoo prefix
and it would have 'worked everywhere' but I think that boat sailed about 10
years ago.

https://wiki.gentoo.org/wiki/Project:Perl/g-cpan is a project is in a
similar space and basically reads perl CPAN metadata to generate stub
ebuilds.
Portage tracks these stub ebuilds (and so for example, it tracks what these
cpan packages install and can remove them afterwards.)

I think this is the most pragmatic approach I've seen used as its mostly an
adapter (to cpan) that just generates ebuilds.
Its plausible that with some careful eclass magic you might be able to make
the installed packages compatible with pip, cargo, etc.

I think its more of a struggle to make it compatible with things like
virtualenv or pip --user though.

-A

[-- Attachment #2: Type: text/html, Size: 5768 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[R0b0t1]

On Wed, Mar 7, 2018 at 11:52 AM, Alec Warner <antarus@gentoo.org> wrote:
> On Wed, Mar 7, 2018 at 11:51 AM, Michael Orlitzky <mjo@gentoo.org> wrote:
>>
>> On 03/07/2018 11:06 AM, anoteros@teknik.io wrote:
>> > Why should portage download some outdated second copy of the
>> > sources for 'bar', rebuild it, and scatter it around the file system
>> > where it cannot be used by other programs installed by cabal?
>>
>
> I'm really not happy with the tone of this email, so I'm going to comment on
> it a bit.
>

I can't help but agree with Mr. Orlitzky's sentiment. All language
package managers suffer from the same sophomoric problems:

1) I usually don't know where things are downloaded from.
2) I can't integrate these changes with my distrbution (Gentoo,
Ubuntu, Debian, Fedora, CentOS) safely without serious work.
3) I can't figure out easily what dependencies a package has. Usually
I see if there are compile or runtime errors. Sometimes the
dependencies are listed somewhere. If the dependency is not what is
currently in e.g. Ubuntu's repository, I may have to maintain separate
versions to be compatible.
4) Sometimes they aren't set up to be built at all. Let the magic
package manager do everything for you. This works, except when your
shared objects are not in the right places. (But it makes me feel
dirty.)

>>
>> These other package managers don't solve any hard problems -- they're
>> basically a fancy interface around wget and "git clone." Portage on the
>> other hand has ~20 years of good ideas and hard work on the hard
>> problems in package management. For example...
>
>
> Portage is also full of not-good ideas; many of these we papered over with
> PMS and EAPI to make
> the actual API people use less horrific. Lets not preach from our ivory
> tower here.
>

The magnitude of "not good" is, I would suggest, very different.

Cabal is a pretty hilarious example. Have you ever tried to build it
without using the release binaries? I suppose this is a second problem
though, where people want to be "self-reliant" and instead just end up
making things impossible to verify or make reproducible.

For the longest time Cabal did not authenticate or verify the code it
would run (as root). Very recently this was fixed, but I still feel
bad any time I let it run, even if it's on a separate development
system.

>>
>> > It seems reasonable to me to 'hook' portage into these other package
>> > managers, so that running 'emerge bar' would actually run 'cabal install
>> > bar'
>>
>> Can "cabal install" build or even identify the C libraries that your
>> Haskell package needs? No, because nobody ever thought of that, and it
>> seems kind of hard now that the cabal build system has no ability to
>> build non-Haskell packages, so no one is ever going to work on it.
>>
>> Can "cabal install" rebuild your Haskell packages when the ABI of some
>> library changes? No, because "cabal install" doesn't have any idea
>> what's installed on your system.
>>
>> Can "cabal install" uninstall a package? Nope, it has no idea what was
>> done during the installation, and thus no idea what to undo.
>>
>> Can "cabal install" verify the integrity of your downloaded source code?
>> No, because by design it fetches and runs code from complete strangers.
>>
>> Can "cabal install" use a local tarball to function without network
>> access? Etc. We're dead in the water.
>>
>> Every other language-specific package manager has the same problems,
>> because they're all written by people who didn't know anything about
>> package management and then got bored when they realized that there's
>> more to it than parsing a json file of dependencies.
>>
>
> They are written by people who are not you, who have different problems than
> you and often don't care about the above use cases.
> It turns out this stuff exists because:
>
> 1) Upstream wants to push 1 single thing and have it work in all distros.
> 2) pip / virtualenv / cargo / whatever work reasonable well.
> 3) Rolling-based distros couldn't keep up with packaging.
> 4) Snapshot-based distros (debian stable / ubuntu) were not designed with
> this in mind as much; because packages were developed with a high velocity.
>

1) What do you mean by this? Distributions are usually not binary
compatible. This "works" by having each distribution customize and
build a project by hand.
2) Virtualenv works well, and cabal now has a local installation
option. Still, these are not perfect.
3) Use a git ebuild, or target stable versions. If a project has so
much churn that I can't keep up with it I will find something more
stable.
4) True, but prefix would be a lot better at fixing that problem. If
not that, something like virtualenv.

In the case of either #3 or #4, the distribution developers prefer you
use their package manager to install packages. You are only safe doing
anything else in an environment like virtualenv, which does not exist
for the vast majority of languages. This is why developers will pass
around VM images, or devote an entire VM to development on a project.
Languages and their packages are not designed to be compartmentalized
and will trash your installation.

There is also (going along with #4):
5) The software may need to work on Windows. They could have used
Cygwin or MSYS2. Anything else is an exercise in futility. People have
come before you and done it better.

Taken together these issues boil down to "the people rewriting package
managers don't realize the problems they actually have" which agrees
with what was said. As above, the typical workflows of these people
involve lots of VMs or containers. Ask yourself "why?"

>
>>
>> If you want to eliminate the duplication of effort, tell these people to
>> use Gentoo Prefix instead of writing the N+1st crippled PM. Doing it the
>> other way around won't work because we'd be replacing one good thing
>> with 75 shitty things.
>
>
> I agree that in theory they could have published ebuilds for Gentoo prefix
> and it would have 'worked everywhere' but I think that boat sailed about 10
> years ago.
>
> https://wiki.gentoo.org/wiki/Project:Perl/g-cpan is a project is in a
> similar space and basically reads perl CPAN metadata to generate stub
> ebuilds.
> Portage tracks these stub ebuilds (and so for example, it tracks what these
> cpan packages install and can remove them afterwards.)
>
> I think this is the most pragmatic approach I've seen used as its mostly an
> adapter (to cpan) that just generates ebuilds.
> Its plausible that with some careful eclass magic you might be able to make
> the installed packages compatible with pip, cargo, etc.
>
> I think its more of a struggle to make it compatible with things like
> virtualenv or pip --user though.
>

This might be a good way to relieve the amount of intervention
required when repackaging code for an actual package manager. The
information should be there. The other option is convincing people to
package for multiple systems at once, which diffuses the effort to the
point people tend to not mind.

1) Language package manager (usually used by Windows consumers).
2) .debs for Ubuntu/Debian.
3) .rpms for Fedora/CentOS.
4) Sometimes there's a Gentoo or Arch release.

If you could sell #4 as a way to generate 1-3 it would likely be
possible to reduce the proliferation of language specific package
managers over time. Prefix would likely play an important role.

Cheers,
     R0b0t1

Re: [gentoo-dev] Integrating Portage with other package managers

[Rich Freeman]

On Wed, Mar 7, 2018 at 12:52 PM, Alec Warner <antarus@gentoo.org> wrote:
>
> https://wiki.gentoo.org/wiki/Project:Perl/g-cpan is a project is in a
> similar space and basically reads perl CPAN metadata to generate stub
> ebuilds.
> Portage tracks these stub ebuilds (and so for example, it tracks what these
> cpan packages install and can remove them afterwards.)
>

This is the right general approach.  If somebody were willing to do
the work I'm sure it would be useful if portage had a more generic
interface for stuff like this, such as a way to do plugins/etc.  The
idea would be to run some outside package manager in some kind of
sandbox, then create a binary package from what gets installed.

Everybody I know has these sorts of complaints about language-based
PMs, whether they prefer Ubuntu, or Debian, or CentOS, or whatever.
Nobody wants random programs downloading random stuff and dropping
orphan files all over their filesystem with no way to identify these
or clean them up.  They're usually written by the same sorts of people
who tell you to pipe curl into sudo bash...

-- 
Rich

Re: [gentoo-dev] Integrating Portage with other package managers

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 8803 bytes --]

On Wed, Mar 7, 2018 at 1:22 PM, R0b0t1 <r030t1@gmail.com> wrote:

> On Wed, Mar 7, 2018 at 11:52 AM, Alec Warner <antarus@gentoo.org> wrote:
> > On Wed, Mar 7, 2018 at 11:51 AM, Michael Orlitzky <mjo@gentoo.org>
> wrote:
> >>
> >> On 03/07/2018 11:06 AM, anoteros@teknik.io wrote:
> >> > Why should portage download some outdated second copy of the
> >> > sources for 'bar', rebuild it, and scatter it around the file system
> >> > where it cannot be used by other programs installed by cabal?
> >>
> >
> > I'm really not happy with the tone of this email, so I'm going to
> comment on
> > it a bit.
> >
>
> I can't help but agree with Mr. Orlitzky's sentiment. All language
> package managers suffer from the same sophomoric problems:
>

> 1) I usually don't know where things are downloaded from.
> 2) I can't integrate these changes with my distrbution (Gentoo,
> Ubuntu, Debian, Fedora, CentOS) safely without serious work.
> 3) I can't figure out easily what dependencies a package has. Usually
> I see if there are compile or runtime errors. Sometimes the
> dependencies are listed somewhere. If the dependency is not what is
> currently in e.g. Ubuntu's repository, I may have to maintain separate
> versions to be compatible.
> 4) Sometimes they aren't set up to be built at all. Let the magic
> package manager do everything for you. This works, except when your
> shared objects are not in the right places. (But it makes me feel
> dirty.)
>
> >>
> >> These other package managers don't solve any hard problems -- they're
> >> basically a fancy interface around wget and "git clone." Portage on the
> >> other hand has ~20 years of good ideas and hard work on the hard
> >> problems in package management. For example...
> >
> >
> > Portage is also full of not-good ideas; many of these we papered over
> with
> > PMS and EAPI to make
> > the actual API people use less horrific. Lets not preach from our ivory
> > tower here.
> >
>
> The magnitude of "not good" is, I would suggest, very different.
>
> Cabal is a pretty hilarious example. Have you ever tried to build it
> without using the release binaries? I suppose this is a second problem
> though, where people want to be "self-reliant" and instead just end up
> making things impossible to verify or make reproducible.
>
> For the longest time Cabal did not authenticate or verify the code it
> would run (as root). Very recently this was fixed, but I still feel
> bad any time I let it run, even if it's on a separate development
> system.
>

Gentoo just got signature checking enabled by default...in ~arch? I'm not
sure if that version of portage is stable yet.
Like I said, be careful how one preaches from the ivory tower.


>
> >>
> >> > It seems reasonable to me to 'hook' portage into these other package
> >> > managers, so that running 'emerge bar' would actually run 'cabal
> install
> >> > bar'
> >>
> >> Can "cabal install" build or even identify the C libraries that your
> >> Haskell package needs? No, because nobody ever thought of that, and it
> >> seems kind of hard now that the cabal build system has no ability to
> >> build non-Haskell packages, so no one is ever going to work on it.
> >>
> >> Can "cabal install" rebuild your Haskell packages when the ABI of some
> >> library changes? No, because "cabal install" doesn't have any idea
> >> what's installed on your system.
> >>
> >> Can "cabal install" uninstall a package? Nope, it has no idea what was
> >> done during the installation, and thus no idea what to undo.
> >>
> >> Can "cabal install" verify the integrity of your downloaded source code?
> >> No, because by design it fetches and runs code from complete strangers.
> >>
> >> Can "cabal install" use a local tarball to function without network
> >> access? Etc. We're dead in the water.
> >>
> >> Every other language-specific package manager has the same problems,
> >> because they're all written by people who didn't know anything about
> >> package management and then got bored when they realized that there's
> >> more to it than parsing a json file of dependencies.
> >>
> >
> > They are written by people who are not you, who have different problems
> than
> > you and often don't care about the above use cases.
> > It turns out this stuff exists because:
> >
> > 1) Upstream wants to push 1 single thing and have it work in all distros.
> > 2) pip / virtualenv / cargo / whatever work reasonable well.
> > 3) Rolling-based distros couldn't keep up with packaging.
> > 4) Snapshot-based distros (debian stable / ubuntu) were not designed with
> > this in mind as much; because packages were developed with a high
> velocity.
> >
>
>
This == upstream has a git tag (release X) and they sync the tag into pip
and they tell people to pip install X and their job is done.
Software, distributed.


> 1) What do you mean by this? Distributions are usually not binary
> compatible. This "works" by having each distribution customize and
> build a project by hand.
>

Right, and because "distributions" move a different speeds (Ubuntu, Debian,
Gentoo) and have different branches (testing, stable, unstable, ~arch,
pmask)
this is all even *more* overhead for developers to worry about. So they use
pip, and they don't (worry about it.)



> 2) Virtualenv works well, and cabal now has a local installation
> option. Still, these are not perfect.
> 3) Use a git ebuild, or target stable versions. If a project has so
> much churn that I can't keep up with it I will find something more
> stable.
>

I don't think Ubuntu ships git-ebuilds (because it doesn't have ebuilds) so
I'm stuck consuming some PPA (either one that upstream builds, or some
shady PPA run by who knows who that build my binaries.)

Easier to pip install!


> 4) True, but prefix would be a lot better at fixing that problem. If
> not that, something like virtualenv.
>
> In the case of either #3 or #4, the distribution developers prefer you
> use their package manager to install packages. You are only safe doing
> anything else in an environment like virtualenv, which does not exist
> for the vast majority of languages. This is why developers will pass
> around VM images, or devote an entire VM to development on a project.
> Languages and their packages are not designed to be compartmentalized
> and will trash your installation.


Containers are certainly awesome, yep.


> There is also (going along with #4):
> 5) The software may need to work on Windows. They could have used
> Cygwin or MSYS2. Anything else is an exercise in futility. People have
> come before you and done it better.
>

> Taken together these issues boil down to "the people rewriting package
> managers don't realize the problems they actually have" which agrees
> with what was said. As above, the typical workflows of these people
> involve lots of VMs or containers. Ask yourself "why?"
>

Because containers are awesome and are way easier to use.

-A


>
> >
> >>
> >> If you want to eliminate the duplication of effort, tell these people to
> >> use Gentoo Prefix instead of writing the N+1st crippled PM. Doing it the
> >> other way around won't work because we'd be replacing one good thing
> >> with 75 shitty things.
> >
> >
> > I agree that in theory they could have published ebuilds for Gentoo
> prefix
> > and it would have 'worked everywhere' but I think that boat sailed about
> 10
> > years ago.
> >
> > https://wiki.gentoo.org/wiki/Project:Perl/g-cpan is a project is in a
> > similar space and basically reads perl CPAN metadata to generate stub
> > ebuilds.
> > Portage tracks these stub ebuilds (and so for example, it tracks what
> these
> > cpan packages install and can remove them afterwards.)
> >
> > I think this is the most pragmatic approach I've seen used as its mostly
> an
> > adapter (to cpan) that just generates ebuilds.
> > Its plausible that with some careful eclass magic you might be able to
> make
> > the installed packages compatible with pip, cargo, etc.
> >
> > I think its more of a struggle to make it compatible with things like
> > virtualenv or pip --user though.
> >
>
> This might be a good way to relieve the amount of intervention
> required when repackaging code for an actual package manager. The
> information should be there. The other option is convincing people to
> package for multiple systems at once, which diffuses the effort to the
> point people tend to not mind.
>
> 1) Language package manager (usually used by Windows consumers).
> 2) .debs for Ubuntu/Debian.
> 3) .rpms for Fedora/CentOS.
> 4) Sometimes there's a Gentoo or Arch release.
>
> If you could sell #4 as a way to generate 1-3 it would likely be
> possible to reduce the proliferation of language specific package
> managers over time. Prefix would likely play an important role.
>
> Cheers,
>      R0b0t1
>
>

[-- Attachment #2: Type: text/html, Size: 11879 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[Michael Orlitzky]

On 03/07/2018 12:52 PM, Alec Warner wrote:
> 
> I'm really not happy with the tone of this email, so I'm going to
> comment on it a bit.
>  

Ok, it would have benefited from a do-I-sound-like-a-dick proofread. I
don't want to sound discouraging because this is an area with lots of
room for improvement. A better conclusion:

Ultimately, people want to integrate the various PMs with portage
because portage is pretty good at keeping your system reliable,
up-to-date, and secure. The language-specific PMs on the other hand only
care about ease of use and how fast they can get bleeding-edge releases
to you. Having both would be ideal, but if we simply shell out to the
language-specific PM, then that would sidestep the good parts of portage
making the integration pointless. The time and attention involved in
ebuild packaging turn out to be critical parts of the product.

Re: [gentoo-dev] Integrating Portage with other package managers

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On Wed, Mar 7, 2018 at 3:12 PM, Michael Orlitzky <mjo@gentoo.org> wrote:

> On 03/07/2018 12:52 PM, Alec Warner wrote:
> >
> > I'm really not happy with the tone of this email, so I'm going to
> > comment on it a bit.
> >
>
> Ok, it would have benefited from a do-I-sound-like-a-dick proofread. I
> don't want to sound discouraging because this is an area with lots of
> room for improvement. A better conclusion:
>
> Ultimately, people want to integrate the various PMs with portage
> because portage is pretty good at keeping your system reliable,
> up-to-date, and secure. The language-specific PMs on the other hand only
> care about ease of use and how fast they can get bleeding-edge releases
> to you. Having both would be ideal, but if we simply shell out to the
> language-specific PM, then that would sidestep the good parts of portage
> making the integration pointless. The time and attention involved in
> ebuild packaging turn out to be critical parts of the product.
>
>
I really appreciate this reply. I also think that portage provides a lot of
value (particularly for complex projects that are perhaps not
so well suited for use with the less featureful tooling.)

-A

[-- Attachment #2: Type: text/html, Size: 1778 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[R0b0t1]

On Wed, Mar 7, 2018 at 1:15 PM, Alec Warner <antarus@gentoo.org> wrote:
>
>
> On Wed, Mar 7, 2018 at 1:22 PM, R0b0t1 <r030t1@gmail.com> wrote:
>>
>> On Wed, Mar 7, 2018 at 11:52 AM, Alec Warner <antarus@gentoo.org> wrote:
>> > On Wed, Mar 7, 2018 at 11:51 AM, Michael Orlitzky <mjo@gentoo.org>
>> > wrote:
>> >>
>> >> On 03/07/2018 11:06 AM, anoteros@teknik.io wrote:
>> >> > Why should portage download some outdated second copy of the
>> >> > sources for 'bar', rebuild it, and scatter it around the file system
>> >> > where it cannot be used by other programs installed by cabal?
>> >>
>> >
>> > I'm really not happy with the tone of this email, so I'm going to
>> > comment on
>> > it a bit.
>> >
>>
>> I can't help but agree with Mr. Orlitzky's sentiment. All language
>> package managers suffer from the same sophomoric problems:
>>
>>
>> 1) I usually don't know where things are downloaded from.
>> 2) I can't integrate these changes with my distrbution (Gentoo,
>> Ubuntu, Debian, Fedora, CentOS) safely without serious work.
>> 3) I can't figure out easily what dependencies a package has. Usually
>> I see if there are compile or runtime errors. Sometimes the
>> dependencies are listed somewhere. If the dependency is not what is
>> currently in e.g. Ubuntu's repository, I may have to maintain separate
>> versions to be compatible.
>> 4) Sometimes they aren't set up to be built at all. Let the magic
>> package manager do everything for you. This works, except when your
>> shared objects are not in the right places. (But it makes me feel
>> dirty.)
>>
>> >>
>> >> These other package managers don't solve any hard problems -- they're
>> >> basically a fancy interface around wget and "git clone." Portage on the
>> >> other hand has ~20 years of good ideas and hard work on the hard
>> >> problems in package management. For example...
>> >
>> >
>> > Portage is also full of not-good ideas; many of these we papered over
>> > with
>> > PMS and EAPI to make
>> > the actual API people use less horrific. Lets not preach from our ivory
>> > tower here.
>> >
>>
>> The magnitude of "not good" is, I would suggest, very different.
>>
>> Cabal is a pretty hilarious example. Have you ever tried to build it
>> without using the release binaries? I suppose this is a second problem
>> though, where people want to be "self-reliant" and instead just end up
>> making things impossible to verify or make reproducible.
>>
>> For the longest time Cabal did not authenticate or verify the code it
>> would run (as root). Very recently this was fixed, but I still feel
>> bad any time I let it run, even if it's on a separate development
>> system.
>
>
> Gentoo just got signature checking enabled by default...in ~arch? I'm not
> sure if that version of portage is stable yet.
> Like I said, be careful how one preaches from the ivory tower.
>

webrsync-gpg has been an option for a long time.

Package maintenance for Gentoo is not the ivory tower. If anything,
the ivory tower is language development, in that tower the language
developers are isolated from the wider reaching consequences of their
actions.

>>
>>
>> >>
>> >> > It seems reasonable to me to 'hook' portage into these other package
>> >> > managers, so that running 'emerge bar' would actually run 'cabal
>> >> > install
>> >> > bar'
>> >>
>> >> Can "cabal install" build or even identify the C libraries that your
>> >> Haskell package needs? No, because nobody ever thought of that, and it
>> >> seems kind of hard now that the cabal build system has no ability to
>> >> build non-Haskell packages, so no one is ever going to work on it.
>> >>
>> >> Can "cabal install" rebuild your Haskell packages when the ABI of some
>> >> library changes? No, because "cabal install" doesn't have any idea
>> >> what's installed on your system.
>> >>
>> >> Can "cabal install" uninstall a package? Nope, it has no idea what was
>> >> done during the installation, and thus no idea what to undo.
>> >>
>> >> Can "cabal install" verify the integrity of your downloaded source
>> >> code?
>> >> No, because by design it fetches and runs code from complete strangers.
>> >>
>> >> Can "cabal install" use a local tarball to function without network
>> >> access? Etc. We're dead in the water.
>> >>
>> >> Every other language-specific package manager has the same problems,
>> >> because they're all written by people who didn't know anything about
>> >> package management and then got bored when they realized that there's
>> >> more to it than parsing a json file of dependencies.
>> >>
>> >
>> > They are written by people who are not you, who have different problems
>> > than
>> > you and often don't care about the above use cases.
>> > It turns out this stuff exists because:
>> >
>> > 1) Upstream wants to push 1 single thing and have it work in all
>> > distros.
>> > 2) pip / virtualenv / cargo / whatever work reasonable well.
>> > 3) Rolling-based distros couldn't keep up with packaging.
>> > 4) Snapshot-based distros (debian stable / ubuntu) were not designed
>> > with
>> > this in mind as much; because packages were developed with a high
>> > velocity.
>> >
>>
>
> This == upstream has a git tag (release X) and they sync the tag into pip
> and they tell people to pip install X and their job is done.
> Software, distributed.
>
>>
>> 1) What do you mean by this? Distributions are usually not binary
>> compatible. This "works" by having each distribution customize and
>> build a project by hand.
>
>
> Right, and because "distributions" move a different speeds (Ubuntu, Debian,
> Gentoo) and have different branches (testing, stable, unstable, ~arch,
> pmask)
> this is all even *more* overhead for developers to worry about. So they use
> pip, and they don't (worry about it.)
>

At some level pip "works" but the solution to more overhead seems to
have been more overhead.

>
>>
>> 2) Virtualenv works well, and cabal now has a local installation
>> option. Still, these are not perfect.
>> 3) Use a git ebuild, or target stable versions. If a project has so
>> much churn that I can't keep up with it I will find something more
>> stable.
>
>
> I don't think Ubuntu ships git-ebuilds (because it doesn't have ebuilds) so
> I'm stuck consuming some PPA (either one that upstream builds, or some shady
> PPA run by who knows who that build my binaries.)
>
> Easier to pip install!
>

Pip may not work, or you may need conflicting versions of something
installed. It doesn't solve these issues.

>>
>> 4) True, but prefix would be a lot better at fixing that problem. If
>> not that, something like virtualenv.
>>
>> In the case of either #3 or #4, the distribution developers prefer you
>> use their package manager to install packages. You are only safe doing
>> anything else in an environment like virtualenv, which does not exist
>> for the vast majority of languages. This is why developers will pass
>> around VM images, or devote an entire VM to development on a project.
>> Languages and their packages are not designed to be compartmentalized
>> and will trash your installation.
>
>
> Containers are certainly awesome, yep.
>
>>
>> There is also (going along with #4):
>> 5) The software may need to work on Windows. They could have used
>> Cygwin or MSYS2. Anything else is an exercise in futility. People have
>> come before you and done it better.
>>
>>
>> Taken together these issues boil down to "the people rewriting package
>> managers don't realize the problems they actually have" which agrees
>> with what was said. As above, the typical workflows of these people
>> involve lots of VMs or containers. Ask yourself "why?"
>
>
> Because containers are awesome and are way easier to use.
>
> -A
>

I think you missed my point: Why are they easier to use?

Cheers,
     R0b0t1

>>
>>
>> >
>> >>
>> >> If you want to eliminate the duplication of effort, tell these people
>> >> to
>> >> use Gentoo Prefix instead of writing the N+1st crippled PM. Doing it
>> >> the
>> >> other way around won't work because we'd be replacing one good thing
>> >> with 75 shitty things.
>> >
>> >
>> > I agree that in theory they could have published ebuilds for Gentoo
>> > prefix
>> > and it would have 'worked everywhere' but I think that boat sailed about
>> > 10
>> > years ago.
>> >
>> > https://wiki.gentoo.org/wiki/Project:Perl/g-cpan is a project is in a
>> > similar space and basically reads perl CPAN metadata to generate stub
>> > ebuilds.
>> > Portage tracks these stub ebuilds (and so for example, it tracks what
>> > these
>> > cpan packages install and can remove them afterwards.)
>> >
>> > I think this is the most pragmatic approach I've seen used as its mostly
>> > an
>> > adapter (to cpan) that just generates ebuilds.
>> > Its plausible that with some careful eclass magic you might be able to
>> > make
>> > the installed packages compatible with pip, cargo, etc.
>> >
>> > I think its more of a struggle to make it compatible with things like
>> > virtualenv or pip --user though.
>> >
>>
>> This might be a good way to relieve the amount of intervention
>> required when repackaging code for an actual package manager. The
>> information should be there. The other option is convincing people to
>> package for multiple systems at once, which diffuses the effort to the
>> point people tend to not mind.
>>
>> 1) Language package manager (usually used by Windows consumers).
>> 2) .debs for Ubuntu/Debian.
>> 3) .rpms for Fedora/CentOS.
>> 4) Sometimes there's a Gentoo or Arch release.
>>
>> If you could sell #4 as a way to generate 1-3 it would likely be
>> possible to reduce the proliferation of language specific package
>> managers over time. Prefix would likely play an important role.
>>
>> Cheers,
>>      R0b0t1
>>
>

Re: [gentoo-dev] Integrating Portage with other package managers

[Rich Freeman]

On Wed, Mar 7, 2018 at 3:55 PM, R0b0t1 <r030t1@gmail.com> wrote:
> On Wed, Mar 7, 2018 at 1:15 PM, Alec Warner <antarus@gentoo.org> wrote:
>>
>> Because containers are awesome and are way easier to use.
>>
>
> I think you missed my point: Why are they easier to use?
>

I suspect that he was equating "containers" with "Docker," which both
runs containers and also is an image management solution (one that I
don't particularly like, but probably because I don't have heavy needs
so I find myself fighting it more than anything else - it doesn't hurt
that for the life of me I can't get it to connect containers to my
bridge, and it seems like it is completely opposed to just using DHCP
to obtain IPs).

But, if you're using Docker, sure, you can run whatever the command is
to download an image of a container with some software pre-installed
and run it.

If you're using something other than Docker to manage your containers
then you still have to get the software you want to use installed in
your container image.

-- 
Rich

Re: [gentoo-dev] Integrating Portage with other package managers

[Benda Xu]

Dear Rich,

Rich Freeman <rich0@gentoo.org> writes:

> Everybody I know has these sorts of complaints about language-based
> PMs, whether they prefer Ubuntu, or Debian, or CentOS, or whatever.
> Nobody wants random programs downloading random stuff and dropping
> orphan files all over their filesystem with no way to identify these
> or clean them up.  They're usually written by the same sorts of people
> who tell you to pipe curl into sudo bash...

One observation.  There are a lot of people on this planet who will
format their disk and reinstall their whatever operating systems upon
getting fscked by random programs downloading random stuff and dropping
orphan files all over their filesystem.  So far the volume of this group
of people is growing fast.

That give an illusion of tolerance from the users, enouraging this evil
to go deeper.

Luckily in Gentoo, we are with competent friends who are taking care of
their own systems.

Cheers,
Benda

Re: [gentoo-dev] Integrating Portage with other package managers

[Benda Xu]

Hi Folks,

Michael Orlitzky <mjo@gentoo.org> writes:

> These other package managers don't solve any hard problems -- they're
> basically a fancy interface around wget and "git clone." Portage on the
> other hand has ~20 years of good ideas and hard work on the hard
> problems in package management. For example...

I just want to point out one very nice article by Michael thoroughly
discussing there issues:

  http://michael.orlitzky.com/articles/motherfuckers_need_package_management.xhtml


This title itself is amusing enough

  Motherfuckers need package management


Benda

Re: [gentoo-dev] Integrating Portage with other package managers

[Dawid Weglinski]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

2018-03-08 3:38 GMT+01:00 Benda Xu <heroxbd@gentoo.org>:

> Dear Rich,
>
> Rich Freeman <rich0@gentoo.org> writes:
>
> > Everybody I know has these sorts of complaints about language-based
> > PMs, whether they prefer Ubuntu, or Debian, or CentOS, or whatever.
> > Nobody wants random programs downloading random stuff and dropping
> > orphan files all over their filesystem with no way to identify these
> > or clean them up.  They're usually written by the same sorts of people
> > who tell you to pipe curl into sudo bash...
>
> One observation.  There are a lot of people on this planet who will
> format their disk and reinstall their whatever operating systems upon
> getting fscked by random programs downloading random stuff and dropping
> orphan files all over their filesystem.  So far the volume of this group
> of people is growing fast.


I'm glad there are people who can take care of their stuff on desktop, but
imagine,
there are plenty of those, who run Gentoo on production and do not wish
some POS
to broke their system.




> That give an illusion of tolerance from the users, enouraging this evil
> to go deeper.
>
> Luckily in Gentoo, we are with competent friends who are taking care of
> their own systems.
>
> Cheers,
> Benda
>
>
>


-- 
Pozdrawiam
Dawid Węgliński

[-- Attachment #2: Type: text/html, Size: 2217 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[Michael Orlitzky]

On 03/08/2018 12:54 AM, Benda Xu wrote:
> 
> This title itself is amusing enough
> 
>   Motherfuckers need package management
> 

Which if it is not clear, is intended to be funny.

Re: [gentoo-dev] Integrating Portage with other package managers

[kuzetsa]

[-- Attachment #1.1: Type: text/plain, Size: 1730 bytes --]

On 03/08/2018 07:25 AM, Michael Orlitzky wrote:
> On 03/08/2018 12:54 AM, Benda Xu wrote:
>>
>> This title itself is amusing enough
>>
>>   Motherfuckers need package management
>>
> 
> Which if it is not clear, is intended to be funny.
> 
> 

the colorful language was proportional to the enthusiasm.

decent info, even though the excited person who wrote it
made me roll my eyes (I clicked for the info ::shrug::)

as per this thread: NuGET (mentioned in the article) is
something I had briefly considered trying to hook into:

an upstream decided that one of their own libraries (it
was written / maintained by upstream) should be unbundled
rather than vendored, but instead of giving it standalone
packaging, they decided to make it only available via
NuGET. Since I had never used .NET before, I had no idea
what that even was, initially. It was a "an experience".

dependencies which can't otherwise be satisfied is the
point of the original poster / creator of this thread:

On 03/07/2018 11:06 AM, anoteros@teknik.io wrote:

> Is this a feature/improvement other Gentoo users/developers
> would be interested in? If so, I would love to help discuss
> and potentially help with its implementation.
>

The problem with something like NuGET, is that I don't
know how to safely support the package manager itself.

I think it's a terrible idea to let "something like that"
loose on a gentoo system...

https://docs.microsoft.com/en-us/visualstudio/mac/nuget-walkthrough

^ I guess they officially support mac, but still.
what happens when the developer of a package manager
decide they need root access to your system, but then
do things "their way"? I've got some concerns.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[kuzetsa]

[-- Attachment #1.1: Type: text/plain, Size: 655 bytes --]

On 03/07/2018 11:06 AM, anoteros@teknik.io wrote:

> It seems reasonable to me to 'hook' portage into these
> other package managers, so that running 'emerge bar'
> would actually run 'cabal install bar' rather than
> downloading sources and running 'ghc'.

it gets tricky when there's no good way to even fetch
the source, I mean. Sorry about not proofreading:

On 03/08/2018 09:47 AM, kuzetsa wrote:
> dependencies which can't otherwise be satisfied is the
> point of the original poster / creator of this thread:

I'm certain I worded that wrong. My intent had been to
say that a point which //was not addressed by// the OP

(oops)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[R0b0t1]

On Wed, Mar 7, 2018 at 3:06 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Wed, Mar 7, 2018 at 3:55 PM, R0b0t1 <r030t1@gmail.com> wrote:
>> On Wed, Mar 7, 2018 at 1:15 PM, Alec Warner <antarus@gentoo.org> wrote:
>>>
>>> Because containers are awesome and are way easier to use.
>>>
>>
>> I think you missed my point: Why are they easier to use?
>>
>
> I suspect that he was equating "containers" with "Docker," which both
> runs containers and also is an image management solution (one that I
> don't particularly like, but probably because I don't have heavy needs
> so I find myself fighting it more than anything else - it doesn't hurt
> that for the life of me I can't get it to connect containers to my
> bridge, and it seems like it is completely opposed to just using DHCP
> to obtain IPs).
>
> But, if you're using Docker, sure, you can run whatever the command is
> to download an image of a container with some software pre-installed
> and run it.
>
> If you're using something other than Docker to manage your containers
> then you still have to get the software you want to use installed in
> your container image.
>

I think I was equating containers to Docker as well. My point was
instead of trying to manage dependencies, containers allow people to
shove everything into an empty root with no conflicts. The
enthusiastic blog post seems to restate this.

Re: [gentoo-dev] Integrating Portage with other package managers

[Rich Freeman]

On Thu, Mar 8, 2018 at 11:44 AM, R0b0t1 <r030t1@gmail.com> wrote:
>
> I think I was equating containers to Docker as well. My point was
> instead of trying to manage dependencies, containers allow people to
> shove everything into an empty root with no conflicts. The
> enthusiastic blog post seems to restate this.
>

That is one of many things they can do.  You can also run a service
like apache in a container even if it is installed in the same root
filesystem as all your other applications.  (In fact, I think this is
sort-of the default behavior if you start apache with the systemd unit
supplied.)

Ultimately on linux the governing functionality are kernel namespaces
and chroot (and I guess you might lump in chuid).  Kernel namespaces
involve the various types of namespaces themselves, and then the
clone/setns/unshare system calls.  There are a lot of things you can
do with various applications of these, and you don't have to run a
process in every possible separated namespace.

I mention it mainly because people tend to limit themselves by
thinking that container=docker, when linux provides a number of system
calls that administrators can employ to do useful things, and you
don't need any kind of fancy management system to use any of them, any
more than you need any fancy tools to run chroot.

If you have util-linux installed then try running (as any user - you
don't have to be root):
unshare -i -m -n -p -u -C -f --mount-proc -U -r  /bin/bash

Congrats.  You are now root in a container.  You're in the same root
filesystem as always.  You'll note that you can't actually see
anything that you couldn't see before.  If you run ps -ea you'll see
that you're the only process running on the system.  Devices like
/dev/sda aren't actually accessible.  A lot of container managers
would mount a new /dev and just hide most of that stuff.  You can
probably imagine how something like this could be useful for isolating
processes.  Try mounting a tmpfs somewhere - you'll see you can do it.
The tmpfs will be invisible to other processes though that aren't
inside the container.

-- 
Rich

Re: [gentoo-dev] Integrating Portage with other package managers

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 3149 bytes --]

On Wed, 7 Mar 2018 11:06:47 -0500
anoteros@teknik.io wrote:

> Having used Gentoo for a few years now, one thing that has been annoying
> to me is the tremendous duplication of effort and uphill battle of
> creating ebuilds (build recipes) for language-specific packages that
> already have their own build systems.
> 
> For example, many languages such as Python (pip), Node (npm), Ruby
> (gems), TeXLive (tlmgr), Haskell (cabal), Rust (cargo) have ways of
> redistributing up-to-date dependencies and ensuring that they build
> properly in most environments.

I was expecting to see Java's Maven in this list. Gentoo Java is in
trouble because we have basically given up trying to keep up with the
ecosystem. The last time anything under dev-java was bumped, other than
JVMs and Tomcat-related stuff, was January 20th. We could be bumping 5
packages a day and still not keep up, never mind all the things we
haven't packaged at all yet.

Java's problem is somewhat unique in that although you can have
optional dependencies at runtime, unless you go out of your way
(and nobody ever does), all these optional dependencies are required
at build time. Suddenly a package that might normally only have 2
dependencies now has 20. But wait, that's just the first level of
dependencies! This can repeat on and on and you end up battered and
bruised having packaged 100 things but still not the one you actually
wanted.

I therefore put some thought into something along the lines of what you
have suggested. Java is supposed to be write (or really build) once,
run anywhere so why are we wasting time building things that we don't
have to? That's certainly why upstreams look at us like we're crazy
whenever we dare to contact them. If we can leverage Maven or whatever
to grab the platform-neutral dependencies for us, we can focus
attention on the few that have native libraries and also things like
icons, desktop entries, etc, that Maven doesn't deal with.

But how would this work? I thought perhaps the dependencies we don't
care about could be installed into a Maven repository, essentially
untracked by Portage but still self-contained, by fetching them in
src_unpack (or src_fetch, which I'd like to introduce) and then
installing them in pkg_preinst. However, unless src_fetch is
implemented, it would not be possible to install Java packages offline.
This approach would also break binary packages, effectively forcing
Java packages to have RESTRICT="bindist", although there would arguably
be little building to do anyway.

Apart from the fact that I haven't had time to do any of this, is this
even what Gentoo users want? I suspect not. But what's worse, this or a
totally stale or practically empty package repository?

Others have experimented with from-source ebuild generators but that
doesn't change the fact that putting all these ebuilds into the tree is
still a huge maintenance burden and because of the build time
dependency issue, end users will still have to build hundreds of
packages they don't even care about.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[R0b0t1]

On Thu, Mar 8, 2018 at 11:50 AM, Rich Freeman <rich0@gentoo.org> wrote:
> If you have util-linux installed then try running (as any user - you
> don't have to be root):
> unshare -i -m -n -p -u -C -f --mount-proc -U -r  /bin/bash
>

Interesting. I hadn't found a good interface to containers and
clone(2) besides Docker. Of course, I didn't look very hard. I half
expect a "new" process model to develop around the kernel namespaces,
as people realize GID separation only is too coarse.

I still see some odd claims about container security, though: claiming
containers are more secure than user accounts still seems odd to me,
as if you don't trust the kernel to enforce user accounts, why trust
it to enforce namespace separation?

Re: [gentoo-dev] Integrating Portage with other package managers

[Rich Freeman]

On Thu, Mar 8, 2018 at 4:00 PM, R0b0t1 <r030t1@gmail.com> wrote:
> On Thu, Mar 8, 2018 at 11:50 AM, Rich Freeman <rich0@gentoo.org> wrote:
>> If you have util-linux installed then try running (as any user - you
>> don't have to be root):
>> unshare -i -m -n -p -u -C -f --mount-proc -U -r  /bin/bash
>>
>
> Interesting. I hadn't found a good interface to containers and
> clone(2) besides Docker. Of course, I didn't look very hard. I half
> expect a "new" process model to develop around the kernel namespaces,
> as people realize GID separation only is too coarse.

Well, the original userspace tools are in app-emulation/lxc (I can see
how it ended up in this category, but obviously it doesn't fit).

Docker is the most popular one.

My personal favorite is systemd-nspawn.  That basically has almost
nothing to do with systemd per-se, but I have no idea if it can work
standalone.  It is a bit more minimal while still being fairly
functional.

unshare/nsenter are the util-linux commands that give relatively
direct access to the system calls themselves.  I used unshare to
illustrate that these really are just system calls and how you use
them is up to you.

>
> I still see some odd claims about container security, though: claiming
> containers are more secure than user accounts still seems odd to me,
> as if you don't trust the kernel to enforce user accounts, why trust
> it to enforce namespace separation?
>

I'm not sure I'd say that containers are more than user accounts
per-se, though with UID mapping it could be easier to avoid sharing
user/group IDs across different services that might otherwise share
them.

I'd say that the security of a container is a bit like the security of
running something in a chroot jail.  It is one additional level of
isolation that something needs to break out of.

Processes running as root inside a container (and I mean the real
root, not root in a user namespace like in my example - unless you ran
the command I gave while logged in as root) are not secure today as
far as I'm aware.  Sure, the isolation is such that a process isn't
going to accidentally escape from the container, so that is a form of
security.  However, a malicious process running as root inside a
container (or a chroot) is generally going to be able to escape if it
wants to.  Maybe some kernel hardening approaches would help with
that.

Also, a process running in a container has less info on the host,
especially if in a chroot (which is typically the case).  If you run a
shell as nobody you could probably glean quite a bit about what is
going on with the host, even if you can't directly touch most of it.
If you could run a process as nobody on many hosts maybe you could
identify which hosts you want to prioritize for further attacks.  If
they're doing the same thing on processes trapped in a
container+chroot then they're basically not going to know about any
other services on the host other than the one they've already hacked
into.

Stuff running as non-root inside a container should be very well
contained, barring kernel bugs.  As we saw a few months ago with
meltdown even running in a VM isn't a perfect guarantee of isolation.
But, I would agree that stuff running under a different UID is also
reasonably secure other than information leaks.

The main benefit of containers is isolation, IMO.  Stuff inside a
container just doesn't see anything outside, which means fewer
incompatibilities.  I run a bunch of nspawn containers mostly running
Gentoo (some with other distros inside), and the nice thing about it
is that when I update a container I know it only provides one service,
so I only have one service I need to test, vs doing a glibc update and
having to test everything.  On the other hand it is that many more
updates to do, and there is a bit more memory cost since libraries
aren't shared across them the way they would be if they all ran on the
host.

-- 
Rich

[gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)

[Benda Xu]

Rich Freeman <rich0@gentoo.org> writes:

> If you have util-linux installed then try running (as any user - you
> don't have to be root): unshare -i -m -n -p -u -C -f --mount-proc -U
> -r /bin/bash
>
> Congrats.  You are now root in a container.  You're in the same root
> filesystem as always.  You'll note that you can't actually see
> anything that you couldn't see before.  If you run ps -ea you'll see
> that you're the only process running on the system.  Devices like
> /dev/sda aren't actually accessible.  A lot of container managers
> would mount a new /dev and just hide most of that stuff.  You can
> probably imagine how something like this could be useful for isolating
> processes.  

Just a side node, this seems to be the ultimate sandbox we (Gentoo and
portage) are after.  With this, we might even be able to have portage
full functional: a build is completely determined and only determined by
the dependencies and USE flags.

Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)

[Rich Freeman]

On Thu, Mar 8, 2018 at 7:46 PM, Benda Xu <heroxbd@gentoo.org> wrote:
> Rich Freeman <rich0@gentoo.org> writes:
>
>> If you have util-linux installed then try running (as any user - you
>> don't have to be root): unshare -i -m -n -p -u -C -f --mount-proc -U
>> -r /bin/bash
>>
>> Congrats.  You are now root in a container.  You're in the same root
>> filesystem as always.  You'll note that you can't actually see
>> anything that you couldn't see before.  If you run ps -ea you'll see
>> that you're the only process running on the system.  Devices like
>> /dev/sda aren't actually accessible.  A lot of container managers
>> would mount a new /dev and just hide most of that stuff.  You can
>> probably imagine how something like this could be useful for isolating
>> processes.
>
> Just a side node, this seems to be the ultimate sandbox we (Gentoo and
> portage) are after.  With this, we might even be able to have portage
> full functional: a build is completely determined and only determined by
> the dependencies and USE flags.
>

I'm not sure how well this would perform, but I had given this a bit
of thought a few years ago.  There are two obvious ways to go about
this: the sandbox or containers.

Either way you first go through the declared dependencies (and @system
under our current policies) and obtain a list of all files installed
by these packages.  Then you either bind mount those files into a
container, or add them to the sandbox configuration as readable.
Right now the sandbox rules are set to allow read access by default on
everything, but this is just a one-liner to change, which would cause
the sandbox to generate an error if a file is read which isn't
explicitly enabled.

The container solution is going to be cleaner because it also helps
curb errors from automagic build systems.  These build systems
wouldn't even be able to detect files not owned by a declared
dependency.  With the sandbox approach those files would be visible,
but would generate errors when accessed (so at least we get a bug
report, but users still have to deal with the issue).  Sure, this
would to some extent hide problems, but then again the automagic build
system would be defeated all the same.

The real question is how long it takes to extract those file lists and
create all those bind mounts.  In the case of @system some of this
could be cached.  I'm pretty sure the file lists would be quite fast -
we only need to go one level deep on the dependency tree (I think).  I
suspect the bind mounts would be fast, but I haven't tried to
benchmark it.  I also have no idea how sandbox performs with a very
long list of rules.

-- 
Rich

Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)

[R0b0t1]

On Thu, Mar 8, 2018 at 7:06 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Thu, Mar 8, 2018 at 7:46 PM, Benda Xu <heroxbd@gentoo.org> wrote:
>> Rich Freeman <rich0@gentoo.org> writes:
>>
>>> If you have util-linux installed then try running (as any user - you
>>> don't have to be root): unshare -i -m -n -p -u -C -f --mount-proc -U
>>> -r /bin/bash
>>>
>>> Congrats.  You are now root in a container.  You're in the same root
>>> filesystem as always.  You'll note that you can't actually see
>>> anything that you couldn't see before.  If you run ps -ea you'll see
>>> that you're the only process running on the system.  Devices like
>>> /dev/sda aren't actually accessible.  A lot of container managers
>>> would mount a new /dev and just hide most of that stuff.  You can
>>> probably imagine how something like this could be useful for isolating
>>> processes.
>>
>> Just a side node, this seems to be the ultimate sandbox we (Gentoo and
>> portage) are after.  With this, we might even be able to have portage
>> full functional: a build is completely determined and only determined by
>> the dependencies and USE flags.
>>
>
> I'm not sure how well this would perform, but I had given this a bit
> of thought a few years ago.  There are two obvious ways to go about
> this: the sandbox or containers.
>

It may be possible to use fanotify(7) (but this would only catch disk
access). There is an extension to fanotify created as someone's thesis
(http://lkml.iu.edu/hypermail/linux/kernel/1703.1/05249.html) that
allows recursive watches. One of the features of fanotify is that it
is possible to deny the events generated. The fanotify API may
simplify the application of the generated rules. It was designed to be
fast, and most processing is done in the kernel when calls are made.

Sadly interest in the patches seems to have waned. The functionality
is not exactly duplicated in containers, but they do make it easier to
find changes.

Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)

[Rich Freeman]

On Thu, Mar 8, 2018 at 9:11 PM, R0b0t1 <r030t1@gmail.com> wrote:
>
> Sadly interest in the patches seems to have waned. The functionality
> is not exactly duplicated in containers, but they do make it easier to
> find changes.
>

Well, the idea with containers wouldn't be to monitor anything, but
instead to ensure that there is nothing in the container the ebuild
shouldn't need.  If the package declares 3 build-time dependencies
then the container would contain @system plus those three
dependencies.  If the build system chokes then it will choke for
everybody (eg for the maintainer testing the package), and not
randomly when some user doesn't have some package installed that
wasn't declared.

Now, a container wouldn't help with figuring out what the dependencies
actually are.  Besides inspection/etc a quick solution there is to
just do a build without a container and check the linking, then go
back and fix the deps, and then rebuild to verify that nothing else
was needed.

-- 
Rich

Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)

[X dej]

2018-03-09 1:46 UTC+01:00, Benda Xu <heroxbd@gentoo.org>:
> Just a side node, this seems to be the ultimate sandbox we (Gentoo and
> portage) are after.  With this, we might even be able to have portage
> full functional: a build is completely determined and only determined by
> the dependencies and USE flags.

If you want to have reproducible binaries, you may also want to alter the clock
of the sandbox system.

Suggestions:

At start, set the date as EPOCH (i.e., in 1970).

Each time a file date seems to be set in the future (by tar, ...), set the date
of the sandbox system to that date plus a milisecond.

Each time a file is created or touched, advance the date of the sandbox system
by one milisecond.

Xdej

Re: [gentoo-dev] Integrating Portage with other package managers

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 5481 bytes --]

On Thu, Mar 8, 2018 at 11:44 AM, R0b0t1 <r030t1@gmail.com> wrote:

> On Wed, Mar 7, 2018 at 3:06 PM, Rich Freeman <rich0@gentoo.org> wrote:
> > On Wed, Mar 7, 2018 at 3:55 PM, R0b0t1 <r030t1@gmail.com> wrote:
> >> On Wed, Mar 7, 2018 at 1:15 PM, Alec Warner <antarus@gentoo.org> wrote:
> >>>
> >>> Because containers are awesome and are way easier to use.
> >>>
> >>
> >> I think you missed my point: Why are they easier to use?
> >>
> >
> > I suspect that he was equating "containers" with "Docker," which both
> > runs containers and also is an image management solution (one that I
> > don't particularly like, but probably because I don't have heavy needs
> > so I find myself fighting it more than anything else - it doesn't hurt
> > that for the life of me I can't get it to connect containers to my
> > bridge, and it seems like it is completely opposed to just using DHCP
> > to obtain IPs).
> >
> > But, if you're using Docker, sure, you can run whatever the command is
> > to download an image of a container with some software pre-installed
> > and run it.
> >
> > If you're using something other than Docker to manage your containers
> > then you still have to get the software you want to use installed in
> > your container image.
> >
>
> I think I was equating containers to Docker as well. My point was
> instead of trying to manage dependencies, containers allow people to
> shove everything into an empty root with no conflicts. The
> enthusiastic blog post seems to restate this.
>
>
I think the premise on which I disagree is that somehow portage (and
package managers in general)
somehow allow you to keep a system 'clean'. Clean being a generic sort of
idea like:

1) All system files are owned by a package.
2) Only the packages that need to be installed are installed.
3) As the administrator, I fully understand the state of the system at all
times.

This is a useful abstraction (because its how many people do systems
administration) but I don't like it for a few reasons.

1) People log in and run stuff as root. Stuff running as root doesn't have
to follow the rules of the PM, and so the system state can diverge (even by
accident.)
  a) (Note that 'People' is often 'me'; I'm not immune!
2) The PM itself (or the code it runs, such as pre / post installs) can
have bugs, and so those bugs also can cause stage to diverge.
3) In general, the state of the machine evolves organically over time.
  a) Start with a clean stage3.
  b) You install your configuration management software (puppet / cfengine
/ chef / whatever your CM is)
  c) You commit some code into your CM, but it has a bug, so you
accidentally mutate some unexpected state)
  d) Your CM is really great at doing things like "deploying a web node"
but its bad at "undeploying the web node" (perhaps you never implemented
this recipe.)
    1) You deploy a web node CM recipe on your server. It works great and
runs as a web node for six months.
    2) The web node is getting busy, so you decide to move servers. You run
your web node recipe on another server and decide to repurpose the server
in 1.
    3) You deploy an LDAP node CM recipe on the server in 1. Maybe its also
still running the web node; maybe it has the web node files but not apache
(because you uninstalled it.)

To me this was systems administration in the 90s / 2000s. The case of 3
(organic machine growth over N years) was rampant. On physical machines and
VMs we commonly addressed this problem by having a machine + service
lifecycle, making reinstalls automated, and reinstalling machines when they
switched roles. Reinstalls were 'clean' and we built new installer images
weekly. It also means I can test my procedures and be sure that I can say,
deploy a new webnode or slave and not worry about never doing deploys; its
common for automation to go stale if unused.

I am reminded a bit of a security mindset; once the machine is compromised
its really hard to prove that its no longer compromised (which is why
backup + re-image is a common remediation in security-land.)
I tend to use this as well in machines. If I logged in and did stuff as
root, its probable its not in configuration management, the package manager
may not know, and its now 'unclean.'

Its also possible that you write CM recipes really well (or distribute all
software in packages maybe.) You don't have bugs very often, or you just
don't care about this idiom of "clean" machines. The intent of this text
was merely to provide context for my experience.

In contrast with disposable containers:

   1. Automated build process for my containers.
      1. If there is a bug in the build, I can throw my buggy containers
      away and build new ones.
   2. Containers are encouraged to be stateless, so logging in to the
   container as root is unlikely to scale well; the containers are likely to
   remain 'clean.'
      1. If my containers are dirty, I can just throw them away and build
      new ones.
   3. If I need to change roles, I can just destroy the webnode container
   and deploy a LDAP node container.

The containers are nominally stateless, so there is less chance of 'gunk'
building up and surprising me later. It also makes the lifecycle simpler.

Obviously its somewhat harder for stateful services (databases, etc.) but I
suspect things like SANs (or Ceph) can really provide the storage backing
for the database.
(database "schema" cleanliness is perhaps a separate issue that I'll defer
for another time ;p)

-A

[-- Attachment #2: Type: text/html, Size: 7319 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[Rich Freeman]

On Fri, Mar 9, 2018 at 11:17 AM, Alec Warner <antarus@gentoo.org> wrote:
>
> In contrast with disposable containers:
>
> Automated build process for my containers.
>
> If there is a bug in the build, I can throw my buggy containers away and
> build new ones.
>
> Containers are encouraged to be stateless, so logging in to the container as
> root is unlikely to scale well; the containers are likely to remain 'clean.'
>
> If my containers are dirty, I can just throw them away and build new ones.
>
> If I need to change roles, I can just destroy the webnode container and
> deploy a LDAP node container.
>
> The containers are nominally stateless, so there is less chance of 'gunk'
> building up and surprising me later. It also makes the lifecycle simpler.
>
> Obviously its somewhat harder for stateful services (databases, etc.) but I
> suspect things like SANs (or Ceph) can really provide the storage backing
> for the database.
> (database "schema" cleanliness is perhaps a separate issue that I'll defer
> for another time ;p)
>

Certainly this is a great way to approach things, but it is also not
the process we have in our handbook, and therefore it seems a bit much
to expect users to actually be following it.

If every one of our profiles shipped with an ansible config file or
similar that just results in a working image with a few edits this
would be somewhat more practical, though I'm not sure I've really seen
a clean solution for something like "docker on the desktop."  Docker
won't even fetch an IP address from my router using DHCP...

-- 
Rich

Re: [gentoo-dev] Integrating Portage with other package managers

[R0b0t1]

On Fri, Mar 9, 2018 at 10:17 AM, Alec Warner <antarus@gentoo.org> wrote:
> The containers are nominally stateless, so there is less chance of 'gunk'
> building up and surprising me later. It also makes the lifecycle simpler.
>
> Obviously its somewhat harder for stateful services (databases, etc.) but I
> suspect things like SANs (or Ceph) can really provide the storage backing
> for the database.
> (database "schema" cleanliness is perhaps a separate issue that I'll defer
> for another time ;p)
>

Restated:

Containers are stateless, which prevents programs from munging state,
because there is none. This is okay except when one needs state, which
one does for most desktop activities.

This implies it doesn't solve the problem. Working around it may be
valid, but only if state can be preserved.

Cheers,
     R0b0t1

Re: [gentoo-dev] Integrating Portage with other package managers

[Rich Freeman]

On Fri, Mar 9, 2018 at 3:11 PM, R0b0t1 <r030t1@gmail.com> wrote:
> On Fri, Mar 9, 2018 at 10:17 AM, Alec Warner <antarus@gentoo.org> wrote:
>> The containers are nominally stateless, so there is less chance of 'gunk'
>> building up and surprising me later. It also makes the lifecycle simpler.
>>
>> Obviously its somewhat harder for stateful services (databases, etc.) but I
>> suspect things like SANs (or Ceph) can really provide the storage backing
>> for the database.
>> (database "schema" cleanliness is perhaps a separate issue that I'll defer
>> for another time ;p)
>>
>
> Containers are stateless, which prevents programs from munging state,
> because there is none. This is okay except when one needs state, which
> one does for most desktop activities.
>

Containers don't have to be stateless.  However, if you want them to
be completely disposable then they obviously need to be.

Bind-mounting something into a container is a simple solution for many
situations.  If your desktop only stores state in /home then that is
probably very practical.  You still need to be able to rapidly build a
suitable desktop container and easily deploy it in the context of
something like a laptop.  But, I don't think this was being proposed
as a solution for desktops in the first place, which of course then
raises the issue that until they do cover desktops containers aren't
really a substitute for keeping your filesystem reasonably clean.

-- 
Rich

Re: [gentoo-dev] Integrating Portage with other package managers

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2415 bytes --]

On Wed, 7 Mar 2018 11:06:47 -0500
anoteros@teknik.io wrote:

> Having used Gentoo for a few years now, one thing that has been annoying
> to me is the tremendous duplication of effort and uphill battle of
> creating ebuilds (build recipes) for language-specific packages that
> already have their own build systems.

As much as this would be nice, for some ecosystems this is entirely impossible.

Notably, Perl would be an impossible target, because instead of using
package level dependencies like Gentoo does, it uses module level dependencies.

A package has >= 1 module. And to make matters interesting, "the
package" a given module ships in can change as a function of time.

Sometimes, 2 packages fuse to become 1, and other times, 1 splits to
become 2.

Within CPAN, this happens pretty transparently, mostly because they
don't actually have an ahead-of-time dependency resolution system.

Dependency resolution is performed via:

   1. Get the module name
   2. Fetch a copy of the index
   3. Look up module name in index
   4. Install resolved package

And that "index" is updated every time a new package is uploaded.

Which basically means that system is only useful for asking the
question "Where is the latest version of X", not "where is Y version of
X"

And to add insult to injury, CPAN's versioning scheme is incompatible
with Gentoo's.

( They have a right to, they had both versions and package management
before we did )

We have a hack of sorts to circumnavigate the version scheme[1]
situation, but it only works in one direction ( eg: you can convert
CPAN versions to normalised Gentoo versions, but not vice versa )

But solving the package/module version resolution missmatch is an
ongoing nightmare which all my efforts have so far concluded, that only
humans are capable of solving, and a large volume of my efforts
engaging with upstream have been driven by this single problem.

Native integration would be really nice, but alas, the best of my
understanding so far is its a giant fantasy that attracts lots of
people with good ideas, that ultimately all fail in application. ( And
part of that is because the upstream you're dealing with is not a
machine, but another human who is prone to deviating from spec on a
regular basis enough to cause headaches for us )

1: https://wiki.gentoo.org/wiki/Project:Perl/Version-Scheme

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Integrating Portage with other package managers

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

On Wed, 7 Mar 2018 14:55:43 -0600
R0b0t1 <r030t1@gmail.com> wrote:

> I think you missed my point: Why are they easier to use?

Because it decouples your projects requirements from your OS and Vendor
requirements.

This becomes really useful if your OS/Vendor wishes to upgrade things
and force people to upgrade things, but your production software isn't
ready for it yet.

Having to block your entire OS from critical upgrades simply because of
that is just not great.

That doesn't mean I think this is the right approach, but it does
explain why people do that sort of thing. 

*especially* for people who aren't using Gentoo.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Functional portage with namespace

[Benda Xu]

Hi Xdej,

X dej <dreplaceelettereejbyeletterea@gmail.com> writes:

> If you want to have reproducible binaries, you may also want to alter
> the clock of the sandbox system.

Ha, indeed many packages hardwrites "date of build" alike.  That is a
hard question to define reproducibility.  I would rather ignore the
timestamps when comparing two binaries.

Benda

Re: [gentoo-dev] Integrating Portage with other package managers

[Benda Xu]

Hi anoteros,

anoteros@teknik.io writes:

> Having used Gentoo for a few years now, one thing that has been
> annoying to me is the tremendous duplication of effort and uphill
> battle of creating ebuilds (build recipes) for language-specific
> packages that already have their own build systems.
>
> For example, many languages such as Python (pip), Node (npm), Ruby
> (gems), TeXLive (tlmgr), Haskell (cabal), Rust (cargo) have ways of
> redistributing up-to-date dependencies and ensuring that they build
> properly in most environments. Portage, it seems, does not take
> advantage of this at all. If I want to install the package 'foo',
> which has 'bar' as a dependency (which has been installed through
> cabal, for instance). Why should portage download some outdated second
> copy of the sources for 'bar', rebuild it, and scatter it around the
> file system where it cannot be used by other programs installed by
> cabal?

There have been quite a lot of effort before.  What I know are:

* R_Overlay

R_Overlay uses roverlay[1] to generate ebuilds from CRAN and BIOC.  It
works for 90% of the packages, and has a dozen of users.

The biggest burden of maintenance is parsing random strings upstream
would put in the dependency fields (e.g. [6]).


* Maven Overlay

It is based on java-ebuilder[2] to generate ebuilds from Java Maven
repository.  The proof of concept overlay works well for Apache
spark[3].

A bit of scripting resolved the incompatibility of version schemes
mentioned by kentnl[5].

I am proposing a GSOC idea to mentor a student on it,

  https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2018/Ideas/Maven_Java_overlay

* g-sorcery

A general purpose ebuild generator in Gentoo (app-portage/g-sorcery)[4],
I have been using it for pip (app-portage/gs-pypi) and elpa
(app-portage/gs-elpa).  It works but a bit out-of-date.  Help welcome.


From my experiences with the ebuild generators, they can be made very
reliable and painless. 

By far IMHO the R Overlay is the most successful.  The overlay is
generated on a server exposed into layman list.  The generation process
needs to be maintained and can have bugs when the upstream evolves.  It
can be a pain for an average user to debug.

Therefore, Gentoo semi-offical automatic overlays from language-specific
repositories provides a promising potential solution.

Yours,
Benda

1. https://github.com/dywisor, developed by dywisor as a GSOC project
   with calchan.

2. https://github.com/gentoo/java-ebuilder, lead by fordfrog of the Java
   Project.

3. https://github.com/heroxbd/maven-overlay

4. developed by jauhien as a GSOC project.

5. https://archives.gentoo.org/gentoo-dev/message/487f5eb7b2d1ba00ae607b3f88bbb8c9

6. https://github.com/dywisor/roverlay/blob/master/config/simple-deprules.d/dev-libs

Re: [gentoo-dev] Functional portage with namespace

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Mon, 12 Mar 2018 07:55:46 +0900
Benda Xu <heroxbd@gentoo.org> wrote:

> Ha, indeed many packages hardwrites "date of build" alike.  That is a
> hard question to define reproducibility.  I would rather ignore the
> timestamps when comparing two binaries.

If a hard-timestamp is to be used, assuming you have portage via git,
then it might be desirable to hard-timestamp based on either:

a) the timestamp of the specific ebuilds last change
b) the timestamp of the most-recent-of specific ebuild+eclass's last change
c) the timestamp of the specific ebuilds initial commit

I'm not sure which one is more practical though.

Sounds like it would be an "experts" tool which would become far more practical
for people who are using custom overlays to maintain their production systems,
and those people can naturally make guarantees about their repos being in git,
and they can decide which of those 3 options ( well, the ones we
provide at least ) are most suited to what they're doing.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Functional portage with namespace

[Francesco Riosa]

Il 17/03/2018 00:40, Kent Fredric ha scritto:
> On Mon, 12 Mar 2018 07:55:46 +0900
> Benda Xu <heroxbd@gentoo.org> wrote:
>
>> Ha, indeed many packages hardwrites "date of build" alike.  That is a
>> hard question to define reproducibility.  I would rather ignore the
>> timestamps when comparing two binaries.
> If a hard-timestamp is to be used, assuming you have portage via git,
> then it might be desirable to hard-timestamp based on either:
>
> a) the timestamp of the specific ebuilds last change
> b) the timestamp of the most-recent-of specific ebuild+eclass's last change
> c) the timestamp of the specific ebuilds initial commit
d) for rsync users the timestamp of the repository, kept in metadata/,
the timestamp of last commit otherwise
>
> I'm not sure which one is more practical though.
>
> Sounds like it would be an "experts" tool which would become far more practical
> for people who are using custom overlays to maintain their production systems,
> and those people can naturally make guarantees about their repos being in git,
> and they can decide which of those 3 options ( well, the ones we
> provide at least ) are most suited to what they're doing.
>