[gentoo-dev] New project: binhost

[gentoo-dev] New project: binhost

[Andreas K. Hüttel]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

Hi all, 

I'm announcing a new project here - "binhost"

"The Gentoo Binhost project aims to provide readily installable, precompiled 
packages for a subset of configurations, via central binary package hosting. 
Currently we are still in the conceptual planning stage. "

https://wiki.gentoo.org/wiki/Project:Binhost

If you're interested in helping out, feel free to add yourself on the wiki 
page.

Note that I see actually *building* the packages not as the central point of 
the project (that could be e.g. a side effect of a tinderbox). I'm more 
concerned about
* what configurations should we use
* what portage features are still needed or need improvements (e.g. binpkg 
signing and verification)
* how should hosting look like
* and how we can test this on a limited scale before it goes "into production"
* ...

Comments, ideas, flamebaits? :D

Cheers, 
Andreas

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, qa, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] New project: binhost

[Lars Wendler]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:

>Hi all, 
>
>I'm announcing a new project here - "binhost"
>
>"The Gentoo Binhost project aims to provide readily installable,
>precompiled packages for a subset of configurations, via central
>binary package hosting. Currently we are still in the conceptual
>planning stage. "
>
>https://wiki.gentoo.org/wiki/Project:Binhost
>
>If you're interested in helping out, feel free to add yourself on the
>wiki page.
>
>Note that I see actually *building* the packages not as the central
>point of the project (that could be e.g. a side effect of a
>tinderbox). I'm more concerned about
>* what configurations should we use
>* what portage features are still needed or need improvements (e.g.
>binpkg signing and verification)
>* how should hosting look like
>* and how we can test this on a limited scale before it goes "into
>production"
>* ...
>
>Comments, ideas, flamebaits? :D
>
>Cheers, 
>Andreas
>

It would be great to improve portage speed with handling binpkgs. I
already have my own binhost for a couple of Gentoo systems and even
though these systems don't have to compile anything themselves,
installing ~100 to ~200 binpkgs takes way more than an hour of
installation time. Arch Linux' pacman only takes a fraction of this
time for the very same task.
I know that I compare apples with pears here but even reducing the
current portage time by 50% would be a huge improvement.

Cheers

-- 
Lars Wendler
Gentoo package maintainer
GPG: 21CC CF02 4586 0A07 ED93  9F68 498F E765 960E 9B39

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] New project: binhost

[Aisha Tammy]

On 2/10/21 12:57 PM, Andreas K. Hüttel wrote:
> Hi all,
>
> I'm announcing a new project here - "binhost"
>
> "The Gentoo Binhost project aims to provide readily installable, precompiled
> packages for a subset of configurations, via central binary package hosting.
> Currently we are still in the conceptual planning stage. "
>
> https://wiki.gentoo.org/wiki/Project:Binhost
>
> If you're interested in helping out, feel free to add yourself on the wiki
> page.
>
> Note that I see actually *building* the packages not as the central point of
> the project (that could be e.g. a side effect of a tinderbox). I'm more
> concerned about
> * what configurations should we use
maybe to start with, desktop profiles (gnome/qt/none + systemd/openrc) are
a good idea, these are the people who would benefit most.
This may not be enough, most people also enable extra flags
polkit/elogind/ttf/otf/wayland/pulseaudio
We can start with a default and gather feedback as to what new flags should
be enabled.
> * what portage features are still needed or need improvements (e.g. binpkg
> signing and verification)
somehow checking valid CFLAG/etc. compatibility, not sure how.
Some packages fail when built with differing flags
> * how should hosting look like
Something like the list of profiles in eselect profile
(replace https by protocol of choice):

   https://binpkg.gentoo.org/amd64/17.1/desktop/
   https://binpkg.gentoo.org/amd64/17.1/desktop/gnome/

could potentially also be mirrored on rsync mirrors.

Cheers,
Aisha
> * and how we can test this on a limited scale before it goes "into production"
> * ...
>
> Comments, ideas, flamebaits? :D
>
> Cheers,
> Andreas
>

Re: [gentoo-dev] New project: binhost

[Rich Freeman]

On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel <dilfridge@gentoo.org> wrote:
>
> * what portage features are still needed or need improvements (e.g. binpkg
> signing and verification)
> * how should hosting look like

Some ideas for portage enhancements:

1.  Ability to fetch binary packages from some kind of repo.
2.  Ability to have multiple binary packages co-exist in a repo (local
or remote) with different build attributes (arch, USE, CFLAGS,
DEPENDS, whatever).
3.  Ability to pick the most appropriate binary packages to use based
on user preferences (with a mix of hard and soft preferences).

One idea I've had around how #2-3 might be implemented is:
1.  Binary packages already contain data on how they were built (USE
flags, dependencies, etc).  Place this in a file using a deterministic
sorting/etc order so that two builds with the same settings will have
the same results.
2.  Generate a hash of the file contents - this can go in the filename
so that the file can co-exist with other files, and be located
assuming you have a full matching set of metadata.
3.  Start dropping attributes from the file based on a list of
priorities and generate additional hashes.  Create symlinked files to
the original file using these hashes (overwriting or not existing
symlinks based on policy).  This allows the binary package to be found
using either an exact set of attributes or a subset of higher-priority
attributes.  This is analogous to shared object symlinking.
4.  The package manager will look for a binary package first using the
user's full config, and then by dropping optional elements of the
config (so maybe it does the search without CFLAGs, then without USE
flags).  Eventually it aborts based on user prefs (maybe the user only
wants an exact match, or is willing to accept alternate CFLAGs but not
USE flags, or maybe anything for the arch is selected.
5.  As always the final selected binary package still gets evaluated
like any other binary package to ensure it is usable.

Such a system can identify whether a potentially usable file exists
using only filename, cutting down on fetching.  In the interests of
avoiding useless fetches we would only carry step 3 reasonably far -
packages would have to match based on architecture and any dynamic
linking requirements.  So we wouldn't generate hashes that didn't
include at least those minimums, and the package manager wouldn't
search for them.

Obviously you could do more (if you have 5 combinations of use flags,
look for the set that matches most closely).  That couldn't be done
using hashes alone in an efficient way.  You could have a small
manifest file alongside the binary package that could be fetched
separately if the package manager wants to narrow things down and
fetch a few of those to narrow it down further.

Or you could skip the hash searching and just fetch all the manifests
for a particular package/arch and just search all of those, but that
is more data to transfer just to do a query.  A metadata cache of some
kind of might be another solution.  Content hashes would probably
still be useful just to allow co-existence of alternate builds.

-- 
Rich

Re: [gentoo-dev] New project: binhost

[Andreas K. Hüttel]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

> Some ideas for portage enhancements:
> 
> 1.  Ability to fetch binary packages from some kind of repo.
> 2.  Ability to have multiple binary packages co-exist in a repo (local
> or remote) with different build attributes (arch, USE, CFLAGS,
> DEPENDS, whatever).
> 3.  Ability to pick the most appropriate binary packages to use based
> on user preferences (with a mix of hard and soft preferences).

The more definite answer should come from Zac, but I think a good part of this 
is already implemented. :)



-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, qa, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] New project: binhost

[Aisha Tammy]

On 2/10/21 2:11 PM, Rich Freeman wrote:
> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel <dilfridge@gentoo.org> wrote:
>> * what portage features are still needed or need improvements (e.g. binpkg
>> signing and verification)
>> * how should hosting look like
> Some ideas for portage enhancements:
>
> 1.  Ability to fetch binary packages from some kind of repo.
> 2.  Ability to have multiple binary packages co-exist in a repo (local
> or remote) with different build attributes (arch, USE, CFLAGS,
> DEPENDS, whatever).
> 3.  Ability to pick the most appropriate binary packages to use based
> on user preferences (with a mix of hard and soft preferences).
>
> One idea I've had around how #2-3 might be implemented is:
> 1.  Binary packages already contain data on how they were built (USE
> flags, dependencies, etc).  Place this in a file using a deterministic
> sorting/etc order so that two builds with the same settings will have
> the same results.
this is provided by FEATURES="binpkg-multi-instance"
(or maybe i misread)
> 2.  Generate a hash of the file contents - this can go in the filename
> so that the file can co-exist with other files, and be located
> assuming you have a full matching set of metadata.
> 3.  Start dropping attributes from the file based on a list of
> priorities and generate additional hashes.  Create symlinked files to
> the original file using these hashes (overwriting or not existing
> symlinks based on policy).  This allows the binary package to be found
> using either an exact set of attributes or a subset of higher-priority
> attributes.  This is analogous to shared object symlinking.
> 4.  The package manager will look for a binary package first using the
> user's full config, and then by dropping optional elements of the
> config (so maybe it does the search without CFLAGs, then without USE
> flags).  Eventually it aborts based on user prefs (maybe the user only
> wants an exact match, or is willing to accept alternate CFLAGs but not
> USE flags, or maybe anything for the arch is selected.
> 5.  As always the final selected binary package still gets evaluated
> like any other binary package to ensure it is usable.
>
> Such a system can identify whether a potentially usable file exists
> using only filename, cutting down on fetching.  In the interests of
> avoiding useless fetches we would only carry step 3 reasonably far -
> packages would have to match based on architecture and any dynamic
> linking requirements.  So we wouldn't generate hashes that didn't
> include at least those minimums, and the package manager wouldn't
> search for them.
>
> Obviously you could do more (if you have 5 combinations of use flags,
> look for the set that matches most closely).  That couldn't be done
> using hashes alone in an efficient way.  You could have a small
> manifest file alongside the binary package that could be fetched
> separately if the package manager wants to narrow things down and
> fetch a few of those to narrow it down further.
>
> Or you could skip the hash searching and just fetch all the manifests
> for a particular package/arch and just search all of those, but that
> is more data to transfer just to do a query.  A metadata cache of some
> kind of might be another solution.  Content hashes would probably
> still be useful just to allow co-existence of alternate builds.
>

Re: [gentoo-dev] New project: binhost

[Frédéric Pierret]

[-- Attachment #1.1: Type: text/plain, Size: 1479 bytes --]



Le 2/10/21 à 6:57 PM, Andreas K. Hüttel a écrit :
> Hi all,
> 
> I'm announcing a new project here - "binhost"
> 
> "The Gentoo Binhost project aims to provide readily installable, precompiled
> packages for a subset of configurations, via central binary package hosting.
> Currently we are still in the conceptual planning stage. "
> 
> https://wiki.gentoo.org/wiki/Project:Binhost
> 
> If you're interested in helping out, feel free to add yourself on the wiki
> page.
> 
> Note that I see actually *building* the packages not as the central point of
> the project (that could be e.g. a side effect of a tinderbox). I'm more
> concerned about
> * what configurations should we use
> * what portage features are still needed or need improvements (e.g. binpkg
> signing and verification)
> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into production"
> * ...
> 
> Comments, ideas, flamebaits? :D
> 
> Cheers,
> Andreas
> 

Hi Andreas,

I'm pretty interested to help for this topic. Notably, for my work on creating and maintaining Gentoo template for Qubes OS, I'm weekly constructing binpkgs mirrors in order to ease rebuilding from scratch Gentoo templates and also for CI purposes. So I would be glad to help the whole Gentoo community in such efforts.

I'm also following a very interesting work here on portage: https://github.com/gentoo/portage/pull/562

Best regards,
Frédéric


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] New project: binhost

[Toralf Förster]

[-- Attachment #1.1: Type: text/plain, Size: 326 bytes --]

On 2/10/21 6:57 PM, Andreas K. Hüttel wrote:
> Comments, ideas, flamebaits? :D

I'd appreciate bin packages from a user perspective too.
With -j1 I do have compile times of about 1 day and more for few packages.

But the topic "repoducible builds" is lurking around the corner!

-- 
Toralf
PGP 23217DA7 9B888F45


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] New project: binhost

[Alexey Sokolov]

ср, 10 февр. 2021 г. в 19:12, Rich Freeman <rich0@gentoo.org>:
>
> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel <dilfridge@gentoo.org> wrote:
> >
> > * what portage features are still needed or need improvements (e.g. binpkg
> > signing and verification)
> > * how should hosting look like
>
> Some ideas for portage enhancements:
>
> 1.  Ability to fetch binary packages from some kind of repo.
> 2.  Ability to have multiple binary packages co-exist in a repo (local
> or remote) with different build attributes (arch, USE, CFLAGS,
> DEPENDS, whatever).
> 3.  Ability to pick the most appropriate binary packages to use based
> on user preferences (with a mix of hard and soft preferences).
>
> One idea I've had around how #2-3 might be implemented is:
> 1.  Binary packages already contain data on how they were built (USE
> flags, dependencies, etc).  Place this in a file using a deterministic
> sorting/etc order so that two builds with the same settings will have
> the same results.
> 2.  Generate a hash of the file contents - this can go in the filename
> so that the file can co-exist with other files, and be located
> assuming you have a full matching set of metadata.
> 3.  Start dropping attributes from the file based on a list of
> priorities and generate additional hashes.  Create symlinked files to
> the original file using these hashes (overwriting or not existing
> symlinks based on policy).  This allows the binary package to be found
> using either an exact set of attributes or a subset of higher-priority
> attributes.  This is analogous to shared object symlinking.
> 4.  The package manager will look for a binary package first using the
> user's full config, and then by dropping optional elements of the
> config (so maybe it does the search without CFLAGs, then without USE
> flags).  Eventually it aborts based on user prefs (maybe the user only
> wants an exact match, or is willing to accept alternate CFLAGs but not
> USE flags, or maybe anything for the arch is selected.

A related idea: if user could specify which USE-flags are mandatory to
be set, which USE flags are mandatory to be not set, and which can be
either way, it's easier to find the matching binary package with less
constraints, where only some flags match.

> 5.  As always the final selected binary package still gets evaluated
> like any other binary package to ensure it is usable.
>
> Such a system can identify whether a potentially usable file exists
> using only filename, cutting down on fetching.  In the interests of
> avoiding useless fetches we would only carry step 3 reasonably far -
> packages would have to match based on architecture and any dynamic
> linking requirements.  So we wouldn't generate hashes that didn't
> include at least those minimums, and the package manager wouldn't
> search for them.
>
> Obviously you could do more (if you have 5 combinations of use flags,
> look for the set that matches most closely).  That couldn't be done
> using hashes alone in an efficient way.  You could have a small
> manifest file alongside the binary package that could be fetched
> separately if the package manager wants to narrow things down and
> fetch a few of those to narrow it down further.
>
> Or you could skip the hash searching and just fetch all the manifests
> for a particular package/arch and just search all of those, but that
> is more data to transfer just to do a query.  A metadata cache of some
> kind of might be another solution.  Content hashes would probably
> still be useful just to allow co-existence of alternate builds.
>
> --
> Rich
>

Re: [gentoo-dev] New project: binhost

[Michał Górny]

On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
> 
> > Hi all, 
> > 
> > I'm announcing a new project here - "binhost"
> > 
> > "The Gentoo Binhost project aims to provide readily installable,
> > precompiled packages for a subset of configurations, via central
> > binary package hosting. Currently we are still in the conceptual
> > planning stage. "
> > 
> > https://wiki.gentoo.org/wiki/Project:Binhost
> > 
> > If you're interested in helping out, feel free to add yourself on the
> > wiki page.
> > 
> > Note that I see actually *building* the packages not as the central
> > point of the project (that could be e.g. a side effect of a
> > tinderbox). I'm more concerned about
> > * what configurations should we use
> > * what portage features are still needed or need improvements (e.g.
> > binpkg signing and verification)
> > * how should hosting look like
> > * and how we can test this on a limited scale before it goes "into
> > production"
> > * ...
> > 
> > Comments, ideas, flamebaits? :D
> > 
> > Cheers, 
> > Andreas
> > 
> 
> It would be great to improve portage speed with handling binpkgs. I
> already have my own binhost for a couple of Gentoo systems and even
> though these systems don't have to compile anything themselves,
> installing ~100 to ~200 binpkgs takes way more than an hour of
> installation time. Arch Linux' pacman only takes a fraction of this
> time for the very same task.
> I know that I compare apples with pears here but even reducing the
> current portage time by 50% would be a huge improvement.

Is that really a problem?  For me, Portage takes about an hour just to
do the dependency processing these days.  In fact, building from sources
is now faster than dependency calculations.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] New project: binhost

[Jaco Kroon]

Hi,

+1 - love the idea, def joining.

However, I suspect the op was aiming at a publicly hosted binpkg
server.  Given the below it becomes one server/host, we care only about
the build parameters, and I suspect profile then has less of an effect
on the built packages.

For publicly *available* infrastructure I do however not recommend that
arbitrary people be able to upload.  We can however from attempted
download logs try to determine which combinations of USE flags are
required (with hashes of stuff this becomes tricky as even only 16 USE
flags are already 65536 potential hashes, and 20 clocks in at over 16
million, still perfectly reversable, but once we start getting to 30+
USE flags ...).  So perhaps a way to feedback "Hey, I looked for this
combo/hash" so we don't need to reverse hashes.

Would definitely join such a project, and it would be greatly beneficial
if we can improve the infra to have a form of distributed build ... ie,
for private infrastucture, improve the mechanisms used to "check for
binpkg availability, i not available, build it and submit back to binary
host", obviously all such nodes would need to be considered "trusted".

Kind Regards,
Jaco

On 2021/02/10 21:11, Rich Freeman wrote:
> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel <dilfridge@gentoo.org> wrote:
>> * what portage features are still needed or need improvements (e.g. binpkg
>> signing and verification)
>> * how should hosting look like
> Some ideas for portage enhancements:
>
> 1.  Ability to fetch binary packages from some kind of repo.
> 2.  Ability to have multiple binary packages co-exist in a repo (local
> or remote) with different build attributes (arch, USE, CFLAGS,
> DEPENDS, whatever).
> 3.  Ability to pick the most appropriate binary packages to use based
> on user preferences (with a mix of hard and soft preferences).
>
> One idea I've had around how #2-3 might be implemented is:
> 1.  Binary packages already contain data on how they were built (USE
> flags, dependencies, etc).  Place this in a file using a deterministic
> sorting/etc order so that two builds with the same settings will have
> the same results.
> 2.  Generate a hash of the file contents - this can go in the filename
> so that the file can co-exist with other files, and be located
> assuming you have a full matching set of metadata.
> 3.  Start dropping attributes from the file based on a list of
> priorities and generate additional hashes.  Create symlinked files to
> the original file using these hashes (overwriting or not existing
> symlinks based on policy).  This allows the binary package to be found
> using either an exact set of attributes or a subset of higher-priority
> attributes.  This is analogous to shared object symlinking.
> 4.  The package manager will look for a binary package first using the
> user's full config, and then by dropping optional elements of the
> config (so maybe it does the search without CFLAGs, then without USE
> flags).  Eventually it aborts based on user prefs (maybe the user only
> wants an exact match, or is willing to accept alternate CFLAGs but not
> USE flags, or maybe anything for the arch is selected.
> 5.  As always the final selected binary package still gets evaluated
> like any other binary package to ensure it is usable.
>
> Such a system can identify whether a potentially usable file exists
> using only filename, cutting down on fetching.  In the interests of
> avoiding useless fetches we would only carry step 3 reasonably far -
> packages would have to match based on architecture and any dynamic
> linking requirements.  So we wouldn't generate hashes that didn't
> include at least those minimums, and the package manager wouldn't
> search for them.
>
> Obviously you could do more (if you have 5 combinations of use flags,
> look for the set that matches most closely).  That couldn't be done
> using hashes alone in an efficient way.  You could have a small
> manifest file alongside the binary package that could be fetched
> separately if the package manager wants to narrow things down and
> fetch a few of those to narrow it down further.
>
> Or you could skip the hash searching and just fetch all the manifests
> for a particular package/arch and just search all of those, but that
> is more data to transfer just to do a query.  A metadata cache of some
> kind of might be another solution.  Content hashes would probably
> still be useful just to allow co-existence of alternate builds.
>

Re: [gentoo-dev] New project: binhost

[Andrew Ammerlaan]

On 10/02/2021 18:57, Andreas K. Hüttel wrote:
> Hi all,
> 
> I'm announcing a new project here - "binhost"
> 
> "The Gentoo Binhost project aims to provide readily installable, precompiled
> packages for a subset of configurations, via central binary package hosting.
> Currently we are still in the conceptual planning stage. "
> 
> https://wiki.gentoo.org/wiki/Project:Binhost
> 
> If you're interested in helping out, feel free to add yourself on the wiki
> page.
> 
> Note that I see actually *building* the packages not as the central point of
> the project (that could be e.g. a side effect of a tinderbox). I'm more
> concerned about
> * what configurations should we use

Others have already suggested starting with a minimal set of flags or 
starting with the profiles, and then adding flags at request. I would 
like to suggest the opposite approach, start with binpkgs of packages 
which have all or most of the flags enabled, and then add more 
specific/minimal configurations of that package later. Because from an 
user perspective it is less of a problem to install a binpkg which has 
more features than you need, than it is to install a binpkg which is 
lacking a certain feature you need. Therefore, I would start with 
configurations of packages that have most/all things enabled, and thus 
are usable for the largest amount of people. This would pull in more 
dependencies, but for binpkgs this is less of a problem since they don't 
add compile time.

> * what portage features are still needed or need improvements (e.g. binpkg
> signing and verification)

I think a bugtracker for this might be a good idea at some point. In 
general, I think that portage's binpkg support is very good already, 
there are however some things that could be improved. Bug 
https://bugs.gentoo.org/687668 comes to mind (and some other things that 
were already mentioned by others).

The wiki guide on binpkgs[1] also mentions that:
"""
The support for multiple binary package servers is somewhat incomplete. 
If several servers serve a binary package for the same package version, 
then only the first one will be considered. This can be problematic when 
these binary packages differ in their USE variable configuration and the 
USE variable configuration of a later binary package would match the 
systems configuration.
"""
I don't know if this is still accurate, but if it is that would 
definitely be something that could use some improvement.

[1] https://wiki.gentoo.org/wiki/Binary_package_guide

> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into production"
> * ...
> 
> Comments, ideas, flamebaits? :D
> 
> Cheers,
> Andreas
> 

Re: [gentoo-dev] New project: binhost

[Marc Schiffbauer]

[-- Attachment #1: Type: text/plain, Size: 2518 bytes --]

* Andreas K. Hüttel schrieb am 10.02.21 um 12:57 Uhr:
> Hi all, 
> 
> I'm announcing a new project here - "binhost"
> 
> "The Gentoo Binhost project aims to provide readily installable, precompiled 
> packages for a subset of configurations, via central binary package hosting. 
> Currently we are still in the conceptual planning stage. "
> 
> https://wiki.gentoo.org/wiki/Project:Binhost
> 
> If you're interested in helping out, feel free to add yourself on the wiki 
> page.
> 
> Note that I see actually *building* the packages not as the central point of 
> the project (that could be e.g. a side effect of a tinderbox). I'm more 
> concerned about
> * what configurations should we use
> * what portage features are still needed or need improvements (e.g. binpkg 
> signing and verification)
> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into production"
> * ...
> 
> Comments, ideas, flamebaits? :D

I like the idea very much. Lets make Gentoo suck less energy ;-)

I have thought of some aspects in the past.

My idea was to maybe have an additional term: 'flavours'. A flavour 
could be something like "most", "slim", "desktop" or "server" and each 
flavour may specify a subset of USE-flags to be enabled for packages or 
not.

That way people could propably make most out of a public binhost if the 
choose one of the available flavours. That way you would not need to set 
USE flags for any single package to match a packge available on a 
binhost and a binhost does not need to have any combination of USE flags 
for any package in order to be most effective.

With a flavour you could just express something like "I want the 
Plasma-Desktop profile with most features enabled". Or "I want the 
pasma-desktop as slim as possible". And we'd have some presets for that.

But before that, I think we'd need some features in portage that make 
binpkgs trustworthy by adding signatures to the packages or at least to 
the Metadata.

And Metadata should also be compressed or possibly be implemented in 
some incremental kind so that you will not have to download all the 
metadata everytime just because some random package changed that you do 
not even have installed or want to install.

If I had more time I would definitely join the project. Maybe this is 
the case in 2-3 years.

Cheers
-Marc

-- 
0xCA3E7BF67F979BE5 - F7FB 78F7 7CC3 79F6 DF07
                     6E9E CA3E 7BF6 7F97 9BE5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] New project: binhost

[Aisha Tammy]

Maybe as starter, if some people have any public binhosts available, 
some volunteers can try using those to build a (test) system to check 
how it fares?

I have two public binhosts available (with very limited set of packages 
and very specific CPUs and very weak servers, so please don't crash them)

   https://bsd.ac/gentoo-binpkgs/ivybridge/Packages  <--- a bit more up 
to date
   https://bsd.ac/gentoo-binpkgs/znver1/Packages   <--- not that 
regularly updated

The packages might be about week or two old, am going to setup weekly 
builds and uploads.

Might be better to start making something, and testing things out. If 
and when things break, we can see how to fix.

I'm sure others have more extensive infrastructure and setups than my 
tiny VPS.
If some other people want to give out information about their binhosts, 
at least some form of trial and error can be started.

Thoughts?
Aisha

On 2/10/21 12:57 PM, Andreas K. Hüttel wrote:
> Hi all,
>
> I'm announcing a new project here - "binhost"
>
> "The Gentoo Binhost project aims to provide readily installable, precompiled
> packages for a subset of configurations, via central binary package hosting.
> Currently we are still in the conceptual planning stage. "
>
> https://wiki.gentoo.org/wiki/Project:Binhost
>
> If you're interested in helping out, feel free to add yourself on the wiki
> page.
>
> Note that I see actually *building* the packages not as the central point of
> the project (that could be e.g. a side effect of a tinderbox). I'm more
> concerned about
> * what configurations should we use
> * what portage features are still needed or need improvements (e.g. binpkg
> signing and verification)
> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into production"
> * ...
>
> Comments, ideas, flamebaits? :D
>
> Cheers,
> Andreas
>

Re: [gentoo-dev] New project: binhost

[Michael Jones]

[-- Attachment #1: Type: text/plain, Size: 1650 bytes --]

On Wed, Feb 10, 2021 at 11:58 AM Andreas K. Hüttel <dilfridge@gentoo.org>
wrote:

> Hi all,
>
> I'm announcing a new project here - "binhost"
>
> "The Gentoo Binhost project aims to provide readily installable,
> precompiled
> packages for a subset of configurations, via central binary package
> hosting.
> Currently we are still in the conceptual planning stage. "
>
> https://wiki.gentoo.org/wiki/Project:Binhost
>
> If you're interested in helping out, feel free to add yourself on the wiki
> page.
>
> Note that I see actually *building* the packages not as the central point
> of
> the project (that could be e.g. a side effect of a tinderbox). I'm more
> concerned about
> * what configurations should we use
> * what portage features are still needed or need improvements (e.g. binpkg
> signing and verification)
> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into
> production"
> * ...
>
> Comments, ideas, flamebaits? :D
>
> Cheers,
> Andreas
>
> --
> Andreas K. Hüttel
> dilfridge@gentoo.org
> Gentoo Linux developer
> (council, qa, toolchain, base-system, perl, libreoffice)




What level of collaboration can be anticipated with non-official overlays?

E.g. Embedded systems such as Raspberry Pi, are particularly well
positioned to benefit from binhosting. There is a project (which I am
contributing to) which aims to provide an installable image and limited
binhosting for the 64bit Raspberry Pi boards

See this forum post for more details:
https://forums.gentoo.org/viewtopic-t-1098232-postdays-0-postorder-asc-start-325.html

[-- Attachment #2: Type: text/html, Size: 2409 bytes --]

Re: [gentoo-dev] New project: binhost

[m1027]

dilfridge:

> Hi all, 
> 
> I'm announcing a new project here - "binhost"
> 
> "The Gentoo Binhost project aims to provide readily installable, precompiled 
> packages for a subset of configurations, via central binary package hosting. 
> Currently we are still in the conceptual planning stage. "
> 
> https://wiki.gentoo.org/wiki/Project:Binhost
> 
> If you're interested in helping out, feel free to add yourself on the wiki 
> page.
> 
> Note that I see actually *building* the packages not as the central point of 
> the project (that could be e.g. a side effect of a tinderbox). I'm more 
> concerned about
> * what configurations should we use
> * what portage features are still needed or need improvements (e.g. binpkg 
> signing and verification)
> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into production"
> * ...
> 
> Comments, ideas, flamebaits? :D

What's with CFLAGS?

I haven't been working with pre-built binary packages yet. But I
imagine something flexible, between distant compiler and a server
keeping the binaries. For my cases, from Raspberry Pis up to AMD
Ryzens, I love to have this server to just receive the requests what
and how to compile and keep exactly the packages which are
requested so it can be distributed if requested again.

Thanks

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2467 bytes --]

On 2/10/21 10:51 AM, Lars Wendler wrote:
> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
> 
>> Hi all, 
>>
>> I'm announcing a new project here - "binhost"
>>
>> "The Gentoo Binhost project aims to provide readily installable,
>> precompiled packages for a subset of configurations, via central
>> binary package hosting. Currently we are still in the conceptual
>> planning stage. "
>>
>> https://wiki.gentoo.org/wiki/Project:Binhost
>>
>> If you're interested in helping out, feel free to add yourself on the
>> wiki page.
>>
>> Note that I see actually *building* the packages not as the central
>> point of the project (that could be e.g. a side effect of a
>> tinderbox). I'm more concerned about
>> * what configurations should we use
>> * what portage features are still needed or need improvements (e.g.
>> binpkg signing and verification)
>> * how should hosting look like
>> * and how we can test this on a limited scale before it goes "into
>> production"
>> * ...
>>
>> Comments, ideas, flamebaits? :D
>>
>> Cheers, 
>> Andreas
>>
> 
> It would be great to improve portage speed with handling binpkgs. I
> already have my own binhost for a couple of Gentoo systems and even
> though these systems don't have to compile anything themselves,
> installing ~100 to ~200 binpkgs takes way more than an hour of
> installation time. Arch Linux' pacman only takes a fraction of this
> time for the very same task.
> I know that I compare apples with pears here but even reducing the
> current portage time by 50% would be a huge improvement.

In order to maximize throughput, we have FEATURES="parallel-install"
that for example allows one package's files to be merged while a
pkg_{pre,post}{inst,rm} phase is running for an unrelated package that
is in the merge queue.

There's also FEATURES="-ebuild-locks" which allows privileged
pkg_{pre,post}{inst,rm} phases to run concurrently with privileged
phases of unrelated packages in the merge queue.

Ultimately, pkg_{pre,post}{inst,rm} phases could be the limiting factor
here. I portage, we should eliminate calls to these phases when
DEFINED_PHASES metadata indicated that they're not defined.

Also, FEATURES=preserve-libs introduces considerable overhead because it
regenerates LinkageMapELF data structures for every merge. It would be
much more efficient if it could incrementally update these data structures.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2052 bytes --]

On 2/11/21 1:17 AM, Michał Górny wrote:
> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>
>>> Hi all, 
>>>
>>> I'm announcing a new project here - "binhost"
>>>
>>> "The Gentoo Binhost project aims to provide readily installable,
>>> precompiled packages for a subset of configurations, via central
>>> binary package hosting. Currently we are still in the conceptual
>>> planning stage. "
>>>
>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>
>>> If you're interested in helping out, feel free to add yourself on the
>>> wiki page.
>>>
>>> Note that I see actually *building* the packages not as the central
>>> point of the project (that could be e.g. a side effect of a
>>> tinderbox). I'm more concerned about
>>> * what configurations should we use
>>> * what portage features are still needed or need improvements (e.g.
>>> binpkg signing and verification)
>>> * how should hosting look like
>>> * and how we can test this on a limited scale before it goes "into
>>> production"
>>> * ...
>>>
>>> Comments, ideas, flamebaits? :D
>>>
>>> Cheers, 
>>> Andreas
>>>
>>
>> It would be great to improve portage speed with handling binpkgs. I
>> already have my own binhost for a couple of Gentoo systems and even
>> though these systems don't have to compile anything themselves,
>> installing ~100 to ~200 binpkgs takes way more than an hour of
>> installation time. Arch Linux' pacman only takes a fraction of this
>> time for the very same task.
>> I know that I compare apples with pears here but even reducing the
>> current portage time by 50% would be a huge improvement.
> 
> Is that really a problem?  For me, Portage takes about an hour just to
> do the dependency processing these days.  In fact, building from sources
> is now faster than dependency calculations.

The ratio of these times is dependent on the complexity of the
dependencies involved, and so is the answer to your question.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2437 bytes --]

On 2/13/21 4:37 PM, Zac Medico wrote:
> On 2/11/21 1:17 AM, Michał Górny wrote:
>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>
>>>> Hi all, 
>>>>
>>>> I'm announcing a new project here - "binhost"
>>>>
>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>> precompiled packages for a subset of configurations, via central
>>>> binary package hosting. Currently we are still in the conceptual
>>>> planning stage. "
>>>>
>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>
>>>> If you're interested in helping out, feel free to add yourself on the
>>>> wiki page.
>>>>
>>>> Note that I see actually *building* the packages not as the central
>>>> point of the project (that could be e.g. a side effect of a
>>>> tinderbox). I'm more concerned about
>>>> * what configurations should we use
>>>> * what portage features are still needed or need improvements (e.g.
>>>> binpkg signing and verification)
>>>> * how should hosting look like
>>>> * and how we can test this on a limited scale before it goes "into
>>>> production"
>>>> * ...
>>>>
>>>> Comments, ideas, flamebaits? :D
>>>>
>>>> Cheers, 
>>>> Andreas
>>>>
>>>
>>> It would be great to improve portage speed with handling binpkgs. I
>>> already have my own binhost for a couple of Gentoo systems and even
>>> though these systems don't have to compile anything themselves,
>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>> installation time. Arch Linux' pacman only takes a fraction of this
>>> time for the very same task.
>>> I know that I compare apples with pears here but even reducing the
>>> current portage time by 50% would be a huge improvement.
>>
>> Is that really a problem?  For me, Portage takes about an hour just to
>> do the dependency processing these days.  In fact, building from sources
>> is now faster than dependency calculations.
> 
> The ratio of these times is dependent on the complexity of the
> dependencies involved, and so is the answer to your question.

Also, in the context of binary packages, dependencies calculations are
generally simpler for binary packages because their USE conditionals and
slot-operator := dependencies are frozen in a particular state. This
dramatically reduces the search space involved in dependency calculation.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 5134 bytes --]

On 2/10/21 11:11 AM, Rich Freeman wrote:
> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel <dilfridge@gentoo.org> wrote:
>>
>> * what portage features are still needed or need improvements (e.g. binpkg
>> signing and verification)
>> * how should hosting look like
> 
> Some ideas for portage enhancements:
> 
> 1.  Ability to fetch binary packages from some kind of repo.

The old PORTAGE_BINHOST functionality has been replaced with a
binrepos.conf file that's very similar to repos.conf:

https://bugs.gentoo.org/668334

It doesn't have explicit support for multiple local binary package
repositories yet, but somebody got it working with src-uri set to a
file:/ uri as described in comments on this bug:

https://bugs.gentoo.org/768957

> 2.  Ability to have multiple binary packages co-exist in a repo (local
> or remote) with different build attributes (arch, USE, CFLAGS,
> DEPENDS, whatever).

We can now enable FEATURES=binpkg-multi-instance by default now that
this bug is fixed:

https://bugs.gentoo.org/571126

> 3.  Ability to pick the most appropriate binary packages to use based
> on user preferences (with a mix of hard and soft preferences).

Current package selection logic for binary packages is basically the
same as for ebuilds. These are the notable differences:

1) Binary packages are sorted in descending order by (version, mtime),
so then most recent builds are preferred when the versions are identical.

2) The --binpkg-respect-use option rejects binary packages what would
need to be rebuilt in order to match local USE settings.

> One idea I've had around how #2-3 might be implemented is:
> 1.  Binary packages already contain data on how they were built (USE
> flags, dependencies, etc).  Place this in a file using a deterministic
> sorting/etc order so that two builds with the same settings will have
> the same results.

This would only be needed to multi-profile binhosts that provide a
variety of configurations for the same package.

Features like this are not necessary if the binhost only intends to
provide packages for a single profile.

> 2.  Generate a hash of the file contents - this can go in the filename
> so that the file can co-exist with other files, and be located
> assuming you have a full matching set of metadata.

For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID
ensure that file names are unique.

> 3.  Start dropping attributes from the file based on a list of
> priorities and generate additional hashes.  Create symlinked files to
> the original file using these hashes (overwriting or not existing
> symlinks based on policy).  This allows the binary package to be found
> using either an exact set of attributes or a subset of higher-priority
> attributes.  This is analogous to shared object symlinking.
> 4.  The package manager will look for a binary package first using the
> user's full config, and then by dropping optional elements of the
> config (so maybe it does the search without CFLAGs, then without USE
> flags).  Eventually it aborts based on user prefs (maybe the user only
> wants an exact match, or is willing to accept alternate CFLAGs but not
> USE flags, or maybe anything for the arch is selected> 5.  As always the final selected binary package still gets evaluated
> like any other binary package to ensure it is usable.
> 
> Such a system can identify whether a potentially usable file exists
> using only filename, cutting down on fetching.  In the interests of
> avoiding useless fetches we would only carry step 3 reasonably far -
> packages would have to match based on architecture and any dynamic
> linking requirements.  So we wouldn't generate hashes that didn't
> include at least those minimums, and the package manager wouldn't
> search for them.
> 
> Obviously you could do more (if you have 5 combinations of use flags,
> look for the set that matches most closely).  That couldn't be done
> using hashes alone in an efficient way.  You could have a small
> manifest file alongside the binary package that could be fetched
> separately if the package manager wants to narrow things down and
> fetch a few of those to narrow it down further.

All of the above is oriented toward multi-profile binhosts, so we'll
have to do a cost/benefit analysis to determine whether it's worth the
effort to introduce the complexity that multi-profile binhosts add.

> Or you could skip the hash searching and just fetch all the manifests
> for a particular package/arch and just search all of those, but that
> is more data to transfer just to do a query.  A metadata cache of some
> kind of might be another solution.  Content hashes would probably
> still be useful just to allow co-existence of alternate builds.

This also relates to the centralized Packages file that's currently used
to distribute the package metadata for all packages in a binhost. We can
make it scale better if we split out a separate index per package, not
unlike a pypi simple index:

https://pypi.org/simple/
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Rich Freeman]

On Sat, Feb 13, 2021 at 8:51 PM Zac Medico <zmedico@gentoo.org> wrote:
>
> > 2.  Generate a hash of the file contents - this can go in the filename
> > so that the file can co-exist with other files, and be located
> > assuming you have a full matching set of metadata.
>
> For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID
> ensure that file names are unique.
>
> > 3.  Start dropping attributes from the file based on a list of
> > priorities and generate additional hashes.  Create symlinked files to
> > the original file using these hashes (overwriting or not existing
> > symlinks based on policy).  This allows the binary package to be found
> > using either an exact set of attributes or a subset of higher-priority
> > attributes.  This is analogous to shared object symlinking.
> > 4.  The package manager will look for a binary package first using the
> > user's full config, and then by dropping optional elements of the
> > config (so maybe it does the search without CFLAGs, then without USE
> > flags).  Eventually it aborts based on user prefs (maybe the user only
> > wants an exact match, or is willing to accept alternate CFLAGs but not
> > USE flags, or maybe anything for the arch is selected> 5.  As always the final selected binary package still gets evaluated
> > like any other binary package to ensure it is usable.
> >
> > Such a system can identify whether a potentially usable file exists
> > using only filename, cutting down on fetching.  In the interests of
> > avoiding useless fetches we would only carry step 3 reasonably far -
> > packages would have to match based on architecture and any dynamic
> > linking requirements.  So we wouldn't generate hashes that didn't
> > include at least those minimums, and the package manager wouldn't
> > search for them.
> >
> > Obviously you could do more (if you have 5 combinations of use flags,
> > look for the set that matches most closely).  That couldn't be done
> > using hashes alone in an efficient way.  You could have a small
> > manifest file alongside the binary package that could be fetched
> > separately if the package manager wants to narrow things down and
> > fetch a few of those to narrow it down further.
>
> All of the above is oriented toward multi-profile binhosts, so we'll
> have to do a cost/benefit analysis to determine whether it's worth the
> effort to introduce the complexity that multi-profile binhosts add.

The hash label on the filenames was also considered around
multi-profiles.  I figured that if you're going to be building
variants of packages you'd want to parallelize and hashes work better
for that.  Plus at least in concept you could potentially identify and
fetch files by hash using info already in the local repo without
having to sync additional metadata from the binhost.  User-contributed
binaries would also work better in such a world though for obvious
security issues that might just take the form of local user-generated
repos (allowing users to supplement the upstream repo with local
builds for a cluster, without having to mirror/reporoduce the entire
upstream.

I do get that multi-profiles aren't entirely an essential feature, but
when you consider stuff like X11 support or stable/unstable it seems
like we're probably going to have to provide at least a few variants
on packages for this to be practical.  You could just put each profile
in a separate repo, but then anything that doesn't actually change
across profiles gets built multiple times.  The hash-based solution is
also a form of deduping.

But, hey, it is great to see anything like this being done at all.
Walking before running isn't a bad thing!

-- 
Rich

[gentoo-dev] Re: New project: binhost

[Michael Haubenwallner]

On 2/10/21 6:57 PM, Andreas K. Hüttel wrote:
> Hi all, 
> 
> I'm announcing a new project here - "binhost"
> 
> "The Gentoo Binhost project aims to provide readily installable, precompiled 
> packages for a subset of configurations, via central binary package hosting. 
> Currently we are still in the conceptual planning stage. "
> 
> https://wiki.gentoo.org/wiki/Project:Binhost
> 
> If you're interested in helping out, feel free to add yourself on the wiki 
> page.
> 
> Note that I see actually *building* the packages not as the central point of 
> the project (that could be e.g. a side effect of a tinderbox). I'm more 
> concerned about
> * what configurations should we use
> * what portage features are still needed or need improvements (e.g. binpkg 
> signing and verification)
> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into production"
> * ...
> 
> Comments, ideas, flamebaits? :D

To me, rather than build time, the time consuming challenge is to review the
content of make.conf to get the most out of my (usually quite powerful) hardware,
when there are updates to Gentoo or even the hardware.

For GSoC 2018 I have drawn the idea of some 'Social Linux Distribution Network',
as in sharing Gentoo config like in some social network, with backing by some
binhost infrastructure to be an optional second step:
https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2018/Ideas#Social_Linux_Distribution_Network

While I unlikely will have enough time to really help out here, I'm a potential user
of binhost with a personal focus on most flexible connectivity while using Xfce.

My 2 ct,
/haubi/

Re: [gentoo-dev] New project: binhost

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 2215 bytes --]

Il giorno mer 10 feb 2021 alle ore 20:15 Andreas K. Hüttel <
dilfridge@gentoo.org> ha scritto:

> > Some ideas for portage enhancements:
> >
> > 1.  Ability to fetch binary packages from some kind of repo.
> > 2.  Ability to have multiple binary packages co-exist in a repo (local
> > or remote) with different build attributes (arch, USE, CFLAGS,
> > DEPENDS, whatever).
> > 3.  Ability to pick the most appropriate binary packages to use based
> > on user preferences (with a mix of hard and soft preferences).
>
> The more definite answer should come from Zac, but I think a good part of
> this
> is already implemented. :)
>
> kind of, from make.conf manpage:

binpkg-multi-instance

  Enable support for  multiple  binary  package  in‐
  stances  per ebuild.  Having multiple instances is
  useful for a number of purposes, such as retaining
  builds that were built with different USE flags or
  linked against different  versions  of  libraries.
  The  location  of  any  particular  package within
  PKGDIR can be expressed as follows:

       ${PKGDIR}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}.xpak

  The  build-id starts at 1 for the first build of a
  particular ebuild, and is  incremented  by  1  for
  each new build. It is possible to share a writable
  PKGDIR over NFS, and  locking  ensures  that  each
  package   added  to  PKGDIR  will  have  a  unique
  build-id. It is not necessary to migrate an exist‐
  ing PKGDIR to the new layout, since portage is ca‐
  pable of working with a mixed PKGDIR layout, where
  packages  using  the old layout are allowed to re‐
  main in place.

  The new PKGDIR layout is backward-compatible  with
  binhost  clients  running older portage, since the
  file format is identical, the per-package PATH at‐
  tribute  in  the  'Packages' index directs them to
  download the file from the correct URI,  and  they
  automatically  use  BUILD_TIME  metadata to select
  the latest builds.

  There is currently no automated way to  prune  old
  builds from PKGDIR, although it is possible to re‐
  move packages manually, and then run 'emaint --fix
  binhost' to update the ${PKGDIR}/Packages index.

[-- Attachment #2: Type: text/html, Size: 2744 bytes --]

Re: [gentoo-dev] New project: binhost

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 2397 bytes --]

Il giorno mer 10 feb 2021 alle ore 19:51 Lars Wendler <
polynomial-c@gentoo.org> ha scritto:

> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>
> >Hi all,
> >
> >I'm announcing a new project here - "binhost"
> >
> >"The Gentoo Binhost project aims to provide readily installable,
> >precompiled packages for a subset of configurations, via central
> >binary package hosting. Currently we are still in the conceptual
> >planning stage. "
> >
> >https://wiki.gentoo.org/wiki/Project:Binhost
> >
> >If you're interested in helping out, feel free to add yourself on the
> >wiki page.
> >
> >Note that I see actually *building* the packages not as the central
> >point of the project (that could be e.g. a side effect of a
> >tinderbox). I'm more concerned about
> >* what configurations should we use
> >* what portage features are still needed or need improvements (e.g.
> >binpkg signing and verification)
> >* how should hosting look like
> >* and how we can test this on a limited scale before it goes "into
> >production"
> >* ...
> >
> >Comments, ideas, flamebaits? :D
> >
> >Cheers,
> >Andreas
> >
>
> It would be great to improve portage speed with handling binpkgs. I
> already have my own binhost for a couple of Gentoo systems and even
> though these systems don't have to compile anything themselves,
> installing ~100 to ~200 binpkgs takes way more than an hour of
> installation time. Arch Linux' pacman only takes a fraction of this
> time for the very same task.
> I know that I compare apples with pears here but even reducing the
> current portage time by 50% would be a huge improvement.
>
> Agreed, nowadays I do use Gentoo in two ways:
- From binpkgs usually for baremetal, server or desktop
- condensing part of the system in a squashfs image, usually for containers

Binpkg performance is acceptable albeit not blazing fast for machines with
500-800 packages (usually server) while for desktops which easily have 2000
packages the time to update can be hours.

While we are here the squashfs images way to distribute is wonderful and
handy, except that it's read-only and managing /etc is more challenging
without the commodity of etc-update/dispatch-conf, would be nicer to have a
comparable tool to be used for this.
Suggestions about the implementation well accepted

Cheers,
Francesco (vivo) Riosa

[-- Attachment #2: Type: text/html, Size: 3015 bytes --]

Re: [gentoo-dev] New project: binhost

[Jaco Kroon]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

Hi,

> Binpkg performance is acceptable albeit not blazing fast for machines
> with 500-800 packages (usually server) while for desktops which easily
> have 2000 packages the time to update can be hours.
I take from this that the binpkg process really should be optimized
somehow.  Without looking at the code it looks like binpkg is
significantly less efficient than without.

I think Michał also hinted at this.  Essentially, from the sounds of the
above I suspect we're at "we're probably burning more CPU on calculating
the dependency tree than merely recompiling".

Which to me defeats the purpose.  This may be a single-core use vs
multiple cores though, so whilst it may take similar amount of time it's
possible that we end up consuming less energy overall.

Kind Regards,
Jaco


[-- Attachment #2: Type: text/html, Size: 1290 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2939 bytes --]

On 2/13/21 4:53 PM, Zac Medico wrote:
> On 2/13/21 4:37 PM, Zac Medico wrote:
>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>
>>>>> Hi all, 
>>>>>
>>>>> I'm announcing a new project here - "binhost"
>>>>>
>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>> precompiled packages for a subset of configurations, via central
>>>>> binary package hosting. Currently we are still in the conceptual
>>>>> planning stage. "
>>>>>
>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>
>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>> wiki page.
>>>>>
>>>>> Note that I see actually *building* the packages not as the central
>>>>> point of the project (that could be e.g. a side effect of a
>>>>> tinderbox). I'm more concerned about
>>>>> * what configurations should we use
>>>>> * what portage features are still needed or need improvements (e.g.
>>>>> binpkg signing and verification)
>>>>> * how should hosting look like
>>>>> * and how we can test this on a limited scale before it goes "into
>>>>> production"
>>>>> * ...
>>>>>
>>>>> Comments, ideas, flamebaits? :D
>>>>>
>>>>> Cheers, 
>>>>> Andreas
>>>>>
>>>>
>>>> It would be great to improve portage speed with handling binpkgs. I
>>>> already have my own binhost for a couple of Gentoo systems and even
>>>> though these systems don't have to compile anything themselves,
>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>> time for the very same task.
>>>> I know that I compare apples with pears here but even reducing the
>>>> current portage time by 50% would be a huge improvement.
>>>
>>> Is that really a problem?  For me, Portage takes about an hour just to
>>> do the dependency processing these days.  In fact, building from sources
>>> is now faster than dependency calculations.
>>
>> The ratio of these times is dependent on the complexity of the
>> dependencies involved, and so is the answer to your question.
> 
> Also, in the context of binary packages, dependencies calculations are
> generally simpler for binary packages because their USE conditionals and
> slot-operator := dependencies are frozen in a particular state. This
> dramatically reduces the search space involved in dependency calculation.

IUSE_RUNTIME will obviously introduce conditionals in binary package
dependencies, but we should welcome these conditionals because they will
provide useful flexibility.

I think IUSE_RUNTIME will be a very nice feature to have for project
binhost, since it will allow binary package dependencies to behave with
flexibility that more closely resembles the flexibility of ebuild
dependencies.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]


Replying to a random post here- I'm sorry, briefly after I started this thread 
real life got in the way and a lot of other things became more urgent.

As soon as I have time (next weekend?) I'll reply to the many e-mails here.

Thanks -A

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3871 bytes --]

On 2/20/21 8:17 PM, Zac Medico wrote:
> On 2/13/21 4:53 PM, Zac Medico wrote:
>> On 2/13/21 4:37 PM, Zac Medico wrote:
>>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>>
>>>>>> Hi all, 
>>>>>>
>>>>>> I'm announcing a new project here - "binhost"
>>>>>>
>>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>>> precompiled packages for a subset of configurations, via central
>>>>>> binary package hosting. Currently we are still in the conceptual
>>>>>> planning stage. "
>>>>>>
>>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>>
>>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>>> wiki page.
>>>>>>
>>>>>> Note that I see actually *building* the packages not as the central
>>>>>> point of the project (that could be e.g. a side effect of a
>>>>>> tinderbox). I'm more concerned about
>>>>>> * what configurations should we use
>>>>>> * what portage features are still needed or need improvements (e.g.
>>>>>> binpkg signing and verification)
>>>>>> * how should hosting look like
>>>>>> * and how we can test this on a limited scale before it goes "into
>>>>>> production"
>>>>>> * ...
>>>>>>
>>>>>> Comments, ideas, flamebaits? :D
>>>>>>
>>>>>> Cheers, 
>>>>>> Andreas
>>>>>>
>>>>>
>>>>> It would be great to improve portage speed with handling binpkgs. I
>>>>> already have my own binhost for a couple of Gentoo systems and even
>>>>> though these systems don't have to compile anything themselves,
>>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>>> time for the very same task.
>>>>> I know that I compare apples with pears here but even reducing the
>>>>> current portage time by 50% would be a huge improvement.
>>>>
>>>> Is that really a problem?  For me, Portage takes about an hour just to
>>>> do the dependency processing these days.  In fact, building from sources
>>>> is now faster than dependency calculations.
>>>
>>> The ratio of these times is dependent on the complexity of the
>>> dependencies involved, and so is the answer to your question.
>>
>> Also, in the context of binary packages, dependencies calculations are
>> generally simpler for binary packages because their USE conditionals and
>> slot-operator := dependencies are frozen in a particular state. This
>> dramatically reduces the search space involved in dependency calculation.
> 
> IUSE_RUNTIME will obviously introduce conditionals in binary package
> dependencies, but we should welcome these conditionals because they will
> provide useful flexibility.
> 
> I think IUSE_RUNTIME will be a very nice feature to have for project
> binhost, since it will allow binary package dependencies to behave with
> flexibility that more closely resembles the flexibility of ebuild
> dependencies.

We can borrow paludis's notion of pbins [1] to generate ebuilds which
install pre-built content, and the generated ebuilds could have USE
flags that behave similarly to IUSE_RUNTIME in the sense that changes to
USE flags will result in different builds of pre-built content being
installed. A content-hash distfiles layout [2] could serve as a
convenient way to store separate builds of pre-built content for
multiple combinations of USE flags, and a generated ebuild would index
the build by USE flag combination.

Also, for the generated ebuilds, we can generate USE flags to include
separate SRC_URI downloads for pre-built content to support things like
FEATURES=split-debug and FEATURES=install-sources.

[1] https://paludis.exherbo.org/overview/pbins.html
[2] https://bugs.gentoo.org/756778
-- 
Thanks,
Zac




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 4849 bytes --]

On 2/13/21 5:51 PM, Zac Medico wrote:
> On 2/10/21 11:11 AM, Rich Freeman wrote:
>> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel <dilfridge@gentoo.org> wrote:
>>>
>>> * what portage features are still needed or need improvements (e.g. binpkg
>>> signing and verification)
>>> * how should hosting look like
>>
>> Some ideas for portage enhancements:
>>
>> 1.  Ability to fetch binary packages from some kind of repo.
> 
> The old PORTAGE_BINHOST functionality has been replaced with a
> binrepos.conf file that's very similar to repos.conf:
> 
> https://bugs.gentoo.org/668334
> 
> It doesn't have explicit support for multiple local binary package
> repositories yet, but somebody got it working with src-uri set to a
> file:/ uri as described in comments on this bug:
> 
> https://bugs.gentoo.org/768957
> 
>> 2.  Ability to have multiple binary packages co-exist in a repo (local
>> or remote) with different build attributes (arch, USE, CFLAGS,
>> DEPENDS, whatever).
> 
> We can now enable FEATURES=binpkg-multi-instance by default now that
> this bug is fixed:
> 
> https://bugs.gentoo.org/571126
> 
>> 3.  Ability to pick the most appropriate binary packages to use based
>> on user preferences (with a mix of hard and soft preferences).
> 
> Current package selection logic for binary packages is basically the
> same as for ebuilds. These are the notable differences:
> 
> 1) Binary packages are sorted in descending order by (version, mtime),
> so then most recent builds are preferred when the versions are identical.
> 
> 2) The --binpkg-respect-use option rejects binary packages what would
> need to be rebuilt in order to match local USE settings.
> 
>> One idea I've had around how #2-3 might be implemented is:
>> 1.  Binary packages already contain data on how they were built (USE
>> flags, dependencies, etc).  Place this in a file using a deterministic
>> sorting/etc order so that two builds with the same settings will have
>> the same results.
> 
> This would only be needed to multi-profile binhosts that provide a
> variety of configurations for the same package.
> 
> Features like this are not necessary if the binhost only intends to
> provide packages for a single profile.
> 
>> 2.  Generate a hash of the file contents - this can go in the filename
>> so that the file can co-exist with other files, and be located
>> assuming you have a full matching set of metadata.
> 
> For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID
> ensure that file names are unique.
> 
>> 3.  Start dropping attributes from the file based on a list of
>> priorities and generate additional hashes.  Create symlinked files to
>> the original file using these hashes (overwriting or not existing
>> symlinks based on policy).  This allows the binary package to be found
>> using either an exact set of attributes or a subset of higher-priority
>> attributes.  This is analogous to shared object symlinking.
>> 4.  The package manager will look for a binary package first using the
>> user's full config, and then by dropping optional elements of the
>> config (so maybe it does the search without CFLAGs, then without USE
>> flags).  Eventually it aborts based on user prefs (maybe the user only
>> wants an exact match, or is willing to accept alternate CFLAGs but not
>> USE flags, or maybe anything for the arch is selected> 5.  As always the final selected binary package still gets evaluated
>> like any other binary package to ensure it is usable.
>>
>> Such a system can identify whether a potentially usable file exists
>> using only filename, cutting down on fetching.  In the interests of
>> avoiding useless fetches we would only carry step 3 reasonably far -
>> packages would have to match based on architecture and any dynamic
>> linking requirements.  So we wouldn't generate hashes that didn't
>> include at least those minimums, and the package manager wouldn't
>> search for them.
>>
>> Obviously you could do more (if you have 5 combinations of use flags,
>> look for the set that matches most closely).  That couldn't be done
>> using hashes alone in an efficient way.  You could have a small
>> manifest file alongside the binary package that could be fetched
>> separately if the package manager wants to narrow things down and
>> fetch a few of those to narrow it down further.
> 
> All of the above is oriented toward multi-profile binhosts, so we'll
> have to do a cost/benefit analysis to determine whether it's worth the
> effort to introduce the complexity that multi-profile binhosts add.

This idea to borrow paludis's notion of pbins could work well for a
multi-profile binhost:

https://archives.gentoo.org/gentoo-dev/message/1aea0de0ff588c67bf652f4c1f8ef304
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 4220 bytes --]

On 2/23/21 11:46 AM, Zac Medico wrote:
> On 2/20/21 8:17 PM, Zac Medico wrote:
>> On 2/13/21 4:53 PM, Zac Medico wrote:
>>> On 2/13/21 4:37 PM, Zac Medico wrote:
>>>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>>>
>>>>>>> Hi all, 
>>>>>>>
>>>>>>> I'm announcing a new project here - "binhost"
>>>>>>>
>>>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>>>> precompiled packages for a subset of configurations, via central
>>>>>>> binary package hosting. Currently we are still in the conceptual
>>>>>>> planning stage. "
>>>>>>>
>>>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>>>
>>>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>>>> wiki page.
>>>>>>>
>>>>>>> Note that I see actually *building* the packages not as the central
>>>>>>> point of the project (that could be e.g. a side effect of a
>>>>>>> tinderbox). I'm more concerned about
>>>>>>> * what configurations should we use
>>>>>>> * what portage features are still needed or need improvements (e.g.
>>>>>>> binpkg signing and verification)
>>>>>>> * how should hosting look like
>>>>>>> * and how we can test this on a limited scale before it goes "into
>>>>>>> production"
>>>>>>> * ...
>>>>>>>
>>>>>>> Comments, ideas, flamebaits? :D
>>>>>>>
>>>>>>> Cheers, 
>>>>>>> Andreas
>>>>>>>
>>>>>>
>>>>>> It would be great to improve portage speed with handling binpkgs. I
>>>>>> already have my own binhost for a couple of Gentoo systems and even
>>>>>> though these systems don't have to compile anything themselves,
>>>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>>>> time for the very same task.
>>>>>> I know that I compare apples with pears here but even reducing the
>>>>>> current portage time by 50% would be a huge improvement.
>>>>>
>>>>> Is that really a problem?  For me, Portage takes about an hour just to
>>>>> do the dependency processing these days.  In fact, building from sources
>>>>> is now faster than dependency calculations.
>>>>
>>>> The ratio of these times is dependent on the complexity of the
>>>> dependencies involved, and so is the answer to your question.
>>>
>>> Also, in the context of binary packages, dependencies calculations are
>>> generally simpler for binary packages because their USE conditionals and
>>> slot-operator := dependencies are frozen in a particular state. This
>>> dramatically reduces the search space involved in dependency calculation.
>>
>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>> dependencies, but we should welcome these conditionals because they will
>> provide useful flexibility.
>>
>> I think IUSE_RUNTIME will be a very nice feature to have for project
>> binhost, since it will allow binary package dependencies to behave with
>> flexibility that more closely resembles the flexibility of ebuild
>> dependencies.
> 
> We can borrow paludis's notion of pbins [1] to generate ebuilds which
> install pre-built content, and the generated ebuilds could have USE
> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
> USE flags will result in different builds of pre-built content being
> installed. A content-hash distfiles layout [2] could serve as a
> convenient way to store separate builds of pre-built content for
> multiple combinations of USE flags, and a generated ebuild would index
> the build by USE flag combination.
> 
> Also, for the generated ebuilds, we can generate USE flags to include
> separate SRC_URI downloads for pre-built content to support things like
> FEATURES=split-debug and FEATURES=install-sources.

Note that all of this can existing EAPI features, since everything new
would be implemented in an ebuild generator that generates a single
ebuild to index pre-built content from multiple binary package builds.

> [1] https://paludis.exherbo.org/overview/pbins.html
> [2] https://bugs.gentoo.org/756778

-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2291 bytes --]

On 2/23/21 12:05 PM, Zac Medico wrote:
> On 2/23/21 11:46 AM, Zac Medico wrote:
>> On 2/20/21 8:17 PM, Zac Medico wrote:
>>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>>> dependencies, but we should welcome these conditionals because they will
>>> provide useful flexibility.
>>>
>>> I think IUSE_RUNTIME will be a very nice feature to have for project
>>> binhost, since it will allow binary package dependencies to behave with
>>> flexibility that more closely resembles the flexibility of ebuild
>>> dependencies.
>>
>> We can borrow paludis's notion of pbins [1] to generate ebuilds which
>> install pre-built content, and the generated ebuilds could have USE
>> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
>> USE flags will result in different builds of pre-built content being
>> installed. A content-hash distfiles layout [2] could serve as a
>> convenient way to store separate builds of pre-built content for
>> multiple combinations of USE flags, and a generated ebuild would index
>> the build by USE flag combination.
>>
>> Also, for the generated ebuilds, we can generate USE flags to include
>> separate SRC_URI downloads for pre-built content to support things like
>> FEATURES=split-debug and FEATURES=install-sources.
> 
> Note that all of this can use existing EAPI features, since everything new
> would be implemented in an ebuild generator that generates a single
> ebuild to index pre-built content from multiple binary package builds.
> 
>> [1] https://paludis.exherbo.org/overview/pbins.html
>> [2] https://bugs.gentoo.org/756778
> 

For generated ebuilds, we'll have a choice to model things as USE flags
or sub-packages. For example, content from FEATURES=split-debug and
FEATURES=install-sources content might make more sense to model as
sub-packages than USE flags. It makes more sense to generate a
sub-package when there is a need for the sub-package to have USE flags.
For example, a split-debug sub-package can have USE flags which index
pre-built content from builds for multiple USE flag combinations.
Similar USE flags could be useful for an install-sources sub-package if
the source code has patches which are conditional on USE flags.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 5490 bytes --]

On 2/23/21 12:33 PM, Zac Medico wrote:
> On 2/23/21 12:05 PM, Zac Medico wrote:
>> On 2/23/21 11:46 AM, Zac Medico wrote:
>>> On 2/20/21 8:17 PM, Zac Medico wrote:
>>>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>>>> dependencies, but we should welcome these conditionals because they will
>>>> provide useful flexibility.
>>>>
>>>> I think IUSE_RUNTIME will be a very nice feature to have for project
>>>> binhost, since it will allow binary package dependencies to behave with
>>>> flexibility that more closely resembles the flexibility of ebuild
>>>> dependencies.
>>>
>>> We can borrow paludis's notion of pbins [1] to generate ebuilds which
>>> install pre-built content, and the generated ebuilds could have USE
>>> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
>>> USE flags will result in different builds of pre-built content being
>>> installed. A content-hash distfiles layout [2] could serve as a
>>> convenient way to store separate builds of pre-built content for
>>> multiple combinations of USE flags, and a generated ebuild would index
>>> the build by USE flag combination.
>>>
>>> Also, for the generated ebuilds, we can generate USE flags to include
>>> separate SRC_URI downloads for pre-built content to support things like
>>> FEATURES=split-debug and FEATURES=install-sources.
>>
>> Note that all of this can use existing EAPI features, since everything new
>> would be implemented in an ebuild generator that generates a single
>> ebuild to index pre-built content from multiple binary package builds.
>>
>>> [1] https://paludis.exherbo.org/overview/pbins.html
>>> [2] https://bugs.gentoo.org/756778
>>
> 
> For generated ebuilds, we'll have a choice to model things as USE flags
> or sub-packages. For example, content from FEATURES=split-debug and
> FEATURES=install-sources content might make more sense to model as
> sub-packages than USE flags. It makes more sense to generate a
> sub-package when there is a need for the sub-package to have USE flags.
> For example, a split-debug sub-package can have USE flags which index
> pre-built content from builds for multiple USE flag combinations.
> Similar USE flags could be useful for an install-sources sub-package if
> the source code has patches which are conditional on USE flags.

Since the generated ebuilds are inspired by pbins, we might call them
"ebins". Once we have designed an ebin generation process that we're
happy with, we should consider making it part of an EAPI, so that
package managers can generate "ebins" on request. The EAPI should
include ways to split out files and distribute them separately based on
USE flags and/or sub-packages, so that binhost users can easily filter
which files are installed based on USE configuration.

We can automatically map user's splitdebug and installsources FEATURES
settings into USE settings, in the same way that FEATURES=test
automatically maps to USE=test.

Each generated ebuild (ebin) will use its SRC_URI metadata to index
builds for each USE flag combination for which a build exists. For
example, for 3 USE flags, up to 8 combinations will be indexed:

IUSE="a b c installsources splitdebug"
SRC_URI="
  !a? !b? !c? ( mirror://binhost/24fe6bd377 )
  !a? !b?  c? ( mirror://binhost/fbe14cbb02 )
  !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
  !a?  b?  c? ( mirror://binhost/ae60f2940d )
   a? !b? !c? ( mirror://binhost/2976e1acc0 )
   a? !b?  c? ( mirror://binhost/f4809db70c )
   a?  b? !c? ( mirror://binhost/ecd08466cf )
   a?  b?  c? ( mirror://binhost/0c00f33b2e )
  installsources? (
    !a? !b? !c? ( mirror://binhost/063a14d6c7 )
    !a? !b?  c? ( mirror://binhost/f67c311625 )
    !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
    !a?  b?  c? ( mirror://binhost/17a673e16a )
     a? !b? !c? ( mirror://binhost/914d1cecfe )
     a? !b?  c? ( mirror://binhost/ca18d86a2b )
     a?  b? !c? ( mirror://binhost/6bce13471a )
     a?  b?  c? ( mirror://binhost/3a6bdcd228 )
  )
  splitdebug? (
    !a? !b? !c? ( mirror://binhost/29b2f38c41 )
    !a? !b?  c? ( mirror://binhost/8adc9bef51 )
    !a?  b? !c? ( mirror://binhost/954d2ce484 )
    !a?  b?  c? ( mirror://binhost/32a614aaca )
     a? !b? !c? ( mirror://binhost/3548a2302d )
     a? !b?  c? ( mirror://binhost/e0c02cdc88 )
     a?  b? !c? ( mirror://binhost/f9cbd3c181 )
     a?  b?  c? ( mirror://binhost/31d4c03474 )
  )
"

For installsources, we can automate deduplication, so that we can
distribute the same file content for multiple USE combinations when
appropriate. If all of the combinations have identical content, then it
will look like this:

  installsources? (
    !a? !b? !c? ( mirror://binhost/063a14d6c7 )
    !a? !b?  c? ( mirror://binhost/063a14d6c7 )
    !a?  b? !c? ( mirror://binhost/063a14d6c7 )
    !a?  b?  c? ( mirror://binhost/063a14d6c7 )
     a? !b? !c? ( mirror://binhost/063a14d6c7 )
     a? !b?  c? ( mirror://binhost/063a14d6c7 )
     a?  b? !c? ( mirror://binhost/063a14d6c7 )
     a?  b?  c? ( mirror://binhost/063a14d6c7 )
  )

In order to ensure that an ebin is not selected for a USE combination
that has not been built yet, combinations for existing builds will be
listed in REQUIRED_USE, like this:

REQUIRED_USE="
|| (
  ( !a !b !c )
  ( !a !b  c )
  ( !a  b !c )
  ( !a  b  c )
  (  a !b !c )
  (  a !b  c )
  (  a  b !c )
  (  a  b  c )
)
"

-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] New project: binhost

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3009 bytes --]

On 2/24/21 2:29 AM, Zac Medico wrote:
> For example, for 3 USE flags, up to 8 combinations will be indexed:
> 
> IUSE="a b c installsources splitdebug"
> SRC_URI="
>   !a? !b? !c? ( mirror://binhost/24fe6bd377 )
>   !a? !b?  c? ( mirror://binhost/fbe14cbb02 )
>   !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
>   !a?  b?  c? ( mirror://binhost/ae60f2940d )
>    a? !b? !c? ( mirror://binhost/2976e1acc0 )
>    a? !b?  c? ( mirror://binhost/f4809db70c )
>    a?  b? !c? ( mirror://binhost/ecd08466cf )
>    a?  b?  c? ( mirror://binhost/0c00f33b2e )
>   installsources? (
>     !a? !b? !c? ( mirror://binhost/063a14d6c7 )
>     !a? !b?  c? ( mirror://binhost/f67c311625 )
>     !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
>     !a?  b?  c? ( mirror://binhost/17a673e16a )
>      a? !b? !c? ( mirror://binhost/914d1cecfe )
>      a? !b?  c? ( mirror://binhost/ca18d86a2b )
>      a?  b? !c? ( mirror://binhost/6bce13471a )
>      a?  b?  c? ( mirror://binhost/3a6bdcd228 )
>   )
>   splitdebug? (
>     !a? !b? !c? ( mirror://binhost/29b2f38c41 )
>     !a? !b?  c? ( mirror://binhost/8adc9bef51 )
>     !a?  b? !c? ( mirror://binhost/954d2ce484 )
>     !a?  b?  c? ( mirror://binhost/32a614aaca )
>      a? !b? !c? ( mirror://binhost/3548a2302d )
>      a? !b?  c? ( mirror://binhost/e0c02cdc88 )
>      a?  b? !c? ( mirror://binhost/f9cbd3c181 )
>      a?  b?  c? ( mirror://binhost/31d4c03474 )
>   )
> "
> 
> For installsources, we can automate deduplication, so that we can
> distribute the same file content for multiple USE combinations when
> appropriate. If all of the combinations have identical content, then it
> will look like this:
> 
>   installsources? (
>     !a? !b? !c? ( mirror://binhost/063a14d6c7 )
>     !a? !b?  c? ( mirror://binhost/063a14d6c7 )
>     !a?  b? !c? ( mirror://binhost/063a14d6c7 )
>     !a?  b?  c? ( mirror://binhost/063a14d6c7 )
>      a? !b? !c? ( mirror://binhost/063a14d6c7 )
>      a? !b?  c? ( mirror://binhost/063a14d6c7 )
>      a?  b? !c? ( mirror://binhost/063a14d6c7 )
>      a?  b?  c? ( mirror://binhost/063a14d6c7 )
>   )
> 
> In order to ensure that an ebin is not selected for a USE combination
> that has not been built yet, combinations for existing builds will be
> listed in REQUIRED_USE, like this:
> 
> REQUIRED_USE="
> || (
>   ( !a !b !c )
>   ( !a !b  c )
>   ( !a  b !c )
>   ( !a  b  c )
>   (  a !b !c )
>   (  a !b  c )
>   (  a  b !c )
>   (  a  b  c )
> )
> "

In https://bugs.gentoo.org/772380 I'm planning to implement a script
that will take an existing $PKGDIR as input, and generate an "ebin"
binhost as output. It will automatically split out pre-built content
bundles for installsources and splitdebug as shown above. If there is
more than one build for a particular package version and USE
combination, then the script will choose the package instance with
latest BUILD_TIME metadata (in alignment with
FEATURES=binpkg-multi-instance).
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

[gentoo-dev] New project: binhost

[Torokhov Sergey]

[-- Attachment #1: Type: text/html, Size: 1823 bytes --]

Re: [gentoo-dev] New project: binhost

[Marco Scardovi]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

Hi there,

Well, actually you could take a look on GRP project as, if I understand,
was similar to what you would like to do:
https://en.wikipedia.org/wiki/Gentoo_Linux#Gentoo_Reference_Platform

Unfortunately this project was closed on 2011, so lots of things are
changed, but could be a good start


Il Sab 13 Mar 2021, 09:44 Torokhov Sergey <torokhov-s-a@yandex.ru> ha
scritto:

> Maybe the expirience of Calculate Linux will be usefull. It's Gentoo based
> distributive that use portage/emerge and have huge set of prebuild packages
> in *.xpack format.
>
> https://wiki.calculate-linux.org/calculate_vs_gentoo
>
> 10.02.2021, 20:58, "Andreas K. Hüttel" <dilfridge@gentoo.org>:
>
> Hi all,
>
> I'm announcing a new project here - "binhost"
>
> "The Gentoo Binhost project aims to provide readily installable,
> precompiled
> packages for a subset of configurations, via central binary package
> hosting.
> Currently we are still in the conceptual planning stage. "
>
> https://wiki.gentoo.org/wiki/Project:Binhost
>
> If you're interested in helping out, feel free to add yourself on the wiki
> page.
>
> Note that I see actually *building* the packages not as the central point
> of
> the project (that could be e.g. a side effect of a tinderbox). I'm more
> concerned about
> * what configurations should we use
> * what portage features are still needed or need improvements (e.g. binpkg
> signing and verification)
> * how should hosting look like
> * and how we can test this on a limited scale before it goes "into
> production"
> * ...
>
> Comments, ideas, flamebaits? :D
>
> Cheers,
> Andreas
>
> --
> Andreas K. Hüttel
> dilfridge@gentoo.org
> Gentoo Linux developer
> (council, qa, toolchain, base-system, perl, libreoffice)
>
>

[-- Attachment #2: Type: text/html, Size: 2731 bytes --]

Re: [gentoo-dev] New project: binhost

[Wolfgang E. Sanyer]

[-- Attachment #1: Type: text/plain, Size: 2144 bytes --]

I'm unable to add myself to the wiki, as i am not a real dev (yet!) but i
am definitely interested in being a part of this team and contributing to
the discussions.

El sáb., 13 de marzo de 2021 5:48 a. m., Marco Scardovi <marco@scardovi.com>
escribió:

> Hi there,
>
> Well, actually you could take a look on GRP project as, if I understand,
> was similar to what you would like to do:
> https://en.wikipedia.org/wiki/Gentoo_Linux#Gentoo_Reference_Platform
>
> Unfortunately this project was closed on 2011, so lots of things are
> changed, but could be a good start
>
>
> Il Sab 13 Mar 2021, 09:44 Torokhov Sergey <torokhov-s-a@yandex.ru> ha
> scritto:
>
>> Maybe the expirience of Calculate Linux will be usefull. It's Gentoo
>> based distributive that use portage/emerge and have huge set of prebuild
>> packages in *.xpack format.
>>
>> https://wiki.calculate-linux.org/calculate_vs_gentoo
>>
>> 10.02.2021, 20:58, "Andreas K. Hüttel" <dilfridge@gentoo.org>:
>>
>> Hi all,
>>
>> I'm announcing a new project here - "binhost"
>>
>> "The Gentoo Binhost project aims to provide readily installable,
>> precompiled
>> packages for a subset of configurations, via central binary package
>> hosting.
>> Currently we are still in the conceptual planning stage. "
>>
>> https://wiki.gentoo.org/wiki/Project:Binhost
>>
>> If you're interested in helping out, feel free to add yourself on the
>> wiki
>> page.
>>
>> Note that I see actually *building* the packages not as the central point
>> of
>> the project (that could be e.g. a side effect of a tinderbox). I'm more
>> concerned about
>> * what configurations should we use
>> * what portage features are still needed or need improvements (e.g.
>> binpkg
>> signing and verification)
>> * how should hosting look like
>> * and how we can test this on a limited scale before it goes "into
>> production"
>> * ...
>>
>> Comments, ideas, flamebaits? :D
>>
>> Cheers,
>> Andreas
>>
>> --
>> Andreas K. Hüttel
>> dilfridge@gentoo.org
>> Gentoo Linux developer
>> (council, qa, toolchain, base-system, perl, libreoffice)
>>
>>

[-- Attachment #2: Type: text/html, Size: 3362 bytes --]

Re: [gentoo-dev] New project: binhost

[Frédéric Pierret]

[-- Attachment #1.1: Type: text/plain, Size: 2371 bytes --]



Le 2/10/21 à 9:04 PM, Frédéric Pierret a écrit :
> 
> 
> Le 2/10/21 à 6:57 PM, Andreas K. Hüttel a écrit :
>> Hi all,
>>
>> I'm announcing a new project here - "binhost"
>>
>> "The Gentoo Binhost project aims to provide readily installable, precompiled
>> packages for a subset of configurations, via central binary package hosting.
>> Currently we are still in the conceptual planning stage. "
>>
>> https://wiki.gentoo.org/wiki/Project:Binhost
>>
>> If you're interested in helping out, feel free to add yourself on the wiki
>> page.
>>
>> Note that I see actually *building* the packages not as the central point of
>> the project (that could be e.g. a side effect of a tinderbox). I'm more
>> concerned about
>> * what configurations should we use
>> * what portage features are still needed or need improvements (e.g. binpkg
>> signing and verification)
>> * how should hosting look like
>> * and how we can test this on a limited scale before it goes "into production"
>> * ...
>>
>> Comments, ideas, flamebaits? :D
>>
>> Cheers,
>> Andreas
>>
> 
> Hi Andreas,
> 
> I'm pretty interested to help for this topic. Notably, for my work on creating and maintaining Gentoo template for Qubes OS, I'm weekly constructing binpkgs mirrors in order to ease rebuilding from scratch Gentoo templates and also for CI purposes. So I would be glad to help the whole Gentoo community in such efforts.
> 
> I'm also following a very interesting work here on portage: https://github.com/gentoo/portage/pull/562
> 
> Best regards,
> Frédéric
> 

Hi,

A quick update here, I'm currently testing and validating the work done in https://github.com/gentoo/portage/pull/562 in order to give some feedback to Zac hoping we could have this new feature for gpkg available soon in Portage.

If anyone wants to try, you just have to use @RinCat branch. I've simply emerged Portage by replacing in portage-9999.ebuild:

EGIT_REPO_URI="https://github.com/RinCat/portage.git"
EGIT_BRANCH="gpkg"

and a quick make.conf example can be found in here: https://github.com/gentoo/portage/pull/562#issuecomment-797112962.

As of today, the gpkg and signature are working properly. I'm currently building a Gentoo from scratch with this feature and then, I would use it as binhost with signature validation enabled.

Best regards,
Frédéric


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] New project: binhost

[Torokhov Sergey]

[-- Attachment #1: Type: text/html, Size: 3505 bytes --]