[gentoo-dev] Open Build Service

[gentoo-dev] Open Build Service

[Samuel Bernardo]

Hi,

I send this email to know the devs opinion about Gentoo integration with
Open Build Service[1].

When creating specialized images and using an automated process for
testing before deployment, I think that Open Build Service would be
useful. It already support all major binary based distros and I think
that Gentoo could be in there also.

There is also a subforum with some interesting posts[2], where was
mentioned some references for Gentoo@OBS.

I reviewed catalyst scripts and Gentoo workflow when creating the
package repository, and I think that it could be integrated in OBS. The
advantage is about creating repositories of binary packages from Gentoo
that would be used to deploy containers or VMs. This way, updates could
be applied to the images. OBS will be responsible to compile all images
that would be associated with their own created binary repository.

To use the binary repository in Gentoo is suggested to use a nfs share
for portage/packages directory[3], but it would be a smoother
integration if emerge gets the packages directly from an url.

You can ask, but for that why not using a binary disto? Well they're not
Gentoo... What I mean with this is that all the Gentoo tools, portage
architecture and the ebuild format that allows for excellent source
package definition (EAPI), turn it unique. Also the freedom associated
with Gentoo distribution that, with less effort than the others, allows
for the creation of new distros. Cross compiling tools are also amazing.

So why shouldn't I wish to use Gentoo always?

Well it don't need to be OBS, but since this opensource project have
some nice ideas, why not starting from there?

Best,

Samuel

[1] http://openbuildservice.org/

[2] https://forums.gentoo.org/viewtopic-p-7829060.html

[3] https://wiki.gentoo.org/wiki/Binary_package_guide

Re: [gentoo-dev] Open Build Service

[Damo Brisbane]

[-- Attachment #1: Type: text/plain, Size: 3331 bytes --]

My two cents worth,

I think broader features not necessarily better; to draw an analogy, dotGo
2015 - Rob Pike - Simplicity is Complicated
<https://www.youtube.com/watch?v=rFejpH_tAHM>, ".. a lot of people talk
about tooling... but the real reason [that the go language is sucessful] is
simplicity..most of the [other languages keep adding] new
features..javascript gets classes.. I realised.. that all of these
languages are changing into the same language"... and more, but interesting
nonetheless.

I am opinionated, I have come to Gentoo because I rejected what I saw as
increased abstraction and concepts being piled onto other distributions,
ironically with such features diminishing rather than expanding choices. So
what would be more powerful? IMHO patience and more concentration on
current build tools in the short term at least, back the current product
and skills within this existing platform, for now. Why? because it is a
good platform, patchy perhaps in some areas, but too good to throw away and
this product is in a perfect position to continue to encouraging community
involvement and improvement against the current toolset. Jumping on, or
dissapating current work across big changes to build concepts, will harm
the product at least in the short term. Long term, let the law of commits
decide.

Kind regards,



On Fri, Nov 10, 2017 at 9:03 AM, Samuel Bernardo <
samuelbernardo.mail@gmail.com> wrote:

> Hi,
>
> I send this email to know the devs opinion about Gentoo integration with
> Open Build Service[1].
>
> When creating specialized images and using an automated process for
> testing before deployment, I think that Open Build Service would be
> useful. It already support all major binary based distros and I think
> that Gentoo could be in there also.
>
> There is also a subforum with some interesting posts[2], where was
> mentioned some references for Gentoo@OBS.
>
> I reviewed catalyst scripts and Gentoo workflow when creating the
> package repository, and I think that it could be integrated in OBS. The
> advantage is about creating repositories of binary packages from Gentoo
> that would be used to deploy containers or VMs. This way, updates could
> be applied to the images. OBS will be responsible to compile all images
> that would be associated with their own created binary repository.
>
> To use the binary repository in Gentoo is suggested to use a nfs share
> for portage/packages directory[3], but it would be a smoother
> integration if emerge gets the packages directly from an url.
>
> You can ask, but for that why not using a binary disto? Well they're not
> Gentoo... What I mean with this is that all the Gentoo tools, portage
> architecture and the ebuild format that allows for excellent source
> package definition (EAPI), turn it unique. Also the freedom associated
> with Gentoo distribution that, with less effort than the others, allows
> for the creation of new distros. Cross compiling tools are also amazing.
>
> So why shouldn't I wish to use Gentoo always?
>
> Well it don't need to be OBS, but since this opensource project have
> some nice ideas, why not starting from there?
>
> Best,
>
> Samuel
>
> [1] http://openbuildservice.org/
>
> [2] https://forums.gentoo.org/viewtopic-p-7829060.html
>
> [3] https://wiki.gentoo.org/wiki/Binary_package_guide
>
>
>

[-- Attachment #2: Type: text/html, Size: 5384 bytes --]

Re: [gentoo-dev] Open Build Service

[Matt Turner]

On Thu, Nov 9, 2017 at 3:03 PM, Samuel Bernardo
<samuelbernardo.mail@gmail.com> wrote:
> Hi,
>
> I send this email to know the devs opinion about Gentoo integration with
> Open Build Service[1].

It sounds like an interesting idea. I've long thought that making our
binary package story more compelling would greatly improve the distro
as a whole.

Also, be prepared. Any threads on gentoo-dev@ asking for opinions on a
matter *will* garner negative replies.

Re: [gentoo-dev] Open Build Service

[Michał Górny]

W dniu czw, 09.11.2017 o godzinie 23∶03 +0000, użytkownik Samuel
Bernardo napisał:
> Hi,
> 
> I send this email to know the devs opinion about Gentoo integration with
> Open Build Service[1].
> 
> When creating specialized images and using an automated process for
> testing before deployment, I think that Open Build Service would be
> useful. It already support all major binary based distros and I think
> that Gentoo could be in there also.
> 
> There is also a subforum with some interesting posts[2], where was
> mentioned some references for Gentoo@OBS.
> 
> I reviewed catalyst scripts and Gentoo workflow when creating the
> package repository, and I think that it could be integrated in OBS. The
> advantage is about creating repositories of binary packages from Gentoo
> that would be used to deploy containers or VMs. This way, updates could
> be applied to the images. OBS will be responsible to compile all images
> that would be associated with their own created binary repository.
> 
> To use the binary repository in Gentoo is suggested to use a nfs share
> for portage/packages directory[3], but it would be a smoother
> integration if emerge gets the packages directly from an url.
> 
> You can ask, but for that why not using a binary disto? Well they're not
> Gentoo... What I mean with this is that all the Gentoo tools, portage
> architecture and the ebuild format that allows for excellent source
> package definition (EAPI), turn it unique. Also the freedom associated
> with Gentoo distribution that, with less effort than the others, allows
> for the creation of new distros. Cross compiling tools are also amazing.
> 
> So why shouldn't I wish to use Gentoo always?
> 
> Well it don't need to be OBS, but since this opensource project have
> some nice ideas, why not starting from there?

You've put a lot of theory in here but not really a single sentence
on what needs to be done. I'm aware that some people used to be working
on adding some OBS ebuilds to Gentoo but I've no clue about their goal.

Long story short, if it requires changes to the ebuild format, then you
have to list them and convince us. If it doesn't, then you are free to
play with OBS any way you like.

> 
> Best,
> 
> Samuel
> 
> [1] http://openbuildservice.org/
> 
> [2] https://forums.gentoo.org/viewtopic-p-7829060.html
> 
> [3] https://wiki.gentoo.org/wiki/Binary_package_guide
> 
> 

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Open Build Service

[Samuel Bernardo]

Hi Michał,

On 11/11/2017 08:09 AM, Michał Górny wrote:
> You've put a lot of theory in here but not really a single sentence
> on what needs to be done. I'm aware that some people used to be working
> on adding some OBS ebuilds to Gentoo but I've no clue about their goal.
>
> Long story short, if it requires changes to the ebuild format, then you
> have to list them and convince us. If it doesn't, then you are free to
> play with OBS any way you like.
I'll review what will need to be done for the integration.
The only feature that would be useful for now is emerge obtaining the
precompiled binary packages to install in containers/VMs from http
rather than nfs[1].
But I'll do a first code review for the integration and then I'll come
back to give the feedback supported by code snippets.

[1] https://wiki.gentoo.org/wiki/Binary_package_guide

Re: [gentoo-dev] Open Build Service

[Peter Volkov]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

On Tue, Nov 14, 2017 at 4:47 AM, Samuel Bernardo <
samuelbernardo.mail@gmail.com> wrote:
The only feature that would be useful for now is emerge obtaining the

> precompiled binary packages to install in containers/VMs from http
> rather than nfs[1].
>

Samuel, probably I miss something but this should work out of box:
https://wiki.gentoo.org/wiki/Binary_package_guide#Web_based_binary_package_host

Or do you mean something else?

--
Peter.

[-- Attachment #2: Type: text/html, Size: 1005 bytes --]

[gentoo-dev] Re: Open Build Service

[Michael Haubenwallner]

Hi,

another two cents:

On 11/10/2017 12:03 AM, Samuel Bernardo wrote:
> Hi,
> 
> I send this email to know the devs opinion about Gentoo integration with
> Open Build Service[1].
> 
> When creating specialized images and using an automated process for
> testing before deployment, I think that Open Build Service would be
> useful. It already support all major binary based distros and I think
> that Gentoo could be in there also.

Well, Gentoo is a Meta Distribution, and it's binary instantiation usually
does emerge on the users machine(s). So users actually do have their private
binary distribution - either with or without caching the binary packages.
(In my opinion, a binary distribution is little more than a cache for binary
packages, to avoid the need for compilation resources on the target machine.)

Of course there is chance that users do have identical profile setups (USE flags,
optimization flags, etc.), which is where OBS (or similar) indeed feels useful.

Rather than sharing binary packages, an idea is to share Gentoo user's profile
setups - with the cache for binary packages to be optional (yet powered by some
open build service). Note that some USE flags disallow binary packaging at all.

Wouldn't we gain something like a "Social Linux Distribution Network" then?

/haubi/

Re: [gentoo-dev] Open Build Service

[Ian Stakenvicius]

[-- Attachment #1.1: Type: text/plain, Size: 1843 bytes --]

On 14/11/17 02:33 AM, Peter Volkov wrote:
> On Tue, Nov 14, 2017 at 4:47 AM, Samuel Bernardo
> <samuelbernardo.mail@gmail.com <mailto:samuelbernardo.mail@gmail.com>>
> wrote:
> The only feature that would be useful for now is emerge obtaining the
> 
>     precompiled binary packages to install in containers/VMs from http
>     rather than nfs[1].
> 
> 
> Samuel, probably I miss something but this should work out of box:
> https://wiki.gentoo.org/wiki/Binary_package_guide#Web_based_binary_package_host
> 
> Or do you mean something else?
>  

I would mirror Peter's concern here -- if you use a "build box" system
to generate binary packages with FEATURES="buildpkg", then the
consumer systems that would use FEATURES="getbinpkg" will fetch the
binary packages with the same standard fetch mechanisms as they do for
distfiles (http, https, ftp) based on the value of PORTAGE_BINHOST in
make.conf.  There shouldn't be any need for NFS from the portage
(emerge) perspective.


To the project in general:

You may want to look into the tindeboxing projects for your basis -- I
think those might get you closer to a fully automated package building
system than working up from catalyst alone.

The biggest issue you will have with doing these types of builds,
though, is dealing with the various use flag differences that various
consumer systems may have.  From what little I've played with binary
builds, if you want to offer binpkg's for an entire system set you
pretty much have to synchronize all use flags across all systems.  So
a realistic deployment will essentially require (a) adherence to a
specific set of static profiles, or (b) a nearly-infinitely-growing
number of build profiles that match each permutation of global and
per-package use flag settings that your consumers would have.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

[gentoo-dev] Re: Open Build Service

[Samuel Bernardo]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

Hi,


On 11/14/2017 10:04 AM, Michael Haubenwallner wrote:
> Well, Gentoo is a Meta Distribution, and it's binary instantiation usually
> does emerge on the users machine(s). So users actually do have their private
> binary distribution - either with or without caching the binary packages.
> (In my opinion, a binary distribution is little more than a cache for binary
> packages, to avoid the need for compilation resources on the target machine.)
>
> Of course there is chance that users do have identical profile setups (USE flags,
> optimization flags, etc.), which is where OBS (or similar) indeed feels useful.
>
> Rather than sharing binary packages, an idea is to share Gentoo user's profile
> setups - with the cache for binary packages to be optional (yet powered by some
> open build service). Note that some USE flags disallow binary packaging at all.
>
> Wouldn't we gain something like a "Social Linux Distribution Network" then?
>
> /haubi/

I think that proposal would be awesome to be achieved!
I like very much the idea. I'll review that.

[-- Attachment #2: Type: text/html, Size: 1525 bytes --]

Re: [gentoo-dev] Open Build Service

[Samuel Bernardo]

Hi,


On 11/14/2017 03:04 PM, Ian Stakenvicius wrote:
> The biggest issue you will have with doing these types of builds,
> though, is dealing with the various use flag differences that various
> consumer systems may have.  From what little I've played with binary
> builds, if you want to offer binpkg's for an entire system set you
> pretty much have to synchronize all use flags across all systems.  So
> a realistic deployment will essentially require (a) adherence to a
> specific set of static profiles, or (b) a nearly-infinitely-growing
> number of build profiles that match each permutation of global and
> per-package use flag settings that your consumers would have.
Just like you described. There will be a profile for each system
environment and that would be integrated in CI/CD using OBS for my
proposed use case. Use flags will be optimized for each case only
possible with a powerful distribution like Gentoo :P

Obviously that creating a new profile would need a strong requirement
for that ;)

That's why I think the effort for maintaining profiles would be awesome
with the idea proposed by Michael Haubenwallner, with the "Social Linux
Distribution Network" sharing Gentoo user's profile
setups - with the cache for binary packages to be optional. So the OBS
reference in this context would be the profile and the binary repository
as optional.
The open-source development way of systems described by the schema
"profile" would be the next step. Containers or appliances cloud be
defined this way for complex deployments.

Thanks ;)

Re: [gentoo-dev] Open Build Service

[Samuel Bernardo]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

Hi Peter,


On 11/14/2017 07:33 AM, Peter Volkov wrote:
> Samuel, probably I miss something but this should work out of box:
> https://wiki.gentoo.org/wiki/Binary_package_guide#Web_based_binary_package_host
>
> Or do you mean something else?
>  
Yes, you're right. I miss that when I read that page last time.
So emerge actually support https. Awesome!

Thanks

[-- Attachment #2: Type: text/html, Size: 1155 bytes --]