[gentoo-user] CI Continuous Integration

[gentoo-user] CI Continuous Integration

[James]

Well,

For all of those who are interested in Continuous Integration, especially
with the demise of Tinderbox, Look at what I just stumbled (googling)
across


"Zentoo Linux also provides a bunch of experimental overlays for special
setups (like Continuous Integration system needing X11 and browsers) and for
testing experimental new packages. "


[1] http://www.zentoo.org/


I have not had a chance to look it over, but at first glance it seems
some folks are already taking  (CI) ideas to the next level?


interesting,
James

Re: [gentoo-user] CI Continuous Integration

[Sam Bishop]

Very interesting. A great example of how something can be both Gentoo
and Not Gentoo. This is 100% Gentoo unlike Funtoo or Sabayon, but it
brings in some of their advantages. Gentoo doesn't prevent us from
having multiple package variants and this leads to cool stuff like
being able to have a set of layman repositories that ebuilds graduate
through in stages, from 'dev' to 'test' to 'stable'.

And this is why I feel so strongly about Gentoo + Git + CI
While github may not be the right place and raw 'git' not the right
tool. I am a big fan of how phabricator + arcanist provides workflow
guarantees on top of using git, such as the 'must pass the linter
rules + tests' workflow and how it can track and reference external
repos side by side with the repos it hosts.

I feel the future belongs to Gentoo as steward of the ebuild format,
portage and related tools more than as a 'meta distro'. CI is the
force multiplier, when anyone who wants to build a "Gentoo powered
distro" has a documented set of tools they can use to 'stand up the
infrastructure' for things like package QA using a CI Server, a Binary
Package build server/server farm, and Binary Package hosting for the
build artefacts. By rights Gentoo not Debian, Arch or Fedora should be
the Distro of choice for creating experimental niche distros from but
we lack the kind of tools to make it 'easy' for people to do. I'm
currently experimenting to see how many of these I can prototype
inside Docker containers or LXC images and it looks quite promising.

The future looks bright for Gentoo. :-)


On 17 December 2014 at 01:15, James <wireless@tampabay.rr.com> wrote:
> Well,
>
> For all of those who are interested in Continuous Integration, especially
> with the demise of Tinderbox, Look at what I just stumbled (googling)
> across
>
>
> "Zentoo Linux also provides a bunch of experimental overlays for special
> setups (like Continuous Integration system needing X11 and browsers) and for
> testing experimental new packages. "
>
>
> [1] http://www.zentoo.org/
>
>
> I have not had a chance to look it over, but at first glance it seems
> some folks are already taking  (CI) ideas to the next level?
>
>
> interesting,
> James
>
>

[gentoo-user] Re: CI Continuous Integration

[James]

Sam Bishop <sam <at> cygnus.email> writes:


> Very interesting. A great example of how something can be both Gentoo
> and Not Gentoo. This is 100% Gentoo unlike Funtoo or Sabayon, but it
> brings in some of their advantages. Gentoo doesn't prevent us from
> having multiple package variants and this leads to cool stuff like
> being able to have a set of layman repositories that ebuilds graduate
> through in stages, from 'dev' to 'test' to 'stable'.

Zentoo is certainly a site worth closer examination for CI ideas.

I also see folks running CI locally on the codes they build
and specifically need to be very robust. Epatch_user is another
need for folks to employ CI on their own (local cluster), imho.



> And this is why I feel so strongly about Gentoo + Git + CI
> While github may not be the right place and raw 'git' not the right
> tool. I am a big fan of how phabricator + arcanist provides workflow
> guarantees on top of using git, such as the 'must pass the linter
> rules + tests' workflow and how it can track and reference external
> repos side by side with the repos it hosts.

I ran across a recent thread [1] on another list about gentoo vs some
of the other more common distros. Folks seem to be firmly in either
camp; but more in the conventional distro camp. What I did find interesting
is lots of corporations are running on hundreds of gentoo systems
and using (chef, puppet, ansible or salt) to ease the management of large
gentoo deployments. It's just nice to know that despite what many say (use a
mainstream distro) Gentoo is alive and doing very well in the corporate world. 

I just wonder why more of them do not openly share management strategies
for large gentoo deployments....? 

[1] http://www.reddit.com/r/linuxadmin/comments/2nkswx/gentoo_in_production/



> I feel the future belongs to Gentoo as steward of the ebuild format,
> portage and related tools more than as a 'meta distro'. CI is the
> force multiplier, when anyone who wants to build a "Gentoo powered
> distro" has a documented set of tools they can use to 'stand up the
> infrastructure' for things like package QA using a CI Server, a Binary
> Package build server/server farm, and Binary Package hosting for the
> build artefacts. By rights Gentoo not Debian, Arch or Fedora should be
> the Distro of choice for creating experimental niche distros from but
> we lack the kind of tools to make it 'easy' for people to do. I'm
> currently experimenting to see how many of these I can prototype
> inside Docker containers or LXC images and it looks quite promising.

I'm just now learning and experimenting with docker and LXC. 'etest'
is an interesting tool one of our devs is putting together in the spirit
of testing combinations of flags for testing [2].

[2] https://github.com/alunduil/etest/

I could not agree more. I think Gentoo is on the verge of an emerging
recognition not only of it's uniqueness, but that it fills a gap sorely
need.

I think that if CI and clusters become, "routine" for the masses of gentoo
users, that will spring-board our rank and file members into jobs deploying
Gentoo deeply into the business world. What extremely talented folks have
done with Gentoo, I've seen many many times. Taking that power and
intentionally making it available to the ordinary linux admin (average
skills) could easily revolutionize the computing landscape. Gentoo will
never be easy, but it is a very flexible and through solution for many
areas of need.

Zentoo and the (corporate usage thread) I posted all tell me that Gentoo
is not only alive and doing well, it is on the move!


James

Re: [gentoo-user] Re: CI Continuous Integration

[Sam Bishop]

Great response, but I just have to express my difference of opinion here.

> Gentoo will never be easy,
> but it is a very flexible and through solution for many areas of need.

That flexibility means we who chose to participate in 'building' Gentoo
have the power to make it as flexible as we want. I want a future where
the ebuild is up there with Make files as a 'standard tool' in the arsenal
of software developers, and nothing stops us making it happen.

There are good examples out there for this:
The AUR on Arch Linux, shows how with the right tools and documentation
like, https://wiki.archlinux.org/index.php/PKGBUILD,
https://wiki.archlinux.org/index.php/Creating_packages#PKGBUILD_functions,
and https://wiki.archlinux.org/index.php/AUR_User_Guidelines#Submitting_packages.
and also NetBSD with their pkgsrc tools are very well developed
(http://www.netbsd.org/docs/pkgsrc/faq.html#faq-pkgtools)
in particular url2pkg and pkglint which let you just 'bootstrap' a package
from just a URL to a source tar ball and pkglint, this kind of code shows
how easy package creation can be, I think we could even have a web
interface for people to submit new software into the 'experimental' pile
so that a hypothetical Gentoo CI system can test them out.

The power at our disposal could even let us 'harvest' the AUR and
other software sources mechanically with daily scripts to download
their packages, analyse them and build new packages for us.
Obviously this would require some effort to build things like
dependency translation lists, but to be honest, anything to do
with package management is a DEEP rabbit hole we can go down.

At the moment the more I learn about the ChromeOS tools the more
I wonder if I should start counting every ChromeOS, device as a
'Gentoo installation that doesn't know it'. Early next year I will probably
buy a ChromeBook Pixel in order to experiment more with just how
easy it is to put the 'full power' back on top of the ChromeOS base. I
suspect it won't be hard, the dev tools let you build packages and push
them to a ChromeOS device over the network, I should be able to just
either push any new package I want from my sever to the laptop.
Or just push portage itself back onto the device.

Mindful of the aforementioned rabbit hole,
I'll stop myself here and sum it up by saying that I don't see any
reason Gentoo must be hard.


On 17 December 2014 at 23:37, James <wireless@tampabay.rr.com> wrote:
> Sam Bishop <sam <at> cygnus.email> writes:
>
>
>> Very interesting. A great example of how something can be both Gentoo
>> and Not Gentoo. This is 100% Gentoo unlike Funtoo or Sabayon, but it
>> brings in some of their advantages. Gentoo doesn't prevent us from
>> having multiple package variants and this leads to cool stuff like
>> being able to have a set of layman repositories that ebuilds graduate
>> through in stages, from 'dev' to 'test' to 'stable'.
>
> Zentoo is certainly a site worth closer examination for CI ideas.
>
> I also see folks running CI locally on the codes they build
> and specifically need to be very robust. Epatch_user is another
> need for folks to employ CI on their own (local cluster), imho.
>
>
>
>> And this is why I feel so strongly about Gentoo + Git + CI
>> While github may not be the right place and raw 'git' not the right
>> tool. I am a big fan of how phabricator + arcanist provides workflow
>> guarantees on top of using git, such as the 'must pass the linter
>> rules + tests' workflow and how it can track and reference external
>> repos side by side with the repos it hosts.
>
> I ran across a recent thread [1] on another list about gentoo vs some
> of the other more common distros. Folks seem to be firmly in either
> camp; but more in the conventional distro camp. What I did find interesting
> is lots of corporations are running on hundreds of gentoo systems
> and using (chef, puppet, ansible or salt) to ease the management of large
> gentoo deployments. It's just nice to know that despite what many say (use a
> mainstream distro) Gentoo is alive and doing very well in the corporate world.
>
> I just wonder why more of them do not openly share management strategies
> for large gentoo deployments....?
>
> [1] http://www.reddit.com/r/linuxadmin/comments/2nkswx/gentoo_in_production/
>
>
>
>> I feel the future belongs to Gentoo as steward of the ebuild format,
>> portage and related tools more than as a 'meta distro'. CI is the
>> force multiplier, when anyone who wants to build a "Gentoo powered
>> distro" has a documented set of tools they can use to 'stand up the
>> infrastructure' for things like package QA using a CI Server, a Binary
>> Package build server/server farm, and Binary Package hosting for the
>> build artefacts. By rights Gentoo not Debian, Arch or Fedora should be
>> the Distro of choice for creating experimental niche distros from but
>> we lack the kind of tools to make it 'easy' for people to do. I'm
>> currently experimenting to see how many of these I can prototype
>> inside Docker containers or LXC images and it looks quite promising.
>
> I'm just now learning and experimenting with docker and LXC. 'etest'
> is an interesting tool one of our devs is putting together in the spirit
> of testing combinations of flags for testing [2].
>
> [2] https://github.com/alunduil/etest/
>
> I could not agree more. I think Gentoo is on the verge of an emerging
> recognition not only of it's uniqueness, but that it fills a gap sorely
> need.
>
> I think that if CI and clusters become, "routine" for the masses of gentoo
> users, that will spring-board our rank and file members into jobs deploying
> Gentoo deeply into the business world. What extremely talented folks have
> done with Gentoo, I've seen many many times. Taking that power and
> intentionally making it available to the ordinary linux admin (average
> skills) could easily revolutionize the computing landscape. Gentoo will
> never be easy, but it is a very flexible and through solution for many
> areas of need.
>
> Zentoo and the (corporate usage thread) I posted all tell me that Gentoo
> is not only alive and doing well, it is on the move!
>
>
> James
>
>
>
>
>
>

Re: [gentoo-user] Re: CI Continuous Integration

[Alan McKinnon]

On 18/12/2014 08:25, Sam Bishop wrote:
> Mindful of the aforementioned rabbit hole,
> I'll stop myself here and sum it up by saying that I don't see any
> reason Gentoo must be hard.


Gentoo isn't "hard" as long as you are up to speed with how to drive it.

It's a high performance race engine that you can turn into anything you
want it to be, and we are the mechanics. The guys in the pits at races
don't consider their engines "hard", they just take it in their stride
because they have large amounts of clue. Just like most of us here :-)

The beauty of Gentoo is that if you do want to make something suitable
for Aunt Tillie to use and nothing else out there fits the bill, you can
write a wrapper around Gentoo to get what you want, and that's where the
complexity lies - in a place Aunt Tillie doesn't need to see it.



-- 
Alan McKinnon
alan.mckinnon@gmail.com