[gentoo-user] automated code validation

[gentoo-user] automated code validation

[James]

Howdy!

Software validation has a long, often divisive history, we should all
be aware of. I've seen several Computer Scientists (PH.Dumasses) go to
fists over "validation". It seems surreal now, but, it was hilarious
at the time as none of the (3) involved in the fists_to_cuff had a clue
about fighting. I'm sure other's have their stories.


But, seriously, Gentoo has taken a big hit recently, with the loss of
Tinderbox and Diego [1]. I surely do not want to dredge up that dispute. I
liked both individuals quite a lot and the outcome is pitiful, to say the
least. One was/is known to be a "buss_crack", but I never had issues with
him. What I could never figure out is why Tinderbox, did not exist in
several different locations; this would allow everyone to test x_64 bit
and then focus on different architectures. The code used should be
open source and mainlined, as everyone in Opensource and elsewhere has
need of robust, automated code validation testing, imho. [2]


Now, an old idea of mine (several racks of "gentoo" systems of various
types and diverse hardware) has an updated sense of urgency. Automated
code testing and development, was the goal, with a twist. I also wanted
a method to rapidly change the entire codebase on  a piece of hardware;
and that was the show stopper until recently. Granted this project "was"
to be focused on embedded gentoo, but, I see no reason it could not be
expanded to include testing gentoo distro codes for x86 (32 and 64 bit
codes) and some other arches too.  

(OK so far?) So you say, great go build it! OK, I've been working on it.
I do need to flesh out some ideas and refine some codes and models I have 
found; for that I would greatly appreciate constructive criticism, ideas
and code contributions. I have the resources to scale up this effort.
I would expect "Gentoo" to also encourage others to build up an automated
test suite environment, at other locations.

Existing resource-codes I expext to leverage:

(1) running many of the test systems from USB devices aka likewhoa_usb.
This solves changing out platforms for discrete tests, by manually moving
platforms around with discrete usb sticks. Sure part of the automated
code testing will be all scripts and ethernet, particular for the mainstream
arches, but, I also wanto to support testing on standalone
boards/system for things such as embedded systems, firewalls, micro-dns
servers, etc etc.

(2) support for linux containers/vm/emulation for some test platforms
(here folks are especially encourage to chim in, as this is not my forte.

(3) The state of the art in massive automated testing is Linaro, imho. [3,4]


Granted Linaro is focused on the latest arm cores, not limited to 64 bit[5],
but what they use works, albeit for rolling out regular binary builds
and not built for a rolling release model. [6] Hopefully, with some
input, we can address this aspect and come up with something (a model).


Maybe some forward looking folks could use "gaming theory" to enhance this
automated  test environment that quickly links to code repositories
where codes are developed and modified. That part of the effort would
be experimental, as I see the opportunity maybe working with one or 2
Overlays that are specifically about a single category of code.


And, just in case you are wondering, yet the cluster I'm working on is
to be use for a myriad of admin tasks and cross compiling  to support
an automated testing platform.


All input is welcome, positive or negative....(obviously, security
is a critical component, after the specification/model is established
realizing that the spec. will be modified based on continuous security
testing and enhancements).

James



[1] https://blog.flameeyes.eu/tag/tinderbox

[2] http://wiki.gentoo.org/wiki/Tinderbox_log_collection_and_analysis

[3] https://launchpad.net/lava

[4] https://validation.linaro.org/

[5] http://www.linaro.org/projects/test-validation/

[6] http://www.linaro.org/blog/lava-blog/lava-fundamentals/

Re: [gentoo-user] automated code validation

[Alex Brandt]

Hey James,

I've removed the original content for length but I love the 
ideas you've put together for an overarching testing strategy.

I've begun work on a small ebuild testing framework, etest [1], 
that I believe fits into your model quite well.  It uses docker 
images for isolation and repeatability.  It allows developers to 
verify the install time and some run time behavior of ebuilds in 
an automated fashion.  It's also extremely alpha and I find new 
issues every time I use it.

That all being said I would love feedback and if anyone is brave 
enough to use it I would love to start cataloging issues that 
people find in github.

James, if this doesn't fit your vision then I apologize for the 
tangential reply to your thread.

[1] https://github.com/alunduil/etest

Thanks,

-- 
Alex Brandt
Cloud Evangelist for Rackspace and Developer for Gentoo
http://blog.alunduil.com

Re: [gentoo-user] automated code validation

[Alec Ten Harmsel]

On 12/04/2014 02:02 PM, Alex Brandt wrote:
> Hey James,
>
> I've removed the original content for length but I love the 
> ideas you've put together for an overarching testing strategy.
>
> I've begun work on a small ebuild testing framework, etest [1], 
> that I believe fits into your model quite well.  It uses docker 
> images for isolation and repeatability.  It allows developers to 
> verify the install time and some run time behavior of ebuilds in 
> an automated fashion.  It's also extremely alpha and I find new 
> issues every time I use it.
>
> That all being said I would love feedback and if anyone is brave 
> enough to use it I would love to start cataloging issues that 
> people find in github.

This looks awesome. I don't want to commit too early, but after this
semester is over I think I can run it on a few ebuilds (inside Jenkins)
and help fix issues.

>
> James, if this doesn't fit your vision then I apologize for the 
> tangential reply to your thread.
>
> [1] https://github.com/alunduil/etest
>
> Thanks,
>

It'd be cool if Gentoo had some sort of automated workflow (with
Jenkins, buildbot, whatever) like this:

1. Receive pull request
2. Detect changed ebuild
3. test ebuild with etest

This will take a lot of work to set up, and depending on how my workload
is next semester, I would love to help out as much as possible.

Alec

[gentoo-user] Re: automated code validation

[James]

Alec Ten Harmsel <alec <at> alectenharmsel.com> writes:

 
> > an automated fashion.  It's also extremely alpha and I find new 
> > issues every time I use it.

alpha is my middle name.


> > That all being said I would love feedback and if anyone is brave 
> > enough to use it I would love to start cataloging issues that 
> > people find in github.

Careful, I'm looking for someone to understand my goals and LEAD.
I'm more of a Klingon, when it come to software, that most.....


> > James, if this doesn't fit your vision then I apologize for the 
> > tangential reply to your thread.

> > [1] https://github.com/alunduil/etest

I have many hammers. WE will make it fit and serve your goals too.
Parties are about everyone having fun, imho.

> It'd be cool if Gentoo had some sort of automated workflow (with
> Jenkins, buildbot, whatever) like this:
> 1. Receive pull request
> 2. Detect changed ebuild
> 3. test ebuild with etest

Let's dumb this down a bit. What all do I need to emerge/tweak to
this running on a single system. Right now I'm using lxde,
but soon I'll have 3 high end workstations with ample ram/8core/amd64
to run a small cluster. Are you proposing for me to use a single system,
a (3) node btrfs/ceph cluster (not yet finished) or work across the net?

Alec has his own cluster at UM, so I need some short term and long
term suggestions what you want, on this list or privately; whatever
is comfortable for you. 

Hopefully we'll get a database_dude involved to keep this organized?


I'm all in!


James

Re: [gentoo-user] automated code validation

[Rich Freeman]

On Thu, Dec 4, 2014 at 2:36 PM, Alec Ten Harmsel
<alec@alectenharmsel.com> wrote:
>
> It'd be cool if Gentoo had some sort of automated workflow (with
> Jenkins, buildbot, whatever) like this:
>
> 1. Receive pull request
> 2. Detect changed ebuild
> 3. test ebuild with etest
>
> This will take a lot of work to set up, and depending on how my workload
> is next semester, I would love to help out as much as possible.
>

I've wondered the same thing, and maybe after sufficient bikeshedding
something like this would be a good GSoC project as well.  Running
repoman is another possibility.

One trick is that a pull request could impact multiple packages
(eclass changes, dependency updates, etc).  In these cases running
repoman probably adds more value than full tests.  Either way there
will potentially be a high cost to do the testing, which could create
a backlog if we queue things up (not queuing things creates other
problems).  Repoman false-alarms is another potential issue with this.

So, it may not be a trivial thing to do, but some kind of CI workflow
would be a wonderful thing to have.

--
Rich

Re: [gentoo-user] automated code validation

[Alec Ten Harmsel]

On 12/04/2014 04:08 PM, Rich Freeman wrote:
> On Thu, Dec 4, 2014 at 2:36 PM, Alec Ten Harmsel
> <alec@alectenharmsel.com> wrote:
>> It'd be cool if Gentoo had some sort of automated workflow (with
>> Jenkins, buildbot, whatever) like this:
>>
>> 1. Receive pull request
>> 2. Detect changed ebuild
>> 3. test ebuild with etest
>>
>> This will take a lot of work to set up, and depending on how my workload
>> is next semester, I would love to help out as much as possible.
>>
> I've wondered the same thing, and maybe after sufficient bikeshedding
> something like this would be a good GSoC project as well.  Running
> repoman is another possibility.

Assuming I have time next semester, I'll look into it (and if I haven't
mentioned anything by mid January, someone should bug me). I have
limited hardware - a single 6-core/32GB box to do testing - but I should
be able to do a lot of testing on small packages.

> One trick is that a pull request could impact multiple packages
> (eclass changes, dependency updates, etc).

This would not be terribly difficult to detect, but insanely expensive
resource-wise. The only thing that comes to mind immediately is ensuring
that the software that detects changes associates a set of packages with
an eclass in some config file, and triggers rebuilds for those packages.

> In these cases running
> repoman probably adds more value than full tests.  Either way there
> will potentially be a high cost to do the testing, which could create
> a backlog if we queue things up (not queuing things creates other
> problems).

I sync weekly, and yeah, there'll need to be some reasonably serious
hardware to do this. Jenkins looks like it supports adding workers that
are firewalled by having the workers initiate connections; if this is
the case (or if whatever we end up supports this), I'll gladly offer up
my workstation to do this.

> Repoman false-alarms is another potential issue with this.

Repoman can be improved ;) Repoman is already run nightly, I just don't
think Gentoo has the manpower to deal with that stuff.

Alec

[gentoo-user] Re: automated code validation

[James]

Alec Ten Harmsel <alec <at> alectenharmsel.com> writes:

> I have time next semester, I'll look into it 

> > Repoman false-alarms is another potential issue with this.
> Repoman can be improved ;) Repoman is already run nightly, I just don't
> think Gentoo has the manpower to deal with that stuff.

Fantastic, GSoc for Alec with Rich mentoring!

I bet our friends at RackSpace will provide all the virtual HorsePower
you need, should google not provide the  hundreds/thousands or cores for
you to run on.

With the evolving etest tool in hand, I'm sure I can put together
a very large pile of ebuilds that need parsing?


James

Re: [gentoo-user] Re: automated code validation

[Rich Freeman]

On Fri, Dec 5, 2014 at 9:05 AM, James <wireless@tampabay.rr.com> wrote:
> Alec Ten Harmsel <alec <at> alectenharmsel.com> writes:
>
>> I have time next semester, I'll look into it
>
>> > Repoman false-alarms is another potential issue with this.
>> Repoman can be improved ;) Repoman is already run nightly, I just don't
>> think Gentoo has the manpower to deal with that stuff.
>
> Fantastic, GSoc for Alec with Rich mentoring!
>
> I bet our friends at RackSpace will provide all the virtual HorsePower
> you need, should google not provide the  hundreds/thousands or cores for
> you to run on.
>
> With the evolving etest tool in hand, I'm sure I can put together
> a very large pile of ebuilds that need parsing?
>

My guess is that the hardware to run all this on is the simplest part
of the problem.  If somebody writes something good and we build the
processes to support it, then I'm sure we can find donors to provide
the CPU cycles.  ChromeOS could probably steal whatever we put
together so there is an obvious synergy there, and I'm sure
Amazon/Rackspace/etc are always looking for use cases to show off.

From past feedback from Diego and such the biggest issue with any kind
of tinderbox is dealing with the output.  As has been pointed out
there are folks running Repoman regularly already, and there have been
past tinderbox efforts.  The real issue is to improve the signal/noise
ratio.  You'll only get so far with that using code - there has to be
process change to support it.

If we were to do something like this long-term I'd envision that this
would run on every commit or something like that, and the commit
doesn't go into the rsync tree until it passes.  If the tree goes red
then people stop doing commits until it is fixed, and somebody gets
their wrist slapped.  That is my sense of how most mature
organizations do CI.  The tinderbox is really just doing verification
- stuff should already be tested BEFORE it goes in the tree.  There
also shouldn't be any false positives.  There would need to be a
mechanism to flag ebuilds with known issues so that the tinderbox can
ignore them, and of course we can monitor that to ensure it isn't
abused.

Basically doing this sort of thing right requires a big change in
mindset.  You can't just throw stuff at the tree and wait for the bug
reports to come in.  You can't just make dealing with the tinderbox
the problem of the poor guy running the tinderbox.  The tinderbox
basically has to become the boss and everybody has to make part of
their job keeping the boss happy.

Then you have to throw in uncooperative upstreams.  If you ARE the
upstream then you can apply these practices yourself and have a build
system that outputs zero warnings and all that.  We're dealing with
what we get, so we're going to have to set the bar considerably lower
if we don't want to exclude anything from the tree which has a lousy
build system.

In any case, focus first on decent code with proposed procedures that
help deal with the output.  Don't go spending a lot of money on
hardware or worrying about setting up clusters and all that.  Of
course, if you want to run stuff in parallel that is something you can
design in from the start (and there is no reason you can't run it in a
few VMs on your laptop - it will be slow but the focus is on ensuring
that it works, not that it is fast).  In the past I've seen lots of
people focus on building empires with dreams of hardware and all that,
and they end up with little in the way of donations and if they do buy
hardware they end up with nothing to run on it.  Write the code first
and get it working at small scale.  If you write the code, the
hardware will probably take care of itself.  People care about Gentoo,
but they're not going to throw lots of equipment at somebody who has
little more than an email chain to show up-front, and I doubt the
Foundation is going to want to buy hardware before the code exists.

The code will be hard.  The culture change will be harder.  The
hardware will be easy.

Bottom line - this sort of thing is a lot harder to pull off than it
might first appear.  I don't want to discourage anybody, but don't
think that you can toss together a few lines in bash to do an emerge
-e world and you're done.  Of course, if you do want to do a manual
run like this and manually submit bugs that have been manually
screened for validity, that is always welcome.

--
Rich

[gentoo-user] Re: automated code validation

[James]

Rich Freeman <rich0 <at> gentoo.org> writes:


> My guess is that the hardware to run all this on is the simplest part
> of the problem.  If somebody writes something good and we build the
> processes to support it, then I'm sure we can find donors to provide
> the CPU cycles.  ChromeOS could probably steal whatever we put
> together so there is an obvious synergy there, and I'm sure
> Amazon/Rackspace/etc are always looking for use cases to show off.

Um, it was a bit of humor ,on a serious topic, which I support.


> From past feedback from Diego and such the biggest issue with any kind
> of tinderbox is dealing with the output.  As has been pointed out
> there are folks running Repoman regularly already, and there have been
> past tinderbox efforts.  The real issue is to improve the signal/noise
> ratio.  You'll only get so far with that using code - there has to be
> process change to support it.

Sound like you have the issues well conceptualized?


> If we were to do something like this long-term I'd envision that this
> would run on every commit or something like that, and the commit
> doesn't go into the rsync tree until it passes.  If the tree goes red
> then people stop doing commits until it is fixed, and somebody gets
> their wrist slapped.  That is my sense of how most mature
> organizations do CI.  The tinderbox is really just doing verification
> - stuff should already be tested BEFORE it goes in the tree.  There
> also shouldn't be any false positives.  There would need to be a
> mechanism to flag ebuilds with known issues so that the tinderbox can
> ignore them, and of course we can monitor that to ensure it isn't
> abused.

Are we talking about a database of know failures? Over time the
database would grow quite large, but be very useful?


> Basically doing this sort of thing right requires a big change in
> mindset.  You can't just throw stuff at the tree and wait for the bug
> reports to come in.  You can't just make dealing with the tinderbox
> the problem of the poor guy running the tinderbox.  The tinderbox
> basically has to become the boss and everybody has to make part of
> their job keeping the boss happy.

Ah, yes; leadership is a fleeting quality.
It always has been and always
will be..... fleeting


> Then you have to throw in uncooperative upstreams.  If you ARE the
> upstream then you can apply these practices yourself and have a build
> system that outputs zero warnings and all that.  We're dealing with
> what we get, so we're going to have to set the bar considerably lower
> if we don't want to exclude anything from the tree which has a lousy
> build system.

POINT very well acknowledged.

Perhaps if we build something that ordinary coders can use and some
folks with resources help those upstream application developers,
then a few will participate with us? This is a huge problem
for the scientific community. Because just because a large, complicated
algorithm runs, does not mean it is correct or robust. They need
tools built upon what your are designing for first the admins
and the secondly for the upstream code developers. Once something
is built for quality code check, and it is reasonable simple to
depoly, I think at the very least many users of those codes will
run these tools on open source code projects. Yes this is a huge effort,
that is sorely needed, imho.

We are all in this together.


> In any case, focus first on decent code with proposed procedures that
> help deal with the output.  Don't go spending a lot of money on
> hardware or worrying about setting up clusters and all that.  Of
> course, if you want to run stuff in parallel that is something you can
> design in from the start (and there is no reason you can't run it in a
> few VMs on your laptop - it will be slow but the focus is on ensuring
> that it works, not that it is fast).  

I totally agree. With excellent leadership, you can keep the secondaries
like me, in-line.  We do need a specification to begin with. Sure it will
evolve, but specs help keep focus and priortize what tools get worked on
and when, imho.

> In the past I've seen lots of
> people focus on building empires with dreams of hardware and all that,
> and they end up with little in the way of donations and if they do buy
> hardware they end up with nothing to run on it.  Write the code first
> and get it working at small scale.  If you write the code, the
> hardware will probably take care of itself.  People care about Gentoo,
> but they're not going to throw lots of equipment at somebody who has
> little more than an email chain to show up-front, and I doubt the
> Foundation is going to want to buy hardware before the code exists.

Easy on the coffee. Now you are jumping to too many conclusions. I agree
well need lots of thinking, planning and an overall robust architecture
design and lots of code. Hardware never is a problem, us EE's are
always way out front of the CS folk, imho......


And yes, let's get coding.......Um what's my homework assignment this
week?


> The code will be hard.  The culture change will be harder.  

And that is why YOU are on the council. My prayers and old fingers
are with you, brah!

> The  hardware will be easy.

Hardware is FUN. Everybody loves new tangible goodies, kids through
old folks. That's why my first degrees where in hardware
(fluids and electronics). I then realize that the microprocessor
is what animates the hardware and allows an EE to become lazy and
successful. So I studied CS, got an advanced degree and continue
to study, build and work on software. It never ends, so we must
"bond" together to get codes to be what we need them to be. I know
this assembler-->Haskel migration is hard on old dudes like me, it
is very, very hard. All I ever do is read and learn
new software tricks without ever mastering anything.....


> Bottom line - this sort of thing is a lot harder to pull off than it
> might first appear.  I don't want to discourage anybody, but don't
> think that you can toss together a few lines in bash to do an emerge
> -e world and you're done.  Of course, if you do want to do a manual
> run like this and manually submit bugs that have been manually
> screened for validity, that is always welcome.

For me, you build things and manually test them. Then you automate
what has worked manually. Sure it often  occurs simultaneously,
but things must be manual and partitionable to facilitate test and debug.


I think with ample thought, planing and a cool, inviting plan, we can get
many kids to participate in GSoC. If we "reach out" there are many other
folks at various distros that have a high level of need (frustration). The
gentoo rank and file experts may just be too busy, too burnt or
disinterested. I dunno. Certainly there is a lot of horsepower in our ranks.

I've had several grad students recently send me unsolicited email in search
of tough problems to solve. Surely we can add some algorithmic challenges,
centric on some of the newer languages to spice this up a bit for those
"hungry brilliant minds" too?

If Gentoo does not do this, Who will?

Who else can (lead) this sort of effort?

For whatever reason, I think this task has fallen on Gentoo.

With the work of LikeWahoa, we can build a usb gentoo image that has this
(GLEP) proposal, and the necessary tools to entire devs from other distros
to download and plug in the USB stick. WE get them addicted to Gentoo
via a live usb stick. They can hide, but it becomes easy to update
the usb stick *everybody* can use it to come up to speed quickly and
catch "gentoo fever". The fever allows folks to at least taste gentoo,
read a bit and send in codes on this GLEP.

I know many  vikings working on some subterranean physics all looking
seriously at gentoo. Gentoo on a usb stick allows for a liveGENTOO that can
be easily updated and they can jump in and out of Gentoo, until the fever
takes hold.


Or some other kind of hook?  Fishing was very difficult till the hook
was refined. WE need a hook, imho. Easy Gentoo and a project to assist
in testing code; you've got a killer idea there rich, and I'm glad to
be on your team!


James