[gentoo-user] Jenkins

[gentoo-user] Jenkins

[James]

Hello one and all,


I want to first install Jenkins on a single multicore amd system, so
I found this brief guide (which seems simple enough):

https://code.google.com/p/godin-gentoo-repository/wiki/Jenkins


What I also need is an example collection of code to run Jenkins-CI scripts
on, sort of as a benchmark. I figure it would be best to learn about
Jenkins by merely trying to follow/duplicate an existing (hopefully small)
Jenkins CI project. Basically a monkey-see-monkey-duplicate sort of endeavour.


This will  also allow me to then duplicate the Jenkins-CI onto the gentoo
mesos clusters (small right now) I have built to debug problems or issues
with Jenkins running on a gentoo-mesos-cluster. Ideas on how to perform
comparison (benchmarks) between the single multi-processor Jenkins-CI and
the one on top of a cluster, would be warmly received too.


After that is completed and stable, then I'm going to attempt to move
other codes to this gentoo-mesos-jenkins-CI framework.


Any suggestions are most welcome.

James

Re: [gentoo-user] Jenkins

[Alec Ten Harmsel]

On Sat, Jun 20, 2015 at 07:03:18PM +0000, James wrote:
> Hello one and all,
> 
> 
> I want to first install Jenkins on a single multicore amd system, so
> I found this brief guide (which seems simple enough):
> 
> https://code.google.com/p/godin-gentoo-repository/wiki/Jenkins
> 

I highly recommend playing with the Jenkins docker container to get a
feel for using the web UI:

    docker run -p 8080:8080 -d jenkins

The official jenkins image comes with the bare minimum of libraries so
good luck building any C/C++ project, but it is helpful just to point
and click around (if you already have docker, that is).

> 
> What I also need is an example collection of code to run Jenkins-CI scripts
> on, sort of as a benchmark. I figure it would be best to learn about
> Jenkins by merely trying to follow/duplicate an existing (hopefully small)
> Jenkins CI project. Basically a monkey-see-monkey-duplicate sort of endeavour.
> 
> 
> This will  also allow me to then duplicate the Jenkins-CI onto the gentoo
> mesos clusters (small right now) I have built to debug problems or issues
> with Jenkins running on a gentoo-mesos-cluster. Ideas on how to perform
> comparison (benchmarks) between the single multi-processor Jenkins-CI and
> the one on top of a cluster, would be warmly received too.
> 

This is not really a fair comparison. Jenkins has no concept of cores,
only "executors". Each executor can run one build at a time. Jenkins was
written with Java in mind (AFAIK), and Java build tools are the reason
for this. `javac` by default launches some number of threads (not sure
how many on other systems, but on my laptop it's usually 3) and compiles
in parallel. I'm not sure it's possible to control this (someone feel
free to correct me if I'm wrong), hence Jenkins does not care about
cores and only about how many builds are allowed to run in parallel at
once.

This is different than make and just other build tools in general that
I've worked with for other languages.

So, to the point; when you add more build slaves with more executors,
you will be able to run more builds in parallel and it will therefore be
faster. This gets even more complicated since you can constrain builds
to only run on certain hosts and various other complicated setups, but
in general more executors means more builds in parallel means faster.

> 
> After that is completed and stable, then I'm going to attempt to move
> other codes to this gentoo-mesos-jenkins-CI framework.
> 
> 
> Any suggestions are most welcome.
> 
> James
> 
> 

My main suggestion is to not bother running a Jenkins cluster and stick
with a single host setup unless:

* You are building so many things that utilization is near 100%
* You are targeting multiple platforms and therefore need multiple build
  hosts

Alec

P.S. I could not find any reference to parallel, threads, processes or
anything else on javac's man page that makes me think it is easy to set
the number of threads it uses.

[gentoo-user] Re: Jenkins

[James]

Alec Ten Harmsel <alec <at> alectenharmsel.com> writes:


> On Sat, Jun 20, 2015 at 07:03:18PM +0000, James wrote:
> > Hello one and all,

> > I want to first install Jenkins on a single multicore amd system, so
> > I found this brief guide (which seems simple enough):
 
> > https://code.google.com/p/godin-gentoo-repository/wiki/Jenkins

> I highly recommend playing with the Jenkins docker container to get a
> feel for using the web UI:

>     docker run -p 8080:8080 -d jenkins

> The official jenkins image comes with the bare minimum of libraries so
> good luck building any C/C++ project, but it is helpful just to point
> and click around (if you already have docker, that is).

Yes, I've used Docker on gentoo before. NO, it's currently un-installed
and using docker for what I am after only complicates things at this point.


> This is not really a fair comparison. Jenkins has no concept of cores,
> only "executors". Each executor can run one build at a time. Jenkins was
> written with Java in mind (AFAIK), and Java build tools are the reason
> for this. `javac` by default launches some number of threads (not sure
> how many on other systems, but on my laptop it's usually 3) and compiles
> in parallel. I'm not sure it's possible to control this (someone feel
> free to correct me if I'm wrong), hence Jenkins does not care about
> cores and only about how many builds are allowed to run in parallel at
> once.

I'll have to drill into javac a bit more, it seems.

There is jenkins-bin on gentoo; it should (?) be a quick install and config.
I'll keep in mind what you are saying, but my main goal is to find (CI)
codes that I can run on gentoo-mesos clusters. Nothing about Jenkins is
a critical need for me. Although, as a consultant I do routinely get
asked about Jenkins experience, so just noodling around with it a bit
is a good idea for me.


> So, to the point; when you add more build slaves with more executors,
> you will be able to run more builds in parallel and it will therefore be
> faster. This gets even more complicated since you can constrain builds
> to only run on certain hosts and various other complicated setups, but
> in general more executors means more builds in parallel means faster.

AT some point, on a single multiprocessor system, Jenkins will slow down
even it more resources are configured for it (overall system load, if
nothing else)? AT that point, I can note the resources and apply those
limits to the cluster and note the performance differences. Surely, a
cluster with more resources will be able to do more (CI_Jenkins) work; so
discovering those details is a primary goal of this endeavour.



> My main suggestion is to not bother running a Jenkins cluster and stick
> with a single host setup unless:

Sorry, the cluster, is the main_goal, not Jenkins or CI. Buds at cisco
are all very braggadocios about CI (Jenkins and others) on their mesos
clusters.....

> * You are building so many things that utilization is near 100%
> * You are targeting multiple platforms and therefore need multiple build
>   hosts

YES, that is exactly the ultimate goal. A CI cluster, open to friends, pals
and customers (maybe). It's all about the mesos cluster, you should know that.

> P.S. I could not find any reference to parallel, threads, processes or
> anything else on javac's man page that makes me think it is easy to set
> the number of threads it uses.


No surprise there. Configuring threads, cgroups, and memory resources
is a hotly contested area of the cluster codes I have (am) working with.
Forget bikeshedding, it Blood_shedding....... especially now that RDMA
is hotly being pursued by the many.

Thanks for your insights,

James