[gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[Robert Larson]

Hello,

I was wondering if anyone has information on a good method to centrally build 
and push packages out to machines with Gentoo.  I have multiple machines 
setup under multiple processor types, and up until now I have been executing 
upgrades on the machines themselves.  Since these are production machines, I 
would like to set up another machine that acts as a compile/install machine.  

This might be compareable to a "gold server", as described at 
infrastructures.org.  I don't suppose I could get around this using distcc?

Configuration management is another issue.  As it is, I have cfengine setup, 
though it is not setup to work with portage and I can't really see these two 
coming together nicely.  Maybe I could create a patch for dispatch-conf...

Mind you, I have no idea on where to even start on something like this, so any 
thoughts, ideas, or information would be helpful.

Thanks!

Robert
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[Christian Heim]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Wednesday 19 October 2005 17:25, Robert Larson wrote:
> Hello,
>
> I was wondering if anyone has information on a good method to centrally
> build and push packages out to machines with Gentoo.  I have multiple
> machines setup under multiple processor types, and up until now I have been
> executing upgrades on the machines themselves.  Since these are production
> machines, I would like to set up another machine that acts as a
> compile/install machine.
>
> This might be compareable to a "gold server", as described at
> infrastructures.org.  I don't suppose I could get around this using distcc?
Nope, distcc is only for distributed compiling ...

You probably want to use some chroots and setup some sort of http/ftp-site for 
that packages which the gold-servers could fetch then ... (via the BINPKG 
features of portage, see man emerge / man make.conf for details)

> Thanks!
>
> Robert

Good luck with that.

Christian
-- 
Christian Heim <phreak@gentoo.org>
Gentoo Linux Developer (VServer)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[Michael Crute]

[-- Attachment #1: Type: text/plain, Size: 1841 bytes --]

On 10/19/05, Robert Larson <robert@sixthings.com> wrote:
>
> Hello,
>
> I was wondering if anyone has information on a good method to centrally
> build
> and push packages out to machines with Gentoo. I have multiple machines
> setup under multiple processor types, and up until now I have been
> executing
> upgrades on the machines themselves. Since these are production machines,
> I
> would like to set up another machine that acts as a compile/install
> machine.
>
> This might be compareable to a "gold server", as described at
> infrastructures.org <http://infrastructures.org>. I don't suppose I could
> get around this using distcc?
>
> Configuration management is another issue. As it is, I have cfengine
> setup,
> though it is not setup to work with portage and I can't really see these
> two
> coming together nicely. Maybe I could create a patch for dispatch-conf...
>
> Mind you, I have no idea on where to even start on something like this, so
> any
> thoughts, ideas, or information would be helpful.
>
>
You could use one server to build binary packages and then emerge them on
the systems that need the upgrades, saves a lot of resources over running
the compile on each machine. Not sure how to handle the multiple archs
however, perhaps you could write some scripts to "change out" your build
environment on the build machine and re-build the packages for each arch.
That approach should be as simple as changing out your /etc/make.conf file
if all the servers use the same build profile otherwise it's going to
involve a bit more work.

-Mike

--
________________________________
Michael E. Crute
Software Developer
SoftGroup Development Corporation

Linux, because reboots are for installing hardware.
"In a world without walls and fences, who needs windows and gates?"

[-- Attachment #2: Type: text/html, Size: 2233 bytes --]

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[xyon]

I was thinking of that, too, but what about the cascade effect of packages
being built for a specific arch? Like say, GCC on the build box was built
against different CFLAGS than what Apache is being built against?

Am I just splitting hairs here?

I'm getting ready to implement two production Gentoo servers and have a
box with a different arch. Maybe the secondary production server with the
exact same arch as the primary would be a better test bed?


On Wed, October 19, 2005 11:50, Michael Crute wrote:
> On 10/19/05, Robert Larson <robert@sixthings.com> wrote:
>>
>> Hello,
>>
>> I was wondering if anyone has information on a good method to centrally
>> build
>> and push packages out to machines with Gentoo. I have multiple machines
>> setup under multiple processor types, and up until now I have been
>> executing
>> upgrades on the machines themselves. Since these are production
>> machines,
>> I
>> would like to set up another machine that acts as a compile/install
>> machine.
>>
>> This might be compareable to a "gold server", as described at
>> infrastructures.org <http://infrastructures.org>. I don't suppose I
>> could
>> get around this using distcc?
>>
>> Configuration management is another issue. As it is, I have cfengine
>> setup,
>> though it is not setup to work with portage and I can't really see these
>> two
>> coming together nicely. Maybe I could create a patch for
>> dispatch-conf...
>>
>> Mind you, I have no idea on where to even start on something like this,
>> so
>> any
>> thoughts, ideas, or information would be helpful.
>>
>>
> You could use one server to build binary packages and then emerge them on
> the systems that need the upgrades, saves a lot of resources over running
> the compile on each machine. Not sure how to handle the multiple archs
> however, perhaps you could write some scripts to "change out" your build
> environment on the build machine and re-build the packages for each arch.
> That approach should be as simple as changing out your /etc/make.conf file
> if all the servers use the same build profile otherwise it's going to
> involve a bit more work.
>
> -Mike
>
> --
> ________________________________
> Michael E. Crute
> Software Developer
> SoftGroup Development Corporation
>
> Linux, because reboots are for installing hardware.
> "In a world without walls and fences, who needs windows and gates?"
>


-- 
Steven McCoy
Site Development/Manager
IndigoRobot Services
http://www.indigorobot.com
mailto:stevenmccoy@indigorobot.com

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[theboywho]

> Nope, distcc is only for distributed compiling ...
>
> You probably want to use some chroots and setup some sort of http/ftp-site
> for that packages which the gold-servers could fetch then ... (via the
> BINPKG features of portage, see man emerge / man make.conf for details)

Sorry if this is a little unclear...

I have used the multiple chroot method, and although it uses lots of disk 
space, i found it the best way to do it.

On the 'gold' server I had multiple chroots, each of which mounted the local 
'/usr/portage', but then mounted unique 'packages' directories over it. 

The multiple arch-depandent packages directories and the global portage 
directories are exported via NFS.

Each of the production machines then mounts the global portage via NFS and 
then their own arch-dependant 'packages' directory over it.

If you would like more information, I would be happy to help.

>
> > Thanks!
> >
> > Robert
>
> Good luck with that.
>
> Christian

-- 
Parmi
theboywho@ruddyperl.com
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[Ian P. Christian]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

On Wednesday 19 October 2005 20:15, theboywho wrote:
> If you would like more information, I would be happy to help.

Absolutly fantastic, I've been meaning to sort this for ages.

I myself would *love* to see this expanded upon.  I could setup a wiki for 
this, or the gentoo wiki might be the best place to do it.  Or, if you like, 
just pencil it out in a tiny bit more detail and I'll document it, I need to 
do this for my company. 

Questions: do you use cron? how do you report when servers need new packages? 
so you automatically install packages? how do you handle restarting services 
on the servers? do you have different profiles like... p4-webserver, 
amd64-mailserver etc...  etc. etc.

Kind Regards,

-- 
Ian P. Christian ~ http://pookey.co.uk

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[theboywho]

On Wednesday 19 October 2005 22:22, Ian P. Christian wrote:
> On Wednesday 19 October 2005 20:15, theboywho wrote:
> > If you would like more information, I would be happy to help.
>
> Absolutly fantastic, I've been meaning to sort this for ages.
>
> I myself would *love* to see this expanded upon.  I could setup a wiki for
> this, or the gentoo wiki might be the best place to do it.  Or, if you
> like, just pencil it out in a tiny bit more detail and I'll document it, I
> need to do this for my company.

I guess the gentoo-wiki would be the best place.
>
> Questions: do you use cron? how do you report when servers need new
> packages? so you automatically install packages? how do you handle
> restarting services on the servers? do you have different profiles like...
> p4-webserver, amd64-mailserver etc...  etc. etc.
>
> Kind Regards,

In regards to doing it automatically - I never did that. I have portage update 
via cron, and then email me a list of the packages that needed updating (i.e. 
the output of 'emerge -uvp world'. 

I keep separate chroots on my desktop machine for all the other machines I 
have in /mnt/chroots/<machinename> then another directory for the separate 
packages folder /mnt/packages/<machinename>

I do the package updates manually in each chroot with the 'buildpkgs' feature 
turned on, and then on the actual machines mount 
the /mnt/packages/<machinename> directory over /usr/portage/packages using 
NFS.

The automating this is something that I would be willing to look in to.

Hope that helps!

-- 
parmi
theboywho@ruddyperl.com
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[Andrea Barisani]

On Wed, Oct 19, 2005 at 10:25:06AM -0500, Robert Larson wrote:
> Hello,
> 
> I was wondering if anyone has information on a good method to centrally build 
> and push packages out to machines with Gentoo.  I have multiple machines 
> setup under multiple processor types, and up until now I have been executing 
> upgrades on the machines themselves.  Since these are production machines, I 
> would like to set up another machine that acts as a compile/install machine.  
> 
> This might be compareable to a "gold server", as described at 
> infrastructures.org.  I don't suppose I could get around this using distcc?
> 
> Configuration management is another issue.  As it is, I have cfengine setup, 
> though it is not setup to work with portage and I can't really see these two 
> coming together nicely.  Maybe I could create a patch for dispatch-conf...
> 
> Mind you, I have no idea on where to even start on something like this, so any 
> thoughts, ideas, or information would be helpful.
> 
> Thanks!
> 
> Robert
> -- 
> gentoo-server@gentoo.org mailing list
>

http://article.gmane.org/gmane.linux.gentoo.sysadmin/65

-- 
Andrea Barisani <lcars@gentoo.org>                            .*.
Gentoo Linux Infrastructure Developer                          V
                                                             (   )
GPG-Key 0x864C9B9E http://dev.gentoo.org/~lcars/pubkey.asc   (   )
    0A76 074A 02CD E989 CE7F AC3F DA47 578E 864C 9B9E        ^^_^^
      "Pluralitas non est ponenda sine necessitate"
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[Dan Podeanu]

Interesting topic. My solution (I finished implementing it several months
ago) for centralized
management is something like this:

Definitions:

Gentoo snapshot: a image of a Gentoo installation, that deployed on a server
will allow
booting of a running system.
Blade root: a server designated at one time as booting server for others.

Objectives:

1. Low maintenance costs: maintaining and applying patches to a single build
(Gentoo snapshots).
2. Low scalability overhead: scalability should be part of the design, it
should not take more than
10 minutes per server to scale up.
3. Redundancy: Permanent hardware failure of N-1 out of N nodes, or
temporary failure (power off)
of all nodes should allow fast (10 minutes) recovery of all nodes in a
cluster.

Restrictions:

1. Single CPU architecture: I consider the cost of maintaining several
architectures to be bigger than
the cost of purchasing a single architecture.

2. Unified packages tree: I consider the cost of maintaining several Gentoo
snapshots just to have
deployed the minimum of packages per server assigned to a specific
application (mail server, web
server etc.) to be bigger than having a common build with all packages and
just starting the required
services (ie. all deployed servers have a both a MTA and Apache installed,
just that web servers
have Apache started, and the mail servers have it stopped and MTA running
instead).

3. An application that can act as a cluster with transparent failover (web
with balancer and health
checking, multiple MX servers, etc.)

4. A remote storage for persistent data (like logs) helps (you will see
why); you can modify the
partitioning or harddisk configuration to maintain a stable filesystem on
individual servers.

Hardware:

1. AMD Opteron blades: 2x Opteron, 4-12gb ram, 1 SATA HDD. Reasons for
choosing:
   - Opteron is cheaper, faster than Xeon, we can replace single cores with
dual cores at any point
     without generating too much heat.

   - 4-12gb RAM & SATA: I prefer the OS to cache a lot of things in RAM to
speed things up, as opposed
     to getting a little ram and expensive SCSI. Harddisks have too many
moving parts and generate a lot
     of heat, which is a problem in a dense CPU environment such as blades
(and you can't trust your
     datacenter to really cool things). And RAM is cheap nowadays anyway.

2. Gigabit network cards with PXE

Software:

One initial server (blade root) is installed with Gentoo. On top of that, in
a directory, another
Gentoo is installed (Gentoo snapshot) that will be replicated on individual
servers as further described,
and all maintenance to the snapshot is done in chroot.

The Blade root runs DHCP and tftp and is able to answer PXE dhcp/tftp
requests (for network boot) and serve
an initial bootloader (grub 0.95 with diskless and diskless-undi patches to
allow detection of Broadcom NICs),
along with an initial initrd filesystem.

The Gentoo snapshot contains all the packages required for all applications
(roughly 2gb on our systems),
along with dhcp/tftp and configs, to allow it to act as Blade root.

In addition, the Blade root contains individual configurations for every
individual deployed server (or, rather,
only changes to the standard Gentoo config, ie. per-blade IPs, custom
application configs, different configuration
for services to start as boot, etc.)

The Gentoo snapshot is compressed as tar.gz in two archives, one with
'running code' (ie. /usr, /bin, etc.)
and another one with additional things we don't really need on every server
(like portage, usr/src, manpages, etc.).
The collection of scripts for compressing everything, initrd, along with all
individual blade configurations and misc
scripts are archived in a 3rd archive.

Booting takes place like this:

The Blade root is powered on and ready to answer PXE.

Blade boots, uses PXE to get an IP via DHCP, an initial grub image via tftp,
that subsequently downloads
the grub configuration file via dhcp.

Boot menu gets displayed, with a default and timeout. Grub downloads the
Linux kernel via tftp, along with
the initrd image. The kernel boots, mounts initrd, executes /linuxrc script.

The initrd contains busybox, a rsync client, fdisk/tar/gzip. When it starts,
it downloads the Gentoo snapshot from the
Blade root, along with the blade configuration and archiving scripts archive
(via rsync), uses fdisk to recreate a
partition table, creates a filesystem, uncompresses the blade configuration
in the target / partition, changes the root
and exec's init, thus booting Gentoo. At the end of Gentoo bootup, grub is
ran locally to install a bootloater to allow
booting in case the Blade root server is unavailable, and the services
required for the particular application the
blade is intended to are started.

The result is a server that has been booted remotely and is an exact image
of the unique blade source. It also contains
everything needed to boot by itself, and everything needed to further boot
other servers. Thanks to the individual
per-server configuration, only services meant for that machine are started,
and the bootup only takes 3-4 minutes more
than usual.

After booting one blade, using it as source to boot the initial Blade root
ensures all servers involved share the
same setup, thus a self-replicating system.

Maintenance costs are related only to updating the Blade root's blade
snapshot. After an emerge -u world,
rebooting the other blades one at a time will distribute the changes. On top
of this you can have whatever
synchronization method you want for the particular application you're
deploying.

I hope this helps.

Cheers,
Dan.


----- Original Message ----- 
From: "Ian P. Christian" <pookey@pookey.co.uk>
To: <gentoo-server@lists.gentoo.org>
Cc: "theboywho" <theboywho@ruddyperl.com>
Sent: Thursday, October 20, 2005 12:22 AM
Subject: Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to
multiple architectures


-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Centralized Gentoo (build -> push/pull) to multiple architectures

[Ramon van Alteren]

Hi Dan,

On 20 Oct , 2005, at 2:41 AM, Dan Podeanu wrote:
> Interesting topic.

Indeed, I'm moving to a different employer and was considering a  
similar setup...
I'm curious about a number of things:

What's the scale of the cluster you're using this setup on ?
Would you be willing / able to share some of the work ?
I'd be very interested to look at your setup before I start my own.

Any comments on the hardware stability of the nodes you're using?
Which make blades are you using?

I was also wondering whether you are familiar with the work of http:// 
www.infrastructures.org/
Your setup has many of it's characteristics.

> Objectives:
>
> 1. Low maintenance costs: maintaining and applying patches to a  
> single build
> (Gentoo snapshots).
> 2. Low scalability overhead: scalability should be part of the  
> design, it
> should not take more than 10 minutes per server to scale up.
> 3. Redundancy: Permanent hardware failure of N-1 out of N nodes, or
> temporary failure (power off) of all nodes should allow fast (10  
> minutes) recovery of all nodes in a
> cluster.

I read below that all nodes include configs for dhcp/tftp in order to  
be able to take over the golden (blade root) server. How do you  
handle that? In case of downtime of the main blade root server which  
of the nodes gets to take over? Is that an automatic or a manual  
process?

Additionally, did you test a all node failure and how did the master  
blade root cope with the strain of all nodes booting at once? What  
hardware are you using for the blade root server ?

> Restrictions:
>
> 1. Single CPU architecture: I consider the cost of maintaining several
> architectures to be bigger than the cost of purchasing a single  
> architecture.

Are you running a full 64-bit setup or 32-bit compatibility mode ?
What are your experiences with stability in 64-bit case ? Especially  
curious about php and it's diverse set of external libs. Do agree  
though, any thoughts on the inevitable upgrade that's going to show  
up some time in the future when your current hardware platform is no  
longer available ?

> 2. Unified packages tree: I consider the cost of maintaining  
> several Gentoo
> snapshots just to have deployed the minimum of packages per server  
> assigned to a specific
> application (mail server, web server etc.) to be bigger than having  
> a common build with all packages and just starting the required  
> services (ie. all deployed servers have a both a MTA and Apache  
> installed, just that web servers have Apache started, and the mail  
> servers have it stopped and MTA running instead).

Agreed, doesn't pay off to have seperate base-sets for the different  
type of nodes, and it's good on redundancy, if needed a former  
webserver can stand in as a database server etc..

> 3. An application that can act as a cluster with transparent  
> failover (web
> with balancer and health checking, multiple MX servers, etc.)

I don't understand this restriction?

> 4. A remote storage for persistent data (like logs) helps (you will  
> see why);
> you can modify the partitioning or harddisk configuration to  
> maintain a stable filesystem on individual servers.

<snipped>

> Software:
>
> One initial server (blade root) is installed with Gentoo. On top of  
> that, in
> a directory, another Gentoo is installed (Gentoo snapshot) that  
> will be replicated on individual
> servers as further described, and all maintenance to the snapshot  
> is done in chroot.
>
> The Blade root runs DHCP and tftp and is able to answer PXE dhcp/tftp
> requests (for network boot) and serve an initial bootloader (grub  
> 0.95 with diskless and diskless-undi patches to allow detection of  
> Broadcom NICs), along with an initial initrd filesystem.
>
> The Gentoo snapshot contains all the packages required for all  
> applications
> (roughly 2gb on our systems), along with dhcp/tftp and configs, to  
> allow it to act as Blade root.

See question above, is switching manual ?

> In addition, the Blade root contains individual configurations for  
> every
> individual deployed server (or, rather, only changes to the  
> standard Gentoo config, ie. per-blade IPs, custom application  
> configs, different configuration for services to start as boot, etc.)

Do you use classes here (e.g. webserver, databaseserver, mailserver,  
cachingserver etc.)?
Or do you maintain individual setups for each server?
What scripting language did you choose for the config scripts and  
stuff and why that script lang?

<booting process snipped>

I'm also curious as to what QA procdures you have in place to prevent  
accidental mistakes on the blade root server. I assume you test  
beforehand ? On all server classes ? Modifications to the third  
archive with the per-server config seem rather difficult to test.

> I hope this helps.

Oh it sure did, it confirmed some ideas i was already thinking about  
and gave me a real world example that it can be done :-)

Thanks,

Ramon
--
Change what you're saying,
Don't change what you said

The Eels



-- 
gentoo-server@gentoo.org mailing list