[gentoo-server] Best practices in managing large server groups

[gentoo-server] Best practices in managing large server groups

[Charles Duffy]

[Apologies if this is a duplicate post; I tried posting this through
GMANE yesterday without subscribing first, and it appears not to have
gone through; likewise when I posted several hours ago today from a 
different address than the one I subscribed].

All,

I'm looking at replacing SuSE SLES9 with Gentoo for an enterprise
application (for reasons of flexibility and licensing) (no, we don't
have an enterprise application budget -- just the reliability
requirements; yaaay, startups!). We're looking to be able to deploy and
manage hundreds of geographically distributed servers.

We have a QA department available to vet each configuration before it is
deployed to the field. We have infrastructure for tracking the progress
of code in svn from creation though QA to deployment; I'm anticipating
tracking a local overlay (containing all packages we use), make.conf,
/etc/portage/*, etc. through this system, autobuilding system images
(either to run virtualized or on real hardware) from the contents of
svn, building binary packages and deploying them to real hardware.

I'm interested in best practices, suggested tools, and/or 3rd party
experiences in this regard.

Some particular questions which come to mind:
  - Should I be using a custom profile or a standard profile with
overrides through make.conf, /etc/portage/* and the like?
  - What's the Right Way to create new system images ready to be loaded
onto a hard drive or run through a virtual machine? gentoo-buildhoster
looks interesting. I've seen Catalyst mentioned as a way to create
stage3 images, but what documentation I've been able to find doesn't
seem very much targeted for my use cases.
  - Any experiences with puppet? With out ratio of servers to staff,
automating configuration and administration is a priority. (We already
have an internal tool written with automating the server configuration
process in mind; it has some functionality puppet doesn't, and puppet
has functionality it doesn't; in theory, I'd like to extend puppet until
our internal tool becomes unnecessary, though I'll need to understand
puppet much better before I can think too hard about that).
  - Have any of 'yall been in the 100s-of-servers situation with
comparable requirements and come up with a different approach to
managing it? How did things work out?

Thank you!


-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Nicolas MASSE]

Quoting Charles Duffy <cduffy@spamcop.net>:

> [Apologies if this is a duplicate post; I tried posting this through
> GMANE yesterday without subscribing first, and it appears not to have
> gone through; likewise when I posted several hours ago today from a
> different address than the one I subscribed].
>
> All,
>
> I'm looking at replacing SuSE SLES9 with Gentoo for an enterprise
> application (for reasons of flexibility and licensing) (no, we don't
> have an enterprise application budget -- just the reliability
> requirements; yaaay, startups!). We're looking to be able to deploy and
> manage hundreds of geographically distributed servers.
>
> We have a QA department available to vet each configuration before it is
> deployed to the field. We have infrastructure for tracking the progress
> of code in svn from creation though QA to deployment; I'm anticipating
> tracking a local overlay (containing all packages we use), make.conf,
> /etc/portage/*, etc. through this system, autobuilding system images
> (either to run virtualized or on real hardware) from the contents of
> svn, building binary packages and deploying them to real hardware.
>
> I'm interested in best practices, suggested tools, and/or 3rd party
> experiences in this regard.

I have absolutely no experience with this, but during one of my  
training courses, I found the following document :
  - http://www.fernhilltec.com.au/~adrian/kara/kara.html

Maybe, it could give some ideas...

Nicolas.


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

--
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Charles Duffy]

Nicolas MASSE wrote:
> I have absolutely no experience with this, but during one of my 
> training courses, I found the following document :
>  - http://www.fernhilltec.com.au/~adrian/kara/kara.html

Thank you for the pointer; the document was worth a readthrough. That 
said, puppet provides a superset of kara's capabilities, is very 
actively maintained, and supports Gentoo as a target platform. 
Obviously, I'm looking for feedback here -- so if someone has tried 
working with puppet and found it inadequate it some respect or in need 
of further extensions, I'd love to hear about it -- but I think I'm 
fairly comfortable with my selection on that particular piece of the 
toolage.

Initial system setup (as opposed to post-install maintenance) is 
somewhat more of an open question, particularly in the context of 
desiring complete reproducibility.
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Ramon van Alteren]

Charles Duffy wrote:
> I'm looking at replacing SuSE SLES9 with Gentoo for an enterprise
> application (for reasons of flexibility and licensing) (no, we don't
> have an enterprise application budget -- just the reliability
> requirements; yaaay, startups!). We're looking to be able to deploy and
> manage hundreds of geographically distributed servers.
We're using gentoo and managed to scale from 1 server to 600servers.
>
> Some particular questions which come to mind:
>  - Should I be using a custom profile or a standard profile with
> overrides through make.conf, /etc/portage/* and the like?
We currently use a standard profile and override if necessary.
>  - What's the Right Way to create new system images ready to be loaded
> onto a hard drive or run through a virtual machine? gentoo-buildhoster
> looks interesting. I've seen Catalyst mentioned as a way to create
> stage3 images, but what documentation I've been able to find doesn't
> seem very much targeted for my use cases.
We''re using a combination of catalyst, pxe and the cli-installer from 
agaffney which we extended to support our configuration database.
catalyst stage4 works fine for us and enables us to repeat build our 
system images. Additionally we use the packages generated by catalyst as 
input for our
binary package server. We're currently looking into the tinderbox target 
to use catalyst to validate new packages on existing system images.

After installing we use a grub config file which allows us to reinstall 
over pxe very easily.
>  - Any experiences with puppet? With out ratio of servers to staff,
> automating configuration and administration is a priority. (We already
> have an internal tool written with automating the server configuration
> process in mind; it has some functionality puppet doesn't, and puppet
> has functionality it doesn't; in theory, I'd like to extend puppet until
> our internal tool becomes unnecessary, though I'll need to understand
> puppet much better before I can think too hard about that).
Puppet works fine for us, we used to run our own scripts for 
configuration stuff but that grew too complex and hard to extend.
We're currently converting to puppet and actually just finished 
converting our web farm to puppet-controlled hosts.
Stability is sometimes an issue and we are actively looking for 
scalability issues with our current setup but overall we're extremely 
happy with the tool.

Ramon
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Ronan Mullally]

On Sun, 20 May 2007, Ramon van Alteren wrote:

> We''re using a combination of catalyst, pxe and the cli-installer from
> agaffney which we extended to support our configuration database.
> catalyst stage4 works fine for us and enables us to repeat build our
> system images. Additionally we use the packages generated by catalyst as
> input for our binary package server. We're currently looking into the
> tinderbox target to use catalyst to validate new packages on existing
> system images.

To add another string to this thread...

I've been using Gentoo on a small number of machines for several years.
Each server is managed individually with its own portage tree, etc.  None
of the boxes are true production boxes.

I've held off on using Gentoo in larger deployments because the idea of
putting a C compiler on a production box is just silly.  I've been meaning
to get to the bottom of this for a while, but never had the time
or incentive.

Does a multi-server gentoo install require a portage tree and gcc on
each box?  I know you can configure a BINHOST server in make.conf, but
that won't stop the install process requiring gcc as part of the profile.
Injecting a dummy stub for gcc will probably get around that, but I can't
see how removing the portage tree is possible.  Granted, in a large
deployment that's not really an issue, /usr/portage can be NFS-mounted
from a central source.  In a more distributed environment however (central
management server, widely distributed production servers) that's not
ideal.

I suspect they way around the gcc question is to use a profile which
doesn't have gcc as a base system dependancy.  Does such a profile exist,
or is does one have to roll their own?  Is there a way to run gentoo
without a portage tree on each box?


-Ronan
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Thilo Bangert]

[-- Attachment #1: Type: text/plain, Size: 245 bytes --]

> I've held off on using Gentoo in larger deployments because the idea of
> putting a C compiler on a production box is just silly.

why?

some production quality software _requires_ a compiler to run.
http://varnish.projects.linpro.no/wiki/FAQ

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-server] Best practices in managing large server groups

[Ronan Mullally]

On Mon, 21 May 2007, Thilo Bangert wrote:

> > I've held off on using Gentoo in larger deployments because the idea of
> > putting a C compiler on a production box is just silly.
>
> why?
>
> some production quality software _requires_ a compiler to run.
> http://varnish.projects.linpro.no/wiki/FAQ

As do many rootkits.  If somebody gets local access to a server with a
suite of development tools they're well on their way to rooting the box.
Removing these tools is simply a good example of security in depth.


-Ronan
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Arturo 'Buanzo' Busleiman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Ronan Mullally wrote:
> As do many rootkits.  If somebody gets local access to a server with a
> suite of development tools they're well on their way to rooting the box.
> Removing these tools is simply a good example of security in depth.

You just slow the attack a little bit by removing the compiler. The attacker will probably use
statically linked binaries, or compile somewhere else. Most rootkits do not depend on external
libraries, neither, except for kernel modules, of course, that depend on the kernel's source.

But, of course, if they got access to the box, then the compiler is the least of your problems at
that time, but I have to admit that the "slowing the attacker down" is an extra layer of protection.
It provides the sysadmins/users/monitoring software more time to detect the breach.

- --
Arturo "Buanzo" Busleiman - Consultor Independiente en Seguridad Informatica
OpenPGP for HTTP: New Web-Auth Scheme: http://freshmeat.net/articles/view/2599
Consulting and Secure Mail Hosting: http://www.buanzo.com.ar/pro/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGUa54AlpOsGhXcE0RCv2JAJ9FBW3UVp/LHa0utGFAcjSoD94fVwCeINiK
94XbD11OieY31dQM6M4/URY=
=4HBQ
-----END PGP SIGNATURE-----
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Ronan Mullally]

On Mon, 21 May 2007, Arturo 'Buanzo' Busleiman wrote:

> But, of course, if they got access to the box, then the compiler is the
> least of your problems at that time, but I have to admit that the
> "slowing the attacker down" is an extra layer of protection. It provides
> the sysadmins/users/monitoring software more time to detect the breach.

Removing development tools is one of just a range of security measures.
It is by no means the be-all and end-all of security - hence my mention of
security in depth.  There are easily a couple of dozen security measures I
implement when installing a box.  Removing development tools (or in most
cases, never installing them) is just one of them, but we diverge...

Back to the original question.  How do I run a number of gentoo boxes
without gcc (or a portage tree)?


-Ronan
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Arturo 'Buanzo' Busleiman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Ronan Mullally wrote:
> Back to the original question.  How do I run a number of gentoo boxes
> without gcc (or a portage tree)?

WIll sound like a joke, but my answer would be "without running gentoo". It's a source-based distro.
There are some binary distros that are created from portage (sadly, I can only recall the
argentinian one, Ututo, but I don't know if it is alive).

- --
Arturo "Buanzo" Busleiman - Consultor Independiente en Seguridad Informatica
OpenPGP for HTTP: New Web-Auth Scheme: http://freshmeat.net/articles/view/2599
Consulting and Secure Mail Hosting: http://www.buanzo.com.ar/pro/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGUbRAAlpOsGhXcE0RCtPXAJ4jkTrMY1FXmHgOFsZDOnHfnSVvVACdG2wO
ld4jsx8vexRI4yJoKCFWvy4=
=PfXB
-----END PGP SIGNATURE-----
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Karl Holz]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

On 21-May-07, at 5:04 AM, Ronan Mullally wrote:

> Does a multi-server gentoo install require a portage tree and gcc on
> each box?

No,  you can use the following command:

# ROOT="/chroot" emerge <pkages you want to install> -pv

this will allow you build a new system, and it wount install the gcc  
packages. all you'll need is a stage 3 tarball for building, edit  
the /etc/make.conf to what ever setting you desire. Just keep in mind  
you will need to install all the programs needed for boot: base- 
system, grep, openssh, pam, core-utills, etc.


> Is there a way to run gentoo without a portage tree on each box?

  yes, if  you setup a build system, using a stage3 tarball, and  
build your system into a directory. Portage will only be under your / 
usr/portage and not into the system image you're building. the good  
thing about using a Stage3 tarball is you can build you system on any  
linux system, build your system image, tarball the image, deploy and  
install grub on x86, yaboot on Mac PPC, silo on Sparc64.


__________________
Karl Gustav B. Holz
----------
karl@new-aeon.com
----------
http://www.new-aeon.com



[-- Attachment #2: Type: text/html, Size: 4030 bytes --]

Re: [gentoo-server] Best practices in managing large server groups

[Christian Bricart]

Arturo 'Buanzo' Busleiman wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Ronan Mullally wrote:
>> Back to the original question.  How do I run a number of gentoo boxes
>> without gcc (or a portage tree)?
>
> WIll sound like a joke, but my answer would be "without running gentoo".
> It's a source-based distro.
> ..

nonsense - of course you're able to maintain your own $BINHOST which
compiles the packages (--buildpkg[only]) and have other servers in your
farm that have the same $arch and USE-flags and use "emerge
--{get,use}binpkg .." on them

http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=3#doc_chap4

then actually you only need "tar" and "b[un]zip2" on the target.

remark: add those keys to $FEATURES in /etc/make.conf and tweak the
servers' repository links to your local buildhost(s) and Portage
mirrors...

Works like charm here - I've got a central host here to build all packages
for identically configured cluster hosts that (e)merge the pre-built
packages. It's quite a time saver

Christian

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Ronan Mullally]

Hi Karl,

On Mon, 21 May 2007, Karl Holz wrote:

> >Is there a way to run gentoo without a portage tree on each box?
>
> yes, if  you setup a build system, using a stage3 tarball, and build your
> system into a directory. Portage will only be under your /usr/portage and not
> into the system image you're building. the good thing about using a Stage3
> tarball is you can build you system on any linux system, build your system
> image, tarball the image, deploy and install grub on x86, yaboot on Mac PPC,
> silo on Sparc64.

How are updates handled?  "emerge -uD <pkg|world>" isn't going to work
without a portage tree, so I presume I'd need to tell each server which
packages need to be updated with "emerge <pkg1> <pkg2> ... <pkgn>" to have
it download them from the binhost?


-Ronan
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Arturo 'Buanzo' Busleiman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Christian Bricart wrote:
> nonsense - of course you're able to maintain your own $BINHOST which
[...]
> Works like charm here - I've got a central host here to build all packages
> for identically configured cluster hosts that (e)merge the pre-built
> packages. It's quite a time saver

Of course it's doable, but there are better choices for binary-based distros than doing all that.
I'm sure it works ok, but if you want to avoid compiling, or having a compiler, then the best idea
sounds like not using a source-based distro. Even cluster-oriented distros, for example.

Anyway, this is a Gentoo list, so I'll never say a "choose another thing instead of Gentoo"-like
phrase never again :P

- --
Arturo "Buanzo" Busleiman - Consultor Independiente en Seguridad Informatica
OpenPGP for HTTP: New Web-Auth Scheme: http://freshmeat.net/articles/view/2599
Consulting and Secure Mail Hosting: http://www.buanzo.com.ar/pro/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGUcCbAlpOsGhXcE0RCkxdAJ0XiKzM1Qr0sz6g7Wv+6nQlPlXC0wCfcqir
WGeA7ANwGmBMbXjqBkuo9VY=
=ZkHV
-----END PGP SIGNATURE-----
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Ryan Gibbons]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]

I believe you will still need a tree either way.

I would just have the master server share it's portage tree over nfs, 
and then when you update the nodes of the cluster, just mount the nfs 
share, run your emerge system or world or whatever, and then when you 
are finished umount the nfs share.

I imagine this could be done easily via scripts, complete with error 
checking for bad mounts bad emerges etc.

Ronan Mullally wrote:
> Hi Karl,
>
> On Mon, 21 May 2007, Karl Holz wrote:
>
>   
>>> Is there a way to run gentoo without a portage tree on each box?
>>>       
>> yes, if  you setup a build system, using a stage3 tarball, and build your
>> system into a directory. Portage will only be under your /usr/portage and not
>> into the system image you're building. the good thing about using a Stage3
>> tarball is you can build you system on any linux system, build your system
>> image, tarball the image, deploy and install grub on x86, yaboot on Mac PPC,
>> silo on Sparc64.
>>     
>
> How are updates handled?  "emerge -uD <pkg|world>" isn't going to work
> without a portage tree, so I presume I'd need to tell each server which
> packages need to be updated with "emerge <pkg1> <pkg2> ... <pkgn>" to have
> it download them from the binhost?
>
>
> -Ronan
>   

[-- Attachment #2: Type: text/html, Size: 1783 bytes --]

Re: [gentoo-server] Best practices in managing large server groups

[Ronan Mullally]

On Mon, 21 May 2007, Ryan Gibbons wrote:

> I believe you will still need a tree either way.
>
> I would just have the master server share it's portage tree over nfs, and then
> when you update the nodes of the cluster, just mount the nfs share, run your
> emerge system or world or whatever, and then when you are finished umount the
> nfs share.
>
> I imagine this could be done easily via scripts, complete with error checking
> for bad mounts bad emerges etc.

That's what I figured.  Okay, last question - how do I stop emerge trying
to update the compiler-less systems to include development tools like gcc?
I presume tweaking the profile is the way to do it.  Is there a stock
profile that already has these excluded?


-Ronan
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

Ronan Mullally kirjoitti:
> On Mon, 21 May 2007, Ryan Gibbons wrote:
> 
>> I believe you will still need a tree either way.
>>
>> I would just have the master server share it's portage tree over nfs, and then
>> when you update the nodes of the cluster, just mount the nfs share, run your
>> emerge system or world or whatever, and then when you are finished umount the
>> nfs share.
>>
>> I imagine this could be done easily via scripts, complete with error checking
>> for bad mounts bad emerges etc.
> 
> That's what I figured.  Okay, last question - how do I stop emerge trying
> to update the compiler-less systems to include development tools like gcc?
> I presume tweaking the profile is the way to do it.  Is there a stock
> profile that already has these excluded?
> 

I think /etc/portage/profile/package.provided could work.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-server] Best practices in managing large server groups

[Ronan Mullally]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 953 bytes --]

On Mon, 21 May 2007, Petteri Räty wrote:

> > That's what I figured.  Okay, last question - how do I stop emerge trying
> > to update the compiler-less systems to include development tools like gcc?
> > I presume tweaking the profile is the way to do it.  Is there a stock
> > profile that already has these excluded?
>
> I think /etc/portage/profile/package.provided could work.

It looks like it might, but you've got to specify packages with version
numbers.  I could see lots of fun and games when newer versions are added
to portage or called as dependancies.  Keeping this file up to date with
the version numbers required would make the process rather tiresome.

That said, it's something that could probably be scripted (run a dummy
emerge on the portage tree and parse the output for any new installs of
gcc, binutils, etc), and if something did slip through, the corresponding
binaries shouldn't be on the binhost.


-Ronan

Re: [gentoo-server] Best practices in managing large server groups

[José Costa]

try portage-utils. they are in C.

On 5/21/07, Ronan Mullally <ronan@iol.ie> wrote:
> On Mon, 21 May 2007, Ryan Gibbons wrote:
>
> > I believe you will still need a tree either way.
> >
> > I would just have the master server share it's portage tree over nfs, and then
> > when you update the nodes of the cluster, just mount the nfs share, run your
> > emerge system or world or whatever, and then when you are finished umount the
> > nfs share.
> >
> > I imagine this could be done easily via scripts, complete with error checking
> > for bad mounts bad emerges etc.
>
> That's what I figured.  Okay, last question - how do I stop emerge trying
> to update the compiler-less systems to include development tools like gcc?
> I presume tweaking the profile is the way to do it.  Is there a stock
> profile that already has these excluded?
>
>
> -Ronan
> --
> gentoo-server@gentoo.org mailing list
>
>
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[José Costa]

ups, replied for the wrong email "question".

how about dealing with /var/*/portage and db stuff?

On 5/21/07, José Costa <meetra@gmail.com> wrote:
> try portage-utils. they are in C.
>
> On 5/21/07, Ronan Mullally <ronan@iol.ie> wrote:
> > On Mon, 21 May 2007, Ryan Gibbons wrote:
> >
> > > I believe you will still need a tree either way.
> > >
> > > I would just have the master server share it's portage tree over nfs, and then
> > > when you update the nodes of the cluster, just mount the nfs share, run your
> > > emerge system or world or whatever, and then when you are finished umount the
> > > nfs share.
> > >
> > > I imagine this could be done easily via scripts, complete with error checking
> > > for bad mounts bad emerges etc.
> >
> > That's what I figured.  Okay, last question - how do I stop emerge trying
> > to update the compiler-less systems to include development tools like gcc?
> > I presume tweaking the profile is the way to do it.  Is there a stock
> > profile that already has these excluded?
> >
> >
> > -Ronan
> > --
> > gentoo-server@gentoo.org mailing list
> >
> >
>

Re: [gentoo-server] Best practices in managing large server groups

[Karl Holz]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

Sure, you'll have a Portage tree, but it will be in you build root.  
here is the way I would set it up

the build root would be under /build, you could also share this over  
nfs too

# cd build
# tar jxvf ~/stage3*.tar.bz2
# cd usr
# tar jxvf ~/portage-latest.tar.bz2
# cd /build

-mounting all the needed directories, binding the root to /build/ 
chroot is only if you want to build packages into it.

# for x in "/sys:/sys" "/dev:/dev/" "/:/chroot"
 > do
 >  mount -o bind $(echo $x|cut -d':' -f1) /build/$(echo $x|cut -d':'  
-f2)
 > done
# mount -t proc none /build/proc
# cat /etc/resolv.conf > /build/etc/resolv.conf
# chroot /build /bin/bash

now you can set your profile, edit your /etc/mack.conf

(chroot) # env-update; source /etc/profile
(chroot) # ROOT="/chroot" emerge <pkg>

for updating you system image use

(chroot) # ROOT="/chroot" emerge <pkg> -uD

you could use world, but it will install the things you may not want.  
you can update the world of the build root /build

(chroot) # emerge --sync ; emerge world -uD

This can all be scripted too, you just neet to make 2 scripts; one to  
start the job and one to work inside the chroot, just change

# chroot /build /bin/bash

to

# chroot /build /start.sh

__________________
Karl Gustav B. Holz
----------
karl@new-aeon.com
----------
http://www.new-aeon.com


On 21-May-07, at 12:27 PM, Ryan Gibbons wrote:

> I believe you will still need a tree either way.
>
> I would just have the master server share it's portage tree over  
> nfs, and then when you update the nodes of the cluster, just mount  
> the nfs share, run your emerge system or world or whatever, and  
> then when you are finished umount the nfs share.
>
> I imagine this could be done easily via scripts, complete with  
> error checking for bad mounts bad emerges etc.
>
> Ronan Mullally wrote:
>> Hi Karl,
>>
>> On Mon, 21 May 2007, Karl Holz wrote:
>>
>>
>>>> Is there a way to run gentoo without a portage tree on each box?
>>>>
>>> yes, if  you setup a build system, using a stage3 tarball, and  
>>> build your
>>> system into a directory. Portage will only be under your /usr/ 
>>> portage and not
>>> into the system image you're building. the good thing about using  
>>> a Stage3
>>> tarball is you can build you system on any linux system, build  
>>> your system
>>> image, tarball the image, deploy and install grub on x86, yaboot  
>>> on Mac PPC,
>>> silo on Sparc64.
>>>
>> How are updates handled?  "emerge -uD <pkg|world>" isn't going to  
>> work
>> without a portage tree, so I presume I'd need to tell each server  
>> which
>> packages need to be updated with "emerge <pkg1> <pkg2> ... <pkgn>"  
>> to have
>> it download them from the binhost?
>>
>>
>> -Ronan
>>


[-- Attachment #2: Type: text/html, Size: 6077 bytes --]

Re: [gentoo-server] Best practices in managing large server groups

[Ramon van Alteren]

Hi Charles

I've been looking for time to answer this more fully than the quick 
oneshot mail I send off earlier.
We run 600 servers on gentoo and started with a single server originally.

So yes it's definitely doable :)

I'll try and answer as much as possible, any questions feel free to mail.

Charles Duffy wrote:
> I'm looking at replacing SuSE SLES9 with Gentoo for an enterprise
> application (for reasons of flexibility and licensing) (no, we don't
> have an enterprise application budget -- just the reliability
> requirements; yaaay, startups!). We're looking to be able to deploy and
> manage hundreds of geographically distributed servers.
See above, what is your planned initial deployment ? Are you starting 
with a hundred or more servers or are you starting with just a couple ?
> We have a QA department available to vet each configuration before it is
> deployed to the field. We have infrastructure for tracking the progress
> of code in svn from creation though QA to deployment; I'm anticipating
> tracking a local overlay (containing all packages we use), make.conf,
> /etc/portage/*, etc. through this system, autobuilding system images
> (either to run virtualized or on real hardware) from the contents of
> svn, building binary packages and deploying them to real hardware.
Having a QA department to offload this work to is certainly a bonus :)
> I'm interested in best practices, suggested tools, and/or 3rd party
> experiences in this regard.
>
> Some particular questions which come to mind:
>  - Should I be using a custom profile or a standard profile with
> overrides through make.conf, /etc/portage/* and the like?
AFAIK you should be able to set all required stuff through overrides.
The point is to keep in mind the benefits of using gentoo and not try 
and work against the system.
Several people discussed running gentoo servers in produktion without a 
build toolchain (gcc etc.)
I have no comments to offer on how desirable this is, but if this is a 
goal/requirement for you I'd strongly suggest using a binary distro.
Gentoo shines as a ultra-configurable source-based distro, running it 
without a build toolchain and / or portage tree is certainly possible.
It would however take away much of the advantages of using gentoo, so 
why not switch to something else in that case.

Removing the portage tree has always been a weird question to me, nobody 
discusses removing the rpm-package database, why are people so keen on 
removing the portage tree ?
It takes roughly 550Mb of space which is quite a lot, but hardly a 
killing requirement given todays diskspace and hardware.
The linux kernel source tree is roughly in the same sizing category.
>  - What's the Right Way to create new system images ready to be loaded
> onto a hard drive or run through a virtual machine? gentoo-buildhoster
> looks interesting. I've seen Catalyst mentioned as a way to create
> stage3 images, but what documentation I've been able to find doesn't
> seem very much targeted for my use cases.
I would recommend catalyst-2. Although documentation is lacking, it 
isn't that hard to setup.
You're probably looking for the stage4 target if you want to build 
system images.
Rolling out gentoo on such a large scale, you need a repeatable system 
image build environment.

The bonus of catalyst is that you automatically get a binary package 
server in the process of generating your images.
Catalyst can be told to use a binary package cache, by carefully setting 
up your catalyst environment you can easily reuse that as source for 
your binpkg server.
I'm not familiar with gentoo-buildhoster, but since it's webpage [1] 
lists it as no longer maintained that would be a no-go area for me.

We combined catalyst with a pxe based boot environment, the quickstart 
installer [2] and puppet[3].
It allows us to provision a server within 30 minutes. That's 30 minutes 
from connecting the hardware to a switch and active in production.
This requires very little manual intervention, which we consider to be a 
good thing (tm)
And yes, that's concurrent, we believe it to be capable of roughly 30 
servers concurrent setup and that appears to be a pxe limitation.

If at all possible, try to build your deployment system thus that you 
can always easily wipe a server and reinstall.
We didn't originally and are refactoring to allow it now.
>  - Any experiences with puppet? With out ratio of servers to staff,
> automating configuration and administration is a priority. (We already
> have an internal tool written with automating the server configuration
> process in mind; it has some functionality puppet doesn't, and puppet
> has functionality it doesn't; in theory, I'd like to extend puppet until
> our internal tool becomes unnecessary, though I'll need to understand
> puppet much better before I can think too hard about that).
We are using puppet extensively. It works, although it's still rough 
around the edges which is as expected from such a young project.
The gentoo provider for puppet is in it's infancy. It works, but 
definitly needs work as well.

Apart from that puppet is a very versatile and powerful tool.
And most importantly it has a very active community of people around it.
They are actively exchanging recipes for server configuration, which is 
useful in itself but becomes extremely useful when combined with
the new module organization in puppet.

Many problems you will face in deploying and configuring such an amount 
of servers will have been solved in whole or in part by someone in the 
puppet community.

I would be curious what functionality is missing from puppet right now 
in your opinion ?
>  - Have any of 'yall been in the 100s-of-servers situation with
> comparable requirements and come up with a different approach to
> managing it? How did things work out?
We've grown very very fast and have tried different methods along the way.
To be honest, we're still in the process of moving most of the 
serverpark under puppet control (nearly 50% done)
And I actually do not expect to find a single set of tools that cope 
with all the issues that you face when deploying such an amount of servers.
Your situation might be different because you are starting with a single 
app.

One vital thing that is missing from the picture is an inventory database.
You need some sort of queryable database that stores servers, location, 
networking info, function, server-identifyable serial of some sort, 
hardware classes, deployment status etc.
Without that you're basically lost. We use a homebrew mysql based system 
with both a cli- and a webinterface.

Apart from that, you'll need lots of infrastructure:
* logservers
* monitoring
* statistics gathering
* backups
* scripts repository
* version controlled configs
* bug / issue tracking
* firewalling
* loadbalancing
* network monitoring and configuration system
* etc. etc. etc.
> Thank you!
You're welcome, I would like to see more people use gentoo in 
large-scale environments and am actively looking for possibilities to 
exchange experiences.
Feel free to contact me if you have any questions. I also hang out at a 
couple of irc-channels [4] with nickname Innocenti

Out of pure curiosity what is your staff to server ratio ?

Grtz Ramon

Senior System Administrator Hyves.nl

[1]http://badpenguins.com/source/
[2]http://dev.gentoo.org/~agaffney/quickstart.php
[3]http://puppet.reductivelabs.com/
[4]#puppet, #gentoo-server, #gentoo-cluster, #gentoo-amd64
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Justin Cataldo]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

>Back to the original question.  How do I run a number of gentoo boxes
>without gcc (or a portage tree)?

This all sounds way over my head, but anyway... could distcc be the answer?

http://www.gentoo.org/doc/en/distcc.xml

JC

On 5/22/07, Ronan Mullally <ronan@iol.ie> wrote:
>
> On Mon, 21 May 2007, Arturo 'Buanzo' Busleiman wrote:
>
> > But, of course, if they got access to the box, then the compiler is the
> > least of your problems at that time, but I have to admit that the
> > "slowing the attacker down" is an extra layer of protection. It provides
> > the sysadmins/users/monitoring software more time to detect the breach.
>
> Removing development tools is one of just a range of security measures.
> It is by no means the be-all and end-all of security - hence my mention of
> security in depth.  There are easily a couple of dozen security measures I
> implement when installing a box.  Removing development tools (or in most
> cases, never installing them) is just one of them, but we diverge...
>
> Back to the original question.  How do I run a number of gentoo boxes
> without gcc (or a portage tree)?
>
>
> -Ronan
> --
> gentoo-server@gentoo.org mailing list
>
>

[-- Attachment #2: Type: text/html, Size: 1677 bytes --]

Re: [gentoo-server] Best practices in managing large server groups

[Brian Kroth]

> Back to the original question.  How do I run a number of gentoo boxes
> without gcc (or a portage tree)?
> 
> 
> -Ronan

Sorry for the late post, I'm still catching up on email.

I haven't looked into removing gcc (but your comments intrigue me).  As 
for no portage tree, all of my machines auto-NFS mount it from a local 
build host that also holds the binary packages.  The NFS mount is set 
read-only so the novice doesn't get to emerge on the machine (yes it's 
easily subverted).  I've found this helps with stability since then 
everything must go through the same build server and I have better 
guarantees about what is actually on the individual machines.  There are 
a couple of different images to take care of the different roles each of 
those servers take on (web server, database server, mail server, etc.).

I've also been looking into PXE booting so I only need to update a 
single image.  To be honest I'm not quite ready for this yet, but it 
might be another option for you.

Brian
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Brian Kroth]

I heard a couple people talking up puppet, but also saying that it's 
young.  Does anyone have experience with cfengine?  Better or worse?

I'm planning on deploying one of these this summer for a 40 server setup.

Thanks,
Brian
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Tomasz Szymczak]

On Sunday 20 of May 2007 20:29:18 Charles Duffy wrote:
> I'm looking at replacing SuSE SLES9 with Gentoo for an enterprise
> application (for reasons of flexibility and licensing) (no, we don't
> have an enterprise application budget -- just the reliability
> requirements; yaaay, startups!). We're looking to be able to deploy and
> manage hundreds of geographically distributed servers.

I'm using some custom made gentoo setup, it's not perfect but works fine for 
me (>30 gentoo servers and >50 gentoo vservers in different locations)

Every gentoo server has mounted snapshot of portage tree compressed by 
squashfs = 40MB (vservers are using bind mounted version), it's extended by 
custom overlays (security, updates, etc). I can do 'big' upgrade of gentoo by 
putting newer portage snapshot.

I've made custom gentoo profiles (f.e. amd64-default-20061004) which are 
setting most things like cflags, use flags, overlays and inherits real 
profile at the end. Also I'm extending it by creating my subprofiles to force 
installing packages, for example:

x86-default-20061004/h  (vserver hardware node)
vserver-default-20061004/www/php5  (www vservers with php5)
vserver-default-20061004/mail  (mail vservers)

Most servers/vservers have /var/lib/portage/world empty, all packages are 
forced by putting it into 'system'. Also, /etc/make.conf on all servers are 
empty too :)

After emerge (compile) of package, it's sent into binhost server (via scp 
in /etc/portage/bashrc) and other servers in same profile are fetching binary 
version.


This solution is a little bit tricky, have some issues but after all - makes 
many things faster and easier for me.

regards
-- 
Tomasz Szymczak                                                   _.-.  jgs
GCS/M d- s-:- a-- C+++$ UL++$>++++$ P++ L++>++++ E W++ N+ w O+  '( ^{_}    (
PS+ PE !Y PGP+ t- X- R tv-- b++>+++ DI D>+++ G e>++++ h! r- y?    `~\`-----'\
Umysł sprawia, że wszystko się zmienia, żeby upozorować upływ czasu  )_)---)_)
--
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Charles Duffy]

Thank you for your advice. It's good to hear that someone else is in a 
similar scenario.

The only thing that's going to be a hard sell around here is Catalyst -- 
our senior sysadmin (who is a considerably more seasoned Gentoo user 
than myself) has taken the position that using Catalyst is unnecessary 
and that emerge (targeting a chroot, with appropriate FEATURES set to 
generate binary packages) should be adequate. He's not immovable on such 
things -- but I'll need to have a very solid understanding of the 
benefits of using Catalyst before arguing any case to the contrary. 
(I've also contacted the author of gentoo-buildhoster to find out why he 
abandoned it, what he's using in its place, and whether he believes the 
project to be worth picking up; if he believes catalyst-2 to be more 
suitable for his use cases, understanding the reasoning behind that 
decision may provide some powerful arguments).

Much of your other advice (ie. quickstart) is completely in line with 
our preexisting plans; thanks again for answering. (As for some of the 
items you're curious about -- such as our staff-to-server ratio -- I'm 
not entirely comfortable discussing all of that on an archived list; 
perhaps on IRC).
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Wendall Cada]

[-- Attachment #1: Type: text/plain, Size: 2774 bytes --]

On Tue, 2007-05-22 at 11:54 -0500, Charles Duffy wrote:
> Thank you for your advice. It's good to hear that someone else is in a 
> similar scenario.
> 
> The only thing that's going to be a hard sell around here is Catalyst -- 
> our senior sysadmin (who is a considerably more seasoned Gentoo user 
> than myself) has taken the position that using Catalyst is unnecessary 
> and that emerge (targeting a chroot, with appropriate FEATURES set to 
> generate binary packages) should be adequate. He's not immovable on such 
> things -- but I'll need to have a very solid understanding of the 
> benefits of using Catalyst before arguing any case to the contrary. 
> (I've also contacted the author of gentoo-buildhoster to find out why he 
> abandoned it, what he's using in its place, and whether he believes the 
> project to be worth picking up; if he believes catalyst-2 to be more 
> suitable for his use cases, understanding the reasoning behind that 
> decision may provide some powerful arguments).
> 
I very much agree with his assessment here. This is what I do for binary
distribution to desktop systems. It is very effective. I have played
with Catalyst quite a bit, but agree that catalyst-2 is much more well
suited for this.

I use also use the method described here: http://m8y.org/gentoosync.txt
to filter the portage tree. It makes for faster sync times, but
primarily, it makes me take a look at any additions or changes to the
tree. I've caught potential problems with updates by simply keeping a
very clean tree, which is important on the build server.

I surprises me how little work has gone into the binary support in
portage with the sheer number of devs around here who use the features.
I asked over three years ago about some specific bugs, and was assured
that they would be fixed soon. I don't think there has been a single
significant change in that time. Would be nice if -g/G was fixed. Here
is the method I use for using -k/K:

For package/*.tbz2 format:
#!/bin/bash
cd /usr/portage/packages/All/
all=`emerge -p $@ | grep "\[ebuild " | sed -e "s/\[ebuild[^]]*] //" | /
awk '{print $1}'`
for pkg in ${all}
do
    `wget -nc http://binhost.tld/portage/packages/${pkg}.tbz2`
done

For *.tbz2 format:
#!/bin/bash
cd /usr/portage/packages/All/
all=`emerge $@ -p | grep "\[ebuild " | sed -e "s/\[ebuild[^]]*] //" / |
sed -e "s/.*\///g" | awk '{print $1}'`
for pkg in ${all}
do
    `wget -nc http://binhost.tld/packages/All/${pkg}.tbz2`
done


Maybe someday, I'll turn it into a proper script, but for now, this
works.

Wendall

-- 
Only wimps use tape backup: _real_ men just upload their important stuff
on ftp, and let the rest of the world mirror it ;)
        -- Linus Torvalds

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-server] Re: Best practices in managing large server groups

[Charles Duffy]

Wendall Cada wrote:
> I very much agree with his assessment here. This is what I do for binary
> distribution to desktop systems. It is very effective. I have played
> with Catalyst quite a bit, but agree that catalyst-2 is much more well
> suited for this.

I'm sorry; which "this" are you referring to? I'm a little unclear on 
whether what you're indicating that you use is Ramon's suggestion (of 
using catalyst-2) or our senior sysadmin's suggestion (of using emerge, 
set to output binary packages and target a chroot).

Thank you!

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Re: Best practices in managing large server groups

[Wendall Cada]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

On Tue, 2007-05-22 at 16:06 -0500, Charles Duffy wrote:
> Wendall Cada wrote:
> > I very much agree with his assessment here. This is what I do for binary
> > distribution to desktop systems. It is very effective. I have played
> > with Catalyst quite a bit, but agree that catalyst-2 is much more well
> > suited for this.
> 
> I'm sorry; which "this" are you referring to? I'm a little unclear on 
> whether what you're indicating that you use is Ramon's suggestion (of 
> using catalyst-2) or our senior sysadmin's suggestion (of using emerge, 
> set to output binary packages and target a chroot).

I use (of using emerge, set to output binary packages and target a
chroot) right now, as Ramon's senior sysadmin recommends. But am looking
into using catalyst-2 in the future.

Wendall
> 
> Thank you!
> 
-- 
Only wimps use tape backup: _real_ men just upload their important stuff
on ftp, and let the rest of the world mirror it ;)
        -- Linus Torvalds

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-server] Best practices in managing large server groups

[Ramon van Alteren]

Charles Duffy wrote:
> Thank you for your advice. It's good to hear that someone else is in a 
> similar scenario.
It's likewise nice to hear someone is planning the same as we do.
> The only thing that's going to be a hard sell around here is Catalyst 
> -- our senior sysadmin (who is a considerably more seasoned Gentoo 
> user than myself) has taken the position that using Catalyst is 
> unnecessary and that emerge (targeting a chroot, with appropriate 
> FEATURES set to generate binary packages) should be adequate. He's not 
> immovable on such things -- but I'll need to have a very solid 
> understanding of the benefits of using Catalyst before arguing any 
> case to the contrary. (I've also contacted the author of 
> gentoo-buildhoster to find out why he abandoned it, what he's using in 
> its place, and whether he believes the project to be worth picking up; 
> if he believes catalyst-2 to be more suitable for his use cases, 
> understanding the reasoning behind that decision may provide some 
> powerful arguments).
Well, basically he's right. Catalyst is a build tool that wraps all the 
stuff you need to do when building packages in a chroot, with some 
additional features
You don't need it.... it just saves time and a lot of work writing your 
own scripts to manage the environment, and it's semi-maintained.

OTOH you lose flexibility, need to invest time in learning how catalyst 
works etc.
Catalyst-1 is completely unusable for building system-images, this is a 
feature that catalyst gained with the -2 release.
Catalyst has gained quite a community of users by now, who keep bugging 
Chris for features and explaination, you can find them on gentoo-catalyst.

The benefit of using catalyst for us is that it solves two wishes in one 
fell swoop:
* an automated and repeatable system image build tool
* a binary package generator for specified system images

Additionally we're looking into using the tinderbox target to validate 
builds on top of our different server images.
> Much of your other advice (ie. quickstart) is completely in line with 
> our preexisting plans; thanks again for answering. (As for some of the 
> items you're curious about -- such as our staff-to-server ratio -- I'm 
> not entirely comfortable discussing all of that on an archived list; 
> perhaps on IRC).
OK, that's cool, we've been thinking of generalizing the changes we did 
to quickstart to get it to talk to our inventory system and submitting 
them back to agaffney.
If somebody finds the time to do the work, that is. I would be very 
curious about the inventory system you are planning, if you're willing 
to shed some light on that I'd be grateful.

Ramon

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Andrew Gaffney]

Ramon van Alteren wrote:
> Well, basically he's right. Catalyst is a build tool that wraps all the 
> stuff you need to do when building packages in a chroot, with some 
> additional features
> You don't need it.... it just saves time and a lot of work writing your 
> own scripts to manage the environment, and it's semi-maintained.

You have an odd definition of "maintained". Chris (catalyst lead) and I (general 
code monkey) are actively working on catalyst. We just don't care much about 
features that don't directly benefit release building, since that's what 
catalyst's primary function is. If someone asks for a feature that doesn't 
affect building releases and submits code, we're very likely to throw it in.

-- 
Andrew Gaffney                                 http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer             Catalyst/Installer + x86 release coordinator
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Ramon van Alteren]

Andrew Gaffney wrote:
> Ramon van Alteren wrote:
>> Well, basically he's right. Catalyst is a build tool that wraps all 
>> the stuff you need to do when building packages in a chroot, with 
>> some additional features
>> You don't need it.... it just saves time and a lot of work writing 
>> your own scripts to manage the environment, and it's semi-maintained.
>
> You have an odd definition of "maintained". Chris (catalyst lead) and 
> I (general code monkey) are actively working on catalyst. We just 
> don't care much about features that don't directly benefit release 
> building, since that's what catalyst's primary function is. If someone 
> asks for a feature that doesn't affect building releases and submits 
> code, we're very likely to throw it in.
Please, that's not what I meant, accept my apologies if that's what came 
across.

What I meant with semi-maintained is that it is maintained as an 
internal tool for building gentoo-releases and not as a general 
system-images buildtool.
Reading it again I should have said that more clearly, sorry.

Ramon

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Andrew Gaffney]

Ramon van Alteren wrote:
> Andrew Gaffney wrote:
>> Ramon van Alteren wrote:
>>> Well, basically he's right. Catalyst is a build tool that wraps all 
>>> the stuff you need to do when building packages in a chroot, with 
>>> some additional features
>>> You don't need it.... it just saves time and a lot of work writing 
>>> your own scripts to manage the environment, and it's semi-maintained.
>>
>> You have an odd definition of "maintained". Chris (catalyst lead) and 
>> I (general code monkey) are actively working on catalyst. We just 
>> don't care much about features that don't directly benefit release 
>> building, since that's what catalyst's primary function is. If someone 
>> asks for a feature that doesn't affect building releases and submits 
>> code, we're very likely to throw it in.
> Please, that's not what I meant, accept my apologies if that's what came 
> across.
> 
> What I meant with semi-maintained is that it is maintained as an 
> internal tool for building gentoo-releases and not as a general 
> system-images buildtool.
> Reading it again I should have said that more clearly, sorry.

That's *very* different from "semi-maintained" :)

-- 
Andrew Gaffney                                 http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer             Catalyst/Installer + x86 release coordinator
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Best practices in managing large server groups

[Thilo Bangert]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

hi all,

> It's likewise nice to hear someone is planning the same as we do.

at work, we are currently planning for a massive expansion of our server 
park and i am in the process of figuring it all out. not having done this 
before i am probably about to fall into a bunch of holes - so any insight 
into how you do stuff is of great help. thanks for all the information so 
far.

what follows are the pieces that we've put together for now. much of this 
has not been tested yet, but i'd wouldn't mind some feedback.

 - use catalyst2 to build different role stage4 images.
 - seed a binhost with the binary packages from catalyst
 - add additional packages to the binhost (as needed)
 - use quickstart to deploy stage4 image
 - use enhost to add the host to a inventory db
 - use puppet for configuration management
 - use munin for resource and performance monitoring
 - use nagios for service monitoring
 - build a tool which creates nagios configuration from the inventory
 - all syslog to a central loghost - using srlog2
 - use nullmailer to forward email to a central smarthost
 - higher level applications will be deployed using capistrano - possibly 
drawing some of the configuration options from the inventory database.

we will not do backups of the nodes - the goal is to be able to reproduce 
the setup faster (or at roughly the same speed)  than a backup recovery. 
application data will most likely be backed up to amazon S3 in (near) 
real time.

we are currently looking into RT or bugzilla for issue tracking...

i'd love to hear what tools other people are using for these and similar 
tasks or what comments people have to the setup outlined above.

> I would be very 
> curious about the inventory system you are planning, if you're willing
> to shed some light on that I'd be grateful.

i haven't given this two much of a thought, just yet. enhost supplies its 
data to ldap - so for a start we'll probably just use that. depending on 
how the deployment procedure of the higher layers works out a relational 
database may turn out to be better suited...

kind regards
Thilo

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]