Re: [gentoo-server] Best practices in managing large server groups

[Thilo Bangert <bangert@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

hi all,

> It's likewise nice to hear someone is planning the same as we do.

at work, we are currently planning for a massive expansion of our server 
park and i am in the process of figuring it all out. not having done this 
before i am probably about to fall into a bunch of holes - so any insight 
into how you do stuff is of great help. thanks for all the information so 
far.

what follows are the pieces that we've put together for now. much of this 
has not been tested yet, but i'd wouldn't mind some feedback.

 - use catalyst2 to build different role stage4 images.
 - seed a binhost with the binary packages from catalyst
 - add additional packages to the binhost (as needed)
 - use quickstart to deploy stage4 image
 - use enhost to add the host to a inventory db
 - use puppet for configuration management
 - use munin for resource and performance monitoring
 - use nagios for service monitoring
 - build a tool which creates nagios configuration from the inventory
 - all syslog to a central loghost - using srlog2
 - use nullmailer to forward email to a central smarthost
 - higher level applications will be deployed using capistrano - possibly 
drawing some of the configuration options from the inventory database.

we will not do backups of the nodes - the goal is to be able to reproduce 
the setup faster (or at roughly the same speed)  than a backup recovery. 
application data will most likely be backed up to amazon S3 in (near) 
real time.

we are currently looking into RT or bugzilla for issue tracking...

i'd love to hear what tools other people are using for these and similar 
tasks or what comments people have to the setup outlined above.

> I would be very 
> curious about the inventory system you are planning, if you're willing
> to shed some light on that I'd be grateful.

i haven't given this two much of a thought, just yet. enhost supplies its 
data to ldap - so for a start we'll probably just use that. depending on 
how the deployment procedure of the higher layers works out a relational 
database may turn out to be better suited...

kind regards
Thilo

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]