Gentoo for many servers (was: Re: [gentoo-user] executing commands on lots of servers at once)

[Alex Schuster <wonko@wonkology.org>]

Alan McKinnon wrote:

> clusterssh will let you log into many machines at once and run emerge
>  -avuND world everywhere

This is way cool. I just started using it on eight Fedora servers I am 
administrating. Nice, now this is an improvement over my 'for $h in 
$HOSTS; do ssh $h "yum install foo"; done' approach.

What do you guys think about using Gentoo for servers? At the institute I 
partially work we chose Fedora. There is no special reason for that - we 
already had some Fedora machines, the setup seemed to work, the reputation 
was good, so we kept it. That was okay for me, why choose many different 
environments and learn everything again. I mentioned Gentoo, but did not 
really suggest to actually use it. Maybe I should have.

These 8 servers I mentioned are basically clones of the one I installed 
manually. Instead of doing this again, I boot a live-cd on a new one, 
create partitions, and extract tar files of the first server's partitions. 
Then I do some extra configuration, like hostname and network setup. Done.

My plan for updating them is to take the first server down, and upgrade 
the installation (if that works - I had some trouble with that before, so 
maybe it will be better to reinstall from scratch). Then I will create a 
snapshot of the new setup, transfer that to the other hosts, and unpack it 
in new logical volumes. I plan to script this so I do not have to do it 
manually every time - but that was before I knew ClusterSSH. When all is 
done and there is some time to take the servers down, I will reboot into 
the new system.

Now I am thinking about a Gentoo installation instead.

Pros:
 - Continuous updates, no downtime for upgrading, only when I decide to 
install a new kernel. This is really really cool. I fear the upgrade from 
Fedora 10 to 12 which has to be done soon.
 - Some improvement in speed. Those machines do A LOT of numbercrunching, 
which jobs often lasting for days, so even small improvements would be 
nice.
 - Easier debugging. When things do not work, I think it's easier to dig 
into the problem. No fancy, but sometimes buggy GUIs hiding basic 
functionality.
 - Heck, Gentoo is _cooler_ than typical distributions. And emerging with 
distcc on about 8*4 cores would be fun :)
 - I am probably the only one who can administrate them.

Cons:
 - If something will not work with this not so common (meta)distribution, 
people will say "always trouble with your Gentoo Schmentoo, it works fine 
in Fedora". Fedora is more mainstream, if something does not work there, 
then it's okay for the people to accept it.
 - I fear that big packages like Matlab are made for and tested on the 
typical distributions, and may have problems with the not-so-common 
Gentoo. I think someone here just had such a problem with Mathematica 
(which we do currently not use).
 - I am probably the only one who can administrate them. I think Gentoo is 
easier to maintain in the long run, but only when you take the time to 
learn it. With Fedora, you do not need much more than the 'yum install' 
command. There is no need to read complicated X.org upgrade guides and 
such.

I think I already made my decision, but I am still interested in your 
opinions, maybe some of you are in a similar position and like to share 
your experiences. Whether I will be allowed to use Gentoo is another 
question, I guess my boss will not like my idea at first, and I am not 
even sure if he is right. But maybe I can test-install Gentoo on one 
machine in a chroot, and see if things work fine.

	Wonko