* [gentoo-user] executing commands on lots of servers at once @ 2009-10-29 18:36 James 2009-10-29 18:57 ` Alan McKinnon ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: James @ 2009-10-29 18:36 UTC (permalink / raw To: gentoo-user I have a cluster of Gentoo boxes I'd like to update all at once, along with emerging specific packages to the servers simultaneously. Does anyone have any experience(s) with a good utility for doing this? I've seen a few scripts online for accomplishing this but I'm not sure how these scripts would handle the large amount of output that emerge generates from all the servers simultaneously. Thoughts / ideas appreciated. -j ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-29 18:36 [gentoo-user] executing commands on lots of servers at once James @ 2009-10-29 18:57 ` Alan McKinnon 2009-10-29 21:42 ` James 2009-11-14 17:36 ` Gentoo for many servers (was: Re: [gentoo-user] " Alex Schuster 2009-10-29 21:36 ` [gentoo-user] executing commands on lots of servers at once Dirk Heinrichs 2009-10-31 12:03 ` he zhitong 2 siblings, 2 replies; 16+ messages in thread From: Alan McKinnon @ 2009-10-29 18:57 UTC (permalink / raw To: gentoo-user On Thursday 29 October 2009 20:36:27 James wrote: > I have a cluster of Gentoo boxes I'd like to update all at once, along > with emerging specific packages to the servers simultaneously. > > Does anyone have any experience(s) with a good utility for doing this? > I've seen a few scripts online for accomplishing this but I'm not sure > how these scripts would handle the large amount of output that emerge > generates from all the servers simultaneously. > > Thoughts / ideas appreciated. I thought we answered this for you two days ago? Put the same world and config on every machine, and build everything on one host called the binhost. emerge -k on every machine will pull binary packages from the binhost. This is identical to working with say Ubuntu, except that it's not a maintainer building packages and putting them on a remote repo, it's you doing it and putting the packages on a machine on your local network. clusterssh will let you log into many machines at once and run emerge -avuND world everywhere -- alan dot mckinnon at gmail dot com ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-29 18:57 ` Alan McKinnon @ 2009-10-29 21:42 ` James 2009-10-30 7:03 ` To James and James (was Re: [gentoo-user] executing commands on lots of servers at once) Dirk Heinrichs 2009-11-14 17:36 ` Gentoo for many servers (was: Re: [gentoo-user] " Alex Schuster 1 sibling, 1 reply; 16+ messages in thread From: James @ 2009-10-29 21:42 UTC (permalink / raw To: gentoo-user Thanks for the response, Alan. I haven't posted on this alias in many months, so it wasn't me who asked 2 days ago. ;) (nor do I see a similar thread from two days ago, but it may have been sent to the bit bucket, so I can't be certain) Thanks! -j On Thu, Oct 29, 2009 at 2:57 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote: > On Thursday 29 October 2009 20:36:27 James wrote: >> I have a cluster of Gentoo boxes I'd like to update all at once, along >> with emerging specific packages to the servers simultaneously. >> >> Does anyone have any experience(s) with a good utility for doing this? >> I've seen a few scripts online for accomplishing this but I'm not sure >> how these scripts would handle the large amount of output that emerge >> generates from all the servers simultaneously. >> >> Thoughts / ideas appreciated. > > > I thought we answered this for you two days ago? > > Put the same world and config on every machine, and build everything on one > host called the binhost. > > emerge -k on every machine will pull binary packages from the binhost. This is > identical to working with say Ubuntu, except that it's not a maintainer > building packages and putting them on a remote repo, it's you doing it and > putting the packages on a machine on your local network. > > clusterssh will let you log into many machines at once and run emerge -avuND > world everywhere > > -- > alan dot mckinnon at gmail dot com > ^ permalink raw reply [flat|nested] 16+ messages in thread
* To James and James (was Re: [gentoo-user] executing commands on lots of servers at once) 2009-10-29 21:42 ` James @ 2009-10-30 7:03 ` Dirk Heinrichs 2009-11-04 14:28 ` [gentoo-user] Re: To James and James (was " James 0 siblings, 1 reply; 16+ messages in thread From: Dirk Heinrichs @ 2009-10-30 7:03 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: Text/Plain, Size: 386 bytes --] Am Donnerstag 29 Oktober 2009 22:42:46 schrieb James: > Thanks for the response, Alan. I haven't posted on this alias in many > months, so it wasn't me who asked 2 days ago. ;) Yes, there are indeed two "James" on the list. Could you please both be so kind and use your full names when posting to the list, to avoid such confusion in the future? Thanks a lot... Dirk [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 190 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* [gentoo-user] Re: To James and James (was Re: executing commands on lots of servers at once) 2009-10-30 7:03 ` To James and James (was Re: [gentoo-user] executing commands on lots of servers at once) Dirk Heinrichs @ 2009-11-04 14:28 ` James 0 siblings, 0 replies; 16+ messages in thread From: James @ 2009-11-04 14:28 UTC (permalink / raw To: gentoo-user Dirk Heinrichs <dirk.heinrichs <at> online.de> writes: > Yes, there are indeed two "James" on the list. Could you please both be so > kind and use your full names when posting to the list, to avoid such confusion > in the future? No, I've been using James on the list since, 2004. I've got grandfather rights. Besides too many hacker/impersonators share my resources; to sort out any instantiation of reality.......(boring). So 'James is the new Sybil ......' http://en.wikipedia.org/wiki/Sybil_(book) pssst, I'm the old, stupid, forgetful one........ pssst, I like diversity too! James ^ permalink raw reply [flat|nested] 16+ messages in thread
* Gentoo for many servers (was: Re: [gentoo-user] executing commands on lots of servers at once) 2009-10-29 18:57 ` Alan McKinnon 2009-10-29 21:42 ` James @ 2009-11-14 17:36 ` Alex Schuster 2009-11-14 19:26 ` Alan McKinnon 2009-11-15 12:09 ` [gentoo-user] Re: Gentoo for many servers Andreas Niederl 1 sibling, 2 replies; 16+ messages in thread From: Alex Schuster @ 2009-11-14 17:36 UTC (permalink / raw To: gentoo-user Alan McKinnon wrote: > clusterssh will let you log into many machines at once and run emerge > -avuND world everywhere This is way cool. I just started using it on eight Fedora servers I am administrating. Nice, now this is an improvement over my 'for $h in $HOSTS; do ssh $h "yum install foo"; done' approach. What do you guys think about using Gentoo for servers? At the institute I partially work we chose Fedora. There is no special reason for that - we already had some Fedora machines, the setup seemed to work, the reputation was good, so we kept it. That was okay for me, why choose many different environments and learn everything again. I mentioned Gentoo, but did not really suggest to actually use it. Maybe I should have. These 8 servers I mentioned are basically clones of the one I installed manually. Instead of doing this again, I boot a live-cd on a new one, create partitions, and extract tar files of the first server's partitions. Then I do some extra configuration, like hostname and network setup. Done. My plan for updating them is to take the first server down, and upgrade the installation (if that works - I had some trouble with that before, so maybe it will be better to reinstall from scratch). Then I will create a snapshot of the new setup, transfer that to the other hosts, and unpack it in new logical volumes. I plan to script this so I do not have to do it manually every time - but that was before I knew ClusterSSH. When all is done and there is some time to take the servers down, I will reboot into the new system. Now I am thinking about a Gentoo installation instead. Pros: - Continuous updates, no downtime for upgrading, only when I decide to install a new kernel. This is really really cool. I fear the upgrade from Fedora 10 to 12 which has to be done soon. - Some improvement in speed. Those machines do A LOT of numbercrunching, which jobs often lasting for days, so even small improvements would be nice. - Easier debugging. When things do not work, I think it's easier to dig into the problem. No fancy, but sometimes buggy GUIs hiding basic functionality. - Heck, Gentoo is _cooler_ than typical distributions. And emerging with distcc on about 8*4 cores would be fun :) - I am probably the only one who can administrate them. Cons: - If something will not work with this not so common (meta)distribution, people will say "always trouble with your Gentoo Schmentoo, it works fine in Fedora". Fedora is more mainstream, if something does not work there, then it's okay for the people to accept it. - I fear that big packages like Matlab are made for and tested on the typical distributions, and may have problems with the not-so-common Gentoo. I think someone here just had such a problem with Mathematica (which we do currently not use). - I am probably the only one who can administrate them. I think Gentoo is easier to maintain in the long run, but only when you take the time to learn it. With Fedora, you do not need much more than the 'yum install' command. There is no need to read complicated X.org upgrade guides and such. I think I already made my decision, but I am still interested in your opinions, maybe some of you are in a similar position and like to share your experiences. Whether I will be allowed to use Gentoo is another question, I guess my boss will not like my idea at first, and I am not even sure if he is right. But maybe I can test-install Gentoo on one machine in a chroot, and see if things work fine. Wonko ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Gentoo for many servers (was: Re: [gentoo-user] executing commands on lots of servers at once) 2009-11-14 17:36 ` Gentoo for many servers (was: Re: [gentoo-user] " Alex Schuster @ 2009-11-14 19:26 ` Alan McKinnon 2009-11-14 22:09 ` Alex Schuster 2009-11-15 12:09 ` [gentoo-user] Re: Gentoo for many servers Andreas Niederl 1 sibling, 1 reply; 16+ messages in thread From: Alan McKinnon @ 2009-11-14 19:26 UTC (permalink / raw To: gentoo-user On Saturday 14 November 2009 19:36:06 Alex Schuster wrote: > Alan McKinnon wrote: > > clusterssh will let you log into many machines at once and run emerge > > -avuND world everywhere > > This is way cool. I just started using it on eight Fedora servers I am > administrating. Nice, now this is an improvement over my 'for $h in > $HOSTS; do ssh $h "yum install foo"; done' approach. I feel your pain :-) We used to have the same problem adding new admins to 87 machines. Now we have a bespoke provisioner that does it all. > What do you guys think about using Gentoo for servers? At the institute I > partially work we chose Fedora. There is no special reason for that - we > already had some Fedora machines, the setup seemed to work, the reputation > was good, so we kept it. That was okay for me, why choose many different > environments and learn everything again. I mentioned Gentoo, but did not > really suggest to actually use it. Maybe I should have. I'm a huge fan of Gentoo and all my personal machines (except the new netbook) have run it for the last 5 years. But I will never install Gentoo on a production server at work. Why? Because it is too time consuming, because no two machines are set up the same, because I can't trust that other admins used the flags they should have. So updates become a case of logging into 80+ machines individually and doing emerge world by hand. Gentoo allows you to customize things to the nth degree - that is it's strength - so people WILL use this one discriminating factor. If OTOH I had a server farm of 80+ machines, all identical, I'd put Gentoo on them in a flash. But I don't have that > These 8 servers I mentioned are basically clones of the one I installed > manually. Instead of doing this again, I boot a live-cd on a new one, > create partitions, and extract tar files of the first server's partitions. > Then I do some extra configuration, like hostname and network setup. Done. > > My plan for updating them is to take the first server down, and upgrade > the installation (if that works - I had some trouble with that before, so > maybe it will be better to reinstall from scratch). Then I will create a > snapshot of the new setup, transfer that to the other hosts, and unpack it > in new logical volumes. I plan to script this so I do not have to do it > manually every time - but that was before I knew ClusterSSH. When all is > done and there is some time to take the servers down, I will reboot into > the new system. > > Now I am thinking about a Gentoo installation instead. > > Pros: > - Continuous updates, no downtime for upgrading, only when I decide to > install a new kernel. This is really really cool. I fear the upgrade from > Fedora 10 to 12 which has to be done soon. Do not upgrade, especially not with a version jump of 2 or more. If you have a lot of machines, I assume you are a decent shop, and that you have some form of formal process for upgrades and changes. What you do instead is a formal migration - copy the data off, reinstall, restore data. If you can't afford to do that every six or twleve months, then I have to ask - what the hell is the organization doing using a distro that is unsupported after 12 months? > - Some improvement in speed. Those machines do A LOT of numbercrunching, > which jobs often lasting for days, so even small improvements would be > nice. Don't fool yourself. Unless you need what Google needs, there is very little speed difference between Gentoo and Fedora. I/O improvements you need can be easily gotten by fiddling the kernel tuning knobs. > - Easier debugging. When things do not work, I think it's easier to dig > into the problem. No fancy, but sometimes buggy GUIs hiding basic > functionality. Errrrrrrrrrrrrrrrmmmmmmmmmmmmmm, Fedora does not require a GUI :-) > - Heck, Gentoo is _cooler_ than typical distributions. And emerging with > distcc on about 8*4 cores would be fun :) Can't argue with that. But that is your ego talking and the machines do not belong to you but to the institute. Your ego has no place in that. > - I am probably the only one who can administrate them. This is not a benefit. It is a severe liability. Where I work, I get fired for trying that :-( > Cons: > - If something will not work with this not so common (meta)distribution, > people will say "always trouble with your Gentoo Schmentoo, it works fine > in Fedora". Fedora is more mainstream, if something does not work there, > then it's okay for the people to accept it. Those same people are likely to say the same about linux vs windows. > - I fear that big packages like Matlab are made for and tested on the > typical distributions, and may have problems with the not-so-common > Gentoo. I think someone here just had such a problem with Mathematica > (which we do currently not use). One or two persons had problems. Many many more replied that they had no problems at all. In Fedora-land, the ratio is the same. > - I am probably the only one who can administrate them. I think Gentoo is > easier to maintain in the long run, but only when you take the time to > learn it. With Fedora, you do not need much more than the 'yum install' > command. There is no need to read complicated X.org upgrade guides and > such. > > I think I already made my decision, but I am still interested in your > opinions, maybe some of you are in a similar position and like to share > your experiences. Whether I will be allowed to use Gentoo is another > question, I guess my boss will not like my idea at first, and I am not > even sure if he is right. But maybe I can test-install Gentoo on one > machine in a chroot, and see if things work fine. Depends how critical these machines are. If you want to change them just because you feel like it, then I do not see how that can possibly be a valid reason. Remember, the institute's needs and desires trump yours every time -- alan dot mckinnon at gmail dot com ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Gentoo for many servers (was: Re: [gentoo-user] executing commands on lots of servers at once) 2009-11-14 19:26 ` Alan McKinnon @ 2009-11-14 22:09 ` Alex Schuster 2009-11-15 6:20 ` Joshua Murphy 0 siblings, 1 reply; 16+ messages in thread From: Alex Schuster @ 2009-11-14 22:09 UTC (permalink / raw To: gentoo-user Alan McKinnon writes: > On Saturday 14 November 2009 19:36:06 Alex Schuster wrote: >> Alan McKinnon wrote: >>> clusterssh will let you log into many machines at once and run emerge >>> -avuND world everywhere >> This is way cool. I just started using it on eight Fedora servers I am >> administrating. Nice, now this is an improvement over my 'for $h in >> $HOSTS; do ssh $h "yum install foo"; done' approach. > > I feel your pain :-) > > We used to have the same problem adding new admins to 87 machines. Now > we have a bespoke provisioner that does it all. Sorry, I just do not get 'bespoke provisioner'. Some sort of software, like clusterssh? Or a person, one admin instead of many? >> What do you guys think about using Gentoo for servers? At the institute >> I partially work we chose Fedora. There is no special reason for that - >> we already had some Fedora machines, the setup seemed to work, the >> reputation was good, so we kept it. That was okay for me, why choose >> many different environments and learn everything again. I mentioned >> Gentoo, but did not really suggest to actually use it. Maybe I should >> have. > > I'm a huge fan of Gentoo Now who would have thought of that! > and all my personal machines (except the new netbook have run it for the > last 5 years. > > But I will never install Gentoo on a production server at work. > > Why? > > Because it is too time consuming, because no two machines are set up the > same, because I can't trust that other admins used the flags they should > have. So updates become a case of logging into 80+ machines individually > and doing emerge world by hand. Gentoo allows you to customize things to > the nth degree - that is it's strength - so people WILL use this one > discriminating factor. > > If OTOH I had a server farm of 80+ machines, all identical, I'd put > Gentoo on them in a flash. But I don't have that Of our 8 machines, 7 are essentially the same and differ only in hard drive space and CPU speed. The other machine is Intel, not AMD, and needs different IDE drivers. At the moment it has a different initrd (I set up a minimal fedora install to generate it after the cloned system did not boot), the rest is - apart from some config files - identical. So I would make sure that about everything is exactly the same, well, maybe except for hostnames, udev net-persistent-rules, ssh keys... what more? The last, a little different machine is a problem though. With optimized CFLAGS, this one would have to compile all stuff again, while for the others I could use binpkgs. Updating them all with clusterssh should not be much more work than updating a single one. Well, not completely true, I would have the double work, as I would upgrade one server first to test if there are problems, and then do it for the others. Maybe I could use the special machine to test stuff, and then update all the others. If they would differ, Gentoo would of course be too much work. I already have this problem now... there is my desktop machine, my notebook running a Gentoo VM, a second desktop machine at my other home, the living-room machine of my flat share, the machine of a fried I also administrate, the server of my flat share I need to set up again... and clusterssh is no option here. >> Now I am thinking about a Gentoo installation instead. >> >> Pros: >> - Continuous updates, no downtime for upgrading, only when I decide to >> install a new kernel. This is really really cool. I fear the upgrade >> from Fedora 10 to 12 which has to be done soon. > > Do not upgrade, especially not with a version jump of 2 or more. If you > have a lot of machines, I assume you are a decent shop, and that you > have some form of formal process for upgrades and changes. Not really, I think. We are not very professional I must admit. We have two capable admins, but one is specialized in network stuff and Windows, the other has to do with our big Sun servers, huuge storage systems and such. They do not much about the Linux cluster. Another user sometimes installs a package on a machine, but usually I do this. For me, it is not my main job, I work only about ten hours per week there, mostly being some 100 km away. We are a research institute. We do neurological research, PET and MRI tomography. The Linux servers do number crunching, and of course they should work and have good uptimes, but it is not as important as if we were an ISP. > What you do instead is a formal migration - copy the data off, > reinstall, restore data. Advice noted. Yes, this sounds like the better idea, giving a cleaner setup. And if some things break I do not have to wonder if it was some strange side effect from the upgrade process. > If you can't afford to do that every six or twleve months, then > I have to ask - what the hell is the organization doing using a distro > that is unsupported after 12 months? Well, I do not think this was considered much. One machine was set up with Fedora for no specific reason, and we kept this distro then. This does not sound too professional, I know. BTW, what distro would you suggest? >> - Some improvement in speed. Those machines do A LOT of >> numbercrunching, which jobs often lasting for days, so even small >> improvements would be nice. > > Don't fool yourself. Unless you need what Google needs, there is very > little speed difference between Gentoo and Fedora. I/O improvements you > need can be easily gotten by fiddling the kernel tuning knobs. I know the difference will not be huge, I see this as a little bonus - nice if is there, but nothing really important. But in the comparison with Ubuntu that came in a thread a few weeks ago, for some applications the speed increase was about 30 percent. Although I would not necessarily expect the difference to be noticeable, I would also not be surprised too much if it were noticeable for some number-crunching applications if they were optimized for the CPU. >> - Easier debugging. When things do not work, I think it's easier to >> dig into the problem. No fancy, but sometimes buggy GUIs hiding basic >> functionality. > > Errrrrrrrrrrrrrrrmmmmmmmmmmmmmm, Fedora does not require a GUI :-) Right, and now that I think of it I do not use it anyway... Well, I did do some things with netsetup (or whatever it's called), now that I know the system a little better I edit things directly in /etc/sysconfig. But the installer is a GUI, right? And if I remember this correctly, I cannot even switch to a text console and do stuff there while installing. Or I could, but did not have utilities like LVM. Something like that. I have to use the installer and its capabilities. >> - Heck, Gentoo is _cooler_ than typical distributions. And emerging >> with distcc on about 8*4 cores would be fun :) > > Can't argue with that. > > But that is your ego talking and the machines do not belong to you but > to the institute. Your ego has no place in that. You're right, thanks for the reminder. But also note the smiley. I know my boss (who is also into geeky things) would also like this - as long as it would work. >> - I am probably the only one who can administrate them. > > This is not a benefit. It is a severe liability. That's why I listed it also on the contra side. Forgot to add a smiley here, it was not meant seriously. But when I think about it... the others also do not know much about Fedora. Not even I do this well. There you use 'yum install <package>', with Gentoo it's 'emerge <package>'. Daily work would be similar. Upgrades would be a different thing, though. Gentoo's portage blockers would not be understood easily, they would prefer to take the servers down and just install the current Fedora distro. Which hopefully would work. >> Cons: >> - If something will not work with this not so common >> (meta)distribution, people will say "always trouble with your Gentoo >> Schmentoo, it works fine in Fedora". Fedora is more mainstream, if >> something does not work there, then it's okay for the people to accept >> it. > > Those same people are likely to say the same about linux vs windows. Right, but we already have Linux, and we need it for our software. Gentoo would not really be needed. >> - I am probably the only one who can administrate them. I think Gentoo >> is easier to maintain in the long run, but only when you take the time >> to learn it. With Fedora, you do not need much more than the 'yum >> install' command. There is no need to read complicated X.org upgrade >> guides and such. >> >> I think I already made my decision, but I am still interested in your >> opinions, maybe some of you are in a similar position and like to share >> your experiences. Whether I will be allowed to use Gentoo is another >> question, I guess my boss will not like my idea at first, and I am not >> even sure if he is right. But maybe I can test-install Gentoo on one >> machine in a chroot, and see if things work fine. > > Depends how critical these machines are. If you want to change them just > because you feel like it, then I do not see how that can possibly be a > valid reason. > > Remember, the institute's needs and desires trump yours every time No, it's not just because I feel like it. The main advantages would be: - No downtime between upgrades. Our jobs run for several days, every downtime has to be planned in advance. People understand this, but they do not like it. They would be very happy if this were not longer necessary. And I would not fear that during the upgrade something breaks, and it would take me long to fix it. - I know this distro well, and this is not at all true about Fedora. I know how to fix problems, I know how things work here. I would feel better with Gentoo, more competent. It just does not feel so well to administrate Fedora. Thanks for your opinions, Alan. As always. Wonko ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Gentoo for many servers (was: Re: [gentoo-user] executing commands on lots of servers at once) 2009-11-14 22:09 ` Alex Schuster @ 2009-11-15 6:20 ` Joshua Murphy 0 siblings, 0 replies; 16+ messages in thread From: Joshua Murphy @ 2009-11-15 6:20 UTC (permalink / raw To: gentoo-user On Sat, Nov 14, 2009 at 5:09 PM, Alex Schuster <wonko@wonkology.org> wrote: > Alan McKinnon writes: > >> On Saturday 14 November 2009 19:36:06 Alex Schuster wrote: >>> Alan McKinnon wrote: > >>>> clusterssh will let you log into many machines at once and run emerge >>>> -avuND world everywhere >>> This is way cool. I just started using it on eight Fedora servers I am >>> administrating. Nice, now this is an improvement over my 'for $h in >>> $HOSTS; do ssh $h "yum install foo"; done' approach. >> >> I feel your pain :-) >> >> We used to have the same problem adding new admins to 87 machines. Now >> we have a bespoke provisioner that does it all. > > Sorry, I just do not get 'bespoke provisioner'. Some sort of software, > like clusterssh? Or a person, one admin instead of many? > > >>> What do you guys think about using Gentoo for servers? At the institute >>> I partially work we chose Fedora. There is no special reason for that - >>> we already had some Fedora machines, the setup seemed to work, the >>> reputation was good, so we kept it. That was okay for me, why choose >>> many different environments and learn everything again. I mentioned >>> Gentoo, but did not really suggest to actually use it. Maybe I should >>> have. >> >> I'm a huge fan of Gentoo > > Now who would have thought of that! > >> and all my personal machines (except the new netbook have run it for the >> last 5 years. >> >> But I will never install Gentoo on a production server at work. >> >> Why? >> >> Because it is too time consuming, because no two machines are set up the >> same, because I can't trust that other admins used the flags they should >> have. So updates become a case of logging into 80+ machines individually >> and doing emerge world by hand. Gentoo allows you to customize things to >> the nth degree - that is it's strength - so people WILL use this one >> discriminating factor. >> >> If OTOH I had a server farm of 80+ machines, all identical, I'd put >> Gentoo on them in a flash. But I don't have that > > Of our 8 machines, 7 are essentially the same and differ only in hard > drive space and CPU speed. The other machine is Intel, not AMD, and needs > different IDE drivers. At the moment it has a different initrd (I set up a > minimal fedora install to generate it after the cloned system did not > boot), the rest is - apart from some config files - identical. > > So I would make sure that about everything is exactly the same, well, > maybe except for hostnames, udev net-persistent-rules, ssh keys... what > more? > The last, a little different machine is a problem though. With optimized > CFLAGS, this one would have to compile all stuff again, while for the > others I could use binpkgs. Updating them all with clusterssh should not > be much more work than updating a single one. Well, not completely true, I > would have the double work, as I would upgrade one server first to test if > there are problems, and then do it for the others. Maybe I could use the > special machine to test stuff, and then update all the others. > > If they would differ, Gentoo would of course be too much work. I already > have this problem now... there is my desktop machine, my notebook running > a Gentoo VM, a second desktop machine at my other home, the living-room > machine of my flat share, the machine of a fried I also administrate, the > server of my flat share I need to set up again... and clusterssh is no > option here. My potentially ill informed thoughts on the above issues/ideas: 1) Pick one machine to host both your make.conf as well as your portage tree and distfiles, potentially splitting them into separate nfs mounts shared out for the rest of the hosts (having the portage tree itself ro on all but its owning machine forces centralization of syncing). 2) /etc/make.conf should simply be a symlink to the centrally located copy. If you must use binpackages, set march to something that will run on every machine involved, then set mcpu to whatever machine is most common if you want to get just a bit more performance here or there. If you don't mind compiling on every host, though, set portage niceness to something friendly to your users and march to native (if you plan to use distcc, this is a BAD idea, use the binpackages). 3) use a replaceable (otherwise identical to the others, and therefore able to be brought back online by just cloning it over) system for your testing and keep frequent scheduled backups of whichever system plays host to your portage tree, binpackages, and distfiles. 4) build your kernel with built in drivers for every piece of boot-time essential hardware in your systems. You'll still be on a far cleaner setup than a mass produced distro provided kernel, you'll only need to maintain one for all your systems, and you'll only have one kernel to worry about building against if you need any out-of-kernel modules as well. 5) script the changing of ssh host keys (or even redistribution of them, if you ), removal of persistent net rules, and prompting for the setting of host name and you'll have a nice, tiny, postinstall tool for the rare case in which you need to re-deploy a system. You may wish to restore things like ssh host keys from backups as well, in the case of re-deployment of systems, since changing them means adjusting known hosts lists elsewhere >>> Now I am thinking about a Gentoo installation instead. >>> >>> Pros: >>> - Continuous updates, no downtime for upgrading, only when I decide to >>> install a new kernel. This is really really cool. I fear the upgrade >>> from Fedora 10 to 12 which has to be done soon. >> >> Do not upgrade, especially not with a version jump of 2 or more. If you >> have a lot of machines, I assume you are a decent shop, and that you >> have some form of formal process for upgrades and changes. > > Not really, I think. We are not very professional I must admit. We have > two capable admins, but one is specialized in network stuff and Windows, > the other has to do with our big Sun servers, huuge storage systems and > such. They do not much about the Linux cluster. Another user sometimes > installs a package on a machine, but usually I do this. For me, it is not > my main job, I work only about ten hours per week there, mostly being some > 100 km away. > We are a research institute. We do neurological research, PET and MRI > tomography. The Linux servers do number crunching, and of course they > should work and have good uptimes, but it is not as important as if we > were an ISP. > >> What you do instead is a formal migration - copy the data off, >> reinstall, restore data. > > Advice noted. Yes, this sounds like the better idea, giving a cleaner > setup. And if some things break I do not have to wonder if it was some > strange side effect from the upgrade process. > >> If you can't afford to do that every six or twleve months, then >> I have to ask - what the hell is the organization doing using a distro >> that is unsupported after 12 months? > > Well, I do not think this was considered much. One machine was set up with > Fedora for no specific reason, and we kept this distro then. This does not > sound too professional, I know. BTW, what distro would you suggest? In the times I've used it, while a bit overweight for my tastes in server work, Ubuntu handled updates quite gracefully, but needed reboots somewhat often. You might get the same or better out of Debian, as it's created a little less directly to be destktop centric, while being the source of the package management that gives Ubuntu what advantages it might have for the role. >>> - Some improvement in speed. Those machines do A LOT of >>> numbercrunching, which jobs often lasting for days, so even small >>> improvements would be nice. >> >> Don't fool yourself. Unless you need what Google needs, there is very >> little speed difference between Gentoo and Fedora. I/O improvements you >> need can be easily gotten by fiddling the kernel tuning knobs. > > I know the difference will not be huge, I see this as a little bonus - > nice if is there, but nothing really important. But in the comparison with > Ubuntu that came in a thread a few weeks ago, for some applications the > speed increase was about 30 percent. Although I would not necessarily > expect the difference to be noticeable, I would also not be surprised too > much if it were noticeable for some number-crunching applications if they > were optimized for the CPU. Are the pieces of software you're using for the number crunching work open source, and will you be recompiling those on Gentoo, with all the optimizations, as well? In the long run, if they're not, you'll get far more out of the I/O improvements Alan mentioned than you ever would out of aggressive use of cflags. >>> - Easier debugging. When things do not work, I think it's easier to >>> dig into the problem. No fancy, but sometimes buggy GUIs hiding basic >>> functionality. >> >> Errrrrrrrrrrrrrrrmmmmmmmmmmmmmm, Fedora does not require a GUI :-) > > Right, and now that I think of it I do not use it anyway... Well, I did do > some things with netsetup (or whatever it's called), now that I know the > system a little better I edit things directly in /etc/sysconfig. > But the installer is a GUI, right? And if I remember this correctly, I > cannot even switch to a text console and do stuff there while installing. > Or I could, but did not have utilities like LVM. Something like that. I > have to use the installer and its capabilities. > >>> - Heck, Gentoo is _cooler_ than typical distributions. And emerging >>> with distcc on about 8*4 cores would be fun :) >> >> Can't argue with that. >> >> But that is your ego talking and the machines do not belong to you but >> to the institute. Your ego has no place in that. > > You're right, thanks for the reminder. But also note the smiley. I know my > boss (who is also into geeky things) would also like this - as long as it > would work. If you've a moderately capable system sitting spare, throw virtualbox or similar on it and bring up a few vms to test the setup in (since with that, you can get away with ). My little core 2 here can handle 3-4 vms without fussing at all, and that's with >>> - I am probably the only one who can administrate them. >> >> This is not a benefit. It is a severe liability. > > That's why I listed it also on the contra side. Forgot to add a smiley > here, it was not meant seriously. > But when I think about it... the others also do not know much about > Fedora. Not even I do this well. There you use 'yum install <package>', > with Gentoo it's 'emerge <package>'. Daily work would be similar. > Upgrades would be a different thing, though. Gentoo's portage blockers > would not be understood easily, they would prefer to take the servers down > and just install the current Fedora distro. Which hopefully would work. > > >>> Cons: >>> - If something will not work with this not so common >>> (meta)distribution, people will say "always trouble with your Gentoo >>> Schmentoo, it works fine in Fedora". Fedora is more mainstream, if >>> something does not work there, then it's okay for the people to accept >>> it. >> >> Those same people are likely to say the same about linux vs windows. > > Right, but we already have Linux, and we need it for our software. Gentoo > would not really be needed. > >>> - I am probably the only one who can administrate them. I think Gentoo >>> is easier to maintain in the long run, but only when you take the time >>> to learn it. With Fedora, you do not need much more than the 'yum >>> install' command. There is no need to read complicated X.org upgrade >>> guides and such. >>> >>> I think I already made my decision, but I am still interested in your >>> opinions, maybe some of you are in a similar position and like to share >>> your experiences. Whether I will be allowed to use Gentoo is another >>> question, I guess my boss will not like my idea at first, and I am not >>> even sure if he is right. But maybe I can test-install Gentoo on one >>> machine in a chroot, and see if things work fine. >> >> Depends how critical these machines are. If you want to change them just >> because you feel like it, then I do not see how that can possibly be a >> valid reason. >> >> Remember, the institute's needs and desires trump yours every time > > No, it's not just because I feel like it. The main advantages would be: > - No downtime between upgrades. Our jobs run for several days, every > downtime has to be planned in advance. People understand this, but they do > not like it. They would be very happy if this were not longer necessary. > And I would not fear that during the upgrade something breaks, and it > would take me long to fix it. > - I know this distro well, and this is not at all true about Fedora. I > know how to fix problems, I know how things work here. I would feel better > with Gentoo, more competent. It just does not feel so well to administrate > Fedora. > > Thanks for your opinions, Alan. As always. > > Wonko As a final note... whatever path you take in either implementing a new setup or just updating the old one, document it, and especially document guides for upkeep and general maintenance. Your boss, Windows guy, and Sun guy're going to be like fish out of water if you get this whole thing put in place and get hit by a bus the next day. *This* is why the "I'm the only one that can.." bit is such a dangerous thing. It's not the fear that you'll try to use it as a bargaining chip down the road, given that, they'd just take the hit, replace you, and then replace the setup with a better documented one... it's the fear that if for any reason you drop out of the picture for them, they're stuck with the cost of doing that. Period. (This is also why, when actively and intentionally done, it's a fire-able offense in many places) -- Poison [BLX] Joshua M. Murphy ^ permalink raw reply [flat|nested] 16+ messages in thread
* [gentoo-user] Re: Gentoo for many servers 2009-11-14 17:36 ` Gentoo for many servers (was: Re: [gentoo-user] " Alex Schuster 2009-11-14 19:26 ` Alan McKinnon @ 2009-11-15 12:09 ` Andreas Niederl 1 sibling, 0 replies; 16+ messages in thread From: Andreas Niederl @ 2009-11-15 12:09 UTC (permalink / raw To: gentoo-user Alex Schuster wrote: > Alan McKinnon wrote: > >> clusterssh will let you log into many machines at once and run emerge >> -avuND world everywhere > > This is way cool. I just started using it on eight Fedora servers I am > administrating. Nice, now this is an improvement over my 'for $h in > $HOSTS; do ssh $h "yum install foo"; done' approach. You could have a look at app-admin/puppet [1][2] which supposedly takes car of these things. [...] > Now I am thinking about a Gentoo installation instead. > > Pros: > - Continuous updates, no downtime for upgrading, only when I decide to > install a new kernel. This is really really cool. I fear the upgrade from > Fedora 10 to 12 which has to be done soon. > - Some improvement in speed. Those machines do A LOT of numbercrunching, > which jobs often lasting for days, so even small improvements would be > nice. > - Easier debugging. When things do not work, I think it's easier to dig > into the problem. No fancy, but sometimes buggy GUIs hiding basic > functionality. These two things would probably be your best selling points for your idea. > - Heck, Gentoo is _cooler_ than typical distributions. And emerging with > distcc on about 8*4 cores would be fun :) Being 'cool' doesn't count, at least last time I looked. > - I am probably the only one who can administrate them. That is a huge disadvantage. > Cons: > - If something will not work with this not so common (meta)distribution, > people will say "always trouble with your Gentoo Schmentoo, it works fine > in Fedora". Fedora is more mainstream, if something does not work there, > then it's okay for the people to accept it. > - I fear that big packages like Matlab are made for and tested on the > typical distributions, and may have problems with the not-so-common > Gentoo. I think someone here just had such a problem with Mathematica > (which we do currently not use). [...] If you're using commercial software which is only supported by Redhat, Novell, etc. then you should think twice about replacing it. But I'm guessing that those packages don't have to be installed on every machine. So, I'd suggest that you use Gentoo on those boxes where you'd have the biggest advantage using it and no or minimal disadvantages. > - I am probably the only one who can administrate them. I think Gentoo is > easier to maintain in the long run, but only when you take the time to > learn it. With Fedora, you do not need much more than the 'yum install' > command. There is no need to read complicated X.org upgrade guides and > such. [...] Please do your colleagues and successors a favor and document your whole setup really good. Regards, Andi [1] http://reductivelabs.com/products/puppet/ [2] http://log.onthebrink.de/2008/05/using-puppet-on-gentoo.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-29 18:36 [gentoo-user] executing commands on lots of servers at once James 2009-10-29 18:57 ` Alan McKinnon @ 2009-10-29 21:36 ` Dirk Heinrichs 2009-10-29 21:44 ` James 2009-10-31 12:03 ` he zhitong 2 siblings, 1 reply; 16+ messages in thread From: Dirk Heinrichs @ 2009-10-29 21:36 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: Text/Plain, Size: 347 bytes --] Am Donnerstag 29 Oktober 2009 19:36:27 schrieb James: > I have a cluster of Gentoo boxes I'd like to update all at once, along > with emerging specific packages to the servers simultaneously. > > Does anyone have any experience(s) with a good utility for doing this? Try out the debian (cough) distributed shell (dsh). HTH... Dirk [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 190 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-29 21:36 ` [gentoo-user] executing commands on lots of servers at once Dirk Heinrichs @ 2009-10-29 21:44 ` James 2009-10-29 22:52 ` Neil Bothwick 0 siblings, 1 reply; 16+ messages in thread From: James @ 2009-10-29 21:44 UTC (permalink / raw To: gentoo-user I'll definitely take a look at this, Dirk. I just took a look at the dsh site and it looks pretty interesting. I'll have to try various different solutions (clusterssh, dsh, etc.) to get a good feel for the pros and cons of each. -j On Thu, Oct 29, 2009 at 5:36 PM, Dirk Heinrichs <dirk.heinrichs@online.de> wrote: > Am Donnerstag 29 Oktober 2009 19:36:27 schrieb James: >> I have a cluster of Gentoo boxes I'd like to update all at once, along >> with emerging specific packages to the servers simultaneously. >> >> Does anyone have any experience(s) with a good utility for doing this? > > Try out the debian (cough) distributed shell (dsh). > > HTH... > > Dirk > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-29 21:44 ` James @ 2009-10-29 22:52 ` Neil Bothwick 2009-10-30 10:42 ` Arnau Bria 0 siblings, 1 reply; 16+ messages in thread From: Neil Bothwick @ 2009-10-29 22:52 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: text/plain, Size: 571 bytes --] On Thu, 29 Oct 2009 17:44:57 -0400, James wrote: > I'll definitely take a look at this, Dirk. I just took a look at the > dsh site and it looks pretty interesting. I'll have to try various > different solutions (clusterssh, dsh, etc.) to get a good feel for > the pros and cons of each. There's also tentakel, just as you were ready to make a choice :) -- Neil Bothwick Standard: (n., adj.) a design target which manufacturers may embellish, improve upon, or ignore as they wish, so long as it can be used profitably in their advertising. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-29 22:52 ` Neil Bothwick @ 2009-10-30 10:42 ` Arnau Bria 2009-10-30 11:07 ` Neil Bothwick 0 siblings, 1 reply; 16+ messages in thread From: Arnau Bria @ 2009-10-30 10:42 UTC (permalink / raw To: gentoo-user On Thu, 29 Oct 2009 22:52:31 +0000 Neil Bothwick wrote: Hi Neil, > There's also tentakel, just as you were ready to make a choice :) I'm taking a look on it, but are few examples... how does tentakel behaves when passing | , $, etc.. ? Cheers -- Arnau Bria http://blog.emergetux.net Bombing for peace is like fucking for virginity ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-30 10:42 ` Arnau Bria @ 2009-10-30 11:07 ` Neil Bothwick 0 siblings, 0 replies; 16+ messages in thread From: Neil Bothwick @ 2009-10-30 11:07 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: text/plain, Size: 540 bytes --] On Fri, 30 Oct 2009 11:42:39 +0100, Arnau Bria wrote: > > There's also tentakel, just as you were ready to make a choice :) > I'm taking a look on it, but are few examples... > > how does tentakel behaves when passing | , $, etc.. ? The commands are executed in a remote shell, but interpreted by the local shell when you call tentakel. So if you want a variable or pipe to be referenced on the remote shell, you'll have to quote or escape it. -- Neil Bothwick COBOL: Completely Obsolete Business Oriented Language [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [gentoo-user] executing commands on lots of servers at once 2009-10-29 18:36 [gentoo-user] executing commands on lots of servers at once James 2009-10-29 18:57 ` Alan McKinnon 2009-10-29 21:36 ` [gentoo-user] executing commands on lots of servers at once Dirk Heinrichs @ 2009-10-31 12:03 ` he zhitong 2 siblings, 0 replies; 16+ messages in thread From: he zhitong @ 2009-10-31 12:03 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: text/plain, Size: 617 bytes --] On Fri, Oct 30, 2009 at 2:36 AM, James <jtp@nc.rr.com> wrote: > I have a cluster of Gentoo boxes I'd like to update all at once, along > with emerging specific packages to the servers simultaneously. > > Does anyone have any experience(s) with a good utility for doing this? > I've seen a few scripts online for accomplishing this but I'm not sure > how these scripts would handle the large amount of output that emerge > generates from all the servers simultaneously. > > Thoughts / ideas appreciated. > > -j > > you can use screen, introduced in last GMN. http://www.gentoo.org/news/en/gmn/20081130-newsletter.xml [-- Attachment #2: Type: text/html, Size: 1027 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2009-11-15 14:01 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-10-29 18:36 [gentoo-user] executing commands on lots of servers at once James 2009-10-29 18:57 ` Alan McKinnon 2009-10-29 21:42 ` James 2009-10-30 7:03 ` To James and James (was Re: [gentoo-user] executing commands on lots of servers at once) Dirk Heinrichs 2009-11-04 14:28 ` [gentoo-user] Re: To James and James (was " James 2009-11-14 17:36 ` Gentoo for many servers (was: Re: [gentoo-user] " Alex Schuster 2009-11-14 19:26 ` Alan McKinnon 2009-11-14 22:09 ` Alex Schuster 2009-11-15 6:20 ` Joshua Murphy 2009-11-15 12:09 ` [gentoo-user] Re: Gentoo for many servers Andreas Niederl 2009-10-29 21:36 ` [gentoo-user] executing commands on lots of servers at once Dirk Heinrichs 2009-10-29 21:44 ` James 2009-10-29 22:52 ` Neil Bothwick 2009-10-30 10:42 ` Arnau Bria 2009-10-30 11:07 ` Neil Bothwick 2009-10-31 12:03 ` he zhitong
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox