[gentoo-user] Managing multiple Gentoo systems

[gentoo-user] Managing multiple Gentoo systems

[Grant]

After a frustrating experience with a Linksys WRT54GL, I've decided to
stick with Gentoo routers.  This increases the number of Gentoo
systems I'm responsible for and they're nearing double-digits.  What
can be done to make the management of multiple Gentoo systems easier?
I think identical hardware in each system would help a lot but I'm not
sure that's practical.  I need to put together a bunch of new
workstations and I'm thinking some sort of server/client arrangement
with the only Gentoo install being on the server could be appropriate.

- Grant

Re: [gentoo-user] Managing multiple Gentoo systems

[Dale]

Grant wrote:
> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.  This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits.  What
> can be done to make the management of multiple Gentoo systems easier?
> I think identical hardware in each system would help a lot but I'm not
> sure that's practical.  I need to put together a bunch of new
> workstations and I'm thinking some sort of server/client arrangement
> with the only Gentoo install being on the server could be appropriate.
>
> - Grant
>
>
>    

If I had a dozen or so rigs, I would want them to be identical at least 
as far as mobos and such.  Drives and such could vary depending on other 
things tho.  If you use the same USE flags, you could update one and 
then copy /etc/portage/* to the other rigs then the updates could just 
be binary installs from the first rig updated.

The only thing that might bite you is if you have different uses for 
each rig.  If you have different packages installed, then you may have 
to compile for that.  Most likely tho, you could use binaries for at 
least the system packages and save some time.

I think Alan has some rigs and may be able to share more ideas.  Maybe 
some others could comment that have dealt with the same thing.

Dale

:-)  :-)

Re: [gentoo-user] Managing multiple Gentoo systems

[Alex Schuster]

Grant writes:

> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.  This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits.  What
> can be done to make the management of multiple Gentoo systems easier?

In the case of editing identical files on different machines (Gentoo or
not), I like clusterssh. It opens an xterm for every destination host
and an additional command window. What you type here happens in all windows.

	Wonko

Re: [gentoo-user] Managing multiple Gentoo systems

[Stroller]

On 2 July 2011, at 23:14, Grant wrote:

> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.

Out of curiosity, could you tell us more about this experience?

The WRT54G(L) is quite dated, and the OpenWRT devs recommend against trying to do anything fancy on it. 

In another post you mentioned that you have a TP-Link TL-WR1043ND, which is a bunch newer, I think, and should run OpenWRT quite well. 

Even if you were to use an Atom-based Gentoo system, I'm sure it would use more power than an ARM- or MIPS-based commodity router - perhaps twice as much. 

Stroller.

Re: [gentoo-user] Managing multiple Gentoo systems

[Grant]

>> After a frustrating experience with a Linksys WRT54GL, I've decided to
>> stick with Gentoo routers.
>
> Out of curiosity, could you tell us more about this experience?

Sure, I was using the stock firmware and I didn't like that you
couldn't specify a source IP address when punching a hole in the
firewall for a particular port, and I also couldn't coax "Remote
Access" into working no matter what I tried.

> The WRT54G(L) is quite dated, and the OpenWRT devs recommend against trying to do anything fancy on it.

I chose the WRT54GL because it has the best ratings on newegg.com.  I
looked into OpenWRT once but decided against it after I decided
installation and possibly management was not nearly as trivial as I
had imagined.

> In another post you mentioned that you have a TP-Link TL-WR1043ND, which is a bunch newer, I think, and should run OpenWRT quite well.

I got rid of the TL-WR1043ND a while back because I couldn't get
packet shaping to work with the stock firmware no matter how I tried.

At this point I've sworn off mystery boxes.  I even had a Dlink router
die on me recently.  If I'm not using mystery boxes for greater
hardware reliability, why am I using them?  Power consumption would be
a good reason but it's not worth it IMO.

- Grant

Re: [gentoo-user] Managing multiple Gentoo systems

[Roman Zilka]

Grant (Sat, 2 Jul 2011 15:14:38 -0700):
> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.  This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits.  What
> can be done to make the management of multiple Gentoo systems easier?
> I think identical hardware in each system would help a lot but I'm not
> sure that's practical.  I need to put together a bunch of new
> workstations and I'm thinking some sort of server/client arrangement
> with the only Gentoo install being on the server could be appropriate.

I used to do it the rsync way. Dozens of machines with varying
hardware, although not profoundly. All machines had all the software any
single machine could need. CFLAGS, kernel config and such were filled
with the greatest common divisor of all the boxes. There was one
'reference box' which did all the compiling and from which all the
other machines would rsync /, minus selected variables, such
as /dev, /etc/mtab, /mnt, /proc, /var/log, etc.

Another set of things excluded from rsync were things that do not
handle themselves locally, but are different accross computers. I had
groups of computers that I wanted to, for example, run different set of
boot-up services. That means that /etc/runlevels was excluded from
rsync'ing, but what was being rsynced were the dirs 
/etc/runlevels-group{1,2,3,...} and every machine had a local
symlink /etc/runlevels -> one of the runlevels-groupX. Administrative
tasks are still limited to working with the single reference box and
its single filesystem (plus a few more runlevels-group's and similar
exceptions).

That way you can have specialized runlevel layouts, specialized fstabs
and other configs, etc. Even specialized kernels for every group of
machines, in theory.

There was also a script being distributed that was called by local crons
and allowed for batch rebooting at midnight, or whatever you may need
to run locally.

You may want to check out lsyncd for keeping dirs in sync in a smart
way on-the-fly, but rsyncing from a cronjob at a safe moment may be the
recommended option, depending on your environment. By 'safe' I mean
when there are no users logged in, for example.

The most tricky part was fine-tuning the set of stuff to exclude
from rsyncing. But if sure can be done in reasonable environments.

HTH,
-rz

Re: [gentoo-user] Managing multiple Gentoo systems

[Simon]

[-- Attachment #1: Type: text/plain, Size: 3721 bytes --]

On Sat, Jul 2, 2011 at 6:14 PM, Grant <emailgrant@gmail.com> wrote:

> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.  This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits.  What
> can be done to make the management of multiple Gentoo systems easier?
> I think identical hardware in each system would help a lot but I'm not
> sure that's practical.  I need to put together a bunch of new
> workstations and I'm thinking some sort of server/client arrangement
> with the only Gentoo install being on the server could be appropriate.
>
> - Grant
>
>

I've been managing 3 very very different gentoo systems for more than 12
months now.  I used to have 2 more machines but I decomissioned them for
they were old and noisy and it's summer and it's hot enough like that! haha

I do not have a lazy-proof way of administrating this whole stuff, but I've
come up with some kind of methodology that helps...

I found it really helps to have the same profile on all machines, same
/etc/portage everything.  The /etc/make.conf can vary, but I would make it
vary in the most minimal way.  For example, all my kernels and
/etc/make.conf are compiled for an i686, global use flags are exactly the
same... in fact the only difference is my host has extra make option "-j5"
and their distfiles mirrors will point to my own computers.  Basically,
downgrade all computers to the lowest common denominator of all your
computers.

I don't use binpkgs but I'm working on that...  and also for the world file,
I have 2 world file "styles", one has "everything" in it and the other is
just the minimal stuff.  I used to maintain the world files directly but
I've opted for creating 2 ebuilds in my own portage overlay which are
basically called sys-apps/everything and another called
sys-apps/minimalist.  Maintenance of world files moved to the maintenance of
these ebuilds.  Also, it helps to have your faster computer install all
packages that would be installed on the others.  If there's any trouble
doing the emerge on that one, you'll have the same problems on the others.
If the solution was to add some keywords in /etc/portage or something like
that, then you can copy the change over to your other PCs and their emerge
should be fine.

To reduce network load, I sync my master server (my server online) against
official sync server, then I sync my main computer at home against my
server, then I sync any other computers at home against the main computer at
home.

But the activity of making your systems up-to-date will vary in intensity
depending on how well it goes.  When things are fine, it used to take 30
minutes to get all my systems updated, with the slowest ones taking 2-3
times that but requiring no intervention until the end.

I'm currently in the process of writing a script that fully automates the
update process.  It basically tries things like "emerge -uDN --keep-going
world" and checks for errors, if there are, it then fires a few safe
"responses" which have helped in the past.  For example, if the above emerge
fails, it then tries a revdep-rebuild, an lafilefixer, python-updater and a
few more, and then it restart from scratch trying the emerge again, over and
over.  Rarely, this will not work and will require that I mask something,
add some keyword or update my ebuild to get rid of some package.  The script
is clearly not intelligent and is wasting a lot of time, but it eventually
gets it done.  Upgrades now take at least 1 hour, but are fully unattended.

Also, I think Wonko's suggestion is pretty good in combination with my
suggestions, I haven't tried but it does seem very attractive.

Simon

[-- Attachment #2: Type: text/html, Size: 4200 bytes --]

Re: [gentoo-user] Managing multiple Gentoo systems

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 585 bytes --]

On Sun, 3 Jul 2011 16:46:30 -0400, Simon wrote:

> I have 2 world file "styles", one has "everything" in it and the other
> is just the minimal stuff.  I used to maintain the world files directly
> but I've opted for creating 2 ebuilds in my own portage overlay which
> are basically called sys-apps/everything and another called
> sys-apps/minimalist.  Maintenance of world files moved to the
> maintenance of these ebuilds.

You can do much the same, but in a simpler way, by using sets.


-- 
Neil Bothwick

Master of all I survey (at the moment, empty pizza boxes)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Managing multiple Gentoo systems

[Nicolas Sebrecht]

On Sat, Jul 02, 2011 at 03:14:38PM -0700, Grant wrote:
> 
> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.  This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits.  What
> can be done to make the management of multiple Gentoo systems easier?
> I think identical hardware in each system would help a lot but I'm not
> sure that's practical.  I need to put together a bunch of new
> workstations and I'm thinking some sort of server/client arrangement
> with the only Gentoo install being on the server could be appropriate.

I maintain multiple Gentoo we mostly use as KVM hosts systems (and
coming embedded routers). As KVM hosts, some of them are very sensible.
Due to the contracts to our customers, I have to do with various update
strategies on top of various hardware.

I've set up a private Gentoo mirror in order to follow updates nicely
(all customers want to update slowly). Well, it's not a true mirror. To
be able to upgrade old systems, I do "private" releases of Gentoo
approximately once a month. A full mirror of all releases would be too
much data. So, I only fetch portage tree and packages from a list I
maintain manually (emerge sucks at that game, by the way). Data is
stored on a nilfs filesystem to improve snapshots size on disk.

-- 
Nicolas Sebrecht

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Grant]

>> After a frustrating experience with a Linksys WRT54GL, I've decided to
>> stick with Gentoo routers.  This increases the number of Gentoo
>> systems I'm responsible for and they're nearing double-digits.  What
>> can be done to make the management of multiple Gentoo systems easier?
>> I think identical hardware in each system would help a lot but I'm not
>> sure that's practical.  I need to put together a bunch of new
>> workstations and I'm thinking some sort of server/client arrangement
>> with the only Gentoo install being on the server could be appropriate.
>
> I maintain multiple Gentoo we mostly use as KVM hosts systems (and
> coming embedded routers). As KVM hosts, some of them are very sensible.
> Due to the contracts to our customers, I have to do with various update
> strategies on top of various hardware.

Thanks to everyone for some very juicy tidbits.  I'm rearranging my
thinking on all of this.  I think the key for me may be to combine
systems with separate functions in the same physical location into a
single system.  Does the KVM thing work well?  Running a bunch of
workstations as nothing more than wireless KVM setups on the same
system?  I should be able to cut my Gentoo systems down to just a few.
 Basically one at each physical location.

- Grant

[gentoo-user] Re: Managing multiple Gentoo systems

[Nicolas Sebrecht]

The 06/07/11, Grant wrote:
> >> After a frustrating experience with a Linksys WRT54GL, I've decided to
> >> stick with Gentoo routers.  This increases the number of Gentoo
> >> systems I'm responsible for and they're nearing double-digits.  What
> >> can be done to make the management of multiple Gentoo systems easier?
> >> I think identical hardware in each system would help a lot but I'm not
> >> sure that's practical.  I need to put together a bunch of new
> >> workstations and I'm thinking some sort of server/client arrangement
> >> with the only Gentoo install being on the server could be appropriate.
> >
> > I maintain multiple Gentoo we mostly use as KVM hosts systems (and
> > coming embedded routers). As KVM hosts, some of them are very sensible.
> > Due to the contracts to our customers, I have to do with various update
> > strategies on top of various hardware.
> 
> Thanks to everyone for some very juicy tidbits.  I'm rearranging my
> thinking on all of this.  I think the key for me may be to combine
> systems with separate functions in the same physical location into a
> single system.  Does the KVM thing work well?

KVM itself works very well here, even with advanced features such as KSM
pages sharing.

The difficulties come with Microsoft products for both good integration
and perfomance (I would recommend RAW format, iSCSI or plain physical
partition instead of qcow2, for example). That beeing said, I finally
have all working well for XP, NT2003 and 2008 servers.

I use libvirt on top of KVM which is in the way to become very good AFA
you don't rely on libvirt's API which tend to move a lot.

>                                                Running a bunch of
> workstations as nothing more than wireless KVM setups on the same
> system?  I should be able to cut my Gentoo systems down to just a few.
>  Basically one at each physical location.

I would be much sceptical for both workstations and wireless guests than
for servers:

1) For workstations, things are currently changing with the very recent
and "not much usable with Gentoo, yet" spice software. I expect a lot of
improvments in the coming months for this use case. I would say it's not
ready for production, yet.

2) About wireless virtualization it's highly depending on what you aim
to do, especially if you intend to use the PCI passthrough feature to
give your wireless card to a guest. For this to work, you MUST have your
hardware (CPU, motherboard and PCI card) VT-d compatible which is
currently NOT a piece of cake, today. It relies on industry and
manufacturers moving not as fast as software. I would expect more widely
VT-d cards in the coming _years_.

Now, if you intend to use the wireless card from you hosts and share
networks using bridge utilities it _MAY_ be OK: Linux bridging does not
always work with all wireless cards (see http://tinyurl.com/ylcutwv for
more information).


In a more general approach, when I hear "routers" and "wireless" I'm
more thinking _embedded_. KVM/qemu would only help you to build your
target systems.


For embedded (or tiny, at least) systems, I would not use LXC.

The drawback with Gentoo is that the current official uclibc stage3 for
embedded/tiny systems is obsolete and marked as experimental. In facts,
it's very _hard_ if not impossible to use it these days. Making your own
cross-compilation environment is not a piece of cake (too), even with
dedicated tools such as crossdev. This topic would ask its own book.
So, if you want to try Gentoo embedded save your time by working on
unofficial stage3.

-- 
Nicolas Sebrecht

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Grant]

>> >> After a frustrating experience with a Linksys WRT54GL, I've decided to
>> >> stick with Gentoo routers.  This increases the number of Gentoo
>> >> systems I'm responsible for and they're nearing double-digits.  What
>> >> can be done to make the management of multiple Gentoo systems easier?
>> >> I think identical hardware in each system would help a lot but I'm not
>> >> sure that's practical.  I need to put together a bunch of new
>> >> workstations and I'm thinking some sort of server/client arrangement
>> >> with the only Gentoo install being on the server could be appropriate.
>> >
>> > I maintain multiple Gentoo we mostly use as KVM hosts systems (and
>> > coming embedded routers). As KVM hosts, some of them are very sensible.
>> > Due to the contracts to our customers, I have to do with various update
>> > strategies on top of various hardware.
>>
>> Thanks to everyone for some very juicy tidbits.  I'm rearranging my
>> thinking on all of this.  I think the key for me may be to combine
>> systems with separate functions in the same physical location into a
>> single system.  Does the KVM thing work well?
>
> KVM itself works very well here, even with advanced features such as KSM
> pages sharing.
>
> The difficulties come with Microsoft products for both good integration
> and perfomance (I would recommend RAW format, iSCSI or plain physical
> partition instead of qcow2, for example). That beeing said, I finally
> have all working well for XP, NT2003 and 2008 servers.
>
> I use libvirt on top of KVM which is in the way to become very good AFA
> you don't rely on libvirt's API which tend to move a lot.
>
>>                                                Running a bunch of
>> workstations as nothing more than wireless KVM setups on the same
>> system?  I should be able to cut my Gentoo systems down to just a few.
>>  Basically one at each physical location.
>
> I would be much sceptical for both workstations and wireless guests than
> for servers:
>
> 1) For workstations, things are currently changing with the very recent
> and "not much usable with Gentoo, yet" spice software. I expect a lot of
> improvments in the coming months for this use case. I would say it's not
> ready for production, yet.
>
> 2) About wireless virtualization it's highly depending on what you aim
> to do, especially if you intend to use the PCI passthrough feature to
> give your wireless card to a guest. For this to work, you MUST have your
> hardware (CPU, motherboard and PCI card) VT-d compatible which is
> currently NOT a piece of cake, today. It relies on industry and
> manufacturers moving not as fast as software. I would expect more widely
> VT-d cards in the coming _years_.
>
> Now, if you intend to use the wireless card from you hosts and share
> networks using bridge utilities it _MAY_ be OK: Linux bridging does not
> always work with all wireless cards (see http://tinyurl.com/ylcutwv for
> more information).
>
>
> In a more general approach, when I hear "routers" and "wireless" I'm
> more thinking _embedded_. KVM/qemu would only help you to build your
> target systems.
>
>
> For embedded (or tiny, at least) systems, I would not use LXC.
>
> The drawback with Gentoo is that the current official uclibc stage3 for
> embedded/tiny systems is obsolete and marked as experimental. In facts,
> it's very _hard_ if not impossible to use it these days. Making your own
> cross-compilation environment is not a piece of cake (too), even with
> dedicated tools such as crossdev. This topic would ask its own book.
> So, if you want to try Gentoo embedded save your time by working on
> unofficial stage3.
>
> --
> Nicolas Sebrecht

I think I'm guilty of assumption regarding your original reference to
KVM.  I assumed you mean keyboard-video-mouse:

http://en.wikipedia.org/wiki/KVM_switch

but now I think you meant Kernel-based Virtual Machine:

http://www.linux-kvm.org/page/Main_Page

And now that I look more closely at KVM switches, it looks like they
provide a method of controlling multiple computers via a single
keyboard, monitor, and mouse.  I need sort of the inverse.  I'd like
to control a single Gentoo computer via multiple sets of keyboards,
monitors, and mice simultaneously.  It would basically be a way to
have the functionality of multiple workstations but the administration
hassle of only a single system.  Wireless communication between the
computer and each keyboard-monitor-mouse would be most convenient, but
that may not be possible so wired would be fine.  Does something like
this exist?

- Grant

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Grant]

> And now that I look more closely at KVM switches, it looks like they
> provide a method of controlling multiple computers via a single
> keyboard, monitor, and mouse.  I need sort of the inverse.  I'd like
> to control a single Gentoo computer via multiple sets of keyboards,
> monitors, and mice simultaneously.  It would basically be a way to
> have the functionality of multiple workstations but the administration
> hassle of only a single system.  Wireless communication between the
> computer and each keyboard-monitor-mouse would be most convenient, but
> that may not be possible so wired would be fine.  Does something like
> this exist?
>
> - Grant

Does this fantasy-arrangement of mine exist?  I guess what I'm after
is a series of dumb terminals to connect to a local Gentoo system so I
don't need to manage a series of Gentoo systems.

- Grant

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[James Wall]

On Fri, Jul 8, 2011 at 2:03 PM, Grant <emailgrant@gmail.com> wrote:
>> And now that I look more closely at KVM switches, it looks like they
>> provide a method of controlling multiple computers via a single
>> keyboard, monitor, and mouse.  I need sort of the inverse.  I'd like
>> to control a single Gentoo computer via multiple sets of keyboards,
>> monitors, and mice simultaneously.  It would basically be a way to
>> have the functionality of multiple workstations but the administration
>> hassle of only a single system.  Wireless communication between the
>> computer and each keyboard-monitor-mouse would be most convenient, but
>> that may not be possible so wired would be fine.  Does something like
>> this exist?
>>
>> - Grant
>
> Does this fantasy-arrangement of mine exist?  I guess what I'm after
> is a series of dumb terminals to connect to a local Gentoo system so I
> don't need to manage a series of Gentoo systems.
>
> - Grant

Have you considered using PXE to network boot your systems? you can
have various configurations set up based on mac addresses to address
different hardware issues. I recommend trying out SystemRescueCD to
experiment with PXE booting for the client and server.
--
No trees were harmed in the sending of this message. However, a large
number of electrons were terribly inconvenienced.

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Grant]

>>> And now that I look more closely at KVM switches, it looks like they
>>> provide a method of controlling multiple computers via a single
>>> keyboard, monitor, and mouse.  I need sort of the inverse.  I'd like
>>> to control a single Gentoo computer via multiple sets of keyboards,
>>> monitors, and mice simultaneously.  It would basically be a way to
>>> have the functionality of multiple workstations but the administration
>>> hassle of only a single system.  Wireless communication between the
>>> computer and each keyboard-monitor-mouse would be most convenient, but
>>> that may not be possible so wired would be fine.  Does something like
>>> this exist?
>>>
>>> - Grant
>>
>> Does this fantasy-arrangement of mine exist?  I guess what I'm after
>> is a series of dumb terminals to connect to a local Gentoo system so I
>> don't need to manage a series of Gentoo systems.
>>
>> - Grant
>
> Have you considered using PXE to network boot your systems? you can
> have various configurations set up based on mac addresses to address
> different hardware issues. I recommend trying out SystemRescueCD to
> experiment with PXE booting for the client and server.

That sounds like exactly what I need.  So, I could set up a Gentoo
server and a bunch of completely diskless clients which would all PXE
boot from the server?  Would the clients basically each control a
different virtual terminal on the server?

> No trees were harmed in the sending of this message. However, a large
> number of electrons were terribly inconvenienced.

That's hilarious. :)

- Grant

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[James Wall]

On Mon, Jul 11, 2011 at 6:39 PM, Grant <emailgrant@gmail.com> wrote:
>>
>> Have you considered using PXE to network boot your systems? you can
>> have various configurations set up based on mac addresses to address
>> different hardware issues. I recommend trying out SystemRescueCD to
>> experiment with PXE booting for the client and server.
>
> That sounds like exactly what I need.  So, I could set up a Gentoo
> server and a bunch of completely diskless clients which would all PXE
> boot from the server?  Would the clients basically each control a
> different virtual terminal on the server?

Each machine can pull a copy of the master boot image to make updates
a lot simpler. The SystemRescueCD PXE boot mechanism just pushes out a
copy of the CD to all the machines to boot them. to update the boot
image just update the files in one location to update all machines.
the machines act as separate fully functioning machine. Check out
http://www.sysresccd.org/Sysresccd-manual-en_PXE_network_booting to
see how to setup the PXE boot environment.

>
>> No trees were harmed in the sending of this message. However, a large
>> number of electrons were terribly inconvenienced.
>
> That's hilarious. :)
>
> - Grant
>
Thanks. A friend shared that with me.



-- 
No trees were harmed in the sending of this message. However, a large
number of electrons were terribly inconvenienced.

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Dale]

James Wall wrote:
> On Mon, Jul 11, 2011 at 6:39 PM, Grant<emailgrant@gmail.com>  wrote:
>    
>
>>> No trees were harmed in the sending of this message. However, a large
>>> number of electrons were terribly inconvenienced.
>>>        
>> That's hilarious. :)
>>
>> - Grant
>>
>>      
> Thanks. A friend shared that with me.
>
>    

I post some of Neil's sig lines to my wall on facebook.  A lot of them 
are pretty neat.  What gets me tho is when one of them applies to the 
topic he is replying too.  I know on a couple occasion Neil has even 
mentioned it himself.

Keep em coming Neil.

Dale

:-)  :-)

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Grant]

>>> Have you considered using PXE to network boot your systems? you can
>>> have various configurations set up based on mac addresses to address
>>> different hardware issues. I recommend trying out SystemRescueCD to
>>> experiment with PXE booting for the client and server.
>>
>> That sounds like exactly what I need.  So, I could set up a Gentoo
>> server and a bunch of completely diskless clients which would all PXE
>> boot from the server?  Would the clients basically each control a
>> different virtual terminal on the server?
>
> Each machine can pull a copy of the master boot image to make updates
> a lot simpler. The SystemRescueCD PXE boot mechanism just pushes out a
> copy of the CD to all the machines to boot them. to update the boot
> image just update the files in one location to update all machines.
> the machines act as separate fully functioning machine. Check out
> http://www.sysresccd.org/Sysresccd-manual-en_PXE_network_booting to
> see how to setup the PXE boot environment.

I think I get it now and it sounds great, exactly what I'm looking for.

Everything can be done in RAM, no disks required?

Can PXE boot be done wirelessly?  Maybe only if the wireless is
onboard?  I tried to Google this but the info returned is terribly
outdated for some reason.

Do you think SystemRescueCD is the best boot image for clients that
only need a browser?

What sort of machine would work well as a client?  Should I just put
together a bunch of motherboards with onboard video and ethernet,
CPUs, RAM, PSUs, and small cases?  Is there a prebuilt system that
works well for this?  Maybe an ARM-15 system as "Tampa Bay" James
referenced, although I think that isn't released yet.

- Grant

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Joshua Murphy]

On Tue, Jul 12, 2011 at 2:23 PM, Grant <emailgrant@gmail.com> wrote:
>>>> Have you considered using PXE to network boot your systems? you can
>>>> have various configurations set up based on mac addresses to address
>>>> different hardware issues. I recommend trying out SystemRescueCD to
>>>> experiment with PXE booting for the client and server.
>>>
>>> That sounds like exactly what I need.  So, I could set up a Gentoo
>>> server and a bunch of completely diskless clients which would all PXE
>>> boot from the server?  Would the clients basically each control a
>>> different virtual terminal on the server?
>>
>> Each machine can pull a copy of the master boot image to make updates
>> a lot simpler. The SystemRescueCD PXE boot mechanism just pushes out a
>> copy of the CD to all the machines to boot them. to update the boot
>> image just update the files in one location to update all machines.
>> the machines act as separate fully functioning machine. Check out
>> http://www.sysresccd.org/Sysresccd-manual-en_PXE_network_booting to
>> see how to setup the PXE boot environment.
>
> I think I get it now and it sounds great, exactly what I'm looking for.
>
> Everything can be done in RAM, no disks required?
>
> Can PXE boot be done wirelessly?  Maybe only if the wireless is
> onboard?  I tried to Google this but the info returned is terribly
> outdated for some reason.
>
> Do you think SystemRescueCD is the best boot image for clients that
> only need a browser?
>
> What sort of machine would work well as a client?  Should I just put
> together a bunch of motherboards with onboard video and ethernet,
> CPUs, RAM, PSUs, and small cases?  Is there a prebuilt system that
> works well for this?  Maybe an ARM-15 system as "Tampa Bay" James
> referenced, although I think that isn't released yet.
>
> - Grant

Well, the first thing you need to decide is whether you want each
client running that browser locally, or whether you want each client
to merely provide an interface to the server, and every user's browser
(and every other application) running on the server itself. If your
clients boot, then run all their own software locally, your server's
under only under load during boot-time and your clients need to be
able to handle that work (not much, but it's more than nothing, just
try running a modern Firefox on 64MB of ram). On the other hand, if
your clients merely boot into a remote connection to the server, a la
VNC or NX, the client does *very* little locally, can run on next to
nothing hardware-wise (a true 'thin client'), and the entirety of the
workload is offloaded to the server. If you want responsive 'eye
candy', 3D graphics work/play, or any form of particularly 'smooth'
animation, you will want that work to be handled on hardware closer to
the user (requiring a far faster processor, more ram, a capable video
device, and likely local storage for swap at the least), while serving
up a simple browser to the user is far more forgiving.

As for wired vs wireless, true hardware PXE booting is generally
limited to wired scenarios, but it would be entirely possible (though
not truly 'diskless') to deploy a minimal kernel+initramfs that
handles initial booting, joining WiFi, pulling down of the system
'image' from your server, and handing control off to that in the same
way your run of the mill kernel+initramfs loads hardware drivers until
it can find the harddrive, attaches to the root partition, and hands
off control to init from there. Changes to the wireless configuration
would require directly visiting each client, and client-side kernel or
initramfs updates easily could as well, if things don't go as planned
(but, since all the user-side software is either run on the server or
loaded from it at boot-time, changes to the client's "loader"
shouldn't be frequent).

There's also the option of pre-made hardware thin clients that
typically boot from internal flash and simply provide a remote
interface to a central server (though most are geared towards RDP or
Citrix), and some are even WiFi capable.

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Grant]

>>>>> Have you considered using PXE to network boot your systems? you can
>>>>> have various configurations set up based on mac addresses to address
>>>>> different hardware issues. I recommend trying out SystemRescueCD to
>>>>> experiment with PXE booting for the client and server.
>>>>
>>>> That sounds like exactly what I need.  So, I could set up a Gentoo
>>>> server and a bunch of completely diskless clients which would all PXE
>>>> boot from the server?  Would the clients basically each control a
>>>> different virtual terminal on the server?
>>>
>>> Each machine can pull a copy of the master boot image to make updates
>>> a lot simpler. The SystemRescueCD PXE boot mechanism just pushes out a
>>> copy of the CD to all the machines to boot them. to update the boot
>>> image just update the files in one location to update all machines.
>>> the machines act as separate fully functioning machine. Check out
>>> http://www.sysresccd.org/Sysresccd-manual-en_PXE_network_booting to
>>> see how to setup the PXE boot environment.
>>
>> I think I get it now and it sounds great, exactly what I'm looking for.
>>
>> Everything can be done in RAM, no disks required?
>>
>> Can PXE boot be done wirelessly?  Maybe only if the wireless is
>> onboard?  I tried to Google this but the info returned is terribly
>> outdated for some reason.
>>
>> Do you think SystemRescueCD is the best boot image for clients that
>> only need a browser?
>>
>> What sort of machine would work well as a client?  Should I just put
>> together a bunch of motherboards with onboard video and ethernet,
>> CPUs, RAM, PSUs, and small cases?  Is there a prebuilt system that
>> works well for this?  Maybe an ARM-15 system as "Tampa Bay" James
>> referenced, although I think that isn't released yet.
>>
>> - Grant
>
> Well, the first thing you need to decide is whether you want each
> client running that browser locally, or whether you want each client
> to merely provide an interface to the server, and every user's browser
> (and every other application) running on the server itself. If your
> clients boot, then run all their own software locally, your server's
> under only under load during boot-time and your clients need to be
> able to handle that work (not much, but it's more than nothing, just
> try running a modern Firefox on 64MB of ram). On the other hand, if
> your clients merely boot into a remote connection to the server, a la
> VNC or NX, the client does *very* little locally, can run on next to
> nothing hardware-wise (a true 'thin client'), and the entirety of the
> workload is offloaded to the server. If you want responsive 'eye
> candy', 3D graphics work/play, or any form of particularly 'smooth'
> animation, you will want that work to be handled on hardware closer to
> the user (requiring a far faster processor, more ram, a capable video
> device, and likely local storage for swap at the least), while serving
> up a simple browser to the user is far more forgiving.

After reading this, my first reaction was to run the browser on the
server and have each client connect via VNC/NX.  Now that I think
about it, I may be better off running the browser locally for
simplicity's sake.  I always try to keep the number of layers I'm
dealing with to a minimum and VNC/NX is one layer I could do without
if I beef up the clients a bit.  How different would the client
hardware requirements be between running the browser locally and
running it via VNC/NX?

I suppose I could also do without the PXE layer and all of its
requirements if I install some sort of minimal storage device (flash
drive, SD card, USB key, etc.) into each workstation for the boot
image.  I could still push updates to the boot image over the network
almost as easily as updating the single boot image on the server.

What is the benefit of loading SystemRescueCD instead of another
monolithic "just work" distro like Ubuntu?

> As for wired vs wireless, true hardware PXE booting is generally
> limited to wired scenarios, but it would be entirely possible (though
> not truly 'diskless') to deploy a minimal kernel+initramfs that
> handles initial booting, joining WiFi, pulling down of the system
> 'image' from your server, and handing control off to that in the same
> way your run of the mill kernel+initramfs loads hardware drivers until
> it can find the harddrive, attaches to the root partition, and hands
> off control to init from there. Changes to the wireless configuration
> would require directly visiting each client, and client-side kernel or
> initramfs updates easily could as well, if things don't go as planned
> (but, since all the user-side software is either run on the server or
> loaded from it at boot-time, changes to the client's "loader"
> shouldn't be frequent).

It sounds like I should stick with ethernet for simplicity's sake.

> There's also the option of pre-made hardware thin clients that
> typically boot from internal flash and simply provide a remote
> interface to a central server (though most are geared towards RDP or
> Citrix), and some are even WiFi capable.

A pre-made thin client could be the way to go.  Do you know of any
that are geared toward open protocols?

- Grant


> Poison [BLX]
> Joshua M. Murphy

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Bill Longman]

On 07/13/2011 12:38 PM, Grant wrote:

> I suppose I could also do without the PXE layer and all of its
> requirements if I install some sort of minimal storage device (flash
> drive, SD card, USB key, etc.) into each workstation for the boot
> image.  I could still push updates to the boot image over the network
> almost as easily as updating the single boot image on the server.
<snip>
> It sounds like I should stick with ethernet for simplicity's sake.

Yeah, PXE on the wire is the place to start if you want to boot across
the network. Start simple. Just get a handful of similar NICs and you
should be set.

>> There's also the option of pre-made hardware thin clients that
>> typically boot from internal flash and simply provide a remote
>> interface to a central server (though most are geared towards RDP or
>> Citrix), and some are even WiFi capable.
> 
> A pre-made thin client could be the way to go.  Do you know of any
> that are geared toward open protocols?

Quick query of "the oracle" yields:

http://www.thinlabs.com/products/thin-clients/aden

I have used AXEL thin client terminals and those require a VNC server
instance on your server per thin client, for the scenario that it sounds
like you're envisioning. It does RDP/VNC but you can get it to do
ssh/telnet on a green screen, with several sessions per seat.

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1667 bytes --]

On Wednesday 13 Jul 2011 21:51:52 Bill Longman wrote:
> On 07/13/2011 12:38 PM, Grant wrote:
> > I suppose I could also do without the PXE layer and all of its
> > requirements if I install some sort of minimal storage device (flash
> > drive, SD card, USB key, etc.) into each workstation for the boot
> > image.  I could still push updates to the boot image over the network
> > almost as easily as updating the single boot image on the server.
> 
> <snip>
> 
> > It sounds like I should stick with ethernet for simplicity's sake.
> 
> Yeah, PXE on the wire is the place to start if you want to boot across
> the network. Start simple. Just get a handful of similar NICs and you
> should be set.
> 
> >> There's also the option of pre-made hardware thin clients that
> >> typically boot from internal flash and simply provide a remote
> >> interface to a central server (though most are geared towards RDP or
> >> Citrix), and some are even WiFi capable.
> > 
> > A pre-made thin client could be the way to go.  Do you know of any
> > that are geared toward open protocols?
> 
> Quick query of "the oracle" yields:
> 
> http://www.thinlabs.com/products/thin-clients/aden
> 
> I have used AXEL thin client terminals and those require a VNC server
> instance on your server per thin client, for the scenario that it sounds
> like you're envisioning. It does RDP/VNC but you can get it to do
> ssh/telnet on a green screen, with several sessions per seat.

We've been using Neoware thin clients for a few years now.  They run some HP 
cooked debian version.

http://www.hp.com/sbso/busproducts_thinclient.html
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Managing multiple Gentoo systems

[kashani]

On 7/2/2011 3:14 PM, Grant wrote:
> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.  This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits.  What
> can be done to make the management of multiple Gentoo systems easier?
> I think identical hardware in each system would help a lot but I'm not
> sure that's practical.  I need to put together a bunch of new
> workstations and I'm thinking some sort of server/client arrangement
> with the only Gentoo install being on the server could be appropriate.
>
> - Grant
>

	You may want to look at something like a config management system. I'm 
using Puppet these days, but Gentoo support isn't spectacular. It would 
be a bit complex to have Puppet install the packages with the correct 
USE flags. However you could use Puppet to manage all the text files and 
then manage the packages somewhat manually.

Here's a snippet of a template for nrpe.cfg

<% if processorcount.to_i >= 12 then -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir') 
%>/check_load -w 35,25,25 -c 35,25,25
<% elsif fqdn =~ /(.*)stage|demo(.*)/ then -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir') 
%>/check_load -w 10,10,10 -c 10,10,10
<% else -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir') 
%>/check_load -w 10,7,5 -c 10,7,5
<% end -%>

If you were managing a make.conf you could set -j<%= processorcount*2 %> 
or whatever as well as pass in your own settings etc. Once you have 
things working it's pretty good at keeping your servers in sync and 
doing minor customization per server based on OS, hardware, IP, 
hostname, etc.

kashani

Re: [gentoo-user] Managing multiple Gentoo systems

[Alan McKinnon]

On Thursday 07 July 2011 11:23:15 kashani did opine thusly:
> On 7/2/2011 3:14 PM, Grant wrote:
> > After a frustrating experience with a Linksys WRT54GL, I've
> > decided to stick with Gentoo routers.  This increases the
> > number of Gentoo systems I'm responsible for and they're
> > nearing double-digits.  What can be done to make the management
> > of multiple Gentoo systems easier? I think identical hardware
> > in each system would help a lot but I'm not sure that's
> > practical.  I need to put together a bunch of new workstations
> > and I'm thinking some sort of server/client arrangement with
> > the only Gentoo install being on the server could be
> > appropriate.
> > 
> > - Grant
> 
> 	You may want to look at something like a config management 
system.
> I'm using Puppet these days, but Gentoo support isn't spectacular.
> It would be a bit complex to have Puppet install the packages with
> the correct USE flags. However you could use Puppet to manage all
> the text files and then manage the packages somewhat manually.

Give chef a try.

It overcomes a lot of the issue puppet ran into, and of course makes 
new ones all of it's won, but by and large chef is more flexible.


> 
> Here's a snippet of a template for nrpe.cfg
> 
> <% if processorcount.to_i >= 12 then -%>
> command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
> %>/check_load -w 35,25,25 -c 35,25,25
> <% elsif fqdn =~ /(.*)stage|demo(.*)/ then -%>
> command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
> %>/check_load -w 10,10,10 -c 10,10,10
> <% else -%>
> command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
> %>/check_load -w 10,7,5 -c 10,7,5
> <% end -%>
> 
> If you were managing a make.conf you could set -j<%=
> processorcount*2 %> or whatever as well as pass in your own
> settings etc. Once you have things working it's pretty good at
> keeping your servers in sync and doing minor customization per
> server based on OS, hardware, IP, hostname, etc.
> 
> kashani
-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Managing multiple Gentoo systems

[kashani]

On 7/7/2011 1:37 PM, Alan McKinnon wrote:
> On Thursday 07 July 2011 11:23:15 kashani did opine thusly:
>> On 7/2/2011 3:14 PM, Grant wrote:
>>> After a frustrating experience with a Linksys WRT54GL, I've
>>> decided to stick with Gentoo routers.  This increases the
>>> number of Gentoo systems I'm responsible for and they're
>>> nearing double-digits.  What can be done to make the management
>>> of multiple Gentoo systems easier? I think identical hardware
>>> in each system would help a lot but I'm not sure that's
>>> practical.  I need to put together a bunch of new workstations
>>> and I'm thinking some sort of server/client arrangement with
>>> the only Gentoo install being on the server could be
>>> appropriate.
>>>
>>> - Grant
>>
>> 	You may want to look at something like a config management
> system.
>> I'm using Puppet these days, but Gentoo support isn't spectacular.
>> It would be a bit complex to have Puppet install the packages with
>> the correct USE flags. However you could use Puppet to manage all
>> the text files and then manage the packages somewhat manually.
>
> Give chef a try.
>
> It overcomes a lot of the issue puppet ran into, and of course makes
> new ones all of it's won, but by and large chef is more flexible.

Too late. I've already put a year in with Puppet and have too much 
working code to switch. Also I'm not much of a programmer so I get a bit 
more out of the DSL though my templates are getting fairly fancy these 
days. For anyone else interested in what we're talking about, here's a 
fairly balanced and up to date link talking about some of the differences.

http://redbluemagenta.com/2011/05/21/puppet-vs-chef/

kashani

Re: [gentoo-user] Managing multiple Gentoo systems

[Alan McKinnon]

On Thursday 07 July 2011 14:01:55 kashani did opine thusly:
> On 7/7/2011 1:37 PM, Alan McKinnon wrote:
> > On Thursday 07 July 2011 11:23:15 kashani did opine thusly:
> >> On 7/2/2011 3:14 PM, Grant wrote:
> >>> After a frustrating experience with a Linksys WRT54GL, I've
> >>> decided to stick with Gentoo routers.  This increases the
> >>> number of Gentoo systems I'm responsible for and they're
> >>> nearing double-digits.  What can be done to make the
> >>> management
> >>> of multiple Gentoo systems easier? I think identical
> >>> hardware
> >>> in each system would help a lot but I'm not sure that's
> >>> practical.  I need to put together a bunch of new
> >>> workstations
> >>> and I'm thinking some sort of server/client arrangement with
> >>> the only Gentoo install being on the server could be
> >>> appropriate.
> >>> 
> >>> - Grant
> >>> 
> >> 	You may want to look at something like a config management
> > 
> > system.
> > 
> >> I'm using Puppet these days, but Gentoo support isn't
> >> spectacular. It would be a bit complex to have Puppet install
> >> the packages with the correct USE flags. However you could
> >> use Puppet to manage all the text files and then manage the
> >> packages somewhat manually.
> > 
> > Give chef a try.
> > 
> > It overcomes a lot of the issue puppet ran into, and of course
> > makes new ones all of it's won, but by and large chef is more
> > flexible.
> 
> Too late. I've already put a year in with Puppet and have too much
> working code to switch. Also I'm not much of a programmer so I get a
> bit more out of the DSL though my templates are getting fairly
> fancy these days. For anyone else interested in what we're talking
> about, here's a fairly balanced and up to date link talking about
> some of the differences.
> 
> http://redbluemagenta.com/2011/05/21/puppet-vs-chef/

At least with puppet you can still work around shortcomings as you 
find them (no black box tricks in puttet)

But regardless of it's quality, it's still 1,000,000's of times better 
than doing it all manually!

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: Managing multiple Gentoo systems

[James]

Grant <emailgrant <at> gmail.com> writes:


> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers.  This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits.  What
> can be done to make the management of multiple Gentoo systems easier?
> I think identical hardware in each system would help a lot but I'm not
> sure that's practical.  I need to put together a bunch of new
> workstations and I'm thinking some sort of server/client arrangement
> with the only Gentoo install being on the server could be appropriate.


Hello Grant,

You have similar goals as I do. In addition to what you are doing
I'm planning on managing thousands of embedded devices, remotely,
for controls purposes.

The new ARM-15 chip is suppose to be an "Intel Killer" in both
the server space and workstation space. It is also is going
to be the chip for 3D video and multi-head devices, such as
you purport to building in your other emails.

TI is very aggressive on the ARM-15 chips based mother boards.
Embedded Gentoo runs on the panda board, thanks to Armin76!

http://www.gentoo.org/proj/en/base/embedded/handbook/?part=4&chap=9

I'm not sure you can wait a few more months, but, in my research
the ARM-15 based devices are going to make significant inroads
into many areas. 

http://www.slashgear.com/ti-omap-5-outed-twin-cortex-a15-cores-kinect-style-tracking-more-07131324/


hth,
James

Re: [gentoo-user] Re: Managing multiple Gentoo systems

[Grant]

>> After a frustrating experience with a Linksys WRT54GL, I've decided to
>> stick with Gentoo routers.  This increases the number of Gentoo
>> systems I'm responsible for and they're nearing double-digits.  What
>> can be done to make the management of multiple Gentoo systems easier?
>> I think identical hardware in each system would help a lot but I'm not
>> sure that's practical.  I need to put together a bunch of new
>> workstations and I'm thinking some sort of server/client arrangement
>> with the only Gentoo install being on the server could be appropriate.
>
>
> Hello Grant,
>
> You have similar goals as I do. In addition to what you are doing
> I'm planning on managing thousands of embedded devices, remotely,
> for controls purposes.
>
> The new ARM-15 chip is suppose to be an "Intel Killer" in both
> the server space and workstation space. It is also is going
> to be the chip for 3D video and multi-head devices, such as
> you purport to building in your other emails.
>
> TI is very aggressive on the ARM-15 chips based mother boards.
> Embedded Gentoo runs on the panda board, thanks to Armin76!
>
> http://www.gentoo.org/proj/en/base/embedded/handbook/?part=4&chap=9
>
> I'm not sure you can wait a few more months, but, in my research
> the ARM-15 based devices are going to make significant inroads
> into many areas.
>
> http://www.slashgear.com/ti-omap-5-outed-twin-cortex-a15-cores-kinect-style-tracking-more-07131324/

Thanks James.  Would ARM-15 machines be a good match for PXE booting?
I'm thinking I just need something minimal so the ARM-15 might be a
great choice if I understand it correctly.  It wouldn't matter that it
runs Gentoo since my clients would be diskless, right?  I'm still
trying to get my mind around this.

- Grant