[gentoo-cluster] High-Availability Howto for Gentoo

[gentoo-cluster] High-Availability Howto for Gentoo

[Mathias Weigt]

Hi!
Is there a special HA-Howto for Gentoo Linux anywhere?
I came across this japanese Document, but unfortunately I don't 
understand a single world.
http://www.gentoo.gr.jp/jpdoc/failover.xml

I want to setup a Gentoo-Server with DRBD and automatic failover but all 
the available HOWTOs are not Gentoo-specific.

I don't have any experience with HA and DRBD and a Document which 
adresses Gentoo in this respect would be really appreciated.

Mathias
-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Hanni Ali]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

Hey,

Not currently, but it's something I'm working on. Anything people have out
there would be welcomed.

I have a high performance and distcc cluster and am working on the failover.
Also I'm looking into a Gentoo clustering LiveCD, suggestions for programs
etc. or help would be welcomed.

Hanni


On 07/04/06, Mathias Weigt <m.weigt@uni-bonn.de> wrote:
>
> Hi!
> Is there a special HA-Howto for Gentoo Linux anywhere?
> I came across this japanese Document, but unfortunately I don't
> understand a single world.
> http://www.gentoo.gr.jp/jpdoc/failover.xml
>
> I want to setup a Gentoo-Server with DRBD and automatic failover but all
> the available HOWTOs are not Gentoo-specific.
>
> I don't have any experience with HA and DRBD and a Document which
> adresses Gentoo in this respect would be really appreciated.
>
> Mathias
> --
> gentoo-cluster@gentoo.org mailing list
>
>


--
E-mail: hanni.ali@gmail.com
Mobile: 07985580147
Website: www.ainkaboot.co.uk

[-- Attachment #2: Type: text/html, Size: 1499 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Jared Greenwald]

Would be nice to have a livecd that you could pop into a set of
systems and have them auto-discover and form a cluster...  :)  You
could use stuff like slp to find each other.

-Jared

On 4/7/06, Hanni Ali <hanni.ali@gmail.com> wrote:
> Hey,
>
> Not currently, but it's something I'm working on. Anything people have out
> there would be welcomed.
>
> I have a high performance and distcc cluster and am working on the failover.
> Also I'm looking into a Gentoo clustering LiveCD, suggestions for programs
> etc. or help would be welcomed.
>
> Hanni
>
>
>
> On 07/04/06, Mathias Weigt <m.weigt@uni-bonn.de> wrote:
> > Hi!
> > Is there a special HA-Howto for Gentoo Linux anywhere?
> > I came across this japanese Document, but unfortunately I don't
> > understand a single world.
> > http://www.gentoo.gr.jp/jpdoc/failover.xml
> >
> > I want to setup a Gentoo-Server with DRBD and automatic failover but all
> > the available HOWTOs are not Gentoo-specific.
> >
> > I don't have any experience with HA and DRBD and a Document which
> > adresses Gentoo in this respect would be really appreciated.
> >
> > Mathias
> > --
> > gentoo-cluster@gentoo.org mailing list
> >
> >
>
>
>
> --
> E-mail: hanni.ali@gmail.com
> Mobile: 07985580147
> Website: www.ainkaboot.co.uk

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Bryan Stalcup]

i have a functioning high availability cluster with heartbeat, drbd  
and drbdlinks (highly recommend, wish it were in the gentoo packages  
repository).

some of my work is documented (piecemeal) at http://bbix.biz/tavi,  
but i'd be happy to flesh it out more and answer any questions, give  
sample config files, etc.

bryan

On Apr 7, 2006, at 7:18 AM, Mathias Weigt wrote:

> Hi!
> Is there a special HA-Howto for Gentoo Linux anywhere?
> I came across this japanese Document, but unfortunately I don't  
> understand a single world.
> http://www.gentoo.gr.jp/jpdoc/failover.xml
>
> I want to setup a Gentoo-Server with DRBD and automatic failover  
> but all the available HOWTOs are not Gentoo-specific.
>
> I don't have any experience with HA and DRBD and a Document which  
> adresses Gentoo in this respect would be really appreciated.
>
> Mathias
> -- 
> gentoo-cluster@gentoo.org mailing list
>

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Hanni Ali]

[-- Attachment #1: Type: text/plain, Size: 2707 bytes --]

Ok,

I suggest we try to put the documentation together on gentoo-wiki.com I've
always found this site an excellent resource.

There are already two stubs which I feel we should build on and kyron has
compiled an excellent list of programs if you follow the links.

http://gentoo-wiki.com/Index:HOWTO#Build_a_Gentoo_High_Performance_Cluster

I suggest we start Build a Gentoo High Availability Cluster.

http://www.gentoo.org/proj/en/cluster/

This is the gentoo cluster page and we only have three How To's All of which
have floors I've kept tabs on problems I've run into with the HPC howto and
distcc howto. I feel we should keep openMosix separate and have a completely
separate set of Howto's for that.

My clusters are generally diskless nodes so I suggest we try to incorporate
this howto into the gentoo-wiki

http://www.gentoo.org/doc/en/diskless-howto.xml

Though this also has it's fair share of difficulties.

I'm prepared to share a certain amount of my work on this. It would be nice
to make this documentation easily understandable for all and I'm always up
for people adding where they run into problems and WHY into these sort of
documents.

I'm looking carefully at HA diskless nodes and ways in which to ensure
redundancy if the master node fails. Suggestions on this would be welcomed.

How many people would be interested in helping out with this. If you've read
this far it must be because it's a Friday afternoon so anything can distract
you!

Cheers

Hanni




On 07/04/06, Bryan Stalcup <bryan@blackboot.biz> wrote:
>
> i have a functioning high availability cluster with heartbeat, drbd
> and drbdlinks (highly recommend, wish it were in the gentoo packages
> repository).
>
> some of my work is documented (piecemeal) at http://bbix.biz/tavi,
> but i'd be happy to flesh it out more and answer any questions, give
> sample config files, etc.
>
> bryan
>
> On Apr 7, 2006, at 7:18 AM, Mathias Weigt wrote:
>
> > Hi!
> > Is there a special HA-Howto for Gentoo Linux anywhere?
> > I came across this japanese Document, but unfortunately I don't
> > understand a single world.
> > http://www.gentoo.gr.jp/jpdoc/failover.xml
> >
> > I want to setup a Gentoo-Server with DRBD and automatic failover
> > but all the available HOWTOs are not Gentoo-specific.
> >
> > I don't have any experience with HA and DRBD and a Document which
> > adresses Gentoo in this respect would be really appreciated.
> >
> > Mathias
> > --
> > gentoo-cluster@gentoo.org mailing list
> >
>
> --
> gentoo-cluster@gentoo.org mailing list
>
>


--
E-mail: hanni.ali@gmail.com
Mobile: 07985580147
Website: www.ainkaboot.co.uk

[-- Attachment #2: Type: text/html, Size: 3634 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Jan Klopper]

[-- Attachment #1: Type: text/plain, Size: 4382 bytes --]

Im running a 5 node gentoo LVS apache cluster.
It has full hearthbeat failover (over ip), and ldirectd functionality.

Im using a tweaked direct route setup in which there are two directors,
checkup on each other trough hearthebeat. which redirect packages to 3 other
nodes, and localnode.
All nodes have a full internet connection though, which allows for phased
setup. (no need to setup the whole portage / binary replication thing first.
each node can keep himself up-to-date.)

If we're going to setup a nice wiki tut, il put up my configs aswell.
ironically i used a win2k + LVS tutorial to get my system to work as it
should.

The whole concept of all those LVS monkey releases is something i didn't
really grasp at first, since gentoo doesn't install the packages like that,
but just keeps them all up to dat trough portage, which is very different to
what the LVS docs are talking about.

The one thing im still strugling with is a way to share and replicate a set
of stable files, they don't change often, but when they do, each cluster
node ought to be updated.

I had an extra NFS server for this, outside the cluster, connected over
local ip's, but the one node i has setup to talk to that (with enough swap
space to keep a local cache i figured), crashed on me,making me damn
reluctant to try it out on the rest.

greets
Jan

On 4/7/06, Hanni Ali <hanni.ali@gmail.com> wrote:
>
> Ok,
>
> I suggest we try to put the documentation together on gentoo-wiki.com I've
> always found this site an excellent resource.
>
> There are already two stubs which I feel we should build on and kyron has
> compiled an excellent list of programs if you follow the links.
>
> http://gentoo-wiki.com/Index:HOWTO#Build_a_Gentoo_High_Performance_Cluster
>
> I suggest we start Build a Gentoo High Availability Cluster.
>
> http://www.gentoo.org/proj/en/cluster/
>
> This is the gentoo cluster page and we only have three How To's All of
> which have floors I've kept tabs on problems I've run into with the HPC
> howto and distcc howto. I feel we should keep openMosix separate and have a
> completely separate set of Howto's for that.
>
> My clusters are generally diskless nodes so I suggest we try to
> incorporate this howto into the gentoo-wiki
>
> http://www.gentoo.org/doc/en/diskless-howto.xml
>
> Though this also has it's fair share of difficulties.
>
> I'm prepared to share a certain amount of my work on this. It would be
> nice to make this documentation easily understandable for all and I'm always
> up for people adding where they run into problems and WHY into these sort of
> documents.
>
> I'm looking carefully at HA diskless nodes and ways in which to ensure
> redundancy if the master node fails. Suggestions on this would be welcomed.
>
> How many people would be interested in helping out with this. If you've
> read this far it must be because it's a Friday afternoon so anything can
> distract you!
>
> Cheers
>
> Hanni
>
>
>
>
>
> On 07/04/06, Bryan Stalcup <bryan@blackboot.biz> wrote:
> >
> > i have a functioning high availability cluster with heartbeat, drbd
> > and drbdlinks (highly recommend, wish it were in the gentoo packages
> > repository).
> >
> > some of my work is documented (piecemeal) at http://bbix.biz/tavi,
> > but i'd be happy to flesh it out more and answer any questions, give
> > sample config files, etc.
> >
> > bryan
> >
> > On Apr 7, 2006, at 7:18 AM, Mathias Weigt wrote:
> >
> > > Hi!
> > > Is there a special HA-Howto for Gentoo Linux anywhere?
> > > I came across this japanese Document, but unfortunately I don't
> > > understand a single world.
> > > http://www.gentoo.gr.jp/jpdoc/failover.xml
> > >
> > > I want to setup a Gentoo-Server with DRBD and automatic failover
> > > but all the available HOWTOs are not Gentoo-specific.
> > >
> > > I don't have any experience with HA and DRBD and a Document which
> > > adresses Gentoo in this respect would be really appreciated.
> > >
> > > Mathias
> > > --
> > > gentoo-cluster@gentoo.org mailing list
> > >
> >
> > --
> > gentoo-cluster@gentoo.org mailing list
> >
> >
>
>
> --
>
> E-mail: hanni.ali@gmail.com
> Mobile: 07985580147
> Website: www.ainkaboot.co.uk
>



--
Vr gr,
Jan Klopper
Innerheight Internet Diensten
http://www.innerheight.com

[-- Attachment #2: Type: text/html, Size: 6608 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Ramon van Alteren]

[-- Attachment #1: Type: text/plain, Size: 1865 bytes --]

Hi List,

We're running a 40 node webserver HA-cluster, with two loadbalancers  
and a 30 node mysql replication setup as datastore behind that which  
we are going to move behind a loadbalancer set soon. We're currently  
building the load-balancers for that.

All the loadbalancers are using a direct routing setup.

Apart from that we also employ a number of storage NFS servers and  
infrastructure related servers such as build-hosts and rsync server etc.

On 7 Apr , 2006, at 10:31 PM, Jan Klopper wrote:
> If we're going to setup a nice wiki tut, il put up my configs  
> aswell. ironically i used a win2k + LVS tutorial to get my system  
> to work as it should.

As soon as stuff cools down a bit here, I'll add configs and docs as  
well to the wiki. Currently it's too busy. I checked the wiki the  
other day but didn't see a page ?

> The whole concept of all those LVS monkey releases is something i  
> didn't really grasp at first, since gentoo doesn't install the  
> packages like that, but just keeps them all up to dat trough  
> portage, which is very different to what the LVS docs are talking  
> about.

Right now we're not usiing any of the ultramonkey stuff but a custom  
setup with LVS and heart-beat. The new set of load-balancers is setup  
to use keepalived.

> How many people would be interested in helping out with this. If  
> you've read this far it must be because it's a Friday afternoon so  
> anything can distract you!

I / we'd be interested. I expect that things cool down a bit  
somewhere at the beginning of may.

If anyone is interested in helping us out, we're hiring :-) You need  
to be located in near to Amsterdam/Holland, but if you are contact me  
off-list.

Grtz Ramon
--
In the beginning, there was nothing. And God said, 'Let there be Light.'
And there was still nothing, but you could see a bit better.




[-- Attachment #2: Type: text/html, Size: 4119 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Hanni Ali]

[-- Attachment #1: Type: text/plain, Size: 2965 bytes --]

Ok this is all good stuff.

This is the specific article I thought as a list we could turn into a
coherent doccument.

http://gentoo-wiki.com/HOWTO_Configure_Gentoo_Linux_for_Clustering

There is also the beginings of an MPI programming guide which although I'm
not so sure it's gentoo specific in any way it certanally would be auseful
thing to anyone setting up MPI etc.

http://gentoo-wiki.com/HOWTO_MPI_Programming

May would be fine I should hopefully have time in June to check through it
thouroughly if it's nearing completion.

I've made a tentative start unfortunatly I'm not sure how much we can take
directly from the current howto on the Gentoo page I don't know who own's
the copyright.

If we use the stub as a sandbox and perhaps the doccument that arises from
it we could call the "Gentoo Clustering Handbook".

Hanni


On 11/04/06, Ramon van Alteren <ramon@vanalteren.nl> wrote:
>
> Hi List,
> We're running a 40 node webserver HA-cluster, with two loadbalancers and a
> 30 node mysql replication setup as datastore behind that which we are going
> to move behind a loadbalancer set soon. We're currently building the
> load-balancers for that.
>
> All the loadbalancers are using a direct routing setup.
>
> Apart from that we also employ a number of storage NFS servers and
> infrastructure related servers such as build-hosts and rsync server etc.
>
> On 7 Apr , 2006, at 10:31 PM, Jan Klopper wrote:
>
> If we're going to setup a nice wiki tut, il put up my configs aswell.
> ironically i used a win2k + LVS tutorial to get my system to work as it
> should.
>
>
> As soon as stuff cools down a bit here, I'll add configs and docs as well
> to the wiki. Currently it's too busy. I checked the wiki the other day but
> didn't see a page ?
>
> The whole concept of all those LVS monkey releases is something i didn't
> really grasp at first, since gentoo doesn't install the packages like that,
> but just keeps them all up to dat trough portage, which is very different to
> what the LVS docs are talking about.
>
>
> Right now we're not usiing any of the ultramonkey stuff but a custom setup
> with LVS and heart-beat. The new set of load-balancers is setup to use
> keepalived.
>
> How many people would be interested in helping out with this. If you've
> > read this far it must be because it's a Friday afternoon so anything can
> > distract you!
> >
>
> I / we'd be interested. I expect that things cool down a bit somewhere at
> the beginning of may.
>
> If anyone is interested in helping us out, we're hiring :-) You need to be
> located in near to Amsterdam/Holland, but if you are contact me off-list.
>
> Grtz Ramon
>  --
> In the beginning, there was nothing. And God said, 'Let there be Light.'
> And there was still nothing, but you could see a bit better.
>
>
>
>


--
E-mail: hanni.ali@gmail.com
Mobile: 07985580147
Website: www.ainkaboot.co.uk

[-- Attachment #2: Type: text/html, Size: 5421 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Ramon van Alteren]

Hanni Ali wrote:

> Ok this is all good stuff.
>
> This is the specific article I thought as a list we could turn into a 
> coherent doccument.
>
> http://gentoo-wiki.com/HOWTO_Configure_Gentoo_Linux_for_Clustering 
> <http://gentoo-wiki.com/HOWTO_Configure_Gentoo_Linux_for_Clustering>
>
> There is also the beginings of an MPI programming guide which although 
> I'm not so sure it's gentoo specific in any way it certanally would be 
> auseful thing to anyone setting up MPI etc.
>
> http://gentoo-wiki.com/HOWTO_MPI_Programming

AFAIK MPI Programming == HPC clustering
Which is an entirely different subject from HA-clustering.

Of course also a very interesting topic, but has relatively few links 
with HA.

Maybe a split in HA-Clustering, HPC-clustering and large-scale sysadmin 
with gentoo.
The large-scale sysadmin is overlapping for both, HA can be usefull in 
smaller setups, HPC is seldomly used in small setup AFAIK

> May would be fine I should hopefully have time in June to check 
> through it thouroughly if it's nearing completion.
>
> I've made a tentative start unfortunatly I'm not sure how much we can 
> take directly from the current howto on the Gentoo page I don't know 
> who own's the copyright.

Suggest we pick one of the usual free licenses (maybe the same as the 
gentoo documentation project uses ?)

>
> If we use the stub as a sandbox and perhaps the doccument that arises 
> from it we could call the "Gentoo Clustering Handbook".
>
I think we should determine topic first (see above)

Regards,

Ramon

-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[busby]

> Hanni Ali wrote:
[ Snip ]
> Maybe a split in HA-Clustering, HPC-clustering and large-scale sysadmin
> with gentoo.
> The large-scale sysadmin is overlapping for both, HA can be usefull in
> smaller setups, HPC is seldomly used in small setup AFAIK
[ Snip ]

I agree; different topics different sections.
I'd like to see something like:

Gentoo Cluster Handbook
  I. Sys Admin Requirements - What reader must know before moving forward
(maybe describe clustering types here?)
 II. SysAdmin for Clusters - tools/support servers/etc
III. HA Clustering - web/email failover/etc
 IV. HPC Clustering - batch processing/etc
  V. Intro to MPI - Merge the other MPI howto and add some?

And of course each volume is full of useful information

/djb

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Hanni Ali]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

Sys Admin is an excelent way to describe the core aspects of configuring
Gentoo for both HP and HA.

Additionally explaining the best wasy to provide failover for HPC is also
useful.

Gentoo Cluster Handbook
 I. Sys Admin Requirements - What reader must know before moving forward
 (maybe describe clustering types here?)
    i) Networking options etc.
    ii) Disked vs Diskless
    iii) more...
 II. SysAdmin for Clusters - tools/support servers/etc

 III. HA Clustering -
   i) web/email
   ii) failover/etc
   iii) mySQL clusters
 IV. HPC Clustering - batch processing/etc
   i) MPICH
   ii) Distributed Compiling? distcc
   iii) Distributed rendering pvmpovray
 V. Intro to MPI - Merge the other MPI howto and add some?
   We need to cover all the different methods and possibly a comparison.

Opinions?

Hanni


On 11/04/06, busby@edoceo.com <busby@edoceo.com> wrote:
>
> > Hanni Ali wrote:
> [ Snip ]
> > Maybe a split in HA-Clustering, HPC-clustering and large-scale sysadmin
> > with gentoo.
> > The large-scale sysadmin is overlapping for both, HA can be usefull in
> > smaller setups, HPC is seldomly used in small setup AFAIK
> [ Snip ]
>
> I agree; different topics different sections.
> I'd like to see something like:
>
> Gentoo Cluster Handbook
>   I. Sys Admin Requirements - What reader must know before moving forward
> (maybe describe clustering types here?)
> II. SysAdmin for Clusters - tools/support servers/etc
> III. HA Clustering - web/email failover/etc
> IV. HPC Clustering - batch processing/etc
>   V. Intro to MPI - Merge the other MPI howto and add some?
>
> And of course each volume is full of useful information
>
> /djb
>
> --
> gentoo-cluster@gentoo.org mailing list
>
>


--
E-mail: hanni.ali@gmail.com
Mobile: 07985580147
Website: www.ainkaboot.co.uk

[-- Attachment #2: Type: text/html, Size: 2628 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[busby]

[ Snip ]
>
>  III. HA Clustering -
>    i) web/email
>    ii) failover/etc
>    iii) mySQL clusters

If we have mySQL I'd like to see PostgreSQL too, I've got a starter on that.

[ Snip ]

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Hanni Ali]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

1. with distcc how many files were being compiled at once?
2. defo include PostgreSQL

Hanni

On 11/04/06, busby@edoceo.com <busby@edoceo.com> wrote:
>
> [ Snip ]
> >
> >  III. HA Clustering -
> >    i) web/email
> >    ii) failover/etc
> >    iii) mySQL clusters
>
> If we have mySQL I'd like to see PostgreSQL too, I've got a starter on
> that.
>
> [ Snip ]
>
> --
> gentoo-cluster@gentoo.org mailing list
>
>


--
E-mail: hanni.ali@gmail.com
Mobile: 07985580147
Website: www.ainkaboot.co.uk

[-- Attachment #2: Type: text/html, Size: 1075 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Ramon van Alteren]

busby@edoceo.com wrote:

>[ Snip ]
>  
>
>> III. HA Clustering -
>>   i) web/email
>>   ii) failover/etc
>>   iii) mySQL clusters
>>    
>>
>
>If we have mySQL I'd like to see PostgreSQL too, I've got a starter on that.
>  
>
You've found a starter on PostgreSQL clustering ??
I'd be most interested, are you sure you don't mean PostgreSQL 
replication with slony ????

I am unaware of any clustering support for postgreSQL but must admit I 
haven't looked in over a year.

Grtz Ramon



-- 
To be stupid and selfish and to have good health are the three requirements for happiness, though if stupidity is lacking, the others are useless.

Gustave Flaubert

  
  

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[busby]

>>
> You've found a starter on PostgreSQL clustering ??
> I'd be most interested, are you sure you don't mean PostgreSQL
> replication with slony ????
>
> I am unaware of any clustering support for postgreSQL but must admit I
> haven't looked in over a year.
>
> Grtz Ramon
>

I think am, these:

PostgreSQL Replication Packages

    * PGCluster - Multi-master no delay synchronus replication for load
sharing or HA. Large objects are now supported.
    * Slony-I - Master to multi-slave cascading and almost-failover.
    * DBBalancer - Alpha
    * pgpool - Connection pooling front end with synchronous replication
    * PostgreSQL table comparator - rsync for PostgreSQL.

from my page:
  http://www.edoceo.com/liber/db-postgresql-replication.php

I've also only slightly dinked with those projects and am no expert on
clustering.

/djb

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Ramon van Alteren]

Hi

Very interesting links, read through them quickly.
Most seem to offer replication-alike setups with various twists.

I'm unsure if there is an official definition of a clustered  
database, but I meant one in the sense that it's used for Oracle  
cluster & Mysql cluster.  IOW fragments of your data reside in  
multiple copies on the database-datanodes. You end up talking to the  
SQL frontend which uses some way of finding the correct fragment of  
data on your data-nodes.

I think (but am not sure) that to call a database solution a cluster  
it would need to be able to store a larger total datavolume than the  
storage-space of a single node.

If it doesn't do that, it's a replication setup.

Don't get me wrong, replication is extremely useful and we're making  
heavy use of it.
However, as far as I've always understood multi-master aka multi- 
write replication is very error-prone and unstable. Which seems  
logical to me if I try to imagine what should happen in a replication- 
setup when two clients are trying to update the same table with an  
autoincrement field at the same time on different replicationhosts.

I'd be interested in anyone's experiences with multi-master  
replication setups, anyone running one ?

Thanx for the links ;-)

Ramon
--
In the beginning, there was nothing. And God said, 'Let there be Light.'
And there was still nothing, but you could see a bit better.



-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Dice R. Random]

Hi all,

On 4/11/06, Hanni Ali <hanni.ali@gmail.com> wrote:
>  Gentoo Cluster Handbook
>   I. Sys Admin Requirements - What reader must know before moving forward
>   (maybe describe clustering types here?)
>      i) Networking options etc.
>      ii) Disked vs Diskless
>      iii) more...

I'm particularly interested in what people are using for diskless
nodes and managing upgrades of system images across the cluster.  I'm
envisioning a system where I have a build environment in which I can
upgrade system software and test functionality on a development
machine and then take a snapshot of the system and copy that up to a
NAS device so that the nodes can then boot it.  It would be even
better if I could specify which packages I wanted (or rather, didn't
want) on the system images so that I could avoid having un-necessary
and potentially attacker-friendly packages such as gcc and portage on
the cluster nodes.

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Eric Thibodeau]

Mr. Rand(),

	I tend to prone the Diskless approach for the reasons you are mentionning. 
You can easily switch between roots with a network boot just by modifying 
your dhcp config and rebooting a node. This is really neat since you can have 
the dev environment on the actual cluster, test the new root with new libs 
and simply reboot some available nodes to test them. Furthermore, this 
approach opens the way to having multiple boot profiles with 
application-specific orientations (/me is thinking of the hellish deal of 
parallel Matlab with a polluted environment and the booting into a really 
optimized one for real MPI work ;)...)

	As for the "un-ncecessary attacker blahblahblah... Put your head behing the 
firewall. Beowulf nodes aren't meant to be publically available if they are 
to be efficient. nonetheless, departmental clusters (by night) could aslo be 
very possible with the diskless approach (even more so since you don't modify 
the current OS, which is most probably some horribly expensive Windows with 
the latest and greates Office suite which the deparment thinks is more 
important that the licences for Matlab...t'is not like we're trying to do 
some scientific work here eh!... (oops...dropped that one)...

Hehe, in any case, I have had only great experiences with diskless nodes at 
the moment and really hope to see the Gentoo community take off on Clusterd 
(with or without disks ;)

Le Mardi 11 Avril 2006 19:11, Dice R. Random a écrit :
> Hi all,
> 
> On 4/11/06, Hanni Ali <hanni.ali@gmail.com> wrote:
> >  Gentoo Cluster Handbook
> >   I. Sys Admin Requirements - What reader must know before moving forward
> >   (maybe describe clustering types here?)
> >      i) Networking options etc.
> >      ii) Disked vs Diskless
> >      iii) more...
> 
> I'm particularly interested in what people are using for diskless
> nodes and managing upgrades of system images across the cluster.  I'm
> envisioning a system where I have a build environment in which I can
> upgrade system software and test functionality on a development
> machine and then take a snapshot of the system and copy that up to a
> NAS device so that the nodes can then boot it.  It would be even
> better if I could specify which packages I wanted (or rather, didn't
> want) on the system images so that I could avoid having un-necessary
> and potentially attacker-friendly packages such as gcc and portage on
> the cluster nodes.
> 

-- 
Eric Thibodeau
Neural Bucket Solutions Inc.
T. (514) 736-1436
C. (514) 710-0517

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Dice R. Random]

Hi Eric,

Sorry, forgot to CC the list on this.  Here goes...

On 4/11/06, Eric Thibodeau <kyron@neuralbs.com> wrote:
> Mr. Rand(),
>
>         I tend to prone the Diskless approach for the reasons you are mentionning.
> You can easily switch between roots with a network boot just by modifying
> your dhcp config and rebooting a node. This is really neat since you can have
> the dev environment on the actual cluster, test the new root with new libs
> and simply reboot some available nodes to test them. Furthermore, this
> approach opens the way to having multiple boot profiles with
> application-specific orientations (/me is thinking of the hellish deal of
> parallel Matlab with a polluted environment and the booting into a really
> optimized one for real MPI work ;)...)

Yes, this is exactly what I had in mind :)

>         As for the "un-ncecessary attacker blahblahblah... Put your head behing the
> firewall. Beowulf nodes aren't meant to be publically available if they are
> to be efficient. nonetheless, departmental clusters (by night) could aslo be
> very possible with the diskless approach

Sorry, I should have mentioned that my application is load balancing /
HA rather than HPC.  There will certainly be some very strict firewall
rulesets in place, however I do have to deal with serving up user
requests.  If someone manages to poke a hole in Apache or whatever and
gets themself a shell I don't want to give them any tools which might
help them along their quest for root.

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Brady Catherman]

Clarifying the differences would hep a bunch. I have talked with  
several 'cluster' admins that had no clue about high performance  
clustering.

If you call Cisco and tell them you are running a cluster they will  
try to sell you a switch with high inbound/outbound traffic but piss  
poor inter-port communication speeds. They try to make there switches  
run really fast for things like HA clusters but fall short on  
performance clusters (where everything talks to everything). Also,  
techniques like Flat Neighborhood Networks are pointless on a HA  
cluster but can be a god send on HPC clusters. These  subtle  
differences are actually much larger than people realize.

If you are planning on writing a MPI programming guide you are  
looking at a bunch of writing. Parallel programming regardless of the  
API used is tricky. Communication methods, shared resource processing  
and the like make parallel programming more of an art form than just  
another API to use.

One last question for you all.. Why is distcc so popular? We used it  
on our 134 node cluster and it actually made compiling much slower  
than just running it on one of the nodes. The network overhead killed  
the performance gain. The only way we found that it helped was  
writing the makefile itself to take advantage of parallelism. Is this  
uncommon for most people?




On Apr 11, 2006, at 7:47 AM, busby@edoceo.com wrote:

>> Hanni Ali wrote:
> [ Snip ]
>> Maybe a split in HA-Clustering, HPC-clustering and large-scale  
>> sysadmin
>> with gentoo.
>> The large-scale sysadmin is overlapping for both, HA can be  
>> usefull in
>> smaller setups, HPC is seldomly used in small setup AFAIK
> [ Snip ]
>
> I agree; different topics different sections.
> I'd like to see something like:
>
> Gentoo Cluster Handbook
>   I. Sys Admin Requirements - What reader must know before moving  
> forward
> (maybe describe clustering types here?)
>  II. SysAdmin for Clusters - tools/support servers/etc
> III. HA Clustering - web/email failover/etc
>  IV. HPC Clustering - batch processing/etc
>   V. Intro to MPI - Merge the other MPI howto and add some?
>
> And of course each volume is full of useful information
>
> /djb
>
> -- 
> gentoo-cluster@gentoo.org mailing list
>

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1172 bytes --]

Brady Catherman wrote:
> One last question for you all.. Why is distcc so popular? We used it on
> our 134 node cluster and it actually made compiling much slower than
> just running it on one of the nodes. The network overhead killed the
> performance gain. The only way we found that it helped was writing the
> makefile itself to take advantage of parallelism. Is this uncommon for
> most people?

Autotools (properly used) create parallelizable Makefiles, so that's not
much of an issue. Clearly if you're just exporting the same job to
another node instead of parallelizing across multiple nodes, you will
see a performance loss.

distcc is particularly useful when not all nodes are attempting to
locally upgrade/install something at the same time, so they team up on a
parallelized compilation for a single node. There's no effective gain by
 using distcc on a large cluster if you're just compiling everything on
every node -- you should be using it in parallel to build binary
packages once, then installing across all nodes.

distcc is also useful when you've got a mixture of slow and fast nodes,
for obvious reasons.

Thanks,
Donnie


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Brady Catherman]

nono.. we where running using distcc across our cluster (you could  
see the compile jobs running on all the different nodes.)

Though I would not be surprised in the least if I configured  
something wrong.. Plus our Mac OS NFS Servers have been... flaky.. at  
best. So there might just be an issue in network file system  
performance.

It appears that my experience with distcc is wrong =) Thats good to  
know.. next time I have to re-build a cluster I will make sure to  
give it another go and see if I can make it perform better =)


On Apr 11, 2006, at 1:51 PM, Donnie Berkholz wrote:

> Brady Catherman wrote:
>> One last question for you all.. Why is distcc so popular? We used  
>> it on
>> our 134 node cluster and it actually made compiling much slower than
>> just running it on one of the nodes. The network overhead killed the
>> performance gain. The only way we found that it helped was writing  
>> the
>> makefile itself to take advantage of parallelism. Is this uncommon  
>> for
>> most people?
>
> Autotools (properly used) create parallelizable Makefiles, so  
> that's not
> much of an issue. Clearly if you're just exporting the same job to
> another node instead of parallelizing across multiple nodes, you will
> see a performance loss.
>
> distcc is particularly useful when not all nodes are attempting to
> locally upgrade/install something at the same time, so they team up  
> on a
> parallelized compilation for a single node. There's no effective  
> gain by
>  using distcc on a large cluster if you're just compiling  
> everything on
> every node -- you should be using it in parallel to build binary
> packages once, then installing across all nodes.
>
> distcc is also useful when you've got a mixture of slow and fast  
> nodes,
> for obvious reasons.
>
> Thanks,
> Donnie
>

-- 
gentoo-cluster@gentoo.org mailing list

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

Brady Catherman wrote:
> nono.. we where running using distcc across our cluster (you could see
> the compile jobs running on all the different nodes.)
> 
> Though I would not be surprised in the least if I configured something
> wrong.. Plus our Mac OS NFS Servers have been... flaky.. at best. So
> there might just be an issue in network file system performance.
> 
> It appears that my experience with distcc is wrong =) Thats good to
> know.. next time I have to re-build a cluster I will make sure to give
> it another go and see if I can make it perform better =)

It's most useful on 100Mbit+ low-latency networks, and most apps will
not parallelize beyond about -j4 anyhow, because there just isn't enough
non-dependent stuff to build.

Personally, distcc is very useful for cross-compilation from a fast x86
development box to slow sparc/arm boxes.

Thanks,
Donnie


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

Ramon van Alteren wrote:
> Suggest we pick one of the usual free licenses (maybe the same as the
> gentoo documentation project uses ?)

Please! Otherwise we can never incorporate the material into any
official documentation.

Thanks,
Donnie


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [gentoo-cluster] High-Availability Howto for Gentoo

[Justin Bronder]

Greetings,

I'm currently employed at a site with some Xserve G5's and a smattering 
of PIII's.
I cannot comment on High Availability Clusters, but I'll be more then 
willing to
discuss the HPC side of clusters.

Right now we primarily run OS X on the G5's, however work is in progress 
to allow
job-submission time switching between OS X and Linux (Debian or Gentoo 
currently,
others in the future possibly) based upon user-submitted requests.

As we run a variety of operating systems, I personally prefer to compile 
the HPC-
orientated applications from source.  Anyways, I noticed a request for 
software
recommendations earlier in this thread, so here's a list of the first 
things I
end up installing when we build a test/development cluster, along with
the versions I have running.

Torque (2.0.0p5)
Mpich (1.2.7)
Mpichgm (Myrinet support, based on 1.2.6 )
Mpiexec (0.80)
Atlas (3.7.11)
HPL (To test the install mainly)

We also find it nice to have server(s) providing:
LDAP
DHCP and related netbooting services.  (We've written our own, highly 
alpha stage right now).
NFS for home directories only.  We've found numerous scalability 
problems with diskless.

Of course the shameless plug for our MyPBS package is also required,
http://sourceforge.net/projects/my-pbs/

This is just a quick list of of what I think any documentation on a HPC 
cluster needs to
cover at minimum.  I'm by no means an expert, but I would like to offer 
my help.

Hanni Ali wrote:

> Ok,
>
> I suggest we try to put the documentation together on gentoo-wiki.com 
> <http://gentoo-wiki.com> I've always found this site an excellent 
> resource.
>
> There are already two stubs which I feel we should build on and kyron 
> has compiled an excellent list of programs if you follow the links.
>
> http://gentoo-wiki.com/Index:HOWTO#Build_a_Gentoo_High_Performance_Cluster
>
> I suggest we start Build a Gentoo High Availability Cluster.
>
> http://www.gentoo.org/proj/en/cluster/
>
> This is the gentoo cluster page and we only have three How To's All of 
> which have floors I've kept tabs on problems I've run into with the 
> HPC howto and distcc howto. I feel we should keep openMosix separate 
> and have a completely separate set of Howto's for that.
>
> My clusters are generally diskless nodes so I suggest we try to 
> incorporate this howto into the gentoo-wiki
>
> http://www.gentoo.org/doc/en/diskless-howto.xml 
> <http://www.gentoo.org/doc/en/diskless-howto.xml>
>
> Though this also has it's fair share of difficulties.
>
> I'm prepared to share a certain amount of my work on this. It would be 
> nice to make this documentation easily understandable for all and I'm 
> always up for people adding where they run into problems and WHY into 
> these sort of documents.
>
> I'm looking carefully at HA diskless nodes and ways in which to ensure 
> redundancy if the master node fails. Suggestions on this would be 
> welcomed.
>
> How many people would be interested in helping out with this. If 
> you've read this far it must be because it's a Friday afternoon so 
> anything can distract you!
>
> Cheers
>
> Hanni
>
>

-- 
Justin Bronder
University of Maine, Orono

Advanced Computing Research Lab
20 Godfrey Dr
Orono, ME 04473
www.clusters.umaine.edu

Mathematics Department
425 Neville Hall
Orono, ME 04469


-- 
gentoo-cluster@gentoo.org mailing list