[gentoo-server] SAN Clustered Filesystem

[gentoo-server] SAN Clustered Filesystem

[Brian Kroth]

Hello all,

I currently manage about 40 Window, OSX, and Hardened Gentoo servers.  I 
will soon have 12 P4 servers that were previously used as video encoders 
free as well as an Apple XRaid.  With all this spare hardware I thought 
I'd research setting up a cluster of servers running Apache for load 
balancing and high availability.  I'm also looking into a MySQL cluster, 
but that wouldn't require a shared filesystem.  I'm wondering if anyone 
has done something like this before and in particular knows a good 
filesystem to use so that each of the servers can access and potentially 
write to the same storage array.  I've accomplished the same thing with 
XServes running OSX, but they like to charge you a pretty penny for the 
XSan software that allows this which I thought I'd try to avoid if 
possible.  So far I've seen only GFS, but haven't gotten much reading 
done on it yet.  Any other tips or insights would be appreciated as well.

Thanks,
Brian
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] SAN Clustered Filesystem

[Sean Cook]

I would actually spend a little more and start looking at iSCSI for attached
storage.  You can generally pickup some decent chassis on ebay for not a lot
of change and it gives you a lot more flexibility.

GFS is ok if you don't want to mess around with a SAN but it has no where
near the performance of fiber or iSCSI attached storage.

Here is exactly what I am talking about...
http://cgi.ebay.com/Dell-EMC-AX100i-iSCSI-12-Slot-SAN-Array-w-4x-250GB-HDD_W0QQitemZ300072200442QQihZ020QQcategoryZ111458QQssPageNameZWDVWQQrdZ1QQcmdZViewItem


On 25-Jan-2007, Brian Kroth wrote:
> Hello all,
> 
> I currently manage about 40 Window, OSX, and Hardened Gentoo servers.  I 
> will soon have 12 P4 servers that were previously used as video encoders 
> free as well as an Apple XRaid.  With all this spare hardware I thought 
> I'd research setting up a cluster of servers running Apache for load 
> balancing and high availability.  I'm also looking into a MySQL cluster, 
> but that wouldn't require a shared filesystem.  I'm wondering if anyone 
> has done something like this before and in particular knows a good 
> filesystem to use so that each of the servers can access and potentially 
> write to the same storage array.  I've accomplished the same thing with 
> XServes running OSX, but they like to charge you a pretty penny for the 
> XSan software that allows this which I thought I'd try to avoid if 
> possible.  So far I've seen only GFS, but haven't gotten much reading 
> done on it yet.  Any other tips or insights would be appreciated as well.
> 
> Thanks,
> Brian
> -- 
> gentoo-server@gentoo.org mailing list
> 
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] SAN Clustered Filesystem

[Brian Kroth]

We actually have an XRaid now sitting in a box.  I was simply going to 
purchase some fiber channel cards for the servers so they can attach to 
it via a QLogic fiber channel switch we have.  But once that's happened 
I want to make sure I can share the same storage pool with all, or 
perhaps just a subset, of the servers.  With OSX this required a special 
XSan client.  I have done very limited research so far to see what it 
would take to get a collection of Gentoo servers to do this, but figured 
I should ask and see if anyone could point me in some directions.

Brian

Sean Cook wrote:
> I would actually spend a little more and start looking at iSCSI for attached
> storage.  You can generally pickup some decent chassis on ebay for not a lot
> of change and it gives you a lot more flexibility.
>
> GFS is ok if you don't want to mess around with a SAN but it has no where
> near the performance of fiber or iSCSI attached storage.
>
> Here is exactly what I am talking about...
> http://cgi.ebay.com/Dell-EMC-AX100i-iSCSI-12-Slot-SAN-Array-w-4x-250GB-HDD_W0QQitemZ300072200442QQihZ020QQcategoryZ111458QQssPageNameZWDVWQQrdZ1QQcmdZViewItem
>
>
> On 25-Jan-2007, Brian Kroth wrote:
>   
>> Hello all,
>>
>> I currently manage about 40 Window, OSX, and Hardened Gentoo servers.  I 
>> will soon have 12 P4 servers that were previously used as video encoders 
>> free as well as an Apple XRaid.  With all this spare hardware I thought 
>> I'd research setting up a cluster of servers running Apache for load 
>> balancing and high availability.  I'm also looking into a MySQL cluster, 
>> but that wouldn't require a shared filesystem.  I'm wondering if anyone 
>> has done something like this before and in particular knows a good 
>> filesystem to use so that each of the servers can access and potentially 
>> write to the same storage array.  I've accomplished the same thing with 
>> XServes running OSX, but they like to charge you a pretty penny for the 
>> XSan software that allows this which I thought I'd try to avoid if 
>> possible.  So far I've seen only GFS, but haven't gotten much reading 
>> done on it yet.  Any other tips or insights would be appreciated as well.
>>
>> Thanks,
>> Brian
>> -- 
>> gentoo-server@gentoo.org mailing list
>>
>>     

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] SAN Clustered Filesystem

[Sean Cook]

OK... it looks like this is just fiber channel... do you have a fiber switch
or are you using direct connects?

In gentoo you will use a qlogic card and just boot the box... the luns that
are presented to the server will just show up as raw disk.  If you want to
get sexy with it, you multipath to set up dual channels to the same disk and
round robin your access to those disks.  This will require a bit more
knowlege of the XRaid and how it presents storage, but it should be
possible.  

If you are using a fiber switch you will want to create a zone from the
device to the nodes based on the wwn that you get off the qlogic interface
which should look something like this:

$ cat /proc/scsi/qla2xxx/0
...
SCSI Device Information:
scsi-qla1-adapter-node=200100e08ba84706;
scsi-qla1-adapter-port=210100e08ba84706;
scsi-qla1-target-0=50060482d52cead8;
...




On 25-Jan-2007, Brian Kroth wrote:
> We actually have an XRaid now sitting in a box.  I was simply going to 
> purchase some fiber channel cards for the servers so they can attach to 
> it via a QLogic fiber channel switch we have.  But once that's happened 
> I want to make sure I can share the same storage pool with all, or 
> perhaps just a subset, of the servers.  With OSX this required a special 
> XSan client.  I have done very limited research so far to see what it 
> would take to get a collection of Gentoo servers to do this, but figured 
> I should ask and see if anyone could point me in some directions.
> 
> Brian
> 
> Sean Cook wrote:
> >I would actually spend a little more and start looking at iSCSI for 
> >attached
> >storage.  You can generally pickup some decent chassis on ebay for not a 
> >lot
> >of change and it gives you a lot more flexibility.
> >
> >GFS is ok if you don't want to mess around with a SAN but it has no where
> >near the performance of fiber or iSCSI attached storage.
> >
> >Here is exactly what I am talking about...
> >http://cgi.ebay.com/Dell-EMC-AX100i-iSCSI-12-Slot-SAN-Array-w-4x-250GB-HDD_W0QQitemZ300072200442QQihZ020QQcategoryZ111458QQssPageNameZWDVWQQrdZ1QQcmdZViewItem
> >
> >
> >On 25-Jan-2007, Brian Kroth wrote:
> >  
> >>Hello all,
> >>
> >>I currently manage about 40 Window, OSX, and Hardened Gentoo servers.  I 
> >>will soon have 12 P4 servers that were previously used as video encoders 
> >>free as well as an Apple XRaid.  With all this spare hardware I thought 
> >>I'd research setting up a cluster of servers running Apache for load 
> >>balancing and high availability.  I'm also looking into a MySQL cluster, 
> >>but that wouldn't require a shared filesystem.  I'm wondering if anyone 
> >>has done something like this before and in particular knows a good 
> >>filesystem to use so that each of the servers can access and potentially 
> >>write to the same storage array.  I've accomplished the same thing with 
> >>XServes running OSX, but they like to charge you a pretty penny for the 
> >>XSan software that allows this which I thought I'd try to avoid if 
> >>possible.  So far I've seen only GFS, but haven't gotten much reading 
> >>done on it yet.  Any other tips or insights would be appreciated as well.
> >>
> >>Thanks,
> >>Brian
> >>-- 
> >>gentoo-server@gentoo.org mailing list
> >>
> >>    
> 
> -- 
> gentoo-server@gentoo.org mailing list
> 
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] SAN Clustered Filesystem

[paul kölle]

Sean Cook schrieb:
> I would actually spend a little more and start looking at iSCSI for attached
> storage.  You can generally pickup some decent chassis on ebay for not a lot
> of change and it gives you a lot more flexibility.
> 
> GFS is ok if you don't want to mess around with a SAN but it has no where
> near the performance of fiber or iSCSI attached storage.
Aren't those apples and oranges? I thought iSCSI is a block level
protocol and doesn't do locking and such whereas GFS does...

sorry, noob wrt above
 Paul
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] SAN Clustered Filesystem

[Sean Cook]

Both are example of potential shared filesystems... GFS is even pitched as a
low cost alternative to a SAN or NAS (see redhats docs).

I would agree that the technologies are vastly different.  GFS is a
clustering Filesystem, where as iSCSI is a block level device similar to FC
and FCAL.

Regards,

Sean


On 25-Jan-2007, paul k?lle wrote:
> Sean Cook schrieb:
> > I would actually spend a little more and start looking at iSCSI for attached
> > storage.  You can generally pickup some decent chassis on ebay for not a lot
> > of change and it gives you a lot more flexibility.
> > 
> > GFS is ok if you don't want to mess around with a SAN but it has no where
> > near the performance of fiber or iSCSI attached storage.
> Aren't those apples and oranges? I thought iSCSI is a block level
> protocol and doesn't do locking and such whereas GFS does...
> 
> sorry, noob wrt above
>  Paul
> -- 
> gentoo-server@gentoo.org mailing list
> 
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] SAN Clustered Filesystem

[Brian Kroth]

paul kölle wrote:
> Sean Cook schrieb:
>>
>> GFS is ok if you don't want to mess around with a SAN but it has no where
>> near the performance of fiber or iSCSI attached storage.
> Aren't those apples and oranges? I thought iSCSI is a block level
> protocol and doesn't do locking and such whereas GFS does...

This is what I was getting at.  I know the basics of working with the 
SAN  to get a set of machines to at least see a storage array.  The next 
step is getting them to read and write to say the same file on a 
filesystem on that storage array without stepping on each others toes or 
corrupting the filesystem that lives on top of that storage array. 
That's where I haven't learned too much yet.

I hadn't actually planned on using the SAN to boot off of, but that 
might be an option for easier configuration/software management.  I 
simply wanted to use it almost as if it were an NFS mount that a group 
of servers stored web content on.  The problem I had with that model is 
that the NFS server is a single point of failure.  If on the other hand 
all the servers are directly attached to the data, any one of them can 
go down and the others won't care or notice.  At least that's the 
working theory behind it right now.
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] SAN Clustered Filesystem

[Sean Cook]

Sorry... I didn't understand exactly what you were trying to do... if you
want to present the same disks to multiple host you would generally use a
cluster "aware" application or a cluster "aware" filesystem that would
establish the quarum and act a more true clustered environment with heartbeat.

Going back to GFS, this is actually capable of doing just that.  So I guess
we are back to GFS :)

http://mail.digicola.com/wiki/index.php?title=User:Martin:GFS

http://www.yolinux.com/TUTORIALS/LinuxClustersAndFileSystems.html


Should give you enough reading to get started... most of the applications we
do this with are either read only data sets and don't need clustering
filesystems or are oracle databases that use ocfs.  So I can only speak in
theory... I would check out what veritas products can do... They have an
amazing track record on almost all nix platforms (solaris,hp ux) and have a
lot of clustering capabilities.

Regards,

sean

On 25-Jan-2007, Brian Kroth wrote:
> 
> 
> paul k?lle wrote:
> >Sean Cook schrieb:
> >>
> >>GFS is ok if you don't want to mess around with a SAN but it has no where
> >>near the performance of fiber or iSCSI attached storage.
> >Aren't those apples and oranges? I thought iSCSI is a block level
> >protocol and doesn't do locking and such whereas GFS does...
> 
> This is what I was getting at.  I know the basics of working with the 
> SAN  to get a set of machines to at least see a storage array.  The next 
> step is getting them to read and write to say the same file on a 
> filesystem on that storage array without stepping on each others toes or 
> corrupting the filesystem that lives on top of that storage array. 
> That's where I haven't learned too much yet.
> 
> I hadn't actually planned on using the SAN to boot off of, but that 
> might be an option for easier configuration/software management.  I 
> simply wanted to use it almost as if it were an NFS mount that a group 
> of servers stored web content on.  The problem I had with that model is 
> that the NFS server is a single point of failure.  If on the other hand 
> all the servers are directly attached to the data, any one of them can 
> go down and the others won't care or notice.  At least that's the 
> working theory behind it right now.
> -- 
> gentoo-server@gentoo.org mailing list
> 
-- 
gentoo-server@gentoo.org mailing list

[gentoo-server] Re: [gentoo-cluster] SAN Clustered Filesystem

[Ramon van Alteren]

Hi Brian,

Brian Kroth wrote:
> I currently manage about 40 Window, OSX, and Hardened Gentoo servers. 
> I will soon have 12 P4 servers that were previously used as video
> encoders free as well as an Apple XRaid.  With all this spare hardware
> I thought I'd research setting up a cluster of servers running Apache
> for load balancing and high availability.  I'm also looking into a
> MySQL cluster, but that wouldn't require a shared filesystem.  I'm
> wondering if anyone has done something like this before and in
> particular knows a good filesystem to use so that each of the servers
> can access and potentially write to the same storage array.  I've
> accomplished the same thing with XServes running OSX, but they like to
> charge you a pretty penny for the XSan software that allows this which
> I thought I'd try to avoid if possible.  So far I've seen only GFS,
> but haven't gotten much reading done on it yet.  Any other tips or
> insights would be appreciated as well.

I currently manage 4 different shared storage systems on gentoo with
SAN-storage based on AoE (coraid 1520 machines)
We choose GFS for the filesystem.

It works, although the throughput is not spectacular.
This has partly to do with GFS and partly because the underlying storage
hardware doesn't support faster throughput.
If you have any specific questions about GFS I'd be happy to answer them.

Apart from that we're currently planning a Lustre deploy for fast-storage.
This is still very much in alpha, so no comments yet.

I've looked at implementing a mysql-cluster in great detail and let it
go because it's mostly meant for a setup with a fixed database-size.
If you experience (large) growth in your database volume clustering is
not very suitable (we're in that situation)
We're currently load-balancing our databases with LVS (ipvsadm) and a
multi-master replication setup.
This works out really well for us, serving estimated average 15Kq/s 
with 50 servers on a very large dataset (> 40Gb) and a fairly random
read pattern.

I would strongly disadvise running multiple databases off GFS.
>From what I gathered it works well for active/passive failover sets
and/or active/active failover, but performance will be hurt compared to
a "normal" fs.

Regards,

Ramon

-- 
gentoo-server@gentoo.org mailing list

[gentoo-server] Re: SAN Clustered Filesystem

[Kevin]

Brian Kroth wrote:
> Hello all,
> 
> I currently manage about 40 Window, OSX, and Hardened Gentoo servers.  I
> will soon have 12 P4 servers that were previously used as video encoders
> free as well as an Apple XRaid.  With all this spare hardware I thought
> I'd research setting up a cluster of servers running Apache for load
> balancing and high availability.  I'm also looking into a MySQL cluster,
> but that wouldn't require a shared filesystem.  I'm wondering if anyone
> has done something like this before and in particular knows a good
> filesystem to use so that each of the servers can access and potentially
> write to the same storage array.  I've accomplished the same thing with
> XServes running OSX, but they like to charge you a pretty penny for the
> XSan software that allows this which I thought I'd try to avoid if
> possible.  So far I've seen only GFS, but haven't gotten much reading
> done on it yet.  Any other tips or insights would be appreciated as well.
> 
> Thanks,
> Brian

You may want to consider OpenAFS: http://www.openafs.org/

Not really sure if this is what you're after or not, but since there have
been several ideas mentioned in this thread in response to your post
already, I thought I'd mention it in case you're not aware of it.

HTH.

-Kevin
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Re: SAN Clustered Filesystem

[Brian Kroth]

Yeah, thanks everyone for your responses.  I'll most likely be testing a 
bunch of these in the coming weeks and months to find the one I think 
works the best.  I'll try and post some results when I'm done.  So far 
just from the reading I think my best bets are Lustre, GFS, and OCFS2. 
The cluster list also had some mention about GlusterFS that I'm curious 
about.

Also, I did some more research on the MySQL end of things and have 
decided that replication is the way to go.  Thanks Hanni for that nice 
graphic.

Brian

Kevin wrote:
> Brian Kroth wrote:
>> Hello all,
>>
>> I currently manage about 40 Window, OSX, and Hardened Gentoo servers.  I
>> will soon have 12 P4 servers that were previously used as video encoders
>> free as well as an Apple XRaid.  With all this spare hardware I thought
>> I'd research setting up a cluster of servers running Apache for load
>> balancing and high availability.  I'm also looking into a MySQL cluster,
>> but that wouldn't require a shared filesystem.  I'm wondering if anyone
>> has done something like this before and in particular knows a good
>> filesystem to use so that each of the servers can access and potentially
>> write to the same storage array.  I've accomplished the same thing with
>> XServes running OSX, but they like to charge you a pretty penny for the
>> XSan software that allows this which I thought I'd try to avoid if
>> possible.  So far I've seen only GFS, but haven't gotten much reading
>> done on it yet.  Any other tips or insights would be appreciated as well.
>>
>> Thanks,
>> Brian
> 
> You may want to consider OpenAFS: http://www.openafs.org/
> 
> Not really sure if this is what you're after or not, but since there have
> been several ideas mentioned in this thread in response to your post
> already, I thought I'd mention it in case you're not aware of it.
> 
> HTH.
> 
> -Kevin
-- 
gentoo-server@gentoo.org mailing list