[gentoo-user] Strategy for using SAN/NAS for storage with Gentoo...

[gentoo-user] Strategy for using SAN/NAS for storage with Gentoo...

[Steve]

I have recently started looking at server resilience and availability in
the context of a hardware failure or hardware upgrade.  I've come to the
conclusion that it would be very desirable if terrabyte-scale data did
not need to be restored from backup.  This isn't a commercial server -
so I'm interested in minimum cost approaches.

With this in mind, I'm interested to discover what represents
state-of-the-art from the perspective of the OS and its configuration. 
Issues I envisage are:

* With NAS, it would be desirable to have a Linux filesystem rather than
access files over CIFS - this raises further questions about protocol...
is NFS as hopelessly outdated as it seems?  Are there any products that
offer NFS access?  Are any of them secure?
* With a SAN, questions of filesystem features are diminished - but
questions of access protocol remain.  What is best supported by gentoo?
* Do any gentooists have any inexpensive hardware configurations that
work especially well?

Any hints or tips?

[gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Harry Putnam]

Steve <gentoo_sjh@shic.co.uk> writes:

> I have recently started looking at server resilience and availability in
> the context of a hardware failure or hardware upgrade.  I've come to the
> conclusion that it would be very desirable if terrabyte-scale data did
> not need to be restored from backup.  This isn't a commercial server -
> so I'm interested in minimum cost approaches.
>
> With this in mind, I'm interested to discover what represents
> state-of-the-art from the perspective of the OS and its configuration. 
> Issues I envisage are:
>
> * With NAS, it would be desirable to have a Linux filesystem rather than
> access files over CIFS - this raises further questions about protocol...
> is NFS as hopelessly outdated as it seems?  Are there any products that
> offer NFS access?  Are any of them secure?
> * With a SAN, questions of filesystem features are diminished - but
> questions of access protocol remain.  What is best supported by gentoo?
> * Do any gentooists have any inexpensive hardware configurations that
> work especially well?
>
> Any hints or tips?

Someone here, a yr or two ago recommended to me when I asked that
question to install opensolaris on a machine and set it up as NAS.

Opensolaris offers the zfs file system, that is really advanced
compared to others.

I'll admit I've had some issues along the way.  And have to do lots of
boning up on opensolaris.

I access the zfs server by cifs from windows machines, and by NFS from
linux.

Opensolaris doesn't use samba by default.  Instead they have their own
CIFS server which works fine.  They do have samba pkgs but no one
hardly use it, preferring their CIFS server.

Mine  is only a home lan setup, but even then the opensolaris server
has over a terabyte of capacity.

I have it setup in 3 mirrors of 2 disks each.

2 prs of 500gb sata hdd and one pr of 750 sata hdd

There are many ways so setup `zraid' systems on `zfs' that are excellent for
reliability. ( I have been told.... I haven't tried that route).

For me the mirror setup seemed good for my small needs.  Even more
reliable I'm told, if a bit higher in disk usage.

opensolaris zfs fs offers a `timeslider' interface to a system of
snapshotting the filesystems in 15 min, 1hr 1day etc snapshots in a
very small footprint way.  You'd have to read up on it.. It would take
too much off topic to cover here.  The system takes an amazing small
amount of disk space for the snapshots based on COW (Copy on write).

There was talk of opensolaris going by the wayside with the Oracle
takeover of Sun... but Oracle has since announced its intention of
puttin even more resources into `opensolaris' development than Sun was
doing.

These may be a good intro to zfs:
http://www.sun.com/bigadmin/features/articles/zfs_part1.scalable.jsp
http://all-unix.blogspot.com/2007/03/zfs-cow-and-relate-features.html
http://wikis.sun.com/display/OpenSolarisInfoPL/How+to+Manage+the+Automatic+ZFS+Snapshot+Service

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Kyle Bader]

+1 on zfs w/ solaris for storage, just don't go cheap and get desktop disks.
-- 

Kyle

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Steve]

On 15/03/2010 15:49, Kyle Bader wrote:
> +1 on zfs w/ solaris for storage, just don't go cheap and get desktop disks.
>   
I have to admit, I do like the idea of ZFS, though not quite enough to
justify maintaining Solaris in addition to my other infrastructure.

I was thinking about something rather different entirely.  I was
thinking about bunging disk on my LAN and shifting as much data from
local storage on my server as possible.  This would mean that the server
could be swapped out with minimum effort.  If 'disk on the net' allowed
mirroring etc. then storage could be expanded and contracted as
necessary without any downtime... essentially, only my hub would then be
a single-point-of-failure.   I'd love to be able to run a VM on my
desktop, for example, and use that as a 'stand-in' while I take-down my
main server for maintenance.  For this to work, I'd need to access the
same file system and be able to switch responsibility for services
between the two 'servers' quickly.

From ages ago, I remember iSCSI being bandied about.  Did that ever go
anywhere (i.e. is this easy to do from Gentoo?)

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Stroller]

On 15 Mar 2010, at 16:26, Steve wrote:
> ...
> From ages ago, I remember iSCSI being bandied about.  Did that ever go
> anywhere (i.e. is this easy to do from Gentoo?)

I believe it is quite widely used - it is mentioned often on the linux- 
poweredge list. I would imagine the Linux kernel allows mounting and  
sharing by iSCSI - check `make menuconfig` and type "/iscsi".

It's hard to be more specific without knowing your usage.

For storage of a "mere terabyte" you can buy a networked storage  
enclosure which will accommodate two drives. These are cheap, do  
mirroring, will accommodate standard 1TB, 1.5TB, 2TB drives, but are  
probably not too fast.

One reads a lot posted by people who have large movie collections  
stored on the network, whether they be MythTV users or the mutineer  
sailors of 17th century galleons. A PC-based solution gives you more  
room for this - you can fit perhaps 4 drives in a standard PC case you  
find at the tip, or you can get 12 or 16 drives in a dedicated  
rackmount server case. This allows capacity of upto 32TB with current  
drives, if you can afford that, or to use cheaper drives (1TB or 1.5TB  
are best gigabytes-per-dollar at present, I think; 500gb drives seem  
recently to have become disproportionately expensive) and have better  
RAID levels.

The Norco one is popular amongst enthusiasts, because it's really  
cheap [1]; it uses 2 x standard ATX power supplies, one for the  
mainboard, one for the drives. You can get similar cases with the  
option of hot-swap PSUs - Chenbro used to be the main brand for this,  
I think, but in the last couple of years TST <http://TSTcom.com> have  
started producing nicer cases; I use a TST ESR-316, which is utterly  
lush, but which was expensive. I have one slight reservation about the  
TST, which I will not spend time detailing unless you ask.

I use only half the TST's capacity at present, but it is a pleasure  
and a relief to have so much room available; expansion of network  
drive capacity is never a problem - just slap a drive in and you're  
ready to go. Even with as many as 6 or 8 drive bays there are corner  
cases which can make expansion a bit of a headache (at least if uptime  
is important).

Since these cases accommodate standard ATX motherboards, you get to  
use an old Pentium 4 motherboard salvaged from an old PC or an Atom- 
based motherboard for £100 or so. The latter price is a bit shocking,  
IMO, compared to (say) the Asus EE-PC, but it reflects the demand for  
them; they're prolly only $100 in the US. These atom motherboards have  
minimal expansion slots, but if you only want to use it for storage  
then you're probably fine with just one.

If you build your own server you can use software or hardware RAID.  
Fast hardware RAID, based on an PCIe controller card, is expensive.  
You can get PCI or PCI-X hardware RAID very cheaply on eBay these  
days, but it's slow. That is to say that PCI or PCI-X hardware RAID is  
fast enough to stream a couple of movies at the same time, fast enough  
to copy 5gb files only a couple of minutes, but production server  
systems (if you were buying a database server for work) would be  
expected to use a PCIe-based hard-drive controller. Hardware RAID is  
nice in its ability to hot-swap out a failed hard-drive without  
interruption. I have not found non-RAID SATA controllers that satisfy  
me with their ability to do hot-swap (although I would love to).

Managing RAID on a PC-based server - rather than a dedicated NAS  
enclosure - very easily allows expansion. With RAID5 or 6 you can just  
add in another drive and expand on to it. I use an old PCI-X (fits in  
a PCI slot) 3ware 9500 card, and it *seems* like if you have a RAID1  
(haven't tried RAID5) on two drives of capacity X, then remove each of  
those drives in turn, rebuilding onto drives of X+Y capacity, then  
upon completion the array appears to the o/s as the larger X+Y size. I  
think some LSI cards do this, also. I would not bet on the ability of  
low-end NAS boxes to do this.

A company called Drobo makes some high-end NAS hardware with space for  
plenty of drives (on some models) and some fancy features. I find UK  
prices a bit shocking, but depending upon your application they might  
be justified; the US prices seem quite reasonable to me.

I wouldn't get too het up about Samba / CIFS vs NFS. Samba / CIFS can  
be faster than NFS, even in an all-Linux environment. Other times it's  
not. This seems pretty much random, depending upon whom is doing the  
benchmarking. On an intellectual level, at least, I find neither  
wholly satisfying - it would be really nice to have a Linux-native  
network filesystem that does authentication / permissions properly.  
But both do work.

I looked at ZFS, but decided that Solaris, from a look at the HCL, was  
too picky over hardware. I think ZFS is great, I no longer think it's  
the future. My selection of cheap hardware is far wider under Linux, I  
can install Gentoo and just `emerge mediatomb` and stream movies to my  
PS3.

So there ya go. Lots of options, budget from dead cheap to mega money.  
Depends how much you can justify.

Stroller.



[1] http://www.newegg.com/product/product.aspx?item=n82e16811219021

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Steve]

On 15/03/2010 18:21, Stroller wrote:
> It's hard to be more specific without knowing your usage.
Yes... I was deliberately vague to see what options came up... but I can
be more specific.  The budget is miniscule - and the performance demands
(bandwidth and latency) are completely non-challenging.  It's in this
context that I'm looking for reliability and availability... and I'd
like to have unix permissions working properly.  Security is a moderate
concern - the physical network is secured - but there is a broadband
connection which exposes various services.

> For storage of a "mere terabyte" you can buy a networked storage
> enclosure which will accommodate two drives. These are cheap, do
> mirroring, will accommodate standard 1TB, 1.5TB, 2TB drives, but are
> probably not too fast.

A cheap NAS enclosure is a definite possibility - there'd be no
performance issue - though this leaves three key questions:
1) Will it support unix file-permissions and can I be (fairly sure) it
will be secure if someone hacks my Wi-Fi?
2) Will I be able to put the (majority of the) gentoo filesystem on it -
or will I need to have a fully booted system to connect?
3) Can I use two entirely separate devices and mirror to both?  (I
expect the failure of the enclosure to be at least as likely as the
failure of a drive.)

> If you build your own server you can use software or hardware RAID.

Hmmm... building my own server - I've done that in the past, but my plan
is to minimize DIY with a view to minimizing the number of components
that might fail.  Ideally, I'd have four devices - one with a CPU and
memory (the server)... booting from Flash or CD or whatever  (+a
replacement in the cupboard); two separate boxes with drives in them
(mirrored storage); one (wired) Ethernet hub and broadband gateway.  I'd
connect to the network from a separate desktop/laptop to interact with
it - either locally or remotely.

> I wouldn't get too het up about Samba / CIFS vs NFS. Samba / CIFS can
> be faster than NFS, even in an all-Linux environment. Other times it's
> not. This seems pretty much random, depending upon whom is doing the
> benchmarking. On an intellectual level, at least, I find neither
> wholly satisfying - it would be really nice to have a Linux-native
> network filesystem that does authentication / permissions properly.
> But both do work.

Well the 'server' will be running Samba - and it's the back-end storage
for that I'm trying to resolve.  CIFS definitely looks problematic -
since Unix permissions for server data are one valuable separation
between publicly accessible services and my private data.  NFS might be
OK (it doesn't "feel" great) - though I *really* don't want to move from
one server to two when I'm aiming for reliability. 

> I looked at ZFS, but decided that Solaris, from a look at the HCL, was
> too picky over hardware. I think ZFS is great, I no longer think it's
> the future. My selection of cheap hardware is far wider under Linux, I
> can install Gentoo and just `emerge mediatomb` and stream movies to my
> PS3.
I like ZFS, conceptually, though I don't like Solaris.  I'm aware that
Apple have toyed with adopting ZFS and that it is available for BSD... A
*really* neat solution would be a (pair of) cheap NAS devices running an
appliance distribution of BSD with ZFS - exporting a NFS mount...
possibly over a VPN?  Hmmm - I'm trying to avoid complexity, too. Hmmm.

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Andrea Conti]

Hi,

> The budget is miniscule - and the performance demands
> (bandwidth and latency) are completely non-challenging.

This IMHO pretty much rules out any kind of server-class hardware, which
tends to be both costly and power-hungry. If you're thinking about
buying used stuff, be sure to factor in the cost and difficulty of
finding spares in some years' time.

Given the point above I would also stick with software RAID. True HW
RAID controllers are quite expensive and generally come with a x8 PCIe
interface which will require a server motherboard -- x16 PCIe video card
slots in commodity boards are usually only certified for x16 and x1
operation, so don't expect them to work reliably with other bus widths.

Linux software RAID also has the advantage that the kernel is not tied
to any specific piece of hardware. In case of a failure, your volumes
will be readable on any other Linux system -- provided the disks
themselves are not toast.

If reliability is your primary concern, I would go for a simple RAID1
setup; if your volumes need to be bigger than a physical disk you can
build a spanned volume over multiple mirrored pairs. Network throughput
will mostly likely be your primary bottleneck, so I'd avoid striping as
it would offer little in the way of performace at the expense of making
data recovery extremely difficult in case the worst should happen.

As for availability, I think the best strategy with a limited budget is
to focus on reducing downtime: make sure your data can survive the
failure of any single component, and choose hardware that you can get
easily and for a reasonable price. Sh*t happens, so make it painless to
clean up.

Network protocol:

If you do not need data sharing (i.e. if your volumes are only mounted
by one client at a time), the simplest solution is to completely avoid
having a FS on the storage server side -- just export the raw block
device via iSCSI, and do everything on the client. In my experience this
also works very well with Windows clients using the free MS iSCSI initiator.

Alternatively, you can consider good old NFS, which performs decently
and tends to behave a bit better -- especially when used over UDP -- in
case of network glitches, like accidentally powering off a switch,
yanking cables, losing wireless connectivity...

CIFS should be avoided at all costs if your clients are not Windows
machines.

File systems: avoid complexity. As technically superior as it might be,
in this kind of setup ZFS is only going to be resource hog and a
maintenance headache; your priority should be having a rock-solid
implementation and a reliable set of diagnostic/repair tools in case
disaster strikes. Tried-and-true ext3 fits the bill nicely if you ask
me; just remember to tune it properly according to your planned use --
eg. if a volume is going to be used to host huge VM disk images, be sure
to create its filesystem with -T largefile4.

Just my 2 cents,

Andrea

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Steve]

On 15/03/2010 22:29, Andrea Conti wrote:
> This IMHO pretty much rules out any kind of server-class hardware, which
> tends to be both costly and power-hungry. If you're thinking about
> buying used stuff, be sure to factor in the cost and difficulty of
> finding spares in some years' time.
>   
I'm considering neither used equipment nor 'server-class' - the workload
simply doesn't demand it.
> Given the point above I would also stick with software RAID.
...
> If reliability is your primary concern, I would go for a simple RAID1
> setup; 
Absolutely.  Software raid is cheaper and implies less hardware to
fail.  Similarly, RAID1 minimises the total number of disks required to
survive a failure. It's the only way for me to go.
> If you do not need data sharing (i.e. if your volumes are only mounted
> by one client at a time), the simplest solution is to completely avoid
> having a FS on the storage server side -- just export the raw block
> device via iSCSI, and do everything on the client.
This idea is on my wavelength. Has anyone on this tried this?  My
concerns are:

1. Are there reliability issues surrounding this technology in Gentoo?
2. Are there any howtos about putting as much of the file-system as
possible onto an iSCSI device.
3. What's the best (most lightweight) way to expose the disk as a block
device.   I don't want to manage three fully-fledged Linux boxes.  Can
(cheap) NAS devices be used to export iSCSI to Gentoo?
4. What would be the strategy to 'secure' this iSCSI device... it would
be a disaster if my WiFi were cracked and my data corrupted from a
non-authorised host.

> In my experience this also works very well with Windows clients using the free MS iSCSI initiator.
>   
That's fantastic - I had no idea that such software existed.  Now, I
wonder, what's the most lightweight solution to get a couple of iSCSI
devices?  Does it help that MS supports attaching devices this way?
> File systems: avoid complexity. As technically superior as it might be,
> in this kind of setup ZFS is only going to be resource hog and a
> maintenance headache; your priority should be having a rock-solid
> implementation and a reliable set of diagnostic/repair tools in case
> disaster strikes. 
Yes. Separate arguments for snapshot support are compelling... but there
are alternatives without tackling the additional complexity.  That said,
the iSCSI approach would work as well with ZFS as something mundane. 
Snap-shots, of course, are only really valuable for non-archive data...
so, in future, I could add a ZFS volume using the same iSCSI strategy.

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Stroller]

On 16 Mar 2010, at 16:32, Steve wrote:
> ...
>> Given the point above I would also stick with software RAID.
> ...
>> If reliability is your primary concern, I would go for a simple RAID1
>> setup;
> Absolutely.  Software raid is cheaper and implies less hardware to
> fail.  Similarly, RAID1 minimises the total number of disks required  
> to
> survive a failure. It's the only way for me to go.

How does your system boot if your RAID1 system volume fails? The one  
you have grub on? I think you mentioned a flash drive, which I've seen  
mentioned before. This seems sound, but just to point out that's  
another, different, single point of failure.

>> If you do not need data sharing (i.e. if your volumes are only  
>> mounted
>> by one client at a time), the simplest solution is to completely  
>> avoid
>> having a FS on the storage server side -- just export the raw block
>> device via iSCSI, and do everything on the client.
> ...
> Snap-shots, of course, are only really valuable for non-archive  
> data...
> so, in future, I could add a ZFS volume using the same iSCSI strategy.

I have wondered if it might be possible to create a large file (`dd  
if=/dev/zero of=/path/to/large/file` constrain at a size of 20gig or  
100gig or whatever) and treat it as a loopback device for stuff like  
this. It's not true snapshotting (in the ZFS / BTFS sense), but you  
can unmount it and make a copy quite quickly.

Stroller.

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

On Tue, 16 Mar 2010 19:57:49 +0000, Stroller wrote:

> How does your system boot if your RAID1 system volume fails?

You put GRUB on both disks, then you can boot from either on its own.


-- 
Neil Bothwick

Growing old is mandatory; growing up is optional!!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Stroller]

On 16 Mar 2010, at 20:04, Neil Bothwick wrote:

> On Tue, 16 Mar 2010 19:57:49 +0000, Stroller wrote:
>
>> How does your system boot if your RAID1 system volume fails?
>
> You put GRUB on both disks, then you can boot from either on its own.

Is this reliable? I don't contest it, I'm just asking. It's just this  
was one of my considerations when choosing hardware RAID.

Stroller.

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[J. Roeleveld]

On Tuesday 16 March 2010 21:13:29 Stroller wrote:
> On 16 Mar 2010, at 20:04, Neil Bothwick wrote:
> > On Tue, 16 Mar 2010 19:57:49 +0000, Stroller wrote:
> >> How does your system boot if your RAID1 system volume fails?
> >
> > You put GRUB on both disks, then you can boot from either on its own.
> 
> Is this reliable? I don't contest it, I'm just asking. It's just this
> was one of my considerations when choosing hardware RAID.
> 
> Stroller.
> 

This is the generally recommended method and I found this method in the Gentoo 
documentation. If this wouldn't be reliable, I would have expected this not to 
be in the docs for long.

I have /boot mirrored, not actually tested with removing a disk yet, but I am 
confident it will work.

If not, I have other methods of booting the system and getting to the data.

--
Joost

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On Tue, 16 Mar 2010 20:13:29 +0000, Stroller wrote:

> >> How does your system boot if your RAID1 system volume fails?  
> >
> > You put GRUB on both disks, then you can boot from either on its
> > own.  
> 
> Is this reliable? I don't contest it, I'm just asking. It's just this  
> was one of my considerations when choosing hardware RAID.

Yes it is, if sda fails unplug it and sdb becomes sda (or hd1 becomes
hd0 in GRUB terms) and the boot continues. Because RAID1 puts the RAID
superblock in a different location from the ordinary one, you can use
either disk from a RAID1 array as a single disk.


-- 
Neil Bothwick

Minds are like parachutes; they only function when fully open. * Sir
James Dewar

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

Am 16.03.2010 22:26, schrieb Neil Bothwick:
> On Tue, 16 Mar 2010 20:13:29 +0000, Stroller wrote:
> 
>>>> How does your system boot if your RAID1 system volume fails?  
>>>
>>> You put GRUB on both disks, then you can boot from either on its
>>> own.  
>>
>> Is this reliable? I don't contest it, I'm just asking. It's just this  
>> was one of my considerations when choosing hardware RAID.
> 
> Yes it is, if sda fails unplug it and sdb becomes sda (or hd1 becomes
> hd0 in GRUB terms) and the boot continues. Because RAID1 puts the RAID
> superblock in a different location from the ordinary one, you can use
> either disk from a RAID1 array as a single disk.
> 
> 

Just for clarification: Is it really necessary to unplug the broken disk
for this to work?
If read access fails on sda and the BIOS tries sdb, would this also
work? Isn't grub's hd0 always the disk on which grub resides (e.g. the
disk from which grub managed to boot)?

Thanks in advance!
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

On Wed, 17 Mar 2010 21:44:34 +0100, Florian Philipp wrote:

> Just for clarification: Is it really necessary to unplug the broken disk
> for this to work?
> If read access fails on sda and the BIOS tries sdb, would this also
> work? Isn't grub's hd0 always the disk on which grub resides (e.g. the
> disk from which grub managed to boot)?

I suspect that may be dependent on the nature of the failure. For
example, if /boot is corrupted, the BIOS will still boot from the broken
disk's MBR before failing later.

Most BIOSes now enable you to disable individual SATA ports, so you could
disappear the disk without unplugging it, although I'm not sure why you'd
want to leave a broken disk in the box.


-- 
Neil Bothwick

This is the day for firm decisions! Or is it?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

Am 17.03.2010 22:00, schrieb Neil Bothwick:
> On Wed, 17 Mar 2010 21:44:34 +0100, Florian Philipp wrote:
> 
>> Just for clarification: Is it really necessary to unplug the broken disk
>> for this to work?
>> If read access fails on sda and the BIOS tries sdb, would this also
>> work? Isn't grub's hd0 always the disk on which grub resides (e.g. the
>> disk from which grub managed to boot)?
> 
> I suspect that may be dependent on the nature of the failure. For
> example, if /boot is corrupted, the BIOS will still boot from the broken
> disk's MBR before failing later.
> 
> Most BIOSes now enable you to disable individual SATA ports, so you could
> disappear the disk without unplugging it, although I'm not sure why you'd
> want to leave a broken disk in the box.
> 
> 

Just in case I ever face high demands on uptime. It's good to know
whether I can still (remote) reboot a machine and it will come up
although one of its drives is broken.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Steve]

On 16/03/2010 19:57, Stroller wrote:
> How does your system boot if your RAID1 system volume fails? The one
> you have grub on? I think you mentioned a flash drive, which I've seen
> mentioned before. This seems sound, but just to point out that's
> another, different, single point of failure.
Well, at the moment, I don't have a RAID system... A flash drive (USB
key) seems a reasonable strategy - I could even have two containing
identical data - so, if the first were to fail then the second would
kick in - if not automatically - then after the duff flash-drive is
removed.  A neat side effect of this would be to eliminate a moving part
on the server - making it quieter... and the drives themselves can be
located at two physically remote places on my LAN.

>>> by one client at a time), the simplest solution is to completely avoid
>>> having a FS on the storage server side -- just export the raw block
>>> device via iSCSI, and do everything on the client.
>> ...
>> Snap-shots, of course, are only really valuable for non-archive data...
>> so, in future, I could add a ZFS volume using the same iSCSI strategy.
> If you do not need data sharing (i.e. if your volumes are only mounted
Yes - I don't think I'd need sharing.  It strikes me that it should be
possible to have a 'live' backup server which just reads until
fail-over...  with a different /var/* - of course.

> I have wondered if it might be possible to create a large file (`dd
> if=/dev/zero of=/path/to/large/file` constrain at a size of 20gig or
> 100gig or whatever) and treat it as a loopback device for stuff like
> this. It's not true snapshotting (in the ZFS / BTFS sense), but you
> can unmount it and make a copy quite quickly.
You could, but the advantage of ZFS is the efficiency of snap-shots.
With your strategy I'd need to process all of the large file every time
I want to make a snapshot... which, even for a mere 100gig, won't be quick.

Re: [gentoo-user] Strategy for using SAN/NAS for storage with Gentoo...

[Andrea Conti]

> 1. Are there reliability issues surrounding this technology in Gentoo?

My only experience is with a Gentoo-based iSCSI target (ie. "server");
my clients are windows-based. The system is a low-end Core 2 duo running
the latest stable kernel and Iscsi Enterprise Target; I have been
running this setup non-stop for a couple of years and it has proven
quite stable.

iSCSI is designed with a dedicated, reliable network in mind, and in my
experience it is quite sensitive to network connectivity issues. It is
best used over gigabit ethernet; fast ethernet is ok, too, if you don't
care about performace. Avoid WiFi if you value your data (and your
mental health)

> 2. Are there any howtos about putting as much of the file-system as
> possible onto an iSCSI device.

Google "root over iscsi". For example:

http://wpkg.org/Diskless_/_remote_boot_with_Open-iSCSI

I have _not_ tried it. It is an interesting concept, but I think that
the OS is better left on a local disk -- the performance penalty is way
too great, especially with the king of budget-oriented storage backend
you are considering.

> 3. What's the best (most lightweight) way to expose the disk as a
> block device. I don't want to manage three fully-fledged Linux boxes.

The only software you need is an iSCSI initiator: a minimal Gentoo
install running sys-block/iscsitarget is enough.

IET allows you to export any kind of raw block device (a disk, a
partition, a RAID volume,...) or even a file on a local filesystem.

Or perhaps you can look into FreeNAS (http://freenas.org), which is less
flexible than a full-fledged OS install but might be enough in your case.

> Can (cheap) NAS devices be used to export iSCSI to Gentoo?

If the NAS device can "speak" iSCSI, well, yes.

> 4. What would be the strategy to 'secure' this iSCSI device... it would
> be a disaster if my WiFi were cracked and my data corrupted from a
> non-authorised host.

iSCSI connections are authenticated with a challenge-response mechanism;
in IET you can also restrict access to specific hosts on a per-volume
basis. That should be enough if you are not transferring the data itself
over WiFi, which is a Bad Thing and should not be done.

> Snap-shots, of course, are only really valuable for non-archive data...
> so, in future, I could add a ZFS volume using the same iSCSI strategy.

ZFS allows you to take FS-level snapshots -- with iSCSI that would be on
the client, onto a network-connected volume, and I don't know what kind
of performance implications that has.

If you want to take snapshots on the server, my first thought would be
to do so at the block level using LVM. No idea if it plays well with
IET, though.

andrea

Re: [gentoo-user] Re: Strategy for using SAN/NAS for storage with Gentoo...

[Iain Buchanan]

On Mon, 2010-03-15 at 09:37 -0500, Harry Putnam wrote:

> There was talk of opensolaris going by the wayside with the Oracle
> takeover of Sun... but Oracle has since announced its intention of
> puttin even more resources into `opensolaris' development than Sun was
> doing.

that will kill it for sure!  (ok, maybe not, but you know the mythical
man month...)

-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Life is like a sewer.  What you get out of it depends on what you put into it.
		-- Tom Lehrer

Re: [gentoo-user] Strategy for using SAN/NAS for storage with Gentoo...

[Keith Dart]

=== On Mon, 03/15, Steve wrote: ===
> Any hints or tips?

===

I recommend setting up your server hardware on a decent mini-PC with
server grade disks and installing openfiler. The openfiler uses XFS for
local storage and exports NFS and CIFS (and iSCSI if you want that). 

http://www.openfiler.com/

It is based on rpath linux and uses a different package management
system than you may be used to. But it's relatively easy to configure
and maintain.


-- Keith Dart

-- 
-- --------------------
Keith Dart
<keith@dartworks.biz>
=======================

Re: [gentoo-user] Strategy for using SAN/NAS for storage with Gentoo...

[Steve]

Keith Dart wrote:
> I recommend setting up your server hardware on a decent mini-PC with
> server grade disks and installing openfiler. The openfiler uses XFS for
> local storage and exports NFS and CIFS (and iSCSI if you want that). 
>
> http://www.openfiler.com/
>
> It is based on rpath linux and uses a different package management
> system than you may be used to. But it's relatively easy to configure
> and maintain.

Both Openfiler and FreeNas look promising from a software perspective.  
Conversely, I'm drawing a bit of a blank trying to find suitable 
hardware to run that software on.  Given that all I need is iSCSI to 
SATA and back... for 1 drive at 100Mbps.... everything I can find seems 
massive overkill.

I've been toying with the idea of abandoning being able to fire-up a 
vmware image to stand in for my server... and shifting to accessing raid 
storage over USB.  It seems a lot less elegant - but it does eliminate 
the need for hardware to run multiple kernels...  When I thought 'iscsi' 
- I'd hoped that I'd find a cheap external drive that supported it 
out-of-the-box for a pittance more than a bare drive.  Was I was being 
hugely overly optimistic?