[gentoo-user] nfs over (lvm over nbd) ?

[gentoo-user] nfs over (lvm over nbd) ?

[Raph]

Hello,
I'm currently trying to setup a network with
a storageless server.
There are 20 computers (2x1.6 Ghz, 1GB RAM, 160GB sata)
and 1 more (outside the room) with the same hardware
(so no real "server"). The whole is connected
with 10/100 ethernet.
Of course, and so sadly, there is not a penny...
In order to make a network where any user can access
his datas whatever the computer he uses, I'm thinking about
the following kind of configuration :
- Each client (one of the 20 computers) runs a MBD daemon
which shares its HD to the server (the 21st) over the
ethernet.
- The "server" handle the 3.2TB in a volume group and
create the logical volumes with LVM
- The "server" also run a NFS daemon
- Then each client can grab it's data through NFS.
- happiness ... ?
------
I never did such a setup but already see a couple of
problems :
- each client need some private HD space to, at least,
boot enough to let the NBD daemon start.
- $10 question : if 20 users login, will the ethernet
be fast enough to support the load ?
------
I though about a more manual way (rsync at login/logout)
but it implies versionning and above all distribution and
mirroring, ... so appears less reliable at first look.
Also I can't imagine a way to abstract the storage with lvm
with a distributed syncing at log{in,out}.

I would greatly appreciate any ideas, critics, ...

Raph

PS-1) Not a penny means that it's already a pain to afford
a 30€, 24 ports switch (the 20 computers were given last year)
So the N.10³ "sata bays" are nothing else than dreams, here.

PS-2) I already saw some subjects outside the specific
gentoo scope on this list but if this one really makes
no sense here, please just kindly precise it so I will
conceal forever :)

Re: [gentoo-user] nfs over (lvm over nbd) ?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

On Wed, 13 May 2009 01:58:18 +0200, Raph wrote:

> I never did such a setup but already see a couple of
> problems :
> - each client need some private HD space to, at least,
> boot enough to let the NBD daemon start.

That's not a problem. Install the OS as usual and mount /home over the
network.

> - $10 question : if 20 users login, will the ethernet
> be fast enough to support the load ?

That depends on how fast the network is and what the users are doing.

A bigger question is what happens if one computer goes down? I suspect
your whole LVM could fail. This seem s an unnecessarily complex approach.
What's wrong with creating each user's home directory on one computer and
exporting it to the others over NFS? If each user has a preferred
workstation, this will also reduce network usage.


-- 
Neil Bothwick

I thought I saw the light at the end of the tunnel...
but it was just some sod with a torch bringing me more work!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] nfs over (lvm over nbd) ?

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

On Wed, 13 May 2009 01:58:18 +0200
Raph <gibboris@gmail.com> wrote:

> Of course, and so sadly, there is not a penny...
> In order to make a network where any user can access
> his datas whatever the computer he uses, I'm thinking about
> the following kind of configuration :
> - Each client (one of the 20 computers) runs a MBD daemon
> which shares its HD to the server (the 21st) over the
> ethernet.
> - The "server" handle the 3.2TB in a volume group and
> create the logical volumes with LVM
> - The "server" also run a NFS daemon
> - Then each client can grab it's data through NFS.
> - happiness ... ?
> ------
> I never did such a setup but already see a couple of
> problems :
> - each client need some private HD space to, at least,
> boot enough to let the NBD daemon start.
> - $10 question : if 20 users login, will the ethernet
> be fast enough to support the load ?

While I've never had any chance to try distributed fs outside the
sandbox, it looks like a best solution in your case, providing both
speed and data redundancy.

Specifically, I mean AFS (there's a bit outdated guide on gentoo-wiki),
lustre and several other less-known projects like tahoe or pohmelfs.

http://allmydata.org/trac/tahoe
http://tservice.net.ru/~s0mbre/old/?section=projects&item=pohmelfs

-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] nfs over (lvm over nbd) ?

[Raph]

Hi,
Neil wrote :
> What's wrong with creating each user's home directory on one computer and
> exporting it to the others over NFS? If each user has a preferred
> workstation, this will also reduce network usage.

If I understand you well, each "client" /etc/exports would be :
/home/user0	server.domain(rw)
...
/home/user378	server.domain(rw)
All the directories have not only to exists but not to be empty
has it would conflict with the others clients serving the "already used"
versions (says for exemple pc14.domain which contains /home/user33/foo.bar)

Then in each "client" /etc/fstab I would put :
server:/home/user0	/home/user0	defaults	0	0
...
server:/home/user378	/home/user378	defaults	0	0

In this case of course I don't rely on a storage server, but I don't know
if nfs v4, v4.1 brings some stuff to "assure the up-to-date status of
the exported directory on the first requested server".

To Mike :
That's interesting, about pohmelfs it seems promising but under
heavy development but I'm even not brave enough to use tahoe.
I will problably look around openAFS (I was considering it as a
potential solution) and/or lustre.

[innocence]
IMHO, what really missed me was a brick easily integratable
with rsync and lvm (a brick to pipe with in a /etc/profile or so :)
which do versionning (how to get an up-to-date directory) and distributed
logical volume management (but lvm means direct access to blocks devices
and implies nbd ...)
[/innocence]
Anyway, thank you for your adivses.

Raph

Re: [gentoo-user] nfs over (lvm over nbd) ?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Thu, 14 May 2009 14:16:46 +0200, Raph wrote:

> > What's wrong with creating each user's home directory on one computer
> > and exporting it to the others over NFS? If each user has a preferred
> > workstation, this will also reduce network usage.  
> 
> If I understand you well, each "client" /etc/exports would be :
> /home/user0	server.domain(rw)
> ...
> /home/user378	server.domain(rw)
> All the directories have not only to exists but not to be empty
> has it would conflict with the others clients serving the "already used"
> versions (says for exemple pc14.domain which
> contains /home/user33/foo.bar)
> 
> Then in each "client" /etc/fstab I would put :
> server:/home/user0	/home/user0	defaults	0	0
> ...
> server:/home/user378	/home/user378	defaults
> 0	0

I meant that each machine would have one or two home directories that
would be exported to the others. In the simplest case, N users and N
computers, each box would have one line in /etc/exports

/home/user1 netmask(rw)

and N-1 lines in /etc/fstab.

box2:/home/user2 /home/user2
...
boxN:/home/userN /home/userN

That way you reduce the impact of one failed machine to only the user
directories on that box, instead of the whole LV.


-- 
Neil Bothwick

"MSDOS didn't get as bad as it is overnight -- it took over ten years
of careful development."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] nfs over (lvm over nbd) ?

[Philipp Riegger]

On Wed, 2009-05-13 at 01:58 +0200, Raph wrote:
> There are 20 computers (2x1.6 Ghz, 1GB RAM, 160GB sata)
> and 1 more (outside the room) with the same hardware
> (so no real "server"). The whole is connected
> with 10/100 ethernet.

This sounds quite ok. The question is, what will you use the systems
for? Because if you do no video, sound or picture editing on a larger
scale, not movie or music sharing, then you don't need that much
storage.

> - Each client (one of the 20 computers) runs a MBD daemon
> which shares its HD to the server (the 21st) over the
> ethernet.
> - The "server" handle the 3.2TB in a volume group and
> create the logical volumes with LVM
> - The "server" also run a NFS daemon
> - Then each client can grab it's data through NFS.
> - happiness ... ?

I'd say no. The setup sounds very very complicated and error-prone.
Before doing this I'd get rid of 1-3 Workstations and put those 1-3
harddisks in the Server to have more storage.

> - each client need some private HD space to, at least,
> boot enough to let the NBD daemon start.

A Gentoo installation is usually between 10 and 20 GB for me, I think.
You can also boot over ethernet.

> - $10 question : if 20 users login, will the ethernet
> be fast enough to support the load ?

Most liekly yes. Maybe it's a little slower, but it will work.


Here is what I'd do:

1) Make normal Gentoo installations on the Workstations. You can most
likely install 1 machine, put the image on the server and copy it
everywhere from there. IP and hostname can be managed by DHCP.

2) Use a NFS shared directory for portage tree and binpackages (for
updating, installing new software).

3) Install the Server. I would recommend using a softraid and therefore
taking the disk from 1-3 Workstations. Then you have 160-320 GB disk
space (RAID1 with 2, RAID5 with 3, RAID6 with 4 disks). For a Server 10
GB for / are enough, make it a little bigger, add some space for the
portage tree and the binpackages, and you have 120-280 GB for /home,
which is really a lot.

This shoulc all be a quite stable setup. Now, if you really need much
more disk space, you can do the nbd/nfs approach (or do it completely
distributed with a Cluster Filesystem and Cluster LVM, but that might be
more work).

Often people need some local disk space and only need it for 1
"session", therefore mount a big partition somewhere (/tmp maybe) and
run tmpwatch regularly. Maybe create a script that automatically
creates /tmp/home/$user for all available accounts. Maybe symlink it
to /home/$user/tmp or something.

Philipp