[gentoo-user] HP Cluster on Gentoo

[gentoo-user] HP Cluster on Gentoo

[Hans-Stefan Bauer]

Hallo,

we are trying to install the gfs file system on a HP Proliant DL385 
Server running Gentoo Linux to set up a cluster system. Here the output 
of "uname -a":

Linux storm2 2.6.18-gentoo-r2 #2 SMP Fri Feb 2 10:03:25 CET 2007 x86_64 
AMD Opteron(tm) Processor 280 GNU/Linux

After configuration and compiling of the kernel (the Gentoo kernel and 
not the vanilla kernel), the following packages are installed:

sys-cluster/ccs                         ~amd64
sys-cluster/magma                       ~amd64
sys-cluster/magma-plugins               ~amd64
sys-cluster/dlm                         ~amd64
sys-cluster/dlm-headers                 ~amd64
sys-cluster/dlm-kernel                  ~amd64
sys-cluster/iddev                       ~amd64
sys-cluster/fence                       ~amd64
sys-cluster/cman                        ~amd64
sys-cluster/cman-headers                ~amd64
sys-cluster/cman-kernel                 ~amd64
sys-fs/gfs                              ~amd64
sys-cluster/gfs-headers                 ~amd64
sys-cluster/gfs-kernel                  ~amd64
sys-fs/clvm                             ~amd64
sys-cluster/rgmanager                   ~amd64

Then I started the necessary services and run into the following problem:

storm1 hsbauer # /etc/init.d/ccsd start
 * Starting ccsd 
...                                                       [ ok ]
storm1 hsbauer # /etc/init.d/cman start
 * Loading cman kernel module 
...                                          [ ok ]
 * Starting cman 
...                                                       [ ok ]
 * Waiting for quorum (300 secs) 
...                                       [ ok ]
storm1 hsbauer # /etc/init.d/clvmd start
 * Loading dlm module ...
 * Loading dlm kernel module 
...                                           [ ok ]
 * Starting fence domain 
...                                               [ ok ]
 * Starting clvmd 
...                                                      [ ok ]
 * Scanning LVM volumes 
...                                                [ ok ]
 * Setting up the Logical Volume Manager ...
  No volume groups 
found                                                   [ ok ]
storm1 hsbauer # /etc/init.d/gfs start
 * Starting gfs cluster:
 * Loading lock_dlm kernel module ...
FATAL: Error inserting lock_dlm (/fs/gfs_locking/lock_dlm/lock_dlm.ko): 
Unknown symbol in module, or unknown parameter (see dmesg)
 * Failed to load lock_dlm kernel 
module                                   [ !! ]


The dmesg message is

....
CMAN 1.03.00 (built Dec  7 2006 15:04:19) installed
NET: Registered protocol family 30
CMAN: Waiting to join or form a Linux-cluster
CMAN: forming a new cluster
CMAN: quorum regained, resuming activity
DLM 1.03.00 (built Feb  1 2007 15:11:38) installed
lock_dlm: Unknown symbol lm_unregister_proto
lock_dlm: Unknown symbol lm_register_proto
....

Apart from the occurring problem it is strange for me that emerge 
installs the kernel module into the root directory of the system (see 
error message above) and not into "/lib/modules/..." as I would expect.

Any help or suggestions are highly appreciated.

Best regards,

Stefan

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] HP Cluster on Gentoo

[Boyd Stephen Smith Jr.]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

On Monday 19 February 2007 02:47, Hans-Stefan Bauer wrote:
> storm1 hsbauer # /etc/init.d/gfs start
>  * Starting gfs cluster:
>  * Loading lock_dlm kernel module ...
> FATAL: Error inserting lock_dlm (/fs/gfs_locking/lock_dlm/lock_dlm.ko):
> Unknown symbol in module, or unknown parameter (see dmesg)
>  * Failed to load lock_dlm kernel
> module                                   [ !! ]

Check your kernel configuration, particularly the "Distributed Lock 
Manager" 
option, CONFIG_DLM.  You may need that part of mainline so that your 
out-of-tree module works.

> Apart from the occurring problem it is strange for me that emerge
> installs the kernel module into the root directory of the system (see
> error message above) and not into "/lib/modules/..." as I would expect.

Hrm, double-check (by looking at your filesystem) to make sure it's in the 
wrong place.  If so, please file a bug, that behavior is broken.

<OT>
By the way, the format of your question was excellent, IMHO.  It had the 
right 
information about how your Gentoo system is set up and the exact, initial 
error message.
</OT>

-- 
Boyd Stephen Smith Jr.                     ,= ,-_-. =. 
bss03@volumehost.net                      ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy           `-'(. .)`-' 
http://iguanasuicide.org/                      \_/     
New GPG Key!  Old key expires 2007-03-25.  Upgrade NOW!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] HP Cluster on Gentoo

[Hans-Stefan Bauer]

Hallo,

Boyd Stephen Smith Jr. schrieb:
> On Monday 19 February 2007 02:47, Hans-Stefan Bauer wrote:
>   
>> storm1 hsbauer # /etc/init.d/gfs start
>>  * Starting gfs cluster:
>>  * Loading lock_dlm kernel module ...
>> FATAL: Error inserting lock_dlm (/fs/gfs_locking/lock_dlm/lock_dlm.ko):
>> Unknown symbol in module, or unknown parameter (see dmesg)
>>  * Failed to load lock_dlm kernel
>> module                                   [ !! ]
>>     
>
> Check your kernel configuration, particularly the "Distributed Lock 
> Manager" 
> option, CONFIG_DLM.  You may need that part of mainline so that your 
> out-of-tree module works.
>   

Thank you for the fast answer. I looked into the file 
"/usr/src/linux/.config". There is no variable CONFIG_DLM in it. Also 
with a "make menuconfig" I did not find a possibility to select it.

>   
>> Apart from the occurring problem it is strange for me that emerge
>> installs the kernel module into the root directory of the system (see
>> error message above) and not into "/lib/modules/..." as I would expect.
>>     
>
> Hrm, double-check (by looking at your filesystem) to make sure it's in the 
> wrong place.  If so, please file a bug, that behavior is broken.
>   

I redirected the output of an emerge into a file with the following command:

emerge sys-cluster/dlm-kernel >& test.out

The full test.out can be found at the following address

machine: ftp.dkrz.de
login     : anonymous
passwd : your Email
cd /pub/Outgoing/stefan_bauer

But, I think, the most important lines are the following:

 >>> Merging sys-cluster/dlm-kernel-1.03.00 to /
--- /cluster/
 >>> /cluster/dlm.ko

If this is a bug, please let me know how I write a bug report - I never 
did this before :-) .

> <OT>
> By the way, the format of your question was excellent, IMHO.  It had the 
> right 
> information about how your Gentoo system is set up and the exact, initial 
> error message.
> </OT>
>
>   
Thank you very much :-) and best regards,

Stefan

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] HP Cluster on Gentoo

[Hans-Werner Hilse]

Hi,

On Tue, 20 Feb 2007 15:33:32 +0100 Hans-Stefan Bauer
<hsbauer@uni-hohenheim.de> wrote:

>  >>> Merging sys-cluster/dlm-kernel-1.03.00 to /
> --- /cluster/
>  >>> /cluster/dlm.ko

Hm. What does "uname -r" say on your system?
What's /lib/modules/$(uname -r)/source linked to? Valid kernel location?

-hwh
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] HP Cluster on Gentoo

[Hans-Stefan Bauer]

Hallo,

Hans-Werner Hilse schrieb:
> Hi,
>
> On Tue, 20 Feb 2007 15:33:32 +0100 Hans-Stefan Bauer
> <hsbauer@uni-hohenheim.de> wrote:
>
>   
>>  >>> Merging sys-cluster/dlm-kernel-1.03.00 to /
>> --- /cluster/
>>  >>> /cluster/dlm.ko
>>     
>
> Hm. What does "uname -r" say on your system?
> What's /lib/modules/$(uname -r)/source linked to? Valid kernel location?
>   

uname -r  --> 2.6.18-gentoo-r2

ls -al /lib/modules/2.6.18-gentoo-r2/source 

provides

lrwxrwxrwx 1 root root   31 Nov 20 17:33 source -> 
/usr/src/linux-2.6.18-gentoo-r2

This is the correct location of the kernel sources.

Best regards,

Stefan
> -hwh
>   

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] HP Cluster on Gentoo

[Boyd Stephen Smith Jr.]

[-- Attachment #1: Type: text/plain, Size: 3339 bytes --]

On Tuesday 20 February 2007, Hans-Stefan Bauer <hsbauer@uni-hohenheim.de> 
wrote about 'Re: [gentoo-user] HP Cluster on Gentoo':
> Hallo,
>
> Boyd Stephen Smith Jr. schrieb:
> > On Monday 19 February 2007 02:47, Hans-Stefan Bauer wrote:
> >> storm1 hsbauer # /etc/init.d/gfs start
> >>  * Starting gfs cluster:
> >>  * Loading lock_dlm kernel module ...
> >> FATAL: Error inserting lock_dlm
> >> (/fs/gfs_locking/lock_dlm/lock_dlm.ko): Unknown symbol in module, or
> >> unknown parameter (see dmesg)
> >>  * Failed to load lock_dlm kernel
> >> module                                   [ !! ]
> >
> > Check your kernel configuration, particularly the "Distributed Lock
> > Manager"
> > option, CONFIG_DLM.  You may need that part of mainline so that your
> > out-of-tree module works.
>
> Thank you for the fast answer. I looked into the file
> "/usr/src/linux/.config". There is no variable CONFIG_DLM in it. Also
> with a "make menuconfig" I did not find a possibility to select it.

Here's the search results against my running kernel (2.6.19-gentoo-r2).

Symbol: DLM [=n]
Prompt: Distributed Lock Manager (DLM)
  Defined at fs/dlm/Kconfig:4
  Depends on: INET && IP_SCTP && EXPERIMENTAL && (IPV6 || IPV6=n)
  Location:
    -> File systems
      -> Distributed Lock Manager
  Selects: CONFIGFS_FS
  Selected by: GFS2_FS_LOCKING_DLM && BLOCK && GFS2_FS

I was unable to find it in my older kernel (2.6.18-gentoo), so perhaps a 
kernel upgrade could solve (part of) your issue.  Of course, any 
out-of-tree modules will have to be compiled after you fix up 
your /usr/src/linux symlink.  'module-rebuild' can help with that.

2.6.19-gentoo-r2 also provides:

Symbol: GFS2_FS_LOCKING_DLM [=n]
Prompt: GFS2 DLM locking module
  Defined at fs/gfs2/Kconfig:34
  Depends on: BLOCK && GFS2_FS
    Location:
     -> File systems
       -> GFS2 file system support (GFS2_FS [=n])
   Selects: DLM

Which probably makes sys-cluster/gfs-kernel unecessary, too.

> >> Apart from the occurring problem it is strange for me that emerge
> >> installs the kernel module into the root directory of the system (see
> >> error message above) and not into "/lib/modules/..." as I would
> >> expect.
> >
> > Hrm, double-check (by looking at your filesystem) to make sure it's in
> > the wrong place.  If so, please file a bug, that behavior is broken.
>
> I redirected the output of an emerge into a file with the following
> command:

I might look at this, but the easiest way to see if it's in the wrong 
location is just find it on the filesystem.  (I don't trust the output of 
emerge all the time anyway -- some packages do stupid things like ignoring 
$ROOT and whotnot.)

>  >>> /cluster/dlm.ko

Hrm, make sure this is modprobed before gfs tries to load lock_dlm.  That 
might solve your issue without a kernel upgrade.  In my first post, I 
didn't consider the fact that you have sys-cluster/dlm-kernel (which 
provides the out-of-tree dlm.ko) despite the fact that you mentioned it.

-- 
Boyd Stephen Smith Jr.                     ,= ,-_-. =. 
bss03@volumehost.net                      ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy           `-'(. .)`-' 
http://iguanasuicide.org/                      \_/     
New GPG Key!  Old key expires 2007-03-25.  Upgrade NOW!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] HP Cluster on Gentoo

[Zac Slade]

On Monday 19 February 2007 02:47:03 Hans-Stefan Bauer wrote:
>  * Loading lock_dlm kernel module ...
> FATAL: Error inserting lock_dlm (/fs/gfs_locking/lock_dlm/lock_dlm.ko):
> Unknown symbol in module, or unknown parameter (see dmesg)
>  * Failed to load lock_dlm kernel
> module                                   [ !! ]

This looks strangely like dlm-kernel was compiled against the wrong kernel.  
Or that dlm-kernel doesn't have support for that kernel. Do you have a newer 
kernel installed than the one you are using?  It very well might have built 
against it instead.

This webpage is a bit dated, but shows steps to get GFS working under gentoo. 
https://open.datacore.ch/DCwiki.open/Wiki.jsp?page=GFS.Install
They talk about a CONFIG_LOCK_DLM needing to be set in the kernel, but I think 
that is taken care of by the dlm-kernel module.

You'll also need to make sure you have device-mapper installed.

Can you give an update on what you've found and if you've made any progress?

-- 
Zac Slade
krakrjak@volumehost.net
ICQ:1415282 YM:krakrjak AIM:ttyp99
--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] HP Cluster on Gentoo

[Hans-Stefan Bauer]

Hallo,

today I updated portage and installed the newest kernel. uname -a now says

Linux storm1 2.6.19-gentoo-r5 #1 SMP Wed Feb 21 15:45:46 CET 2007 x86_64 
AMD Opteron(tm) Processor 280 AuthenticAMD GNU/Linux

Now DLM and GFS2 are compiled as modules

storm1 linux # grep DLM .config
CONFIG_GFS2_FS_LOCKING_DLM=m
CONFIG_DLM=m
# CONFIG_DLM_DEBUG is not set

storm1 linux # grep GFS .config
CONFIG_GFS2_FS=m
# CONFIG_GFS2_FS_LOCKING_NOLOCK is not set
CONFIG_GFS2_FS_LOCKING_DLM=m
CONFIG_CONFIGFS_FS=m

So far so good.

However, then I tried to re-install the cluster manager software 
(packages cman, cman-headers and cman-kernel), since cman is not 
included into the kernel. No problems ocurred with cman and 
cman-headers. But  an emerge of cman-kernel crashed.

...
* Determining the location of the kernel source code
 * Found kernel source directory:
 *     /usr/src/linux
 * Found kernel object directory:
 *     /lib/modules/2.6.19-gentoo-r5/build
 * Found sources for kernel version:
 *     2.6.19-gentoo-r5

--> this is the correct location of the sources !!

...

make[2]: Entering directory `/usr/src/linux-2.6.19-gentoo-r5'
  CC [M]  
/var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/cnxman.o
/var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/cnxman.c: 
In function 'do_ioctl_join_cluster':
/var/tmp/portage/sys-cluster/cman-kernel-1.03.00/work/cluster-1.03.00/cman-kernel/src/cnxman.c:1751: 
error: 'system_utsname' undeclared (first use in this function)

....

!!! ERROR: sys-cluster/cman-kernel-1.03.00 failed.
Call stack:
  ebuild.sh, line 1614:   Called dyn_compile
  ebuild.sh, line 971:   Called qa_call 'src_compile'
  environment, line 4024:   Called src_compile
  cman-kernel-1.03.00.ebuild, line 37:   Called die

!!! compile error
!!! If you need support, post the topmost build error, and the call 
stack if relevant.
!!! A complete build log is located at 
'/var/tmp/portage/sys-cluster/cman-kernel-1.03.00/temp/build.log'.


The variable mentioned in the above error message (system_utsname) is 
nowhere defined in the kernel source tree 2.6.19. In the older kernel 
2.6.18 it was defined in:

/usr/src/linux/init/version.c

For me this means that the cnan-kernel package is not made for this 
kernel. Is there a newer package of cman for gentoo available?


Best regards,

Stefan


-- 
gentoo-user@gentoo.org mailing list