LVM2+mdraid+systemd (was Re: [gentoo-user] systemd and lvm)

LVM2+mdraid+systemd (was Re: [gentoo-user] systemd and lvm)

[Canek Peláez Valdés]

On Fri, Sep 13, 2013 at 7:42 AM, Stefan G. Weichinger <lists@xunil.at> wrote:
> Am 12.09.2013 20:23, schrieb Canek Peláez Valdés:
>
>> Stefan, what initramfs are you using?
>
> dracut, run via your kerninst-script.
>
>> Could you please explain how is exactly your layout? From drives to
>> partitions to PVs, VGs and LVs? And throw in there also the LUKS and
>> RAID (if used) setup. I will try to replicate that in a VM. Next week,
>> since we have a holiday weekend coming.
>
> thanks for your offer.
>
> I wil happily list my setup BUT let me tell at first that the latest
> sys-fs/lvm2-2.02.100 seems to have fixed that semaphore-issue.
>
> After booting my desktop with it I quickly tested:
>
> # lvcreate -n test -L1G VG03
>   Logical volume "test" created
> #
>
> fine!
>
> Three times ok ...
>
> But I still face the fact that the LVs weren't activated at boot time.
> Manual "vgchange -ay" needed ... or that self-written lvm.service
> enabled as mentioned somewhat earlier.
>
> Here my setup:
>
> # lsblk
> NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
> sda                    8:0    0 931,5G  0 disk
> ├─sda1                 8:1    0     2M  0 part
> ├─sda2                 8:2    0     2G  0 part  [SWAP]
> ├─sda3                 8:3    0   600G  0 part
> │ └─md127              9:127  0 595,1G  0 raid1
> │   ├─VG03-music     253:0    0   190G  0 lvm   /mnt/music
> │   ├─VG03-platz     253:1    0   200G  0 lvm   /mnt/platz
> │   ├─VG03-media     253:2    0    45G  0 lvm   /mnt/media
> │   ├─VG03-home      253:3    0    30G  0 lvm
> │   ├─VG03-oopsfiles 253:4    0    12G  0 lvm   /mnt/oopsfiles
> │   ├─VG03-dropbox   253:5    0     5G  0 lvm   /mnt/dropbox
> │   ├─VG03-distfiles 253:6    0    10G  0 lvm   /usr/portage/distfiles
> │   ├─VG03-gentoo32  253:7    0    15G  0 lvm   /mnt/gentoo32
> │   ├─VG03-xp        253:8    0    40G  0 lvm
> │   └─VG03-test      253:9    0     1G  0 lvm
> └─sda6                 8:6    0    50G  0 part
>   └─md4                9:4    0    50G  0 raid1
> sdb                    8:16   0 931,5G  0 disk
> ├─sdb1                 8:17   0   100M  0 part
> ├─sdb2                 8:18   0  98,8G  0 part
> ├─sdb3                 8:19   0    50G  0 part
> │ └─md4                9:4    0    50G  0 raid1
> ├─sdb4                 8:20   0  12,4G  0 part
> └─sdb6                 8:22   0 595,1G  0 part
>   └─md127              9:127  0 595,1G  0 raid1
>     ├─VG03-music     253:0    0   190G  0 lvm   /mnt/music
>     ├─VG03-platz     253:1    0   200G  0 lvm   /mnt/platz
>     ├─VG03-media     253:2    0    45G  0 lvm   /mnt/media
>     ├─VG03-home      253:3    0    30G  0 lvm
>     ├─VG03-oopsfiles 253:4    0    12G  0 lvm   /mnt/oopsfiles
>     ├─VG03-dropbox   253:5    0     5G  0 lvm   /mnt/dropbox
>     ├─VG03-distfiles 253:6    0    10G  0 lvm   /usr/portage/distfiles
>     ├─VG03-gentoo32  253:7    0    15G  0 lvm   /mnt/gentoo32
>     ├─VG03-xp        253:8    0    40G  0 lvm
>     └─VG03-test      253:9    0     1G  0 lvm
> sdc                    8:32   0  55,9G  0 disk
> ├─sdc1                 8:33   0    25G  0 part  /
> ├─sdc2                 8:34   0     2G  0 part
> └─sdc3                 8:35   0  28,9G  0 part  /home
> sr0                   11:0    1  1024M  0 rom
>
>
>
> This pretty much says it all, right?
>
> 2 hdds sda and sdb
> 1 ssd sdc
>
> root-fs and /home on ssd ...
>
> sda and sdb build two RAID-arrays (rather ugly names and partitions ...
> grown over time):
>
> # cat /proc/mdstat
> Personalities : [raid1]
> md4 : active raid1 sdb3[0] sda6[2]
>       52395904 blocks super 1.2 [2/2] [UU]
>
> md127 : active raid1 sdb6[0] sda3[1]
>       623963072 blocks [2/2] [UU]
>
> unused devices: <none>
>
>
> # pvs
>   PV         VG   Fmt  Attr PSize   PFree
>   /dev/md127 VG03 lvm2 a--  595,05g 47,05g

Sorry I took my time, I was busy.

Well, yours' a complex setup. This is a similar, although simpler, version:

NAME                 MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0                   11:0    1 1024M  0 rom
vda                  253:0    0    5G  0 disk
|-vda1               253:1    0   95M  0 part  /boot
|-vda2               253:2    0  1.9G  0 part  [SWAP]
`-vda3               253:3    0    3G  0 part  /home
vdb                  253:16   0    5G  0 disk
`-vdb1               253:17   0    5G  0 part  /
vdc                  253:32   0    5G  0 disk
`-vdc1               253:33   0    5G  0 part
  `-md127              9:127  0    5G  0 raid1
    |-vg-vol1 (dm-0) 254:0    0    2G  0 lvm   /home/canek/Music
    |-vg-vol2 (dm-1) 254:1    0    2G  0 lvm   /home/canek/Pictures
    `-vg-vol3 (dm-2) 254:2    0 1020M  0 lvm   /home/canek/Videos
vdd                  253:48   0    5G  0 disk
`-vdd1               253:49   0    5G  0 part
  `-md127              9:127  0    5G  0 raid1
    |-vg-vol1 (dm-0) 254:0    0    2G  0 lvm   /home/canek/Music
    |-vg-vol2 (dm-1) 254:1    0    2G  0 lvm   /home/canek/Pictures
    `-vg-vol3 (dm-2) 254:2    0 1020M  0 lvm   /home/canek/Videos

/boot on vda1 as ext2, / (root) on vdb1 as ext4, /home on vda3 as
ext4,  vda2 as swap, and vdc1 and vdd1 as md127 raid1, from which a PV
is created, and from it a VG, and from it vol1, vol2 and vol3 are 3
LVs. The LVs are mounted inside /home which is mounted inside /
(root), to see if it could cause troubles.

Again, everything just works. I did not touch one single configuration
file for LVM2, mdadm, nor almost anything else. Since /boot is outside
of mdraid and/or LVM2, it was even simpler that my last experiment.

What I did:

1. The partitions used for mdraid (vdc1 and vdd1) are of type "Linux
raid autodetect" (fd in fdisk).

2. The raid and LVM2 volumes were created following (more or less)
[1]. I did not mknod nothing, mdadm --create does everything.

3. I did *NOT* used --metadata=0.90 when creating md1.

4. I used mkfs.ext[24] -L *always*. All the mounted partitions/LVs
that I use have labels; even the swap partition is labeled.

5. My fstab:

LABEL=Root  /  ext4  acl,noatime   0 1
LABEL=Boot  /boot  ext2  acl,noatime   0 2
LABEL=Home  /home  ext4  acl,noatime   0 2
LABEL=Music  /home/canek/Music ext4 acl,noatime   0 2
LABEL=Pictures  /home/canek/Pictures ext4 acl,noatime   0 2
LABEL=Videos  /home/canek/Videos ext4 acl,noatime   0 2
LABEL=Swap  none  swap  sw    0 0
shm   /dev/shm tmpfs  nodev,nosuid,noexec  0 0
tmpfs   /tmp  tmpfs  defaults,nosuid,size=100% 0 0

Music/Pictures/Videos are the LVs inside the raid; they have labels. I
do not put anything special in fstab, I just put the label.

6. In /etc/portage/make.conf:

DRACUT_MODULES="caps systemd lvm mdraid"

7. USE flags:

sys-kernel/dracut device-mapper
sys-boot/grub device-mapper mount

8. In /etc/dracut.conf:

add_dracutmodules+="lvm mdraid systemd"

I didn't modified lvmconf nor mdadmconf, so both are by default set to "no".

9. In /etc/default/grub:

GRUB_PRELOAD_MODULES="mdraid1x"

10. I compiled the kernel with the options suggested by [1] and [2].
make, make install, make modules_install

11. I generated my initramfs with: dracut -f -H /boot/initrd-3.10.7 3.10.7

12. GRUB2: grub2-mkconfig -o /boot/grub2/grub.cfg

13. I repeat, I didn't touched /etc/lvm/lvm.conf, nor /etc/mdadm.conf.

Versions installed:

sys-apps/systemd-204
sys-boot/grub-2.00_p5107-r1
sys-fs/lvm2-2.02.100-r2
sys-fs/mdadm-3.3
sys-kernel/dracut-033 (compile with USE="optimization")
sys-kernel/vanilla-sources-3.11.1

Everything just works. I didn't needed to search for service files for
LVM2 nor for mdraid; the ones installed by the package (and the
generated one using generators) just worked.

I masked the following unit files, since they are not installed in my
Virtual Machine:

cryptsetup.target
fcoe.service
iscsi.service

I also masked lvm2-activation-net.service, since it gave me a failure
(lvm2-activation-net.service: main process exited, code=exited,
status=5/NOTINSSTALLED); however everything works and the LVs get
mounted automatically.

Since /usr is not even in its own partition, the initramfs is not
necessary; I can boot just fine without it.

lvm ~ # systemctl status home-canek-Music.mount
home-canek-Music.mount - /home/canek/Music
   Loaded: loaded (/etc/fstab)
   Active: active (mounted) since Fri 2013-09-20 03:40:41 CDT; 1min 38s ago
    Where: /home/canek/Music
     What: /dev/mapper/vg-vol1
  Process: 1271 ExecMount=/bin/mount /dev/disk/by-label/Music
/home/canek/Music -t ext4 -o acl,noatime (code=exited,
status=0/SUCCESS)

Sep 20 03:40:41 lvm systemd[1]: Mounting /home/canek/Music...
Sep 20 03:40:41 lvm systemd[1]: Mounted /home/canek/Music.

lvm ~ # systemd-analyze blame
           949ms systemd-udev-trigger.service
           161ms systemd-fsck@dev-disk-by\x2dlabel-Pictures.service
           142ms systemd-fsck@dev-disk-by\x2dlabel-Home.service
           135ms systemd-fsck-root.service
           131ms systemd-fsck@dev-disk-by\x2dlabel-Boot.service
           105ms sshd.service
           100ms lvm2-activation-early.service
            96ms systemd-logind.service
            95ms systemd-fsck@dev-disk-by\x2dlabel-Music.service
            95ms sys-kernel-debug.mount
            95ms dhcpcd@enp0s3.service
            92ms systemd-vconsole-setup.service
            90ms systemd-fsck@dev-disk-by\x2dlabel-Videos.service
            90ms dev-hugepages.mount
            88ms systemd-tmpfiles-setup-dev.service
            86ms dev-mqueue.mount
            73ms lvm2-activation.service
            66ms mdadm.service
            55ms home-canek-Music.mount
            52ms systemd-sysctl.service
            42ms boot.mount
            39ms systemd-remount-fs.service
            39ms systemd-user-sessions.service
            38ms systemd-tmpfiles-setup.service
            32ms home.mount
            31ms systemd-journal-flush.service
            29ms systemd-udev-settle.service
            27ms home-canek-Pictures.mount
            24ms systemd-random-seed-load.service
            22ms home-canek-Videos.mount
            21ms tmp.mount
            18ms systemd-udevd.service
            12ms dev-disk-by\x2dlabel-Swap.swap

Again, I don't see what the problem is. Now I have installed Gentoo in
a virtual machine with everything using LVM2 (even /boot), and in
another using LVM2+mdraid. I suppose I just need to try
LVM2+mdraid+LUKS.

Regards.

[1] http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml
[2] https://wiki.gentoo.org/wiki/LVM
-- 
Canek Peláez Valdés
Posgrado en Ciencia e Ingeniería de la Computación
Universidad Nacional Autónoma de México

Re: LVM2+mdraid+systemd (was Re: [gentoo-user] systemd and lvm)

[Stefan G. Weichinger]

Am 20.09.2013 10:46, schrieb Canek Peláez Valdés:

> Sorry I took my time, I was busy.
> 
> Well, yours' a complex setup. This is a similar, although simpler, version:

At first: thank your for the extended test setup you did and described
... I will dig through it as soon as I find time ... I am quite busy
these days as well.

Thanks, regards, Stefan!

Re: LVM2+mdraid+systemd (was Re: [gentoo-user] systemd and lvm)

[Stefan G. Weichinger]

I haven't yet worked through all your suggestions/descriptions.

Edited USE-flags and dracut-modules, worked around bug

https://bugs.gentoo.org/show_bug.cgi?id=485202

and rebuilt kernel and initrd.

Didn't activate LVs ...

Now I edited fstab:

I had the option "systemd.automount" enabled, like in

/dev/mapper/VG03-media /mnt/media		ext4
noatime,user_xattr,comment=systemd.automount 0 2

The/my idea behind that: the boot-process should not need to wait for
the LVs activated/fscked/mounted ... and my root-fs and /home are both
on the SSD anyway (not LVM-based).

I removed that option and after the next boot the LVs were activated and
mounted (though the booting was a bit slower, as expected).

OK. I send this message now and test another few reboots.

Thanks, Stefan

Re: LVM2+mdraid+systemd (was Re: [gentoo-user] systemd and lvm)

[Canek Peláez Valdés]

On Fri, Sep 20, 2013 at 11:17 AM, Stefan G. Weichinger <lists@xunil.at> wrote:
>
> I haven't yet worked through all your suggestions/descriptions.
>
> Edited USE-flags and dracut-modules, worked around bug
>
> https://bugs.gentoo.org/show_bug.cgi?id=485202
>
> and rebuilt kernel and initrd.
>
> Didn't activate LVs ...
>
> Now I edited fstab:
>
> I had the option "systemd.automount" enabled, like in
>
> /dev/mapper/VG03-media /mnt/media               ext4
> noatime,user_xattr,comment=systemd.automount 0 2
>
> The/my idea behind that: the boot-process should not need to wait for
> the LVs activated/fscked/mounted ... and my root-fs and /home are both
> on the SSD anyway (not LVM-based).
>
> I removed that option and after the next boot the LVs were activated and
> mounted (though the booting was a bit slower, as expected).
>
> OK. I send this message now and test another few reboots.

Forgot to mention it: I also enabled mdadm.service.

Regards.
-- 
Canek Peláez Valdés
Posgrado en Ciencia e Ingeniería de la Computación
Universidad Nacional Autónoma de México

Re: LVM2+mdraid+systemd (was Re: [gentoo-user] systemd and lvm)

[Stefan G. Weichinger]

Am 20.09.2013 18:50, schrieb Canek Peláez Valdés:

>> OK. I send this message now and test another few reboots.
> 
> Forgot to mention it: I also enabled mdadm.service.

That service is enabled here as well and running fine.



# systemctl status lvm2-activation-net.service
lvm2-activation-net.service - Activation of LVM2 logical volumes
   Loaded: loaded (/etc/lvm/lvm.conf)
   Active: inactive (dead) since Fr 2013-09-20 20:57:15 CEST
     Docs: man:lvm(8)
           man:vgchange(8)
  Process: 580 ExecStart=/sbin/lvm vgchange -aay --sysinit (code=exited,
status=0/SUCCESS)
  Process: 366 ExecStartPre=/usr/bin/udevadm settle (code=exited,
status=0/SUCCESS)
 Main PID: 580 (code=exited, status=0/SUCCESS)

Sep 20 20:57:13 hiro.oops.intern lvm[580]: 10 logical volume(s) in
volume group "VG03" now active
Sep 20 20:57:15 hiro.oops.intern systemd[1]: Started Activation of LVM2
logical volumes.


nice ... but not at every boot ... one time they are activated, one time
not.

*sigh*

Thanks for all your patience ... I could live with that lvm.service ;-)

Considering to convert the mdadm-RAID to metadata 1.2 (wouldn't hurt
anyway, right?)

Stefan