[gentoo-user] systemd/dracut: Correctly boot using flash drive with LUKS key?

[gentoo-user] systemd/dracut: Correctly boot using flash drive with LUKS key?

[Andrew Udvare]

[-- Attachment #1.1: Type: text/plain, Size: 1316 bytes --]

I have a set up like this to boot:

GRUB - access non-encrypted EFI partition to find kernel, initramfs, etc
initramfs has enough on it to boot minus the encryption key
(Removable) storage must be inserted (UUID matching) to:
- mount to expected point
- use key to decrypt root partition (systemd job in the first systemd
launched within initramfs)

This set up works, but it has one annoying thing coming from systemd
after initramfs is done.

When I take out the flash drive after booting up, I continuously get a
message that the flash drive wants to be mounted again.

Nov 16 00:01:52 limelight systemd[1]:
dev-disk-by\x2duuid-BD3B\x2d03BD.device: Job
dev-disk-by\x2duuid-BD3B\x2d03BD.device/start timed out.
Nov 16 00:01:52 limelight systemd[1]: Timed out waiting for device
Silicon-Power16G CHUAN.
Nov 16 00:01:52 limelight systemd[1]: Dependency failed for /mnt/chuan.
Nov 16 00:01:52 limelight systemd[1]: mnt-chuan.mount: Job
mnt-chuan.mount/start failed with result 'dependency'.
Nov 16 00:01:52 limelight systemd[1]:
dev-disk-by\x2duuid-BD3B\x2d03BD.device: Job
dev-disk-by\x2duuid-BD3B\x2d03BD.device/start failed with result 'timeout'.

I think this is because of the entry in /etc/fstab.

Anyone have a similar issue? And how can I fix this in Dracut?

Thanks
Andrew


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] systemd/dracut: Correctly boot using flash drive with LUKS key?

[Rich Freeman]

On Fri, Nov 16, 2018 at 12:11 AM Andrew Udvare <audvare@gmail.com> wrote:
>
> - use key to decrypt root partition (systemd job in the first systemd
> launched within initramfs)
> ...
> Nov 16 00:01:52 limelight systemd[1]: Dependency failed for /mnt/chuan.
> Nov 16 00:01:52 limelight systemd[1]: mnt-chuan.mount: Job
> mnt-chuan.mount/start failed with result 'dependency'.

So, I haven't messed around that much with systemd and removable
media, but a few things I'd start by looking at:

systemctl show mnt-chuan.mount
systemctl list-dependencies --reverse --all mnt-chuan.mount

I'd have to read up the manpages on how automounting works and what
the appropriate flags are in fstab to trigger that.  Obviously that is
someplace else to check.

Clearly you have things set up so that the usb drive is mounted before
your script goes and reads the key off of it.  If you're doing that in
a service and have a dependency on the usb drive (or system somehow
figured out that there should be a dependency on it), and that service
is left in a running state after the decryption is done, that could be
causing issues as systemd thinks that drive is still needed and is
desperately trying to get it back.

With systemd mountpoints are units, and can be started/stopped like
any service and can be dependencies of services.  This is usually a
good thing, but services tend to be persistent and removable drives
tend not to be, so you probably need to use some care when mixing the
two.  Note that due to generators system will create mount units
automatically from fstab and other sources.  Usually traditional
commands like mount/umount and so on just do the right thing, but if
you've inadvertently enabled some kind of automounting logic or
systemd believes there is a dependency with another unit then you
might be fighting the system.

Here is another route you could take.  When you want to remove that
usb drive, instead of just pulling it out or manually unmounting it,
try running:
systemctl stop mnt-chuan.mount
That is asking systemd to nicely unmount it.  If systemd doesn't want
to unmount it then it will nicely tell you take a hike, but perhaps
the error output will give you some clues as to why.

-- 
Rich

Re: [gentoo-user] systemd/dracut: Correctly boot using flash drive with LUKS key?

[Andrew Udvare]

> On 2018-11-16, at 08:16, Rich Freeman <rich0@gentoo.org> wrote:
> 
> On Fri, Nov 16, 2018 at 12:11 AM Andrew Udvare <audvare@gmail.com> wrote:
>> 
>> - use key to decrypt root partition (systemd job in the first systemd
>> launched within initramfs)
>> ...
>> Nov 16 00:01:52 limelight systemd[1]: Dependency failed for /mnt/chuan.
>> Nov 16 00:01:52 limelight systemd[1]: mnt-chuan.mount: Job
>> mnt-chuan.mount/start failed with result 'dependency'.
> 
> So, I haven't messed around that much with systemd and removable
> media, but a few things I'd start by looking at:
> 
> systemctl show mnt-chuan.mount
> systemctl list-dependencies --reverse --all mnt-chuan.mount

So this output seems to make sense, and running systemctl stop on the mount gave me similar messages about dependencies. I"m not sure there is a way around this, as the dependency tree always relies on that mount before systemd-cryptsetup@root can work. If unmounted it will be flagged as in invalid state.

I am not sure if there is a way to move the systemd-cryptsetup@home.service up the dependency tree once it's working, which would then remove the mnt-chuan.mount dependency.

Tree from the device:

dev-disk-by\x2...device
● └─mnt-chuan.mount
    │ └─...
●   ├─systemd-cryptsetup@home.service
●   │ ├─dev-disk-by\x2duuid-....device
    │ │   └─...
●   │ ├─dev-mapper-home.device
●   │ └─cryptsetup.target
●   │   └─sysinit.target

systemd-cryptsetup@home.service is red.

-- 
Andrew Udvare

Re: [gentoo-user] systemd/dracut: Correctly boot using flash drive with LUKS key?

[Rich Freeman]

On Fri, Nov 16, 2018 at 12:15 PM Andrew Udvare <audvare@gmail.com> wrote:
>
> I am not sure if there is a way to move the systemd-cryptsetup@home.service up the dependency tree once it's working, which would then remove the mnt-chuan.mount dependency.
>

Ok, I did a bit more reading.  You're using the cryptsetup generator
most likely.  It sets up units to be oneshot+remainafterexit, which
means they're "active" whenever the LUKS device is mounted (without
any processes - but they show as active so that you can stop them and
unmount the device).  It sets the RequiresMountsFor parameter for the
device the key file is contained on, which makes that mount service a
Required dependency.  That means that it can't be unmounted while the
cryptsetup device is in use, and in theory attempting to unmount the
key file should make systemd attempt to unmount the cryptsetup device
(though busy filesystems could interfere with that).

I'm not sure if there is a cleaner solution for this without adding
some kind of new dependency type to systemd.  You really do want that
mountpoint to be considered "Required" when the service starts as that
file is essential, but you don't need it to stick around after the
unit starts.  Of course, you don't want it to automatically stop right
away either (that would create a race condition between unmounting the
key and reading it).

This is built-in systemd behavior.  I wouldn't be surprised if the
folks on the #systemd IRC channel would be interested in talking about
this use case and ways to improve it.  The fact that they anticipate
the key being on a removable mount makes me think it is something they
want to support, but it would clearly be more secure to not leave that
file mounted all the time.

I'd encourage you to ask about it on IRC.  If that isn't something you
want to do I might inquire about it myself, but I don't want to steal
your thunder.

--
Rich

Re: [gentoo-user] systemd/dracut: Correctly boot using flash drive with LUKS key?

[Andrew Udvare]

[-- Attachment #1.1: Type: text/plain, Size: 1943 bytes --]

On 16/11/2018 14:43, Rich Freeman wrote:
> On Fri, Nov 16, 2018 at 12:15 PM Andrew Udvare <audvare@gmail.com> wrote:
>>
>> I am not sure if there is a way to move the systemd-cryptsetup@home.service up the dependency tree once it's working, which would then remove the mnt-chuan.mount dependency.
>>
> 
> Ok, I did a bit more reading.  You're using the cryptsetup generator
> most likely.  It sets up units to be oneshot+remainafterexit, which
> means they're "active" whenever the LUKS device is mounted (without
> any processes - but they show as active so that you can stop them and
> unmount the device).  It sets the RequiresMountsFor parameter for the
> device the key file is contained on, which makes that mount service a
> Required dependency.  That means that it can't be unmounted while the
> cryptsetup device is in use, and in theory attempting to unmount the
> key file should make systemd attempt to unmount the cryptsetup device
> (though busy filesystems could interfere with that).

So it is a bit strange that /mnt/chuan was considered a dependency just
because of mention in /etc/crypttab. However I found out that the reason
has something to do with the /mnt/chuan entry in /etc/fstab in my real
root, and this is not a necessary line (it is the only entry in the
initrd fstab). I removed the line and now the dependency is still show
with list-dependencies, but it is white instead of red. My system is
still shown as running rather than degraded.

Removing the line from /etc/fstab only partially solves the problem, as
it's not explained what happens with the USB drive once the root is
switched because after that it's not shown to be mounted. I am pretty
sure it's not safely unmounted before the switch, which leaves it in a
strange state requiring fsck. Don't know the best way around this other
than wait till systemd supports the keyscript option in /etc/crypttab.

-- 
Andrew


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]