[gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Kenton Groombridge]

[-- Attachment #1: Type: text/plain, Size: 2318 bytes --]

Hi everyone,

I noticed that there are many systemd units which are shipped by various
packages which could be hardened, some further than they are currently and some
that could use some hardening in general.

For those who are unaware, systemd units support many options which can be used
to restrict privileges of the processes run by the service. Some of these
options include things like making specified paths inaccessible or read-only,
setting the no_new_privs flag, protecting kernel sysctls, preventing the loading
of kernel modules, applying a seccomp filter to restrict syscalls, and more. I
frequently reference systemd.exec(5)[1] and this page[2] for reference.

Many of these options are fairly easy to apply from a user perspective - a user
only needs to harden something like miniflux.service by overriding/settings via
'systemctl edit miniflux.service' (or manually editing
/etc/systemd/system/miniflux.service.d/override.conf). But, I want to propose an
initiative to set some of these options by default for systemd units shipped in
::gentoo.

Care must be taken though, as some of these options may end up breaking some
functionality that could be expected by users. An example of this may be if the
package maintainer made the root filesystem read-only for a service except for
its private /var/lib, but a user was using an entirely different directory for
the service's read-writable data. Something like this may need to be
communicated via post-installation messages or simply left out by default,
depending on the circumstances. On the other hand, there are many options like
restricting syscalls via SystemCallFilter=@system-service or restricting
privilege escalation via NoNewPrivileges=true that I think are generally safe to
apply, but each service is different and needs to be handled and tested
accordingly.

As for getting units updated, I think a good place to start would be to create a
new tracker bug for identifying packages providing systemd units that could be
improved in this regard, and each bug filed could include recommendations for
some of the more common options like ProtectSystem=, ProtectHome=,
ProtectDevices=, and others.

What do you think?

[1] https://www.freedesktop.org/software/systemd/man/systemd.exec.html
[2] https://docs.arbitrary.ch/security/systemd.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 2622 bytes --]

On Mon, Aug 22, 2022 at 02:10:47PM -0400, Kenton Groombridge wrote:
> Hi everyone,
> 
> I noticed that there are many systemd units which are shipped by various
> packages which could be hardened, some further than they are currently and some
> that could use some hardening in general.
> 
> For those who are unaware, systemd units support many options which can be used
> to restrict privileges of the processes run by the service. Some of these
> options include things like making specified paths inaccessible or read-only,
> setting the no_new_privs flag, protecting kernel sysctls, preventing the loading
> of kernel modules, applying a seccomp filter to restrict syscalls, and more. I
> frequently reference systemd.exec(5)[1] and this page[2] for reference.
> 
> Many of these options are fairly easy to apply from a user perspective - a user
> only needs to harden something like miniflux.service by overriding/settings via
> 'systemctl edit miniflux.service' (or manually editing
> /etc/systemd/system/miniflux.service.d/override.conf). But, I want to propose an
> initiative to set some of these options by default for systemd units shipped in
> ::gentoo.
> 
> Care must be taken though, as some of these options may end up breaking some
> functionality that could be expected by users. An example of this may be if the
> package maintainer made the root filesystem read-only for a service except for
> its private /var/lib, but a user was using an entirely different directory for
> the service's read-writable data. Something like this may need to be
> communicated via post-installation messages or simply left out by default,
> depending on the circumstances. On the other hand, there are many options like
> restricting syscalls via SystemCallFilter=@system-service or restricting
> privilege escalation via NoNewPrivileges=true that I think are generally safe to
> apply, but each service is different and needs to be handled and tested
> accordingly.
> 
> As for getting units updated, I think a good place to start would be to create a
> new tracker bug for identifying packages providing systemd units that could be
> improved in this regard, and each bug filed could include recommendations for
> some of the more common options like ProtectSystem=, ProtectHome=,
> ProtectDevices=, and others.
> 
> What do you think?

This would be a great improvement! systemd users can also see some of
this via `systemd-analyze security`.

> [1] https://www.freedesktop.org/software/systemd/man/systemd.exec.html
> [2] https://docs.arbitrary.ch/security/systemd.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]

> What do you think?

I support it! These options go criminally underused, and are extremely 
useful for what you "pay" to use them (and, as you mentioned, they are 
user overridable, so it's not possible to make some security choices 
that are irreparable).

It'd also be nice to promote them to more upstreams ;)

-- 
Arsen Arsenović

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Mike Gilbert]

On Mon, Aug 22, 2022 at 2:10 PM Kenton Groombridge <concord@gentoo.org> wrote:
> What do you think?

I am concerned that people will start mass filing bugs with
suggestions without fully understanding them or without testing them
thoroughly. Please don't do that.

Also, ideally we would not need to provide systemd units at a distro
level, and they would instead be provided by upstream. I certainly
don't want to start installing distro-customized units where upstream
already provides unit files.

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 858 bytes --]

On 22/08/2022 20.10, Kenton Groombridge wrote:
> Hi everyone,
> 
> I noticed that there are many systemd units which are shipped by various
> packages which could be hardened, some further than they are currently and some
> that could use some hardening in general.

Yes, please. Nevertheless, as others have already pointed out, this is 
an upstream issue and should be treated as such. That is, please feel 
encouraged and encourage others to submit patches upstream that adds 
hardening measurements to their systemd units. I usually find

https://ruderich.org/simon/notes/systemd-service-hardening

a good starting point when I want to harden a service.

By addressing this upstream, everyone benefits, and potential issues 
caused by the hardening measurements are fixed faster (as they affect a 
more extensive user base).

- Flow

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 25757 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Kenton Groombridge]

[-- Attachment #1: Type: text/plain, Size: 2016 bytes --]

On 22/08/22 03:42PM, Mike Gilbert wrote:
> On Mon, Aug 22, 2022 at 2:10 PM Kenton Groombridge <concord@gentoo.org> wrote:
> > What do you think?
> 
> I am concerned that people will start mass filing bugs with
> suggestions without fully understanding them or without testing them
> thoroughly. Please don't do that.
> 

I had thought of this potentially being a problem as well. I think with this in
mind perhaps it would be better to start with creating some documentation on
these systemd service options on the wiki, with notes geared for both users and
developers and when these options would/would not be a good idea to enable from
both perspectives. That way we can at least have some solid reference material
when addressing such bugs and providing guidance to developers to improve
systemd units in their packages.

> Also, ideally we would not need to provide systemd units at a distro
> level, and they would instead be provided by upstream. I certainly
> don't want to start installing distro-customized units where upstream
> already provides unit files.
> 

I agree! Unfortunately I know of a very small amount of packages where hardened
systemd unit files are available but are not supported by upstream. One such
upstream includes a hardened systemd unit in their contrib/, but nevertheless it
is not installed by default for fear of breaking users' configurations.

I think the best way to address this is to have packages ship unit override
files instead of unit files themselves which enable these options. For example,
instead of Gentoo shipping a modified miniflux.service unit file, we can instead
install a file to /etc/system/miniflux.service.d/00gentoo.conf using the
existing systemd_install_serviced helper in systemd.eclass which enables these
options. Then, over time we can merge the modifications we make upstream if
upstream wants them. If not, we can continue to ship this override file without
changing how the original unit file gets installed.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Florian Schmaus]

On 25/08/2022 15.25, Kenton Groombridge wrote:
> I think the best way to address this is to have packages ship unit override
> files instead of unit files themselves which enable these options. For example,
> instead of Gentoo shipping a modified miniflux.service unit file, we can instead
> install a file to /etc/system/miniflux.service.d/00gentoo.conf using the
> existing systemd_install_serviced helper in systemd.eclass which enables these
> options.

Wouldn't the proper place for overrides installed by a distributions 
package manager be

/usr/lib/systemd/system/miniflux.service.d/gentoo.conf


- Flow

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Kenton Groombridge]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On 22/08/25 04:06PM, Florian Schmaus wrote:
> Wouldn't the proper place for overrides installed by a distributions package
> manager be
> 
> /usr/lib/systemd/system/miniflux.service.d/gentoo.conf
> 

Yes... I was wondering that too. Currently systemd_install_serviced installs to
/etc/systemd/system instead of /usr/lib/systemd/system appears to be wrong.
systemd's own documentation says distributions should be installing contents to
/usr/lib/systemd/system while /etc/systemd/system is intended for "System units
created by the administrator" (users).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Michał Górny]

On Thu, 2022-08-25 at 16:06 +0200, Florian Schmaus wrote:
> On 25/08/2022 15.25, Kenton Groombridge wrote:
> > I think the best way to address this is to have packages ship unit override
> > files instead of unit files themselves which enable these options. For example,
> > instead of Gentoo shipping a modified miniflux.service unit file, we can instead
> > install a file to /etc/system/miniflux.service.d/00gentoo.conf using the
> > existing systemd_install_serviced helper in systemd.eclass which enables these
> > options.
> 
> Wouldn't the proper place for overrides installed by a distributions 
> package manager be
> 
> /usr/lib/systemd/system/miniflux.service.d/gentoo.conf
> 

These files are meant to be modifiable by the sysadmin, so they don't
belong in /usr.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Mike Gilbert]

On Thu, Aug 25, 2022 at 10:29 AM Kenton Groombridge <concord@gentoo.org> wrote:
>
> On 22/08/25 04:06PM, Florian Schmaus wrote:
> > Wouldn't the proper place for overrides installed by a distributions package
> > manager be
> >
> > /usr/lib/systemd/system/miniflux.service.d/gentoo.conf
> >
>
> Yes... I was wondering that too. Currently systemd_install_serviced installs to
> /etc/systemd/system instead of /usr/lib/systemd/system appears to be wrong.
> systemd's own documentation says distributions should be installing contents to
> /usr/lib/systemd/system while /etc/systemd/system is intended for "System units
> created by the administrator" (users).

The existing function is meant to install "placeholder" drop-ins that
would be populated by the sysadmin.

We could introduce a new function to install distro-specific overrides
in [/usr]/lib/systemd/system.

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Kenton Groombridge]

[-- Attachment #1: Type: text/plain, Size: 276 bytes --]

On 22/08/25 01:04PM, Mike Gilbert wrote:
> We could introduce a new function to install distro-specific overrides
> in [/usr]/lib/systemd/system.
> 

I think that's a good idea. systemd_{new,do}serviceconf maybe?

As I understand it these should go to /usr/lib/[...].

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Mike Gilbert]

On Thu, Aug 25, 2022 at 1:41 PM Kenton Groombridge <concord@gentoo.org> wrote:
>
> On 22/08/25 01:04PM, Mike Gilbert wrote:
> > We could introduce a new function to install distro-specific overrides
> > in [/usr]/lib/systemd/system.
> >
>
> I think that's a good idea. systemd_{new,do}serviceconf maybe?
>
> As I understand it these should go to /usr/lib/[...].

The correct path to use depends on the type of unit and the
"split-usr" USE flag.

With split-usr enabled, overrides for system service units go in
/lib/systemd/system/foo.service.d.
With split-usr disabled, overrides for system services units go in
/usr/lib/systemd/system/foo.service.d.
Overrides for user service units would always go in
/usr/lib/systemd/user/foo.service.d.

We will be phasing out split-usr later this year due to pressure from
systemd upstream to stop supporting it.

Anyway, there are already functions to get the correct path based on
pkg-config in the systemd.eclass.

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Florian Schmaus]

On 25/08/2022 17.03, Michał Górny wrote:
> On Thu, 2022-08-25 at 16:06 +0200, Florian Schmaus wrote:
>> On 25/08/2022 15.25, Kenton Groombridge wrote:
>>> I think the best way to address this is to have packages ship unit override
>>> files instead of unit files themselves which enable these options. For example,
>>> instead of Gentoo shipping a modified miniflux.service unit file, we can instead
>>> install a file to /etc/system/miniflux.service.d/00gentoo.conf using the
>>> existing systemd_install_serviced helper in systemd.eclass which enables these
>>> options.
>>
>> Wouldn't the proper place for overrides installed by a distributions
>> package manager be
>>
>> /usr/lib/systemd/system/miniflux.service.d/gentoo.conf
> 
> These files are meant to be modifiable by the sysadmin, so they don't
> belong in /usr.

While then can not be modified, settings made in /usr/lib/systemd/system 
can be overridden by the sysadmin by placing a file in /etc/systemd/system.

I am not aware of a reason why a package manger should install systemd 
configuration files under /etc.

- Flow

Re: [gentoo-dev] [RFC] Encouraging using hardening options in systemd units

[Rich Freeman]

On Fri, Aug 26, 2022 at 4:57 AM Florian Schmaus <flow@gentoo.org> wrote:
>
> While then can not be modified, settings made in /usr/lib/systemd/system
> can be overridden by the sysadmin by placing a file in /etc/systemd/system.
>
> I am not aware of a reason why a package manger should install systemd
> configuration files under /etc.
>

I'd say the biggest use case would be settings that typically require
modification that are needed at service launch time.  Think of
packages that stick stuff in /etc/conf.d for non-systemd settings.

An example of this can be found in sys-devel/distcc/files/distccd.service.conf

Sure, you could stick that in /usr, and then the user could copy it to
/etc (or use systemctl edit).  However, since it is something that is
basically intended to be edited you can argue that it makes more sense
to just stick it in /etc.  This also means that if a new setting gets
added the user will be made aware via config protection.  For a
drop-in that changes in /usr the user would receive no notice of this,
and the new settings would get merged with theirs or ignored depending
on how it was done.

For a distro override that wouldn't typically be modified, like
integrating one service with another that only service the local host,
then maybe /usr would make more sense.

Using systemctl edit is also a little awkward due to the way it
presents you with the original config at the bottom and you have to
copy the stuff you want to modify to the top.  I assume it lets you
copy comments as well, but it is a bit like opening up two editors and
copying from a skeleton file to a new file, vs just editing one in
place.  Well, except you aren't working between two files but editing
at two places in one file, so if your editor doesn't handle split
screen well you will scroll around a lot if the file is large.

I don't have very strong feelings about this either way, but I can
definitely see why things were done the way they're done.  I'm not
sure if there are any examples of fairly large drop-ins in /etc that
we install but I'd probably want to look at one before changing the
approach to see if it makes sense.

-- 
Rich