[gentoo-user] VRFs / Jails / Containers

[gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

Does Gentoo have any support for VRFs or (chroot) Jails or Containers 
without going down the Docker (et al) path?

I'm wanting to do some things with a Gentoo router that is trivial to do 
with network namespaces via manual commands ~> scripts.  But that's far 
from standard Gentoo init script based system.  And I'd like something 
more Gentoo standards based.

Does Gentoo have or support anything like this natively?  Or am I 
getting into territory where I'm rolling my own?

Re: [gentoo-user] VRFs / Jails / Containers

[Bill Kenworthy]

On 3/2/19 10:32 am, Grant Taylor wrote:
> Does Gentoo have any support for VRFs or (chroot) Jails or Containers
> without going down the Docker (et al) path?
>
> I'm wanting to do some things with a Gentoo router that is trivial to
> do with network namespaces via manual commands ~> scripts.  But that's
> far from standard Gentoo init script based system.  And I'd like
> something more Gentoo standards based.
>
> Does Gentoo have or support anything like this natively?  Or am I
> getting into territory where I'm rolling my own


LXC containers ??


BillK

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/2/19 7:36 PM, Bill Kenworthy wrote:
> LXC containers ??

Maybe.

I just feel like that's more heavy weight than I want.

I'm functionally running a series of ip commands to configure networking 
in a special way.

Maybe I should look into what it takes to extend netifrc to support what 
I want.  I sort of think that VRF could model off of bonding and / or 
bridge and / or VLAN devices.  At least in the master / slave aspect.

I'm sure that veth will be a new concept, but it may be able to model 
after a tunnel interface.

It would be really nice to have network namespace support.  But I don't 
see anything that could be modeled off of.

Re: [gentoo-user] VRFs / Jails / Containers

[Michael Jones]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

systemd-nspawn is also an option, but I don't think that'll work with
OpenRC.

On Sat, Feb 2, 2019 at 9:56 PM Grant Taylor <
gtaylor@gentoo.tnetconsulting.net> wrote:

> On 2/2/19 7:36 PM, Bill Kenworthy wrote:
> > LXC containers ??
>
> Maybe.
>
> I just feel like that's more heavy weight than I want.
>
> I'm functionally running a series of ip commands to configure networking
> in a special way.
>
> Maybe I should look into what it takes to extend netifrc to support what
> I want.  I sort of think that VRF could model off of bonding and / or
> bridge and / or VLAN devices.  At least in the master / slave aspect.
>
> I'm sure that veth will be a new concept, but it may be able to model
> after a tunnel interface.
>
> It would be really nice to have network namespace support.  But I don't
> see anything that could be modeled off of.
>
>

[-- Attachment #2: Type: text/html, Size: 1240 bytes --]

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/2/19 9:39 PM, Michael Jones wrote:
> systemd-nspawn is also an option, but I don't think that'll work with 
> OpenRC.

Ya....  I moved (back to) Gentoo to get away from systemd.  I'm not 
going to voluntarily opt to use it, or any of it's children.  That's 
/my/ opinion.  I know others opinions differ.

Thank you for the information all the same.

Re: [gentoo-user] VRFs / Jails / Containers

[Bill Kenworthy]

On 3/2/19 12:52 pm, Grant Taylor wrote:
> On 2/2/19 9:39 PM, Michael Jones wrote:
>> systemd-nspawn is also an option, but I don't think that'll work with
>> OpenRC.
>
> Ya....  I moved (back to) Gentoo to get away from systemd.  I'm not
> going to voluntarily opt to use it, or any of it's children.  That's
> /my/ opinion.  I know others opinions differ.
>
> Thank you for the information all the same.
>
I am unclear on what you are trying to do.  I find the gentoo scripts
good for the simple case but a complex case almost always needs extra
help.  If its networking, could something like shorewall help?

BillK

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/2/19 11:09 PM, Bill Kenworthy wrote:
> I am unclear on what you are trying to do.

See my reply to Rich's message for a description.

> I find the gentoo scripts good for the simple case but a complex case 
> almost always needs extra help.

Yep.

I was hoping that there was something that I was unaware of or could 
extend to do what I want to do.

> If its networking, could something like shorewall help?

No, I don't think that Shorewall or a similar firewall config management 
system will help.

I also find those systems annoying.  Sure, they have their benefits. 
But why do I need them when I should be able to do the same thing on a 
stock Gentoo (or other) Linux system?  After all they are using the same 
kernel.  (Maybe a different version or config there of.)

I will occasionally look at those solutions and treat them like themed 
Lego sets.  I build them, look at them, analyze them, and pull out the 
distinct Lego bricks that I want to use in my own system.  }:-)

Re: [gentoo-user] VRFs / Jails / Containers

[Rich Freeman]

On Sat, Feb 2, 2019 at 11:52 PM Grant Taylor
<gtaylor@gentoo.tnetconsulting.net> wrote:
>
> On 2/2/19 9:39 PM, Michael Jones wrote:
> > systemd-nspawn is also an option, but I don't think that'll work with
> > OpenRC.
>
> Ya....  I moved (back to) Gentoo to get away from systemd.  I'm not
> going to voluntarily opt to use it, or any of it's children.  That's
> /my/ opinion.  I know others opinions differ.
>

Nothing wrong with that approach.  I use systemd-nspawn to run a bunch
of containers, hosted in Gentoo, and many of which run Gentoo.
However, these all run systemd and I don't believe you can run nspawn
without a systemd host (the guest/container can be anything).  These
are containers running full distros with systemd in my case, not just
single-process containers, in my case.  However, nspawn does support
single-process containers, and that includes with veth, but nspawn
WON'T initialize networking in those containers (ie DHCP/etc), leaving
this up to the guest (it does provide a config file for
systemd-networkd inside the guest if it is in use to autoconfigure
DHCP).

I'm not exactly certain what you're trying to accomplish, but
namespaces are just a kernel system call when it comes down to it (two
of them I think offhand).  Two util-linux programs provide direct
access to them for shell scripts: unshare and nsenter.  If you're just
trying to run a process in a separate namespace so that it can use
veth/etc then you could probably initialize that in a script run from
unshare.  If you don't need more isolation you could run it right from
the host filesystem without a separate mount or process namespace.  Or
you could create a new mount namespace but only modify specific parts
of it like /var/lib or whatever.

People generally equate containers with docker but as you seem to get
you can do a lot with namespaces without basically running completely
independent distros.  Now, I will point out that there are good
reasons for keeping things separate - they may or may not apply to
your application.  If you just want to run a single daemon on 14
different IPs and have each of those daemons see the same filesystem
minus /var/lib and /etc that is something you could certainly do with
namespaces and the only resource cost would be the storage of the
extra /var/lib and /etc directories (they could even use the same
shared libraries in RAM, and indeed the same process image itself I
think).

The only gotcha is that I'm not sure how much of it is already done,
so you may have to roll your own.  If you find generic solutions for
running services in partially-isolated namespaces with network
initialization taken care of for you I'd be very interested in hearing
about it.

-- 
Rich

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/3/19 5:37 AM, Rich Freeman wrote:
> Nothing wrong with that approach.  I use systemd-nspawn to run a bunch 
> of containers, hosted in Gentoo, and many of which run Gentoo.  However, 
> these all run systemd and I don't believe you can run nspawn without a 
> systemd host (the guest/container can be anything).  These are containers 
> running full distros with systemd in my case, not just single-process 
> containers, in my case.  However, nspawn does support single-process 
> containers, and that includes with veth, but nspawn WON'T initialize 
> networking in those containers (ie DHCP/etc), leaving this up to the guest 
> (it does provide a config file for systemd-networkd inside the guest if 
> it is in use to autoconfigure DHCP).

ACK

That makes me think that systemd-nspawn is less of a fit for what I'm 
wanting to do.

> I'm not exactly certain what you're trying to accomplish, but namespaces 
> are just a kernel system call when it comes down to it (two of them I 
> think offhand).  Two util-linux programs provide direct access to them 
> for shell scripts: unshare and nsenter.  If you're just trying to run a 
> process in a separate namespace so that it can use veth/etc then you could 
> probably initialize that in a script run from unshare.  If you don't need 
> more isolation you could run it right from the host filesystem without 
> a separate mount or process namespace.  Or you could create a new mount 
> namespace but only modify specific parts of it like /var/lib or whatever.

That's quite close to what I'm doing.  I'm actually using unshare to 
create a mount / network / UTS namespace (set) and then running some 
commands in them.

The namespaces are functioning as routers.  I have an OvS switch 
connected to the main / default (unnamed) namespace and nine (internal) 
OvS ports, each one in a different namespace.  Thus forming a backbone 
between the ten network namespaces.

Each of the nine network namespaces then has a veth pair that connects 
back to the main network namespace as an L2 interface that VirtualBox 
(et al) can glom onto as necessary.

This way I can easily have nine completely different networks that VMs 
can use.  My main home network has a route to these networks via my 
workstation.  (I'm actually using routing protocols to distribute this.)

So the main use of the network namespaces is as a basic IP router. 
There doesn't /need/ to be any processes running in them.  I do run BIRD 
in the network namespaces for simplicity reasons.  But that's more 
ancillary.

I don't strictly need the mount namespaces for what I'm currently doing. 
  That's left over from when I was running Quagga and /needed/ to alter 
some mounts to run multiple instances of Quagga on the same machine.

I do like the UTS namespace so that each ""router has a different host 
name when I enter it.

Maybe this helps explain /what/ I'm doing.  As for /why/ I'm doing it, 
well because reasons.  Maybe not even good reasons.  But I'm still doing 
it.  ¯\_(ツ)_/¯  I'm happy to discuss this in a private thread if anyone 
is really curious.

> People generally equate containers with docker but as you seem to get 
> you can do a lot with namespaces without basically running completely 
> independent distros.

Yep.  I feel like independent distros, plus heavier weight management 
daemons on top are a LOT more than I want.

As stated, I don't really /need/ to run processes in the containers.  I 
do because it's easy.  The only thing I /need/ is the separate IP stack 
/ configuration.

> Now, I will point out that there are good reasons for keeping things 
> separate - they may or may not apply to your application.  If you just 
> want to run a single daemon on 14 different IPs and have each of those 
> daemons see the same filesystem minus /var/lib and /etc that is something 
> you could certainly do with namespaces and the only resource cost would 
> be the storage of the extra /var/lib and /etc directories (they could 
> even use the same shared libraries in RAM, and indeed the same process 
> image itself I think).

Yep.

> The only gotcha is that I'm not sure how much of it is already done, so 
> you may have to roll your own.  If you find generic solutions for running 
> services in partially-isolated namespaces with network initialization 
> taken care of for you I'd be very interested in hearing about it.

I think there are a LOT of solutions for creating and managing 
containers.  (I'm using the term "container" loosely here.)  The thing 
is that many of them are each their own heavy weight entity.  I have yet 
to find any that integrate well with OS init scripts.

I feel like what I want to do can /almost/ be done with netifrc.  Or 
that netifrc could be extended to do what (I think is) /little/ 
additional work to do it.

I don't know that network namespaces are strictly required.  I've been 
using them for years.  That being said, the current incarnation of 
Virtual Routing and Forwarding (VRF) provided by l3mdev seems to be very 
promising.  I expect that I could make VRF (l3mdev) do what I wanted to 
do too.  At least the part that I /need/.  I'm not sure how to launch 
processes associated with the VRF (l3mdev).  I'm confident it's 
possible, but I've not done it.

But, even VRF (l3mdev) is not supported by netifrc.  I feel like the 
Policy Based Routing (PBR) is even a kludge and largely consists of 
(parts of) the ip / tc commands being put into the /etc/conf.d/net file.

I feel like bridging / bonding / VLANs have better support than PBR 
does.  All of which are way better supported than VRF (l3mdev) which is 
better supported than network namespaces.

Though, I'm not really surprised.  All of the init scripts that I've 
seen seem to be designed around the premise of a singular system and 
have no knowledge that there might be other (virtual) systems.  What 
little I know about Docker is that even it's configuration is singular 
system in nature and still only applies to the instance that it's 
working on.  I've not seen any OS init scripts that are aware of the 
fact that they might be working on other systems.  I think the closest 
I've seen is FreeBSD jails.  But even that is separate init scripts, 
which are again somewhat focused on the jail.

I need to do some thinking about /what/ /specifically/ I want to do 
before I start thinking about /how/ to go about doing it.

That being said, I think it would be really nice to have various 
interfaces tagged with what NetNS they belong to and use the same 
net.$interface type init scripts for them.

Re: [gentoo-user] VRFs / Jails / Containers

[Michael Orlitzky]

On 2/2/19 10:56 PM, Grant Taylor wrote:
> On 2/2/19 7:36 PM, Bill Kenworthy wrote:
>> LXC containers ??
> 
> Maybe.
> 
> I just feel like that's more heavy weight than I want.
> 
> I'm functionally running a series of ip commands to configure networking
> in a special way.
> 

You can add commands to your existing network configuration that will be 
run when an interface comes up. For example, in /etc/conf.d/net,

   ifup_wlan0="iwconfig \$int key s:secretkey enc open essid foobar"

(taken from the example file that ships with OpenRC).

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/3/19 6:26 AM, Michael Orlitzky wrote:
> You can add commands to your existing network configuration that will be 
> run when an interface comes up. For example, in /etc/conf.d/net,
> 
>    ifup_wlan0="iwconfig \$int key s:secretkey enc open essid foobar"

Ya....  I find that to be an absolute kludge.  Does it work?  Yes.  Is 
it clean?  Probably not.  Is it graceful?  Absolutely not.

Think about how it's possible to configure bridging / bonding / VLANs 
via various parameters and having netifrc construct the commands that 
are run in the background.

I'd love to see something that assumes the commands run in the main / 
default / unnamed network namespace / VRF unless otherwise specified.

I'd love to be able to add a parameter to a configuration file that 
tells sshd to run in a specific VRF like Alarig was wanting to do. 
Heck, I'd like to see init scripts gracefully deal with the fact that 
there should be multiple instances of a daemon running, even if they are 
simply on different ports, much less different VRFs or namespaces.

Re: [gentoo-user] VRFs / Jails / Containers

[Michael Orlitzky]

On 2/3/19 12:39 PM, Grant Taylor wrote:
> On 2/3/19 6:26 AM, Michael Orlitzky wrote:
>> You can add commands to your existing network configuration that will be
>> run when an interface comes up. For example, in /etc/conf.d/net,
>>
>>     ifup_wlan0="iwconfig \$int key s:secretkey enc open essid foobar"
> 
> Ya....  I find that to be an absolute kludge.  Does it work?  Yes.  Is
> it clean?  Probably not.  Is it graceful?  Absolutely not.
> 
> Think about how it's possible to configure bridging / bonding / VLANs
> via various parameters and having netifrc construct the commands that
> are run in the background.
> 

Ultimately netifrc is just a shell script that parses another shell 
script to construct a third shell script. I don't think doing it with 
only two shell scripts is that much less elegant =)

You could go all the way and write your own OpenRC service as 
/etc/init.d/whatever. You can make it depend on the network being up, 
and then just write everything that you want it to do into the start 
function with the corresponding "undo" steps in the stop function.

If the series of commands is long and complicated and if you sometimes 
want to do/undo this subset of the configuration independently, then 
that's how I'd do it.

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 02/03/2019 11:23 AM, Michael Orlitzky wrote:
> Ultimately netifrc is just a shell script that parses another shell 
> script to construct a third shell script. I don't think doing it with 
> only two shell scripts is that much less elegant =)

The elegance, or lack there of, is not in the number of shell scripts. 
Rather the fact that tc (QoS) parameters are stuffed into a command line 
verses having things split out and parsed is what I dislike.  Take VLANs 
for example, there is a netifrc parameter for specifying the VLAN IDs 
that belong on an interface.  Netifrc will then construct the commands. 
People don't need to know how to construct the commands themselves to 
utilize VLANs.  tc (QoS) is not anywhere nearly as nice.

Bridging and bonding is similarly more graceful than tc (QoS).

> You could go all the way and write your own OpenRC service as 
> /etc/init.d/whatever.

That's sort of where I'm gravitating at the moment.  Something I can 
(re)start/stop via standard init commands.

> You can make it depend on the network being up, and then just write 
> everything that you want it to do into the start function with the 
> corresponding "undo" steps in the stop function.

Maybe it will need to depend on the lowest level of networking.  Maybe. 
Seeing as how it would provide networking between the host and the 
namespaces (containers), I think it would functionally be parallel to 
the networking services.  I think namespaces could be up even if the 
main network was not.

> If the series of commands is long and complicated and if you sometimes 
> want to do/undo this subset of the configuration independently, then 
> that's how I'd do it.

The number of commands is really dependent on what I'm doing at a higher 
level.  I can see having relatively similar commands for different 
namespaces broken out into separate files such that it's easy to 
(re)start/stop individual namespaces.  I might see if there's a way to 
re-use the same file much like net.<device> is a sym-link to net.lo.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] VRFs / Jails / Containers

[Michael Orlitzky]

On 2/4/19 3:50 PM, Grant Taylor wrote:
> On 02/03/2019 11:23 AM, Michael Orlitzky wrote:
>> Ultimately netifrc is just a shell script that parses another shell 
>> script to construct a third shell script. I don't think doing it with 
>> only two shell scripts is that much less elegant =)
> 
> The elegance, or lack there of, is not in the number of shell scripts. 
> Rather the fact that tc (QoS) parameters are stuffed into a command line 
> verses having things split out and parsed is what I dislike.  Take VLANs 
> for example, there is a netifrc parameter for specifying the VLAN IDs 
> that belong on an interface.  Netifrc will then construct the commands. 
> People don't need to know how to construct the commands themselves to 
> utilize VLANs.  tc (QoS) is not anywhere nearly as nice.
> 

If you're willing to go all-out to do this right, you can implement that
parsing yourself.

You can create a package called netifrc-qos that consists of (only?) two
files: an OpenRC service script, and an associated conf.d file. Inside
the conf.d file you can use whatever variables you like, and then have
the init script parse them and run the resulting commands. Afterwards
you'd create an ebuild for it, install it from an overlay, and then edit
the conf.d file accordingly. It's a lot of work, but that's the most
Gentoo way to do it IMO.

You might also want to check with the OpenRC/netifrc projects to see if
they have anything like this in the works already. Either way they may
be willing to collaborate on a design.

Re: [gentoo-user] VRFs / Jails / Containers

[Alarig Le Lay]

For the VRF part, Gentoo supports it; it’s in the upstream kernel
sources.

I only tried it once, but failed because my sshd should have been lunch
in my VRF and I didn’t quickly find a way to do it.

But otherwise, it worked.

-- 
Alarig

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/3/19 1:50 AM, Alarig Le Lay wrote:
> For the VRF part, Gentoo supports it; it’s in the upstream kernel 
> sources.

Yep.  I've been doing Network Namespaces, and VRF to a lesser degree, 
for quite a while now.  It's just all been manual or ad-hock scripts.

> I only tried it once, but failed because my sshd should have been lunch 
> in my VRF and I didn’t quickly find a way to do it.

Yep.

That's the type of integration that I've found lacking.

I'm only currently asking about how to configure the various network 
components, not even how to run processes inside of the various systems.

> But otherwise, it worked.

It absolutely manually works.  I'm looking for the thing(s) to allow the 
Gentoo OS init scripts to take over some of the management.  That's what 
I'm finding lacking.  I asked my question because I was hoping that 
someone would know about something I didn't.  ;-)

Re: [gentoo-user] VRFs / Jails / Containers

[Laurence Perkins]

On Sat, 2019-02-02 at 19:32 -0700, Grant Taylor wrote:
> Does Gentoo have any support for VRFs or (chroot) Jails or
> Containers 
> without going down the Docker (et al) path?
> 
> I'm wanting to do some things with a Gentoo router that is trivial to
> do 
> with network namespaces via manual commands ~> scripts.  But that's
> far 
> from standard Gentoo init script based system.  And I'd like
> something 
> more Gentoo standards based.
> 
> Does Gentoo have or support anything like this natively?  Or am I 
> getting into territory where I'm rolling my own?
> 

Have you tried firejail?  It gives you convenient ways to set up the
container parameters consistently and is in the repo.  Its invocation
is also simple enough to not clutter up your startup scripts.

LMP

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 02/04/2019 09:23 AM, Laurence Perkins wrote:
> Have you tried firejail?  It gives you convenient ways to set up the 
> container parameters consistently and is in the repo.

No, I have not.  Thank you for the pointer.

> Its invocation is also simple enough to not clutter up your startup 
> scripts.

I don't think I mind adding things to start up scripts.  I'm more 
looking for the most Gentoo<ish> way to do what I'm wanting to do 
without relying on something on top of Gentoo.  So if that involves 
adding things to start up scripts, I'm cool with it.

I just don't want to add an entire subsystem, like Docker (et al), if I 
don't actually have to.

I'm starting to wonder if I'm going to be better off writing new scripts 
that will match existing init scripts and their methodology to 
(re)start/stop namespaces / containers / jails.  Perhaps firejail will 
give me what I want or provide insight.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] VRFs / Jails / Containers

[Rich Freeman]

On Mon, Feb 4, 2019 at 1:44 PM Grant Taylor
<gtaylor@gentoo.tnetconsulting.net> wrote:
>
> I'm starting to wonder if I'm going to be better off writing new scripts
> that will match existing init scripts and their methodology to
> (re)start/stop namespaces / containers / jails.  Perhaps firejail will
> give me what I want or provide insight.
>

IMO I would separate your container logic from your service manager logic.

If you have a script that launches a container, then all you need is a
generic init.d script that runs it.

I launch nspawn containers from systemd units all the time.  The only
logic in the units is running the command line to start nspawn.

IMO if you start mixing the two it will just make it harder to
maintain.  Sure, an init.d script CAN do anything, but that doesn't
mean that you should do it this way.

Without creating a separate reply I wanted to react to your other
email detailing your config.  It strikes me that you might not even
need containers to set up all those interfaces and the routing between
them.  However, the container probably still makes sense so that
random processes trying to listen on 0.0.0.0 on the host don't end up
attaching to all those virtual interfaces.

Really all you need is some initialization inside each container and
then the kernel is doing all the work.  You don't really need any
userspace process running in the container except for the fact that
kernel namespaces are attached to processes.  As a result, I'd suggest
considering using sysvinit inside your containers to do the work.  You
might run openrc/netifrc to do the network setup inside each
container, or just have sysvinit run a shell script that initializes
and then terminates, leaving init running childless indefinitely (I
assume it supports this).  If you want a process to noop indefinitely
at minimal cost that is basically the definition of what sysvinit
does...

-- 
Rich

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 02/04/2019 11:55 AM, Rich Freeman wrote:
> IMO I would separate your container logic from your service manager logic.

I'm not exactly sure what you mean by "container logic" vs "service 
manager logic" and how they differ.  I'm assuming that the former 
creates / destroys the container and that the latter manages 
(re)starting/stopping services where ever they are at.

> If you have a script that launches a container, then all you need is a 
> generic init.d script that runs it.

I guess that's one way to do it.  But that doesn't seem very Gentoo<ish> 
to me.

I'd like to see a way that I can have standard service init scripts and 
use them where ever I want them, either inside a container or outside on 
the host.

As long as I don't want to run the same service in multiple places, I 
don't see a problem with doing that.  Multiple instances starts to get 
more tricky, but is still possible, and should be location agnostic.

> I launch nspawn containers from systemd units all the time.  The only 
> logic in the units is running the command line to start nspawn.
> 
> IMO if you start mixing the two it will just make it harder to maintain. 
> Sure, an init.d script CAN do anything, but that doesn't mean that you 
> should do it this way.

I'm wanting to avoid having an init script that creates the container 
and starts services therein.  I'd rather start the container and then 
start the services therein using the same type of init scripts, just 
called within different contexts.

> Without creating a separate reply I wanted to react to your other email 
> detailing your config.  It strikes me that you might not even need 
> containers to set up all those interfaces and the routing between them. 
> However, the container probably still makes sense so that random processes 
> trying to listen on 0.0.0.0 on the host don't end up attaching to all 
> those virtual interfaces.

Yes, I could have all the interfaces on the host.  But I'm doing a 
number of different things and don't want to spoil the host.

The nice containers that I mentioned are long standing containers.  I 
routinely stand up 10 ~ 100 more for various tests.

I'm also using network namespaces as an isolation so that I can easily 
do various things with networking without the added complexity of 
isolating things from each other via command line or policy based 
routing.  Each network namespace can easily have it's view of 0.0.0.0 
(as a good example) and it's own routing table.  I don't need to bother 
with PBR / ip rules / iptables complexities.  Each NetNS just knows 
about it's local interfaces.

> Really all you need is some initialization inside each container and 
> then the kernel is doing all the work.  You don't really need any 
> userspace process running in the container except for the fact that 
> kernel namespaces are attached to processes.

I mostly agree.  I am running BIRD inside the container, but that's more 
of a would be nice to have and I can work around not having it.  There 
are also the occasional commands that I want to run to do 
troubleshooting (ping, traceroute, etc) as well as dynamically modifying 
the containers which is usually done via "nsenter …" or "ip netns exec 
$NetNSname …" commands.

> As a result, I'd suggest considering using sysvinit inside your 
> containers to do the work.

That is a possibility.  But I feel like that's tantamount to saying 
"Gentoo doesn't have an answer for what you're wanting to do, so just 
use Sys V init scripts."  I don't like it.

I like the idea of re-using standard OpenRC / NetifRC scripts inside the 
containers too.  Especially if the services don't conflict anywhere.  To 
me, this re-uses the existing Gentoo methodology in different contexts.

> You might run openrc/netifrc to do the network setup inside each 
> container, or just have sysvinit run a shell script that initializes 
> and then terminates, leaving init running childless indefinitely (I 
> assume it supports this).  If you want a process to noop indefinitely 
> at minimal cost that is basically the definition of what sysvinit does...

The more that I think about it, largely in response to emails in this 
thread, I believe that I want the same overall thing to create the 
network between the default / main / unnamed NetNS and the container, as 
well as likely re-using the OpenRC / NetifRC scripts to configure things 
inside the container.

I think, and would be curious to have someone confirm or refute, that I 
could add configuration information to /etc/conf.d/net for the xyz123 
interface inside the container and use an /etc/init.d/net.xyz123 init 
script sym-linked to /etc/init.d/net.lo script.

My host would not have net.xyz123 in any runlevel.  Certainly not boot 
or default.

I think that would mean that I could run rc-service net.xyz123 start 
inside the container and re-use existing Gentoo methodology.

Now I wonder if I could use custom runlevels for each container and rely 
on standard init system.  }:-)  But that's a different question.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] VRFs / Jails / Containers

[Rich Freeman]

So, I think we're miscommunicating a bit here...

On Mon, Feb 4, 2019 at 4:10 PM Grant Taylor
<gtaylor@gentoo.tnetconsulting.net> wrote:
>
> On 02/04/2019 11:55 AM, Rich Freeman wrote:
> > IMO I would separate your container logic from your service manager logic.
>
> I'm not exactly sure what you mean by "container logic" vs "service
> manager logic" and how they differ.  I'm assuming that the former
> creates / destroys the container and that the latter manages
> (re)starting/stopping services where ever they are at.

I'm saying that an init.d script shouldn't try to do anything other
than initialize a service, which should be implemented outside the
init.d script.

So, if you have a shell script that launches a container, then you
should call it from the init.d script.  You shouldn't merge them into
a single init.d script that has 30 lines of container setup logic or
whatever.

>
> I'd like to see a way that I can have standard service init scripts and
> use them where ever I want them, either inside a container or outside on
> the host.

Of course.  That shell script that launches a container could very
well just launch sysvinit which runs openrc which runs another set of
init.d scripts INSIDE the container to initialize it.

> I'm wanting to avoid having an init script that creates the container
> and starts services therein.  I'd rather start the container and then
> start the services therein using the same type of init scripts, just
> called within different contexts.

Yup - though I would think the scripts inside the container would be
fairly different, as they are doing different things.  The scripts
inside the container aren't starting containers, for a start...

> > As a result, I'd suggest considering using sysvinit inside your
> > containers to do the work.
>
> That is a possibility.  But I feel like that's tantamount to saying
> "Gentoo doesn't have an answer for what you're wanting to do, so just
> use Sys V init scripts."  I don't like it.
>
> I like the idea of re-using standard OpenRC / NetifRC scripts inside the
> containers too.  Especially if the services don't conflict anywhere.  To
> me, this re-uses the existing Gentoo methodology in different contexts.

OpenRC/Netifrc are run by sysvinit in Gentoo, as I mention later on.
These two are not mutually exclusive.

> The more that I think about it, largely in response to emails in this
> thread, I believe that I want the same overall thing to create the
> network between the default / main / unnamed NetNS and the container, as
> well as likely re-using the OpenRC / NetifRC scripts to configure things
> inside the container.

Not sure how much of it would be re-use.  The scripts inside/outside
the container would likely have different roles.

> I think, and would be curious to have someone confirm or refute, that I
> could add configuration information to /etc/conf.d/net for the xyz123
> interface inside the container and use an /etc/init.d/net.xyz123 init
> script sym-linked to /etc/init.d/net.lo script.
>
> My host would not have net.xyz123 in any runlevel.  Certainly not boot
> or default.

Honestly, I wouldn't go sticking container init.d scripts inside the
host init.d.  I mean, I guess you could, but again, separation of
concerns and all that.  You're going to have to use a separate
/etc/runlevels, so why not just a whole separate /etc?

-- 
Rich

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 02/04/2019 02:58 PM, Rich Freeman wrote:
> So, I think we're miscommunicating a bit here...

It happens.

> I'm saying that an init.d script shouldn't try to do anything other 
> than initialize a service, which should be implemented outside the 
> init.d script.

It sounds like you are saying that an init script shouldn't do anything 
other than (re)start/stop a service and that there should be a separate 
script (binary / command) that is the service.

(I'm going to assume that's accurate unless / until you say otherwise.)

Does this apply if the ""service is something as simple as enabling or 
disabling IP forwarding?  Should the init script call a separate script 
to write the proper value to the requisite proc entry?

> So, if you have a shell script that launches a container, then you should 
> call it from the init.d script.  You shouldn't merge them into a single 
> init.d script that has 30 lines of container setup logic or whatever.

I think the issue that I'm having, and part of what you're calling out 
is that the script for a ""container (network namespace) is more than it 
should be.

Consider the following commands to start the ""container:

ip netns add myContainer
ip link add myContainer type veth peer name myHost netns myContainer
ip link set myContainer up
ip addr add 192.0.2.1/24 dev myContainer
ip netns exec myContainer ip link set myHost up
ip netns exec myContainer ip addr add 192.0.2.2/24 dev myHost

Consider the following command to stop the ""container:

ip netns del myContainer

I feel like those two sections could easily fit within an OpenRC init 
script:

start() {
    ip netns add myContainer
    ip link add myContainer type veth peer name myHost netns myContainer
    ip link set myContainer up
    ip addr add 192.0.2.1/24 dev myContainer
    ip netns exec myContainer ip link set myHost up
    ip netns exec myContainer ip addr add 192.0.2.2/24 dev myHost
}

stop() {
    ip netns del myContainer
}

So, I'm not sure why those commands need to or should live inside 
something other than the init script.

Please help me understand what I'm missing or not understanding.

> Of course.  That shell script that launches a container could very well 
> just launch sysvinit which runs openrc which runs another set of init.d 
> scripts INSIDE the container to initialize it.

Now I'm starting to think that you are under the impression that the 
""container(s) that I'm working with are more complicated and have many 
things running in them, more akin to a full OS.  That's not the case for 
me or my use case.

About the only other added complication might be launching BIRD and / or 
an additional network interface.

> Yup - though I would think the scripts inside the container would be 
> fairly different, as they are doing different things.  The scripts inside 
> the container aren't starting containers, for a start...

The contents of the ""container(s) that I'm using are identical to the 
host.  They actually /are/ the host.  I'm not using mount namespaces. 
So the ""container ~> network namespace sees the exact same files as the 
host.

The only reason that I (sometimes) use the UTS namespace is so that 
uname (et al) return a different name when run inside the NetNS.

> OpenRC/Netifrc are run by sysvinit in Gentoo, as I mention later on. 
> These two are not mutually exclusive.

Okay.

> Not sure how much of it would be re-use.  The scripts inside/outside 
> the container would likely have different roles.

I would think that I could (re)start / stop BIRD inside the NetNS the 
exact same way I do on the host.  I would expect that I could use the 
same "rc-service bird …" command inside and outside.

> Honestly, I wouldn't go sticking container init.d scripts inside the host 
> init.d.  I mean, I guess you could, but again, separation of concerns 
> and all that.  You're going to have to use a separate /etc/runlevels, 
> so why not just a whole separate /etc?

Why do I need to use a separate /etc/runlevels?

Why can't I have a single /etc/runlevels/myContainer that is never used 
outside of the container and only used inside the container?  Remember 
that the host and container share the same file system.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/4/19 5:10 PM, Grant Taylor wrote:
> Consider the following commands to start the ""container:
> 
> ip netns add myContainer
> ip link add myContainer type veth peer name myHost netns myContainer
> ip link set myContainer up
> ip addr add 192.0.2.1/24 dev myContainer
> ip netns exec myContainer ip link set myHost up
> ip netns exec myContainer ip addr add 192.0.2.2/24 dev myHost
> 
> Consider the following command to stop the ""container:
> 
> ip netns del myContainer

This is the other method that I'm starting containers.

unshare --mount=/run/mountns/$container --net=/run/netns/$container 
--uts=/run/utsns/$container /bin/true

nsenter --mount=/run/mountns/$container --net=/run/netns/$container 
--uts=/run/utsns/$container /bin/hostname $container

I can use nsenter to execute similar ip link & addr commands to bring 
the links up.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] VRFs / Jails / Containers

[Rich Freeman]

On Mon, Feb 4, 2019 at 7:10 PM Grant Taylor
<gtaylor@gentoo.tnetconsulting.net> wrote:
>
> On 02/04/2019 02:58 PM, Rich Freeman wrote:
> > So, I think we're miscommunicating a bit here...
>
> It happens.
>

Yeah, I think you're over-reading into my posts.  I'm mostly reacting
to your ideas and not trying to be prescriptive.

For example, you talked about running openrc inside the container, and
then I talked about how you might do that, and then you asked why
would you want to run openrc inside the container.  Well, since I
don't know exactly what you're doing I don't know - if it doesn't make
sense then don't do it...  :)

> > I'm saying that an init.d script shouldn't try to do anything other
> > than initialize a service, which should be implemented outside the
> > init.d script.
>
> It sounds like you are saying that an init script shouldn't do anything
> other than (re)start/stop a service and that there should be a separate
> script (binary / command) that is the service.

Essentially.  It isn't like it won't work if you do something else -
it is just a design principle.

Suppose you want to manually launch a container without using the
init.d script, just as you might launch a non-forking server process
to do some console debugging if you were having an issue.  And so on.

It is just a shell script, so it isn't like it won't work if you do it
all in the script.

>
> ip netns add myContainer
> ip link add myContainer type veth peer name myHost netns myContainer
> ip link set myContainer up
> ip addr add 192.0.2.1/24 dev myContainer
> ip netns exec myContainer ip link set myHost up
> ip netns exec myContainer ip addr add 192.0.2.2/24 dev myHost
>

Interesting.  I didn't realize that linux even supported creating
network namespaces without an associated process.  Maybe you don't
need one after all.

I guess since network interfaces can do their netfilter/etc logic
without any processes actually listening on them it makes sense that
these namespaces might have their own existence.

> So, I'm not sure why those commands need to or should live inside
> something other than the init script.

They don't need to.

IMO they should, but that is like saying that your 5000 line C program
should actually have 5000 lines and whitespace, and not look like the
javascript source to gmail.  It is somewhat subjective, as gcc doesn't
care if the whole thing is one big mass of punctuation...  :)

> Why can't I have a single /etc/runlevels/myContainer that is never used
> outside of the container and only used inside the container?  Remember
> that the host and container share the same file system.

Can you actually start openrc in a container using a parameter-driven
runlevel that isn't a number?  I believe you can pass a numeric
runlevel to init and it will start on that runlevel (though I'm not
sure you can start openrc that way in non-trivial configs as that
might skip the boot runlevel, assuming openrc doesn't override this
sysvinit behavior).  You can't use the kernel command line since
containers don't have a separate kernel.

You can of course change the default runlevel for openrc using config
files in /etc, but those are shared with the host in your proposed
design.

Maybe you could hack something together here, but honestly I'm not
sure what you're getting by not having a separate /etc or at least a
bind mount for the openrc config.

Again, assuming you need openrc in the container at all.

-- 
Rich

Re: [gentoo-user] VRFs / Jails / Containers

[Grant Taylor]

On 2/5/19 10:55 AM, Rich Freeman wrote:
> Yeah, I think you're over-reading into my posts.  I'm mostly reacting 
> to your ideas and not trying to be prescriptive.

So we have a feedback loop.  I'm trying to understand why you're saying 
what you're saying.

I'm still looking for possibilities and pros / cons of each.

> For example, you talked about running openrc inside the container, and 
> then I talked about how you might do that, and then you asked why would 
> you want to run openrc inside the container.  Well, since I don't know 
> exactly what you're doing I don't know - if it doesn't make sense then 
> don't do it...  :)

Seeing as how my container really is the same as the host, at least when 
it comes to file system and files, OpenRC is what is there.  Hence my 
interest in re-using what's already there.

That is /if/ doing such does not have too many cons.

> Essentially.  It isn't like it won't work if you do something else - 
> it is just a design principle.

ACK

> Suppose you want to manually launch a container without using the init.d 
> script, just as you might launch a non-forking server process to do some 
> console debugging if you were having an issue.  And so on.

I see what you're saying by starting a typical daemon manually vs init 
scripts.

Given that what I'm talking about doing doesn't leave a process running 
(I'm ignoring BIRD for the moment) there's nothing to kill to stop it.

What I'm doing is really a series of commands that stand the service up 
and a command that stops it.

I guess that I could have an independent script for this and then have 
the init script do nothing more than call the script with a start 
parameter.  But I feel like my independent script is functionally 
identical to an init script.

> It is just a shell script, so it isn't like it won't work if you do it 
> all in the script.

*nod*

> Interesting.  I didn't realize that linux even supported creating network 
> namespaces without an associated process.  Maybe you don't need one 
> after all.

I've been using network namespaces without a process for quite a while. 
They are extremely handy.

> I guess since network interfaces can do their netfilter/etc logic 
> without any processes actually listening on them it makes sense that 
> these namespaces might have their own existence.

;-)

> They don't need to.

Which is why I was back to putting the (re)start / stop commands in the 
init script.

> IMO they should, but that is like saying that your 5000 line C program 
> should actually have 5000 lines and whitespace, and not look like the 
> javascript source to gmail.  It is somewhat subjective, as gcc doesn't 
> care if the whole thing is one big mass of punctuation...  :)

Yes, there is subjectivity to it.  But there are also Gentoo 
methodologies and guidelines.

> Can you actually start openrc in a container using a parameter-driven 
> runlevel that isn't a number?

I don't know.  I'll have to try and find out.

> I believe you can pass a numeric runlevel to init and it will start 
> on that runlevel (though I'm not sure you can start openrc that way 
> in non-trivial configs as that might skip the boot runlevel, assuming 
> openrc doesn't override this sysvinit behavior).  You can't use the 
> kernel command line since containers don't have a separate kernel.

I think I understand what you're saying.  I also doubt that I will need 
a boot runlevel in a container the way that I'm doing them.

I can see needing a net.<something> and maybe a BIRD init script in the 
target runlevel.  But other than that, there's virtually nothing that 
does not already exist from the host's file system / environment.

> You can of course change the default runlevel for openrc using config 
> files in /etc, but those are shared with the host in your proposed design.

ACK

I /can/ use a mount namespace and have a different /etc.  But I'd rather 
not do that unless there is a need to do so.

> Maybe you could hack something together here, but honestly I'm not sure 
> what you're getting by not having a separate /etc or at least a bind 
> mount for the openrc config.

I currently have no need for a separate /etc.  So why have something 
that I don't currently need?

> Again, assuming you need openrc in the container at all.

Seeing as how the container has the same files as the host, that means 
that OpenRC is already in the container.  So if I can make use of it 
without causing problems, then why not?