[gentoo-embedded] root on squashfs+aufs

[gentoo-embedded] root on squashfs+aufs

[Ed W]

Hi, I'm interested in hearing how others have managed a root filesystem 
based on squashfs (or similar) with layering using aufs (or unionfs)?

I am building an embedded box and it looks like it would work well to 
have a simple package manager based around a bunch of squashfs 
partitions which layer on new functionality for certain builds (and it 
would allow some versioning).  On the top of that we need a writeable 
bunch of configuration files which we might do either with symlinks or a 
writeable top level partition (still undecided).

Mainly interested to hear how others have managed the boot process and 
built their initrd/linuxrc?  I'm sure I read about at least one person 
on this list calling a similar setup "squashfs packages"?

Thanks

Ed W

Re: [gentoo-embedded] root on squashfs+aufs

[joerg.faschingbauer]

>>>>> "Ed" == Ed W <lists@wildgooses.com> writes:

Ed> Hi, I'm interested in hearing how others have managed a root
Ed> filesystem based on squashfs (or similar) with layering using aufs (or
Ed> unionfs)?

Ed> I am building an embedded box and it looks like it would work well to
Ed> have a simple package manager based around a bunch of squashfs
Ed> partitions which layer on new functionality for certain builds (and it
Ed> would allow some versioning).  On the top of that we need a writeable
Ed> bunch of configuration files which we might do either with symlinks or
Ed> a writeable top level partition (still undecided).

Ed> Mainly interested to hear how others have managed the boot process and
Ed> built their initrd/linuxrc?  I'm sure I read about at least one person
Ed> on this list calling a similar setup "squashfs packages"?

I did a setup using aufs. Not squashfs, but iso9660 instead (people
like to use Windows to create packages). Anyway, it shouldn't make a
real difference which filesystem is used.

It starts with an initramfs containing busybox, /init, and a few
helper scripts that implement the stacking of the packages. init and
the scripts interpret a bunch of custom kernel commandline parameters
of their own, like 'Where are my packages?' (ISOs are files on a FAT32
partition on the compact flash), and 'Where will I write to?'  (this
can be either another directory on a filesystem, or a ramfs instance).

When all packages have been stacked, hook scripts that come with the
packages are run in a chroot. For example, packages can add their own
init scripts by calling rc-update, or add users, or whatnot.

When that is done, the Busybox switch_root command is used to hand
over the boot to the Gentoo system that is the sum of the packages.

What I am extremely excited about is that everything worked right out
of the box. Not even one single problem. I expected aufs to fail with
many branches (currently we have some 50 of them), busybox to be not
compatible enough with the shell, initramfs to impose too many
restrictions ... - nothing.

Cheers,
Joerg

Re: [gentoo-embedded] root on squashfs+aufs

[Ed W]

Hi, this seems extremely similar to what I am targeting!


joerg.faschingbauer@aon.at wrote:
> When all packages have been stacked, hook scripts that come with the
> packages are run in a chroot. For example, packages can add their own
> init scripts by calling rc-update, or add users, or whatnot.
>   

I assume you run these hook scripts each time you boot then?  This seems 
like a good idea, I had been thinking about putting that on the writable 
partition and dreaming up some way to ensure that the scripts were only 
run once - I think your design is cleaner (although need to watch boot 
speeds)

I would be very interested to pick over your implementation details if 
you were willing to share (offlist if you don't think it's sensible to 
post publically?).  I'm quite interested in how you picked the package 
list and ordered them and exactly how you implemented the hook scripts 
and failure conditions?

> What I am extremely excited about is that everything worked right out
> of the box. Not even one single problem. I expected aufs to fail with
> many branches (currently we have some 50 of them), busybox to be not
> compatible enough with the shell, initramfs to impose too many
> restrictions ... - nothing.
>   

That's really good to hear!  I am currently only planning about 3-4 
packages, but based on what you say I might look to split my packages 
more granually...

Thanks for sharing

Ed W

Re: [gentoo-embedded] root on squashfs+aufs

[joerg.faschingbauer]

>>>>> "Ed" == Ed W <lists@wildgooses.com> writes:

>> When all packages have been stacked, hook scripts that come with the
>> packages are run in a chroot. For example, packages can add their own
>> init scripts by calling rc-update, or add users, or whatnot.
>> 

Ed> I assume you run these hook scripts each time you boot then?  This
Ed> seems like a good idea, I had been thinking about putting that on the
Ed> writable partition and dreaming up some way to ensure that the scripts
Ed> were only run once - I think your design is cleaner (although need to
Ed> watch boot speeds)

You definitely have to. Although our platform is "embedded" in a sense
that it has no keyboard and mouse, and that it has to "just work", it
has a dual core 2GHz Intel. This is not to say that boot is slow
(rc-update does not crunch numbers), it's just that it is fast enough
and we didn't invest much to speed it up.

Ed> I would be very interested to pick over your implementation details if
Ed> you were willing to share (offlist if you don't think it's sensible to
Ed> post publically?).  I'm quite interested in how you picked the package
Ed> list and ordered them and exactly how you implemented the hook scripts
Ed> and failure conditions?

Unfortunately I did the work for my employer, so I cannot share it
with you, at least not character by character.

One package (one ISO) contains several directories which become aufs
branches during boot, and a dedicated single directory that contains
links like

 00base-os -> ../base-os
 10core-modules -> ../blah

It's these links that give the order of the stack, and it's actually
the links that are being stacked. You need this order because (at
least) the base OS install has to come first. Other packages may
overlay files from there, for example /etc/X11/xorg.conf or
/etc/conf.d/net.

>> What I am extremely excited about is that everything worked right out
>> of the box. Not even one single problem. I expected aufs to fail with
>> many branches (currently we have some 50 of them), busybox to be not
>> compatible enough with the shell, initramfs to impose too many
>> restrictions ... - nothing.
>> 

Ed> That's really good to hear!  I am currently only planning about 3-4
Ed> packages, but based on what you say I might look to split my packages
Ed> more granually...

Definitely. What I did was to keep things separate when they are not
related to each other. Some of the directories have a tree structure
that hosts only one single file. I don't have 50 ISO files around
though - I aggregate many directories into one ISO.

Cheers,
Joerg

Re: [gentoo-embedded] root on squashfs+aufs

[Ed W]

Hi

> One package (one ISO) contains several directories which become aufs
> branches during boot, and a dedicated single directory that contains
> links like
>
>  00base-os -> ../base-os
>  10core-modules -> ../blah
>
> It's these links that give the order of the stack, and it's actually
> the links that are being stacked. You need this order because (at
> least) the base OS install has to come first. 

Hmm, I think I need to read up on the aufs mount options.  I'm still at 
the planning stage, but if aufs is really this flexible then it will 
help a lot.  Sounds like you can pretty much do "ls -1 | sort" in your 
case and use the result as your mount command?

Perhaps you don't even need your hook scripts then?  They could just 
become normal /etc/init.d scripts and would get run normally as part of 
the standard boot process?

> Definitely. What I did was to keep things separate when they are not
> related to each other. Some of the directories have a tree structure
> that hosts only one single file. I don't have 50 ISO files around
> though - I aggregate many directories into one ISO.
>
>   

Did you consider gpg signing stuff?  I can see this would make a lot of 
sense for my app, but worried this is going to add a ton of time to the 
boot process...

It will possibly be enough for this application to do the gpg check when 
adding new images, and then some effort into disallowing other routes to 
tamper with the boot images.  After all if the main boot process can be 
subverted then the gpg checks can be removed altogether. (linuxbios if I 
was really paranoid I guess...)

Thanks for sharing

Ed W

Re: [gentoo-embedded] root on squashfs+aufs

[joerg.faschingbauer]

>>>>> "Ed" == Ed W <lists@wildgooses.com> writes:

Ed> Hmm, I think I need to read up on the aufs mount options.  I'm still
Ed> at the planning stage, but if aufs is really this flexible then it
Ed> will help a lot.  Sounds like you can pretty much do "ls -1 | sort" in
Ed> your case and use the result as your mount command?

That's basically what I do.

Ed> Perhaps you don't even need your hook scripts then?  They could just
Ed> become normal /etc/init.d scripts and would get run normally as part
Ed> of the standard boot process?

They don't run just because they are there. Well, I could add the
symlinks in /etc/runlevels/default which is what rc-update does, but
that's no fun.

We enjoy the flexibility of the scripts because they provide a
convenient way to glue the system together. For example, several
packages contribute kernel modules, and it's a hook script where we
call 'depmod -a' to create a consistent view of the entire
system. Much like env-update is called from a hook to gather
environment settings from different packages.

Ed> Did you consider gpg signing stuff?  I can see this would make a lot
Ed> of sense for my app, but worried this is going to add a ton of time to
Ed> the boot process...

We sign ISO files, this involves ECDSA and smartcards. I cannot tell
you more, or I'll have to kill you. (Honestly, I don't quite
understand what's going on exactly.)

Cheers,
Joerg