Re: [gentoo-dev] Managing updates on many identical Gentoo systems

[Alec Warner <antarus@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 3107 bytes --]

On Thu, Jan 18, 2018 at 6:46 AM, Anthony G. Basile <blueness@gentoo.org>
wrote:

> Hi everyone,
>
> I'm trying to design an update system for many identical Gentoo systems.
>  Using a binhost is obvious, but there are still problems with this
> approach.
>
> Unless there's some magic I don't know about (and this is why I'm
> sending this email) each machine still needs to have the portage tree
> installed locally (1.5 GB) or somehow mounted by a network filesystem
> (which is not practical if the machines are not on a local network).
>

+Zac

I don't believe this is true; with the correct binhost configuration
portage should:

1) Contact the binhost to get a dump of packages available on it.
2) Use that dump to create a 'virtual portage tree' (bindb).
3) Use the bindb for package discovery (is foo available) and dependency
calculation.


> Furthermore, each machine would have to run emerge locally to do the
> calculation of what packages need updating.
>

The good news is that with a bindb; the package tree itself is much
smaller. ::gentoo is 23000 packages.
But you bindb is only as large as you build binpkgs for and publish them.
So say stage3 + some other stuff.
Around 500 packages perhaps.

A couple of exciting problems might include:

1) How to expose the bindb to machines?
2) How to secure bindb updates (HTTPS?)


> This procedure is redundant because each machine is housing the same
> data and doing the same dependence-tree calculation.  It should be
> possible to do this calculation on a centralized binhost and simply
> communicate the update information to the remote machines.  They would
> then only have to download the .tbz2's and install them, keeping a tidy
> /var/db/pkg.  Thus they avoid having to house the portage tree and
> burning cpu cycles that just calculate redundant information.
>

Emerge used to have support for emerge package.tar.gz; I wonder if it still
works?
I haven't seen anyone use it in forever?

antarus@woodpecker ~ $ emerge --help
emerge: command-line interface to the Portage system
Usage:
   emerge [ options ] [ action ] [ ebuild | tbz2 | file | @set | atom ] [ ... ]
   emerge [ options ] [ action ] < @system | @world >

Note that 'tbz2' is still listed ;)


>
> I'm inspired here by OpenBSD's pkg_add which doesn't require all of
> ports to be installed, and mender which is a
>

An alternative is that your central build hosts builds images instead of
packages.
You then boot your devices into a new image to get updates. This is a
pretty common pattern for embedded devices.
You need a secure method to deliver new images and probably a sane method
for rollback of a bad image.
Typically this is done with a bootloader and 3 image slots:

SLOT A: Current image.
SLOT B: Proposed update image.
SLOT C: Recovery Image # usually RO.

Bootloader boots A normally; but during update boots B. If A and B fail to
boot, Boots C.


>
> Any ideas?
>
> --
> Anthony G. Basile, Ph.D.
> Gentoo Linux Developer [Hardened]
> E-Mail    : blueness@gentoo.org
> GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
> GnuPG ID  : F52D4BBA
>
>

[-- Attachment #2: Type: text/html, Size: 4735 bytes --]