From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 17BA2138206 for ; Thu, 18 Jan 2018 16:13:45 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 80278E089A; Thu, 18 Jan 2018 16:13:38 +0000 (UTC) Received: from mail-vk0-x235.google.com (mail-vk0-x235.google.com [IPv6:2607:f8b0:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 1D76AE088F for ; Thu, 18 Jan 2018 16:13:37 +0000 (UTC) Received: by mail-vk0-x235.google.com with SMTP id n4so13935594vkd.6 for ; Thu, 18 Jan 2018 08:13:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scriptkitty-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=x74lxypZC3bhrkXKEQzhG4jMDNQi6vREnFQxQanid7w=; b=VvI1tXDnCYZmxv2hdvs3EOuB6PmLOvnpOu4ZYJduUsHtTtXFaQz2SC8A5PhfIw+3cS yGbvLVgtzT8jMI6SDY47/hV0xdXK9VQlX4y4CdYovIS3o5d5zy2bY6R9ZsSiSv7cmGgq pJi7yb7/+wD77MzdPZeKbVDRKRN6fQcHaMb7LKZ8TvxztbOeZA3l7yOUy2RdFeaRcp8D 0K2g2C45wlYoh3F7F5Ee5NKrnDnOoircbQOeePUuOvqVlX4qEBOlJOZEmja2Vn1iPLaT wzicK2d4oB+fHAMuegmzoA9WulNTZX7aJAPSwZZvF1+euOm0IV/RPpNiBXxf9VEtLlW1 4fng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=x74lxypZC3bhrkXKEQzhG4jMDNQi6vREnFQxQanid7w=; b=mzTmrN7IFqqEBAsE2U3pUzzwGMzVWLvGtC0AA8ORu2lKzFrgJST/420QbwcKy4SVpK S1kgtmtNZ5evxo6xPtMtZPnVySunaEOkLKHfMsEkIeHYuPr3ipBYXzW89I+tfVu5rTBJ /msDOYyViIW5YmLBtBK3VnAgI+wnWfsZ5HEXXTfXGnSlbyfhKBMjtkjSl0274tD7vOKV J/FMUbM01+bV8tLmwXoZpdbzAFYPu9XdksAzlJEZlol4ABn/F5hN0cPUb2RFUYnIxVVO 2fPzfqLoDktOAmscY7OlmHfpTI9BYEdeBiLU6nMxrQC7NGzuaCA5T9IOLGvMbz2Scd8f ChFw== X-Gm-Message-State: AKwxytfsUbOGTEHHmSXSrJ5KYwn2hC8kzC/VbhJlGEOV9jvYMhqqZwx7 kzZ77RX2eu1QN5PgFusDHIgUuV51VDI7Mo+euOUKTUqE X-Google-Smtp-Source: ACJfBotCyPvfvnAC1W1E1dwGkoe+9EbK5HY8m949rR6E9qXlkIRjjS5QVEokceEpRJXBUnlF4xupOq2sCfk+wCmFj+0= X-Received: by 10.31.161.214 with SMTP id k205mr1049556vke.82.1516292016770; Thu, 18 Jan 2018 08:13:36 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Sender: antarus@scriptkitty.com Received: by 10.176.29.20 with HTTP; Thu, 18 Jan 2018 08:13:36 -0800 (PST) X-Originating-IP: [2620:0:1003:410:70fb:b96d:b61a:b6c5] In-Reply-To: <2686de8e-334c-084b-4828-6109b10dd536@gentoo.org> References: <2686de8e-334c-084b-4828-6109b10dd536@gentoo.org> From: Alec Warner Date: Thu, 18 Jan 2018 11:13:36 -0500 X-Google-Sender-Auth: Ug1WWhKHIPBWStc8sKasHhT76ww Message-ID: Subject: Re: [gentoo-dev] Managing updates on many identical Gentoo systems To: Gentoo Dev , Zac Medico Content-Type: multipart/alternative; boundary="001a1143f1680925ff05630f3fd3" X-Archives-Salt: 25352915-f698-4e55-8946-4a300bb2333f X-Archives-Hash: e94cc662c3ad6aee1a9ec3619c3beca1 --001a1143f1680925ff05630f3fd3 Content-Type: text/plain; charset="UTF-8" On Thu, Jan 18, 2018 at 6:46 AM, Anthony G. Basile wrote: > Hi everyone, > > I'm trying to design an update system for many identical Gentoo systems. > Using a binhost is obvious, but there are still problems with this > approach. > > Unless there's some magic I don't know about (and this is why I'm > sending this email) each machine still needs to have the portage tree > installed locally (1.5 GB) or somehow mounted by a network filesystem > (which is not practical if the machines are not on a local network). > +Zac I don't believe this is true; with the correct binhost configuration portage should: 1) Contact the binhost to get a dump of packages available on it. 2) Use that dump to create a 'virtual portage tree' (bindb). 3) Use the bindb for package discovery (is foo available) and dependency calculation. > Furthermore, each machine would have to run emerge locally to do the > calculation of what packages need updating. > The good news is that with a bindb; the package tree itself is much smaller. ::gentoo is 23000 packages. But you bindb is only as large as you build binpkgs for and publish them. So say stage3 + some other stuff. Around 500 packages perhaps. A couple of exciting problems might include: 1) How to expose the bindb to machines? 2) How to secure bindb updates (HTTPS?) > This procedure is redundant because each machine is housing the same > data and doing the same dependence-tree calculation. It should be > possible to do this calculation on a centralized binhost and simply > communicate the update information to the remote machines. They would > then only have to download the .tbz2's and install them, keeping a tidy > /var/db/pkg. Thus they avoid having to house the portage tree and > burning cpu cycles that just calculate redundant information. > Emerge used to have support for emerge package.tar.gz; I wonder if it still works? I haven't seen anyone use it in forever? antarus@woodpecker ~ $ emerge --help emerge: command-line interface to the Portage system Usage: emerge [ options ] [ action ] [ ebuild | tbz2 | file | @set | atom ] [ ... ] emerge [ options ] [ action ] < @system | @world > Note that 'tbz2' is still listed ;) > > I'm inspired here by OpenBSD's pkg_add which doesn't require all of > ports to be installed, and mender which is a > An alternative is that your central build hosts builds images instead of packages. You then boot your devices into a new image to get updates. This is a pretty common pattern for embedded devices. You need a secure method to deliver new images and probably a sane method for rollback of a bad image. Typically this is done with a bootloader and 3 image slots: SLOT A: Current image. SLOT B: Proposed update image. SLOT C: Recovery Image # usually RO. Bootloader boots A normally; but during update boots B. If A and B fail to boot, Boots C. > > Any ideas? > > -- > Anthony G. Basile, Ph.D. > Gentoo Linux Developer [Hardened] > E-Mail : blueness@gentoo.org > GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA > GnuPG ID : F52D4BBA > > --001a1143f1680925ff05630f3fd3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On T= hu, Jan 18, 2018 at 6:46 AM, Anthony G. Basile <blueness@gentoo.org&= gt; wrote:
Hi e= veryone,

I'm trying to design an update system for many identical Gentoo systems= .
=C2=A0Using a binhost is obvious, but there are still problems with this approach.

Unless there's some magic I don't know about (and this is why I'= ;m
sending this email) each machine still needs to have the portage tree
installed locally (1.5 GB) or somehow mounted by a network filesystem
(which is not practical if the machines are not on a local network).

+Zac

I don't = believe this is true; with the correct binhost configuration portage should= :

1) Contact the binhost to get a dump of packages= available on it.
2) Use that dump to create a 'virtual porta= ge tree' (bindb).
3) Use the bindb for package discovery (is = foo available) and dependency calculation.
=C2=A0
Furthermore, each machine would= have to run emerge locally to do the
calculation of what packages need updating.

=
The good news is that with a bindb; the package tree itself is much sm= aller. ::gentoo is 23000 packages.
But you bindb is only as large= as you build binpkgs for and publish them. So say stage3 + some other stuf= f.
Around 500 packages perhaps.

A couple= of exciting problems might include:

1) How to exp= ose the bindb to machines?
2) How to secure bindb updates (HTTPS?= )


This procedure is redundant because each machine is housing the same
data and doing the same dependence-tree calculation.=C2=A0 It should be
possible to do this calculation on a centralized binhost and simply
communicate the update information to the remote machines.=C2=A0 They would=
then only have to download the .tbz2's and install them, keeping a tidy=
/var/db/pkg.=C2=A0 Thus they avoid having to house the portage tree and
burning cpu cycles that just calculate redundant information.

Emerge used to have support for emerge package.tar.g= z; I wonder if it still works?
I haven't seen anyone use it i= n forever?

antarus@woodpecker ~ $ emerge --he=
lp
emerge: command-line interface to the Portage system
Usage:
   emerge [ options ] [ action ] [ ebuild | tbz2 | file | @set | atom ] [ .=
.. ]
   emerge [ options ] [ action ] < @system | @world >
Note= that 'tbz2' is still listed ;)
=C2=A0

I'm inspired here by OpenBSD's pkg_add which doesn't require al= l of
ports to be installed, and mender which is a

An alternative is that your central build hosts builds images instead= of packages.
You then boot your devices into a new image to get = updates. This is a pretty common pattern for embedded devices.
Yo= u need a secure method to deliver new images and probably a sane method for= rollback of a bad image.
Typically this is done with a bootloade= r and 3 image slots:

SLOT A: Current image.
<= div>SLOT B: Proposed update image.
SLOT C: Recovery Image # usual= ly RO.

Bootloader boots A normally; but during upd= ate boots B. If A and B fail to boot, Boots C.
=C2=A0

Any ideas?

--
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail=C2=A0 =C2=A0 : blueness@gento= o.org
GnuPG FP=C2=A0 : 1FED FAD9 D82C 52A5 3BAB=C2=A0 DC79 9384 FA6E F52D 4BBA GnuPG ID=C2=A0 : F52D4BBA


--001a1143f1680925ff05630f3fd3--