[gentoo-dev] Gentoo LTS or: proper backward compatibility?

[gentoo-dev] Gentoo LTS or: proper backward compatibility?

[m1027]

Hi and happy new year.

When we create apps on Gentoo they become easily incompatible for
older Gentoo systems in production where unattended remote world
updates are risky. This is due to new glibc, openssl-3 etc.

So, what we've thought of so far is:

(1) Keeping outdated developer boxes around and compile there. We
would freeze portage against accidental emerge sync by creating a
git branch in /var/db/repos/gentoo. This feels hacky and requires a
increating number of develper VMs. And sometimes we are hit by a
silent incompatibility we were not aware of.

(2) Using Ubuntu LTS for production and Gentoo for development is
hit by subtile libjpeg incompatibilites and such.

(3) Distributing apps as VMs or docker: Even those tools advance and
become incompatible, right? And not suitable when for smaller Arm
devices.

(4) Flatpak: No experience, does it work well?

(5) Inventing a full fledged OTA Gentoo OS updater and distribute
that together with the apps... Nah.

Hm... Comments welcome.

Thanks

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Rich Freeman]

On Mon, Jan 2, 2023 at 7:48 AM m1027 <m1027@posteo.net> wrote:
>
> When we create apps on Gentoo they become easily incompatible for
> older Gentoo systems in production where unattended remote world
> updates are risky. This is due to new glibc, openssl-3 etc.

So, unless you're proposing some improvement this might be
better-suited for the -user list.  Officially Gentoo doesn't really
"support" a stable/LTS/release-based environment.  That isn't to say
that it couldn't be made to do this, however, a more release-based
process is one of the core competencies of most other distros, so I'm
not sure how much Gentoo would add here vs just running something
else.

> (1) Keeping outdated developer boxes around and compile there. We
> would freeze portage against accidental emerge sync by creating a
> git branch in /var/db/repos/gentoo. This feels hacky and requires a
> increating number of develper VMs. And sometimes we are hit by a
> silent incompatibility we were not aware of.

If you're running a large production environment and want a stable
platform to target, I'd think you'd want to minimize the number of
those you have.  You wouldn't want every server running a different
base OS, and then have a branch to target development at it.

The idea of basing your own releases on Gentoo is probably something
many companies do.  All you need is to just host one repository and
distfile mirror for it, and point all your production hosts at it.
Then you could have staging/development environments, and promote the
core OS as appropriate.  Then your migration path is the same for all
hosts and you can account for major vs minor changes.

You do need to mirror distfiles as one of the weaknesses of Gentoo for
a release-based strategy is that we do not have a solid solution for
archiving things like patches/services/etc that are distributed
outside of the repo.  If a package is removed from the current repo no
QA process ensures that the SRC_URIs for anything they used remain
stable.

Of course somebody would need to do the work but it shouldn't be THAT
difficult to have a Gentoo Reference Release project that basically
just snapshots the repository/distfiles at points in time and provides
a guarantee of clean updates between milestones.  Of course if you
want backported security updates on top of that this would be quite a
bit more effort.  It would mainly be useful for those who aren't
updating regularly, but completely ignoring all updates isn't a good
practice regardless which is probably part of why this hasn't
happened.  You could greatly minimize the cost of backporting security
updates if you only targeted packages of interest to you, or where the
updates are low-risk to you, but that depends quite a bit on your own
needs.

> (3) Distributing apps as VMs or docker: Even those tools advance and
> become incompatible, right? And not suitable when for smaller Arm
> devices.

I don't see why containers would be unsuitable for ARM.  VMs certainly
are memory hogs but well-designed containers don't really consume much
more memory than the service they are hosting.  Of course if you dump
a whole OS in a container that will consume a fair bit of RAM.

An advantage of containers is that they make the host OS less
relevant.  You could run a release-based distro with security-only
updates on the host, and then application containers are low-risk to
update remotely and can be built in whatever environment they are most
suited to.

There is a reason k8s is popular.

-- 
Rich

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Michael Orlitzky]

On Mon, 2023-01-02 at 12:48 +0000, m1027 wrote:
> Hi and happy new year.
> 
> When we create apps on Gentoo they become easily incompatible for
> older Gentoo systems in production where unattended remote world
> updates are risky. This is due to new glibc, openssl-3 etc.
> 

Just update them. YOLO.

Seriously though, most of us run Gentoo in some sort of production
setting. Updating often is your best bet as it keeps the list of
suspects short when things do break.

OTOH I'm only about 15km from our servers when I set them on fire, so
keep that in mind if yours are on another continent. If updating
frequently is truly out of the question, a rolling release probably
isn't the best deployment target for you.

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]



> On 2 Jan 2023, at 12:48, m1027 <m1027@posteo.net> wrote:
> 
> Hi and happy new year.
> 
> When we create apps on Gentoo they become easily incompatible for
> older Gentoo systems in production where unattended remote world
> updates are risky. This is due to new glibc, openssl-3 etc.
> 
> So, what we've thought of so far is:
> 
> (1) Keeping outdated developer boxes around and compile there. We
> would freeze portage against accidental emerge sync by creating a
> git branch in /var/db/repos/gentoo. This feels hacky and requires a
> increating number of develper VMs. And sometimes we are hit by a
> silent incompatibility we were not aware of.
> 
> (2) Using Ubuntu LTS for production and Gentoo for development is
> hit by subtile libjpeg incompatibilites and such.
> 
> (3) Distributing apps as VMs or docker: Even those tools advance and
> become incompatible, right? And not suitable when for smaller Arm
> devices.
> 
> (4) Flatpak: No experience, does it work well?
> 
> (5) Inventing a full fledged OTA Gentoo OS updater and distribute
> that together with the apps... Nah.
> 
> Hm... Comments welcome.

I'd really suggest just using stable in production and a mix
for developers so you can catch any problems beforehand.

We try to be quite conservative about things like OpenSSL 3,
glibc updates, etc.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[m1027]

sam:

> > On 2 Jan 2023, at 12:48, m1027 <m1027@posteo.net> wrote:
> > 
> > Hi and happy new year.
> > 
> > When we create apps on Gentoo they become easily incompatible for
> > older Gentoo systems in production where unattended remote world
> > updates are risky. This is due to new glibc, openssl-3 etc.

[...]

> I'd really suggest just using stable in production and a mix
> for developers so you can catch any problems beforehand.
> 
> We try to be quite conservative about things like OpenSSL 3,
> glibc updates, etc.

Thanks, a misunderstanding then: I am talking about stable only.

Whilst Gentoo may be conservative and all of you do an excellent job
on keeping things smooth, incompatibilities of software being newly
created on up-to-date developer systems (and then tried to be
distributed to outdated production systems) may arise multiple times
per year; *any* of the following alone may trigger a incompatibility.

Just some random examples:

(1) Most prominent, glibc updates. It has its sophisticated function
versioning. You may or may not be hit. It is not depending on
glibc's version but the internal function versions which are updated
in a quite subtile way. (Function versioning is without doubt an
impressive feature.)

(2) libjpeg: In the past, libjpeg reached version 9 (like on Ubuntu
today) but later was versioned 62 or 6.2 AFAIK. If you have an old
Gentoo production system still on libjpeg-9, you have a hard time to
update and distribute a new version of your app, as this version of
libjpeg is not present in Gentoo anymore. (Don't get me wrong, there
have probably been good reasons for this downgrade.) BTW: This
little incompatibility is one of the reasons why it is hard to
compile a app on Ubuntu for Gentoo.

(3) An older one: libressl was removed. Well, we all remember the
debate whether to keep it or not.

(4) There is openssl-3 showing up on the horizon. I expect
incompatibilities when distributing newly built software.

(5) Portage EAPIs: If there is a new EAPI and you emerge --sync,
then you need to update portage. This however, might require
surprisingly many other updates. New python, setuptools and friends.

I am not complaining here. Hey, we are on rolling release. Some of
you may even know individual solutions to work around each of it.
However, we just may get into trouble when distributing newly
compiled apps (on new Gentoo systems) to older Gentoo systems. And
we don't know in advance. I am looking for the best way to avoid
that.

Thanks

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Alarig Le Lay]

Hi,

On Mon 02 Jan 2023 23:31:17 GMT, m1027 wrote:
> I am not complaining here. Hey, we are on rolling release. Some of
> you may even know individual solutions to work around each of it.
> However, we just may get into trouble when distributing newly
> compiled apps (on new Gentoo systems) to older Gentoo systems. And
> we don't know in advance. I am looking for the best way to avoid
> that.

It’s not really a production environment, but I have this hobby infra
I’ve set up when I was a student an my solution is to have a VM
compiling the big packages (gcc, glibc, llvm, php, etc.) and publish a
binary repo from that. Then, every VM pulls this, and compile what’s
specific on top of that.
Everything is handled by crons. So far it works. One thing to get in
mind is that, if a VM got stuck to an old commit (e.g. because one
package needed a use change and emerge fails to update world due to
that), I’m always re-compiling everything, because I’m scared of glibc
or ncurses ABI breaks.

My scale is ~30 VMs on 4 physical servers.

-- 
Alarig

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Peter Stuge]

Hi,

m1027 wrote:
> So, what we've thought of so far is:
> 
> (1) Keeping outdated developer boxes around and compile there. We
> would freeze portage against accidental emerge sync by creating a
> git branch in /var/db/repos/gentoo. This feels hacky and requires a
> increating number of develper VMs. And sometimes we are hit by a
> silent incompatibility we were not aware of.
..
> (5) Inventing a full fledged OTA Gentoo OS updater and distribute
> that together with the apps... Nah.
> 
> Hm... Comments welcome.

I recommend taking ownership (and responsibility) of your OS.

Gentoo tooling (catalyst) is really fantastic for doing so.

Essentially you will be maintaining a private fork of gentoo.git, but
one where you only really need to manually process the packages you
care to control, only when you care to control them.

Use catalyst to build tarballs (and binpkgs) from snapshots of that repo.

emerge can install binpkg at least from FTP.


//Peter

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Peter Stuge]

Peter Stuge wrote:
> Essentially you will be maintaining a private fork of gentoo.git,

If this seems too heavy handed then you can just as well do the reverse:

Maintain an overlay repo with the packages you care to control in the
state you care to have them, set that in the catalyst stage4.spec
portage_overlay and add unwanted package versions in gentoo.git to
the package.mask directory used by catalyst.

This may sound complicated but it isn't bad at all.

For total control also make your own profile, e.g. based on embedded,
but that's not per se neccessary, only if the standard profiles has too
much conflicts with what you want in @system.

catalyst will rebuild @system according to spec file but with too much
difference that just becomes annoying and feels more trouble than a
controlled profile.

This approach falls somewhere between your options (1) and (5).


Good luck!

//Peter

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[m1027]

peter:

> Peter Stuge wrote:
> > Essentially you will be maintaining a private fork of gentoo.git,
> 
> If this seems too heavy handed then you can just as well do the reverse:
> 
> Maintain an overlay repo with the packages you care to control in the
> state you care to have them, set that in the catalyst stage4.spec
> portage_overlay and add unwanted package versions in gentoo.git to
> the package.mask directory used by catalyst.
> 
> This may sound complicated but it isn't bad at all.
> 
> For total control also make your own profile, e.g. based on embedded,
> but that's not per se neccessary, only if the standard profiles has too
> much conflicts with what you want in @system.
> 
> catalyst will rebuild @system according to spec file but with too much
> difference that just becomes annoying and feels more trouble than a
> controlled profile.
> 
> This approach falls somewhere between your options (1) and (5).

Wow, wasn't aware of catalyst at all. What a beast in terms of
control.

(FYI: I enjoyed the links on catalyst you sent me directly.
Unfortunatelly I cannot answer you directly due to the default TLS
guarantee kicked in by my provider: "TLS is required, but was not
offered by host". That's usually to be fixed on the receiving side.)

While being able to build exact environments with catalyst I wonder
how it could help fixing the issue of my original post. To sum up:
Whenever we need to deliver a updated app to customers whose OS is
too old (but updating it is too risky), we could either a) keep
evenly outdated dev build OSes around forever (oh no!), or b) post
our newly built app in a container (leaving the lovely native
world); and both ignore the fact that customers wish maintenance of
the entire OS actually, too. So, ideally, there is c): In a
hypothetic case we would prepare a entire OS incl. our app (maybe
via catalyst?) which would require a bootloader-like mini-OS on the
customer's side, to receive updates over the internet, switch the OS
at boot time, and fallback. I was recently playing with systemd-boot
and it's interesting try-boot feature.

Thanks

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Frederik Pfautsch - fpprogs]

[-- Attachment #1.1: Type: text/plain, Size: 2488 bytes --]

Am 04.01.23 um 23:00 schrieb m1027:
> 
> Wow, wasn't aware of catalyst at all. What a beast in terms of
> control.
> 
> (FYI: I enjoyed the links on catalyst you sent me directly.
> Unfortunatelly I cannot answer you directly due to the default TLS
> guarantee kicked in by my provider: "TLS is required, but was not
> offered by host". That's usually to be fixed on the receiving side.)
> 
> While being able to build exact environments with catalyst I wonder
> how it could help fixing the issue of my original post. To sum up:
> Whenever we need to deliver a updated app to customers whose OS is
> too old (but updating it is too risky), we could either a) keep
> evenly outdated dev build OSes around forever (oh no!), or b) post
> our newly built app in a container (leaving the lovely native
> world); and both ignore the fact that customers wish maintenance of
> the entire OS actually, too. So, ideally, there is c): In a
> hypothetic case we would prepare a entire OS incl. our app (maybe
> via catalyst?) which would require a bootloader-like mini-OS on the
> customer's side, to receive updates over the internet, switch the OS
> at boot time, and fallback. I was recently playing with systemd-boot
> and it's interesting try-boot feature.

So essentially it sounds like you want something similar to Yocto / Poky 
/ Petalinux for the non-embedded world (and based on Gentoo of course; 
it sounds like catalyst is something like that)?

Petalinux / Yocto is essentially a cross compiling environment with 
rootfs, etc. And after everything has been built, the rootfs is just 
packaged into a flashable ext4-image or something. So your devs wouldn't 
need to work on a (slightly) outdated system, just use the (in this case 
non-)cross-compiling environment with its own, separate rootfs. Is, 
e.g., crossdev able to build a non-cross, native compiler? It already 
provides a rootfs-folder and a separate emerge-config, etc...

Just throwing crazy ideas around, what about using net-boot for your 
customer? This way they just need to store an image somewhere and can 
update it whenever necessary. Or (ab-)using an initramfs. Or u-boot 
bootloader, just like in the embedded world. Depending on the size of 
the actual OS/rootfs, taking ideas from e.g. Android with their A/B 
bootslots (i.e. two root-partitions or something, where one is active 
and the other can be updated with clever scripts, after a reboot they 
are swapped).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[m1027]

frederik.pfautsch:

> > So, ideally, there is c): In a hypothetic case we would prepare
> > a entire OS incl. our app (maybe via catalyst?) which would
> > require a bootloader-like mini-OS on the customer's side, to
> > receive updates over the internet, switch the OS at boot time,
> > and fallback. I was recently playing with systemd-boot and it's
> > interesting try-boot feature.
> 
> So essentially it sounds like you want something similar to Yocto / Poky 
> / Petalinux for the non-embedded world (and based on Gentoo of course; 
> it sounds like catalyst is something like that)?

I've had a look on that: Wow, another interesting approach to build
customized OSes. Thanks!

> Just throwing crazy ideas around, what about using net-boot for your 
> customer? This way they just need to store an image somewhere and can 
> update it whenever necessary. Or (ab-)using an initramfs. Or u-boot 
> bootloader, just like in the embedded world. Depending on the size of 
> the actual OS/rootfs, taking ideas from e.g. Android with their A/B 
> bootslots (i.e. two root-partitions or something, where one is active 
> and the other can be updated with clever scripts, after a reboot they 
> are swapped).

... exactly what is on my wishlist currently! I am missing such an
alternative when in need for updating a remote (customer's) OS,
where ssh + emerge @world is just no option. If we had that, I'd see
Gentoo (e.g. with catalyst or via Yocto) shining bright here as it
is perfect in stipping down things to the required minimum.

Thanks.

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Raphaël Barrois]

On 05/01/2023 12:09, m1027 wrote:
> frederik.pfautsch:  > >>> So, ideally, there is c): In a hypothetic case we would prepare 
a >>> entire OS incl. our app (maybe via catalyst?) which would require 
 >>> a bootloader-like mini-OS on the customer's side, to receive >>> 
updates over the internet, switch the OS at boot time, and >>> fallback. 
I was recently playing with systemd-boot and it's >>> interesting 
try-boot feature. >> >> So essentially it sounds like you want something 
similar to Yocto / >> Poky / Petalinux for the non-embedded world (and 
based on Gentoo of >> course; it sounds like catalyst is something like 
that)? > > I've had a look on that: Wow, another interesting approach to 
build > customized OSes. Thanks! > >> Just throwing crazy ideas around, 
what about using net-boot for >> your customer? This way they just need 
to store an image somewhere >> and can update it whenever necessary. Or 
(ab-)using an initramfs. >> Or u-boot bootloader, just like in the 
embedded world. Depending on >> the size of the actual OS/rootfs, taking 
ideas from e.g. Android >> with their A/B bootslots (i.e. two 
root-partitions or something, >> where one is active and the other can 
be updated with clever >> scripts, after a reboot they are swapped). > > 
... exactly what is on my wishlist currently! I am missing such an > 
alternative when in need for updating a remote (customer's) OS, where > 
ssh + emerge @world is just no option. If we had that, I'd see Gentoo > 
(e.g. with catalyst or via Yocto) shining bright here as it is > perfect 
in stipping down things to the required minimum. > > Thanks.
Interesting projects in that space could be mender.io, or swupdate.

Both are based on the idea of having at least two system partitions (+ 
maybe a couple of data partitions), and downloading the update on an 
inactive partion. On next boot, the bootloader tries to boot from the 
new partition; if that fails, it will fall back to the previous (known 
working) partition after a few tries.


-- 
Xelnor

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Peter Stuge]

m1027 wrote:
> Wow, wasn't aware of catalyst at all. What a beast in terms of control.

It's not so well-known maybe because it was created by and for
gentoo-releng but if you know what you want it's a fantastic tool.


> (FYI: I enjoyed the links on catalyst you sent me directly.
> Unfortunatelly I cannot answer you directly due to the default
> TLS guarantee

Thanks! Glad you liked them. And yes, TLS is on my list.


> While being able to build exact environments with catalyst I wonder
> how it could help fixing the issue of my original post.

Thank you for connecting back to the original post! Let's see:


> Whenever we need to deliver a updated app to customers whose OS is
> too old (but updating it is too risky), we could either
> a) keep evenly outdated dev build OSes around forever (oh no!), or
> b) post our newly built app in a container (leaving the lovely native
> world); and both ignore the fact that customers wish maintenance of
> the entire OS actually, too.
> So, ideally, there is c): In a hypothetic case we would prepare a
> entire OS incl. our app (maybe via catalyst?) which would require
> a bootloader-like mini-OS on the customer's side, to receive updates
> over the internet, switch the OS at boot time, and fallback.

I had a) in mind. Why "oh no!" ? I didn't mean forever.

I think it's already a very good value proposition that you can deliver
updates of your apps for the particular systems that your customers run.

catalyst building specific, "old-like" OSes on which you build new (binpkg)
versions of your app to be installable by emerge on old customer OSes is
admittedly a lot of work, at least initially.

Separate from those app updates you can then use catalyst to also tackle
OS maintenance/updates. You can create either full milestone OSes or just
individual (binpkg) packages through which customers' OSes can become
less fragmented over time, at the pace each customer is comfortable with.
OS updates can take only small steps at a time, since catalyst allows
exact control.

This could reduce target OS diversity drastically over time with probably
relatively moderate development effort. More effort might go into holding
customer hands along the bespoke update paths.

So, I see controlled paths forward in both problem dimensions. I don't
know if the effort is actually practical for you but it /is/ doable and
most importantly it can be customized for the individual systems
currently in production at your customers.


//Peter

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[m1027]

peter:

> > Whenever we need to deliver a updated app to customers whose OS is
> > too old (but updating it is too risky), we could either
> > a) keep evenly outdated dev build OSes around forever (oh no!), or
> > b) post our newly built app in a container (leaving the lovely native
> > world); and both ignore the fact that customers wish maintenance of
> > the entire OS actually, too.
> > So, ideally, there is c): In a hypothetic case we would prepare a
> > entire OS incl. our app (maybe via catalyst?) which would require
> > a bootloader-like mini-OS on the customer's side, to receive updates
> > over the internet, switch the OS at boot time, and fallback.
> 
> I had a) in mind. Why "oh no!" ? I didn't mean forever.

Why "Oh no": Well, it works and we are currently going that way. But
see the other discussion branch in this thread: It's just a growing
maintainment mess over time. In short: Too many OS versions resp.
according dev VMs to rebuild new apps against it.


> catalyst building specific, "old-like" OSes on which you build new (binpkg)
> versions of your app to be installable by emerge on old customer OSes is
> admittedly a lot of work, at least initially.

Yes, that brings in another approach: *If* we had catalyst
controlled versions of the (already...) delivered OSes, we could
maybe setup a CI/CD like pipeline to re-create the according developer
VM (for building a updated app with the exact old dependencies as on
the formerly delivered old customer OS). So to say: VM just in time.
Well I'll keep that in mind.

That may help against the VM mess (see above). But building exact
(older) OSes still requires a solution to distribute the entire OS
incl. app remotely and savely. See also the other discussion branch
on that if you like.

Thanks

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Alec Warner]

On Mon, Jan 2, 2023 at 4:48 AM m1027 <m1027@posteo.net> wrote:
>
> Hi and happy new year.
>
> When we create apps on Gentoo they become easily incompatible for
> older Gentoo systems in production where unattended remote world
> updates are risky. This is due to new glibc, openssl-3 etc.

I wrote a very long reply, but I've removed most of it: I basically
have a few questions, and then some comments:

I don't quite grasp your problem statement, so I will repeat what I
think it is and you can confirm / deny.
  - Your devs build using gentoo synced against some recent tree, they
have recent packages, and they build some software that you deploy to
prod.
  - Your prod machines are running gentoo synced against some recent
tree, but not upgraded (maybe only glsa-check runs) and so they are
running 'old' packages because you are afraid to update them[0]
  - Your software builds OK in dev, but when you deploy it in prod it
breaks, because prod is really old, and your developments are using
packages that are too new.

My main feedback here is:
 - Your "build" environment should be like prod. You said you didn't
want to build "developer VMs" but I am unsure why. For example I run
Ubuntu and I do all my gentoo development (admittedly very little
these days)
   in a systemd-nspawn container, and I have a few shell scripts to
mount everything and set it up (so it has a tree snapshot, some git
repos, some writable space etc.)
 - Your "prod" environment is too risky to upgrade, and you have
difficulty crafting builds that run in every prod environment. I think
this is fixable by making a build environment more like the prod
environment.
    The challenge here is that if you have not done that (kept the
copies of ebuilds around, the distfiles, etc) it can be challenging to
"recreate" the existing older prod environments.
    But if you do the above thing (where devs build in a container)
and you can make that container like the prod environments, then you
can enable devs to build for the prod environment (in a container on
their local machine) and get the outcome you want.
 - Understand that not upgrading prod is like, to use a finance term,
picking up pennies in front of a steamroller. It's a great strategy,
but eventually you will actually *need* to upgrade something. Maybe
for a critical security issue, maybe for a feature. Having a build
environment that matches prod is good practice, you should do it, but
you should also really schedule maintenance for these prod nodes to
get them upgraded. (For physical machines, I've often seen businesses
just eat the risk and assume the machine will physically fail before
the steamroller comes, but this is less true with virtualized
environments that have longer real lifetimes.)

>
> So, what we've thought of so far is:
>
> (1) Keeping outdated developer boxes around and compile there. We
> would freeze portage against accidental emerge sync by creating a
> git branch in /var/db/repos/gentoo. This feels hacky and requires a
> increating number of develper VMs. And sometimes we are hit by a
> silent incompatibility we were not aware of.

In general when you build binaries for some target, you should build
on that target when possible. To me, this is the crux of your issue
(that you do not) and one of the main causes of your pain.
You will need to figure out a way to either:
 - Upgrade the older environments to new packages.
 - Build in copies of the older environments.

I actually expect the second one to take 1-2 sprints (so like 1 engineer month?)
 - One sprint to make some scripts that makes a new production 'container'
 - One sprint to sort of integrate that container into your dev
workflow, so devs build in the container instead of what they build in
now.

It might be more or less daunting depending on how many distinct
(unique?) prod environments you have (how many containers will you
actually need for good build coverage?), how experienced in Gentoo
your developers are, and how many artifacts from prod you have.
 - A few crazy ideas are like:
   - Snapshot an existing prod machine, strip of it machine-specific
bits, and use that as your container.
   - Use quickpkg to generate a bunch of bin pkgs from a prod machine,
use that to bootstrap a container.
   - Probably some other exciting ideas on the list ;)

>
> (2) Using Ubuntu LTS for production and Gentoo for development is
> hit by subtile libjpeg incompatibilites and such.

I would advise, if possible, to make dev and prod as similar as
possible[1]. I'd be curious what blockers you think there are to this
pattern.
Remember that "dev" is not "whatever your devs are using" but is
ideally some maintained environment; segmented from their daily driver
computer (somehow).

>
> (3) Distributing apps as VMs or docker: Even those tools advance and
> become incompatible, right? And not suitable when for smaller Arm
> devices.

I think if your apps are small and self-contained and easily rebuilt,
your (3) and (4) can be workable.

If you need 1000 dependencies at runtime, your containers are going to
be expensive to build, expensive to maintain, you are gonna have to
build them often (for security issues), it can be challenging to
support incremental builds and incremental updates...you generally
want a clearer problem statement to adopt this pain. Two problem
statements that might be worth it are below ;)

If you told me you had 100 different production environments, or
needed to support 12 different OSes, I'd tell you to use containers
(or similar)
If you told me you didn't control your production environment (because
users installed the software wherever) I'd tell you use containers (or
similar)

>
> (4) Flatpak: No experience, does it work well?

Flatpak is conceptually similar to your (3). I know you are basically
asking "does it work" and the answer is "probably", but see the other
questions for (3). I suspect it's less about "does it work" and more
about "is some container deployment thing really a great idea."

>
> (5) Inventing a full fledged OTA Gentoo OS updater and distribute
> that together with the apps... Nah.

This sounds like a very expensive solution that is likely rife with
very exciting security problems, fwiw.

>
> Hm... Comments welcome.
>

Peter's comment about basically running your own fork of gentoo.git
and sort of 'importing the updates' is workable. Google did this for
debian testing (called project Rodete)[2]. I can't say it's a
particularly cheap solution (significant automation and testing
required) but I think as long as you are keeping up (I would advise
never falling more than 365d behind time.now() in your fork) then I
think it provides some benefits.
  - You control when you take updates.
  - You want to stay "close" to time.now() in the tree, since a
rolling distro is how things are tested.
  - This buys you 365d or so to fix any problem you find.
  - It nominally requires that you test against ::gentoo and
::your-gentoo-fork, so you find problems in ::gentoo before they are
pulled into your fork, giving you a heads up that you need to put work
in.

[0] FWIW this is basically what #gentoo-infra does on our boxes and
it's terrible and I would not recommend it to most people in the
modern era. Upgrade your stuff regularly.
[1] When I was at Google we had a hilarious outage because someone
switched login managers (gdm vs kdm) and kdm had a different default
umask somehow? Anyway it resulted in a critical component having the
wrong permissions and it caused a massive outage (luckily we had
sufficient redundancy that it was not user visible) but it was one of
the scariest outages I had ever seen. I was in charge of investigating
(being on the dev OS team at the time) and it was definitely very
difficult to figure out "what changed" to produce the bad build. We
stopped building on developer workstations soon after, FWIW.
[2] https://cloud.google.com/blog/topics/developers-practitioners/how-google-got-to-rolling-linux-releases-for-desktops

> Thanks
>
>

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[m1027]

Many thanks for your detailed thoughs for sharing the rich
experiences on this! See below:

antarus:

> On Mon, Jan 2, 2023 at 4:48 AM m1027 <m1027@posteo.net> wrote:
> >
> > Hi and happy new year.
> >
> > When we create apps on Gentoo they become easily incompatible for
> > older Gentoo systems in production where unattended remote world
> > updates are risky. This is due to new glibc, openssl-3 etc.
> 
> I wrote a very long reply, but I've removed most of it: I basically
> have a few questions, and then some comments:
> 
> I don't quite grasp your problem statement, so I will repeat what I
> think it is and you can confirm / deny.
>
>   - Your devs build using gentoo synced against some recent tree, they
> have recent packages, and they build some software that you deploy to
> prod.

Yes.

>   - Your prod machines are running gentoo synced against some recent
> tree, but not upgraded (maybe only glsa-check runs) and so they are
> running 'old' packages because you are afraid to update them[0]

Well, we did sync (without updading packages) in the past but today we
even fear to sync against recent trees. Without going into details,
as a rule of thumb, weekly or monthly sync + package updates work
near to perfect. (It's cool to see what a good job emerge does on our
own internal production systems.) Updating systems older than 12
months or so may, however, be a hugh task. And too risky for remote
production systems of customers.


>   - Your software builds OK in dev, but when you deploy it in prod it
> breaks, because prod is really old, and your developments are using
> packages that are too new.

Exactly.


> My main feedback here is:
>  - Your "build" environment should be like prod. You said you didn't
> want to build "developer VMs" but I am unsure why. For example I run
> Ubuntu and I do all my gentoo development (admittedly very little
> these days)
>    in a systemd-nspawn container, and I have a few shell scripts to
> mount everything and set it up (so it has a tree snapshot, some git
> repos, some writable space etc.)

Okay, yes. That is way (1) I mentioned in my OP. It works indeed but
has the mentioned drawbacks: VMs and maintenance pile up, and for
each developer. And you don't know when there is the moment to
create a new VM. But yes it seems to me one of the ways to go:
*Before* creating a production system you need to freeze portage,
create dev VMs, and prevent updates on the VMs, too. (Freezing aka
not updating has many disadvantages, of course.)


>  - Your "prod" environment is too risky to upgrade, and you have
> difficulty crafting builds that run in every prod environment. I think
> this is fixable by making a build environment more like the prod
> environment.
>     The challenge here is that if you have not done that (kept the
> copies of ebuilds around, the distfiles, etc) it can be challenging to
> "recreate" the existing older prod environments.
>     But if you do the above thing (where devs build in a container)
> and you can make that container like the prod environments, then you
> can enable devs to build for the prod environment (in a container on
> their local machine) and get the outcome you want.

Not sure I got your point here. But yes, it comes down to what was
said above.


>  - Understand that not upgrading prod is like, to use a finance term,
> picking up pennies in front of a steamroller. It's a great strategy,
> but eventually you will actually *need* to upgrade something. Maybe
> for a critical security issue, maybe for a feature. Having a build
> environment that matches prod is good practice, you should do it, but
> you should also really schedule maintenance for these prod nodes to
> get them upgraded. (For physical machines, I've often seen businesses
> just eat the risk and assume the machine will physically fail before
> the steamroller comes, but this is less true with virtualized
> environments that have longer real lifetimes.)

Yes, haha, I agree. And yes, I totally ignored backporting security
here, as well as the need that we might *require* a dependend
package upgrade (e.g. to fix a known memory leak). I left that out
for simlicity only.


> > So, what we've thought of so far is:
> >
> > (1) Keeping outdated developer boxes around and compile there. We
> > would freeze portage against accidental emerge sync by creating a
> > git branch in /var/db/repos/gentoo. This feels hacky and requires a
> > increating number of develper VMs. And sometimes we are hit by a
> > silent incompatibility we were not aware of.
> 
> In general when you build binaries for some target, you should build
> on that target when possible. To me, this is the crux of your issue
> (that you do not) and one of the main causes of your pain.
> You will need to figure out a way to either:
>  - Upgrade the older environments to new packages.
>  - Build in copies of the older environments.
> 
> I actually expect the second one to take 1-2 sprints (so like 1 engineer month?)
>  - One sprint to make some scripts that makes a new production 'container'
>  - One sprint to sort of integrate that container into your dev
> workflow, so devs build in the container instead of what they build in
> now.
> 
> It might be more or less daunting depending on how many distinct
> (unique?) prod environments you have (how many containers will you
> actually need for good build coverage?), how experienced in Gentoo
> your developers are, and how many artifacts from prod you have.
>  - A few crazy ideas are like:
>    - Snapshot an existing prod machine, strip of it machine-specific
> bits, and use that as your container.
>    - Use quickpkg to generate a bunch of bin pkgs from a prod machine,
> use that to bootstrap a container.
>    - Probably some other exciting ideas on the list ;)

Thanks for the enthusiasm on it. ;-) Well:

We cannot build (develop) on that exact target. Imagine hardware
being sold to customers. They just want/need a software update of
our app.

And, unfortunatelly we don't have hardware clones of all the
different customer's hardware at ours to build, test etc.

So, we come back on the question how to have a solid LTS-like
software OS / stack onto which newly compiled developer apps can be
distributed and just work. And all this in Gentoo. :-)


> > (2) Using Ubuntu LTS for production and Gentoo for development is
> > hit by subtile libjpeg incompatibilites and such.
> 
> I would advise, if possible, to make dev and prod as similar as
> possible[1]. I'd be curious what blockers you think there are to this
> pattern.
> Remember that "dev" is not "whatever your devs are using" but is
> ideally some maintained environment; segmented from their daily driver
> computer (somehow).

That is again VMs per "release" and per dev, right? See above "way
(1)".


> > (3) Distributing apps as VMs or docker: Even those tools advance and
> > become incompatible, right? And not suitable when for smaller Arm
> > devices.
> 
> I think if your apps are small and self-contained and easily rebuilt,
> your (3) and (4) can be workable.
> 
> If you need 1000 dependencies at runtime, your containers are going to
> be expensive to build, expensive to maintain, you are gonna have to
> build them often (for security issues), it can be challenging to
> support incremental builds and incremental updates...you generally
> want a clearer problem statement to adopt this pain. Two problem
> statements that might be worth it are below ;)
> 
> If you told me you had 100 different production environments, or
> needed to support 12 different OSes, I'd tell you to use containers
> (or similar)
> If you told me you didn't control your production environment (because
> users installed the software wherever) I'd tell you use containers (or
> similar)
> 
> >
> > (4) Flatpak: No experience, does it work well?
> 
> Flatpak is conceptually similar to your (3). I know you are basically
> asking "does it work" and the answer is "probably", but see the other
> questions for (3). I suspect it's less about "does it work" and more
> about "is some container deployment thing really a great idea."

Well thanks for your comments on containers and flatpak. It's
motivating to investigate that further.

Admittedly, we've been sticking to natively built apps for reasons
that might not be relevant these days. (Hardware bound apps, bus
systems etc, performance reasons on IoT like devices, no real
experience in lean containers yet, only Qemu.)


> Peter's comment about basically running your own fork of gentoo.git
> and sort of 'importing the updates' is workable. Google did this for
> debian testing (called project Rodete)[2]. I can't say it's a
> particularly cheap solution (significant automation and testing
> required) but I think as long as you are keeping up (I would advise
> never falling more than 365d behind time.now() in your fork) then I
> think it provides some benefits.
>   - You control when you take updates.
>   - You want to stay "close" to time.now() in the tree, since a
> rolling distro is how things are tested.
>   - This buys you 365d or so to fix any problem you find.
>   - It nominally requires that you test against ::gentoo and
> ::your-gentoo-fork, so you find problems in ::gentoo before they are
> pulled into your fork, giving you a heads up that you need to put work
> in.

I haven't commented on Peter yet but yes I'll have a look on what he
added. Something tells me that distributing apps in a container
might be the cheaper way for us. We'll see.
 

> [0] FWIW this is basically what #gentoo-infra does on our boxes and
> it's terrible and I would not recommend it to most people in the
> modern era. Upgrade your stuff regularly.
> [1] When I was at Google we had a hilarious outage because someone
> switched login managers (gdm vs kdm) and kdm had a different default
> umask somehow? Anyway it resulted in a critical component having the
> wrong permissions and it caused a massive outage (luckily we had
> sufficient redundancy that it was not user visible) but it was one of
> the scariest outages I had ever seen. I was in charge of investigating
> (being on the dev OS team at the time) and it was definitely very
> difficult to figure out "what changed" to produce the bad build. We
> stopped building on developer workstations soon after, FWIW.
> [2] https://cloud.google.com/blog/topics/developers-practitioners/how-google-got-to-rolling-linux-releases-for-desktops

Thanks for sharing! Very interesting insights.

To sum up:

You described interesting ways to create and control own releases of
Gentoo. So production and developer systems could be aligned on
that. The effort depends.

Another way is containers.

Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Alec Warner]

On Mon, Jan 2, 2023 at 4:55 PM m1027 <m1027@posteo.net> wrote:
>
> Many thanks for your detailed thoughs for sharing the rich
> experiences on this! See below:
>
> antarus:
>
> > On Mon, Jan 2, 2023 at 4:48 AM m1027 <m1027@posteo.net> wrote:
> > >
> > > Hi and happy new year.
> > >
> > > When we create apps on Gentoo they become easily incompatible for
> > > older Gentoo systems in production where unattended remote world
> > > updates are risky. This is due to new glibc, openssl-3 etc.
> >
> > I wrote a very long reply, but I've removed most of it: I basically
> > have a few questions, and then some comments:
> >
> > I don't quite grasp your problem statement, so I will repeat what I
> > think it is and you can confirm / deny.
> >
> >   - Your devs build using gentoo synced against some recent tree, they
> > have recent packages, and they build some software that you deploy to
> > prod.
>
> Yes.
>
> >   - Your prod machines are running gentoo synced against some recent
> > tree, but not upgraded (maybe only glsa-check runs) and so they are
> > running 'old' packages because you are afraid to update them[0]
>
> Well, we did sync (without updading packages) in the past but today we
> even fear to sync against recent trees. Without going into details,
> as a rule of thumb, weekly or monthly sync + package updates work
> near to perfect. (It's cool to see what a good job emerge does on our
> own internal production systems.) Updating systems older than 12
> months or so may, however, be a hugh task. And too risky for remote
> production systems of customers.

My primary risk I think is that even if you ship your app in a
container you still need somewhere to run the containers. Currently
that is a fleet of different hardware and gentoo configurations, and
while containers certainly simplify your life there, they won't fix
all your problems. Now instead of worrying that upgrading your Gentoo
OS will break your app, it will instead break your container runtime.
It is likely a smaller surface area, but it is not zero. Not saying
don't use containers, just that there is no free lunch here
necessarily.

>
>
> >   - Your software builds OK in dev, but when you deploy it in prod it
> > breaks, because prod is really old, and your developments are using
> > packages that are too new.
>
> Exactly.
>
>
> > My main feedback here is:
> >  - Your "build" environment should be like prod. You said you didn't
> > want to build "developer VMs" but I am unsure why. For example I run
> > Ubuntu and I do all my gentoo development (admittedly very little
> > these days)
> >    in a systemd-nspawn container, and I have a few shell scripts to
> > mount everything and set it up (so it has a tree snapshot, some git
> > repos, some writable space etc.)
>
> Okay, yes. That is way (1) I mentioned in my OP. It works indeed but
> has the mentioned drawbacks: VMs and maintenance pile up, and for
> each developer. And you don't know when there is the moment to
> create a new VM. But yes it seems to me one of the ways to go:
> *Before* creating a production system you need to freeze portage,
> create dev VMs, and prevent updates on the VMs, too. (Freezing aka
> not updating has many disadvantages, of course.)

Oh sorry, I failed to understand you were doing that already. I agree
it's challenging, I think if you don't have a great method to simplify
here, it might not be a great avenue going forward.
 - Trying to figure out when you can make a new VM.
 - Trying to figure out when you can take a build and deploy it to a
customer safely.

I've seen folks try to group customers in some way to reduce the
number of prod artifacts required, but if you cannot it might be

The benefit of containers here is that you can basically deploy your
app at whatever rate you want, and only the OS upgrades remain risky
(because they might break the container runtime.)
Depending on your business needs, it might be advantageous to go that route.

>
>
> >  - Your "prod" environment is too risky to upgrade, and you have
> > difficulty crafting builds that run in every prod environment. I think
> > this is fixable by making a build environment more like the prod
> > environment.
> >     The challenge here is that if you have not done that (kept the
> > copies of ebuilds around, the distfiles, etc) it can be challenging to
> > "recreate" the existing older prod environments.
> >     But if you do the above thing (where devs build in a container)
> > and you can make that container like the prod environments, then you
> > can enable devs to build for the prod environment (in a container on
> > their local machine) and get the outcome you want.
>
> Not sure I got your point here. But yes, it comes down to what was
> said above.
>
>
> >  - Understand that not upgrading prod is like, to use a finance term,
> > picking up pennies in front of a steamroller. It's a great strategy,
> > but eventually you will actually *need* to upgrade something. Maybe
> > for a critical security issue, maybe for a feature. Having a build
> > environment that matches prod is good practice, you should do it, but
> > you should also really schedule maintenance for these prod nodes to
> > get them upgraded. (For physical machines, I've often seen businesses
> > just eat the risk and assume the machine will physically fail before
> > the steamroller comes, but this is less true with virtualized
> > environments that have longer real lifetimes.)
>
> Yes, haha, I agree. And yes, I totally ignored backporting security
> here, as well as the need that we might *require* a dependend
> package upgrade (e.g. to fix a known memory leak). I left that out
> for simlicity only.

Ahh my worry is that the easy parts are easy and the edge cases are
what really makes things intractable here.

>
>
> > > So, what we've thought of so far is:
> > >
> > > (1) Keeping outdated developer boxes around and compile there. We
> > > would freeze portage against accidental emerge sync by creating a
> > > git branch in /var/db/repos/gentoo. This feels hacky and requires a
> > > increating number of develper VMs. And sometimes we are hit by a
> > > silent incompatibility we were not aware of.
> >
> > In general when you build binaries for some target, you should build
> > on that target when possible. To me, this is the crux of your issue
> > (that you do not) and one of the main causes of your pain.
> > You will need to figure out a way to either:
> >  - Upgrade the older environments to new packages.
> >  - Build in copies of the older environments.
> >
> > I actually expect the second one to take 1-2 sprints (so like 1 engineer month?)
> >  - One sprint to make some scripts that makes a new production 'container'
> >  - One sprint to sort of integrate that container into your dev
> > workflow, so devs build in the container instead of what they build in
> > now.
> >
> > It might be more or less daunting depending on how many distinct
> > (unique?) prod environments you have (how many containers will you
> > actually need for good build coverage?), how experienced in Gentoo
> > your developers are, and how many artifacts from prod you have.
> >  - A few crazy ideas are like:
> >    - Snapshot an existing prod machine, strip of it machine-specific
> > bits, and use that as your container.
> >    - Use quickpkg to generate a bunch of bin pkgs from a prod machine,
> > use that to bootstrap a container.
> >    - Probably some other exciting ideas on the list ;)
>
> Thanks for the enthusiasm on it. ;-) Well:
>
> We cannot build (develop) on that exact target. Imagine hardware
> being sold to customers. They just want/need a software update of
> our app.
>
> And, unfortunatelly we don't have hardware clones of all the
> different customer's hardware at ours to build, test etc.

Ahh sorry, I meant mostly the software configuration here (my
apologies). It sounds like above you are doing that already and are
finding that keeping numerous software configurations (VMs) around is
too costly.
If that is the case it sounds like containers, flatpak, or snap
packages could be the way to go (the last one only if your prod
environment is systemd compatible.)

>
> So, we come back on the question how to have a solid LTS-like
> software OS / stack onto which newly compiled developer apps can be
> distributed and just work. And all this in Gentoo. :-)
>
>
> > > (2) Using Ubuntu LTS for production and Gentoo for development is
> > > hit by subtile libjpeg incompatibilites and such.
> >
> > I would advise, if possible, to make dev and prod as similar as
> > possible[1]. I'd be curious what blockers you think there are to this
> > pattern.
> > Remember that "dev" is not "whatever your devs are using" but is
> > ideally some maintained environment; segmented from their daily driver
> > computer (somehow).
>
> That is again VMs per "release" and per dev, right? See above "way
> (1)".

At a previous job we built some scripts to build a VM per release; but
in our scheme we only had to build 9 VMs worst case (3 targets, and 3
OS tracks, so 9 total.) We shared the base VM images (per release)
with the entire development team of 9 people. It was feasible with
decent internet (100mbit). We had some shared storage we put signed
images on. But often you would only use 1 image to test locally, then
push to the CI pipeline that would test on the other 8 images (because
it was cheaper / whatever in the datacenter.)

I continue to agree with you in that if you can't get your # of
targets down near that kind of number (10-20ish) it's probably not
going to be a great time for you.


>
>
> > > (3) Distributing apps as VMs or docker: Even those tools advance and
> > > become incompatible, right? And not suitable when for smaller Arm
> > > devices.
> >
> > I think if your apps are small and self-contained and easily rebuilt,
> > your (3) and (4) can be workable.
> >
> > If you need 1000 dependencies at runtime, your containers are going to
> > be expensive to build, expensive to maintain, you are gonna have to
> > build them often (for security issues), it can be challenging to
> > support incremental builds and incremental updates...you generally
> > want a clearer problem statement to adopt this pain. Two problem
> > statements that might be worth it are below ;)
> >
> > If you told me you had 100 different production environments, or
> > needed to support 12 different OSes, I'd tell you to use containers
> > (or similar)
> > If you told me you didn't control your production environment (because
> > users installed the software wherever) I'd tell you use containers (or
> > similar)
> >
> > >
> > > (4) Flatpak: No experience, does it work well?
> >
> > Flatpak is conceptually similar to your (3). I know you are basically
> > asking "does it work" and the answer is "probably", but see the other
> > questions for (3). I suspect it's less about "does it work" and more
> > about "is some container deployment thing really a great idea."
>
> Well thanks for your comments on containers and flatpak. It's
> motivating to investigate that further.
>
> Admittedly, we've been sticking to natively built apps for reasons
> that might not be relevant these days. (Hardware bound apps, bus
> systems etc, performance reasons on IoT like devices, no real
> experience in lean containers yet, only Qemu.)

Depending on your app, you can get pretty lean containers. We have a
golang app (https://gitweb.gentoo.org/sites/soko.git/tree/Dockerfile)
whose docker image is 39MB, but it mostly just has a large statically
compiled go-binary in it.  We run gitlab-ce in a large container that
is 2.5GB, so the sizes can definitely get large if you are not
careful.

Another potential issue for containers is the container runtime shares
a kernel with the host, so if your host kernel is very old, but you
need new kernel features (or syscalls) they may be missing on the host
kernel, so there are still some gotchas (but as mentioned, probably
fewer than you experience with a full OS build.)

Good Luck!

-A

>
>
> > Peter's comment about basically running your own fork of gentoo.git
> > and sort of 'importing the updates' is workable. Google did this for
> > debian testing (called project Rodete)[2]. I can't say it's a
> > particularly cheap solution (significant automation and testing
> > required) but I think as long as you are keeping up (I would advise
> > never falling more than 365d behind time.now() in your fork) then I
> > think it provides some benefits.
> >   - You control when you take updates.
> >   - You want to stay "close" to time.now() in the tree, since a
> > rolling distro is how things are tested.
> >   - This buys you 365d or so to fix any problem you find.
> >   - It nominally requires that you test against ::gentoo and
> > ::your-gentoo-fork, so you find problems in ::gentoo before they are
> > pulled into your fork, giving you a heads up that you need to put work
> > in.
>
> I haven't commented on Peter yet but yes I'll have a look on what he
> added. Something tells me that distributing apps in a container
> might be the cheaper way for us. We'll see.
>
>
> > [0] FWIW this is basically what #gentoo-infra does on our boxes and
> > it's terrible and I would not recommend it to most people in the
> > modern era. Upgrade your stuff regularly.
> > [1] When I was at Google we had a hilarious outage because someone
> > switched login managers (gdm vs kdm) and kdm had a different default
> > umask somehow? Anyway it resulted in a critical component having the
> > wrong permissions and it caused a massive outage (luckily we had
> > sufficient redundancy that it was not user visible) but it was one of
> > the scariest outages I had ever seen. I was in charge of investigating
> > (being on the dev OS team at the time) and it was definitely very
> > difficult to figure out "what changed" to produce the bad build. We
> > stopped building on developer workstations soon after, FWIW.
> > [2] https://cloud.google.com/blog/topics/developers-practitioners/how-google-got-to-rolling-linux-releases-for-desktops
>
> Thanks for sharing! Very interesting insights.
>
> To sum up:
>
> You described interesting ways to create and control own releases of
> Gentoo. So production and developer systems could be aligned on
> that. The effort depends.
>
> Another way is containers.
>
>