Re: [Newsletter] Re: [gentoo-catalyst] [PATCH 2/2] Move from PORTDIR_OVERLAY to repos.conf

[Matt Turner <mattst88@gentoo.org>]

On Sun, Oct 18, 2020 at 9:58 AM Felix Bier <Felix.Bier@rohde-schwarz.com> wrote:
>
> Am Samstag, den 17.10.2020, 13:11 -0700 schrieb Matt Turner:
> > On Sat, Oct 17, 2020 at 12:00 PM Felix Bier
> > <Felix.Bier@rohde-schwarz.com> wrote:
> > >
> > > This commit fixes the following issues:
> > >
> > >   * The PORTDIR_OVERLAY variable has been deprecated by Gentoo.
> > >
> > >     With this commit, the variable is no longer written to the
> > >     generated make.conf. Instead, a config file
> > >     /etc/portage/repos.conf/<repo-name>.conf
> > >     is generated for each overlay. The repo name is read from the
> > >     overlay using the portage API. Internally, portage parses
> > >     metadata/layout.conf and profiles/repo_name to obtain the name.
> > >
> > >     References:
> > >     https://wiki.gentoo.org/wiki//etc/portage/make.conf
> > >     https://wiki.gentoo.org/wiki//etc/portage/repos.conf
> > >
> > >   * All overlays were copied into the same target directory. If the
> > >     same file name occurred in multiple overlays, the last overlay
> > >     would overwrite all previous files with this name. In
> > > particular,
> > >     only the metadata/layout.conf of the last overlay was retained,
> > >     so it was not possible to reference the other overlays e.g. via
> > >     the masters entry in the layout.conf or the portage-2 syntax
> > >     for specifying a parent profile from another overlay. Also,
> > >     this created problems when the overlays contained ebuilds
> > >     for the same package, but with differing versions, because
> > >     after copying, the target directory contained both versions of
> > > the
> > >     ebuild but only the manifest file of the last overlay.
> > >
> > >     With this commit, each overlay is copied into a separate
> > >     sub-directory, e.g. /var/gentoo/repos/local/<repo-name>/.
> > >     This directory is referenced via the location entry in the
> > >     generated /etc/portage/repos.conf/<repo-name>.conf.
> > > ---
> >
> > Hello,
> >
> > Thank you for the patches. I'm happy to see them.
> >
> > I cannot apply the patches however. This one in particular is badly
> > line wrapped by your mail client. I tried fixing it up, but either
> > failed or don't know what this is supposed to apply to.
> >
> > It looks like the intention is for this to apply to the
> > catalyst-3.0-stable branch since it discusses copying overlays into a
> > subdirectory, rather than any mention of squashfs snapshots.
> >
> > I really don't want new feature work on the catalyst-3.0-stable
> > branch, for example. If that is the target, then please consider
> > rebasing the work onto the master branch.
> >
> > Matt
>
> Hello Matt,
> thank you for the initial review.
>
> I'm sorry for the formatting issues, I will check my mail client's
> format settings and resend the patches.
>
> The patches are intendend for the master branch. I agree that this
> feature should not be in the stable branch.
>
> My understanding is that in the master branch, the squashfs is only
> used for the main repository, whereas the overlays are still copied as
> directories (in the StageBase.portage_overlay() function).
>
> This patch only removes PORTDIR_OVERLAY, not PORTDIR, so it should not
> be a problem that the main repository is squashed. I kept the helper
> functions that this patch adds general, so that they can be reused for
> a later PORTDIR removal, but I would prefer to do that in a separate
> step.

Right. I guess I was expecting a transition from PORTDIR +
PORTDIR_OVERLAY to the modern just-a-bunch-of-repos repos.conf system.
That's totally fine to not make the leap all at once.

There's actually a patch in https://bugs.gentoo.org/560518 that
creates repos.conf, if you're interested in any follow on work :)

> I don't know if there are plans to also switch the overlays to
> squashfs. My intuition would be that the overlays would usually be much
> smaller than the main repository, so squashing them would have less
> benefit. Even if there is a plan to switch the overlays to squashfs (or
> bind-mounting, which I would prefer), I think my patch would still be
> helpful for that, since it would not be possible to create multiple
> mounts with the same target directory (such as /var/db/repo/local). So
> creating subdirectories (/var/db/repo/local/overlay1,
> /var/db/repo/local/overlay2 etc.) would still be needed.

An additional advantage of squashfs snapshots is that they're
snapshots. In that way, we should be able to reproduce a build from
exactly the same inputs. You're right that the other advantages of
squashfs snapshots (smaller size, not copying 1GB of small files,
quick cleanup, etc) don't really apply to the same degree to small
overlays.