[gentoo-portage-dev] How to have several gentoo repos on one machine?

[gentoo-portage-dev] How to have several gentoo repos on one machine?

[Joakim Tjernlund]

I need to more than one gentoo repo in my computer.
So I add to repos.conf:
 [tm-cusfpv3]
 auto-sync = yes
 sync-type = rsync
 sync-uri = rsync://devsrv.transmode.se/tm-cusfpv3
 location = /usr/local/portage/tm-cusfpv3

this did not work as "portageq repositories_configuration /" complains:
!!! Section 'tm-cusfpv3' in repos.conf has name different from repository name 'gentoo' set inside repository

I figured the name in repos.conf would just override /usr/local/portage/tm-cusfpv3/profiles/repo_name ?

 Jocke

[gentoo-portage-dev] Re: How to have several gentoo repos on one machine?

[Duncan]

Joakim Tjernlund posted on Wed, 21 Oct 2015 11:08:02 +0000 as excerpted:

> I need to more than one gentoo repo in my computer.
> 
> So I add to repos.conf:
>  [tm-cusfpv3]
>  auto-sync = yes
>  sync-type = rsync
>  sync-uri = rsync://devsrv.transmode.se/tm-cusfpv3
>  location = /usr/local/portage/tm-cusfpv3
> 
> this did not work as "portageq repositories_configuration /" complains:
> !!! Section 'tm-cusfpv3' in repos.conf has name different from
> repository name 'gentoo' set inside repository
> 
> I figured the name in repos.conf would just override
> /usr/local/portage/tm-cusfpv3/profiles/repo_name ?

While it's not quite clear to me either why you'd need two identical 
gentoo repos (and if they're not identical, why is the non-gentoo-
official mirror still using the gentoo name?) or exactly what this config-
line does, the aliases= attribute, along with force=aliases, in 
repos.conf, may be what you're looking for.

See the portage (5) manpage, repos.conf section, attributes supported in 
sections of repositories subsection, under aliases and force.  
Unfortunately, the description for aliases is anything but clear, tho the 
usage (including comment) further down in the example subsection does 
help some.

If that doesn't help, then while the portage devs may have some other 
suggestions, the workaround that occurs to me is to use rsync's exclude/
filter options, so rsync ignores that file and doesn't sync it.  I do[1] 
that with a few custom files/dirs that I don't want synced, and rsync 
ignores them just as I told it to. =:^)

See the rsync (1) manpage, --exclude, --exclude-from, --include, --
include-from and --filter=RULE options, as well as the filter rules, 
include/exclude pattern rules, and anchoring include/exclude patterns 
sections.  However, be prepared to spend a bit of time studying, as these 
options are very powerful/flexible/configurable and thus take some time 
to figure out.

Once you have rsync ignoring the repo_name file, you can rename it as you 
like.

However, do be aware that (as the repos.conf force option docs mention) 
messing with this is very likely to invalidate the pre-generated metadata 
cache, and if you don't regenerate it (egencache), portage will take a 
*VERY* long time figuring stuff out, *MUCH* longer than usual, as it 
won't have the benefit of the metadata cache for that repo.

Which again has me asking why you need two separate gentoo repos.  Either 
they're identical and the one should suffice, or the unofficial one 
should be named something other than gentoo.  In fact, at least in 
theory, in addition to all the headaches not using a different name is 
forcing on users, that's potentially trademark violation if it's publicly 
available, different from the official gentoo mirror, and yet still 
calling itself gentoo.

---
[1] rsync exclude options: I actually use gentoo's git-based usersync repo 
on github, now, and thus don't rsync any repos all any more, here, and 
git of course has its git-ignore feature/files, which I use now.  But I 
used rsync's exclude as suggested above, for years.  Worked fine. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Re: How to have several gentoo repos on one machine?

[Joakim Tjernlund]

On Thu, 2015-10-22 at 02:29 +0000, Duncan wrote:
> Joakim Tjernlund posted on Wed, 21 Oct 2015 11:08:02 +0000 as excerpted:
> 
> > I need to more than one gentoo repo in my computer.
> > 
> > So I add to repos.conf:
> >  [tm-cusfpv3]
> >  auto-sync = yes
> >  sync-type = rsync
> >  sync-uri = rsync://devsrv.transmode.se/tm-cusfpv3
> >  location = /usr/local/portage/tm-cusfpv3
> > 
> > this did not work as "portageq repositories_configuration /" complains:
> > !!! Section 'tm-cusfpv3' in repos.conf has name different from
> > repository name 'gentoo' set inside repository
> > 
> > I figured the name in repos.conf would just override
> > /usr/local/portage/tm-cusfpv3/profiles/repo_name ?
> 
> While it's not quite clear to me either why you'd need two identical 
> gentoo repos (and if they're not identical, why is the non-gentoo-
> official mirror still using the gentoo name?) or exactly what this config-
> line does, the aliases= attribute, along with force=aliases, in 
> repos.conf, may be what you're looking for.

I use one for my host and the other for cross building our products root FS
and they are not in sync. That rules out the aliases I guess?

> 
> See the portage (5) manpage, repos.conf section, attributes supported in 
> sections of repositories subsection, under aliases and force.  
> Unfortunately, the description for aliases is anything but clear, tho the 
> usage (including comment) further down in the example subsection does 
> help some.
> 
> If that doesn't help, then while the portage devs may have some other 
> suggestions, the workaround that occurs to me is to use rsync's exclude/
> filter options, so rsync ignores that file and doesn't sync it.  I do[1] 
> that with a few custom files/dirs that I don't want synced, and rsync 
> ignores them just as I told it to. =:^)

Right, I will test this but I consider it an workaround and it will only
work with rsync.

> 
> See the rsync (1) manpage, --exclude, --exclude-from, --include, --
> include-from and --filter=RULE options, as well as the filter rules, 
> include/exclude pattern rules, and anchoring include/exclude patterns 
> sections.  However, be prepared to spend a bit of time studying, as these 
> options are very powerful/flexible/configurable and thus take some time 
> to figure out.
> 
> Once you have rsync ignoring the repo_name file, you can rename it as you 
> like.

I don't plan on renaming anything in the repo_name file, it should just
be ignored and the name I have select in repos.conf should used.

I don't see any value in repo_name file now that we have the new repos.conf,
possibly it could be a fallback only for PORTDIR users.

> 
> However, do be aware that (as the repos.conf force option docs mention) 
> messing with this is very likely to invalidate the pre-generated metadata 
> cache, and if you don't regenerate it (egencache), portage will take a 
> *VERY* long time figuring stuff out, *MUCH* longer than usual, as it 
> won't have the benefit of the metadata cache for that repo.

OK, will have to look into that. Our main use will be binary pks only and
even this is slow. emerge is usling a lot of time to figure out what to
update even when using binary packets only(even when PKGDIR is empty or almost empty)

> 
> Which again has me asking why you need two separate gentoo repos.  Either 
> they're identical and the one should suffice, or the unofficial one 
> should be named something other than gentoo.  In fact, at least in 
> theory, in addition to all the headaches not using a different name is 
> forcing on users, that's potentially trademark violation if it's publicly 
> available, different from the official gentoo mirror, and yet still 
> calling itself gentoo.
> 
> ---
> [1] rsync exclude options: I actually use gentoo's git-based usersync repo 
> on github, now, and thus don't rsync any repos all any more, here, and 
> git of course has its git-ignore feature/files, which I use now.  But I 
> used rsync's exclude as suggested above, for years.  Worked fine. =:^)

Nice, I am heading the same was, using git all the way but I not there yet.
One problem is that using git is disk space I think. Files are just
ignored but still present in the repo so syncing to our embedded target
will take a lot more space. 
Any thoughts on that?

 Jocke

[gentoo-portage-dev] Re: How to have several gentoo repos on one machine?

[Duncan]

Joakim Tjernlund posted on Thu, 22 Oct 2015 06:48:06 +0000 as excerpted:

> On Thu, 2015-10-22 at 02:29 +0000, Duncan wrote:
>> Joakim Tjernlund posted on Wed, 21 Oct 2015 11:08:02 +0000 as
>> excerpted:
>> 
>> > I need to more than one gentoo repo in my computer.


>> > this did not work as "portageq repositories_configuration /"
>> > complains:
>> > !!! Section 'tm-cusfpv3' in repos.conf has name different from
>> > repository name 'gentoo' set inside repository
>> > 
>> > I figured the name in repos.conf would just override
>> > /usr/local/portage/tm-cusfpv3/profiles/repo_name ?
>> 
>> While it's not quite clear to me either why you'd need two identical
>> gentoo repos[...]
> 
> I use one for my host and the other for cross building our products root
> FS and they are not in sync. That rules out the aliases I guess?

I think so, yes.  However, as a user I'd really like to understand 
aliases, their purpose, and at high level how they work, and the current 
manpage doesn't help so much there.  Without that I really don't know 
enough about aliases to say anything further.

But meanwhile, I was sort of in your situation for awhile as I was 
building for my main amd64 system and in a 32-bit chroot for a 32-bit-
only netbook, with a separate portage config for each, and while in my 
case they both pointed at the same gentoo repo and overlays using bind-
mounts into the 32-bit chroot, without those bind-mounts it would have 
been two parallel and separate portage installations, one configured for 
32-bit x86 in the chroot, one configured for amd64 outside the chroot.

And that's what I'd use in your case, two separate portage installations, 
which could then of course have separate configs.

That said, while I understand the principle of stability, and if it's 
private there shouldn't be legal issues, I still wonder at the idea.  One 
of the reasons I could and did use bind-mounts and thus literally the 
same repos in my case, was that the gentoo repo is the gentoo repo, and 
other than the possibility of snapshotting it for archiving purposes (and 
of using one of those snapshots should it be needed, say because I left 
the netbook unupgraded for too long and it could no longer jump from the 
version on it to current), I considered the gentoo repo the gentoo repo, 
and a local copy that wasn't synced would no longer represent the present 
state of the gentoo repo.

If I were to un-sync for other than very temporary recovery purposes, I'd 
thus want to call the repo something other than gentoo, since it would no 
longer represent the current state of the true gentoo repo.

And if I made changes to that unsynced repo, say to stabilize it further 
(and if I wasn't doing so, what would be the purpose of keeping it 
unsynced for so long), that'd be even /more/ reason to call it something 
other than gentoo, because then it would no longer properly represent 
that state of the true gentoo repo at /any/ time.

But having the git repo available changes the way that works 
dramatically, see below...

> I don't plan on renaming anything in the repo_name file, it should just
> be ignored and the name I have select in repos.conf should used.
> 
> I don't see any value in repo_name file now that we have the new
> repos.conf, possibly it could be a fallback only for PORTDIR users.

The portage devs are welcome to contradict me if they like, but AFAIK, it 
still serves the useful purpose of double-checking that you don't for 
instance have two repos accidently syncing to the same place, and that 
the names used to refer to the repo stay consistent.  (Again, part of the 
need for consistency would be due to the metadata and thus metadata cache 
being repo-specific, automatically invalidating the cache if the remote 
name and local name don't agree.  Locally regenerating the metadata cache 
will go a long way to avoiding that problem, but it's an expensive 
operation that most users won't want to do, and keeping the names in sync 
helps avoid inadvertent cache invalidation.)

>> I actually use gentoo's git-based usersync
>> repo on github, now, and thus don't rsync any repos all any more, here,
>> and git of course has its git-ignore feature/files, which I use now. 
>> But I used rsync's exclude as suggested above, for years.  Worked fine.
>> =:^)
> 
> Nice, I am heading the same was, using git all the way but I not there
> yet.
> One problem is that using git is disk space I think. Files are just
> ignored but still present in the repo so syncing to our embedded target
> will take a lot more space.
> Any thoughts on that?

Well, at least once your trailing target (presumably the embedded repo) 
is safely past the git repo's epoc (the date imported from cvs, for our 
purposes), git flexibility will let you checkout older versions on-
demand, then checkout HEAD once again.

In a scenario where both copies aren't likely to be used at once, you can 
use a single local git repo and just checkout the version of it you want 
dynamically.

In a concurrent-use scenario, there's a few ways you could go.  What I'd 
probably do would be two git repos, one synced to gentoo-remote, 
presumably with full git history (or at least git history back to the 
other checkout), the other locally checked out from the "current" repo, 
at the checkout of interest.

If you're doing this sort of thing then the sort of space the git repo 
takes up shouldn't be a big concern, but in case it is, it's worth noting 
that given the right filesystem and dedup tools, there will only actually 
be the one copy of "common" data on-storage, with each of those two git 
repos reflinking (think a lower-level hard-link) data that's common 
between them, which will be pretty much everything in the earlier one 
since the current one will have the earlier one as history.

I'm a regular on the btrfs list, for instance, and on btrfs, a very space 
efficient solution would be to originally do an initial git checkout of 
the older, presumably embedded target repo, create a btrfs snapshot out 
of it, and then (in the working copy, not the snapshot) git-pull from the 
remote to update to current.  The btrfs snapshot will have locked in 
place the older version in the snapshot, while the git pull in the 
working copy will create any new files, delete any remote-deleted ones 
(but they'll still be in the btrfs snapshot), reflink any old files, and 
reflink but then cow (copy-on-write) any updated files.  For this 
scenario you wouldn't even need any additional dedup tools, tho if you 
had them, they'd probably save even more space (multiple versions of the 
same package often have very nearly the same ebuilds, for instance, 
differing in little more than name, and dedup would catch and dedup these 
as well, while the pure native btrfs snapshot method probably wouldn't).

Of course I'm conservative enough that I only call btrfs "stabilizing and 
maturing, but not fully stable or mature yet", for various reasons you'll 
see enumerated in my posts on the btrfs list, but if you're following the 
standard sysadmin backup rule, if it's not backed up, by definition you 
value it less than the time/resource necessary for doing the backup, 
factored against the risk of actually needing the backup (thus nicely 
dealing with second and third and Nth level backups as well, since the 
risk of actually needing them drops accordingly, but they may well be 
worth it out to some higher value of N for very highly valued data), then 
in general I and others have found it stable /enough/.

I guess xfs and ext4 both have dedup features as well, but I went 
straight from reiserfs to btrfs and am thus not really familiar with 
them.  (And zfs of course is the more mature btrfs, but there's some down 
sides like needing loads of ecc-strongly-recommended ram, as well as 
license concerns for people like me, that may eliminate it from 
consideration even if it'd otherwise really be a stable and reliable 
version of where btrfs is still headed, but hasn't yet arrived.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] Re: How to have several gentoo repos on one machine?

[Joakim Tjernlund]

On Thu, 2015-10-22 at 11:26 +0000, Duncan wrote:
> Joakim Tjernlund posted on Thu, 22 Oct 2015 06:48:06 +0000 as excerpted:
> 
> > On Thu, 2015-10-22 at 02:29 +0000, Duncan wrote:
> > > Joakim Tjernlund posted on Wed, 21 Oct 2015 11:08:02 +0000 as
> > > excerpted:
> > > 
> > > > I need to more than one gentoo repo in my computer.
> 
> 
> > > > this did not work as "portageq repositories_configuration /"
> > > > complains:
> > > > !!! Section 'tm-cusfpv3' in repos.conf has name different from
> > > > repository name 'gentoo' set inside repository
> > > > 
> > > > I figured the name in repos.conf would just override
> > > > /usr/local/portage/tm-cusfpv3/profiles/repo_name ?
> > > 
> > > While it's not quite clear to me either why you'd need two identical
> > > gentoo repos[...]
> > 
> > I use one for my host and the other for cross building our products root
> > FS and they are not in sync. That rules out the aliases I guess?
> 
> I think so, yes.  However, as a user I'd really like to understand 
> aliases, their purpose, and at high level how they work, and the current 
> manpage doesn't help so much there.  Without that I really don't know 
> enough about aliases to say anything further.
> 
> But meanwhile, I was sort of in your situation for awhile as I was 
> building for my main amd64 system and in a 32-bit chroot for a 32-bit-
> only netbook, with a separate portage config for each, and while in my 
> case they both pointed at the same gentoo repo and overlays using bind-
> mounts into the 32-bit chroot, without those bind-mounts it would have 
> been two parallel and separate portage installations, one configured for 
> 32-bit x86 in the chroot, one configured for amd64 outside the chroot.
> 
> And that's what I'd use in your case, two separate portage installations, 
> which could then of course have separate configs.

I don't need separate portage installations but I do need separate configs and
separate gentoo ebuild repos as the embedded target has a less frequent update
cycle and I want to be able to rebuild the exact same pkg at any time.

> 
> That said, while I understand the principle of stability, and if it's 
> private there shouldn't be legal issues, I still wonder at the idea.  One 
> of the reasons I could and did use bind-mounts and thus literally the 
> same repos in my case, was that the gentoo repo is the gentoo repo, and 
> other than the possibility of snapshotting it for archiving purposes (and 
> of using one of those snapshots should it be needed, say because I left 
> the netbook unupgraded for too long and it could no longer jump from the 
> version on it to current), I considered the gentoo repo the gentoo repo, 
> and a local copy that wasn't synced would no longer represent the present 
> state of the gentoo repo.
> 
> If I were to un-sync for other than very temporary recovery purposes, I'd 
> thus want to call the repo something other than gentoo, since it would no 
> longer represent the current state of the true gentoo repo.
> 
> And if I made changes to that unsynced repo, say to stabilize it further 
> (and if I wasn't doing so, what would be the purpose of keeping it 
> unsynced for so long), that'd be even /more/ reason to call it something 
> other than gentoo, because then it would no longer properly represent 
> that state of the true gentoo repo at /any/ time.
> 
> But having the git repo available changes the way that works 
> dramatically, see below...
> 
> > I don't plan on renaming anything in the repo_name file, it should just
> > be ignored and the name I have select in repos.conf should used.
> > 
> > I don't see any value in repo_name file now that we have the new
> > repos.conf, possibly it could be a fallback only for PORTDIR users.
> 
> The portage devs are welcome to contradict me if they like, but AFAIK, it 
> still serves the useful purpose of double-checking that you don't for 
> instance have two repos accidently syncing to the same place, and that 
> the names used to refer to the repo stay consistent.  (Again, part of the 
> need for consistency would be due to the metadata and thus metadata cache 
> being repo-specific, automatically invalidating the cache if the remote 
> name and local name don't agree.  Locally regenerating the metadata cache 
> will go a long way to avoiding that problem, but it's an expensive 
> operation that most users won't want to do, and keeping the names in sync 
> helps avoid inadvertent cache invalidation.)

In any case, the use of repo_name must be the same for all tools, now emerge
and portageq are different.

> 
> > > I actually use gentoo's git-based usersync
> > > repo on github, now, and thus don't rsync any repos all any more, here,
> > > and git of course has its git-ignore feature/files, which I use now. 
> > > But I used rsync's exclude as suggested above, for years.  Worked fine.
> > > =:^)
> > 
> > Nice, I am heading the same was, using git all the way but I not there
> > yet.
> > One problem is that using git is disk space I think. Files are just
> > ignored but still present in the repo so syncing to our embedded target
> > will take a lot more space.
> > Any thoughts on that?
> 
> Well, at least once your trailing target (presumably the embedded repo) 
> is safely past the git repo's epoc (the date imported from cvs, for our 
> purposes), git flexibility will let you checkout older versions on-
> demand, then checkout HEAD once again.
> 
> In a scenario where both copies aren't likely to be used at once, you can 
> use a single local git repo and just checkout the version of it you want 
> dynamically.

Hmm, that is an good ide but for now I do want to keep them separate as
the host repo is managed by IT and the other is R&D and having to coordinate
this common repo with IT is not something I want ATM.

My space concerns are a bit different, currently I exclude lots of
categories from the embedded target, (No X11, games etc.) and I do
this using rsync exclude on the server side.

> 
> In a concurrent-use scenario, there's a few ways you could go.  What I'd 
> probably do would be two git repos, one synced to gentoo-remote, 
> presumably with full git history (or at least git history back to the 
> other checkout), the other locally checked out from the "current" repo, 
> at the checkout of interest.
> 
> If you're doing this sort of thing then the sort of space the git repo 
> takes up shouldn't be a big concern, but in case it is, it's worth noting 
> that given the right filesystem and dedup tools, there will only actually 
> be the one copy of "common" data on-storage, with each of those two git 
> repos reflinking (think a lower-level hard-link) data that's common 
> between them, which will be pretty much everything in the earlier one 
> since the current one will have the earlier one as history.
> 
> I'm a regular on the btrfs list, for instance, and on btrfs, a very space 
> efficient solution would be to originally do an initial git checkout of 
> the older, presumably embedded target repo, create a btrfs snapshot out 
> of it, and then (in the working copy, not the snapshot) git-pull from the 
> remote to update to current.  The btrfs snapshot will have locked in 
> place the older version in the snapshot, while the git pull in the 
> working copy will create any new files, delete any remote-deleted ones 
> (but they'll still be in the btrfs snapshot), reflink any old files, and 
> reflink but then cow (copy-on-write) any updated files.  For this 
> scenario you wouldn't even need any additional dedup tools, tho if you 
> had them, they'd probably save even more space (multiple versions of the 
> same package often have very nearly the same ebuilds, for instance, 
> differing in little more than name, and dedup would catch and dedup these 
> as well, while the pure native btrfs snapshot method probably wouldn't).

We use BTRFS too :) this is the perfect FS for safe upgrades and rollback.
However, we create a btrfs snapshot and do the upgrades into the snapshot.
Once we a happy we set default subvolume to the snapshot and reboot.

We feel this is much safer than upgrading into the running system, much can go
wrong halfway through an upgrade and our systems sits in the bush somewhere.
Even if the upgrade goes well, you may end up with a system you cannot login to
if pam was upgraded until you reboot.

 Jocke