[gentoo-dev] Current portage will now truncate your repo's git history to 1

[gentoo-dev] Current portage will now truncate your repo's git history to 1

[Florian Schmaus]

This is a public service announcement that the recently stabilized 
portage version will truncate you repo's git history to 1.

While this is a good thing for the majority of Gentoo users, it affects 
developers that develop in a git known to portage (like me). If I 
understand the portage maintainers vision correctly, then future portage 
will assume full control over its configured repositories and 
potentially perform destructive operations on them, for example "git 
clean" [1].

I personally would prefer portage simply adjusting its behavior based on 
the owner of the repository. That is, if it's the 'portage' user then 
assume full control, and if it is a different user, then fall back to a 
preserving, conservative mode of operation. Unfortunately, for me, this 
idea was received skeptically at best in a recent discussion in 
#gentoo-portage.

So this is a heads up for fellow developers using a similar workflow 
like me, that they are probably required to change their workflow to use 
PORTDIR_OVERLAY and multiple repositories on their system: a 
system-wide, managed by portage, and a dev repository (in your HOME), 
scoped in via PORTDIR_OVERLAY.

As everyone knows, there is nothing better than to change the workflow 
that has served you well over multiple years. But apparently the 
PORTDIR_OVERLAY approach works well for others, so I am confident that I 
(and others) will be able to make the transition with a minimal amount 
of ranting. ;)

- Flow



1: https://github.com/gentoo/portage/pull/939

[gentoo-dev] Re: Current portage will now truncate your repo's git history to 1

[Florian Schmaus]

On 15/12/2022 20.22, Florian Schmaus wrote:
> As everyone knows, there is nothing better than to change the workflow 
> that has served you well over multiple years. But apparently the 
> PORTDIR_OVERLAY approach works well for others, so I am confident that I 
> (and others) will be able to make the transition with a minimal amount 
> of ranting. ;)

As it was pointed out that some consider my wording here inappropriate 
and/or rude, I want to apologize to everyone who was offended by the 
prior paragraph. I had

     https://xkcd.com/1172/

in mind when writing it.

I was also criticized for not providing a rationale for the behavior 
change in portage. The motivation is driven by the noble goal to make 
portage more user friendly and robust. Which I fully support. Just like 
I support shallow clones per default. I just believe portage should do a 
better job figuring out if it potentially messes with a "user-owned" 
repository and adjust its behavior accordingly. In a similar spirit I 
submitted https://github.com/gentoo/portage/pull/960

- Flow

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[Toralf Förster]

[-- Attachment #1.1: Type: text/plain, Size: 751 bytes --]

On 12/15/22 20:22, Florian Schmaus wrote:
> o use PORTDIR_OVERLAY and multiple repositories on their system: a 
> system-wide, managed by portage, and a dev repository (in your HOME), 
> scoped in via PORTDIR_OVERLAY.

Isn't this covered by /etc/portage/repos.conf/*

eg here's my config:

cat /etc/portage/repos.conf/all.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /var/db/repos/gentoo
auto-sync = yes
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo.git
#sync-git-verify-commit-signature = true
priority = 10

[local]
location = /var/db/repos/local
auto-sync = no
priority = 99

[tgro]
location = /var/db/repos/tgro
auto-sync = no

priority = 90

-- 
Toralf
PGP 23217DA7 9B888F45


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[Florian Schmaus]

On 15/12/2022 21.10, Toralf Förster wrote:
> On 12/15/22 20:22, Florian Schmaus wrote:
>> o use PORTDIR_OVERLAY and multiple repositories on their system: a 
>> system-wide, managed by portage, and a dev repository (in your HOME), 
>> scoped in via PORTDIR_OVERLAY.
> 
> Isn't this covered by /etc/portage/repos.conf/*

Absolutely, but this requires a manual intervention from the user. And, 
of course, you can totally opt-out from portage managing (syncing) the 
repository, but then you have to take care of syncing yourself.

The point is that with the new portage release, portage's behavior 
changes. And I would argue that portage should not, in its effort to 
become more user friendly, disregard ebuild-developer friendliness. 
Assuming it is achievable with a reasonable amount of additional code 
complexity.

- Flow

[gentoo-dev] Re: Current portage will now truncate your repo's git history to 1

[Duncan]

Florian Schmaus posted on Thu, 15 Dec 2022 21:40:19 +0100 as excerpted:

> On 15/12/2022 21.10, Toralf Förster wrote:
>> On 12/15/22 20:22, Florian Schmaus wrote:
>>> o use PORTDIR_OVERLAY and multiple repositories on their system: a
>>> system-wide, managed by portage, and a dev repository (in your HOME),
>>> scoped in via PORTDIR_OVERLAY.
>> 
>> Isn't this covered by /etc/portage/repos.conf/*
> 
> Absolutely, but this requires a manual intervention from the user. And,
> of course, you can totally opt-out from portage managing (syncing) the
> repository, but then you have to take care of syncing yourself.
> 
> The point is that with the new portage release, portage's behavior
> changes. And I would argue that portage should not, in its effort to
> become more user friendly, disregard ebuild-developer friendliness.
> Assuming it is achievable with a reasonable amount of additional code
> complexity.

This bit me too, and making things worse, the truncate killed the git 
history that presumably had the answer I needed to fix it up. 
=:^(  Fortunately I had a bit of a clue due to preemptively following the 
portage changelog where I had seen a hint, so I was able to dig it up 
again without the git log help that's definitely now my first instinct.

Long story short and for the record, manual intervention indeed and I wish 
it had at least come with a news item, but here's the magic that fixed it 
for me.

I had one of these previously, IIRC clone depth, but both are now needed.  
I put these in the [DEFAULT] section here because I run several overlays 
and I "want" access to proper git logs and history by default. (FWIW 
"want" is the polite, cooled down, version; the situation was considerably 
more sweary when I lost that history and instinctively I tried to look in 
the git history for why, but of course it was GONE!)

repos.conf, [DEFAULT] (or [gentoo]) section:

clone-depth     = 0
sync-depth      = 0

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Current portage will now truncate your repo's git history to 1

[Brian Evans]

[-- Attachment #1.1: Type: text/plain, Size: 1646 bytes --]

On 12/15/22 20:08, Duncan wrote:
> Florian Schmaus posted on Thu, 15 Dec 2022 21:40:19 +0100 as excerpted:
> 
>> On 15/12/2022 21.10, Toralf Förster wrote:
>>> On 12/15/22 20:22, Florian Schmaus wrote:
>>>> o use PORTDIR_OVERLAY and multiple repositories on their system: a
>>>> system-wide, managed by portage, and a dev repository (in your HOME),
>>>> scoped in via PORTDIR_OVERLAY.
>>>
>>> Isn't this covered by /etc/portage/repos.conf/*
>>
>> Absolutely, but this requires a manual intervention from the user. And,
>> of course, you can totally opt-out from portage managing (syncing) the
>> repository, but then you have to take care of syncing yourself.
>>
>> The point is that with the new portage release, portage's behavior
>> changes. And I would argue that portage should not, in its effort to
>> become more user friendly, disregard ebuild-developer friendliness.
>> Assuming it is achievable with a reasonable amount of additional code
>> complexity.
> 
> This bit me too, and making things worse, the truncate killed the git
> history that presumably had the answer I needed to fix it up.
> =:^(  Fortunately I had a bit of a clue due to preemptively following the
> portage changelog where I had seen a hint, so I was able to dig it up
> again without the git log help that's definitely now my first instinct.
> 

Thankfully, if anyone does accidentally gets shallowed, just executing 
'git fetch --unshallow' will revert the default '--depth 1'

I really don't care for that 'git clean' patch.  If that makes it in 
without a way to opt-out, it will be patched out for me personally.

Brian

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 3328 bytes --]



> On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:
> 
> This is a public service announcement that the recently stabilized portage version will truncate you repo's git history to 1.

I wish you'd shown us in #gentoo-portage before sending this, as it's a bit misleading / alarmist. We were actively all speaking
at the time.

Not only to reconsider the phrasing and include some important details, but also to mention the rationale, which I describe
below.

If you felt the Portage team should've written a news item for it, you were (and still are) free to say so, and we'd
do it.

> 
> While this is a good thing for the majority of Gentoo users, it affects developers that develop in a git known to portage (like me). If I understand the portage maintainers vision correctly, then future portage will assume full control over its configured repositories and potentially perform destructive operations on them, for example "git clean" [1].
> 
... only for repositories of sync-type=git & auto-sync=yes.

The rationale for this is that most people who use sync-type=git are doing so because they want
a quicker sync (to complete), a more reliable one (no Manifest race condition issues), and to
get changes from Gentoo faster.

Whenever Portage is syncing something, our view was that we should prioritise successful
completion of syncs, especially given it may be performed unattended.

If git is pulling from a CDN, Portage may on one sync receive state X, but
on a subsequent sync receive state X-1. This can cause an odd state
where git wants to resolve conflicts and manual intervention is required.

This situation was often worse with sync-depth=1 and would lead
to orphaned files, hence the 'git clean' PR referenced.

The motivation behind the sync depth part was because of
disk space growing unbounded otherwise.

> I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.

This wouldn't work for Prefix and also FEATURES="-usersync".

--

Now, looking forward in a constructive manner, I think we can
make both camps happy here with an option to indicate
If Portage can assume the repository will be touched by something
other than Portage.

I propose a configuration option called 'volatile' (thanks
Arsen for the name suggestion) which indicates that the
repository may be changed outside of Portage by any time,
and hence no destructive operations should be carried out.

If the option is on, it also prohibits some optimisations
which require assuming the integrity of the repository.

I have a draft of these changes at https://github.com/gentoo/portage/pull/961.

(https://github.com/gentoo/portage/pull/961.patch if want to view as a raw patch series.0

I am undecided on whether volatile should be default on or not. In the PR
as it is, it defaults to off, which would require a news item. I may just turn it
on given the tone of this thread, as none of it has been very encouraging
for further work on Portage.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[Michael]

> The motivation behind the sync depth part was because of
> disk space growing unbounded otherwise.

I recently did `git config --local fetch.prune true` to the repos I
don't develop (user mode).
And that seems to help with the uncontrollable growth.

# grep prune /usr/portage/.git/config -B1
[fetch]
       prune = true

Michael

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 184 bytes --]



> On 17 Dec 2022, at 05:42, Sam James <sam@gentoo.org> wrote:
>> 
> ... only for repositories of sync-type=git & auto-sync=yes.

Sorry, of course, the auto sync part doesn't matter.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] Re: Current portage will now truncate your repo's git history to 1

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 2375 bytes --]

On Fri, Dec 16, 2022 at 11:01:13PM -0500, Brian Evans wrote:
> On 12/15/22 20:08, Duncan wrote:
> > Florian Schmaus posted on Thu, 15 Dec 2022 21:40:19 +0100 as excerpted:
> > 
> >> On 15/12/2022 21.10, Toralf Förster wrote:
> >>> On 12/15/22 20:22, Florian Schmaus wrote:
> >>>> o use PORTDIR_OVERLAY and multiple repositories on their system: a
> >>>> system-wide, managed by portage, and a dev repository (in your HOME),
> >>>> scoped in via PORTDIR_OVERLAY.
> >>>
> >>> Isn't this covered by /etc/portage/repos.conf/*
> >>
> >> Absolutely, but this requires a manual intervention from the user. And,
> >> of course, you can totally opt-out from portage managing (syncing) the
> >> repository, but then you have to take care of syncing yourself.
> >>
> >> The point is that with the new portage release, portage's behavior
> >> changes. And I would argue that portage should not, in its effort to
> >> become more user friendly, disregard ebuild-developer friendliness.
> >> Assuming it is achievable with a reasonable amount of additional code
> >> complexity.
> > 
> > This bit me too, and making things worse, the truncate killed the git
> > history that presumably had the answer I needed to fix it up.
> > =:^(  Fortunately I had a bit of a clue due to preemptively following the
> > portage changelog where I had seen a hint, so I was able to dig it up
> > again without the git log help that's definitely now my first instinct.
> > 
> 
> Thankfully, if anyone does accidentally gets shallowed, just executing 
> 'git fetch --unshallow' will revert the default '--depth 1'
> 
> I really don't care for that 'git clean' patch.  If that makes it in 
> without a way to opt-out, it will be patched out for me personally.
> 
> Brian

By "that 'git clean'" patch, do you mean [1]? Why are you speaking as
if you're talking about a 3rd party and not people that are also on
the gentoo-dev mailing list? It seems like the more productive thing
to do would be to at least offer substantive criticism about a
specific PR, maybe even in the PR itself.

While I'm at it: you used to be reachable on IRC, but you never
migrated away from Freenode. Why? It seems like every few days someone
is looking for you only to be surprised that you're not in Gentoo's
IRC community anymore.

[1] https://github.com/gentoo/portage/pull/939

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 3867 bytes --]

On Sat, Dec 17, 2022 at 05:42:43AM +0000, Sam James wrote:
> 
> 
> > On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:
> > 
> > This is a public service announcement that the recently stabilized portage version will truncate you repo's git history to 1.
> 
> I wish you'd shown us in #gentoo-portage before sending this, as it's a bit misleading / alarmist. We were actively all speaking
> at the time.
> 
> Not only to reconsider the phrasing and include some important details, but also to mention the rationale, which I describe
> below.
> 
> If you felt the Portage team should've written a news item for it, you were (and still are) free to say so, and we'd
> do it.
> 
> > 
> > While this is a good thing for the majority of Gentoo users, it affects developers that develop in a git known to portage (like me). If I understand the portage maintainers vision correctly, then future portage will assume full control over its configured repositories and potentially perform destructive operations on them, for example "git clean" [1].
> > 
> ... only for repositories of sync-type=git & auto-sync=yes.
> 
> The rationale for this is that most people who use sync-type=git are doing so because they want
> a quicker sync (to complete), a more reliable one (no Manifest race condition issues), and to
> get changes from Gentoo faster.
> 
> Whenever Portage is syncing something, our view was that we should prioritise successful
> completion of syncs, especially given it may be performed unattended.
> 
> If git is pulling from a CDN, Portage may on one sync receive state X, but
> on a subsequent sync receive state X-1. This can cause an odd state
> where git wants to resolve conflicts and manual intervention is required.
> 
> This situation was often worse with sync-depth=1 and would lead
> to orphaned files, hence the 'git clean' PR referenced.
> 
> The motivation behind the sync depth part was because of
> disk space growing unbounded otherwise.

Just to make this more abundantly explicit: it repeatedly happened to
me (but others, too) that on a system using git to sync a shallow
::gentoo clone, during unattended syncs, the repository attempted a
merge with the upstream, yielding a repo in a merging state, with a
huge list of dirtied files in the working tree, plus untracked files.

> > I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.
> 
> This wouldn't work for Prefix and also FEATURES="-usersync".
> 
> --
> 
> Now, looking forward in a constructive manner, I think we can
> make both camps happy here with an option to indicate
> If Portage can assume the repository will be touched by something
> other than Portage.
> 
> I propose a configuration option called 'volatile' (thanks
> Arsen for the name suggestion) which indicates that the
> repository may be changed outside of Portage by any time,
> and hence no destructive operations should be carried out.
> 
> If the option is on, it also prohibits some optimisations
> which require assuming the integrity of the repository.
> 
> I have a draft of these changes at https://github.com/gentoo/portage/pull/961.
> 
> (https://github.com/gentoo/portage/pull/961.patch if want to view as a raw patch series.0
> 
> I am undecided on whether volatile should be default on or not. In the PR
> as it is, it defaults to off, which would require a news item. I may just turn it
> on given the tone of this thread, as none of it has been very encouraging
> for further work on Portage.



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[Florian Schmaus]

On 17.12.22 06:42, Sam James wrote:
>> On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:
>> I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.
> 
> This wouldn't work for Prefix and also FEATURES="-usersync".

Simply do not apply the approach on Prefix. That should be fine. I am 
not sure how usersync being disabled (or enabled) plays a role here, though.

I believe something like this would make everyone happy:

if volatile_explicitly_configured:
   volatile = explicitly_configured_volatile_value
else if prefix
   volatile = false
else if Path(repo_dir).user != (root|portage) # Or, if uid >= 1000
   volatile = true
else
   volatile = false

Assuming that the "if target repo is not shallow, then no shallow clone 
(unless explicitly requested)" check is only applied if volatile=true, 
then you even have the desired effect that users get their repo 
automatically converted to shallow ones. Which makes you happy. And the 
above logic would make me happy, since the "if Path(repo_dir).uid >= 
1000" branch would be taken for me.

That sounds like a win/win to me.

- Flow

Re: [gentoo-dev] Current portage will now truncate your repo's git history to 1

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]



> On 18 Dec 2022, at 10:19, Florian Schmaus <flow@gentoo.org> wrote:
> 
> On 17.12.22 06:42, Sam James wrote:
>>> On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:
>>> I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.
>> This wouldn't work for Prefix and also FEATURES="-usersync".
> 
> Simply do not apply the approach on Prefix. That should be fine. I am not sure how usersync being disabled (or enabled) plays a role here, though.
> 

The owner of the repository and its permissions will be affected by the permissions used by Portage to sync.

> I believe something like this would make everyone happy:
> 
> if volatile_explicitly_configured:
>  volatile = explicitly_configured_volatile_value
> else if prefix
>  volatile = false
> else if Path(repo_dir).user != (root|portage) # Or, if uid >= 1000
>  volatile = true
> else
>  volatile = false
> 
> Assuming that the "if target repo is not shallow, then no shallow clone (unless explicitly requested)" check is only applied if volatile=true, then you even have the desired effect that users get their repo automatically converted to shallow ones. Which makes you happy. And the above logic would make me happy, since the "if Path(repo_dir).uid >= 1000" branch would be taken for me.
> 

Feel free to suggest that on the PR, it sounds like a decent compromise, if a bit automagic - but it wouldn't be the first or last case of that in Portage.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]