[gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Matt Turner]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

tl;dr: In app-portage/gentoolkit-0.5.1 there's a new tool I wrote,
called merge-driver-ekeyword that can automatically resolve git merge
conflicts involving the KEYWORDS=... line in ebuilds.

Since the KEYWORDS=... assignment is a single line, git struggles to
handle conflicts. When rebasing a series of commits that modify the
KEYWORDS=... it's usually easier to throw them away and reapply on the
new tree than it is to manually handle conflicts during the rebase.

git allows a 'merge driver' program to handle conflicts; this program
handles conflicts in the KEYWORDS=... assignment. E.g., given an ebuild
with these keywords:

KEYWORDS="~alpha amd64 arm arm64 ~hppa ppc ppc64 x86"

One developer drops the ~alpha keyword and pushes to gentoo.git, and
another developer stabilizes hppa. Without this merge driver, git
requires the second developer to manually resolve the conflict which is
tedious and prone to mistakes when rebasing a long series of patches.
With the custom merge driver, it automatically resolves the conflict.

To use the merge driver, configure your gentoo.git as such:

gentoo.git/.git/config:

     [merge "keywords"]
	    name = KEYWORDS merge driver
	    driver = merge-driver-ekeyword %O %A %B %P

gentoo.git/.git/info/attributes:

     *.ebuild merge=keywords

With that configured, git merge conflicts on the KEYWORDS=... line will
be resolved automatically (e.g. during git pull --rebase).

Enjoy!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 636 bytes --]

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Michael Orlitzky]

On Mon, 2021-03-01 at 22:54 -0500, Matt Turner wrote:
> tl;dr: In app-portage/gentoolkit-0.5.1 there's a new tool I wrote,
> called merge-driver-ekeyword that can automatically resolve git merge
> conflicts involving the KEYWORDS=... line in ebuilds.
> 
> Since the KEYWORDS=... assignment is a single line,

Is that enforced? If not, will the merge driver handle other formats
correctly? And if it is... why don't we just enforce putting each
keyword on a separate line instead, so that we don't have this problem
in the first place?

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]


> On 2 Mar 2021, at 05:10, Michael Orlitzky <mjo@gentoo.org> wrote:
> 
> On Mon, 2021-03-01 at 22:54 -0500, Matt Turner wrote:
>> tl;dr: In app-portage/gentoolkit-0.5.1 there's a new tool I wrote,
>> called merge-driver-ekeyword that can automatically resolve git merge
>> conflicts involving the KEYWORDS=... line in ebuilds.
>> 
>> Since the KEYWORDS=... assignment is a single line,
> 
> Is that enforced? If not, will the merge driver handle other formats
> correctly? And if it is... why don't we just enforce putting each
> keyword on a separate line instead, so that we don't have this problem
> in the first place?

Yeah, it’s policy: https://projects.gentoo.org/qa/policy-guide/ebuild-format.html#pg0105.

I also removed all of the trivial infringers not long ago: https://github.com/gentoo/gentoo/pull/19467. There
aren’t many left.

Anyway, yes, the format is still a pain because diff doesn’t handle it well. But I don’t really want
to see ebuilds become longer...

sam


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Wolfgang E. Sanyer]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

Git diff has --color-word, doesn't this suggest that it has the capability
of identifying these word-by-word changes?

Is there perhaps a merge option that is being overlooked?

El mar., 2 de marzo de 2021 12:21 a. m., Sam James <sam@gentoo.org>
escribió:

>
> > On 2 Mar 2021, at 05:10, Michael Orlitzky <mjo@gentoo.org> wrote:
> >
> > On Mon, 2021-03-01 at 22:54 -0500, Matt Turner wrote:
> >> tl;dr: In app-portage/gentoolkit-0.5.1 there's a new tool I wrote,
> >> called merge-driver-ekeyword that can automatically resolve git merge
> >> conflicts involving the KEYWORDS=... line in ebuilds.
> >>
> >> Since the KEYWORDS=... assignment is a single line,
> >
> > Is that enforced? If not, will the merge driver handle other formats
> > correctly? And if it is... why don't we just enforce putting each
> > keyword on a separate line instead, so that we don't have this problem
> > in the first place?
>
> Yeah, it’s policy:
> https://projects.gentoo.org/qa/policy-guide/ebuild-format.html#pg0105.
>
> I also removed all of the trivial infringers not long ago:
> https://github.com/gentoo/gentoo/pull/19467. There
> aren’t many left.
>
> Anyway, yes, the format is still a pain because diff doesn’t handle it
> well. But I don’t really want
> to see ebuilds become longer...
>
> sam
>
>

[-- Attachment #2: Type: text/html, Size: 2053 bytes --]

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Matt Turner]

On Tue, Mar 2, 2021 at 6:11 AM Wolfgang E. Sanyer
<wolfgangesanyer@gmail.com> wrote:
> Git diff has --color-word, doesn't this suggest that it has the capability of identifying these word-by-word changes?

No, git diff has that feature. That doesn't imply that the default
merge driver has that feature.

> Is there perhaps a merge option that is being overlooked?

At least not one the upstream git developers know about. (I asked)

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Matt Turner]

On Tue, Mar 2, 2021 at 12:11 AM Michael Orlitzky <mjo@gentoo.org> wrote:
>                                 why don't we just enforce putting each
> keyword on a separate line instead, so that we don't have this problem
> in the first place?

Why don't we change 30 thousand ebuilds rather than use this script?

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Alessandro Barbieri]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

Il Mar 2 Mar 2021, 20:03 Matt Turner <mattst88@gentoo.org> ha scritto:

> On Tue, Mar 2, 2021 at 12:11 AM Michael Orlitzky <mjo@gentoo.org> wrote:
> >                                 why don't we just enforce putting each
> > keyword on a separate line instead, so that we don't have this problem
> > in the first place?
>
> Why don't we change 30 thousand ebuilds rather than use this script?
>

IMO before this the keyword mechanism should be reworked to be three-state:
1 Arch known working
2 Arch known not working
3 Arch untested/status unknown

And let the user choose to use packages from 3 or not

[-- Attachment #2: Type: text/html, Size: 1352 bytes --]

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Brian Evans]

[-- Attachment #1.1: Type: text/plain, Size: 924 bytes --]

On 3/2/2021 2:36 PM, Alessandro Barbieri wrote:
> Il Mar 2 Mar 2021, 20:03 Matt Turner <mattst88@gentoo.org 
> <mailto:mattst88@gentoo.org>> ha scritto:
> 
>     On Tue, Mar 2, 2021 at 12:11 AM Michael Orlitzky <mjo@gentoo.org
>     <mailto:mjo@gentoo.org>> wrote:
>      >                                 why don't we just enforce putting
>     each
>      > keyword on a separate line instead, so that we don't have this
>     problem
>      > in the first place?
> 
>     Why don't we change 30 thousand ebuilds rather than use this script?
> 
> 
> IMO before this the keyword mechanism should be reworked to be three-state:
> 1 Arch known working
> 2 Arch known not working
> 3 Arch untested/status unknown
> 
> And let the user choose to use packages from 3 or not
> 
This is how it already is.  1 and 3 are common but 2 is uncommon except 
for binary only packages.

Brian


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Michał Górny]

On Tue, 2021-03-02 at 00:10 -0500, Michael Orlitzky wrote:
> On Mon, 2021-03-01 at 22:54 -0500, Matt Turner wrote:
> > tl;dr: In app-portage/gentoolkit-0.5.1 there's a new tool I wrote,
> > called merge-driver-ekeyword that can automatically resolve git merge
> > conflicts involving the KEYWORDS=... line in ebuilds.
> > 
> > Since the KEYWORDS=... assignment is a single line,
> 
> Is that enforced? If not, will the merge driver handle other formats
> correctly? And if it is... why don't we just enforce putting each
> keyword on a separate line instead, so that we don't have this problem
> in the first place?

Are you volunteering to fix all the tools to support the new format
correctly?

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Michael Orlitzky]

On Tue, 2021-03-02 at 14:02 -0500, Matt Turner wrote:
> On Tue, Mar 2, 2021 at 12:11 AM Michael Orlitzky <mjo@gentoo.org> wrote:
> >                                 why don't we just enforce putting each
> > keyword on a separate line instead, so that we don't have this problem
> > in the first place?
> 
> Why don't we change 30 thousand ebuilds rather than use this script?
> 

If we're going to enforce a format, it makes more sense to sed things
into the easy format once than it does to leave things in the hard
format forever and maintain a custom tool that makes the hard format
more like the easy format.

With everything currently on one line, QA could easy change them all at
once in one big commit.

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Michael Orlitzky]

On Tue, 2021-03-02 at 20:42 +0100, Michał Górny wrote:
> 
> Are you volunteering to fix all the tools to support the new format
> correctly?
> 

When you already spend all day fixing other peoples' software, this
doesn't sound that scary.

The PMS says that KEYWORDS is whitespace-separated. Probably only
repoman/pkgcheck would require trivial changes. Anything not enforcing
::gentoo policy should (ha ha) already support the PMS format.

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Michał Górny]

On Tue, 2021-03-02 at 15:16 -0500, Michael Orlitzky wrote:
> On Tue, 2021-03-02 at 20:42 +0100, Michał Górny wrote:
> > 
> > Are you volunteering to fix all the tools to support the new format
> > correctly?
> > 
> 
> When you already spend all day fixing other peoples' software, this
> doesn't sound that scary.
> 
> The PMS says that KEYWORDS is whitespace-separated. Probably only
> repoman/pkgcheck would require trivial changes. Anything not enforcing
> ::gentoo policy should (ha ha) already support the PMS format.

Thank you for your detailed research.  However, you seem to have missed
ekeyword and nattka.  And the fact that we still don't have a proper
bash parsing library that could reliably edit complex bash in ebuilds.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Matt Turner]

On Tue, Mar 2, 2021 at 2:36 PM Alessandro Barbieri
<lssndrbarbieri@gmail.com> wrote:
>
> Il Mar 2 Mar 2021, 20:03 Matt Turner <mattst88@gentoo.org> ha scritto:
>>
>> On Tue, Mar 2, 2021 at 12:11 AM Michael Orlitzky <mjo@gentoo.org> wrote:
>> >                                 why don't we just enforce putting each
>> > keyword on a separate line instead, so that we don't have this problem
>> > in the first place?
>>
>> Why don't we change 30 thousand ebuilds rather than use this script?
>
>
> IMO before this the keyword mechanism should be reworked to be three-state:
> 1 Arch known working
> 2 Arch known not working
> 3 Arch untested/status unknown
>
> And let the user choose to use packages from 3 or not

Are you actually replying to me? I was restating mjo's question to
demonstrate the absurdity, not suggesting a different course of
action.

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Matt Turner]

On Tue, Mar 2, 2021 at 3:12 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On Tue, 2021-03-02 at 14:02 -0500, Matt Turner wrote:
> > On Tue, Mar 2, 2021 at 12:11 AM Michael Orlitzky <mjo@gentoo.org> wrote:
> > >                                 why don't we just enforce putting each
> > > keyword on a separate line instead, so that we don't have this problem
> > > in the first place?
> >
> > Why don't we change 30 thousand ebuilds rather than use this script?
> >
>
> If we're going to enforce a format, it makes more sense to sed things
> into the easy format once than it does to leave things in the hard
> format forever and maintain a custom tool that makes the hard format
> more like the easy format.

Sorry, I'm not interested in engaging on this topic. Please start a new thread.

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Patrick McLean]

On Mon, 1 Mar 2021 22:54:45 -0500
Matt Turner <mattst88@gentoo.org> wrote:

> tl;dr: In app-portage/gentoolkit-0.5.1 there's a new tool I wrote,
> called merge-driver-ekeyword that can automatically resolve git merge
> conflicts involving the KEYWORDS=... line in ebuilds.
> 
> To use the merge driver, configure your gentoo.git as such:
> 
> gentoo.git/.git/config:
> 
>      [merge "keywords"]
> 	    name = KEYWORDS merge driver
> 	    driver = merge-driver-ekeyword %O %A %B %P
> 
> gentoo.git/.git/info/attributes:
> 
>      *.ebuild merge=keywords
> 
> With that configured, git merge conflicts on the KEYWORDS=... line will
> be resolved automatically (e.g. during git pull --rebase).
> 
Excellent, this will eliminate one source of annoyance with our
workflow. Perhaps we should document this in the wiki:
https://wiki.gentoo.org/wiki/Gentoo_git_workflow

That way new developers, or developers setting up a new machine will
have instructions on setting this up.

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

>>>>> On Tue, 02 Mar 2021, Michael Orlitzky wrote:

>> Are you volunteering to fix all the tools to support the new format
>> correctly?

> The PMS says that KEYWORDS is whitespace-separated. Probably only
> repoman/pkgcheck would require trivial changes.

No, there are other tools as well, e.g. ekeyword and ebuild-mode.

As matter of fact, ebuild-mode accepts any ASCII whitespace characters
(horizontal tab, new line, vertical tab, form feed, carriage return,
and space) in KEYWORDS, but will change the line to its canonical format
whenever a keyword is updated. That is, in correct order and with single
spaces as separators.

BTW, do package managers allow any Unicode whitespace (like "​" ZERO
WIDTH SPACE or " " OGHAM SPACE MARK) in places where PMS says
"whitespace"?

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

[gentoo-dev] Re: [gentoo-dev-announce] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]


> On 2 Mar 2021, at 03:54, Matt Turner <mattst88@gentoo.org> wrote:
> 
> tl;dr: In app-portage/gentoolkit-0.5.1 there's a new tool I wrote,
> called merge-driver-ekeyword that can automatically resolve git merge
> conflicts involving the KEYWORDS=... line in ebuilds.
> 

Thanks a lot for your work here, Matt! It’s something that’s
proven genuinely helpful in day-to-day work and solves
an actual problem during arch testing.

This is a great improvement on the status quo and obviously this isn’t
the place to bikeshed over the ebuild format.

> [snip]

Best,
Sam

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] New tool: merge-driver-ekeyword automatically resolves git merge conflicts involving KEYWORDS=...

[Michael Orlitzky]

On Wed, 2021-03-03 at 13:09 +0100, Ulrich Mueller wrote:
> > > > > > On Tue, 02 Mar 2021, Michael Orlitzky wrote:
> > > Are you volunteering to fix all the tools to support the new format
> > > correctly?
> > The PMS says that KEYWORDS is whitespace-separated. Probably only
> > repoman/pkgcheck would require trivial changes.
> 
> No, there are other tools as well, e.g. ekeyword and ebuild-mode.
> 

But only repoman/pkgcheck should be enforcing ::gentoo QA policy. The
others are already buggy if they don't support newlines and should be
fixed independently. Then GOTO 1: only repoman/pkgcheck require trivial
updates.