[gentoo-dev] Package stabilization groups

[gentoo-dev] Package stabilization groups

[Matt Turner]

Hello,

Many of us have started using `pkgdev bugs` to file stabilization
bugs. It works well (Thanks Arthur!) and I encourage everyone to give
it a try.

Where possible, it files one stabilization bug per package. This makes
arch testers' jobs easier and makes the task easier to automate.

But sometimes we do want to stabilize packages together. For example
major versions of x11-wm/mutter and gnome-base/gnome-shell are tied
together. If a new mutter is stabilized without the new gnome-shell,
the tree will still be consistent, but emerge -u @world will warn
users that the mutter upgrade is blocked.

There was some brief discussion on IRC about how to document these
groupings, and two main ideas were suggested:

- add a field to metadata.xml to specify the group by an arbitrary name.
  E.g. <stable-group name="..."/>
  Each package in the group would specify the same value of name="..."

- maintain the groups in a separate place (similar to portage @sets).

Can anyone think of particular advantages or disadvantages to one
solution versus the other? Any other (better) ideas?

Thanks,
Matt

Re: [gentoo-dev] Package stabilization groups

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1484 bytes --]

On 16-07-2023 10:57:54 -0400, Matt Turner wrote:
> Hello,
> 
> Many of us have started using `pkgdev bugs` to file stabilization
> bugs. It works well (Thanks Arthur!) and I encourage everyone to give
> it a try.
> 
> Where possible, it files one stabilization bug per package. This makes
> arch testers' jobs easier and makes the task easier to automate.
> 
> But sometimes we do want to stabilize packages together. For example
> major versions of x11-wm/mutter and gnome-base/gnome-shell are tied
> together. If a new mutter is stabilized without the new gnome-shell,
> the tree will still be consistent, but emerge -u @world will warn
> users that the mutter upgrade is blocked.
> 
> There was some brief discussion on IRC about how to document these
> groupings, and two main ideas were suggested:
> 
> - add a field to metadata.xml to specify the group by an arbitrary name.
>   E.g. <stable-group name="..."/>
>   Each package in the group would specify the same value of name="..."
> 
> - maintain the groups in a separate place (similar to portage @sets).
> 
> Can anyone think of particular advantages or disadvantages to one
> solution versus the other? Any other (better) ideas?

I don't know how widespread the problem is, and how much it can be
generalised, but could you perhaps use a virtual, such that
stabilisation of the virtual means the deps must be satisfied?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Matt Turner]

On Sun, Jul 16, 2023 at 11:15 AM Fabian Groffen <grobian@gentoo.org> wrote:
>
> On 16-07-2023 10:57:54 -0400, Matt Turner wrote:
> > Hello,
> >
> > Many of us have started using `pkgdev bugs` to file stabilization
> > bugs. It works well (Thanks Arthur!) and I encourage everyone to give
> > it a try.
> >
> > Where possible, it files one stabilization bug per package. This makes
> > arch testers' jobs easier and makes the task easier to automate.
> >
> > But sometimes we do want to stabilize packages together. For example
> > major versions of x11-wm/mutter and gnome-base/gnome-shell are tied
> > together. If a new mutter is stabilized without the new gnome-shell,
> > the tree will still be consistent, but emerge -u @world will warn
> > users that the mutter upgrade is blocked.
> >
> > There was some brief discussion on IRC about how to document these
> > groupings, and two main ideas were suggested:
> >
> > - add a field to metadata.xml to specify the group by an arbitrary name.
> >   E.g. <stable-group name="..."/>
> >   Each package in the group would specify the same value of name="..."
> >
> > - maintain the groups in a separate place (similar to portage @sets).
> >
> > Can anyone think of particular advantages or disadvantages to one
> > solution versus the other? Any other (better) ideas?
>
> I don't know how widespread the problem is, and how much it can be
> generalised, but could you perhaps use a virtual, such that
> stabilisation of the virtual means the deps must be satisfied?

Heh, I guess we could do that if we had no other options.

Re: [gentoo-dev] Package stabilization groups

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 3130 bytes --]

On 16/07/2023 17.57, Matt Turner wrote:
> Hello,
> 
> Many of us have started using `pkgdev bugs` to file stabilization
> bugs. It works well (Thanks Arthur!) and I encourage everyone to give
> it a try.

Gladly :) You can semi-blame mgorny for the creation of `pkgdev bugs`,
because I started to file stablereqs for @python packages, and at some
point it was too tiring and repetitive, so I was brought to the drawing
table to think of a solution.

> Where possible, it files one stabilization bug per package. This makes
> arch testers' jobs easier and makes the task easier to automate.
> 
> But sometimes we do want to stabilize packages together. For example
> major versions of x11-wm/mutter and gnome-base/gnome-shell are tied
> together. If a new mutter is stabilized without the new gnome-shell,
> the tree will still be consistent, but emerge -u @world will warn
> users that the mutter upgrade is blocked.
> 
> There was some brief discussion on IRC about how to document these
> groupings, and two main ideas were suggested:
> 
> - add a field to metadata.xml to specify the group by an arbitrary name.
>   E.g. <stable-group name="..."/>
>   Each package in the group would specify the same value of name="..."
> 
> - maintain the groups in a separate place (similar to portage @sets).
> 
> Can anyone think of particular advantages or disadvantages to one
> solution versus the other? Any other (better) ideas?

Let me list some things as advantages to each one (since I see an
advantage to one as disadvantage to other).

Advantages of field in metadata.xml:

- local to package, easier to not miss. Easier to follow for the maintainer.

- easier to find which group the current package relates to

Advantages to group files:

- easier to index (one file listing all children, instead of searching
across repo who defines it)

- easier to not repeat group. In the metadata field it might happen to
repeat, less likely since it is easy to search, but similar group names
might be created, merging them into one by mistake, or creating very
similar names and mistaking them. When we have a single file, it is
easier to see the whole picture and verify things.

- since this is compressed information (special files instead of spread
over repo), we could load all of them into app's cache, and make all
computation easier.

- enables an easier syntax as `pkgdev bugs @group1` to file a single bug
for all of them. In Gnome's case as an example, we could have "gnome1",
"gnome2", "gnome3", "gnome4", etc - groups standing for multiple
"layers/stages" of stabilizing, and then you could just file `pkgdev
bugs @gnome{1..4}` to file "at one click" the whole gnome stablereqs.

> Thanks,
> Matt


Now I'll speak from the point of implementer of `pkgdev bugs`. For me I
think both approaches are good, but I would prefer the latter over the
former. Nicer syntax, easy cache of all groups, easier to solve the
"graph problems" in the tool.

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]

>>>>> On Sun, 16 Jul 2023, Arthur Zamarin wrote:

> Let me list some things as advantages to each one (since I see an
> advantage to one as disadvantage to other).

> Advantages of field in metadata.xml:

> - local to package, easier to not miss. Easier to follow for the maintainer.

> - easier to find which group the current package relates to

> Advantages to group files:

> - easier to index (one file listing all children, instead of searching
> across repo who defines it)

> - easier to not repeat group. In the metadata field it might happen to
> repeat, less likely since it is easy to search, but similar group names
> might be created, merging them into one by mistake, or creating very
> similar names and mistaking them. When we have a single file, it is
> easier to see the whole picture and verify things.

> - since this is compressed information (special files instead of spread
> over repo), we could load all of them into app's cache, and make all
> computation easier.

> - enables an easier syntax as `pkgdev bugs @group1` to file a single bug
> for all of them. In Gnome's case as an example, we could have "gnome1",
> "gnome2", "gnome3", "gnome4", etc - groups standing for multiple
> "layers/stages" of stabilizing, and then you could just file `pkgdev
> bugs @gnome{1..4}` to file "at one click" the whole gnome stablereqs.

Can't we do the same that we do for local use flags? Namely, define them
in metadata.xml, but collect them in one central location?

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 919 bytes --]

On 16/07/2023 21.11, Ulrich Mueller wrote:
>>>>>> On Sun, 16 Jul 2023, Arthur Zamarin wrote:
> 
>> - enables an easier syntax as `pkgdev bugs @group1` to file a single bug
>> for all of them. In Gnome's case as an example, we could have "gnome1",
>> "gnome2", "gnome3", "gnome4", etc - groups standing for multiple
>> "layers/stages" of stabilizing, and then you could just file `pkgdev
>> bugs @gnome{1..4}` to file "at one click" the whole gnome stablereqs.
> 
> Can't we do the same that we do for local use flags? Namely, define them
> in metadata.xml, but collect them in one central location?

Do note that this grouping is done at "metadata repo", and not the
normal git repo where development is done. Meaning you need to manually
add the package to the central "index".

> Ulrich

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Matt Turner]

On Sun, Jul 16, 2023 at 2:04 PM Arthur Zamarin <arthurzam@gentoo.org> wrote:
> Now I'll speak from the point of implementer of `pkgdev bugs`. For me I
> think both approaches are good, but I would prefer the latter over the
> former. Nicer syntax, easy cache of all groups, easier to solve the
> "graph problems" in the tool.

Sounds good to me. Should we have infra create a new git repo for us
for this purpose?

Re: [gentoo-dev] Package stabilization groups

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 864 bytes --]

On 17/07/2023 16.50, Matt Turner wrote:
> On Sun, Jul 16, 2023 at 2:04 PM Arthur Zamarin <arthurzam@gentoo.org> wrote:
>> Now I'll speak from the point of implementer of `pkgdev bugs`. For me I
>> think both approaches are good, but I would prefer the latter over the
>> former. Nicer syntax, easy cache of all groups, easier to solve the
>> "graph problems" in the tool.
> 
> Sounds good to me. Should we have infra create a new git repo for us
> for this purpose?
> 

No. I think it should go under normal git repo, and not separate repo. I
see no gains from it being under separate repository, only headache (how
to sync between them).

I think a main index files under
"/metadata/${some_good_name}/${group_name}" would be best.

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 1081 bytes --]

On Mon, Jul 17, 2023 at 19:39:30 +0300, Arthur Zamarin wrote:
> On 17/07/2023 16.50, Matt Turner wrote:
> > On Sun, Jul 16, 2023 at 2:04 PM Arthur Zamarin <arthurzam@gentoo.org> wrote:
> >> Now I'll speak from the point of implementer of `pkgdev bugs`. For me I
> >> think both approaches are good, but I would prefer the latter over the
> >> former. Nicer syntax, easy cache of all groups, easier to solve the
> >> "graph problems" in the tool.
> > 
> > Sounds good to me. Should we have infra create a new git repo for us
> > for this purpose?
> > 
> 
> No. I think it should go under normal git repo, and not separate repo. I
> see no gains from it being under separate repository, only headache (how
> to sync between them).
> 
> I think a main index files under
> "/metadata/${some_good_name}/${group_name}" would be best.
> 

/metadata/stable-goups/${group_name} perhaps? Then you can adapt Ionen's
example as:

    $ cat /path/to/repo/metadata/stable-groups/qt
    dev-qt/package
    dev-qt/package2

Easy to read, easy to write :)

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 2188 bytes --]


Matt Turner <mattst88@gentoo.org> writes:

> Hello,
>
> Many of us have started using `pkgdev bugs` to file stabilization
> bugs. It works well (Thanks Arthur!) and I encourage everyone to give
> it a try.
>
> Where possible, it files one stabilization bug per package. This makes
> arch testers' jobs easier and makes the task easier to automate.
>
> But sometimes we do want to stabilize packages together. For example
> major versions of x11-wm/mutter and gnome-base/gnome-shell are tied
> together. If a new mutter is stabilized without the new gnome-shell,
> the tree will still be consistent, but emerge -u @world will warn
> users that the mutter upgrade is blocked.
>

Big fan of the idea & very much in support of it. This also serves
to give us logical groupings of packages which are closely related
and should be bumped together.

> There was some brief discussion on IRC about how to document these
> groupings, and two main ideas were suggested:
>
> - add a field to metadata.xml to specify the group by an arbitrary name.
>   E.g. <stable-group name="..."/>
>   Each package in the group would specify the same value of name="..."
>
> - maintain the groups in a separate place (similar to portage @sets).
>
> Can anyone think of particular advantages or disadvantages to one
> solution versus the other? Any other (better) ideas?
>

When we discussed this a few months ago on IRC, I also brought up a
related point:

[2023-05-02T18:38:51+0100] <@sam_> do we want to repeat the group members in each member, or do tools need to scan for each thing?
[2023-05-02T18:39:07+0100] <@sam_> i.e. does each member have <stable-group><pkg>...</pkg></stable-group>, or do we do <stable-group name=".../>?
[2023-05-02T18:39:26+0100] <@arthurzam> I think each package says which groups it is part of
[2023-05-02T18:39:44+0100] <@radhermit> I would do the latter
[...]
[2023-05-02T18:42:42+0100] <@radhermit> technically you could also maintain them in a separate place like metadata/groups and layer inter-group dependencies on top of that somehow in the format

I'd prefer the metadata/ at repo root idea because I can see updating
various metadata.xmls being a nuisance.

best,
sam


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 2494 bytes --]

On 17/07/2023 19.37, Sam James wrote:
> 
> Big fan of the idea & very much in support of it. This also serves
> to give us logical groupings of packages which are closely related
> and should be bumped together.
> 
>> There was some brief discussion on IRC about how to document these
>> groupings, and two main ideas were suggested:
>>
>> - add a field to metadata.xml to specify the group by an arbitrary name.
>>   E.g. <stable-group name="..."/>
>>   Each package in the group would specify the same value of name="..."
>>
>> - maintain the groups in a separate place (similar to portage @sets).
>>
>> Can anyone think of particular advantages or disadvantages to one
>> solution versus the other? Any other (better) ideas?
>>
> 
> When we discussed this a few months ago on IRC, I also brought up a
> related point:
> 
> [2023-05-02T18:38:51+0100] <@sam_> do we want to repeat the group members in each member, or do tools need to scan for each thing?
> [2023-05-02T18:39:07+0100] <@sam_> i.e. does each member have <stable-group><pkg>...</pkg></stable-group>, or do we do <stable-group name=".../>?
> [2023-05-02T18:39:26+0100] <@arthurzam> I think each package says which groups it is part of
> [2023-05-02T18:39:44+0100] <@radhermit> I would do the latter
> [...]
> [2023-05-02T18:42:42+0100] <@radhermit> technically you could also maintain them in a separate place like metadata/groups and layer inter-group dependencies on top of that somehow in the format

If you read carefully my messages in IRC linked above, you can see I was
supporting per package metadata entry. If you read my latest post to ML,
you can see I now prefer central files. After many considerations since
then I understood my initial preference was a bad idea :)

(I'm noting it here just so folks understand the mismatch between texts
and my stance).

> I'd prefer the metadata/ at repo root idea because I can see updating
> various metadata.xmls being a nuisance.

Hmm, I was thinking the opposite (maintaining it in parallel place to
the package would be harder), but if you say so (and you help maintain
huge clusters of packages so I believe you) then I think we don't have
any good reason to go with per package metadata.xml entry?

Let's wait for more input, and then we can go with defining the syntax,
rules and such...

> best,
> sam
> 

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On Mon, Jul 17, 2023 at 08:34:45PM +0300, Arthur Zamarin wrote:
> Hmm, I was thinking the opposite (maintaining it in parallel place to
> the package would be harder), but if you say so (and you help maintain
> huge clusters of packages so I believe you) then I think we don't have
> any good reason to go with per package metadata.xml entry?

imo seeing/setting it all in one place is simpler, ultimately it's
about managing a group (whole) and I don't think is super important
on the individual packages.

e.g.

[qt]
dev-qt/package
dev-qt/package2

Can visualize and don't have to go set this in each one of them.

fwiw repo-cd could be given a hook to grep that file, and be able
to say that a package is part of a stabilization group -- and if
nattka automatically looks at groups it'll also be hard to overlook
when filing bugs.

Regardless of approach, some kind of central file (generated or not)
would certainly be nice not to end up grepping all metadata.xml in
the tree to figure out which packages are part of a group.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Package stabilization groups

[Agostino Sarubbo]

Hello,

although it might seem offtopic, but while talking about metadata.xml fileds 
and `pkgdev bugs` I would like to ask opinions about have a metadata.xml field 
that allows the maintainer to exclude the package from `pkgdev bugs`

That should be useful for core packages (gcc/glibc) and for packages that can 
be properly tested by have an account somewhere.

Thanks