[gentoo-dev] Standard parsable format for profiles/package.mask file

[gentoo-dev] Standard parsable format for profiles/package.mask file

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 4072 bytes --]

Hi all

I want to suggest a standard format for profiles/package.mask, for
multiple reasons:

1. Easier to write simple to understand mask or last-rites entries. When
all entries are in similar format, the reader knows where to expect
important information and such. Also easier for writer to convey all
needed information.

2. We can teach tools to parse it and render nicely, or help you fill
the file. For example I've tried to implement a parser for
packages.gentoo.org so it shows as nice as possible the message, see as
example [1]. On the other hand, `pkgdev mask` [2] can help you fill the
message (including bug number, last-rite until date, author & email
line). Both of them mostly works, but when someone "breaks" the
unofficial syntax, the tools fail sadly.

This is why I want to recommend we create a mostly standard syntax, so
we can all expect the same thing and have nice things.
Also please note that for now I want to formalize the format only for
profiles/package.mask file, and not the one inside all the different
profiles. If you think we better apply to all of them, we can think on
it separately please :)

The current format is mostly acceptable, but let's tighten it. I will
implement a pkgcheck check that will validate the format and error out
if invalid.

[1] https://packages.gentoo.org/packages/sys-fs/eudev
[2] https://pkgcore.github.io/pkgdev/man/pkgdev/mask.html

===== "Formal" format =====

Each entry is composed of 2 parts: "#"-prefixed explanation block and
list of "${CATEGORY}/${PN}" packages. Entries are separated when a new
explanation block starts (meaning first "#"-prefixed line after packages
list). You may add newlines between packages in packages list.

The first line of the "#"-prefixed explanation block must be of the
format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
format YYYY-MM-DD, in UTC timezone.

If this is a last-rite message, the last line must list the last-rite
last date (removal date) and the last-rite bug number. You can also list
other bugs relevant to the last-rite. So I think a format of: "Removal
on ${REMOVAL_DATE}.  Bug #NNNNNN, #NNNNNN." Where the bug list is comma
and space separated, we have at least one space (" +" regex) between the
removal date and bug list, and the date is of YYYY-MM-DD format.
I prefer this line is separate (and not continuous of prefix message text).

The explanation block itself can reference bugs, by matching the regex
"[Bb]ugs? #\d+(, +#\d+)*" (For example: "bug #713106, #753134"). I think
this is quite a simple one, but powerful enough for most.

Lines with single newline between them (so no blank line between them)
are considered as single paragraph continuum. If you want to start new
paragraph, leave a blank line (still prefixed with #) - think similar to
markdown. A line matching the last-rite line is always it's own paragraph.

========= Example =========

After all of those rambling, here is an example (it will result in 3
paragraphs, 2 explanation and 1 last-rite finish):

# Arthur Zamarin <arthurzam@gentoo.org> (2023-09-21)
# Very broken, no idea why packaged, need to drop ASAP. The project
# is done with supporting this package. See for history bug #667889.
#
# As a better plan, you should migrate to dev-lang/perl, which has
# better compatibility with dev-lang/ruby when used with dev-lang/lua
# bindings.
# Removal on 2023-10-21.  Bug #667687, #667689.
dev-lang/python

==== Call for comments ====

So how does it sound? I know it is easy to try to limit the syntax for
me (since I"ll need to implement parsing of it), but I think this format
above matches most of the currently used once, and the one created by
`pkgdev mask`. But i needed, I'm open to improve it by comments.

Should it be a GLEP, I don't think so? But I'm unsure about it. We do
need to document it (for example header of that exact file).


-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 759 bytes --]

On 21/09/2023 21.40, Arthur Zamarin wrote:
> Hi all
> 
> I want to suggest a standard format for profiles/package.mask, for
> multiple reasons:

Sounds sensible. +1

> The first line of the "#"-prefixed explanation block must be of the
> format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
> format YYYY-MM-DD, in UTC timezone.
                      ^^^^^^^^^^^^^^^^

Can we drop this? Or, at least, relax this.

I usually just enter my locale date here and like to avoid having to 
think about that UTC is potentially in a different date. I also can not 
remember any situation where the date being in UTC matters. Plus, if you 
want accurate timestamps, then the git commit/author date is here for 
you. :)

- Flow

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 17273 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

>>>>> On Thu, 21 Sep 2023, Florian Schmaus wrote:

>> The first line of the "#"-prefixed explanation block must be of the
>> format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
>> format YYYY-MM-DD, in UTC timezone.
>                     ^^^^^^^^^^^^^^^^

> Can we drop this? Or, at least, relax this.

I think UTC makes a lot of sense in an international context like ours.
It also avoids flapping of the date between entries (i.e. a newer entry
having an older date than the previous one).

> I usually just enter my locale date here and like to avoid having to
> think about that UTC is potentially in a different date. I also can
> not remember any situation where the date being in UTC matters. Plus,
> if you want accurate timestamps, then the git commit/author date is
> here for you. :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Sam James]

Ulrich Mueller <ulm@gentoo.org> writes:

> [[PGP Signed Part:Undecided]]
>>>>>> On Thu, 21 Sep 2023, Florian Schmaus wrote:
>
>>> The first line of the "#"-prefixed explanation block must be of the
>>> format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
>>> format YYYY-MM-DD, in UTC timezone.
>>                     ^^^^^^^^^^^^^^^^
>
>> Can we drop this? Or, at least, relax this.
>
> I think UTC makes a lot of sense in an international context like ours.
> It also avoids flapping of the date between entries (i.e. a newer entry
> having an older date than the previous one).

Yes, I don't think we need to punish people for getting it wrong, but at
the same time, I'd like us to standardise on UTC - saying this as
someone who isn't in UTC half of the year.

That means others are free to correct it if they notice it's the
wrong date and so on.

>> I usually just enter my locale date here and like to avoid having to
>> think about that UTC is potentially in a different date. I also can
>> not remember any situation where the date being in UTC matters. Plus,
>> if you want accurate timestamps, then the git commit/author date is
>> here for you. :)

Users consume p.mask entries directly rather than via git.

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 2001 bytes --]

On 21/09/2023 23.48, Sam James wrote:
> Ulrich Mueller <ulm@gentoo.org> writes:
>>>>>>> On Thu, 21 Sep 2023, Florian Schmaus wrote:
>>>> The first line of the "#"-prefixed explanation block must be of the
>>>> format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
>>>> format YYYY-MM-DD, in UTC timezone.
>>>                      ^^^^^^^^^^^^^^^^
>>
>>> Can we drop this? Or, at least, relax this.
>>
>> I think UTC makes a lot of sense in an international context like ours.

It does, but mostly if you also care about the hour and minute of the 
timestamp [1].


>> It also avoids flapping of the date between entries (i.e. a newer entry
>> having an older date than the previous one).

This appears to be mainly an esthetic issue (and probably also rarely 
happens).


> the same time, I'd like us to standardise on UTC

But why? Which problem does it solve for p.mask entries?


>>> I usually just enter my locale date here and like to avoid having to
>>> think about that UTC is potentially in a different date. I also can
>>> not remember any situation where the date being in UTC matters. Plus,
>>> if you want accurate timestamps, then the git commit/author date is
>>> here for you. :)
> 
> Users consume p.mask entries directly rather than via git.

Correct. But how many users reading p.mask entries thought about the 
timezone the date timestamp is in? I'd assume at most only a few, 
because it is not relevant to know if its +00 or +02 or -07.

Some, including me, consider timestamps without timezone specifiers to 
be in local time (either of the consumer or producer of the timestamp). 
Hence, if you really must have UTC here, then at least consider making 
it explicit my requiring the 'Z' timezone specifier (which, if you want 
to be ISO compatible, probably means that the timestamp must include 
HH:MM too).

- Flow


1: That is probably why I don't see a timezone specifier for calendar 
dates in ISO 8601.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 17273 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 608 bytes --]

On 22/09/2023 08.39, Florian Schmaus wrote:
> Some, including me, consider timestamps without timezone specifiers to 
> be in local time (either of the consumer or producer of the timestamp). 
> Hence, if you really must have UTC here, then at least consider making 
> it explicit my requiring the 'Z' timezone specifier.
I forgot the to include the motivation for doing so:

Timestamps with a timezone specifier make it clear to the producer of 
the timestamp that they should produce an offset-ed timestamp and make 
it evident to the consumer of the timestamp which timezone it is in.

- Flow

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 17273 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

>>>>> On Fri, 22 Sep 2023, Florian Schmaus wrote:

> Some, including me, consider timestamps without timezone specifiers to
> be in local time (either of the consumer or producer of the
> timestamp). Hence, if you really must have UTC here, then at least
> consider making it explicit my requiring the 'Z' timezone specifier
> (which, if you want to be ISO compatible, probably means that the
> timestamp must include HH:MM too).

How about converting package.mask to XML? The xs:date type would allow
a date followed by a time zone [1].

/me hides

[1] https://www.w3.org/TR/xmlschema-2/#deviantformats

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

>>>>> On Fri, 22 Sep 2023, Ulrich Mueller wrote:

>>>>> On Fri, 22 Sep 2023, Florian Schmaus wrote:
>> Some, including me, consider timestamps without timezone specifiers to
>> be in local time (either of the consumer or producer of the
>> timestamp). Hence, if you really must have UTC here, then at least
>> consider making it explicit my requiring the 'Z' timezone specifier
>> (which, if you want to be ISO compatible, probably means that the
>> timestamp must include HH:MM too).

> How about converting package.mask to XML? The xs:date type would allow
> a date followed by a time zone [1].

> /me hides

Seriously, this isn't a hill I am willing to die on. I still prefer UTC
there, but I'd be fine if the wording said "should" instead of "must".

Ulrich

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Sam James]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Fri, 22 Sep 2023, Ulrich Mueller wrote:
>
>>>>>> On Fri, 22 Sep 2023, Florian Schmaus wrote:
>>> Some, including me, consider timestamps without timezone specifiers to
>>> be in local time (either of the consumer or producer of the
>>> timestamp). Hence, if you really must have UTC here, then at least
>>> consider making it explicit my requiring the 'Z' timezone specifier
>>> (which, if you want to be ISO compatible, probably means that the
>>> timestamp must include HH:MM too).
>
>> How about converting package.mask to XML? The xs:date type would allow
>> a date followed by a time zone [1].
>
>> /me hides
>
> Seriously, this isn't a hill I am willing to die on. I still prefer UTC
> there, but I'd be fine if the wording said "should" instead of "must".

Yes, I want the UTC bit in there, but it's fine if it's "should" and not
"must". I was trying to articulate that before but I didn't do so very
clearly.

>
> Ulrich

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Alex Boag-Munroe]

Any reason for the parseable parts to not be in an established human
readable/editable format? e.g. the config ini style format, or TOML?

To crib from the OP example with something configparser understands:
[PREAMBLE]
Timestamp: 2023-09-21 15:07:42+00:00
Author: Arthur Zamarin <arthurzam@gentoo.org>
Justification: Very broken, no idea why packaged, need to drop ASAP.
    The project is done with supporting this package.
Bugs: 667687, 667689
Removal Date: 2023-10-21
Packages: dev-lang/python

The format is well documented already and simple to check for
validity, so any GLEP would just need to cover correct keys/values.

Just a thought.

--
Ninpo

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Sam James]

Alex Boag-Munroe <ninpo@qap.la> writes:

> Any reason for the parseable parts to not be in an established human
> readable/editable format? e.g. the config ini style format, or TOML?

The only issue really is that depending on how it's done (do we do
it for the whole file, or just comments), it may need a new (profile)
EAPI which will take a while to implement and deploy.

If it's just for comments, then we can do it immediately though.

>
> To crib from the OP example with something configparser understands:
> [PREAMBLE]
> Timestamp: 2023-09-21 15:07:42+00:00
> Author: Arthur Zamarin <arthurzam@gentoo.org>
> Justification: Very broken, no idea why packaged, need to drop ASAP.
>     The project is done with supporting this package.
> Bugs: 667687, 667689
> Removal Date: 2023-10-21
> Packages: dev-lang/python
>
> The format is well documented already and simple to check for
> validity, so any GLEP would just need to cover correct keys/values.
>

But yeah, I agree it's worth thinking about a proper format rather than
fragile text mangling going into the future.

> Just a thought.

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Alex Boag-Munroe]

On Fri, 22 Sept 2023 at 15:37, Sam James <sam@gentoo.org> wrote:
>
>
> Alex Boag-Munroe <ninpo@qap.la> writes:
>
> > Any reason for the parseable parts to not be in an established human
> > readable/editable format? e.g. the config ini style format, or TOML?
>
> The only issue really is that depending on how it's done (do we do
> it for the whole file, or just comments), it may need a new (profile)
> EAPI which will take a while to implement and deploy.
>
> If it's just for comments, then we can do it immediately though.
>
> >
> > To crib from the OP example with something configparser understands:
> > [PREAMBLE]
> > Timestamp: 2023-09-21 15:07:42+00:00
> > Author: Arthur Zamarin <arthurzam@gentoo.org>
> > Justification: Very broken, no idea why packaged, need to drop ASAP.
> >     The project is done with supporting this package.
> > Bugs: 667687, 667689
> > Removal Date: 2023-10-21
> > Packages: dev-lang/python
> >
> > The format is well documented already and simple to check for
> > validity, so any GLEP would just need to cover correct keys/values.
> >
>
> But yeah, I agree it's worth thinking about a proper format rather than
> fragile text mangling going into the future.
>
Perhaps eventually it could/should be used for the whole file but as
an interim/beginning there's no reason you couldn't start with
comments:

# [PREAMBLE]
# Timestamp: 2023-09-21 15:07:42+00:00
# Author: Arthur Zamarin <arthurzam@gentoo.org>
# Justification: Very broken, no idea why packaged, need to drop ASAP.
#     The project is done with supporting this package.
# Bugs: 667687, 667689
# Packages: dev-lang/python
dev-lang/python

This simply adds a pre parse step of stripping the comments then
feeding directly into configparser or probably more suitable, TOML
since TOML has better syntax for directly delivering things like a
"Packages:" key as a list.

Redoing a bunch of package.* parsing probably wasn't in scope of the
OP but I've always wondered and this felt an opportune moment to
ask/suggest :)

--
Ninpo

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 2635 bytes --]

On 22/09/2023 17.50, Alex Boag-Munroe wrote:
> On Fri, 22 Sept 2023 at 15:37, Sam James <sam@gentoo.org> wrote:
>>
>>
>> Alex Boag-Munroe <ninpo@qap.la> writes:
>>
>>> Any reason for the parseable parts to not be in an established human
>>> readable/editable format? e.g. the config ini style format, or TOML?
>>
>> The only issue really is that depending on how it's done (do we do
>> it for the whole file, or just comments), it may need a new (profile)
>> EAPI which will take a while to implement and deploy.
>>
>> If it's just for comments, then we can do it immediately though.
>>
>>>
>>> To crib from the OP example with something configparser understands:
>>> [PREAMBLE]
>>> Timestamp: 2023-09-21 15:07:42+00:00
>>> Author: Arthur Zamarin <arthurzam@gentoo.org>
>>> Justification: Very broken, no idea why packaged, need to drop ASAP.
>>>     The project is done with supporting this package.
>>> Bugs: 667687, 667689
>>> Removal Date: 2023-10-21
>>> Packages: dev-lang/python
>>>
>>> The format is well documented already and simple to check for
>>> validity, so any GLEP would just need to cover correct keys/values.
>>>
>>
>> But yeah, I agree it's worth thinking about a proper format rather than
>> fragile text mangling going into the future.
>>
> Perhaps eventually it could/should be used for the whole file but as
> an interim/beginning there's no reason you couldn't start with
> comments:
> 
> # [PREAMBLE]
> # Timestamp: 2023-09-21 15:07:42+00:00
> # Author: Arthur Zamarin <arthurzam@gentoo.org>
> # Justification: Very broken, no idea why packaged, need to drop ASAP.
> #     The project is done with supporting this package.
> # Bugs: 667687, 667689
> # Packages: dev-lang/python
> dev-lang/python
> 
> This simply adds a pre parse step of stripping the comments then
> feeding directly into configparser or probably more suitable, TOML
> since TOML has better syntax for directly delivering things like a
> "Packages:" key as a list.
> 
> Redoing a bunch of package.* parsing probably wasn't in scope of the
> OP but I've always wondered and this felt an opportune moment to
> ask/suggest :)

Thanks for the idea. Yes, it was out of scope such suggestions for me
originally, but thinking more about it, I take it more positively.
Please let me (and others) to consider it for some days, cause this is
very interesting proposal. Things to consider is how much effort it is
to file in future, which format to use, etc.

> --
> Ninpo
> 

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Sam James]

Arthur Zamarin <arthurzam@gentoo.org> writes:

> [[PGP Signed Part:Undecided]]
> On 22/09/2023 17.50, Alex Boag-Munroe wrote:
>> On Fri, 22 Sept 2023 at 15:37, Sam James <sam@gentoo.org> wrote:
>>>
>>>
>>> Alex Boag-Munroe <ninpo@qap.la> writes:
>>>
>>>> Any reason for the parseable parts to not be in an established human
>>>> readable/editable format? e.g. the config ini style format, or TOML?
>>>
>>> The only issue really is that depending on how it's done (do we do
>>> it for the whole file, or just comments), it may need a new (profile)
>>> EAPI which will take a while to implement and deploy.
>>>
>>> If it's just for comments, then we can do it immediately though.
>>>
>>>>
>>>> To crib from the OP example with something configparser understands:
>>>> [PREAMBLE]
>>>> Timestamp: 2023-09-21 15:07:42+00:00
>>>> Author: Arthur Zamarin <arthurzam@gentoo.org>
>>>> Justification: Very broken, no idea why packaged, need to drop ASAP.
>>>>     The project is done with supporting this package.
>>>> Bugs: 667687, 667689
>>>> Removal Date: 2023-10-21
>>>> Packages: dev-lang/python
>>>>
>>>> The format is well documented already and simple to check for
>>>> validity, so any GLEP would just need to cover correct keys/values.
>>>>
>>>
>>> But yeah, I agree it's worth thinking about a proper format rather than
>>> fragile text mangling going into the future.
>>>
>> Perhaps eventually it could/should be used for the whole file but as
>> an interim/beginning there's no reason you couldn't start with
>> comments:
>> 
>> # [PREAMBLE]
>> # Timestamp: 2023-09-21 15:07:42+00:00
>> # Author: Arthur Zamarin <arthurzam@gentoo.org>
>> # Justification: Very broken, no idea why packaged, need to drop ASAP.
>> #     The project is done with supporting this package.
>> # Bugs: 667687, 667689
>> # Packages: dev-lang/python
>> dev-lang/python
>> 
>> This simply adds a pre parse step of stripping the comments then
>> feeding directly into configparser or probably more suitable, TOML
>> since TOML has better syntax for directly delivering things like a
>> "Packages:" key as a list.
>> 
>> Redoing a bunch of package.* parsing probably wasn't in scope of the
>> OP but I've always wondered and this felt an opportune moment to
>> ask/suggest :)
>
> Thanks for the idea. Yes, it was out of scope such suggestions for me
> originally, but thinking more about it, I take it more positively.
> Please let me (and others) to consider it for some days, cause this is
> very interesting proposal. Things to consider is how much effort it is
> to file in future, which format to use, etc.
>

It's fine with me if we just go with the original for now, but I think
the awkwardness (which is not your fault, but the nature of wrangling
free-form text) of the specification shows that we should at some point
move to something structured.

But I don't consider it a blocker for doing something better than
the status quo.

Maybe we'll be better off just going straight for Exherbo-style
p.mask in future: https://ciaranm.wordpress.com/2011/03/27/classifying-repository-masks/

Ultimately, text munging sucks and there's only so much we can do to
polish it. But your original proposal is a good improvement on how things are now.

>> --
>> Ninpo
>> 

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

>>>>> On Fri, 22 Sep 2023, Alex Boag-Munroe wrote:

> Perhaps eventually it could/should be used for the whole file but as
> an interim/beginning there's no reason you couldn't start with
> comments:

> # [PREAMBLE]
> # Timestamp: 2023-09-21 15:07:42+00:00
> # Author: Arthur Zamarin <arthurzam@gentoo.org>
> # Justification: Very broken, no idea why packaged, need to drop ASAP.
> #     The project is done with supporting this package.
> # Bugs: 667687, 667689
> # Packages: dev-lang/python
> dev-lang/python

And I thought my suggestion to use XML was far-fetched and an obvious
joke. :(

This seems rather restrictive, adds unnecessary redundancy, and would
make it hard to type an entry without the aid of special tools.

Also, there are other files like use.mask which probably shouldn't have
a completely different format. Their entries often have the author/date
line plus a one line comment which says all that is needed. Adding
massive header blocks there would be excessive.

IMHO Arthur's original proposal was fine. Let's not over-complicate
things.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Sam James]

Ulrich Mueller <ulm@gentoo.org> writes:

> [[PGP Signed Part:Undecided]]
>>>>>> On Fri, 22 Sep 2023, Alex Boag-Munroe wrote:
>
>> Perhaps eventually it could/should be used for the whole file but as
>> an interim/beginning there's no reason you couldn't start with
>> comments:
>
>> # [PREAMBLE]
>> # Timestamp: 2023-09-21 15:07:42+00:00
>> # Author: Arthur Zamarin <arthurzam@gentoo.org>
>> # Justification: Very broken, no idea why packaged, need to drop ASAP.
>> #     The project is done with supporting this package.
>> # Bugs: 667687, 667689
>> # Packages: dev-lang/python
>> dev-lang/python
>
> And I thought my suggestion to use XML was far-fetched and an obvious
> joke. :(

This is rather disrespectful to someone (who is a new contributor as
well) making a suggestion in earnest. You could've dropped this sentence
and the rest of your critique would stand.

Please consider phrasing in future.

>
> This seems rather restrictive, adds unnecessary redundancy, and would
> make it hard to type an entry without the aid of special tools.
>
> Also, there are other files like use.mask which probably shouldn't have
> a completely different format. Their entries often have the author/date
> line plus a one line comment which says all that is needed. Adding
> massive header blocks there would be excessive.
>
> IMHO Arthur's original proposal was fine. Let's not over-complicate
> things.
>
> Ulrich
>
> [[End of PGP Signed Part]]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Alex Boag-Munroe]

On Sat, 23 Sept 2023 at 08:03, Ulrich Mueller <ulm@gentoo.org> wrote:
<snip>
> This seems rather restrictive, adds unnecessary redundancy, and would
> make it hard to type an entry without the aid of special tools.
>
> Also, there are other files like use.mask which probably shouldn't have
> a completely different format. Their entries often have the author/date
> line plus a one line comment which says all that is needed. Adding
> massive header blocks there would be excessive.
>
> IMHO Arthur's original proposal was fine. Let's not over-complicate
> things.
>
> Ulrich

I'm confused, you're against adding "massive header blocks" but you're
fine with Arthur's 9 line entry but not my 8 line one.

My idea was a stop gap to add something easily parsed once the
comments are stripped but keeping the comments in place currently for
backwards compatibility.

Since parsing was part of the OP it made sense to me to suggest an
"already existing made for humans to read/write while tools can
already parse it" structure. What special tools do you think are
needed to write it? I wrote the above in Kate after double checking
the configparser docs. A standard template would be trivial.

--
Ninpo

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1781 bytes --]

>>>>> On Sat, 23 Sep 2023, Alex Boag-Munroe wrote:

> I'm confused, you're against adding "massive header blocks" but you're
> fine with Arthur's 9 line entry but not my 8 line one.

Your 8 line entry was this (please correct me if you meant to refer to
an entry from a different message):

--- 8< ---
# [PREAMBLE]
# Timestamp: 2023-09-21 15:07:42+00:00
# Author: Arthur Zamarin <arthurzam@gentoo.org>
# Justification: Very broken, no idea why packaged, need to drop ASAP.
#     The project is done with supporting this package.
# Bugs: 667687, 667689
# Packages: dev-lang/python
dev-lang/python
--- >8 ---

And Arthur's was this:

--- 8< ---
# Arthur Zamarin <arthurzam@gentoo.org> (2023-09-21)
# Very broken, no idea why packaged, need to drop ASAP. The project
# is done with supporting this package. See for history bug #667889.
#
# As a better plan, you should migrate to dev-lang/perl, which has
# better compatibility with dev-lang/ruby when used with dev-lang/lua
# bindings.
# Removal on 2023-10-21.  Bug #667687, #667689.
dev-lang/python
--- >8 ---

Of course it is longer when it contains 4 additional lines of
explanation.

> My idea was a stop gap to add something easily parsed once the
> comments are stripped but keeping the comments in place currently for
> backwards compatibility.

Yes, understood. I think we should keep the "simple line-based file"
format [1] with comments. If we would change it to something completely
different, we would also impose that format on users' (public or
private) overlays.

Also, we'd either have to change the files in /etc/portage (not sure how
popular that would be) or live with two different and incompatible
formats in profiles and user configuration.

Ulrich

[1] https://projects.gentoo.org/pms/8/pms.html#x1-480005.2.5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

[gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Jonas Stein]

[-- Attachment #1.1: Type: text/plain, Size: 1239 bytes --]

Hi all,


>> I want to suggest a standard format for profiles/package.mask, for
>> multiple reasons:
> 
> Sounds sensible. +1
> 
>> The first line of the "#"-prefixed explanation block must be of the
>> format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
>> format YYYY-MM-DD, in UTC timezone.
>                       ^^^^^^^^^^^^^^^^
> 
> Can we drop this? Or, at least, relax this.
> 
> I usually just enter my locale date here and like to avoid having to 
> think about that UTC is potentially in a different date. I also can not 
> remember any situation where the date being in UTC matters. Plus, if you 
> want accurate timestamps, then the git commit/author date is here for 
> you. :)

I think for the same reason UTC full timestamp makes sense:

The person who adds an entry to the package.mask file does not care 
about a few hours and simply rounds the time up.

The scripts which evaluate the .mask file have a proper time stamp. For 
a script it is trivial to work with time, if you have UTC in a standard 
format. But it is painful, if you just have a date and do not know the 
timezone.

So I think you and I really want UTC here.

-- 
Best,
Jonas

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 2401 bytes --]

>>>>> On Thu, 21 Sep 2023, Arthur Zamarin wrote:

> ===== "Formal" format =====

> Each entry is composed of 2 parts: "#"-prefixed explanation block and
> list of "${CATEGORY}/${PN}" packages. Entries are separated when a new
> explanation block starts (meaning first "#"-prefixed line after packages
> list). You may add newlines between packages in packages list.

"Must" rather than "may" here? You certainly cannot list several
packages in the same line.

> The first line of the "#"-prefixed explanation block must be of the
> format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
> format YYYY-MM-DD, in UTC timezone.

> If this is a last-rite message, the last line must list the last-rite
> last date (removal date) and the last-rite bug number. You can also list
> other bugs relevant to the last-rite. So I think a format of: "Removal
> on ${REMOVAL_DATE}.  Bug #NNNNNN, #NNNNNN." Where the bug list is comma
> and space separated, we have at least one space (" +" regex) between the
> removal date and bug list, and the date is of YYYY-MM-DD format.
> I prefer this line is separate (and not continuous of prefix message text).

> The explanation block itself can reference bugs, by matching the regex
> "[Bb]ugs? #\d+(, +#\d+)*" (For example: "bug #713106, #753134"). I think
> this is quite a simple one, but powerful enough for most.

> Lines with single newline between them (so no blank line between them)
> are considered as single paragraph continuum. If you want to start new
> paragraph, leave a blank line (still prefixed with #) - think similar to
> markdown. A line matching the last-rite line is always it's own paragraph.

Is this rule about paragraphs needed? It is at odds with the rule that
the removal date and bug must be on their own line (i.e. that line is
_not_ part of a "paragraph continuum").

What about the introductory comment block in the file? Should there be a
defined syntax for a separator between it and the rest of the file? For
example, everything above the first line matching "^#[ \t]*---" could be
ignored by automatic tools, and they would insert new entries below that
separator.

> Should it be a GLEP, I don't think so? But I'm unsure about it. We do
> need to document it (for example header of that exact file).

It shouldn't be too difficult to wrap this up as a GLEP. OTOH, we don't
have a GLEP for eclassdoc either.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Tim Harder]

On 2023-09-21 Thu 15:22, Ulrich Mueller wrote:
>>>>>> On Thu, 21 Sep 2023, Arthur Zamarin wrote:
>> Should it be a GLEP, I don't think so? But I'm unsure about it. We do
>> need to document it (for example header of that exact file).
>
>It shouldn't be too difficult to wrap this up as a GLEP.

To me standardizing a format in Gentoo (outside of PMS-related
functionality) requires a GLEP or at the very least some semi-formal
documentation outside the file in question in a place like the
devmanual. Consider it due diligence of the process that allows people
writing code to target the format without having to chase details down
into code bases or mailing list threads.

> OTOH, we don't have a GLEP for eclassdoc either.

This is a poor example since it's partly the reason why an awk script
with issues relating to extensibility and maintainability is still used
to generate eclass manpages.

I mainly let it slide when writing pkgcore/pkgcheck parsing
functionality because the devmanual [0] was a passable resource at the
time.

Tim

[0]: https://devmanual.gentoo.org/eclass-writing/#documenting-eclasses

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Sam James]

Tim Harder <radhermit@gentoo.org> writes:

> On 2023-09-21 Thu 15:22, Ulrich Mueller wrote:
>>>>>>> On Thu, 21 Sep 2023, Arthur Zamarin wrote:
>>> Should it be a GLEP, I don't think so? But I'm unsure about it. We do
>>> need to document it (for example header of that exact file).
>>
>>It shouldn't be too difficult to wrap this up as a GLEP.
>
> To me standardizing a format in Gentoo (outside of PMS-related
> functionality) requires a GLEP or at the very least some semi-formal
> documentation outside the file in question in a place like the
> devmanual. Consider it due diligence of the process that allows people
> writing code to target the format without having to chase details down
> into code bases or mailing list threads.

+1

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

On Thu, Sep 21, 2023 at 23:22:27 +0200, Ulrich Mueller wrote:
> >>>>> On Thu, 21 Sep 2023, Arthur Zamarin wrote:
> 
> > ===== "Formal" format =====
> 
> > Each entry is composed of 2 parts: "#"-prefixed explanation block and
> > list of "${CATEGORY}/${PN}" packages. Entries are separated when a new
> > explanation block starts (meaning first "#"-prefixed line after packages
> > list). You may add newlines between packages in packages list.
> 
> "Must" rather than "may" here? You certainly cannot list several
> packages in the same line.
> 

I read it to mean something like this would be allowed:

    # Blurb about package1, package2, and package3
    category/package1
    category/package2

    category/package3

Whether it makes sense to allow that is a different question.

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 3273 bytes --]

On 22/09/2023 00.22, Ulrich Mueller wrote:
>>>>>> On Thu, 21 Sep 2023, Arthur Zamarin wrote:
> 
>> ===== "Formal" format =====
> 
>> Each entry is composed of 2 parts: "#"-prefixed explanation block and
>> list of "${CATEGORY}/${PN}" packages. Entries are separated when a new
>> explanation block starts (meaning first "#"-prefixed line after packages
>> list). You may add newlines between packages in packages list.
> 
> "Must" rather than "may" here? You certainly cannot list several
> packages in the same line.

Agreed, poor choice of words.

>> The first line of the "#"-prefixed explanation block must be of the
>> format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
>> format YYYY-MM-DD, in UTC timezone.
> 
>> If this is a last-rite message, the last line must list the last-rite
>> last date (removal date) and the last-rite bug number. You can also list
>> other bugs relevant to the last-rite. So I think a format of: "Removal
>> on ${REMOVAL_DATE}.  Bug #NNNNNN, #NNNNNN." Where the bug list is comma
>> and space separated, we have at least one space (" +" regex) between the
>> removal date and bug list, and the date is of YYYY-MM-DD format.
>> I prefer this line is separate (and not continuous of prefix message text).
> 
>> The explanation block itself can reference bugs, by matching the regex
>> "[Bb]ugs? #\d+(, +#\d+)*" (For example: "bug #713106, #753134"). I think
>> this is quite a simple one, but powerful enough for most.
> 
>> Lines with single newline between them (so no blank line between them)
>> are considered as single paragraph continuum. If you want to start new
>> paragraph, leave a blank line (still prefixed with #) - think similar to
>> markdown. A line matching the last-rite line is always it's own paragraph.
> 
> Is this rule about paragraphs needed? It is at odds with the rule that
> the removal date and bug must be on their own line (i.e. that line is
> _not_ part of a "paragraph continuum").

Hmm, yeah, rereading my text shows I've over-complicated it. What I
wanted is that last paragraph (yes, if there are many bugs it might wrap
into new line) can be not separated with blank line from "main
explanation block".

> What about the introductory comment block in the file? Should there be a
> defined syntax for a separator between it and the rest of the file? For
> example, everything above the first line matching "^#[ \t]*---" could be
> ignored by automatic tools, and they would insert new entries below that
> separator.

Good point, and I should address it as you recommended. I will mention
the ignore-until-this-line, and that new entries should be added as
first entry after that ignore-until-this-line.

>> Should it be a GLEP, I don't think so? But I'm unsure about it. We do
>> need to document it (for example header of that exact file).
> 
> It shouldn't be too difficult to wrap this up as a GLEP. OTOH, we don't
> have a GLEP for eclassdoc either.

Yeah, after all the input, yes, I will work on a formal GLEP. It will
take time, but I hope to prepare a first draft in the coming 2 weeks.

> Ulrich

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

On Thu, Sep 21, 2023 at 22:40:05 +0300, Arthur Zamarin wrote:
> ===== "Formal" format =====
> 
> Each entry is composed of 2 parts: "#"-prefixed explanation block and
> list of "${CATEGORY}/${PN}" packages. Entries are separated when a new
> explanation block starts (meaning first "#"-prefixed line after packages
> list). You may add newlines between packages in packages list.
> 

What about mandatory blank line(s) between entries? That way it ensures
they are visually separated when skimming through the file. Plus, you
can easily jump from entry to entry in editors that support
paragraph-wise movement.

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

>>>>> On Fri, 22 Sep 2023, Oskari Pirhonen wrote:

>> Each entry is composed of 2 parts: "#"-prefixed explanation block and
>> list of "${CATEGORY}/${PN}" packages. Entries are separated when a new
>> explanation block starts (meaning first "#"-prefixed line after packages
>> list). You may add newlines between packages in packages list.

> What about mandatory blank line(s) between entries? That way it ensures
> they are visually separated when skimming through the file. Plus, you
> can easily jump from entry to entry in editors that support
> paragraph-wise movement.

Yes, please. Mandatory blank lines between entries, and no blank lines
(or lines containing only whitespace) within entries. Especially, no
blank lines in the list of packages.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] Standard parsable format for profiles/package.mask file

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 1195 bytes --]

On 22/09/2023 12.21, Ulrich Mueller wrote:
>>>>>> On Fri, 22 Sep 2023, Oskari Pirhonen wrote:
> 
>>> Each entry is composed of 2 parts: "#"-prefixed explanation block and
>>> list of "${CATEGORY}/${PN}" packages. Entries are separated when a new
>>> explanation block starts (meaning first "#"-prefixed line after packages
>>> list). You may add newlines between packages in packages list.
> 
>> What about mandatory blank line(s) between entries? That way it ensures
>> they are visually separated when skimming through the file. Plus, you
>> can easily jump from entry to entry in editors that support
>> paragraph-wise movement.
> 
> Yes, please. Mandatory blank lines between entries, and no blank lines
> (or lines containing only whitespace) within entries. Especially, no
> blank lines in the list of packages.

Yeah I agree. Originally I wanted to allow blank lines between packages
in same entry (to enable you to group them), but as further
considerations and your input, this is a bad idea (if you want to divide
the group, create separate entries).

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, pkgcore stack, Arch Teams, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 673 bytes --]

On 21/09/2023 21.40, Arthur Zamarin wrote:
> If this is a last-rite message, the last line must list the last-rite
> last date (removal date) and the last-rite bug number. You can also list

FWIW, I would assume the last-rite date to be the date where the 
package's last rites where initiated, i.e., where it was p.masked. The 
removal date would be the date on which the packages was cleaned from 
the tree.

> other bugs relevant to the last-rite. So I think a format of: "Removal
> on ${REMOVAL_DATE}.

I now we, myself included, often wrote "Removal on…", however, I wonder 
if we should reflect reality and write "Removal after…" instead.

- Flow

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 17273 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

[gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Jaco Kroon]

Hi,

On 2023/09/22 13:16, Florian Schmaus wrote:
> On 21/09/2023 21.40, Arthur Zamarin wrote:
>> If this is a last-rite message, the last line must list the last-rite
>> last date (removal date) and the last-rite bug number. You can also list
>
> FWIW, I would assume the last-rite date to be the date where the 
> package's last rites where initiated, i.e., where it was p.masked. The 
> removal date would be the date on which the packages was cleaned from 
> the tree.
>
>> other bugs relevant to the last-rite. So I think a format of: "Removal
>> on ${REMOVAL_DATE}.
>
> I now we, myself included, often wrote "Removal on…", however, I 
> wonder if we should reflect reality and write "Removal after…" instead.

+1

"on or after" would probably be most accurate though but gets very wordy.

Kind regards,
Jaco

[gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Jonas Stein]

[-- Attachment #1.1: Type: text/plain, Size: 148 bytes --]

Hi all,

> # Removal on 2023-10-21.  Bug #667687, #667689.

We should use "after" instead of "on":

# Removal after T

-- 
Best,
Jonas


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Ulrich Mueller]

>>>>> On Sun, 24 Sep 2023, Jonas Stein wrote:

>> # Removal on 2023-10-21.  Bug #667687, #667689.

> We should use "after" instead of "on":

> # Removal after T

I wonder if we even need to specify the wording in such detail. For any
tools parsing the file, it might be enough to say that the line must
contain, in this order:

- "Removal" (case insensitive?) as the first word,
- exactly one date in YYYY-MM-DD format,
- optionally, the word "bug" followed by one or more bug numbers.

Ulrich

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

On Mon, Sep 25, 2023 at 14:03:26 +0200, Ulrich Mueller wrote:
> >>>>> On Sun, 24 Sep 2023, Jonas Stein wrote:
> 
> >> # Removal on 2023-10-21.  Bug #667687, #667689.
> 
> > We should use "after" instead of "on":
> 
> > # Removal after T
> 
> I wonder if we even need to specify the wording in such detail. For any
> tools parsing the file, it might be enough to say that the line must
> contain, in this order:
> 
> - "Removal" (case insensitive?) as the first word,
> - exactly one date in YYYY-MM-DD format,
> - optionally, the word "bug" followed by one or more bug numbers.
> 

With the scheme above, the following would be valid (among other more
nonsensical looking things):

    # Removal before YYYY-MM-DD.  Bug #1, #2

IMO it definitely should require a word with similar meaning to "on" or
"after". How exactly that gets worded in the end, IDK. Maybe it'll end
up as "exactly one of 'on' or 'after'" or something like that.

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]