[gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 5442 bytes --]

Hey all,

here's the updated proposal for a new "arches.desc" file, after feedback.

In short:
We introduce a new file "arches.desc" which essentially describes if an arch 
(not a profile) is stable or not. The meaning of profiles.desc is not affected; 
profiles.desc still describes single profiles of each arch.
This is introduced in particular to help moving arches from stable to testing 
while keeping the "~arch" deptree consistent, or moving arches from testing to 
stable and easily preparing a consistent "arch" deptree.

Details:

1] File location:
profiles/arches.desc    or   metadata/arches.desc


2] File format: whitespace-separated, two columns, lines starting with # are 
comments
first column: arch name ("amd64")
second column: one of the three values "stable", "testing", "unstable"
Example:
===========================
# Example arches.desc file
amd64	stable
mips	testing
m68k	unstable
===========================


3] Meaning of the three values "stable", "testing", "unstable" for repoman

* stable: When a profile of arch is tested, then repoman checks consistency for 
"arch" and for "~arch" separately. 
Which profiles of the arch are tested is still controlled by profiles.desc (and 
-d / -e switches). 
This is the current behaviour and should be the default if nothing is specified 
for an arch.

* testing: When a profile of arch is tested, then repoman treats "arch" as 
"~arch", and tests consistency only for "~arch".
Which profiles of the arch are tested is still controlled by profiles.desc (and 
-d / -e switches). 
A new switch for repoman may be provided to temporarily upgrade an arch from 
"testing" to "stable" (for arch team work).

* unstable: When a profile of arch is tested, then repoman treats "arch" as an 
error and aborts. Consistency is only tested for "~arch".
Which profiles of the arch are tested is still controlled by profiles.desc (and 
-d / -e switches). 

* optionally, additional value "broken":
Do not test this arch at all. (In my opinion not necessary, since we can 
always set all profiles of an arch to exp.)


4] Meaning for other tools
All arches set to "stable" are considered "stable arches", meaning
* they get listed first in eshowkw
* they require stabilization requests in bugzilla
* ...


5] Initial value in the Gentoo repository
On introduction, the setting will be "stable" for all stable arches, "testing" 
for all arches where "inofficial" stable keywords exist (sh, s390, ...), and 
"unstable" everywhere else. 


6] Rationale
At the moment we have several cases where repoman finds its limits:

a) An arch loses its stable status (imagine s390), but the arch team doesnt 
want to drop all the stable keywords since they are useful for stage building. 
Since stable keywords on s390 are an arch-team-internal thing, everyone can 
drop the latest stable version of a package, and it's the arch team's 
responsibility to keep their "unofficial stable tree" straight. 
Right now this means that we have to set all *profiles* of this arch to "exp", 
otherwise repoman throws errors about a broken stable deptree. If we do that, 
repoman does however also not check ~s390 consistency, meaning that the 
~s390 deptree will soon be broken as well.
With arches.conf, one could set (arch: testing, profile: stable), which results 
in stable keywords being ignored, but the ~s390 deptree still being enforced.

b) An arch prepares for becoming a stable arch (think arm64).
So, first the ~arm64 deptree should be fine, and then stable keywords can be 
added. However, to avoid repoman errors as long as the stable deptree isnt 
complete yet the profiles need to be set to dev/exp, and again this brings the 
danger of the ~arm64 deptree getting inadvertently broken. Again the 
combination (arch: testing, profile: stable) helps.

Finally, at the moment the "semi-official" algorithm to figure out if an *arch* 
is stable (i.e., requires stabilization requests etc bla bla) is to check if 
the arch has any stable profile. This makes non-stable arches which want to 
keep a consistent deptree (think mips) impossible. Reading the arch status 
from arches.desc instead solves this problem.


7] Compatibility

a) No arches.desc and new system, or arch not listed in arches.desc
Arches are treated as "stable" by repoman (current behaviour), with profile 
status according to profiles.desc. So, business as usual.
Gentoolkit and other tools trying to determine a list of stable arches should 
fall back to current method of scanning profiles.desc for stable profiles.

b) arches.desc and old system
Tools ignore the unknown file (?).
Repoman and other tools may emit surplus dependency errors when profiles are 
checked on arches that are "testing" (they check the consistency of the stable 
tree alone, which is not OK, since "arch" is supposed to be treated like 
"~arch"). This affects only development work and can be fixed by updating 
repoman.

c) PMS may need to be amended to allow the additional file. 


8] Several repositories

If arches.desc is present in several repositories, then the strictest setting 
for an arch wins. [I don't really see many usecases for this though.]


Congratulations for getting this far. What's your opinion?

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer (council, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

On 26-03-2017 20:54:51 +0200, Andreas K. Huettel wrote:
> Congratulations for getting this far. What's your opinion?

When you say "arch" you actually mean a keyword as per GLEP-53[1] right?

Thanks,
Fabian


[1] https://wiki.gentoo.org/wiki/GLEP:53

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

>>>>> On Sun, 26 Mar 2017, Fabian Groffen wrote:

> When you say "arch" you actually mean a keyword as per GLEP-53[1]
> right?

Which doesn't agree with actual usage in the tree, though.

Ulrich

> [1] https://wiki.gentoo.org/wiki/GLEP:53

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 8224 bytes --]

On Sun, 26 Mar 2017 20:54:51 +0200
"Andreas K. Huettel" <dilfridge@gentoo.org> wrote:

> Hey all,
> 
> here's the updated proposal for a new "arches.desc" file, after
> feedback.
> 
> In short:
> We introduce a new file "arches.desc" which essentially describes if
> an arch (not a profile) is stable or not. The meaning of
> profiles.desc is not affected; profiles.desc still describes single
> profiles of each arch. This is introduced in particular to help
> moving arches from stable to testing while keeping the "~arch"
> deptree consistent, or moving arches from testing to stable and
> easily preparing a consistent "arch" deptree.
> 
> Details:
> 
> 1] File location:
> profiles/arches.desc    or   metadata/arches.desc
> 
> 
> 2] File format: whitespace-separated, two columns, lines starting
> with # are comments
> first column: arch name ("amd64")
> second column: one of the three values "stable", "testing", "unstable"
> Example:
> ===========================
> # Example arches.desc file
> amd64	stable
> mips	testing
> m68k	unstable
> ===========================
> 


While this is a simple format.  This is not a standard data input file
format for language tools to map it into native language variables and
types.

I would much prefer for any new files to be created in a format that
most languages have data input modules for and are easily read/edited
by humans.  While this can be read and split easily in python code.  It
is not future proof for additional data being added and/or removed.

 For the repoman stage3 rewrites. I am moving all configurable data
 from the code into yaml based files in /metadata/repoman.  These files
 will be easily edited by all developers for updates to banned eclasses
 and various other values not needing code changes.   

So with a general file name of arches.desc  Is there any other data
that we want to include in that file?  Possibly migrated from other
file(s).  In that case a dictionary format yaml file might be best.
My example below has additional info over what was proposed.  
It is an example only to show the possible benefit of such a file type.

  ie:
-----------------------------------------------------------------------
---
# the --- in the first line indicates this as a yaml file
# comments are allowed as well as whitespace
# DATA format for arches is dictionary

amd64:
    stability: stable
    bits: 64
    description:  Includes CPU manufaturers such as Intel, AMD, others...
    comments: The most common/popular arch in the tree...
    email: amd64@...

mips:
    stability: testing
    bits: 32
    description: Risc based achitecture ... (not fact checked)
    comments: Primarily used in some laptops and single board development systems...
    email: foo@...

m68k:
    stability: unstable
    bits: 8
    description: Motorola...
    comments: embedded...
    email: embedded@...
EOF
-----------------------------------------------------------------------------
The above example makes it easily modified to include new / remove obsolete data in the future.
The above list could add or remove the email field without affecting the proposed use of the 
stability field used by any applications.  That is not the case with a position dependent 
space separated list as was proposed.

Nearly all language types have native parsers for this type of file (no custom parser code required).
the above file info can easily be manipulated into various lists such as archlist, stable_arches, testing_arches,...
or any other data that is included for them.

> 
> 3] Meaning of the three values "stable", "testing", "unstable" for
> repoman
> 
> * stable: When a profile of arch is tested, then repoman checks
> consistency for "arch" and for "~arch" separately. 
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and -d / -e switches). 
> This is the current behaviour and should be the default if nothing is
> specified for an arch.
> 
> * testing: When a profile of arch is tested, then repoman treats
> "arch" as "~arch", and tests consistency only for "~arch".
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and -d / -e switches). 
> A new switch for repoman may be provided to temporarily upgrade an
> arch from "testing" to "stable" (for arch team work).
> 
> * unstable: When a profile of arch is tested, then repoman treats
> "arch" as an error and aborts. Consistency is only tested for "~arch".
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and -d / -e switches). 
> 
> * optionally, additional value "broken":
> Do not test this arch at all. (In my opinion not necessary, since we
> can always set all profiles of an arch to exp.)
> 
> 
> 4] Meaning for other tools
> All arches set to "stable" are considered "stable arches", meaning
> * they get listed first in eshowkw
> * they require stabilization requests in bugzilla
> * ...
> 
> 
> 5] Initial value in the Gentoo repository
> On introduction, the setting will be "stable" for all stable arches,
> "testing" for all arches where "inofficial" stable keywords exist
> (sh, s390, ...), and "unstable" everywhere else. 
> 
> 
> 6] Rationale
> At the moment we have several cases where repoman finds its limits:
> 
> a) An arch loses its stable status (imagine s390), but the arch team
> doesnt want to drop all the stable keywords since they are useful for
> stage building. Since stable keywords on s390 are an
> arch-team-internal thing, everyone can drop the latest stable version
> of a package, and it's the arch team's responsibility to keep their
> "unofficial stable tree" straight. Right now this means that we have
> to set all *profiles* of this arch to "exp", otherwise repoman throws
> errors about a broken stable deptree. If we do that, repoman does
> however also not check ~s390 consistency, meaning that the ~s390
> deptree will soon be broken as well. With arches.conf, one could set
> (arch: testing, profile: stable), which results in stable keywords
> being ignored, but the ~s390 deptree still being enforced.
> 
> b) An arch prepares for becoming a stable arch (think arm64).
> So, first the ~arm64 deptree should be fine, and then stable keywords
> can be added. However, to avoid repoman errors as long as the stable
> deptree isnt complete yet the profiles need to be set to dev/exp, and
> again this brings the danger of the ~arm64 deptree getting
> inadvertently broken. Again the combination (arch: testing, profile:
> stable) helps.
> 
> Finally, at the moment the "semi-official" algorithm to figure out if
> an *arch* is stable (i.e., requires stabilization requests etc bla
> bla) is to check if the arch has any stable profile. This makes
> non-stable arches which want to keep a consistent deptree (think
> mips) impossible. Reading the arch status from arches.desc instead
> solves this problem.
> 
> 
> 7] Compatibility
> 
> a) No arches.desc and new system, or arch not listed in arches.desc
> Arches are treated as "stable" by repoman (current behaviour), with
> profile status according to profiles.desc. So, business as usual.
> Gentoolkit and other tools trying to determine a list of stable
> arches should fall back to current method of scanning profiles.desc
> for stable profiles.
> 
> b) arches.desc and old system
> Tools ignore the unknown file (?).
> Repoman and other tools may emit surplus dependency errors when
> profiles are checked on arches that are "testing" (they check the
> consistency of the stable tree alone, which is not OK, since "arch"
> is supposed to be treated like "~arch"). This affects only
> development work and can be fixed by updating repoman.
> 
> c) PMS may need to be amended to allow the additional file. 
> 
> 
> 8] Several repositories
> 
> If arches.desc is present in several repositories, then the strictest
> setting for an arch wins. [I don't really see many usecases for this
> though.]
> 
> 
> Congratulations for getting this far. What's your opinion?
> 



-- 
Brian Dolbec <dolsen>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Ciaran McCreesh]

On Sun, 26 Mar 2017 13:04:18 -0700
Brian Dolbec <dolsen@gentoo.org> wrote:
> While this is a simple format.  This is not a standard data input file
> format for language tools to map it into native language variables and
> types.
> 
> I would much prefer for any new files to be created in a format that
> most languages have data input modules for and are easily read/edited
> by humans.  While this can be read and split easily in python code.
> It is not future proof for additional data being added and/or removed.
> 
>  For the repoman stage3 rewrites. I am moving all configurable data
>  from the code into yaml based files in /metadata/repoman.  These
> files will be easily edited by all developers for updates to banned
> eclasses and various other values not needing code changes.   
> 
> So with a general file name of arches.desc  Is there any other data
> that we want to include in that file?  Possibly migrated from other
> file(s).  In that case a dictionary format yaml file might be best.
> My example below has additional info over what was proposed.  
> It is an example only to show the possible benefit of such a file
> type.

It's bad enough that we have to parse XML inside the package mangler for
optional data. Adding YAML (with all its format bugs: YAML files
created with libyaml can't be read by syck, and vice-versa) for files
that the package mangler has to read is even worse.

Plain text *is* a standard format.

-- 
Ciaran McCreesh

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

>>>>> On Sun, 26 Mar 2017, Brian Dolbec wrote:

> I would much prefer for any new files to be created in a format that
> most languages have data input modules for and are easily read/edited
> by humans.  While this can be read and split easily in python code.  It
> is not future proof for additional data being added and/or removed.

>  For the repoman stage3 rewrites. I am moving all configurable data
>  from the code into yaml based files in /metadata/repoman.  These files
>  will be easily edited by all developers for updates to banned eclasses
>  and various other values not needing code changes.   

> So with a general file name of arches.desc  Is there any other data
> that we want to include in that file?  Possibly migrated from other
> file(s).  In that case a dictionary format yaml file might be best.
> My example below has additional info over what was proposed.  
> It is an example only to show the possible benefit of such a file type.

Please don't. Any such sophisticated file formats don't play well with
standard Unix tools like grep. Also the file will have have to be used
by tools like app-emacs/ebuild-mode, and to my knowledge there exists
no solid parser for yaml in elisp.

A simple text file with one record per line has the advantage that it
can be easily parsed in any language.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Walter Dnes]

On Sun, Mar 26, 2017 at 01:04:18PM -0700, Brian Dolbec wrote
>
> So with a general file name of arches.desc  Is there any other data
> that we want to include in that file?  Possibly migrated from other
> file(s).  In that case a dictionary format yaml file might be best.
> My example below has additional info over what was proposed.


On Sun, Mar 26, 2017 at 10:29:17PM +0200, Ulrich Mueller wrote
> 
> Please don't. Any such sophisticated file formats don't play well with
> standard Unix tools like grep. Also the file will have have to be used
> by tools like app-emacs/ebuild-mode, and to my knowledge there exists
> no solid parser for yaml in elisp.
> 
> A simple text file with one record per line has the advantage that it
> can be easily parsed in any language.

  A compromise that addresses both concerns.  How about a collection of
records in the format...

arch:attribute:data

  Examples

amd64:stability: stable
amd64:bits: 64
amd64:description:  Includes CPU manufaturers such as Intel, AMD, others...
amd64:comments: The most common/popular arch in the tree...
amd64:email: amd64@...

mips:stability: testing
mips:bits: 32
mips:description: Risc based achitecture ... (not fact checked)
mips:comments: Primarily used in some laptops and single board development systems...
mips:email: foo@...

m68k:stability: unstable
m68k:bits: 8
m68k:description: Motorola...
m68k:comments: embedded...
m68k:email: embedded@...

  Hopefully, there's no need to go JSON format (bleagh).  To simplify
parsing, fields for one arch should be clustered together.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

On 26-03-2017 22:02:38 +0200, Ulrich Mueller wrote:
> >>>>> On Sun, 26 Mar 2017, Fabian Groffen wrote:
> 
> > When you say "arch" you actually mean a keyword as per GLEP-53[1]
> > right?
> 
> Which doesn't agree with actual usage in the tree, though.

That surprises me.  Do you have an example of that?

Thanks,
Fabian

> 
> Ulrich
> 
> > [1] https://wiki.gentoo.org/wiki/GLEP:53



-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

On Sun, 26 Mar 2017 13:04:18 -0700
Brian Dolbec <dolsen@gentoo.org> wrote:

>   While this can be read and split easily in python code.  It
> is not future proof for additional data being added and/or removed.

This is why in my earlier comments to this proposal, I asked for

a: more descriptive terms than 'stable', 'unstable', or 'testing', because they're
all contextually ambiguous without inherently clear meaning

b: a format of <index>\s<fields> where <fields> was a list of space-delimited <key>=<value> pairs.

This at least means we stop relying on columns for data, and means that the data
is also trivially parseable with simple tools like grep/sed.

Whereas defining it as a multi-line YAML parser may seem great if you can
assume every tool at users disposal has YAML parsers and standard YAML parsing libraries,
but in reality, some of the tools at our disposal are "bash" and "sed", and decoding
and interpreting YAML from Bash is rather complicated.

( And there are other fun problems when you start talking about YAML )

Though, you could cheat and mandate a packed 1-line-per-arch YAML format.

This, iirc, is legal YAML:

amd64: {stability: "stable", bits: 64, description:  "Includes CPU manufaturers such as Intel, AMD, others...", comments: "The most common/popular arch in the tree...", email: "amd64@..." }


But at that point ... 

s/\b(([^=]+=(\S+)\b/{$1: "$2"}, / && parse_yaml ... 





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

On Sun, 26 Mar 2017 22:10:23 -0400
"Walter Dnes" <waltdnes@waltdnes.org> wrote:

> Hopefully, there's no need to go JSON format (bleagh).  To simplify
> parsing, fields for one arch should be clustered together.

"should be committed only after running it through sort with LC_ALL=C" 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

>>>>> On Mon, 27 Mar 2017, Fabian Groffen wrote:

>> > When you say "arch" you actually mean a keyword as per GLEP-53[1]
>> > right?
>> 
>> Which doesn't agree with actual usage in the tree, though.

> That surprises me.  Do you have an example of that?

The GLEP says about the OS suffix:

"The right hand part indicates the operating system or distribution,
such as linux, macos, solaris or fbsd. If the right hand part is
omitted, it implies the operating system/distribution type is
GNU/Linux."

So if I understand this correctly, x86-linux should be equivalent to
x86. But in reality, the linux suffix denotes that it is a prefix
arch. I'm not saying that this is bad, only it's not what the GLEP
says.

Until recently there was also x64-freebsd vs amd64-fbsd, where both
the arch and the OS part denoted the same, but used different tokens
to distinguish between prefix and non-prefix. (And I don't understand
why amd64 is called x64 on prefix. A different OS suffix should be
sufficient.)

Ulrich

>> > [1] https://wiki.gentoo.org/wiki/GLEP:53

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]

On 27-03-2017 09:56:50 +0200, Ulrich Mueller wrote:
> >>>>> On Mon, 27 Mar 2017, Fabian Groffen wrote:
> 
> >> > When you say "arch" you actually mean a keyword as per GLEP-53[1]
> >> > right?
> >> 
> >> Which doesn't agree with actual usage in the tree, though.
> 
> > That surprises me.  Do you have an example of that?
> 
> The GLEP says about the OS suffix:
> 
> "The right hand part indicates the operating system or distribution,
> such as linux, macos, solaris or fbsd. If the right hand part is
> omitted, it implies the operating system/distribution type is
> GNU/Linux."
> 
> So if I understand this correctly, x86-linux should be equivalent to
> x86. But in reality, the linux suffix denotes that it is a prefix
> arch. I'm not saying that this is bad, only it's not what the GLEP
> says.

I see.  The lack of explicit mentioning what the difference means allows
for different interpretations.  I always *assumed* it meant Gentoo (1
part) vs Gentoo/Alt (2 parts).

> Until recently there was also x64-freebsd vs amd64-fbsd, where both
> the arch and the OS part denoted the same, but used different tokens
> to distinguish between prefix and non-prefix. (And I don't understand
> why amd64 is called x64 on prefix. A different OS suffix should be
> sufficient.)

It kind of proves the point that two fields in a keyword isn't "enough
for everyone".

Back to the topic of the thread, is it possible to make the difference
between e.g. x86, x86-linux, x86-solaris and x86-macos in this proposal?

Thanks,
Fabian


> >> > [1] https://wiki.gentoo.org/wiki/GLEP:53


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Mart Raudsepp]

Ühel kenal päeval, E, 27.03.2017 kell 11:07, kirjutas Fabian Groffen:
> On 27-03-2017 09:56:50 +0200, Ulrich Mueller wrote:
> > > > > > > On Mon, 27 Mar 2017, Fabian Groffen wrote:
> > > > > When you say "arch" you actually mean a keyword as per GLEP-
> > > > > 53[1]
> > > > > right?
> > > > 
> > > > Which doesn't agree with actual usage in the tree, though.
> > > That surprises me.  Do you have an example of that?
> > 
> > The GLEP says about the OS suffix:
> > 
> > "The right hand part indicates the operating system or
> > distribution,
> > such as linux, macos, solaris or fbsd. If the right hand part is
> > omitted, it implies the operating system/distribution type is
> > GNU/Linux."
> > 
> > So if I understand this correctly, x86-linux should be equivalent
> > to
> > x86. But in reality, the linux suffix denotes that it is a prefix
> > arch. I'm not saying that this is bad, only it's not what the GLEP
> > says.
> 
> I see.  The lack of explicit mentioning what the difference means
> allows
> for different interpretations.  I always *assumed* it meant Gentoo (1
> part) vs Gentoo/Alt (2 parts).
> 
> > Until recently there was also x64-freebsd vs amd64-fbsd, where both
> > the arch and the OS part denoted the same, but used different
> > tokens
> > to distinguish between prefix and non-prefix. (And I don't
> > understand
> > why amd64 is called x64 on prefix. A different OS suffix should be
> > sufficient.)
> 
> It kind of proves the point that two fields in a keyword isn't
> "enough
> for everyone".
> 
> Back to the topic of the thread, is it possible to make the
> difference
> between e.g. x86, x86-linux, x86-solaris and x86-macos in this
> proposal?
> 

I believe the intention here is for this file to declare stuff about an
"arch" in terms of what repoman names it as such (--include-arches=,
etc), and what profiles.desc has as the first column value (in comments
it also names that column "arch").
The filename "arches.desc" just comes from that convention, while
indeed it really matches what you put in KEYWORDS in terms of ebuild
usage. I guess that filename is a shed to paint then too.


Mart

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

>>>>> On Mon, 27 Mar 2017, Mart Raudsepp wrote:

> Ühel kenal päeval, E, 27.03.2017 kell 11:07, kirjutas Fabian Groffen:
>> Back to the topic of the thread, is it possible to make the
>> difference between e.g. x86, x86-linux, x86-solaris and x86-macos
>> in this proposal?

> I believe the intention here is for this file to declare stuff
> about an "arch" in terms of what repoman names it as such
> (--include-arches=, etc), and what profiles.desc has as the first
> column value (in comments it also names that column "arch").

> The filename "arches.desc" just comes from that convention, while
> indeed it really matches what you put in KEYWORDS in terms of ebuild
> usage. I guess that filename is a shed to paint then too.

It also agrees with the PMS usage of the term "architecture":
https://projects.gentoo.org/pms/6/pms.html#x1-610005.3.2
https://projects.gentoo.org/pms/6/pms.html#x1-690007.3.2

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Mart Raudsepp]

This looks good overall, thanks.

If we stay with the whitespace separated columns, the spec should be
clear that implementations should be able to deal with future
additional "columns" in their parsing code.

Below some paint choices from me.

> We introduce a new file "arches.desc" which essentially describes if
> an arch 
> (not a profile) is stable or not. The meaning of profiles.desc is not
> affected;

Essentially the proposal extends profiles/arch.list but due to
backwards compatibility can't just add details there.
As such, in my opinion the file should be called arch.desc (not plural
arches.desc) to go along with that.

> 1] File location:
> profiles/arches.desc    or   metadata/arches.desc

profiles/arch.desc or metadata/repoman/arch.desc

> 3] Meaning of the three values "stable", "testing", "unstable" for
> repoman
> 
> * stable: When a profile of arch is tested, then repoman checks
> consistency for 
> "arch" and for "~arch" separately. 
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and 
> -d / -e switches). 
> This is the current behaviour and should be the default if nothing is
> specified 
> for an arch.
> 
> * testing: When a profile of arch is tested, then repoman treats
> "arch" as 
> "~arch", and tests consistency only for "~arch".
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and 
> -d / -e switches). 
> A new switch for repoman may be provided to temporarily upgrade an
> arch from 
> "testing" to "stable" (for arch team work).
> 
> * unstable: When a profile of arch is tested, then repoman treats
> "arch" as an 
> error and aborts. Consistency is only tested for "~arch".
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and 
> -d / -e switches). 

This sounds more like "testing" to me - architecture is only meant to
have "testing" keywords, which is what I tend to call ~arch because
it's in testing to become "stable" in ~30days or so, instead of calling
it "unstable" (which feels appropriate only for a package that doesn't
carry any stable keywords in older versions either).
While taken from another perspective, the meaning for "testing" as in
this proposal makes sense too - treat all as "testing" keywords.
This goes back to the overloaded terminology concerns that have been
echoed by others as well.
But I don't have any good suggestions for alternatives either right
now. stable/no_stable_check/testing_only? shrug.

> 4] Meaning for other tools
> All arches set to "stable" are considered "stable arches", meaning
> * they get listed first in eshowkw
> * they require stabilization requests in bugzilla
> * ...

If other tools use this, then maybe the repoman specific
metadata/repoman/ path isn't appropriate afterall. So then
profiles/arch.desc or metadata/arch.desc


Mart

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[M. J. Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1989 bytes --]

On 27/03/17 11:10, Mart Raudsepp wrote:
>
>> 3] Meaning of the three values "stable", "testing", "unstable" for
>> repoman
>>
>> * stable: When a profile of arch is tested, then repoman checks
>> consistency for 
>> "arch" and for "~arch" separately. 
>> Which profiles of the arch are tested is still controlled by
>> profiles.desc (and 
>> -d / -e switches). 
>> This is the current behaviour and should be the default if nothing is
>> specified 
>> for an arch.
>>
>> * testing: When a profile of arch is tested, then repoman treats
>> "arch" as 
>> "~arch", and tests consistency only for "~arch".
>> Which profiles of the arch are tested is still controlled by
>> profiles.desc (and 
>> -d / -e switches). 
>> A new switch for repoman may be provided to temporarily upgrade an
>> arch from 
>> "testing" to "stable" (for arch team work).
>>
>> * unstable: When a profile of arch is tested, then repoman treats
>> "arch" as an 
>> error and aborts. Consistency is only tested for "~arch".
>> Which profiles of the arch are tested is still controlled by
>> profiles.desc (and 
>> -d / -e switches). 
> This sounds more like "testing" to me - architecture is only meant to
> have "testing" keywords, which is what I tend to call ~arch because
> it's in testing to become "stable" in ~30days or so, instead of calling
> it "unstable" (which feels appropriate only for a package that doesn't
> carry any stable keywords in older versions either).
> While taken from another perspective, the meaning for "testing" as in
> this proposal makes sense too - treat all as "testing" keywords.
> This goes back to the overloaded terminology concerns that have been
> echoed by others as well.
> But I don't have any good suggestions for alternatives either right
> now. stable/no_stable_check/testing_only? shrug.
>
>
'unstable' should surely be applied to masked packages, no? Everything
not-stable and not-unstable becomes therefore 'testing' ...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Tue, 28 Mar 2017 00:00:30 +0100
"M. J. Everitt" <m.j.everitt@iee.org> wrote:

> 'unstable' should surely be applied to masked packages, no? Everything
> not-stable and not-unstable becomes therefore 'testing' ...

Nah, he's trying to make the phrase "stable arch" mean something
in a way tools can understand.

Because we currently have stable arches as a concept, but as far
as portage is concerned, we only have stable *profiles*, but we can
only identify specific profiles with arches ... which means ...

We can't have a value of ~arch that we can test without also
implying the experimental profiles of that arch that don't matter.

Hence, 

stable - what it currently means

testing - for architectures where there will be no promises
          beyond "Somebody tested it once" and a 'stable' KEYWORD
          value does not mean anything more than a '~' KEYWORD value.

          Where the objective is to make sure at least for an architecture
          developers should spend effort to keep that keywording in place,
          but not ever bother with stabilizing.

unstable - This architecture is so undermaintained that no encouragement
           is made of developers to keep keywords consistent, and they can be freely
           ignored.

This is why I preferred alternative wording that was descriptive of what
its doing instead of so obscure and generic and over-conflated.

   keyword-consistency=literal-match  # 'stable'

   keyword-consistency=mixed  # 'testing'

   keyword-consistency=none # 'unstable'

Or something along those lines.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]