[gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4378 bytes --]

Hello, everyone.

Currently PMS defines two variables that are being repeatedly abused to
access repository data in unpredictable and breaking manners -- PORTDIR
and ECLASSDIR. They both reference only so-called 'master
repository', are permitted in source builds and src_* phases only.

For quite some time, QA is monitoring their use and repeatedly
reporting abuses and spec violations. I'd like to run a joint QA & PMS
team effort in cleaning up those variables for sane multi-repository
support or banning them altogether. For this reason, I would like to
know your opinion.


Licenses [1]
------------

So far, the most common use of ${PORTDIR} was to access the licenses
subdirectory. That has a number of issues -- most importantly, it fails
when the license is provided by another repository. It is also unusable
in binary packages.

So far I see two major possibilities here. We can either decide that:

a. ebuilds don't need to access licenses directly and if they do,
the licenses are usually included in distfiles or can be obtained
independently of ebuild tree, or

b. we provide a proper, safe mechanism for obtaining licenses that
works with multiple repositories and binary packages. In particular,
I was thinking of establishing a LICENSEDIR that would contain copies
or symlinks to all needed licenses, both in source and binary installs.

Few relevant current bugs:

- sci-visualization/veusz: replaces bundled license with PORTDIR
  symlink, which will fail if repository is moved or when using binary
  packages [2],

- sci-biology/foldingathome: tries to reference the license
  in pkg_setup(), while PORTDIR is valid only in src_* [3],

- sys-block/tw_cli: copies license into install. The license is not
  included in tarball, yet we have different opinions whether copying
  it is necessary or not [4].


Eclasses [5]
------------

The next thing is ECLASSDIR, sometimes disguised as equivalent
${PORTDIR}/eclass. It is used only by a single developer, for two
reasons. One is to create eclass-manpages whose ebuild is a huge hack
relying on random deprecated Portage variables anyway, so not worth
noting. The other is to access a huge collection of patches (over 100
files) which are stored in eclass/ELT-patches and not ever used on most
of Gentoo systems.

We've considered different options on how to make ECLASSDIR sane and so
far seems there's no real way of doing it while keeping the ELT-patches
thing working (the way suggested for licenses won't work, for example).
So for less sane solutions, again we have two:

A. ban ECLASSDIR completely, and move ELT-patches to a dedicated
package [6]. This way, it could be cleanly managed, versioned
and filtered to install only files relevant to user's system (i.e. not
for all potential prefixes we support), and libtool.eclass can simply
DEPEND on it. Furthermore, a lot of the code could be moved to
a reusable external patcher tool.

B. Restrict ECLASSDIR to be available only in global scope of
eclasses (i.e. while sourcing them), and set it to the repository from
which eclass is sourced. This is ugly, barely predictable but should
work. Well, as long as you copy the whole ELT-patches structure along
with libtool.eclass.


PORTDIR [7]
-----------

Almost all uses of PORTDIR are covered by the two above categories.
The only remaining use is games-mods.eclass using it to read header.txt
file from the repository [8]. Which in fact could be trivially replaced
if games team members had a little different attitude towards QA but
I guess that's not my problem.

Therefore, I think that if both of the above issues are solved either
way, PORTDIR should be banned completely. Until then, we'd like it to
be QA-deprecated and discouraged from being used unless absolutely
necessary.


What are your thoughts?


[1]:https://bugs.gentoo.org/show_bug.cgi?id=566412
[2]:https://bugs.gentoo.org/show_bug.cgi?id=341653
[3]:https://bugs.gentoo.org/show_bug.cgi?id=566402
[4]:https://bugs.gentoo.org/show_bug.cgi?id=566416
[5]:https://bugs.gentoo.org/show_bug.cgi?id=373351
[6]:https://bugs.gentoo.org/show_bug.cgi?id=566424
[7]:https://bugs.gentoo.org/show_bug.cgi?id=373349
[8]:https://bugs.gentoo.org/show_bug.cgi?id=416739

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[NP-Hardass]

[-- Attachment #1: Type: text/plain, Size: 5132 bytes --]

Probably not the ideal solution given that you seem to prefer removal of such variables, but a REPODIR variable which is set to the directory where the repo is (basically making PORTDIR dynamic and setting it on a per package basis) could enable developers to reference their repo when needed, allowing variable use in a multi repo system. Additionally, if that idea is liked, I think an array of the repos masters and/or their dirs (or some functions to access that information) could also be useful. Like get_masters and get_repo_dir functions.

Just some ideas and I have no real attachments, but just figured I would brainstorm and put them out there. 

On November 21, 2015 4:36:57 PM EST, "Michał Górny" <mgorny@gentoo.org> wrote:
>Hello, everyone.
>
>Currently PMS defines two variables that are being repeatedly abused to
>access repository data in unpredictable and breaking manners -- PORTDIR
>and ECLASSDIR. They both reference only so-called 'master
>repository', are permitted in source builds and src_* phases only.
>
>For quite some time, QA is monitoring their use and repeatedly
>reporting abuses and spec violations. I'd like to run a joint QA & PMS
>team effort in cleaning up those variables for sane multi-repository
>support or banning them altogether. For this reason, I would like to
>know your opinion.
>
>
>Licenses [1]
>------------
>
>So far, the most common use of ${PORTDIR} was to access the licenses
>subdirectory. That has a number of issues -- most importantly, it fails
>when the license is provided by another repository. It is also unusable
>in binary packages.
>
>So far I see two major possibilities here. We can either decide that:
>
>a. ebuilds don't need to access licenses directly and if they do,
>the licenses are usually included in distfiles or can be obtained
>independently of ebuild tree, or
>
>b. we provide a proper, safe mechanism for obtaining licenses that
>works with multiple repositories and binary packages. In particular,
>I was thinking of establishing a LICENSEDIR that would contain copies
>or symlinks to all needed licenses, both in source and binary installs.
>
>Few relevant current bugs:
>
>- sci-visualization/veusz: replaces bundled license with PORTDIR
>  symlink, which will fail if repository is moved or when using binary
>  packages [2],
>
>- sci-biology/foldingathome: tries to reference the license
>  in pkg_setup(), while PORTDIR is valid only in src_* [3],
>
>- sys-block/tw_cli: copies license into install. The license is not
>  included in tarball, yet we have different opinions whether copying
>  it is necessary or not [4].
>
>
>Eclasses [5]
>------------
>
>The next thing is ECLASSDIR, sometimes disguised as equivalent
>${PORTDIR}/eclass. It is used only by a single developer, for two
>reasons. One is to create eclass-manpages whose ebuild is a huge hack
>relying on random deprecated Portage variables anyway, so not worth
>noting. The other is to access a huge collection of patches (over 100
>files) which are stored in eclass/ELT-patches and not ever used on most
>of Gentoo systems.
>
>We've considered different options on how to make ECLASSDIR sane and so
>far seems there's no real way of doing it while keeping the ELT-patches
>thing working (the way suggested for licenses won't work, for example).
>So for less sane solutions, again we have two:
>
>A. ban ECLASSDIR completely, and move ELT-patches to a dedicated
>package [6]. This way, it could be cleanly managed, versioned
>and filtered to install only files relevant to user's system (i.e. not
>for all potential prefixes we support), and libtool.eclass can simply
>DEPEND on it. Furthermore, a lot of the code could be moved to
>a reusable external patcher tool.
>
>B. Restrict ECLASSDIR to be available only in global scope of
>eclasses (i.e. while sourcing them), and set it to the repository from
>which eclass is sourced. This is ugly, barely predictable but should
>work. Well, as long as you copy the whole ELT-patches structure along
>with libtool.eclass.
>
>
>PORTDIR [7]
>-----------
>
>Almost all uses of PORTDIR are covered by the two above categories.
>The only remaining use is games-mods.eclass using it to read header.txt
>file from the repository [8]. Which in fact could be trivially replaced
>if games team members had a little different attitude towards QA but
>I guess that's not my problem.
>
>Therefore, I think that if both of the above issues are solved either
>way, PORTDIR should be banned completely. Until then, we'd like it to
>be QA-deprecated and discouraged from being used unless absolutely
>necessary.
>
>
>What are your thoughts?
>
>
>[1]:https://bugs.gentoo.org/show_bug.cgi?id=566412
>[2]:https://bugs.gentoo.org/show_bug.cgi?id=341653
>[3]:https://bugs.gentoo.org/show_bug.cgi?id=566402
>[4]:https://bugs.gentoo.org/show_bug.cgi?id=566416
>[5]:https://bugs.gentoo.org/show_bug.cgi?id=373351
>[6]:https://bugs.gentoo.org/show_bug.cgi?id=566424
>[7]:https://bugs.gentoo.org/show_bug.cgi?id=373349
>[8]:https://bugs.gentoo.org/show_bug.cgi?id=416739
>
>-- 
>Best regards,
>Michał Górny
><http://dev.gentoo.org/~mgorny/>

-- 
NP-Hardass

[-- Attachment #2: Type: text/html, Size: 6240 bytes --]

Re: [gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 21/11/15 06:29 PM, NP-Hardass wrote:
> Probably not the ideal solution given that you seem to prefer
> removal of such variables, but a REPODIR variable which is set to
> the directory where the repo is (basically making PORTDIR dynamic
> and setting it on a per package basis) could enable developers
> to reference their repo when needed, allowing variable use in a
> multi repo system. Additionally, if that idea is liked, I think
> an array of the repos masters and/or their dirs (or some
> functions to access that information) could also be useful. Like
> get_masters and get_repo_dir functions.
> 

- From your description here REPODIR would only point to the current
repo, so for licence-file access when the license is in the main
gentoo repo, but the current repo is an overlay, it would still fail.

Similar cases could occur for eclasses, especially since 'masters'
in repo metadata allows multiple repos to be chained.

All told, I think i'm in favour of banning the variables, and
potentially providing getter functions that would output the path of
these files if they need to be accessed -- '$(get_eclasspath
[name])' or $(get_licensepath [name]) or the like.  I don't know if
these could be implemented in-eclass or if they would be something
that would have to be added to EAPI7..?



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlZRAu4ACgkQAJxUfCtlWe1r4wEA3bnt0LtExuJCFTFMzZfzoTgl
q+T0eEPvKw0HK3De9rsA/iZZlZ0VKBTErx7mXk4YjiykN9Ruk/ZswkDDSGWM29IW
=JEGh
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

On Sat, 21 Nov 2015 18:29:31 -0500
NP-Hardass <NP-Hardass@gentoo.org> wrote:

> Probably not the ideal solution given that you seem to prefer removal of such variables, but a REPODIR variable which is set to the directory where the repo is (basically making PORTDIR dynamic and setting it on a per package basis) could enable developers to reference their repo when needed, allowing variable use in a multi repo system. Additionally, if that idea is liked, I think an array of the repos masters and/or their dirs (or some functions to access that information) could also be useful. Like get_masters and get_repo_dir functions.

I'd really prefer if we implemented stuff that people need rather than
random ideas that people may one day find some abusive use for. Plus,
we don't want to bind EAPI into specific repository structure. Plus, as
axs pointed out, that doesn't help anyone unless they want to
reimplement whole masters logic in every ebuild needing that.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]

On Sat, 21 Nov 2015 18:49:03 -0500
Ian Stakenvicius <axs@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 21/11/15 06:29 PM, NP-Hardass wrote:
> > Probably not the ideal solution given that you seem to prefer
> > removal of such variables, but a REPODIR variable which is set to
> > the directory where the repo is (basically making PORTDIR dynamic
> > and setting it on a per package basis) could enable developers
> > to reference their repo when needed, allowing variable use in a
> > multi repo system. Additionally, if that idea is liked, I think
> > an array of the repos masters and/or their dirs (or some
> > functions to access that information) could also be useful. Like
> > get_masters and get_repo_dir functions.
> >   
> 
> - From your description here REPODIR would only point to the current
> repo, so for licence-file access when the license is in the main
> gentoo repo, but the current repo is an overlay, it would still fail.
> 
> Similar cases could occur for eclasses, especially since 'masters'
> in repo metadata allows multiple repos to be chained.

There's also the eclass-overrides problem.

> All told, I think i'm in favour of banning the variables, and
> potentially providing getter functions that would output the path of
> these files if they need to be accessed -- '$(get_eclasspath
> [name])' or $(get_licensepath [name]) or the like.  I don't know if
> these could be implemented in-eclass or if they would be something
> that would have to be added to EAPI7..?

get_eclasspath doesn't solve the issue of storing random stuff under
eclass/ subdirectories, I think. And if I understand your idea
correctly, get_licensepath would still not work for binary packages.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[Martin Vaeth]

Ian Stakenvicius <axs@gentoo.org> wrote:
> '$(get_eclasspath [name])' or $(get_licensepath [name]) or the like

Since we are at a brainstorm level:

How about filling associative arrays instead?

This would BTW also provide the names of the master repositories
(although one could perhaps do the latter more explicit in an array,
too). I suppose that this cannot be done without a new EAPI.

Re: [gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[Daniel Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/21/2015 01:36 PM, Michał Górny wrote:
> Hello, everyone.
> 
> Currently PMS defines two variables that are being repeatedly
> abused to access repository data in unpredictable and breaking
> manners -- PORTDIR and ECLASSDIR. They both reference only
> so-called 'master repository', are permitted in source builds and
> src_* phases only.
> 
> For quite some time, QA is monitoring their use and repeatedly 
> reporting abuses and spec violations. I'd like to run a joint QA &
> PMS team effort in cleaning up those variables for sane
> multi-repository support or banning them altogether. For this
> reason, I would like to know your opinion.
> 
> 
> Licenses [1] ------------
> 
> So far, the most common use of ${PORTDIR} was to access the
> licenses subdirectory. That has a number of issues -- most
> importantly, it fails when the license is provided by another
> repository. It is also unusable in binary packages.
> 
> So far I see two major possibilities here. We can either decide
> that:
> 
> a. ebuilds don't need to access licenses directly and if they do, 
> the licenses are usually included in distfiles or can be obtained 
> independently of ebuild tree, or
> 
> b. we provide a proper, safe mechanism for obtaining licenses that 
> works with multiple repositories and binary packages. In
> particular, I was thinking of establishing a LICENSEDIR that would
> contain copies or symlinks to all needed licenses, both in source
> and binary installs.
> 
> [snip]
> 

I have no opinions on the others since I'm still rather inexperienced,
but LICENSEDIR seems like a sane solution to the licenses problem and
should be able to handle the multi-repo use case.
- -- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWUrujAAoJEAEkDpRQOeFwqd8QAKca2FvSB9iLHDwGnwh/8c14
CoFxAn6vwXUvtCkZBqCzwKsaDKoSNWPxjCki6MQyYWb4KMKH++952vx3T5ILOqV5
6vo7PfvG3bjTOaES/XeRfWt+pjYUfGAfJ5UAfXECH7mrF7ZmA6EEyTrFr1SjLvVR
jlnk+6Os8va/n8+KjZzAjhcf9BnNChv7MEwZPFrJFESjxr9Cx7Icie41jLfrOqKq
grRL/F4RJJrsn1Hlfo8MWrDnLJDlAYNg8BdXzCzR7s6DfpGqTuqXdr2+WtHcNJfh
vhKMeiTF8VJCRZgrVX6J8+3RMHVwiT4PaImf5SqX/xD692z+Og4Rl0sTb1IcJDSH
pg0QkYp/ugp88GrXLvIVsaMQTr2wgMK9IZpibK6VfX8mpTHKBRBJxAZYZTHyK0mM
+qI5qL3Q9XMTCIEGJrpGf+48T34iHxIyI2Th5jPn/SqyHGLSyEkHoRI/Yr9txy6n
6/fZhuy9iEPkZNdEmzG8OOSvsObIvTGrDCbKMSaW9cEovEXmBgDFwLWViEwC5Dlj
MK0OKkwASV9ukzyOJq28qf7/S+yt/eHlgSwo3xPyLBxdZ8ulpLoJq5SE0LbNU+4v
DUV8EBt2e6NBlKuHXkPMp5sCu1t4oInYe16jXqU9N2j0dbluphuTLXdSIt0QD34t
lKXAnfBcf55t2nj4f3mT
=uAl3
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [EAPI 7] The future goals for PORTDIR, ECLASSDIR and... LICENSEDIR?

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

>>>>> On Sun, 22 Nov 2015, Michał Górny wrote:

> On Sat, 21 Nov 2015 18:49:03 -0500
> Ian Stakenvicius <axs@gentoo.org> wrote:

>> All told, I think i'm in favour of banning the variables, and
>> potentially providing getter functions that would output the path
>> of these files if they need to be accessed -- '$(get_eclasspath
>> [name])' or $(get_licensepath [name]) or the like.  I don't know if
>> these could be implemented in-eclass or if they would be something
>> that would have to be added to EAPI7..?

The more I think about the idea of having a LICENSEDIR containing
symlinks, the less I like it. It would add the overhead of creating
the dir to all ebuilds, whereas it would be used in a handful of cases
only. Also the symlinks won't be permanent, so postinst messages like
"read the license at ${LICENSEDIR}/GPL-2" cannot be used, unless the
ebuild would follow the link (e.g. using readlink) which seems clumsy.

So IMO a function like get_license_path is the better solution here.

> And if I understand your idea correctly, get_licensepath would still
> not work for binary packages.

Is there a scenario where LICENSEDIR would work, but the function
would not?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]