[gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1294 bytes --]

Hi all,

The ~arch version of portage hs a new QA check that reports installation
of files outside of directories that have been whitelisted [1]. The
current whitelist includes:

directories common to / and /usr
================================
bin lib lib32 lib64 libx32 sbin

top level directories
================================
boot dev etc opt srv usr var

/usr level directories
================================
include libexec share src

/usr/share/doc level directories
================================
/usr/share/doc/${PF}

The first bug report [2] is for qt-core, which installs documentation
into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
includes ebuild revision such as -r1, -r2, and so on).

I have created a patch that will allow ebuilds to whitelist directories
by setting a QA_INSTALL_PATHS variable [3], however @mgorny said:

On Mon, Oct 1, 2018 at 12:34 AM Michał Górny <mgorny@gentoo.org> wrote:
> QA NAK on this.  The purpose of QA checks is not for ebuilds to
> officially override them because histerical raisins.

[1]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=d5f97eaa464736a454c8ad410f4acd4fccdf2324
[2] https://bugs.gentoo.org/667280
[3] https://bugs.gentoo.org/667378
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Mike Gilbert]

On Mon, Oct 1, 2018 at 11:19 AM Zac Medico <zmedico@gentoo.org> wrote:
>
> Hi all,
>
> The ~arch version of portage hs a new QA check that reports installation
> of files outside of directories that have been whitelisted [1]. The
> current whitelist includes:
>
> directories common to / and /usr
> ================================
> bin lib lib32 lib64 libx32 sbin
>
> top level directories
> ================================
> boot dev etc opt srv usr var
>
> /usr level directories
> ================================
> include libexec share src
>
> /usr/share/doc level directories
> ================================
> /usr/share/doc/${PF}
>
> The first bug report [2] is for qt-core, which installs documentation
> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
> includes ebuild revision such as -r1, -r2, and so on).
>
> I have created a patch that will allow ebuilds to whitelist directories
> by setting a QA_INSTALL_PATHS variable [3], however @mgorny said:
>
> On Mon, Oct 1, 2018 at 12:34 AM Michał Górny <mgorny@gentoo.org> wrote:
> > QA NAK on this.  The purpose of QA checks is not for ebuilds to
> > officially override them because histerical raisins.

I don't think QA has the final word on features that are added to the
package manager. They can prevent abuse of those features by ebuilds
in the repository, however.

Personally, I think compatibility with historical code is a perfectly
valid reason to make an exception here.

[gentoo-dev] Re: RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Michael Orlitzky]

On 10/01/2018 11:19 AM, Zac Medico wrote:
> 
> The first bug report [2] is for qt-core, which installs documentation
> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF}

It would be helpful to know why qtcore installs into the wrong directory
in the first place.

If ${PF} really causes serious problems, then would a symlink from
${PN}-${PV} to ${PF} fare any better?

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1788 bytes --]

On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
> Hi all,
> 
> The ~arch version of portage hs a new QA check that reports installation
> of files outside of directories that have been whitelisted [1]. The
> current whitelist includes:
> 
> directories common to / and /usr
> ================================
> bin lib lib32 lib64 libx32 sbin
> 
> top level directories
> ================================
> boot dev etc opt srv usr var
> 
> /usr level directories
> ================================
> include libexec share src
> 
> /usr/share/doc level directories
> ================================
> /usr/share/doc/${PF}
> 
> The first bug report [2] is for qt-core, which installs documentation
> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
> includes ebuild revision such as -r1, -r2, and so on).

No, it doesn't.  There's no /usr/share/doc/qtcore-5.11.1 on my system.

> I have created a patch that will allow ebuilds to whitelist directories
> by setting a QA_INSTALL_PATHS variable [3], however @mgorny said:

@mgorny was seriously concerned about trigger-happy patch authors who
find it urgent to silence QA warnings without even bothering to properly
discuss the problem.  And is continuously concerned about people who
want something but don't bother starting the discussion, and instead
rely on somebody else to start the discussion, even without having
the appropriate knowledge on what the problem is in the first place.

In other words, this is something that should be discussed on a case-by-
case basis.  Not the usual Gentoo thing of 'I don't like this QA
warning, let's silence it quickly and go on ignoring everyone, whether
my package is broken or not'.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1304 bytes --]

On 10/01/2018 08:48 AM, Michał Górny wrote:
> On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
>> I have created a patch that will allow ebuilds to whitelist directories
>> by setting a QA_INSTALL_PATHS variable [3], however @mgorny said:
> 
> @mgorny was seriously concerned about trigger-happy patch authors who
> find it urgent to silence QA warnings without even bothering to properly
> discuss the problem.

Discussion takes time, and I'm a very busy person, sorry about that.
Anyway, eventually we are forced to discuss those issues that really matter.

> And is continuously concerned about people who
> want something but don't bother starting the discussion, and instead
> rely on somebody else to start the discussion, even without having
> the appropriate knowledge on what the problem is in the first place.

> In other words, this is something that should be discussed on a case-by-
> case basis.  Not the usual Gentoo thing of 'I don't like this QA
> warning, let's silence it quickly and go on ignoring everyone, whether
> my package is broken or not'.

Maybe we should implement a policy requiring gentoo-dev mailing list
proposals for QA_INSTALL_PATHS usage in gentoo ebuilds (similar to the
policy for news items and eclass changes)?
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Andreas Sturmlechner]

On Montag, 1. Oktober 2018 17:48:16 CEST Michał Górny wrote:
> On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
> > /usr/share/doc level directories
> > ================================
> > /usr/share/doc/${PF}
> > 
> > The first bug report [2] is for qt-core, which installs documentation
> > into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
> > includes ebuild revision such as -r1, -r2, and so on).
> 
> No, it doesn't.  There's no /usr/share/doc/qtcore-5.11.1 on my system.

This is coming from dev-qt/qt-docs. It is a problem because any other package 
building QCH API docs with cross-references to Qt API needs to install below 
this path, and will generate the same QA warning (currently kde-frameworks/* 
does this).

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1120 bytes --]

On Mon, 2018-10-01 at 19:23 +0200, Andreas Sturmlechner wrote:
> On Montag, 1. Oktober 2018 17:48:16 CEST Michał Górny wrote:
> > On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
> > > /usr/share/doc level directories
> > > ================================
> > > /usr/share/doc/${PF}
> > > 
> > > The first bug report [2] is for qt-core, which installs documentation
> > > into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
> > > includes ebuild revision such as -r1, -r2, and so on).
> > 
> > No, it doesn't.  There's no /usr/share/doc/qtcore-5.11.1 on my system.
> 
> This is coming from dev-qt/qt-docs.

Nope, still not /usr/share/doc/qt*core*-...

>  It is a problem because any other package 
> building QCH API docs with cross-references to Qt API needs to install below 
> this path, and will generate the same QA warning (currently kde-frameworks/* 
> does this).

Yes.  That is why I believe that hardcoding the exception in every
package is simply wrong.  Wouldn't it be cleaner to account for the path
in the QA check?

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1261 bytes --]

On 10/01/2018 11:16 AM, Michał Górny wrote:
> On Mon, 2018-10-01 at 19:23 +0200, Andreas Sturmlechner wrote:
>> On Montag, 1. Oktober 2018 17:48:16 CEST Michał Górny wrote:
>>> On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
>>>> /usr/share/doc level directories
>>>> ================================
>>>> /usr/share/doc/${PF}
>>>>
>>>> The first bug report [2] is for qt-core, which installs documentation
>>>> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
>>>> includes ebuild revision such as -r1, -r2, and so on).
>>>
>>> No, it doesn't.  There's no /usr/share/doc/qtcore-5.11.1 on my system.
>>
>> This is coming from dev-qt/qt-docs.
> 
> Nope, still not /usr/share/doc/qt*core*-...
> 
>>  It is a problem because any other package 
>> building QCH API docs with cross-references to Qt API needs to install below 
>> this path, and will generate the same QA warning (currently kde-frameworks/* 
>> does this).
> 
> Yes.  That is why I believe that hardcoding the exception in every
> package is simply wrong.  Wouldn't it be cleaner to account for the path
> in the QA check?

There may be cases where we want to fix the ebuild to use
/usr/share/doc/${PF} though, right?
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Sergei Trofimovich]

On Mon, 1 Oct 2018 08:19:29 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> Hi all,
> 
> The ~arch version of portage hs a new QA check that reports installation
> of files outside of directories that have been whitelisted [1]. The
> current whitelist includes:
> 
> directories common to / and /usr
> ================================
> bin lib lib32 lib64 libx32 sbin
> 
> top level directories
> ================================
> boot dev etc opt srv usr var
> 
> /usr level directories
> ================================
> include libexec share src
> 
> /usr/share/doc level directories
> ================================
> /usr/share/doc/${PF}

As this will break existing ebuilds I'd suggest guarding it against
next EAPI. Out of top of by head the change will break at least
crossdev outright. At least for two reasons:
1. building a gcc cross-compiler usually refers to $SYSROOT/sys-include.
   'git grep sys-include' will show a bunch of ebuilds that create that
   symlinks outside the space.
2. cross-building target libc is currently done on the host's emerge
  and installs into /usr/$CTARGET/.

I think whitelist would be able to cover these use cases as they are
a function of ${CTARGET} (or $CATEGORY) value.

-- 

  Sergei

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1790 bytes --]

On 10/01/2018 01:04 PM, Sergei Trofimovich wrote:
> On Mon, 1 Oct 2018 08:19:29 -0700
> Zac Medico <zmedico@gentoo.org> wrote:
> 
>> Hi all,
>>
>> The ~arch version of portage hs a new QA check that reports installation
>> of files outside of directories that have been whitelisted [1]. The
>> current whitelist includes:
>>
>> directories common to / and /usr
>> ================================
>> bin lib lib32 lib64 libx32 sbin
>>
>> top level directories
>> ================================
>> boot dev etc opt srv usr var
>>
>> /usr level directories
>> ================================
>> include libexec share src
>>
>> /usr/share/doc level directories
>> ================================
>> /usr/share/doc/${PF}
> 
> As this will break existing ebuilds I'd suggest guarding it against
> next EAPI. Out of top of by head the change will break at least
> crossdev outright. At least for two reasons:
> 1. building a gcc cross-compiler usually refers to $SYSROOT/sys-include.
>    'git grep sys-include' will show a bunch of ebuilds that create that
>    symlinks outside the space.
> 2. cross-building target libc is currently done on the host's emerge
>   and installs into /usr/$CTARGET/.
> 
> I think whitelist would be able to cover these use cases as they are
> a function of ${CTARGET} (or $CATEGORY) value.

The QA check currently does not die, it only calls eqawarn. So the worst
case is a message like this:

> QA: other
> The ebuild is installing to one or more unexpected paths:
>
>   /usr/share/doc/qt-5.11.1
>
> Please fix the ebuild to use correct FHS/Gentoo policy paths.

It will be annoying if this message triggers for every package installed
by crossdev, so we should do something about that.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 2285 bytes --]

On Mon, 1 Oct 2018 13:48:46 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> On 10/01/2018 01:04 PM, Sergei Trofimovich wrote:
> > On Mon, 1 Oct 2018 08:19:29 -0700
> > Zac Medico <zmedico@gentoo.org> wrote:
> >   
> >> Hi all,
> >>
> >> The ~arch version of portage hs a new QA check that reports installation
> >> of files outside of directories that have been whitelisted [1]. The
> >> current whitelist includes:
> >>
> >> directories common to / and /usr
> >> ================================
> >> bin lib lib32 lib64 libx32 sbin
> >>
> >> top level directories
> >> ================================
> >> boot dev etc opt srv usr var
> >>
> >> /usr level directories
> >> ================================
> >> include libexec share src
> >>
> >> /usr/share/doc level directories
> >> ================================
> >> /usr/share/doc/${PF}  
> > 
> > As this will break existing ebuilds I'd suggest guarding it against
> > next EAPI. Out of top of by head the change will break at least
> > crossdev outright. At least for two reasons:
> > 1. building a gcc cross-compiler usually refers to $SYSROOT/sys-include.
> >    'git grep sys-include' will show a bunch of ebuilds that create that
> >    symlinks outside the space.
> > 2. cross-building target libc is currently done on the host's emerge
> >   and installs into /usr/$CTARGET/.
> > 
> > I think whitelist would be able to cover these use cases as they are
> > a function of ${CTARGET} (or $CATEGORY) value.  
> 
> The QA check currently does not die, it only calls eqawarn. So the worst
> case is a message like this:
> 
> > QA: other
> > The ebuild is installing to one or more unexpected paths:
> >
> >   /usr/share/doc/qt-5.11.1
> >
> > Please fix the ebuild to use correct FHS/Gentoo policy paths.  
> 
> It will be annoying if this message triggers for every package installed
> by crossdev, so we should do something about that.

Ah, QA warning is fine. We'll have some time to address it.

For crossdev it should be only libc packages that do unusual paths
(their initial version is installed into /. We'll fix it like
https://bugs.gentoo.org/642604#c6 but it takes time).

Normal packages should be installed into $SYSROOT proper.

-- 

  Sergei

[-- Attachment #2: Цифровая подпись OpenPGP --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

On Mon, 1 Oct 2018 08:19:29 -0700 Zac Medico wrote:
> Hi all,
> 
> The ~arch version of portage hs a new QA check that reports installation
> of files outside of directories that have been whitelisted [1]. The
> current whitelist includes:
> 
> directories common to / and /usr
> ================================
> bin lib lib32 lib64 libx32 sbin
> 
> top level directories
> ================================
> boot dev etc opt srv usr var
> 
> /usr level directories
> ================================
> include libexec share src
> 
> /usr/share/doc level directories
> ================================
> /usr/share/doc/${PF}
> 
> The first bug report [2] is for qt-core, which installs documentation
> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
> includes ebuild revision such as -r1, -r2, and so on).

Sometimes documentation contains files used at run-time by the
application. Since application is usually not aware of Gentoo
specific revision numbers it is reasonable to install docs in
${PN}-${PV}. Such checks *must* be overrideable.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1764 bytes --]

On 10/01/2018 11:23 AM, Zac Medico wrote:
> On 10/01/2018 11:16 AM, Michał Górny wrote:
>> On Mon, 2018-10-01 at 19:23 +0200, Andreas Sturmlechner wrote:
>>> On Montag, 1. Oktober 2018 17:48:16 CEST Michał Górny wrote:
>>>> On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
>>>>> /usr/share/doc level directories
>>>>> ================================
>>>>> /usr/share/doc/${PF}
>>>>>
>>>>> The first bug report [2] is for qt-core, which installs documentation
>>>>> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
>>>>> includes ebuild revision such as -r1, -r2, and so on).
>>>>
>>>> No, it doesn't.  There's no /usr/share/doc/qtcore-5.11.1 on my system.
>>>
>>> This is coming from dev-qt/qt-docs.
>>
>> Nope, still not /usr/share/doc/qt*core*-...
>>
>>>  It is a problem because any other package 
>>> building QCH API docs with cross-references to Qt API needs to install below 
>>> this path, and will generate the same QA warning (currently kde-frameworks/* 
>>> does this).
>>
>> Yes.  That is why I believe that hardcoding the exception in every
>> package is simply wrong.  Wouldn't it be cleaner to account for the path
>> in the QA check?
> 
> There may be cases where we want to fix the ebuild to use
> /usr/share/doc/${PF} though, right?

Until this QA check has adjustable whitelist support, we can consider it
an unstable work in progress. Therefore, I'd like for the QA team to
move it gentoo/metadata/install-qa-check.d/08gentoo-paths until it has
matured. It's safe to commit it to the gentoo repository now, and it
will become active when portage's internal copy is removed. I've filed
this bug report to track progress:

https://bugs.gentoo.org/667604
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Michael Orlitzky]

On 10/03/2018 12:38 PM, Zac Medico wrote:
> 
> Until this QA check has adjustable whitelist support, we can consider it
> an unstable work in progress.

Has anyone said why these things need to be in ${PN}-${PV} rather than
${PF}?

If they don't need to be in ${PN}-${PV} in the first place, then the QA
check is just doing exactly what it's supposed to. If, on the other
hand, there's a good reason for something to install docs outside of
${PF}, then we should just fix the check (and our policy) to allow it.

(A whitelist is IMO the wrong approach, the QA check should never report
"correct" things in the first place.)

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Andreas Sturmlechner]

On Mittwoch, 3. Oktober 2018 20:26:23 CEST Michael Orlitzky wrote:
> Has anyone said why these things need to be in ${PN}-${PV} rather than
> ${PF}?

Because API does not change due to Gentoo revisions, and breaking/having to 
rebuild docs for that is unacceptable.

It will lead to more docs being installed outside of /usr/share/docs.

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Wed, 2018-10-03 at 20:29 +0200, Andreas Sturmlechner wrote:
> On Mittwoch, 3. Oktober 2018 20:26:23 CEST Michael Orlitzky wrote:
> > Has anyone said why these things need to be in ${PN}-${PV} rather than
> > ${PF}?
> 
> Because API does not change due to Gentoo revisions, and breaking/having to 
> rebuild docs for that is unacceptable.
> 

...except when it does.  We have valid cases for packages simultaneously
installing different documentation for different PF but matching PV
(e.g. slotted GTK+2 + GTK+3 packages).

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Andreas Sturmlechner]

On Montag, 1. Oktober 2018 20:16:28 CET Michał Górny wrote:
> >  It is a problem because any other package
> > building QCH API docs with cross-references to Qt API needs to install
> > below this path, and will generate the same QA warning (currently
> > kde-frameworks/* does this).
> 
> Yes.  That is why I believe that hardcoding the exception in every
> package is simply wrong.  Wouldn't it be cleaner to account for the path
> in the QA check?

What's the current status of this? I'd like to know if there's going to be a 
whitelist or QA_ var to put into kde5.eclass, if not, I can also simply 
(EAPI-7-)move kde handbook directory out of /usr/share/doc into /usr/share/
kde-doc or similar and be done with it.

Regards,
Andreas