[gentoo-dev] rfc: empty directories in ${D}

[gentoo-dev] rfc: empty directories in ${D}

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

All,

I just happened to notice the following warning from portage when
bumping dhcpcd.

> One or more empty directories installed to /var:

>  /var/lib/dhcpcd

> If those directories need to be preserved, please make sure to create
> or mark them for keeping using 'keepdir'. Future versions of Portage
> will strip empty directories from installation image.

If we are going to require emptty  directories to be marked with
keepdir, I think we should hard fail the emerge rather than quietly
strip the empty directories. If we just strip the directories, this
will, more than likely, lead to broken packages. In the case of dhcpcd,
the upstream build system installs the /var/lib/dhcpcd directory, then
dhcpcd writes to the directory.

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[Michał Górny]

W dniu czw, 29.03.2018 o godzinie 09∶39 -0500, użytkownik William Hubbs
napisał:
> All,
> 
> I just happened to notice the following warning from portage when
> bumping dhcpcd.
> 
> > One or more empty directories installed to /var:
> >  /var/lib/dhcpcd
> > If those directories need to be preserved, please make sure to create
> > or mark them for keeping using 'keepdir'. Future versions of Portage
> > will strip empty directories from installation image.
> 
> If we are going to require emptty  directories to be marked with
> keepdir, I think we should hard fail the emerge rather than quietly
> strip the empty directories. If we just strip the directories, this
> will, more than likely, lead to broken packages. In the case of dhcpcd,
> the upstream build system installs the /var/lib/dhcpcd directory, then
> dhcpcd writes to the directory.
> 

Are you saying that dozens of packages should suddenly start failing
for users so that developers would feel more obliged to fix them?
Provided that the packages are still maintained, and it won't be
'hey, we just made it impossible to install this package, maybe someone
will fix it one day'.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] rfc: empty directories in ${D}

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

On 29-03-2018 16:47:51 +0200, Michał Górny wrote:
> W dniu czw, 29.03.2018 o godzinie 09∶39 -0500, użytkownik William Hubbs
> napisał:
> > All,
> > 
> > I just happened to notice the following warning from portage when
> > bumping dhcpcd.
> > 
> > > One or more empty directories installed to /var:
> > >  /var/lib/dhcpcd
> > > If those directories need to be preserved, please make sure to create
> > > or mark them for keeping using 'keepdir'. Future versions of Portage
> > > will strip empty directories from installation image.
> > 
> > If we are going to require emptty  directories to be marked with
> > keepdir, I think we should hard fail the emerge rather than quietly
> > strip the empty directories. If we just strip the directories, this
> > will, more than likely, lead to broken packages. In the case of dhcpcd,
> > the upstream build system installs the /var/lib/dhcpcd directory, then
> > dhcpcd writes to the directory.
> > 
> 
> Are you saying that dozens of packages should suddenly start failing
> for users so that developers would feel more obliged to fix them?
> Provided that the packages are still maintained, and it won't be
> 'hey, we just made it impossible to install this package, maybe someone
> will fix it one day'.

I agree, packages shouldn't suddenly start failing.  Not during install,
not during runtime either.  For changes like this EAPIs were invented.

Fabian

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1897 bytes --]

On Thu, Mar 29, 2018 at 05:15:28PM +0200, Fabian Groffen wrote:
> On 29-03-2018 16:47:51 +0200, Michał Górny wrote:
> > W dniu czw, 29.03.2018 o godzinie 09∶39 -0500, użytkownik William Hubbs
> > napisał:
> > > All,
> > > 
> > > I just happened to notice the following warning from portage when
> > > bumping dhcpcd.
> > > 
> > > > One or more empty directories installed to /var:
> > > >  /var/lib/dhcpcd
> > > > If those directories need to be preserved, please make sure to create
> > > > or mark them for keeping using 'keepdir'. Future versions of Portage
> > > > will strip empty directories from installation image.
> > > 
> > > If we are going to require emptty  directories to be marked with
> > > keepdir, I think we should hard fail the emerge rather than quietly
> > > strip the empty directories. If we just strip the directories, this
> > > will, more than likely, lead to broken packages. In the case of dhcpcd,
> > > the upstream build system installs the /var/lib/dhcpcd directory, then
> > > dhcpcd writes to the directory.
> > > 
> > 
> > Are you saying that dozens of packages should suddenly start failing
> > for users so that developers would feel more obliged to fix them?
> > Provided that the packages are still maintained, and it won't be
> > 'hey, we just made it impossible to install this package, maybe someone
> > will fix it one day'.
> 
> I agree, packages shouldn't suddenly start failing.  Not during install,
> not during runtime either.  For changes like this EAPIs were invented.

Fabian has the right idea I think.

If we quietly strip away empty directories installed by upstream build
systems, there is a possibility we will cause runtime breakage. If we are
going to have breakage, then yes, I would rather it be at emerge time to
force us to fix it before runtime breakage hits users.

Thanks,

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2822 bytes --]

On Thu, Mar 29, 2018 at 10:47 AM, Michał Górny <mgorny@gentoo.org> wrote:

> W dniu czw, 29.03.2018 o godzinie 09∶39 -0500, użytkownik William Hubbs
> napisał:
> > All,
> >
> > I just happened to notice the following warning from portage when
> > bumping dhcpcd.
> >
> > > One or more empty directories installed to /var:
> > >  /var/lib/dhcpcd
> > > If those directories need to be preserved, please make sure to create
> > > or mark them for keeping using 'keepdir'. Future versions of Portage
> > > will strip empty directories from installation image.
> >
> > If we are going to require emptty  directories to be marked with
> > keepdir, I think we should hard fail the emerge rather than quietly
> > strip the empty directories. If we just strip the directories, this
> > will, more than likely, lead to broken packages. In the case of dhcpcd,
> > the upstream build system installs the /var/lib/dhcpcd directory, then
> > dhcpcd writes to the directory.
> >
>
> Are you saying that dozens of packages should suddenly start failing
> for users so that developers would feel more obliged to fix them?
> Provided that the packages are still maintained, and it won't be
> 'hey, we just made it impossible to install this package, maybe someone
> will fix it one day'.
>
>
I have a more concrete question regarding what the feature is trying to do,
and how it accomplishes it.

The problem seems to be that there are empty directories in ${D} that get
merged to the livefs but are not used.
These directories are either made by the ebuild directly (e.g. a bug in
Gentoo) or by upstream's build system (a bug upstream.)

Is there any particular reason we need to remove them?

We have elected (based on current portage) to warn people about empty
directories. If the goal is to remove the empty directories; this seems
reasonable; folks can then
figure out if the bug is in Gentoo, or Upstream[0], and submit a fix. The
struggle here comes where we want to do two sorts of activities.

1) Stop new bugs (e.g. by stripping empty directories and causing packages
to fail to work properly because the livefs is missing directories.)
2) Fix existing bugs (e.g. by warning folks.)

Ultimately this is what I feel like EAPI was made for; we could propose
stripping in EAPI7, and warning in all other EAPIs.

-A

[0] There is another issue whereby upstreams might not care about this bug,
or might introduce new bugs because they lack this QA mechanism. I tend to
be pessemistic on these efforts for this reason; now Gentoo has to carry
some Gentoo-only patches to fix this random "QA" problem, for what to me is
a minor benefit. Some upstreams will never merge these patches. So I'm
curious what the gain is?


--
> Best regards,
> Michał Górny
>
>
>

[-- Attachment #2: Type: text/html, Size: 3762 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[Michael Orlitzky]

On 03/29/2018 11:28 AM, Alec Warner wrote:
> 
> Is there any particular reason we need to remove them?
> 

The PMS says that empty directories are undefined, so the portage
behavior of installing them and leaving them alone leads to
incompatibilities. Ebuilds rely on the portage behavior, and if another
PM (within its rights) deletes them, then the package breaks with the
non-portage PM.

Re: [gentoo-dev] rfc: empty directories in ${D}

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

On Thu, Mar 29, 2018 at 11:47 AM, Michael Orlitzky <mjo@gentoo.org> wrote:

> On 03/29/2018 11:28 AM, Alec Warner wrote:
> >
> > Is there any particular reason we need to remove them?
> >
>
> The PMS says that empty directories are undefined, so the portage
> behavior of installing them and leaving them alone leads to
> incompatibilities. Ebuilds rely on the portage behavior, and if another
> PM (within its rights) deletes them, then the package breaks with the
> non-portage PM.
>
>
So we could simply change the PMS to keep the empty directories?

Why is removing them *better* is my question.

[-- Attachment #2: Type: text/html, Size: 1045 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

On Thu, Mar 29, 2018 at 11:47:21AM -0400, Michael Orlitzky wrote:
> On 03/29/2018 11:28 AM, Alec Warner wrote:
> > 
> > Is there any particular reason we need to remove them?
> > 
> 
> The PMS says that empty directories are undefined, so the portage
> behavior of installing them and leaving them alone leads to
> incompatibilities. Ebuilds rely on the portage behavior, and if another
> PM (within its rights) deletes them, then the package breaks with the
> non-portage PM.

Maybe so, but you just made the argument for doing nothing different in
current eapis and proposing stripping empty directories in eapi 7.
However, this should be stripping empty directories combined with
failing the emerge.

The issue is that if an upstream build system creates a directory, then
the upstream software tries to access the directory without creating
it, there will be runtime breakage.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[Michael Orlitzky]

On 03/29/2018 11:57 AM, Alec Warner wrote:
> 
> So we could simply change the PMS to keep the empty directories?
> 
> Why is removing them *better* is my question.

I proposed this idea but basically everyone with a say in the process
was against it: https://bugs.gentoo.org/644366

My last comment: "If no one complains, then you were right."

Re: [gentoo-dev] rfc: empty directories in ${D}

[Michael Orlitzky]

On 03/29/2018 11:57 AM, William Hubbs wrote:
>>
>> The PMS says that empty directories are undefined, so the portage
>> behavior of installing them and leaving them alone leads to
>> incompatibilities. Ebuilds rely on the portage behavior, and if another
>> PM (within its rights) deletes them, then the package breaks with the
>> non-portage PM.
> 
> Maybe so, but you just made the argument for doing nothing different in
> current eapis and proposing stripping empty directories in eapi 7.
> However, this should be stripping empty directories combined with
> failing the emerge.

If we strip them only in EAPI=7, then that still leaves all of these
packages broken with respect to the PMS in the other EAPIs.

Stripping the empty directories isn't my favorite approach, but leaving
things broken looks bad on paper too.

Re: [gentoo-dev] rfc: empty directories in ${D}

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

On Thu, Mar 29, 2018 at 12:24:38PM -0400, Michael Orlitzky wrote:
> On 03/29/2018 11:57 AM, William Hubbs wrote:
> >>
> >> The PMS says that empty directories are undefined, so the portage
> >> behavior of installing them and leaving them alone leads to
> >> incompatibilities. Ebuilds rely on the portage behavior, and if another
> >> PM (within its rights) deletes them, then the package breaks with the
> >> non-portage PM.
> > 
> > Maybe so, but you just made the argument for doing nothing different in
> > current eapis and proposing stripping empty directories in eapi 7.
> > However, this should be stripping empty directories combined with
> > failing the emerge.
> 
> If we strip them only in EAPI=7, then that still leaves all of these
> packages broken with respect to the PMS in the other EAPIs.
> 
> Stripping the empty directories isn't my favorite approach, but leaving
> things broken looks bad on paper too.

If we are going to strip the empty directories, we should hard fail the
emerge at the same time. Otherwise there is no way to know whether the
package we successfully install will now run.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[Michał Górny]

W dniu czw, 29.03.2018 o godzinie 11∶34 -0500, użytkownik William Hubbs
napisał:
> On Thu, Mar 29, 2018 at 12:24:38PM -0400, Michael Orlitzky wrote:
> > On 03/29/2018 11:57 AM, William Hubbs wrote:
> > > > 
> > > > The PMS says that empty directories are undefined, so the portage
> > > > behavior of installing them and leaving them alone leads to
> > > > incompatibilities. Ebuilds rely on the portage behavior, and if another
> > > > PM (within its rights) deletes them, then the package breaks with the
> > > > non-portage PM.
> > > 
> > > Maybe so, but you just made the argument for doing nothing different in
> > > current eapis and proposing stripping empty directories in eapi 7.
> > > However, this should be stripping empty directories combined with
> > > failing the emerge.
> > 
> > If we strip them only in EAPI=7, then that still leaves all of these
> > packages broken with respect to the PMS in the other EAPIs.
> > 
> > Stripping the empty directories isn't my favorite approach, but leaving
> > things broken looks bad on paper too.
> 
> If we are going to strip the empty directories, we should hard fail the
> emerge at the same time. Otherwise there is no way to know whether the
> package we successfully install will now run.
> 

The developer is supposed to *look* at what the package installs.
If people just commit ebuilds based on 'Portage did not explode',
then maybe they shouldn't have commit access in the first place.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] rfc: empty directories in ${D}

[Paweł Hajdan, Jr.]

[-- Attachment #1.1: Type: text/plain, Size: 992 bytes --]

On 29/03/2018 19:00, Michał Górny wrote:
> W dniu czw, 29.03.2018 o godzinie 11∶34 -0500, użytkownik William Hubbs
> napisał:
>> If we are going to strip the empty directories, we should hard fail the
>> emerge at the same time. Otherwise there is no way to know whether the
>> package we successfully install will now run.
> 
> The developer is supposed to *look* at what the package installs.
> If people just commit ebuilds based on 'Portage did not explode',
> then maybe they shouldn't have commit access in the first place.

You're IMO both making good points.

I have an impression proponents of each solution comment on parts of the
opposing argument, but I'm yet to see a more organized pros and cons list.

There's an element of design taste, and it may help to have some
reasonable guidelines, like "avoid breaking users' systems" - as opposed
to expecting volunteer developers never to make mistakes, or optimizing
for anyone but the user.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[Andrey Utkin]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On Thu, Mar 29, 2018 at 11:57:06AM -0400, Alec Warner wrote:
> On Thu, Mar 29, 2018 at 11:47 AM, Michael Orlitzky <mjo@gentoo.org> wrote:
> 
> > On 03/29/2018 11:28 AM, Alec Warner wrote:
> > >
> > > Is there any particular reason we need to remove them?
> > >
> >
> > The PMS says that empty directories are undefined, so the portage
> > behavior of installing them and leaving them alone leads to
> > incompatibilities. Ebuilds rely on the portage behavior, and if another
> > PM (within its rights) deletes them, then the package breaks with the
> > non-portage PM.
> >
> >
> So we could simply change the PMS to keep the empty directories?
> 
> Why is removing them *better* is my question.

Right, I am not aware why PMS has left this explicitly undefined. Have
read through https://bugs.gentoo.org/644366 but there's no hint on why,
too. I appreciate mjo's proposal. I think it would be good for ebuild
maintainer to have a switch "empty dirs are ok by default". The
disagreement seems to be based on a prejudice and distrust towards
upstreams' build systems.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1014 bytes --]

Re: [gentoo-dev] rfc: empty directories in ${D}

[Michał Górny]

W dniu sob, 31.03.2018 o godzinie 20∶46 +0100, użytkownik Andrey Utkin
napisał:
> On Thu, Mar 29, 2018 at 11:57:06AM -0400, Alec Warner wrote:
> > On Thu, Mar 29, 2018 at 11:47 AM, Michael Orlitzky <mjo@gentoo.org> wrote:
> > 
> > > On 03/29/2018 11:28 AM, Alec Warner wrote:
> > > > 
> > > > Is there any particular reason we need to remove them?
> > > > 
> > > 
> > > The PMS says that empty directories are undefined, so the portage
> > > behavior of installing them and leaving them alone leads to
> > > incompatibilities. Ebuilds rely on the portage behavior, and if another
> > > PM (within its rights) deletes them, then the package breaks with the
> > > non-portage PM.
> > > 
> > > 
> > 
> > So we could simply change the PMS to keep the empty directories?
> > 
> > Why is removing them *better* is my question.
> 
> Right, I am not aware why PMS has left this explicitly undefined. Have
> read through https://bugs.gentoo.org/644366 but there's no hint on why,
> too. I appreciate mjo's proposal. I think it would be good for ebuild
> maintainer to have a switch "empty dirs are ok by default". The
> disagreement seems to be based on a prejudice and distrust towards
> upstreams' build systems.

You can accuse developers of 'prejudice and distrust', or you can
believe that they actually have some experience and knowledge to prove
the point. But if you start with prejudice towards Gentoo developers,
then I suppose there's no point in arguing further.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] rfc: empty directories in ${D}

[Ciaran McCreesh]

On Sat, 31 Mar 2018 20:46:56 +0100
Andrey Utkin <andrey_utkin@gentoo.org> wrote:
> Right, I am not aware why PMS has left this explicitly undefined.

Because when we wrote the spec, Portage's actual behaviour on empty
directories was extremely tricky to define and depended upon things
like whether you were upgrading or reinstalling a package.

-- 
Ciaran McCreesh