[gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1774 bytes --]

Hello,

As most of you probably doesn't know, PMS guarantees that ${D} always
ends with a slash. It seems that this particular wording was enforced
by historical portage behavior (instead of fixing the ebuilds...) yet
it didn't ever get really widespread.

Specifically, it is awfully counter-intuitive and even PMS gets it
wrong in most of the time (see: einstall). I could sum up the issues
with it in three snippets:

1) "${D}"usr/bin/foo

2) emake install DESTDIR="${D%/}"

3) local foodir=/usr/share/foo
   insinto ${foodir}
   ...
   bar "${D}${foodir#/}"

Ugly, isn't it? And since it's completely counter-intuitive, most
ebuilds and eclasses get it wrong. Either by appending a slash just to
be sure (and having double slashes) or even conditionally stripping one.

While we could keep 'fixing' the ebuilds to follow the weird PMS
requirement, I'd rather go the other way -- fixing PMS to reflect
common sense.

The plan would be to:

1) lift the requirement that ${D} needs to end with slash,

2) fix all the ebuilds and eclasses which rely on that to use explicit
slash -- resulting in double slashes in the transition period,

3) make portage not append the slash anymore,

4) update PMS to specify that all directory variables do not end with
a slash,

5) slowly update the remaining ebuilds and eclasses to remove
the conditional slash removal/appending code.

This is mostly possible since majority of systems supported by Gentoo
does work fine with double slashes in paths, and the custom Cygwin port
fails randomly anyway due to this specific issue. We'd break it
completely for the transition period but afterwards, it will all be
clean and sane again.

Your thoughts?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Gregory M. Turner]

On 4/13/2013 2:23 PM, Michał Górny wrote:
> Your thoughts?
+1, this causes untold agonies for my pet platform.  See #465772 for 
more thorough griping :)

-gmt

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 47 bytes --]

Michał Górny wrote:
> Your thoughts?

+1

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

On Sat, 13 Apr 2013 23:23:57 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> As most of you probably doesn't know, PMS guarantees that ${D} always
> ends with a slash. It seems that this particular wording was enforced
> by historical portage behavior (instead of fixing the ebuilds...) yet
> it didn't ever get really widespread.

I forgot to add the bug-link:

https://bugs.gentoo.org/show_bug.cgi?id=465772

There you can find at least my rough estimates of how many ebuilds
are directly affected. Shortly saying, if we fix PMS, we need to fix,
say, 200 ebuilds. If we leave it as is, we have around 1750 obviously
broken ebuilds + over 3000 if we count DESTDIR=${D}. Not to mention all
the hidden issues.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Sun, 14 Apr 2013 13:32:29 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> There you can find at least my rough estimates of how many ebuilds
> are directly affected. Shortly saying, if we fix PMS, we need to fix,
> say, 200 ebuilds. If we leave it as is, we have around 1750 obviously
> broken ebuilds + over 3000 if we count DESTDIR=${D}. Not to mention all
> the hidden issues.

Sorry, packages not ebuilds. Ebuild counts are a bit larger
for the first one, and guess for the second one.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/13/2013 11:23 PM, Michał Górny wrote:
> Your thoughts?
> 

Sounds ok, but how will the transition happen exactly? Will the
behavior just change for new EAPIs or for all? If for all, who will do
the tinderbox run?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRarnhAAoJEFpvPKfnPDWzyHgIAI5O3zh1a83A2SkUwrKj1imU
kpy5Bvsb0NCXfwNHnC95A8201Z8F9KoSn0LB+9BKk8Jwaj0XsCNfljatKcbvNB3j
WmhxRk4GTtdkANgvCDDYrNhc4N/1e91bJwEdMsxr91MkKGM3JHfGZEYxWVooOZ2J
XVUgV5OSpBu//mAtipV/iyA7ALZXrFGvKUqevN1BRj4FN0TfE15WLiYYHwL1tLyR
EPDgxqG8a3RPUyzOrNikkP4R15bpZ/4sGVl3yqlQgM06rmiR0PPXA9V8R6963dvX
dOrm0/oZ7DwGGx+zr7NOwErZ6+UEGkF22cMJPCGpIPrEHBJZsFevUrl6EweGfDc=
=fkG2
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

On Sat, 13 Apr 2013 23:23:57 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> As most of you probably doesn't know, PMS guarantees that ${D} always
> ends with a slash. It seems that this particular wording was enforced
> by historical portage behavior (instead of fixing the ebuilds...) yet
> it didn't ever get really widespread.

Should ROOT usually be empty then?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On Sun, 14 Apr 2013 15:22:42 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Sat, 13 Apr 2013 23:23:57 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > As most of you probably doesn't know, PMS guarantees that ${D} always
> > ends with a slash. It seems that this particular wording was enforced
> > by historical portage behavior (instead of fixing the ebuilds...) yet
> > it didn't ever get really widespread.
> 
> Should ROOT usually be empty then?

Considering its use, probably yes.

  ${ROOT}/usr/bin/foo
  ${EPREFIX}/usr/bin/foo
  ${D}${EPREFIX}/usr/bin/foo

All seem clean and consistent to me.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Michał Górny]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Sun, 14 Apr 2013 16:14:57 +0200
hasufell <hasufell@gentoo.org> wrote:

> On 04/13/2013 11:23 PM, Michał Górny wrote:
> > Your thoughts?
> > 
> 
> Sounds ok, but how will the transition happen exactly? Will the
> behavior just change for new EAPIs or for all? If for all, who will do
> the tinderbox run?

I am fine with whatever suits people.

New EAPI would be simpler for ebuilds but harder for eclasses (which
will have to support both cases). Additionally, there will be
the increased risk that thoughtless EAPI switch will silently trigger
failure.

Retroactive change has the benefit that most of the current uses become
valid and all ebuilds start to work correctly the moment we finish it.
Noticeably, the transition will occur with a portage upgrade so any
testing can be done through p.masking a single package.

That said, the tinderbox run would probably end up being community
effort in an --emptytree @world update.

- -- 
Best regards,
Michał Górny
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iQJ8BAEBCgBmBQJRasTgXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ1RUJGMjBGOTk2RkIzQzIyQ0M2RkNBNDBC
QUJGMUQ1RkY4QzgxMTBBAAoJELq/HV/4yBEK0XkQAJkvkv86axzdMJyr7YtUm7w/
ESqLlS5mhqw66Q8Su6n1Rj8d0cfSd8KD9gbu0GdXOyrJ54HpxD5GrAByEjoE+guc
3/cBb1WIYCq30XZbGkBv2uJL92ysKukSaqbfTd+KimFvzc3okd7W0R+5Matiq6BZ
KktZZ+vMPXmVmbdnm7zoHqvvEB2rsQy/xwpNj3hXNheLdtvZ1oahAQoiCRi743tD
wViY+sCsnn4plsWOLpTibWpOwQD2Qn8sLoaC/exvVNsT7EkEX6REpkw0/V3uz566
jIXv92lcKpTGJdROBAMfhJ5dp8fsQlhJExzlIXPIuxkn4fs4dgKzlHNJ3M78xxGj
3Tt/QeAptEiCWWPMbxZW3yDGiMcGIo2faR2gPOotplBuq7F9KdSedYCw//WXfD9V
c5MjslDZCezFLYBhCK7JBILXsavSoAhkRXgjVREcquJeGUaZpkguSGDripZgFwWy
9N4buJ5WOscHT8foVgMNm9SxC565SgmbU6BKboSK1nJMNTwvaaLaJTIfuQvldbVF
HLMDYh13N66XJLZfLyy8jnR6HaYfSNTTUndu+n0a+nFkFA4vGYh9B135ikiR4d+N
DxO9KUnjuqTYiup8PAUsdfek62OJuFBMkN9biZlIy5ZahEixOPRQNquX36xze68b
z1EmfdaF7bjB7H7tHy1p
=CftZ
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On Sun, 14 Apr 2013 16:58:07 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> On Sun, 14 Apr 2013 15:22:42 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > On Sat, 13 Apr 2013 23:23:57 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> > > As most of you probably doesn't know, PMS guarantees that ${D}
> > > always ends with a slash. It seems that this particular wording
> > > was enforced by historical portage behavior (instead of fixing
> > > the ebuilds...) yet it didn't ever get really widespread.
> > 
> > Should ROOT usually be empty then?
> 
> Considering its use, probably yes.
> 
>   ${ROOT}/usr/bin/foo
>   ${EPREFIX}/usr/bin/foo
>   ${D}${EPREFIX}/usr/bin/foo
> 
> All seem clean and consistent to me.

How many things test whether ROOT=/ ? I seem to recall that being
fairly common, back when something accidentally set it to //.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]

On Sun, 14 Apr 2013 17:39:44 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Sun, 14 Apr 2013 16:58:07 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > On Sun, 14 Apr 2013 15:22:42 +0100
> > Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > > On Sat, 13 Apr 2013 23:23:57 +0200
> > > Michał Górny <mgorny@gentoo.org> wrote:
> > > > As most of you probably doesn't know, PMS guarantees that ${D}
> > > > always ends with a slash. It seems that this particular wording
> > > > was enforced by historical portage behavior (instead of fixing
> > > > the ebuilds...) yet it didn't ever get really widespread.
> > > 
> > > Should ROOT usually be empty then?
> > 
> > Considering its use, probably yes.
> > 
> >   ${ROOT}/usr/bin/foo
> >   ${EPREFIX}/usr/bin/foo
> >   ${D}${EPREFIX}/usr/bin/foo
> > 
> > All seem clean and consistent to me.
> 
> How many things test whether ROOT=/ ? I seem to recall that being
> fairly common, back when something accidentally set it to //.

Yes, that sucks a fair bit. There are many ebuilds doing that, and most
of those ebuilds append additional slash after it anyway... we really
suck at consistency.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Aaron W. Swenson]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

On Sun, Apr 14, 2013 at 06:59:19PM +0200, Michał Górny wrote:
> On Sun, 14 Apr 2013 17:39:44 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> 
> > On Sun, 14 Apr 2013 16:58:07 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> > > On Sun, 14 Apr 2013 15:22:42 +0100
> > > Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > > > On Sat, 13 Apr 2013 23:23:57 +0200
> > > > Michał Górny <mgorny@gentoo.org> wrote:
> > > > Should ROOT usually be empty then?
> > > 
> > > Considering its use, probably yes.
> > > 
> > >   ${ROOT}/usr/bin/foo
> > >   ${EPREFIX}/usr/bin/foo
> > >   ${D}${EPREFIX}/usr/bin/foo
> > > 
> > > All seem clean and consistent to me.
> > 
> > How many things test whether ROOT=/ ? I seem to recall that being
> > fairly common, back when something accidentally set it to //.
> 
> Yes, that sucks a fair bit. There are many ebuilds doing that, and most
> of those ebuilds append additional slash after it anyway... we really
> suck at consistency.
> 

We're only accountable insofar as ROOT defaults to "/".

ROOT being a user set variable, having ROOT be an empty string by
default still does not guarantee that ROOT won't end with a
slash. Even if we change it so that it defaults to an empty string, it
won't negate the need to do ${ROOT%/}/some/path.

The only thing that would help is if PMS defined that ROOT must not
end with a slash. In which case it would be up to the package manage
to modify ROOT before it gets evaluated in an ebuild.

-- 
Mr. Aaron W. Swenson
Gentoo Linux Developer
Email : titanofold@gentoo.org
GnuPG FP : 2C00 7719 4F85 FB07 A49C 0E31 5713 AA03 D1BB FDA0
GnuPG ID : D1BBFDA0

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] [RFC] Cleaning up PMS to have ${D} not end with a slash

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

On Mon, 15 Apr 2013 10:56:45 +0000
"Aaron W. Swenson" <titanofold@gentoo.org> wrote:
> ROOT being a user set variable, having ROOT be an empty string by
> default still does not guarantee that ROOT won't end with a
> slash. Even if we change it so that it defaults to an empty string, it
> won't negate the need to do ${ROOT%/}/some/path.

The spec guarantees that ROOT will be non-empty and end in a slash. If
Portage isn't enforcing this, file a bug.


-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: [RFC] Cleaning up PMS to have ${D} not end with a slash

[Steven J. Long]

On Mon, Apr 15, 2013 at 12:01:24PM +0100, Ciaran McCreesh wrote:
> On Mon, 15 Apr 2013 10:56:45 +0000
> "Aaron W. Swenson" <titanofold@gentoo.org> wrote:
> > ROOT being a user set variable, having ROOT be an empty string by
> > default still does not guarantee that ROOT won't end with a
> > slash. Even if we change it so that it defaults to an empty string, it
> > won't negate the need to do ${ROOT%/}/some/path.
> 
> The spec guarantees that ROOT will be non-empty and end in a slash. If
> Portage isn't enforcing this, file a bug.

Yes, but his point was this:

> > The only thing that would help is if PMS defined that ROOT must not
> > end with a slash.
 
ie, let the mangler enforce empty or a valid directory not ending in a
slash, which is hardly difficult, and makes ROOT easier to work with,
along the lines of how any shellscripter would do it, similarly to the
motivation for the change to D. Only with ROOT it's much more important
that the path is correctly-formed for cross-platform compatibility.

-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: [RFC] Cleaning up PMS to have ${D} not end with a slash

[Gregory M. Turner]

On 4/15/2013 10:31 AM, Steven J. Long wrote:
> On Mon, Apr 15, 2013 at 12:01:24PM +0100, Ciaran McCreesh wrote:
>> On Mon, 15 Apr 2013 10:56:45 +0000
>> "Aaron W. Swenson" <titanofold@gentoo.org> wrote:
>>> ROOT being a user set variable, having ROOT be an empty string by
>>> default still does not guarantee that ROOT won't end with a
>>> slash. Even if we change it so that it defaults to an empty string, it
>>> won't negate the need to do ${ROOT%/}/some/path.
>> The spec guarantees that ROOT will be non-empty and end in a slash. If
>> Portage isn't enforcing this, file a bug.
> Yes, but his point was this:
>
>>> The only thing that would help is if PMS defined that ROOT must not
>>> end with a slash.
>   
> ie, let the mangler enforce empty or a valid directory not ending in a
> slash, which is hardly difficult, and makes ROOT easier to work with,
> along the lines of how any shellscripter would do it, similarly to the
> motivation for the change to D. Only with ROOT it's much more important
> that the path is correctly-formed for cross-platform compatibility.
That's how EPREFIX works; indeed, you don't need to do ${EPREFIX%/}, 
ever (unless, perhaps, we are manually scrubbing user inputs that 
haven't been scrubbed by portage).  Try it yourself: EPREFIX="/" 
portageq envvar EPREFIX.

If we "wanted to" do the same thing to {E,}ROOT we could, at least in 
future EAPIs.

However, given that ROOT is named "ROOT", "/" makes some kind of 
reasonable, intuitive sense.  I see double-slash mistakes involving ROOT 
here and there, but they are not hopelessly ubiquitous as they are with 
D, where, as Michał demonstrates in #465772, approximately 10 ebuilds in 
portage use the variable incorrectly for every 1 getting it right!

-gmt

[gentoo-dev] Re: Re: [RFC] Cleaning up PMS to have ${D} not end with a slash

[Steven J. Long]

On Mon, Apr 15, 2013 at 11:54:22AM -0700, Gregory M. Turner wrote:
> On 4/15/2013 10:31 AM, Steven J. Long wrote:
> > On Mon, Apr 15, 2013 at 12:01:24PM +0100, Ciaran McCreesh wrote:
> >> The spec guarantees that ROOT will be non-empty and end in a slash. If
> >> Portage isn't enforcing this, file a bug.
> > Yes, but his point was this:
> >
> >>> The only thing that would help is if PMS defined that ROOT must not
> >>> end with a slash.
> >   
> > ie, let the mangler enforce empty or a valid directory not ending in a
> > slash, which is hardly difficult, and makes ROOT easier to work with,
> > along the lines of how any shellscripter would do it, similarly to the
> > motivation for the change to D. Only with ROOT it's much more important
> > that the path is correctly-formed for cross-platform compatibility.

> That's how EPREFIX works; indeed, you don't need to do ${EPREFIX%/}, 
> ever (unless, perhaps, we are manually scrubbing user inputs that 
> haven't been scrubbed by portage).  Try it yourself: EPREFIX="/" 
> portageq envvar EPREFIX.

And you took that decision because it makes writing ebuilds simpler.

> If we "wanted to" do the same thing to {E,}ROOT we could, at least in 
> future EAPIs.

Sure, exactly the point: ED and EROOT (equiv D and ROOT) should just be
easy to work with. After all that is the whole reason to provide an ebuild
infrastructure in the first place: to make it easy to write scriptlets that
compile and install software. Anything that improves that, while staying
robust as shell, is worth doing imo. Especially low-hanging fruit like this.

> However, given that ROOT is named "ROOT", "/" makes some kind of 
> reasonable, intuitive sense.

Ofc it does, and I understand where it came from. It just happens to be
useless in shellscript, which is what ebuilds are. Nor is anyone arguing
that it should change in configuration, or eg in portageq. Just in ebuild
space, which was the point Aaron brought up, that Ciaran decided to snip
from his reply, while making some random point about portage that had
nothing to do with anything.

>  I see double-slash mistakes involving ROOT 
> here and there, but they are not hopelessly ubiquitous as they are with 
> D, where, as Michał demonstrates in #465772, approximately 10 ebuilds in 
> portage use the variable incorrectly for every 1 getting it right!

Why not just be consistent across all directory variables, which can be
done very simply in ebuild.sh, and never have to worry about it for any of
them?

Seems a lot simpler and more robust to me, and we can get on with our lives
instead of reading long-winded discussions about whether it's "worth the
effort" to support prefix or alt-installs.

IMO some of those people should "make the effort" to learn the language
they're implementing in, instead of criticizing what they clearly don't
grok. And the rest should go back to the swamp.

Regards,
steveL.
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)