[gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Sam James]

Prompted by yet another instance of this, this time at
https://forums.gentoo.org/viewtopic-t-1171999.html.

The results of these tests are often hardcoded into installed files
which causes issues if using a binpkg of them from a merged-usr system
on a non-merged-usr system. Just set the cache variables to avoid that.

Bug: https://bugs.gentoo.org/927837
Bug: https://bugs.gentoo.org/927864
Bug: https://bugs.gentoo.org/935721
Signed-off-by: Sam James <sam@gentoo.org>
---
 profiles/default/linux/make.defaults | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/profiles/default/linux/make.defaults b/profiles/default/linux/make.defaults
index 12dd86f25c531..f74c0cb8e3a5d 100644
--- a/profiles/default/linux/make.defaults
+++ b/profiles/default/linux/make.defaults
@@ -58,3 +58,14 @@ enable_year2038="no"
 # on others. Spoof the same result as configure gets on a modern glibc system
 # for now. See bug #447970 and bug #922652.
 gl_cv_func_getcwd_path_max="yes"
+
+# Sam James <sam@gentoo.org> (2024-12-04)
+# The results of these tests are often hardcoded into installed files
+# which causes issues if using a binpkg of them from a merged-usr system
+# on a non-merged-usr system. Just set the cache variables to avoid that.
+ac_cv_path_SED="sed"
+ac_cv_path_EGREP="grep -E"
+ac_cv_path_EGREP_TRADITIONAL="grep -E"
+ac_cv_path_FGREP="grep -F"
+ac_cv_path_GREP="grep"
+ac_cv_path_lt_DD="dd"
-- 
2.47.1

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Michael Orlitzky]

On 2024-12-04 22:55:22, Sam James wrote:
> Prompted by yet another instance of this, this time at
> https://forums.gentoo.org/viewtopic-t-1171999.html.
> 
> The results of these tests are often hardcoded into installed files
> which causes issues if using a binpkg of them from a merged-usr system
> on a non-merged-usr system. Just set the cache variables to avoid that.
> ...
> +ac_cv_path_SED="sed"

Obviously it would defeat the purpose to put a full path in there, but
won't this break software that expects them to be a path (since that's
what the autoconf macros are documented to do)?

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Sam James]

Michael Orlitzky <mjo@gentoo.org> writes:

> On 2024-12-04 22:55:22, Sam James wrote:
>> Prompted by yet another instance of this, this time at
>> https://forums.gentoo.org/viewtopic-t-1171999.html.
>> 
>> The results of these tests are often hardcoded into installed files
>> which causes issues if using a binpkg of them from a merged-usr system
>> on a non-merged-usr system. Just set the cache variables to avoid that.
>> ...
>> +ac_cv_path_SED="sed"
>
> Obviously it would defeat the purpose to put a full path in there, but
> won't this break software that expects them to be a path (since that's
> what the autoconf macros are documented to do)?

That's a fair point and I'm not sure how to handle it. We could use
profile.bashrc to set these?

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

On Tue, Dec 10, 2024 at 05:46:38AM +0000, Sam James wrote:
> Michael Orlitzky <mjo@gentoo.org> writes:
> 
> > On 2024-12-04 22:55:22, Sam James wrote:
> >> Prompted by yet another instance of this, this time at
> >> https://forums.gentoo.org/viewtopic-t-1171999.html.
> >> 
> >> The results of these tests are often hardcoded into installed files
> >> which causes issues if using a binpkg of them from a merged-usr system
> >> on a non-merged-usr system. Just set the cache variables to avoid that.
> >> ...
> >> +ac_cv_path_SED="sed"
> >
> > Obviously it would defeat the purpose to put a full path in there, but
> > won't this break software that expects them to be a path (since that's
> > what the autoconf macros are documented to do)?
> 
> That's a fair point and I'm not sure how to handle it. We could use
> profile.bashrc to set these?

Assume you're already aware, but still to remind that profile.bashrc
is not in PMS and ideally shouldn't rely on this for anything but
optional sanity checks or such.

Albeit I don't understand what would be doing in there, setting the
full path for the current profile wouldn't accomplish anything for
binpkg-on-different-usr-type-profile.

(on a related another note, do hope autoconf moves to being handled
like cmake/meson in the future so these things could just be in an
eclass -- we could do more complex things if needed too without
needing profile.bashrc)

That aside, I personally feel that an option could instead be
to consider merged-usr binpkg on non-merged-usr as unsupported
and only support merged-usr for binary packages.

The issue still semi-exists for users migrating their systems, but
it's at least mitigated by 23.0 suggesting emerge -e @world and thus
updating every paths.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Sam James]

Ionen Wolkens <ionen@gentoo.org> writes:

> On Tue, Dec 10, 2024 at 05:46:38AM +0000, Sam James wrote:
>> Michael Orlitzky <mjo@gentoo.org> writes:
>> 
>> > On 2024-12-04 22:55:22, Sam James wrote:
>> >> Prompted by yet another instance of this, this time at
>> >> https://forums.gentoo.org/viewtopic-t-1171999.html.
>> >> 
>> >> The results of these tests are often hardcoded into installed files
>> >> which causes issues if using a binpkg of them from a merged-usr system
>> >> on a non-merged-usr system. Just set the cache variables to avoid that.
>> >> ...
>> >> +ac_cv_path_SED="sed"
>> >
>> > Obviously it would defeat the purpose to put a full path in there, but
>> > won't this break software that expects them to be a path (since that's
>> > what the autoconf macros are documented to do)?
>> 
>> That's a fair point and I'm not sure how to handle it. We could use
>> profile.bashrc to set these?
>
> Assume you're already aware, but still to remind that profile.bashrc
> is not in PMS and ideally shouldn't rely on this for anything but
> optional sanity checks or such.
>
> Albeit I don't understand what would be doing in there, setting the
> full path for the current profile wouldn't accomplish anything for
> binpkg-on-different-usr-type-profile.

Oh, of course...

>
> (on a related another note, do hope autoconf moves to being handled
> like cmake/meson in the future so these things could just be in an
> eclass -- we could do more complex things if needed too without
> needing profile.bashrc)
>
> That aside, I personally feel that an option could instead be
> to consider merged-usr binpkg on non-merged-usr as unsupported
> and only support merged-usr for binary packages.

I start to think it might be a better idea, although the transition
might be a bit rough. The main problem with these bugs to begin with of
course isn't the build failures when they're obvious but when they're
more subtle.

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 2811 bytes --]

On 12/10/24 3:02 AM, Ionen Wolkens wrote:
> Albeit I don't understand what would be doing in there, setting the
> full path for the current profile wouldn't accomplish anything for
> binpkg-on-different-usr-type-profile.


It surely would -- the point of setting it for the current profile is to
make ./configure stop storing overly-specific build time paths, which
means that the resulting binpkg is now well-formed and works on both
profiles.

It is a bug when e.g. `libgcrypt-config` stores contents like this:

    for i in $includes $cflags_final; do
       if echo "$tmp" | /usr/bin/grep -F -v -- "$i" >/dev/null; then


But the suggested profile change would make this be:


    for i in $includes $cflags_final; do
       if echo "$tmp" | grep -F -v -- "$i" >/dev/null; then



And Michael is suggesting we instead encode the "API filename":


    for i in $includes $cflags_final; do
       if echo "$tmp" | /bin/grep -F -v -- "$i" >/dev/null; then



Since on both split and merged-usr systems, grep is always in /bin --
same as bash and sh


> (on a related another note, do hope autoconf moves to being handled
> like cmake/meson in the future so these things could just be in an
> eclass -- we could do more complex things if needed too without
> needing profile.bashrc)


tbh, /usr/share/config.site exists for this. Despite previous history of
being used for fragile caching of arbitrary configure checks that ended
up dismantled and given up as too dangerous, it remains a perfectly
suitable place for configuring impossible-to-change information such as
policy-defined paths.


> That aside, I personally feel that an option could instead be
> to consider merged-usr binpkg on non-merged-usr as unsupported
> and only support merged-usr for binary packages.


Sure, we could declare tons of things as unsupported. We could declare
split-usr as unsupported, in fact, which would feel a bit more honest
than saying "split-usr is supported, we have profiles for it, just if
you use it there are weird footguns and then your system breaks".

It's not a hard fix. It's correcting something that should be the case
regardless, even for merged-usr systems. It's quite silly for scripts to
configure the absolute path to `grep`.


> The issue still semi-exists for users migrating their systems, but
> it's at least mitigated by 23.0 suggesting emerge -e @world and thus
> updating every paths.


I don't see how that's supposed to help, especially given that if you
migrate your system then affected packages are defined as "affected"
based on whether they are broken and fail to work, and that remains true
during -e as well since packages depending on the broken package will
fail to successfully -e.


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

On Tue, Dec 10, 2024 at 03:26:38AM -0500, Eli Schwartz wrote:
> On 12/10/24 3:02 AM, Ionen Wolkens wrote:
> > Albeit I don't understand what would be doing in there, setting the
> > full path for the current profile wouldn't accomplish anything for
> > binpkg-on-different-usr-type-profile.
> 
> 
> It surely would -- the point of setting it for the current profile is to
> make ./configure stop storing overly-specific build time paths, which
> means that the resulting binpkg is now well-formed and works on both
> profiles.

I said *full* path. I was trying to figure out the reasoning of using
profile.bashrc vs just doing ="sed" without a path in profile itself,
and guessing what may have wanted to do there (like a conditional
to set a different path, or read EPREFIX.. but that just replicate
what ./configure will figure out itself).
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 2238 bytes --]

On Tue, Dec 10, 2024 at 03:26:38AM -0500, Eli Schwartz wrote:
> On 12/10/24 3:02 AM, Ionen Wolkens wrote:
> > That aside, I personally feel that an option could instead be
> > to consider merged-usr binpkg on non-merged-usr as unsupported
> > and only support merged-usr for binary packages.
> 
> 
> Sure, we could declare tons of things as unsupported. We could declare
> split-usr as unsupported, in fact, which would feel a bit more honest
> than saying "split-usr is supported, we have profiles for it, just if
> you use it there are weird footguns and then your system breaks".

No need to take it to extremes, binpkgs aren't everything and there's
no need to drop support for something just because the binpkgs don't
support it. I think it's fair if binpkgs only support the primary
configurations, much like how they use default profile USEs given
supporting all USE configurations is virtually impossible.

Ideally it'd be nice if binpkgs could make the distinction between
profiles they are compatible with or not so it wouldn't be a "footgun".
Also an issue for non-binpkg where users switch to profiles that need
migration without doing the migration (at best we've been trying to
stop them with profile.bashrc or ebuild checks).

Aka, distinction between profiles that do or need something special
vs profiles that just change a few USE/masks isn't clear cut.

> > The issue still semi-exists for users migrating their systems, but
> > it's at least mitigated by 23.0 suggesting emerge -e @world and thus
> > updating every paths.
> 
> 
> I don't see how that's supposed to help, especially given that if you
> migrate your system then affected packages are defined as "affected"
> based on whether they are broken and fail to work, and that remains true
> during -e as well since packages depending on the broken package will
> fail to successfully -e.

I said semi-affected given there's a compat symlink that keeps things
working, the paths are just kind of technically wrong but works.

It would be an issue for merged-usr -> non-merged-usr switch, but we
don't have a migration path nor tools for the other way around so users
always been a bit on their own there.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1084 bytes --]

On 12/5/24 7:44 PM, Michael Orlitzky wrote:
> On 2024-12-04 22:55:22, Sam James wrote:
>> Prompted by yet another instance of this, this time at
>> https://forums.gentoo.org/viewtopic-t-1171999.html.
>>
>> The results of these tests are often hardcoded into installed files
>> which causes issues if using a binpkg of them from a merged-usr system
>> on a non-merged-usr system. Just set the cache variables to avoid that.
>> ...
>> +ac_cv_path_SED="sed"
> 
> Obviously it would defeat the purpose to put a full path in there, but
> won't this break software that expects them to be a path (since that's
> what the autoconf macros are documented to do)?


It "should" only ever break software that uses them to configure shebang
lines, but this is uncommon for anything other than SH and BASH.
Attempting to treat EGREP, FGREP, and EGREP_TRADITIONAL as paths of any
sort will be outright broken, even, since "/bin/grep -E" can't be used
as a full filepath anyway.

What circumstances other than a shebang might break without a full path?


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Michael Orlitzky]

On 2024-12-10 00:54:11, Eli Schwartz wrote:
>
> What circumstances other than a shebang might break without a full path?

When PATH is not reliable, like inside a cron job. Or arguments to execve().

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1685 bytes --]

On 12/10/24 1:31 PM, Michael Orlitzky wrote:
> On 2024-12-10 00:54:11, Eli Schwartz wrote:
>>
>> What circumstances other than a shebang might break without a full path?
> 
> When PATH is not reliable, like inside a cron job. Or arguments to execve().


execve doesn't make any sense to me as an argument. I don't think I've
ever in my life either seen or heard of software that did a
configure-time check for the full abspath of a file in order to solve
the fact that execve doesn't search on PATH. They simply used execvpe
instead.

There may be other reasons for doing such a configure-time check, but
"PATH is not reliable" is not one of them...

Inside cron doesn't make a ton of sense to me either. A cron job will
use a basic system PATH including these utilities already, and it
frequently surprises people that the PATH set up in their ~/.bashrc
isn't applied -- things like /opt/barware/bin and $HOME/.local/bin.

Hardcoding the path to /bin/grep won't help in the slightest here, but
hardcoding the path to a script installed in a non-default location
would help. The script is then going to run with a basic
PATH=/bin:/usr/bin and any core system utilities (such as grep, sed, dd)
will be found but custom user scripts in the same directory as the
configured crontab script will *not* be found.

In both cases, there's nothing to gain from ensuring that autoconf sees
full paths.

For the execve case it's possible to write a proof of concept
demonstrating the issue, but writing a PoC to demonstrate that people
can write incorrect code doesn't prove that people *do* write incorrect
code so the logic is circular.

-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-dev] [PATCH] profiles/default/linux: export cache variables for sed and friends

[Michael Orlitzky]

On 2024-12-10 14:33:26, Eli Schwartz wrote:
> On 12/10/24 1:31 PM, Michael Orlitzky wrote:
> > On 2024-12-10 00:54:11, Eli Schwartz wrote:
> >>
> >> What circumstances other than a shebang might break without a full path?
> > 
> > When PATH is not reliable, like inside a cron job. Or arguments to execve().
> 
> execve doesn't make any sense to me as an argument. I don't think I've
> ever in my life either seen or heard of software that did a
> configure-time check for the full abspath of a file in order to solve
> the fact that execve doesn't search on PATH. They simply used execvpe
> instead.
> 
> ...
> 
> For the execve case it's possible to write a proof of concept
> demonstrating the issue, but writing a PoC to demonstrate that people
> can write incorrect code doesn't prove that people *do* write incorrect
> code so the logic is circular.

I think you think I'm making an argument that I'm not making. Here's
the scenario I imagine:

  * I want to run sed from another program
  * I put AC_PROG_SED in configure.ac
  * I notice that $SED is an absolute path, so I use execve() to run
    it

Or, maybe more realistically, suppose I don't know WTF I'm doing and
choose execve() from the list of exec* methods because I tried it and
works. I'm not using execve (as opposed to execvpe) to solve any
particular problem here, it's just what happened.