[gentoo-dev] escaping variables in sed expressions

[gentoo-dev] escaping variables in sed expressions

[Marijn Schouten (hkBst)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi list,

it seems I have been using some fragile sed expression and I'd like to tap the collective
wisdom for avoiding doing that in the future.

dev-scheme/slib-3.1.5-r1 currently does

sed "s_prefix = /usr/local/_prefix = ${D}/usr/_" -i Makefile

to make it not violate the sandbox. However a user had set
PORTAGE_TMPDIR=/home/gentoo_overflow/tmp causing the sed expression to contain too may
underscores and failing.[1]

There are several option to handle this. I could use a less common delimiter or I could
escape it: ${D//_/\_} instead of ${D}. I could use a sed expression that doesn't suffer
from this problem (thanks to dleverton):

sed -ne '\_^prefix = /usr/local_!{p;d}' -e "iprefix = ${D}" -i Makefile

Comments?

Marijn

[1]: http://bugs.gentoo.org/show_bug.cgi?id=217735

- --
Marijn Schouten (hkBst), Gentoo Lisp project, Gentoo ML
<http://www.gentoo.org/proj/en/lisp/>, #gentoo-{lisp,ml} on FreeNode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgEjjEACgkQp/VmCx0OL2zGDQCcCcgx1/g/UXpB38HIjKjNhmL6
S4MAoK1aXJS6SW9FaZT4i2iaeo6AlD2u
=Id31
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Santiago M. Mola]

On Tue, Apr 15, 2008 at 1:14 PM, Marijn Schouten (hkBst)
<hkBst@gentoo.org> wrote:
>
>  Hi list,
>
>  it seems I have been using some fragile sed expression and I'd like to tap
> the collective
>  wisdom for avoiding doing that in the future.
>
>  dev-scheme/slib-3.1.5-r1 currently does
>
>  sed "s_prefix = /usr/local/_prefix = ${D}/usr/_" -i Makefile
>
>  to make it not violate the sandbox. However a user had set
>  PORTAGE_TMPDIR=/home/gentoo_overflow/tmp causing the sed expression to
> contain too may
>  underscores and failing.[1]
>
>  There are several option to handle this. I could use a less common
> delimiter or I could
>  escape it: ${D//_/\_} instead of ${D}. I could use a sed expression that
> doesn't suffer
>  from this problem (thanks to dleverton):
>
>  sed -ne '\_^prefix = /usr/local_!{p;d}' -e "iprefix = ${D}" -i Makefile
>
>  Comments?
>

Currently is use ':' as sed delimiter when paths are involved. I'd
also like to hear from you about proper delimiters if you think ':' is
not safe enough.

AFAIK, the only corner case which would make this fail would be
Windows paths (C:/gentoo-prefix).

Regards,

-- 
Santiago M. Mola
Jabber ID: cooldwind@gmail.com
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Fabian Groffen]

On 15-04-2008 13:05:26 +0200, Santiago M. Mola wrote:
> On Tue, Apr 15, 2008 at 1:14 PM, Marijn Schouten (hkBst)
> <hkBst@gentoo.org> wrote:
> >
> >  Hi list,
> >
> >  it seems I have been using some fragile sed expression and I'd like to tap
> > the collective
> >  wisdom for avoiding doing that in the future.
> >
> >  dev-scheme/slib-3.1.5-r1 currently does
> >
> >  sed "s_prefix = /usr/local/_prefix = ${D}/usr/_" -i Makefile
> >
> >  to make it not violate the sandbox. However a user had set
> >  PORTAGE_TMPDIR=/home/gentoo_overflow/tmp causing the sed expression to
> > contain too may
> >  underscores and failing.[1]
> >
> >  There are several option to handle this. I could use a less common
> > delimiter or I could
> >  escape it: ${D//_/\_} instead of ${D}. I could use a sed expression that
> > doesn't suffer
> >  from this problem (thanks to dleverton):
> >
> >  sed -ne '\_^prefix = /usr/local_!{p;d}' -e "iprefix = ${D}" -i Makefile
> >
> >  Comments?
> >
> 
> Currently is use ':' as sed delimiter when paths are involved. I'd
> also like to hear from you about proper delimiters if you think ':' is
> not safe enough.

I met one case where : was indeed a problem, but that was in
CFLAGS/LDFLAGS replacements.  Some linkers accept (and do require)
arguments that are like "-mg:2512s".

> AFAIK, the only corner case which would make this fail would be
> Windows paths (C:/gentoo-prefix).

C:\ iirc, but Cygwin seems to map this as /cygdrive/C, Interix as
/dev/fs/C, command prompt I have no clue how portage could ever normally
work there.  SpikeSource's SpikeWAMP uses Cygwin underneath, so
Portage/ebuilds will see the mapped paths only, never heard of any
problems from them regarding this either.


-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Frank Gruellich]

* Santiago M. Mola <coldwind@gentoo.org> 15. Apr 08:
> On Tue, Apr 15, 2008 at 1:14 PM, Marijn Schouten (hkBst)
> Currently is use ':' as sed delimiter when paths are involved. I'd
> also like to hear from you about proper delimiters if you think ':' is
> not safe enough.
> 
> AFAIK, the only corner case which would make this fail would be
> Windows paths (C:/gentoo-prefix).

Even though it's probably stupid to use it, but ':' is a valid character
within a path.  I've no solution for this problem, however.

Kind regards,
 Frank.
-- 
Sigmentation fault
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Marius Mauch]

On Tue, 15 Apr 2008 16:17:54 +0200
Frank Gruellich <frank@der-frank.org> wrote:

> * Santiago M. Mola <coldwind@gentoo.org> 15. Apr 08:
> > On Tue, Apr 15, 2008 at 1:14 PM, Marijn Schouten (hkBst)
> > Currently is use ':' as sed delimiter when paths are involved. I'd
> > also like to hear from you about proper delimiters if you think ':'
> > is not safe enough.
> > 
> > AFAIK, the only corner case which would make this fail would be
> > Windows paths (C:/gentoo-prefix).
> 
> Even though it's probably stupid to use it, but ':' is a valid
> character within a path.  I've no solution for this problem, however.

Valid maybe (but then pretty much every character is valid), but colon
is used as path delimiter in many other contexts (e.g. $PATH) so it's
rather unlikely to be used.

Marius
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Frank Gruellich]

* Marius Mauch <genone@gentoo.org> 15. Apr 08:
> On Tue, 15 Apr 2008 16:17:54 +0200
> Frank Gruellich <frank@der-frank.org> wrote:
> > * Santiago M. Mola <coldwind@gentoo.org> 15. Apr 08:
> > > On Tue, Apr 15, 2008 at 1:14 PM, Marijn Schouten (hkBst)
> > > Currently is use ':' as sed delimiter when paths are involved. I'd
> > > also like to hear from you about proper delimiters if you think ':'
> > > is not safe enough.
> > Even though it's probably stupid to use it, but ':' is a valid
> > character within a path.  I've no solution for this problem, however.
> Valid maybe (but then pretty much every character is valid),

I've been a bigmouth so I couldn't sleep last night thinking about that
problem (which in fact happened to me sometimes).  The very last
character I'd expect in a path would be the NUL char (\x00).  I was not
able to create a filename or path containing it.  (Anyone else?)
Unfortunately that stupid sed does not work with \x00 as delimiter...

But because a path will not contain a \x00 you can replace all
"$delimiter_of_your_dreams" with a \x00 and later change it back to the
original without adding any new.

Looks like:

(0) frank@bane [~] % echo '/foo/bar/foo:baz/baz bar/laber_rabarber/' |tr '/' '\0' |sed 's/a/o/g' |tr '\0' '/'
/foo/bor/foo:boz/boz bor/lober_roborber/
(0) frank@bane [~] %

Moving that to OP's problem he could maybe use something like:

tr '/' '\0' <Makefile |sed "s_prefix = /usr/local/_prefix = ${D}/usr/_" |tr '\0' '/' >Makefile.new

This obviously introduces the problem that a Makefile contains more than
just paths and there could be a \x00 somewhere... but... well...

> but colon is used as path delimiter in many other contexts (e.g.
> $PATH) so it's rather unlikely to be used.

I'm *very* paranoid.  ;-)

Kind regards,
 Frank.
-- 
Sigmentation fault
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

On Wed, 16 Apr 2008 19:17:51 +0200
Frank Gruellich <frank@der-frank.org> wrote:
> I was not able to create a filename or path containing it.  (Anyone
> else?)

Unix file names can't contain / or null.

> Unfortunately that stupid sed does not work with \x00 as
> delimiter...

Nor do most Unix apps, since they tend to be written in C using all
those C library functions that work on null terminated strings.

Null introduces far more problems than it solves, character-wise...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: escaping variables in sed expressions

[Duncan]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> posted
20080416182405.651698dd@snowcone, excerpted below, on  Wed, 16 Apr 2008
18:24:05 +0100:

> On Wed, 16 Apr 2008 19:17:51 +0200
> Frank Gruellich <frank@der-frank.org> wrote:
>> I was not able to create a filename or path containing it.  (Anyone
>> else?)
> 
> Unix file names can't contain / or null.

Feel free to correct me if I'm mistaken, but I believe I was reading 
somewhere and /thought/ it was this list...

While you are almost certainly correct on POSIX/Unix filenames and the 
shell won't accept / in a filename, IIRC (from reading) it's often 
possible for C programs to code a literal / in a filename, and possible 
for some filesystems (also written in C, generally) to accept it.  Thus, 
while POSIX/Unix standards don't allow it, in practice, it's sometimes 
possible, if rare.

This was an entirely new idea to me when I read it, but it sounded like 
just the sort of filesystem implementation detail one might overlook, so 
I remembered it, I /believe/ accurately.  Whatever your faults, you /do/ 
tend to be quite accurate on such things, so if you'd either confirm this 
or disabuse me of my misinformation, I'd definitely appreciate it.

If it's correct, it's certainly worth considering before one starts 
making absolutist assumptions and statements that could be wrong in some 
cases, particularly as such bad assumptions seem to often lead ultimately 
to security faults.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: escaping variables in sed expressions

[Rémi Cardona]

Duncan a écrit :
> Whatever your faults, you /do/ 
> tend to be quite accurate on such things.

Wow, you've managed to turn a nice technical discussion (which is rare 
enough in recent history) into a let's-start-bashing-people thread.

You've lost all credibility in just one sentence... Pity.

> If it's correct, it's certainly worth considering before one starts 
> making absolutist assumptions and statements that could be wrong in some 
> cases, particularly as such bad assumptions seem to often lead ultimately 
> to security faults.

Well gee, thanks for "considering" Gentoo security, I feel so much 
better now.

Seriously though, please leave the condescending tone of your post at 
the door. This post of yours is seriously out of line (imho).

Thanks

Rémi
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: escaping variables in sed expressions

[Duncan]

Rémi Cardona <remi@gentoo.org> posted 4806E4E3.3080909@gentoo.org,
excerpted below, on  Thu, 17 Apr 2008 07:49:23 +0200:

> Duncan a écrit :
>> Whatever your faults, you /do/
>> tend to be quite accurate on such things.
> 
> Wow, you've managed to turn a nice technical discussion (which is rare
> enough in recent history) into a let's-start-bashing-people thread.
> 
> You've lost all credibility in just one sentence... Pity.

Thanks, someone, for mailing me offlist about this.  I'm incredibly sorry 
it was apparently taken negatively, as I had an entirely different 
meaning in mind, but it obviously didn't come thru as I intended.

I'm not sure I can explain without inserting my foot even further into my 
mouth, but I'll try.  I was intending to acknowledge the potential wound 
but communicate my respect for the person and the technical expertise and 
appreciation for fineness of detail I've repeatedly seen him 
demonstrate.  Unfortunately, it seems I screwed up big time and that's 
not the way it came out; rather than respect, it seemed like dissing.

So I unconditionally apologize for screwing up.  Again, thanks for 
rightfully pointing that out, and thanks as well to the person that 
notified me and tried to help.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: escaping variables in sed expressions

[Santiago M. Mola]

On Thu, Apr 17, 2008 at 6:31 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>
>  While you are almost certainly correct on POSIX/Unix filenames and the
>  shell won't accept / in a filename, IIRC (from reading) it's often
>  possible for C programs to code a literal / in a filename, and possible
>  for some filesystems (also written in C, generally) to accept it.  Thus,
>  while POSIX/Unix standards don't allow it, in practice, it's sometimes
>  possible, if rare.

If that's possible, we shouldn't support it anyway. If someone wants
to use /var/tmp/port\/age we'll just stab him, if someone releases a
tarball with such filenames we'll stab him, too.

-- 
Santiago M. Mola
Jabber ID: cooldwind@gmail.com
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: escaping variables in sed expressions

[Duncan]

"Santiago M. Mola" <coldwind@gentoo.org> posted
3c32af40804170251w353a1329o26433de9256ecba9@mail.gmail.com, excerpted
below, on  Thu, 17 Apr 2008 11:51:32 +0200:

> If that's possible, we shouldn't support it anyway. If someone wants to
> use /var/tmp/port\/age we'll just stab him, if someone releases a
> tarball with such filenames we'll stab him, too.

Agreed.  It's not a question of Gentoo support I'm worried about, but of 
the absolute statement and someone coming across it and relying on it in 
a (possibly unrelated) context in which it perhaps isn't possible to rely 
on, to unknown negative consequences possibly including security issues.

Also, as it's in the upthread, just in case someone doesn't get the 
apology I just sent, let me again unconditionally apologize for the very 
big screwup I made of things above.  It didn't come out the way I 
intended at all, and I thank someone for rightfully pointing that out, 
and also the person who brought it to my attention offlist and allowing 
me to try to correct the problem -- tho I realize it's not something that 
really /can/ be entirely corrected.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: escaping variables in sed expressions

[Olivier Galibert]

On Thu, Apr 17, 2008 at 04:31:59AM +0000, Duncan wrote:
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> posted
> 20080416182405.651698dd@snowcone, excerpted below, on  Wed, 16 Apr 2008
> 18:24:05 +0100:
> 
> > On Wed, 16 Apr 2008 19:17:51 +0200
> > Frank Gruellich <frank@der-frank.org> wrote:
> >> I was not able to create a filename or path containing it.  (Anyone
> >> else?)
> > 
> > Unix file names can't contain / or null.
> 
> Feel free to correct me if I'm mistaken, but I believe I was reading 
> somewhere and /thought/ it was this list...
> 
> While you are almost certainly correct on POSIX/Unix filenames and the 
> shell won't accept / in a filename, IIRC (from reading) it's often 
> possible for C programs to code a literal / in a filename, and possible 
> for some filesystems (also written in C, generally) to accept it.  Thus, 
> while POSIX/Unix standards don't allow it, in practice, it's sometimes 
> possible, if rare.

You can't access or create a file with a visible '/' in it on linux
using filesystem access calls because the character is taken into
account at the vfs level, i.e. before it hits the filesystem.  That
doesn't preclude three things though:

- Some filesystems are supposed to allow it (hfs for instance).  Dunno
  what the filesystem driver does in that case, perhaps an encoding or a
  mapping.  FS-dependant in any case

- You can change the fs at the sector level and forcibly put a '/' in
  a filename there.  What happens then is also fs-dependant, in the
  usual "interesting things can happen when your filesystem is corrupt"
  way.

- You can have series of codes (such as c0 af) which can be
  interpreted by a permissive utf-8 decoder as an encoding of '/'.
  They're valid in a file name, but they're an incorrect utf8
  sequence.  That's a pure userspace issue though.

  OG.
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: escaping variables in sed expressions

[Steve Long]

Ciaran McCreesh wrote:

> Nor do most Unix apps, since they tend to be written in C using all
> those C library functions that work on null terminated strings.
> 
> Null introduces far more problems than it solves, character-wise...
> 
..but it's fine as a terminator, if you know what you're doing.


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[David Leverton]

On Tuesday 15 April 2008 12:14:57 Marijn Schouten (hkBst) wrote:
> There are several option to handle this. I could use a less common
> delimiter or I could escape it: ${D//_/\_} instead of ${D}. I could use a
> sed expression that doesn't suffer from this problem (thanks to dleverton):
>
> sed -ne '\_^prefix = /usr/local_!{p;d}' -e "iprefix = ${D}" -i Makefile

Just to clarify, I didn't think of escaping when I suggested this.  Escaping 
is probably cleaner, and certainly easier to understand.
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Ulrich Mueller]

>>>>> On Tue, 15 Apr 2008, Marijn Schouten (hkBst) wrote:

> There are several option to handle this. I could use a less common
> delimiter or I could escape it: ${D//_/\_} instead of ${D}. I could
> use a sed expression that doesn't suffer from this problem (thanks
> to dleverton):

> sed -ne '\_^prefix = /usr/local_!{p;d}' -e "iprefix = ${D}" -i Makefile

Hi Marijn,

both approaches won't work since you would have to escape not only the
pattern delimiter, but also sed special characters (like "\" and "&")
_and_ make special characters (e.g. the dollar sign).

Probably better to pass it to make via a command argument or
environment variable.

Ulrich
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]

Marijn Schouten (hkBst) kirjoitti:
> Hi list,
> 
> it seems I have been using some fragile sed expression and I'd like to 
> tap the collective
> wisdom for avoiding doing that in the future.
> 
> dev-scheme/slib-3.1.5-r1 currently does
> 
> sed "s_prefix = /usr/local/_prefix = ${D}/usr/_" -i Makefile
> 
> to make it not violate the sandbox. However a user had set
> PORTAGE_TMPDIR=/home/gentoo_overflow/tmp causing the sed expression to 
> contain too may
> underscores and failing.[1]
> 
> There are several option to handle this. I could use a less common 
> delimiter or I could
> escape it: ${D//_/\_} instead of ${D}. I could use a sed expression that 
> doesn't suffer
> from this problem (thanks to dleverton):
> 
> sed -ne '\_^prefix = /usr/local_!{p;d}' -e "iprefix = ${D}" -i Makefile
> 
> Comments?
> 
> Marijn
> 
> [1]: http://bugs.gentoo.org/show_bug.cgi?id=217735
> 

You should just fix the Makefile to respect DESTDIR and send the patch 
upstream.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: [gentoo-dev] escaping variables in sed expressions

[Marijn Schouten (hkBst)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Petteri Räty wrote:
| You should just fix the Makefile to respect DESTDIR and send the patch
| upstream.

You're right of course, but that is only the right way for this specific instance and not
a general way of handling tricky sed expressions.

Marijn

- --
Marijn Schouten (hkBst), Gentoo Lisp project, Gentoo ML
<http://www.gentoo.org/proj/en/lisp/>, #gentoo-{lisp,ml} on FreeNode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgEyckACgkQp/VmCx0OL2z7+QCdHx8n7Py3CQJy1tJMmYkyCmJW
yUgAoIVlu60bCfPJqC1iNIYc8AegKrCm
=1ItB
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] escaping variables in sed expressions

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 568 bytes --]

On Tuesday 15 April 2008, Marijn Schouten (hkBst) wrote:
> Hi list,
>
> it seems I have been using some fragile sed expression and I'd like to tap
> the collective wisdom for avoiding doing that in the future.
>
> dev-scheme/slib-3.1.5-r1 currently does
>
> sed "s_prefix = /usr/local/_prefix = ${D}/usr/_" -i Makefile

ignoring the escape character (which is currently standardized in the tree as 
a colon), expressions like this are inherently fragile due to makefile 
changes.  i tend to do:
sed -i '/^prefix[[:space:]]*=/s:=.*:=/usr' Makefile
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]