Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[rindeal]

Have you guys read
https://stackoverflow.com/questions/16529716/awk-save-modifications-inplace
?

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]

rindeal <dev.rindeal@gmail.com> writes:
> Have you guys read
> https://stackoverflow.com/questions/16529716/awk-save-modifications-inplace
> ?

a) mawk doesn't support it.
b) 4.1 is not stabilized, yet.

-- Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[rindeal]

> a) mawk doesn't support it.
> b) 4.1 is not stabilized, yet.

a) if you depend on mawk then you can really create your own wrapper
in your ebuild or do it inline
b) then perhaps put your efforts that way

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

On Sat, 21 May 2016 20:45:18 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> On 21 May 2016 17:08, Amadeusz Żołnowski wrote:
> > This function is too generic to be in
> > rebar.eclass, so I have decided to move it to eutils. What is the
> > problem with adding it to eutils?  
> 
> because no one else is using it or cares about it.  we've gotten by
> for more than 10 years at this point w/out someone asking for this.

Which makes me think of the terribly ugly evar_push which is used only
by you, as part of bad epatch implementation and one eblit. Yet it is
in eutils, and from time to time we have to tell our users 'you can do
things more correctly than using a crappy function that looks like it's
official'.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]

On 21 May 2016 17:08, Amadeusz Żołnowski wrote:
> Mike Frysinger <vapier@gentoo.org> writes:
> >> The same "sed -i" is used. I have some configs to edit in place in
> >> src_prepare(). It's easier with awk rather than sed.
> >
> > again, provide an example.  one or two uncommon use cases doesn't justify
> > being added to eutils.
> 
> See rebar.eclass review where it is used. Later ejabberd ebuild I'm
> going to add uses it as well.

neither of those are terribly compelling and can be written in sed.
you want to do a match & replace in that just as easily.  roughly:
	sed -i \
		-e '/{[[:space:]]*deps[[:space:]]*,\/,\/}/{s:.*:{deps, []}:}' \
		foo

> This function is too generic to be in
> rebar.eclass, so I have decided to move it to eutils. What is the
> problem with adding it to eutils?

because no one else is using it or cares about it.  we've gotten by
for more than 10 years at this point w/out someone asking for this.

if you really want it in rebar, then just DEPEND on gawk and use the
inplace flag in your code.  then there's no need for eawk anywhere.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

Michał Górny <mgorny@gentoo.org> writes:>
>> +	# Following commands should always succeed unless something weird is going
>> +	# on.
>> +	cat "${tmpf}" >"${f}" || die 'failed to replace source file' || return
>> +	rm "${tmpf}" || die "failed to remove temporary file"
>
> Any reason not to use mv here? Aside to making this simpler, it would
> also prevent this from rewriting contents of hardlinked files.

Permissions of ${f} wouldn't be preserved.

>> +	return 1
>> +}
>> +
>> +tbegin "preserves permissions"
>> +
>> +cd "${tmpdir}" || tend $?
>
> This doesn't terminate tests. So you'll end up creating files
> in the wrong directory.

Of course! :-/ Probably it would be best to put each test case in a
function. I have the same issue in tests for rebar.eclass. Thanks!

PS. As of discussion with robbat2, I am probably going to depend on
gawk-4.1 (which has in-place edit feature) and use it explicitely in
rebar.eclass and some other ebuilds where I meant to use eawk. robbat2
has already requested stabilization of gawk-4.1.

Thanks,
-- Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]

On Sat, 21 May 2016 14:49:41 +0100
aidecoe@gentoo.org wrote:

> From: Amadeusz Żołnowski <aidecoe@gentoo.org>
> 
> awk doesn't have the -i option like sed and if editing file in place is
> desired, additional steps are required. eawk uses tmp file to make it
> look to the caller editing happens in place.
> 
> New version of gawk (not stabilized yet) does support editing in place
> but forcing user to install specific awk implementation is not desired.
> ---
>  eclass/eutils.eclass        | 16 +++++++++++
>  eclass/tests/eutils_eawk.sh | 66 +++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 82 insertions(+)
>  create mode 100755 eclass/tests/eutils_eawk.sh
> 
> diff --git a/eclass/eutils.eclass b/eclass/eutils.eclass
> index dbedffe..963a692 100644
> --- a/eclass/eutils.eclass
> +++ b/eclass/eutils.eclass
> @@ -20,6 +20,22 @@ _EUTILS_ECLASS=1
>  
>  inherit multilib toolchain-funcs
>  
> +# @FUNCTION: eawk
> +# @USAGE: <file> <args>
> +# @DESCRIPTION:
> +# Edit file <file> in place with awk. Pass all arguments following <file> to
> +# awk.
> +eawk() {
> +	local f="$1"; shift
> +	local tmpf="$(emktemp)"
> +
> +	awk "$@" "${f}" >"${tmpf}" || die -n 'awk failed' || return

You can't use 'die -n' before EAPI 6. So you either need a conditional
there, or make the function available only in EAPI 6.

> +	# Following commands should always succeed unless something weird is going
> +	# on.
> +	cat "${tmpf}" >"${f}" || die 'failed to replace source file' || return
> +	rm "${tmpf}" || die "failed to remove temporary file"

Any reason not to use mv here? Aside to making this simpler, it would
also prevent this from rewriting contents of hardlinked files.

> +}
> +
>  # @FUNCTION: eqawarn
>  # @USAGE: [message]
>  # @DESCRIPTION:
> diff --git a/eclass/tests/eutils_eawk.sh b/eclass/tests/eutils_eawk.sh
> new file mode 100755
> index 0000000..7e0d1d4
> --- /dev/null
> +++ b/eclass/tests/eutils_eawk.sh
> @@ -0,0 +1,66 @@
> +#!/bin/bash
> +# Copyright 1999-2016 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +# $Id$
> +
> +source tests-common.sh
> +
> +inherit eutils
> +
> +# Mock die so it doesn't break tests.
> +die() {
> +	echo "die: $*" 1>&2
> +	return 1
> +}
> +
> +tbegin "preserves permissions"
> +
> +cd "${tmpdir}" || tend $?

This doesn't terminate tests. So you'll end up creating files
in the wrong directory.

> +
> +cat <<EOF >'test.txt'

Then, you aren't checking for failure here.

> +testme1
> +testme2
> +testme3
> +EOF
> +
> +cat <<EOF >'test_expected.txt'
> +testme1
> +foo
> +testme3
> +EOF
> +
> +chmod 704 'test.txt' || tend $?
> +eumask_push 000
> +eawk 'test.txt' '/^testme2$/ {print "foo"; next;} 1' || tend $?
> +eumask_pop
> +
> +diff 'test.txt' 'test_expected.txt'
> +expected=$?
> +
> +[[ $(stat -c '%a' 'test.txt') = 704 ]]
> +perms=$?
> +
> +[[ ${expected}${perms} = 00 ]]
> +
> +tend $?
> +
> +
> +tbegin "doesn't alter file on failure"
> +
> +cd "${tmpdir}" || tend $?
> +
> +cat <<EOF >'test.txt'
> +testme1
> +testme2
> +testme3
> +EOF
> +
> +cat 'test.txt' >'test_expected.txt'
> +
> +# eawk should file because of syntax error.
> +eawk 'test.txt' '/^testme2$/ print "foo"; next;} 1' 2>/dev/null && tend 1
> +diff 'test.txt' 'test_expected.txt'
> +
> +tend $?
> +
> +texit



-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 836 bytes --]

On 05/21/2016 06:08 PM, Amadeusz Żołnowski wrote:
> Mike Frysinger <vapier@gentoo.org> writes:
>>> The same "sed -i" is used. I have some configs to edit in place in
>>> src_prepare(). It's easier with awk rather than sed.
>>
>> again, provide an example.  one or two uncommon use cases doesn't justify
>> being added to eutils.
> 
> See rebar.eclass review where it is used. Later ejabberd ebuild I'm
> going to add uses it as well. This function is too generic to be in
> rebar.eclass, so I have decided to move it to eutils. What is the
> problem with adding it to eutils?

Is the use-case sufficiently common for it to be sourced in all ebuilds
inheriting eutils?

-- 
Kristian Fiskerstrand
OpenPGP certificate reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 547 bytes --]

Mike Frysinger <vapier@gentoo.org> writes:
>> The same "sed -i" is used. I have some configs to edit in place in
>> src_prepare(). It's easier with awk rather than sed.
>
> again, provide an example.  one or two uncommon use cases doesn't justify
> being added to eutils.

See rebar.eclass review where it is used. Later ejabberd ebuild I'm
going to add uses it as well. This function is too generic to be in
rebar.eclass, so I have decided to move it to eutils. What is the
problem with adding it to eutils?

-- Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On 20 May 2016 21:09, Amadeusz Żołnowski wrote:
> Mike Frysinger <vapier@gentoo.org> writes:
> > On 18 May 2016 22:25, aidecoe@gentoo.org wrote:
> >> awk doesn't have the -i option like sed and if editing file in place is
> >> desired, additional steps are required. eawk uses tmp file to make it
> >> look to the caller editing happens in place.
> >
> > what's your real use case ?  i've never once thought "man, i wish i could
> > run an awk script and modify a file in place".  and i write awk most days.
> 
> The same "sed -i" is used. I have some configs to edit in place in
> src_prepare(). It's easier with awk rather than sed.

again, provide an example.  one or two uncommon use cases doesn't justify
being added to eutils.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[aidecoe]

From: Amadeusz Żołnowski <aidecoe@gentoo.org>

awk doesn't have the -i option like sed and if editing file in place is
desired, additional steps are required. eawk uses tmp file to make it
look to the caller editing happens in place.

New version of gawk (not stabilized yet) does support editing in place
but forcing user to install specific awk implementation is not desired.
---
 eclass/eutils.eclass        | 16 +++++++++++
 eclass/tests/eutils_eawk.sh | 66 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+)
 create mode 100755 eclass/tests/eutils_eawk.sh

diff --git a/eclass/eutils.eclass b/eclass/eutils.eclass
index dbedffe..963a692 100644
--- a/eclass/eutils.eclass
+++ b/eclass/eutils.eclass
@@ -20,6 +20,22 @@ _EUTILS_ECLASS=1
 
 inherit multilib toolchain-funcs
 
+# @FUNCTION: eawk
+# @USAGE: <file> <args>
+# @DESCRIPTION:
+# Edit file <file> in place with awk. Pass all arguments following <file> to
+# awk.
+eawk() {
+	local f="$1"; shift
+	local tmpf="$(emktemp)"
+
+	awk "$@" "${f}" >"${tmpf}" || die -n 'awk failed' || return
+	# Following commands should always succeed unless something weird is going
+	# on.
+	cat "${tmpf}" >"${f}" || die 'failed to replace source file' || return
+	rm "${tmpf}" || die "failed to remove temporary file"
+}
+
 # @FUNCTION: eqawarn
 # @USAGE: [message]
 # @DESCRIPTION:
diff --git a/eclass/tests/eutils_eawk.sh b/eclass/tests/eutils_eawk.sh
new file mode 100755
index 0000000..7e0d1d4
--- /dev/null
+++ b/eclass/tests/eutils_eawk.sh
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Copyright 1999-2016 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+# $Id$
+
+source tests-common.sh
+
+inherit eutils
+
+# Mock die so it doesn't break tests.
+die() {
+	echo "die: $*" 1>&2
+	return 1
+}
+
+tbegin "preserves permissions"
+
+cd "${tmpdir}" || tend $?
+
+cat <<EOF >'test.txt'
+testme1
+testme2
+testme3
+EOF
+
+cat <<EOF >'test_expected.txt'
+testme1
+foo
+testme3
+EOF
+
+chmod 704 'test.txt' || tend $?
+eumask_push 000
+eawk 'test.txt' '/^testme2$/ {print "foo"; next;} 1' || tend $?
+eumask_pop
+
+diff 'test.txt' 'test_expected.txt'
+expected=$?
+
+[[ $(stat -c '%a' 'test.txt') = 704 ]]
+perms=$?
+
+[[ ${expected}${perms} = 00 ]]
+
+tend $?
+
+
+tbegin "doesn't alter file on failure"
+
+cd "${tmpdir}" || tend $?
+
+cat <<EOF >'test.txt'
+testme1
+testme2
+testme3
+EOF
+
+cat 'test.txt' >'test_expected.txt'
+
+# eawk should file because of syntax error.
+eawk 'test.txt' '/^testme2$/ print "foo"; next;} 1' 2>/dev/null && tend 1
+diff 'test.txt' 'test_expected.txt'
+
+tend $?
+
+texit
-- 
2.8.2

[gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[aidecoe]

From: Amadeusz Żołnowski <aidecoe@gentoo.org>

awk doesn't have the -i option like sed and if editing file in place is
desired, additional steps are required. eawk uses tmp file to make it
look to the caller editing happens in place.

New version of gawk (not stabilized yet) does support editing in place
but forcing user to install specific awk implementation is not desired.
---
 eclass/eutils.eclass        | 16 +++++++++++
 eclass/tests/eutils_eawk.sh | 65 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 81 insertions(+)
 create mode 100755 eclass/tests/eutils_eawk.sh

diff --git a/eclass/eutils.eclass b/eclass/eutils.eclass
index dbedffe..963a692 100644
--- a/eclass/eutils.eclass
+++ b/eclass/eutils.eclass
@@ -20,6 +20,22 @@ _EUTILS_ECLASS=1
 
 inherit multilib toolchain-funcs
 
+# @FUNCTION: eawk
+# @USAGE: <file> <args>
+# @DESCRIPTION:
+# Edit file <file> in place with awk. Pass all arguments following <file> to
+# awk.
+eawk() {
+	local f="$1"; shift
+	local tmpf="$(emktemp)"
+
+	awk "$@" "${f}" >"${tmpf}" || die -n 'awk failed' || return
+	# Following commands should always succeed unless something weird is going
+	# on.
+	cat "${tmpf}" >"${f}" || die 'failed to replace source file' || return
+	rm "${tmpf}" || die "failed to remove temporary file"
+}
+
 # @FUNCTION: eqawarn
 # @USAGE: [message]
 # @DESCRIPTION:
diff --git a/eclass/tests/eutils_eawk.sh b/eclass/tests/eutils_eawk.sh
new file mode 100755
index 0000000..b06f377
--- /dev/null
+++ b/eclass/tests/eutils_eawk.sh
@@ -0,0 +1,65 @@
+#!/bin/bash
+# Copyright 1999-2016 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+# $Id$
+
+source tests-common.sh
+
+inherit eutils
+
+# Mock die so it doesn't break tests.
+die() {
+	echo "die: $*" 1>&2
+	return 1
+}
+
+tbegin "preserves permissions"
+
+cd "$(emktemp -d)"
+
+cat <<EOF >'test.txt'
+testme1
+testme2
+testme3
+EOF
+
+cat <<EOF >'test_expected.txt'
+testme1
+foo
+testme3
+EOF
+
+chmod 704 'test.txt'
+eumask_push 000
+eawk 'test.txt' '/^testme2$/ {print "foo"; next;} 1'
+eumask_pop
+
+diff 'test.txt' 'test_expected.txt'
+expected=$?
+
+[[ $(stat -c '%a' 'test.txt') = 704 ]]
+perms=$?
+
+[[ ${expected}${perms} = 00 ]]
+
+tend $?
+
+
+tbegin "doesn't alter file on failure"
+
+cd "$(emktemp -d)"
+
+cat <<EOF >'test.txt'
+testme1
+testme2
+testme3
+EOF
+
+cat 'test.txt' >'test_expected.txt'
+
+eawk 'test.txt' '/^testme2$/ print "foo"; next;} 1' 2>/dev/null
+diff 'test.txt' 'test_expected.txt'
+
+tend $?
+
+texit
-- 
2.8.2

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

Amadeusz Żołnowski <aidecoe@gentoo.org> writes:
> Indeed. So one of these:
>
>   a) eawk <file> <awk_args...>
>   b) eawk <files...> -- <awk_args...>
>   c) eawk <awk_args...> -- <files...>

Ups, actually (c) wouldn't be correct either, so only (a) and (b) and
I'd just stick to (a) to not complicate things.

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Mike Frysinger <vapier@gentoo.org> writes:
> On 18 May 2016 22:25, aidecoe@gentoo.org wrote:
>> awk doesn't have the -i option like sed and if editing file in place is
>> desired, additional steps are required. eawk uses tmp file to make it
>> look to the caller editing happens in place.
>
> what's your real use case ?  i've never once thought "man, i wish i could
> run an awk script and modify a file in place".  and i write awk most days.

The same "sed -i" is used. I have some configs to edit in place in
src_prepare(). It's easier with awk rather than sed.

-- Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[rindeal]

Reasons for inplace editing are the same for any language and tool,
but having a wrapper just because some tool doesn't support it, is a
bloat. Especially when gawk>=4.1 supports it.

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Patrice Clement]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

Thursday 19 May 2016 23:08:05, Mike Frysinger wrote :
> On 18 May 2016 22:25, aidecoe@gentoo.org wrote:
> > awk doesn't have the -i option like sed and if editing file in place is
> > desired, additional steps are required. eawk uses tmp file to make it
> > look to the caller editing happens in place.
> 
> what's your real use case ?  i've never once thought "man, i wish i could
> run an awk script and modify a file in place".  and i write awk most days.
> -mike

Same question. Why would you ever want to run awk and have it in-place edit a
file? Have you ever heard about Perl? In place editing a file with it is as
easy as ABC:

$ perl -i'' -ne 's/foo/bar/g;' file.txt

And you're done. If you don't fancy Perl, you could use Ruby with the exact
same syntax and it works just as well. Then add either of these to DEPEND and
you're set.

Please walk us through why you actually want this function to be added cause I
read this thread and your diff like 10 times and couldn't come up with a
sensible reason which would justify this change.

-- 
Patrice Clement
Gentoo Linux developer
http://www.gentoo.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

"Robin H. Johnson" <robbat2@gentoo.org> writes:
>> > cp -f "${tmpf}" "$f" || die "copy back failed"
>> 
>> Why '-f' is required?
> Corner cases w/ copying and bad ebuild usage (specifically running
> ebuild as two different non-root users during development). Mostly
> habit.

Reading:

  http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html

in 3.a.ii it is stated 'cp' should behave like redirection if file
already exists, but in case it must actually force (3.a.iii) it does
unlink first and in that case permissions of destination are not
preserved. So it should either be 'cp' alone or redirection as I already
have.

> Another thought I had, is that if you're using eawk on something in
> $ROOT, you're also going to clobber ownership, permissions and
> extended attributes.

Yes, that's why I have used stream redirection in the first place to
make it explicit/clear that permissions and attributes are preserved.

> If we declare the calling format in BSD style:
> <args> <files>
>
> Then this is negated, but means you need to be able to find the first
> file in the list, which requires parsing the options correctly.

Which is not even an option here. (Pun intended.)

> Another variation that you haven't considered above is passing
> multiple programs to awk via -e or -f options, in which case there is
> no non-option program argument.

Indeed. So one of these:

  a) eawk <file> <awk_args...>
  b) eawk <files...> -- <awk_args...>
  c) eawk <awk_args...> -- <files...>
 
in (b) and (c) "--" is required. This makes it easy to separate options
From files without parsing args. With (c) we would have interface almost
similar to awk (only that we require "--"). (a) is simplest in
implementation and doesn't require "--". I am not sure which is most
intuitive. What do you think?

-- Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

Mart Raudsepp <leio@gentoo.org> writes:
> Similarly, I would like that all ebuild that call 'sed' actually DEPEND
> on sed, not just half of them.
> I would also like that no ebuild would assume packages in @system to be
> present, beyond those dictated by PMS (unpackers and such).

Please don't divert from the subject of the thread.

Thanks,
-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

>>>>> On Thu, 19 May 2016, Amadeusz Żołnowski wrote:

> Ulrich Mueller <ulm@gentoo.org> writes:
>> Yes, of course. "|| die -n || return" if you want the function to
>> return at that point, if it was called under nonfatal.
>>
>>> https://blogs.gentoo.org/mgorny/2015/11/13/the-ultimate-guide-to-eapi-6
>>
>> Note that the ${?} in the code example there is redundant.

> Why redundant? I imagine we can call it like:

>     if ! nonfatal efoo; then
>         do_something_on_error
>     fi

> can't we?

The status of the last command will be returned in any case, so there
is no difference between "|| return" and "|| return $?".

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

On 18 May 2016 22:25, aidecoe@gentoo.org wrote:
> awk doesn't have the -i option like sed and if editing file in place is
> desired, additional steps are required. eawk uses tmp file to make it
> look to the caller editing happens in place.

what's your real use case ?  i've never once thought "man, i wish i could
run an awk script and modify a file in place".  and i write awk most days.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On 20 May 2016 02:16, Mart Raudsepp wrote:
> I would like if such a function would explicitly document that calling
> this function means the caller should DEPEND on an awk provider.
> Similarly, I would like that all ebuild that call 'sed' actually DEPEND
> on sed, not just half of them.
> I would also like that no ebuild would assume packages in @system to be
> present, beyond those dictated by PMS (unpackers and such).

this is all a terrible terrible idea and has been hashed/explained a
number of times on this list.  have you ever tried looking at the tools
configure scripts run ?
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Göktürk Yüksek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Mart Raudsepp:
> Ühel kenal päeval, K, 18.05.2016 kell 22:25, kirjutas 
> aidecoe@gentoo.org:
>> From: Amadeusz Żołnowski <aidecoe@gentoo.org>
>> 
>> awk doesn't have the -i option like sed and if editing file in 
>> place is desired, additional steps are required. eawk uses tmp 
>> file to make it look to the caller editing happens in place. ---
>>  eclass/eutils.eclass | 13 +++++++++++++ 1 file changed, 13 
>> insertions(+)
>> 
>> diff --git a/eclass/eutils.eclass b/eclass/eutils.eclass index 
>> dbedffe..e331f1b 100644 --- a/eclass/eutils.eclass +++ 
>> b/eclass/eutils.eclass @@ -20,6 +20,19 @@ _EUTILS_ECLASS=1
>> 
>> inherit multilib toolchain-funcs
>> 
>> +# @FUNCTION: eawk +# @USAGE: <file> <args> +# @DESCRIPTION: +# 
>> Edit file <file> in place with awk. Pass all arguments following
>>  <file> to +# awk.
>> 
> 
> I would like if such a function would explicitly document that 
> calling this function means the caller should DEPEND on an awk 
> provider. Similarly, I would like that all ebuild that call 'sed' 
> actually DEPEND on sed, not just half of them.

According to PMS, sed is guaranteed to be present[0].

> I would also like that no ebuild would assume packages in @system 
> to be present, beyond those dictated by PMS (unpackers and such).
> 
Quoting PMS [1]:

"[...] in most cases this means that the only packages that can be
assumed to be present are those listed in the base profile or
equivalent [...]"

Implicit system dependencies are documented in devmanual as well. May
I ask why you are against it?

> Mart
> 

[0] https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-12700011.3.1.1
[1] https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-12600011.3.1
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJXPkxQAAoJEIT4AuXAiM4zH/gH/2KJIdAbL1x8TfvWIeOnjQ4E
IatAXY9DentQgGVFHU+bwOj+bdgMfaU3QACjDAD/GOPYTwrOSlHD2dK7spkou7RS
VY3voejyvK1tHASooMvEaaH8VQ9HMe82JDS64YTfxMLFb8FcwJOYde3plVSFPArA
UGhNCzEpgMJBkd89lS/BsmtHqS0czuAO+VJdVttMo1EyW8CR1s6ou+FVzymgJM/+
P8Zg6eVIcyJOQqQcnpRbDPvSw2lGNP7BofZjFng578DNz3F1LAtdBFBujNNT3kUW
FgbifW7Y53hnx3nYjjBigBp8ERm3HuhDXQiQMNEOsFlfIWKarp4017NC8BmH9i4=
=jHCS
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Mart Raudsepp]

Ühel kenal päeval, K, 18.05.2016 kell 22:25, kirjutas
aidecoe@gentoo.org:
> From: Amadeusz Żołnowski <aidecoe@gentoo.org>
> 
> awk doesn't have the -i option like sed and if editing file in place
> is
> desired, additional steps are required. eawk uses tmp file to make it
> look to the caller editing happens in place.
> ---
>  eclass/eutils.eclass | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/eclass/eutils.eclass b/eclass/eutils.eclass
> index dbedffe..e331f1b 100644
> --- a/eclass/eutils.eclass
> +++ b/eclass/eutils.eclass
> @@ -20,6 +20,19 @@ _EUTILS_ECLASS=1
>  
>  inherit multilib toolchain-funcs
>  
> +# @FUNCTION: eawk
> +# @USAGE: <file> <args>
> +# @DESCRIPTION:
> +# Edit file <file> in place with awk. Pass all arguments following
> <file> to
> +# awk.
> 

I would like if such a function would explicitly document that calling
this function means the caller should DEPEND on an awk provider.
Similarly, I would like that all ebuild that call 'sed' actually DEPEND
on sed, not just half of them.
I would also like that no ebuild would assume packages in @system to be
present, beyond those dictated by PMS (unpackers and such).

Mart

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Robin H. Johnson]

On Thu, May 19, 2016 at 10:50:48PM +0100, Amadeusz Żołnowski wrote:
> "Robin H. Johnson" <robbat2@gentoo.org> writes:
> > 1.
> > If your AWK command has an error, then this clobbers the file, a
> > better variant, with die, would be:
> > awk "$@" "$f" >"${tmpf}" || die "awk failed..."
> > cp -f "${tmpf}" "$f" || die "copy back failed"
> 
> Thanks! On the one hand it's a simple function, and on the other hand
> there are several things which can go wrong. I will submit corrected
> version with unit test.
> 
> Why '-f' is required?
Corner cases w/ copying and bad ebuild usage (specifically running
ebuild as two different non-root users during development). Mostly
habit.

Another thought I had, is that if you're using eawk on something in
$ROOT, you're also going to clobber ownership, permissions and extended
attributes.

> >> # @USAGE: <file> <args>
> > This calling format is also a problem if they intended to pass multiple
> > files to the command, or if they do so accidentally. 
> >
> > Eg, an unprotected glob in the first argument, eg:
> > eawk foo* '/foobar/{$1="x"; print $0}'
> >
> > If foo1 and foo2 existed, this would be considered as:
> > awk foo2 /foobar/{$1="x"; print $0}' foo1
> >
> > which really will NOT have the desired effect.
> 
> I am aware of this. I considered before:
> 
>   a) eawk files... -- awk_args...
>   b) eawk awk_args -- pattern files...
> 
> making '--' required, but that just complicates usage and function
> itself. And anyway isn't it true for any other command with careless
> glob use?
> 
> I have documented that function takes a file, not files. The effect of
> misuse eawk would be awk error. Although maybe it would be useful to
> support multiple files? Then I'd go for variant (b) probably.
If we declare the calling format in BSD style:
<args> <files>

Then this is negated, but means you need to be able to find the first
file in the list, which requires parsing the options correctly.

Another variation that you haven't considered above is passing multiple
programs to awk via -e or -f options, in which case there is no
non-option program argument.

I agree generally that this is a useful idea, but there's a lot of
caveats and corner-cases :-(.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

"Robin H. Johnson" <robbat2@gentoo.org> writes:
> 1.
> If your AWK command has an error, then this clobbers the file, a
> better variant, with die, would be:
> awk "$@" "$f" >"${tmpf}" || die "awk failed..."
> cp -f "${tmpf}" "$f" || die "copy back failed"

Thanks! On the one hand it's a simple function, and on the other hand
there are several things which can go wrong. I will submit corrected
version with unit test.

Why '-f' is required?


>> # @USAGE: <file> <args>
> This calling format is also a problem if they intended to pass multiple
> files to the command, or if they do so accidentally. 
>
> Eg, an unprotected glob in the first argument, eg:
> eawk foo* '/foobar/{$1="x"; print $0}'
>
> If foo1 and foo2 existed, this would be considered as:
> awk foo2 /foobar/{$1="x"; print $0}' foo1
>
> which really will NOT have the desired effect.

I am aware of this. I considered before:

  a) eawk files... -- awk_args...
  b) eawk awk_args -- pattern files...

making '--' required, but that just complicates usage and function
itself. And anyway isn't it true for any other command with careless
glob use?

I have documented that function takes a file, not files. The effect of
misuse eawk would be awk error. Although maybe it would be useful to
support multiple files? Then I'd go for variant (b) probably.

Cheers,

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

Ulrich Mueller <ulm@gentoo.org> writes:
> Yes, of course. "|| die -n || return" if you want the function to
> return at that point, if it was called under nonfatal.
>
>> https://blogs.gentoo.org/mgorny/2015/11/13/the-ultimate-guide-to-eapi-6
>
> Note that the ${?} in the code example there is redundant.

Why redundant? I imagine we can call it like:

    if ! nonfatal efoo; then
        do_something_on_error
    fi

can't we?

-- Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

>>>>> On Thu, 19 May 2016, Mart Raudsepp wrote:

>> Ulrich Mueller <ulm@gentoo.org> writes:
>> > The EAPI 6 method would be to use "|| die -n". Then the caller
>> > either call "eawk" if the function should die automatically, or
>> > "nonfatal eawk" if it should return with an error status.

> Don't you possibly need a || return as well in there if it isn't the
> last things in the function?

Yes, of course. "|| die -n || return" if you want the function to
return at that point, if it was called under nonfatal.

> https://blogs.gentoo.org/mgorny/2015/11/13/the-ultimate-guide-to-eapi-6

Note that the ${?} in the code example there is redundant.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Mart Raudsepp]

Ühel kenal päeval, N, 19.05.2016 kell 18:51, kirjutas Amadeusz
Żołnowski:
> Ulrich Mueller <ulm@gentoo.org> writes:
> > The EAPI 6 method would be to use "|| die -n". Then the caller can
> > either call "eawk" if the function should die automatically, or
> > "nonfatal eawk" if it should return with an error status.

Don't you possibly need a || return as well in there if it isn't the
last things in the function?
https://blogs.gentoo.org/mgorny/2015/11/13/the-ultimate-guide-to-eapi-6

> Yes, that seems better. Hasn't it been introduced earlier than in
> EAPI 6?

die -n argument is new in EAPI 6
nonfatal was added in EAPI 4, together with the fatality defaults for
PM utilities.
See https://devmanual.gentoo.org/ebuild-writing/eapi/


Mart

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]

Ulrich Mueller <ulm@gentoo.org> writes:
> The EAPI 6 method would be to use "|| die -n". Then the caller can
> either call "eawk" if the function should die automatically, or
> "nonfatal eawk" if it should return with an error status.

Yes, that seems better. Hasn't it been introduced earlier than in
EAPI 6?

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

>>>>> On Wed, 18 May 2016, Amadeusz Żołnowski wrote:

>> Also, wouldn't the absence of 'die' cause silent breakages?

> I want to caller decide whether die or not and what error messge to
> give.

The EAPI 6 method would be to use "|| die -n". Then the caller can
either call "eawk" if the function should die automatically, or
"nonfatal eawk" if it should return with an error status.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Göktürk Yüksek]

Jeroen Roovers:
> On Wed, 18 May 2016 19:31:38 -0400
> Göktürk Yüksek <gokturk@gentoo.org> wrote:
> 
>> There could be some performance implications. cat will usually do
>> slow, buffered I/O. cp tries to be smarter with allocation, i.e. it
>> may take advantage of the btrfs specific clone to do a O(1) copy.
> 
> Really? We're talking about editing streams of usually a couple of
> kilobytes and performance is what you're worried about? Because of
> not using one particular filesystem maybe? Really?
> 
> 
I wasn't against the use of redirection, just wondering if there was a
reason to prefer that over cp. To me, an extra dup2() in redirection
seems useless. You can also argue that an extra syscall() has no visible
impact and I wouldn't object. cp should work as efficient as shell
redirection and potentially better in some cases. Sure, it won't make a
noticeable difference with such small files.

>      jer
> 

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Jeroen Roovers]

On Wed, 18 May 2016 19:31:38 -0400
Göktürk Yüksek <gokturk@gentoo.org> wrote:

> There could be some performance implications. cat will usually do
> slow, buffered I/O. cp tries to be smarter with allocation, i.e. it
> may take advantage of the btrfs specific clone to do a O(1) copy.

Really? We're talking about editing streams of usually a couple of
kilobytes and performance is what you're worried about? Because of
not using one particular filesystem maybe? Really?


     jer

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Göktürk Yüksek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Amadeusz Żołnowski:
> Göktürk Yüksek <gokturk@gentoo.org> writes:
>>> +	cat "${f}" >"${tmpf}" || return 1
>> Why shell redirection with cat instead of cp? both are in
>> coreutils.
> 
> I thought cp could overwrite file mode of already existing tmp
> file, but actually it doesn't, so cp can be here as well. Is there
> actual benefit of using cp here?
> 
There could be some performance implications. cat will usually do
slow, buffered I/O. cp tries to be smarter with allocation, i.e. it
may take advantage of the btrfs specific clone to do a O(1) copy.

>> Also, wouldn't the absence of 'die' cause silent breakages?
> 
> I want to caller decide whether die or not and what error messge
> to give.
> 
Maybe that should be part of the documentation. People are more likely
to assume that it will die on its own, I think. Is there a case in
which you would not want to die?

> Cheers,
> 

-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJXPPtVAAoJEIT4AuXAiM4zhX4H+gNivSvNvQzf3tsjmbVatGWB
DLIvIxUY7h79c0YDTXnaU+4LLf7CvED26XT9QNWp297OfNpsXKWGTR5GbdG+w0ab
SR490l9Uh/qn8HBKvh1hmPTEyeRHXm0ZoQ6/jrgwlz7ehSawQlzGRUPqFOHj88Fc
qwWhKk1p2fc7TUnRO0SUY/xcRyZeEyUqEG3ueVYfPlTo10rC5+B20IFjnO09vwbT
INV27KHqjLnpRT4xGWvCyCiEfw6DDypxZ/PKfr4c7omHF8OF+5GGnf+FFZ8UJFkt
93TVIPDeUC9XeAsmqirgN5LuYWsNNUwTEdiDfjglLrTb6OgXA0eIIwXsGH/G3d4=
=J3uT
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]

Göktürk Yüksek <gokturk@gentoo.org> writes:
>> +	cat "${f}" >"${tmpf}" || return 1
> Why shell redirection with cat instead of cp? both are in coreutils.

I thought cp could overwrite file mode of already existing tmp file, but
actually it doesn't, so cp can be here as well. Is there actual benefit
of using cp here?

> Also, wouldn't the absence of 'die' cause silent breakages?

I want to caller decide whether die or not and what error messge to
give.

Cheers,

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Robin H. Johnson]

On Wed, May 18, 2016 at 10:25:02PM +0100, aidecoe@gentoo.org wrote:
> From: Amadeusz Żołnowski <aidecoe@gentoo.org>
> 
> awk doesn't have the -i option like sed and if editing file in place is
> desired, additional steps are required. eawk uses tmp file to make it
> look to the caller editing happens in place.
NAK.

1.
If your AWK command has an error, then this clobbers the file, a better
variant, with die, would be:
awk "$@" "$f" >"${tmpf}" || die "awk failed..."
cp -f "${tmpf}" "$f" || die "copy back failed"

> # @USAGE: <file> <args>
This calling format is also a problem if they intended to pass multiple
files to the command, or if they do so accidentally. 

Eg, an unprotected glob in the first argument, eg:
eawk foo* '/foobar/{$1="x"; print $0}'

If foo1 and foo2 existed, this would be considered as:
awk foo2 /foobar/{$1="x"; print $0}' foo1

which really will NOT have the desired effect.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Re: [gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[Göktürk Yüksek]

aidecoe@gentoo.org:
> From: Amadeusz Żołnowski <aidecoe@gentoo.org>
> 
> awk doesn't have the -i option like sed and if editing file in place is
> desired, additional steps are required. eawk uses tmp file to make it
> look to the caller editing happens in place.
> ---
>  eclass/eutils.eclass | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/eclass/eutils.eclass b/eclass/eutils.eclass
> index dbedffe..e331f1b 100644
> --- a/eclass/eutils.eclass
> +++ b/eclass/eutils.eclass
> @@ -20,6 +20,19 @@ _EUTILS_ECLASS=1
>  
>  inherit multilib toolchain-funcs
>  
> +# @FUNCTION: eawk
> +# @USAGE: <file> <args>
> +# @DESCRIPTION:
> +# Edit file <file> in place with awk. Pass all arguments following <file> to
> +# awk.
> +eawk() {
> +	local f="$1"; shift
> +	local tmpf="$(emktemp)"
> +
> +	cat "${f}" >"${tmpf}" || return 1
Why shell redirection with cat instead of cp? both are in coreutils.
Also, wouldn't the absence of 'die' cause silent breakages?

> +	awk "$@" "${tmpf}" >"${f}"
> +}
> +
>  # @FUNCTION: eqawarn
>  # @USAGE: [message]
>  # @DESCRIPTION:
> 

[gentoo-dev] [PATCH] eutils.eclass: Add awk wrapper - eawk - edit file in place

[aidecoe]

From: Amadeusz Żołnowski <aidecoe@gentoo.org>

awk doesn't have the -i option like sed and if editing file in place is
desired, additional steps are required. eawk uses tmp file to make it
look to the caller editing happens in place.
---
 eclass/eutils.eclass | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/eclass/eutils.eclass b/eclass/eutils.eclass
index dbedffe..e331f1b 100644
--- a/eclass/eutils.eclass
+++ b/eclass/eutils.eclass
@@ -20,6 +20,19 @@ _EUTILS_ECLASS=1
 
 inherit multilib toolchain-funcs
 
+# @FUNCTION: eawk
+# @USAGE: <file> <args>
+# @DESCRIPTION:
+# Edit file <file> in place with awk. Pass all arguments following <file> to
+# awk.
+eawk() {
+	local f="$1"; shift
+	local tmpf="$(emktemp)"
+
+	cat "${f}" >"${tmpf}" || return 1
+	awk "$@" "${tmpf}" >"${f}"
+}
+
 # @FUNCTION: eqawarn
 # @USAGE: [message]
 # @DESCRIPTION:
-- 
2.8.2