[gentoo-dev] emerge harmless buglet?

[gentoo-dev] emerge harmless buglet?

[Jani Monoses]

With latest stable portage I always get errors like below while emerging
packages. 
Changing a line in portage.py makes this go away.Whether spawn is not
supposed to be used with a multiple commands piping eachother I don't
know but anyway isn't using bunzip2 easier ?

at line 4775 :
        print ">>> extracting",mypkg
#       notok=spawn("cat "+mytbz2+"| bzip2 -dq | tar xpf -",free=1)
        notok=spawn("bunzip2 -q "+mytbz2, free=1)

The borken pipe error is there at every emerge.

i.e
>>> extracting info
>>> extracting gcc-config-1.3.3-r1
cat: write error: Broken pipe
>>> Merging sys-devel/gcc-config-1.3.3-r1 to /


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Marius Mauch]

On 08/04/03  Jani Monoses wrote:

> With latest stable portage I always get errors like below while
> emerging packages. 
> Changing a line in portage.py makes this go away.Whether spawn is not
> supposed to be used with a multiple commands piping eachother I don't
> know but anyway isn't using bunzip2 easier ?
> 
> at line 4775 :
>         print ">>> extracting",mypkg
> #       notok=spawn("cat "+mytbz2+"| bzip2 -dq | tar xpf -",free=1)
>         notok=spawn("bunzip2 -q "+mytbz2, free=1)
> 
> The borken pipe error is there at every emerge.
> 
> i.e
> >>> extracting info
> >>> extracting gcc-config-1.3.3-r1
> cat: write error: Broken pipe
> >>> Merging sys-devel/gcc-config-1.3.3-r1 to /


It's fixed in current 49_pre versions. And only using bunzip2 won't work
as it only decompresses the package, but it does not untar it. You have
to send the uncompressed tarball to tar. The fixed line is
	notok=spawn("bzip2 -dc "+mytbz2+" | tar xpf -",free=1)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[splite-gentoo]

On Mon, Aug 04, 2003 at 03:32:28PM +0200, Marius Mauch wrote:
>
> It's fixed in current 49_pre versions. And only using bunzip2 won't work
> as it only decompresses the package, but it does not untar it. You have
> to send the uncompressed tarball to tar. The fixed line is
> 	notok=spawn("bzip2 -dc "+mytbz2+" | tar xpf -",free=1)

Why not:

 	notok=spawn("tar xpjf "+mytbz2,free=1)

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Mike Frysinger]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 598 bytes --]

On Monday 04 August 2003 10:34, splite-gentoo@sigint.cs.purdue.edu wrote:
> On Mon, Aug 04, 2003 at 03:32:28PM +0200, Marius Mauch wrote:
> > The fixed line is
> > 	notok=spawn("bzip2 -dc "+mytbz2+" | tar xpf -",free=1)

actually thats wrong, you'll still get a warning msg ;)
if you look at the bug i filed sometime ago (#25293) the better line is:
notok=spawn("bzcat -q "+mytbz2+" | tar xpf -",free=1)

> Why not:
>
>  	notok=spawn("tar xpjf "+mytbz2,free=1)

that assumes that tar supports bzip2 ... anyone who has used bsd/older *nixs 
will tell you, dont plan on it :)

-mike

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[splite-gentoo]

On Mon, Aug 04, 2003 at 03:52:58PM -0400, Mike Frysinger wrote:
Content-Description: signed data
> On Monday 04 August 2003 10:34, splite-gentoo@sigint.cs.purdue.edu wrote:
>
> > Why not:
> >
> >  	notok=spawn("tar xpjf "+mytbz2,free=1)
> 
> that assumes that tar supports bzip2 ... anyone who has used bsd/older *nixs 
> will tell you, dont plan on it :)

Non-GNU tars don't support the "-z", "--files-from", "--no-recursion",
"--exclude", or "--no-same-owner" flags either.  emerge-webrsync and
quickpkg already use the "-j" flag.

# fgrep -r 'tar ' /usr/portage/eclass /usr/lib/portage
/usr/portage/eclass/eutils.eclass:      # to tar which will make tar not extract anything and exit with 0
/usr/portage/eclass/eutils.eclass:      local out="`(tail +${skip} ${src} | gzip -cd | tar -x --no-same-owner -f -) 2>&1`"
/usr/portage/eclass/eutils.eclass:              tail +${skip} ${src} | tar -x --no-same-owner -f - \
/usr/portage/eclass/rpm.eclass:# 2. if it is a source rpm, it finds all .tar .tar.gz, .tgz, .tbz2, .tar.bz2,
/usr/portage/eclass/rpm.eclass:# output tar again but directly extracts to ${WORKDIR}
/usr/portage/eclass/selinux-policy.eclass:      tar -C /etc/security/selinux/src  --exclude policy.12 --exclude tmp \
/usr/lib/portage/bin/ebuild.sh:                         tar ${tarvars} -xf ${DISTDIR}/${x} || die "$myfail"
/usr/lib/portage/bin/ebuild.sh:                         tar ${tarvars} -xzf ${DISTDIR}/${x} || die "$myfail"
/usr/lib/portage/bin/ebuild.sh:                         bzip2 -dc ${DISTDIR}/${x} | tar ${tarvars} -xf - || die "$myfail"
/usr/lib/portage/bin/ebuild.sh:                                 tar ${tarvars} -xzf ${DISTDIR}/${x} || die "$myfail"
/usr/lib/portage/bin/ebuild.sh:                                 bzip2 -dc ${DISTDIR}/${x} | tar ${tarvars} -xf - || die "$myfail"
/usr/lib/portage/bin/ebuild.sh: tar cpvf - * | bzip2 -f > ../bin.tar.bz2 || die "Failed to create tarball"
/usr/lib/portage/bin/ebuild.sh: tar czf /usr/src/redhat/SOURCES/${PF}.tar.gz ${O}/${PF}.ebuild ${O}/files || die "Failed to create base rpm tarball."
/usr/lib/portage/bin/emerge-webrsync:   tar jxf $file
/usr/lib/portage/bin/quickpkg:  tar cjvf ${MYDIR}/bin.tar.bz2 --files-from=${MYDIR}/filelist --no-recursion

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Mike Frysinger]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1471 bytes --]

On Monday 04 August 2003 16:28, splite-gentoo@sigint.cs.purdue.edu wrote:
> Non-GNU tars don't support the "-z", "--files-from", "--no-recursion",
> "--exclude", or "--no-same-owner" flags either.  emerge-webrsync and
> quickpkg already use the "-j" flag.

ugh, quickpkg also uses 'no-recursion' ... and unpack() in ebuild.sh uses 
'no-same-owner' ... any ideas on how to get around the owner bug ?  if we 
want the userpriv to be a reality we need that ...

how about we just replace bsd-tar's with GNU tars ;)

seriously though, the problem i have with piping into tar is that you cant 
catch errors ... (i hope someone can correct me though)

for example:
bzcat t.tbz2 | tar xf - || echo "oh no i failed"
assume that t.tbz2 either doesnt exist, or is invalid bz2 file, or cant be 
read ... whatever, the point is bzcat fails ...
that failure results in a pretty little msg that informs us that t.tbz2 was 
unable to be decompressed.  so, bzcat returns non 0 ... but what does that 
mean for tar ?
bzcat produced 0 bytes in its stdout pipe so tar got 0 bytes in its stdin pipe 
... tar, unfortunately, doesnt seem to mind getting 0 bytes.  it'll exit 
happily with a return of 0.

anyone have thoughts on how to handle this as intended ?

the ugly hack that i produced (as can be found in unpack_makeself() in 
eutils.eclass) is something like this:
out="`bzcat t.tbz2 | tar xf - 2>1`"
[ ! -z "${out}" ] && echo "oh no i failed"

-mike

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[Aron Griffis]

Vapier wrote:	[Mon Aug 04 2003, 06:15:50PM EDT]
> anyone have thoughts on how to handle this as intended ?

Use PIPE_STATUS.

I've thought about writing another one like die/assert.  It would be
assert_pipe, which would require that each element in the pipe returns
zero exit status.  What do you think?  (History: I wrote "die" in the
first place.)

Aron

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Mike Frysinger]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 572 bytes --]

On Monday 04 August 2003 18:20, Aron Griffis wrote:
> Vapier wrote:	[Mon Aug 04 2003, 06:15:50PM EDT]
>
> > anyone have thoughts on how to handle this as intended ?
>
> Use PIPE_STATUS.
>
> I've thought about writing another one like die/assert.  It would be
> assert_pipe, which would require that each element in the pipe returns
> zero exit status.  What do you think?

please do and i'll make sure it is integrated into portage :)
right now unpack() is affected by this too ...

> (History: I wrote "die" in the first place.)

i'll handle your 'reward' off list
-mike

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[Aron Griffis]

Vapier wrote:	[Mon Aug 04 2003, 06:43:49PM EDT]
> please do and i'll make sure it is integrated into portage :)
> right now unpack() is affected by this too ...

Actually, a scan of the portage tree reveals that every case where
assert is called there is only one element in the pipeline preceding
(i.e. there is no pipeline).

This means that it would also be an option to change the behavior of
assert instead of adding a new API to portage.  It wouldn't affect any
of the ebuilds currently in the tree.  I think this would be a better
option since right now, assert is essentially the same thing as || die.

> > (History: I wrote "die" in the first place.)
> 
> i'll handle your 'reward' off list -mike

:-)  I didn't write that for rewards or self-gratification; I was...
qualifying my intent to write an additional API.

Aron

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Mike Frysinger]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 515 bytes --]

On Monday 04 August 2003 22:07, Aron Griffis wrote:
> Vapier wrote:	[Mon Aug 04 2003, 06:43:49PM EDT]
>
> > please do and i'll make sure it is integrated into portage :)
> > right now unpack() is affected by this too ...
>
> Actually, a scan of the portage tree reveals that every case where
> assert is called there is only one element in the pipeline preceding
> (i.e. there is no pipeline).

that'd be cool ...
put it together and file a bug or send it to me and ill handle it from there 
:)
-mike

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[Aron Griffis]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

Vapier wrote:	[Mon Aug 04 2003, 10:11:05PM EDT]
> put it together and file a bug or send it to me and ill handle it from
> there 

Here's the entire patch.

Aron

[-- Attachment #2: ebuild.sh.patch --]
[-- Type: text/plain, Size: 511 bytes --]

--- /usr/lib/portage/bin/ebuild.sh	2003-07-27 22:35:02.000000000 -0400
+++ ebuild.sh	2003-08-04 22:52:57.000000000 -0400
@@ -50,7 +50,7 @@
 # It _must_ preceed all the calls to die and assert.
 shopt -s expand_aliases
 alias die='diefunc "$FUNCNAME" "$LINENO" "$?"'
-alias assert='_retval=$?; [ $_retval = 0 ] || diefunc "$FUNCNAME" "$LINENO" "$_retval"'
+alias assert='_pipestatus="${PIPESTATUS[*]}"; [[ "${_pipestatus// /}" -eq 0 ]] || diefunc "$FUNCNAME" "$LINENO" "$_pipestatus"'
 
 OCC="$CC"
 OCXX="$CXX"


[-- Attachment #3: Type: text/plain, Size: 37 bytes --]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[splite-gentoo]

On Mon, Aug 04, 2003 at 06:15:50PM -0400, Mike Frysinger wrote:
Content-Description: signed data
>
> anyone have thoughts on how to handle this as intended ?
> 
> the ugly hack that i produced (as can be found in unpack_makeself() in 
> eutils.eclass) is something like this:
> out="`bzcat t.tbz2 | tar xf - 2>1`"
> [ ! -z "${out}" ] && echo "oh no i failed"

Also a bit ugly, but one could do:

{ bzcat t.tbz2 || echo "bzcat failed" >&2; } | tar xf - || echo "tar failed" >&2

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Marius Mauch]

On 08/04/03  splite-gentoo@sigint.cs.purdue.edu wrote:

> On Mon, Aug 04, 2003 at 03:32:28PM +0200, Marius Mauch wrote:
> >
> > It's fixed in current 49_pre versions. And only using bunzip2 won't
> > work as it only decompresses the package, but it does not untar it.
> > You have to send the uncompressed tarball to tar. The fixed line is
> > 	notok=spawn("bzip2 -dc "+mytbz2+" | tar xpf -",free=1)
> 
> Why not:
>  	notok=spawn("tar xpjf "+mytbz2,free=1)

I didn't fix it so I can only guess: the -j option for tar is a GNU
extension and portage aims at OS compatibility with *BSD, MacOS and
cygwin, these systems are not using GNU tar by default. The pipe
solutions should work on all these systems.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1126 bytes --]

On Tuesday 05 August 2003 04:28, Marius Mauch wrote:
> On 08/04/03  splite-gentoo@sigint.cs.purdue.edu wrote:
> > On Mon, Aug 04, 2003 at 03:32:28PM +0200, Marius Mauch wrote:
> > > It's fixed in current 49_pre versions. And only using bunzip2 won't
> > > work as it only decompresses the package, but it does not untar it.
> > > You have to send the uncompressed tarball to tar. The fixed line is
> > > 	notok=spawn("bzip2 -dc "+mytbz2+" | tar xpf -",free=1)
> >
> > Why not:
> >  	notok=spawn("tar xpjf "+mytbz2,free=1)
>
> I didn't fix it so I can only guess: the -j option for tar is a GNU
> extension and portage aims at OS compatibility with *BSD, MacOS and
> cygwin, these systems are not using GNU tar by default. The pipe
> solutions should work on all these systems.

It is actually an unofficial extension that has the same meaning as the z 
option except it is for bz2. For portability one should use 
"--use-compress-program bzip2" though, which is exactly what it is a 
shorthand for.

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.cs.kun.nl/~pauldv

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[Spider]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

begin  quote
On Tue, 5 Aug 2003 04:28:17 +0200
Marius Mauch <genone@genone.homeip.net> wrote:


> I didn't fix it so I can only guess: the -j option for tar is a GNU
> extension and portage aims at OS compatibility with *BSD, MacOS and
> cygwin, these systems are not using GNU tar by default. The pipe
> solutions should work on all these systems.
> 

Does this mean we can start implementing "ln -s star tar"  in our
systems?   (star is another tar implementation thats more posix
compliant than GNU tar is. )


//Spider



-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[Donny Davies]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

Hi Spider

On Tue, Aug 05, 2003 at 12:43:56PM +0200, Spider wrote:
>Does this mean we can start implementing "ln -s star tar"  in our
>systems?   (star is another tar implementation thats more posix
>compliant than GNU tar is. )

It may produce archives more in-line with POSIX standards but
the command-line options are different.

Donny.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[Spider]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

begin  quote
On Tue, 5 Aug 2003 11:29:29 -0400
Donny Davies <woodchip@gentoo.org> wrote:

> Hi Spider
> 
> On Tue, Aug 05, 2003 at 12:43:56PM +0200, Spider wrote:
> >Does this mean we can start implementing "ln -s star tar"  in our
> >systems?   (star is another tar implementation thats more posix
> >compliant than GNU tar is. )
>
> It may produce archives more in-line with POSIX standards but
> the command-line options are different.

Yes, but if portage starts to become more POSIX compliant and compliant
with the BSD implementation it will make such a transition easier. 


//Spider

-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] emerge harmless buglet?

[Aron Griffis]

Spider wrote:	[Tue Aug 05 2003, 06:43:56AM EDT]
> Does this mean we can start implementing "ln -s star tar"  in our
> systems?   (star is another tar implementation thats more posix
> compliant than GNU tar is. )

This already seems to be the case on my system.

$ epm -qf /usr/bin/tar
star-1.5_alpha14

Aron

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] emerge harmless buglet?

[Robin H.Johnson]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Tue, Aug 05, 2003 at 12:05:43PM -0400, Aron Griffis wrote:
> Spider wrote:	[Tue Aug 05 2003, 06:43:56AM EDT]
> > Does this mean we can start implementing "ln -s star tar"  in our
> > systems?   (star is another tar implementation thats more posix
> > compliant than GNU tar is. )
> This already seems to be the case on my system.
> 
> $ epm -qf /usr/bin/tar
> star-1.5_alpha14
GNU tar is installed in /bin/tar
Star installs in /usr/bin/tar

While Star may be more posix compliant, be very careful with it, as
things that depend on GNU tar don't like it (eg amanda).

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]