[gentoo-user] Tailing compressed build logs

[gentoo-user] Tailing compressed build logs

[Bryan Gardiner]

Hi folks,

How can I follow Portage's compressed build logs in real time as they
are generated?

I keep build logs and use FEATURES=compress-build-logs so that they
don't get too large.  I can peek at how a build is going with zless on
build.log.gz, which doesn't update (understandably), but I would
really like to be able to watch a log with some "tail -f" equivalent.
I get streaming output with

    tail -c +1 -f build.log.gz | od -t x1

but the following hangs with no output:

    tail -c +1 -f build.log.gz | gunzip

even with a build log that is 72KB compressed (2.4MB uncompressed),
which should be larger than any pipe buffers...  Any idea why gunzip
can't handle this, or what I should I should be doing instead?

Thanks,
Bryan

Re: [gentoo-user] Tailing compressed build logs

[Mickaël Bucas]

Le mar. 7 mars 2023 à 05:36, Bryan Gardiner <bog@khumba.net> a écrit :
>
> Hi folks,
>
> How can I follow Portage's compressed build logs in real time as they
> are generated?
>
> I keep build logs and use FEATURES=compress-build-logs so that they
> don't get too large.  I can peek at how a build is going with zless on
> build.log.gz, which doesn't update (understandably), but I would
> really like to be able to watch a log with some "tail -f" equivalent.
> I get streaming output with
>
>     tail -c +1 -f build.log.gz | od -t x1
>
> but the following hangs with no output:
>
>     tail -c +1 -f build.log.gz | gunzip
>
> even with a build log that is 72KB compressed (2.4MB uncompressed),
> which should be larger than any pipe buffers...  Any idea why gunzip
> can't handle this, or what I should I should be doing instead?
>
> Thanks,
> Bryan
>
Hi

Reading the man page, "zless" is just a wrapper around "less".
You can check with:
$ file $(which zless)
/usr/bin/zless: POSIX shell script, ASCII text executable
$ less $(which zless)

So it should support the same options including typing "F" at the end
of a file to keep trying to read when the end of file is reached.

I made a small test, but it didn't work:
# Create a growing file
$ yes | nl | gzip > zless-test.gz &
# Try to follow at the end
$ zless zless-test.gz

With ">" to go to the end and "F" to continue, I didn't get the
expected behavior, it stood still at the point I was viewing.
I don't know if it's really a bug or if I made a mistake...
(Don't forget to stop the growing file :) )

Best regards

Mickaël Bucas

Re: [gentoo-user] Tailing compressed build logs

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1917 bytes --]

On Tuesday, 7 March 2023 07:52:01 GMT Mickaël Bucas wrote:
> Le mar. 7 mars 2023 à 05:36, Bryan Gardiner <bog@khumba.net> a écrit :
> > Hi folks,
> > 
> > How can I follow Portage's compressed build logs in real time as they
> > are generated?
> > 
> > I keep build logs and use FEATURES=compress-build-logs so that they
> > don't get too large.  I can peek at how a build is going with zless on
> > build.log.gz, which doesn't update (understandably), but I would
> > really like to be able to watch a log with some "tail -f" equivalent.
> > I get streaming output with
> > 
> >     tail -c +1 -f build.log.gz | od -t x1
> > 
> > but the following hangs with no output:
> >     tail -c +1 -f build.log.gz | gunzip
> > 
> > even with a build log that is 72KB compressed (2.4MB uncompressed),
> > which should be larger than any pipe buffers...  Any idea why gunzip
> > can't handle this, or what I should I should be doing instead?
> > 
> > Thanks,
> > Bryan
> 
> Hi
> 
> Reading the man page, "zless" is just a wrapper around "less".
> You can check with:
> $ file $(which zless)
> /usr/bin/zless: POSIX shell script, ASCII text executable
> $ less $(which zless)
> 
> So it should support the same options including typing "F" at the end
> of a file to keep trying to read when the end of file is reached.
> 
> I made a small test, but it didn't work:
> # Create a growing file
> $ yes | nl | gzip > zless-test.gz &
> # Try to follow at the end
> $ zless zless-test.gz
> 
> With ">" to go to the end and "F" to continue, I didn't get the
> expected behavior, it stood still at the point I was viewing.
> I don't know if it's really a bug or if I made a mistake...
> (Don't forget to stop the growing file :) )
> 
> Best regards
> 
> Mickaël Bucas

You could try:

tail -c +1 -f build.log.gz | gunzip | less 

I think it should work, but I haven't tried it.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Tailing compressed build logs

[Bryan Gardiner]

[-- Attachment #1: Type: text/plain, Size: 3138 bytes --]

Hi,

On Tue, 07 Mar 2023 09:01:34 +0000
Michael <confabulate@kintzios.com> wrote:

> On Tuesday, 7 March 2023 07:52:01 GMT Mickaël Bucas wrote:
> > Le mar. 7 mars 2023 à 05:36, Bryan Gardiner <bog@khumba.net> a
> > écrit :  
> > > Hi folks,
> > > 
> > > How can I follow Portage's compressed build logs in real time as
> > > they are generated?
> > > 
> > > ...
> > > 
> > >     tail -c +1 -f build.log.gz | od -t x1
> > > 
> > > but the following hangs with no output:
> > >     tail -c +1 -f build.log.gz | gunzip
> > 
> > Reading the man page, "zless" is just a wrapper around "less".

Neat, zless is a lot simpler than I thought.

> > So it should support the same options including typing "F" at the
> > end of a file to keep trying to read when the end of file is
> > reached.
> > 
> > I made a small test, but it didn't work:
> > # Create a growing file
> > $ yes | nl | gzip > zless-test.gz &
> > # Try to follow at the end
> > $ zless zless-test.gz
> > 
> > With ">" to go to the end and "F" to continue, I didn't get the
> > expected behavior, it stood still at the point I was viewing.
> > I don't know if it's really a bug or if I made a mistake...
> > (Don't forget to stop the growing file :) )

I suppose it makes sense that a simple "gunzip -c file.gz | less"
doesn't work, assuming zless is equivalent to that.  gunzip sees the
end of the file and exits rather than waiting for additional content.
Then less sees its pipe close.

So I think the "tail -c +1 -f build.log.gz" bit is needed, and appears
to work since od shows data appended every few seconds.

> You could try:
> 
> tail -c +1 -f build.log.gz | gunzip | less 
> 
> I think it should work, but I haven't tried it.

Thanks, but I suspect that less is still at the whim of gunzip here,
and gunzip isn't producing anything.

Maybe gunzip just has a massive output buffer.  If I do something a bit
more complicated:

    (N=0; while true; do
      (( N++ ))
      echo "$N" | gzip
      if (( N % 4000 == 0 )); then sleep 1; fi
    done) | gunzip | od -A d -t x1

then nothing is shown for 15 seconds or so, then od quickly dumps
51952 bytes of hex output, then another long pause before dumping
another 60480 bytes of hex.  If I try this with "pigz -cd" instead of
"gunzip" then output starts flowing after just ~400 bytes, but sadly
that still doesn't work for build.log.gz.

Anyway, I've had another thought, that since portage is flushing lines
to the gzipped log fairly frequently, it's probably not getting the
best compression ratio.  So I went and tested recompressing some logs,
and got a decent improvement:

- nodejs-18.9.1: build.log.gz is 112KB, recompressed it's 68.9KB
  (38.5% savings).

- qtwebengine-5.15.8_p20230112: build.log.gz is 1.44MB, recompressed
  it's 1.01MB (29.9% savings).

- gtk+-3.24.37: build.log.gz is 157KB, recompressed it's 120KB
  (23.6% savings).

So maybe I'm better off abandoning this feature, and setting something
up to compress build logs after the fact.  Then good ol' tail -f will
work just fine.

Thanks, cheers,
Bryan

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Tailing compressed build logs

[Andreas Stiasny]

On 07.03.23 05:35, Bryan Gardiner wrote:

> but the following hangs with no output:
> 
>      tail -c +1 -f build.log.gz | gunzip

I think you should either use gunzip -f --stdout instead of just gunzip 
or use zcat.


Andreas

Re: [gentoo-user] Tailing compressed build logs

[Bryan Gardiner]

On Wed, 8 Mar 2023 14:13:35 +0100
Andreas Stiasny <gentoo@lists.stiasny.priv.at> wrote:

> On 07.03.23 05:35, Bryan Gardiner wrote:
> 
> > but the following hangs with no output:
> > 
> >      tail -c +1 -f build.log.gz | gunzip  
> 
> I think you should either use gunzip -f --stdout instead of just
> gunzip or use zcat.
> 
> 
> Andreas
> 

Hi, thanks!  --force and --stdout don't seem to help in this case.

- Bryan

Re: [gentoo-user] Tailing compressed build logs

[Andreas Stiasny]

On 09.03.23 08:46, Bryan Gardiner wrote:

> Hi, thanks!  --force and --stdout don't seem to help in this case.

Sorry, I was wrong. The problem is not the actual use of gunzip but the 
fact that you are trying to decompress a file without reading it from 
the beginning. Usually this is not possible.

zcat build.log.gz | tail -n 30

would show you the last 30 lines but no new content because zcat or 
gunzip is not waiting for new input. You could try gztool

https://github.com/circulosmeos/gztool/

"gztool -T" should work like "tail -f".


Andreas

Re: [gentoo-user] Tailing compressed build logs

[Peter Humphrey]

On Thursday, 9 March 2023 14:31:32 GMT Andreas Stiasny wrote:

>You could try gztool
> 
> https://github.com/circulosmeos/gztool/
> 
> "gztool -T" should work like "tail -f".

That looks very handy. But:

Compiling it from the github source fails because I haven't found all the 
prerequisites.

Fetching a Debian compiled version and running deb2targz on it ends up with a 
.so file, which where to put?

I haven't found it in any Gentoo overlays.

How can I get hold of a version for Gentoo?

-- 
Regards,
Peter.

Re: [gentoo-user] Tailing compressed build logs

[Matt Connell]

On Thu, 2023-03-09 at 15:49 +0000, Peter Humphrey wrote:
> Fetching a Debian compiled version and running deb2targz on it ends up with a 
> .so file, which where to put?

app-arch/deb2targz exists.  Would probably satisfy the need.

Re: [gentoo-user] Tailing compressed build logs

[Dale]

Peter Humphrey wrote:
> On Thursday, 9 March 2023 14:31:32 GMT Andreas Stiasny wrote:
>
>> You could try gztool
>>
>> https://github.com/circulosmeos/gztool/
>>
>> "gztool -T" should work like "tail -f".
> That looks very handy. But:
>
> Compiling it from the github source fails because I haven't found all the 
> prerequisites.
>
> Fetching a Debian compiled version and running deb2targz on it ends up with a 
> .so file, which where to put?
>
> I haven't found it in any Gentoo overlays.
>
> How can I get hold of a version for Gentoo?
>


I found it in a overlay.  At least google and github page says it is
there. 

https://github.com/gentoo-mirror/slonko

It appears to be called gztool. 

Hope that helps.

Dale

:-)  :-) 

Re: [gentoo-user] Tailing compressed build logs

[Peter Humphrey]

On Thursday, 9 March 2023 16:01:16 GMT Dale wrote:
> Peter Humphrey wrote:
> > On Thursday, 9 March 2023 14:31:32 GMT Andreas Stiasny wrote:
> >> You could try gztool
> >> 
> >> https://github.com/circulosmeos/gztool/
> >> 
> >> "gztool -T" should work like "tail -f".
> > 
> > That looks very handy. But:
> > 
> > Compiling it from the github source fails because I haven't found all the
> > prerequisites.
> > 
> > Fetching a Debian compiled version and running deb2targz on it ends up
> > with a .so file, which where to put?
> > 
> > I haven't found it in any Gentoo overlays.
> > 
> > How can I get hold of a version for Gentoo?
> 
> I found it in a overlay.  At least google and github page says it is
> there. 
> 
> https://github.com/gentoo-mirror/slonko
> 
> It appears to be called gztool. 
> 
> Hope that helps.

It does - thanks!

-- 
Regards,
Peter.

Re: [gentoo-user] Tailing compressed build logs

[Bryan Gardiner]

On Thu, 9 Mar 2023 10:01:16 -0600
Dale <rdalek1967@gmail.com> wrote:

> Peter Humphrey wrote:
> > On Thursday, 9 March 2023 14:31:32 GMT Andreas Stiasny wrote:
> >  
> >> You could try gztool
> >>
> >> https://github.com/circulosmeos/gztool/
> >>
> >> "gztool -T" should work like "tail -f".  
> > That looks very handy. But:
> >
> > Compiling it from the github source fails because I haven't found
> > all the prerequisites.
> >
> > Fetching a Debian compiled version and running deb2targz on it ends
> > up with a .so file, which where to put?
> >
> > I haven't found it in any Gentoo overlays.
> >
> > How can I get hold of a version for Gentoo?
> 
> I found it in a overlay.  At least google and github page says it is
> there. 
> 
> https://github.com/gentoo-mirror/slonko
> 
> It appears to be called gztool. 
> 
> Hope that helps.

Ah, this is great.  "gztool -TW" does exactly what it should.  Nice
find, thanks a bunch!

- Bryan