[gentoo-dev] rfc: escape sequences in logs

[gentoo-dev] rfc: escape sequences in logs

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
> Dnia 2013-09-01, o godz. 16:49:34
> William Hubbs <williamh@gentoo.org> napisał(a):
> 
> > On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
> > > Dnia 2013-08-31, o godz. 11:26:30
> > > Ulrich Mueller <ulm@gentoo.org> napisał(a):
> > > 
> > > > >>>>> On Sat, 31 Aug 2013, Michał Górny wrote:
> > > > 
> > > > > And time for a small update. 
> > > > 
> > > > In git-r3_checkout:
> > > > 
> > > >             git --no-pager diff --color --stat \
> > > >                 ${old_commit_id}..${new_commit_id}
> > > > 
> > > > I'd rather omit the --color option, otherwise log files will contain
> > > > escape sequences.
> > > 
> > > I'd rather leave it. The diff is more for pretty-printing anyway, it
> > > shouldn't really matter in the logs.
> > 
> > Please don't. I also do not want escape sequences in log files.
> 
> Ok, '--color' removed. However, I think we should work something out to
> get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
> logs from escape sequences since many people actually benefit from them.

All,

I'm starting a new thread on this, because I think it might warrant some
discussion.

I can see why someone might want to use escape codes for color displays,
etc. However, imo, escape codes do not belong in log files.

mgorny says many people benefit from having escape codes in log
files, but I see no benefit from it, and I don't like going through
build.log because of them. If you load a build.log into an editor, the
escape sequences are basically trash characters that get in the way.

Another consideration is if someone puts messages from a build.log
directly in a bug and the messages contain escape codes, this will crash
things like pybugz because bugzilla doesn't filter out the escape
character.

Does anyone else have an opinion on this?

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: escape sequences in logs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2684 bytes --]

Dnia 2013-09-02, o godz. 14:21:52
William Hubbs <williamh@gentoo.org> napisał(a):

> On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
> > Dnia 2013-09-01, o godz. 16:49:34
> > William Hubbs <williamh@gentoo.org> napisał(a):
> > 
> > > On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
> > > > Dnia 2013-08-31, o godz. 11:26:30
> > > > Ulrich Mueller <ulm@gentoo.org> napisał(a):
> > > > 
> > > > > >>>>> On Sat, 31 Aug 2013, Michał Górny wrote:
> > > > > 
> > > > > > And time for a small update. 
> > > > > 
> > > > > In git-r3_checkout:
> > > > > 
> > > > >             git --no-pager diff --color --stat \
> > > > >                 ${old_commit_id}..${new_commit_id}
> > > > > 
> > > > > I'd rather omit the --color option, otherwise log files will contain
> > > > > escape sequences.
> > > > 
> > > > I'd rather leave it. The diff is more for pretty-printing anyway, it
> > > > shouldn't really matter in the logs.
> > > 
> > > Please don't. I also do not want escape sequences in log files.
> > 
> > Ok, '--color' removed. However, I think we should work something out to
> > get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
> > logs from escape sequences since many people actually benefit from them.
> 
> All,
> 
> I'm starting a new thread on this, because I think it might warrant some
> discussion.
> 
> I can see why someone might want to use escape codes for color displays,
> etc. However, imo, escape codes do not belong in log files.

Well, colors are not the only thing that uses escape sequences. They
are used pretty often for various kinds of progress reporting --
percentages, progress bars. Many tools simply don't implement any other
kind of progress reporting, so sometimes it's either escape sequences
or long waiting with no signs of life.

> mgorny says many people benefit from having escape codes in log
> files, but I see no benefit from it, and I don't like going through
> build.log because of them. If you load a build.log into an editor, the
> escape sequences are basically trash characters that get in the way.

I said people benefit from having them output during the build process.
Logs are a different thing, that's why I suggested having a feature
that would remove those from output when generating logs.

> Another consideration is if someone puts messages from a build.log
> directly in a bug and the messages contain escape codes, this will crash
> things like pybugz because bugzilla doesn't filter out the escape
> character.

That is a bug in pybugz and not an argument, you know.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] rfc: escape sequences in logs

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 3272 bytes --]

On Mon, Sep 02, 2013 at 09:41:28PM +0200, Michał Górny wrote:
> Dnia 2013-09-02, o godz. 14:21:52
> William Hubbs <williamh@gentoo.org> napisał(a):
> 
> > On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
> > > Dnia 2013-09-01, o godz. 16:49:34
> > > William Hubbs <williamh@gentoo.org> napisał(a):
> > > 
> > > > On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
> > > > > Dnia 2013-08-31, o godz. 11:26:30
> > > > > Ulrich Mueller <ulm@gentoo.org> napisał(a):
> > > > > 
> > > > > > >>>>> On Sat, 31 Aug 2013, Michał Górny wrote:
> > > > > > 
> > > > > > > And time for a small update. 
> > > > > > 
> > > > > > In git-r3_checkout:
> > > > > > 
> > > > > >             git --no-pager diff --color --stat \
> > > > > >                 ${old_commit_id}..${new_commit_id}
> > > > > > 
> > > > > > I'd rather omit the --color option, otherwise log files will contain
> > > > > > escape sequences.
> > > > > 
> > > > > I'd rather leave it. The diff is more for pretty-printing anyway, it
> > > > > shouldn't really matter in the logs.
> > > > 
> > > > Please don't. I also do not want escape sequences in log files.
> > > 
> > > Ok, '--color' removed. However, I think we should work something out to
> > > get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
> > > logs from escape sequences since many people actually benefit from them.
> > 
> > All,
> > 
> > I'm starting a new thread on this, because I think it might warrant some
> > discussion.
> > 
> > I can see why someone might want to use escape codes for color displays,
> > etc. However, imo, escape codes do not belong in log files.
> 
> Well, colors are not the only thing that uses escape sequences. They
> are used pretty often for various kinds of progress reporting --
> percentages, progress bars. Many tools simply don't implement any other
> kind of progress reporting, so sometimes it's either escape sequences
> or long waiting with no signs of life.
> 
> > mgorny says many people benefit from having escape codes in log
> > files, but I see no benefit from it, and I don't like going through
> > build.log because of them. If you load a build.log into an editor, the
> > escape sequences are basically trash characters that get in the way.
> 
> I said people benefit from having them output during the build process.
> Logs are a different thing, that's why I suggested having a feature
> that would remove those from output when generating logs.

There is a NOCOLOR variable you can set in make.conf, but that would
require users to set it, and it also affects the display.

How does portage write build.log?

> > Another consideration is if someone puts messages from a build.log
> > directly in a bug and the messages contain escape codes, this will crash
> > things like pybugz because bugzilla doesn't filter out the escape
> > character.
> 
> That is a bug in pybugz and not an argument, you know.

I said "things like pybugz".

Bugzilla allowing control characters in the xml is the issue. The python
xmlrpc library raises an exception for malformed xml because of it, so
there isn't much pybugz, or anything that uses that library, can do
about it.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: escape sequences in logs

[Ulrich Mueller]

>>>>> On Mon, 2 Sep 2013, William Hubbs wrote:

> I can see why someone might want to use escape codes for color
> displays, etc. However, imo, escape codes do not belong in log
> files.

> mgorny says many people benefit from having escape codes in log
> files, but I see no benefit from it, and I don't like going through
> build.log because of them. If you load a build.log into an editor,
> the escape sequences are basically trash characters that get in the
> way.

> Another consideration is if someone puts messages from a build.log
> directly in a bug and the messages contain escape codes, this will
> crash things like pybugz because bugzilla doesn't filter out the
> escape character.

> Does anyone else have an opinion on this?

Log files should be plain text without any embedded font attributes or
other formatting.

I'd consider any tool as broken if it outputs escape sequences when
the output doesn't go to a terminal. (Unless such output was
explicitly asked for.)

Ulrich

Re: [gentoo-dev] rfc: escape sequences in logs

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

On 3 September 2013 09:22, Ulrich Mueller <ulm@gentoo.org> wrote:

> I'd consider any tool as broken if it outputs escape sequences when
> the output doesn't go to a terminal. (Unless such output was
> explicitly asked for.)
>


However, what about when output is going to a terminal *and* a log file?

It seems smarter to output escape sequences in that case.

Additionally, although many people find escape sequences in log files
useless, I find them somewhat useful, for instance, if somebody uploads a
logfile with escape sequences in it, I can just run the file into "less -R"
or "less -r" depending on my mood and see the log file *Exactly* as the
user saw it.

I can see why you might not like it if you're reading it in a non-escape
aware editor, but I don't really use editors for reading log files, that
strikes me as doing something wrong.

And it is quite simple to remove escape sequences from log files if I
desire it.

using: app-text/ansifilter

 zcat
/var/log/portage/build/app-accessibility/at-spi2-core-2.6.3:20130825-143158.log.gz
| ansifilter

using perl and Term::ANSIColor ( standard issue with Perl itself )

zcat
/var/log/portage/build/app-accessibility/at-spi2-core-2.6.3:20130825-143158.log.gz
| perl -MTerm::ANSIColor=colorstrip -ple '$_ = colorstrip($_)'


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2041 bytes --]

Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2105 bytes --]

On Mon, 2 Sep 2013 14:21:52 -0500
William Hubbs <williamh@gentoo.org> wrote:

> I can see why someone might want to use escape codes for color
> displays, etc. However, imo, escape codes do not belong in log files.

They belong there so future display can remain colorful.

Why do they not belong there? What do people have to do who want them?

> mgorny says many people benefit from having escape codes in log
> files, but I see no benefit from it, and I don't like going through
> build.log because of them. If you load a build.log into an editor, the
> escape sequences are basically trash characters that get in the way.

They make some sections differ and therefore stand out; so, you get a
much faster impression of what you are looking at is what you want.

Why do you open them in an editor and not in a viewer?

> Does anyone else have an opinion on this?

There are definitely enough people that want them; so, why not have
them by default and strip them if you don't want them? The opposite,
adding escape codes where there are none; is a much harder thing to do.

Actually, I would love to see even more codes in build.log such that
they come more machine parseable; for instance, there is no indication
which process outputs a certain message, or whether the message
happened on stdout or stderr.

Such information is quite crucial, yet it is missing; it would be neat
if I could just grep the last stderr lines of the last process that's
not emerge or make, giving me exactly the gcc error I need to see. Once
I found that line, grepping context is easy.

Until then, I'm stuck with having to scroll up to that point; and while
I know -j1 helps with that, it would be crude to ignore non -j1 logs...

If you're trying to more efficiently parse logs; consider adding more
information instead, because dropping information does not really gain.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: escape sequences in logs

[Zac Medico]

On 09/02/2013 01:29 PM, William Hubbs wrote:
> On Mon, Sep 02, 2013 at 09:41:28PM +0200, Michał Górny wrote:
>> Dnia 2013-09-02, o godz. 14:21:52
>> William Hubbs <williamh@gentoo.org> napisał(a):
>>
>>> On Mon, Sep 02, 2013 at 01:33:19PM +0200, Michał Górny wrote:
>>>> Dnia 2013-09-01, o godz. 16:49:34
>>>> William Hubbs <williamh@gentoo.org> napisał(a):
>>>>
>>>>> On Sat, Aug 31, 2013 at 10:48:32PM +0200, Michał Górny wrote:
>>>>>> Dnia 2013-08-31, o godz. 11:26:30
>>>>>> Ulrich Mueller <ulm@gentoo.org> napisał(a):
>>>>>>
>>>>>>>>>>>> On Sat, 31 Aug 2013, Michał Górny wrote:
>>>>>>>
>>>>>>>> And time for a small update. 
>>>>>>>
>>>>>>> In git-r3_checkout:
>>>>>>>
>>>>>>>             git --no-pager diff --color --stat \
>>>>>>>                 ${old_commit_id}..${new_commit_id}
>>>>>>>
>>>>>>> I'd rather omit the --color option, otherwise log files will contain
>>>>>>> escape sequences.
>>>>>>
>>>>>> I'd rather leave it. The diff is more for pretty-printing anyway, it
>>>>>> shouldn't really matter in the logs.
>>>>>
>>>>> Please don't. I also do not want escape sequences in log files.
>>>>
>>>> Ok, '--color' removed. However, I think we should work something out to
>>>> get both parties satisfied. Maybe portage feature or a tool to 'uncruft'
>>>> logs from escape sequences since many people actually benefit from them.
>>>
>>> All,
>>>
>>> I'm starting a new thread on this, because I think it might warrant some
>>> discussion.
>>>
>>> I can see why someone might want to use escape codes for color displays,
>>> etc. However, imo, escape codes do not belong in log files.
>>
>> Well, colors are not the only thing that uses escape sequences. They
>> are used pretty often for various kinds of progress reporting --
>> percentages, progress bars. Many tools simply don't implement any other
>> kind of progress reporting, so sometimes it's either escape sequences
>> or long waiting with no signs of life.
>>
>>> mgorny says many people benefit from having escape codes in log
>>> files, but I see no benefit from it, and I don't like going through
>>> build.log because of them. If you load a build.log into an editor, the
>>> escape sequences are basically trash characters that get in the way.
>>
>> I said people benefit from having them output during the build process.
>> Logs are a different thing, that's why I suggested having a feature
>> that would remove those from output when generating logs.
> 
> There is a NOCOLOR variable you can set in make.conf, but that would
> require users to set it, and it also affects the display.
> 
> How does portage write build.log?

It uses its PipeLogger class to read the output from a pipe, and write
to the log (and stdout if appropriate):


http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=blob_plain;f=pym/portage/util/_async/PipeLogger.py;hb=HEAD

So yes, it would be possible to do translations on-the-fly. However, the
event loop that handles the logging is currently single threaded, and
might become a bottleneck if it has to do lots of output processing
(especially for larger values of --jobs).
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: escape sequences in logs

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On 09/02/2013 03:21 PM, William Hubbs wrote:
> All,
> 
> I'm starting a new thread on this, because I think it might warrant some
> discussion.
> 
> I can see why someone might want to use escape codes for color displays,
> etc. However, imo, escape codes do not belong in log files.
> 
> mgorny says many people benefit from having escape codes in log
> files, but I see no benefit from it, and I don't like going through
> build.log because of them. If you load a build.log into an editor, the
> escape sequences are basically trash characters that get in the way.

I find looking at logs with escape characters to be more pleasant than
looking at them without escape characters. I admit that it is annoying
to view them in a web browser where the escape characters are not
parsed, but that is easily resolved at the terminal.

Just my two cents.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] rfc: escape sequences in logs

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

On 3 September 2013 16:17, Richard Yao <ryao@gentoo.org> wrote:

> I admit that it is annoying
> to view them in a web browser where the escape characters are not
> parsed, but that is easily resolved at the terminal
>


You could plausibly also have a filter in bugzilla that detects escape
codes in the log, and adds a display link that routes to a page rendered
through `ansifilter -H` , similar to how we have a diff formatter for
patches.

I'm not sure the exact layers of glue required to make that viable in
bugzilla, but it doesn't seem impossible.


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[-- Attachment #2: Type: text/html, Size: 1285 bytes --]

Re: [gentoo-dev] rfc: escape sequences in logs

[Tobias Klausmann]

Hi! 


My two cents for viewing-logs-in-vim (which is a use case for
me): 

$ eix ansiesc
* app-vim/ansiesc
     Available versions:  (~)12
     Homepage:            http://www.vim.org/scripts/script.php?script_id=302
     Description:         vim plugin: ansi escape sequences concealed, but highlighted as specified

Regards,
Tobias

Re: [gentoo-dev] rfc: escape sequences in logs

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Mon, Sep 2, 2013 at 8:21 PM, William Hubbs <williamh@gentoo.org> wrote:

>
> mgorny says many people benefit from having escape codes in log
> files, but I see no benefit from it, and I don't like going through
> build.log because of them. If you load a build.log into an editor, the
> escape sequences are basically trash characters that get in the way.


You do know that many eclasses (including the Ruby ones) have a setting
that enables/disable color output? I keep it enabled on my system but I've
always been disabling it for tinderboxes..


Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

[-- Attachment #2: Type: text/html, Size: 1136 bytes --]

Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Ulrich Mueller]

>>>>> On Tue, 3 Sep 2013, Tom Wijsman wrote:

> On Mon, 2 Sep 2013 14:21:52 -0500
> William Hubbs <williamh@gentoo.org> wrote:

>> I can see why someone might want to use escape codes for color
>> displays, etc. However, imo, escape codes do not belong in log files.

> They belong there so future display can remain colorful.

> Why do they not belong there? What do people have to do who want
> them?

Escape sequences have been designed for communication with peripheral
devices, not for markup or as a storage format.

Also "future colorful display" generally won't be portabe because
escape sequences depend on the setting of the TERM variable. (And
again, software that emits them with TERM=dumb or TERM unset is
broken.)

Ulrich

Re: [gentoo-dev] rfc: escape sequences in logs

[Ulrich Mueller]

>>>>> On Tue, 3 Sep 2013, Tobias Klausmann wrote:

> My two cents for viewing-logs-in-vim (which is a use case for me):

> $ eix ansiesc
> * app-vim/ansiesc
>      Available versions:  (~)12
>      Homepage:            http://www.vim.org/scripts/script.php?script_id=302
>      Description:         vim plugin: ansi escape sequences concealed, but highlighted as specified

Well, Emacs has this (M-x list-packages):

  ansi-color         3.4.2        built-in   translate ANSI escape sequences into faces

It doesn't support such broken concepts like escape sequences in files
out of the box, so some manual configuration is needed for it.

Ulrich

[gentoo-dev] Re: rfc: escape sequences in logs

[Duncan]

Kent Fredric posted on Tue, 03 Sep 2013 11:59:13 +1200 as excerpted:

> And it is quite simple to remove escape sequences from log files if I
> desire it.
> 
> using: app-text/ansifilter
> 
>  zcat
> /var/log/portage/build/app-accessibility/at-spi2-
core-2.6.3:20130825-143158.log.gz
> | ansifilter
> 
> using perl and Term::ANSIColor ( standard issue with Perl itself )
> 
> zcat
> /var/log/portage/build/app-accessibility/at-spi2-
core-2.6.3:20130825-143158.log.gz
> | perl -MTerm::ANSIColor=colorstrip -ple '$_ = colorstrip($_)'

This has long bothered me, but not enough to be worth investigating 
myself...

Zac mentioned in a different subthread that portage's logger is single-
threaded ATM, and could become a bottleneck if it had to do too much 
output processing.

How feasible might it be to make something like this a (possibly optional 
but presumably enabled by default) portage dep, and set portage's default 
logging to use it when writing the log file (as opposed to the screen 
output)?  Presumably that would pipe to another app to do the actual 
logfile filtering and writing, so the threading issue would disappear.

Actually, presuming there's already a python implementation to parallel 
that perl one, it's possible the only thing that would need changed would 
be portage's default logging command.  Pythonistas?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: rfc: escape sequences in logs

[Ulrich Mueller]

>>>>> On Tue, 3 Sep 2013, Duncan  wrote:

[Please check your e-mail agent. The "References:" header in your
posting was broken.]

> Zac mentioned in a different subthread that portage's logger is
> single-threaded ATM, and could become a bottleneck if it had to do
> too much output processing.

> How feasible might it be to make something like this a (possibly
> optional but presumably enabled by default) portage dep, and set
> portage's default logging to use it when writing the log file (as
> opposed to the screen output)? Presumably that would pipe to another
> app to do the actual logfile filtering and writing, so the threading
> issue would disappear.

> Actually, presuming there's already a python implementation to
> parallel that perl one, it's possible the only thing that would need
> changed would be portage's default logging command. Pythonistas?

Isn't it just a matter of regexp matching? For example,
Emacs' ansi-color.el discards any string that matches
<ESC>\[\([ABCDsuK]\|2J\|=[0-9]+[hI]\|[0-9;]*[Hfm]\)
and it seems to work in the common cases.

Ulrich

[gentoo-dev] Re: rfc: escape sequences in logs

[Duncan]

Ulrich Mueller posted on Tue, 03 Sep 2013 15:01:28 +0200 as excerpted:

> [Please check your e-mail agent. The "References:" header in your
> posting was broken.]

Thanks.  Upstream is aware of the bug but hasn't patched it yet.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Tue, Sep 03, 2013 at 10:25:19AM +0200, Ulrich Mueller wrote:
> >>>>> On Tue, 3 Sep 2013, Tom Wijsman wrote:
> 
> > On Mon, 2 Sep 2013 14:21:52 -0500
> > William Hubbs <williamh@gentoo.org> wrote:
> 
> >> I can see why someone might want to use escape codes for color
> >> displays, etc. However, imo, escape codes do not belong in log files.
> 
> > They belong there so future display can remain colorful.
> 
> > Why do they not belong there? What do people have to do who want
> > them?
> 
> Escape sequences have been designed for communication with peripheral
> devices, not for markup or as a storage format.
> 
> Also "future colorful display" generally won't be portabe because
> escape sequences depend on the setting of the TERM variable. (And
> again, software that emits them with TERM=dumb or TERM unset is
> broken.)

Ulrich has summed this up well here. The bottom line is that escape
sequences are for communicating with the user's peripheral devices.
Since yours may not be the same as the user's who created the log, you
don't even know that yours will respond the same way.

I don't care what goes to the user's terminal, but these escape
sequences do not belong in log files.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Walter Dnes]

On Tue, Sep 03, 2013 at 10:25:19AM +0200, Ulrich Mueller wrote

> Escape sequences have been designed for communication with peripheral
> devices, not for markup or as a storage format.
> 
> Also "future colorful display" generally won't be portabe because
> escape sequences depend on the setting of the TERM variable. (And
> again, software that emits them with TERM=dumb or TERM unset is
> broken.)

  Similar to...

USE="foo bar" emerge blah blah blah

...can the average user do something like...

TERM="dumb" emerge blah blah blah

  I don't believe very many users or admins babysit an entire 2 hour
"emerge --deep --update @world" session, hitting {CTRL-S} when some
colour pops up.  In my case, I examine /var/log/portage/elog after the
emerge finishes, successfully or unsuccessfully.  And I don't use an
"editor" to view log files.  I use mc (Midnight Commander) which views
plain text, but doesn't decode ANSI.  A nice feature of mc is that I can
sort the file list by various options.  When looking at emerge output in
/var/log/portage/elog, I want to sort by file-modify-time, so I can
easily see which files were output in the most recent emerge run.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Tue, 3 Sep 2013 14:03:14 -0500
William Hubbs <williamh@gentoo.org> wrote:

> On Tue, Sep 03, 2013 at 10:25:19AM +0200, Ulrich Mueller wrote:
> > >>>>> On Tue, 3 Sep 2013, Tom Wijsman wrote:
> > 
> > > On Mon, 2 Sep 2013 14:21:52 -0500
> > > William Hubbs <williamh@gentoo.org> wrote:
> > 
> > >> I can see why someone might want to use escape codes for color
> > >> displays, etc. However, imo, escape codes do not belong in log
> > >> files.
> > 
> > > They belong there so future display can remain colorful.
> > 
> > > Why do they not belong there? What do people have to do who want
> > > them?
> > 
> > Escape sequences have been designed for communication with
> > peripheral devices, not for markup or as a storage format.
> > 
> > Also "future colorful display" generally won't be portabe because
> > escape sequences depend on the setting of the TERM variable. (And
> > again, software that emits them with TERM=dumb or TERM unset is
> > broken.)
> 
> Ulrich has summed this up well here. The bottom line is that escape
> sequences are for communicating with the user's peripheral devices.

Nice summary. But, it is also communicating with my peripheral device.

> Since yours may not be the same as the user's who created the log, you
> don't even know that yours will respond the same way.

Sounds like something we need to standardize. Although, from the many
build logs I have came across; it does respond fairly well in most of
the cases, I am yet to come across a build log that displays bad.

And for that occasional build log, stripping isn't a hard thing to do.

> I don't care what goes to the user's terminal, but these escape
> sequences do not belong in log files.

And then I asked the questions that I'd like to see answered:

Why do they not belong there? What do people have to do who want them?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

On Tue, 3 Sep 2013 15:43:44 -0400
"Walter Dnes" <waltdnes@waltdnes.org> wrote:

>   Similar to...
> 
> USE="foo bar" emerge blah blah blah
> 
> ...can the average user do something like...
> 
> TERM="dumb" emerge blah blah blah

That sounds like a very rare occasion, from all the bugs I have dealt
with I have not seen any dumb terminal escape codes so far. And if
there would be one, it wouldn't be hard to strip them for that one...
 
> I don't believe very many users or admins babysit an entire 2 hour
> "emerge --deep --update @world" session, hitting {CTRL-S} when some
> colour pops up.

That is not what this is about, this is about having escape sequences
in build logs obtained from Bugzilla; because, they aid in skimming
through logs (until we implement the feature I asked for in subject).

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Alan McKinnon]

On 03/09/2013 22:11, Tom Wijsman wrote:
> On Tue, 3 Sep 2013 14:03:14 -0500
> William Hubbs <williamh@gentoo.org> wrote:
> 
>> On Tue, Sep 03, 2013 at 10:25:19AM +0200, Ulrich Mueller wrote:
>>>>>>>> On Tue, 3 Sep 2013, Tom Wijsman wrote:
>>>
>>>> On Mon, 2 Sep 2013 14:21:52 -0500
>>>> William Hubbs <williamh@gentoo.org> wrote:
>>>
>>>>> I can see why someone might want to use escape codes for color
>>>>> displays, etc. However, imo, escape codes do not belong in log
>>>>> files.
>>>
>>>> They belong there so future display can remain colorful.
>>>
>>>> Why do they not belong there? What do people have to do who want
>>>> them?
>>>
>>> Escape sequences have been designed for communication with
>>> peripheral devices, not for markup or as a storage format.
>>>
>>> Also "future colorful display" generally won't be portabe because
>>> escape sequences depend on the setting of the TERM variable. (And
>>> again, software that emits them with TERM=dumb or TERM unset is
>>> broken.)
>>
>> Ulrich has summed this up well here. The bottom line is that escape
>> sequences are for communicating with the user's peripheral devices.
> 
> Nice summary. But, it is also communicating with my peripheral device.
> 
>> Since yours may not be the same as the user's who created the log, you
>> don't even know that yours will respond the same way.
> 
> Sounds like something we need to standardize. Although, from the many
> build logs I have came across; it does respond fairly well in most of
> the cases, I am yet to come across a build log that displays bad.
> 
> And for that occasional build log, stripping isn't a hard thing to do.
> 
>> I don't care what goes to the user's terminal, but these escape
>> sequences do not belong in log files.
> 
> And then I asked the questions that I'd like to see answered:
> 
> Why do they not belong there? What do people have to do who want them?
> 

escape sequences in logs (any kind of logs) are basically noise, they
make search and grep hard to use. They also make the log impossible to
read properly if your terminal type is not the same as what was in
effect when the log was created. And they are essentially candy. A log
without escapes that you wish had them is still usable. A log with
escapes you wish were absent is impossible to use sanely.

The solution is obvious - default to writing plain text to log files and
give the user an option to enable escapes in the log if {s,}he chooses
to have it. This does mean you can't use tricks with tee.

I *do* like colorized text on my terminal, but I do believe we ought to
keep defaults sane - the minimum that could possibly work. Everything
extra should be optional

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Magnus Granberg]

tisdag 03 september 2013 22.41.14 skrev  Alan McKinnon:
....
> I *do* like colorized text on my terminal, but I do believe we ought to
> keep defaults sane - the minimum that could possibly work. Everything
> extra should be optional

What about NOCOLOR="false" in make.conf see man make.conf for more info.
/Magnus

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Rich Freeman]

On Tue, Sep 3, 2013 at 4:41 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> The solution is obvious - default to writing plain text to log files and
> give the user an option to enable escapes in the log if {s,}he chooses
> to have it. This does mean you can't use tricks with tee.

Not sure it is so obvious.

Log files are about capturing information.  Escapes are about the
presentation of information - a reporting feature not unlike
pagination/etc.  It wouldn't make sense to embed page numbers in a log
file - if they are desired they should be added at the time the
information is presented, just like escape sequences.

It seems to me that the cleaner situation would be to capture
information in the logs, and use a pretty-printer of some kind to make
it look nice.  Terminate output should be made to look nice when
directed at a terminal.

Of course, the ultimate result of the whole logs-are-information
concept is something like the systemd journal, but I won't go there.
I'm not sure that would really help here - the challenge is more about
the actual content and not the metadata.

Rich

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Alan McKinnon]

On 03/09/2013 23:00, Magnus Granberg wrote:
> tisdag 03 september 2013 22.41.14 skrev  Alan McKinnon:
> ....
>> I *do* like colorized text on my terminal, but I do believe we ought to
>> keep defaults sane - the minimum that could possibly work. Everything
>> extra should be optional
> 
> What about NOCOLOR="false" in make.conf see man make.conf for more info.
> /Magnus


That affects terminal output too.

It does not seem desirable to force terminal and logs to always have the
same setting. To my mind the sane approach would be individual config
settings for both with sane defaults.

But, and here's the catch, that then means one can't simply tee output
to two places. There would have to be two output threads. This seems
like a LOT of work to implement


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

Dnia 2013-09-03, o godz. 17:03:39
Rich Freeman <rich0@gentoo.org> napisał(a):

> On Tue, Sep 3, 2013 at 4:41 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> > The solution is obvious - default to writing plain text to log files and
> > give the user an option to enable escapes in the log if {s,}he chooses
> > to have it. This does mean you can't use tricks with tee.
> 
> Not sure it is so obvious.
> 
> Log files are about capturing information.  Escapes are about the
> presentation of information - a reporting feature not unlike
> pagination/etc.  It wouldn't make sense to embed page numbers in a log
> file - if they are desired they should be added at the time the
> information is presented, just like escape sequences.
> 
> It seems to me that the cleaner situation would be to capture
> information in the logs, and use a pretty-printer of some kind to make
> it look nice.  Terminate output should be made to look nice when
> directed at a terminal.

How would you handle progress reporting with this? Something like
'capture one thousand lines of updated percentages and merge them with
a magical pretty printer'? I don't see real gain compared to what we
have now.

Except that now escape sequences cause that thousand lines to be just
one ultimately long line with escapes in the editor.

You can't strip logs out of information and then magically inject it
back without having it somewhere. It just can't work.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2465 bytes --]

On Tue, 03 Sep 2013 22:41:14 +0200
Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> escape sequences in logs (any kind of logs) are basically noise, they
> make search and grep hard to use.

But then why not implement matters that actually make search and grep
easier to use, see the new subject for example. Currently the logs
aren't search and grep compatible because you have no indication where
the last error is and which process has output that; and changing
escape sequences won't do anything about that, this isn't an argument.

> They also make the log impossible to
> read properly if your terminal type is not the same as what was in
> effect when the log was created.

Haven't experienced this, can you give me two attachments from
Bugzilla where at least one of both will display incorrectly for me?

It's easy to say, but I bet it is extremely hard to find; if I can't
find them, they must be very rare. Can you find them?

> And they are essentially candy. A log
> without escapes that you wish had them is still usable.

Since I have to deal with logs a lot, I'd rather work with the most
usable logs; until subject is implemented, escape codes do help me.

> A log with escapes you wish were absent is impossible to use sanely.

Then why do I never have problems with this? Why not strip them?

> The solution is obvious - default to writing plain text to log files
> and give the user an option to enable escapes in the log if {s,}he
> chooses to have it. This does mean you can't use tricks with tee.

What are you trying to solve here? This is not about the users; from
their perspective, I can only hope that the functionality will stay the
same as they don't have a problem with it, some maintainers do...
 
> I *do* like colorized text on my terminal, but I do believe we ought
> to keep defaults sane - the minimum that could possibly work.
> Everything extra should be optional

Sorry, but I'd rather get the most out of build logs; by being
restricted to minimum output, it takes more time to find the relevant
details from the build log.

If you're trying to more efficiently parse logs; consider adding more
information instead, because dropping information does not really gain.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Rich Freeman]

On Tue, Sep 3, 2013 at 5:12 PM, Michał Górny <mgorny@gentoo.org> wrote:
> How would you handle progress reporting with this? Something like
> 'capture one thousand lines of updated percentages and merge them with
> a magical pretty printer'? I don't see real gain compared to what we
> have now.

There is no value of sending incremental progress reporting to the log
in the first place.

However, as Alan already pointed out, implementing this isn't trivial.
 You could have a variation on tee that just strips out the junk, but
it isn't really the same as having fit-for-purpose output streams,
perhaps taking as input a highly structured raw log format like xml.
You'll never get that from a diverse set of build systems.

Rich

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Alan McKinnon]

On 03/09/2013 23:03, Rich Freeman wrote:
> On Tue, Sep 3, 2013 at 4:41 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> The solution is obvious - default to writing plain text to log files and
>> give the user an option to enable escapes in the log if {s,}he chooses
>> to have it. This does mean you can't use tricks with tee.
> 
> Not sure it is so obvious.
> 
> Log files are about capturing information.  Escapes are about the
> presentation of information - a reporting feature not unlike
> pagination/etc.  It wouldn't make sense to embed page numbers in a log
> file - if they are desired they should be added at the time the
> information is presented, just like escape sequences.
> 
> It seems to me that the cleaner situation would be to capture
> information in the logs, and use a pretty-printer of some kind to make
> it look nice.  Terminate output should be made to look nice when
> directed at a terminal.

This implies that the log can only then be viewed with a pretty-printer.

I wouldn't want to be the maintainer that has to deal with outraged
users if that becomes the case.

> Of course, the ultimate result of the whole logs-are-information
> concept is something like the systemd journal, but I won't go there.
> I'm not sure that would really help here - the challenge is more about
> the actual content and not the metadata.

So do what we've always done in Unix-land: leave the decision up to the
user. Build logs can always be regenerated (run emerge again) if the
ANSI sequences truly are vital whilst troubleshooting a specific log.
And we already do this for localized logs if the dev can't make sense of
non-English messages. It only becomes a problem if the log is for one of
that small number of loooooong builds (libreoffice, kdelibs etc)

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

On Tue, 3 Sep 2013 17:03:39 -0400
Rich Freeman <rich0@gentoo.org> wrote:

> Log files are about capturing information.  Escapes are about the
> presentation of information - a reporting feature not unlike
> pagination/etc.  It wouldn't make sense to embed page numbers in a log
> file - if they are desired they should be added at the time the
> information is presented, just like escape sequences.

Exactly, and the only way to add them when you present them is by
adding more information to the build log (eg. see subject for a start)
such that at presentation time a viewer can use this information to
color it; that way, you don't need the escape sequences anymore,
because you've made the build log more useful.

Or in other words, that allows people to search and filter it much more
efficiently; why try to match what might be an error if we can get the
stderr output of the last process instead?

> It seems to me that the cleaner situation would be to capture
> information in the logs, and use a pretty-printer of some kind to make
> it look nice.

That is the idea I try to from with this new subject; either I would
like to see more information that allows me to more efficiently deal
with the build log, if not stick with the escape sequences if people
object; but if we end up with neither of both it is going to be a step
backward that bothers all the people who rely on this feature.

> Terminate output should be made to look nice when
> directed at a terminal.

+1 Please don't make us go back to scrolling through black and white.
 
> Of course, the ultimate result of the whole logs-are-information
> concept is something like the systemd journal, but I won't go there.

It's a prime example; though, we probably might want to write our own
format and tools because we have quite a different goal.

> I'm not sure that would really help here - the challenge is more about
> the actual content and not the metadata.

It really is about both, not just content; reconsider metadata, that's
the key to log parsing and from there on we get more efficient searching
(last error lines), indexing (tinderbox) and so on ...

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

On Tue, 03 Sep 2013 23:10:38 +0200
Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> On 03/09/2013 23:00, Magnus Granberg wrote:
> > tisdag 03 september 2013 22.41.14 skrev  Alan McKinnon:
> > ....
> >> I *do* like colorized text on my terminal, but I do believe we
> >> ought to keep defaults sane - the minimum that could possibly
> >> work. Everything extra should be optional
> > 
> > What about NOCOLOR="false" in make.conf see man make.conf for more
> > info. /Magnus
> 
> That affects terminal output too.
> 
> It does not seem desirable to force terminal and logs to always have
> the same setting. To my mind the sane approach would be individual
> config settings for both with sane defaults.

This discussion should not be about our users; we should not change user
behavior or introduce unnecessary config settings based on a discussion
that intends to deal with the build logs that we download from Bugzilla.

> But, and here's the catch, that then means one can't simply tee output
> to two places. There would have to be two output threads. This seems
> like a LOT of work to implement

Yes, I believe Zac stated that in another sub thread.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1896 bytes --]

On Tue, 3 Sep 2013 17:17:49 -0400
Rich Freeman <rich0@gentoo.org> wrote:

> On Tue, Sep 3, 2013 at 5:12 PM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > How would you handle progress reporting with this? Something like
> > 'capture one thousand lines of updated percentages and merge them
> > with a magical pretty printer'? I don't see real gain compared to
> > what we have now.
> 
> There is no value of sending incremental progress reporting to the log
> in the first place.
>
> However, as Alan already pointed out, implementing this isn't trivial.
>  You could have a variation on tee that just strips out the junk, but
> it isn't really the same as having fit-for-purpose output streams,
> perhaps taking as input a highly structured raw log format like xml.
> You'll never get that from a diverse set of build systems.

Starting lines with something like STDERR:NAME:PID: would be a good
start; xml would indeed be overkill, let's keep it a textual file with
a minimal amount of currently missing information we need to add.

I'm not asking for magically discovering missing details; what I am
asking for, is that details that are known but not currently inserted in
the build log be made available in the build log so we can use them.

Currently, there is no indication which process outputs what on which
stream; so, this often leaves us wonder "is this an actual error" and
"which process did output this" (for less common build utilities).

On top of that, inserting lines for processes that return a non-zero
code would also be neat to have; because then you can obtain which
process fails (NAME:PID pair) and output its last stderr lines.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

On Tue, 03 Sep 2013 23:22:21 +0200
Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> So do what we've always done in Unix-land: leave the decision up to
> the user. Build logs can always be regenerated (run emerge again) if
> the ANSI sequences truly are vital whilst troubleshooting a specific
> log. And we already do this for localized logs if the dev can't make
> sense of non-English messages. It only becomes a problem if the log
> is for one of that small number of loooooong builds (libreoffice,
> kdelibs etc)

+1 I am still not convinced we are experiencing an actual practical
problem for the majority of the build logs that are attached; we've
been doing this for years, why is it so suddenly considered a problem?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Rich Freeman]

On Tue, Sep 3, 2013 at 5:22 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 03/09/2013 23:03, Rich Freeman wrote:
>> It seems to me that the cleaner situation would be to capture
>> information in the logs, and use a pretty-printer of some kind to make
>> it look nice.  Terminate output should be made to look nice when
>> directed at a terminal.
>
> This implies that the log can only then be viewed with a pretty-printer.

The log would just contain the information - you could view it with
cat.  If you wanted to view it with colorized escape codes then you'd
use the pretty-printer.  The pretty-printer wouldn't strip out escape
codes - it would add them.  Of course, this means that the printer
would need to be able to parse the log output.

The alternative is something like journal where you load up the log
with metadata and have to use a formatting program to display it.
Arguably that is what directing escape codes to a log does - it just
isn't fielded like journal, but also somewhat easier to view without
special software.

I do agree that none of this seems terribly practical.  I think the
furthest I'd take this is just optionally stripping escape codes when
dumping to a log (tee on steroids), and leave it at that.

Rich

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Walter Dnes]

On Tue, Sep 03, 2013 at 10:15:39PM +0200, Tom Wijsman wrote

> That is not what this is about, this is about having escape sequences
> in build logs obtained from Bugzilla; because, they aid in skimming
> through logs (until we implement the feature I asked for in subject).

  "The road to binary syslog files is paved with good intentions", or
something like that.  Question... why does it have to be escape
sequences?  Can't it be readable plain text?  E.g. something like...

//STDERR.OUT.START
foo
bar
blah blah blah
//STDERR.OUT.END

  This would be easy to grep and/or parse in bash.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Walter Dnes]

On Tue, Sep 03, 2013 at 11:44:45PM +0200, Tom Wijsman wrote

> +1 I am still not convinced we are experiencing an actual practical
> problem for the majority of the build logs that are attached; we've
> been doing this for years, why is it so suddenly considered a problem?

  As I pointed out in a message to the previous thread...

==========================================================================
WARN: prepare
It seems that you need to set USE_PYTHON to make sure that legacy
packages will be built with respect to PYTHON_TARGETS correctly:

        USE_PYTHON='.[35;1m2.7.[0m'

==========================================================================

  See https://bugs.gentoo.org/show_bug.cgi?id=463954

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2627 bytes --]

On 4 September 2013 08:11, Tom Wijsman <TomWij@gentoo.org> wrote:

> And then I asked the questions that I'd like to see answered:
>
>
> Why do they not belong there? What do people have to do who want them?
>


If anyone needs a poster child for the sort of escape sequence outputs that
most definitely are of no value to somebody reading the log after the fact,
the output from vims' test suite is a good example.

Granted, it uses a very wide variety of terminal escape codes, which
include, but are not limited to, window resizing control characters.

I think given that context, it may be sane to restrict log control
characters to a specific subset of control characters, specifically basic
colour/highlighting control characters, which are at least somewhat
standardized and not too device specific ( not all devices support all
colour control characters, or bold/italic/underline control characters, but
the negative side effect of not supporting a given control character in
this case is minimal )

You could perhaps have an ENV var that controls the manner in which log
files are written, say:

PORTAGE_LOG_ESCAPES="all"  # no stripping of control characters
PORTAGE_LOG_ESCAPES="color" # strips all but color control characters
PORTAGE_LOG_ESCAPES="color style" # strips all but
color/bold/italic/underline/reverse/conceal control characters
PORTAGE_LOG_ESCAPES="none" # strip all control characters

Somewhere in these control modes somebody needs to define how "\r" and "\b"
are treated, simply removing those characters could conceivably make log
files worse, not better, because it would reintroduce messages/characters
that were not seen by the end user.

Perhaps something like

PORTAGE_LOG_ESCAPES="erasures:strip" # remove erasure characters
PORTAGE_LOG_ESCAPES="erasures:apply" # Apply the logical effect of
erasures, deleting characters from the output log to mimic how it would be
displayed
PORTAGE_LOG_ESCAPES="erasures:keep" # keep \r and \b

Either way, if you were to introduce such a variable, you could have a
standard defined value that Gentoo agree upon, which I'd imagine might be
"none erasures:apply"  or something like that, and end users can turn them
back on if they want, with a caveat that users should apply a standard
stripping to these logs before submitting them to bugzilla, with a tool
that comes standard with portage to facilitate stripping logs for
submission to bugzilla.

This gives users the control they want, the features they want if they need
them, and still gives gentoo staff an easier time if it is determined
escape codes are not useful for bug reports.






-- 
Kent

[-- Attachment #2: Type: text/html, Size: 3791 bytes --]

[gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Duncan]

Tom Wijsman posted on Tue, 03 Sep 2013 23:16:11 +0200 as excerpted:

>  Currently the logs aren't
> search and grep compatible because you have no indication where the last
> error is and which process has output that

Quite apart from the ansi-color discussion, I've had reasonable luck 
simply searching/grepping for "error" here.  Sure, a lot of packages 
build *error* files of some sort and they're normally false-positives, 
but it does cut down the search space considerably, and there's usually 
only a couple such false-positives to worry about per-package.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

Dnia 2013-09-03, o godz. 18:57:12
"Walter Dnes" <waltdnes@waltdnes.org> napisał(a):

> On Tue, Sep 03, 2013 at 10:15:39PM +0200, Tom Wijsman wrote
> 
> > That is not what this is about, this is about having escape sequences
> > in build logs obtained from Bugzilla; because, they aid in skimming
> > through logs (until we implement the feature I asked for in subject).
> 
>   "The road to binary syslog files is paved with good intentions", or
> something like that.  Question... why does it have to be escape
> sequences?  Can't it be readable plain text?  E.g. something like...
> 
> //STDERR.OUT.START
> foo
> bar
> blah blah blah
> //STDERR.OUT.END
> 
>   This would be easy to grep and/or parse in bash.

Especially if one is interspersed with the other. I suggest trying
first, then suggesting it to others.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Tue, 3 Sep 2013 21:43:24 -0400
"Walter Dnes" <waltdnes@waltdnes.org> wrote:

> On Tue, Sep 03, 2013 at 11:44:45PM +0200, Tom Wijsman wrote
> 
> > +1 I am still not convinced we are experiencing an actual practical
> > problem for the majority of the build logs that are attached; we've
> > been doing this for years, why is it so suddenly considered a
> > problem?
> 
>   As I pointed out in a message to the previous thread...
>
> ==========================================================================
> WARN: prepare
> It seems that you need to set USE_PYTHON to make sure that legacy
> packages will be built with respect to PYTHON_TARGETS correctly:
> 
>         USE_PYTHON='.[35;1m2.7.[0m'
> 
> ==========================================================================
> 
>   See https://bugs.gentoo.org/show_bug.cgi?id=463954 

That's an easy one, it is clearly 2.7 surrounded by escape codes; but
that is not something the maintainer that downloads the build.log from
Bugzilla needs to know. I don't really see this as a counter example.

The user won't see this at the end of the output or in elogv; and even
when catting it to his TERM (which is the original one almost always)
it will also show colored. So, I don't see the problem here.

But let me restate, this discussion is _not_ about our users; we are
discussing the build logs as downloaded from Bugzilla, users _already_
have the means to disable (in make.conf) or strip it (we discussed many
examples) on their own if they don't want to use an appropriate viewer.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

On Wed, 4 Sep 2013 06:25:14 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> Tom Wijsman posted on Tue, 03 Sep 2013 23:16:11 +0200 as excerpted:
> 
> >  Currently the logs aren't
> > search and grep compatible because you have no indication where the
> > last error is and which process has output that
> 
> Quite apart from the ansi-color discussion, I've had reasonable luck 
> simply searching/grepping for "error" here.  Sure, a lot of packages 
> build *error* files of some sort and they're normally
> false-positives, but it does cut down the search space considerably,
> and there's usually only a couple such false-positives to worry about
> per-package.

When you have to go through tenths to hundreds of those on a daily
base, those "couple such false-positives" add up a lot; considering
that stderr indication will get the exact line you actually need.

Directly seeing the error (or even have it on my clipboard, 'cause I am
a bug wrangler whom places it in the summary) is much faster than
having to scroll through the output looking for the right line (and
still having to select and copy it).

In terms of what I need to process, we talk about changing O(n) to O(1).

Oh, then you should know, it's not always "error" it says; other times,
for example when failing tests, it says "fail"; and when you run some
other generation tool (doc, ...) you see something different, sometimes
it is an exception you get to see. Those are all easy to exactly match
with [see subject], but not at all exact with our current logs.

Rather than that we bike shed over something so unimportant as escape
codes; I'd rather see our build logs improve with necessary information,
in a way we can more efficiently deal with them than having to do the
matching completely manually ourselves.

It is in our philosophy to "to design tools and systems that allow a
user to do that work as pleasantly and efficiently as possible, as they
see fit." (user is in this case Gentoo Developer) so I don't see why we
would keep dealing with build log unpleasant an inefficient.

http://www.gentoo.org/main/en/philosophy.xml

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Wed, 4 Sep 2013 18:03:14 +1200
Kent Fredric <kentfredric@gmail.com> wrote:

> On 4 September 2013 08:11, Tom Wijsman <TomWij@gentoo.org> wrote:
> 
> > And then I asked the questions that I'd like to see answered:
> >
> >
> > Why do they not belong there? What do people have to do who want
> > them?
> >
> 
> If anyone needs a poster child for the sort of escape sequence
> outputs that most definitely are of no value to somebody reading the
> log after the fact, the output from vims' test suite is a good
> example.
> 
> Granted, it uses a very wide variety of terminal escape codes, which
> include, but are not limited to, window resizing control characters.

But how much of these are actually relevant to the majority of the
build logs that we receive; do they even contain those characters, or
are they actually already stripped? I haven't seen any build log do
window resizing so far for example. 

> I think given that context, it may be sane to restrict log control
> characters to a specific subset of control characters, specifically
> basic colour/highlighting control characters, which are at least
> somewhat standardized and not too device specific [... SNIP ...]

+1 The idea of just having the useful subset sounds really nice.

> Either way, if you were to introduce such a variable, you could have a
> standard defined value that Gentoo agree upon, which I'd imagine
> might be "none erasures:apply"  or something like that, and end users
> can turn them back on if they want, with a caveat that users should
> apply a standard stripping to these logs before submitting them to
> bugzilla, with a tool that comes standard with portage to facilitate
> stripping logs for submission to bugzilla.

-1 I'd rather not make submission more complex than it needs to be; if
we want any processing to happen here, we should probably let Bugzilla
do this to keep things easy for the user.

> This gives users the control they want, the features they want if
> they need them, and still gives gentoo staff an easier time if it is
> determined escape codes are not useful for bug reports.

The hard part will be the determination.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

On Wed, 4 Sep 2013 09:17:11 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 2013-09-03, o godz. 18:57:12
> "Walter Dnes" <waltdnes@waltdnes.org> napisał(a):
> 
> > On Tue, Sep 03, 2013 at 10:15:39PM +0200, Tom Wijsman wrote
> > 
> > > That is not what this is about, this is about having escape
> > > sequences in build logs obtained from Bugzilla; because, they aid
> > > in skimming through logs (until we implement the feature I asked
> > > for in subject).
> > 
> >   "The road to binary syslog files is paved with good intentions",
> > or something like that.  Question... why does it have to be escape
> > sequences?  Can't it be readable plain text?  E.g. something like...
> > 
> > //STDERR.OUT.START
> > foo
> > bar
> > blah blah blah
> > //STDERR.OUT.END
> > 
> >   This would be easy to grep and/or parse in bash.
> 
> Especially if one is interspersed with the other. I suggest trying
> first, then suggesting it to others.

Definitely do not want them on their own line; instead something like

    OUT:make:1000: ...
    ERR:gcc:1001: ...
    ERR:gcc:1001: ...
    ERR:gcc:1001: ...
    ERR:gcc:1001: ...
    ERR:make:1000: *** [...] Error 1
    ERR:make:1000: make[2]: *** Waiting for unfinished jobs....

If you then grep the last non-make and non-portage STDERR, you simply
get just the gcc lines you actually want. From there you can grep the
lines around it for context.

You could then have a tool that tells you something like

    ...
    === ERROR at build.log:123 ===
    ...
    ...
    ...
    ...
    === /ERROR at build.log:123 ===
    *** [...] Error 1
    make[2]: *** Waiting for unfinished jobs....

and that allows you to immediately see the error right away;
additionally, it shows a few lines of context before and after. If you
need more context, the line number allows you to jump right to it in
the build.log if you need even more context.

We could even build this feature into Bugzilla...

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

Dnia 2013-09-04, o godz. 11:24:22
Tom Wijsman <TomWij@gentoo.org> napisał(a):

> On Wed, 4 Sep 2013 09:17:11 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > Dnia 2013-09-03, o godz. 18:57:12
> > "Walter Dnes" <waltdnes@waltdnes.org> napisał(a):
> > 
> > > On Tue, Sep 03, 2013 at 10:15:39PM +0200, Tom Wijsman wrote
> > > 
> > > > That is not what this is about, this is about having escape
> > > > sequences in build logs obtained from Bugzilla; because, they aid
> > > > in skimming through logs (until we implement the feature I asked
> > > > for in subject).
> > > 
> > >   "The road to binary syslog files is paved with good intentions",
> > > or something like that.  Question... why does it have to be escape
> > > sequences?  Can't it be readable plain text?  E.g. something like...
> > > 
> > > //STDERR.OUT.START
> > > foo
> > > bar
> > > blah blah blah
> > > //STDERR.OUT.END
> > > 
> > >   This would be easy to grep and/or parse in bash.
> > 
> > Especially if one is interspersed with the other. I suggest trying
> > first, then suggesting it to others.
> 
> Definitely do not want them on their own line; instead something like
> 
>     OUT:make:1000: ...
>     ERR:gcc:1001: ...
>     ERR:gcc:1001: ...
>     ERR:gcc:1001: ...
>     ERR:gcc:1001: ...
>     ERR:make:1000: *** [...] Error 1
>     ERR:make:1000: make[2]: *** Waiting for unfinished jobs....
> 
> If you then grep the last non-make and non-portage STDERR, you simply
> get just the gcc lines you actually want. From there you can grep the
> lines around it for context.

And how are you going to implement this? I doubt that fd/vt input has
any sort of 'writing process id' indicator.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

On Wed, 4 Sep 2013 11:59:37 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> And how are you going to implement this? I doubt that fd/vt input has
> any sort of 'writing process id' indicator.

Yeah, will require some inspection into how this works and what
information we have available; if that indicator isn't present and
upstream doesn't want to add it, we can still opt to hook into the
write calls and make the information available that way.

I can't imagine there is completely no way to do this; a first example,
strace, comes to mind that can do this. So a stripped down form of that
would be one way to do it. Assuming we can't get direct information.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1512 bytes --]

On 4 September 2013 21:59, Michał Górny <mgorny@gentoo.org> wrote:

> And how are you going to implement this? I doubt that fd/vt input has
> any sort of 'writing process id' indicator.
>

In one terminal:

 cat -vET

In another:

pgrep -x cat # 199935


ls -la /proc/199935/fd/

dr-x------ 2 kent kent  0 Sep  4 23:29 .
dr-xr-xr-x 8 kent kent  0 Sep  4 23:28 ..
lrwx------ 1 kent kent 64 Sep  4 23:29 0 -> /dev/pts/3
lrwx------ 1 kent kent 64 Sep  4 23:29 1 -> /dev/pts/3
lrwx------ 1 kent kent 64 Sep  4 23:29 2 -> /dev/pts/3

So you can certainly get the information the ohter way round.


now...

Term 1:
$ tty
/dev/pts/3
$ cat -vET

Term 2:

for i in /proc/[0-9]*; do cmd=$(cat $i/comm); for j in $i/fd/*; do  echo -n
"$cmd:$j:"; readlink $j ; echo ; done  done | grep "pts/3"


bash:/proc/159261/fd/0:/dev/pts/3
bash:/proc/159261/fd/1:/dev/pts/3
bash:/proc/159261/fd/2:/dev/pts/3
bash:/proc/159261/fd/255:/dev/pts/3
gvim:/proc/175475/fd/0:/dev/pts/3
gvim:/proc/175475/fd/1:/dev/pts/3
gvim:/proc/175475/fd/2:/dev/pts/3
gvim:/proc/175642/fd/0:/dev/pts/3
gvim:/proc/175642/fd/1:/dev/pts/3
gvim:/proc/175642/fd/2:/dev/pts/3
gvim:/proc/199494/fd/0:/dev/pts/3
gvim:/proc/199494/fd/1:/dev/pts/3
gvim:/proc/199494/fd/2:/dev/pts/3
cat:/proc/207567/fd/0:/dev/pts/3
cat:/proc/207567/fd/1:/dev/pts/3
cat:/proc/207567/fd/2:/dev/pts/3

I see. I have a few gvim instances also reading/writing to that terminal I
didn't know about, interesting.



-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2629 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On 4 September 2013 21:59, Michał Górny <mgorny@gentoo.org> wrote:

> And how are you going to implement this? I doubt that fd/vt input has
> any sort of 'writing process id' indicator.
>


Though granted, my other post is not going to be useful on a line-by-line
basis.

The obvious easy approach is have an exec launcher of some kind that
launches processes with IO redirected to an intermediary process that
simply prepends arbitrary strings to every line sent to them, prior to
those processes piping the data to the final aggregate output.

But this would require large amounts of pipe work and fork+ipc magic.

Its surely doable, I recall seeing parts of paludis that do stuff like this.

Just those pipes are inherently easy to break in my experience.



-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1341 bytes --]

Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

Dnia 2013-09-04, o godz. 23:45:44
Kent Fredric <kentfredric@gmail.com> napisał(a):

> On 4 September 2013 21:59, Michał Górny <mgorny@gentoo.org> wrote:
> 
> > And how are you going to implement this? I doubt that fd/vt input has
> > any sort of 'writing process id' indicator.
> >
> 
> 
> Though granted, my other post is not going to be useful on a line-by-line
> basis.
> 
> The obvious easy approach is have an exec launcher of some kind that
> launches processes with IO redirected to an intermediary process that
> simply prepends arbitrary strings to every line sent to them, prior to
> those processes piping the data to the final aggregate output.
> 
> But this would require large amounts of pipe work and fork+ipc magic.
> 
> Its surely doable, I recall seeing parts of paludis that do stuff like this.
> 
> Just those pipes are inherently easy to break in my experience.

That's my point. There's no simple way of doing such a thing, and any
'hard' way involves a lot of fragile hacks and assumptions. And more of
that, more likely it will actually break something that relies on
at least some sanity of host.

I can't really imagine going anywhere without another LD_PRELOAD
library. Then, I don't know if platforms other than Linux have any
tools for proper chaining of LD_PRELOAD (think of sandbox). That's just
one issue.

Then, think of pipes. Some programs use stdin/stdout to communicate,
maybe even stderr. You need to know which handles you can actually
touch and which not. That should be fairly easy to achieve but still I
suspect it would be fragile.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

[gentoo-dev] Re: rfc: escape sequences in logs

[Chris Brannon]

William Hubbs <williamh@gentoo.org> writes:

> On Mon, Sep 02, 2013 at 09:41:28PM +0200, Michał Górny wrote:

>> That is a bug in pybugz and not an argument, you know.
>
> I said "things like pybugz".
>
> Bugzilla allowing control characters in the xml is the issue. The python
> xmlrpc library raises an exception for malformed xml because of it, so
> there isn't much pybugz, or anything that uses that library, can do
> about it.

Right.  The bug is in bugzilla itself, not pybugz.  I did some research.
Escape and all of the other non-whitespace ASCII control characters are
illegal in XML.  It is also not valid to escape them with entities, like
&#27;.  However, Bugzilla's XML-RPC interface sends them anyway.
This is only a problem for build logs included inline in bug comments.
Attached logs are fine, with or without the color codes.
Maybe the codes could be stripped by pybugz before processing the XML?

-- Chris

[gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: rfc: escape sequences in logs)

[Duncan]

Kent Fredric posted on Wed, 04 Sep 2013 23:38:40 +1200 as excerpted:

> I see. I have a few gvim instances also reading/writing to that terminal
> I didn't know about, interesting.

Which brings up the privacy point.  Anything getting this fancy and 
convoluted in terms of implementation is going to have to be very careful 
to narrow the scope of what it's looking at, lest it become the NSA in 
terms of dragnet.  Even then, how many people would REALLY trust it, 
knowing the lengths to which it has to go to narrow down and focus in on 
JUST the info it needs?  I don't think I'd trust it, for the same reason 
I don't trust the NSA doing similar things under "just trust us" 
pretenses.  Would YOU? (Granted, our implementation would presumably be 
open, something the NSA most certainly is NOT, but still...)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: rfc: escape sequences in logs)

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]

On Thu, 5 Sep 2013 04:54:46 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> Kent Fredric posted on Wed, 04 Sep 2013 23:38:40 +1200 as excerpted:
> 
> > I see. I have a few gvim instances also reading/writing to that
> > terminal I didn't know about, interesting.
> 
> Which brings up the privacy point.

Not really, because those instances wouldn't appear in logs.

> Anything getting this fancy and convoluted in terms of
> implementation is going to have to be very careful to narrow the
> scope of what it's looking at, lest it become the NSA in terms of
> dragnet.

It would be narrowed down to just the processes of the log; so, I don't
think anything special needs to be done.

> Even then, how many people would REALLY trust it, knowing
> the lengths to which it has to go to narrow down and focus in on JUST
> the info it needs?  I don't think I'd trust it, for the same reason I
> don't trust the NSA doing similar things under "just trust us"
> pretenses.  Would YOU? (Granted, our implementation would presumably
> be open, something the NSA most certainly is NOT, but still...)

You already put trust in us; a tiny feature difference that doesn't
really reveal anything (compared to eg. `emerge --info --verbose`),
isn't going to make a difference in that.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]