[gentoo-dev] Parsing emerge

[gentoo-dev] Parsing emerge

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 2533 bytes --]

I'm putting this here for informative purposes as to achieving proper
parsing of portage for use in extending the already present behaviour to
better suit one's needs.  A few things I'll bring into the table from my
experience:

1) NOCOLOR=true is a good idea.  It's rather hard to account for both
color and no color in regex parsing (unless someone has a spectacularly
fun regex for that...) and I find it easiest to parse.

2) stdout is home to all the portage output, !!! type stuff goes to
stderr.

3) Portage is actually fairly parseable if you try.  Lucky I also found
that the compile itself goes to stdout, giving you even more flexibility
for reporting type situations.

4) Parsing can, in some cases, help you add extra functionality to
portage without even having to hack the portage code.  I've already
worked on a system that can buffer 20 lines or so backlog, catch an
error in the compile, and give you the next few lines.  By doing this,
it  makes it easier to give bug reports by narrowing down what's failing
in a compile, expecially if you're getting spewed 100 compile messages
and your backlog can't keep up.  I've already talked with ferringb
regarding that and have looked at some added functionality to make such
a system more realiable.

I wrote a small perl script to exemplify what I'm trying to go at:

http://dev.gentoo.org/~chriswhite/emerge_parse.pl

In order to install this, you'll need to `g-cpan.pl Expect` in order to
get the Expect module (which interestingly enough also installs the dep
of IO::Tty).

I'm hoping that this information will be somewhat helpful to people here
in realizing the full potentional of parsing.  Basically, you run the
script with -pv arguments (for now.. more to come) like so:

`emerge_parse.pl -pv xine-lib`

and you'll get a nice translation of the emerge -pv layout:

root@secures chris # emerge_parse.pl -pv xine-lib
[Pretend] emerge would do a(n) re-install of
media-libs/xine-lib-1_rc8-r1

root@secures chris # emerge_parse.pl -pv java-sdk-docs
[Pretend] emerge would do a(n) fetchonly re-install of
dev-java/java-sdk-docs-1.4.2

root@secures dev-util # emerge_parse.pl -pv flawfinder
[Pretend] emerge would do a(n) new install of dev-util/flawfinder-1.24

.. etc.. etc..

As you can see, extra functionality has been added, and now you get the
translation of emerge's abbreviated install method format to something
more human readable.  I'll hope to have more to come with regards to
emerge parsing.

[-- Attachment #2: このメッセージにはデジタル署名された部分があります --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Parsing emerge

[Ian Leitch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris White wrote:
| I'm putting this here for informative purposes as to achieving proper
| parsing of portage for use in extending the already present behaviour to
| better suit one's needs.  A few things I'll bring into the table from my
| experience:
|
| 1) NOCOLOR=true is a good idea.  It's rather hard to account for both
| color and no color in regex parsing (unless someone has a spectacularly
| fun regex for that...) and I find it easiest to parse.
|
| 2) stdout is home to all the portage output, !!! type stuff goes to
| stderr.
|
| 3) Portage is actually fairly parseable if you try.  Lucky I also found
| that the compile itself goes to stdout, giving you even more flexibility
| for reporting type situations.
|
| 4) Parsing can, in some cases, help you add extra functionality to
| portage without even having to hack the portage code.  I've already
| worked on a system that can buffer 20 lines or so backlog, catch an
| error in the compile, and give you the next few lines.  By doing this,
| it  makes it easier to give bug reports by narrowing down what's failing
| in a compile, expecially if you're getting spewed 100 compile messages
| and your backlog can't keep up.  I've already talked with ferringb
| regarding that and have looked at some added functionality to make such
| a system more realiable.
|
| I wrote a small perl script to exemplify what I'm trying to go at:
|
| http://dev.gentoo.org/~chriswhite/emerge_parse.pl
|
| In order to install this, you'll need to `g-cpan.pl Expect` in order to
| get the Expect module (which interestingly enough also installs the dep
| of IO::Tty).
|
| I'm hoping that this information will be somewhat helpful to people here
| in realizing the full potentional of parsing.  Basically, you run the
| script with -pv arguments (for now.. more to come) like so:
|
| `emerge_parse.pl -pv xine-lib`
|
| and you'll get a nice translation of the emerge -pv layout:
|
| root@secures chris # emerge_parse.pl -pv xine-lib
| [Pretend] emerge would do a(n) re-install of
| media-libs/xine-lib-1_rc8-r1
|
| root@secures chris # emerge_parse.pl -pv java-sdk-docs
| [Pretend] emerge would do a(n) fetchonly re-install of
| dev-java/java-sdk-docs-1.4.2
|
| root@secures dev-util # emerge_parse.pl -pv flawfinder
| [Pretend] emerge would do a(n) new install of dev-util/flawfinder-1.24
|
| .. etc.. etc..
|
| As you can see, extra functionality has been added, and now you get the
| translation of emerge's abbreviated install method format to something
| more human readable.  I'll hope to have more to come with regards to
| emerge parsing.

Sorry to spoil your post Chris, but I think this is just another
testimonial to how badly we all need that portage API (plus the portage
daemon for tools such as Porthole who currently also have to parse
emerge output). This is by no means the first attempt at parsing I've seen.

But hey, nice work in the absence of the API, it does look like the
script has uses.

Regards,
Ian Leitch
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFCKEHIefZ4eWAXRGIRAtzxAJ0WzeL5erEZZISc2RtHN7fmUiAKlgCgj7nv
GdAhWXwT+BUrhrgkFJIU8eI=
=pqYj
-----END PGP SIGNATURE-----
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parsing emerge

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

At 2005-03-04 11:08 +0000 Ian Leitch Wrote:

> Sorry to spoil your post Chris, but I think this is just another
> testimonial to how badly we all need that portage API (plus the portage
> daemon for tools such as Porthole who currently also have to parse
> emerge output). This is by no means the first attempt at parsing I've seen.

Well, I do indeed see that as a valid argument, but at the same time,
something like this may take a bit shorter time as far as implementation
than getting the works of a new portage API.  In fact, even if we did
get a portage API, I still think that parsing could extend the
flexibility of portage as you have you seen.  On another point, I would
assume (correct me if I'm wrong), that the portage API would be python
based.  For those who don't want to use python, you can still add
functionality (this was written in perl) that you desire.

> But hey, nice work in the absence of the API, it does look like the
> script has uses.
> 
> Regards,
> Ian Leitch


[-- Attachment #2: このメッセージにはデジタル署名された部分があります --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Parsing emerge

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

At 2005-03-04 11:08 +0000 Ian Leitch Wrote:

> Sorry to spoil your post Chris, but I think this is just another
> testimonial to how badly we all need that portage API (plus the portage
> daemon for tools such as Porthole who currently also have to parse
> emerge output). This is by no means the first attempt at parsing I've seen.

Well, I do indeed see that as a valid argument, but at the same time,
something like this may take a bit shorter time as far as implementation
than getting the works of a new portage API.  In fact, even if we did
get a portage API, I still think that parsing could extend the
flexibility of portage as you have you seen.  On another point, I would
assume (correct me if I'm wrong), that the portage API would be python
based.  For those who don't want to use python, you can still add
functionality (this was written in perl) that you desire.

> But hey, nice work in the absence of the API, it does look like the
> script has uses.
> 
> Regards,
> Ian Leitch


[-- Attachment #2: このメッセージにはデジタル署名された部分があります --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Parsing emerge

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

On Sat, 2005-03-05 at 00:05 +0900, Chris White wrote:
> based.  For those who don't want to use python, you can still add
> functionality (this was written in perl) that you desire.

Blasphemer!  Get the rope, boys.  ;]

Anyway, this script looks nice.  It is always fun to play around with
portage.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Parsing emerge

[Ian Leitch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris White wrote:
~ > Well, I do indeed see that as a valid argument, but at the same time,
| something like this may take a bit shorter time as far as implementation
| than getting the works of a new portage API.  In fact, even if we did
| get a portage API, I still think that parsing could extend the
| flexibility of portage as you have you seen.  On another point, I would
| assume (correct me if I'm wrong), that the portage API would be python
| based.  For those who don't want to use python, you can still add
| functionality (this was written in perl) that you desire.

Indeed, it is a nice tool to have while we wait for the API, I should
have made this clear in my first post, sorry :)

As for using other languages, once the API is in place it would then be
possible to create bindings for those languages.

BTW, no need to send to robin.g.o and lists.g.o anymore, either will do.

Regards,
Ian Leitch

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFCKKBCefZ4eWAXRGIRAl4kAJ0U5akp5ksMD30nOllbvF9uRxJfxACfSXL1
ykRWHqesC3oxtietmNOjDI4=
=BIVn
-----END PGP SIGNATURE-----
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parsing emerge

[Jason Stubbs]

On Friday 04 March 2005 20:08, Ian Leitch wrote:
> Chris White wrote:
> | I'm putting this here for informative purposes as to achieving proper
> | parsing of portage for use in extending the already present behaviour to
> | better suit one's needs.  A few things I'll bring into the table from my
> | experience:
>
> Sorry to spoil your post Chris, but I think this is just another
> testimonial to how badly we all need that portage API (plus the portage
> daemon for tools such as Porthole who currently also have to parse
> emerge output). This is by no means the first attempt at parsing I've seen.

I've asked several times before but I may as well ask again. What are the 
requirements? From my point of view, there is not much point in doing 
anything without some specific requirements. Take the analogy of a teenage 
daughter asking her parents for a transportation vehicle. A tractor would fit 
the bill as would a Boeing 747...

Regards,
Jason Stubbs
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Parsing emerge

[Chris White]

[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]

2005-03-04 (金) の 17:29 +0900 に Chris White さんは書きました:
I've updated the script now, it has a better more functional layout than
before (A large page script :).

Re-cap of the install is here:

> 
> http://dev.gentoo.org/~chriswhite/emerge_parse.pl
> 
> In order to install this, you'll need to `g-cpan.pl Expect` in order to
> get the Expect module (which interestingly enough also installs the dep
> of IO::Tty).

the main update was the parsing of category, package, and versions from
ebuilds.  The code of interest comes from these lines of code:

sub package_nvsplit {
        $i=0;
        while($_[$i] !~ /^\d/) {
                $i++;
        }
        $package_end=$i-1;
        $pn=join("-",@_[0..$package_end]);
        $pv=@_[$i..$#package_array];
        return $pn, $pv;
}

So for those interested in parsing out package versions and what not,
this is basically how to go about it.  First, you splitup the category
and the package name+version (or $P).  So...

media-video/xine-lib-1.0 gets splitup into:
media-video (category) and xine-lib-1.0 ($P).  Now then, to get package
and version seperation, one simply splits up $P with the - seperator.
This gives us:

xine lib 1.0

Which, a simple loop checks to see if we've hit a section that starts
with a digit (the version number), and re-combines the values before it
with -'s, giving us:

$pv = 1.0
$pn = xine-lib

So there you have it, the basis on seperation of package category, name,
and version.  Hopefully this will help those who wish to hack around a
little get an idea of what's involved in the minute areas of portage.
Note that while the code is perl, the concept is very language
non-specific (I could do one in C.. but I'd rather save some of my
sanity ;).

Next stop will be getting USE flag descriptions in a more speedy manner
than simple grep-ing.  Please note that suggestions for
improvement/comments/etc. are welcome, and I hope this turns into a nice
informative developer's discussion.  

_big note_
I'd also like to thank Nick (carpaski) for his insight on the above
code :).
_big note_

[-- Attachment #2: このメッセージにはデジタル署名された部分があります --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Parsing emerge

[Aron Griffis]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

Chris White wrote:	[Sat Mar 05 2005, 12:11:24AM EST]
> sub package_nvsplit {
>         $i=0;
>         while($_[$i] !~ /^\d/) {
>                 $i++;
>         }
>         $package_end=$i-1;
>         $pn=join("-",@_[0..$package_end]);
>         $pv=@_[$i..$#package_array];
>         return $pn, $pv;
> }

my ($cat, $name, $ver) = m{(.*)/(.*?)-(\d.*)};

There are much better regexes that will actually validate, but the
above regex does everything you described.

If you're interested in completed Perl code to correctly sort ebuilds
according to portage rules, see
http://dev.gentoo.org/~agriffis/darcs/skel/bashrc.funcs and search for
sortvers.

Regards,
Aron

--
Aron Griffis
Gentoo Linux Developer


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]