[gentoo-portage-dev] Portage API

[gentoo-portage-dev] Portage API

[Jason Stubbs]

Hi all,

Most will know that I worked on and mostly completed an API for portage and 
that I've had it to throw it away due to being directly counter to portage's 
future goals. Thus, an API has been pushed back to the release following 
2.0.51. So, what I need is a list of things that are required of portage from 
a client application writer's perspective.

A couple of goals for the API:

* Loose enough to not inhibit portage's direction
* Light enough that it is easy to maintain
* Simple enough that the way to do things is obvious
* Minimilist enough that there is only one way to do things

So, let the suggestions roll. :)

Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Brian]

On Sun, 2004-10-17 at 20:01 +0900, Jason Stubbs wrote:

> So, let the suggestions roll. :)
> 
> Regards,
> Jason Stubbs

Dan has been the porthole developer that has done the most integrating
portage modules for porthole's use, but it looks like we are loosing him
from gentoo for now.  I have not had much reason to think about how much
more we might need from portage.



Portage modules/variables currently used directly by Porthole:

portage.grabfile():: used for obtaining [keywords.desc, use.desc, use.local.desc]
	
portage.config(clone=portage.settings).environ()[var] :: where var = 1 of ['PORTDIR_OVERLAY','PORTDIR', "USE"]
	
portage.auxdbkeys

"""Extract installed versions from full name."""
return portage.db['/']['vartree'].dep_match(full_name)

portage.db['/']['porttree'].getallnodes()

portage.db['/']['vartree'].getallnodes()

portage.catpkgsplit(ebuild)

portage.portdb.aux_get(ebuild, [property])[0]

portage.best(versions)

# showing complete porthole function for (possibly) more clarity
def get_properties(ebuild):
    """Get all ebuild variables in one chunk."""
    return Properties(dict(zip(keys,
                               portage.portdb.aux_get(ebuild,
                                                      portage.auxdbkeys))))


def get_versions(self, include_masked = True):
        """Returns all versions of the available ebuild"""
        # Note: this slow, especially when include_masked is false
        criterion = include_masked and 'match-all' or 'match-visible'
        return portage.portdb.xmatch(criterion, self.full_name)





Wish list:
	
Portages version comparison/handling code from bug: http://bugs.gentoo.org/show_bug.cgi?id=37406
was would be implemented soon.  I see browsing thru 2.0.51_rc9 that it is not yet implemented.
If it is being implemented Porthole should use the following from portage's code:
			
ver_regexp = re.compile("^(cvs-)?(\\d+)((\\.\\d+)*)([a-zA-Z]?)((_(pre|p|beta|alpha|rc)\\d*)*)(-r(\\d+))?$")
suffix_regexp = re.compile("^(alpha|beta|rc|pre|p)(\\d*)$")
# modified portage comparison suffix values for sorting in desired precedence
suffix_value = {"alpha": '0', "beta": '1', "pre": '2', "rc": '3', "p": '4'}
	
That would allow porthole's modified version of that code to follow any changes in portage's values.

-- 
Brian <dol-sen@telus.net>


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Jason Stubbs]

On Monday 18 October 2004 01:25, Brian wrote:
> Dan has been the porthole developer that has done the most integrating
> portage modules for porthole's use, but it looks like we are loosing him
> from gentoo for now.

Yeah, I saw that..

> I have not had much reason to think about how much more we might need from 
> portage. 

I'm more interested in what you are already using and more importantly why. 
Whatever the API ends up being, it'll probably resemble portage's current API 
very little.

I'll try to ascertain the why's from the what's that you've listed. Let me 
know if I get any wrong.

> portage.grabfile():: used for obtaining [keywords.desc, use.desc,
> use.local.desc]

Descriptions of USE flags. (I don't know what keywords.desc is.)

> portage.config(clone=portage.settings).environ()[var] :: where var = 1 of
> ['PORTDIR_OVERLAY','PORTDIR', "USE"]

The current USE settings. I'm don't see why PORTDIR or PORTDIR_OVERLAY is 
needed.

> portage.auxdbkeys

I'm not sure what's required here either.

> """Extract installed versions from full name."""
> return portage.db['/']['vartree'].dep_match(full_name)

Querying the installed package database.

> portage.db['/']['porttree'].getallnodes()

Querying the ebuild package database.

> portage.db['/']['vartree'].getallnodes()

As above.

> portage.catpkgsplit(ebuild)

Not sure why this is needed.

> portage.portdb.aux_get(ebuild, [property])[0]

Querying individual packages.

> portage.best(versions)

Package version comparison.

> # showing complete porthole function for (possibly) more clarity
> def get_properties(ebuild):
>     """Get all ebuild variables in one chunk."""
>     return Properties(dict(zip(keys,
>                                portage.portdb.aux_get(ebuild,
>                                                       portage.auxdbkeys))))
>
>
> def get_versions(self, include_masked = True):
>         """Returns all versions of the available ebuild"""
>         # Note: this slow, especially when include_masked is false
>         criterion = include_masked and 'match-all' or 'match-visible'
>         return portage.portdb.xmatch(criterion, self.full_name)

I think these are both covered above.

> Wish list:
>
> Portages version comparison/handling code from bug:
> http://bugs.gentoo.org/show_bug.cgi?id=37406 was would be implemented soon.
>  I see browsing thru 2.0.51_rc9 that it is not yet implemented. If it is
> being implemented Porthole should use the following from portage's code:
>
> ver_regexp =
> re.compile("^(cvs-)?(\\d+)((\\.\\d+)*)([a-zA-Z]?)((_(pre|p|beta|alpha|rc)\\
>d*)*)(-r(\\d+))?$") suffix_regexp =
> re.compile("^(alpha|beta|rc|pre|p)(\\d*)$")
> # modified portage comparison suffix values for sorting in desired
> precedence suffix_value = {"alpha": '0', "beta": '1', "pre": '2', "rc":
> '3', "p": '4'}

This is covered above with version comparison.

> That would allow porthole's modified version of that code to follow any
> changes in portage's values.

I strongly recommend against doing any of this sort of stuff manually. I know 
that there isn't really much choice at the moment, but that's exactly what 
we're trying to get away from.

I believe porthole also makes a lot of use of emerge. If possible, please let 
me know what options it is called with and what output is parsed.

Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Brian]

[-- Attachment #1: Type: text/plain, Size: 5492 bytes --]

On Mon, 2004-10-18 at 22:53 +0900, Jason Stubbs wrote:

> I'm more interested in what you are already using and more importantly why. 
> Whatever the API ends up being, it'll probably resemble portage's current API 
> very little.
> 
> I'll try to ascertain the why's from the what's that you've listed. Let me 
> know if I get any wrong.
> 
> > portage.grabfile():: used for obtaining [keywords.desc, use.desc,
> > use.local.desc]
> 
> Descriptions of USE flags. (I don't know what keywords.desc is.)
> 

keywords.desc does not exist on my system: func() call =
portage.grabfile('/usr/portage/profiles/keywords.desc')

It seems to be a depricated function in porthole, the output it stored
if found was not used anywhere.   I am deleting it.

> > portage.config(clone=portage.settings).environ()[var] :: where var = 1 of
> > ['PORTDIR_OVERLAY','PORTDIR', "USE"]
> 
> The current USE settings. I'm don't see why PORTDIR or PORTDIR_OVERLAY is 
> needed.

Porthole searches the directories for ebuilds, etc..  Better to use the
correct directories, so get the correct directories.  I believe Dan was
using portage rather than creating new code to scan thru /etc/make.conf
as well as the globals & defaults.

> 
> > portage.auxdbkeys
> 
> I'm not sure what's required here either.

I'll look into it further.


> 
> > """Extract installed versions from full name."""
> > return portage.db['/']['vartree'].dep_match(full_name)
> 
> Querying the installed package database.
> 
> > portage.db['/']['porttree'].getallnodes()
> 
> Querying the ebuild package database.
> 
> > portage.db['/']['vartree'].getallnodes()
> 
> As above.
> 
> > portage.catpkgsplit(ebuild)
> 
> Not sure why this is needed.

looks like = convienence, why re-invent the code to split the
cat/package-version.
 
> 
> > portage.portdb.aux_get(ebuild, [property])[0]
> 
> Querying individual packages.
> 
> > portage.best(versions)

for finding picking specific ebuild properties to display, etc.


> 
> > # showing complete porthole function for (possibly) more clarity
> > def get_properties(ebuild):
> >     """Get all ebuild variables in one chunk."""
> >     return Properties(dict(zip(keys,
> >                                portage.portdb.aux_get(ebuild,
> >                                                       portage.auxdbkeys))))
> >
> >
> > def get_versions(self, include_masked = True):
> >         """Returns all versions of the available ebuild"""
> >         # Note: this slow, especially when include_masked is false
> >         criterion = include_masked and 'match-all' or 'match-visible'
> >         return portage.portdb.xmatch(criterion, self.full_name)
> 
> I think these are both covered above.
> 
> > Wish list:
> >
> > Portages version comparison/handling code from bug:
> > http://bugs.gentoo.org/show_bug.cgi?id=37406 was would be implemented soon.
> >  I see browsing thru 2.0.51_rc9 that it is not yet implemented. If it is
> > being implemented Porthole should use the following from portage's code:
> >
> > ver_regexp =
> > re.compile("^(cvs-)?(\\d+)((\\.\\d+)*)([a-zA-Z]?)((_(pre|p|beta|alpha|rc)\\
> >d*)*)(-r(\\d+))?$") suffix_regexp =
> > re.compile("^(alpha|beta|rc|pre|p)(\\d*)$")
> > # modified portage comparison suffix values for sorting in desired
> > precedence suffix_value = {"alpha": '0', "beta": '1', "pre": '2', "rc":
> > '3', "p": '4'}
> 
> This is covered above with version comparison.
> 

No it isn't.   Portage's vercmp() is used to return the best ebuild of
two supplied.   Porthole needs to sort a list of available ebuilds to
proper order.  I based that code from Marius's changed code for vercmp()
so that porthole could display the ebuild version list in the correct
order.


> > That would allow porthole's modified version of that code to follow any
> > changes in portage's values.
> 
> I strongly recommend against doing any of this sort of stuff manually. I know 
> that there isn't really much choice at the moment, but that's exactly what 
> we're trying to get away from.

To use portage's vercmp() I would have to write a python bubble sort to
sort a list of ebuild versions.   What I did was use Marius's code and
modified it to pad the version string parts to 3 digits then used
python's builtin list.sort().

> 
> I believe porthole also makes a lot of use of emerge. If possible, please let 
> me know what options it is called with and what output is parsed.
> 
Porthole can currently send any emerge option by use of the advanced
emerge dialog for the most common advanced options or the run_custom
dialog which inputs any command to send to the terminal.  I have run ls,
python-updater and a few others as a test.

Due to porthole-terminal not being interactive commands such as "emerge
-a" will hang waiting for input that it can not receive.


We currently filter out any warning, caution, summary info including any
">>>" statements, einfo statements, errors.   For a complete list look
in the configuration.xml file attached.

I would like to change the terminal to use the vte gtk terminal widget
which could allow it to be a full function terminal with the message
filtering, su & chroot ability.  I did have a somewhat functional
version I tested but the python interface was incomplete and is not
really developed.  I would need to re-code the terminal in c. In case
you didn't know vte is gnome-terminals major workings that was split
into the main virtual terminal widget and the gnome-terminal
enhancements.


-- 
Brian <dol-sen@telus.net>

[-- Attachment #2: configuration.xml --]
[-- Type: text/xml, Size: 2329 bytes --]

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<portholeconfig ver="0.3">
  <re_filters>
    <info py_type="list" item_count="5">
      <info-1>^&gt;&gt;&gt;[\s][^/]</info-1>
      <info-2>^\s\*\s</info-2>
      <info-3>.*[Rr]emember\sto</info-3>
      <info-4>.*[Ff]orget\sto</info-4>
      <info-5>^\!\!\!</info-5>
    </info>
    <notinfo py_type="list" item_count="6">
      <notinfo-1>^&gt;&gt;&gt; Waiting 5 seconds</notinfo-1>
      <notinfo-2>^&gt;&gt;&gt; \([Cc]ontrol-[Cc] to abort</notinfo-2>
      <notinfo-3>^&gt;&gt;&gt; [Uu]nmerging in:</notinfo-3>
      <notinfo-4>^\s\*\s[Cc]aching service dependencies</notinfo-4>
      <notinfo-5>.*libtool.*remember to run.*libtool</notinfo-5>
      <notinfo-6>.*libtool.*has not been installed in</notinfo-6>
    </notinfo>
    <warning py_type="list" item_count="4">
      <warning-1>.*[Ww][Aa][Rr][Nn][Ii][Nn][Gg][:\!,\s]</warning-1>
      <warning-2>.*[Ff]ile.*not found</warning-2>
      <warning-3>.*\s[Ee][Rr][Rr][Oo][Rr][:\-\!,]\s</warning-3>
      <warning-4>.*([Ss]yntax[Ee]rror|[Ss]yntax [Ee]rror[:\-\!,])</warning-4>
    </warning>
    <notwarning py_type="list" item_count="7">
      <notwarning-1>.*[Ii]f you wanted to set the \-\-build type</notwarning-1>
      <notwarning-2>.*(checking|check).* compiler warning flags</notwarning-2>
      <notwarning-3>.*(checking|check).* warning flags.* compiler</notwarning-3>
      <notwarning-4>.*remember to run.*libtool</notwarning-4>
      <notwarning-5>.*warning: \-jN forced in submake:</notwarning-5>
      <notwarning-6>.*has not been installed in `/usr/lib</notwarning-6>
      <notwarning-7>^libtool\:.*seems to be moved</notwarning-7>
    </notwarning>
    <error py_type="list" item_count="3">
      <error-1>^\!\!\!\sERROR\:.*\sfailed\.</error-1>
      <error-2>^\!\!\!\s[Ee]rror calculating dependencies.*[Pp]lease correct</error-2>
      <error-3>^\!\!\!\s[Ee]rror\:.*\sconflicts with another package</error-3>
    </error>
    <noterror py_type="list" item_count="0">
    </noterror>
    <caution py_type="list" item_count="1">
      <caution-1>.*[Cc][Aa][Uu][Tt][Ii][Oo][Nn][:!,\s]</caution-1>
    </caution>
    <notcaution py_type="list" item_count="0">
    </notcaution>
    <emerge>^&gt;&gt;&gt; emerge [^/]</emerge>
    <ebuild>^\d+\s+\[ebuild.*</ebuild>
  </re_filters>
</portholeconfig>


[-- Attachment #3: Type: text/plain, Size: 45 bytes --]

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Daniel Taylor]

On Oct 17, 2004, at 12:25, Brian wrote:
> On Sun, 2004-10-17 at 20:01 +0900, Jason Stubbs wrote:
>
>> So, let the suggestions roll. :)
>>
>> Regards,
>> Jason Stubbs
>
> Dan has been the porthole developer that has done the most integrating
> portage modules for porthole's use, but it looks like we are loosing 
> him
> from gentoo for now.  I have not had much reason to think about how 
> much
> more we might need from portage.

I think Jason and I talked about this somewhat a while back. It was 
actually Fredrik Arnerup that did most of the work with portagelib (I 
originally used file system calls to get portage info, which is bad bad 
bad!). I'm not sure he's on this list so you may want to mail him.

And just so everyone knows, I have moved away from Gentoo and towards 
debian-based distributions because of my current situation. I am sorry, 
but life has to move on, and I need to spend more time on my studies 
now. Debian is just faster and easier for me. I will try and 
participate as well as I can, though, and try to stay active on the 
various mailing lists and help out if I can.

I need to go now, but expect another email from me about this stuff 
after I get a chance to read it. I think the interfacing with portage 
can be made MUCH better for third party application developers.
--
Daniel G. Taylor
http://programmer-art.org


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Brian]

On Sun, 2004-10-17 at 20:01 +0900, Jason Stubbs wrote:

> A couple of goals for the API:
> 
> * Loose enough to not inhibit portage's direction
> * Light enough that it is easy to maintain
> * Simple enough that the way to do things is obvious
> * Minimilist enough that there is only one way to do things
> 
> So, let the suggestions roll. :)
> 
> Regards,
> Jason Stubbs
> 

You might want to email pythonhead, he is importing portage and
gentoolkit for his portage frontend named portal.  I don't know if he is
on this list.

> --
> gentoo-portage-dev@gentoo.org mailing list
> 
-- 
Brian <dol-sen@telus.net>


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Brian]

Here is the portion of Fredriks reply on the porthole-dev list for those
that are not on that list.

Hi,

Yes, I confess. I wrote most of portagelib.py. I am a little rusty
though, since I haven't been involved in porthole development for quite
som time.

mån 2004-10-18 klockan 22.14 skrev Brian:
> On Mon, 2004-10-18 at 22:53 +0900, Jason Stubbs wrote:
> > The current USE settings. I'm don't see why PORTDIR or
PORTDIR_OVERLAY is 
> > needed.
> 
> Porthole searches the directories for ebuilds, etc..  Better to use
the
> correct directories, so get the correct directories.  I believe Dan
was
> using portage rather than creating new code to scan
thru /etc/make.conf
> as well as the globals & defaults.

Yep, except it was I who did it :-)
 
> > > portage.auxdbkeys
> > 
> > I'm not sure what's required here either.

auxdbkeys is/should be a list of all properties an ebuild can have. It
was used to get all properties at once. The point of this was, if I
recall correctly, to convert the properties into fields in a python
object. Not necessary, but high in hack value, I think :-)

> > > # showing complete porthole function for (possibly) more clarity
> > > def get_properties(ebuild):
> > >     """Get all ebuild variables in one chunk."""
> > >     return Properties(dict(zip(keys,
> > >                                portage.portdb.aux_get(ebuild,
> > >
portage.auxdbkeys))))
> > >
> > >

See previous point.

> 
> To use portage's vercmp() I would have to write a python bubble sort
to
> sort a list of ebuild versions.   What I did was use Marius's code and
> modified it to pad the version string parts to 3 digits then used
> python's builtin list.sort().
> 

Ah, but list.sort() takes an option argument, which is a comparison
function. I haven't looked at it, but I would guess vercmp() is designed
to work with list.sort(). No need to write that bubble sort.

-- 
Fredrik Arnerup <e97_far@e.kth.se>
http://www.stacken.kth.se/~foo/

-- 
Brian <dol-sen@telus.net>


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Daniel Taylor]

Hey,

Glad to see you got a hold of Fredrik. He knows far more about Portage 
internals and what we are using than anyone else, even if he hasn't 
worked with Porthole for a while!

Note that what follows are the rantings of someone who will probably 
not be using Portage or Porthole anymore, so take it as you will.

Now, as Brian and Fredrik have answered many of the questions and 
pointed out what and why in their replies, I'll talk about my take on 
the whole issue. I think there are a few issues that need to be 
addressed for the API to work.

Currently, interfacing with Portage is complicated. The Portage 
internals aren't very documented, and much of the code is far from 
self-documenting (check out the dependency stuff, I still don't get 
it!). I do realize the point of the API is to have a well documented, 
standard way of interfacing with Portage, so I see the API as a 
solution to this issue. This is why I was bugging you so much, Jason ;)

Everytime that Porthole does anything other than load package 
information, and external script (emerge) is run. This seems a bit 
inefficient, when we already have Portage loaded in memory as a module. 
What I think would be much more convenient and perhaps provide more 
functionality would be to emerge through the API; 
portage.emerge(app_name, pipe), where pipe would allow reading output 
and entering input (if a question needs to be answered or something). 
You should be able to internally streamline this process so that 
another instance of Portage doesn't need to be loaded. This principle 
applies to other functions, such as syncing, as well.

 From what I remember of the docs you showed me for the now defunct API, 
everything was a bit obscure. You emerged or pretended based on which 
tree was sent to a function (from what I understood of it, anyway). 
These functions should be made as easy to understand for new developers 
as can be. They should try to be self-explanatory.

This isn't part of the API, but I thought I'd mention it. Portage is 
notorious for not being internationalized. I did a mockup of a gconf- 
and GNOME-using, il8n-ized, HIGified, multi-backend Porthole just a few 
weeks back. I think Brian is adding il8n to the current codebase. Even 
if we were to support il8n, package descriptions and various output 
would all only be available in English. This needs to change.

One thing that really impressed me with the API you were working on is 
that you used it to make an 'emerge' script that acted just like the 
official one. This is how this stuff needs to be done. Portage should 
just be a library, and it should contain (and export) all of the 
functionality for the system to function properly (managing packages 
and information, updating, whatever). All tools that are written should 
use the API. This puts more force behind putting core functionality in 
the library instead of the scripts, which is a pain at the moment (take 
a look at emerge).

Well, enough ranting for now. I'm sure I am wrong about some of these 
points, so be gentle ;-)
--
Daniel G. Taylor
http://programmer-art.org


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Brian]

On Tue, 2004-10-19 at 18:56 -0400, Daniel Taylor wrote:

> Everytime that Porthole does anything other than load package 
> information, and external script (emerge) is run. This seems a bit 
> inefficient, when we already have Portage loaded in memory as a module. 
> What I think would be much more convenient and perhaps provide more 
> functionality would be to emerge through the API; 
> portage.emerge(app_name, pipe), where pipe would allow reading output 
> and entering input (if a question needs to be answered or something). 
> You should be able to internally streamline this process so that 
> another instance of Portage doesn't need to be loaded. This principle 
> applies to other functions, such as syncing, as well.
> 

I don't know that much about threading, but currently porthole must be
run as root in order to call emerge to sync or install.  That means
anything that porthole spawns runs as root, including browsers.  I
believe that emerge should still be a spawned process so that porthole
can have the ability to run as a user with only the spawned processes
that require root priveliges can be run so.   Unfortunately there is not
currently a python interface to libgksu & libgksuUI.  I have thought
about using swig to generate one.

Any experienced volunteers???

That way porthole could also run with a different portage for the gui
than the target system such as a chroot install/maintainance (I think).


> This isn't part of the API, but I thought I'd mention it. Portage is 
> notorious for not being internationalized. I did a mockup of a gconf- 
> and GNOME-using, il8n-ized, HIGified, multi-backend Porthole just a few 
> weeks back. I think Brian is adding il8n to the current codebase. Even 
> if we were to support il8n, package descriptions and various output 
> would all only be available in English. This needs to change.
> 

I currently only have all text strings marked for gettext support.  I
need to study up on what else needs to be done.  I have one volunteer
for vietnamese translation.  I am sure others will volunteer once
support is complete.


I agree with Dan that better dependancy connections are needed.
Porthole does not I believe always agree with what an emerge -p comes up
with.  But it is now much better than it used to be.

-- 
Brian <dol-sen@telus.net>


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Portage API

[Jason Stubbs]

On Tuesday 19 October 2004 10:08, Brian wrote:
> Here is the portion of Fredriks reply on the porthole-dev list for those
> that are not on that list.

Heh.. I got this three times. ;)

Just letting you all know that I haven't forgotten about this thread. Just 
been busy with the release of 2.0.51. Will get back to this soon.

Regards,
Jason Stubbs

--
gentoo-portage-dev@gentoo.org mailing list