[gentoo-guis] One backend for alle portage GUIs

[gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi guys,

today while idling in #gentoo-guis bheekling came up with
www.packagekit.org, which wants to be one frontend for many package
managers. We had some discussion about it - but this is not want I want
to propose.

While thinking about it, I wondered, why we have so many different
portage frontends - and each having its own portage backend. As there
are different approaches in defining a GUI, there is only one approach
of getting your information from portage. So instead of having one GUI,
I want to have one backend, which can be used by all different GUIs.
This makes the development of new GUIs simple - and might even be useful
for small scripts.

Also this backend should it make possible to access different package
managers (portage, paludis, pkgcore, equo) through one standardized API.
Thus in the end all GUIs will support all different Gentoo Package Manglers.

HOW?
====

Basically I want to have one service in the middle, that is accessed via
DBus. This service itself calls pkg-manager backends which in turn get
information out of the pkg manager.

For the second part there are two different approaches:
1) Have all backends in the service itself.
2) Have the backends extra - and they are called by DBus itself.

Pros & Cons:
1)
- - C: All backends should be written in the same language, what could be
difficult, as different pkg-managers use different languages.
- - P: Faster?

2)
- - P: Different languages can be chosen - and it is easy to add a new
backend.
- - P: The backend might be provided by the package manager itself, which
makes it easier to maintain.
- - C: More overhead.

Open Issues
===========

- -  How could we send complex objects with dbus?


So guys ... now the question: Could you imagine to put efforts together
and plan such a thing? =)

Regards,
- -- Necoro =)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG8hir4UOg/zhYFuARAsDYAJ90E3ef0ROwXIBFDfSnVh3z7hYIhgCfW1t5
75vIYXskq3E34EBA8/V3I2E=
=ypIU
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[Luis Francisco Araujo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann wrote:
> While thinking about it, I wondered, why we have so many different
> portage frontends - and each having its own portage backend. As there
> are different approaches in defining a GUI, there is only one approach
> of getting your information from portage. So instead of having one GUI,

It might theoretically sound like that , but different GUI approaches
might bring different backend approaches too.

> I want to have one backend, which can be used by all different GUIs.
> This makes the development of new GUIs simple - and might even be useful
> for small scripts.
> 
> Also this backend should it make possible to access different package
> managers (portage, paludis, pkgcore, equo) through one standardized API.
> Thus in the end all GUIs will support all different Gentoo Package Manglers.
> 

It really sounds nice, i suppose it'd be a matter of defining what would
be the exact scope of this back-end. I don't think it might offer all
the necessary stuff for different package managers to make truly
independent a GUI from back-end code , nevertheless , it could really
offer a set of sharable standard procedures (probably as a library?).

> HOW?
> ====
> 
> Basically I want to have one service in the middle, that is accessed via
> DBus. This service itself calls pkg-manager backends which in turn get
> information out of the pkg manager.
> 
> For the second part there are two different approaches:
> 1) Have all backends in the service itself.

This would be pretty much like a library, there is no need for DBus in
such a case i think.

> 2) Have the backends extra - and they are called by DBus itself.
> 
> Pros & Cons:
> 1)
> - C: All backends should be written in the same language, what could be
> difficult, as different pkg-managers use different languages.

Indeed, as i mentioned before, there might exist some dependencies in
the way objects are handled between a GUI and a back-end (Controller),
and its programming language.

> - P: Faster?
> 
I think so.

> 2)
> - P: Different languages can be chosen - and it is easy to add a new
> backend.

This makes a more proper usage of DBus i suppose ; nevertheless here we
could face the problem of DBus not getting or handling languages
features the programmer would like to take advantage of; this also would
involve creating different sub-sets of API for each language to
circumvent some of these issues ... and the thing might get more complex....

> - P: The backend might be provided by the package manager itself, which
> makes it easier to maintain.

The package manager or the GUI front-end?, if its the former, we would
have to talk the different package managers developers into this , and
if it is the later, .... it's pretty much what all the GUI front-ends
are already doing i guess.

> - C: More overhead.
> 
> Open Issues
> ===========
> 
> -  How could we send complex objects with dbus?
> 
> 
> So guys ... now the question: Could you imagine to put efforts together
> and plan such a thing? =)

I don't think it could exist something like 'one back-end to rule them
all', and i barely see the advantage of using something like DBus, but
it could exist some kind of library offering sharable procedures for
different packages managers , and from there on, each GUI package
manager could use whatever fits best its design and model. So, as i
said, it is probably a matter of defining a scope for this idea.

Thanks and Regards,

- --

Luis F. Araujo "araujo at gentoo.org"
Gentoo Linux

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.6 (GNU/Linux)

iD8DBQFG8vO5BCmRZan6aegRAoa9AJ45D5HyGXSghtJYZohxk/XNhdEr1QCghpgD
RlLT3SRt/LZqrVTSmclJjMI=
=xUQe
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Just to bring some new better terms:
- - backend: is the whole thing which has to be written
- - provider: the part between backend and package manager

Luis Francisco Araujo schrieb:
> René 'Necoro' Neumann wrote:
>> While thinking about it, I wondered, why we have so many different
>> portage frontends - and each having its own portage backend. As there
>> are different approaches in defining a GUI, there is only one approach
>> of getting your information from portage. So instead of having one GUI,
> 
> It might theoretically sound like that , but different GUI approaches
> might bring different backend approaches too.

Ok - I only thought about the object oriented approach. That is true.
> 
>> I want to have one backend, which can be used by all different GUIs.
>> This makes the development of new GUIs simple - and might even be useful
>> for small scripts.
> 
>> Also this backend should it make possible to access different package
>> managers (portage, paludis, pkgcore, equo) through one standardized API.
>> Thus in the end all GUIs will support all different Gentoo Package Manglers.
> 
> 
> It really sounds nice, i suppose it'd be a matter of defining what would
> be the exact scope of this back-end. I don't think it might offer all
> the necessary stuff for different package managers to make truly
> independent a GUI from back-end code , nevertheless , it could really
> offer a set of sharable standard procedures (probably as a library?).

Hmm ... this has to be defined. I don't know in detail what are the
differences between the package managers. But I think all of them extend
portage in a way. So one could agree on defining portage as the basic
set of operations supported - and everything else has to be implemented
by the GUI itself.

> 
>> HOW?
>> ====
> 
>> Basically I want to have one service in the middle, that is accessed via
>> DBus. This service itself calls pkg-manager backends which in turn get
>> information out of the pkg manager.
> 
>> For the second part there are two different approaches:
>> 1) Have all backends in the service itself.
> 
> This would be pretty much like a library, there is no need for DBus in
> such a case i think.

The library still has the problem that you have to find a common
demoninator as the language to write the library in... (And it produces
some overhead, as you have to access portage using C for example).

Ok ... you could write e.g. the portage provider in Python and put a
small C layer on top to have the normal bindings...

>> 2)
>> - P: Different languages can be chosen - and it is easy to add a new
>> backend.
> 
> This makes a more proper usage of DBus i suppose ; 
I would still prefer this way - if Dbus hadn't some open issues like
authentication

> nevertheless here we
> could face the problem of DBus not getting or handling languages
> features the programmer would like to take advantage of
You and your Haskell ;P --- but you are true: DBus is object oriented

> this also would involve creating different sub-sets of API for each language to
> circumvent some of these issues ... and the thing might get more complex.... 
no - there would be still be one single API ... the GUI itself then had
to map it to get the way it wants it to be


>> - P: The backend might be provided by the package manager itself, which
>> makes it easier to maintain.
> 
> The package manager or the GUI front-end?, if its the former, we would
> have to talk the different package managers developers into this , and
> if it is the later, .... it's pretty much what all the GUI front-ends
> are already doing i guess.

I meant the "backend providers" for the service to be maintained by the
normal package managers.

Regards,
- - Nec

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG80UK4UOg/zhYFuARAs/RAJ4zoVQv2CwAmVKnKATej/+7CiS7tgCeOHRO
R5qeYPqLctN3UQFuaL9TeGM=
=08iL
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

To bundle the efforts of discussing this topic a little bit, I started
putting some thoughts into a wiki [1]. There exists a forum on the same
page too :)

Please feel free to join the discussion and get some things done ;).

Regards,
- - Necoro

[1] http://portato.origo.ethz.ch/wiki/backend
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHAro74UOg/zhYFuARAmCkAJ9ZjPelBnzvwWpiqSoyZs1GnXFNwACffF6b
Dg+BosJYIjTw4TV5+8AmY2c=
=ajf+
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[Rolf Herzog]

Tobias Scherbaum mentioned in his new book (in german only: Gentoo Linux. The 
Meta-Distribution) that such an API is already planned. He called it EAPI in 
the book and it should give all portage frontends a well defined acces point 
into the emerge system. Has anyone heard about that?
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rolf Herzog schrieb:
> Tobias Scherbaum mentioned in his new book (in german only: Gentoo Linux. The 
> Meta-Distribution) that such an API is already planned. He called it EAPI in 
> the book and it should give all portage frontends a well defined acces point 
> into the emerge system. Has anyone heard about that?
The EAPI I know about only defines the things which can be used in an
ebuild ...
But I will check again =)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHA18c4UOg/zhYFuARApXLAJ9BXqd/xvneaSH3n2i174PTgMhRqwCdFEIn
sanhq3zBCWxzmZsEtjYNAR4=
=hAw/
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann schrieb:
> [1] http://portato.origo.ethz.ch/wiki/backend

First:
- ------

Some changes:
To not mix the portato subversion with the new backend, I created a new
project on origo [1] and copied the wiki pages there.
The project is named "Catapult" until we find a better name ;)

(@dol-sen: I also copied your forum post and added you as a project
member :))

Second:
- -------

I created a small showcase (a server and a client). The details can be
found in [2] - I don't want to copy them :). Inspired by dol-sen I'll
write some tests to see, if dbus is slower than the normal import.

Regards,
- - Necoro

[1] http://catapult.origo.ethz.ch
[2] http://catapult.origo.ethz.ch/node/19
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHA8Ou4UOg/zhYFuARAsZiAJ45Pc8ca9v6qx0ZPXG7YC36l8tBrwCfZsgJ
PT8aZBearu8+dszxbBZslXc=
=2J65
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann schrieb:
> Inspired by dol-sen I'll
> write some tests to see, if dbus is slower than the normal import.
> 
> Regards,
> - Necoro

Ok ... I did some testing, and the results are quite positive ... please
see [1] for results :)

(Excuse me that I'm only posting links, but I don't want to duplicate
information)

Regards,
- - Necoro

[1] http://catapult.origo.ethz.ch/wiki/timing
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHA9al4UOg/zhYFuARAuGRAJ4ztJWE9/CmIFUOdw6ExOosf6iwzgCeKDKT
LWaTaADCur60L0l/XikrFVg=
=uosX
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It's going stepwise forward :) Today during some boring university
classes I created a first provider for portage [1]. It is strongly based
on the old backend for portato -- with other words, the API is far from
being considered stable. If you want to take a look at what the current
API is, you find the files under [1] and (as it is not really
documented) some comments here [2]. The comments are not 1:1, but you
should get the idea :)

Further, I created two ebuilds: "app-portage/catapult" and
"app-portage/catapult-portage". You find all info about them in the wiki
[3].

If you want to check the new provider: I have copied the client for the
showcase and changed it to the provider API. You have two possibilities
to start it:

1.) Go into the provider dir: Launch start.py& and then the client.py
2.) Install the two ebuilds (just do "USE='portage' emerge -av
catapult"), then go into the provider dir and launch client.py (Yes -
the daemon is started automatically ... dbus ftw ;P)

At the beginning of next week, I'm planning to make portato use this new
amazing backend :). We'll see if this is going to work... (Rumors say
that currently dbus times out on the first connect...)

Regards,
- - a tired Nec

[1] https://svn.origo.ethz.ch/catapult/portage
[2]
https://svn.origo.ethz.ch/portato/trunk/portato/backend/system_interface.py
https://svn.origo.ethz.ch/portato/trunk/portato/backend/package.py
[3] http://catapult.origo.ethz.ch/wiki/directory_structure_and_ebuilds
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHBXSm4UOg/zhYFuARAiQpAKCFpQV7wHdv8kvVZqpinnUkVhtnowCfXuS3
ZVSn9oR5pgNnUldZsQCmghI=
=4Uoy
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It's going stepwise forward :) Today during some boring university
classes I created a first provider for portage [1]. It is strongly based
on the old backend for portato -- with other words, the API is far from
being considered stable. If you want to take a look at what the current
API is, you find the files under [1] and (as it is not really
documented) some comments here [2]. The comments are not 1:1, but you
should get the idea :)

Further, I created two ebuilds: "app-portage/catapult" and
"app-portage/catapult-portage". You find all info about them in the wiki
[3].

If you want to check the new provider: I have copied the client for the
showcase and changed it to the provider API. You have two possibilities
to start it:

1.) Go into the provider dir: Launch start.py& and then the client.py
2.) Install the two ebuilds (just do "USE='portage' emerge -av
catapult"), then go into the provider dir and launch client.py (Yes -
the daemon is started automatically ... dbus ftw ;P)

At the beginning of next week, I'm planning to make portato use this new
amazing backend :). We'll see if this is going to work... (Rumors say
that currently dbus times out on the first connect...)

Regards,
- - a tired Nec

[1] https://svn.origo.ethz.ch/catapult/portage
[2]
https://svn.origo.ethz.ch/portato/trunk/portato/backend/system_interface.py
https://svn.origo.ethz.ch/portato/trunk/portato/backend/package.py
[3] http://catapult.origo.ethz.ch/wiki/directory_structure_and_ebuilds
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHBXSm4UOg/zhYFuARAiQpAKCFpQV7wHdv8kvVZqpinnUkVhtnowCfXuS3
ZVSn9oR5pgNnUldZsQCmghI=
=4Uoy
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> 
> At the beginning of next week, I'm planning to make portato use this new
> amazing backend :). We'll see if this is going to work... (Rumors say
> that currently dbus times out on the first connect...)

As promised, I finished a portato version which is using the new
catapult backend... (Btw - the rumor has been proved being wrong ;))

You can get the code doing:
svn co https://svn.origo.ethz.ch/portato/trunk
or you can install portato with catapult support:
1. layman -o http://portato.sourceforge.net/layman.xml -f
2. layman -o http://portato.sourceforge.net/layman.xml -a portato
3. USE="catapult" emerge -av "=portato-9999"


Now another question occurs: Am I the only one interested in this
project? Because there is nearly no feedback/suggestions/discussions
(except with bheekling in IRC)

Regards,
- - Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHCm5m4UOg/zhYFuARAqyDAJ9v+w/zYSO7xlij9c+dk/E3PrhcGQCfZ2Ox
M38GnvkNAKBmugL0yQrEue8=
=0+Gn
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[Dawid Węgliński]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

Dnia 08-10-2007, pon o godzinie 19:52 +0200, René 'Necoro' Neumann
napisał(a):
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> > 
> > At the beginning of next week, I'm planning to make portato use this new
> > amazing backend :). We'll see if this is going to work... (Rumors say
> > that currently dbus times out on the first connect...)
> 
> As promised, I finished a portato version which is using the new
> catapult backend... (Btw - the rumor has been proved being wrong ;))
> 
> You can get the code doing:
> svn co https://svn.origo.ethz.ch/portato/trunk
> or you can install portato with catapult support:
> 1. layman -o http://portato.sourceforge.net/layman.xml -f
> 2. layman -o http://portato.sourceforge.net/layman.xml -a portato
> 3. USE="catapult" emerge -av "=portato-9999"
> 
> 
> Now another question occurs: Am I the only one interested in this
> project? Because there is nearly no feedback/suggestions/discussions
> (except with bheekling in IRC)
> 
> Regards,
> - - Necoro
From my side, i'm following your work and your posts, but currently i
have other work to do, as p.g.o or my gentoo-x86 access, so i can't help
you in anyway. The second is, i'm not python coder. I bet jokey or
araujo also have some work to do atm :)

Cheers

PS. And personally i think you do great job. Keep it going.
-- 
,-----------------------------.
| Dawid Węgliński             |
| cla@gentoo.org              |
| cla @ irc.freenode.net      |
| GPG: 295E72D9               |
`-----------------------------'


[-- Attachment #2: To jest część listu podpisana cyfrowo --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dawid Węgliński schrieb:
> From my side, i'm following your work and your posts, but currently i
> have other work to do, as p.g.o or my gentoo-x86 access, so i can't help
> you in anyway. The second is, i'm not python coder. I bet jokey or
> araujo also have some work to do atm :)

Hehe ... good points :)
To get rid of the python dependency (for understanding), I'll do some
API-documentation tomorrow (in the disgusting "Consulting and Technical
Sales" class =|)

> 
> Cheers
> 
> PS. And personally i think you do great job. Keep it going.

Thanks - I'll try =) (at the moment I have plenty of time ... will
change somewhere in the future)

Regards,
- - Necoro

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHCoIk4UOg/zhYFuARAs3rAJwPAk96Jm5/p/+MnRlMHVYZN0Zg8gCeIA4r
b6vyAZVEH1HJm/49x7SNQC8=
=WYb5
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[Brian]

On Mon, 2007-08-10 at 19:52 +0200, René 'Necoro' Neumann wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> > 
> > At the beginning of next week, I'm planning to make portato use this new
> > amazing backend :). We'll see if this is going to work... (Rumors say
> > that currently dbus times out on the first connect...)
> 
> As promised, I finished a portato version which is using the new
> catapult backend... (Btw - the rumor has been proved being wrong ;))
> 
> You can get the code doing:
> svn co https://svn.origo.ethz.ch/portato/trunk
> or you can install portato with catapult support:
> 1. layman -o http://portato.sourceforge.net/layman.xml -f
> 2. layman -o http://portato.sourceforge.net/layman.xml -a portato
> 3. USE="catapult" emerge -av "=portato-9999"
> 
> 
> Now another question occurs: Am I the only one interested in this
> project? Because there is nearly no feedback/suggestions/discussions
> (except with bheekling in IRC)
> 
> Regards,
> - - Necoro

No, I am interested in working with you to develop an (hopefully)
universal back-end.  Like others have said, I have been busy and not had
more time to work on it.  I also need to finish some things and a few
bug fixes to get a release out.  Any new back-end support will need to
wait for a future release.


I almost sent a reply early this weekend, but you did say you would not
be starting until this week and that you wanted to do some testing to
prove or debunk some rumours as to it's performance.

I am glad that you have been able to debunk them :)

Now for some more nitty gritty things.  So far we have not
discussed/decided which of the models you laid out we would base this
new back-end on.  So far it is just portato's existing back-end wrapped
in dbus.  I think it is great for initial testing without expending a
lot of effort in restructuring any front-end code.  But I do not think
it would be usable for me at all.  Porthole has grown so much over the
years that it's code was getting somewhat like portage's, huge files.
For this next version I have spilt things up quite a bit, modularizing
things better.  Have cut down memory usage quite a bit and sped things
up quite a bit in the process.


One of the things that I did was to create a separate db module. In the
past much of it was incorporated into our portagelib.  That makes it
more difficult to change back-end, package managers and update code for
changes from upstream.  So far your catapult back-end is creating a
package object for your front-end.  That may be fine for a fully
integrated program, but now you are separating it out into it's own
process and passing that structure to your front-end.  I don't know if
dbus is passing references (pointers) or making copies.  I think that
there is potential for large memory leaks that way.  Also portholes
definition of a package object is different than portato's as I'm sure
kuroo's and himerge's is.  I believe that the back-end should be
restricted to only interfacing the front-end to the package manager
enquiries along with some utility code for odds and ends to provide a
more complete back-end service.  By odds and ends I mean code chunks
needed to be able to provide missing features/functions of the different
package managers we may support.

I think that pothole's potagelib.py contents is more of what a back-end
should provide.  I'll be the first to admit (I'm biased) it needs work
and cleanup and there is room for it to grow/improve.  I do not think
that it should be providing package structures with embedded package
manager calls.  I think it should be restricted to the basic data types
returned by the package managers.  Any more complex structures should be
handled by the front-end code or any intermediate code it uses.

Anyway... my thoughts so far.  How about the others?  What do you see as
your needs of a back-end?
 

Another question, I have subscribed to the gentoo-guis list.  Is
everyone else that is interested also?  Should we just use that list for
now?  So I/we don't get 4 or five copies of an email from different
directions.

As for IRC, I'm not that big on it.  Also I'm in the Canada/US Pacific
time zone,  Necoro I believe is in Europe.  That usually means when I'm
going to bed, he's just coming online and vice versa.

-- 
Brian <dol-sen@telus.net>

-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian schrieb:
> On Mon, 2007-08-10 at 19:52 +0200, René 'Necoro' Neumann wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>> At the beginning of next week, I'm planning to make portato use this new
>>> amazing backend :). We'll see if this is going to work... (Rumors say
>>> that currently dbus times out on the first connect...)
>> As promised, I finished a portato version which is using the new
>> catapult backend... (Btw - the rumor has been proved being wrong ;))
>>
>> You can get the code doing:
>> svn co https://svn.origo.ethz.ch/portato/trunk
>> or you can install portato with catapult support:
>> 1. layman -o http://portato.sourceforge.net/layman.xml -f
>> 2. layman -o http://portato.sourceforge.net/layman.xml -a portato
>> 3. USE="catapult" emerge -av "=portato-9999"
>>
>>
>> Now another question occurs: Am I the only one interested in this
>> project? Because there is nearly no feedback/suggestions/discussions
>> (except with bheekling in IRC)
>>
>> Regards,
>> - - Necoro
> 
> No, I am interested in working with you to develop an (hopefully)
> universal back-end.  Like others have said, I have been busy and not had
> more time to work on it.  I also need to finish some things and a few
> bug fixes to get a release out.  Any new back-end support will need to
> wait for a future release.

Yeah - sorry me =| ... I've been a little too impatient :)
> 
> 
> I almost sent a reply early this weekend, but you did say you would not
> be starting until this week and that you wanted to do some testing to
> prove or debunk some rumours as to it's performance.
> 
> I am glad that you have been able to debunk them :)
> 
> Now for some more nitty gritty things.  So far we have not
> discussed/decided which of the models you laid out we would base this
> new back-end on.
We decided (where "we" means: bheekling, araujo and me) to use (try?)
the structure mentioned here [1] as "No Daemon".

> So far it is just portato's existing back-end wrapped
> in dbus.  I think it is great for initial testing without expending a
> lot of effort in restructuring any front-end code.  But I do not think
> it would be usable for me at all.  Porthole has grown so much over the
> years that it's code was getting somewhat like portage's, huge files.
> For this next version I have spilt things up quite a bit, modularizing
> things better.  Have cut down memory usage quite a bit and sped things
> up quite a bit in the process.
> 
> 
> So far your catapult back-end is creating a
> package object for your front-end.  
Wrong... basically the "Package" and "System" classes are just a bunch
of functions. They do not behave in an object-oriented manner (ie. you
have to provide a CPV (cat/pkg-ver) for each call to the Package
object). So this really is functional. Perhaps they can be merged into
one object - just wanted to structurize a little bit.
I thought of doing it really object-oriented, i.e. creating a
dbus-object for each package in the tree (so you would then have for
example: "org/gentoo/catapult/packages/sys-apps/portage") but this kinda
screwed up dbus ;).

Portato (which supported switching backends for some time already) is
wrapping these functional calls into its own objects... see the
CatapultPackage object [2] as an example.

> That may be fine for a fully
> integrated program, but now you are separating it out into it's own
> process and passing that structure to your front-end.  I don't know if
> dbus is passing references (pointers) or making copies.  I think that
> there is potential for large memory leaks that way.  Also portholes
> definition of a package object is different than portato's as I'm sure
> kuroo's and himerge's is.
Does not apply -- see above :)

> I believe that the back-end should be
> restricted to only interfacing the front-end to the package manager
> enquiries along with some utility code for odds and ends to provide a
> more complete back-end service.  By odds and ends I mean code chunks
> needed to be able to provide missing features/functions of the different
> package managers we may support.

Yes - this is what we want to achieve.
What we need is a stable API - so that we can say: All backends have to
provide the following functions: ...

> 
> I think that pothole's potagelib.py contents is more of what a back-end
> should provide.  I'll be the first to admit (I'm biased) it needs work
> and cleanup and there is room for it to grow/improve.  I do not think
> that it should be providing package structures with embedded package
> manager calls.  I think it should be restricted to the basic data types
> returned by the package managers.  Any more complex structures should be
> handled by the front-end code or any intermediate code it uses.

See 3 above ;)

> Anyway... my thoughts so far.  How about the others?  What do you see as
> your needs of a back-end?
>  
> 
> Another question, I have subscribed to the gentoo-guis list.  Is
> everyone else that is interested also?  Should we just use that list for
> now?  So I/we don't get 4 or five copies of an email from different
> directions.

I already removed you from the CC list. Should the Portato-Developers be
removed too?

> 
> As for IRC, I'm not that big on it.  Also I'm in the Canada/US Pacific
> time zone,  Necoro I believe is in Europe.  That usually means when I'm
> going to bed, he's just coming online and vice versa.
> 
*stabs the freaking timezones*

Regards,
- - Necoro

[1] http://catapult.origo.ethz.ch/wiki/structure
[2]
http://svn.origo.ethz.ch/wsvn/portato/trunk/portato/backend/catapult/package.py
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHC26Z4UOg/zhYFuARArXYAJ47EhHSWgynsdsDzYPc+XBGOvedPQCghYQj
10G/yRYmYS0Ukd1Q+wAYEDo=
=0TFY
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> Another question, I have subscribed to the gentoo-guis list.  Is
>> everyone else that is interested also?  Should we just use that list for
>> now?  So I/we don't get 4 or five copies of an email from different
>> directions.
> 
> I already removed you from the CC list. Should the Portato-Developers be
> removed too?
> 

Should read "Porthole-Developers" of course ... *slaps Freud*
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHC4h44UOg/zhYFuARAtGeAJ44ZZZiY+VvxyvZgyn3BKgZUrtRkwCfb8g9
4IPT6il5Hq5tVrhkNIcQbks=
=WQrf
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[Luis Francisco Araujo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian wrote:
> On Mon, 2007-08-10 at 19:52 +0200, René 'Necoro' Neumann wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>> At the beginning of next week, I'm planning to make portato use this new
>>> amazing backend :). We'll see if this is going to work... (Rumors say
>>> that currently dbus times out on the first connect...)
>> As promised, I finished a portato version which is using the new
>> catapult backend... (Btw - the rumor has been proved being wrong ;))
>>
>> You can get the code doing:
>> svn co https://svn.origo.ethz.ch/portato/trunk
>> or you can install portato with catapult support:
>> 1. layman -o http://portato.sourceforge.net/layman.xml -f
>> 2. layman -o http://portato.sourceforge.net/layman.xml -a portato
>> 3. USE="catapult" emerge -av "=portato-9999"
>>
>>
>> Now another question occurs: Am I the only one interested in this
>> project? Because there is nearly no feedback/suggestions/discussions
>> (except with bheekling in IRC)
>>
>> Regards,
>> - - Necoro
> 
> One of the things that I did was to create a separate db module. In the
> past much of it was incorporated into our portagelib.  That makes it
> more difficult to change back-end, package managers and update code for
> changes from upstream.  So far your catapult back-end is creating a
> package object for your front-end.  That may be fine for a fully
> integrated program, but now you are separating it out into it's own
> process and passing that structure to your front-end.  I don't know if
> dbus is passing references (pointers) or making copies.  I think that
> there is potential for large memory leaks that way.  Also portholes
> definition of a package object is different than portato's as I'm sure
> kuroo's and himerge's is.  I believe that the back-end should be
> restricted to only interfacing the front-end to the package manager
> enquiries along with some utility code for odds and ends to provide a
> more complete back-end service.  By odds and ends I mean code chunks
> needed to be able to provide missing features/functions of the different
> package managers we may support.
> 

That's right, different front-ends have different package object
representation. One of my idea was to write an initial library (probably
on C so it can easily be used through many languages) , creating a set
of sharable procedures between front-ends, this also could give us some
ideas of how far a general package object representation is possible
between the different front-ends ; i think that'd be a good way to start
building this project.

> I think that pothole's potagelib.py contents is more of what a back-end
> should provide.  I'll be the first to admit (I'm biased) it needs work
> and cleanup and there is room for it to grow/improve.  I do not think
> that it should be providing package structures with embedded package
> manager calls.  I think it should be restricted to the basic data types
> returned by the package managers.  Any more complex structures should be
> handled by the front-end code or any intermediate code it uses.
> 

Right, I also agree that this back-end should only generate general data
that can be easily shared by many front-ends so they can handle these
objects in whatever way fits better for each one. Now, the question is,
what kind of data exactly would this be?, it's here where i think the
library project would help to figure this out. But as to give an initial
idea, this library could be usable by any kind of portage front-end (no
only graphical) and should be easily used through different tools.

> Anyway... my thoughts so far.  How about the others?  What do you see as
> your needs of a back-end?
>  
> 
> Another question, I have subscribed to the gentoo-guis list.  Is
> everyone else that is interested also?  Should we just use that list for
> now?  So I/we don't get 4 or five copies of an email from different
> directions.
> 

Yes, We are using this mailing list to discuss this project, so, anyone
interested on this subject, please subscribe to it.

> As for IRC, I'm not that big on it.  Also I'm in the Canada/US Pacific
> time zone,  Necoro I believe is in Europe.  That usually means when I'm
> going to bed, he's just coming online and vice versa.
> 

Don't worry about that, everyone is spread around the globe, if you have
some time, try to stop by it too and say hi :-)

Regards,

- --

Luis F. Araujo "araujo at gentoo.org"
Gentoo Linux

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHC/OVBCmRZan6aegRAh0OAJ0Y5Md+nl+B8z7v0iwUnzJWWtVw1ACgsySa
cUbTLZn4gUV6r0/NOW/56KM=
=YfPV
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luis Francisco Araujo schrieb:
> Brian wrote:
>> One of the things that I did was to create a separate db module. In the
>> past much of it was incorporated into our portagelib.  That makes it
>> more difficult to change back-end, package managers and update code for
>> changes from upstream.  So far your catapult back-end is creating a
>> package object for your front-end.  That may be fine for a fully
>> integrated program, but now you are separating it out into it's own
>> process and passing that structure to your front-end.  I don't know if
>> dbus is passing references (pointers) or making copies.  I think that
>> there is potential for large memory leaks that way.  Also portholes
>> definition of a package object is different than portato's as I'm sure
>> kuroo's and himerge's is.  I believe that the back-end should be
>> restricted to only interfacing the front-end to the package manager
>> enquiries along with some utility code for odds and ends to provide a
>> more complete back-end service.  By odds and ends I mean code chunks
>> needed to be able to provide missing features/functions of the different
>> package managers we may support.
> 
> 
> That's right, different front-ends have different package object
> representation. One of my idea was to write an initial library (probably
> on C so it can easily be used through many languages) , creating a set
> of sharable procedures between front-ends, this also could give us some
> ideas of how far a general package object representation is possible
> between the different front-ends ; i think that'd be a good way to start
> building this project.

This is the thing I'm waiting for atm ... the functions to implement (ie
the API)... I think, that it should be done in a functional way is
general consensus, isn't it?

> 
>> I think that pothole's potagelib.py contents is more of what a back-end
>> should provide.  I'll be the first to admit (I'm biased) it needs work
>> and cleanup and there is room for it to grow/improve.  I do not think
>> that it should be providing package structures with embedded package
>> manager calls.  I think it should be restricted to the basic data types
>> returned by the package managers.  Any more complex structures should be
>> handled by the front-end code or any intermediate code it uses.
> 
> 
> Right, I also agree that this back-end should only generate general data
> that can be easily shared by many front-ends so they can handle these
> objects in whatever way fits better for each one. Now, the question is,
> what kind of data exactly would this be?, it's here where i think the
> library project would help to figure this out. But as to give an initial
> idea, this library could be usable by any kind of portage front-end (no
> only graphical) and should be easily used through different tools.

This is the point where dbus comes in handy ... it can be used by all
languages having dbus bindings (and if you don't have them - use the
ones in C and do some wrapping) -- even in shell scripts (using the
program "dbus-send"). But all depends on one thing: a stable API (sorry
if I'm repeating myself ...).

And we should not forget to implement the API in a way to allow
API-versioning (i.e. to be able to say: "I'm using Catapult-APIv0" and
the provider behaves in the right way).

Regards,
- - Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHC/ir4UOg/zhYFuARAohJAJ0e7BLLh9M+0csd12XKiqyu44o0VQCdHD1k
4spX4jB9eUN482eICrnSCoM=
=Qbfd
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[Brian]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

On Tue, 2007-09-10 at 23:54 +0200, René 'Necoro' Neumann wrote:

> This is the thing I'm waiting for atm ... the functions to implement (ie
> the API)... I think, that it should be done in a functional way is
> general consensus, isn't it?
> 

Ok, I've put together a small list of portage calls mostly unigue, but
many have variations of the basic call (not listed)repeated for
different info returned. A few lines of code I left in to make it more
clear, but should give an overall list of the portage function calls
used so far.


I will put together a list of functions that I created in portagelib to
add functionality directly related to portage.  Some of them are copies
of code chunks from lifted from gentoolkit.

> Regards,
> - - Necoro

-- 
Brian <dol-sen@telus.net>

[-- Attachment #2: portage_calls --]
[-- Type: text/plain, Size: 2196 bytes --]

try:
    import portage
    import portage_const
    import portage_manifest
    print >>stderr, ("PORTAGELIB: portage version = " + portage.VERSION)
except ImportError:
    
return portage.settings.virtuals
portage.settings["ARCH"]
portage.catpkgsplit
portage.pkgsplit(ebuild)
portage.grabfile
portage.config(clone=portage.settings).environ()[var]
return portage.db['/']['vartree'].dep_match(str(package_name))
return portage.portdb.xmatch(*args, **kwargs)[:] # make a copy.  needed for <portage-svn-r5382
for x in portage.portdb.mysettings.pmaskdict[full_name]:
for x in portage.portdb.mysettings.punmaskdict[full_name]:
return portage.best(versions)
portage.archlist[:]
portage.getmaskingreason(ebuild)
portdb.findname(mycpv)
mf = portage_manifest.Manifest(pkgdir, settings["DISTDIR"])
    fetchlist = portdb.getfetchlist(mycpv, mysettings=settings, all=True)[1]
mysum[0] = mf.getDistfilesSize(fetchlist)
portage.db['/']['porttree'].dbapi.finddigest(ebuild)
if portage.portdb.cpv_exists(ebuild): # if in portage tree
Properties(dict(zip(keys, portage.portdb.aux_get(ebuild, portage.auxdbkeys))))
        vartree = portage.db['/']['vartree']
        if vartree.dbapi.cpv_exists(ebuild): # elif in installed pkg tree
            return Properties(dict(zip(keys, vartree.dbapi.aux_get(ebuild, portage.auxdbkeys))))
        else: return Properties()
non_virtual_atom = portage.dep_virtual([atom], portage.settings)[0]
dir,ovl = portage.portdb.findname2(cpv)
def get_system_pkgs(): # lifted from gentoolkit
	"""Returns a tuple of lists, first list is resolved system packages,
	second is a list of unresolved packages."""
	pkglist = settings.packages
return portage.db['/']['porttree'].getallnodes()[:] # copy
return portage.db['/']['vartree'].getallnodes()[:] # try copying...
return portage.db['/']['vartree'].getebuildpath(fullname)
trees = portage.create_trees(trees=trees, **kwargs)
settings = trees[myroot]["vartree"].settings
    mtimedbfile = os.path.join("/", portage.CACHE_PATH.lstrip(os.path.sep), "mtimedb")
    mtimedb = portage.MtimeDB(mtimedbfile)
    return settings, trees, mtimedb
settings, trees, mtimedb = load_emerge_config()
portdb = trees[settings["ROOT"]]["porttree"].dbapi

Re: [gentoo-guis] One backend for all portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian schrieb:
> On Tue, 2007-09-10 at 23:54 +0200, René 'Necoro' Neumann wrote:
> 
>> This is the thing I'm waiting for atm ... the functions to implement (ie
>> the API)... I think, that it should be done in a functional way is
>> general consensus, isn't it?
>>
> 
> Ok, I've put together a small list of portage calls mostly unigue, but
> many have variations of the basic call (not listed)repeated for
> different info returned. A few lines of code I left in to make it more
> clear, but should give an overall list of the portage function calls
> used so far.
> 
> 
> I will put together a list of functions that I created in portagelib to
> add functionality directly related to portage.  Some of them are copies
> of code chunks from lifted from gentoolkit.
> 
>> Regards,
>> - - Necoro
> 
Thanks =)

I've put it into the Wiki [1] too. :)

I'll try again today to create some API documentation and will merge
your requirements.

- - Necoro

[1] http://catapult.origo.ethz.ch/wiki/API
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDGRO4UOg/zhYFuARAjFZAJsFL+zyl+msKMqXj8HL0Mm8vmjVGACfQcyn
+0lokSd1G+M6oEpEcieUqIM=
=FSjp
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[René 'Necoro' Neumann]

[-- Attachment #1: Type: text/plain, Size: 2728 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann schrieb:
> Brian schrieb:
>> On Tue, 2007-09-10 at 23:54 +0200, René 'Necoro' Neumann wrote:
> 
>>> This is the thing I'm waiting for atm ... the functions to implement (ie
>>> the API)... I think, that it should be done in a functional way is
>>> general consensus, isn't it?
>>>
>> Ok, I've put together a small list of portage calls mostly unigue, but
>> many have variations of the basic call (not listed)repeated for
>> different info returned. A few lines of code I left in to make it more
>> clear, but should give an overall list of the portage function calls
>> used so far.
> 
> 
>> I will put together a list of functions that I created in portagelib to
>> add functionality directly related to portage.  Some of them are copies
>> of code chunks from lifted from gentoolkit.
> 
>>> Regards,
>>> - - Necoro
> Thanks =)
> 
> I've put it into the Wiki [1] too. :)
> 
> I'll try again today to create some API documentation and will merge
> your requirements.

Ok - I've added the current API to the wiki [1]. I did not manage to
include all of the needs of porthole, as I don't know what some of the
calls do, and whether they are really needed ;).
Please see the attached file for the comments.

@dol-sen: Would be great if you could scan through the list and see,
which ones are really needed. Because portage calls inside functions
that are then moved into catapult don't need to have an equivalent in
the API (at least in most cases). - And for the vast majority I was just
clueless ^^. (and too lazy to lookup portage code).

There are certain issues with the current API:

1.) Add some more stuff needed by other frontends.
2.) Perhaps merge the Package and the System object as they both just
provide functions.
3.) Merge package.is_in_overlay() and package.get_overlay_path()
4.) Get rid of the different get_*_settings functions and add special
ones: get_homepage, get_depend(which), get_arch, ...
5.) Currently there are two functions returning use flags which were set
at installation time of a package:
	- get_use_flags(): Returning ALL set useflags
	- get_installed_use_flags(): Return only the set useflags which are
also in IUSE.
This should perhaps be merged into one function with an attribute
"only_iuse" or similar.

Some comments are appreciated ;)

- - Necoro

[1] http://catapult.origo.ethz.ch/wiki/API
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDRAq4UOg/zhYFuARAmyxAJ4jqUKg8pAIi6AGv/QYcGZ4XOagkACeOFJK
9XOv2sBduIknjtsKv8CbYqM=
=1FVU
-----END PGP SIGNATURE-----

[-- Attachment #2: portage_calls_porthole_commented.txt --]
[-- Type: text/plain, Size: 2456 bytes --]

# already implemented
portage.settings["ARCH"] # system.get_global_settings("ARCH")
portage.catpkgsplit # system.split_cpv()
return portage.best(versions) # system.find_best()
portage.getmaskingreason(ebuild) # package.get_masking_reason()
if portage.portdb.cpv_exists(ebuild): # package.is_in_system()
if vartree.dbapi.cpv_exists(ebuild): # package.is_installed()
dir,ovl = portage.portdb.findname2(cpv) # package.is_in_overlay(); package.get_overlay_path()
return portage.db['/']['vartree'].getebuildpath(fullname) # package.get_ebuild_path()
portdb.findname(mycpv) # package.get_ebuild_path()

# needs to be implemented
return portage.settings.virtuals # ??

# unknown; guess is that it needs to be implemented
portage.pkgsplit(ebuild) # what does this do?
portage.archlist[:] # ??
portage.config(clone=portage.settings).environ()[var] # ??
return portage.db['/']['vartree'].dep_match(str(package_name)) # ??
return portage.portdb.xmatch(*args, **kwargs)[:] # ??
fetchlist = portdb.getfetchlist(mycpv, mysettings=settings, all=True)[1] # ???
mysum[0] = mf.getDistfilesSize(fetchlist) # ???
portage.db['/']['porttree'].dbapi.finddigest(ebuild) # ???
non_virtual_atom = portage.dep_virtual([atom], portage.settings)[0] # ??

# unknown
mf = portage_manifest.Manifest(pkgdir, settings["DISTDIR"]) # ????
return portage.db['/']['porttree'].getallnodes()[:] # ??
return portage.db['/']['vartree'].getallnodes()[:] # ??
mtimedbfile = os.path.join("/", portage.CACHE_PATH.lstrip(os.path.sep), "mtimedb") # ??
mtimedb = portage.MtimeDB(mtimedbfile) # ??
settings, trees, mtimedb = load_emerge_config() # ??

# should not be needed, as they are catapult internal and there should be no reason to use them in the frontend
for x in portage.portdb.mysettings.pmaskdict[full_name]: # do we need this?
for x in portage.portdb.mysettings.punmaskdict[full_name]: # do we need this?
Properties(dict(zip(keys, portage.portdb.aux_get(ebuild, portage.auxdbkeys)))) # use the get_*_settings?
return Properties(dict(zip(keys, vartree.dbapi.aux_get(ebuild, portage.auxdbkeys)))) # use the get_*_settings?
vartree = portage.db['/']['vartree'] # not needed -> should be handled internally; apps should not need vartree
trees = portage.create_trees(trees=trees, **kwargs) # not needed
settings = trees[myroot]["vartree"].settings # not needed
portdb = trees[settings["ROOT"]]["porttree"].dbapi # not needed

# other
portage.grabfile # write yourself or move into library

Re: [gentoo-guis] One backend for all portage GUIs

[Brian]

On Wed, 2007-10-10 at 19:47 +0200, René 'Necoro' Neumann wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> René 'Necoro' Neumann schrieb:
> > Brian schrieb:
> >> On Tue, 2007-09-10 at 23:54 +0200, René 'Necoro' Neumann wrote:
> > 
> >>> This is the thing I'm waiting for atm ... the functions to implement (ie
> >>> the API)... I think, that it should be done in a functional way is
> >>> general consensus, isn't it?
> >>>
> >> Ok, I've put together a small list of portage calls mostly unigue, but
> >> many have variations of the basic call (not listed)repeated for
> >> different info returned. A few lines of code I left in to make it more
> >> clear, but should give an overall list of the portage function calls
> >> used so far.
> > 
> > 
> >> I will put together a list of functions that I created in portagelib to
> >> add functionality directly related to portage.  Some of them are copies
> >> of code chunks from lifted from gentoolkit.
> > 
> >>> Regards,
> >>> - - Necoro
> > Thanks =)
> > 
> > I've put it into the Wiki [1] too. :)
> > 
> > I'll try again today to create some API documentation and will merge
> > your requirements.
> 
> Ok - I've added the current API to the wiki [1]. I did not manage to
> include all of the needs of porthole, as I don't know what some of the
> calls do, and whether they are really needed ;).
> Please see the attached file for the comments.
> 
> @dol-sen: Would be great if you could scan through the list and see,
> which ones are really needed. Because portage calls inside functions
> that are then moved into catapult don't need to have an equivalent in
> the API (at least in most cases). - And for the vast majority I was just
> clueless ^^. (and too lazy to lookup portage code).


Yeah, I know...  I made the list only for reference as to the calls to
portage used.  To know what they are used for, etc. you need to go thru
portagelib.  Many of the ones needed you already have blank defs for and
raise NotImplemented ERROR.  I didn't expect you to implement them all
now anyway :)


> 
> There are certain issues with the current API:
> 
> 1.) Add some more stuff needed by other frontends.
> 2.) Perhaps merge the Package and the System object as they both just
> provide functions.

I had a chance to look over your code a little more.  I think it would
be less confusing if some of the files and classes were renamed.  You've
used package.py and class XXXPackage several times.  It gets confusing
with a frontend's Package class...   Since they are just function groups
then perhaps 

class CPVFunctions

Anyway, I have some ideas on how we might organize groups of functions,
but I need more time to think it thru and get them down in print. 

> 3.) Merge package.is_in_overlay() and package.get_overlay_path()
> 4.) Get rid of the different get_*_settings functions and add special
> ones: get_homepage, get_depend(which), get_arch, ...
> 5.) Currently there are two functions returning use flags which were set
> at installation time of a package:
> 	- get_use_flags(): Returning ALL set useflags
> 	- get_installed_use_flags(): Return only the set useflags which are
> also in IUSE.
> This should perhaps be merged into one function with an attribute
> "only_iuse" or similar.
> 
> Some comments are appreciated ;)
> 
> - - Necoro

Sorry, I have to work again tonight.  I won't have time to look yet.
Only enough time to respond with this email.

-- 
Brian <dol-sen@telus.net>

-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for all portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian schrieb:
> On Wed, 2007-10-10 at 19:47 +0200, René 'Necoro' Neumann wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> René 'Necoro' Neumann schrieb:
>>
>> Ok - I've added the current API to the wiki [1]. I did not manage to
>> include all of the needs of porthole, as I don't know what some of the
>> calls do, and whether they are really needed ;).
>> Please see the attached file for the comments.
>>
>> @dol-sen: Would be great if you could scan through the list and see,
>> which ones are really needed. Because portage calls inside functions
>> that are then moved into catapult don't need to have an equivalent in
>> the API (at least in most cases). - And for the vast majority I was just
>> clueless ^^. (and too lazy to lookup portage code).
> 
> 
> Yeah, I know...  I made the list only for reference as to the calls to
> portage used.  To know what they are used for, etc. you need to go thru
> portagelib.  Many of the ones needed you already have blank defs for and
> raise NotImplemented ERROR.  I didn't expect you to implement them all
> now anyway :)
> 

Oh oh ... it seems like we are mixing things up here ... Please do not
refer to the Portato Code anymore. The only important thing is the
catapult code [1] and the API documentation. Sorry if I confused you all  =|

> 
>> There are certain issues with the current API:
>>
>> 1.) Add some more stuff needed by other frontends.
>> 2.) Perhaps merge the Package and the System object as they both just
>> provide functions.
> 
> I had a chance to look over your code a little more.  I think it would
> be less confusing if some of the files and classes were renamed.  You've
> used package.py and class XXXPackage several times.  It gets confusing
> with a frontend's Package class...   Since they are just function groups
> then perhaps 
> 
> class CPVFunctions
> 
> Anyway, I have some ideas on how we might organize groups of functions,
> but I need more time to think it thru and get them down in print. 
> 

As I said: Ignore portato's code. ;). And the naming of the classes is
really arbitrary - we might change this if you want. What is more
important is the DBus-Object Path...

>> 3.) Merge package.is_in_overlay() and package.get_overlay_path()
>> 4.) Get rid of the different get_*_settings functions and add special
>> ones: get_homepage, get_depend(which), get_arch, ...
>> 5.) Currently there are two functions returning use flags which were set
>> at installation time of a package:
>> 	- get_use_flags(): Returning ALL set useflags
>> 	- get_installed_use_flags(): Return only the set useflags which are
>> also in IUSE.
>> This should perhaps be merged into one function with an attribute
>> "only_iuse" or similar.
>>
>> Some comments are appreciated ;)
>>
>> - - Necoro
> 
> Sorry, I have to work again tonight.  I won't have time to look yet.
> Only enough time to respond with this email.
> 

- - Necoro

[1] https://svn.origo.ethz.ch/catapult/portage/catapult_portage/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHDbQd4UOg/zhYFuARAi4TAJ45M1hT94FiRmxnZZymu9ID5NcYIACeI6ZC
KYfWtnM1mPRY5w5YygBc8xs=
=1KFQ
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ok ... some new points =):

1.) The dbus daemon is running the whole time the system is up, when
started once. Therefore the settings should be reloaded each time a
config file (/etc/portage/*, /etc/make.conf ...) is changed. I thought
of using inotify for this issue. Unfortunately, a well known editor
creates some problems: ViM (and I guess other editors behave similary):
It messes around and creates swap-files, temporary files - and instead
of modifying a file directly, it first moves it, then creates a new
modified one and finally deletes the old one (thus the inotify-watch is
gone afterwards).
The creation of the swap files is an issue, when watching a directory
for changes (e.g. /etc/portage). As the files are created _always_, the
settings would be reloaded even if the user only reads the files (and
another time, when he closes it, as the swap files are deleted). This is
definitly a non-optimal solution.

2.) If noone objects in the following days, I would like to write to the
portage-dev-ml about this project (or better: the plans of the project)
and perhaps get some proposals from this side :)

3.) As it is quite some work to maintain two versions of the
portage-backend, I plan to use catapult in Portato by default. But this
depends on how soon we find a solution for 1.).

Regards,
- -- Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHHNNX4UOg/zhYFuARAgz9AJ9ZHoNuGDoPnrZ/EySe28abPw6R5gCeLT3Y
QrwuokUomXr121St9QnCnpE=
=9bff
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey,

I have to admit: Catapult did not work as expected. I wrote, that I used
it in Portato. But this wasn't correct as I saw today - after doing two
dbus calls he switched to the normal backend. I fixed that and
discovered several bugs, which got fixed. I also managed to put the
"update_world" call into an extra thread, so it does not block the whole
daemon if it is called. - More calls will follow. I don't know if it is
a good idea to do so with all calls (overhead?) or if only selected
calls should get their own private thread.

Regards,
Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHKmps4UOg/zhYFuARAjKgAJ99uFXBF3QK5Usr1dPsmeYyBCUYYwCfVOEI
gdifp9O7tFkAPi5jcCwQoEc=
=h0En
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[Luis Francisco Araujo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann wrote:
> Hey,
> 
> I have to admit: Catapult did not work as expected. I wrote, that I used
> it in Portato. But this wasn't correct as I saw today - after doing two
> dbus calls he switched to the normal backend. I fixed that and
> discovered several bugs, which got fixed. I also managed to put the
> "update_world" call into an extra thread, so it does not block the whole
> daemon if it is called. - More calls will follow. I don't know if it is
> a good idea to do so with all calls (overhead?) or if only selected
> calls should get their own private thread.
> 
> Regards,
> Necoro

Can you explain what problems did you have?

I just could glanced over some irc logs saying that was because of DBus
not being smart enough to handle different process in parallel?

Regards,

- --

Luis F. Araujo "araujo at gentoo.org"
Gentoo Linux

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHKyz8BCmRZan6aegRAk2jAKCWP20UkS4l8cDkwoloXlczHrQr5QCg04Yx
4uNkxohbnqd211HwLsZLuZA=
=JD+1
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luis Francisco Araujo schrieb:
> René 'Necoro' Neumann wrote:
>> Hey,
> 
>> I have to admit: Catapult did not work as expected. I wrote, that I used
>> it in Portato. But this wasn't correct as I saw today - after doing two
>> dbus calls he switched to the normal backend. I fixed that and
>> discovered several bugs, which got fixed. I also managed to put the
>> "update_world" call into an extra thread, so it does not block the whole
>> daemon if it is called. - More calls will follow. I don't know if it is
>> a good idea to do so with all calls (overhead?) or if only selected
>> calls should get their own private thread.
> 
>> Regards,
>> Necoro
> 
> Can you explain what problems did you have?
> 
> I just could glanced over some irc logs saying that was because of DBus
> not being smart enough to handle different process in parallel?
> 
> Regards,
> 
The main problem was as following:

If client A starts a time consuming process (e.g. update_world), the
server blocks all following requests (regardless which client has sent
them) until the process has finished.
This could be solved by making the call asynchronous on server site:
The function gets two callbacks (one for success, one for error). Then
it can launch a thread doing the calculation which calls the appropriate
callback eventually. So the function can return immediately and other
calls are not blocked.

Finding this was not the problem - it just did not work. After two hours
I finally figured out, that I have to explicitly allow threads for glib.
So this was more kind of a python/glib problem and not a dbus one.

A second problem:
If a client calls a time consuming process, it could happen that the
dbus call times out (i.e. the server takes too long to get the answer).
This can be easily solved by making the call asynchronous on client site. :)

Another problems/bugs were minor and trivial ones.

Regards,
- - Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHK2MG4UOg/zhYFuARAuRoAJwMuyntpRlLqoSMduEljanWnRvq+gCeLIyL
cmzlowE0nAm612gjPoM6fvM=
=oFtc
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann schrieb:
> Ok ... some new points =):
> 
> 1.) The dbus daemon is running the whole time the system is up, when
> started once. Therefore the settings should be reloaded each time a
> config file (/etc/portage/*, /etc/make.conf ...) is changed. I thought
> of using inotify for this issue. Unfortunately, a well known editor
> creates some problems: ViM (and I guess other editors behave similary):
> It messes around and creates swap-files, temporary files - and instead
> of modifying a file directly, it first moves it, then creates a new
> modified one and finally deletes the old one (thus the inotify-watch is
> gone afterwards).
> The creation of the swap files is an issue, when watching a directory
> for changes (e.g. /etc/portage). As the files are created _always_, the
> settings would be reloaded even if the user only reads the files (and
> another time, when he closes it, as the swap files are deleted). This is
> definitly a non-optimal solution.

I finally solved that issue:
Now all events occuring at watched pathes are checked whether they refer
to a temporary file. If this is true, they are ignored.

A temporary file is defined as:
- - either ends or starts with "#"
- - is a hidden file (starts with ".")
- - is a backup file (ends with "~" or ".bkp")
- - consits only of digits (yes ViM does such crazy stuff)

And to reduce the reload-calls even more: All events that trigger a
reload in a specific time frame (at the moment: 2 seconds) are taken as
one trigger.

> 
> 2.) If noone objects in the following days, I would like to write to the
> portage-dev-ml about this project (or better: the plans of the project)
> and perhaps get some proposals from this side :)

this is still a todo.

> 3.) As it is quite some work to maintain two versions of the
> portage-backend, I plan to use catapult in Portato by default. But this
> depends on how soon we find a solution for 1.).

Catapult now seems quite reliable. Some functionality still has to be
implemented:

- - different API versions (*)
- - central library (for use flags, central administration etc) (*)
- - allow the server to be stopped from outside. this is esp. important
when installing a new version - the old version running should then stop
and the new one should start ;)
- - extended API ... if there are no more API requests I will take the API
as version 0 and all future extensions will be API v1.

(*): not a blocker: so if this cannot be done in time, it can be delayed

@araujo: How are the chances that catapult can get into tree in some
weeks? - Would you (proxy-)maintain it?
If yes, the next portato version (which will come in like 3 to 4 weeks I
hope) would only use catapult as backend (except of course another major
bug occurs).

This would allow me, to focus on maintaining the catapult backend and
not having to deal with two things. (For example the update_world needs
some extension...)

Regards,
Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHK2gZ4UOg/zhYFuARArVSAJwMZGGKhtN19/+Xg49eM5iTE+1IvwCfRVTO
adrteqWShmv0eXGIQnpSW6o=
=4CF5
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann schrieb:
>
> Catapult now seems quite reliable. Some functionality still has to be
> implemented:
[...]
> - allow the server to be stopped from outside. this is esp. important
> when installing a new version - the old version running should then stop
> and the new one should start ;)

This is now allowed. I added another DBus-Interface
(org.gentoo.catapult.Admin) which currently holds the two methods
"stop()" and "ping()". The first one does the job :). "Ping" is kind of
a start up function: it pings the service - and if the service is not
started already, dbus launches it.

I also extended the catapult-start.sh: I renamed it to "catapult.sh" and
make it accept four commands:

- - start: Starts the service - the script does not return # old
catapult-start.sh bevavior
- - stop: Stops the service
- - ping: Pings the service --> startup WITH returning (see note above)
- - status: Checks whether the service has been started. As the catapult
services a run on the session bus, it can only give information about
the current user.

So a restart of e.g. the catapult portage service would look like:

if catapult status portage; then
    catapult stop portage
    catapult ping portage
fi

A similar algorithm is used by the catapult-portage ebuild in the
pkg_postinst function to restart the root's service (or whoever ran the
emerge command).

Regards,
Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHLlr44UOg/zhYFuARAjghAJ9IRA5R29O25OF0sKKoLqwaHBp89ACfTGKe
2CkmnA8aS3nyw8dzkS4LqDA=
=smMS
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list

Re: [gentoo-guis] One backend for alle portage GUIs

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I just worked a little bit at the central catapult library:

It now defines several variables (CATAPULT_SYSTEM_INTERFACE,
CATAPULT_PACKAGE_INTERFACE,..., CATAPULT_PACKAGE_BUS, ...) which might
be used in providers and other applications using dbus to not have to
copy'n'paste all the DBus specifications.

Also available is the function "get_dbus_address(backend_t)" which gets
a backend_t (DEFAULT, PORTAGE, PALUDIS, ...) and returns the appropriate
DBus address. This way the applications can either use the system
default (DEFAULT) or directly announce on of the other providers.
Currently the default is hard-coded to "portage" but it is meant to be
read out of a central configuration file in the future. (Feel free to
implement this ;)).

I know - this is really trivial: but I wanted more like implement a
basic build system:

I therefore used cmake (looks better than the normal autohell ... I
tried SCons too, but did not really like it though I _am_ a python
fanboy ^^).
I also added a first wrapper (to Python) using SWIG. SWIG has the
advantage, that the same interface file can be used to create a whole
bunch of wrappers (tcl, perl, ruby, ... (see "equery u swig" for whole
list ;))). And it is easy to use together with cmake, too.

Yeah ... the ebuilds are up2date to be used, so try it out, if you like
;) ...

Regards,
Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHM0qc4UOg/zhYFuARAn34AJ9NUiAr3dU59KvXhNZeT8ZcBg6StwCfd6hf
izRNn7/1nmvccDabTp/Xlh0=
=rCFR
-----END PGP SIGNATURE-----
-- 
gentoo-guis@gentoo.org mailing list