[gentoo-guis] Why I don't like catapult

[gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 5486 bytes --]

Since all catapult discussion appears to be happening here, and I don't
feel like registering from some forum that may or may not be read...

Catapult, as it stands, probably isn't sanely usable in general and
definitely isn't usable for Paludis. Here's why.

The API is Broken
-----------------

* use of cpv throughout. cpv isn't enough to uniquely identify an ID,
  especially if there're overlays. cpvr is closer, but still doesn't
  work if you need to deal with virtuals directly (rather than the thing
  provided by the virtual)

* use of regexes throughout. Every language defines its own, different
  regex dialect. There's no portability between languages.

* No mention of EAPI anywhere.

* No mention of error handling anywhere.

* Generally lots of hard-coded ebuildy things that don't map well onto
  future EAPIs or handling of non-ebuilds (e.g. native CRAN support).

* Essentially an abstraction of Portage internals, rather than an
  abstraction of a general package manager.

* It's not clear how much of the API would be useful, or that the API
  provides enough to do something useful.

* It's tied onto a single user configuration setup, and has no sane way
  of handling multiple configurations or unconfigured operation.

* sets don't map on to package manager sets.

* Administration:

- version. But no name?

* System:

- find_best. What does this do if you give it foo/bar-1 foo/baz-1?
  What's the point of having this function at all?

- find_best_match. What's a search key? If it's a spec, it needs an
  associated EAPI. What if only_installed is false? Does this mean we
  want both not installed and installed stuff?

- find_packages. As find_best_match for search key. What disambiguation
  is performed upon the atom? What happens if you pass it something like
  'git', which is ambiguous? Regular expressions at this level is silly
  -- which of the zillion regex flavours do we use? Does it operate on
  installed, not installed or what?

- get_config_path. Unportable. What if the user is using some other
  config format?

- get_deep_option. Deep's a mess; making package managers emulate it via
  a command line switch is silly.

- get_environment. In no way portable.

- get_global_key. Again, unportable -- Paludis has no notion of global
  configuration.

- get_merge_command. There's no nice way of using whatever this returns.
  And it's pointless -- why not just ask the package manager to do the
  merge? And what about package managers that require an explicit
  --install? When you ask for --sync or whatever, off what do you base
  the command?

- get_newuse_option. Again, unportable and messy.

- get_oneshot_option. Again, unportable and messy.

- get_pretend_option. Again, unportable and messy. And probably useless
  -- what would you do with this?

- get_sync_command. What's the point? Why not just ask the package
  manager to do the sync?

- get_unmerge_option. Same.

- get_update_option. Same as for get_deep_option.

- get_updated_packages. Highly unclear what this really does.

- get_use_desc. package should be a cpvr.

- get_world_file_path. Unportable.

- list_categories. String or regex?

- sort_package_list. You don't want stuff sorted the way it's sorted by
  Paludis. You might want a well defined sort.

- split_cpv. Why the weird revision thing?

- update_world. Huh? In some ways pointlessly specific, in some ways way
  too vague. And what's this use flag stuff? We finally just about got
  people away from setting use flags in places other than config files,
  and for good reason.

* Package

- What do all of these do on unknown EAPI?

- compare_version. But it takes packages. What's the ordering on
  foo/bar-1 vs bar/baz-1?

- get_actual_use_flags. What's the point of this?

- get_dep_packages. Hardcoding three dep classes is icky and doesn't
  work nicely with future EAPIs. And what's the return value? What about
  when || is in operation?

- get_global_settings. No such thing as a global setting.

- get_iuse_flags. What does this do for IUSE defaults? For installed
  packages, all flags are forced.

- get_installed_use_flags. Empty list? Seriously?

- get_masking_reason. Is the return value supposed to be used by
  anything other than a human?

- get_matched_dep_packages. Huh?

- get_overlay_path. What if it's in an overlay with no on-disk location?
  What if it's in some format the end tool doesn't recognise?

- get_package_path. Ditto.

- get_package_settings. What do you mean by everything? Just metadata?
  And what about EAPI? Some things don't make sense for some EAPIs or
  some package formats.

- get_use_flags. Bad. No legit use.

- is_in_system. System is a bad name.

- is_testing. How does package.keywords alter whether a package is
  testing?

- use_expanded. Not clear how this works when things aren't really
  expanded or when expanded keys aren't known.

Why DBus?
---------

What's the point in using DBus? What does it add over a simplified
library with bindings?

How does the whole privs thing work? Should anyone who can talk by DBus
be allowed to perform privileged operations? Should anyone who can't
perform privileged operations be allowed to do unprivileged operations?

What about persistence? What's classed as a session? Why force it down
to a single session at once?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh schrieb:
> Since all catapult discussion appears to be happening here, and I don't
> feel like registering from some forum that may or may not be read...
> 
> Catapult, as it stands, probably isn't sanely usable in general and
> definitely isn't usable for Paludis. Here's why.

Thanks for your comments :). I don't have time at the moment too read
and discuss them all, but I will do so in the evening ;)

The reason I wrote this mail is just to say, that the current API is
what it is, because there was no discussion at all (though I asked
several people) - so at the moment it is more or less Portato's portage
backend.

As you are the "Package Management" guru around, I would be glad if we
can make catapult something useful together.

So again: Thanks for your comments :)

- - Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH72iI4UOg/zhYFuARApSFAJ41qCcp2xAeQ6ymtT39sGbwUPgaagCfTlPf
mBdX0QrjyUQdcFSCNvBBP7U=
=xSqx
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

My replies ;)

One important remark: I always see the things from the API-user view. I
have to use this API in a package manager GUI and thus want to have the
API to specify certain functionality.

Ciaran McCreesh schrieb:
> * use of cpv throughout. cpv isn't enough to uniquely identify an ID,
>   especially if there're overlays. cpvr is closer, but still doesn't
>   work if you need to deal with virtuals directly (rather than the thing
>   provided by the virtual)

CPVR sounds reasonable ... and what about using virtual/* for virtuals?
Btw: There was the idea of providing a "get_virtuals" function....

> * use of regexes throughout. Every language defines its own, different
>   regex dialect. There's no portability between languages.

True - I thought of using some kind of globbing - thus only allowing "*"
and "?" (and perhaps "^" and "$")

> * No mention of EAPI anywhere.

Why would this be needed? This is only true if "catapult API" == "EAPI".

> * No mention of error handling anywhere.

True. Currently exceptions are only passed to the caller - but this
should definitly be specified somewhere.

> * Generally lots of hard-coded ebuildy things that don't map well onto
>   future EAPIs or handling of non-ebuilds (e.g. native CRAN support).

For me - gentoo package management is all about ebuilds. So I can't see,
what should changed there.
The only thing that should be provided is a feature similar to EAPI (the
 "catapult API"), so that additional stuff can be specified later on. I
don't see a point in thinking about things, that may come somewhere in
the future.

> * It's tied onto a single user configuration setup, and has no sane way
>   of handling multiple configurations or unconfigured operation.

I don't understand this ...

> * sets don't map on to package manager sets.

This can easily be changed. But nevertheless, some catapult-wide sets
should be defined, that map onto the specific ones in the package
managers (e.g. catapult.SET_ALL should be replaced by the specific set
in the different managers)

> * Administration:
> 
> - version. But no name?

I saw this myself some days ago... Really a flaw - but one simple to fix ;)

> - find_best. What does this do if you give it foo/bar-1 foo/baz-1?
Undefined atm.
>   What's the point of having this function at all?
See very first remark.

> - find_best_match. What's a search key? If it's a spec, it needs an
>   associated EAPI.
True.
> What if only_installed is false? Does this mean we
>   want both not installed and installed stuff?
Yes. - But perhaps, instead of the "only_installed" flag used here and
there, a set-name should be provided. (Which then can be "installed",
"uninstalled" or whatever).

> - find_packages. As find_best_match for search key. What disambiguation
>   is performed upon the atom? What happens if you pass it something like
>   'git', which is ambiguous?
Return everything that fits. It should be on the users side to translate
it into something useful.

> - get_config_path. Unportable. What if the user is using some other
>   config format?
But the configs (read: package.mask, package.use etc.) have to be at a
specified unique location, don't they?

> - get_deep_option. Deep's a mess; making package managers emulate it via
>   a command line switch is silly.
If you don't have an equivalent: Return "".

> - get_environment. In no way portable.
True. Should be dropped. I used it to get unwanted things out of the
environment. (E.g. modify EMERGE_DEFAULT_OPTS). But this is really a
user thing and should not be here.

> - get_global_key. Again, unportable -- Paludis has no notion of global
>   configuration.
get_global_settings and get_package_settings should be merged anyway.


> - get_merge_command. There's no nice way of using whatever this returns.
>   And it's pointless -- why not just ask the package manager to do the
>   merge?
Because catapult should be passive: I.E. only provide information and
does not do anything that could be harmful.

> And what about package managers that require an explicit
>   --install?
Return: ["pkgmgr", "--install"]. Everything that has "get_xxx_command"
should return the complete commandline.

> - get_newuse_option. Again, unportable and messy.
> 
> - get_oneshot_option. Again, unportable and messy.
Why unportable? - See the remarks to "--deep".

> - get_pretend_option. Again, unportable and messy. And probably useless
>   -- what would you do with this?
True.

> - get_updated_packages. Highly unclear what this really does.
Returns a list of packages, that can be updated. Similar to what
"eix-sync" shows at the end of the run.

> - get_world_file_path. Unportable.
True.

> - list_categories. String or regex?
Has to be defined.

> - sort_package_list. You don't want stuff sorted the way it's sorted by
>   Paludis. You might want a well defined sort.
True.

> - split_cpv. Why the weird revision thing?
What's so weired about it?

> - update_world. Huh? In some ways pointlessly specific, in some ways way
>   too vague. And what's this use flag stuff? We finally just about got
>   people away from setting use flags in places other than config files,
>   and for good reason.
Rember: Passive. It should return the packages, that WOULD be updated,
if a world-update (or alike) would be run. The parameters are there, to
run different scenarios. (Portato for example uses this, to generate the
list - taking into account the user saved flags, which aren't saved to
configs yet.)

> - What do all of these do on unknown EAPI?
Again: Why does the API have to care about EAPI? - It's the providers
job to care about it.

> - compare_version. But it takes packages. What's the ordering on
>   foo/bar-1 vs bar/baz-1?
Currently it would return: ("bar-1", "baz-1") as bar < baz. But this can
easily specified different. And a good point to think about error
handling again.

> - get_actual_use_flags. What's the point of this?
In general: Returns all useflags currently set (even the package
specific ones). It additionally has the ability to merge in some other
flag settings. (Ok - the description in the API-document is a little bit
... strange).

> - get_dep_packages. Hardcoding three dep classes is icky and doesn't
>   work nicely with future EAPIs.
Is only hard-coded in the document. (Count this as a doc-error.)

> And what's the return value? What about when || is in operation?
WWPMD (what would the package manager do): Simplified: Pass the
dep-string to the package manager and see what he does with it. The
result is returned.

> - get_iuse_flags. What does this do for IUSE defaults? For installed
>   packages, all flags are forced.
defaults are returned verbatim. I used to define "forced flags" as: "The
flags the user can't change." (like userland_GNU or use.mask'ed flags)

> - get_installed_use_flags. Empty list? Seriously?
DBus does not know the "empty value". The signature has to be kept in
all cases. So: yes - empty list. (Other choice would be an exception...)

> - get_masking_reason. Is the return value supposed to be used by
>   anything other than a human?
I doubt it.

> - get_matched_dep_packages. Huh?
Return all packages the cpv depends on, but which are already installed.
"get_dep_packages" and "get_matched_dep_packages" can surely be merged.

> - get_overlay_path. What if it's in an overlay with no on-disk location?
>   What if it's in some format the end tool doesn't recognise?
The end tool's problem I'd say.

> - get_use_flags. Bad. No legit use.
Doc error. Has already been removed several weeks ago.

> - is_in_system. System is a bad name.
Renaming is easy :)

> - is_testing. How does package.keywords alter whether a package is
>   testing?
Just a naming issue - I use "testing" for things that are keyworded
"~KEYWORD".

> - use_expanded. Not clear how this works when things aren't really
>   expanded.
Return "". The naming is bad here too.

> or when expanded keys aren't known.
When should that happen?

> 
> Why DBus?
> ---------
> 
> What's the point in using DBus? What does it add over a simplified
> library with bindings?

Library bindings have several disadvantages:
- - Providers have to be coded in C.
- - How to access portage internals from C-Code? Yes: You can implement
python-calls in C... but I think this is messy.

Dbus advantages:
- - allows some interesting structures: Like dbus+ssh: Contact the
Catapult-Service on one machine from another one (remote adminstration)
- - bindings already provided
- - caching: For example the portage internal caches are kept and don't
have to be recreated each time something using catapult is started.

> How does the whole privs thing work? Should anyone who can talk by DBus
> be allowed to perform privileged operations? Should anyone who can't
> perform privileged operations be allowed to do unprivileged operations?
I thought of having Catapult being a passive thing. So there is no way
of doing privileged actions.

> What about persistence? What's classed as a session? Why force it down
> to a single session at once?
Easier to implement. But it isn't forced. If the provider implements
sessions (e.g. using the Dbus-sending-adress as a key) w/o extending
API, I can't see a problem.

Regards,
Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQFH784w4UOg/zhYFuARAnZRAJ4hnjiH/2yycAiDCXLnst2EpesyuQCWJNNs
6gPL9qWeiaFgCGzpwbT9cA==
=Pdi+
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 8804 bytes --]

On Sun, 30 Mar 2008 19:30:24 +0200
René 'Necoro' Neumann <lists@necoro.eu> wrote:
> Ciaran McCreesh schrieb:
> > * use of cpv throughout. cpv isn't enough to uniquely identify an
> > ID, especially if there're overlays. cpvr is closer, but still
> > doesn't work if you need to deal with virtuals directly (rather
> > than the thing provided by the virtual)
> 
> CPVR sounds reasonable ... and what about using virtual/* for
> virtuals? Btw: There was the idea of providing a "get_virtuals"
> function....

The issue with virtuals is that all the package managers do them a bit
differently. For Paludis, if you have two different providers of the
same virtual installed with the same version and slot, you'll get two
IDs that have the same cpvr and that are only distinguished by the
package for which they are a virtual. This isn't an issue for us since
we use a different PackageID instance for each, but if instead you're
using a string to identify things (eww) you're in trouble.

> > * use of regexes throughout. Every language defines its own,
> > different regex dialect. There's no portability between languages.
> 
> True - I thought of using some kind of globbing - thus only allowing
> "*" and "?" (and perhaps "^" and "$")

Why support it at all? That's not something that should be in at the
package manager level.

> > * No mention of EAPI anywhere.
> 
> Why would this be needed? This is only true if "catapult API" ==
> "EAPI".

Well no. When doing pretty much anything with IDs, you have to be aware
of the EAPI. Client programs mustn't attempt to do anything with IDs
whose EAPI they don't recognise.

> > * Generally lots of hard-coded ebuildy things that don't map well
> > onto future EAPIs or handling of non-ebuilds (e.g. native CRAN
> > support).
> 
> For me - gentoo package management is all about ebuilds. So I can't
> see, what should changed there.

Future-direction-wise, a lot of people are wanting to handle things
like CPAN, CRAN, Gems etc natively in the package manager rather than
by writing wrapper ebuilds.

> > * It's tied onto a single user configuration setup, and has no sane
> > way of handling multiple configurations or unconfigured operation.
> 
> I don't understand this ...

Paludis supports multiple configurations. You can, for example,
have /etc/paludis, /etc/paludis-mychroot and ~/.paludis-anotherchroot,
and Paludis can operate with all of thse. Paludis can also operate on
systems where there's no configuration -- for example, you shouldn't
need a full config setup just to be able to run QA checks on one
particular repository.

Portage also sort of has some of this in a very crude way, through
PORTAGE_CONFIGROOT.

> > * sets don't map on to package manager sets.
> 
> This can easily be changed. But nevertheless, some catapult-wide sets
> should be defined, that map onto the specific ones in the package
> managers (e.g. catapult.SET_ALL should be replaced by the specific set
> in the different managers)

I'm not really sure that your concept of sets is right...

> > - find_best. What does this do if you give it foo/bar-1 foo/baz-1?
> Undefined atm.
> >   What's the point of having this function at all?
> See very first remark.

Sure, but you should still define your API in such a way that it's
easily generalisable and maps well onto the underlying data.

> > What if only_installed is false? Does this mean we
> >   want both not installed and installed stuff?
> Yes. - But perhaps, instead of the "only_installed" flag used here and
> there, a set-name should be provided. (Which then can be "installed",
> "uninstalled" or whatever).

Again, set confusion. Are you viewing set as a collection of IDs or a
collection of specs? (It's the latter.)

> > - find_packages. As find_best_match for search key. What
> > disambiguation is performed upon the atom? What happens if you pass
> > it something like 'git', which is ambiguous?
> Return everything that fits. It should be on the users side to
> translate it into something useful.

That's messy.

> > - get_config_path. Unportable. What if the user is using some other
> >   config format?
> But the configs (read: package.mask, package.use etc.) have to be at a
> specified unique location, don't they?

For Paludis? No. NoConfigEnvironment, for example, uses no
configuration directory at all.

> > - get_deep_option. Deep's a mess; making package managers emulate
> > it via a command line switch is silly.
> If you don't have an equivalent: Return "".

It shouldn't be in the API at all though.

> > - get_merge_command. There's no nice way of using whatever this
> > returns. And it's pointless -- why not just ask the package manager
> > to do the merge?
> Because catapult should be passive: I.E. only provide information and
> does not do anything that could be harmful.

Then it shouldn't even try to provide a way to do the merge stuff. It
needs to either do this properly or not at all.

> > - get_newuse_option. Again, unportable and messy.
> > 
> > - get_oneshot_option. Again, unportable and messy.
> Why unportable? - See the remarks to "--deep".

Neither maps on exactly to the Paludis resolver. The Paludis resolver
is quite a bit more flexible option-wise than what Portage offers, but
there's no exact correspondence between options.

> > - split_cpv. Why the weird revision thing?
> What's so weired about it?

Why is the revision considered special?

> > - What do all of these do on unknown EAPI?
> Again: Why does the API have to care about EAPI? - It's the providers
> job to care about it.

If you perform an API call that works on an ID with an unknown EAPI,
what's supposed to happen? What about if you perform an API call that
works upon an ID with an EAPI known to the package manager but not to
the tool?

(One example: what happens if we go with DEPENDENCIES for EAPI 2, and if
you ask for DEPEND on an EAPI 2 package?)

> > - compare_version. But it takes packages. What's the ordering on
> >   foo/bar-1 vs bar/baz-1?
> Currently it would return: ("bar-1", "baz-1") as bar < baz. But this
> can easily specified different. And a good point to think about error
> handling again.

Why the ordering on PN rather than CATEGORY/PN?

> > And what's the return value? What about when || is in operation?
> WWPMD (what would the package manager do): Simplified: Pass the
> dep-string to the package manager and see what he does with it. The
> result is returned.

The package manager has to have special handling for || throughout. The
behaviour of || blocks is rather tricky.

> > - get_iuse_flags. What does this do for IUSE defaults? For installed
> >   packages, all flags are forced.
> defaults are returned verbatim. I used to define "forced flags" as:
> "The flags the user can't change." (like userland_GNU or use.mask'ed
> flags)

The user can't change any flags for installed packages.

> > - get_masking_reason. Is the return value supposed to be used by
> >   anything other than a human?
> I doubt it.

Specify that.

> > - get_overlay_path. What if it's in an overlay with no on-disk
> > location? What if it's in some format the end tool doesn't
> > recognise?
> The end tool's problem I'd say.

You're providing API calls that make this sort of thing difficult. Not
an ideal situation...

> > or when expanded keys aren't known.
> When should that happen?

Installed packages. USE_EXPAND wasn't (and still isn't by some package
managers iirc) saved to VDB.

> > What's the point in using DBus? What does it add over a simplified
> > library with bindings?
> 
> Library bindings have several disadvantages:
> - - Providers have to be coded in C.
> - - How to access portage internals from C-Code? Yes: You can
> implement python-calls in C... but I think this is messy.

You can do the translation either way.

> > How does the whole privs thing work? Should anyone who can talk by
> > DBus be allowed to perform privileged operations? Should anyone who
> > can't perform privileged operations be allowed to do unprivileged
> > operations?
> I thought of having Catapult being a passive thing. So there is no way
> of doing privileged actions.

Things like querying an uncached package are privileged.

> > What about persistence? What's classed as a session? Why force it
> > down to a single session at once?
> Easier to implement. But it isn't forced. If the provider implements
> sessions (e.g. using the Dbus-sending-adress as a key) w/o extending
> API, I can't see a problem.

But without well-defined session management, you're stuck when it comes
to multiple configurations and the like.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh schrieb:
> On Sun, 30 Mar 2008 19:30:24 +0200
> René 'Necoro' Neumann <lists@necoro.eu> wrote:
>> Ciaran McCreesh schrieb:
>>> * use of cpv throughout. cpv isn't enough to uniquely identify an
>>> ID, especially if there're overlays. cpvr is closer, but still
>>> doesn't work if you need to deal with virtuals directly (rather
>>> than the thing provided by the virtual)
>> CPVR sounds reasonable ... and what about using virtual/* for
>> virtuals? Btw: There was the idea of providing a "get_virtuals"
>> function....
> 
> The issue with virtuals is that all the package managers do them a bit
> differently. For Paludis, if you have two different providers of the
> same virtual installed with the same version and slot, you'll get two
> IDs that have the same cpvr and that are only distinguished by the
> package for which they are a virtual. This isn't an issue for us since
> we use a different PackageID instance for each, but if instead you're
> using a string to identify things (eww) you're in trouble.

Hmmm - ok then virtuals really need to be thought about.

>>> * use of regexes throughout. Every language defines its own,
>>> different regex dialect. There's no portability between languages.
>> True - I thought of using some kind of globbing - thus only allowing
>> "*" and "?" (and perhaps "^" and "$")
> 
> Why support it at all? That's not something that should be in at the
> package manager level.

That's true ... but catapult is a bit higher than package manager level.
And at least the "find_" functions should allow them, to make searching
and alike simpler.

>>> * No mention of EAPI anywhere.
>> Why would this be needed? This is only true if "catapult API" ==
>> "EAPI".
> 
> Well no. When doing pretty much anything with IDs, you have to be aware
> of the EAPI. Client programs mustn't attempt to do anything with IDs
> whose EAPI they don't recognise.

Hmm ... no. I see you are coming from the package-manager theory side.
And there EAPI is important. But I can't see any point here where EAPI
plays a role (besides package-IDs (which can be resolved using catapult
API versions)), that can't be covered _inside_ the provider.
I don't doubt that e.g. the catapult-paludis provider has to deal with
EAPI in some way. But the _user_ (or better: application dev) using
catapult API does not need to be aware of it. Catapult is of higher
level than package management itself.

>>> * Generally lots of hard-coded ebuildy things that don't map well
>>> onto future EAPIs or handling of non-ebuilds (e.g. native CRAN
>>> support).
>> For me - gentoo package management is all about ebuilds. So I can't
>> see, what should changed there.
> 
> Future-direction-wise, a lot of people are wanting to handle things
> like CPAN, CRAN, Gems etc natively in the package manager rather than
> by writing wrapper ebuilds.

Ok. Perhaps we can postpone this until it gets real. (And then perhaps
use scpvr's ;) ... like: cpan:bla-foo/SomePerlPkg-0.13)

>>> * It's tied onto a single user configuration setup, and has no sane
>>> way of handling multiple configurations or unconfigured operation.
>> I don't understand this ...
> 
> Paludis supports multiple configurations. You can, for example,
> have /etc/paludis, /etc/paludis-mychroot and ~/.paludis-anotherchroot,
> and Paludis can operate with all of thse. Paludis can also operate on
> systems where there's no configuration -- for example, you shouldn't
> need a full config setup just to be able to run QA checks on one
> particular repository.
> 
> Portage also sort of has some of this in a very crude way, through
> PORTAGE_CONFIGROOT.

As catapult only queries, this can be resolved _inside_ the paludis
provider.
This of course gets hairy if it comes to libcatapult to provide
config-file manipulation...

>>> * sets don't map on to package manager sets.
>> This can easily be changed. But nevertheless, some catapult-wide sets
>> should be defined, that map onto the specific ones in the package
>> managers (e.g. catapult.SET_ALL should be replaced by the specific set
>> in the different managers)
> 
> I'm not really sure that your concept of sets is right...
> 

A "set" is a "set of packages" for me. You mention a "set of specs" -
don't get what kind of "specs" you are referring to here.

>>> - find_best. What does this do if you give it foo/bar-1 foo/baz-1?
>> Undefined atm.
>>>   What's the point of having this function at all?
>> See very first remark.
> 
> Sure, but you should still define your API in such a way that it's
> easily generalisable and maps well onto the underlying data.
>

Hmm ... Catapult should provide an API to ease developing applications
that deal with package managers. So there might be the case here or
there that one provider has to do some work to map to the underlying
package manager.

>>> - find_packages. As find_best_match for search key. What
>>> disambiguation is performed upon the atom? What happens if you pass
>>> it something like 'git', which is ambiguous?
>> Return everything that fits. It should be on the users side to
>> translate it into something useful.
> 
> That's messy.
>

Why? - It's a search function. And I guess in most cases the "return
everything that matches" is in case intended.

>>> - get_config_path. Unportable. What if the user is using some other
>>>   config format?
>> But the configs (read: package.mask, package.use etc.) have to be at a
>> specified unique location, don't they?
> 
> For Paludis? No. NoConfigEnvironment, for example, uses no
> configuration directory at all.
> 

This really is an issue... And what is happening if the user wants to
set an option (e.g. a useflag) in the "NoConfigEnvironment"? - Is a
config directory created and the environment switched?

>>> - get_deep_option. Deep's a mess; making package managers emulate
>>> it via a command line switch is silly.
>> If you don't have an equivalent: Return "".
> 
> It shouldn't be in the API at all though.

Hmmm ... it is of course portage-inspired. So you are of course right,
that something like this should not be allowed in the API. Nevertheless,
I want to provide applications with the ability to run a package manager
with different option switches which follow some global semantics.
Or do you think, something like this is just not possible and we have to
look for another way of doing so?

>>> - get_merge_command. There's no nice way of using whatever this
>>> returns. And it's pointless -- why not just ask the package manager
>>> to do the merge?
>> Because catapult should be passive: I.E. only provide information and
>> does not do anything that could be harmful.
> 
> Then it shouldn't even try to provide a way to do the merge stuff. It
> needs to either do this properly or not at all.

Think of the following use case: An application provides an interface
that allows users to select a package for installation. Now this
application uses catapult and wants to query for the command which has
to be run to get this package installed. Using get_merge_command, the
app could do:

spawn(catapult.system.get_merge_command(), package_list)

Catapult isn't the system merging here - it just provides the
information to actually _do_ the merge.

>>> - split_cpv. Why the weird revision thing?
>> What's so weired about it?
> 
> Why is the revision considered special?

I'm waiting what's the result of the "-r0"-thread on gentoo-dev before
giving an anwer ;)

>>> - What do all of these do on unknown EAPI?
>> Again: Why does the API have to care about EAPI? - It's the providers
>> job to care about it.
> 
> If you perform an API call that works on an ID with an unknown EAPI,
> what's supposed to happen? What about if you perform an API call that
> works upon an ID with an EAPI known to the package manager but not to
> the tool?
> 
> (One example: what happens if we go with DEPENDENCIES for EAPI 2, and if
> you ask for DEPEND on an EAPI 2 package?)

We have the following cases:
1.) "DEPEND" is a fixed catapult term/constant. Then the provider has to
map to "DEPENDENCIES" if it encounters the term and EAPI2 is used.
2.) "DEPEND" is directly passed to the package manager (for example
using "get_package_settings"). I think this isn't preferred behavior and
only a given set of constants for the "settings" and alike should be
used. -> see 1.)

>>> - compare_version. But it takes packages. What's the ordering on
>>>   foo/bar-1 vs bar/baz-1?
>> Currently it would return: ("bar-1", "baz-1") as bar < baz. But this
>> can easily specified different. And a good point to think about error
>> handling again.
> 
> Why the ordering on PN rather than CATEGORY/PN?

Hehe ;) - my fault... misread it. Of course it is first sorted using the
category. But haven't seen, they are different.

>>> And what's the return value? What about when || is in operation?
>> WWPMD (what would the package manager do): Simplified: Pass the
>> dep-string to the package manager and see what he does with it. The
>> result is returned.
> 
> The package manager has to have special handling for || throughout. The
> behaviour of || blocks is rather tricky.

I guess it is a major problem if the package manager isn't able to
handle "||", right? - So I can't see your point here.

>>> - get_iuse_flags. What does this do for IUSE defaults? For installed
>>>   packages, all flags are forced.
>> defaults are returned verbatim. I used to define "forced flags" as:
>> "The flags the user can't change." (like userland_GNU or use.mask'ed
>> flags)
> 
> The user can't change any flags for installed packages.

True. But he can re-install it using the changed flags. (Though he isn't
using the flags of the installed packages here, but the ones of the
uninstalled and overwriting the installed one.)
In other words: For this use case, the useflags of installed packages
are not seen as "forced flags" per se.

>>> - get_masking_reason. Is the return value supposed to be used by
>>>   anything other than a human?
>> I doubt it.
> 
> Specify that.

I guess I'm again missing something ;). But I can't see, why it matters
if it is intended for human readers or not :). And if there _is_ a
difference, there should be no problem in adding a "for_humans" boolean
argument to this call. :)
Though I don't know, what the masking reason should look like for
non-human targets.

>>> - get_overlay_path. What if it's in an overlay with no on-disk
>>> location? What if it's in some format the end tool doesn't
>>> recognise?
>> The end tool's problem I'd say.
> 
> You're providing API calls that make this sort of thing difficult. Not
> an ideal situation...

Just return some kind of an URL. The end tool has to deal with it.

>>> or when expanded keys aren't known.
>> When should that happen?
> 
> Installed packages. USE_EXPAND wasn't (and still isn't by some package
> managers iirc) saved to VDB.

Ok - so you are trying to cover the case, that USE_EXPAND changes after
installation of a package.
My answer would be: This is the fault of the underlying package manager.
If USE_EXPAND can't be resolved correctly, the wrong result (what ever
that means in the special case) is passed back.

Catapult should try to get rid of some flaws - but if it is not
possible, we can't do anything here. Bad luck.

>>> What's the point in using DBus? What does it add over a simplified
>>> library with bindings?
>> Library bindings have several disadvantages:
>> - - Providers have to be coded in C.
>> - - How to access portage internals from C-Code? Yes: You can
>> implement python-calls in C... but I think this is messy.
> 
> You can do the translation either way.

Right - but I consider it messy. (And I personally am a Python fan boy
and really dislike to code in C or even C++).
And I think it is quite strange, if you have a Python app, that accesses
Catapult-Portage:

Python (app) <---> Python (catapult bindings) <---> C (catapult) <--->
Python (portage)

>>> How does the whole privs thing work? Should anyone who can talk by
>>> DBus be allowed to perform privileged operations? Should anyone who
>>> can't perform privileged operations be allowed to do unprivileged
>>> operations?
>> I thought of having Catapult being a passive thing. So there is no way
>> of doing privileged actions.
> 
> Things like querying an uncached package are privileged.

Yes? - Why? Any user can open an ebuild and have a look at the internals.

And if it still matters: DBus allows to define in config files, which
users are allowed to do which actions on which bus.

>>> What about persistence? What's classed as a session? Why force it
>>> down to a single session at once?
>> Easier to implement. But it isn't forced. If the provider implements
>> sessions (e.g. using the Dbus-sending-adress as a key) w/o extending
>> API, I can't see a problem.
> 
> But without well-defined session management, you're stuck when it comes
> to multiple configurations and the like.

What exactly do you want to store in / link to the session?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH8BYr4UOg/zhYFuARAlLnAJ0fdoxjIa3tj3vOz8iMjUDOgLD7cACghPk5
AnSwpVHV1KRy0txxDGHJ0a4=
=XoHx
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 10116 bytes --]

On Mon, 31 Mar 2008 00:37:31 +0200
René 'Necoro' Neumann <lists@necoro.eu> wrote:
> Hmmm - ok then virtuals really need to be thought about.

Or just forcibly ignored...

> >>> * use of regexes throughout. Every language defines its own,
> >>> different regex dialect. There's no portability between languages.
> >> True - I thought of using some kind of globbing - thus only
> >> allowing "*" and "?" (and perhaps "^" and "$")
> > 
> > Why support it at all? That's not something that should be in at the
> > package manager level.
> 
> That's true ... but catapult is a bit higher than package manager
> level. And at least the "find_" functions should allow them, to make
> searching and alike simpler.

Sounds like you need a layered architecture. A low level catapult to
package manager layer, and a high level package manager independent
extras layer.

> Hmm ... no. I see you are coming from the package-manager theory side.
> And there EAPI is important. But I can't see any point here where EAPI
> plays a role (besides package-IDs (which can be resolved using
> catapult API versions)), that can't be covered _inside_ the provider.

As soon as you query any kind of information or perform any kind of
information on an ID your code must be EAPI aware.

> I don't doubt that e.g. the catapult-paludis provider has to deal with
> EAPI in some way. But the _user_ (or better: application dev) using
> catapult API does not need to be aware of it. Catapult is of higher
> level than package management itself.

But application developers do have to be aware of EAPI! Otherwise they
aren't allowed to query anything about the IDs they receive.

> Ok. Perhaps we can postpone this until it gets real. (And then perhaps
> use scpvr's ;) ... like: cpan:bla-foo/SomePerlPkg-0.13)

Paludis does it already.

> > Paludis supports multiple configurations. You can, for example,
> > have /etc/paludis, /etc/paludis-mychroot and
> > ~/.paludis-anotherchroot, and Paludis can operate with all of thse.
> > Paludis can also operate on systems where there's no configuration
> > -- for example, you shouldn't need a full config setup just to be
> > able to run QA checks on one particular repository.
> > 
> > Portage also sort of has some of this in a very crude way, through
> > PORTAGE_CONFIGROOT.
> 
> As catapult only queries, this can be resolved _inside_ the paludis
> provider.
> This of course gets hairy if it comes to libcatapult to provide
> config-file manipulation...

How does Paludis know which configuration set it should be using to
respond to queries?

> A "set" is a "set of packages" for me. You mention a "set of specs" -
> don't get what kind of "specs" you are referring to here.

A set isn't a set of packages. It's a set of dependency specifications
(foo/bar or >=foo/bar-2 or whatever).

> >>> - find_packages. As find_best_match for search key. What
> >>> disambiguation is performed upon the atom? What happens if you
> >>> pass it something like 'git', which is ambiguous?
> >> Return everything that fits. It should be on the users side to
> >> translate it into something useful.
> > 
> > That's messy.
> 
> Why? - It's a search function. And I guess in most cases the "return
> everything that matches" is in case intended.

Have a look at what existing tools do when given an ambiguous package
name part. They generally error, not do everything. Coding various
things gets tricky when you have to manually check that you've only
been given a single package name.

> >>> - get_config_path. Unportable. What if the user is using some
> >>> other config format?
> >> But the configs (read: package.mask, package.use etc.) have to be
> >> at a specified unique location, don't they?
> > 
> > For Paludis? No. NoConfigEnvironment, for example, uses no
> > configuration directory at all.
> > 
> 
> This really is an issue... And what is happening if the user wants to
> set an option (e.g. a useflag) in the "NoConfigEnvironment"? - Is a
> config directory created and the environment switched?

No. Users can't set options in NoConfigEnvironment. There is literally
no configuration beyond the constructing parameters.

> Nevertheless, I want to provide applications with the ability to run
> a package manager with different option switches which follow some
> global semantics. Or do you think, something like this is just not
> possible and we have to look for another way of doing so?

Doing this via an exec() is the wrong way to do it. If you want it to
be sanely usable you need to do it via a proper API.

> Think of the following use case: An application provides an interface
> that allows users to select a package for installation. Now this
> application uses catapult and wants to query for the command which has
> to be run to get this package installed. Using get_merge_command, the
> app could do:
> 
> spawn(catapult.system.get_merge_command(), package_list)
> 
> Catapult isn't the system merging here - it just provides the
> information to actually _do_ the merge.

Which is the wrong way of doing it. You might as well ask for a command
that outputs DESCRIPTION for a given ID rather than providing an API
call to do it directly.

> >>> - split_cpv. Why the weird revision thing?
> >> What's so weired about it?
> > 
> > Why is the revision considered special?
> 
> I'm waiting what's the result of the "-r0"-thread on gentoo-dev before
> giving an anwer ;)

Doesn't make much difference either way... Why is the revision the only
version part you treat specially?

> We have the following cases:
> 1.) "DEPEND" is a fixed catapult term/constant. Then the provider has
> to map to "DEPENDENCIES" if it encounters the term and EAPI2 is used.

Not doable. DEPENDENCIES can't map to DEPEND. DEPEND isn't powerful
enough syntactically.

> 2.) "DEPEND" is directly passed to the package manager (for example
> using "get_package_settings"). I think this isn't preferred behavior
> and only a given set of constants for the "settings" and alike should
> be used. -> see 1.)

Then you'd get a spurious blank result.

> >>> And what's the return value? What about when || is in operation?
> >> WWPMD (what would the package manager do): Simplified: Pass the
> >> dep-string to the package manager and see what he does with it. The
> >> result is returned.
> > 
> > The package manager has to have special handling for || throughout.
> > The behaviour of || blocks is rather tricky.
> 
> I guess it is a major problem if the package manager isn't able to
> handle "||", right? - So I can't see your point here.

The package manager handles || on a case by case basis. Anything
containing || cannot be flattened to a simple list.

> >>> - get_iuse_flags. What does this do for IUSE defaults? For
> >>> installed packages, all flags are forced.
> >> defaults are returned verbatim. I used to define "forced flags" as:
> >> "The flags the user can't change." (like userland_GNU or
> >> use.mask'ed flags)
> > 
> > The user can't change any flags for installed packages.
> 
> True. But he can re-install it using the changed flags. (Though he
> isn't using the flags of the installed packages here, but the ones of
> the uninstalled and overwriting the installed one.)
> In other words: For this use case, the useflags of installed packages
> are not seen as "forced flags" per se.

They are forced for the installed ID. They aren't necessarily forced
for other different IDs that can be installed to replace the currently
installed ID.

The foo/bar-2.0 that you have installed is a completely different ID to
any foo/bar-2.0 you happen to have in a repository somewhere.

> 
> >>> - get_masking_reason. Is the return value supposed to be used by
> >>>   anything other than a human?
> >> I doubt it.
> > 
> > Specify that.
> 
> I guess I'm again missing something ;). But I can't see, why it
> matters if it is intended for human readers or not :). And if there
> _is_ a difference, there should be no problem in adding a
> "for_humans" boolean argument to this call. :)
> Though I don't know, what the masking reason should look like for
> non-human targets.

If it's intended to be machine usable, the format has to be fixed.

> 
> >>> - get_overlay_path. What if it's in an overlay with no on-disk
> >>> location? What if it's in some format the end tool doesn't
> >>> recognise?
> >> The end tool's problem I'd say.
> > 
> > You're providing API calls that make this sort of thing difficult.
> > Not an ideal situation...
> 
> Just return some kind of an URL. The end tool has to deal with it.

Then the name needs changing, and it needs to be documented that the
return value is entirely meaningless.

> >>> How does the whole privs thing work? Should anyone who can talk by
> >>> DBus be allowed to perform privileged operations? Should anyone
> >>> who can't perform privileged operations be allowed to do
> >>> unprivileged operations?
> >> I thought of having Catapult being a passive thing. So there is no
> >> way of doing privileged actions.
> > 
> > Things like querying an uncached package are privileged.
> 
> Yes? - Why? Any user can open an ebuild and have a look at the
> internals.

But they can't write to cache on disk. Remember that Portage needs you
to be in the 'portage' group to do anything, and anyone in the
'portage' group can easily obtain root on the box.

> >>> What about persistence? What's classed as a session? Why force it
> >>> down to a single session at once?
> >> Easier to implement. But it isn't forced. If the provider
> >> implements sessions (e.g. using the Dbus-sending-adress as a key)
> >> w/o extending API, I can't see a problem.
> > 
> > But without well-defined session management, you're stuck when it
> > comes to multiple configurations and the like.
> 
> What exactly do you want to store in / link to the session?

Which configuration set is active, for one...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh schrieb:
> 
> Sounds like you need a layered architecture. A low level catapult to
> package manager layer, and a high level package manager independent
> extras layer.
> 

Currently working on transforming it ... Will post the results later on :)

>> Ok. Perhaps we can postpone this until it gets real. (And then perhaps
>> use scpvr's ;) ... like: cpan:bla-foo/SomePerlPkg-0.13)
> 
> Paludis does it already.
>

So we can set it as the package format? (Though needing further
definitions.)

>>> Paludis supports multiple configurations. You can, for example,
>>> have /etc/paludis, /etc/paludis-mychroot and
>>> ~/.paludis-anotherchroot, and Paludis can operate with all of thse.
>>> Paludis can also operate on systems where there's no configuration
>>> -- for example, you shouldn't need a full config setup just to be
>>> able to run QA checks on one particular repository.
>>>
>>> Portage also sort of has some of this in a very crude way, through
>>> PORTAGE_CONFIGROOT.
>> As catapult only queries, this can be resolved _inside_ the paludis
>> provider.
>> This of course gets hairy if it comes to libcatapult to provide
>> config-file manipulation...
> 
> How does Paludis know which configuration set it should be using to
> respond to queries?
> 

Ok - so we need sessions ...

I see two possibilites to implement it:
- - pass it as a parameter each time to each function ... Problems:
	+ each function needs an additional parameter ...
	+ how and what to pass using DBus? - I only can think of id-strings ...
but this leads to the second possibility
- - use the unique dbus service name and store sessions internally: see
http://lists.freedesktop.org/archives/dbus/2008-March/009550.html -->
now it is back to provider logic

In other words: The provider, which needs sessions, can use an internal
hashmap with the dbus service name of the client as the key and store
the data. This service name is defined to be unique as long as the dbus
bus is running. -> No need to support sessions in the official API.

>> A "set" is a "set of packages" for me. You mention a "set of specs" -
>> don't get what kind of "specs" you are referring to here.
> 
> A set isn't a set of packages. It's a set of dependency specifications
> (foo/bar or >=foo/bar-2 or whatever).
> 

k - understood

>>>>> - find_packages. As find_best_match for search key. What
>>>>> disambiguation is performed upon the atom? What happens if you
>>>>> pass it something like 'git', which is ambiguous?
>>>> Return everything that fits. It should be on the users side to
>>>> translate it into something useful.
>>> That's messy.
>> Why? - It's a search function. And I guess in most cases the "return
>> everything that matches" is in case intended.
> 
> Have a look at what existing tools do when given an ambiguous package
> name part. They generally error, not do everything. Coding various
> things gets tricky when you have to manually check that you've only
> been given a single package name.
> 

As written above, I'm currently working on changing this.

>>>>> - get_config_path. Unportable. What if the user is using some
>>>>> other config format?
>>>> But the configs (read: package.mask, package.use etc.) have to be
>>>> at a specified unique location, don't they?
>>> For Paludis? No. NoConfigEnvironment, for example, uses no
>>> configuration directory at all.
>>>
>> This really is an issue... And what is happening if the user wants to
>> set an option (e.g. a useflag) in the "NoConfigEnvironment"? - Is a
>> config directory created and the environment switched?
> 
> No. Users can't set options in NoConfigEnvironment. There is literally
> no configuration beyond the constructing parameters.

For the moment we can remove this call. But as soon as we want to have
libcatapult to provide a central place to allow config file changes, we
need a solution for this problem.

>> Nevertheless, I want to provide applications with the ability to run
>> a package manager with different option switches which follow some
>> global semantics. Or do you think, something like this is just not
>> possible and we have to look for another way of doing so?
> 
> Doing this via an exec() is the wrong way to do it. If you want it to
> be sanely usable you need to do it via a proper API.
> 
>> Think of the following use case: An application provides an interface
>> that allows users to select a package for installation. Now this
>> application uses catapult and wants to query for the command which has
>> to be run to get this package installed. Using get_merge_command, the
>> app could do:
>>
>> spawn(catapult.system.get_merge_command(), package_list)
>>
>> Catapult isn't the system merging here - it just provides the
>> information to actually _do_ the merge.
> 
> Which is the wrong way of doing it. You might as well ask for a command
> that outputs DESCRIPTION for a given ID rather than providing an API
> call to do it directly.

That's true. But I want it for three reasons:
- - no security issues
- - safe way is more important than nice way: In case the information
catapult provided (e.g. dep tree) is not correct (for whatever reason),
I don't want it to crash the users system. Thus, it is better to let the
package manager to have the final say. - This reason gets obsolete if
the providers are maintained by the package manager teams themselves,
but this is not the case atm.
- - at least in portage I can't see a way to say "hey - portage: install
the following packages please", as the logic behind this is done in
emerge itself.

Is there any other reason to vote against this besides "isn't the nice way"?

>>>>> - split_cpv. Why the weird revision thing?
>>>> What's so weired about it?
>>> Why is the revision considered special?
>> I'm waiting what's the result of the "-r0"-thread on gentoo-dev before
>> giving an anwer ;)
> 
> Doesn't make much difference either way... Why is the revision the only
> version part you treat specially?

We could also return "" instead "-r0" if no revision is given. Does not
make a difference.

>> We have the following cases:
>> 1.) "DEPEND" is a fixed catapult term/constant. Then the provider has
>> to map to "DEPENDENCIES" if it encounters the term and EAPI2 is used.
> 
> Not doable. DEPENDENCIES can't map to DEPEND. DEPEND isn't powerful
> enough syntactically.

Now I see, why you want the user to know of the EAPI... Do you think it
would be enough to allow to query the EAPI value of a package? Then
(taking this example) the client can decide whether to ask for "DEPEND"
or "DEPENDENCIES" depending on the EAPI.
If for some reason, he asks for "DEPEND" and the package uses EAPI2 an
EAPIError or something similar is being thrown.

>>>>> And what's the return value? What about when || is in operation?
>>>> WWPMD (what would the package manager do): Simplified: Pass the
>>>> dep-string to the package manager and see what he does with it. The
>>>> result is returned.
>>> The package manager has to have special handling for || throughout.
>>> The behaviour of || blocks is rather tricky.
>> I guess it is a major problem if the package manager isn't able to
>> handle "||", right? - So I can't see your point here.
> 
> The package manager handles || on a case by case basis. Anything
> containing || cannot be flattened to a simple list.

When installing the the package the package manager is taking one of the
choices of "||", based on certain information. A similar approach should
be used here too.

>>>>> - get_iuse_flags. What does this do for IUSE defaults? For
>>>>> installed packages, all flags are forced.
>>>> defaults are returned verbatim. I used to define "forced flags" as:
>>>> "The flags the user can't change." (like userland_GNU or
>>>> use.mask'ed flags)
>>> The user can't change any flags for installed packages.
>> True. But he can re-install it using the changed flags. (Though he
>> isn't using the flags of the installed packages here, but the ones of
>> the uninstalled and overwriting the installed one.)
>> In other words: For this use case, the useflags of installed packages
>> are not seen as "forced flags" per se.
> 
> They are forced for the installed ID. They aren't necessarily forced
> for other different IDs that can be installed to replace the currently
> installed ID.
> 
> The foo/bar-2.0 that you have installed is a completely different ID to
> any foo/bar-2.0 you happen to have in a repository somewhere.

Postponing this ... have to think about it.

>>>>> - get_masking_reason. Is the return value supposed to be used by
>>>>>   anything other than a human?
>>>> I doubt it.
>>> Specify that.
>> I guess I'm again missing something ;). But I can't see, why it
>> matters if it is intended for human readers or not :). And if there
>> _is_ a difference, there should be no problem in adding a
>> "for_humans" boolean argument to this call. :)
>> Though I don't know, what the masking reason should look like for
>> non-human targets.
> 
> If it's intended to be machine usable, the format has to be fixed.

So let's decide to have it human readable only.

>>>>> - get_overlay_path. What if it's in an overlay with no on-disk
>>>>> location? What if it's in some format the end tool doesn't
>>>>> recognise?
>>>> The end tool's problem I'd say.
>>> You're providing API calls that make this sort of thing difficult.
>>> Not an ideal situation...
>> Just return some kind of an URL. The end tool has to deal with it.
> 
> Then the name needs changing, and it needs to be documented that the
> return value is entirely meaningless.

Name change ok - perhaps s/path/url/ ?

"entirely meaningless" - I would use "might be meaningless" instead.

>>>>> How does the whole privs thing work? Should anyone who can talk by
>>>>> DBus be allowed to perform privileged operations? Should anyone
>>>>> who can't perform privileged operations be allowed to do
>>>>> unprivileged operations?
>>>> I thought of having Catapult being a passive thing. So there is no
>>>> way of doing privileged actions.
>>> Things like querying an uncached package are privileged.
>> Yes? - Why? Any user can open an ebuild and have a look at the
>> internals.
> 
> But they can't write to cache on disk. Remember that Portage needs you
> to be in the 'portage' group to do anything, and anyone in the
> 'portage' group can easily obtain root on the box.

Nope - you can query even if you are not in the portage group.

Despite the current open issues (btw: I tried to collect them on a wiki
page: http://catapult.origo.ethz.ch/wiki/current_issues ), do you think
that the whole system can be brought to something useful in the end?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH8kJ+4UOg/zhYFuARAjGOAJ9P4D5kZBJPu5FgS7ONmXw7T7E6kACaA0IU
YNS2iJrPw7gxB8dJy8U/mEY=
=UAQ9
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

René 'Necoro' Neumann schrieb:
> Ciaran McCreesh schrieb:
>> Sounds like you need a layered architecture. A low level catapult to
>> package manager layer, and a high level package manager independent
>> extras layer.
> 
> 
> Currently working on transforming it ... Will post the results later on :)
> 

Ok - I got some stuff done today:

- - find_packages now only supports package specs. It throws an
AmbiguousPackage error if there is something ambiguous.
	Note: if "" is passed as the spec, _everything_ is returned

- - list_categories always returns all categories

- - filter_list has been implemented as part of libcatapult which takes a
globbing string (allowing "*" and "?") and a list - and filters. (Note:
This is currently not implemented in a nice way. The python wrappers are
longer than the function itself ;))

- - removed get_environment and get_world_file_path from the API

- - added the "name()" method to the admin interface

- - added first exception/error support. For a detailed listing of
currently known errors: http://catapult.origo.ethz.ch/wiki/api_exceptions

And I had an idea about getting rid of dbus (at least in some way):
Instead of forcing dbus, we first implement catapult as a library. The
dbus support is additional and would be a small wrapper on top of this
library. So we had both worlds combined.

To get the portage code ported, I'm going to have a look into
Cython/Pyrex. This is probably not the most performant way - but who
cares ;)

Only one thing has to be solved: How is it possible to load additional
"catapult providers" on the fly? - So similar to the dbus way: You
install catapult and additional providers (e.g. catapult-portage) and
then load the correct libs at runtime.
I guess this can be answered by someone who has a deeper knowledge of C
than me ;)

For reference - the latest revisions having this changes implemented:
catapult rev: 25
catapult-portage rev: 41
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH8rBM4UOg/zhYFuARAiykAJ9U/ZZhYz0hBbe0mnVrZj+Yva7NIgCfXyMX
vGSyAUL8lFjWsNJLQHaCdY0=
=OqIs
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On Tue, 01 Apr 2008 23:59:40 +0200
René 'Necoro' Neumann <lists@necoro.eu> wrote:
> Only one thing has to be solved: How is it possible to load additional
> "catapult providers" on the fly? - So similar to the dbus way: You
> install catapult and additional providers (e.g. catapult-portage) and
> then load the correct libs at runtime.
> I guess this can be answered by someone who has a deeper knowledge of
> C than me ;)

Unix, rather than C... dlopen. If you're using C++, make provider an
ABC and have a virtual ctor that dlopen()s .so files that export a
function to make a concrete subclass. If you're using C, use a struct
full of function pointers instead, and cry when you're forced to use
1970s style programming. Or use dlsym() a zillion times and cry even
more...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 7346 bytes --]

On Tue, 01 Apr 2008 16:11:10 +0200
René 'Necoro' Neumann <lists@necoro.eu> wrote:
> >> Ok. Perhaps we can postpone this until it gets real. (And then
> >> perhaps use scpvr's ;) ... like: cpan:bla-foo/SomePerlPkg-0.13)
> > 
> > Paludis does it already.
> 
> So we can set it as the package format? (Though needing further
> definitions.)

Mm? With Paludis you get things like CRAN as 'just another repository'.

> > How does Paludis know which configuration set it should be using to
> > respond to queries?

When you invoke a Paludis client you can use --environment to specify
a configuration set.

> Ok - so we need sessions ...
> 
> I see two possibilites to implement it:

The third is to have different DBus services per configuration.

> That's true. But I want it for three reasons:
> - - no security issues
> - - safe way is more important than nice way: In case the information
> catapult provided (e.g. dep tree) is not correct (for whatever
> reason), I don't want it to crash the users system. Thus, it is
> better to let the package manager to have the final say. - This
> reason gets obsolete if the providers are maintained by the package
> manager teams themselves, but this is not the case atm.
> - - at least in portage I can't see a way to say "hey - portage:
> install the following packages please", as the logic behind this is
> done in emerge itself.
> 
> Is there any other reason to vote against this besides "isn't the
> nice way"?

Yeah. You can't use it to write a decent client. More on that below.

> > Doesn't make much difference either way... Why is the revision the
> > only version part you treat specially?
> 
> We could also return "" instead "-r0" if no revision is given. Does
> not make a difference.

Why split revision at all?

> Now I see, why you want the user to know of the EAPI... Do you think
> it would be enough to allow to query the EAPI value of a package? Then
> (taking this example) the client can decide whether to ask for
> "DEPEND" or "DEPENDENCIES" depending on the EAPI.
> If for some reason, he asks for "DEPEND" and the package uses EAPI2 an
> EAPIError or something similar is being thrown.

The issue with doing things that way is that every single client has to
be updated for every new EAPI.

It's probably useful to look at how Paludis and Portage handle EAPIs.
Portage basically just has some hardcoded if-else logic ("if the EAPI
is 1 then ...") in relevant areas (and doesn't generally enforce EAPI
things). This has the advantage that it's quick to code if you have a
small number of relatively similar EAPIs. It has the disadvantage that
when we introduce a new EAPI someone has to go through and check all
those code paths, and add a bunch more. It doesn't scale well to large
changes.

Paludis does things differently. EAPI-dependent code never directly
queries EAPI. Instead, it queries a central class for capabilities
(things like "does this EAPI support slot deps?" or "does this EAPI
use an ECONF_SOURCE-aware src_compile?"). And rather than hardcoding
configuration keys ("get me the DEPEND string" and so on), we have an
abstracted high-level interface ("get me the key that holds
dependencies that have been parsed into an abstract higher level
format") that's sufficiently powerful to deal with all EAPIs, along with
non-ebuild packages. This means that clients written using the Paludis
EAPI (and, indeed, the core Paludis library) have absolutely no mention
or awareness of EAPI anywhere, and yet always behave correctly for all
EAPIs.

Forcing every client to be EAPI aware and updated for every new EAPI is
majorly sucky. On the other hand, the Paludis approach quite possibly
only works because we have strong static checking and rich classes...

> > The package manager handles || on a case by case basis. Anything
> > containing || cannot be flattened to a simple list.
> 
> When installing the the package the package manager is taking one of
> the choices of "||", based on certain information. A similar approach
> should be used here too.

That certain information is the key issue. That certain information, in
the dependency resolution case, is a rather complex context that can
include things like speculative branch taking on earlier || cases,
speculative unmasking and knowledge of packages that aren't installed
yet but that will be by the time the ID providing the || is reached for
install.

For example... The Paludis handling code for || ( ) deps when resolving
a normal dependency goes very roughly like this:

- Is the || ( ) block empty after removing use? blocks? If so, pretend
it's empty.

- Try to rewrite the || ( ) block to a single multi-operator dep spec,
and use that instead of the || ( ) block if we can.

- Are any of the || ( ) children already installed? (Note that this
handles complex blocks and the like too...) If so, take that child.

- If we have something like || ( a >=b-2 ) and b-1 is installed, take
the b-2 branch if we can do so.

- Go for the first viable option.

- Fall back and go for the first option to get an exception.

You can't emulate this logic by flattening.

> >> Just return some kind of an URL. The end tool has to deal with it.
> > 
> > Then the name needs changing, and it needs to be documented that the
> > return value is entirely meaningless.
> 
> Name change ok - perhaps s/path/url/ ?
> 
> "entirely meaningless" - I would use "might be meaningless" instead.

So what's the URL for what Paludis calls VirtualsRepository or
InstalledVirtualsRepository?

> > But they can't write to cache on disk. Remember that Portage needs
> > you to be in the 'portage' group to do anything, and anyone in the
> > 'portage' group can easily obtain root on the box.
> 
> Nope - you can query even if you are not in the portage group.

Really not a good idea...

> Despite the current open issues (btw: I tried to collect them on a
> wiki page: http://catapult.origo.ethz.ch/wiki/current_issues ), do
> you think that the whole system can be brought to something useful in
> the end?

I think you have one big issue: you don't have a clear purpose for the
API.

On the one hand, you're saying that it should be a nice simple API for
clients.

On the other hand, you're saying that you want to use it for writing a
nice fancy GUI.

Of the Paludis clients we've experimented with, the one that makes by
far the most complex use of the API is the GUI client. This isn't
merely because we can -- the only time a GUI becomes useful is when it
offers functionality that can't easily be provided quickly by a non-GUI
client. A GUI should be able to do things like display a resolution of
'update world with deps', and have little clicky things on each item
saying things like "exclude this from the update" (greyed out if the
item can't be excluded), "show me a tree of why this package is
included in the list" and "select a different item from the || ( )
branch that pulled this dep in". You can't do this without an extremely
rich API (and you probably can't do it without lambdas hooked into the
resolver...). You definitely can't do this by having a crude "give me a
command I can exec()" way of installing things.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> Despite the current open issues (btw: I tried to collect them on a
>> wiki page: http://catapult.origo.ethz.ch/wiki/current_issues ), do
>> you think that the whole system can be brought to something useful in
>> the end?
> 
> I think you have one big issue: you don't have a clear purpose for the
> API.
> 
> On the one hand, you're saying that it should be a nice simple API for
> clients.
> 
> On the other hand, you're saying that you want to use it for writing a
> nice fancy GUI.

We just have a lack of a common base here. I can't see why these two
points are disjoint. The current API is already used for a GUI - and works.
The problem I see, is that you (and paludis) always want to be a little
bit more fancy and more complex.

> Of the Paludis clients we've experimented with, the one that makes by
> far the most complex use of the API is the GUI client. This isn't
> merely because we can -- the only time a GUI becomes useful is when it
> offers functionality that can't easily be provided quickly by a non-GUI
> client.

Nope - a GUI should provide things which are cumbersome using the
commandline. If a GUI offers things, that can't be done using CLI /
scripts - the CLI is bad.

> A GUI should be able to do things like display a resolution of
> 'update world with deps', and have little clicky things on each item
> saying things like "exclude this from the update" (greyed out if the
> item can't be excluded), "show me a tree of why this package is
> included in the list" and "select a different item from the || ( )
> branch that pulled this dep in". You can't do this without an extremely
> rich API (and you probably can't do it without lambdas hooked into the
> resolver...). You definitely can't do this by having a crude "give me a
> command I can exec()" way of installing things.

Your description is fancy ... _but_ the things you are describing is an
integrated GUI. It's on the same level the CLI is on.
This is nothing which is achievable with catapult (at least at the
moment). Catapult _uses_ the package manager and is not part of it. Thus
also the GUI would _use_ the PM.

And as mentioned in my earlier mail, avoiding the exec() stuff and use
the manager itself is bad in my eyes for different reasons. And for
portage it won't work at all.

Thus - I still vote for providing the exec interface. It can be dropped
if a nicer solution is found later on.

To sum up: I vote for the simple API - to get things done. It can be
enhanced later on.

And just because I feel like it - the get_x_option and their return
values as I see it for paludis:

deep : []
newuse: ["--dl-reinstall", "if-use-changed"]
pretend: ["--pretend"]
oneshot: ["--preserve-world"]

- - Necoro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH83Ta4UOg/zhYFuARAkofAJ4u7VQrzBzSyQLUEL8a7jfLSUxiCgCfT7f/
PBX30hWht2dQvL4RHOgd0YI=
=b4xY
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2252 bytes --]

On Wed, 02 Apr 2008 13:58:18 +0200
René 'Necoro' Neumann <lists@necoro.eu> wrote:
> We just have a lack of a common base here. I can't see why these two
> points are disjoint. The current API is already used for a GUI - and
> works. The problem I see, is that you (and paludis) always want to be
> a little bit more fancy and more complex.

We want to provide the functionality users need to get the job done.

> > Of the Paludis clients we've experimented with, the one that makes
> > by far the most complex use of the API is the GUI client. This isn't
> > merely because we can -- the only time a GUI becomes useful is when
> > it offers functionality that can't easily be provided quickly by a
> > non-GUI client.
> 
> Nope - a GUI should provide things which are cumbersome using the
> commandline. If a GUI offers things, that can't be done using CLI /
> scripts - the CLI is bad.

Realistically, there are some potentially useful things that just don't
map well onto a command line environment. Dynamic selection of || ( )
choices is a good example of this -- there's no sane way of offering it
in a CLI, but in a GUI (or even in ncurses if you push it) it isn't too
cumbersome.

> Your description is fancy ... _but_ the things you are describing is
> an integrated GUI. It's on the same level the CLI is on.
> This is nothing which is achievable with catapult (at least at the
> moment). Catapult _uses_ the package manager and is not part of it.
> Thus also the GUI would _use_ the PM.

And thus the GUI will end up delivering a half-arsed minimally
functional toy. That isn't what users need.

> And as mentioned in my earlier mail, avoiding the exec() stuff and use
> the manager itself is bad in my eyes for different reasons. And for
> portage it won't work at all.

You could implement it for Portage by using exec() yourself...

> Thus - I still vote for providing the exec interface. It can be
> dropped if a nicer solution is found later on.
> 
> To sum up: I vote for the simple API - to get things done. It can be
> enhanced later on.

To get what done? If the API can't be used to write a decent app,
people are going to stick with the package manager APIs.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh schrieb:
>> Your description is fancy ... _but_ the things you are describing is
>> an integrated GUI. It's on the same level the CLI is on.
>> This is nothing which is achievable with catapult (at least at the
>> moment). Catapult _uses_ the package manager and is not part of it.
>> Thus also the GUI would _use_ the PM.
> 
> And thus the GUI will end up delivering a half-arsed minimally
> functional toy. That isn't what users need.

This ended in a fully functional tool - but with a restricted feature
set. This isn't an issue for most users though - because the additional
features won't be needed by most users.

Again: There is a working GUI with it's user base already existing,
which _can_ use Catapult already. And I won't describe it as "minimally
functional toy".

>> And as mentioned in my earlier mail, avoiding the exec() stuff and use
>> the manager itself is bad in my eyes for different reasons. And for
>> portage it won't work at all.
> 
> You could implement it for Portage by using exec() yourself...
>

To be honest, this is something I haven't thought about yet ...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH83wI4UOg/zhYFuARAq8pAJ4mDNWqHwaTTL8tptuB/99MVkvnrgCeNBVB
LAMM2k9aJAb3WCYWJg6lqE4=
=pLOp
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

On Wed, 02 Apr 2008 14:28:56 +0200
René 'Necoro' Neumann <lists@necoro.eu> wrote:
> > And thus the GUI will end up delivering a half-arsed minimally
> > functional toy. That isn't what users need.
> 
> This ended in a fully functional tool - but with a restricted feature
> set. This isn't an issue for most users though - because the
> additional features won't be needed by most users.

Ah, the Gnome mentality... Except it doesn't work. Most users need at
least one feature that most users don't need.

> Again: There is a working GUI with it's user base already existing,
> which _can_ use Catapult already. And I won't describe it as
> "minimally functional toy".

I would.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-guis] Why I don't like catapult

[René 'Necoro' Neumann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh schrieb:
> On Wed, 02 Apr 2008 14:28:56 +0200
> René 'Necoro' Neumann <lists@necoro.eu> wrote:
>>> And thus the GUI will end up delivering a half-arsed minimally
>>> functional toy. That isn't what users need.
>> This ended in a fully functional tool - but with a restricted feature
>> set. This isn't an issue for most users though - because the
>> additional features won't be needed by most users.
> 
> Ah, the Gnome mentality... Except it doesn't work. Most users need at
> least one feature that most users don't need.
> 

Hehe :D

Ok - so I would declare this match as won by you ;). Now it's your turn:
What API additions do you propose?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH85C+4UOg/zhYFuARAuiTAJ0cOsv/Pjmq6YaKC6em1pyPnZ60AgCdFy92
QcbLY2KcsFcfE4ZZttXnYMI=
=7Atm
-----END PGP SIGNATURE-----
-- 
gentoo-guis@lists.gentoo.org mailing list

Re: [gentoo-guis] Why I don't like catapult

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

On Wed, 02 Apr 2008 15:57:18 +0200
René 'Necoro' Neumann <lists@necoro.eu> wrote:
> Ok - so I would declare this match as won by you ;). Now it's your
> turn: What API additions do you propose?

That would depend upon the class of tools I was targetting. I'd start
by getting a decent set of use cases.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]