[gentoo-dev] portage database management

[gentoo-dev] portage database management

[Ingo Krabbe]

Hi all,

I'm currently working on a Berkeley DB project for my customer.  While
doing this I'm asking if the package management of portage shouldn't go
to database one day ?  I mean it is getting slow ... it could be much
faster though.

Has anybody thought about this topic ? Are there current developments
on this topic ?

BYE INGO

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Rendhalver [Peter Brown]]

>>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:

    Ingo> Hi all,
    Ingo> I'm currently working on a Berkeley DB project for my customer.  While
    Ingo> doing this I'm asking if the package management of portage shouldn't go
    Ingo> to database one day ?  I mean it is getting slow ... it could be much
    Ingo> faster though.

    Ingo> Has anybody thought about this topic ? Are there current developments
    Ingo> on this topic ?

a while ago i thought about trying to generate an xml representation
of the portage "database"

i thought it would be cool to have it as xml cause then we could use
it for various purposes it would include the entire portage tree and
the database with extra bits to say wether a package was installed and
when it was installed its only an idea right now but i dont think it
would be too tricky to do  

i would write it in perl but thats what i am good at
and perl has some nifty XML modules

anyone else interested in this ??

-- 
 XEmacs Advocate  | Do not try the patience of Wizards,
 Gentoo Developer | for they are subtle and quick to anger.
 Perl Hacker      |   - Elric (Technomage) , Babylon 5.
 Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 06:37:43PM +1000, Rendhalver [Peter Brown] wrote:
> 
> a while ago i thought about trying to generate an xml representation
> of the portage "database"
> 
> i thought it would be cool to have it as xml cause then we could use
> it for various purposes it would include the entire portage tree and
> the database with extra bits to say wether a package was installed and
> when it was installed its only an idea right now but i dont think it
> would be too tricky to do  
> 
> i would write it in perl but thats what i am good at
> and perl has some nifty XML modules
> 
> anyone else interested in this ??
> 
> -- 
>  XEmacs Advocate  | Do not try the patience of Wizards,
>  Gentoo Developer | for they are subtle and quick to anger.
>  Perl Hacker      |   - Elric (Technomage) , Babylon 5.
>  Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>
> 
> 
> --
> gentoo-dev@gentoo.org mailing list
> 
> 

Would this give any speed up ? I don't know XML database features with
perl.  Of course you can store some constraint information with packages
in XML but what I was thinking about went more in the direction of fast
search trees and package hash tables, that the system administrator can
search for portage packages and keywords.

BYE INGO

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Rendhalver [Peter Brown]]

>>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:

    Ingo> On Sat, Feb 01, 2003 at 06:37:43PM +1000, Rendhalver [Peter Brown] wrote:
    >> 
    >> a while ago i thought about trying to generate an xml representation
    >> of the portage "database"
    >> 
    >> i thought it would be cool to have it as xml cause then we could use
    >> it for various purposes it would include the entire portage tree and
    >> the database with extra bits to say wether a package was installed and
    >> when it was installed its only an idea right now but i dont think it
    >> would be too tricky to do  
    >> 
    >> i would write it in perl but thats what i am good at
    >> and perl has some nifty XML modules
    >> 
    >> anyone else interested in this ??
    >> 
[snip]

    Ingo> Would this give any speed up ? I don't know XML database features with
    Ingo> perl.  Of course you can store some constraint information with packages
    Ingo> in XML but what I was thinking about went more in the direction of fast
    Ingo> search trees and package hash tables, that the system administrator can
    Ingo> search for portage packages and keywords.

ah ok so you want to put the actual portage tree into a database yes?


-- 
 XEmacs Advocate  | Do not try the patience of Wizards,
 Gentoo Developer | for they are subtle and quick to anger.
 Perl Hacker      |   - Elric (Technomage) , Babylon 5.
 Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 07:16:44PM +1000, Rendhalver [Peter Brown] wrote:
> ah ok so you want to put the actual portage tree into a database yes?

nope, I think it would be much nicer to portage to create a mirror
image of the portage tree in a database, together with all textual
information available.  I want to leave the portage as it is for
installing and updating packages but I want to be able to get events
from portage when new packages arrive (rsync), are installed (emerge)
or uninstalled (unmerge) or updated (emerge -u), so I can keep track of
it.

Of course it would be a solution to manage everything in a database, hmm
it is a tree you know, a big tree in recent times, so it should be a
database object.  But the evolution of portage was filesystem oriented,
which is understandable for portability, stability and transparency.  At
least the filesystem is a database too, a slow one though, but fast
enough for the installation purposes, measured against the compilation
and download times.

I often bothered about the problem of searching a package by keyword or
package name (emerge -s foo) and (emerge -S foo), when I just want to
get a quick overview about a topic or want to look up this new package I
just heard of in this newgroup yesterday.

This operation takes much too long for my taste and thats what I like to
keep in a database.  I know there are textual database systems like
htref, but I don't understand their installation and configuration
syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
put everything in a Berkeley DB put a job in the background and fire
some events or raise some signals.

BYE INGO

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Rendhalver [Peter Brown]]

>>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:

    Ingo> On Sat, Feb 01, 2003 at 07:16:44PM +1000, Rendhalver [Peter Brown] wrote:
    >> ah ok so you want to put the actual portage tree into a database yes?

    Ingo> nope, I think it would be much nicer to portage to create a mirror
    Ingo> image of the portage tree in a database, together with all textual
    Ingo> information available.  I want to leave the portage as it is for
    Ingo> installing and updating packages but I want to be able to get events
    Ingo> from portage when new packages arrive (rsync), are installed (emerge)
    Ingo> or uninstalled (unmerge) or updated (emerge -u), so I can keep track of
    Ingo> it.

sounds like a fun and useful project

    Ingo> Of course it would be a solution to manage everything in a database, hmm
    Ingo> it is a tree you know, a big tree in recent times, so it should be a
    Ingo> database object.  But the evolution of portage was filesystem oriented,
    Ingo> which is understandable for portability, stability and transparency.  At
    Ingo> least the filesystem is a database too, a slow one though, but fast
    Ingo> enough for the installation purposes, measured against the compilation
    Ingo> and download times.

    Ingo> I often bothered about the problem of searching a package by keyword or
    Ingo> package name (emerge -s foo) and (emerge -S foo), when I just want to
    Ingo> get a quick overview about a topic or want to look up this new package I
    Ingo> just heard of in this newgroup yesterday.

yeah i know what you mean there

you would have to make it so the database can be easily updated from
the output of a emerge rsync (or a cvs update for us developer types)
or do some kind of check on the portage tree for modifications to ebuilds

    Ingo> This operation takes much too long for my taste and thats what I like to
    Ingo> keep in a database.  I know there are textual database systems like
    Ingo> htref, but I don't understand their installation and configuration
    Ingo> syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
    Ingo> put everything in a Berkeley DB put a job in the background and fire
    Ingo> some events or raise some signals.

i know what you mean there :)
i would use something that can talk to multiple database backends so
the user has a choice in database to use

maybe you could use libgda/libgnomedb/mergeant for this ??
they can talk to lots of databases and the list is growing as well
last i looked there was an LDAP backend for libgda

-- 
 XEmacs Advocate  | Do not try the patience of Wizards,
 Gentoo Developer | for they are subtle and quick to anger.
 Perl Hacker      |   - Elric (Technomage) , Babylon 5.
 Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[John Nilsson]

On this topic, I'd like to ad some ideas. How about a package database
on some central server ( and mirrors...). This db would have more
indepth information of every package, HOWTOS, bugs, discussions all that
kind of information you would wan't (mostly just a gentoo specific info
text and link to a homepage I suspect, but you COULD add more).

This way you could have a forum for each package. This is probably
wanted if gentoo keeps growing, gentoo-user would be heavy as
linux-kernel.

/John

On Sat, 2003-02-01 at 11:17, Rendhalver [Peter Brown] wrote:
> >>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:
> 
>     Ingo> On Sat, Feb 01, 2003 at 07:16:44PM +1000, Rendhalver [Peter Brown] wrote:
>     >> ah ok so you want to put the actual portage tree into a database yes?
> 
>     Ingo> nope, I think it would be much nicer to portage to create a mirror
>     Ingo> image of the portage tree in a database, together with all textual
>     Ingo> information available.  I want to leave the portage as it is for
>     Ingo> installing and updating packages but I want to be able to get events
>     Ingo> from portage when new packages arrive (rsync), are installed (emerge)
>     Ingo> or uninstalled (unmerge) or updated (emerge -u), so I can keep track of
>     Ingo> it.
> 
> sounds like a fun and useful project
> 
>     Ingo> Of course it would be a solution to manage everything in a database, hmm
>     Ingo> it is a tree you know, a big tree in recent times, so it should be a
>     Ingo> database object.  But the evolution of portage was filesystem oriented,
>     Ingo> which is understandable for portability, stability and transparency.  At
>     Ingo> least the filesystem is a database too, a slow one though, but fast
>     Ingo> enough for the installation purposes, measured against the compilation
>     Ingo> and download times.
> 
>     Ingo> I often bothered about the problem of searching a package by keyword or
>     Ingo> package name (emerge -s foo) and (emerge -S foo), when I just want to
>     Ingo> get a quick overview about a topic or want to look up this new package I
>     Ingo> just heard of in this newgroup yesterday.
> 
> yeah i know what you mean there
> 
> you would have to make it so the database can be easily updated from
> the output of a emerge rsync (or a cvs update for us developer types)
> or do some kind of check on the portage tree for modifications to ebuilds
> 
>     Ingo> This operation takes much too long for my taste and thats what I like to
>     Ingo> keep in a database.  I know there are textual database systems like
>     Ingo> htref, but I don't understand their installation and configuration
>     Ingo> syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
>     Ingo> put everything in a Berkeley DB put a job in the background and fire
>     Ingo> some events or raise some signals.
> 
> i know what you mean there :)
> i would use something that can talk to multiple database backends so
> the user has a choice in database to use
> 
> maybe you could use libgda/libgnomedb/mergeant for this ??
> they can talk to lots of databases and the list is growing as well
> last i looked there was an LDAP backend for libgda

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 12:06:39PM +0100, John Nilsson wrote:
> On this topic, I'd like to ad some ideas. How about a package database
> on some central server ( and mirrors...). This db would have more
> indepth information of every package, HOWTOS, bugs, discussions all that
> kind of information you would wan't (mostly just a gentoo specific info
> text and link to a homepage I suspect, but you COULD add more).
> 
> This way you could have a forum for each package. This is probably
> wanted if gentoo keeps growing, gentoo-user would be heavy as
> linux-kernel.
> 
> /John

Nice idea, this would focus more on the Perl->XML side of this
discussion.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Dylan Carlson]

On Saturday 01 February 2003 06:06 am, John Nilsson wrote:
> On this topic, I'd like to ad some ideas. How about a package database
> on some central server ( and mirrors...). This db would have more
> indepth information of every package, HOWTOS, bugs, discussions all that
> kind of information you would wan't (mostly just a gentoo specific info
> text and link to a homepage I suspect, but you COULD add more).
>
> This way you could have a forum for each package. This is probably
> wanted if gentoo keeps growing, gentoo-user would be heavy as
> linux-kernel.
>
> /John
>

That is an EXCELLENT idea.   It seems like such an obvious one, but alas, 
nobody else has come out with it.  

FreeBSD, for example, indexes all of their ports on the web, with extra 
info that (among other things) allow you to quickly jump into CVS to see 
what revisions have been made.   But there are no forums for each port.

Cheers,
Dylan Carlson [absinthe@pobox.com]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Mark Constable]

> On Saturday 01 February 2003 06:06 am, John Nilsson wrote:
> ...
> This way you could have a forum for each package. This is probably
> wanted if gentoo keeps growing, gentoo-user would be heavy as
> linux-kernel.

Yes PLEASE, this would be excellent. Something a little like the manual
pages at php.net would be spectacular, and even better would be a Wiki.

Many moons ago, when Debian only had 1500 packages, I set up a gated
mailing-list->newsgroup (non-usenet) for each package and kinda got
laughed out of town for being overkill. I regret not pursuing it harder.

--markc


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Jim Nutt]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On Sun, 2 Feb 2003 13:29:49 +1000
Mark Constable <markc@renta.net> wrote:

> > On Saturday 01 February 2003 06:06 am, John Nilsson wrote:
> > ...
> > This way you could have a forum for each package. This is probably
> > wanted if gentoo keeps growing, gentoo-user would be heavy as
> > linux-kernel.
> 
> Yes PLEASE, this would be excellent. Something a little like the
> manual pages at php.net would be spectacular, and even better would be
> a Wiki.

Something like gforge (http://gforge.org) would probably work, with a project for each package.

jim

-- 
jim nutt

home: 	jim@nuttz.org		jabber:	jimnutt@jabber.com
work: 	jimnutt@vestek.com	ms msg: jim@nuttz.org
pgp id: 1ECBCC78

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] portage database management

[Dylan Carlson]

On Saturday 01 February 2003 04:25 am, Ingo Krabbe wrote:

>
> This operation takes much too long for my taste and thats what I like to
> keep in a database.  I know there are textual database systems like
> htref, but I don't understand their installation and configuration
> syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
> put everything in a Berkeley DB put a job in the background and fire
> some events or raise some signals.
>

I agree, there needs to be a way to speed up those operations eventually... 
if only to index the tree so that seeks/searches are performed faster.

Problem with berkeleydb (sleepycat) is... things that get coded around 
berkeleydb often stay married to berkeleydb forever, for better or for 
worse. I had some issues with berkeleydb in the past year with versioning; 
more than once it broke things on minor upgrades along the same branch.  
YMMV.

I think it's perhaps better to make this modular.  Write db modules, one 
for sleepycat, one for pgsql, one for mysql, et al.  Leave the indexing 
data store up to the admin.   The data itself indexed in a B-tree.   

XML is a nice output format for interfacing with other applications, but 
that doesn't mean you should store your data in it.

Cheers,
Dylan Carlson [absinthe@pobox.com]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 10:34:30AM -0500, Dylan Carlson wrote:
> I agree, there needs to be a way to speed up those operations eventually... 
> if only to index the tree so that seeks/searches are performed faster.
> 
> Problem with berkeleydb (sleepycat) is... things that get coded around 
> berkeleydb often stay married to berkeleydb forever, for better or for 
> worse. I had some issues with berkeleydb in the past year with versioning; 
> more than once it broke things on minor upgrades along the same branch.  
> YMMV.
> 
> I think it's perhaps better to make this modular.  Write db modules, one 
> for sleepycat, one for pgsql, one for mysql, et al.  Leave the indexing 
> data store up to the admin.   The data itself indexed in a B-tree.   
> 
> XML is a nice output format for interfacing with other applications, but 
> that doesn't mean you should store your data in it.
> 
> Cheers,
> Dylan Carlson [absinthe@pobox.com]
> 

Thanks, this is an answer I longed for.  I like berkeley DB for its easy
and small C API.  Of course there would be not much problem to use a
modular approach.  I may try a easy prototypish approach with C in the
evening or tomorrow.  Please inform me (mailto:<i.krabbe@dokom.net>) if
you want to have a look upon it.

BYE INGO

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: portage database management

[Christian Plessl]

Ingo Krabbe wrote:

> Hi all,
> 
> I'm currently working on a Berkeley DB project for my customer.  While
> doing this I'm asking if the package management of portage shouldn't go
> to database one day ?  I mean it is getting slow ... it could be much
> faster though.
> 
> Has anybody thought about this topic ? Are there current developments
> on this topic ?

To your first argument I agree, portage is not very fast when searching for
packages etc.

But, I would vote strongly againgst moving it to a database approach. Having
all portage information in plain textfiles because:

* This enables easy creation of tools, that display/manage portage
information in various ways because text files can be processed with any
programming language

* If something goes wrong there is a chance to correct this if textfiles are
used. Putting all into a database which stores everything in binary format
only seems rather scary to me. Although the potential speedup sounds
interessting this idea reminds me of the Windows registry, which was a nice
idea in the beginning, but turned out to be a nightmare.

If performance of portage is a problem for you, I would suggest using the
current portage information to build a fast searchable index for the
portage information.

Regards,
 Christian

-- 
Christian Plessl <plessl@tik.ee.ethz.ch>
Computer Engineering and Networks Lab, ETH Zurich, Switzerland


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: portage database management

[Marko Mikulicic]

Christian Plessl wrote:

> 
> If performance of portage is a problem for you, I would suggest using the
> current portage information to build a fast searchable index for the
> portage information.

I agree with you. I think it should be better to have separate index 
which can be rebuild from scratch upon corruption or whatever.
  If only the searching speed is needed then only gentoolkit's qpkg 
could be changed, reducing the possibility of breaking portage
  However an extension to portage itself could be an option if we want 
alse speed up the rependency resolution (expecially for emerge -up 
world) or for future features like auto-useflag-change-detection.
  Portage's access to package metainformation could be separated in 
front  and backend and modularized with failback on the text-files backend.
  The backends could be for text, berkleydb (little library dependency, 
useful on stage1 files or livecds), relational db or some other 
backstore. The frontends could include portage itself, a xml extractor 
(which could be an interface for other programs),...


Marko


--
gentoo-dev@gentoo.org mailing list