[gentoo-dev] portage database management

[gentoo-dev] portage database management

[Ingo Krabbe]

Hi all,

I'm currently working on a Berkeley DB project for my customer.  While
doing this I'm asking if the package management of portage shouldn't go
to database one day ?  I mean it is getting slow ... it could be much
faster though.

Has anybody thought about this topic ? Are there current developments
on this topic ?

BYE INGO

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Rendhalver [Peter Brown]]

>>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:

    Ingo> Hi all,
    Ingo> I'm currently working on a Berkeley DB project for my customer.  While
    Ingo> doing this I'm asking if the package management of portage shouldn't go
    Ingo> to database one day ?  I mean it is getting slow ... it could be much
    Ingo> faster though.

    Ingo> Has anybody thought about this topic ? Are there current developments
    Ingo> on this topic ?

a while ago i thought about trying to generate an xml representation
of the portage "database"

i thought it would be cool to have it as xml cause then we could use
it for various purposes it would include the entire portage tree and
the database with extra bits to say wether a package was installed and
when it was installed its only an idea right now but i dont think it
would be too tricky to do  

i would write it in perl but thats what i am good at
and perl has some nifty XML modules

anyone else interested in this ??

-- 
 XEmacs Advocate  | Do not try the patience of Wizards,
 Gentoo Developer | for they are subtle and quick to anger.
 Perl Hacker      |   - Elric (Technomage) , Babylon 5.
 Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 06:37:43PM +1000, Rendhalver [Peter Brown] wrote:
> 
> a while ago i thought about trying to generate an xml representation
> of the portage "database"
> 
> i thought it would be cool to have it as xml cause then we could use
> it for various purposes it would include the entire portage tree and
> the database with extra bits to say wether a package was installed and
> when it was installed its only an idea right now but i dont think it
> would be too tricky to do  
> 
> i would write it in perl but thats what i am good at
> and perl has some nifty XML modules
> 
> anyone else interested in this ??
> 
> -- 
>  XEmacs Advocate  | Do not try the patience of Wizards,
>  Gentoo Developer | for they are subtle and quick to anger.
>  Perl Hacker      |   - Elric (Technomage) , Babylon 5.
>  Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>
> 
> 
> --
> gentoo-dev@gentoo.org mailing list
> 
> 

Would this give any speed up ? I don't know XML database features with
perl.  Of course you can store some constraint information with packages
in XML but what I was thinking about went more in the direction of fast
search trees and package hash tables, that the system administrator can
search for portage packages and keywords.

BYE INGO

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Rendhalver [Peter Brown]]

>>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:

    Ingo> On Sat, Feb 01, 2003 at 06:37:43PM +1000, Rendhalver [Peter Brown] wrote:
    >> 
    >> a while ago i thought about trying to generate an xml representation
    >> of the portage "database"
    >> 
    >> i thought it would be cool to have it as xml cause then we could use
    >> it for various purposes it would include the entire portage tree and
    >> the database with extra bits to say wether a package was installed and
    >> when it was installed its only an idea right now but i dont think it
    >> would be too tricky to do  
    >> 
    >> i would write it in perl but thats what i am good at
    >> and perl has some nifty XML modules
    >> 
    >> anyone else interested in this ??
    >> 
[snip]

    Ingo> Would this give any speed up ? I don't know XML database features with
    Ingo> perl.  Of course you can store some constraint information with packages
    Ingo> in XML but what I was thinking about went more in the direction of fast
    Ingo> search trees and package hash tables, that the system administrator can
    Ingo> search for portage packages and keywords.

ah ok so you want to put the actual portage tree into a database yes?


-- 
 XEmacs Advocate  | Do not try the patience of Wizards,
 Gentoo Developer | for they are subtle and quick to anger.
 Perl Hacker      |   - Elric (Technomage) , Babylon 5.
 Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 07:16:44PM +1000, Rendhalver [Peter Brown] wrote:
> ah ok so you want to put the actual portage tree into a database yes?

nope, I think it would be much nicer to portage to create a mirror
image of the portage tree in a database, together with all textual
information available.  I want to leave the portage as it is for
installing and updating packages but I want to be able to get events
from portage when new packages arrive (rsync), are installed (emerge)
or uninstalled (unmerge) or updated (emerge -u), so I can keep track of
it.

Of course it would be a solution to manage everything in a database, hmm
it is a tree you know, a big tree in recent times, so it should be a
database object.  But the evolution of portage was filesystem oriented,
which is understandable for portability, stability and transparency.  At
least the filesystem is a database too, a slow one though, but fast
enough for the installation purposes, measured against the compilation
and download times.

I often bothered about the problem of searching a package by keyword or
package name (emerge -s foo) and (emerge -S foo), when I just want to
get a quick overview about a topic or want to look up this new package I
just heard of in this newgroup yesterday.

This operation takes much too long for my taste and thats what I like to
keep in a database.  I know there are textual database systems like
htref, but I don't understand their installation and configuration
syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
put everything in a Berkeley DB put a job in the background and fire
some events or raise some signals.

BYE INGO

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Rendhalver [Peter Brown]]

>>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:

    Ingo> On Sat, Feb 01, 2003 at 07:16:44PM +1000, Rendhalver [Peter Brown] wrote:
    >> ah ok so you want to put the actual portage tree into a database yes?

    Ingo> nope, I think it would be much nicer to portage to create a mirror
    Ingo> image of the portage tree in a database, together with all textual
    Ingo> information available.  I want to leave the portage as it is for
    Ingo> installing and updating packages but I want to be able to get events
    Ingo> from portage when new packages arrive (rsync), are installed (emerge)
    Ingo> or uninstalled (unmerge) or updated (emerge -u), so I can keep track of
    Ingo> it.

sounds like a fun and useful project

    Ingo> Of course it would be a solution to manage everything in a database, hmm
    Ingo> it is a tree you know, a big tree in recent times, so it should be a
    Ingo> database object.  But the evolution of portage was filesystem oriented,
    Ingo> which is understandable for portability, stability and transparency.  At
    Ingo> least the filesystem is a database too, a slow one though, but fast
    Ingo> enough for the installation purposes, measured against the compilation
    Ingo> and download times.

    Ingo> I often bothered about the problem of searching a package by keyword or
    Ingo> package name (emerge -s foo) and (emerge -S foo), when I just want to
    Ingo> get a quick overview about a topic or want to look up this new package I
    Ingo> just heard of in this newgroup yesterday.

yeah i know what you mean there

you would have to make it so the database can be easily updated from
the output of a emerge rsync (or a cvs update for us developer types)
or do some kind of check on the portage tree for modifications to ebuilds

    Ingo> This operation takes much too long for my taste and thats what I like to
    Ingo> keep in a database.  I know there are textual database systems like
    Ingo> htref, but I don't understand their installation and configuration
    Ingo> syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
    Ingo> put everything in a Berkeley DB put a job in the background and fire
    Ingo> some events or raise some signals.

i know what you mean there :)
i would use something that can talk to multiple database backends so
the user has a choice in database to use

maybe you could use libgda/libgnomedb/mergeant for this ??
they can talk to lots of databases and the list is growing as well
last i looked there was an LDAP backend for libgda

-- 
 XEmacs Advocate  | Do not try the patience of Wizards,
 Gentoo Developer | for they are subtle and quick to anger.
 Perl Hacker      |   - Elric (Technomage) , Babylon 5.
 Apache God       |  <mailto:rendhalver at gentoo.org> <GnuPG KeyID: AE51D190>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[John Nilsson]

On this topic, I'd like to ad some ideas. How about a package database
on some central server ( and mirrors...). This db would have more
indepth information of every package, HOWTOS, bugs, discussions all that
kind of information you would wan't (mostly just a gentoo specific info
text and link to a homepage I suspect, but you COULD add more).

This way you could have a forum for each package. This is probably
wanted if gentoo keeps growing, gentoo-user would be heavy as
linux-kernel.

/John

On Sat, 2003-02-01 at 11:17, Rendhalver [Peter Brown] wrote:
> >>>>> "Ingo" == Ingo Krabbe <i.krabbe@dokom.net> writes:
> 
>     Ingo> On Sat, Feb 01, 2003 at 07:16:44PM +1000, Rendhalver [Peter Brown] wrote:
>     >> ah ok so you want to put the actual portage tree into a database yes?
> 
>     Ingo> nope, I think it would be much nicer to portage to create a mirror
>     Ingo> image of the portage tree in a database, together with all textual
>     Ingo> information available.  I want to leave the portage as it is for
>     Ingo> installing and updating packages but I want to be able to get events
>     Ingo> from portage when new packages arrive (rsync), are installed (emerge)
>     Ingo> or uninstalled (unmerge) or updated (emerge -u), so I can keep track of
>     Ingo> it.
> 
> sounds like a fun and useful project
> 
>     Ingo> Of course it would be a solution to manage everything in a database, hmm
>     Ingo> it is a tree you know, a big tree in recent times, so it should be a
>     Ingo> database object.  But the evolution of portage was filesystem oriented,
>     Ingo> which is understandable for portability, stability and transparency.  At
>     Ingo> least the filesystem is a database too, a slow one though, but fast
>     Ingo> enough for the installation purposes, measured against the compilation
>     Ingo> and download times.
> 
>     Ingo> I often bothered about the problem of searching a package by keyword or
>     Ingo> package name (emerge -s foo) and (emerge -S foo), when I just want to
>     Ingo> get a quick overview about a topic or want to look up this new package I
>     Ingo> just heard of in this newgroup yesterday.
> 
> yeah i know what you mean there
> 
> you would have to make it so the database can be easily updated from
> the output of a emerge rsync (or a cvs update for us developer types)
> or do some kind of check on the portage tree for modifications to ebuilds
> 
>     Ingo> This operation takes much too long for my taste and thats what I like to
>     Ingo> keep in a database.  I know there are textual database systems like
>     Ingo> htref, but I don't understand their installation and configuration
>     Ingo> syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
>     Ingo> put everything in a Berkeley DB put a job in the background and fire
>     Ingo> some events or raise some signals.
> 
> i know what you mean there :)
> i would use something that can talk to multiple database backends so
> the user has a choice in database to use
> 
> maybe you could use libgda/libgnomedb/mergeant for this ??
> they can talk to lots of databases and the list is growing as well
> last i looked there was an LDAP backend for libgda

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 12:06:39PM +0100, John Nilsson wrote:
> On this topic, I'd like to ad some ideas. How about a package database
> on some central server ( and mirrors...). This db would have more
> indepth information of every package, HOWTOS, bugs, discussions all that
> kind of information you would wan't (mostly just a gentoo specific info
> text and link to a homepage I suspect, but you COULD add more).
> 
> This way you could have a forum for each package. This is probably
> wanted if gentoo keeps growing, gentoo-user would be heavy as
> linux-kernel.
> 
> /John

Nice idea, this would focus more on the Perl->XML side of this
discussion.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Dylan Carlson]

On Saturday 01 February 2003 06:06 am, John Nilsson wrote:
> On this topic, I'd like to ad some ideas. How about a package database
> on some central server ( and mirrors...). This db would have more
> indepth information of every package, HOWTOS, bugs, discussions all that
> kind of information you would wan't (mostly just a gentoo specific info
> text and link to a homepage I suspect, but you COULD add more).
>
> This way you could have a forum for each package. This is probably
> wanted if gentoo keeps growing, gentoo-user would be heavy as
> linux-kernel.
>
> /John
>

That is an EXCELLENT idea.   It seems like such an obvious one, but alas, 
nobody else has come out with it.  

FreeBSD, for example, indexes all of their ports on the web, with extra 
info that (among other things) allow you to quickly jump into CVS to see 
what revisions have been made.   But there are no forums for each port.

Cheers,
Dylan Carlson [absinthe@pobox.com]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Mark Constable]

> On Saturday 01 February 2003 06:06 am, John Nilsson wrote:
> ...
> This way you could have a forum for each package. This is probably
> wanted if gentoo keeps growing, gentoo-user would be heavy as
> linux-kernel.

Yes PLEASE, this would be excellent. Something a little like the manual
pages at php.net would be spectacular, and even better would be a Wiki.

Many moons ago, when Debian only had 1500 packages, I set up a gated
mailing-list->newsgroup (non-usenet) for each package and kinda got
laughed out of town for being overkill. I regret not pursuing it harder.

--markc


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Jim Nutt]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On Sun, 2 Feb 2003 13:29:49 +1000
Mark Constable <markc@renta.net> wrote:

> > On Saturday 01 February 2003 06:06 am, John Nilsson wrote:
> > ...
> > This way you could have a forum for each package. This is probably
> > wanted if gentoo keeps growing, gentoo-user would be heavy as
> > linux-kernel.
> 
> Yes PLEASE, this would be excellent. Something a little like the
> manual pages at php.net would be spectacular, and even better would be
> a Wiki.

Something like gforge (http://gforge.org) would probably work, with a project for each package.

jim

-- 
jim nutt

home: 	jim@nuttz.org		jabber:	jimnutt@jabber.com
work: 	jimnutt@vestek.com	ms msg: jim@nuttz.org
pgp id: 1ECBCC78

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] portage database management

[Dylan Carlson]

On Saturday 01 February 2003 04:25 am, Ingo Krabbe wrote:

>
> This operation takes much too long for my taste and thats what I like to
> keep in a database.  I know there are textual database systems like
> htref, but I don't understand their installation and configuration
> syntax.  Hmm, I'm a C Programmer you know, it is much easier to me to
> put everything in a Berkeley DB put a job in the background and fire
> some events or raise some signals.
>

I agree, there needs to be a way to speed up those operations eventually... 
if only to index the tree so that seeks/searches are performed faster.

Problem with berkeleydb (sleepycat) is... things that get coded around 
berkeleydb often stay married to berkeleydb forever, for better or for 
worse. I had some issues with berkeleydb in the past year with versioning; 
more than once it broke things on minor upgrades along the same branch.  
YMMV.

I think it's perhaps better to make this modular.  Write db modules, one 
for sleepycat, one for pgsql, one for mysql, et al.  Leave the indexing 
data store up to the admin.   The data itself indexed in a B-tree.   

XML is a nice output format for interfacing with other applications, but 
that doesn't mean you should store your data in it.

Cheers,
Dylan Carlson [absinthe@pobox.com]

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Ingo Krabbe]

On Sat, Feb 01, 2003 at 10:34:30AM -0500, Dylan Carlson wrote:
> I agree, there needs to be a way to speed up those operations eventually... 
> if only to index the tree so that seeks/searches are performed faster.
> 
> Problem with berkeleydb (sleepycat) is... things that get coded around 
> berkeleydb often stay married to berkeleydb forever, for better or for 
> worse. I had some issues with berkeleydb in the past year with versioning; 
> more than once it broke things on minor upgrades along the same branch.  
> YMMV.
> 
> I think it's perhaps better to make this modular.  Write db modules, one 
> for sleepycat, one for pgsql, one for mysql, et al.  Leave the indexing 
> data store up to the admin.   The data itself indexed in a B-tree.   
> 
> XML is a nice output format for interfacing with other applications, but 
> that doesn't mean you should store your data in it.
> 
> Cheers,
> Dylan Carlson [absinthe@pobox.com]
> 

Thanks, this is an answer I longed for.  I like berkeley DB for its easy
and small C API.  Of course there would be not much problem to use a
modular approach.  I may try a easy prototypish approach with C in the
evening or tomorrow.  Please inform me (mailto:<i.krabbe@dokom.net>) if
you want to have a look upon it.

BYE INGO

--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: portage database management

[Christian Plessl]

Ingo Krabbe wrote:

> Hi all,
> 
> I'm currently working on a Berkeley DB project for my customer.  While
> doing this I'm asking if the package management of portage shouldn't go
> to database one day ?  I mean it is getting slow ... it could be much
> faster though.
> 
> Has anybody thought about this topic ? Are there current developments
> on this topic ?

To your first argument I agree, portage is not very fast when searching for
packages etc.

But, I would vote strongly againgst moving it to a database approach. Having
all portage information in plain textfiles because:

* This enables easy creation of tools, that display/manage portage
information in various ways because text files can be processed with any
programming language

* If something goes wrong there is a chance to correct this if textfiles are
used. Putting all into a database which stores everything in binary format
only seems rather scary to me. Although the potential speedup sounds
interessting this idea reminds me of the Windows registry, which was a nice
idea in the beginning, but turned out to be a nightmare.

If performance of portage is a problem for you, I would suggest using the
current portage information to build a fast searchable index for the
portage information.

Regards,
 Christian

-- 
Christian Plessl <plessl@tik.ee.ethz.ch>
Computer Engineering and Networks Lab, ETH Zurich, Switzerland


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: portage database management

[Marko Mikulicic]

Christian Plessl wrote:

> 
> If performance of portage is a problem for you, I would suggest using the
> current portage information to build a fast searchable index for the
> portage information.

I agree with you. I think it should be better to have separate index 
which can be rebuild from scratch upon corruption or whatever.
  If only the searching speed is needed then only gentoolkit's qpkg 
could be changed, reducing the possibility of breaking portage
  However an extension to portage itself could be an option if we want 
alse speed up the rependency resolution (expecially for emerge -up 
world) or for future features like auto-useflag-change-detection.
  Portage's access to package metainformation could be separated in 
front  and backend and modularized with failback on the text-files backend.
  The backends could be for text, berkleydb (little library dependency, 
useful on stage1 files or livecds), relational db or some other 
backstore. The frontends could include portage itself, a xml extractor 
(which could be an interface for other programs),...


Marko


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Brian Friday]

-- Begin Newbie Rant

First (as it is my first post), I'd like to say thanks to all the people
who've made gentoo what it is today. I've used lots of linux distributions
but Gentoo was the first to make me feel at home. So thanks for creating
something I have, am and hope to continue enjoying.

-- End Newbie Rant

Perhaps I'm totally missing the point but....

On the large scale, there will be soon if not already a need to know
when a package was released and/or what version of the distribution it
first occurred in. In addition, making dependency information more
accessible, up to date and more detailed would be helpful as well. These
needs would be in addition to the general information portage currently
stores about a packages (ie category, name, version, size, homepage, and
description).

For the large scale, a database would seem to be the logical choice for
storing all this information. With a web or similarly easy interface to
allow casual browsing and or administration. You'd want to have primary
servers which would ideally feed mirrors who then in turn feed users.
Though they could feed the end users as well.

For the individual system, I think the DB idea is overkill but I like
the plugin ability. Maybe a step similar to choosing your cron/syslog but
instead you would be choosing your portage backend.

One of the key strengths of Gentoo I see as an end user is that most/all
of the files which determine how I can build/configure my system are
directly editable.  In practical terms, this openness has encouraged me to
pro-actively attack problems rather than wait for fixes. Ultimately
allowing me to feel a direct part of the Gentoo community, even if I'm
only hacking my system.

Now what if the portage on end user systems retained a XML text file
format similar to the example below. This would give us the flexibility of
text, as it is readable by the normal human, while at the same time
machine friendly. The example below is just a quick idea how a ebuild file
might look in an XML portage system.

Example XML file for a package:

<package>
<name></name>
<version></version>
<size></size>
<homepage></homepage>
<license></license>
<portage_release_date></portage_release_date>
<slot></slot>
<ebuild_comment></ebuild_comment>
<description>
    <keywords></keywords>
    <abstract>one line info text</abstract>
    <full_text>full about text</full_text>
    <ebuild_comment></ebuild_comment>
</description>
<dependency>
    <name></name>
    <version></version>
    <mandatory>yes or no</mandatory>
    <reason>why it might be good to have this</reason>
</dependency>
<ebuild_commands>
    <ebuild_comment></ebuild_comment>
    <pkg_setup>
          <ebuild_comment></ebuild_comment>
          <commands></commands>
    </pkg_setup>
    <src_unpack></src_unpack>
    <src_compile></src_compile>
    <src_install></src_install>
    <pkg_preinst></pkg_preinst>
    <pkg_postinst></pkg_postinst>
    <pkg_prerm></pkg_prerm>
    <pkg_postrm></pkg_postrm>
    <pkg_config></pkg_config>
</ebuild_commands>
</package>

The nice thing about XML is it could also be used with the plugin idea,
allowing for a intermediate structured format for transfer between the
gentoo servers/mirrors and the end user system. Backends are given the
data in a format they can parse through without to much trouble and you
could also export a ebuild out in a structured format perhaps allowing for
remote submission of ebuilds.

Now that I'm thinking about it you could have two XML package template
versions, detailed and optimised versions. The detailed version would be
as shown above containing absolutely everything about the ebuild. The
optimised version would contain just minimum elements needed for the
portage system. This would have the secondary effect of having optimised
mirrors which only have those bare bones ebuild files....

I'm rambling so I'll end this. Thoughts? Flames? Expressive Grunts?


--
Brian





--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Jared H. Hudson]

I like XML as much as the next guy, but one thing you should remember is 
that ebuilds are not some arbitrary pkg format -- they're bash scripts. 
They may not look it, since they don't have #!/bin/bash at the top, but 
they are bash scripts that are sourced with other bash scripts. So, for 
example, most ebuilds have bash functins like src_unpack, src_install, 
ect, but others have their own functions that are defined and called by 
these src_* functions. Converting this to XML would mean that portage 
would have to convert everything to bash, plus there'd have to be enough 
flexibility in the XML stylesheet we'd to include the possibility of 
unknown-ebuild-specific global variables, functions, ect.

Just food for thought.

-Jared H.

Brian Friday wrote:
> -- Begin Newbie Rant
> 
> First (as it is my first post), I'd like to say thanks to all the people
> who've made gentoo what it is today. I've used lots of linux distributions
> but Gentoo was the first to make me feel at home. So thanks for creating
> something I have, am and hope to continue enjoying.
> 
> -- End Newbie Rant
> 
> Perhaps I'm totally missing the point but....
> 
> On the large scale, there will be soon if not already a need to know
> when a package was released and/or what version of the distribution it
> first occurred in. In addition, making dependency information more
> accessible, up to date and more detailed would be helpful as well. These
> needs would be in addition to the general information portage currently
> stores about a packages (ie category, name, version, size, homepage, and
> description).
> 
> For the large scale, a database would seem to be the logical choice for
> storing all this information. With a web or similarly easy interface to
> allow casual browsing and or administration. You'd want to have primary
> servers which would ideally feed mirrors who then in turn feed users.
> Though they could feed the end users as well.
> 
> For the individual system, I think the DB idea is overkill but I like
> the plugin ability. Maybe a step similar to choosing your cron/syslog but
> instead you would be choosing your portage backend.
> 
> One of the key strengths of Gentoo I see as an end user is that most/all
> of the files which determine how I can build/configure my system are
> directly editable.  In practical terms, this openness has encouraged me to
> pro-actively attack problems rather than wait for fixes. Ultimately
> allowing me to feel a direct part of the Gentoo community, even if I'm
> only hacking my system.
> 
> Now what if the portage on end user systems retained a XML text file
> format similar to the example below. This would give us the flexibility of
> text, as it is readable by the normal human, while at the same time
> machine friendly. The example below is just a quick idea how a ebuild file
> might look in an XML portage system.
> 
> Example XML file for a package:
> 
> <package>
> <name></name>
> <version></version>
> <size></size>
> <homepage></homepage>
> <license></license>
> <portage_release_date></portage_release_date>
> <slot></slot>
> <ebuild_comment></ebuild_comment>
> <description>
>     <keywords></keywords>
>     <abstract>one line info text</abstract>
>     <full_text>full about text</full_text>
>     <ebuild_comment></ebuild_comment>
> </description>
> <dependency>
>     <name></name>
>     <version></version>
>     <mandatory>yes or no</mandatory>
>     <reason>why it might be good to have this</reason>
> </dependency>
> <ebuild_commands>
>     <ebuild_comment></ebuild_comment>
>     <pkg_setup>
>           <ebuild_comment></ebuild_comment>
>           <commands></commands>
>     </pkg_setup>
>     <src_unpack></src_unpack>
>     <src_compile></src_compile>
>     <src_install></src_install>
>     <pkg_preinst></pkg_preinst>
>     <pkg_postinst></pkg_postinst>
>     <pkg_prerm></pkg_prerm>
>     <pkg_postrm></pkg_postrm>
>     <pkg_config></pkg_config>
> </ebuild_commands>
> </package>
> 
> The nice thing about XML is it could also be used with the plugin idea,
> allowing for a intermediate structured format for transfer between the
> gentoo servers/mirrors and the end user system. Backends are given the
> data in a format they can parse through without to much trouble and you
> could also export a ebuild out in a structured format perhaps allowing for
> remote submission of ebuilds.
> 
> Now that I'm thinking about it you could have two XML package template
> versions, detailed and optimised versions. The detailed version would be
> as shown above containing absolutely everything about the ebuild. The
> optimised version would contain just minimum elements needed for the
> portage system. This would have the secondary effect of having optimised
> mirrors which only have those bare bones ebuild files....
> 
> I'm rambling so I'll end this. Thoughts? Flames? Expressive Grunts?
> 
> 
> --
> Brian
> 
> 
> 
> 
> 
> --
> gentoo-dev@gentoo.org mailing list


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Alain Penders]

That's indeed the main problem with XML.  If you want to see the practicality 
of this, take a look at Ant (http://ant.apache.org/).  Ant is a Java based 
build system, kinda like Makefiles but WAY more powerful.  It's slowly 
becoming the defacto standard for building Java projects, and especially for 
large Java projects it's invaluable.

It defines a project in XML, and comes with a lot of modules (comparable to 
eclasses and the default ebuild commands) which can be used inside that XML.

Implementing an ebuild in Ant would be very simple, and probably with one or 
two additional modules one could implement all of portage in ant.  Doing the 
latter would be way slower than portage is today though, and doing the first 
requires adding Java... not very useful right now either.

Looking at the cheer size of Ant -- all the stuff they had to put in before it 
became a really useful system, I'd vote against trying to do this for portage.  
Having an XML definition for each package, yes... replacing the actual build 
code by XML - no.

My $0.02...

Alain


On Thu, Feb 06, 2003 at 03:41:29AM -0600, Jared H. Hudson wrote:
> I like XML as much as the next guy, but one thing you should remember is 
> that ebuilds are not some arbitrary pkg format -- they're bash scripts. 
> They may not look it, since they don't have #!/bin/bash at the top, but 
> they are bash scripts that are sourced with other bash scripts. So, for 
> example, most ebuilds have bash functins like src_unpack, src_install, 
> ect, but others have their own functions that are defined and called by 
> these src_* functions. Converting this to XML would mean that portage 
> would have to convert everything to bash, plus there'd have to be enough 
> flexibility in the XML stylesheet we'd to include the possibility of 
> unknown-ebuild-specific global variables, functions, ect.
> 
> Just food for thought.
> 
> -Jared H.

--
gentoo-dev@gentoo.org mailing list

RE: [gentoo-dev] portage database management

[Riyad Kalla]

agreed

> -----Original Message-----
> From: Alain Penders [mailto:alain@gentoo.org] 
> Sent: Thursday, February 06, 2003 12:38 PM
> To: gentoo-dev@gentoo.org
> Subject: Re: [gentoo-dev] portage database management
> 
> 
> That's indeed the main problem with XML.  If you want to see 
> the practicality 
> of this, take a look at Ant (http://ant.apache.org/).  Ant is 
> a Java based 
> build system, kinda like Makefiles but WAY more powerful.  
> It's slowly 
> becoming the defacto standard for building Java projects, and 
> especially for 
> large Java projects it's invaluable.
> 
> It defines a project in XML, and comes with a lot of modules 
> (comparable to 
> eclasses and the default ebuild commands) which can be used 
> inside that XML.
> 
> Implementing an ebuild in Ant would be very simple, and 
> probably with one or 
> two additional modules one could implement all of portage in 
> ant.  Doing the 
> latter would be way slower than portage is today though, and 
> doing the first 
> requires adding Java... not very useful right now either.
> 
> Looking at the cheer size of Ant -- all the stuff they had to 
> put in before it 
> became a really useful system, I'd vote against trying to do 
> this for portage.  
> Having an XML definition for each package, yes... replacing 
> the actual build 
> code by XML - no.
> 
> My $0.02...
> 
> Alain
> 
> 
> On Thu, Feb 06, 2003 at 03:41:29AM -0600, Jared H. Hudson wrote:
> > I like XML as much as the next guy, but one thing you 
> should remember 
> > is
> > that ebuilds are not some arbitrary pkg format -- they're 
> bash scripts. 
> > They may not look it, since they don't have #!/bin/bash at 
> the top, but 
> > they are bash scripts that are sourced with other bash 
> scripts. So, for 
> > example, most ebuilds have bash functins like src_unpack, 
> src_install, 
> > ect, but others have their own functions that are defined 
> and called by 
> > these src_* functions. Converting this to XML would mean 
> that portage 
> > would have to convert everything to bash, plus there'd have 
> to be enough 
> > flexibility in the XML stylesheet we'd to include the 
> possibility of 
> > unknown-ebuild-specific global variables, functions, ect.
> > 
> > Just food for thought.
> > 
> > -Jared H.
> 
> --
> gentoo-dev@gentoo.org mailing list
> 


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Brian Friday]

<quote who="Alain Penders">
> Looking at the cheer size of Ant -- all the stuff they had to put in
> before it  became a really useful system, I'd vote against trying to do
> this for portage.   Having an XML definition for each package, yes...
> replacing the actual build  code by XML - no.

I agree as well, the example I gave earlier of a possible XML package file
was  based on my incorrect assumption portage was python and not bash
based. Not sure why I got that impression, especially as I realize all the
conf files have bash syntax...

My general thought was this: Craft a XML file which clearly identifies the
sections of the current ebuild system. Once this is done (again I was
thinking of python or perl here not bash) create a wrapper which acts as a
transition layer between portage and the new package XML file.

Clearly though I don't know enough about portage so please forgive my
past/current ignorance as I go back and read the manual.

--
Brian



--
gentoo-dev@gentoo.org mailing list

RE: [gentoo-dev] portage database management

[Riyad Kalla]

come to think of it, I thought it was all python as well....

anybody know why we thought this?

> -----Original Message-----
> From: Brian Friday [mailto:bfriday@lasierra.edu] 
> Sent: Thursday, February 06, 2003 3:21 PM
> To: gentoo-dev@gentoo.org
> Subject: Re: [gentoo-dev] portage database management
> 
> 
> 
> <quote who="Alain Penders">
> > Looking at the cheer size of Ant -- all the stuff they had 
> to put in 
> > before it  became a really useful system, I'd vote against 
> trying to do
> > this for portage.   Having an XML definition for each 
> package, yes...
> > replacing the actual build  code by XML - no.
> 
> I agree as well, the example I gave earlier of a possible XML 
> package file was  based on my incorrect assumption portage 
> was python and not bash based. Not sure why I got that 
> impression, especially as I realize all the conf files have 
> bash syntax...
> 
> My general thought was this: Craft a XML file which clearly 
> identifies the sections of the current ebuild system. Once 
> this is done (again I was thinking of python or perl here not 
> bash) create a wrapper which acts as a transition layer 
> between portage and the new package XML file.
> 
> Clearly though I don't know enough about portage so please 
> forgive my past/current ignorance as I go back and read the manual.
> 
> --
> Brian
> 
> 
> 
> --
> gentoo-dev@gentoo.org mailing list
> 


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] portage database management

[Mario Witt]

On Thursday 06 February 2003 23:30, Riyad Kalla wrote:
> come to think of it, I thought it was all python as well....
>
> anybody know why we thought this?
>

Ehm, bovine spongiform encephalopathy?

(Hint: Google)

--
Mario Witt


--
gentoo-dev@gentoo.org mailing list

RE: [gentoo-dev] portage database management

[Matt Tucker]

-- Riyad Kalla <rsk@u.arizona.edu> spake thusly:

>> -----Original Message-----
>> From: Brian Friday [mailto:bfriday@lasierra.edu] 
>> 
>> I agree as well, the example I gave earlier of a possible XML 
>> package file was  based on my incorrect assumption portage 
>> was python and not bash based. Not sure why I got that 
>> impression, especially as I realize all the conf files have 
>> bash syntax...

> come to think of it, I thought it was all python as well....
> 
> anybody know why we thought this?

Portage itself is implemented in Python, but the ebuild mechanism is
all done with shell scripts. It goes something like:

    emerge [python] loads:
    portage.py [python] and calls
    portage.doebuild() [python] which executes
    ebuild.sh [shell] which sources
    category/pkg/pkg-version.ebuild [shell]

The line is pretty blurred between the two though, so I could certainly
understand the confusion.


--
gentoo-dev@gentoo.org mailing list