[gentoo-dev] maybe it is time to put portage queries into a database.

[gentoo-dev] maybe it is time to put portage queries into a database.

[Tony Clark]

I wanted to quickly find a package with emerge -s whatever.  I noticed that it 
seemed to be taking a long time now for basic searches.  A couple of random 
example are presented below.  Now in my case on the machine in question, it 
is probably a bit slower than need be as /usr/portage is nfs mounted but 
neither the nfs server, this client are what would be called slow machines 
and I run 100baset ethernet. 

12 months ago I thought the search times where acceptable, nowdays they are 
pushing it and with the number of packages going into portage sooner rather 
than later it is going to be unacceptal.  I would also suggest a centralised 
server approach using something like mysql as apposed to localised databases 
based on berekley or something else.  reason being that a number a ppl are 
running home lans and they are increasing in popularity and there is no need 
to store N machines worth of data. mysql comes to mind as it is a popular 
database as there is a good chance that for some other application it is 
already running on the network.  I know in my case thats so.

I know this has been discussed in the past but always put down for reasons I 
largely agreed with, but the number of packages and times have changed.  I 
don't mind helping out with this with some perl etc.  What I was thinking on 
for the initial implementation would be just to store the information 
required for searching.  I don't see a need to store all dependcies etc so 
the installation process actually becomes database driven.  In this case the 
queriey times are so short to the actually installation time, no noticable 
speed up will take place.

The floor is open, let the flames begin :)

tony

tony@power tony $ time emerge -s zope
Searching...
[ Results for search key : zope ]
[ Applications found : 2 ]

*  app-admin/zope-config
      Latest version available: 0.1-r1
      Latest version installed: [ Not Installed ]
      Size of downloaded files: 0 kB
      Homepage:
      Description: A Gentoo Zope multi-Instance configure tool.

*  net-zope/zope
      Latest version available: 2.6.1
      Latest version installed: [ Not Installed ]
      Size of downloaded files: 2,317 kB
      Homepage:    http://www.zope.org
      Description: Zope is a web application platform used for building 
high-performance, dynamic web sites.



real    0m6.968s
user    0m1.310s
sys     0m0.660s


tony@power tony $ time emerge -S zope
Searching...
[ Results for search key : zope ]
[ Applications found : 27 ]

*  app-admin/zope-config
      Latest version available: 0.1-r1
      Latest version installed: [ Not Installed ]
      Size of downloaded files: 0 kB
      Homepage:
      Description: A Gentoo Zope multi-Instance configure tool.

*  net-zope/zope
      Latest version available: 2.6.1
      Latest version installed: [ Not Installed ]
      Size of downloaded files: 2,317 kB
      Homepage:    http://www.zope.org
      Description: Zope is a web application platform used for building 
high-performance, dynamic web sites.

*  app-admin/zprod-manager-0.1
      Latest version available: 0.1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.gentoo.org/
      Description: Gentoo Zope Product selection tool

*  app-admin/zprod-update-0.1 [ Masked ]
      Latest version available: 0.1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:
      Description: Gentoo Zope Product selection tool.

*  net-zope/abracadabraobject-1.5.1
      Latest version available: 1.5.1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zope.org/Members/mjablonski/AbracadabraObject
      Description: This can add pre-configured ZOPE-objects to folders through 
ZMI.

*  net-zope/btreefolder2-0.5.0
      Latest version available: 0.5.0
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://hathaway.freezope.org/Software/BTreeFolder2/
      Description: Acts like a Zope folder but can store many more items.

*  net-zope/cmfcollectorng-0.20
      Latest version available: 0.20
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    
http://www.zope.org/Members/ajung/CMFCollectorNG/Wiki/FrontPage
      Description: Zope/CMF-based bugtracking system.

*  net-zope/cmfforum-1.0
      Latest version available: 1.0
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.sf.net/projects/collective/
      Description: This is a zope product

*  net-zope/cmfphoto-0.2
      Latest version available: 0.2
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://sourceforge.net/projects/collective/
      Description: Zope product to have photos.

*  net-zope/cmfphotoalbum-0.2
      Latest version available: 0.2
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://sourceforge.net/projects/collective/
      Description: Zope/CMF product to organize e-pics into hierarchical photo 
album.

*  net-zope/extfile-1.2.0_beta2
      Latest version available: 1.2.0_beta2
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zope.org/Members/shh/ExtFile
      Description: Zope proxy objects for files on the filesystem

*  net-zope/filesystemsite-1.2
      Latest version available: 1.2
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    
http://www.zope.org/Members/k_vertigo/Products/FileSystemSite
      Description: Zope proxy objects for files on the filesystem.

*  net-zope/groups-0.3.1
      Latest version available: 0.3.1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zope.org/Members/faassen/Groups
      Description: Group support (including local roles) for Zope

*  net-zope/ldapuserfolder-2.1_beta3
      Latest version available: 2.1_beta3
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.dataflake.org/software/ldapuserfolder/
      Description: LDAP User Authentication for Zope.

*  net-zope/localfs-1.0.0
      Latest version available: 1.0.0
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://sourceforge.net/projects/localfs/
      Description: Zope product for accessing the local filesystem

*  net-zope/localizer-1.0.0-r1
      Latest version available: 1.0.0-r1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.localizer.org
      Description: Helps to build multilingual zope websites and zope 
products.

*  net-zope/neoportallibrary-0.9b
      Latest version available: 0.9b
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zoper.net/
      Description: Collection of modules to build Zope/CMF/Plone products.

*  net-zope/parsedxml-1.3.1
      Latest version available: 1.3.1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zope.org/Members/faassen/ParsedXML
      Description: XML objects for Zope.

*  net-zope/photo-1.2.3
      Latest version available: 1.2.3
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zope.org/Members/rbickers/Photo
      Description: Zope product for managing photos and photo albums

*  net-zope/plone-1.0.1-r1
      Latest version available: 1.0.1-r1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://plone.org
      Description: A Zope Content Management System, based on Zope CMF.

*  net-zope/squishdot-1.5.0
      Latest version available: 1.5.0
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://squishdot.org/
      Description: A web-based news publishing and discussion product for 
Zope.

*  net-zope/translationservice-0.3
      Latest version available: 0.3
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zope.org/Members/efge/TranslationService/
      Description: Translation servive for zope. i18n tags in ZPT.

*  net-zope/xmlwidgets-0.8.5
      Latest version available: 0.8.5
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zope.org/Members/faassen/XMLWidgets
      Description: UI widgets for Zope XML objects.

*  net-zope/zmysqlda-2.0.8
      Latest version available: 2.0.8
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://sourceforge.net/projects/mysql-python
      Description: A MySQL Database Adapter(DA) for zope.

*  net-zope/zphotoslides-1.1-r1 [ Masked ]
      Latest version available: 1.1-r1
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.zphotoslides.org/
      Description: Present your photos quickly in zope.

*  net-zope/zpsycopgda-1.1.2
      Latest version available: 1.1.2
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://www.initd.org/software/psycopg.py
      Description: PostgreSQL database adapter for Zope.

*  net-zope/zwiki-0.17.0
      Latest version available: 0.17.0
      Latest version installed: [ Not Installed ]
      Size of downloaded files:  [no/bad digest]
      Homepage:    http://zwiki.org/
      Description: A zope wiki-clone for easy-to-edit collaborative websites.



real    1m23.409s
user    0m26.580s
sys     0m11.130s

-- 
Contract ASIC and FPGA design.
Telephone +46 702 894 667
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x633E2623



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[jesse]

[-- Attachment #1: Type: text/plain, Size: 11749 bytes --]

I would say options, tools and the ability to do this sorta stuff should
be optional. I don't wan't to have to run mysql on my laptop however
having portage db in a dbfile would be handy. On the other hand i run 8
machines in my house on Gentoo, and having each machines info and db's
in my mysql server here would be a great boon ( already using local
syncing and nfs mounted dist dirs)

using DBI this could make it pretty easy to support
mysql/postrgesql/dbfiles. Isn't there already a project out there doin
somthing with mysql ?


0.02$

On Tue, 2003-06-24 at 15:00, Tony Clark wrote:
> I wanted to quickly find a package with emerge -s whatever.  I noticed that it 
> seemed to be taking a long time now for basic searches.  A couple of random 
> example are presented below.  Now in my case on the machine in question, it 
> is probably a bit slower than need be as /usr/portage is nfs mounted but 
> neither the nfs server, this client are what would be called slow machines 
> and I run 100baset ethernet. 
> 
> 12 months ago I thought the search times where acceptable, nowdays they are 
> pushing it and with the number of packages going into portage sooner rather 
> than later it is going to be unacceptal.  I would also suggest a centralised 
> server approach using something like mysql as apposed to localised databases 
> based on berekley or something else.  reason being that a number a ppl are 
> running home lans and they are increasing in popularity and there is no need 
> to store N machines worth of data. mysql comes to mind as it is a popular 
> database as there is a good chance that for some other application it is 
> already running on the network.  I know in my case thats so.
> 
> I know this has been discussed in the past but always put down for reasons I 
> largely agreed with, but the number of packages and times have changed.  I 
> don't mind helping out with this with some perl etc.  What I was thinking on 
> for the initial implementation would be just to store the information 
> required for searching.  I don't see a need to store all dependcies etc so 
> the installation process actually becomes database driven.  In this case the 
> queriey times are so short to the actually installation time, no noticable 
> speed up will take place.
> 
> The floor is open, let the flames begin :)
> 
> tony
> 
> tony@power tony $ time emerge -s zope
> Searching...
> [ Results for search key : zope ]
> [ Applications found : 2 ]
> 
> *  app-admin/zope-config
>       Latest version available: 0.1-r1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files: 0 kB
>       Homepage:
>       Description: A Gentoo Zope multi-Instance configure tool.
> 
> *  net-zope/zope
>       Latest version available: 2.6.1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files: 2,317 kB
>       Homepage:    http://www.zope.org
>       Description: Zope is a web application platform used for building 
> high-performance, dynamic web sites.
> 
> 
> 
> real    0m6.968s
> user    0m1.310s
> sys     0m0.660s
> 
> 
> tony@power tony $ time emerge -S zope
> Searching...
> [ Results for search key : zope ]
> [ Applications found : 27 ]
> 
> *  app-admin/zope-config
>       Latest version available: 0.1-r1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files: 0 kB
>       Homepage:
>       Description: A Gentoo Zope multi-Instance configure tool.
> 
> *  net-zope/zope
>       Latest version available: 2.6.1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files: 2,317 kB
>       Homepage:    http://www.zope.org
>       Description: Zope is a web application platform used for building 
> high-performance, dynamic web sites.
> 
> *  app-admin/zprod-manager-0.1
>       Latest version available: 0.1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.gentoo.org/
>       Description: Gentoo Zope Product selection tool
> 
> *  app-admin/zprod-update-0.1 [ Masked ]
>       Latest version available: 0.1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:
>       Description: Gentoo Zope Product selection tool.
> 
> *  net-zope/abracadabraobject-1.5.1
>       Latest version available: 1.5.1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zope.org/Members/mjablonski/AbracadabraObject
>       Description: This can add pre-configured ZOPE-objects to folders through 
> ZMI.
> 
> *  net-zope/btreefolder2-0.5.0
>       Latest version available: 0.5.0
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://hathaway.freezope.org/Software/BTreeFolder2/
>       Description: Acts like a Zope folder but can store many more items.
> 
> *  net-zope/cmfcollectorng-0.20
>       Latest version available: 0.20
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    
> http://www.zope.org/Members/ajung/CMFCollectorNG/Wiki/FrontPage
>       Description: Zope/CMF-based bugtracking system.
> 
> *  net-zope/cmfforum-1.0
>       Latest version available: 1.0
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.sf.net/projects/collective/
>       Description: This is a zope product
> 
> *  net-zope/cmfphoto-0.2
>       Latest version available: 0.2
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://sourceforge.net/projects/collective/
>       Description: Zope product to have photos.
> 
> *  net-zope/cmfphotoalbum-0.2
>       Latest version available: 0.2
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://sourceforge.net/projects/collective/
>       Description: Zope/CMF product to organize e-pics into hierarchical photo 
> album.
> 
> *  net-zope/extfile-1.2.0_beta2
>       Latest version available: 1.2.0_beta2
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zope.org/Members/shh/ExtFile
>       Description: Zope proxy objects for files on the filesystem
> 
> *  net-zope/filesystemsite-1.2
>       Latest version available: 1.2
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    
> http://www.zope.org/Members/k_vertigo/Products/FileSystemSite
>       Description: Zope proxy objects for files on the filesystem.
> 
> *  net-zope/groups-0.3.1
>       Latest version available: 0.3.1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zope.org/Members/faassen/Groups
>       Description: Group support (including local roles) for Zope
> 
> *  net-zope/ldapuserfolder-2.1_beta3
>       Latest version available: 2.1_beta3
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.dataflake.org/software/ldapuserfolder/
>       Description: LDAP User Authentication for Zope.
> 
> *  net-zope/localfs-1.0.0
>       Latest version available: 1.0.0
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://sourceforge.net/projects/localfs/
>       Description: Zope product for accessing the local filesystem
> 
> *  net-zope/localizer-1.0.0-r1
>       Latest version available: 1.0.0-r1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.localizer.org
>       Description: Helps to build multilingual zope websites and zope 
> products.
> 
> *  net-zope/neoportallibrary-0.9b
>       Latest version available: 0.9b
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zoper.net/
>       Description: Collection of modules to build Zope/CMF/Plone products.
> 
> *  net-zope/parsedxml-1.3.1
>       Latest version available: 1.3.1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zope.org/Members/faassen/ParsedXML
>       Description: XML objects for Zope.
> 
> *  net-zope/photo-1.2.3
>       Latest version available: 1.2.3
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zope.org/Members/rbickers/Photo
>       Description: Zope product for managing photos and photo albums
> 
> *  net-zope/plone-1.0.1-r1
>       Latest version available: 1.0.1-r1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://plone.org
>       Description: A Zope Content Management System, based on Zope CMF.
> 
> *  net-zope/squishdot-1.5.0
>       Latest version available: 1.5.0
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://squishdot.org/
>       Description: A web-based news publishing and discussion product for 
> Zope.
> 
> *  net-zope/translationservice-0.3
>       Latest version available: 0.3
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zope.org/Members/efge/TranslationService/
>       Description: Translation servive for zope. i18n tags in ZPT.
> 
> *  net-zope/xmlwidgets-0.8.5
>       Latest version available: 0.8.5
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zope.org/Members/faassen/XMLWidgets
>       Description: UI widgets for Zope XML objects.
> 
> *  net-zope/zmysqlda-2.0.8
>       Latest version available: 2.0.8
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://sourceforge.net/projects/mysql-python
>       Description: A MySQL Database Adapter(DA) for zope.
> 
> *  net-zope/zphotoslides-1.1-r1 [ Masked ]
>       Latest version available: 1.1-r1
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.zphotoslides.org/
>       Description: Present your photos quickly in zope.
> 
> *  net-zope/zpsycopgda-1.1.2
>       Latest version available: 1.1.2
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://www.initd.org/software/psycopg.py
>       Description: PostgreSQL database adapter for Zope.
> 
> *  net-zope/zwiki-0.17.0
>       Latest version available: 0.17.0
>       Latest version installed: [ Not Installed ]
>       Size of downloaded files:  [no/bad digest]
>       Homepage:    http://zwiki.org/
>       Description: A zope wiki-clone for easy-to-edit collaborative websites.
> 
> 
> 
> real    1m23.409s
> user    0m26.580s
> sys     0m11.130s

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Brian Jackson]

Totally agree. I have 5 Gentoo boxes at work (probably more soon), and it 
takes a ridiculously long amount of time to do searches. One thing I would 
suggest is to develop as an external tool, and once it matures, then shoot 
for inclusion into portage. another suggestion would be for it to support 
more than one type of db (sleepycat, postgres, mysql, gdbm for example). If 
you try to make some postgres zealot put mysql on just to query packages, you 
are going to get some very unhappy words directed your way. I can try to help 
also in my spare time.

--Brian Jackson

On Tuesday 24 June 2003 05:00 pm, Tony Clark wrote:
> I wanted to quickly find a package with emerge -s whatever.  I noticed that
> it seemed to be taking a long time now for basic searches.  A couple of
> random example are presented below.  Now in my case on the machine in
> question, it is probably a bit slower than need be as /usr/portage is nfs
> mounted but neither the nfs server, this client are what would be called
> slow machines and I run 100baset ethernet.
>
> 12 months ago I thought the search times where acceptable, nowdays they are
> pushing it and with the number of packages going into portage sooner rather
> than later it is going to be unacceptal.  I would also suggest a
> centralised server approach using something like mysql as apposed to
> localised databases based on berekley or something else.  reason being that
> a number a ppl are running home lans and they are increasing in popularity
> and there is no need to store N machines worth of data. mysql comes to mind
> as it is a popular database as there is a good chance that for some other
> application it is already running on the network.  I know in my case thats
> so.
>
> I know this has been discussed in the past but always put down for reasons
> I largely agreed with, but the number of packages and times have changed. 
> I don't mind helping out with this with some perl etc.  What I was thinking
> on for the initial implementation would be just to store the information
> required for searching.  I don't see a need to store all dependcies etc so
> the installation process actually becomes database driven.  In this case
> the queriey times are so short to the actually installation time, no
> noticable speed up will take place.
>
> The floor is open, let the flames begin :)
>
> tony
>
<snipped times>

-- 
OpenGFS -- http://opengfs.sourceforge.net
Home -- http://www.brianandsara.net


--
gentoo-dev@gentoo.org mailing list

RE: [gentoo-dev] maybe it is time to put portage queries into a database.

[Anthony Floyd]

[snip]
>tony@power tony $ time emerge -S zope
[snip]
>real    1m23.409s
>user    0m26.580s
>sys     0m11.130s

How important is this?  Well consider the following on an "old" 
machine (PP200):

# time emerge -S zope
[snip]
real    6m38.645s
user    2m16.580s
sys     0m10.930s

So, while I agree that this pushes the bounds of patience, I don't want
to be required to install MySQL or any other db engine just to run 
portage.  I'm sure many others don't want that other.  However, having 
an optional component, say capable of handling multiple databases and 
activated with USE flags ... that makes sense.  Ideally, optimizing the 
current system without requiring a massive db engine would be better, 
though.

Just some thoughts.

A>

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Tony Clark]

On Wednesday 25 June 2003 00.16, jesse wrote:
> I would say options, tools and the ability to do this sorta stuff should
> be optional. I don't wan't to have to run mysql on my laptop however
> having portage db in a dbfile would be handy. On the other hand i run 8
> machines in my house on Gentoo, and having each machines info and db's
> in my mysql server here would be a great boon ( already using local
> syncing and nfs mounted dist dirs)

Yes as soon as I posted it, I realised it should be an optional add on.  I did 
another quick test and emerge -s actually runs much faster on the nfs server, 
x5, however,  emerge -S is just as big a pain.
 
>
> using DBI this could make it pretty easy to support
> mysql/postrgesql/dbfiles. Isn't there already a project out there doin
> somthing with mysql ?

Not that I am aware of.  If there is let me know.

>
>
> 0.02$
>
> On Tue, 2003-06-24 at 15:00, Tony Clark wrote:
> > I wanted to quickly find a package with emerge -s whatever.  I noticed
> > that it seemed to be taking a long time now for basic searches.  A couple
> > of random example are presented below.  Now in my case on the machine in
> > question, it is probably a bit slower than need be as /usr/portage is nfs
> > mounted but neither the nfs server, this client are what would be called
> > slow machines and I run 100baset ethernet.

tony
-- 
Contract ASIC and FPGA design.
Telephone +46 702 894 667
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x633E2623



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Henti Smith]

On 25 Jun 2003 00:52:42 +0200
Julien Herfurth <jherfurth@arafox.com> wrote:

> On Wed, 2003-06-25 at 00:16, jesse wrote:
> > I would say options, tools and the ability to do this sorta stuff should
> > be optional. I don't wan't to have to run mysql on my laptop however
> > having portage db in a dbfile would be handy. On the other hand i run 8
> > machines in my house on Gentoo, and having each machines info and db's
> > in my mysql server here would be a great boon ( already using local
> > syncing and nfs mounted dist dirs)
> > 
> > using DBI this could make it pretty easy to support
> > mysql/postrgesql/dbfiles. Isn't there already a project out there doin
> > somthing with mysql ?
> 
> see also sqlite

isn't db a system lib .. and installed on every machine ? 

can't you just use that ? 

-- 
Henti Smith
bain@tcsn.co.za
Senior Administrator
The Computer-Smith Networking
http://www.tcsn.co.za

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Marius Mauch]

On Wed, 25 Jun 2003 00:00:00 +0200 Tony Clark wrote:

> 12 months ago I thought the search times where acceptable, nowdays
> they are pushing it and with the number of packages going into portage
> sooner rather than later it is going to be unacceptal.  I would also
> suggest a centralised server approach using something like mysql as
> apposed to localised databases based on berekley or something else. 
> reason being that a number a ppl are running home lans and they are
> increasing in popularity and there is no need to store N machines
> worth of data. mysql comes to mind as it is a popular database as
> there is a good chance that for some other application it is already
> running on the network.  I know in my case thats so.
> 
> I know this has been discussed in the past but always put down for
> reasons I largely agreed with, but the number of packages and times
> have changed.  I don't mind helping out with this with some perl etc. 
> What I was thinking on for the initial implementation would be just to
> store the information required for searching.  I don't see a need to
> store all dependcies etc so the installation process actually becomes
> database driven.  In this case the queriey times are so short to the
> actually installation time, no noticable speed up will take place.

I'd suggest a modular approach, so that portage can support different
backends (and frontends) easier. Those who want a minimal systen can
continue to use the filesystem while people who look for more
performance can use a database server. That would require a lot of work
on portage though, probably more than carpaski alone can handle. If
there is interest on the dev side I'd be more than willing to help with
this (although my python skills are lacking, I'm more a Java and C
programmer, but I'm learning fast ;-).

BTW, I also noted the slowness of the search function, especially when
searching in descriptions, my worst measured result was 25 minutes! Most
likely caused by a serious lack of ram, but half an hour just to find a
package is really annoying.

Marius

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Tony Clark]

On Wednesday 25 June 2003 00.16, Brian Jackson wrote:
> Totally agree. I have 5 Gentoo boxes at work (probably more soon), and it
> takes a ridiculously long amount of time to do searches. One thing I would
> suggest is to develop as an external tool, and once it matures, then shoot
> for inclusion into portage. another suggestion would be for it to support
> more than one type of db (sleepycat, postgres, mysql, gdbm for example). If

Yes, it would be nice to make it that way.  I guess something like client -> 
client-db-interface-layer -> db-of-choice.

> you try to make some postgres zealot put mysql on just to query packages,
> you are going to get some very unhappy words directed your way. I can try
> to help also in my spare time.

Well there are plenty of perl modules to handle client-db-interface-layer but 
I suspect they will all have different APIs for the client.  My thoughts are 
get one system to work then build on that.  So long as the design is modular, 
that should be too difficult for those interested. :)

tony

>
> --Brian Jackson
>
> On Tuesday 24 June 2003 05:00 pm, Tony Clark wrote:
> > I wanted to quickly find a package with emerge -s whatever.  I noticed
> > that it seemed to be taking a long time now for basic searches.  A couple
> > of random example are presented below.  Now in my case on the machine in
> > question, it is probably a bit slower than need be as /usr/portage is nfs
> > mounted but neither the nfs server, this client are what would be called
> > slow machines and I run 100baset ethernet.
> >
> > 12 months ago I thought the search times where acceptable, nowdays they
> > are pushing it and with the number of packages going into portage sooner
> > rather than later it is going to be unacceptal.  I would also suggest a
> > centralised server approach using something like mysql as apposed to
> > localised databases based on berekley or something else.  reason being
> > that a number a ppl are running home lans and they are increasing in
> > popularity and there is no need to store N machines worth of data. mysql
> > comes to mind as it is a popular database as there is a good chance that
> > for some other application it is already running on the network.  I know
> > in my case thats so.
> >
> > I know this has been discussed in the past but always put down for
> > reasons I largely agreed with, but the number of packages and times have
> > changed. I don't mind helping out with this with some perl etc.  What I
> > was thinking on for the initial implementation would be just to store the
> > information required for searching.  I don't see a need to store all
> > dependcies etc so the installation process actually becomes database
> > driven.  In this case the queriey times are so short to the actually
> > installation time, no noticable speed up will take place.
> >
> > The floor is open, let the flames begin :)
> >
> > tony
>
> <snipped times>

-- 
Contract ASIC and FPGA design.
Telephone +46 702 894 667
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x633E2623



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Julien Herfurth]

On Wed, 2003-06-25 at 00:16, jesse wrote:
> I would say options, tools and the ability to do this sorta stuff should
> be optional. I don't wan't to have to run mysql on my laptop however
> having portage db in a dbfile would be handy. On the other hand i run 8
> machines in my house on Gentoo, and having each machines info and db's
> in my mysql server here would be a great boon ( already using local
> syncing and nfs mounted dist dirs)
> 
> using DBI this could make it pretty easy to support
> mysql/postrgesql/dbfiles. Isn't there already a project out there doin
> somthing with mysql ?

see also sqlite


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Henti Smith]

On 24 Jun 2003 16:04:58 -0700
jesse <yoda@f00bar.com> wrote:

> > isn't db a system lib .. and installed on every machine ? 
> > 
> > can't you just use that ? 
> 
> My proposition was to use some connector lib that is compatable with all
> those db formats. so you don't have to re-write a bunch of backend
> drivers or DB specific queries. Im not a python programmer C/Perl are my
> main forte, and you can use DBI to interface with many many db's as long
> as you keep your queries db agnostic, and use the dbi methods. At least
> with postgresql, mysql, and oracle to (some extent) this is the case.  

but since the db system is already part of the default system, why not just use that.
It's really a system function .. and I knwo th "selectable" databse of choice is a nice to have
but do we (gentoo) really want people to run different DB's for portage backend stuff .. and handle the bug tracking hell that will result because of it ? 

-- 
Henti Smith
bain@tcsn.co.za
Senior Administrator
The Computer-Smith Networking
http://www.tcsn.co.za

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Tony Clark]

On Wednesday 25 June 2003 00.33, Anthony Floyd wrote:
> [snip]
>
> >tony@power tony $ time emerge -S zope
>
> [snip]
>
> >real    1m23.409s
> >user    0m26.580s
> >sys     0m11.130s
>
> How important is this?  Well consider the following on an "old"
> machine (PP200):
>
> # time emerge -S zope
> [snip]
> real    6m38.645s
> user    2m16.580s
> sys     0m10.930s
>
> So, while I agree that this pushes the bounds of patience, I don't want
> to be required to install MySQL or any other db engine just to run
> portage.  I'm sure many others don't want that other.  However, having
> an optional component, say capable of handling multiple databases and
> activated with USE flags ... that makes sense.  Ideally, optimizing the
> current system without requiring a massive db engine would be better,
> though.
The problem is that the current system doesn't scale well as it is all text 
based files.  There is already caching so if you make the same query and it 
is still in the cache things go a lot faster.  It is also creating a lot of 
unnecessary network traffic where portage isn't local to the machine.  Not a 
big issue yet most likely but it could be an issue when more media streaming 
comes into play.

tony
-- 
Contract ASIC and FPGA design.
Telephone +46 702 894 667
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x633E2623



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Tony Clark]

On Wednesday 25 June 2003 00.52, Julien Herfurth wrote:
> On Wed, 2003-06-25 at 00:16, jesse wrote:
> > I would say options, tools and the ability to do this sorta stuff should
> > be optional. I don't wan't to have to run mysql on my laptop however
> > having portage db in a dbfile would be handy. On the other hand i run 8
> > machines in my house on Gentoo, and having each machines info and db's
> > in my mysql server here would be a great boon ( already using local
> > syncing and nfs mounted dist dirs)
> >
> > using DBI this could make it pretty easy to support
> > mysql/postrgesql/dbfiles. Isn't there already a project out there doin
> > somthing with mysql ?
>
> see also sqlite

Looks good, but I see that as being handled by the client-db-interface layer 
sometime in the future.

tony
-- 
Contract ASIC and FPGA design.
Telephone +46 702 894 667
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x633E2623



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[jesse]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]


> isn't db a system lib .. and installed on every machine ? 
> 
> can't you just use that ? 

My proposition was to use some connector lib that is compatable with all
those db formats. so you don't have to re-write a bunch of backend
drivers or DB specific queries. Im not a python programmer C/Perl are my
main forte, and you can use DBI to interface with many many db's as long
as you keep your queries db agnostic, and use the dbi methods. At least
with postgresql, mysql, and oracle to (some extent) this is the case.  

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[kl4rk]

> I'd suggest a modular approach, so that portage can support different
> backends (and frontends) easier. Those who want a minimal systen can
> continue to use the filesystem while people who look for more
> performance can use a database server. That would require a lot of work
> on portage though, probably more than carpaski alone can handle. If
> there is interest on the dev side I'd be more than willing to help with
> this (although my python skills are lacking, I'm more a Java and C
> programmer, but I'm learning fast ;-).
> 
> BTW, I also noted the slowness of the search function, especially when
> searching in descriptions, my worst measured result was 25 minutes! Most
> likely caused by a serious lack of ram, but half an hour just to find a
> package is really annoying.
> 
> Marius

It would be great to have some kind of 'portage server' that could serve not only search requests etc, but also src tarballs for the packages and even compiled packages (that could be served if the USE settings they were compiled with were the same as the USE flags and architecture the client was about to compile with). That way, each LAN could have its gentoo server other boxes would just query. That server could act as a cache between public mirrors (rsync and ftp mirrors) and clients, thus reducing external bandwidth usage.

That would make gentoo better for big lans with many Gentoo boxes, and maybe it could be a starting point towards having public servers to offer some compiled packages. I know many people don't like the idea, but I would love to have the option of using those compiled packages for at least big apps, eg kde.

Josep Sanjuas

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Brian Jackson]

On Tuesday 24 June 2003 05:58 pm, Henti Smith wrote:
> On 24 Jun 2003 16:04:58 -0700
>
> jesse <yoda@f00bar.com> wrote:
> > > isn't db a system lib .. and installed on every machine ?
> > >
> > > can't you just use that ?
> >
> > My proposition was to use some connector lib that is compatable with all
> > those db formats. so you don't have to re-write a bunch of backend
> > drivers or DB specific queries. Im not a python programmer C/Perl are my
> > main forte, and you can use DBI to interface with many many db's as long
> > as you keep your queries db agnostic, and use the dbi methods. At least
> > with postgresql, mysql, and oracle to (some extent) this is the case.
>
> but since the db system is already part of the default system, why not just
> use that. It's really a system function .. and I knwo th "selectable"
> databse of choice is a nice to have but do we (gentoo) really want people
> to run different DB's for portage backend stuff .. and handle the bug
> tracking hell that will result because of it ?

I don't think that would scale well(sure a couple of boxes, but I'm probably 
going to have over ten by the end of the year, and if it scales well enough, 
that would be a boon for cluster environments where there can be thousands of 
nodes). I like the idea of having many backends available. That way, people 
could use whatever suited their environment. Aside from that the libs you are 
talking about are c libs. If this project aims to be included as part of 
portage at some point, it needs to be written in portage's native language. I 
think python has some basic db lib interface built in, I don't know about 
mysql/postgres/etc.

--Brian Jackson

-- 
OpenGFS -- http://opengfs.sourceforge.net
Home -- http://www.brianandsara.net


--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] maybe it is time to put portage queries into a database.

[Tony Clark]

My mail relay died so I haven't been able to reply but here are my basic 
thoughts of the database layouts.  db,table and fields only, at this stage.
DB name gentoo # We may find uses for other gentoo related tables later.
Table portage
Field Name				Type	Length
PortageGroup				text		20		# [media-tv]
PortagePackage			text		30		# [xawtv]
PortageInstalledVersion		text		20		#Current installed version. [3.86-r1]
PortageStableVersion		text		20		#Newest unmasked version (x86)
PortageTestingVersion		text		20		#Newest testing version (~x86)
PortagePackageHomepage	text		40		
PortagePackageDescription	text		160

Some ideas for commands

portsearch -lg <PortageGroup> # If portage group is specified list all
						    # packages in that group else list everything.
						    # Allow partial matches.
portsearch -lp <PortagePackage>  # Return all packages which do a partial
							# match.  basically the same as
							# emerge -s
portsearch -ld	<phrase>  		# return all matchs in description.  emerge -S

These are just the basic ones, there are more but I think you get the general 
idea.  I'm not sure if the dependancies should be in there as well as a list 
of files belong to the package.
If we put package dependancies in it package it would make it easy to remove 
packages with all their dependants, which as far as I am aware, isn't handled 
by portage yet.

tony
-- 
Contract ASIC and FPGA design.
Telephone +46 702 894 667
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x633E2623



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[jesse]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

this is doable for distfiles and rsync now (and for arch built packages)
its really managing the rest of portage that needs some way of bein
centralized

there was a thread  on forums that got some play in the GWN:

http://forums.gentoo.org/viewtopic.php?t=59134


On Tue, 2003-06-24 at 16:14, kl4rk wrote:
> > I'd suggest a modular approach, so that portage can support different
> > backends (and frontends) easier. Those who want a minimal systen can
> > continue to use the filesystem while people who look for more
> > performance can use a database server. That would require a lot of work
> > on portage though, probably more than carpaski alone can handle. If
> > there is interest on the dev side I'd be more than willing to help with
> > this (although my python skills are lacking, I'm more a Java and C
> > programmer, but I'm learning fast ;-).
> > 
> > BTW, I also noted the slowness of the search function, especially when
> > searching in descriptions, my worst measured result was 25 minutes! Most
> > likely caused by a serious lack of ram, but half an hour just to find a
> > package is really annoying.
> > 
> > Marius
> 
> It would be great to have some kind of 'portage server' that could serve not only search requests etc, but also src tarballs for the packages and even compiled packages (that could be served if the USE settings they were compiled with were the same as the USE flags and architecture the client was about to compile with). That way, each LAN could have its gentoo server other boxes would just query. That server could act as a cache between public mirrors (rsync and ftp mirrors) and clients, thus reducing external bandwidth usage.
> 
> That would make gentoo better for big lans with many Gentoo boxes, and maybe it could be a starting point towards having public servers to offer some compiled packages. I know many people don't like the idea, but I would love to have the option of using those compiled packages for at least big apps, eg kde.
> 
> Josep Sanjuas
> 
> --
> gentoo-dev@gentoo.org mailing list


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Tony Clark]

On Wednesday 25 June 2003 01.57, Brian Jackson wrote:
> On Tuesday 24 June 2003 05:58 pm, Henti Smith wrote:
> > On 24 Jun 2003 16:04:58 -0700
> >
> > jesse <yoda@f00bar.com> wrote:
> > > > isn't db a system lib .. and installed on every machine ?
> > > >
> > > > can't you just use that ?
> > >
> > > My proposition was to use some connector lib that is compatable with
> > > all those db formats. so you don't have to re-write a bunch of backend
> > > drivers or DB specific queries. Im not a python programmer C/Perl are
> > > my main forte, and you can use DBI to interface with many many db's as
> > > long as you keep your queries db agnostic, and use the dbi methods. At
> > > least with postgresql, mysql, and oracle to (some extent) this is the
> > > case.
> >
> > but since the db system is already part of the default system, why not
> > just use that. It's really a system function .. and I knwo th
> > "selectable" databse of choice is a nice to have but do we (gentoo)
> > really want people to run different DB's for portage backend stuff .. and
> > handle the bug tracking hell that will result because of it ?
>
> I don't think that would scale well(sure a couple of boxes, but I'm
> probably going to have over ten by the end of the year, and if it scales
> well enough, that would be a boon for cluster environments where there can
> be thousands of nodes). I like the idea of having many backends available.
> That way, people could use whatever suited their environment. Aside from
> that the libs you are talking about are c libs. If this project aims to be
> included as part of portage at some point, it needs to be written in
> portage's native language. I think python has some basic db lib interface
> built in, I don't know about mysql/postgres/etc.

I know SFA about python and see an initial implementation in perl as the 
fastest way to get something going at least for me.

tony

-- 
Contract ASIC and FPGA design.
Telephone +46 702 894 667
http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x633E2623



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[jesse]

[-- Attachment #1: Type: text/plain, Size: 1813 bytes --]

I have 10 machines at home and I am building 15 right now for work. This
sort of stuff is very interesting to me. 

the point wasn't about what to do localy, but what options to have for
some central management. I think that the idea that maybe this project
should start outside of portage was a good one. Sure for local db berkDB
or whatever sounds great. My point was why not make it DB agnostic so
that it can integrate with whatever the USER wants. After all one of the
reasons i like Gentoo over other distros is .. _choice_ . . 

The fact that im not a python guy ( never even really wrote hello world
) has held me back from even attempting such a feat. Writing this in
anything but python will probly ensure it wont get into portage :P 





On Tue, 2003-06-24 at 15:58, Henti Smith wrote:
> On 24 Jun 2003 16:04:58 -0700
> jesse <yoda@f00bar.com> wrote:
> 
> > > isn't db a system lib .. and installed on every machine ? 
> > > 
> > > can't you just use that ? 
> > 
> > My proposition was to use some connector lib that is compatable with all
> > those db formats. so you don't have to re-write a bunch of backend
> > drivers or DB specific queries. Im not a python programmer C/Perl are my
> > main forte, and you can use DBI to interface with many many db's as long
> > as you keep your queries db agnostic, and use the dbi methods. At least
> > with postgresql, mysql, and oracle to (some extent) this is the case.  
> 
> but since the db system is already part of the default system, why not just use that.
> It's really a system function .. and I knwo th "selectable" databse of choice is a nice to have
> but do we (gentoo) really want people to run different DB's for portage backend stuff .. and handle the bug tracking hell that will result because of it ? 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Matthew Kennedy]

Marius Mauch <genone@genone.homeip.net> writes:

> On Wed, 25 Jun 2003 00:00:00 +0200 Tony Clark wrote:
>
>
> BTW, I also noted the slowness of the search function, especially when
> searching in descriptions, my worst measured result was 25 minutes! Most
> likely caused by a serious lack of ram, but half an hour just to find a
> package is really annoying.

It would be interesting to see how much of a difference a Berkely DB
search would make (for emerge -s).


-- 
Matthew Kennedy
Gentoo Linux Developer

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 2002 bytes --]

On Wed, Jun 25, 2003 at 12:36:00AM +0200, Tony Clark wrote:
[... Portage with DB backend ...]
> Not that I am aware of.  If there is let me know.

Yes, well, no... not really that I know of. Hmm, let me explain :)

We (a small research group) are currently investigating bringing Gentoo Linux
desktops to companies. One of the things we're investigating is having
Portage on a remote MySQL database _but_ still having a local Portage for
when the network goes down. This is indeed rather easy (it needs some patches
to portage.py so that it queries the server and not parses it's own cache or
ebuild collection) but our current implementation is faulty (well, it works
great, but it isn't flexible).

The next choice would be to have Portage enhanced with databasefunctionality
(using Python's DB-API) and support for remote databases. This way a local
Gentoo desktop (as most of the people have) would store it's information in
normal ebuilds (default behaviour), a local Berkeley DB (optional) or a
remote SQL-server (optional). This would mean that Portage be enhanced with a
medium-independent layer, and several sublayers (one for each medium) which
takes care of the medium-dependent stuff.

Alain was working on a similar layer-implementation, but has dropped it in
favor of another project. 

Also, as you might now, this would be a major rewrite, so it's not the
current priority :) So if you want to fasten the search-functionality, write
a script that parses /usr/portage/*/*/*.ebuild and puts it in a database,
create a small script that does a SQL-statement against the database, and use
that script instead of "emerge -[sS]".

Wkr,
	Sven Vermeulen

-- 
Thanks to DRM, you know that something has been built in environment of 
unspecified degree of security, from source you cannot check, written by 
programmers you don't know, released after passing QA of unknown quality and 
which is released under a license that disclaims any responsibility...

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Henti Smith]

On Tue, 24 Jun 2003 15:33:37 -0700
"Anthony Floyd" <Anthony.Floyd@composites.ubc.ca> wrote:

> [snip]
> >tony@power tony $ time emerge -S zope
> [snip]
> >real    1m23.409s
> >user    0m26.580s
> >sys     0m11.130s
> 
> How important is this?  Well consider the following on an "old" 
> machine (PP200):
> 
> # time emerge -S zope
> [snip]
> real    6m38.645s
> user    2m16.580s
> sys     0m10.930s
> 

feedback 

Pentium 200 mmx 64MB ram *Smile* yes it's a production machine ;P 
uptime: 09:40:24 up  4:35,  4 users,  load average: 4.14, 3.17, 2.90
real    121m10.028s
user    78m12.865s
sys     29m34.110s

*yes ... it's true*

PII 350 128MB ram
uptime: 00:38:07 up 13 days, 10:39,  3 users,  load average: 3.60, 3.42, 3.21
real    33m13.179s
user    16m19.768s
sys     7m12.774s

Celeron 633 256MB ram
uptime: 00:26:53 up 7 days,  5:35, 15 users,  load average: 0.02, 0.12, 0.55
real    1m55.049s
user    1m23.220s
sys     0m13.600s

PPRO 200 128MB ram
uptime: 02:28:30 up 20 days, 11:20,  1 user,  load average: 0.00, 0.00, 0.00
real    39m31.880s
user    31m26.380s
sys     6m12.910s

Dual PPRO 200 128MB ram
02:38:14 up 111 days,  9:30,  2 users,  load average: 0.11, 0.35, 0.18
real    3m25.934s
user    2m35.240s
sys     0m18.110s

PII 350 64MB ram
uptime: 02:28:30 up 20 days, 11:20,  1 user,  load average: 0.00, 0.00, 0.00
real    2m44.061s
user    1m17.700s
sys     0m5.250s






-- 
Henti Smith
bain@tcsn.co.za
Senior Administrator
The Computer-Smith Networking
http://www.tcsn.co.za

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1127 bytes --]

On Wednesday 25 June 2003 00:41, Marius Mauch wrote:
>
> BTW, I also noted the slowness of the search function, especially when
> searching in descriptions, my worst measured result was 25 minutes! Most
> likely caused by a serious lack of ram, but half an hour just to find a
> package is really annoying.
>

The problem is the enormous amount of ebuilds we offer. The amount of ebuilds 
is currently: 8553 and the amount of files is: 50435.

Try the following script to make searching faster:
find /usr/portage/ -name "*.ebuild" -exec grep -H DESCRIPTION {} \; \
  |awk -F: '{ sub("(/usr/portage/[^/]*/)","",$1); sub("/.*","",$1); \
  printf("%-20s:%s\n", $1, $2) }'|sort |uniq >~/descriptions

This will make a nice file with all descriptions for all packages, filtered 
for duplicates. Just use grep on this file instead of emerge -S and you can 
find the package. If you put the creation of this file in crontab, you have 
yourself a fast and flexible solution, with only 4726 lines of descriptions.

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[kl4rk]

On Wed, 25 Jun 2003 12:39:28 +0200
Paul de Vrieze <pauldv@gentoo.org> wrote:

> On Wednesday 25 June 2003 00:41, Marius Mauch wrote:
> >
> > BTW, I also noted the slowness of the search function, especially when
> > searching in descriptions, my worst measured result was 25 minutes! Most
> > likely caused by a serious lack of ram, but half an hour just to find a
> > package is really annoying.
> >
> 
> This will make a nice file with all descriptions for all packages, filtered 
> for duplicates. Just use grep on this file instead of emerge -S and you can 
> find the package. If you put the creation of this file in crontab, you have 
> yourself a fast and flexible solution, with only 4726 lines of descriptions.
> 
> Paul

Maybe instead of having a description in the ebuild itself, there could be a single file per package cointaining the description (and maybe other stuff). That'd make searches faster. There's no point in having the description in the ebuild as it is something static that won't probably change between versions. Having it in the ebuild means (not 100% sure here) parsing the entire ebuild just to read the descr, which is too much work for a single line of text. If the number of packages continues to grow that'll become a bigger and bigger pain.

Having a sepparate file for descriptions might allow one day easlily adding bigger descriptions (for the user not to need visiting the package's homepage).

Josep

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[rob holland]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]


--On Wednesday, June 25, 2003 13:01:36 +0200 kl4rk <kl4rkmail@jazzfree.com> 
wrote:

> Having a sepparate file for descriptions might allow one day easlily
> adding bigger descriptions (for the user not to need visiting the
> package's homepage).

No disrespect intended but this has been said about a bazillion times on 
the -dev list now.

There will possibly be description stuff in the metadata file which paul 
has posted to the list about before. Please read the dtd he's put forward 
and comment on that.

Regards,

Rob

--

robh@gentoo.org / robh:irc.freenode.net
http://cvs.gentoo.org/~robh/robh@gentoo.org.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Mark Bainter]

Julien Herfurth [jherfurth@arafox.com] wrote:
> On Wed, 2003-06-25 at 00:16, jesse wrote:
> > I would say options, tools and the ability to do this sorta stuff should
> > be optional. I don't wan't to have to run mysql on my laptop however
> > having portage db in a dbfile would be handy. On the other hand i run 8
> > machines in my house on Gentoo, and having each machines info and db's
> > in my mysql server here would be a great boon ( already using local
> > syncing and nfs mounted dist dirs)
> > 
> > using DBI this could make it pretty easy to support
> > mysql/postrgesql/dbfiles. Isn't there already a project out there doin
> > somthing with mysql ?
> 
> see also sqlite

See also firebird.  It's small, and thus likely to raise less objections
from people like me as an installation dependency.  It can also be used 
locally or over the network.  And it supports sql syntax.

-- 
The urge to save humanity is almost always only a false-face for the urge
to rule it.  
				--H.L. Mencken

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] maybe it is time to put portage queries into a database.

[Eric Sammer]

Someone stated:
>I would say options, tools and the ability to do this sorta stuff should
>be optional. I don't wan't to have to run mysql on my laptop however
>having portage db in a dbfile would be handy. On the other hand i run 8
>machines in my house on Gentoo, and having each machines info and db's
>in my mysql server here would be a great boon ( already using local
>syncing and nfs mounted dist dirs)

Daring to broach the "enterprise" related issues, I would think that 
this would be one of those things that makes deploying and managing a 
medium network (>= 500 servers) difficult. To be the chameleon of an OS 
the average */Linux distro needs to be (netapp, workstation, kiosk, 
"normal" server, cluster node, etc.) these (hold nose here) "enterprise" 
features should be implemented in such a way as to easily switch between 
  these applications (of usage).

Good example of quick changes: nsswitch as it relates to host info (many 
options, simple to change between).

Bad example: switching file system types (many options, but not "flick 
of switch" easy to change).

Maybe, as an option, moving to a directory service (openldap, et al) for 
portage naming and access should be considered. Tools similiar to 
Apple/NeXTSTEP/OpenSTEP 'niload, nicl, nidump' could be used to switch 
between and migrate easily (or at least more so).

Pros:
1. extensible
2. distributed
3. access bindings for most common languages (python, perl, c, c++, 
etc.) which avoid that particular hot-button.
4. easy to replicate (see #1 + #2)
5. opens the option of moving other system services easily into the same 
system (passwd, groups, hosts, email, etc.)

Cons:
1. can be complex for newbies (overcome with nice tools like 
mirrorselect, ufed, emerge, et al)
2. dependencies (see #3 below)
3. potential bloat (arguable and should be considered in relation to the 
"chameleon" introductory and migration paragraph).

The virt host email howto could give the user another option to drop 
mysql for the directory service thus achieving even more use and sharing 
of resources. This also opens the company wide enterprise directory (as 
in addressbook) issue making Gentoo that much more appealing to such 
entities.

Of course, this is something that has probably been kicked around. In 
this instance, the option of making Gentoo more accessible to corporate 
and other (hold nose) "enterprise" environments is available.

I've worked on many similar ideas and systems in the past and would love 
to work on something like this for Gentoo.

(sorry if this is an old suggestion)

Regards...

-- 
Eric Sammer
eric@ineoconcepts.com
http://www.ineoconcepts.com


--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Eric Sammer]

Just to add something to my last email about portage and database / 
directory services...

While talking to my wife this morning (a sysadmin and security 
professional and fellow gentoo user) about this situation, we realized 
something else about portage in openldap. This would (or could, 
depending on implementation) severely limit the rsync bottle neck by 
allowing for a hierarchy of directory servers to be replicated from by 
users. With clever use of referals and replication, you could 
effectively remove the rsync issues of bandwidth and the 
"stop-syncing-so-often-it's-rude" problem. Updates to portage would be 
propagated down the line when commited moving a smaller (but steady) 
stream of traffic rather than unpredictable bursts (note: that's an 
assumption). This would eliminate the need for 'emerge sync' (in theory).

Again, this is all very academic as the data to back up these ideas is 
out of the public eye (thankfully). Maybe just food for thought...

Thanks to all devs for all the great work.
-- 
Eric Sammer
eric@ineoconcepts.com
http://www.ineoconcepts.com


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Rigo Ketelings]

Op vr 27-06-2003, om 14:23 schreef Eric Sammer:
> Just to add something to my last email about portage and database / 
> directory services...
> 
> While talking to my wife this morning (a sysadmin and security 
> professional and fellow gentoo user) about this situation, we realized 
> something else about portage in openldap. This would (or could, 
> depending on implementation) severely limit the rsync bottle neck by 
> allowing for a hierarchy of directory servers to be replicated from by 
> users.

I REALLY like this idea..

>  With clever use of referals and replication, you could 
> effectively remove the rsync issues of bandwidth and the 
> "stop-syncing-so-often-it's-rude" problem. Updates to portage would be 
> propagated down the line when commited moving a smaller (but steady) 
> stream of traffic rather than unpredictable bursts (note: that's an 
> assumption). This would eliminate the need for 'emerge sync' (in theory).
> 
> Again, this is all very academic as the data to back up these ideas is 
> out of the public eye (thankfully). Maybe just food for thought...
> 
> Thanks to all devs for all the great work.

Yup, can't say that enough too ;)...


Rogi



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Zack Gilburd]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1876 bytes --]

On Friday 27 June 2003 05:58 am, Rigo Ketelings wrote:
> Op vr 27-06-2003, om 14:23 schreef Eric Sammer:
> > Just to add something to my last email about portage and database /
> > directory services...
> >
> > While talking to my wife this morning (a sysadmin and security
> > professional and fellow gentoo user) about this situation, we realized
> > something else about portage in openldap. This would (or could,
> > depending on implementation) severely limit the rsync bottle neck by
> > allowing for a hierarchy of directory servers to be replicated from by
> > users.
>
> I REALLY like this idea..

I /tried/ using LDAP as my authentication for pam a while back, in hopes of 
having a nice, network-wide, roaming profile.  However, such hopes were 
quickly shattered once I got authentication going even locally.  My login 
attempts would take /quite/ a while and it was very inefficient.  I would 
rather see a MySQL database.

> >  With clever use of referals and replication, you could
> > effectively remove the rsync issues of bandwidth and the
> > "stop-syncing-so-often-it's-rude" problem. Updates to portage would be
> > propagated down the line when commited moving a smaller (but steady)
> > stream of traffic rather than unpredictable bursts (note: that's an
> > assumption). This would eliminate the need for 'emerge sync' (in theory).
> >
> > Again, this is all very academic as the data to back up these ideas is
> > out of the public eye (thankfully). Maybe just food for thought...
> >
> > Thanks to all devs for all the great work.
>
> Yup, can't say that enough too ;)...

Granted I am incorrect about my assertions above, I would like to see this 
*work* in a real-world situation before I say, "Yea, sure, let's give LDAP a 
try..."

-- 
Zack Gilburd
 http://tehunlose.com
  GnuPG Key ID: A79A45668240AB6C

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[c.wegener]

On Fri, Jun 27, 2003 at 07:58:39AM -0700, Zack Gilburd wrote:
Content-Description: signed data
> On Friday 27 June 2003 05:58 am, Rigo Ketelings wrote:
> > Op vr 27-06-2003, om 14:23 schreef Eric Sammer:
> > > Just to add something to my last email about portage and database /
> > > directory services...
> > >
> > > While talking to my wife this morning (a sysadmin and security
> > > professional and fellow gentoo user) about this situation, we realized
> > > something else about portage in openldap. This would (or could,
> > > depending on implementation) severely limit the rsync bottle neck by
> > > allowing for a hierarchy of directory servers to be replicated from by
> > > users.
> >
> > I REALLY like this idea..
> 
> I /tried/ using LDAP as my authentication for pam a while back, in hopes of 
> having a nice, network-wide, roaming profile.  However, such hopes were 
> quickly shattered once I got authentication going even locally.  My login 
> attempts would take /quite/ a while and it was very inefficient.  I would 
> rather see a MySQL database.
> 

As directory services are optimized for read performance, there are
disadvantages for write access to a directory service. But compared to
the amount of user needing read access to a directory service the
number of write access updates should REALLY small.

I think your bad experience with LDAP as authentication service may
have to do with misconfiguration or lack of optimization.

> > >  With clever use of referals and replication, you could
> > > effectively remove the rsync issues of bandwidth and the
> > > "stop-syncing-so-often-it's-rude" problem. Updates to portage would be
> > > propagated down the line when commited moving a smaller (but steady)
> > > stream of traffic rather than unpredictable bursts (note: that's an
> > > assumption). This would eliminate the need for 'emerge sync' (in theory).
> > >
> > > Again, this is all very academic as the data to back up these ideas is
> > > out of the public eye (thankfully). Maybe just food for thought...
> > >
> > > Thanks to all devs for all the great work.
> >
> > Yup, can't say that enough too ;)...
> 
> Granted I am incorrect about my assertions above, I would like to see this 
> *work* in a real-world situation before I say, "Yea, sure, let's give LDAP a 
> try..."

christoph
--
^*^

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Patrick Kursawe]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On Fri, Jun 27, 2003 at 05:21:02PM +0200, c.wegener@itcampus.de wrote:

[portage in LDAP]

> As directory services are optimized for read performance, there are
> disadvantages for write access to a directory service. But compared to
> the amount of user needing read access to a directory service the
> number of write access updates should REALLY small.
> 
> I think your bad experience with LDAP as authentication service may
> have to do with misconfiguration or lack of optimization.

I wonder how LDAP should have a better performance than rsync?
Are there implementations that support compression while transmitting
only the delta between entries?

I have next to no experience with directory services, so could you
please outline a bit further why you think this is better than rsync
or a database?

Yours sceptically,

Patrick	

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Eric Sammer]

Zack Gilburd wrote:

> I /tried/ using LDAP as my authentication for pam a while back, in hopes of 
> having a nice, network-wide, roaming profile.  However, such hopes were 
> quickly shattered once I got authentication going even locally.  My login 
> attempts would take /quite/ a while and it was very inefficient.

Certainly, setup (in its current incarnation) is flawed, at a minimum. I 
think it could work if it were "distro sanctioned" and therefore had 
full support. A good example of directory services working from the 
get-go is Mac OS X. I know a lot of Linux users disapprove of MOSX. I 
don't bring it up to start that debate but to illustrate a working 
example. MOSX ships with NetInfo (enabled) and all auth, groups, hosts, 
and other related items are using the directory by default. This makes 
managing a network of MOSX boxes much like a visit to the candy shop - 
while sticky, still very sweet.

> I would 
> rather see a MySQL database.

The problem with this approach is that mysql is a bit (ok, a lot) 
heavier than openldap (or ldap in general). The other reason is because 
there are major differences between databases to such a degree that it 
matters much more than, say, replacing openldap with another 
implementation. Without getting into the debate of mysql vs. postgres, 
there are major differences between the two and if something system 
level (i.e. portage) were to use mysql, it would be pretty difficult 
(due to the differences in things like datatypes and the like) to move 
it over to postgresql or sapdb. All of that said, ldap was (or has 
evolved to be) designed for holding this type of information and is a 
bit more standardized in terms of types and schema. Also, as I said in 
my first email, it also provides for the replication, referrals, 
addressbook services, and other fantastic features. Say it quietly to 
yourself - "no more 'emerge sync'... simple propagation... platform 
agnostic standards..." - it sounds pretty good. ;)

> Granted I am incorrect about my assertions above, I would like to see this 
> *work* in a real-world situation before I say, "Yea, sure, let's give LDAP a 
> try..."

Absolutely. This is a architectural change and would require simple 
tools to convert between flat files and the directory service 
equivelents and other "goodies" for less caring / experienced / 
concerned users. Like everything else in Gentoo already is, it would 
have to be simple, powerful, extensible, and sexy as hell... (like the 
init scripts, portage, gentoolkit, mirrorselect, ufed, net-setup, 
env-update, modules-update, and all those other tools that make me all 
full of smiles)

Sorry... the whole idea of working, integrated, supported directory 
services makes me froth at the mouth...

;)

-- 
Eric Sammer
eric@ineoconcepts.com
http://www.ineoconcepts.com


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Patrick Kursawe]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

On Fri, Jun 27, 2003 at 11:28:12AM -0400, Eric Sammer wrote:
> Say it quietly to 
> yourself - "no more 'emerge sync'... simple propagation... platform 
> agnostic standards..." - it sounds pretty good. ;)

Even if said quietly together with "dialup network"?
Or with "I only sync when there's a GLSA"?
> 
Just wondering

Patrick

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Eric Sammer]

Patrick Kursawe wrote:
> I wonder how LDAP should have a better performance than rsync?
> Are there implementations that support compression while transmitting
> only the delta between entries?

Directory services work very differently than rsync. I do believe that 
compression is supported (specifically in openldap) between servers 
during replication. The way ldap works (quick and dirty) is that when a 
change is made to the master, the change is written to a log file. 
Another daemon (again, specific implementation of openldap) is 
constantly reading the transaction log and sends the updates down the 
line to slaves. In theory, it is similar to the way DNS works 
(propagation and referals). If you decide that replication is 
undesirable, referrals allow you to tell your *local* ldap server to 
resend the request to a designated parent (which then can resend and 
resend and so on) - this is much like DNS. What's cool is that a stream 
of single transactions would be consistently flowing through the 
(hypotetical) network of Gentoo master ldap servers which could be 
accessed by region by users using portage. The performance would 
probably be better because (AFAIK) less overall traffic should be 
required and ldap has provisions for defering load (replication and 
referrals).

> I have next to no experience with directory services, so could you
> please outline a bit further why you think this is better than rsync
> or a database?

Further reading might describe it better than I can. Check out 
http://www.openldap.org for the details.

> Yours sceptically,

Hehhehe... if everyone just jumped at it, things would get messy fast. ;)

-- 
Eric Sammer
eric@ineoconcepts.com
http://www.ineoconcepts.com


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Eric Sammer]

Patrick Kursawe wrote:
> On Fri, Jun 27, 2003 at 11:28:12AM -0400, Eric Sammer wrote:
> 
>>Say it quietly to 
>>yourself - "no more 'emerge sync'... simple propagation... platform 
>>agnostic standards..." - it sounds pretty good. ;)
> 
> 
> Even if said quietly together with "dialup network"?
> Or with "I only sync when there's a GLSA"?

Sure. You could choose to replicate all of portage and keep a local copy 
within your private network. It would work sort of like using fetchmail 
to grab your email from an external mail server and transfering it to a 
mail server inside your network and then disconnecting from the 'net.

It could potentially work a number of ways...

> Just wondering

No problem... it's a good question.

-- 
Eric Sammer
eric@ineoconcepts.com
http://www.ineoconcepts.com


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Directory services (was Re: [gentoo-dev] maybe it is time to put portage queries into a database.)

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 591 bytes --]

On Friday 27 June 2003 16:58, Zack Gilburd wrote:
>
> I /tried/ using LDAP as my authentication for pam a while back, in hopes of
> having a nice, network-wide, roaming profile.  However, such hopes were
> quickly shattered once I got authentication going even locally.  My login
> attempts would take /quite/ a while and it was very inefficient.  I would
> rather see a MySQL database.

For that you need a good setup AND, you need to run nscd. That makes things a 
LOT faster.

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]