[gentoo-dev] RFC - Gentoo on the Lab

[gentoo-dev] RFC - Gentoo on the Lab

[Ricardo Loureiro]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

Hi all,

As part of the final project for my graduation in Informatics
Engineering (kinda Computer Science but that's the official name),
I'm gonna develop a "distributed" portage so we can have a test
lab at our uni with Gentoo (Starting September 1st). The idea is to
have a central machine with the portage tree inside an SQL server and
hacked portage so it makes remote sql query's. Also, the machines
should not compile any software but either request binary packages
that would be custom build according to their specs somewhere else.
This would allow a central machine to request the world file of every
client, calculate the needed upgrades, compile them according to
their specs and make them available on the lan as a binary package.
And this would make at least my job a lot easier. The project is
bigger, but I just described the necessary as background to my
questions, although I'd love to discuss the whole thing with other
ppl.

Some questions:

1- Can I RSYNC_EXCLUDE everything except profiles and have an usable
system?

2- There was a portagesql effort, is it dead?

3- If I tweak a custom profile not to have dev packages such as gcc,
what kind of problems should I expect? Which packages from the base
profile MUST exist? The idea is that the client machines should not
be capable of compiling software.

4- Any ideas on how the conf files should be handled?

The final result of the project will be released to the community
(GPL or BSD, still need to think), so I'd love to ear from users
dealing with this kind of scenario, question, comments, whatever you
think I should focus on. Oh, a name for the project would be welcome
=)

Thanks in advance,
Ricardo Loureiro
--
http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Andrew Gaffney]

Ricardo Loureiro wrote:
> Some questions:
> 
> 1- Can I RSYNC_EXCLUDE everything except profiles and have an usable
> system?

Depending on exactly how you're doing this, you'll probably want to keep 
eclasses/ as well.

> 2- There was a portagesql effort, is it dead?

Yes. It has been for a while, afaik.

> 3- If I tweak a custom profile not to have dev packages such as gcc,
> what kind of problems should I expect? Which packages from the base
> profile MUST exist? The idea is that the client machines should not
> be capable of compiling software.

You'll probably run into an ebuild somewhere that directly depends on gcc and/or 
related dev packages (very likely one of the leftover 'system' packages).

> 4- Any ideas on how the conf files should be handled?

Nope :)

-- 
Andrew Gaffney                            http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer                                   Installer Project
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Donnie Berkholz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ricardo Loureiro wrote:
| As part of the final project for my graduation in Informatics
| Engineering (kinda Computer Science but that's the official name),
| I'm gonna develop a "distributed" portage so we can have a test
| lab at our uni with Gentoo (Starting September 1st).

You might want to get on the gentoo-portage-dev list to discuss this in
more detail to the people most relevant and interested in it.

| The final result of the project will be released to the community
| (GPL or BSD, still need to think), so I'd love to ear from users
| dealing with this kind of scenario, question, comments, whatever you
| think I should focus on. Oh, a name for the project would be welcome
| =)

I strongly encourage you to release it under the same license as Portage
is (GPL-2), so parts (or all) of it could be incorporated into Gentoo.

Thanks,
Donnie
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDCUfiXVaO67S1rtsRAmZaAJ0ct39eYsPB3mXEkCRElhBBFN7bCgCgtU2s
K+LVTrh5LIaaQEUQQRmPZeQ=
=iqZX
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Ivan Yosifov]

On Sun, 2005-08-21 at 20:34 -0700, Donnie Berkholz wrote:
> | The final result of the project will be released to the community
> | (GPL or BSD, still need to think), so I'd love to ear from users
> | dealing with this kind of scenario, question, comments, whatever you
> | think I should focus on. Oh, a name for the project would be welcome
> | =)
> 
> I strongly encourage you to release it under the same license as Portage
> is (GPL-2), so parts (or all) of it could be incorporated into Gentoo.
> 
If portage is GPL-2 it he is building a derivative work does he have a
choice really ?


-- 
Cheers,
Ivan Yosifov.

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: RFC - Gentoo on the Lab

[Duncan]

Donnie Berkholz posted <430947E3.1000201@gentoo.org>, excerpted below,  on
Sun, 21 Aug 2005 20:34:59 -0700:

> Ricardo Loureiro wrote:
> 
> | The final result of the project will be released to the community
> | (GPL or BSD, still need to think), so I'd love to ear from users
> | dealing with this kind of scenario, question, comments, whatever you
> | think I should focus on. Oh, a name for the project would be welcome
> | =)
> 
> I strongly encourage you to release it under the same license as Portage
> is (GPL-2), so parts (or all) of it could be incorporated into Gentoo.

If I'm not mistaken (and I admit I could be, IANAL), BSD licensed
material, because the BSD license is a more permissive license with
generally few conditions, can be co-mingled in a GPL (2) project without
issue, and with the whole thing continuing to be GPL (2) licensed.  I'm
not sure whether BSD markers must remain attached or not.  In fact, I
believe it's fairly common for the kernel, for instance, to borrow BSD
code, and that said borrowed code doesn't have to be treated specially at
all (save for the traditional ACK of original source, but that's only
responsible origin tracing, certainly within the context of the SCOs of
the world).

Of course, the other possibility, if the permissiveness of BSD is desired,
would be a dual-license BSD/GPL.  That clears up any possible conflicts
directly and immediately.  OTOH, with portage itself already GPL2
licensed, I'm not sure I see much point in BSD licensing any portage
dependent new code, in any case, since it's dependent on GPL2 code anyway.
Still, the dual license certainly can't harm, and would likely be my
choice if I wanted the BSD permissiveness to apply to my code, under the
circumstances.  (FWIW, I prefer GPL, so there's no question that's how I'd
license it if it were me, but it's not, so that doesn't count.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: RFC - Gentoo on the Lab

[Ricardo Loureiro]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Mon, 22 Aug 2005 04:31:09 -0700
Duncan <1i5t5.duncan@cox.net> wrote:

> Of course, the other possibility, if the permissiveness of BSD is
> desired,
> would be a dual-license BSD/GPL.  That clears up any possible
> conflicts
> directly and immediately.  OTOH, with portage itself already GPL2
> licensed, I'm not sure I see much point in BSD licensing any portage
> dependent new code, in any case, since it's dependent on GPL2 code
> anyway.
> Still, the dual license certainly can't harm, and would likely be my
> choice if I wanted the BSD permissiveness to apply to my code,
> under the
> circumstances.  (FWIW, I prefer GPL, so there's no question that's
> how I'd
> license it if it were me, but it's not, so that doesn't count.)
> 

BSD licence allows any fork to be created with any other licence, so
a BSD project can have a GPL fork. I didn't thought about changing
Portage itself simply because I don't know if the changes are welcome
to Gentoo itself, probably more a Gentoo server project. But if that
is a possibility I have no problems with GPL.


Ricardo Loureiro
--
http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Marius Mauch]

On 08/22/05  Ricardo Loureiro wrote:

> 1- Can I RSYNC_EXCLUDE everything except profiles and have an usable
> system?

Define "usable". As only portage uses the tree it would be the only
thing that might break.

> 2- There was a portagesql effort, is it dead?

As far as I know, yes. But it wasn't what you wanted anyway (only
implemented a SQL cache for faster searching, interesting that almost
every "rewrite" attempt implements searching first)

> 3- If I tweak a custom profile not to have dev packages such as gcc,
> what kind of problems should I expect? Which packages from the base
> profile MUST exist? The idea is that the client machines should not
> be capable of compiling software.

I'd guess baselayout + it's deps + libc are the absolute minimum
(excluding baselayout-lite and other embedded solutions).

> 4- Any ideas on how the conf files should be handled?

Depends on your client nodes, if they are (almost) identical I'd just
sync them from a master node. If not it gets complicated.

Anyway, I hope you realize that your project doesn't only involve
hacking on portage, but rewriting almost all of it for the client part.
Actually I'd rather suggest you start from scratch (so you also make it
work completely without a tree), or wait for Brians rewrite in HEAD (not
a good idea though if you have a deadline). Server should be less of an
issue, mostly config tweaks there.
But as Donnie said, gentoo-portage-dev is the better list for this
discussion.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Ricardo Loureiro]

[-- Attachment #1: Type: text/plain, Size: 3250 bytes --]

On Mon, 22 Aug 2005 16:38:11 +0200
Marius Mauch <genone@gentoo.org> wrote:

> Define "usable". As only portage uses the tree it would be the only
> thing that might break.

Usable in the way that the client machines should be able to use
portage, except it's the hacked (or new package) version that should
do everything from the SQL server. For example, a emerge package
would behave in 2 possible ways;1- calculate it's dependencies from
the portage tree on the SQL server and request the binary packages,
2- Request the package and the server would calculate dependencies
and get the binary done. I'm more keen on the second since it takes
away processor time from the clients, but that involves sending
sensitive information such as world files and make.conf over the
network.

> As far as I know, yes. But it wasn't what you wanted anyway (only
> implemented a SQL cache for faster searching, interesting that
> almost
> every "rewrite" attempt implements searching first)

I asked because it would be nice to talk to their devs, so I could
know in advance what were their problems and what they would have
done different. Anyway the project will be different, as you said,
but another goal was to produce a single machine mode that would use
a relational database engine as portage tree.

> 
> I'd guess baselayout + it's deps + libc are the absolute minimum
> (excluding baselayout-lite and other embedded solutions).

Thanks, that was exactly what I was looking for.

> 
> > 4- Any ideas on how the conf files should be handled?
> 
> Depends on your client nodes, if they are (almost) identical I'd
> just
> sync them from a master node. If not it gets complicated.

That's the problem, very different machines (and maybe some time
later even arch's). The best way was to produce yet another
etc-update, but 5 months for a single person is too little time for
that. In general most of the times if the config files are not
changed it's safe to overwrite, else don't, but sometimes pakcage
versions have config files re-written, and that's a problem. Just
wanted to know what you ppl do in these situations and maybe found
something I was not aware of.

> 
> Anyway, I hope you realize that your project doesn't only involve
> hacking on portage, but rewriting almost all of it for the client
> part.
> Actually I'd rather suggest you start from scratch (so you also
> make it
> work completely without a tree), or wait for Brians rewrite in HEAD
> (not
> a good idea though if you have a deadline). Server should be less
> of an
> issue, mostly config tweaks there.

My initial thought was a from scratch portage in python that could
use many of the code already done, that would be better since portage
itself doesn't need a client-server mode and I could learn a lot more
this way. Waiting is not an option, no pressure on other ppl and
limited time for the project, but I hope to have the time to change
it after the deadline as a personal hobbie.

> But as Donnie said, gentoo-portage-dev is the better list for this
> discussion.

Did already, thanks for your help already.


Ricardo Loureiro
--
http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Grobian]

Ricardo Loureiro wrote:
> Usable in the way that the client machines should be able to use
> portage, except it's the hacked (or new package) version that should
> do everything from the SQL server. For example, a emerge package
> would behave in 2 possible ways;1- calculate it's dependencies from
> the portage tree on the SQL server and request the binary packages,
> 2- Request the package and the server would calculate dependencies
> and get the binary done. I'm more keen on the second since it takes
> away processor time from the clients, but that involves sending
> sensitive information such as world files and make.conf over the
> network.

Sounds like in your setup you would like to keep a profile of the client 
on the server, so you don't have to send over that information, because 
it is already in the DBMS.  That also allows you to update 
'push-driven', because the server can easily tell which clients have 
packages that are out of date.


-- 
Fabian Groffen
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Kristian Benoit]

On Mon, 2005-08-22 at 16:38 +0200, Marius Mauch wrote:

> Anyway, I hope you realize that your project doesn't only involve
> hacking on portage, but rewriting almost all of it for the client part.
> Actually I'd rather suggest you start from scratch

I do agree with that, portage probably need a rewrite/better
modularization anyway. There is/was a project called portage-ng () you
might want to have a look at. I did a little in that direction recently,
and it seems that there is not too many people working on it since
drobbins left, but you can contact Pieter (pvdabeel@gentoo.org). I might
get on that too at some point in the future too.

There is also Patrick Mclean (chutzpah@linuxfreak.ca) that has a nice
lab setup using Gentoo at McGill. It's a diskless system so might not be
to much what you are looking for, but as you probably want to dig before
getting to work, you should really see his setup. He did not released
anything yet, but we talked about starting a wiki for gentoo labs
sometime and have his setup released there.

Kristian

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1652 bytes --]

On Mon, Aug 22, 2005 at 01:49:14PM -0400, Kristian Benoit wrote:
> On Mon, 2005-08-22 at 16:38 +0200, Marius Mauch wrote:
> 
> > Anyway, I hope you realize that your project doesn't only involve
> > hacking on portage, but rewriting almost all of it for the client part.
> > Actually I'd rather suggest you start from scratch
> 
> I do agree with that, portage probably need a rewrite/better
> modularization anyway. There is/was a project called portage-ng () you
> might want to have a look at. I did a little in that direction recently,
> and it seems that there is not too many people working on it since
> drobbins left, but you can contact Pieter (pvdabeel@gentoo.org). I might
> get on that too at some point in the future too.

Portage-ng never resulted in anything tangible (read: code), further 
the doc wasn't really useful for anything then jotting down what's 
desired.  Unless something's changed, that doc should've been yanked 
down.  She's dead, jim.

Regarding modularization of portage, it requires that, but 
fundamentally it requires a rewrite of the core; there is no internal 
package abstraction, repository abstraction, hell, even a clean config 
abstraction (let alone cache abstraction).

The 2.1 code that was pushed out for inspection addresses the cache 
issue mostly, and modularization as much as possible.  Everything else 
falls to the rewrite which is underway- I'd suggest contacting portage 
devs, since what you're after is pretty much what's been designed to 
allow for, without requiring hacks to portage- just would be plugins.

That and help would always be welcome :P
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Stephen Bennett]

On Mon, 22 Aug 2005 13:49:14 -0400
Kristian Benoit <kbenoit@opersys.com> wrote:

> I do agree with that, portage probably need a rewrite/better
> modularization anyway. There is/was a project called portage-ng () you
> might want to have a look at. I did a little in that direction
> recently, and it seems that there is not too many people working on
> it since drobbins left, but you can contact Pieter
> (pvdabeel@gentoo.org). I might get on that too at some point in the
> future too.

portage-ng is dead. There is a rewrite going on, but it'll take a while
to get anywhere near usable.
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Re: RFC - Gentoo on the Lab

[Duncan]

Ricardo Loureiro posted <20050822142658.0f5f7aab@acme.rjlouro.org>,
excerpted below,  on Mon, 22 Aug 2005 14:26:58 +0100:

> BSD licence allows any fork to be created with any other licence, so
> a BSD project can have a GPL fork. I didn't thought about changing
> Portage itself simply because I don't know if the changes are welcome
> to Gentoo itself, probably more a Gentoo server project. But if that
> is a possibility I have no problems with GPL.

The idea is welcome, and a Gentoo Server project (with its implications
for portage) has been discussed and even started (AFAIK) a number of
times.  Whether or not it would actually be added to portage is another
matter, no promises (nor as a user could I give them), of course, but the
idea is not to complicate any such possibilities with an incompatible
license.

Certainly, such projects have been the beginnings  of serious advances in
any number of open source projects...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Kristian Benoit]

On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote:
> That and help would always be welcome :P

Then where do I find the code (I'm an official dev yet, so I only have
access to what's in the mirrors and the patchs on mailing lists)

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Kristian Benoit]

On Mon, 2005-08-22 at 21:41 +0100, Stephen Bennett wrote:
> On Mon, 22 Aug 2005 13:49:14 -0400
> Kristian Benoit <kbenoit@opersys.com> wrote:
> 
> > I do agree with that, portage probably need a rewrite/better
> > modularization anyway. There is/was a project called portage-ng () you
> > might want to have a look at. I did a little in that direction
> > recently, and it seems that there is not too many people working on
> > it since drobbins left, but you can contact Pieter
> > (pvdabeel@gentoo.org). I might get on that too at some point in the
> > future too.
> 
> portage-ng is dead. There is a rewrite going on, but it'll take a while
> to get anywhere near usable.

Here is my recent communication with Pieter:

On Sat, 2005-08-13 at 21:59 +0200, Pieter Van den Abeele wrote: 
> On 13 Aug 2005, at 19:16, Kristian Benoit wrote:
> 
> > I've heard that you might be the last to know something about
> > portage-ng. What's the actual status,
> 
> Here's what I've done so far:
> 
> I've build upon Andreas' Zeller idea of using Smolka's feature logic  
> for software configuration management. Zeller's approach was ok for  
> determining consistency/inconsistency of software configurations. In  
> his phd thesis he described the algorithm involved and discussed time  
> complexity (which goes to NP if you allow for quantification). My  
> impression after implementing his idea was that for automated  
> construction there were a few things that were lacking, and some  
> things were incorrect. I revisited Zellers feature logic, and ended  
> up with a clean implementation of a declarative language which has  
> some very desirable properties for automated software construction.  
> I'd say my approach is closer to the spirit of Smolka's feature logic  
> than Zellers approach. My non-destructive feature unification has a  
> worst case time complexity of O(n x m) where n is the length of the  
> feature term describing your system, and m is the length of the  
> feature term describing the component to be added. In practice  
> performance is very good. An emerge --vp --emptytree kde with the  
> regular portage takes about 55 seconds on my Open Desktop Workstation  
> and produces a list of 200 packages. A similar action using my  
> implementation:
> 
> =====================================================================
> Total time: 14.55 seconds
> =====================================================================
> Predicate                       Box Entries =    Calls+Redos     Time
> =====================================================================
> vunify/4                            341,900 =  341,900+0        72.6%
> $garbage_collect/1                      326 =      326+0        13.6%
> lists:append/3                      684,320 =  684,320+0         4.0%
> genterm/2                               520 =      520+0         3.9%
> hunify/4                                520 =      520+0         3.3%
> is/2                                342,420 =  342,420+0         1.8%
>  >/2                                 342,160 =  342,160+0         0.8%
> buildsystem/2                             1 =        1+0         0.0%
> val/3                                 5,200 =    5,200+0         0.0%
> unify/3                                 260 =      260+0         0.0%
> 
> % 9,531,861 inferences, 13.96 CPU in 14.55 seconds (96% CPU, 682798  
> Lips)
> 
> The search space is kept as small as possible because I've introduced  
> lazy feature evaluation and both multi valued and open features.  
> Those concepts are missing in Zellers approach. Comparing current  
> portage and my implementation performance-wise is difficult. In  
> general mine is faster, but current portage doesn't use sql either.  
> What is for sure is that mine allows you to express various  
> constraints. You can prevent a dependency from being built with a  
> specific use flag. My implementation automatically solves 'blockers'.  
> Circular dependencies are also solved correctly. For instance, if you  
> want to install unixodbc with the qt use flag enabled and you want to  
> install qt wit the unixodbc use flag enabled, my portage knows that  
> it needs to:
> 
> emerge unixodbc without qt
> emerge qt with unixodbc
> remerge unixodbc with qt support
> 
> So it makes revdep-rebuild unnecessary basically.
> 
> 
> Once I was done implementing this new declarative language in which  
> for instance ebuild concepts can be easily expressed (but also rpm,  
> deb, fink, darwinports). I implemented a knowledge base in which  
> those feature terms can be stored/looked up efficiently. I used an  
> odbc bridge design pattern and have tested the thing with postgresql/ 
> mysql and sqlite. I've also implemented a DCG parser which parses  
> ebuilds (no more having to start a bash process for each ebuild like  
> current portage does) and converts it to feature logic representation  
> very efficiently. One of the things I'm doing now is finishing up the  
> documentation and experimenting with an RDF based system for  
> 'ebuildcasting'. The idea is that users who write ebuilds could  
> "ebuildcast" it like some people podcast their own radio shows. The  
> knowledge base can be subscribed to several sources (basically the  
> idea of portage overlays but combined with some ideas from the  
> semantic web).
> 
> 
> > can we see the code,
> 
> yes, the first release (and all releases after that) will be GPL.
> 
> 
> > do you think there is still a hope ?
> 
> As for me finishing this and doing something with it, yes.
> 
> 
> > Anyway, I'd like to learn what there is to learn about it and perhaps
> > help to get it restarted.
> 
> For your convenience I've put some interesting papers online on  
> http://www.metadistribution.org/seminar-tinf/
> 
> I'd love to discuss the idea of feature logic with you while I'm  
> finishing things up.
> 
> Pieter
> 
> 
> > thank for any info.
> >
> > Krisitian

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Ricardo Loureiro]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

On Mon, 22 Aug 2005 21:41:35 +0100
Stephen Bennett <spb@gentoo.org> wrote:

> portage-ng is dead. There is a rewrite going on, but it'll take a
> while
> to get anywhere near usable.

I searched a bit to find information about portage-ng but the only
doc I found was Daniel's pdf with just a drawing of the arch itself,
nothing very useful to me. Also the latest news go back almost 2
years. Is there a place with more information on what portage-ng
should be? 



Ricardo Loureiro
--
http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Ricardo Loureiro]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Mon, 22 Aug 2005 15:39:10 -0500
Brian Harring <ferringb@gentoo.org> wrote:

> The 2.1 code that was pushed out for inspection addresses the cache 
> issue mostly, and modularization as much as possible.  Everything
> else 
> falls to the rewrite which is underway- I'd suggest contacting
> portage 
> devs, since what you're after is pretty much what's been designed
> to 
> allow for, without requiring hacks to portage- just would be
> plugins.

Writing a plugin for portage would be perfect, but I can't wait for
2.1 to be released due to deadlines. Looks like the best shot at this
point is a new package using as much as possible the actual python
libs created for portage.

When I start the project itself (September 1st) I'm gonna start to
write and organize all the ideas, but you all have been a great help
so far.

Ricardo Loureiro
--
http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 6199 bytes --]

Lot of text left inline, pardon, just scroll and deal with it :P

On Tue, Aug 23, 2005 at 12:28:08PM -0400, Kristian Benoit wrote:
> Here is my recent communication with Pieter:
> 
> On Sat, 2005-08-13 at 21:59 +0200, Pieter Van den Abeele wrote: 
> > On 13 Aug 2005, at 19:16, Kristian Benoit wrote:
> > 
> > > I've heard that you might be the last to know something about
> > > portage-ng. What's the actual status,
> > 
> > Here's what I've done so far:
> > 
> > I've build upon Andreas' Zeller idea of using Smolka's feature logic  
> > for software configuration management. Zeller's approach was ok for  
> > determining consistency/inconsistency of software configurations. In  
> > his phd thesis he described the algorithm involved and discussed time  
> > complexity (which goes to NP if you allow for quantification). My  
> > impression after implementing his idea was that for automated  
> > construction there were a few things that were lacking, and some  
> > things were incorrect. I revisited Zellers feature logic, and ended  
> > up with a clean implementation of a declarative language which has  
> > some very desirable properties for automated software construction.  
> > I'd say my approach is closer to the spirit of Smolka's feature logic  
> > than Zellers approach. My non-destructive feature unification has a  
> > worst case time complexity of O(n x m) where n is the length of the  
> > feature term describing your system, and m is the length of the  
> > feature term describing the component to be added. In practice  
> > performance is very good. An emerge --vp --emptytree kde with the  
> > regular portage takes about 55 seconds on my Open Desktop Workstation  
> > and produces a list of 200 packages. A similar action using my  
> > implementation:
> > 
> > =====================================================================
> > Total time: 14.55 seconds
> > =====================================================================
> > Predicate                       Box Entries =    Calls+Redos     Time
> > =====================================================================
> > vunify/4                            341,900 =  341,900+0        72.6%
> > $garbage_collect/1                      326 =      326+0        13.6%
> > lists:append/3                      684,320 =  684,320+0         4.0%
> > genterm/2                               520 =      520+0         3.9%
> > hunify/4                                520 =      520+0         3.3%
> > is/2                                342,420 =  342,420+0         1.8%
> >  >/2                                 342,160 =  342,160+0         0.8%
> > buildsystem/2                             1 =        1+0         0.0%
> > val/3                                 5,200 =    5,200+0         0.0%
> > unify/3                                 260 =      260+0         0.0%
> > 
> > % 9,531,861 inferences, 13.96 CPU in 14.55 seconds (96% CPU, 682798  
> > Lips)
Stating it as nicely as possible, without knowing what that's doing, 
stats can be construed several ways; faster backend access (both vdb 
and ebuild/cache), dodging/caching virtuals, etc; language 
implementation is a point of curiousity also.

Faster implementation doesn't surprise me- stable portage is fricking 
*absolutely* retarded about caching, parsing of deps and cpvs, loading 
of profile, etc.  Either way, interest remains in seeing it :)

> > The search space is kept as small as possible because I've introduced  
> > lazy feature evaluation and both multi valued and open features.  
> > Those concepts are missing in Zellers approach. Comparing current  
> > portage and my implementation performance-wise is difficult. In  
> > general mine is faster, but current portage doesn't use sql either.  
> > What is for sure is that mine allows you to express various  
> > constraints. You can prevent a dependency from being built with a  
> > specific use flag. My implementation automatically solves 'blockers'.  
> > Circular dependencies are also solved correctly. For instance, if you  
> > want to install unixodbc with the qt use flag enabled and you want to  
> > install qt wit the unixodbc use flag enabled, my portage knows that  
> > it needs to:
> > 
> > emerge unixodbc without qt
> > emerge qt with unixodbc
> > remerge unixodbc with qt support
> > 
> > So it makes revdep-rebuild unnecessary basically.
revdep-rebuild is still necessary, regardless of use deps or full 
state graphs- anyone doubting that should take a look at the breakage 
that has occured whenever mysql/ssl have decided to change their 
soname while maintaining compatibility.
Need a bincompat metadata attribute to kill off revdep-rebuild.


> > Once I was done implementing this new declarative language in which  
> > for instance ebuild concepts can be easily expressed (but also rpm,  
> > deb, fink, darwinports).

This sounds a lot like my restriction subsystem in the rewrite (code's 
available to anyone after it).

With the restriction setup, anyone who wants it could just plugin a 
different repo/pkg plugin, and have deps based on sonames fex; granted 
anyone doing it would need some form of a binary repository, but the 
agnostic aspect of representing matching criteria is the key to it.

> > I implemented a knowledge base in which  
> > those feature terms can be stored/looked up efficiently. I used an  
> > odbc bridge design pattern and have tested the thing with postgresql/ 
> > mysql and sqlite. I've also implemented a DCG parser which parses  
> > ebuilds (no more having to start a bash process for each ebuild like  
> > current portage does)

Strictly a stable slow down, and relevant when pulling metadata; 2.1 
(ebd patchset) is a daeonized ebuild processor effectively.  Shaves a 
good 30% off of runtime, while being proper (sans bugs).

Alternative implementations welcome of course, but ebd is pretty much 
something whipped out that addresses long standing bugs without 
requiring a full rewrite of parsing (interested parties can do it 
another day, I care not to after filter-env).
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On Tue, Aug 23, 2005 at 12:25:03PM -0400, Kristian Benoit wrote:
> On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote:
> > That and help would always be welcome :P
> 
> Then where do I find the code (I'm an official dev yet, so I only have
> access to what's in the mirrors and the patchs on mailing lists)
Timing mildy sucks; just switched over to svn, making our anoncvs 
access pointless for the rewrite.

So... after anon svn to replace loss of anoncvs provided by carpaski.
Meantime, anyone who wants snapshots can scream at me and I'll post 
them- anon svn if it is agreed to be implemented infra wise, once it's 
up the info will be posted in gentoo-portage-dev ml, and tagged into 
/topic in irc (the norms).
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

On Tue, Aug 23, 2005 at 12:25:03PM -0400, Kristian Benoit wrote:
> On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote:
> > That and help would always be welcome :P
> 
> Then where do I find the code (I'm an official dev yet, so I only have
> access to what's in the mirrors and the patchs on mailing lists)
If you're after a snapshot, give a yell; right now still waiting for 
svn crap to straighten out (mainly for the rewrite to be moved fully 
to svn), but once done I can point you at wherever I dump it in my 
devspace till anonsvn is up.

No clue on eta of anonsvn, that's infra's thing unfortunately.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Marius Mauch]

On 08/23/05  Ricardo Loureiro wrote:

> On Mon, 22 Aug 2005 21:41:35 +0100
> Stephen Bennett <spb@gentoo.org> wrote:
> 
> > portage-ng is dead. There is a rewrite going on, but it'll take a
> > while
> > to get anywhere near usable.
> 
> I searched a bit to find information about portage-ng but the only
> doc I found was Daniel's pdf with just a drawing of the arch itself,
> nothing very useful to me. Also the latest news go back almost 2
> years. Is there a place with more information on what portage-ng
> should be? 

Nope. It was basically one of Daniels naive visions mixed with Pieters
science blabla (nothing against science, but I wouldn't call a package
repository a knowledge base). If there is any actual information in
portage-ng, you have to ask Pieter, but officially it's dead (was never
alive really), but don't hope to get anything more than "it's using
advanced science, it works and I'll release the code someday".

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC - Gentoo on the Lab

[Kristian Benoit]

On Tue, 2005-08-23 at 12:27 -0500, Brian Harring wrote:
> On Tue, Aug 23, 2005 at 12:25:03PM -0400, Kristian Benoit wrote:
> > On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote:
> > > That and help would always be welcome :P
> > 
> > Then where do I find the code (I'm an official dev yet, so I only have
> > access to what's in the mirrors and the patchs on mailing lists)
> If you're after a snapshot, give a yell; right now still waiting for 
> svn crap to straighten out (mainly for the rewrite to be moved fully 
> to svn), but once done I can point you at wherever I dump it in my 
> devspace till anonsvn is up.

Yeah, I'd really like having a snapshot, even if I'd prefer having
cvs/svn access. You can send me it by mail or make it available
somewhere.

> No clue on eta of anonsvn, that's infra's thing unfortunately.
> ~harring

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] portage rewrite snapshot (was RFC - Gentoo on the Lab)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On Tue, Aug 23, 2005 at 06:58:53PM -0400, Kristian Benoit wrote:
> Yeah, I'd really like having a snapshot, even if I'd prefer having
> cvs/svn access. You can send me it by mail or make it available
> somewhere.
Pardon the delay, wanted to iron out building code before 
pushing the snapshot up- it's available at
http://dev.gentoo.org/~ferringb/new-08-25-05.tar.bz2

anonsvn is in the air, but in the interim I'll be doing snapshot's as 
stuff comes in.

/me notes again, that it is not usable yet.
You can build packages with it, but the classes for merge operations 
aren't in place, meaning no merge.
Bugs probably exist up the ying yang also; if you want to hack on it, 
then by all means go for it, otherwise kindly wait (general disclaimer 
that one).
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]