[gentoo-dev] GRP set consolidation

[gentoo-dev] GRP set consolidation

[Joshua Brindle]

Ok, we just had a conversation in #gentoo-dev and it seems most people 
are interested in this. The problem as it stands is that our mirrors are 
  low on space and so other projects can't get space, such as embedded, 
hardened and even other archs. The main user of space is having 5 grp 
packagecd's for x86 alone. The idea is to build a consolidated grp set 
with a generic -mcpu (i686 or so) and then use that.

As I'm sure this will become a large and flaming thread, and probably 
won't read them all I'll just go ahead and suggest that we add this to 
the next meeting to discuss it if there are large amounts of disagreement.

Joshua Brindle

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GRP set consolidation

[Stuart Herbert]

[-- Attachment #1: Type: text/plain, Size: 574 bytes --]

On Friday 21 May 2004 23:11, Chris Gianelloni wrote:
> The other take on this is gentoopix and what it will provide.  

What is gentoopix?  Does it have a GLEP number? ;-)

Best regards,
Stu
-- 
Stuart Herbert                                              stuart@gentoo.org
Gentoo Developer                                       http://www.gentoo.org/
                                                   http://stu.gnqs.org/diary/

GnuPG key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319  C549 0C2F 80BA F9AF C57C
--

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GRP set consolidation

[Chris Gianelloni]

On Fri, 2004-05-21 at 17:34, Joshua Brindle wrote:
> Ok, we just had a conversation in #gentoo-dev and it seems most people 
> are interested in this. The problem as it stands is that our mirrors are 
>   low on space and so other projects can't get space, such as embedded, 
> hardened and even other archs. The main user of space is having 5 grp 
> packagecd's for x86 alone. The idea is to build a consolidated grp set 
> with a generic -mcpu (i686 or so) and then use that.
> 
> As I'm sure this will become a large and flaming thread, and probably 
> won't read them all I'll just go ahead and suggest that we add this to 
> the next meeting to discuss it if there are large amounts of disagreement.

Just to reiterate what I had said on -dev, I think it is a good idea. 
I'm not either "for" or "against" it, I just think having only one set
of packages to QA would make all our lives easier.  I *can* see the need
for 2 GRP sets, perhaps a x86 and a i686 set, simply to please the users
that find GRP useful.

The other take on this is gentoopix and what it will provide.  If it can
provide the same functionality as doing a GRP install, then I would say
we need to get moving on gentoopix and get rid of GRP (but not before).

Just my .02c

-- 
Chris Gianelloni
Developer
Games/LiveCD Teams
Gentoo Linux

Is your power animal a penguin?


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GRP set consolidation

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

On Fri, 21 May 2004 16:34:09 -0500 Joshua Brindle <method@gentoo.org>
wrote:
| Ok, we just had a conversation in #gentoo-dev and it seems most people
| are interested in this. The problem as it stands is that our mirrors
| are low on space and so other projects can't get space, such as
| embedded, hardened and even other archs. The main user of space is
| having 5 grp  packagecd's for x86 alone. The idea is to build a
| consolidated grp set with a generic -mcpu (i686 or so) and then use
| that.

We already do this for sparc. Rather than providing GRP and stages for
v7, v8, supersparc, hypersparc, cypress, v9, ultrasparc and ultrasparc3,
we just provide a set for sparc and a set for sparc64.

-- 
Ciaran McCreesh, Gentoo XMLcracy Member G03X276
(Sparc, MIPS, Vim, si hoc legere scis nimium eruditionis habes)
Mail:    ciaranm at gentoo.org
Web:     http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GRP set consolidation

[John Davis]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

On Fri, 2004-05-21 at 17:34, Joshua Brindle wrote:
> Ok, we just had a conversation in #gentoo-dev and it seems most people 
> are interested in this. The problem as it stands is that our mirrors are 
>   low on space and so other projects can't get space, such as embedded, 
> hardened and even other archs. The main user of space is having 5 grp 
> packagecd's for x86 alone. The idea is to build a consolidated grp set 
> with a generic -mcpu (i686 or so) and then use that.
> 
> As I'm sure this will become a large and flaming thread, and probably 
> won't read them all I'll just go ahead and suggest that we add this to 
> the next meeting to discuss it if there are large amounts of disagreement.
> 
> Joshua Brindle
> 
> --
> gentoo-dev@gentoo.org mailing list

Josh -
Seems like something good to research to me. GRP does take up a lot of
space on the mirrors, not mentioning the fact that it takes a ton of
time to build. 

But..

Before we do anything, I would like to gather some user input. I
remember klieber talking about a system that he was setting up that we
could use to survey users. Kurt, any progress? I don't want to make a
relatively major change to releases without at least some user input.

Cheers,
//zhen
-- 
John Davis
Gentoo Linux Developer
<http://dev.gentoo.org/~zhen>

----
GnuPG Public Key: <http://dev.gentoo.org/~zhen/zhen_pub.asc>
Fingerprint: 2364 71BD 4BC2 705D F338  FF70 6650 1235 1946 2D47


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GRP set consolidation

[Pieter Van den Abeele]

On 22 May 2004, at 00:09, Stuart Herbert wrote:

> On Friday 21 May 2004 23:11, Chris Gianelloni wrote:
>> The other take on this is gentoopix and what it will provide.
>
> What is gentoopix?  Does it have a GLEP number? ;-)

I didn't commit a GLEP for it, but I described and implemented the idea 
I think Chris is talking about for the livecd scripts I wrote and used 
for all ppc/g3/g4/g5 releases until Catalyst was put into production 
(My scripts available on http://www.metadistribution.org/gentoo/ and in 
gentoo cvs). The idea is the following:

- stage1 is a set of packages which are build on an existing system 
(doesn't have to be gentoo).
- stage2 is stage1 + extra packages
- stage3 is stage2 + extra packages
- grp is stage3 + extra packages
- livecd is grp + extra packages which make the system ( which consists 
of all the unpacked packages ) bootable as a livecd.

If my script is given a stage2 and asked to build a livecd, it will 
upgrade to stage3, then build grp and add a few packages to make the 
produced set of GRP bootable. For Historical reasons, Gentoo release 
tools (even current catalyst) are not following this design/idea: If I 
give catalyst a stage2 or a stage3 , and ask it to produce a set of GRP 
it will do so, but if I ask catalyst afterwards to build a livecd, it 
will just start from the stage2/stage3 again and compile kde/gnome (in 
case of the kde/gnome livecd), all over again. Such a thing leads to 
countless wasted hours for the engineers building our releases (incl. 
me). I have been working closely together with zhen, our current 
catalyst maintainer on the design of the tool, my remarks can be found 
in the REMARKS file in the catalyst cvsroot.

My original proposal was to just give the people a (kde/gnome, minimal, 
developer, game, whatever) livecd and then use a tool which 'derives' 
packages from the live environment on that cd, using the gentoo package 
manager knowledge. You just build one bootable, optimized livecd (with 
a compressed loop filesystem, you're able to put a +2G system on a 
regular 700M CD) for each of your subarchs (or just one generic), and 
when the user has booted into this system, he/she can create stages, 
GRP, or even a new livecd, using the packages derived from the livecd. 
For those packages which cannot be derived, portage can use it's 
knowledge (ebuilds) to build/compile/create the package. It will have 
to use its knowledge anyway if you're changing the CFLAGS, USE, 
whatever...

This system makes stages unnecessary for distribution: A stage3 
contains a stage2, so, from a mirror perspective, it makes sense to 
just have the packages, or the knowledge how to derive them, and have 
'virtual stages', or better 'metastages', declaring what packages a 
configuration consists of (our livecds run the stages they distribute 
anyway):

- a desktop stage, a server stage, an xfree stage, the original 
stage2...

In theory we could include milions of configurations/metastages on the 
cd (declared in a DB, text file, whatever) and just have portage 
assemble the actual tarball on the fly by deriving the packages from 
the current live environment or using its knowledge to create the 
missing ones. That saves us at least a few gigs on the mirrors. Once 
portage gets a faster DB backend, the limiting factor will become the 
expressiveness (conflicting packages, glep 23, glep 21, glep 19, glep 
22 (especially) , emerge -e world && auto-use ...) and the speed drop 
that comes with increasing that, (we go from a tree walking algorithm 
(nothing fancy) to actual reasoning/planning, which is a bit more 
challenging :-)

Best regards,

Pieter Van den Abeele


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GRP set consolidation

[John Davis]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

On Fri, 2004-05-21 at 19:41, Pieter Van den Abeele wrote:
> On 22 May 2004, at 00:09, Stuart Herbert wrote:
> 
> > On Friday 21 May 2004 23:11, Chris Gianelloni wrote:
> >> The other take on this is gentoopix and what it will provide.
> >
> > What is gentoopix?  Does it have a GLEP number? ;-)
> 


Pieter -
I am slowly but steadily working through that REMARKS file ;) I like the
idea of what you are proposing, in fact, I really like it. I think the
users would be keen to dropping in a graphical LiveCD and being able to
emerge packages right off of the livecd fs.

Rac just submitted some packages to me for a "stage4" - which is a
stage3 w/ more packages. Perhaps we could extend this into the gentoopix
idea and use stage4s to build our livecds? (btw, what does gentoopix
stand for?).

I think we could implement this fairly easily, let me know what you are
thinking. We might want to keep our stages and minimal livecds around to
save users from having to download big CDs if they want to install
Gentoo (for our low bandwidth folks).

Cheers,
-- 
John Davis
Gentoo Linux Developer
<http://dev.gentoo.org/~zhen>

----
GnuPG Public Key: <http://dev.gentoo.org/~zhen/zhen_pub.asc>
Fingerprint: 2364 71BD 4BC2 705D F338  FF70 6650 1235 1946 2D47


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GRP set consolidation

[Joseph Booker]

John Davis said:
> (btw, what does gentoopix
> stand for?).

john: look at knoppix.net

is this sorta like a knoppix disc that can be generated from catalyste's
conf files? if so, anyway of possibly sharing some work on the init
scripts and stuff like persistant home dirs and things like klik ( a tool
to install programs on knoppix while its booted)?

-- 
 Joe Booker

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GRP set consolidation

[Pieter Van den Abeele]

On 22 May 2004, at 02:01, John Davis wrote:

> Rac just submitted some packages to me for a "stage4" - which is a
> stage3 w/ more packages.
>
> Perhaps we could extend this into the gentoopix
> idea and use stage4s to build our livecds? (btw, what does gentoopix
> stand for?).
>
> I think we could implement this fairly easily, let me know what you are
> thinking. We might want to keep our stages and minimal livecds around 
> to
> save users from having to download big CDs if they want to install
> Gentoo (for our low bandwidth folks).

We already have small components 'ebuilds' representing knowledge to 
build packages. We also have packages, the result of applying the 
knowledge. Sometimes we make a snapshot of our knowledge and build a 
lot of packages from ebuilds in this snapshot, we call the collection 
of packages a 'reference platform'.

The GRP set of packages is a superset of the stage 3 set of packages,
The stage3 set of packages is a superset of the stage2 set of packages,
The stage2 set of packages is a superset of the stage1 set of packages.
...

A stage is a huge tarball of unpacked packages. Imho that's not ok: In 
the past users suggested switching to 'diff stages': A stage3 tarball 
would then only include the packages not in the stage2 tarball and not 
in the stage1. Stage2 only includes what is not in stage1. I'm a bit 
more ambitious and think tarballs of packages aren't needed because we 
already provide packages: either installed (and in use on the livecd 
the user is booting with) or packed (as small tarballs, which can be 
used for installation).

A package can be created from a set of packages (portage can do that), 
so given a livecd, portage can come up with a set of packages.
Instead of having lots of huge tarballs, or switch to complicated 'diff 
stages',  we can just declare what packages a stage3 consists of. The 
(declarative) statement:

"stage foobar consists of package foo and package bar"

is all that is needed for portage to create a tarball consisting of the 
foo and bar packages. How portage computes the foo and bar package 
(fetch from a repository, derive from an installed system, use ebuild 
knowledge to compile from source, ... ) is portages business (a 
different issue).

The advantage is this:

"stage foobarwithx consists of package foo and package bar and package 
x"
"stage foobar consists of package foo and package bar"
"stage foo consists of package foo"
"stage foowithx consists of package foo and package x"
"stage bar consist of package bar"

Assume every package (foo, bar or x) is 100M in size (compressed). 
Realizing the above with stage tarballs would take 900M, where just 
having the packages and the declarative representation of the stages 
would be only 300M in size. In our current system adding a new stage4 
tarball is impossible, storage wise. Adding a stage4,5,6,7,8,9,10... 
"virtual" representation is absolutely possible.

Best regards,

Pieter Van den Abeele





> Cheers,
> -- 
> John Davis
> Gentoo Linux Developer
> <http://dev.gentoo.org/~zhen>
>
> ----
> GnuPG Public Key: <http://dev.gentoo.org/~zhen/zhen_pub.asc>
> Fingerprint: 2364 71BD 4BC2 705D F338  FF70 6650 1235 1946 2D47
>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GRP set consolidation

[John Davis]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

On Fri, 2004-05-21 at 20:08, Joseph Booker wrote:
> John Davis said:
> > (btw, what does gentoopix
> > stand for?).
> 
> john: look at knoppix.net
> 
> is this sorta like a knoppix disc that can be generated from catalyste's
> conf files? if so, anyway of possibly sharing some work on the init
> scripts and stuff like persistant home dirs and things like klik ( a tool
> to install programs on knoppix while its booted)?

This is something that I have been mulling over but have not had the
chance to act upon, mostly because I have been coding catalyst to get it
to work correctly ;)

Our initscripts are very Gentoo-centric, so sharing those would probably
not be in our best interest (they work as is), but persistant homedirs
would be cool (as well as a slew of other things). So would boot
configurations that could be saved on a floppy or usb keychain. We can
use portage in place of klik. You get the idea - there are many exciting
opportunities here :)

First priority though is to get Catalyst the tool stabilizied. Then we
can move on to things that are more fun ;)

Cheers,
-- 
John Davis
Gentoo Linux Developer
<http://dev.gentoo.org/~zhen>

----
GnuPG Public Key: <http://dev.gentoo.org/~zhen/zhen_pub.asc>
Fingerprint: 2364 71BD 4BC2 705D F338  FF70 6650 1235 1946 2D47


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GRP set consolidation

[Joseph Booker]

John Davis said:
> Our initscripts are very Gentoo-centric, so sharing those would probably
> not be in our best interest (they work as is), but persistant homedirs
> would be cool (as well as a slew of other things). So would boot
> configurations that could be saved on a floppy or usb keychain. We can
> use portage in place of klik. You get the idea - there are many exciting
> opportunities here :)

using portage on a livecd.............I dont know weither to cry or laugh,
its like the perfect way to show off how long it takes to install oo.o
from source :P

> First priority though is to get Catalyst the tool stabilizied. Then we
> can move on to things that are more fun ;)
>

ha, i figured it was stablized if your using it for the stages and livecds
already. if i knew python i would find atleast something in catalyst to
write some patchs for to help ;)


-- 
 Joe Booker

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GRP set consolidation

[Nick Rout]

On Sat, 2004-05-22 at 11:15, John Davis wrote:
> On Fri, 2004-05-21 at 17:34, Joshua Brindle wrote:
> > Ok, we just had a conversation in #gentoo-dev and it seems most people 
> > are interested in this. The problem as it stands is that our mirrors are 
> >   low on space and so other projects can't get space, such as embedded, 
> > hardened and even other archs. The main user of space is having 5 grp 
> > packagecd's for x86 alone. The idea is to build a consolidated grp set 
> > with a generic -mcpu (i686 or so) and then use that.
> > 
> > As I'm sure this will become a large and flaming thread, and probably 
> > won't read them all I'll just go ahead and suggest that we add this to 
> > the next meeting to discuss it if there are large amounts of disagreement.
> > 
> > Joshua Brindle
> > 
> > --
> > gentoo-dev@gentoo.org mailing list
> 
> Josh -
> Seems like something good to research to me. GRP does take up a lot of
> space on the mirrors, not mentioning the fact that it takes a ton of
> time to build. 
> 
> But..
> 
> Before we do anything, I would like to gather some user input. I
> remember klieber talking about a system that he was setting up that we
> could use to survey users. Kurt, any progress? I don't want to make a
> relatively major change to releases without at least some user input.
> Cheers,
> //zhen

Just to stick in an oar here from someone who has used grp for a group
install, it is handy to have the individual grp packages on the mirrors,
rather than an iso.

Then if you want kde but not gnome, you just download the parts you
want.

we used PORTAGE_BINHOST= and emerge -g to do this from our own server
(had downloaded isos for 3 or 4 architectures first, and loopback
mounted them in /var/www/localhost/htdocs/gentoo/grp/$SUBARCH/, then a
fresh stage3 installee could type emerge -g kde and wander off for a
coffee.

Anyway I can see that space on the servers is limited, so I guess
putting individual packages on the server could not be additional to the
iso's.


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GRP set consolidation

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Fri, May 21, 2004 at 11:09:20PM +0100 or thereabouts, Stuart Herbert wrote:
> What is gentoopix?  Does it have a GLEP number? ;-)

Gentoopix is simply a Knoppix-like Gentoo CD.  It does not do the
installation -- it merely facilitates it by providing a full, usable
kde/gnome environment to the end user.  This gives the end user full use of
their computer so they can continue working while waiting for gentoo to
install in the background.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]