[gentoo-releng] x86 (ppc?) cd layout for 2004.0

[gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Daniel Robbins]

Hi guys,

I realize that this is a bit late to bring this up, but I wanted to
suggest an optimized organization for our x86 CDs.

Traditionally, we've had 5 disc 1's and 5 disc 2's. What I'm proposing
is that we move to 1 disc 1 and 5 disc 2's. There would be one disc 1
for all our sub-architectures. This would help the store in that we'd
have fewer unique CDs to produce. It would also remove about 2GB from
the mirrors.

Here's how disc1 would be organized (I'm using shorthand names for the
files):

(runtime data, about 75 MB)
/stages
	x86-stage1.tar.bz2
	x86-stage3.tar.bz2
	i686-stage3.tar.bz2
	pentium3-stage3.tar.bz2
	pentium4-stage3.tar.bz2
	athlon-xp-stage3.tar.bz2
/snapshots
	portage.tar.bz2
/distfiles
	225MB of sources (everything needed to go from
	stage1 to stage3 and to boot the system, plus kernel 	sources,
kernel-dependent ebuild sources, and if 	there's	room, X sources.

Necessary changes to the docs to accomodate this: mention that if the
user wants to install from stage2, the stage2 will need to be grabbed
from our mirrors using wget. Most users don't use stage2, so this won't
impact very many people.

Users no longer need to do a cp -a /mnt/cdrom/packages/All/* step for
CD1. This simplifies installation. The downside to this is that
things like metalog and vixie-cron will need to be compiled from
source. This isn't too horrible since they don't take particularly
long to compile. And the sources for these packages will be included
on CD1.

The lack of .tbz2s on CD1 would also mean that for the entire
installation process up to the first reboot, you don't need to use
"emerge -k." But using emerge with a -k would still continue to work.
This could allow for simplification of the docs (remove unnecessary '-k'
references.)

All in all, this would require very few changes to our existing install
procedure. No significant changes would be necessary.

CD2 would contain pre-built packages (the *entire* GRP set) for a
particular sub-arch, such as athlon-xp. The user would be able to pull
from the GRP set after he/she reboots the box into his new install and
mounts CD2.

I think this new layout is a good idea for several reasons. First of
all, it helps the store to deal with our continually increasing order
level, which will probably skyrocket with the 2004.0 release. I will
need to outsource the production of at least some of our CDs, and doing
1 run of 2500 CDs is a *lot* cheaper than doing 5 runs of 500 CDs.

Second, it takes some load off our mirrors, and gives users only one CD1
to download for all of x86, and the CD1 that they get is a lot more
versatile than what we currently offer. The 2GB freed up on the mirrors
could be used for the upcoming x86 KDE/GNOME LiveCD, as well as one or
two other special catalyst LiveCDs. This allows us to offer more cool
stuff to our users without adding any additional bloat to our mirrors.

Likewise, if this is adopted for PPC, it would allow us to have
potentially 1 instead of 3 (or maybe even 4?) disc 1's, and reduce our
mirror load by an additional 900MB to 1.4GB. It would also give ppc
users one CD1 to download that provides them with pre-built stage3's for
ppc, g3, g4 (32-bit g5?) which I think many would find convenient.

Please let me know what you think of this new layout proposal.

Regards,

Daniel


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 2194 bytes --]

On Wed, Feb 18, 2004 at 12:52:54AM -0700, Daniel Robbins wrote:
[...]
> Necessary changes to the docs to accomodate this: mention that if the
> user wants to install from stage2, the stage2 will need to be grabbed
> from our mirrors using wget. Most users don't use stage2, so this won't
> impact very many people.

This is fairly trivial.

> Users no longer need to do a cp -a /mnt/cdrom/packages/All/* step for
> CD1. This simplifies installation. The downside to this is that
> things like metalog and vixie-cron will need to be compiled from
> source. This isn't too horrible since they don't take particularly
> long to compile. And the sources for these packages will be included
> on CD1.

The main job here (documentation-wise) is to move all references to GRP
outside it's current location [1] and towards the beginning of the
installation (not the end as not all architectures follow the same
structure). That's not a real problem (at least not on first sight) but
translations of the Gentoo Handbook may lag behind a bit if this is pursued
for 2004.0.

> The lack of .tbz2s on CD1 would also mean that for the entire
> installation process up to the first reboot, you don't need to use
> "emerge -k." But using emerge with a -k would still continue to work.
> This could allow for simplification of the docs (remove unnecessary '-k'
> references.)

Those are already removed in the draft 'cause I hated them. Too many users
thought it was required.

> Please let me know what you think of this new layout proposal.

Well, personally I'm a bit disappointed as I won't be able to tell people
that Gentoo can be installed in less than 20 minutes without network
connection (it is possible using PORTAGE_BINHOST and an I'net connection).

I do agree with the advantages and don't see any showstoppers.

Wkr,
	Sven Vermeulen

[1] http://www.gentoo.org/doc/en/handbook/draft/handbook.xml?part=1&chap=5

-- 
  FOSDEM 2004               Free and Open Source Developers European Meeting
  21 - 22 februari          Brussels, Belgium          http://www.fosdem.org

  http://www.gentoo.org     Documentation & PR              swift@gentoo.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Pieter Van den Abeele]

On 18 Feb 2004, at 09:36, Sven Vermeulen wrote:

> Well, personally I'm a bit disappointed as I won't be able to tell 
> people
> that Gentoo can be installed in less than 20 minutes without network
> connection (it is possible using PORTAGE_BINHOST and an I'net 
> connection).

Isn't the smallest subset of packages needed to be able to boot netless 
into a stage3:

a kernel
a bootloader

only? (Ok, maybe some wierd things like pcmcia/hotplug (anybody using a 
pcmcia/usb cdrom?), ...) If the kernel is copied from the cd, only 
compiling the bootloader
shouldn't add much to those 20 minutes? I think I can do a ten minutes 
install using this technique (at least on ppc).

Pieter


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Daniel Robbins]

On Wed, 2004-02-18 at 07:22, Pieter Van den Abeele wrote:
> Isn't the smallest subset of packages needed to be able to boot netless 
> into a stage3:
> 
> a kernel
> a bootloader
> 
> only? (Ok, maybe some wierd things like pcmcia/hotplug (anybody using a 
> pcmcia/usb cdrom?), ...) If the kernel is copied from the cd, only 
> compiling the bootloader
> shouldn't add much to those 20 minutes? I think I can do a ten minutes 
> install using this technique (at least on ppc).

Thinking about it, you are correct. But moving logger and cron
installation to the "packages install" phase will require fairly
significant changes to our docs, and I don't know if it is wise to make
those changes so late.

But if we could, then yes, we could put all the loggers and crons on
CD2, and you could install 2004.0 without having to compile anything
from source except the kernel, just like 1.4.

Regards,

Daniel


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

On Wed, Feb 18, 2004 at 10:18:44AM -0700, Daniel Robbins wrote:
> Thinking about it, you are correct. But moving logger and cron
> installation to the "packages install" phase will require fairly
> significant changes to our docs, and I don't know if it is wise to make
> those changes so late.

It's also too important part of the installation. Perhaps not for the cron
(not everybody needs a cron daemon) but it is for the system logger. You
cannot ask the user to install a system logger after he has rebooted into his
Gentoo system. It'll upset the user as all information is printed on his
terminal.
 
Correct me if I'm wrong though, I too would appreciate it if a stage3 install
is done faster.

The 20 minutes is from bootup -> end, including partitioning, formatting,
configuring etc. I'm not able to do it much faster than that, sorry :)

Wkr,
	Sven Vermeulen

-- 
  FOSDEM 2004               Free and Open Source Developers European Meeting
  21 - 22 februari          Brussels, Belgium          http://www.fosdem.org

  http://www.gentoo.org     Documentation & PR              swift@gentoo.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Pieter Van den Abeele]

On 18 Feb 2004, at 21:23, Sven Vermeulen wrote:

> It's also too important part of the installation. Perhaps not for the 
> cron
> (not everybody needs a cron daemon) but it is for the system logger. 
> You
> cannot ask the user to install a system logger after he has rebooted 
> into his
> Gentoo system. It'll upset the user as all information is printed on 
> his
> terminal.

It is indeed a cosmetic issue, and when no logger is installed the 
login process spawns an ugly message making the user think his system 
is broken.
Looking at this from another perspective:

a stage3 is a stage2 + emerge system

The system profile includes virtuals (logger, cron, kernel, 
bootloader). The problem is that there are a few possibilities to 
realize these virtuals and thus there are different realizations for a 
given profile. We could choose a default realization of such a virtual, 
but for something like a logger this might be difficult, a user is not 
bound to like our choices :-). This brings up my earlier suggestion 
about a "virtual stage3". It would be trivial to have portage (-ng) 
automatically suggest a realization for a system profile using GRP, 
rather than to create different realizations. Not having to include 
stage3's, would allow to create only 1 cd instead of 2 cd sets. Instead 
of 1 bootable CD + 5 non-bootable CDs we would have only 5 bootable 
CDs.

Pieter


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Seemant Kulleen]

[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]

On Tue, 2004-02-17 at 23:52, Daniel Robbins wrote:
> Hi guys,
> 
> I realize that this is a bit late to bring this up, but I wanted to
> suggest an optimized organization for our x86 CDs.
> 
> Traditionally, we've had 5 disc 1's and 5 disc 2's. What I'm proposing
> is that we move to 1 disc 1 and 5 disc 2's. There would be one disc 1
> for all our sub-architectures. This would help the store in that we'd
> have fewer unique CDs to produce. It would also remove about 2GB from
> the mirrors.
> 
> Here's how disc1 would be organized (I'm using shorthand names for the
> files):
> 
> (runtime data, about 75 MB)
> /stages
> 	x86-stage1.tar.bz2
> 	x86-stage3.tar.bz2
> 	i686-stage3.tar.bz2
> 	pentium3-stage3.tar.bz2
> 	pentium4-stage3.tar.bz2
> 	athlon-xp-stage3.tar.bz2
> /snapshots
> 	portage.tar.bz2
> /distfiles
> 	225MB of sources (everything needed to go from
> 	stage1 to stage3 and to boot the system, plus kernel 	sources,
> kernel-dependent ebuild sources, and if 	there's	room, X sources.
> 

This layout really appeals to me (not that my opinion matters, but
anyway).  I think this is a great way to leverage the common aspects of
each sub architecture, and leads to less overall bloat and maintenance.
-- 
Seemant Kulleen
Developer and Project Co-ordinator,
Gentoo Linux					http://dev.gentoo.org/~seemant

Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x3458780E
Key fingerprint = 23A9 7CB5 9BBB 4F8D 549B 6593 EDA2 65D8 3458 780E


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

On Wed, 18 Feb 2004 00:52:54 -0700 Daniel Robbins <drobbins@gentoo.org>
wrote:
| Traditionally, we've had 5 disc 1's and 5 disc 2's. What I'm proposing
| is that we move to 1 disc 1 and 5 disc 2's. There would be one disc 1
| for all our sub-architectures. This would help the store in that we'd
| have fewer unique CDs to produce. It would also remove about 2GB from
| the mirrors.

I like this. One small request -- any chance of keeping the 'minimal' CD
around? No stages, no source, no binaries, just the runtime data. This
would be useful for:

* rescue CD
* stage1 type people with reasonable network connections that still
don't want to download lots of things they won't use (I suspect that
this covers a lot of our users...)
* people who make their own stages
* use with updated stages

A random poll in #gentoo suggests that the minimal CDs are extremely
popular, and I know that I'd miss it. 

As an additional suggestion from Seemant:

10:25 <@seemant> ciaranm: if no stage1, it should include a text file
with links to mirrors where stages are located
10:25 <@seemant> so like lynx file:///blah/stage-mirrors.html
10:25 <@seemant> and then instant "click" access to mirrors

-- 
Ciaran McCreesh
Mail:    ciaranm at gentoo.org
Web:     http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Daniel Robbins]

On Wed, 2004-02-18 at 03:35, Ciaran McCreesh wrote:
> I like this. One small request -- any chance of keeping the 'minimal' CD
> around? No stages, no source, no binaries, just the runtime data. This
> would be useful for:

I am pretty sure that the beejay is building a new minimal CD for 2004.0
(I believe he told me this last night.) I agree that it's incredibly
useful for lots of people and a relatively quick download.

Regards,

Daniel


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Pieter Van den Abeele]

Hi,

On 18 Feb 2004, at 08:52, Daniel Robbins wrote:
> Likewise, if this is adopted for PPC, it would allow us to have
> potentially 1 instead of 3 (or maybe even 4?) disc 1's, and reduce our
> mirror load by an additional 900MB to 1.4GB. It would also give ppc
> users one CD1 to download that provides them with pre-built stage3's 
> for
> ppc, g3, g4 (32-bit g5?) which I think many would find convenient.

We were already using the format you described in your email for the 
1.4 ppc discs, with the exception that I created different disc1's 
containing a subset of all stages. For the rest they were identical. 
There was no grp on disc 1.
Biggest advantage is that you can update grp only, without having to 
recreate the disc one.
Also, it removes the requirement to maintain two lists of binary 
packages.

The reason for choosing this format for the ppc disc made building a 
release much easier:

Unpack a stage3
emerge --buildpkg $PACKAGES
emerge --unmerge $UNWANTED

When finished you have a complete set of GRP, and a kde/gnome cd.
Creating the smaller cds, is just rerunning the script using the GRP 
built for kde/gnome but a smaller set of $PACKAGES and a different $USE 
of course.

On ppc , pre-releases are for generic ppc only (compatible with 
G3,G4,G5). Fosdem release is therefore in the format you described. For 
the official release, I want to reduce the set even further by making 
only one disc 1 for G3,G4 and G5 (G5 uses a different kernel, but 
identical initrd and live environment).

Pieter

>
> Please let me know what you think of this new layout proposal.
>
> Regards,
>
> Daniel
>
>
> --
> gentoo-releng@gentoo.org mailing list
>


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0[Message Scanned]

[John Davis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> 
> Please let me know what you think of this new layout proposal.
> 


Works for me -
Beejay, how is the LiveCD problem coming? I would vote to drop 2.4 and 
keep 2.6 if this is going to be a show stopper, but I will lead that up 
to the leads.

Regards,
//zhen

- -- 
John Davis
Gentoo Linux Developer
<http://dev.gentoo.org/~zhen>

- ----
GnuPG Public Key: <http://dev.gentoo.org/~zhen/zhen_pub.asc>
Fingerprint: 2364 71BD 4BC2 705D F338  FF70 6650 1235 1946 2D47
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFAM4U9ZlASNRlGLUcRAv2EAKDNJPvpw+Y7GxxIt+JuoN1SwPNf0wCfSyow
JEeHSitDToFSqDLlp7CmEKw=
=17Hp
-----END PGP SIGNATURE-----

--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] x86 (ppc?) cd layout for 2004.0

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Wed, Feb 18, 2004 at 12:52:54AM -0700, Daniel Robbins wrote:
> Necessary changes to the docs to accomodate this: mention that if the
> user wants to install from stage2, the stage2 will need to be grabbed
> from our mirrors using wget. Most users don't use stage2, so this won't
> impact very many people.
[...]

I've updated the documentation to reflect this change. The tarball [1] and
PDF [2] are still available from my dev-page.

Wkr,
	Sven Vermeulen

[1] http://dev.gentoo.org/~swift/handbook.tar.gz
[2] http://dev.gentoo.org/~swift/handbook.pdf

-- 
  FOSDEM 2004               Free and Open Source Developers European Meeting
  21 - 22 februari          Brussels, Belgium          http://www.fosdem.org

  http://www.gentoo.org     Documentation & PR              swift@gentoo.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]