[gentoo-catalyst] Keeping a test environment using catalyst

[gentoo-catalyst] Keeping a test environment using catalyst

[Paul Smith]

Hi again;

Thanks for the help the other day; I solved those problems.

Right now I'm working on getting some other packages installed into the
livecd.  The problem I'm having is that there doesn't seem to be any
fully usable "test environment" that reflects the livecd I can use for
development.  What I'm looking for is something that has the contents of
the livecd, but that also has the Portage database and tools installed
so that I can work with emerge, ebuild, etc. and get new packages
installed and working the way I want them to.

I can see the livecd-stage1, for example, leaves behind an image that I
could chroot to, but it's missing all the portage info so I can't really
do much in terms of testing there.


How do you all develop and test live CDs?  Is there some trick to
getting a test image available?  Or, do you do development via another
method and only start using catalyst when everything pretty much works?

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT: HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1516 bytes --]

On Tue, 2005-05-10 at 14:34 -0400, Paul Smith wrote:
> Right now I'm working on getting some other packages installed into the
> livecd.  The problem I'm having is that there doesn't seem to be any
> fully usable "test environment" that reflects the livecd I can use for
> development.  What I'm looking for is something that has the contents of
> the livecd, but that also has the Portage database and tools installed
> so that I can work with emerge, ebuild, etc. and get new packages
> installed and working the way I want them to.

You could just copy the portage tree into your livecd-stage1 directory.

> I can see the livecd-stage1, for example, leaves behind an image that I
> could chroot to, but it's missing all the portage info so I can't really
> do much in terms of testing there.

Do what catalyst does.  Bind mount it.

> How do you all develop and test live CDs?  Is there some trick to
> getting a test image available?  Or, do you do development via another
> method and only start using catalyst when everything pretty much works?

Honestly, I've been doing catalyst development (and use) long enough
that I pretty much know what to expect.  When something doesn't work, I
rebuild the entire image each time.  The reason for this is simple.  I
do not want to do *any* manual steps outside of catalyst, since it'll do
nothing but give you trouble in the long run.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Paul Smith]

%% Chris Gianelloni <wolf31o2@gentoo.org> writes:

  cg> Do what catalyst does.  Bind mount it.

Yes, that would work for /usr/portage, but I'm still missing all the
emerge, etc. commands.

  >> How do you all develop and test live CDs?  Is there some trick to
  >> getting a test image available?  Or, do you do development via another
  >> method and only start using catalyst when everything pretty much works?

  cg> Honestly, I've been doing catalyst development (and use) long
  cg> enough that I pretty much know what to expect.  When something
  cg> doesn't work, I rebuild the entire image each time.  The reason
  cg> for this is simple.  I do not want to do *any* manual steps
  cg> outside of catalyst, since it'll do nothing but give you trouble
  cg> in the long run.

This is good if you're just debugging catalyst itself, but I've got a
whole set of steps such as creating and debugging some ebuilds,
configuring things, creating rc scripts to do some stuff, etc. etc.

Obviously at the end it needs to build and boot correctly "from scratch"
but doing it from scratch for every incremental change is pretty slow.


If there are no fancy tricks that I'm missing I'm sure I can come up
with some myself :-).

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Robert Paskowitz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Portage is still around after livecd-stage1, it's needed to install
stuff in livecd-stage2.

Paul Smith wrote:
> %% Chris Gianelloni <wolf31o2@gentoo.org> writes:
> 
>   cg> Do what catalyst does.  Bind mount it.
> 
> Yes, that would work for /usr/portage, but I'm still missing all the
> emerge, etc. commands.
> 
>   >> How do you all develop and test live CDs?  Is there some trick to
>   >> getting a test image available?  Or, do you do development via another
>   >> method and only start using catalyst when everything pretty much works?
> 
>   cg> Honestly, I've been doing catalyst development (and use) long
>   cg> enough that I pretty much know what to expect.  When something
>   cg> doesn't work, I rebuild the entire image each time.  The reason
>   cg> for this is simple.  I do not want to do *any* manual steps
>   cg> outside of catalyst, since it'll do nothing but give you trouble
>   cg> in the long run.
> 
> This is good if you're just debugging catalyst itself, but I've got a
> whole set of steps such as creating and debugging some ebuilds,
> configuring things, creating rc scripts to do some stuff, etc. etc.
> 
> Obviously at the end it needs to build and boot correctly "from scratch"
> but doing it from scratch for every incremental change is pretty slow.
> 
> 
> If there are no fancy tricks that I'm missing I'm sure I can come up
> with some myself :-).
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCgSFsZwjIiODIZ4oRAmDbAJ4yXaq7UVTkr/6DUe/QbUZMxUuQwgCfTxxP
NdD1RhRWyIPcyPTLFelTCcU=
=uGAY
-----END PGP SIGNATURE-----
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Paul Smith]

%% Robert Paskowitz <r2d2@gentoo.org> writes:

  rp> Portage is still around after livecd-stage1, it's needed to install
  rp> stuff in livecd-stage2.

You're right: hm.  I must have been looking at stage2 when I thought I
was looking at stage1.  OK, that works.


Next question: I want to create a new runlevel, so I need a directory
/etc/runlevels/newlevel created in my image.  I thought I would do it in
the fsscript, but that happens too late: the rcadd stuff runs before
that and fails.

I guess I could put the rc-update into the fsscript too, after the
mkdir, rather than leaving it in the livecd/rcadd section.  Too bad
though.  Is there another easy way to get that directory created
(without adding a whole ebuild package or something for it)?

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Robert Paskowitz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I do the mkdir then rc-update stuff in my fsscript for the additonal
runlevels. Had to be careful to add some of the more mundane scripts
that are in default.

You could edit your seed stage to have the directories, or some dummy
ebuild, but it seems more effort than it's worth.

Maybe a livecd/runleveladd could be added. It should be pretty simple,
just needs to do some mkdir in the backend.


Paul Smith wrote:
> %% Robert Paskowitz <r2d2@gentoo.org> writes:
> 
>   rp> Portage is still around after livecd-stage1, it's needed to install
>   rp> stuff in livecd-stage2.
> 
> You're right: hm.  I must have been looking at stage2 when I thought I
> was looking at stage1.  OK, that works.
> 
> 
> Next question: I want to create a new runlevel, so I need a directory
> /etc/runlevels/newlevel created in my image.  I thought I would do it in
> the fsscript, but that happens too late: the rcadd stuff runs before
> that and fails.
> 
> I guess I could put the rc-update into the fsscript too, after the
> mkdir, rather than leaving it in the livecd/rcadd section.  Too bad
> though.  Is there another easy way to get that directory created
> (without adding a whole ebuild package or something for it)?
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCgVN6ZwjIiODIZ4oRArNDAJ4/xeg73L1khBzyFXTwtnddMonZCgCeOO+L
4EhTGLFc3oX4axTeaWPD5kc=
=Jipp
-----END PGP SIGNATURE-----
-- 
gentoo-catalyst@gentoo.org mailing list

[gentoo-catalyst] C++ programs

[Paul Smith]

Hm.  I installed a C++ program on the livecd, but it fails to
run because it's missing libstdc++.

Checking I see that on Gentoo, that library appears to live in the gcc
package.  The livecd unmerges the gcc package, so at runtime that
library no longer exists.

I understand that this library is tied very closely with the compiler,
but it seems incorrect to me that if you don't have the compiler package
installed on your system, all your C++ programs fail to start!  In
Debian for example this library is distributed as a separate package.


How do others manage this?  Do you unmerge it but copy that library to
/lib by hand?  Leave it merged but use livecd/empty and livecd/rm to
clean out the stuff you don't want?  Leave the whole package merged and
not worry about the wasted space? ... or ... ?

Hm, I see there's a sys-libs/libstdc++, but it's masked... is merging
that package how this issue is normally handled?


Cheers!

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]

On Tue, 2005-05-10 at 16:56 -0400, Paul Smith wrote:
> %% Chris Gianelloni <wolf31o2@gentoo.org> writes:
> 
>   cg> Do what catalyst does.  Bind mount it.
> 
> Yes, that would work for /usr/portage, but I'm still missing all the
> emerge, etc. commands.

You should have these during the livecd-stage* builds, as portage is not
unmerged.

>   >> How do you all develop and test live CDs?  Is there some trick to
>   >> getting a test image available?  Or, do you do development via another
>   >> method and only start using catalyst when everything pretty much works?
> 
>   cg> Honestly, I've been doing catalyst development (and use) long
>   cg> enough that I pretty much know what to expect.  When something
>   cg> doesn't work, I rebuild the entire image each time.  The reason
>   cg> for this is simple.  I do not want to do *any* manual steps
>   cg> outside of catalyst, since it'll do nothing but give you trouble
>   cg> in the long run.
> 
> This is good if you're just debugging catalyst itself, but I've got a
> whole set of steps such as creating and debugging some ebuilds,
> configuring things, creating rc scripts to do some stuff, etc. etc.
> 
> Obviously at the end it needs to build and boot correctly "from scratch"
> but doing it from scratch for every incremental change is pretty slow.

We don't do any shortcuts simply because we *are* the catalyst
developers.  When we find shortcomings in catalyst, we fix them.  ;]

> If there are no fancy tricks that I'm missing I'm sure I can come up
> with some myself :-).

Like I said before, I'm not sure I really follow what you're trying to
do.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1203 bytes --]

On Tue, 2005-05-10 at 20:23 -0400, Paul Smith wrote:
> Next question: I want to create a new runlevel, so I need a directory
> /etc/runlevels/newlevel created in my image.  I thought I would do it in
> the fsscript, but that happens too late: the rcadd stuff runs before
> that and fails.
> 
> I guess I could put the rc-update into the fsscript too, after the
> mkdir, rather than leaving it in the livecd/rcadd section.  Too bad
> though.  Is there another easy way to get that directory created
> (without adding a whole ebuild package or something for it)?

The "proper" way would be via an ebuild first, and fsscript second.

Remember also, that catalyst is not designed to be everything to
everyone.  One of the primary reasons for having catalyst calling tons
of external scripts, rather than incorporating everything into a single
monolithic script/executable, is to allow you, the user, to replace
portions of it to do what you want.  It is designed to be flexible, but
sometimes it requires someone to rewrite a module or modify it to get
the exact results desired.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] C++ programs

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1298 bytes --]

On Tue, 2005-05-10 at 21:13 -0400, Paul Smith wrote:
> Hm.  I installed a C++ program on the livecd, but it fails to
> run because it's missing libstdc++.
> 
> Checking I see that on Gentoo, that library appears to live in the gcc
> package.  The livecd unmerges the gcc package, so at runtime that
> library no longer exists.
> 
> I understand that this library is tied very closely with the compiler,
> but it seems incorrect to me that if you don't have the compiler package
> installed on your system, all your C++ programs fail to start!  In
> Debian for example this library is distributed as a separate package.
> 
> 
> How do others manage this?  Do you unmerge it but copy that library to
> /lib by hand?  Leave it merged but use livecd/empty and livecd/rm to
> clean out the stuff you don't want?  Leave the whole package merged and
> not worry about the wasted space? ... or ... ?
> 
> Hm, I see there's a sys-libs/libstdc++, but it's masked... is merging
> that package how this issue is normally handled?

I unmask the package and merge it.

This works perfectly for ut2004, fluxbox, and americas-army ebuilds, all
which require libstdc++ to function.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Paul Smith]

%% Chris Gianelloni <wolf31o2@gentoo.org> writes:

  cg> We don't do any shortcuts simply because we *are* the catalyst
  cg> developers.  When we find shortcomings in catalyst, we fix them.  ;]

  >> If there are no fancy tricks that I'm missing I'm sure I can come up
  >> with some myself :-).

  cg> Like I said before, I'm not sure I really follow what you're
  cg> trying to do.

Hi Chris; thanks for taking the time to reply!

I completely see your viewpoint here.  My situation is that I'm trying
to add somewhat more esoteric software to my livecd than comes on a
"typical" livecd.  Some of it doesn't even have any Gentoo ebuilds, so
I'm writing them myself.  For ebuilds that are in Gentoo, I need to run
emerge -p to see what would be pulled in, investigate the USE flags in
the ebuilds to see if I can turn off features I don't like, then see
what would be pulled in, etc.  Also, to find out what I can unmerge and
still have the thing work properly.

In a situation like this it's very time consuming to have to rebuild
livecd1 and then livecd2 every time I need to tweak an ebuild or add or
remove a package, so I was looking for a "development mode" where things
were maybe not as clean, but were "good enough" to allow development
work to proceed quickly.  Then obviously there's polishing at the end
that would have to be done more cleanly.


I quite understand your position: you are using ostensibly stable Gentoo
packages and your work is focused on making sure Catalyst works.  I'm
using unstable/development packages and assuming that Catalyst is stable.

Anyway, I think I have enough now to work with for the moment; adding in
libstdc++ works (I did this last night but forgot to send email), and I
can bind-mount the snapshot and chroot and that works as well.

My current project is understanding where the kernel boot parameters
come from (my bootline has "nodhcp" which I don't want).  My other
problem is that the hardware detection stuff is running too late in the
"default" boot process; it brings up eth0, then sshd, then apache,
_THEN_ runs the hardware detection.  So, I can't quite figure out how it
brings up eth0 without detecting the hardware... maybe it runs twice?
Hm.


Cheers for now!

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 3599 bytes --]

On Wed, 2005-05-11 at 12:39 -0400, Paul Smith wrote:
> %% Chris Gianelloni <wolf31o2@gentoo.org> writes:
> 
>   cg> We don't do any shortcuts simply because we *are* the catalyst
>   cg> developers.  When we find shortcomings in catalyst, we fix them.  ;]
> 
>   >> If there are no fancy tricks that I'm missing I'm sure I can come up
>   >> with some myself :-).
> 
>   cg> Like I said before, I'm not sure I really follow what you're
>   cg> trying to do.
> 
> Hi Chris; thanks for taking the time to reply!
> 
> I completely see your viewpoint here.  My situation is that I'm trying
> to add somewhat more esoteric software to my livecd than comes on a
> "typical" livecd.  Some of it doesn't even have any Gentoo ebuilds, so
> I'm writing them myself.  For ebuilds that are in Gentoo, I need to run
> emerge -p to see what would be pulled in, investigate the USE flags in
> the ebuilds to see if I can turn off features I don't like, then see
> what would be pulled in, etc.  Also, to find out what I can unmerge and
> still have the thing work properly.

OK.  This makes sense to me now.

> In a situation like this it's very time consuming to have to rebuild
> livecd1 and then livecd2 every time I need to tweak an ebuild or add or
> remove a package, so I was looking for a "development mode" where things
> were maybe not as clean, but were "good enough" to allow development
> work to proceed quickly.  Then obviously there's polishing at the end
> that would have to be done more cleanly.

Your best bet here is then to chroot into your livecd-stage1 and start
playing.  Remember that catalyst just uses "emerge" for most of its
work, so you can easily duplicate the steps.

You might also want to look into the tinderbox target.  It will allow
you to determine just what is required above and beyond a stage3 tarball
to get your packages working.

Personally, I prefer the USE="-*" approach, rather than deciding what
you don't need, try to figure out what you want.  Turn everything off,
then only turn on what you really need.

> I quite understand your position: you are using ostensibly stable Gentoo
> packages and your work is focused on making sure Catalyst works.  I'm
> using unstable/development packages and assuming that Catalyst is stable.

Catalyst is anything but stable.  It works quite well for building a
*Gentoo* release, but is not well suited for other things (yet) simply
due to that being the only focus of catalyst until recently.

> Anyway, I think I have enough now to work with for the moment; adding in
> libstdc++ works (I did this last night but forgot to send email), and I
> can bind-mount the snapshot and chroot and that works as well.

Cool.

> My current project is understanding where the kernel boot parameters
> come from (my bootline has "nodhcp" which I don't want).  My other
> problem is that the hardware detection stuff is running too late in the
> "default" boot process; it brings up eth0, then sshd, then apache,
> _THEN_ runs the hardware detection.  So, I can't quite figure out how it
> brings up eth0 without detecting the hardware... maybe it runs twice?

All of the boot parameters come from the archscript.  Normally,
autoconfig is started first due to it being first alphabetically.  The
"hwsetup" portion you see on the Gentoo releases is done by autoconfig,
and runs before coldplug.

Just curious, but what arch are you building?  x86?  amd64?  ppc?

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Paul Smith]

%% Chris Gianelloni <wolf31o2@gentoo.org> writes:

  >> I quite understand your position: you are using ostensibly stable
  >> Gentoo packages and your work is focused on making sure Catalyst
  >> works.  I'm using unstable/development packages and assuming that
  >> Catalyst is stable.

  cg> Catalyst is anything but stable.  It works quite well for building
  cg> a *Gentoo* release, but is not well suited for other things (yet)
  cg> simply due to that being the only focus of catalyst until
  cg> recently.

:-)  Yes, understood.

  >> My current project is understanding where the kernel boot parameters
  >> come from (my bootline has "nodhcp" which I don't want).  My other
  >> problem is that the hardware detection stuff is running too late in the
  >> "default" boot process; it brings up eth0, then sshd, then apache,
  >> _THEN_ runs the hardware detection.  So, I can't quite figure out how it
  >> brings up eth0 without detecting the hardware... maybe it runs twice?

  cg> All of the boot parameters come from the archscript.

I found this now.

  cg> Normally, autoconfig is started first due to it being first
  cg> alphabetically.

Oops.  That's not terribly robust :-).  I'm adding "apache2" and that
comes before "autoconfig".  Not sure why SSH is starting earlier, but
apache2 depends on "net" and presumably (although there isn't a "net"
runscript) that somehow translates into net.eth0 etc.

  cg> The "hwsetup" portion you see on the Gentoo releases is done by
  cg> autoconfig, and runs before coldplug.

Hm.  In my version of autoconfig this seems to be commented out:

  depend() {
          need modules
          use alsasound
  #       provide gpm pcmcia apmd acpid coldplug
  }

Maybe that's my problem?  I'm getting this from livecd-tools-1.0.20.

I don't see anything depending on coldplug either.

It sure would be nice if the rc scripts tools had some debugging
capabilities, like a "don't do it" mode that just printed what would be
run for a given runlevel or init script, and why.  It would make this
stuff a lot simpler...

It seems like basic things like "net" should depend on coldplug and/or
autoconfig.

  cg> Just curious, but what arch are you building?  x86?  amd64?  ppc?

x86

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2513 bytes --]

On Wed, 2005-05-11 at 14:39 -0400, Paul Smith wrote:
> Oops.  That's not terribly robust :-).  I'm adding "apache2" and that
> comes before "autoconfig".  Not sure why SSH is starting earlier, but
> apache2 depends on "net" and presumably (although there isn't a "net"
> runscript) that somehow translates into net.eth0 etc.

Well, in Gentoo, unless it is moved due to dependency, it will start
alphabetically.  Also 2 init scripts with the same dependencies (say
apache2 and apcupsd) will start alphabetically, since both have "need
net" and nothing to do with each other.

I am adding a "before net" to autoconfig, which should resolve this
issue.

>   cg> The "hwsetup" portion you see on the Gentoo releases is done by
>   cg> autoconfig, and runs before coldplug.
> 
> Hm.  In my version of autoconfig this seems to be commented out:

>   depend() {
>           need modules
>           use alsasound
>   #       provide gpm pcmcia apmd acpid coldplug
>   }
> 
> Maybe that's my problem?  I'm getting this from livecd-tools-1.0.20.

hwsetup, not the depend.  That line is commented on purpose, as I still
haven't found a good way to get this to work without throwing warnings
all over the screen about "$blah already provided by autoconfig,
skipping"... for some reason, users like to file bug about anything that
shows up on the screen that they think could possibly be an error... ;]

> I don't see anything depending on coldplug either.

No, but read the script.  It starts coldplug itself.  If you add
coldplug via rcadd/rc-update, then you're short-circuiting the boot.
Remember that some of this stuff is a hack that "just works" when used
for the releases... ;]

> It sure would be nice if the rc scripts tools had some debugging
> capabilities, like a "don't do it" mode that just printed what would be
> run for a given runlevel or init script, and why.  It would make this
> stuff a lot simpler...

Yeah, the init system still is kinda a black art.

> It seems like basic things like "net" should depend on coldplug and/or
> autoconfig.

Except that net requires neither.  That and the baselayout guys don't
like to "pollute" the base init scripts, but rather would suggest that
autoconfig should "use net", which I have just added.

As for coldplug, it is started by autoconfig itself, and should never be
added manually to a LiveCD.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Paul Smith]

%% Chris Gianelloni <wolf31o2@gentoo.org> writes:

  cg> Well, in Gentoo, unless it is moved due to dependency, it will start
  cg> alphabetically.

Understand: my point was that relying on "autoconfig" appearing first
alphabetically doesn't seem too robust (as proven by the fact it didn't
DTRT anymore when I added "apache", which comes first :-)).

  cg> I am adding a "before net" to autoconfig, which should resolve this
  cg> issue.

OK, that's a good idea.

I added "autoconfig:boot" to the livecd/rcadd section to start it in the
boot sequence instead of the default sequence.

That seemed to work but your way looks like it might be more clean.

  >> I don't see anything depending on coldplug either.

  cg> No, but read the script.  It starts coldplug itself.  If you add
  cg> coldplug via rcadd/rc-update, then you're short-circuiting the boot.
  cg> Remember that some of this stuff is a hack that "just works" when used
  cg> for the releases... ;]

  cg> Yeah, the init system still is kinda a black art.

It's very cool, it's just not fully featured yet.  It seems like they
got it working but then stopped and didn't add any usability or
developer capabilities :-(.  And even the ones that are there are not
documented.  Oh well.


I feel like I've just about finished the part of this that deals heavily
with Catalyst and got it all working: the remaining bits are getting all
my extra packages working properly.

Random question: has anyone thought about or looked at using unionfs or
mini_fo or one of the other "translucent" or "fan-out" or whatever you
want to call them, filesystems?  I mean, for allowing a writable
configuration for a livecd without needing to strew symlinks etc. around
the base filesystem?

Cheers!

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: [gentoo-catalyst] Keeping a test environment using catalyst

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Thu, 2005-05-12 at 12:39 -0400, Paul Smith wrote:
> Random question: has anyone thought about or looked at using unionfs or
> mini_fo or one of the other "translucent" or "fan-out" or whatever you
> want to call them, filesystems?  I mean, for allowing a writable
> configuration for a livecd without needing to strew symlinks etc. around
> the base filesystem?

Remember that any such changes, from a Release Engineering standpoint,
are bad.  They take a long time to ensure they work, and we try our best
to not use anything that isn't available/stable on all arches we
support.

That being said, I would love to see some work being done by somebody on
poking around with unionfs.  I think it would be very good for the
upcoming LiveCD/Installer disk, as it would allow people to configure
their settings, then transfer them to the running system once the
installation is complete.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Unionfs (was: Re: [gentoo-catalyst] Keeping a test environment using catalyst)

[Paul Smith]

%% Chris Gianelloni <wolf31o2@gentoo.org> writes:

  cg> That being said, I would love to see some work being done by
  cg> somebody on poking around with unionfs.  I think it would be very
  cg> good for the upcoming LiveCD/Installer disk, as it would allow
  cg> people to configure their settings, then transfer them to the
  cg> running system once the installation is complete.

So, I've compared the Gentoo initrd with the Knoppix 3.8.2
(unionfs-based) initrd.  The Knoppix init is different in some
interesting ways: for one thing they never pivot_root, nor chroot or
anything.  They basically use the same / directory, that was created by
the kernel during initrd, for the entire lifetime of the system.

What they do, basically, is this (I've stripped out a lot of detail of
course):

    mkdir /ramdisk /UNIONFS

    mount -o tmpfs /ramdisk /ramdisk
    mount -o unionfs -o dirs=/ramdisk=rw:/cdrom=ro /UNIONFS /UNIONFS

Now they have a unionfs filesystem at /UNIONFS, which has at the top a
ramdisk (really located in /ramdisk) and underneath that the cdrom
contents (really located in /cdrom).

They then proceed to, essentially, make everything in / be a symlink
pointing into /UNIONFS (obviously there are various complexities
involved here).

Then they just exec the real init.

I don't know why they don't use pivot_root; that seems cleaner to me.
But, maybe something about unionfs doesn't allow pivot_root to work.
I'll keep poking at this.


I _thought_ that Knoppix allowed you to make the top of the unionfs an
actual device, like a USB keydisk or something, instead of a ramdisk,
but offhand I don't see anything in their initrd that does anything like
that: it seems hardcoded to go to a ramdisk as far as I can tell.  I'll
make sure of this; to me that's the real benefit of unionfs: to use it
to customize/personalize a generic livecd, and make those customizations
persistent.  If you don't have this I'm not sure what the advantage of
unionfs is over the more traditional methods, like Gentoo's.  I guess
the only real advantage is that the entire disk is modifiable rather
than just specific areas.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: Unionfs (was: Re: [gentoo-catalyst] Keeping a test environment using catalyst)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2513 bytes --]

On Thu, 2005-05-19 at 11:37 -0400, Paul Smith wrote:
> What they do, basically, is this (I've stripped out a lot of detail of
> course):
> 
>     mkdir /ramdisk /UNIONFS
> 
>     mount -o tmpfs /ramdisk /ramdisk
>     mount -o unionfs -o dirs=/ramdisk=rw:/cdrom=ro /UNIONFS /UNIONFS
> 
> Now they have a unionfs filesystem at /UNIONFS, which has at the top a
> ramdisk (really located in /ramdisk) and underneath that the cdrom
> contents (really located in /cdrom).
> 
> They then proceed to, essentially, make everything in / be a symlink
> pointing into /UNIONFS (obviously there are various complexities
> involved here).
> 
> Then they just exec the real init.
> 
> I don't know why they don't use pivot_root; that seems cleaner to me.
> But, maybe something about unionfs doesn't allow pivot_root to work.
> I'll keep poking at this.

I'm just curious why this needs to be done within the initrd, at all.  I
guess it could be, but I could also see it working outside the initrd.

Anyway, if we were to do it, we would allow for mounting a USB key or
other such device.  Most likely, it would be via a unionfs=/dev/$blah
option, that would mount that device, along with the CD, as a unionfs.
One thing I probably would *not* do is make the entire CD a unionfs, as
I just think it would be a waste of energy and would make
troubleshooting much harder.  Things like /var, /etc, /root, and /home
would definitely be cool to not only be writable, but also survivable
from a reboot.

> I _thought_ that Knoppix allowed you to make the top of the unionfs an
> actual device, like a USB keydisk or something, instead of a ramdisk,
> but offhand I don't see anything in their initrd that does anything like
> that: it seems hardcoded to go to a ramdisk as far as I can tell.  I'll
> make sure of this; to me that's the real benefit of unionfs: to use it
> to customize/personalize a generic livecd, and make those customizations
> persistent.  If you don't have this I'm not sure what the advantage of
> unionfs is over the more traditional methods, like Gentoo's.  I guess
> the only real advantage is that the entire disk is modifiable rather
> than just specific areas.

Having the entire disk modifiable, in many cases, isn't a good idea.

I would prefer that only certain parts be writable.  After all, who
really needs to write to /usr or /bin on a CD?

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Unionfs (was: Re: [gentoo-catalyst] Keeping a test environment using catalyst)

[Paul Smith]

%% Chris Gianelloni <wolf31o2@gentoo.org> writes:

  cg> I'm just curious why this needs to be done within the initrd, at
  cg> all.  I guess it could be, but I could also see it working outside
  cg> the initrd.

If you do it after initrd, you'll have to be very, very careful.
"Outside" the initrd you've invoked the real init, and it's starting to
bring up the system.  If you haven't already set up the unionfs before
that, you better be darn sure you do it immediately... if not you will
have applications coming up with the wrong configs, failing because they
can't write to /etc and/or /var, etc.

  cg> Anyway, if we were to do it, we would allow for mounting a USB key
  cg> or other such device.  Most likely, it would be via a
  cg> unionfs=/dev/$blah option, that would mount that device, along
  cg> with the CD, as a unionfs.

Yes.  I have this working right now, although it just uses a ramdisk
today.  Obviously using my current method, in order to use a keydisk or
maybe even a file on a FAT partition which contains an ext2 partition
and is mounted as a loop FS, you would need to add smarts to the initrd
(linuxrc) to detect the hardware and mount those partitions.

  cg> One thing I probably would *not* do is make the entire CD a
  cg> unionfs, as I just think it would be a waste of energy and would
  cg> make troubleshooting much harder.

I'm not sure what you mean by "not make the entire CD a unionfs"...?  Do
you just mean that some root directories would not link to the unionfs
but instead would go to the cdrom?  Or...?

  cg> Things like /var, /etc, /root, and /home would definitely be cool
  cg> to not only be writable, but also survivable from a reboot.

Yes, that's the ultimate goal of course.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: Unionfs (was: Re: [gentoo-catalyst] Keeping a test environment using catalyst)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2579 bytes --]

On Wed, 2005-05-25 at 13:54 -0400, Paul Smith wrote:
> %% Chris Gianelloni <wolf31o2@gentoo.org> writes:
> 
>   cg> I'm just curious why this needs to be done within the initrd, at
>   cg> all.  I guess it could be, but I could also see it working outside
>   cg> the initrd.
> 
> If you do it after initrd, you'll have to be very, very careful.
> "Outside" the initrd you've invoked the real init, and it's starting to
> bring up the system.  If you haven't already set up the unionfs before
> that, you better be darn sure you do it immediately... if not you will
> have applications coming up with the wrong configs, failing because they
> can't write to /etc and/or /var, etc.

That's a good point.  I was just thinking that by holding off until a
bit later, it allows us to have our full module list, rather than the
limited subset provided in the initrd.

Essentially, anything that we can use to boot a CD from, we can use as a
unionfs device.  This would include IDE/SCSI/SATA/USB/Firewire devices.

>   cg> Anyway, if we were to do it, we would allow for mounting a USB key
>   cg> or other such device.  Most likely, it would be via a
>   cg> unionfs=/dev/$blah option, that would mount that device, along
>   cg> with the CD, as a unionfs.
> 
> Yes.  I have this working right now, although it just uses a ramdisk
> today.  Obviously using my current method, in order to use a keydisk or
> maybe even a file on a FAT partition which contains an ext2 partition
> and is mounted as a loop FS, you would need to add smarts to the initrd
> (linuxrc) to detect the hardware and mount those partitions.

Right.  This would take more logic in the initrd.

>   cg> One thing I probably would *not* do is make the entire CD a
>   cg> unionfs, as I just think it would be a waste of energy and would
>   cg> make troubleshooting much harder.
> 
> I'm not sure what you mean by "not make the entire CD a unionfs"...?  Do
> you just mean that some root directories would not link to the unionfs
> but instead would go to the cdrom?  Or...?

Correct.  I don't really see a point in having the entire CD be
writable.  I can understand it for configuration directories and such,
but nobody really needs to write to /bin or /usr on a CD.

>   cg> Things like /var, /etc, /root, and /home would definitely be cool
>   cg> to not only be writable, but also survivable from a reboot.
> 
> Yes, that's the ultimate goal of course.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Unionfs (was: Re: [gentoo-catalyst] Keeping a test environment using catalyst)

[Paul Smith]

%% Chris Gianelloni <wolf31o2@gentoo.org> writes:

  cg> That's a good point.  I was just thinking that by holding off
  cg> until a bit later, it allows us to have our full module list,
  cg> rather than the limited subset provided in the initrd.

Well, you still can have your full module list during linuxrc.  After
you mount the cdrom you can just change /lib to point to the version on
the cdrom partition (whether you do it directly or through the unionfs,
it doesn't really matter).

Now, making this change involves a little bit of fancy footwork, since
some applications, even in the initrd, will be dynamically linked and
trying to swap out /lib can give them a migraine.  However, taking a
page from the Knoppix linuxrc I got it to work by invoking the runtime
linker from the cdrom directly; here's a snippet:

  # Run a command using "safe" paths to the ORIGFS.  Only valid once
  # that's been mounted!  Note that the command MUST have a
  # fully-qualified path!
  runit() {
    cmd="$1"
    shift
    "${ORIGFS}/lib/ld-linux.so.2" --library-path "${ORIGFS}/lib" \
      "${ORIGFS}$cmd" "$@"
  }

  cd /
  mkdir -p /.old
  for f in etc lib bin boot sbin var opt root usr; do
    runit /usr/bin/test -e "/$f" \
      && runit /bin/mv "/$f" /.old \
      && runit /bin/ln -snf "${UNIONFS}/$f" "/$f"
  done
  $DEBUG || rm -rf /.old

Here, ORIGFS would be the mount point for the cdrom.  Of course you can
change this into two loops easily enough if you don't want all of these
to go through the unionfs.


Anyway, once you get this done your "/" contains the fully-expressed
squashfs contents, not just the stuff in the initrd, so you can load up
any other modules you think are necessary, that might not exist in the
initrd filesystem.


Cheers!

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list

Re: Unionfs (was: Re: [gentoo-catalyst] Keeping a test environment using catalyst)

[Paul Smith]

In retrospect, is busybox in the initrd linked statically?  I didn't
check.  If it is then we SHOULD be able to hardcode the busybox versions
of those tools (test, mv, ln) rather than using the cdrom versions.

One simple way to do it (using my algorithm of moving stuff into /.old
then deleting it all at the end) would be to put /.old/bin on the PATH,
like this:

    PATH=/.old/bin:/bin

Hm, does the busybox shell do command search hashing?  If so that's a
bit of a problem.  Well, it should be simple to work around.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@nortel.com>           HASMAT--HA Software Mthds & Tools
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
-------------------------------------------------------------------------------
   These are my opinions---Nortel Networks takes no responsibility for them.
-- 
gentoo-catalyst@gentoo.org mailing list