[gentoo-releng] catalyst livecd improvements

[gentoo-releng] catalyst livecd improvements

[Daniel Robbins]

Hi guys,

I've just committed a batch of improvements to catalyst. They are not
tested yet (will try to run some tests now) but should work. (Let me
know if you have any problems.)

* The boot/kernel/foo/extraversion parameter is now optional rather
  than required. modules/targets.py and
  examples/livecd/runscript/default-runscript.sh 
  have been updated to work smoothly if it is not specified.

* New parameter /boot/kernel/foo/packages to specify packages to
  merge after your kernel is built. Use this to merge any kernel-
  dependent things like nvidia-kernel, e1000, etc.

* New parameter /boot/kernel/foo/use to specify USE settings to
  export to the environment. These settings will be used for kernel
  source merge and /boot/kernel/foo/packages merge.

* emerge of kernel sources no longer uses the pkgcache. There is
  no real time benefit to pkgcache in this case, and this allows us
  to use the /boot/kernel/foo/use setting to tweak what patches may
  be included in the kernel sources we are compiling. Also, packages
  specified in /boot/kernel/foo/packages are not built using
  pkgcache, since they are kernel-dependent ebuilds and can't be
  re-used if the kernel sources change.

* targets/livecd-stage2/* has been deprecated. These steps are all
  included in the runscript now. Use the latest default-runscript.sh
  to take advantage of this.

As always, let me know if you need any catalyst features implemented. As
people start using catalyst on their own, we sometimes don't realize
that certain features are needed, so we need your requests. (If catalyst
doesn't do what you need, it is likely
because we haven't added the feature yet rather than an issue of
catalyst not being able to do it.)

Regards,

Daniel


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] catalyst livecd improvements

[tigger]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On Wed, Feb 11, 2004 at 12:22:16PM -0700, Daniel Robbins wrote:

> * targets/livecd-stage2/* has been deprecated. These steps are all
>   included in the runscript now. Use the latest default-runscript.sh
>   to take advantage of this.

The runscript doesn't do livecd/unmerge or livecd/empty as far as I can
see. So stage2 is still needed. Let me know if I'm missing something.

Also:

I added /var/ccache to cleanables so that it was deleted before creating
the livecd. Now someone has changed the catalyst code to sometimes put
ccache stuff in /root/.ccache, but they haven't added any code to clean
it up.

I couldn't easily figure out how to do it now that its a moving target,
I had four attempts and none of them working so I just added
/root/.ccache to the livecd/empty variable :/

-- 
rob holland - [ tigger@gentoo.org ]
irc://irc.freenode.net/#gentoo as tigger^
http://dev.gentoo.org/~tigger/tigger@gentoo.org.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] catalyst livecd improvements

[Benjamin Judas]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

Am Mi, den 11.02.2004 schrieb Daniel Robbins um 20:22:
> Hi guys,
> 
> I've just committed a batch of improvements to catalyst. They are not
> tested yet (will try to run some tests now) but should work. (Let me
> know if you have any problems.)
[...]
> * New parameter /boot/kernel/foo/packages to specify packages to
>   merge after your kernel is built. Use this to merge any kernel-
>   dependent things like nvidia-kernel, e1000, etc.
[...]

This doesn't work. I used this parameter in my spec-file like that :
<paste>
boot/kernel: gentoo smp
boot/kernel/gentoo/sources: =sys-kernel/xfs-sources-2.4.24
boot/kernel/gentoo/config: examples/livecd/x86/config-2.4.24-x86
boot/kernel/gentoo/packages: nvnet pcmcia-cs linux-wlan-ng slmodem  <--
boot/kernel/smp/sources: =sys-kernel/gentoo-dev-sources-2.6.1-r1
boot/kernel/smp/config: examples/livecd/x86/config-2.6.1-x86
</paste>

I redirected catalysts output to a log-file. Searching in that logfile
(/pcmcia-cs) didn't get any result. I have taken a look into catalyst
but I don't really see the point. (I might need some time to work my way
through the sources). So for now I will try to continue with my old
approach (i.e. creating a specialized version of default-runscript that
fulfills my needs). Maybe someone of you has an idea why
boot/kernel/gentoo/packages isn't processed.

Regards,
  Benni

-- 
Benjamin Judas                                  http://dev.gentoo.org/~beejay
Gentoo-developer                                        http://www.gentoo.org
Germany

GPG-Public-Key : http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC31DEDD8
Key-Fingerprint : 4E65 AAFE 785B 61D8 E4D9  1671 E017 87B7 C31D EDD8

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] catalyst livecd improvements

[Daniel Robbins]

On Thu, 2004-02-12 at 04:04, tigger@gentoo.org wrote:
> The runscript doesn't do livecd/unmerge or livecd/empty as far as I can
> see. So stage2 is still needed. Let me know if I'm missing something.

This is handled by the python code.

> I added /var/ccache to cleanables so that it was deleted before creating
> the livecd. Now someone has changed the catalyst code to sometimes put
> ccache stuff in /root/.ccache, but they haven't added any code to clean
> it up.

Please do not commit directly to the catalyst code. You can send me
patches if you have any changes that need to go in.

catalyst puts the ccache dir inside /var in the chroot, and sets the
CCACHE_DIR environment variable accordingly.

> I couldn't easily figure out how to do it now that its a moving target,
> I had four attempts and none of them working so I just added
> /root/.ccache to the livecd/empty variable :/

Again, please don't make direct commits to the catalyst code. Thanks.

Sincerely,

Daniel


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] catalyst livecd improvements

[Daniel Robbins]

On Thu, 2004-02-12 at 09:20, Benjamin Judas wrote:
> This doesn't work. I used this parameter in my spec-file like that :
> <paste>
> boot/kernel: gentoo smp
> boot/kernel/gentoo/sources: =sys-kernel/xfs-sources-2.4.24
> boot/kernel/gentoo/config: examples/livecd/x86/config-2.4.24-x86
> boot/kernel/gentoo/packages: nvnet pcmcia-cs linux-wlan-ng slmodem  <--
> boot/kernel/smp/sources: =sys-kernel/gentoo-dev-sources-2.6.1-r1
> boot/kernel/smp/config: examples/livecd/x86/config-2.6.1-x86
> </paste>

OK, will do my own testing today and get that fixed for you.

Regards,

Daniel


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] catalyst livecd improvements

[Daniel Robbins]

On Thu, 2004-02-12 at 12:03, Daniel Robbins wrote:
> On Thu, 2004-02-12 at 04:04, tigger@gentoo.org wrote:
> > The runscript doesn't do livecd/unmerge or livecd/empty as far as I can
> > see. So stage2 is still needed. Let me know if I'm missing something.
> 
> This is handled by the python code.

Actually, you're right in that there is a
targets/livecd-stage2/unmerge.sh that is called by the python code. The
livecd/empty part is handled completely in python. unmerge.sh is
staying, but the livecd-stage2.sh file shouldn't be needed anymore so
it's been removed.

Regards,

Daniel


--
gentoo-releng@gentoo.org mailing list

Re: [gentoo-releng] catalyst livecd improvements

[tigger]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

On Thu, Feb 12, 2004 at 12:03:30PM -0700, Daniel Robbins wrote:
> On Thu, 2004-02-12 at 04:04, tigger@gentoo.org wrote:
> > The runscript doesn't do livecd/unmerge or livecd/empty as far as I can
> > see. So stage2 is still needed. Let me know if I'm missing something.
>
> This is handled by the python code.

I'm well aware of that. However its only done if target=livecd-stage2.
Is this target now set outside of the specfile or something?

> > I added /var/ccache to cleanables so that it was deleted before creating
> > the livecd. Now someone has changed the catalyst code to sometimes put
> > ccache stuff in /root/.ccache, but they haven't added any code to clean
> > it up.
> 
> catalyst puts the ccache dir inside /var in the chroot, and sets the
> CCACHE_DIR environment variable accordingly.

No, please read the code:

modules/targets.py:                             ccdir="/root/.ccache"

This directory is being created with cache stuff in it and is not being
cleaned up.

> Again, please don't make direct commits to the catalyst code. Thanks.

Then please respond to my pings on irc or my emails for once.

-- 
rob holland - [ tigger@gentoo.org ]
irc://irc.freenode.net/#gentoo as tigger^
http://dev.gentoo.org/~tigger/tigger@gentoo.org.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] catalyst livecd improvements

[Daniel Robbins]

On Fri, 2004-02-13 at 03:38, tigger@gentoo.org wrote:
> > Again, please don't make direct commits to the catalyst code. Thanks.
> 
> Then please respond to my pings on irc or my emails for once.

I'm very sorry that I did not reply to the one private email that you
sent me. Sometimes, emails I receive do slip through the cracks.
In the future, if you can't reach me, email zhen or post to the -releng
list with any issues relating to catalyst. 

There are people that we allow to make commits directly to catalyst
code. In general, they are listed on the catalyst project page
(http://www.gentoo.org/proj/en/releng/catalyst/). We also allow arch
leads to make changes to the settings in their respective arch plugin,
and beejay has made some commits to non-arch parts of the code. The
difference here is that I am working with beejay to debug certain parts
of the code (he's also the acting x86 arch lead.)

For any kind of development, it's important that there is at least one
person who has a full understanding of what changes are being made to
the code. That person is me, with Zhen as my assistant.

I understand that you had only the best intentions for making your
commits, and they were fairly minor, but I just want to be clear about
our policy. Also, I know that we have a very unstructured style of
development on this project, so I can certainly understand the confusion
that this can cause. To help remedy this, I've started to document
Gentoo development standards here on cvs:

gentoo/xml/htdocs/proj/en/lead/dev-project-stds.xml

(It's plain ASCII currently, not XML yet but the writing has been
through several drafts. What is there is fairly refined; I'll be adding
several more chapters before putting it online.)

I think that reading this document will help you understand where I'm
coming from. For catalyst, we are trying to use the "surgical team"
model of development and ensure that we preserve catalyst's conceptual
integrity by making sure that all code gets reviewed by me or Zhen
before hitting the tree.

Also, my initial reply to you *was* very snippy; I apologize for that. I
had a *very* stressful day yesterday (kids sick) and it came across in
my email. Please forgive me for that.

Regards,

Daniel


--
gentoo-releng@gentoo.org mailing list

[gentoo-releng] catalyst bugs?

[Kumba]

Not sure if I've hit two bugs here or not (I suspect they're bugs).

First one; I'm giving a shot at mips grp builds with a limited set of 
packages, so I follow the guide and do something similar to this in my 
grp spec file:

grp: cd1

grp/cd1/type: pkgset
grp/cd1/packages:

I specify my packages, and save the file, exit, and attempt to invoke 
catalyst.  Then I get this:

# ./catalyst -f mips4-grp-20040210.spec
Setting storedir to config file value "/usr/catalyst"
Setting portdir to default value "/usr/portage"
Setting distdir to config file value "/usr/portage/distfiles"
Setting options to config file value "pkgcache ccache"
Setting sharedir to config file value "/usr/catalyst"
Compiler cache support enabled.
Package cache support enabled.

catalyst: Required argument "grp/c/packages" not specified.


It looks like to me that the parsing code is not separating on 
spaces/whitespace, but is instead grabbing only the first character.


Second Bug;  Skipping around the first bug above by changing all 
instances of "cd1" to just "1", I was able to continue onwards -- a 
little.  GRP started to compile a few things, it got as far as merging 
portage in the chroot and then died with this:

 >>> Auto-cleaning packages ...

 >>> No outdated packages were found on your system.


  * GNU info directory index is up-to-date.
/bin/bash: line 46: unexpected EOF while looking for matching `"'
/bin/bash: line 49: syntax error: unexpected end of file

catalyst: GRP build aborting due to error.


I suspect that catalyst invoked an external bash script for this stage 
of grp building, but I do not know the internals of catalyst too well. 
I suspect this external bash script has issues that need to be addressed.



--Kumba

-- 
"Such is oft the course of deeds that move the wheels of the world: 
small hands do them because they must, while the eyes of the great are 
elsewhere."  --Elrond

--
gentoo-releng@gentoo.org mailing list