[gentoo-catalyst] small distros - setups and practices?

[gentoo-catalyst] small distros - setups and practices?

[Max Arnold]

Don't sure if this is right place to post, but since I'm experimenting with catalyst
maybe someone will share some thoughts and suggests additional reading.

What I'm trying to do:

1. Create small, task-oriented stage4 for fixed hardware configuration
2. Deploy it to several machines (now looking for deployment variants and installers)
3. Maintain it (updates)

The most interesting and challenging part is (3). There are my thoughts and 
associated questions:

I want special build host, where all binary targets are built. Its setup
should be easily repeatable by other developers (i.e. every task should be scripted,
without manual tweaks like 'chroot here and patch/emerge this'). Catalyst with
specs and overlays seems fine to me.

- Which parts of host system affect on resulting target?

- I probably do not want update portage tree and world on build host very often,
  but want newer snapshots and seed stages to build targets. Is it possible?

I want incremental selective binary updates to be pushed on nodes. I.e. generate
stage4 once, and push only changes for a long period of time. Then repeat.

- Is it possible with catalyst to produce icremental binary updates (i.e. feed it with
  new snapshot and take set of updated packages as a result)? To me seems it is not.

- Is portage tree required on nodes in order to install binary packages (never used them)?

- My first plan is to unpack stage4, put fresh tree snapshot, clean PKGDIR and update world.
  Then push everything from PKGDIR to nodes (still don't know how to manage configs).

- What about sequential stage4 generation without purging cache?

- Get rid of portage on nodes and push binary diffs (or rsync).

And small questions about catalyst internals. Is it possible to use some functions manually?
For example, I want to mount-bind required parts (/proc, /dev, snapshot, distfiles...),
chroot to storedir/tmp/default/stageX, do or test something manually and then exit and unmount.
Is there are any handy scripts to do this?  Manual catalyst execution step-by-step?
Also it will be good to just look on emerge -pv output, or list URLs which catalyst
tries to fetch (for manual fetching and offline operations).


Any thoughts?  How do you guys organize things like this?
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] small distros - setups and practices?

[Chris Gianelloni]

On Tue, 2008-04-22 at 20:18 +0800, Max Arnold wrote:
> 1. Create small, task-oriented stage4 for fixed hardware configuration
> 2. Deploy it to several machines (now looking for deployment variants and installers)
> 3. Maintain it (updates)

This is rather funny, since I'm adding support for exactly this stuff to
catalyst (and via other tools which will utilize catalyst).

> I want special build host, where all binary targets are built. Its setup
> should be easily repeatable by other developers (i.e. every task should be scripted,
> without manual tweaks like 'chroot here and patch/emerge this'). Catalyst with
> specs and overlays seems fine to me.

Sure.  In fact, I have 2 different "host types" right now, a "master"
and a "build host" though they can reside on the same machine.

> - Which parts of host system affect on resulting target?

None, if you build *all* of the stages, in order.

> - I probably do not want update portage tree and world on build host very often,
>   but want newer snapshots and seed stages to build targets. Is it possible?

Set portdir in catalyst.conf to anything that you want.

> I want incremental selective binary updates to be pushed on nodes. I.e. generate
> stage4 once, and push only changes for a long period of time. Then repeat.
> 
> - Is it possible with catalyst to produce icremental binary updates (i.e. feed it with
>   new snapshot and take set of updated packages as a result)? To me seems it is not.

Huh?  Have you ever looked at the package cache?

ls -l /var/tmp/catalyst/packages/stage4-i686-custom/All > ~/before.txt
catalyst -f ~/stage4-i686-custom.spec
ls -l /var/tmp/catalyst/packages/stage4-i686-custom/All > ~/after.txt
diff -u ~/before.txt ~/after.txt

> - Is portage tree required on nodes in order to install binary packages (never used them)?

Yes and no.  You don't need a complete tree, but you do need the
profiles.

> - What about sequential stage4 generation without purging cache?

What about it?

> And small questions about catalyst internals. Is it possible to use some functions manually?
> For example, I want to mount-bind required parts (/proc, /dev, snapshot, distfiles...),
> chroot to storedir/tmp/default/stageX, do or test something manually and then exit and unmount.
> Is there are any handy scripts to do this?  Manual catalyst execution step-by-step?

Feel free to write one.  You can't really use just parts of catalyst.

> Also it will be good to just look on emerge -pv output, or list URLs which catalyst
> tries to fetch (for manual fetching and offline operations).

Well, I think that you need to spend a bit more time figuring out what
catalyst can already do before asking for additional support.  Much of
what you've asked here, catalyst already does.  ;]

> Any thoughts?  How do you guys organize things like this?

There's this thing called Gentoo.... ;]

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] small distros - setups and practices?

[Max Arnold]

On Tue, Apr 22, 2008 at 09:58:57AM -0700, Chris Gianelloni wrote:
> On Tue, 2008-04-22 at 20:18 +0800, Max Arnold wrote:
> > 1. Create small, task-oriented stage4 for fixed hardware configuration
> > 2. Deploy it to several machines (now looking for deployment variants and installers)
> > 3. Maintain it (updates)
> 
> This is rather funny, since I'm adding support for exactly this stuff to
> catalyst (and via other tools which will utilize catalyst).

Yes, synchronicity exists :)  Maybe I will be an early adopter?

> > I want special build host, where all binary targets are built. Its setup
> > should be easily repeatable by other developers (i.e. every task should be scripted,
> > without manual tweaks like 'chroot here and patch/emerge this'). Catalyst with
> > specs and overlays seems fine to me.
> 
> Sure.  In fact, I have 2 different "host types" right now, a "master"
> and a "build host" though they can reside on the same machine.

Which roles they have?

> > - Is it possible with catalyst to produce icremental binary updates (i.e. feed it with
> >   new snapshot and take set of updated packages as a result)? To me seems it is not.
> 
> Huh?  Have you ever looked at the package cache?
> 
> ls -l /var/tmp/catalyst/packages/stage4-i686-custom/All > ~/before.txt
> catalyst -f ~/stage4-i686-custom.spec
> ls -l /var/tmp/catalyst/packages/stage4-i686-custom/All > ~/after.txt
> diff -u ~/before.txt ~/after.txt
> 
> > - What about sequential stage4 generation without purging cache?
> 
> What about it?

You described my thoughts more clear (package cache, before.txt and after.txt)
My doubts was about using grp target.

> Well, I think that you need to spend a bit more time figuring out what
> catalyst can already do before asking for additional support.  Much of
> what you've asked here, catalyst already does.  ;]

Yes, that is what I'm currently doing :)
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] small distros - setups and practices?

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

On Wed, 2008-04-23 at 10:03 +0800, Max Arnold wrote:
> > > I want special build host, where all binary targets are built. Its setup
> > > should be easily repeatable by other developers (i.e. every task should be scripted,
> > > without manual tweaks like 'chroot here and patch/emerge this'). Catalyst with
> > > specs and overlays seems fine to me.
> > 
> > Sure.  In fact, I have 2 different "host types" right now, a "master"
> > and a "build host" though they can reside on the same machine.
> 
> Which roles they have?

I'm not sure that I follow.  Are you asking what specific tasks would be
performed by the roles above?

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-catalyst] small distros - setups and practices?

[Max Arnold]

On Thu, May 01, 2008 at 10:04:26AM -0700, Chris Gianelloni wrote:
> On Wed, 2008-04-23 at 10:03 +0800, Max Arnold wrote:
> > > > I want special build host, where all binary targets are built. Its setup
> > > > should be easily repeatable by other developers (i.e. every task should be scripted,
> > > > without manual tweaks like 'chroot here and patch/emerge this'). Catalyst with
> > > > specs and overlays seems fine to me.
> > > 
> > > Sure.  In fact, I have 2 different "host types" right now, a "master"
> > > and a "build host" though they can reside on the same machine.
> > 
> > Which roles they have?
> 
> I'm not sure that I follow.  Are you asking what specific tasks would be
> performed by the roles above?

Yes, what specific tasks would be performed by these host types and
what are benefits of such distinction?

I guess a "master" purpose is to serve updates, or things are more tricky?
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] small distros - setups and practices?

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On Fri, 2008-05-02 at 10:20 +0800, Max Arnold wrote:
> > > > Sure.  In fact, I have 2 different "host types" right now, a "master"
> > > > and a "build host" though they can reside on the same machine.
> > > 
> > > Which roles they have?
> > 
> > I'm not sure that I follow.  Are you asking what specific tasks would be
> > performed by the roles above?
> 
> Yes, what specific tasks would be performed by these host types and
> what are benefits of such distinction?
> 
> I guess a "master" purpose is to serve updates, or things are more tricky?

Yeah, you're thinking way too small.  ;]

The master server will be the control interface for custom profiles and
binary repositories, which are built by the build servers.  It will
(eventually) also be the configuration engine for each server.  I hope
to eventually have a complete system capable of controlling a Gentoo
system from birth to death, from a centralized location.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-catalyst] small distros - setups and practices?

[Max Arnold]

On Fri, May 02, 2008 at 08:03:47AM -0700, Chris Gianelloni wrote:
> On Fri, 2008-05-02 at 10:20 +0800, Max Arnold wrote:
> > > > > Sure.  In fact, I have 2 different "host types" right now, a "master"
> > > > > and a "build host" though they can reside on the same machine.
> > > > 
> > > > Which roles they have?
> > > 
> > > I'm not sure that I follow.  Are you asking what specific tasks would be
> > > performed by the roles above?
> > 
> > Yes, what specific tasks would be performed by these host types and
> > what are benefits of such distinction?
> > 
> > I guess a "master" purpose is to serve updates, or things are more tricky?
> 
> Yeah, you're thinking way too small.  ;]
> 
> The master server will be the control interface for custom profiles and
> binary repositories, which are built by the build servers.  It will
> (eventually) also be the configuration engine for each server.  I hope
> to eventually have a complete system capable of controlling a Gentoo
> system from birth to death, from a centralized location.

Seems like a powerful addition to Gentoo infrastructure.  Good luck in your work!
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] small distros - setups and practices?

[Robert Šmol]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]

Will take chance for question here.

We ran into similar scenario as well when we designed virtualized Gentoo
infrastructure and we found out that emerge/portage has not complete (read
perfect) support for binaries (i.e. different USE flags for different
machines).

Can you explain more about your project? It seems it really fits our needs.
Some of our thoughts were to have this builder machine available as XEN
domU, it would build the packages in chroot and then serve them to machines
url  as:

PORTAGE_BINHOST="http://packages.builder.domain/<machine-type>/"

Some other crazy ideas involved some extension to emerge/portage like:
Slave - hey, this is my /var/lib/portage/world, my USE flags
(/etc/make.conf) and I want such and such package
Master - calculating a bit and providing Slave with packages

(but this is overkill)

At the moment I have working prototype which is fine because machines are
updated and it is doable, at the same moment it needs a lot of love because
it is just a dirty hack (like scripts to easily start chroot, config apache,
etc...).

I do not know catalyst much, all I know it is possible to do stage4 with
this (i.e. build up2date XEN domu machine). I appreciate any hints on how to
incorporate this into something slightly bigger.

I am looking for anyone interested in this, lets join effort guys.

Aloha,

Robert

[-- Attachment #2: Type: text/html, Size: 1500 bytes --]

Re: [gentoo-catalyst] small distros - setups and practices?

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

On Mon, 2008-05-05 at 20:02 +0200, Robert Šmol wrote:
> Can you explain more about your project?

Not really... it's something under development and quite a ways away
from seeing the light of day.

> PORTAGE_BINHOST="http://packages.builder.domain/<machine-type>/"

Well, I'm using catalyst to do the building.

> Slave - hey, this is my /var/lib/portage/world, my USE flags
> (/etc/make.conf) and I want such and such package

Wow!  That sure sounds a lot like what catalyst does.  If only we had a
way to specify settings for packages and such... something like a
profile.  Maybe even if we had a file we could give catalyst, some kind
of specification... Oh wait... *grin*

> Master - calculating a bit and providing Slave with packages

catalyst -f packages.spec && rsync --archive
--delete-after /var/tmp/catalyst/packages/$rel_type/$target-$subarch-$rel_ver/* master:/exports/packages/$target/$subarch/$rel_ver
> 
> (but this is overkill)

...or is it?

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-catalyst] small distros - setups and practices?

[Ramon van Alteren]

On May 2, 2008, at 5:03 PM, Chris Gianelloni wrote:

> On Fri, 2008-05-02 at 10:20 +0800, Max Arnold wrote:
>>>>> Sure.  In fact, I have 2 different "host types" right now, a  
>>>>> "master"
>>>>> and a "build host" though they can reside on the same machine.
>>>>
>>>> Which roles they have?
>>>
>>> I'm not sure that I follow.  Are you asking what specific tasks  
>>> would be
>>> performed by the roles above?
>>
>> Yes, what specific tasks would be performed by these host types and
>> what are benefits of such distinction?
>>
>> I guess a "master" purpose is to serve updates, or things are more  
>> tricky?
>
> Yeah, you're thinking way too small.  ;]
>
> The master server will be the control interface for custom profiles  
> and
> binary repositories, which are built by the build servers.  It will
> (eventually) also be the configuration engine for each server.  I hope
> to eventually have a complete system capable of controlling a Gentoo
> system from birth to death, from a centralized location.

Is there any public repository where you are developing this stuff in ?
I'd like to contribute if possible.

Additionally I'd be interested in your ideas on managing the  
configuration from this master host....
We're currently using puppet for that which works fine for us, curious  
why you'd want to re-implement either puppet of cf-engine.

Ramon
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] small distros - setups and practices?

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On Mon, 2008-05-05 at 21:41 +0200, Ramon van Alteren wrote:
> Is there any public repository where you are developing this stuff
> in ?

Nope.

> curious  
> why you'd want to re-implement either puppet of cf-engine.

I never said anything about re-implementing anything.  I plan on
leveraging existing technologies, where possible.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-catalyst] small distros - setups and practices?

[Ramon van Alteren]

On May 7, 2008, at 8:46 AM, Chris Gianelloni wrote:

> On Mon, 2008-05-05 at 21:41 +0200, Ramon van Alteren wrote:
>> Is there any public repository where you are developing this stuff
>> in ?
>
> Nope.

Too bad :-(
I'd be nice if there was a possibility to contribute......

Ramon
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] small distros - setups and practices?

[John Eckhart]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

Chris,

   The work you mention for catalyst has been something I've been very
interested in leveraging for a while now (read years). If you want any
contribution help, I'ld be interested in helping out.

   On a side note, I've been using a locally mounted squashfs overlayed with
a bare writable fs (JFS in this case, but it could be any) via aufs to track
my portage trees for a while. It lets me compress portage from 600+ mb down
to about 50 mb AND speeds up portage access by an order of magnitude. This
might be an interesting method to distribute/deploy the tree or even simply
track it within catalyst as catalyst builds the updates. Let me know if this
might be of interest to you.

John

On Fri, May 9, 2008 at 6:02 AM, Ramon van Alteren <ramon@vanalteren.nl>
wrote:

>
> On May 7, 2008, at 8:46 AM, Chris Gianelloni wrote:
>
>  On Mon, 2008-05-05 at 21:41 +0200, Ramon van Alteren wrote:
>>
>>> Is there any public repository where you are developing this stuff
>>> in ?
>>>
>>
>> Nope.
>>
>
> Too bad :-(
> I'd be nice if there was a possibility to contribute......
>
>
> Ramon
> --
> gentoo-catalyst@lists.gentoo.org mailing list
>
>

[-- Attachment #2: Type: text/html, Size: 1909 bytes --]