[gentoo-catalyst] [PATCH 0/4] HOWTO to get newbies started

[gentoo-catalyst] [PATCH 0/4] HOWTO to get newbies started

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

Here are my notes from figuring out how catalyst works, after browsing
around online and getting help from y'all.  On IRC this morning, Chris
White said he had some similar ideas floating around in his head.  I
haven't filed a bug with the HOWTO [1], but I can if anyone wants to
discuss it that way.

[1]: http://www.gentoo.org/proj/en/releng/catalyst/faq.xml#howto

W. Trevor King (4):
  doc/HOWTO: First pass at a gentle Catalyst introduction
  Makefile: Add DOCS for building files/HOWTO.html
  files/.gitignore: Ignore $(DOCS) and $(DOC_SIDE_EFFECTS)
  doc/HOWTO.txt: Explain how to run catalyst without installing

 Makefile         |  15 +++-
 doc/HOWTO.txt    | 219 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 files/.gitignore |   3 +-
 3 files changed, 233 insertions(+), 4 deletions(-)
 create mode 100644 doc/HOWTO.txt

-- 
1.8.1.336.g94702dd

[gentoo-catalyst] [PATCH 1/4] doc/HOWTO: First pass at a gentle Catalyst introduction

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

---
 doc/HOWTO.txt | 204 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)
 create mode 100644 doc/HOWTO.txt

diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt
new file mode 100644
index 0000000..ea9c8f4
--- /dev/null
+++ b/doc/HOWTO.txt
@@ -0,0 +1,204 @@
+Catalyst is a release-buildcing tool for Gentoo.  If you use Gentoo
+and want to roll your own live CD or bootable USB stick, this is the
+way to go.  First, get a Gentoo development box and install the
+necessary tools:
+
+    # emerge -av dev-util/catalyst
+
+Configure catalyst by editing `/etc/catalyst/catalyst.conf`, which is
+well commented.  This sets up defaults such as hashing algorithms and
+storage directories.  The defaults will probably be fine unless disk
+space is an issue.
+
+Assembling a starting point
+---------------------------
+
+Portage snapshot
+~~~~~~~~~~~~~~~~
+
+Create a snapshot of your current Portage tree (you may want to
+`emerge --sync` first):
+
+    # catalyst --snapshot 20130131
+    # ls /var/tmp/catalyst/snapshots/
+    portage-20130131.tar.bz2
+    portage-20130131.tar.bz2.CONTENTS
+    portage-20130131.tar.bz2.DIGESTS
+
+where the storage location is relative to the default
+`$storedir=/var/tmp/catalyst`.
+
+Stage3 tarball
+~~~~~~~~~~~~~~
+
+Get a stage3 tarball (containing the build tools you'll need to
+construct your stage1) from your local
+http://www.gentoo.org/main/en/mirrors.xml[Gentoo mirror].
+
+    $GENTOO_MIRROR/releases/$ARCH/current-stage3/
+
+For example,
+
+    http://distfiles.gentoo.org/releases/amd64/current-stage3/stage3-amd64-20121213.tar.bz2
+
+Grab the tarball and put it where catalyst will find it:
+
+    # wget http://…/stage3-amd64-20121213.tar.bz2
+    # wget http://…/stage3-amd64-20121213.tar.bz2.CONTENTS
+    # wget http://…/stage3-amd64-20121213.tar.bz2.DIGESTS.asc
+    # sha512sum -c stage3-amd64-20121213.tar.bz2.DIGESTS.asc
+    # gpg --verify stage3-amd64-20121213.tar.bz2.DIGESTS.asc
+    # mv stage3-amd64-20121213.tar.bz2* /var/tmp/catalyst/builds/default/
+
+where the storage dir is `$storedir/builds/$source_subpath`
+(`$storedir` from `catalyst.conf`, `$source_subpath` from your
+`*.spec` file).
+
+`.*spec` files
+~~~~~~~~~~~~~~
+
+`.*spec` files tell catalyst about the system you're trying to build.
+There are a number of examples distributed with catalyst.  Look in
+`/usr/share/doc/catalyst-*/examples/`.  A minimal `*.spec` file for
+this example is:
+
+    # cat default-stage1-amd64-2013.1.spec
+    subarch: amd64
+    version_stamp: 2013.1
+    target: stage1
+    rel_type: default
+    profile: default/linux/amd64/10.0/no-multilib
+    snapshot: 20130131
+    source_subpath: default/stage3-amd64-20121213
+
+You may need to adjust the `subarch`, `snapshot`, and `source_subpath`
+fields of the `*.spec` to match your target host, Portage snapshot,
+and stage3 tarball name respectively.
+
+For more details on what can go into a `*.spec` file, see
+`catalyst-spec(5)`.
+
+Building stage1
+---------------
+
+Now that everything's setup, run catalyst:
+
+    # catalyst -f default-stage1-amd64-2013.1.spec
+
+which will build the target and install something like:
+
+    # ls /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.*
+    /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2
+    /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.CONTENTS
+    /var/tmp/catalyst/builds/default/stage1-amd64-2013.1.tar.bz2.DIGESTS
+
+The name is an expansion of
+`$storedir/builds/$rel_type/$target-$subarch-$version_stamp…`.
+
+Building stage2 and stage3
+--------------------------
+
+Once you've built the stage1 from your seed stage3, you can use that
+stage1 to build a stage2 and stage3.  The `*.spec` files are similar:
+
+    $ diff -u default-stage{1,2}-amd64-2013.1.spec
+    --- default-stage1-amd64-2013.1.spec
+    +++ default-stage2-amd64-2013.1.spec
+    @@ -1,7 +1,7 @@
+     subarch: amd64
+     version_stamp: 2013.1
+    -target: stage1
+    +target: stage2
+     rel_type: default
+     profile: default/linux/amd64/10.0/no-multilib
+     snapshot: 20130131
+    -source_subpath: default/stage3-amd64-20121213
+    +source_subpath: default/stage1-amd64-2013.1
+    $ diff default-stage{2,3}-amd64-2013.1.spec
+    --- default-stage2-amd64-2013.1.spec
+    +++ default-stage3-amd64-2013.1.spec
+    @@ -1,7 +1,7 @@
+     subarch: amd64
+     version_stamp: 2013.1
+    -target: stage2
+    +target: stage3
+     rel_type: default
+     profile: default/linux/amd64/10.0/no-multilib
+     snapshot: 20130131
+    -source_subpath: default/stage1-amd64-2013.1
+    +source_subpath: default/stage2-amd64-2013.1
+
+Gentoo stages
+-------------
+
+You can't compile a big pile of source code without an already
+compiled toolchain, which is where Gentoo's stages come in.  The “base
+system” contains the necessary build tools and supporting
+infrastructure to get things going.  The stages are:
+
+1. System must be bootstrapped and the base system must be compiled
+   (a new toolchain built with external seed tools).
+2. Stage1 + bootstrapped (a new toolchain build with stage1 tools).
+3. Stage2 + base system compiled (the base system built with stage2 tools).
+4. Stage3 + non-base packages.
+
+For more details on the differences between the stages, look at the
+target helper scripts (e.g. `targets/stage1/*.sh`).
+
+Building with a kernel
+----------------------
+
+If you're shooting for a live CD or bootable USB stick, you'll need to
+compile your own kernel.  Here's how that works.
+
+Genkernel
+~~~~~~~~~
+
+When you don't know exactly which kernel options you need, add
+something like the following to your `*.spec`:
+
+    boot/kernel: gentoo
+    boot/kernel/gentoo/sources: gentoo-sources
+
+You can still set `boot/kernel/<label>/config` when you're using
+genkernel if you want to give genkernel some hints.
+
+Genkernel alternatives
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you don't want to use a genkernel, your options are fairly limited.
+The currently suggested route is to create your own binary kernel
+package.
+
+Stage4
+------
+
+`examples/stage4_template.spec` is a good template for building a
+stage4 tarball.  Besides setting `target: stage4` and adjusting
+`source_subpath`, I usually use `stage4/packages`, `stage4/rcadd`, and
+the `boot/kernel` stuff described above.  This gives an almost
+bootable stage that you can dump on a USB flash drive.
+
+Live CDs
+--------
+
+Live CDs should be built in two stages: `livecd-stage1` (based on a
+stage3 source) for building extra packages (along the same lines as a
+stage4) and `livecd-stage2` (based on `livecd-stage1`) for setting up
+the kernel, bootloader, filesystem, and other details.  See
+`examples/livecd-stage*_template.spec` for some ideas.
+
+Live USBs
+---------
+
+The easiest way to create a live USB is currently to install a live CD
+ISO using
+http://www.syslinux.org/wiki/index.php/Doc/isolinux#HYBRID_CD-ROM.2FHARD_DISK_MODE[isohybrid]
+and `dd`:
+
+    # isohybrid filename.iso
+    # dd if=filename.iso of=/dev/sdX
+
+replacing `X` with the appropriate drive letter for your USB disk.
+See https://bugs.gentoo.org/show_bug.cgi?id=251719[bug 251719] for
+details.
-- 
1.8.1.336.g94702dd

[gentoo-catalyst] [PATCH 2/4] Makefile: Add DOCS for building files/HOWTO.html

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

The rules are more general though; any new *.txt file in doc/ that
doesn't match *.?.txt or one of the explicitly-listed *.generated.txt
files will be automatically built and distributed.

The generated `files/docbook-xsl.css` is a side effect of the xhtml
target.
---
 Makefile | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 016120a..98accbe 100644
--- a/Makefile
+++ b/Makefile
@@ -5,15 +5,18 @@ PACKAGE_VERSION = `fgrep '__version__=' catalyst | sed 's|^__version__="\(.*\)"$
 MAN_PAGE_SOURCES = $(wildcard doc/*.?.txt)
 MAN_PAGES = $(patsubst doc/%.txt,files/%,$(MAN_PAGE_SOURCES))
 MAN_PAGE_INCLUDES = doc/subarches.generated.txt doc/targets.generated.txt
-EXTRA_DIST = $(MAN_PAGES)
-GENERATED_FILES = $(MAN_PAGES) $(MAN_PAGE_INCLUDES)
+DOC_SOURCES = $(filter-out $(MAN_PAGE_SOURCES) $(MAN_PAGE_INCLUDES),$(wildcard doc/*.txt))
+DOCS = $(patsubst doc/%.txt,files/%.html,$(DOC_SOURCES))
+DOC_SIDE_EFFECTS  = files/docbook-xsl.css
+EXTRA_DIST = $(MAN_PAGES) $(DOCS) $(DOC_SIDE_EFFECTS)
+GENERATED_FILES = $(MAN_PAGES) $(MAN_PAGE_INCLUDES) $(DOCS) $(DOC_SIDE_EFFECTS)
 
 distdir = catalyst-$(PACKAGE_VERSION)
 
 
 all: $(EXTRA_DIST)
 
-files/%: doc/%.txt doc/asciidoc.conf Makefile catalyst
+$(MAN_PAGES): files/%: doc/%.txt doc/asciidoc.conf Makefile catalyst
 	a2x --conf-file=doc/asciidoc.conf --attribute="catalystversion=$(PACKAGE_VERSION)" \
 		 --format=manpage -D files "$<"
 
@@ -27,6 +30,12 @@ doc/subarches.generated.txt: $(wildcard arch/*.py) doc/make_subarch_table_guidex
 doc/targets.generated.txt: doc/make_target_table.py $(wildcard modules/*_target.py)
 	"./$<" > "$@"
 
+$(DOCS): files/%.html: doc/%.txt doc/asciidoc.conf Makefile
+	a2x --conf-file=doc/asciidoc.conf --attribute="catalystversion=$(PACKAGE_VERSION)" \
+		 --format=xhtml -D files "$<"
+
+$(DOC_SIDE_EFFECTS): $(firstword $(DOCS))
+
 clean:
 	rm -f $(GENERATED_FILES)
 	find -name '*.pyo' -delete
-- 
1.8.1.336.g94702dd

[gentoo-catalyst] [PATCH 3/4] files/.gitignore: Ignore $(DOCS) and $(DOC_SIDE_EFFECTS)

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

These are generated by the Makefile from sources in doc/.
---
 files/.gitignore | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/files/.gitignore b/files/.gitignore
index 4a4f038..9dbf1f1 100644
--- a/files/.gitignore
+++ b/files/.gitignore
@@ -1,3 +1,4 @@
 catalyst.1
 catalyst-spec.5
-
+*.html
+*.css
-- 
1.8.1.336.g94702dd

[gentoo-catalyst] [PATCH 4/4] doc/HOWTO.txt: Explain how to run catalyst without installing

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

Thanks to Chris White for explaining this to me.
---
 doc/HOWTO.txt | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt
index ea9c8f4..5717f80 100644
--- a/doc/HOWTO.txt
+++ b/doc/HOWTO.txt
@@ -202,3 +202,18 @@ and `dd`:
 replacing `X` with the appropriate drive letter for your USB disk.
 See https://bugs.gentoo.org/show_bug.cgi?id=251719[bug 251719] for
 details.
+
+Running catalyst from a Git checkout
+------------------------------------
+
+If you're developing catalyst, you'll want to test your altered
+version.  An easy way to run it without reinstalling is to setup a
+local configuration file and run:
+
+    # ./catalyst -c catalyst.conf -f path/to/your.spec
+
+The local configuration file can use all the defaults except for
+`sharedir`, which you should change to point to your development
+directory:
+
+    sharedir="/home/wking/src/catalyst"
-- 
1.8.1.336.g94702dd

[gentoo-catalyst] Re: [PATCH 0/4] HOWTO to get newbies started

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

On Sat, Feb 09, 2013 at 03:40:29PM -0500, W. Trevor King wrote:
> Here are my notes from figuring out how catalyst works, after browsing
> around online and getting help from y'all.  On IRC this morning, Chris
> White said he had some similar ideas floating around in his head.  I
> haven't filed a bug with the HOWTO [1], but I can if anyone wants to
> discuss it that way.

Thoughts?  Anyone?  I can also show up on IRC…

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Re: [PATCH 0/4] HOWTO to get newbies started

[Matt Turner]

On Thu, Feb 21, 2013 at 12:35 PM, W. Trevor King <wking@tremily.us> wrote:
> On Sat, Feb 09, 2013 at 03:40:29PM -0500, W. Trevor King wrote:
>> Here are my notes from figuring out how catalyst works, after browsing
>> around online and getting help from y'all.  On IRC this morning, Chris
>> White said he had some similar ideas floating around in his head.  I
>> haven't filed a bug with the HOWTO [1], but I can if anyone wants to
>> discuss it that way.
>
> Thoughts?  Anyone?  I can also show up on IRC…

I like all the patches. I was hoping someone else would review them
before me, but I think I'll just handle them. Hopefully tonight.

Thanks a bunch for the patches. Sorry about the lack of response.
Please don't take that as a lack of interest or appreciation. :)

Matt

Re: [gentoo-catalyst] Re: [PATCH 0/4] HOWTO to get newbies started

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 408 bytes --]

On Thu, Feb 21, 2013 at 01:47:45PM -0800, Matt Turner wrote:
> Thanks a bunch for the patches. Sorry about the lack of response.
> Please don't take that as a lack of interest or appreciation. :)

No problem.  I understand that folks are busy…

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Re: [PATCH 0/4] HOWTO to get newbies started

[Matt Turner]

On Thu, Feb 21, 2013 at 1:58 PM, W. Trevor King <wking@tremily.us> wrote:
> On Thu, Feb 21, 2013 at 01:47:45PM -0800, Matt Turner wrote:
>> Thanks a bunch for the patches. Sorry about the lack of response.
>> Please don't take that as a lack of interest or appreciation. :)
>
> No problem.  I understand that folks are busy…
>
> --
> This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
> For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

First six patches are pushed!

Re: [gentoo-catalyst] [PATCH 1/4] doc/HOWTO: First pass at a gentle Catalyst introduction

[Matt Turner]

On Sat, Feb 9, 2013 at 12:40 PM, W. Trevor King <wking@tremily.us> wrote:
> From: "W. Trevor King" <wking@tremily.us>
>
> ---
>  doc/HOWTO.txt | 204 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 204 insertions(+)
>  create mode 100644 doc/HOWTO.txt
>
> diff --git a/doc/HOWTO.txt b/doc/HOWTO.txt
> new file mode 100644
> index 0000000..ea9c8f4
> --- /dev/null
> +++ b/doc/HOWTO.txt
> @@ -0,0 +1,204 @@
> +Catalyst is a release-buildcing tool for Gentoo.  If you use Gentoo
                              ^
Typo. I'll fix it up before I push the series.

Re: [gentoo-catalyst] [PATCH 2/4] Makefile: Add DOCS for building files/HOWTO.html

[Matt Turner]

On Sat, Feb 9, 2013 at 12:40 PM, W. Trevor King <wking@tremily.us> wrote:
> From: "W. Trevor King" <wking@tremily.us>
>
> The rules are more general though; any new *.txt file in doc/ that
> doesn't match *.?.txt or one of the explicitly-listed *.generated.txt
> files will be automatically built and distributed.
>
> The generated `files/docbook-xsl.css` is a side effect of the xhtml
> target.
> ---
>  Makefile | 15 ++++++++++++---
>  1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 016120a..98accbe 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -5,15 +5,18 @@ PACKAGE_VERSION = `fgrep '__version__=' catalyst | sed 's|^__version__="\(.*\)"$
>  MAN_PAGE_SOURCES = $(wildcard doc/*.?.txt)
>  MAN_PAGES = $(patsubst doc/%.txt,files/%,$(MAN_PAGE_SOURCES))
>  MAN_PAGE_INCLUDES = doc/subarches.generated.txt doc/targets.generated.txt
> -EXTRA_DIST = $(MAN_PAGES)
> -GENERATED_FILES = $(MAN_PAGES) $(MAN_PAGE_INCLUDES)
> +DOC_SOURCES = $(filter-out $(MAN_PAGE_SOURCES) $(MAN_PAGE_INCLUDES),$(wildcard doc/*.txt))
> +DOCS = $(patsubst doc/%.txt,files/%.html,$(DOC_SOURCES))
> +DOC_SIDE_EFFECTS  = files/docbook-xsl.css
> +EXTRA_DIST = $(MAN_PAGES) $(DOCS) $(DOC_SIDE_EFFECTS)
> +GENERATED_FILES = $(MAN_PAGES) $(MAN_PAGE_INCLUDES) $(DOCS) $(DOC_SIDE_EFFECTS)
>
>  distdir = catalyst-$(PACKAGE_VERSION)
>
>
>  all: $(EXTRA_DIST)
>
> -files/%: doc/%.txt doc/asciidoc.conf Makefile catalyst
> +$(MAN_PAGES): files/%: doc/%.txt doc/asciidoc.conf Makefile catalyst
>         a2x --conf-file=doc/asciidoc.conf --attribute="catalystversion=$(PACKAGE_VERSION)" \
>                  --format=manpage -D files "$<"
>
> @@ -27,6 +30,12 @@ doc/subarches.generated.txt: $(wildcard arch/*.py) doc/make_subarch_table_guidex
>  doc/targets.generated.txt: doc/make_target_table.py $(wildcard modules/*_target.py)
>         "./$<" > "$@"
>
> +$(DOCS): files/%.html: doc/%.txt doc/asciidoc.conf Makefile
> +       a2x --conf-file=doc/asciidoc.conf --attribute="catalystversion=$(PACKAGE_VERSION)" \
> +                --format=xhtml -D files "$<"
> +
> +$(DOC_SIDE_EFFECTS): $(firstword $(DOCS))
> +
>  clean:
>         rm -f $(GENERATED_FILES)
>         find -name '*.pyo' -delete
> --
> 1.8.1.336.g94702dd

What's the rationale behind generating HTML instead of man pages like
the other docs?

Re: [gentoo-catalyst] [PATCH 2/4] Makefile: Add DOCS for building files/HOWTO.html

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

On Fri, Feb 22, 2013 at 06:04:49PM -0800, Matt Turner wrote:
> What's the rationale behind generating HTML instead of man pages like
> the other docs?

HTML is easy to post on the web so folks can see how to use Catalyst
before installing it.  Once you've got it installed and setup your
first run, there's probably not much in the HOWTO that you haven't
already figured out, so I don't foresee lots of use for a man page.
It's easy enough to do though, so maybe we should build both?

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH 0/4] HOWTO to get newbies started

[Matt Turner]

On Sat, Feb 9, 2013 at 12:40 PM, W. Trevor King <wking@tremily.us> wrote:
> From: "W. Trevor King" <wking@tremily.us>
>
> Here are my notes from figuring out how catalyst works, after browsing
> around online and getting help from y'all.  On IRC this morning, Chris
> White said he had some similar ideas floating around in his head.  I
> haven't filed a bug with the HOWTO [1], but I can if anyone wants to
> discuss it that way.
>
> [1]: http://www.gentoo.org/proj/en/releng/catalyst/faq.xml#howto
>
> W. Trevor King (4):
>   doc/HOWTO: First pass at a gentle Catalyst introduction
>   Makefile: Add DOCS for building files/HOWTO.html
>   files/.gitignore: Ignore $(DOCS) and $(DOC_SIDE_EFFECTS)
>   doc/HOWTO.txt: Explain how to run catalyst without installing
>
>  Makefile         |  15 +++-
>  doc/HOWTO.txt    | 219 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  files/.gitignore |   3 +-
>  3 files changed, 233 insertions(+), 4 deletions(-)
>  create mode 100644 doc/HOWTO.txt
>
> --
> 1.8.1.336.g94702dd
>
>

I've committed this series. Thanks!