[gentoo-soc] SCM Snapshot management

[gentoo-soc] SCM Snapshot management

[Colleen Josephson]

[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]

The SCM snapshot management project (
http://www.gentoo.org/proj/en/userrel/soc/ideas.xml#doc_chap2_sect18),
paired with adding tags to Portage, search improvements, and a couple other
small Portage feature additions.

To quote Luca Barbato:
The idea is to work on the concept so that:

- by emerging a live ebuild you automatically generate the snapshot
ebuild and record it...instead of recording it as live ebuild in the vdb you
record
it as snapshot ebuild. The generation step could include a version
generation algorithm more complicated than you expect

- you have a emaint command to generate the snapshot ebuild on the fly.

Where exactly would the information of a live vs. snapshot ebuild be
recorded in the vdb? I checked out my own VDB,
and it had some text files like:
BUILD_TIME  CBUILD  CHOST COUNTER  DEFINED_PHASES  DESCRIPTION  FEATURES
INHERITED  KEYWORDS  LICENSE  NEEDED.ELF.2  RDEPEND  SLOT environment.bz2
repository CATEGORY CFLAGS  CONTENTS  CXXFLAGS  DEPEND  EAPI  HOMEPAGE  IUSE
LDFLAGS  NEEDED   PF  SIZE  USE jack-audio-connection-kit-0.120.1.ebuild

I checked the contents of most of them, but none seemed like particularly
likely locations to store that information.

Also, the project description says:

   - If the writer has no access (the ebuild is not uploaded to main tree or
   a listed overlay), the ebuild behaves as a live ebuild.


   - Otherwise, the snapshot manager daemon packages the snapshot and posts
   it on Gentoo mirrors. The ebuild fetches the snapshot and uses it. (this
   will require a manifest update upon/after commit...)

So I think that means that if the ebuild writer is a dev, the snapshot will
be posted to Gentoo mirrors. If not, the ebuild is not snapshot and is a
live ebuild.
I think it would be nice to have the ability to use locally-stored snapshots
instead of forcing those who are not devs to do live ebuilds.

I was thinking for the versioning, perhaps something revision numbers would
be sensible?

Thanks,
-Colleen

[-- Attachment #2: Type: text/html, Size: 2216 bytes --]

Re: [gentoo-soc] SCM Snapshot management

[Zac Medico]

On 04/02/2011 03:44 PM, Colleen Josephson wrote:
> The SCM snapshot management project (
> http://www.gentoo.org/proj/en/userrel/soc/ideas.xml#doc_chap2_sect18),
> paired with adding tags to Portage, search improvements, and a couple other
> small Portage feature additions.
> 
> To quote Luca Barbato:
> The idea is to work on the concept so that:
> 
> - by emerging a live ebuild you automatically generate the snapshot
> ebuild and record it...instead of recording it as live ebuild in the vdb you
> record
> it as snapshot ebuild. The generation step could include a version
> generation algorithm more complicated than you expect
> 
> - you have a emaint command to generate the snapshot ebuild on the fly.

Also, it would be nice if you could specify a specific revision or tag
to take the snapshot from. Maybe it would even make sense to specify a
branch in some cases, but I imagine that in most cases you'd probably
have separate ebuilds for separate branches.

> Where exactly would the information of a live vs. snapshot ebuild be
> recorded in the vdb? I checked out my own VDB,
> and it had some text files like:
> BUILD_TIME  CBUILD  CHOST COUNTER  DEFINED_PHASES  DESCRIPTION  FEATURES
> INHERITED  KEYWORDS  LICENSE  NEEDED.ELF.2  RDEPEND  SLOT environment.bz2
> repository CATEGORY CFLAGS  CONTENTS  CXXFLAGS  DEPEND  EAPI  HOMEPAGE  IUSE
> LDFLAGS  NEEDED   PF  SIZE  USE jack-audio-connection-kit-0.120.1.ebuild

If you generate a version, that's probably going to be encoded into the
name of the directory, and it will also show up in PF. You could use a
special version suffix to distinguish live/snapshot/normal ebuilds, like
in GLEP 54. It's also conceivable to use tokens in either the PROPERTIES
or RESTRICT variables to distinguish between live/snapshot/normal ebuilds.

> I checked the contents of most of them, but none seemed like particularly
> likely locations to store that information.
> 
> Also, the project description says:
> 
>    - If the writer has no access (the ebuild is not uploaded to main tree or
>    a listed overlay), the ebuild behaves as a live ebuild.
> 
> 
>    - Otherwise, the snapshot manager daemon packages the snapshot and posts
>    it on Gentoo mirrors. The ebuild fetches the snapshot and uses it. (this
>    will require a manifest update upon/after commit...)

I'm not sure if automatic upload is really necessary. If a developer is
has tested and is satisfied with a particular snapshot, it shouldn't be
too much trouble to upload the snapshot manually.

> So I think that means that if the ebuild writer is a dev, the snapshot will
> be posted to Gentoo mirrors. If not, the ebuild is not snapshot and is a
> live ebuild.
> I think it would be nice to have the ability to use locally-stored snapshots
> instead of forcing those who are not devs to do live ebuilds.
> 
> I was thinking for the versioning, perhaps something revision numbers would
> be sensible?

I think a timestamp is probably going to be the most manageable
approach, since it will provide uniformity, regardless of the underlying
SCM. For example, you could encode the timestamp as a _p2011041916
version suffix if you wanted 1 minute precision.

[1] http://www.gentoo.org/proj/en/glep/glep-0054.html
-- 
Thanks,
Zac

Re: [gentoo-soc] SCM Snapshot management

[Luca Barbato]

On 04/04/2011 09:16 PM, Zac Medico wrote:

> [1] http://www.gentoo.org/proj/en/glep/glep-0054.html

I'd rather focus on:

http://dev.gentoo.org/~lu_zero/glep/liveebuild.rst



-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: [gentoo-soc] SCM Snapshot management

[Colleen Josephson]

I've submitted my proposal for the SCM Snapshot Management Infrastructure [1].
It is a little rough around the edges. Any comments/suggestions are
appreciated, I'll do my best to add to it before the deadline!

[1] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/cjosephson/1

-Colleen