[gentoo-soc] Framework for automated ebuild generators: final report

[gentoo-soc] Framework for automated ebuild generators: final report

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

Hi all,

summary
~~~~~~~

I'm working on `Framework for automated ebuild generators [1]. My mentor
is Rafael Martins. The aim of this project is to facilitate creation of
ebuild-generators for 3d party software providers. For those interested
in the advance of a project there is a Kanban board [2] and a
git-repository with code [3].

Final report
~~~~~~~~~~~~

Status: finished.

A framework for automated ebuild generators was created as was planned.
A developer's documentation can be found at [4]. During the project 3
working backends were implemented:

-- for ELPA repositories of emacs elisp packages
-- for CTAN (not so useful, as there is good CTAN suppport in Gentoo,
but usable if something is missing in the main tree)
-- for PyPI

A brief user's documentation is available at [5]. Also there are man
pages available after installing g-sorcery.

Usage is simple:

1. Add jauhien overlay with

	# layman -a jauhien

2. Emerge app-portage/g-sorcery (note, it depends on
app-portage/layman-9999)

	# emerge -va g-sorcery

3. List available overlays with

	# layman -L

4. Add a g-sorcery overlay you like (note, there is dependency between
overlays)

	# layman -a gnu-elpa -a marmalade

5. Emerge package you want

	# emerge -va app-emacs/clojure-mode

Credits. Thanks to:

	-- Rafael Martins for mentoring
	-- Brian Dolbec for lots of advices
	-- Denis Dupeyron for general coordination
	-- Auke Booij for implementing during GSoC 2010 idea that I found
interesting and that inspired me to work on ebuild generation
	-- anybody from Gentoo community who helped with answers to my
questions (not only during this GSoC ;-) )

Also I would like to know if somebody wants a backend for any repository
type. At the moment I'm planning to have a look at Leiningen packaging
system as I have plans to do some work with Clojure.

[1] http://www.google-melange.com/gsoc/project/google/gsoc2013/jauhien/17001

[2]
https://trello.com/board/framework-for-automated-ebuild-generators/51ad0f829c32858e72000c5f

[3] http://git.overlays.gentoo.org/gitweb/?p=proj/g-sorcery.git;a=summary

[4]
https://github.com/jauhien/g-sorcery/blob/master/docs/developer_instructions.rst

[5] https://github.com/jauhien/g-sorcery/blob/master/README.md

--
Regards,
Jauhien


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-soc] Framework for automated ebuild generators: final report

[EBo]

Jauhien,

First congratulations on finishing the project!  I have a couple of 
questions and suggestions for possible future work.  I will inline to 
keep the thread.

On Sep 25 2013 2:58 PM, Jauhien Piatlicki wrote:
> Hi all,
>
> summary
> ~~~~~~~
>
> I'm working on `Framework for automated ebuild generators [1]. My 
> mentor
> is Rafael Martins. The aim of this project is to facilitate creation 
> of
> ebuild-generators for 3d party software providers. For those 
> interested
> in the advance of a project there is a Kanban board [2] and a
> git-repository with code [3].
>
> Final report
> ~~~~~~~~~~~~
>
> Status: finished.
>
> A framework for automated ebuild generators was created as was 
> planned.
> A developer's documentation can be found at [4]. During the project 3
> working backends were implemented:
>
> -- for ELPA repositories of emacs elisp packages
> -- for CTAN (not so useful, as there is good CTAN suppport in Gentoo,
> but usable if something is missing in the main tree)
> -- for PyPI

It would interesting if this work could work with other efforts to 
tackle specific other specific backends like R and the R_Overlay 
project.  I also know that ruby, and rubygems is another management 
headache (which I recently smacked my forehead against -- for the third 
time in as many years).

> A brief user's documentation is available at [5]. Also there are man
> pages available after installing g-sorcery.
>
> Usage is simple:
>
> 1. Add jauhien overlay with
>
> 	# layman -a jauhien
>
> 2. Emerge app-portage/g-sorcery (note, it depends on
> app-portage/layman-9999)

With Gentoo's distain for live ebuilds, do you have a sense for which 
version of layman might support the necessary functionality?

> 	# emerge -va g-sorcery
>
> 3. List available overlays with
>
> 	# layman -L
>
> 4. Add a g-sorcery overlay you like (note, there is dependency 
> between
> overlays)
>
> 	# layman -a gnu-elpa -a marmalade
>
> 5. Emerge package you want
>
> 	# emerge -va app-emacs/clojure-mode
>
> Credits. Thanks to:
>
> 	-- Rafael Martins for mentoring
> 	-- Brian Dolbec for lots of advices
> 	-- Denis Dupeyron for general coordination
> 	-- Auke Booij for implementing during GSoC 2010 idea that I found
> interesting and that inspired me to work on ebuild generation
> 	-- anybody from Gentoo community who helped with answers to my
> questions (not only during this GSoC ;-) )
>
> Also I would like to know if somebody wants a backend for any 
> repository
> type. At the moment I'm planning to have a look at Leiningen 
> packaging
> system as I have plans to do some work with Clojure.

recently I started looking at OSv, and it uses gradle.  The current 
gentoo ebuilds I have found are -bin installs.  Since OSv has a Java VM 
as one of its primary pieces, it might be both instructive and useful to 
contact them and get the skinny on why they use gradle since you are 
looking at Clojure.

I mentioned rubygems/ruby above.  Maybe that would be an interesting 
avenue also.

other than that, thanks for all the great work!

   EBo --

> [1] 
> http://www.google-melange.com/gsoc/project/google/gsoc2013/jauhien/17001
>
> [2]
> 
> https://trello.com/board/framework-for-automated-ebuild-generators/51ad0f829c32858e72000c5f
>
> [3] 
> http://git.overlays.gentoo.org/gitweb/?p=proj/g-sorcery.git;a=summary
>
> [4]
> 
> https://github.com/jauhien/g-sorcery/blob/master/docs/developer_instructions.rst
>
> [5] https://github.com/jauhien/g-sorcery/blob/master/README.md
>
> --
> Regards,
> Jauhien

Re: [gentoo-soc] Framework for automated ebuild generators: final report

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 4871 bytes --]

On Wed, 2013-09-25 at 23:29 -0600, EBo wrote:
> Jauhien,
> 
> First congratulations on finishing the project!  I have a couple of 
> questions and suggestions for possible future work.  I will inline to 
> keep the thread.
> 
> On Sep 25 2013 2:58 PM, Jauhien Piatlicki wrote:
> > Hi all,
> >
> > summary
> > ~~~~~~~
> >
> > I'm working on `Framework for automated ebuild generators [1]. My 
> > mentor
> > is Rafael Martins. The aim of this project is to facilitate creation 
> > of
> > ebuild-generators for 3d party software providers. For those 
> > interested
> > in the advance of a project there is a Kanban board [2] and a
> > git-repository with code [3].
> >
> > Final report
> > ~~~~~~~~~~~~
> >
> > Status: finished.
> >
> > A framework for automated ebuild generators was created as was 
> > planned.
> > A developer's documentation can be found at [4]. During the project 3
> > working backends were implemented:
> >
> > -- for ELPA repositories of emacs elisp packages
> > -- for CTAN (not so useful, as there is good CTAN suppport in Gentoo,
> > but usable if something is missing in the main tree)
> > -- for PyPI
> 
> It would interesting if this work could work with other efforts to 
> tackle specific other specific backends like R and the R_Overlay 
> project. 

R needed more fine grained control over the ebuilds generated,  hence
R_Overlay, with it's database of rules/overrides used to generate the
ebuilds.  In fact for the pypi backend some code inspiration is taken
from R-overlay to produce a database which can/will be downloaded to
speed up overlay generation/update (that feature may not be complete
yet).

>  I also know that ruby, and rubygems is another management 
> headache (which I recently smacked my forehead against -- for the third 
> time in as many years).
> 
> > A brief user's documentation is available at [5]. Also there are man
> > pages available after installing g-sorcery.
> >
> > Usage is simple:
> >
> > 1. Add jauhien overlay with
> >
> > 	# layman -a jauhien
> >
> > 2. Emerge app-portage/g-sorcery (note, it depends on
> > app-portage/layman-9999)
> 
> With Gentoo's distain for live ebuilds, do you have a sense for which 
> version of layman might support the necessary functionality?


I will include it in the next release of layman.  But none of the
current releases have it.  I have to make some changes for the new
portage config changes now in the newer portage versions.  So, it
shouldn't be too long.

And what's so hard about:

echo "=app-portage/layman **" >> /etc/portage/package.accept_keywords
emerge layman

and for easier live ebuild updates:

emerge app-portage/smart-live-rebuild

> 
> > 	# emerge -va g-sorcery
> >
> > 3. List available overlays with
> >
> > 	# layman -L
> >
> > 4. Add a g-sorcery overlay you like (note, there is dependency 
> > between
> > overlays)
> >
> > 	# layman -a gnu-elpa -a marmalade
> >
> > 5. Emerge package you want
> >
> > 	# emerge -va app-emacs/clojure-mode
> >
> > Credits. Thanks to:
> >
> > 	-- Rafael Martins for mentoring
> > 	-- Brian Dolbec for lots of advices
> > 	-- Denis Dupeyron for general coordination
> > 	-- Auke Booij for implementing during GSoC 2010 idea that I found
> > interesting and that inspired me to work on ebuild generation
> > 	-- anybody from Gentoo community who helped with answers to my
> > questions (not only during this GSoC ;-) )
> >
> > Also I would like to know if somebody wants a backend for any 
> > repository
> > type. At the moment I'm planning to have a look at Leiningen 
> > packaging
> > system as I have plans to do some work with Clojure.
> 
> recently I started looking at OSv, and it uses gradle.  The current 
> gentoo ebuilds I have found are -bin installs.  Since OSv has a Java VM 
> as one of its primary pieces, it might be both instructive and useful to 
> contact them and get the skinny on why they use gradle since you are 
> looking at Clojure.
> 
> I mentioned rubygems/ruby above.  Maybe that would be an interesting 
> avenue also.

I don't know ruby/rubygems, but I have a feeling it might be better off
using R_overlays code to generate a hosted Ruby_overlay

> 
> other than that, thanks for all the great work!
> 
>    EBo --
> 
> > [1] 
> > http://www.google-melange.com/gsoc/project/google/gsoc2013/jauhien/17001
> >
> > [2]
> > 
> > https://trello.com/board/framework-for-automated-ebuild-generators/51ad0f829c32858e72000c5f
> >
> > [3] 
> > http://git.overlays.gentoo.org/gitweb/?p=proj/g-sorcery.git;a=summary
> >
> > [4]
> > 
> > https://github.com/jauhien/g-sorcery/blob/master/docs/developer_instructions.rst
> >
> > [5] https://github.com/jauhien/g-sorcery/blob/master/README.md
> >
> > --
> > Regards,
> > Jauhien
> 
> 


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: [gentoo-soc] Framework for automated ebuild generators: final report

[Rich Freeman]

On Thu, Sep 26, 2013 at 6:21 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
>
> And what's so hard about:
>
> echo "=app-portage/layman **" >> /etc/portage/package.accept_keywords
> emerge layman
>
> and for easier live ebuild updates:
>
> emerge app-portage/smart-live-rebuild

There is nothing "hard" about running live ebuilds - they can simply
never be unmasked, and there is no QA whatsoever.  Smart-live-rebuild
just means that if you have a live package that does happen to work
the first commit that introduces a regression breaks it.  :)

Nothing wrong with having to depend on something live for something
cutting-edge.  However, it really isn't in a usable state until that
changes.  It will get there eventually.

Looking forward to getting rid of g-cpan whenever this becomes stable...

Rich

Re: [gentoo-soc] Framework for automated ebuild generators: final report

[EBo]

On Sep 26 2013 4:21 AM, Brian Dolbec wrote:
> On Wed, 2013-09-25 at 23:29 -0600, EBo wrote:
>> On Sep 25 2013 2:58 PM, Jauhien Piatlicki wrote:
>> ...
>> It would interesting if this work could work with other efforts to
>> tackle specific other specific backends like R and the R_Overlay
>> project.
>
> R needed more fine grained control over the ebuilds generated,  hence
> R_Overlay, with it's database of rules/overrides used to generate the
> ebuilds.  In fact for the pypi backend some code inspiration is taken
> from R-overlay to produce a database which can/will be downloaded to
> speed up overlay generation/update (that feature may not be complete
> yet).

Good to know.

>> > ...
>> > 2. Emerge app-portage/g-sorcery (note, it depends on
>> > app-portage/layman-9999)
>>
>> With Gentoo's distain for live ebuilds, do you have a sense for 
>> which
>> version of layman might support the necessary functionality?
>
>
> I will include it in the next release of layman.  But none of the
> current releases have it.  I have to make some changes for the new
> portage config changes now in the newer portage versions.  So, it
> shouldn't be too long.
>
> And what's so hard about:
>
> echo "=app-portage/layman **" >> /etc/portage/package.accept_keywords
> emerge layman
>
> and for easier live ebuild updates:
>
> emerge app-portage/smart-live-rebuild

it is not an issue with it being hard, but stability and consistency -- 
the code that I am you checked out of a repository and built last week 
is not necessarily the same code I checked out and emerged today.  
Someone might have made a change the day before yesterday which 
introduced a bug, and when I post a bug report there is no way to know 
what code either of us are compiling.

I always preferred using date-stamped ebuilds which check out a 
particular revision of a code base.  So I know that on 20130923 this is 
when I tested revision #219 with a hashtag of "fa1f0b5c4db5".  Then when 
I go back to test or depend on something, I know exactly what version we 
are talking about.

>> > ...
>> recently I started looking at OSv, and it uses gradle.  The current
>> gentoo ebuilds I have found are -bin installs.  Since OSv has a Java 
>> VM
>> as one of its primary pieces, it might be both instructive and 
>> useful to
>> contact them and get the skinny on why they use gradle since you are
>> looking at Clojure.
>>
>> I mentioned rubygems/ruby above.  Maybe that would be an interesting
>> avenue also.
>
> I don't know ruby/rubygems, but I have a feeling it might be better 
> off
> using R_overlays code to generate a hosted Ruby_overlay

sounds about right.  Since you were asking for suggestions I thought I 
would oblige ;-)))

Anyway, thanks again for the great work, and I hope that this and the 
R_Overlay continues to grow and progress.

   Best regards,

   EBo --

Re: [gentoo-soc] Framework for automated ebuild generators: final report

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 235 bytes --]

2013/9/26 Rich Freeman <rich0@gentoo.org>

>
> Looking forward to getting rid of g-cpan whenever this becomes stable...
>
>
There is no CPAN backend currently. But I can add it. I'll look at it in
the nearest future.

Regrads,
Jauhien

[-- Attachment #2: Type: text/html, Size: 685 bytes --]

Re: [gentoo-soc] Framework for automated ebuild generators: final report

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

2013/9/26 EBo <ebo@sandien.com>

> First congratulations on finishing the project!  I have a couple of
> questions and suggestions for possible future work.  I will inline to keep
> the thread.
>
>
Thanks

recently I started looking at OSv, and it uses gradle.  The current gentoo
> ebuilds I have found are -bin installs.  Since OSv has a Java VM as one of
> its primary pieces, it might be both instructive and useful to contact them
> and get the skinny on why they use gradle since you are looking at Clojure.
>
> I mentioned rubygems/ruby above.  Maybe that would be an interesting
> avenue also.
>
>
Will look at it.

Regards,
Jauhien

[-- Attachment #2: Type: text/html, Size: 1197 bytes --]

Re: [gentoo-soc] Framework for automated ebuild generators: final report

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]

On Thu, 2013-09-26 at 06:02 -0600, EBo wrote:
> it is not an issue with it being hard, but stability and consistency -- 
> the code that I am you checked out of a repository and built last week 
> is not necessarily the same code I checked out and emerged today.  
> Someone might have made a change the day before yesterday which 
> introduced a bug, and when I post a bug report there is no way to know 
> what code either of us are compiling.
> 
> I always preferred using date-stamped ebuilds which check out a 
> particular revision of a code base.  So I know that on 20130923 this is 
> when I tested revision #219 with a hashtag of "fa1f0b5c4db5".  Then when 
> I go back to test or depend on something, I know exactly what version we 
> are talking about.

>    Best regards,
> 
>    EBo --
> 
> 

FYI layman-9999 uses the git revision in the version, so submitting a
bug can be very specific about the version it occurred in.  It also
helps me to track down a possible regression.  But for dependency use,
yeah, date versions are better.
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 620 bytes --]