[gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[darkdefende]

Hi there!
My name is Sebastian Parborg and I'm currently a student at the university
in Linköping (LiU), Sweden. I usually go by the nick "DarkDefender".
I've been using Gentoo for at least five years (I've lost track of the time
I have been using it :P) and I've been thinking about participating in GSOC
for some time now. So naturally Gentoo was one of the first things that I
looked at.

The ebuild generator in the ideas list on the wiki caught my attention and
I would really like to work on this.
I've discussed some ideas in the IRC channel but it will be good to have
it up here also.

I was thinking about doing a ebuild generator like a "install wizard". What
I mean with this is that it will have input fields for deps, useflags, where
to get the sources and how (tarball?/Git?/SVN?...) and the different stages
in the ebuild (unpack,compile,install...).
However I aim at making it as simple so that it will basically "ask" you how
you would build it normally and it will put together an ebuild for you.
This will obviously be usefull for beginners or people (perhaps devs?) that
doesn't use gentoo. It might also come in handy with small projects.
But that doesn´t mean that I'll not try to make it usefull for larger projects
as well.

An other thing is that I will make the generator independet of the GUI so that
you can write clients to it in whatever you wish. But I will mainly focus on
a GTK client first to get it working. After that I will try to make a webapp
for it as dol-sen on the IRC channel suggested.
So you can input all needed paramaters on a webpage and it will output the
ebuild so that you can copy/paste it.

C&C is hugely appreciated to flesh out this proposal.

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Fabian Groffen]

Hi Sebastian,

On 20-03-2011 19:18:54 +0100, darkdefende@gmail.com wrote:
> I was thinking about doing a ebuild generator like a "install wizard". What
> I mean with this is that it will have input fields for deps, useflags, where
> to get the sources and how (tarball?/Git?/SVN?...) and the different stages
> in the ebuild (unpack,compile,install...).

Have you considered a 100% automatic ebuild generator?  Based on
heuristics you can come quite far in generating a first shot.  Think of
tools like autoscan, and what brew create[1] for instance do.

> However I aim at making it as simple so that it will basically "ask" you how
> you would build it normally and it will put together an ebuild for you.

What are the questions you envisage to present the user here?

> This will obviously be usefull for beginners or people (perhaps devs?) that
> doesn't use gentoo. It might also come in handy with small projects.
> But that doesn´t mean that I'll not try to make it usefull for larger projects
> as well.

In general, trying to come up with a use-case (or two) helps to get
yourself a focus on what you would like to do, and what not.  A clearly
defined project has multiple advantages for everyone involved.

> An other thing is that I will make the generator independent of the GUI so that
> you can write clients to it in whatever you wish. But I will mainly focus on
> a GTK client first to get it working. After that I will try to make a webapp
> for it as dol-sen on the IRC channel suggested.

How about interaction with an editor, such as Vim?  Would a command-line
based core be acceptable for you?  It would help you to get it accepted
in a broader audience.

> C&C is hugely appreciated to flesh out this proposal.

What language(s) would you like to use to work this thing out?
Perhaps try to fit approaches like Perl's GPAN into your proposal.
Likewise, Python's pypi packages may inspire you how to fully
automatically create an ebuild from just an url.


[1] http://mxcl.github.com/homebrew/

Regards,

-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[darkdefende]

Fabian Groffen writes:

> Hi Sebastian,
> 
> Have you considered a 100% automatic ebuild generator?  Based on
> heuristics you can come quite far in generating a first shot.  Think of
> tools like autoscan, and what brew create[1] for instance do.
>
Well it would be quite hard to make it 100% automatic as there are so many
ways you can unpack->prepare->compile->install.
Perhaps it would work for simple makefile projects but then you have scons,
cmake, python and java projects.

I have no idea how I would automaticly figure out the useflags, deps and
the build process without user input!

> What are the questions you envisage to present the user here?
>
It won't really be questions prompted to the user but inputfields where
you'll write how you would build it normally.
For example in the "How would you build/install" field you would for a
simple makefile project type:
make && make install
And the program should convert that to the corresponding commands in
src_compile() and src_install()

> In general, trying to come up with a use-case (or two) helps to get
> yourself a focus on what you would like to do, and what not.  A clearly
> defined project has multiple advantages for everyone involved.
>
I thought that this would be to help with generation of small to medium
sized projects. But if I have time I will try to make it able to generate
a working firefox/blender ebuild ;)

I have some small projects that it will try to get it to build first 
ofcourse. That way I can set milestones hopefully.

> How about interaction with an editor, such as Vim?  Would a command-line
> based core be acceptable for you?  It would help you to get it accepted
> in a broader audience.
>
Why indeed! I use most of my stuff from commandline, including vim.

> What language(s) would you like to use to work this thing out?
> Perhaps try to fit approaches like Perl's GPAN into your proposal.
> Likewise, Python's pypi packages may inspire you how to fully
> automatically create an ebuild from just an url.
>
I've only had courses in python and lisp (scheme/racket) so far.
I'm planning to write this in python.

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Fabian Groffen]

On 21-03-2011 12:00:34 +0100, darkdefende@gmail.com wrote:
> > Have you considered a 100% automatic ebuild generator?  Based on
> > heuristics you can come quite far in generating a first shot.  Think of
> > tools like autoscan, and what brew create[1] for instance do.
> >
> Well it would be quite hard to make it 100% automatic as there are so many
> ways you can unpack->prepare->compile->install.
> Perhaps it would work for simple makefile projects but then you have scons,
> cmake, python and java projects.

How hard do you think it would be to detect that you are actually
dealing with one of those?

> I have no idea how I would automaticly figure out the useflags, deps and
> the build process without user input!

Of course you can't make it 100% perfect.  But would you be happy if you
could do a fairly good job in 70% of the cases?
Of course guessing and heuristics are your main tools here, but often
you find that they are actually right, and hence are reducing the work
significantly.  (And so increading the usefulness with the same amount.)

> > What are the questions you envisage to present the user here?
> >
> It won't really be questions prompted to the user but inputfields where
> you'll write how you would build it normally.

Why do you think a user knows?

> For example in the "How would you build/install" field you would for a
> simple makefile project type:
> make && make install
> And the program should convert that to the corresponding commands in
> src_compile() and src_install()

I challenge you to think of a way to unpack a tarball, analyse its
files and then conclude it's aparently a autoconf-based package, so
configure, make and make install will likely be the way to do it.

> > In general, trying to come up with a use-case (or two) helps to get
> > yourself a focus on what you would like to do, and what not.  A clearly
> > defined project has multiple advantages for everyone involved.
> >
> I thought that this would be to help with generation of small to medium
> sized projects. But if I have time I will try to make it able to generate
> a working firefox/blender ebuild ;)

This sounds more like making an example, not a use-case that describes
how your technique should be used, and why it is really useful/much
better than the current way of doing things.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[darkdefende]

Fabian Groffen writes:

> How hard do you think it would be to detect that you are actually
> dealing with one of those?
>
That depends on how much the build process differs from the standard way
of doing things. And as you said if it does a good job in most cases perhaps
that's good enough.

> Why do you think a user knows?
>
Hmm, you have a good point there. If they already know then perhaps it would
be easier for them to write the ebuild by hand instead.
So a generator with minimal user input might be the way to go here.

> I challenge you to think of a way to unpack a tarball, analyse its
> files and then conclude it's aparently a autoconf-based package, so
> configure, make and make install will likely be the way to do it.
>
If you wan't to be my mentor then I'll gladly accept your challenge and
look forward working with you!

> This sounds more like making an example, not a use-case that describes
> how your technique should be used, and why it is really useful/much
> better than the current way of doing things.
>
Indeed. Now that I've thought about it perhaps your suggestion is the
way to go. :)

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2710 bytes --]

On Mon, 2011-03-21 at 18:37 +0100, darkdefende@gmail.com wrote:
> Fabian Groffen writes:
> 
> > How hard do you think it would be to detect that you are actually
> > dealing with one of those?
> >
> That depends on how much the build process differs from the standard way
> of doing things. And as you said if it does a good job in most cases perhaps
> that's good enough.
> 

I think that most of the standard ways should be easily detected


> > Why do you think a user knows?

That user could be a dev for the app, but may not be a gentoo user.
Just helping to create an ebuild to get his/her pkg into gentoo.

Probably only if it is web based, and gets auto-submitted to a holding
area that users/devs can review, download, etc.

> >
> Hmm, you have a good point there. If they already know then perhaps it would
> be easier for them to write the ebuild by hand instead.
> So a generator with minimal user input might be the way to go here.
> 
> > I challenge you to think of a way to unpack a tarball, analyse its
> > files and then conclude it's aparently a autoconf-based package, so
> > configure, make and make install will likely be the way to do it.
> >
> If you wan't to be my mentor then I'll gladly accept your challenge and
> look forward working with you!
> 

Basic ability doesn't sound like much of a challenge to me.  What would
probably be best (and a challenge) is to have some kind of learning
mechanism, that the generator could use to better refine it's detection
and solutions.  After all there is a whole tree, plus many overlays that
can be run through as examples for it to learn from.  (yes I know there
are many that could/should be improved)

There is an app called shogun-toolbox  http://www.shogun-toolbox.org/

that is designed specifically for machine learning.  It has many
interface capabilities including python (which they state has the most
complete documentation).  There is also many examples for it's use.

I personally don't know nearly enough about it to know what modeling
methods might be suitable or even if there is one suitable for this type
of model.  But I think should be investigated.  Implemented this would
have the potential to increase it's success rate over time, reducing
developer load even more.

It is licensed GPL-3


> > This sounds more like making an example, not a use-case that describes
> > how your technique should be used, and why it is really useful/much
> > better than the current way of doing things.
> >
> Indeed. Now that I've thought about it perhaps your suggestion is the
> way to go. :)
> 

:) more ideas for thought
-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1531 bytes --]

On 16:56 Mon 21 Mar     , Brian Dolbec wrote:
> There is an app called shogun-toolbox  http://www.shogun-toolbox.org/
> 
> that is designed specifically for machine learning.  It has many 
> interface capabilities including python (which they state has the most 
> complete documentation).  There is also many examples for it's use.
> 
> I personally don't know nearly enough about it to know what modeling 
> methods might be suitable or even if there is one suitable for this 
> type of model.  But I think should be investigated.  Implemented this 
> would have the potential to increase it's success rate over time, 
> reducing developer load even more.

This sounds like something that might be beyond a GSoC project to 
develop a standalone code base. It would probably work better next year 
as an enhancement, if this year's project to build an initial working 
application worked out well.

What I would do instead is a "smart" detection using things like 
flex/bison/pybison. You could first detect the buildsystem type 
(autotools/cmake/distutils/etc) based on file existence. Then you would 
parse the build files using the syntax described for your lexical 
analyzer/parser to build a basic understanding of what build-time 
options are available and offer them as USE flags. You could also do 
similar things to detect dependencies by understanding their syntax in 
the source code.

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1904 bytes --]

On Mon, 2011-03-21 at 20:27 -0500, Donnie Berkholz wrote:
> On 16:56 Mon 21 Mar     , Brian Dolbec wrote:
> > There is an app called shogun-toolbox  http://www.shogun-toolbox.org/
> > 
> > that is designed specifically for machine learning.  It has many 
> > interface capabilities including python (which they state has the most 
> > complete documentation).  There is also many examples for it's use.
> > 
> > I personally don't know nearly enough about it to know what modeling 
> > methods might be suitable or even if there is one suitable for this 
> > type of model.  But I think should be investigated.  Implemented this 
> > would have the potential to increase it's success rate over time, 
> > reducing developer load even more.
> 
> This sounds like something that might be beyond a GSoC project to 
> develop a standalone code base. It would probably work better next year 
> as an enhancement, if this year's project to build an initial working 
> application worked out well.

Absolutely.  But I wanted to get the thought out there.  Then maybe this
years code could be designed to be easier to upgrade it's AI next year.


> 
> What I would do instead is a "smart" detection using things like 
> flex/bison/pybison. You could first detect the buildsystem type 
> (autotools/cmake/distutils/etc) based on file existence. Then you would 
> parse the build files using the syntax described for your lexical 
> analyzer/parser to build a basic understanding of what build-time 
> options are available and offer them as USE flags. You could also do 
> similar things to detect dependencies by understanding their syntax in 
> the source code.
> 

Yes there are several areas that a machine learning system could be
beneficial in this whole process.  Some of which are listed as other
project ideas this year.

-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[darkdefende]

Brian Dolbec writes:

> Absolutely.  But I wanted to get the thought out there.  Then maybe this
> years code could be designed to be easier to upgrade it's AI next year.
>
I have never written anything related to AI code generation before so I
don't really know how I would design the code to be easily modifed to
support this.
If there any easy rules to follow or do I have know how it works on a very
advanced level?

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

On Tue, 2011-03-22 at 18:49 +0100, darkdefende@gmail.com wrote:
> Brian Dolbec writes:
> 
> > Absolutely.  But I wanted to get the thought out there.  Then maybe this
> > years code could be designed to be easier to upgrade it's AI next year.
> >
> I have never written anything related to AI code generation before so I
> don't really know how I would design the code to be easily modified to
> support this.
> If there any easy rules to follow or do I have know how it works on a very
> advanced level?
> 

Neither have I.  Nor do I even pretend to know anything about
implementing this type of thing.   But for the purpose of your proposal,
don't worry about it.  I intend to find out more about it.   As I find
out the info needed. I'll make it known.  At that point it can be
decided if a slight change to the design is warranted.
-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]

On 23:34 Tue 22 Mar     , Brian Dolbec wrote:
> On Tue, 2011-03-22 at 18:49 +0100, darkdefende@gmail.com wrote:
> > Brian Dolbec writes:
> > > Absolutely.  But I wanted to get the thought out there.  Then 
> > > maybe this years code could be designed to be easier to upgrade 
> > > it's AI next year.
> >
> > I have never written anything related to AI code generation before 
> > so I don't really know how I would design the code to be easily 
> > modified to support this. If there any easy rules to follow or do I 
> > have know how it works on a very advanced level?
> 
> Neither have I.  Nor do I even pretend to know anything about 
> implementing this type of thing.  But for the purpose of your 
> proposal, don't worry about it.  I intend to find out more about it.  
> As I find out the info needed. I'll make it known.  At that point it 
> can be decided if a slight change to the design is warranted.

I have some knowledge about machine learning (ML), particularly as it 
relates to dimensionality reduction and clustering but also some other 
areas.

Essentially the idea would be to come up with a training set and a test 
set, both with known answers, using ebuilds that are already written. 
You "train" your ML tool on the training set to learn how source code 
and/or build tools relate to ebuild syntax and dependencies, then "test" 
it on the test set to see whether it learned things correctly. Iterate 
by fiddling around with settings until things actually work; when they 
do, you have a complete backend.

The complexity of that approach is why I suggest a simpler one that 
relies on defining the format of various files, since we already know 
what autotools syntax looks like, what ebuild syntax looks like, what C 
source code looks like, etc. ML approaches are more useful when you 
don't know the connection between inputs and outputs, but we do. On the 
ebuild side of this syntax-based approach, there would be some potential 
integration to another of our ongoing GSoC projects, libbash.

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[darkdefende]

Donnie Berkholz writes:

> The complexity of that approach is why I suggest a simpler one that 
> relies on defining the format of various files, since we already know 
> what autotools syntax looks like, what ebuild syntax looks like, what C 
> source code looks like, etc. ML approaches are more useful when you 
> don't know the connection between inputs and outputs, but we do. On the 
> ebuild side of this syntax-based approach, there would be some potential 
> integration to another of our ongoing GSoC projects, libbash.
>
Agreed, I would be to great of a task for me to write it like that.
Thanks for explaining how it would work though! Perhaps I will try
something like that later. :)

About the libbash project and/or other projects:
How should we collaborate during GSoC? Will there be a strong
"my code, their code" line or can we help each other if we are going to
use code from one of the other GSoC projects?

Re: [gentoo-soc] [SOC Ebuild generator project] Introduction and C&C

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On 03/26/2011 10:06 PM, darkdefende@gmail.com wrote:
> Donnie Berkholz writes:
> 
>> The complexity of that approach is why I suggest a simpler one that
>> relies on defining the format of various files, since we already know
>> what autotools syntax looks like, what ebuild syntax looks like, what
>> C source code looks like, etc. ML approaches are more useful when you
>> don't know the connection between inputs and outputs, but we do. On
>> the ebuild side of this syntax-based approach, there would be some
>> potential integration to another of our ongoing GSoC projects, libbash.
>>
> Agreed, I would be to great of a task for me to write it like that.
> Thanks for explaining how it would work though! Perhaps I will try
> something like that later. :)
> 
> About the libbash project and/or other projects:
> How should we collaborate during GSoC? Will there be a strong
> "my code, their code" line or can we help each other if we are going to
> use code from one of the other GSoC projects?
> 

Open source projects should be encouraging people to contribute so I
don't think fences make sense :) The only objectionable thing would be
implementing half the project for the other student but I doubt anyone
is planning to do that.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]