[gentoo-soc] Two ideas: R package installer (CRAN) and per-user daemons

[gentoo-soc] Two ideas: R package installer (CRAN) and per-user daemons

[Auke Booij]

Hey there,

Google Summer of Code is slowly taking off, I'd like to claim a spot.

Two ideas I'd like to expand on behalf of Gentoo are:
1. R package installer (also see gentoo-wiki.com). R has a lot of
additional packages, so it is not feasible to manually create and
maintain ebuilds for all of them. There are a number of databases
containing these additional packages, including CRAN and Bioconductor.
Three reasonable solutions are: 1. create a big ebuild with loads and
loads of USE flags which compiles and installs the wanted R packages
as one big package (not the best, since this is exactly what we want
to avoid with Gentoo, and we would indeed need loads of use flags), 2.
generate ebuilds for all packages (this would need regular running of
scripts, and would probably need for its own overlay), 3. install to
the filesystem without letting the package manager know about anything
(ehm... I guess this is not a reasonable solution, after all). Are
there any ideas about how to solve this, and how R usually installs
them? Sebastien fabbro, I cc'd you, are you still interested in
mentoring this?

2. Per-user daemons. Currently, there are some daemons, like
PulseAudio, jackd, several download daemons, and soon possibly X.org
(which will no longer need to run as root, and really should run as a
normal user), which run as specific users, instead of root or a
systemwide daemon-specific user. There is currently no framework for
launching these daemons at all, and the current solution is to either
make users figure out their own solution (a popular one is an extra
line in .xinitrc), or to set the username in some conf.d file, thus
disabling the possibility to run several of them for different users.
I would like to develop an extension to our current init.d system
which will enable users to write and start their own init scripts,
which at first only run as themselves, not as another user. This would
make it possible to cleanly launch several identical daemons for
several users. This project would involve either writing a new
additional init.d system, or extend our current system. I would like
to know 1. if you guys are understanding a word of what I'm saying at
all 2. if this would be an interesting idea (I think it is, of course)
3. anyone has any ideas or would like to mentor me.

Thanks for your ideas.
Auke Booij "tulcod".

Re: [gentoo-soc] Two ideas: R package installer (CRAN) and per-user daemons

[Petteri Räty]

On 18.3.2010 16.58, Auke Booij wrote:

> 2. Per-user daemons. Currently, there are some daemons, like
> PulseAudio, jackd, several download daemons, and soon possibly X.org
> (which will no longer need to run as root, and really should run as a
> normal user), which run as specific users, instead of root or a
> systemwide daemon-specific user. There is currently no framework for
> launching these daemons at all, and the current solution is to either
> make users figure out their own solution (a popular one is an extra
> line in .xinitrc), or to set the username in some conf.d file, thus
> disabling the possibility to run several of them for different users.
> I would like to develop an extension to our current init.d system
> which will enable users to write and start their own init scripts,
> which at first only run as themselves, not as another user. This would
> make it possible to cleanly launch several identical daemons for
> several users. This project would involve either writing a new
> additional init.d system, or extend our current system. I would like
> to know 1. if you guys are understanding a word of what I'm saying at
> all 2. if this would be an interesting idea (I think it is, of course)
> 3. anyone has any ideas or would like to mentor me.
> 

Isn't this something that should be handled by the upstream projects
instead of doing it at the distribution level? Also see:
http://standards.freedesktop.org/autostart-spec/autostart-spec-latest.html

Regards,
Petteri

[gentoo-soc] Re: Two ideas: R package installer (CRAN) and per-user daemons

[Sébastien Fabbro]

On Thursday 18 March, Auke Booij wrote:

> 1. R package installer (also see gentoo-wiki.com). R has a lot of
> additional packages, so it is not feasible to manually create and
> maintain ebuilds for all of them. There are a number of databases
> containing these additional packages, including CRAN and Bioconductor.
> Three reasonable solutions are: 1. create a big ebuild with loads and
> loads of USE flags which compiles and installs the wanted R packages
> as one big package (not the best, since this is exactly what we want
> to avoid with Gentoo, and we would indeed need loads of use flags), 2.
> generate ebuilds for all packages (this would need regular running of
> scripts, and would probably need for its own overlay), 3. install to
> the filesystem without letting the package manager know about anything
> (ehm... I guess this is not a reasonable solution, after all). Are
> there any ideas about how to solve this, and how R usually installs
> them? Sebastien fabbro, I cc'd you, are you still interested in
> mentoring this?

Hi,

CRAN/Bioconductor support in Gentoo is definitely something worth
exploring and has been requested by several Gentoo users.
Solutions 1. and 3. are not something I would pursue. With the
thousands R packages to deal with, you would want either on-the-fly
ebuild generation or the package manager to directly deal with the CRAN
format (which is I think what the paludis folks started doing until CRAN
changed its format again). Any solution leading to complex handling,
the users will return to the R CMD INSTALL quickly.
Keep in mind that one of the main reason not to use the R default
package management system but a Gentoo one is to get R-external
dependencies from the portage tree.

I am not sure I will find time to mentor the full Summer. If anyone
applying for this project makes a good application, I'll push this to
my fellow devs to give a hand in the mentoring process.

--
Sebastien

Re: [gentoo-soc] Two ideas: R package installer (CRAN) and per-user daemons

[Auke Booij]

Petteri,

Thanks, there are indeed some solutions for users running X, but I
think a lot of services could benefit from being able to be started
without the user logging in (for example, webservers really should
launch clients' websites as fastcgi services at boot time) and without
having to run X anywhere.

As for who should handle this... there are some other ideas on the
ideas page concerning OpenRC, which I think is the most appropriate
project for this idea. And depending on the exact implementation, it
might very well be dependent upon the system's init.d software (in
this case OpenRC, but different distros use different systems). Ergo,
I'd say this idea does fit inside Gentoo's scope.

Auke.

On Fri, Mar 19, 2010 at 12:22 PM, Petteri Räty <betelgeuse@gentoo.org> wrote:
> On 18.3.2010 16.58, Auke Booij wrote:
>
>> 2. Per-user daemons. Currently, there are some daemons, like
>> PulseAudio, jackd, several download daemons, and soon possibly X.org
>> (which will no longer need to run as root, and really should run as a
>> normal user), which run as specific users, instead of root or a
>> systemwide daemon-specific user. There is currently no framework for
>> launching these daemons at all, and the current solution is to either
>> make users figure out their own solution (a popular one is an extra
>> line in .xinitrc), or to set the username in some conf.d file, thus
>> disabling the possibility to run several of them for different users.
>> I would like to develop an extension to our current init.d system
>> which will enable users to write and start their own init scripts,
>> which at first only run as themselves, not as another user. This would
>> make it possible to cleanly launch several identical daemons for
>> several users. This project would involve either writing a new
>> additional init.d system, or extend our current system. I would like
>> to know 1. if you guys are understanding a word of what I'm saying at
>> all 2. if this would be an interesting idea (I think it is, of course)
>> 3. anyone has any ideas or would like to mentor me.
>>
>
> Isn't this something that should be handled by the upstream projects
> instead of doing it at the distribution level? Also see:
> http://standards.freedesktop.org/autostart-spec/autostart-spec-latest.html
>
> Regards,
> Petteri
>
>

Re: [gentoo-soc] Re: Two ideas: R package installer (CRAN) and per-user daemons

[Auke Booij]

Hi Sebastien,

Please forgive me for my lack of knowledge about package managers, but
what systems are in place to generate ebuilds on the fly? And is there
any compatibility between the different package managers in that
sense? Would it be possible to make an overlay build a repository of R
packages on the fly? If ebuilds are generated on the fly, how will we
deal with nontrivial ebuilds?

As always, there are a number of advantages to make the package
manager do the hard work. I guess some more benefits would, among
many, be: emerge world updates R packages too (ie. centralized
updating), packages in portage can pull in R packages without user
interference, and users can relatively easily install R packages with
patches.

Thanks,
Auke.

2010/3/19 Sébastien Fabbro <bicatali@gentoo.org>:
> On Thursday 18 March, Auke Booij wrote:
>
>> 1. R package installer (also see gentoo-wiki.com). R has a lot of
>> additional packages, so it is not feasible to manually create and
>> maintain ebuilds for all of them. There are a number of databases
>> containing these additional packages, including CRAN and Bioconductor.
>> Three reasonable solutions are: 1. create a big ebuild with loads and
>> loads of USE flags which compiles and installs the wanted R packages
>> as one big package (not the best, since this is exactly what we want
>> to avoid with Gentoo, and we would indeed need loads of use flags), 2.
>> generate ebuilds for all packages (this would need regular running of
>> scripts, and would probably need for its own overlay), 3. install to
>> the filesystem without letting the package manager know about anything
>> (ehm... I guess this is not a reasonable solution, after all). Are
>> there any ideas about how to solve this, and how R usually installs
>> them? Sebastien fabbro, I cc'd you, are you still interested in
>> mentoring this?
>
> Hi,
>
> CRAN/Bioconductor support in Gentoo is definitely something worth
> exploring and has been requested by several Gentoo users.
> Solutions 1. and 3. are not something I would pursue. With the
> thousands R packages to deal with, you would want either on-the-fly
> ebuild generation or the package manager to directly deal with the CRAN
> format (which is I think what the paludis folks started doing until CRAN
> changed its format again). Any solution leading to complex handling,
> the users will return to the R CMD INSTALL quickly.
> Keep in mind that one of the main reason not to use the R default
> package management system but a Gentoo one is to get R-external
> dependencies from the portage tree.
>
> I am not sure I will find time to mentor the full Summer. If anyone
> applying for this project makes a good application, I'll push this to
> my fellow devs to give a hand in the mentoring process.
>
> --
> Sebastien
>
>

Re: [gentoo-soc] Re: Two ideas: R package installer (CRAN) and per-user daemons

[Sébastien Fabbro]

On Friday 19 March, Auke Booij wrote:

> Please forgive me for my lack of knowledge about package managers, but
> what systems are in place to generate ebuilds on the fly? And is there
> any compatibility between the different package managers in that
> sense? Would it be possible to make an overlay build a repository of R
> packages on the fly? If ebuilds are generated on the fly, how will we
> deal with nontrivial ebuilds?
> 
> As always, there are a number of advantages to make the package
> manager do the hard work. I guess some more benefits would, among
> many, be: emerge world updates R packages too (ie. centralized
> updating), packages in portage can pull in R packages without user
> interference, and users can relatively easily install R packages with
> patches.

Experiment with g-cpan and g-ctan in the portage tree to have an idea
of what is done. Indeed working directly with the package manager is
more appealing for many reasons: dependencies, QA, convenience, ...For
differences between package managers, see with the different parties
involved. To my knowledge only paludis had some CRAN integration at
some point. Also Debian has a cran2deb utility which is probably
worth looking into, they actually had a GSoC 2008 project about it.

Sebastien

Re: [gentoo-soc] Re: Two ideas: R package installer (CRAN) and per-user daemons

[Auke Booij]

Alright, so I experimented a bit with G-CPAN and read some of its
code, and it kind of works as I expected. I also spoke with some
paludis devs (including the chief), and they told me that basically a
lot of CRAN packages are illegal in some of way. It's usually the
version that's got a wrong format, but the metadata files are slightly
incorrect in general. If I would get this to work, we will need to
file loads of bugs upstream to improve their repo quality. Having
spoken with some guys in #R, this seems to be no problem, the CRAN
maintainers would love QA improvements.

I also looked into cran2deb, and it's basically not unlike G-CPAN,
except that it directly builds and then generates a .deb file. The
magic happens in /R/build.R, all the way at the bottom.

Another thing. Bioconductor, another source of R packages, uses
exactly the same format as CRAN, so if I can get CRAN packages to
build, Bioconductor packages will install just as smoothly.

I guess this sums up my findings, and I don't think I'm really missing
any critical information. I would like to hear any further ideas, and
would appreciate a mentor, but I guess mentors are usually assigned
after the application period, so perhaps I should start writing an
application.

Thanks,
tulcod.

PS: an overview of the quality of packages in CRAN can be found at this link:
http://cran.r-project.org/web/checks/check_summary.html

2010/3/19 Sébastien Fabbro <bicatali@gentoo.org>:
> On Friday 19 March, Auke Booij wrote:
>
>> Please forgive me for my lack of knowledge about package managers, but
>> what systems are in place to generate ebuilds on the fly? And is there
>> any compatibility between the different package managers in that
>> sense? Would it be possible to make an overlay build a repository of R
>> packages on the fly? If ebuilds are generated on the fly, how will we
>> deal with nontrivial ebuilds?
>>
>> As always, there are a number of advantages to make the package
>> manager do the hard work. I guess some more benefits would, among
>> many, be: emerge world updates R packages too (ie. centralized
>> updating), packages in portage can pull in R packages without user
>> interference, and users can relatively easily install R packages with
>> patches.
>
> Experiment with g-cpan and g-ctan in the portage tree to have an idea
> of what is done. Indeed working directly with the package manager is
> more appealing for many reasons: dependencies, QA, convenience, ...For
> differences between package managers, see with the different parties
> involved. To my knowledge only paludis had some CRAN integration at
> some point. Also Debian has a cran2deb utility which is probably
> worth looking into, they actually had a GSoC 2008 project about it.
>
> Sebastien
>
>

[gentoo-soc] Re: Two ideas: R package installer (CRAN) and per-user daemons

[Auke Booij]

I'd like to get more thoughts on this. Should it share code with
OpenRC (that's what makes init.d scripts work, right?), or be a
completely separate program? If it's a new program, should it use the
same init.d "standard"? I was also thinking of inter-user
dependencies, where system init.d scripts can depend on user init.d
scripts, and, if the other users allows this, users can depend others'
init.d scripts. Would such possibilities be useful? What about root
not allowing users to create new init.d scripts, but only launch an
existing one on their behalf (ie., root gives you a list of daemons
you can launch, but you cannot create your own)? I guess the
configuration files should be based on Bash (especially considering
some env variables could be useful)?

Thanks in advance for all your ideas,
tulcod.

On Thu, Mar 18, 2010 at 3:58 PM, Auke Booij <auke@tulcod.com> wrote:
> 2. Per-user daemons. Currently, there are some daemons, like
> PulseAudio, jackd, several download daemons, and soon possibly X.org
> (which will no longer need to run as root, and really should run as a
> normal user), which run as specific users, instead of root or a
> systemwide daemon-specific user. There is currently no framework for
> launching these daemons at all, and the current solution is to either
> make users figure out their own solution (a popular one is an extra
> line in .xinitrc), or to set the username in some conf.d file, thus
> disabling the possibility to run several of them for different users.
> I would like to develop an extension to our current init.d system
> which will enable users to write and start their own init scripts,
> which at first only run as themselves, not as another user. This would
> make it possible to cleanly launch several identical daemons for
> several users. This project would involve either writing a new
> additional init.d system, or extend our current system. I would like
> to know 1. if you guys are understanding a word of what I'm saying at
> all 2. if this would be an interesting idea (I think it is, of course)
> 3. anyone has any ideas or would like to mentor me.

Re: [gentoo-soc] Re: Two ideas: R package installer (CRAN) and per-user daemons

[Hans de Graaff]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On Mon, 2010-03-22 at 14:33 +0100, Auke Booij wrote:
> Alright, so I experimented a bit with G-CPAN and read some of its
> code, and it kind of works as I expected. I also spoke with some
> paludis devs (including the chief), and they told me that basically a
> lot of CRAN packages are illegal in some of way.

With Ruby's gems we are seeing similar problems, to the point that we no
longer use the rubygem package manager application but instead wrote our
own eclasses to deal with the gems directly. This means more work for
the ruby herd, but it does enable us to properly express dependencies
and do things like patching and running tests. There is a balance here
between extra work and proper quality that depends on the quality of the
underlying repository.

Kind regards,

Hans

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-soc] Re: Two ideas: R package installer (CRAN) and per-user daemons

[Auke Booij]

[-- Attachment #1: Type: text/plain, Size: 2612 bytes --]

Alright, here's some proof I know how to use git. patch -p1 processes
it just fine on devmanual.git.

On Thu, Mar 18, 2010 at 3:58 PM, Auke Booij <auke@tulcod.com> wrote:
> Hey there,
>
> Google Summer of Code is slowly taking off, I'd like to claim a spot.
>
> Two ideas I'd like to expand on behalf of Gentoo are:
> 1. R package installer (also see gentoo-wiki.com). R has a lot of
> additional packages, so it is not feasible to manually create and
> maintain ebuilds for all of them. There are a number of databases
> containing these additional packages, including CRAN and Bioconductor.
> Three reasonable solutions are: 1. create a big ebuild with loads and
> loads of USE flags which compiles and installs the wanted R packages
> as one big package (not the best, since this is exactly what we want
> to avoid with Gentoo, and we would indeed need loads of use flags), 2.
> generate ebuilds for all packages (this would need regular running of
> scripts, and would probably need for its own overlay), 3. install to
> the filesystem without letting the package manager know about anything
> (ehm... I guess this is not a reasonable solution, after all). Are
> there any ideas about how to solve this, and how R usually installs
> them? Sebastien fabbro, I cc'd you, are you still interested in
> mentoring this?
>
> 2. Per-user daemons. Currently, there are some daemons, like
> PulseAudio, jackd, several download daemons, and soon possibly X.org
> (which will no longer need to run as root, and really should run as a
> normal user), which run as specific users, instead of root or a
> systemwide daemon-specific user. There is currently no framework for
> launching these daemons at all, and the current solution is to either
> make users figure out their own solution (a popular one is an extra
> line in .xinitrc), or to set the username in some conf.d file, thus
> disabling the possibility to run several of them for different users.
> I would like to develop an extension to our current init.d system
> which will enable users to write and start their own init scripts,
> which at first only run as themselves, not as another user. This would
> make it possible to cleanly launch several identical daemons for
> several users. This project would involve either writing a new
> additional init.d system, or extend our current system. I would like
> to know 1. if you guys are understanding a word of what I'm saying at
> all 2. if this would be an interesting idea (I think it is, of course)
> 3. anyone has any ideas or would like to mentor me.
>
> Thanks for your ideas.
> Auke Booij "tulcod".
>

[-- Attachment #2: 0001-Testing-patch-creation.patch --]
[-- Type: application/octet-stream, Size: 1303 bytes --]

From 464c992e45b31d6d1e28ad1b0e4d0bdca3e57c26 Mon Sep 17 00:00:00 2001
From: Auke Booij <auke@tulcod.com>
Date: Sat, 3 Apr 2010 14:45:02 +0200
Subject: [PATCH] Testing patch creation

---
 ebuild-writing/eapi/text.xml |    4 ++++
 text.xml                     |    2 +-
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/ebuild-writing/eapi/text.xml b/ebuild-writing/eapi/text.xml
index 94a1b50..2bc3ee5 100644
--- a/ebuild-writing/eapi/text.xml
+++ b/ebuild-writing/eapi/text.xml
@@ -293,6 +293,10 @@ src_compile() {
 		the <c>src_compile</c> phase is equivalent to a call to the function
 		named <c>default_src_compile</c>.
 		</p>
+		<p>
+		G-CRAN absolutely abuses this functionality. Sooner or later, it'll set
+		your cat on fire.
+		</p>
 	</li>
 </ul>
 </body>
diff --git a/text.xml b/text.xml
index 15acc9d..897f60a 100644
--- a/text.xml
+++ b/text.xml
@@ -17,7 +17,7 @@ detailed, up to date technical content.
             Ciaran McCreesh, Grant Goodyear, Aaron Walker, Robert Coie,
             Tom Martin, Paul Varner, Ilya Volynets-Evenbakh,
             Diego Pettenò, Fernando J. Pereda, Simon Stelling, Alin Dobre,
-            Joseph Jezak, Tim Yamin, Mark Loeser
+            Joseph Jezak, Tim Yamin, Mark Loeser, Auke Booij
         </p>
     </dd>
 </dl>
-- 
1.7.0.2