From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 21BBE138010 for ; Tue, 21 Aug 2012 18:14:25 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 94E18E074B for ; Tue, 21 Aug 2012 18:14:19 +0000 (UTC) Received: from mail-wg0-f53.google.com (mail-wg0-f53.google.com [74.125.82.53]) by pigeon.gentoo.org (Postfix) with ESMTP id 822C3E03E4 for ; Tue, 21 Aug 2012 16:27:09 +0000 (UTC) Received: by wgbfm10 with SMTP id fm10so5161090wgb.10 for ; Tue, 21 Aug 2012 09:27:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mailerd.de; s=mails; h=mime-version:sender:x-originating-ip:date:x-google-sender-auth :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=G6xf15ye9XkxFE+TOcYDiw0MhSxZuTqj2vgxHBWI47E=; b=FJ3n3IObYp+qjxjBBb82YM6x2drhfTVyG0989q4pgdbKR6rtsmwaV3vp3mR+sLxPWN dVbMANDy1qIWDABS32pVWnGAaHASxbqqPPqSUyLRb9oP6KSSDrj70r66xco/wXa5Yy7v YsfNYTguXh1ypOzr8knV/2rUq1Oy5EevPLuDw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:date:x-google-sender-auth :message-id:subject:from:to:cc:content-type :content-transfer-encoding:x-gm-message-state; bh=G6xf15ye9XkxFE+TOcYDiw0MhSxZuTqj2vgxHBWI47E=; b=iYkslWcgBwjd7zntbJiovCGdvoVafYDVaX2om16kHeQBSpRTRDWen2wbdc9fCrMN00 UHxAss/+84zmE/nAvoyUbTUQh2kaYAWoyVzxBlrkRT6BAIFMqFL2WTOBi9AWVcLebIfO TFI7j4e+aRH7hy3VTOUs3fAvMg84m32MkSEE+FdYMJStaqlaTXxUcKF/1pgT9qRaHD11 rE5vR7ycf3PxOMAKRb+yKHWB4zr8V3h6DLzPywxSpTaWMq4GVzTLE30fCMe7SIgUb62S ah5PHctb0oBQC6uT9YMlLxz0QMG9todjvMQsDEbD0ebEDHrnyCjoYAuOZkfXwC7hReq2 wheA== Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-soc@lists.gentoo.org Reply-to: gentoo-soc@lists.gentoo.org MIME-Version: 1.0 Received: by 10.180.91.228 with SMTP id ch4mr39292681wib.7.1345566428705; Tue, 21 Aug 2012 09:27:08 -0700 (PDT) Sender: andre@mailerd.de Received: by 10.194.13.104 with HTTP; Tue, 21 Aug 2012 09:27:08 -0700 (PDT) X-Originating-IP: [94.216.110.220] Date: Tue, 21 Aug 2012 18:27:08 +0200 X-Google-Sender-Auth: Sym-rT9k4b03x08Qj_RLfSCmmFc Message-ID: Subject: [gentoo-soc] Automatically generated overlay of R packages - final report From: =?ISO-8859-1?Q?Andr=E9_Erdmann?= To: gentoo-soc@lists.gentoo.org Cc: Denis Dupeyron Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQmVTDpK2/F9nfjIpLU2mYjsB7ac694sUjqretxdik9p1PP1m/p/hVHTuFt9LzyB3BpOe+l1 X-Archives-Salt: 8f4e155d-7808-4f0c-982c-af602349854c X-Archives-Hash: d4f671ff17449e6f5d0470d0ecff9051 Hi everyone, =3D=3D Brief summary of this project =3D=3D The aim of this project is to create scripts that automate the process of overlay creation/maintenance for R packages from repositories such as CRAN and Bioconductor. Longer: For the ebuild creation of a single package one needs to extract the package, copy-paste data from its description file to the ebuild and look up dependencies, which is time-consuming. Although trivial for a few number of packages, this is practically impossible to do by hand for repositories like CRAN (> 3500 packages), especially 'cause it also requires tracking changes (new / updated / removed packages). The solution is to automate that process and this is what this project is about. The project's git repository is located at http://git.overlays.gentoo.org/gitweb/?p=3Dproj/R_overlay.git =3D=3D Current state of this project and future directions =3D=3D Automatically generated overlay of R packages has now reached the end of GSoC 2012's coding period. The result briefly described below is roverlay, a script and modules written in python. I tried to keep the code as extensible as possible, making future extensions like other ways to get R packages (git, svn, ...) easy. It has two user-accessible main parts: * overlay creation, which accepts R packages as input and creates a portage overlay for them * repository management, download R packages from remotes and use them as input for overlay creation The minimal requirement for downloading packages is that a remote offers http access to its packages. The preferred way is rsync, which is used for CRAN and BIOC. The http support has been added later to include repos like R-Forge and Omegahat. Overlay creation is able to work incrementally so that existing ebuilds don't have to be recreated. It involves several tasks: * reading R package metadata and fixing errors like misspelled data fields along the way, e.g. 'Depents' is read as 'Depends'. Package reading is configurable. * ebuild creation, which tries to create an ebuild for an R package using its metadata -> dependency resolution that creates correct DEPEND/RDEPEND ebuild variables. It's realized by a dictionary approach extended by version-relative lookups. * overlay writing -> per-Package metadata.xml/Manifest creation Currently, the ebuild creation success rate is slightly higher than 95%. Ca. 900 out of 32000 creations fail due to various reasons: os type not supported, dependency unresolvable, R package format not supported (.Z-compressed tarballs, ...). Extensive documentation is available at [0] and covers usage, configuration, installation, what to expect and how roverlay works. All in all, I accomplished most objectives of my proposal. Some have been dropped like getting packages via svn, some have been added like getting packages via http and the version-relative dependency resolution. What's really missing is the integration into Gentoo's infrastructure so that end-users can add the resulting overlay using Layman. This will hopefully happen in the near future. As for the future, I'll focus on adding features based on real world/production usage needs. At last, I'd like to thank Denis (Calchan), my mentor, for guidance throughout the last months. I don't tend to ask many questions, but whenever I had one, he was able to answer it ;) Overall, taking part in gsoc for Gentoo has been a good experience. [0] http://git.overlays.gentoo.org/gitweb/?p=3Dproj/R_overlay.git;a=3Dblob_= plain;f=3Ddoc/html/usage.html;hb=3DHEAD --=20 Regards, Andr=E9 E.