From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <gentoo-python+bounces-256-garchives=archives.gentoo.org@lists.gentoo.org>
Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80])
by finch.gentoo.org (Postfix) with ESMTP id 579DA1381F3
for <garchives@archives.gentoo.org>; Thu, 29 Nov 2012 16:29:45 +0000 (UTC)
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
by pigeon.gentoo.org (Postfix) with SMTP id 8AFCC21C076;
Thu, 29 Nov 2012 16:29:40 +0000 (UTC)
Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183])
(using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by pigeon.gentoo.org (Postfix) with ESMTPS id EDCC121C076
for <gentoo-python@lists.gentoo.org>; Thu, 29 Nov 2012 16:29:39 +0000 (UTC)
Received: from mail-ee0-f53.google.com (mail-ee0-f53.google.com [74.125.83.53])
(using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
(No client certificate requested)
(Authenticated sender: floppym)
by smtp.gentoo.org (Postfix) with ESMTPSA id E0C4733DAF8
for <gentoo-python@lists.gentoo.org>; Thu, 29 Nov 2012 16:29:38 +0000 (UTC)
Received: by mail-ee0-f53.google.com with SMTP id c50so8829664eek.40
for <gentoo-python@lists.gentoo.org>; Thu, 29 Nov 2012 08:29:36 -0800 (PST)
Precedence: bulk
List-Post: <mailto:gentoo-python@lists.gentoo.org>
List-Help: <mailto:gentoo-python+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-python+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-python+subscribe@lists.gentoo.org>
List-Id: Discussions centering around the Python ecosystem in Gentoo Linux <gentoo-python.gentoo.org>
X-BeenThere: gentoo-python@gentoo.org
X-BeenThere: gentoo-python@lists.gentoo.org
MIME-Version: 1.0
Received: by 10.14.225.72 with SMTP id y48mr79885513eep.46.1354206576153; Thu,
29 Nov 2012 08:29:36 -0800 (PST)
Received: by 10.14.94.207 with HTTP; Thu, 29 Nov 2012 08:29:35 -0800 (PST)
In-Reply-To: <1354188694-7229-1-git-send-email-mgorny@gentoo.org>
References: <1354188694-7229-1-git-send-email-mgorny@gentoo.org>
Date: Thu, 29 Nov 2012 11:29:35 -0500
Message-ID: <CAJ0EP42fXdG+pq=eMfTg8QktynWVNhZa60oga_Jp5Rw1dzvTDA@mail.gmail.com>
Subject: [gentoo-python] Re: distutils-r1: a bit of clean up + parallel builds
From: Mike Gilbert <floppym@gentoo.org>
To: gentoo-python@lists.gentoo.org
Cc: python@gentoo.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Archives-Salt: c4d6503d-3af9-40fe-a10e-68f248286e05
X-Archives-Hash: 28ca800c2f167d970b5e281c0cb29c65
On Thu, Nov 29, 2012 at 6:31 AM, Micha=C5=82 G=C3=B3rny <mgorny@gentoo.org>=
wrote:
> 1) setup.py installs files to intermediate root (alike python.eclass).
>
> This way anything we do on the installed files doesn't collide with
> other merges potentially running in parallel. This also means that we
> don't have to delay installing the wrapper till all setup.py invocations
> have completed.
>
> This is done directly in distutils-r1_python_install. The setup.py is
> given a different --root, the renaming is done on intermediate image
> and the image is quickly merged to the destination.
>
> In order to perform the merge efficiently, I used:
>
> cp --archive --link --no-clobber
>
> so that copy should preserve everything and use hard-links whenever
> possible. --no-clobber is necessary to avoid error on colliding files
> (cp doesn't want to overwrite when hardlinking).
>
>
> 2) the wrapper is installed in distutils-r1_python_install.
>
> Previously, distutils-r1_python_install only renamed the installed
> executables (because of distutils no-clobber behavior),
> and distutils-r1_python_install_all installed the wrapper.
>
> Now we can install both in the same function, since distutils installs
> into intermediate images. Therefore, the wrapper being installed
> in another intermediate image or even the real image won't collide.
>
>
> 3) the sub-phases are run in parallel.
>
> Since distutils itself is unable to do parallel builds, building Python
> packages with C extensions for multiple Python implementations can get
> very slow. In order to circumvent that, we're using the multiprocessing
> eclass to run sub-phases in parallel.
>
> This means that with 4 implementations enabled and -j4, all four
> implementations will be built at the same time. And if they have C
> extensions, 4 source files will be built at the same time. This also
> makes it possible to use distcc.
>
> As stated in the last patch:
>
> dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3:
>
> - non-parallel: 11 min 23 sec
> - parallel: 7 min 49 sec (with a bit of swapping)
> - parallel w/ distcc: 3 min 40 sec
>
> main machine: Core2 2x1.6 GHz and almost 2 GiBs of RAM
> distcc host: Athlon64 2x2 GHz and 3 GiBs of RAM
>
I was just thinking to myself last night that parallel builds/installs
would be nice. You must be psychic.
My thought was to add a flag/variable for python_foreach_impl, but
your method should work just as well.
Building the temporary image in sub-directories of ${D} feels a bit
strange, but I guess it works. I think distutils.eclass used ${T}?
Anyway, +1 from me.