From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 13B3B138202 for ; Thu, 29 Nov 2012 11:30:24 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 35A1221C005; Thu, 29 Nov 2012 11:30:18 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 3F69A21C005 for ; Thu, 29 Nov 2012 11:30:17 +0000 (UTC) Received: from pomiocik.lan (unknown [81.219.163.24]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: mgorny) by smtp.gentoo.org (Postfix) with ESMTPSA id 74B2D33D8E5; Thu, 29 Nov 2012 11:30:15 +0000 (UTC) From: =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?= To: gentoo-python@lists.gentoo.org Cc: python@gentoo.org Subject: [gentoo-python] distutils-r1: a bit of clean up + parallel builds Date: Thu, 29 Nov 2012 12:31:28 +0100 Message-Id: <1354188694-7229-1-git-send-email-mgorny@gentoo.org> X-Mailer: git-send-email 1.8.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Discussions centering around the Python ecosystem in Gentoo Linux X-BeenThere: gentoo-python@gentoo.org X-BeenThere: gentoo-python@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Archives-Salt: ddb0475b-7705-47ce-bd64-28e78b744d02 X-Archives-Hash: 7e90048f9e5463e47853b269798d727d 1) setup.py installs files to intermediate root (alike python.eclass). This way anything we do on the installed files doesn't collide with other merges potentially running in parallel. This also means that we don't have to delay installing the wrapper till all setup.py invocations have completed. This is done directly in distutils-r1_python_install. The setup.py is given a different --root, the renaming is done on intermediate image and the image is quickly merged to the destination. In order to perform the merge efficiently, I used: cp --archive --link --no-clobber so that copy should preserve everything and use hard-links whenever possible. --no-clobber is necessary to avoid error on colliding files (cp doesn't want to overwrite when hardlinking). 2) the wrapper is installed in distutils-r1_python_install. Previously, distutils-r1_python_install only renamed the installed executables (because of distutils no-clobber behavior), and distutils-r1_python_install_all installed the wrapper. Now we can install both in the same function, since distutils installs into intermediate images. Therefore, the wrapper being installed in another intermediate image or even the real image won't collide. 3) the sub-phases are run in parallel. Since distutils itself is unable to do parallel builds, building Python packages with C extensions for multiple Python implementations can get very slow. In order to circumvent that, we're using the multiprocessing eclass to run sub-phases in parallel. This means that with 4 implementations enabled and -j4, all four implementations will be built at the same time. And if they have C extensions, 4 source files will be built at the same time. This also makes it possible to use distcc. As stated in the last patch: dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3: - non-parallel: 11 min 23 sec - parallel: 7 min 49 sec (with a bit of swapping) - parallel w/ distcc: 3 min 40 sec main machine: Core2 2x1.6 GHz and almost 2 GiBs of RAM distcc host: Athlon64 2x2 GHz and 3 GiBs of RAM