From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1M1PDi-0006vd-SX for garchives@archives.gentoo.org; Tue, 05 May 2009 18:18:23 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id A942EE084E; Tue, 5 May 2009 18:18:19 +0000 (UTC) Received: from amun.cheops.ods.org (amun.cheops.ods.org [82.95.138.191]) by pigeon.gentoo.org (Postfix) with ESMTP id 246E4E084E for ; Tue, 5 May 2009 18:18:19 +0000 (UTC) Received: from tefnut.cheops.ods.org ([2001:888:1022:0:211:24ff:fe37:e46e] helo=gentoo.org) by amun.cheops.ods.org with esmtps (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1M1PDc-0004cB-Ld for gentoo-dev@lists.gentoo.org; Tue, 05 May 2009 20:18:18 +0200 Date: Tue, 5 May 2009 20:18:14 +0200 From: Fabian Groffen To: gentoo-dev@lists.gentoo.org Subject: [gentoo-dev] An Introduction to Gentoo Prefix Message-ID: <20090505181814.GQ29207@gentoo.org> Mail-Followup-To: gentoo-dev@lists.gentoo.org Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.19 (Darwin 8.11.0, VIM - Vi IMproved 7.2) Organization: Gentoo Foundation, Inc. X-Content-Scanned: by amun.cheops.ods.org (Exim Exiscan) using SpamAssassin and ClamAV X-Archives-Salt: e6d488b9-8243-4af0-a20a-e2d8c3819487 X-Archives-Hash: 6cfb555437e46cc41e3b9ed80433bc18 While some of you may have heard of it, we -- the Prefix team -- have the impression that for most developers the Gentoo Prefix project is in general an unknown, hidden and vague project, that primarily generates a lot of commits. Therefore, we have decided to try and explain what Gentoo Prefix is, and why we Prefix devs are so passionately about it. Let us start by pointing out that Gentoo Prefix is all but a "new" project. While the archeological signs of the project only go back to 2006[1], the project has existed before that in the minds of people no longer with us any more. The Gentoo Prefix project aims to bring the Gentoo Portage Tree to non-Gentoo and/or non-Linux systems, through the following two characteristics: 1) a free to choose target directory offset (prefix), in which packages are installed, and 2) the ability to install and operate without adminstrative privileges To understand why these two characteristics were chosen, it is best to get into the mindset of many Gentoo Prefix users (and devs). There are plenty of reasons, of which a few described in [2]. In short, the users are looking for convenience on their platform of choice, or platform they are doomed to use. Some people just have to work with Windows. Others just like Mac OS X over any GNOME or KDE. Some work with Solaris, others with AIX, or even their Atari. All of these systems have their merits and demerits. (Gentoo) Linux users are used to a very rich and convenient userland consisting of many GNU tools, such as sed, awk, grep, tar, etc., but also development tools such as compilers, autotools, etc. While these tools are absent, simply outdated or not as flexible as on Gentoo, users of said systems are in need of the flexible tool suite as delivered by Gentoo. An obvious solution is to get Portage running and to install the tools on the system in use. However, due to requirements, privileges, or an insane love relation, the original system may not be harmed by replacing or modifying existing software. Yes, in some cases, such modifications are even impossible by lack of privileges: no root access! In a really compact form, we can say that for Gentoo Prefix users: - an offset installation is necessary not to break the host OS software - sometimes root privileges are unavailable - unprivileged installations add an extra level of protection/safety - `rm -Rf ` removes the entire Prefix and all traces of it Gentoo Prefix delivers unprivileged offset installations, which we call a "prefix". The prefix here is the offset used for the entire installation, such as /home/sally/gentoo. A Gentoo Prefix system is bootstrapped into such prefix, by following a number of instructions, e.g.[3]. Using this prefix, the file system layout chosen by Prefix Portage is exactly the same as for a normal Gentoo Linux system, but shifted into the prefix. This would for instance result in /home/sally/gentoo/usr/bin for the previously mentioned example. In principle, it serves no use to have programs installed in bin, usr/bin, sbin, usr/sbin, etc. under the prefix, and it would make more sense if they would be installed all in one place. However, following the Gentoo Linux layout enables direct backwards compatability when the chosen prefix is the empty string, and requires no extra patching or changes. With this in mind, we would like to stress that it is not the desire of the current Gentoo Prefix project to go beyond the ability to install in an unprivileged install-time fixed offset. We consider the current approach as a milestone that delivers offset installations. Pilots within our own team have shown that on top of this milestone, further steps can be made, such as variable offset locations and close cooperation with e.g. a Gentoo Linux host system. However, we do not consider these works in progress as part of the *current* Gentoo Prefix project. Unfortunately, support for installing packages into a prefix, does not come for free. Most ebuilds need small changes, some ebuilds need extensive patching to get programs to install and run correctly with regard to the (offset) prefix. To get this all running, we set up our own tree, which contains all our "converted" ebuilds, extra profiles and modified eclasses[4]. At some point our tree became pretty big[1], and it turned out our enthusiasm (and that of our users) brought overlays.g.o down to its knees. At that point, our hobby got way out of hand, and to resolve the issue we had to switch to rsync, resulting in our own generated tree with metadata. As side-effect this greatly sped up the user experience[5], contributing to getting our hobby even more out of hand. We, as Prefix team, feel this puts our project in a bit weird position, where we resemble closely to an in-house fork of the Gentoo project itself. It is our desire and intention to move away from this situation, which not only consumes an awful lot of resources for maintaining the fork, but also keeps a lot of work outside of the mainstream Gentoo infrastructure. As Prefix team, we feel that the current shape of the Gentoo Prefix implementation is mature. That is, it has been proven to be useful for a number of users/scenarios, and it has been able to work for a substantial number of different systems, without having to change the core part. Therefore, we plan to focus on merging back the many patches and extensions from our Prefix overlay into the Gentoo mainline tree. For us, this roadmap looks as follows: 1) get the prefix USE-flag masked in profiles/base We use a special USE-flag 'prefix' to conditionalise actions in ebuilds and eclasses that really depend on being in an offset installation or not. Obviously, this flag should be disabled in normal profiles. 2) move the prefix profiles to the mainline tree The Prefix project maintains a profiles/prefix directory with Gentoo Prefix specific profiles. They inherit from base, and in the Linux case even from the default/linux profiles. 3) add prefix-only ebuilds to the mainline tree Gentoo Prefix has different systems, which sometimes need different packages. Specialised ebuilds now live in the Prefix tree only. 4) file bugs for eclass changes to their maintainers Like ebuilds, eclasses need changes to work properly using an offset installation. These changes are usually related to dealing with the offset prefix, but sometimes also add specialised support for the Prefix architectures. 5) bring back trivial patches Different systems bring different compile or installation problems. These patches can be trivial, and hence non-intrusive. We like to move those patches to the Gentoo mainline tree. 6) solve complicated ebuild changes For a number of ebuilds, mostly for the toolchain, a set of extensive patches are applied in the Prefix tree. While our changes are mostly designed to be backwards compatible, we prefer to get them reviewed by the respective maintainers and get them merged with their blessings. 7) merging prefix branch of Portage into trunk Prefix Portage currently lives in its own branch of the Portage sources. With an empty offset string, Prefix Portage is equal in behaviour to trunk sources. We would like to merge the Prefix branch with trunk and have a release, instead of keeping the branch and adding a Prefix Portage ebuild. 8) add prefix bits to ebuilds Prefix involves some variables which need to be used in ebuilds to make them offset-aware. Often this is trivial, but requires understanding on what is exactly going on. We plan to provide information about this. At the same time we do this, we want to move over our Prefix arches keywords, such that we can start using the Gentoo mainline tree. We hope this informative email sheds a light on the activities of the Gentoo Prefix team, and the actions we aim to take in the near future. We are open for any questions or comments. [1] http://stats.prefix.freens.org/keywords-packages.png [2] http://www.gentoo.org/proj/en/gentoo-alt/prefix/usecases.xml [3] http://www.gentoo.org/proj/en/gentoo-alt/prefix/bootstrap-solaris.xml [4] http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay [5] http://stats.prefix.freens.org/rsync-usage.png -- Fabian Groffen Gentoo on a different level