From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1PteQq-0007TD-DO for garchives@archives.gentoo.org; Sun, 27 Feb 2011 11:04:56 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 095D71C057 for ; Sun, 27 Feb 2011 11:04:55 +0000 (UTC) Received: from mail1.nippynetworks.com (mail1.nippynetworks.com [91.220.24.129]) by pigeon.gentoo.org (Postfix) with ESMTP id EB72A1C013 for ; Sun, 27 Feb 2011 10:43:26 +0000 (UTC) Received: from localhost (mail1.nippynetworks.com [127.0.0.1]) by mail1.nippynetworks.com (Postfix) with ESMTP id 476CD340305 for ; Sun, 27 Feb 2011 10:43:26 +0000 (GMT) X-Virus-Scanned: amavisd-new at nippynetworks.com Received: from mail1.nippynetworks.com ([127.0.0.1]) by localhost (mail1.nippynetworks.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id NSAmH97MO-kD for ; Sun, 27 Feb 2011 10:43:26 +0000 (GMT) Received: from Ed-Wildgooses-MacBook-Pro.local (office.nippynetworks.com [212.69.49.94]) (Authenticated sender: edward@wildgooses.com) by mail1.nippynetworks.com (Postfix) with ESMTPSA id AB78234030B for ; Sun, 27 Feb 2011 10:43:25 +0000 (GMT) Message-ID: <4D6A2ACB.5010406@wildgooses.com> Date: Sun, 27 Feb 2011 10:43:23 +0000 From: Ed W User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] avoiding urgent stabilizations References: <4D501BA4.6040802@gentoo.org> <20110221001106.GB22774@nibiru.local> <4D678390.7050109@wildgooses.com> <201102250308.21246.matt@deploylinux.net> <4D679480.2000508@wildgooses.com> <20110225175310.GG22774@nibiru.local> <4D68E7E1.1050109@wildgooses.com> <20110226155750.GH22774@nibiru.local> In-Reply-To: <20110226155750.GH22774@nibiru.local> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable X-Archives-Salt: X-Archives-Hash: 34c2911a3c985a6713c88359c4f6ffa6 On 26/02/2011 15:57, Enrico Weigelt wrote: > * Ed W schrieb: > >> I'm just building some embedded devices on the side using >> gentoo and my minimal builds are only a few MB? > How to do you get out all the buildtime stuff (portage, toolchain, etc)= ? > >> Seems like your complaint is that you have gentoo installs which are >> full featured with a toolchain and portage, which you are comparing to= >> an installation you built with a different tool that doesn't have a >> toolchain installed? > Well, with Briegel, I'm strictly differenciating between the build > host and the target image. Everything's crosscompiled by design. > So I can explicitly define which packages need to be in the final > image (and only those are built). The build host can by any fairly > recent GNU/Linux system (happens to be Gentoo at my site) I think you are missing the point that the gentoo build system is quite=20 happily building you a destination build with as little as one package=20 in it? Whether that destination boots is a function of careful package=20 choice, but the point is you are definitely not constrained from=20 building a highly minimal target if you wish? Just from the top of my head I should imagine this would likely boot: mkdir -p /newroot/{proc,dev,sys} cp -a /dev/{null,zero} /newroot/dev/ emerge =97root /newroot glibc shadow baselayout openrc util-linux udev If you want to go more minimal then I have a uclibc chroot that I use as = a build environment, the first line of my build script for my minimal=20 embedded device is simply: emerge --root /embedded baselayout uclibc busybox I haven't checked, but give or take creating /proc, /sys and some=20 minimal devices in /dev, the above should boot fine. If you were booting into a "container" based virtualisation then you=20 don't need to manage /dev and mounting devices and so you can get even=20 more minimal than the top example I posted (ie I don't think you need=20 udev, etc) Obviously such minimal builds will have minimal package management, but=20 this may not be a problem if you have a super fixed configuration?=20 Simply construct your environment such that the data is separate from=20 the OS and also make sure any OS customisation is scriptable and now you = would have a way to occasionally build a new image which you rolled out=20 to all virtual machines >> 1) Build your base images without a toolchain or portage and use a >> minimal package installer to install pre-built binary packages. >> This seems fraught with issues long term though... > How can this be done w/ Gentoo ? > AFAIK it always has a system set containing things like portage, > toolchain, etc, etc. "System" is just a statement of some defaults. At the level you are=20 working at you can largely ignore it. Build what you like into the target build.... > Would be still too much for my goals: I don't wanna have anything in > the final system that's not necessary for runtime. Building and even > installation should be driven from outside (actually, the final system > could even have /bin, /usr, etc mounted ro and /var w/o noexec) OK, so sounds like a simple build script like the one above would suit yo= u? For more advanced use consider learning to use catalyst or Funtoos build = system? These are full on scriptable ways to repeatably build non=20 trivial target builds. (remember they can build as little as single=20 package targets or more complicated builds) > Note that Gentoo's binary packages (which are practically snapshots > of what will be copied in by merge stage) are alway bound to specific > domains, eg. the full configuration (make.conf, useflags, etc, etc), > and there may be a lot of interdependencies between installed packages > (and versions), so it takes some care to use it in stable ways. > > This might be very fine for you (eg. if you have dozens of really > equal systems), but you'll have to be aware of the limitations. Well, your requirements suggest that you don't need a package manager=20 and you will instead build a new read-only OS build every so often and=20 deploy that. However, in the interests of exploring the idea further: - By definition binary packages only work where you have equal systems?=20 This is true of gentoo AND rpm/deb. - If you need binary packages for non equal systems then you need some=20 kind of namespacing method to separate the binaries. Gentoo has no=20 automated way to do this, but it's reasonably workable to have binary=20 package repos and simply tell your guest machine to use the appropriate=20 binary repo - Gentoo's binaries are slightly cleverer than some notice in that they=20 have some awareness of when they don't match the target and the operator = has the option to either ignore that and continue the install, OR to=20 ignore the binary package and generate it's own binaries using the local = compiler, etc. So, I don't really see the problem, in fact Gentoo binaries can be seen=20 as slightly more flexible than RPMs to some extent if you allow them to=20 be "optional". > As long as you have the same make.conf/useflag/etc settings for all > systems (within a guest type), this might work well. But that's true of Redhat/Debian/Gentoo? RPMs don't allow you to have a single binary repo where some of the=20 packages are linked against perl 5.8 and some against 5.12? I guess it=20 would be possible to enhance the packaging format to cater for that, but = largely the whole premise of binary packages is that you lock down your=20 target options and build all the systems identically? There is no easy=20 way around this? (In fact it's what you want) > At this point, Gentoo's binary packages could become tricky. > The big questions eg. are: can it handle multiple binpkgs of the > same (source) package with differing useflags ? How are version- > specific dependencies handled, etc, etc. As it stands, no it can't. However, neither can RPM/DEB... Remember the premise of binaries is that they are all the same... Pick a = system configuration and build all your binaries with that=20 configuration. If you need to support multiple configurations then setup = multiple binary repositories >> articles, but perhaps start with: >> http://en.gentoo-wiki.com/wiki/Tiny_Gentoo >> >> Roughly speaking you could "freshen" your current installation with >> (from memory): >> ROOT=3D"/tmp/new_build" emerge -av world > Is this just the installation root or also the sysroot for the > toolchain ? I think you need to read through the tiny_gentoo type wiki pages. The=20 general build process is: 1) Start with a working machine 2) Build a build environment 3) Jump into the build environment and repeat step 2) to build a target=20 environment 4) Jump into the target environment Now in the absence of cross compiling you can often skip step 2) and=20 build the target environment directly, but it's normally useful to have=20 that build environment so that you can customise things a bit. Also this will help explain some of the catalyst recipes and why you see = everything built twice, once to get a build environment, then again to=20 build the target build >> Further, if you wish you can make those VMs have a reduced or missing >> toolchain etc. > hmm, do I have to unmerge certain packages explicitly or can I > tell it to only install those packages required for runtime in > the first place ? Well your really simplest "build" would be to take a stage3/4 and simply = "unmerge" anything you don't need... bit crude though However, from the email above you can see that it's more desirable to=20 instead "build up", starting with an empty build and from there add the=20 packages you actually want. >> You are almost thinking about it all wrong. ("There is no spoon...") >> >> This is gentoo, so at this more advanced level, stop thinking about >> "standard system set" and instead free your mind to start with >> "nothing". > With "standard system set" I mean those packages which are in @system > on a particular arch. In my vision, @system would be simply empty, > instead @world tells which packages you want and the dependencies > are pulled in automatically. Seriously, "there is no spoon"... If you want to make your own build script then there isn't even an=20 @system... > No idea if that's valid anymore, but some (long?) time ago, there > was the rule that individual packages weren't allowed to explicitly > DEPEND=3D on system packages. Obviously this wouldn't work with an > empty system set (you'd have to find out the deps and add them > to world set manually). I believe this is correct. However, there aren't many packages in system = and so whilst it's kind of buyer beware once you start chopping stuff=20 out of system, the point is only if you don't install gzip or gcc then=20 you need to beware stuff which depends on either. I think in practice manually managing those dependencies is going to be=20 very straightforward, ie you are more going to fix the applications to=20 not require the dependencies. After all if you don't want gcc, then you=20 really don't want gcc, simple really... eg, if some simple application=20 uses make to update it's config files then you will likely look at some=20 workaround rather than installing make... > Well, that's where it gets complicated: if some packages depend on some= > library libfoo in @system (which I would empty in this case), portage > lacks information to compute the full dependency tree, so I'd have to > find them out and add them to @world manually. Exactly what I do not wa= nt. Oh come on. You are working at the level of building your OS image from=20 scratch and you don't know if any of your applications require (at=20 runtime) gcc, bison, m4, wget, man, os-headers, portage or ssh? It seems likely that you can quickly determine if you can live without=20 most of the above (at runtime)...?!! > OTOH, Briegel doenst have that issue, as all dependencies are stated > explicitly in each package (well, with one exception: libc). So, you can probably already use that information to figure out your=20 minimal set of system packages that you likely need (for runtime). Just checking that Briegel tracks both runtime and build time=20 dependencies separately? eg you might need gzip to build something, but=20 not to run it? So far I think you could easily convert your briegel recipes to a simple = ebuild script and run them side by side? It's actually such an easy=20 thing to do, to build a new stage2/3/4 that it's a shame folks don't do=20 it more! Good luck Ed W