[gentoo-user] Binary package cruncher?

[gentoo-user] Binary package cruncher?

[Simon]

Hey there!

   I've purchased a linode some time ago because my hardware is very (*very*) 
old and it was taking very long to do updates.  Now, what I've done is I've 
setup my linode to be exactly the same as my host, portage-wise.  And on the 
linode, I have installed the sum of all packages that are installed on all my 
PCs at home...  this way, when doing a `emerge -uDN world` on the linode, all 
packages are updated for all my machines (no fancy scripts to compile each 
machine's pkg independantly, etc).

   So, this is a binary package maker (or cruncher as it does it really well!). 
  However, one detail that is interesting is that to make sure all compilations 
work fine, I decided to really install everything on the host, yes, including 
stuff that requires a X server and so on (which I never use on the host).

   This is fine with me except it's taking a lot of space to have all that 
around...  so, since my host is really fast, i thought it might not be a problem 
for me to build dependencies in a sandbox, build the program, keep only the 
binpkg and remove everything including the dependencies that "had to be 
installed" for proper compilation.

   Ideally, I would see my file /var/lib/portage/world to contain strictly the 
software used on the host (ie. apache, mysql, etc...) and another file like 
/var/lib/portage/binworld which would contain all that is in world plus all the 
other packages that are in the worlds of my other PCs.  Packages in binworld 
would be compiled, as well as their dependancies.  However, if a binpkg is 
available for a dependancy, then this one is used instead.  And this way, all 
the "ghost" dependancies and "ghost" packages will all be compiled exactly as it 
is now, with some overhead for decompressing the deps binpkg for each pkg 
compilation.

   This is not super efficient, but the way I've thought it, should be simple to 
do... a simple gentoo hack so to speak.  However, I'm wondering if anybody has 
suggestion for better ways to do this and if you could give me pointers to such 
projects.  Also keep in mind that I really really want to update ALL my PCs with 
a single `emerge -uDN world` on the host, then copy new pkgs (using rsync or 
other) to the PCs and do an `emerge -k -uDN world` on them.  Nothing more.  (So 
unless your suggestion is simpler than my current (fully installed) setup, 
please tell me!)

Thanks in advance!
   Simon

Re: [gentoo-user] Binary package cruncher?

[Alan McKinnon]

On Saturday 15 November 2008 19:40:50 Simon wrote:
>    This is not super efficient, but the way I've thought it, should be
> simple to do... a simple gentoo hack so to speak.  However, I'm wondering
> if anybody has suggestion for better ways to do this and if you could give
> me pointers to such projects.  Also keep in mind that I really really want
> to update ALL my PCs with a single `emerge -uDN world` on the host, then
> copy new pkgs (using rsync or other) to the PCs and do an `emerge -k -uDN
> world` on them.  Nothing more.  (So unless your suggestion is simpler than
> my current (fully installed) setup, please tell me!)

I would build all that stuff in a chroot. The logic is that the your buildhost 
system is not quite the same thing as the machine hosting the buildhost.

One thing you cannot get away from is that to build say X for your slow 
hardware, it has to be done on a machine that has all X's build dependencies 
fully installed and working. You might not want that on your production 
server. Some fancy tricks with bind-mounts into the chroot would let you 
share common stuff. Or, you could simply buy a bit more storage if you are 
running out. It's cheap enough and if you've gone to this much trouble 
already, some more storage would be minor

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Binary package cruncher?

[Peter Humphrey]

On Sunday 16 November 2008 10:05:53 Alan McKinnon wrote:
> On Saturday 15 November 2008 19:40:50 Simon wrote:
> >    This is not super efficient, but the way I've thought it, should be
> > simple to do... a simple gentoo hack so to speak.  However, I'm
> > wondering if anybody has suggestion for better ways to do this and if
> > you could give me pointers to such projects.  Also keep in mind that I
> > really really want to update ALL my PCs with a single `emerge -uDN
> > world` on the host, then copy new pkgs (using rsync or other) to the
> > PCs and do an `emerge -k -uDN world` on them.  Nothing more.  (So
> > unless your suggestion is simpler than my current (fully installed)
> > setup, please tell me!)
>
> I would build all that stuff in a chroot. The logic is that the your
> buildhost system is not quite the same thing as the machine hosting the
> buildhost.
>
> One thing you cannot get away from is that to build say X for your slow
> hardware, it has to be done on a machine that has all X's build
> dependencies fully installed and working. You might not want that on your
> production server. Some fancy tricks with bind-mounts into the chroot
> would let you share common stuff. Or, you could simply buy a bit more
> storage if you are running out. It's cheap enough and if you've gone to
> this much trouble already, some more storage would be minor

Another possibility is to use distcc. I haven't used it recently, but an 
earlier version seemed ok.

-- 
Rgds
Peter

Re: [gentoo-user] Binary package cruncher?

[Simon]

> I would build all that stuff in a chroot. The logic is that the your buildhost 
> system is not quite the same thing as the machine hosting the buildhost.

That's the big trick I was thinking about...

1) Make a proper development environment under a chroot, place the "binworld" on 
top of world.
2) emerge -k -uDN world
    (this will install, in the chroot env all dependencies using previously 
built binpkgs, if those dependencies don't exist, they will be built and binpkg 
will be made for everything that is built)
3) un-chroot properly (unbind directories, etc)
4) delete the chroot directory containing installed dependencies, not relevant 
to the production environment

This would build the packages and install nothing on the host.  But I'm not sure 
if chroot is really proper for this... maybe I would have to use a unionfs to 
make sure that nothing gets overwritten in /lib or gets installed in the real 
/lib for example...

After everything is built, i would just redo the `emerge -k -uDN world` on the 
host in the real production environment to update it.

> One thing you cannot get away from is that to build say X for your slow 
> hardware, it has to be done on a machine that has all X's build dependencies 
> fully installed and working. You might not want that on your production 
> server. Some fancy tricks with bind-mounts into the chroot would let you 
> share common stuff. Or, you could simply buy a bit more storage if you are 
> running out. It's cheap enough and if you've gone to this much trouble 
> already, some more storage would be minor

You're right and this is why I had installed X on my host, with all its deps, 
even though none of it was used.  But using the chroot (and/or unionfs trick) 
I'd be able to build it with its deps without really installing it on the prod 
environment.

However, diskspace on a VPS is very costly.  For example, one extra GB is 
2$/m... and with the cheapest VPS i have only 12GB now.  I'm sure my full 
production environment would not move much higher than 4GB (without counting 
/usr/portage) while it probably is filling my disk now...

So, basically, if gentoo is a meta-distribution, my host is "my" distribution 
and serves to build the packages that are then available to my PCs...

Thanks,
   Simon

Re: [gentoo-user] Binary package cruncher?

[Simon]

> Another possibility is to use distcc. I haven't used it recently, but an 
> earlier version seemed ok.

Distcc is used to speed compilation time.  But the point is that even if I set 
distcc hosts to not be on the localhost (ie, just compile remotely and only use 
localhost to manage the whole thing)...  it still takes way too long.  Plus, 
many packages don't compile with distcc, will compile localhost only.  Plus, 
many parts of the emerge are not distributed, like ./configure, etc...

I mean, installing a binpkg is already slow on my PCs, imagine the compilation 
(yea, i am very patient).  That's why I purchased the VPS to compile and make 
binpkg...  One day I might purchase a second VPS and set distcc between the two 
(a binpkg farm?)... but until then distcc is really near useless in my setup. 
Only exception to what I just said is with the kernel, which I always compile 
with distcc.

Thanks,
   Simon