[gentoo-dev] Building custom package for multi-arch/system

[gentoo-dev] Building custom package for multi-arch/system

[Beber]

[-- Attachment #1: Type: text/plain, Size: 2120 bytes --]

Hi,

This mail is continuation of a discussion last month on IRC on
#gentoo-portage. As I think it's larger than just portage array, here it
is.

I have one server dedicated for compilation (xm-compile), and many
applicative servers (xmred5, lemonhead, etc).

Currently I use the --buildpkg on xm-compile to build packages
emergeable by --getbinpkgonly. Everything is fine except that I'm not
able to --getbinpkg function of USE flags, CFLAGS, CHOST nor ACCEPT_KEYWORDS.

So all packages on all servers are build in the same way with same USE.
But I would like to avoid it. I have mx servers (exim) that need mysql
lib, so mysql USE is there. I have mysql servers so -minimal USE is
there. I don't want to install a full mysql on mx servers, I just need
mysql lib.

So I'd like to build custom packages for almost everything.

I don't want to build packages on servers for security reasons (somes
could reproach me that, but it's not the topic ;) and for a limitation
of disk usage.

So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
names function of USE, like : 
  CHOST="x86_64-pc-linux-gnu"
  ACCEPT_KEYWORDS="amd64"
  tcpdump

  USE="+chroot +ipv6 -ssl -test -samba -smi"
  is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2

  USE="+chroot +ipv6 +ssl -test +samba -smi"
  is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2

Like this, I can easily add on client side a portage bashrc to get the file
and place it in the good place to don't have portage to download it as
it would find the right file with good USE. This implies I need to
emerge 1 package per 1 package which is horrible after some times.

So, do you guys plan to implement a such thing ? That's one of the
features that is mostly missing imho. The principal miss in on client
side as I have tools to manage packages but would like to not have too
much specific scripts on client side.

Thanks,

-- 
Beber

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Building custom package for multi-arch/system

[Jacob Godserv]

(I'm not an official developer.)

On Thu, Jan 28, 2010 at 10:17, Beber <beber@meleeweb.net> wrote:
> So, do you guys plan to implement a such thing ? That's one of the
> features that is mostly missing imho. The principal miss in on client
> side as I have tools to manage packages but would like to not have too
> much specific scripts on client side.

This is also high on my "things to hack" list. (Not that I'd ever come
up with something acceptable, but it's worth a try. :)

I would love to see the whole PKGDIR variable vanish altogether and
merge binary builds into ebuilds, creating a sort of apt-get / portage
mixture, all with a "standard" ebuild setup. By allowing binaries to
take on all the functionality of ebuilds, features like USE flags,
multiple versions per repository, varying CFLAGs, and other such
features are a given. The binary can be downloaded via standard ebuild
methods, even.

Something to chew on.

-- 

    "For then there will be great distress, unequaled
    from the beginning of the world until now — and never
    to be equaled again. If those days had not been cut
    short, no one would survive, but for the sake of the
    elect those days will be shortened."

    Are you ready?

Re: [gentoo-dev] Building custom package for multi-arch/system

[Gilles Dartiguelongue]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

Hi,

I have a somewhat similar need and to satisfy it, I'm building packages
in a chroot that has all the common packages then one chroot per
variation. It's probably nowhere scalable enough for you though and
would most likely but much more efficient if package building could be
wrapped so that it builds the package for one setup, then reconfigures
the package and build it for the new setup without cleaning everything
in the middle. Some scripting could probably do that but then I think
this joins well with multiple ABIs discussions as it basically means
portage needs to be able to build the same package several times with
different configuration.

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Building custom package for multi-arch/system

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
> names function of USE, like : 
>   CHOST="x86_64-pc-linux-gnu"
>   ACCEPT_KEYWORDS="amd64"
>   tcpdump
> 
>   USE="+chroot +ipv6 -ssl -test -samba -smi"
>   is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
> 
>   USE="+chroot +ipv6 +ssl -test +samba -smi"
>   is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
A word of warning on this.

Simply capturing the USE flags is not enough, the libraries that a
binary is linked against can cause problems as well, as the dependency
imposed by linking is much stricter than the RDEPEND in the ebuild.

tcpdump for example has:
RDEPEND="... ssl? ( >=dev-libs/openssl-0.9.6m )"

But if you built and linked against openssl-0.9.7*, then upgraded to
openssl-0.9.8, you can expect breakage.

Similar things happened with binaries linked against libmysqlclient in
the past.

As to my own take on a solution to the problem for work stuff, we simply
maintain one chroot per unique environment (we only have 5 envs), and
build in the chroot, then use those binpkgs on the boxes.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] Building custom package for multi-arch/system

[Philipp Riegger]

On Thu, 28 Jan 2010 22:31:04 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> But if you built and linked against openssl-0.9.7*, then upgraded to
> openssl-0.9.8, you can expect breakage.

But can't this be kind of ignored and solves the Gentoo way? If the
binpkg does not work, don't install it and simply rebuild it (if not
stated otherwise on the command line)?

Philipp

Re: [gentoo-dev] Building custom package for multi-arch/system

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

On Fri, Jan 29, 2010 at 11:52:40AM +0100, Philipp Riegger wrote:
> On Thu, 28 Jan 2010 22:31:04 +0000
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> > But if you built and linked against openssl-0.9.7*, then upgraded to
> > openssl-0.9.8, you can expect breakage.
> But can't this be kind of ignored and solves the Gentoo way? If the
> binpkg does not work, don't install it and simply rebuild it (if not
> stated otherwise on the command line)?
The gentoo "way" in this case is revdep-rebuild, and you've already
stated that your servers do not have a compiler.

You need to either expose the linkage dependencies outside the binpkg,
or introspect it when you're about to install it to find that it's NOT
going to work.

zmedico/portage folk: Can we have a phase during binpkg install, that
lets us check ELF linkages before the files get moving to ${ROOT}?

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] Building custom package for multi-arch/system

[Zac Medico]

On 01/29/2010 01:44 PM, Robin H. Johnson wrote:
> zmedico/portage folk: Can we have a phase during binpkg install, that
> lets us check ELF linkages before the files get moving to ${ROOT}?

Yeah, portage-2.2 has a LinkageMap class that we can use to verify
that all the needed sonames are present.
-- 
Thanks,
Zac

Re: [gentoo-dev] Building custom package for multi-arch/system

[Bertrand Jacquin]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]

D'ar yaou 28 a viz Genver 2010 e 23 eur 31, « Robin H. Johnson » he deus skrivet :
> On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> > So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
> > names function of USE, like : 
> >   CHOST="x86_64-pc-linux-gnu"
> >   ACCEPT_KEYWORDS="amd64"
> >   tcpdump
> > 
> >   USE="+chroot +ipv6 -ssl -test -samba -smi"
> >   is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
> > 
> >   USE="+chroot +ipv6 +ssl -test +samba -smi"
> >   is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
> A word of warning on this.
> 
> Simply capturing the USE flags is not enough, the libraries that a
> binary is linked against can cause problems as well, as the dependency
> imposed by linking is much stricter than the RDEPEND in the ebuild.
> 
> tcpdump for example has:
> RDEPEND="... ssl? ( >=dev-libs/openssl-0.9.6m )"

Of course, but real *DEPEND could be added in the xpak part of tbz2 and
resolv depedencies by the way.

> But if you built and linked against openssl-0.9.7*, then upgraded to
> openssl-0.9.8, you can expect breakage.
> 
> Similar things happened with binaries linked against libmysqlclient in
> the past.
> 
> As to my own take on a solution to the problem for work stuff, we simply
> maintain one chroot per unique environment (we only have 5 envs), and
> build in the chroot, then use those binpkgs on the boxes.

I did that already, but it's a bit heavy to maintain and do much
repetitive (maybe it's not the good word) as many packages would be
build with the same USE (in the case of same arch, chost etc).

-- 
Beber

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Building custom package for multi-arch/system

[Philipp Riegger]

On Thu, 28 Jan 2010 16:17:41 +0100
Beber <beber@meleeweb.net> wrote:

> So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
> names function of USE, like : 
>   CHOST="x86_64-pc-linux-gnu"
>   ACCEPT_KEYWORDS="amd64"
>   tcpdump
> 
>   USE="+chroot +ipv6 -ssl -test -samba -smi"
>   is placed
> under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
> 
>   USE="+chroot +ipv6 +ssl -test +samba -smi"
>   is placed
> under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
> 

So these are the SHA1 sums of the packages? Nice.

I would like to see something like this:

With different CFLAGS, CHOST etc. the user should set a seperate PKGDIR.

USE-flags are encoded in the filename:

Sort USE-flags alphabetically, use a 1 for every enabled and a 0 for
every disabled USE-flag, make this array smaller using hex code ore
base64.

>   USE="+chroot +ipv6 -ssl -test -samba -smi"

110000

tcpdump-4.0.1_pre20090709+c0.tbz2

>   USE="+chroot +ipv6 +ssl -test +samba -smi"

111010

tcpdump-4.0.1_pre20090709+e2.tbz2

Problem with this is, if anything with the USEflags as we see it from
outside changes (USE-flags are added, deletes, renamed and therefore
change position), this gets none-unique, which can be fixed in several
ways.

Philipp

Re: [gentoo-dev] Building custom package for multi-arch/system

[Bertrand Jacquin]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

D'ar gwener 29 a viz Genver 2010 e 00 eur 23, « Philipp Riegger » he deus skrivet :
> On Thu, 28 Jan 2010 16:17:41 +0100
> Beber <beber@meleeweb.net> wrote:
> 
> > So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
> > names function of USE, like : 
> >   CHOST="x86_64-pc-linux-gnu"
> >   ACCEPT_KEYWORDS="amd64"
> >   tcpdump
> > 
> >   USE="+chroot +ipv6 -ssl -test -samba -smi"
> >   is placed
> > under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7b4416738.tbz2
> > 
> >   USE="+chroot +ipv6 +ssl -test +samba -smi"
> >   is placed
> > under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb495980bd.tbz2
> 
> So these are the SHA1 sums of the packages? Nice.

No, the SHA1 is USE var SHA1.

sha1sum <<< "$USE"

-- 
Beber

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Building custom package for multi-arch/system

[Max Arnold]

On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> So, do you guys plan to implement a such thing ? That's one of the
> features that is mostly missing imho. The principal miss in on client
> side as I have tools to manage packages but would like to not have too
> much specific scripts on client side.

I like the way it done in OpenEmbedded. You have the tree of recipes (think of portage tree)
and bunch of targets. For each target BitBake can generate binary release and package feed.
Client package management is lightweight and does not require BitBake, recipes tree and even
python. At least this is my lame interpretation of how it works :)

Maybe this "metadistribution" approach is cleaner than binary package support in emerge. If
user wants to compile packages on the client, he uses portage. If not - he can setup build
server for multiple targets and completely drop portage from client machines. The only thing
client should know is feed url with full list of binary packages. And I do not think client
should deal with USE flags - for large installations unification is the only sane way to scale.

Re: [gentoo-dev] Building custom package for multi-arch/system

[Antoni Grzymala]

Max Arnold dixit (2010-01-29, 12:24):

> On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> > So, do you guys plan to implement a such thing ? That's one of the
> > features that is mostly missing imho. The principal miss in on
> > client side as I have tools to manage packages but would like to not
> > have too much specific scripts on client side.
> 
> I like the way it done in OpenEmbedded. You have the tree of recipes
> (think of portage tree) and bunch of targets. For each target BitBake
> can generate binary release and package feed. Client package
> management is lightweight and does not require BitBake, recipes tree
> and even python. At least this is my lame interpretation of how it
> works :)
> 
> Maybe this "metadistribution" approach is cleaner than binary package
> support in emerge. If user wants to compile packages on the client, he
> uses portage. If not - he can setup build server for multiple targets
> and completely drop portage from client machines. The only thing
> client should know is feed url with full list of binary packages. And
> I do not think client should deal with USE flags - for large
> installations unification is the only sane way to scale.

I think something similar is very nicely done on R-Path based
distributions (with "flavours" [1]). Conary itself also seems to be a
pretty well thought out package manager.

[1] http://docs.rpath.com/conary/Conaryopedia/sect-flavors.html

-- 
[a]

Re: [gentoo-dev] Building custom package for multi-arch/system

[Zac Medico]

On 01/28/2010 09:24 PM, Max Arnold wrote:
> On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
>> So, do you guys plan to implement a such thing ? That's one of the
>> features that is mostly missing imho. The principal miss in on client
>> side as I have tools to manage packages but would like to not have too
>> much specific scripts on client side.
> 
> I like the way it done in OpenEmbedded. You have the tree of recipes (think of portage tree)
> and bunch of targets. For each target BitBake can generate binary release and package feed.
> Client package management is lightweight and does not require BitBake, recipes tree and even
> python. At least this is my lame interpretation of how it works :)

You can do something similar using the emerge --config-root option.
You'd just use a different --config-root for each target, and each
of those would have a separate $PKGDIR.

> Maybe this "metadistribution" approach is cleaner than binary package support in emerge. If
> user wants to compile packages on the client, he uses portage. If not - he can setup build
> server for multiple targets and completely drop portage from client machines. The only thing
> client should know is feed url with full list of binary packages. And I do not think client
> should deal with USE flags - for large installations unification is the only sane way to scale.

The clients need sys-apps/portage installed, but not the whole
portage tree (although the profiles directory can be useful for
the profile and package moves). The clients should set
PORTAGE_BINHOST in make.conf, so that binary packages are
automatically downloaded with the emerge -g option.
-- 
Thanks,
Zac

Re: [gentoo-dev] Building custom package for multi-arch/system

[Max Arnold]

On Sat, Jan 30, 2010 at 03:59:27AM -0800, Zac Medico wrote:
> On 01/28/2010 09:24 PM, Max Arnold wrote:
> > On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> >> So, do you guys plan to implement a such thing ? That's one of the
> >> features that is mostly missing imho. The principal miss in on client
> >> side as I have tools to manage packages but would like to not have too
> >> much specific scripts on client side.
> > 
> > I like the way it done in OpenEmbedded. You have the tree of recipes (think of portage tree)
> > and bunch of targets. For each target BitBake can generate binary release and package feed.
> > Client package management is lightweight and does not require BitBake, recipes tree and even
> > python. At least this is my lame interpretation of how it works :)
> 
> You can do something similar using the emerge --config-root option.
> You'd just use a different --config-root for each target, and each
> of those would have a separate $PKGDIR.

Can someone share his experience of using Catalyst or Metro as generator of binary package
updates for precompiled client machines?  How smooth it is in the long run?

Re: [gentoo-dev] Building custom package for multi-arch/system

[Jacob Godserv]

On Sat, Jan 30, 2010 at 10:13, Max Arnold <lwarxx@gmail.com> wrote:
> Can someone share his experience of using Catalyst or Metro as generator of binary package
> updates for precompiled client machines?  How smooth it is in the long run?

I use metro (wrote a little HOWTO on it: <
http://neuvoo.org/wiki/index.php?title=Create_Image/Building >) and it
asks emerge to --buildpkg at every stage. It keeps binaries separate
per stage. The end result, as long as you told it not to clean, is
three folders with a binary repo in each. It will re-use these
binaries in future stage builds, if you don't clean it out.

If you don't want it building stages, it's very easy to configure
metro to do exactly what you want. It still has to be figured out to
some extent, but the format is much simpler than catalyst and takes
less time to customize properly.

I'm not sure what "long run" looks like as I have yet to enter the
"long run". As far as I can tell, metro will always build and update
binary repositories when you ask it to. ;)

-- 
    Jacob

    "For then there will be great distress, unequaled
    from the beginning of the world until now — and never
    to be equaled again. If those days had not been cut
    short, no one would survive, but for the sake of the
    elect those days will be shortened."

    Are you ready?

[gentoo-dev] Re: Building custom package for multi-arch/system

[Duncan]

Zac Medico posted on Sat, 30 Jan 2010 03:59:27 -0800 as excerpted:

> The clients need sys-apps/portage installed, but not the whole portage
> tree (although the profiles directory can be useful for the profile and
> package moves). The clients should set PORTAGE_BINHOST in make.conf, so
> that binary packages are automatically downloaded with the emerge -g
> option.

Thanks.  I knew the PORTAGE_BINHOST and --config-root bits, but hadn't 
thought about the tree not being needed, save for the profiles.  I 
recently setup a netbook based on a Gentoo image compiled elsewhere, and 
while it's now functional, I'm still tweaking stuff, including the portage 
arrangement.   That I don't need the entire gentoo tree for emerging 
binpkgs is thus quite interesting new information at this point. =:^)

Does that happen to be documented anywhere?  I'd not seen it before.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Building custom package for multi-arch/system

[Zac Medico]

On 01/30/2010 10:11 AM, Duncan wrote:
> Thanks.  I knew the PORTAGE_BINHOST and --config-root bits, but hadn't 
> thought about the tree not being needed, save for the profiles.  I 
> recently setup a netbook based on a Gentoo image compiled elsewhere, and 
> while it's now functional, I'm still tweaking stuff, including the portage 
> arrangement.   That I don't need the entire gentoo tree for emerging 
> binpkgs is thus quite interesting new information at this point. =:^)
> 
> Does that happen to be documented anywhere?  I'd not seen it before.

No, not really. The binhost support was updated in portage-2.1.6,
with support for the $PKGDIR/Packages file. Since then it's a lot
more appealing.
-- 
Thanks,
Zac