[gentoo-dev] Berkeley DB, coexistence of different versions

[gentoo-dev] Berkeley DB, coexistence of different versions

[Stefaan]

Hi all!!

Here's an issue Seemant and I have been struggling with, and doesn't
seem as easy to solve as like touching one ebuild.

Berkeley DB comes in many flavors.  Judging by the slots in the
ebuilds (1, 3, 4, 4.1, 4.2, 4.3) there's many different interface
versions.  This comes with the added fun of having packages depend on
certain versions of DB.  Nothing shocking so far.

However, looking at the heimdal-package we noticed several problems
(which seem to be more widespread than just heimdal):
1) The package won't build when both db-4 and db-3 are installed (bug
#100723).  Underlying reason is that the package only knew about
db-4.0, and thus assumed the include for db-4.x to be called
<db4/db.h>.  Because the installed db-4.2 didn't have this name for
its include, and because the package was still trying to link with a
4.x library, this gave problems.
2) Detecting of available version of db-includes and db-libraries is
done separately.  So we have a first algorithm looking for the most
recent include files (out of a list of possibilities known at the time
the package is created), and a second algorithm doing the same for the
libraries.  This is a reason the error in (1) was possible.
3) In an attempt to circumvent the problem, we could start hard coding
packages to require a specific db-version. But this would have
distinct disadvantages: a) as it seems logical to depend on the latest
version available at ebuild-creation time, installing a system with
ebuilds of different ages results in a system with many db-versions,
even when it's not necessary; b) we undo the effort put in a package
to make it compile with different db-versions
4) Note that even then, the problem in (1) hasn't disappeared, as the
configure-scripts of the package still crawl around the system to find
a suitable include-library combo and, not knowing the full truth,
select an unsuitable one.  There even seems to be a disagreement (on
upstream level) on whether libdb3.so.* or libdb-3.so.* is the
preferred naming.

Looking at other distributions:
- Fedora Core seems to try and solve this problem by allowing only
db-4 (I guess db-4.x's are compatible to some level then?)  It does
however package db-1 together with libgnome-1 in a single rpm to
account for a dependency problem there.  This is not an option for
gentoo, as nvi, xemacs, libtool, partysip, nss_ldap, hotkeys, jigdo
etc. seem to depend on older versions.  And er, what when db-5
appears?
- Debian has a clean solution: install as many db-versions as you
want, but only one development package at a time.  The development
package contains /usr/include/db.h AND /usr/lib/libdb.so.  Adapting
packages if necessary to use those two is an easy task.  However, this
isn't an option for gentoo either, as we don't split packages into
-libs and -devel.

So I'm wondering, what would be a clean solution for this problem?

Cheers,
Stefaan

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Berkeley DB, coexistence of different versions

[Georgi Georgiev]

maillog: 13/09/2005-00:50:47(+0200): Stefaan types
<snip>
> So I'm wondering, what would be a clean solution for this problem?

The same way that gtk-1 / gtk-2 goes -- add a bunch of useflags and
force the package to use a particular version. :)

So for heimdal it becomes:

db1? ( sys-libs/db-1* )
!db1? ( db3? ( sys-libs/db-3* ) )
!db1? ( !db3? ( db-4.0? ( sys-libs/db-4.0* ) ) )
...

db.eclass anyone? :)

After all, it is not only important to make heimdal compile and link
properly, you also have to track the proper dependency and only make the
package depend on the library that it is linked against. Or is a dep on
sys-libs/db going to make the package depend on all slotted versions of
db?

-- 
(*   Georgi Georgiev   (* Thus spake the master programmer: "When a    (*
*)    chutz@gg3.net    *) program is being tested, it is too late to   *)
(*  +81(90)2877-8845   (* make design changes." -- Geoffrey James,     (*
*) ------------------- *) "The Tao of Programming"                     *)
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Berkeley DB, coexistence of different versions

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 2281 bytes --]

On Tue, Sep 13, 2005 at 12:50:47AM +0200, Stefaan wrote:
> Hi all!!
> 
> Here's an issue Seemant and I have been struggling with, and doesn't
> seem as easy to solve as like touching one ebuild.
You're making the problem seem much larger than it is.

Pauldv and myself have been managing berkdb pretty well, I haven't seen
bugs on it in a long time.
> 
> Berkeley DB comes in many flavors.  Judging by the slots in the
> ebuilds (1, 3, 4, 4.1, 4.2, 4.3) there's many different interface
> versions.  This comes with the added fun of having packages depend on
> certain versions of DB.  Nothing shocking so far.
> 
> However, looking at the heimdal-package we noticed several problems
> (which seem to be more widespread than just heimdal):
[snip]

Fix your package to compile with any version of db, that's the general
approach that has been taken so far, and it's been largely successful.

You should use these libraries and headers:
/usr/lib/libdb.{so,a}
/usr/include/db.h

If you want to limit it to a certain reason of DB for some other
reason, then consider /usr/include/db1, /usr/include/db3,
/usr/include/db4, /usr/include/db4.X. 

And use the unversioned functions (db_create instead of db_create_4002).
To deal with the varying number of arguments, have a look at
PHP/nss_ldap - the fix is pretty simple, just an extra argument if the
version is 4 or newer (IIRC, but check).

Detection of some specific version is bad! Just figure out what major
version, and do the correct thing for that series.

In a bunch of cases, the best approach has been to just rip out
upstreams stuff from configure.in, and write it properly, and submit it
back to them (definetly see PHP4 in this regard).

Also take a look at db4-fix.eclass.

You claim the interface versions have changed, but the code is still
largely compatible. What's a larger issue is the fact that you can
upgrade an existing database, but not downgrade it.

If you are stuck updating the package for this, then give me or pauldv
a shout.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 241 bytes --]

Re: [gentoo-dev] Berkeley DB, coexistence of different versions

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 3830 bytes --]

On Tuesday 13 September 2005 04:23, Robin H. Johnson wrote:
> On Tue, Sep 13, 2005 at 12:50:47AM +0200, Stefaan wrote:
> > Hi all!!
> >
> > Here's an issue Seemant and I have been struggling with, and doesn't
> > seem as easy to solve as like touching one ebuild.
>
> You're making the problem seem much larger than it is.
>
> Pauldv and myself have been managing berkdb pretty well, I haven't seen
> bugs on it in a long time.
>
> > Berkeley DB comes in many flavors.  Judging by the slots in the
> > ebuilds (1, 3, 4, 4.1, 4.2, 4.3) there's many different interface
> > versions.  This comes with the added fun of having packages depend on
> > certain versions of DB.  Nothing shocking so far.
> >
> > However, looking at the heimdal-package we noticed several problems
> > (which seem to be more widespread than just heimdal):
>
> [snip]
>
> Fix your package to compile with any version of db, that's the general
> approach that has been taken so far, and it's been largely successful.
>
> You should use these libraries and headers:
> /usr/lib/libdb.{so,a}
> /usr/include/db.h
>
> If you want to limit it to a certain reason of DB for some other
> reason, then consider /usr/include/db1, /usr/include/db3,
> /usr/include/db4, /usr/include/db4.X.
>
> And use the unversioned functions (db_create instead of
> db_create_4002). To deal with the varying number of arguments, have a
> look at
> PHP/nss_ldap - the fix is pretty simple, just an extra argument if the
> version is 4 or newer (IIRC, but check).
>
> Detection of some specific version is bad! Just figure out what major
> version, and do the correct thing for that series.
>
> In a bunch of cases, the best approach has been to just rip out
> upstreams stuff from configure.in, and write it properly, and submit it
> back to them (definetly see PHP4 in this regard).
>
> Also take a look at db4-fix.eclass.
>
> You claim the interface versions have changed, but the code is still
> largely compatible. What's a larger issue is the fact that you can
> upgrade an existing database, but not downgrade it.
>
> If you are stuck updating the package for this, then give me or pauldv
> a shout.

The major issue with berkeley db is only that we build using "versioned 
symbols", a setup that prevents header/lib mismatches and allows 
different library versions to coexist in one binary (a situation that 
unfortunately happens with db as it's not strong on binary compatibility. 
Version 4.3 however does not offer this symbols anymore, which is the 
main reason it is masked. If you want to do some nice work you could work 
on using linking scripts to do versioning properly (and use the ELF 
versioning system). That leaves the poor detection of db packages by 
other packages/configure scripts. Unfortunately many configure scripts 
are just broken. The proper name of a library name according to the 
standards for library names is db-4.2.so, db-4.so and db.so to exist for 
all versions. A package should try to look for the most appropriate one 
of those, but many configure scripts look for redhat-isms like "db4.so". 

The db.eclass maintains shared info by all db3 and db4 versions such that 
/usr/include/db.h and /usr/lib/db.so allways point to the most recent 
version of db installed.

On the configure scripts, the best approach is to have a test program that 
includes the db header and then checks whether the header and library 
versions are equal. The problem with checks for db_create is that this 
function is called db_create4002 for db4.2 and this mapping is provided 
in the header file. If the header file is not included in the test 
program, the test fails.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Berkeley DB, coexistence of different versions

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

On Wednesday 14 September 2005 16:28, Paul de Vrieze wrote:
> The db.eclass maintains shared info by all db3 and db4 versions such that
> /usr/include/db.h and /usr/lib/db.so allways point to the most recent
> version of db installed.
Thus breaking Gentoo/FreeBSD as db.h there is something else, ehm :)
Yeah I know i'm annoying, but I'm waiting for that fix ;)

-- 
Diego "Flameeyes" Pettenò
Gentoo Developer - http://dev.gentoo.org/~flameeyes/
(Gentoo/FreeBSD, Video, Gentoo/AMD64, Sound, PAM)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Berkeley DB, coexistence of different versions

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

On Wednesday 14 September 2005 16:47, Diego 'Flameeyes' Pettenò wrote:
> On Wednesday 14 September 2005 16:28, Paul de Vrieze wrote:
> > The db.eclass maintains shared info by all db3 and db4 versions such
> > that /usr/include/db.h and /usr/lib/db.so allways point to the most
> > recent version of db installed.
>
> Thus breaking Gentoo/FreeBSD as db.h there is something else, ehm :)
> Yeah I know i'm annoying, but I'm waiting for that fix ;)

I'll look at fixing it. Is it only db.h or is there a problem 
with /usr/lib/libdb.so too?

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Berkeley DB, coexistence of different versions

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 355 bytes --]

On Wednesday 14 September 2005 17:01, Paul de Vrieze wrote:
> I'll look at fixing it. Is it only db.h or is there a problem
> with /usr/lib/libdb.so too?
Just db.h, there's no libdb.so in FreeBSD's base system.

-- 
Diego "Flameeyes" Pettenò
Gentoo Developer - http://dev.gentoo.org/~flameeyes/
(Gentoo/FreeBSD, Video, Gentoo/AMD64, Sound, PAM)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]