[gentoo-dev] VDB access

[gentoo-dev] VDB access

[Ciaran McCreesh]

The following things access VDB by hand:

* gnome2-utils.eclass. Will be fixed once a portage with proper
env saving goes stable, which isn't too far off. Bug 155993.

* net-misc/l7-filter. This can be fixed as soon as portage with proper
env saving goes stable.

* kde.eclass, for slot_rebuild. This seems like it's a really dumb hack
to cater for users who don't know how to use revdep-rebuild (there
seems to be a lot of that going around, making elog worthless, but
that's a different issue...), and should probably just be removed.

* pcmcia.eclass. Supposedly deprecated. Bug 140289.

* eutils.eclass. For built_with_use.

Of these, only the last appears to be of any use. So how about the
following?

* Modify built_with_use so that it calls native_built_with_use if such
a function exists, and falls back to its own implementation otherwise.

* Allow package managers that implement native_built_with_use to sandbox
off vdb completely, and ban all ebuild access to vdb.

This will let package managers use a format other than VDB. A well
designed replacement can shave a minute off cold cache command times.

-- 
Ciaran McCreesh
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: VDB access

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 1500 bytes --]

Hi,

Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk>:
> The following things access VDB by hand:
> * gnome2-utils.eclass. Will be fixed once a portage with proper
> env saving goes stable, which isn't too far off. Bug 155993.
> 
> * net-misc/l7-filter. This can be fixed as soon as portage with proper
> env saving goes stable.
> 
> * kde.eclass, for slot_rebuild. This seems like it's a really dumb
> hack to cater for users who don't know how to use revdep-rebuild
> (there seems to be a lot of that going around, making elog worthless,
> but that's a different issue...), and should probably just be removed.
> 
> * pcmcia.eclass. Supposedly deprecated. Bug 140289.

 Are there open bugs for it?  Or is anyone aware of and working on it?
 
> * Modify built_with_use so that it calls native_built_with_use if such
> a function exists, and falls back to its own implementation otherwise.
> 
> * Allow package managers that implement native_built_with_use to
> sandbox off vdb completely, and ban all ebuild access to vdb.

 Sounds reasonable as far as I can see. 
 
> This will let package managers use a format other than VDB. A well
> designed replacement can shave a minute off cold cache command times.

 How much gain can be expected?  And what package managers will allow
native_built_with_use?

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://www.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: VDB access

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1947 bytes --]

On Tue, 5 Feb 2008 08:43:55 +0100
Christian Faulhammer <opfer@gentoo.org> wrote:
> > This will let package managers use a format other than VDB. A well
> > designed replacement can shave a minute off cold cache command
> > times.
> 
>  How much gain can be expected?

There're two common operations that're really really slow (because they
need lots of filesystem access) using VDB: turning a pkg into a cat/pkg
and finding all PROVIDEd packages. A redesigned format can, for example,
make the former a single filesystem operation and the latter a single
directory read.

If you're looking for numbers, you can see very roughly how long a VDB
load takes off cold cache using:

echo 2 | sudo dd of=/proc/sys/vm/drop_caches
find /var/db/pkg/ -type f -not -name 'environment.bz2' \
    | xargs cat >/dev/null

A different on-disk format could avoid 90% of those filesystem accesses
for many operations.

There's a more subtle issue with VDB and the scope of locking required
by multithreaded implementations. This one's not an issue for
interpreted languages, but it's a minor nuisance in places where
there's no language-induced locking.

> And what package managers will allow native_built_with_use?

I doubt Portage will, since if someone's going to make changes to
Portage that'd let it use something other than VDB, they've clearly got
enough time to implement use deps as a side project, which would make
built_with_use obsolete. Paludis would certainly implement it -- we've
already got a redesigned on-disk db format that we use for tracking
unpackaged packages, and repurposing that to handle ebuilds wouldn't be
very hard.

Incidentally, if anyone is still thinking that built_with_use is
fine... It doesn't handle IUSE defaults, which means package managers
have to store a hacked version of IUSE in VDB for EAPI 1. This isn't
documented anywhere and is highly unobvious.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: VDB access

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

Ciaran McCreesh kirjoitti:
> 
> Incidentally, if anyone is still thinking that built_with_use is
> fine... It doesn't handle IUSE defaults, which means package managers
> have to store a hacked version of IUSE in VDB for EAPI 1. This isn't
> documented anywhere and is highly unobvious.
> 

I don't understand this. IUSE defaults affect whether a use flag is on 
or off when installing the ebuild. Why would it need to be stored in the 
VDB? There should only be a need of the value of $USE. Please elaborate.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] Re: VDB access

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

On Wed, 06 Feb 2008 08:43:20 +0200
Petteri Räty <betelgeuse@gentoo.org> wrote:
> Ciaran McCreesh kirjoitti:
> > Incidentally, if anyone is still thinking that built_with_use is
> > fine... It doesn't handle IUSE defaults, which means package
> > managers have to store a hacked version of IUSE in VDB for EAPI 1.
> > This isn't documented anywhere and is highly unobvious.
> 
> I don't understand this. IUSE defaults affect whether a use flag is
> on or off when installing the ebuild. Why would it need to be stored
> in the VDB? There should only be a need of the value of $USE. Please
> elaborate.

VDB stores exact copies of ebuild metadata variables, except for IUSE.
IUSE is the only one that gets modified by the package manager.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] VDB access

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh wrote:
> * Modify built_with_use so that it calls native_built_with_use if such
> a function exists, and falls back to its own implementation otherwise.
> 
> * Allow package managers that implement native_built_with_use to sandbox
> off vdb completely, and ban all ebuild access to vdb.

The problem that I see with native_built_with_use is that the
package manager will then need to precisely implement the
build_with_use interface along with any and all supported options
(and this interface would effectively become part of the EAPI).

If the package manager exposes a slightly lower level interface to
the USE flags then build_with_use can use that instead, and the
package manager won't have to implement the full build_with_use
interface.  For example, portageq currently supports a "metadata"
command that can be used to query installed package metadata such as
USE and IUSE. Perhaps we should use some type of interface similar
to that.

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHqNNc/ejvha5XGaMRAjseAKDQ73gjzGKR9Rdnz8o1EYxVI63v3QCgxAfA
QCCmjfOLes0cZTpcPBuRCCI=
=S5M6
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] VDB access

[Vlastimil Babka]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Zac Medico wrote:
> If the package manager exposes a slightly lower level interface to
> the USE flags then build_with_use can use that instead, and the
> package manager won't have to implement the full build_with_use
> interface.  For example, portageq currently supports a "metadata"
> command that can be used to query installed package metadata such as
> USE and IUSE. Perhaps we should use some type of interface similar
> to that.

I'd say it depends on whether we want to support native_built_with_use
just as a temporary workaround until there are use deps (then just
package managers not using vdb would implement it, and portage need not
care), or we would want to use the general metadata query also for other
purposes.

Anyway, restricting ebuilds from vdb access sounds like a good idea to me.
-- 
Vlastimil Babka (Caster)
Gentoo/Java







[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] VDB access

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vlastimil Babka wrote:
> Zac Medico wrote:
>> If the package manager exposes a slightly lower level interface to
>> the USE flags then build_with_use can use that instead, and the
>> package manager won't have to implement the full build_with_use
>> interface.  For example, portageq currently supports a "metadata"
>> command that can be used to query installed package metadata such as
>> USE and IUSE. Perhaps we should use some type of interface similar
>> to that.
> 
> I'd say it depends on whether we want to support native_built_with_use
> just as a temporary workaround until there are use deps (then just
> package managers not using vdb would implement it, and portage need not
> care), or we would want to use the general metadata query also for other
> purposes.

The point is that if we simplify the interface and avoid duplication
of logic, there will be less potential for EAPI change. If somebody
wants to add a new option to built_with_use then they'll either have
to bump the EAPI or just accept that anybody using an incompatible
native_built_with_use implementation may experience api breakage.

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHqWaf/ejvha5XGaMRAgMPAKDhVfdAcdC/m+ZCiIPUGKkeSzAZuACgr+6Y
dgxIgIeV8FhsDHPxLI9egEc=
=W8Pg
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] VDB access

[Michael Haubenwallner]

On Wed, 2008-02-06 at 07:30 +0100, Vlastimil Babka wrote:
> Zac Medico wrote:

> or we would want to use the general metadata query also for other
> purposes.

> Anyway, restricting ebuilds from vdb access sounds like a good idea to me.

+1 for general metadata query interface.

In prefix (specifically on AIX) we have some nasty requirement to merge
object files inside archive files, because shared-objects should be
members of a static libraries, fex libstdc++.so.6 is an archive member
of libstdc++.a.

This (still experimental) inside-library-merge currently is done using
some hackery in profile.bashrc, but this requires access to vdb/CONTENTS
in pre_pkg_postrm() to clean up after unmerge.

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] VDB access

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On Wed, 06 Feb 2008 14:24:28 +0100
Michael Haubenwallner <haubi@gentoo.org> wrote:
> +1 for general metadata query interface.
> 
> In prefix (specifically on AIX) we have some nasty requirement to
> merge object files inside archive files, because shared-objects
> should be members of a static libraries, fex libstdc++.so.6 is an
> archive member of libstdc++.a.
> 
> This (still experimental) inside-library-merge currently is done using
> some hackery in profile.bashrc, but this requires access to
> vdb/CONTENTS in pre_pkg_postrm() to clean up after unmerge.

CONTENTS isn't metadata in the conventional sense... If someone's
redesigning VDB, they're very unlikely to keep CONTENTS in the same
format. So you'd need a second, different interface for that...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] VDB access

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

On Tue, 05 Feb 2008 13:21:34 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> > * Allow package managers that implement native_built_with_use to
> > sandbox off vdb completely, and ban all ebuild access to vdb.
> 
> The problem that I see with native_built_with_use is that the
> package manager will then need to precisely implement the
> build_with_use interface along with any and all supported options
> (and this interface would effectively become part of the EAPI).

True.

> If the package manager exposes a slightly lower level interface to
> the USE flags then build_with_use can use that instead, and the
> package manager won't have to implement the full build_with_use
> interface.  For example, portageq currently supports a "metadata"
> command that can be used to query installed package metadata such as
> USE and IUSE. Perhaps we should use some type of interface similar
> to that.

That's also a possibility. How about something like:

if [[ -n "${E_INSTALLED_PKG_METADATA_CMD}" ]] ; then
    ${E_INSTALLED_PKG_METADATA_CMD} >=cat/pkg-1.23 USE foo
else
    # old vdb stuff
fi

with heavy (and checked) restrictions upon legal metadata keys (stuff
that's in individual files, not environment.bz2)?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] VDB access

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh wrote:
> On Tue, 05 Feb 2008 13:21:34 -0800
> Zac Medico <zmedico@gentoo.org> wrote:
>> If the package manager exposes a slightly lower level interface to
>> the USE flags then build_with_use can use that instead, and the
>> package manager won't have to implement the full build_with_use
>> interface.  For example, portageq currently supports a "metadata"
>> command that can be used to query installed package metadata such as
>> USE and IUSE. Perhaps we should use some type of interface similar
>> to that.
> 
> That's also a possibility. How about something like:
> 
> if [[ -n "${E_INSTALLED_PKG_METADATA_CMD}" ]] ; then
>     ${E_INSTALLED_PKG_METADATA_CMD} >=cat/pkg-1.23 USE foo
> else
>     # old vdb stuff
> fi
> 
> with heavy (and checked) restrictions upon legal metadata keys (stuff
> that's in individual files, not environment.bz2)?
> 

That seems fine except it would be handy if it supports something
like the built_with_use -a and -o options since, given your
interface, it would be best to implement those on the package
manager side.

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHqrYq/ejvha5XGaMRArngAJ0SlTbooCI+5jjxesdF0MnkU0Ym6ACgsstw
5Mp+QGZPTVPGw2LyY7Cmwpk=
=8xmy
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] VDB access

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 563 bytes --]

On Wed, 06 Feb 2008 23:41:31 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> > if [[ -n "${E_INSTALLED_PKG_METADATA_CMD}" ]] ; then
> >     ${E_INSTALLED_PKG_METADATA_CMD} >=cat/pkg-1.23 USE foo
> > else
> >     # old vdb stuff
> > fi
> 
> it would be handy if it supports something like the built_with_use -a
> and -o options

Mmm, or we could just have it return the raw value. Slight difficulty
there in that it'd have to return IUSE etc in whatever format the
installed package's EAPI uses, but that's not too big a deal.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] VDB access

[Rémi Cardona]

Ciaran McCreesh a écrit :
> The following things access VDB by hand:
> 
> * gnome2-utils.eclass. Will be fixed once a portage with proper
> env saving goes stable, which isn't too far off. Bug 155993.

Quick follow-up on that for everyone.

The eclass has been modified not to access VDB anymore yet with all the 
bells and whistles to improve performance, and the concerned ebuilds 
have been updated.

Cheers

-- 
Rémi Cardona
LRI, INRIA
remi.cardona@lri.fr
remi@gentoo.org
-- 
gentoo-dev@lists.gentoo.org mailing list