[gentoo-dev] Prelinking in Gentoo and interactions with portage

[gentoo-dev] Prelinking in Gentoo and interactions with portage

[Stefan Jones]

DISCLAIMER: Prelink is experimental and ^may^ break EVERYTHING
( having said this I have suffered no serious breakages, but others
have)

Recently I have started working on prelinking in gentoo.

When a shared executable is run all the references to functions in
shared libraries have to be cross referenced and offsets applied to the
function pointers. Prelink modifies the executables so that this work is
done before the binary is run. This speeds start up times by a large
factor (up to 50% for some applications like mozilla and KDE).

To use prelink you need a prelink enabled glibc (2.3.1 satisfies this),
and a new version of binutils (2.13.90.0.10 will do).

I have put a masked prelink ebuild into portage, this provides the tool
that modifies the binaries. Emerge it and read "man prelink" for more
details.

Now the portage details:

Prelink needs to be rerun on shared librarys and executables every time
their dependant librarys get modified. So when you upgrade a library
prelink needs to be rerun on the executables. This means the md5sum of a
binary will change from the portage digest.

There is a work around for that, prelink has the ability to spool out
the original binary's contents to stdout. This can be harnessed by
portage. The details how to do so follow:

On Install:
1) The MD5sun is first calculated for the CONTENTS file
2) The files are merged (installed) into the filesystem
3) prelink is run on all the installed binaries and libraries

On Uninstall:
* If the file is an executable / library pipe it though prelink when
calculating the MD5sum.

All the above would be ifdefed out if some PRELINK_ENABLE variable was
not set somewhere (eg. make.global), so there will be no slowdown for
ppl not using prelink.

Note: the prelink command cannot be put in
/usr/lib/portage/bin/prepallstrip as prelink may have to be rerun on the
file causing the MD5sum to change again.

Also some small bash script needs to be made to allow the user to
re-prelink files after a library upgrade. Prelinking doesn't take long,
prelinking from scratch /usr/bin only takes a few minutes. This could be
in a cron-job or manually run by the user though a new portage command.

Note: re-prelinking a file is very quick, so it may be feasible to
prelink all dependants in emerge after the library install. But this
would require reverse dependency checking.

Once the details are fixed, I will happing code this up and submit the
patches.

Food for thought,

	Stefan

-- 
Stefan Jones <cretin@gentoo.org>
Gentoo Linux


--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: [gentoo-core] Prelinking in Gentoo and interactions with portage

[Daniel Robbins]

On Sun, 2002-11-17 at 06:50, Stefan Jones wrote:
> Note: re-prelinking a file is very quick, so it may be feasible to
> prelink all dependants in emerge after the library install. But this
> would require reverse dependency checking.
> 
> Once the details are fixed, I will happing code this up and submit the
> patches.

Thanks very much for this info and for looking into this. Recently,
Mandrake claimed they are superior in performance to Gentoo Linux (in
one of their newsletters, I think) because they prelink and we don't.
Maybe we should. Maybe we shouldn't. Opinions welcome :) I'd also like
to track down that Mandrake comparison -- woodchip, do you have the
reference?

Best Regards,

-- 
Daniel Robbins
Chief Architect, Gentoo Linux
http://www.gentoo.org


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [gentoo-core] Prelinking in Gentoo and interactions with portage

[Alexander Futasz]

On 17 Nov 2002 16:39:45 -0700, Daniel Robbins wrote:
> Thanks very much for this info and for looking into this. Recently,
> Mandrake claimed they are superior in performance to Gentoo Linux (in
> one of their newsletters, I think) because they prelink and we don't.
> Maybe we should. Maybe we shouldn't. Opinions welcome :) I'd also like
> to track down that Mandrake comparison -- woodchip, do you have the
> reference?

http://mandrakeforum.com/article.php?sid=2472&lang=en

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [gentoo-core] Prelinking in Gentoo and interactions with portage

[david mattatall]

You gotta make dev-libs/libelf a dep in your ebuild. Yes I realize it is only 
experimental as of right now...but still...
-- 
*-_-_-_-_-_-_-_-_-_-_-_-_-_-*
AIM: StabbyMcforehead
ICQ: 73080754
Email: davidsmind@linuxbasics.com
Weblog: http://davidsmind.diaryland.com


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Prelinking in Gentoo and interactions with portage

[Felipe Ghellar]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii; format=flowed, Size: 502 bytes --]

What about including it in the variable FEATURES in make.conf?

FEATURES="sandbox ccache prelink"


Stefan Jones wrote:
> All the above would be ifdefed out if some PRELINK_ENABLE variable was
> not set somewhere (eg. make.global), so there will be no slowdown for
> ppl not using prelink.

_______________________________________________________________________
Yahoo! Encontros
O lugar certo para encontrar a sua alma gêmea.
http://br.encontros.yahoo.com/

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Prelinking in Gentoo and interactions with portage

[David Olsen]

On 2002-11-18-09:53:03, Felipe Ghellar <fghellar2@yahoo.com.br> wrote:
> What about including it in the variable FEATURES in make.conf?
> 
> FEATURES="sandbox ccache prelink"

Prelinking may be more suited as a USE variable, just an opinion though.

Best Regards, 

David Olsen
Administrator
linuxpimpz.com
Gentoo user since 1.0_rc3
"Geek -- once it was a four-letter insult, now it's a six-figure salary."


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Prelinking in Gentoo and interactions with portage

[Stefan Jones]

On Mon, 2002-11-18 at 17:41, David Olsen wrote:
> On 2002-11-18-09:53:03, Felipe Ghellar <fghellar2@yahoo.com.br> wrote:
> > What about including it in the variable FEATURES in make.conf?
> > 
> > FEATURES="sandbox ccache prelink"
> 
> Prelinking may be more suited as a USE variable, just an opinion though.
> 

I am siding on the side of "FEATURES" as prelinking globally effects
binaries and libraries, you cannot always say that certain libraries are
not to be prelinked, as USE flags would imply.

Also you would have a messy problem on uninstall having to read the USE
flags. Although you could choose to always uninstall in the prelink safe
way.

Plus it fits better into the portage overview to have USE flags effect
the behaviour of the ebuild code and FEATURES flags effect the portage
(python) code.

My two pence ,

-- 
Stefan Jones <cretin@gentoo.org>
Gentoo Linux


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Prelinking in Gentoo and interactions with portage

[Stefan Jones]

Hi ppl, I have make a draft version of the portage changes needed.

I am not a python programmer, or want to be! The code can be optimised a
bit but I believe that prelink was well programmed so all checks for
file type etc are best handled by C based prelink, as long as os.system
is fast.

I have created code which adds support for the FEATURE flag prelink.
If it exists it uses prelink to verify the md5sums.
Also it prelinks files as they are installed.

Most importantly it works for me! Happy testing!

code:
download from
http://cvs.gentoo.org/~cretin/portage.py
This is based on the latest CVS portage

TODO: 
* Safety checks for existence of /usr/sbin/prelink
* optimise to minimise number of calls to prelink
* see if a less IO intensive md5sum method exists  (of course it does!)

now to sleep .....

-- 
Stefan Jones <cretin@gentoo.org>
Gentoo Linux


--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Prelinking in Gentoo and interactions with portage

[Stefan Jones]

patch: http://cvs.gentoo.org/~cretin/portage-prelink.patch
diffed against the latest portage.py

I think the patch included here is close to the final product, or at
least the principles involved. Some of it can still be polished a bit
more by the gurus, email me with any ideas plz!

Prelink features are only enabled if you have prelink in FEATURES in
make.conf. You also must have prelink installed at /usr/sbin/prelink
(the default location)

What it does:

On unmerges it uses prelink to calculate the md5sums.

On emerge a list of all files installed with the tokens "bin" or "lib"
in their path or name is made. (ie /bin/ls or /opt/something/bin/ls or
libblibble.la ) An attempt is then made to prelink these. This is dirty
but safe. Patches welcome for better code.

How to install:

patch /usr/lib/python2.2/site-packages/portage.py with this patch

/bin/cp -f portage.py /usr/lib/python2.2/site-packages/

/bin/rm -f /usr/lib/python2.2/site-packages/portage.py[co]

/usr/bin/python -c "import py_compile;
py_compile.compile('/usr/lib/python2.2/site-packages/portage.py')" 
/usr/bin/python -O -c "import py_compile;
py_compile.compile('/usr/lib/python2.2/site-packages/portage.py')"
			    	
add prelink to the FEATURES list and emerge prelink

Have fun!

-- 
Stefan Jones <cretin@gentoo.org>
Gentoo Linux


--
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Prelinking in Gentoo and interactions with portage

[Javier Marcet]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

* david mattatall <davidsmind@linuxbasics.com> [021118 14:19]:

>You gotta make dev-libs/libelf a dep in your ebuild. Yes I realize it is only 
>experimental as of right now...but still...

Also, there are some issues with QT's xinerama and xfree's libGLU being
compiled as non PIC by default.

Disabling xinerama in QT and adding

echo "#define BuildLibGlxWithoutPIC NO" >> config/cf/host.def

fixes both problems.


BTW, anyone managed to compile xfree-4.2.99.3? I'm using xfree 4.2.99
compiled on October 23rd. Since then I have two nasty problems to
compile it, what also happens with the 4.2.99.3 snapshot on portage.

One of the problems is a sandbox violation, trying to access a Makefile
in /usr/lib which I haven't been able to trace down, and there's one
more which I don't remember right now.

Do you know any fix? It's really nice to use the cvs version with xft2
:)


-- 
Javier Marcet <jmarcet@pobox.com>

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Prelinking in Gentoo and interactions with portage

[stefanajones]

 > >You gotta make dev-libs/libelf a dep in your ebuild. Yes I realize it is only 
> >experimental as of right now...but still...
> 
> Also, there are some issues with QT's xinerama and xfree's libGLU being
> compiled as non PIC by default.
> 
> Disabling xinerama in QT and adding
> 
> echo "#define BuildLibGlxWithoutPIC NO" >> config/cf/host.def
> 
> fixes both problems.
> 

I am aware about these issues, I have read the very good thread at
http://forums.gentoo.org/viewtopic.php?t=21117

Will try to push these into the portage tree when a senior developers agrees.

In the meantime sandbox is starting to dislike prelinked binaries again. I am having to merge jrray's patch for user compiles into my patch. ( http://cvs.gentoo.org/~jrray/ )

will reveal more when I have it,

  Stefan

--
gentoo-dev@gentoo.org mailing list