[gentoo-dev] Reverse dependency Scanning

[gentoo-dev] Reverse dependency Scanning

[Spider]

[-- Attachment #1: Type: text/plain, Size: 1434 bytes --]

Well,  this is something I spent quite a bit of time on yesterday, with
some help from bonsaikitten on figuring out the python madness.


Before ferringb tells it again, no, I know this isn't foolproof, cannot
be 100% in coverage, won't cover dlopen, conditional includes, bash
source inherit or other such fancy things that people do :)

However, it somewhat works ;)
What it does :
 given a list of files ( `etcat files category/package`  generates a
nice one ) it scans them, classifies into "scripts, perl, python,
libraries, executables" ,  then does its best to parse theese into
generating a list of -needed- files.

This means that it outputs a list of shared libraries linked against,
python modules imported, perl modules,  and script interpreters
( #!/bin/bash ...)

So,  taking that list you can then play around a bit with `qpkg -f` and
you'll end up with a nice list of packages that are actually used.


So, I know it isn't foolproof, don't expect it to be.  However, feedback
would be nice, patches and so on.   

oh, where it lives?  cvs : gentoo/users/spider/depreverse  , expects to
be ran from the current directory as it calls "./..."  for the included
scripts.

http://www.gentoo.org/cgi-bin/viewcvs.cgi/users/spider/depreverse/?root=gentoo


//Spider


-- 
begin  .signature
Tortured users / Laughing in pain
See Microsoft KB Article Q265230 for more information.
end


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reverse dependency Scanning

[Juergen Hoetzel]

Why not reverse it:

1. *DONT* unmerge an old lib (the *.so.* file) if the soname changes due
    an upgrade. Just leave this single file on the file-system
2. Scan your system in regular intervals for unused old shared libs

This seems much simpler and doesn't break systems.

Jürgen


> Well,  this is something I spent quite a bit of time on yesterday, with
> some help from bonsaikitten on figuring out the python madness.
> 
> 
> Before ferringb tells it again, no, I know this isn't foolproof, cannot
> be 100% in coverage, won't cover dlopen, conditional includes, bash
> source inherit or other such fancy things that people do :)
> 
> However, it somewhat works ;)
> What it does :
>  given a list of files ( `etcat files category/package`  generates a
> nice one ) it scans them, classifies into "scripts, perl, python,
> libraries, executables" ,  then does its best to parse theese into
> generating a list of -needed- files.
> 
> This means that it outputs a list of shared libraries linked against,
> python modules imported, perl modules,  and script interpreters
> ( #!/bin/bash ...)
> 
> So,  taking that list you can then play around a bit with `qpkg -f` and
> you'll end up with a nice list of packages that are actually used.
> 
> 
> So, I know it isn't foolproof, don't expect it to be.  However, feedback
> would be nice, patches and so on.   
> 
> oh, where it lives?  cvs : gentoo/users/spider/depreverse  , expects to
> be ran from the current directory as it calls "./..."  for the included
> scripts.
> 
> http://www.gentoo.org/cgi-bin/viewcvs.cgi/users/spider/depreverse/?root=gentoo
> 
> 
> //Spider
> 
> 
> -- 
> begin  .signature
> Tortured users / Laughing in pain
> See Microsoft KB Article Q265230 for more information.
> end
> 


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reverse dependency Scanning

[Thomas de Grenier de Latour]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

On Sat, 23 Apr 2005 12:15:32 +0200
Spider <spider@gentoo.org> wrote:

> So, I know it isn't foolproof, don't expect it to be.  However,
> feedback would be nice, patches and so on.   

I think that for binaries you could filter the ldd output to keep
only the libs that are directly linked to, since that's what
should really become dependencies. See atttached patch for a
possible quick and dirty implementation.

Example without the patch:
% echo /usr/games/bin/abuse.sdl | ./depreverse.orig
/lib/libdl.so.2
/lib/libncursesw.so.5
/lib/libnsl.so.1
/lib/tls/libc.so.6
/lib/tls/libm.so.6
/lib/tls/libpthread.so.0
/usr/lib/gcc/i686-pc-linux-gnu/3.4.3-20050110/libgcc_s.so.1
/usr/lib/gcc/i686-pc-linux-gnu/3.4.3-20050110/libstdc++.so.6
/usr/lib/libaa.so.1
/usr/lib/libcaca.so.0
/usr/lib/libSDL-1.2.so.0
/usr/lib/libslang-utf8.so.1
/usr/lib/libvga.so.1
/usr/lib/libX11.so.6
/usr/lib/libXext.so.6
/usr/lib/libXxf86vm.so.1
/usr/lib/opengl/xorg-x11/lib/libGL.so.1

The same with the patch:
% echo /usr/games/bin/abuse.sdl | ./depreverse
/lib/libnsl.so.1
/lib/tls/libc.so.6
/lib/tls/libm.so.6
/lib/tls/libpthread.so.0
/usr/lib/gcc/i686-pc-linux-gnu/3.4.3-20050110/libgcc_s.so.1
/usr/lib/gcc/i686-pc-linux-gnu/3.4.3-20050110/libstdc++.so.6
/usr/lib/libSDL-1.2.so.0
/usr/lib/opengl/xorg-x11/lib/libGL.so.1

-- 
TGL.

[-- Attachment #2: depreverse--filter_libs.patch --]
[-- Type: text/x-patch, Size: 628 bytes --]

--- depreverse.orig	2005-04-23 13:35:49.000000000 +0200
+++ depreverse	2005-04-23 14:03:07.000000000 +0200
@@ -111,7 +111,8 @@
 # since we already tested if executables are executable, we only need one pass to test both theese lists.
 for FILE in $EXELIST $LIBLIST ; do
 	if [ -r $FILE ]; then
-		ldd $FILE |grep -v 'libNoVersion.so' | grep -v 'linux-gate.so' | awk '/=>/ { print $3 }'
+		needed_libs_regexp="$(objdump -p $FILE | awk '/NEEDED/ { rex=rex "\\|\\<"$2"\\>"} ; END {print substr(rex,3)}')"
+		ldd $FILE | grep "${needed_libs_regexp}" | awk '/=>/ { print $3 }'
 	fi
 done | sort -u | sed "s/['\"]/\\\&/g" | sort -u
 

Re: [gentoo-dev] Reverse dependency Scanning

[Spider]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On Sat, 2005-04-23 at 14:21 +0200, Thomas de Grenier de Latour wrote:
> On Sat, 23 Apr 2005 12:15:32 +0200
> Spider <spider@gentoo.org> wrote:
> 
> > So, I know it isn't foolproof, don't expect it to be.  However,
> > feedback would be nice, patches and so on.   
> 
> I think that for binaries you could filter the ldd output to keep
> only the libs that are directly linked to, since that's what
> should really become dependencies. See atttached patch for a
> possible quick and dirty implementation.

Thanks, Applied said quick&dirty implementation,  committing as well :)

//Spider

-- 
begin  .signature
Tortured users / Laughing in pain
See Microsoft KB Article Q265230 for more information.
end


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reverse dependency Scanning

[Thomas de Grenier de Latour]

I've made a small python wrapper around depreverse, which is
available here:  http://tdegreni.free.fr/gentoo/dotfinder.py

It has a similar purpose as your "finder.sh" wrapper, but outputs
its results as a graph in dot format (with 4 layers: requested
packages, contents files, depended-on files, and depended-on
packages). For example, the following command:
  ./dotfinder.py galeon | dot -Tps > galeon.ps
will results in something like this:
  http://tdegreni.free.fr/gentoo/galeon.ps

I'm not sure it's really useful, but oh well, it was worth writing
to learn dot's very basics :)

-- 
TGL.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reverse dependency Scanning

[Spider]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Mon, 2005-04-25 at 03:51 +0200, Thomas de Grenier de Latour wrote:
> I've made a small python wrapper around depreverse, which is
> available here:  http://tdegreni.free.fr/gentoo/dotfinder.py
> 
> It has a similar purpose as your "finder.sh" wrapper, but outputs
> its results as a graph in dot format (with 4 layers: requested
> packages, contents files, depended-on files, and depended-on
> packages). For example, the following command:
>   ./dotfinder.py galeon | dot -Tps > galeon.ps
> will results in something like this:
>   http://tdegreni.free.fr/gentoo/galeon.ps
> 
> I'm not sure it's really useful, but oh well, it was worth writing
> to learn dot's very basics :)

Hehe,  Rather cool, but the diagram is a bit hard to get an overview of.
Though, I recognise the reason for doing it :)


 Thanks ,
   Spider

-- 
begin  .signature
Tortured users / Laughing in pain
See Microsoft KB Article Q265230 for more information.
end


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reverse dependency Scanning

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

On Monday 25 April 2005 11:29, Spider wrote:
> On Mon, 2005-04-25 at 03:51 +0200, Thomas de Grenier de Latour wrote:
> > I've made a small python wrapper around depreverse, which is
> > available here:  http://tdegreni.free.fr/gentoo/dotfinder.py
> >
> > It has a similar purpose as your "finder.sh" wrapper, but outputs
> > its results as a graph in dot format (with 4 layers: requested
> > packages, contents files, depended-on files, and depended-on
> > packages). For example, the following command:
> >   ./dotfinder.py galeon | dot -Tps > galeon.ps
> > will results in something like this:
> >   http://tdegreni.free.fr/gentoo/galeon.ps
> >
> > I'm not sure it's really useful, but oh well, it was worth writing
> > to learn dot's very basics :)
>
> Hehe,  Rather cool, but the diagram is a bit hard to get an overview
> of. Though, I recognise the reason for doing it :)

Making the diagram vertical would probably be a big improvement. one thing 
I miss however is a destinction between libraries linked into the 
application, and those that only are taken in by another library. The so 
files do make this distinction so it should be possible to extract it.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reverse dependency Scanning

[Thomas de Grenier de Latour]

On Mon, 25 Apr 2005 14:52:18 +0200
Paul de Vrieze <pauldv@gentoo.org> wrote:

> Making the diagram vertical would probably be a big improvement.

Yep, that's very true. I've uploaded (same url) a new version
which adds 'rankdir=LR' to the graph, and I also think it's
much more readable this way (on "medium size" package sure, it's
still and will probably always be absolutly messy on big packages
like mozilla for instance). 

Another change I've made is the addition of a "-r/--real-paths"
option which make the script resolve all symlinks while searching
for packages owning some depended-on files. The point is that if
you don't do that, then it won't be accurate in finding deps: a
program may be linked to "/usr/X11R6/lib/foo.so" whereas the file
was installed as "/usr/lib/foo.so" for instance (or the opposite -
I've found both on my system actually).
The following example shows the difference i think:
http://tdegreni.free.fr/gentoo/xdtv-symlinks.ps
http://tdegreni.free.fr/gentoo/xdtv-realpaths.ps
(the ebuild is not from portage, so i'm the one to blame for the
/usr/X11R6/lib linkings)

The drawback of that "real-paths" approach is that you then can't
distinguish a dep on Bash (/bin/bash) from one on any Bourne shell
(/bin/sh), and that's why i made it optional. Probably the best
would be to do both kinds of search at once and show the
difference in the graph, or something like that...

> one thing I miss however is a destinction between libraries
> linked into the application, and those that only are taken in by
> another library. The so files do make this distinction so it
> should be possible to extract it.

If i don't misunderstand what you mean, then that was exactly the
point of a patch i've submitted 2 days ago, and which is in the
current depreverse version. Shared libs that are listed as deps
should be only those which are marked NEEDED in the objects
headers, not those that are indirectly linked to.

-- 
TGL.
-- 
gentoo-dev@gentoo.org mailing list