[gentoo-dev] Implicit system dependencies

[gentoo-dev] Implicit system dependencies

[Michael]

Hi,

I wrote a script to search for discrepancies between linked libraries and 
what's defined in (R)DEPEND, with the intention of improving QA for minimal 
package installs.

The devmanual seems to suggest that it's a bad thing for packages to 
explicitly depend on packages in the system set, while many packages in the 
tree do indeed explicitly depend upon packages such as zlib and ncurses.

Any thoughts to as how far I should go with filing bugs regarding this 
issue?

Best regards,
Michael

Re: [gentoo-dev] Implicit system dependencies

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 593 bytes --]

On Monday, August 22, 2011 15:21:23 Michael wrote:
> I wrote a script to search for discrepancies between linked libraries and
> what's defined in (R)DEPEND, with the intention of improving QA for minimal
> package installs.
> 
> The devmanual seems to suggest that it's a bad thing for packages to
> explicitly depend on packages in the system set, while many packages in the
> tree do indeed explicitly depend upon packages such as zlib and ncurses.

file bugs for missing zlib/ncurses deps.  if there are other packages you're 
concerned with, ask now and we'll tell you.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Implicit system dependencies

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 363 bytes --]

On 8/22/11 12:21 PM, Michael wrote:
> I wrote a script to search for discrepancies between linked libraries and 
> what's defined in (R)DEPEND, with the intention of improving QA for minimal 
> package installs.

It would be great to integrate this into portage and make it a part of
the developer profile. This would just help prevent future breakages.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] Implicit system dependencies

[Maciej Mrozowski]

[-- Attachment #1: Type: Text/Plain, Size: 835 bytes --]

On Tuesday 23 of August 2011 18:54:04 Paweł Hajdan, Jr. wrote:
> On 8/22/11 12:21 PM, Michael wrote:
> > I wrote a script to search for discrepancies between linked libraries and
> > what's defined in (R)DEPEND, with the intention of improving QA for
> > minimal package installs.
> 
> It would be great to integrate this into portage and make it a part of
> the developer profile. This would just help prevent future breakages.

It seems there are a couple of such home-grown solutions already.
dberkholz used to have on in his dev webspace[1], we have one[2] in kde 
overlay...

1.  http://dev.gentoo.org/~dberkholz/scripts/ (linking_libs, included_headers)
2. 
http://git.overlays.gentoo.org/gitweb/?p=proj/kde.git;a=tree;f=Documentation/maintainers;hb=master 
(dynlink_scanner + try_dlopen.c)

-- 
regards
MM

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Implicit system dependencies

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

On 19:08 Tue 23 Aug     , Maciej Mrozowski wrote:
> On Tuesday 23 of August 2011 18:54:04 Paweł Hajdan, Jr. wrote:
> > On 8/22/11 12:21 PM, Michael wrote:
> > > I wrote a script to search for discrepancies between linked libraries and
> > > what's defined in (R)DEPEND, with the intention of improving QA for
> > > minimal package installs.
> > 
> > It would be great to integrate this into portage and make it a part of
> > the developer profile. This would just help prevent future breakages.
> 
> It seems there are a couple of such home-grown solutions already.
> dberkholz used to have on in his dev webspace[1], we have one[2] in kde 
> overlay...
> 
> 1.  http://dev.gentoo.org/~dberkholz/scripts/ (linking_libs, included_headers)
> 2. 
> http://git.overlays.gentoo.org/gitweb/?p=proj/kde.git;a=tree;f=Documentation/maintainers;hb=master 
> (dynlink_scanner + try_dlopen.c)


Also we've got a couple of really cool GSoC projects along these lines.

One is called autodep [1, 2], and it integrates into portage to let you 
know if you're missing deps or specified too many deps.

The other one is an ebuild generator [3, 4] that works for autotools but not a 
lot else at this point.

1. http://archives.gentoo.org/gentoo-soc/msg_7491f6b5bffdccaedc939beb6e1c4f0d.xml 
2. http://dev.gentoo.org/~neurogeek/guidexml/ (temporary homepage)
3. http://archives.gentoo.org/gentoo-soc/msg_e29529cb6f3dfc762f4f7e313b106deb.xml
4. http://soc.dev.gentoo.org/~darkdefender/ebuildgenerator

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: Implicit system dependencies

[Michael]

Mike Frysinger wrote:

> On Monday, August 22, 2011 15:21:23 Michael wrote:
>> I wrote a script to search for discrepancies between linked libraries and
>> what's defined in (R)DEPEND, with the intention of improving QA for
>> minimal package installs.
>> 
>> The devmanual seems to suggest that it's a bad thing for packages to
>> explicitly depend on packages in the system set, while many packages in
>> the tree do indeed explicitly depend upon packages such as zlib and
>> ncurses.
> 
> file bugs for missing zlib/ncurses deps.  if there are other packages
> you're concerned with, ask now and we'll tell you.
> -mike

What about packages such as bzip2, xz-utils, or really, app-arch/*?

Also, on a related note, would it be appropriate to report the following 
undeclared deps, given that the package in question already has DEPEND for 
gtk+?

* Checking app-arch/guitar-0.1.4 for undeclared dependencies
 *   /usr/bin/guitar links to /usr/lib64/libX11.so.6
 *     Missing dependency on x11-libs/libX11
 *   /usr/bin/guitar links to /usr/lib64/libXi.so.6
 *     Missing dependency on x11-libs/libXi
 *   /usr/bin/guitar links to /usr/lib64/libXext.so.6
 *     Missing dependency on x11-libs/libXext
 *   /usr/bin/guitar links to /usr/lib64/libxcb.so.1
 *     Missing dependency on x11-libs/libxcb
 *   /usr/bin/guitar links to /usr/lib64/libXau.so.6
 *     Missing dependency on x11-libs/libXau
 *   /usr/bin/guitar links to /usr/lib64/libXdmcp.so.6
 *     Missing dependency on x11-libs/libXdmcp


Best regards,
Michael

Re: [gentoo-dev] Re: Implicit system dependencies

[Nathan Phillip Brink]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

On Wed, Aug 24, 2011 at 03:53:26AM +1000, Michael wrote:
> Also, on a related note, would it be appropriate to report the following 
> undeclared deps, given that the package in question already has DEPEND for 
> gtk+?
> 
> * Checking app-arch/guitar-0.1.4 for undeclared dependencies
>  *   /usr/bin/guitar links to /usr/lib64/libX11.so.6
>  *     Missing dependency on x11-libs/libX11
>  *   /usr/bin/guitar links to /usr/lib64/libXi.so.6
>  *     Missing dependency on x11-libs/libXi
>  *   /usr/bin/guitar links to /usr/lib64/libXext.so.6
>  *     Missing dependency on x11-libs/libXext
>  *   /usr/bin/guitar links to /usr/lib64/libxcb.so.1
>  *     Missing dependency on x11-libs/libxcb
>  *   /usr/bin/guitar links to /usr/lib64/libXau.so.6
>  *     Missing dependency on x11-libs/libXau
>  *   /usr/bin/guitar links to /usr/lib64/libXdmcp.so.6
>  *     Missing dependency on x11-libs/libXdmcp

Those look like indirect linker dependencies to me:

ohnopublishing mnt # readelf -d $(which guitar) | grep -e NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libgtk-1.2.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgdk-1.2.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libglib-1.2.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
ohnopublishing mnt #

See that X11 is brought in by libgtk and recursively so on and so on:
ohnopublishing mnt # readelf -d /usr/lib64/libgtk-1.2.so.0 | grep -e NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libgdk-1.2.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgmodule-1.2.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libglib-1.2.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libX11.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

-- 
binki

Look out for missing or extraneous apostrophes!

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Implicit system dependencies

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1796 bytes --]

On Tuesday, August 23, 2011 13:53:26 Michael wrote:
> Mike Frysinger wrote:
> > On Monday, August 22, 2011 15:21:23 Michael wrote:
> >> I wrote a script to search for discrepancies between linked libraries
> >> and what's defined in (R)DEPEND, with the intention of improving QA for
> >> minimal package installs.
> >> 
> >> The devmanual seems to suggest that it's a bad thing for packages to
> >> explicitly depend on packages in the system set, while many packages in
> >> the tree do indeed explicitly depend upon packages such as zlib and
> >> ncurses.
> > 
> > file bugs for missing zlib/ncurses deps.  if there are other packages
> > you're concerned with, ask now and we'll tell you.
> 
> What about packages such as bzip2, xz-utils, or really, app-arch/*?

assuming you're referring to them because the SRC_URI is compressed by the 
relevant formats, atm only bzip2 and gzip is allowed to be assumed.  
everything else has to be in DEPEND.  there is work/discussion to automate 
this (implicit unpack depends based on SRC_URI suffixes).

if your package is actually linking against any of the libs these guys 
provided (like libbz2.so), then you need to treat it like any other link-time 
dependency.

> Also, on a related note, would it be appropriate to report the following
> undeclared deps, given that the package in question already has DEPEND for
> gtk+?

like Nathan said, only if the package directly uses/links against the X11 
API/libs in question.  if the package only uses gtk+ funcs, then it should not 
be declaring a dependency on the X11 libs.

keep in mind that an app that only uses the gtk api can run on a system where 
gtk is not using X11 as its backend.  think raw framebuffer, win32, or 
something equally ridiculous.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Implicit system dependencies

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2292 bytes --]

On Tue, 23 Aug 2011 15:02:00 -0400
Nathan Phillip Brink <binki@gentoo.org> wrote:

> On Wed, Aug 24, 2011 at 03:53:26AM +1000, Michael wrote:
> > Also, on a related note, would it be appropriate to report the
> > following undeclared deps, given that the package in question
> > already has DEPEND for gtk+?
> > 
> > * Checking app-arch/guitar-0.1.4 for undeclared dependencies
> >  *   /usr/bin/guitar links to /usr/lib64/libX11.so.6
> >  *     Missing dependency on x11-libs/libX11
> >  *   /usr/bin/guitar links to /usr/lib64/libXi.so.6
> >  *     Missing dependency on x11-libs/libXi
> >  *   /usr/bin/guitar links to /usr/lib64/libXext.so.6
> >  *     Missing dependency on x11-libs/libXext
> >  *   /usr/bin/guitar links to /usr/lib64/libxcb.so.1
> >  *     Missing dependency on x11-libs/libxcb
> >  *   /usr/bin/guitar links to /usr/lib64/libXau.so.6
> >  *     Missing dependency on x11-libs/libXau
> >  *   /usr/bin/guitar links to /usr/lib64/libXdmcp.so.6
> >  *     Missing dependency on x11-libs/libXdmcp
> 
> Those look like indirect linker dependencies to me:
> 
> ohnopublishing mnt # readelf -d $(which guitar) | grep -e NEEDED
>  0x0000000000000001 (NEEDED)             Shared library:
> [libgtk-1.2.so.0] 0x0000000000000001 (NEEDED)             Shared
> library: [libgdk-1.2.so.0] 0x0000000000000001 (NEEDED)
> Shared library: [libglib-1.2.so.0] 0x0000000000000001
> (NEEDED)             Shared library: [libc.so.6] ohnopublishing mnt #
> 
> See that X11 is brought in by libgtk and recursively so on and so on:
> ohnopublishing mnt # readelf -d /usr/lib64/libgtk-1.2.so.0 | grep -e
> NEEDED 0x0000000000000001 (NEEDED)             Shared library:
> [libgdk-1.2.so.0] 0x0000000000000001 (NEEDED)             Shared
> library: [libgmodule-1.2.so.0] 0x0000000000000001
> (NEEDED)             Shared library: [libglib-1.2.so.0]
> 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
> 0x0000000000000001 (NEEDED)             Shared library: [libX11.so.6]
> 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

I think --as-needed may make a difference here. And often packages
supply broken pkgconfig files, with deps for static linking put into
the common ones.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

[gentoo-dev] Re: Implicit system dependencies

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

Il giorno mar, 23/08/2011 alle 05.21 +1000, Michael ha scritto:
> 
> Any thoughts to as how far I should go with filing bugs regarding
> this 
> issue? 

Please if you're going to file bugs about this, do so only on a system
built with _forced_ --as-needed, otherwise you're going to hit a long
list of false positives (gpgstats seems to be one of those, since it
does not use symbols from libassuan or gpg-error, even though it links
to those on your system.

Especially since I'm not sure most fo the developers would know how to
confirm your reports (hint: scanelf -n on the files on a system that is
built with forced --as-needed, I can provide such data myself from the
tinderbox).

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Implicit system dependencies

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On 2011-08-23 Tue 12:12, Mike Frysinger wrote:
> assuming you're referring to them because the SRC_URI is compressed by the 
> relevant formats, atm only bzip2 and gzip is allowed to be assumed.  
> everything else has to be in DEPEND.  there is work/discussion to automate 
> this (implicit unpack depends based on SRC_URI suffixes).

I thought xz-utils can be assumed as well since it was added to the
system set almost six months ago [1].

Tim

[1]: http://archives.gentoo.org/gentoo-dev/msg_998b4e7fdf578346bb5cfc66be340f7d.xml

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: Re: Implicit system dependencies

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

Il giorno mar, 23/08/2011 alle 23.52 -0700, Tim Harder ha scritto:
> I thought xz-utils can be assumed as well since it was added to the
> system set almost six months ago [1].

It has really very little to do with being in the system set or not. And
the answer is no, you cannot assume that...

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Re: Implicit system dependencies

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

On 2011-08-24 Wed 00:21, Diego Elio Pettenò wrote:
> Il giorno mar, 23/08/2011 alle 23.52 -0700, Tim Harder ha scritto:
> > I thought xz-utils can be assumed as well since it was added to the
> > system set almost six months ago [1].

> It has really very little to do with being in the system set or not. And
> the answer is no, you cannot assume that...

So then why do bzip2 and gzip get special exceptions? Majority rule? :)

Anyway, I think getting implicit unpack depends based on SRC_URI
suffixes like Mike mentioned will be great.

Tim

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: Implicit system dependencies

[Michael]

Diego Elio Pettenò wrote:

> Il giorno mar, 23/08/2011 alle 05.21 +1000, Michael ha scritto:
>> 
>> Any thoughts to as how far I should go with filing bugs regarding
>> this
>> issue?
> 
> Please if you're going to file bugs about this, do so only on a system
> built with _forced_ --as-needed, otherwise you're going to hit a long
> list of false positives (gpgstats seems to be one of those, since it
> does not use symbols from libassuan or gpg-error, even though it links
> to those on your system.
> 
> Especially since I'm not sure most fo the developers would know how to
> confirm your reports (hint: scanelf -n on the files on a system that is
> built with forced --as-needed, I can provide such data myself from the
> tinderbox).
> 

Fair enough, sorry for the bugspam, and thanks for the tip.

While I now see that gpgstats does not actually depend on libassuan or gpg-
error, the build process still fails without them:

x86_64-pc-linux-gnu-g++ -Wl,--hash-style=gnu -Wl,--as-needed -Wall -W 
error.o utils.o array.o iarray.o s.o -o gpgstats `gpgme-config --libs`
/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.3/../../../../x86_64-pc-linux-
gnu/bin/ld: cannot find -lassuan
collect2: ld returned 1 exit status
make: *** [gpgstats] Error 1
emake failed

Is /this/ considered to be a bug? Do I just write a patch to stop linking 
against libs that aren't needed?

Best regards,
Michael

Re: [gentoo-dev] Re: Re: Implicit system dependencies

[Brian Harring]

On Wed, Aug 24, 2011 at 12:31:29AM -0700, Tim Harder wrote:
> On 2011-08-24 Wed 00:21, Diego Elio Petten?? wrote:
> > Il giorno mar, 23/08/2011 alle 23.52 -0700, Tim Harder ha scritto:
> > > I thought xz-utils can be assumed as well since it was added to the
> > > system set almost six months ago [1].
> 
> > It has really very little to do with being in the system set or not. And
> > the answer is no, you cannot assume that...
> 
> So then why do bzip2 and gzip get special exceptions? Majority rule? :)

Age.  Making the mistake once doesn't mean we should make it twice. ;)

> Anyway, I think getting implicit unpack depends based on SRC_URI
> suffixes like Mike mentioned will be great.

Proposals welcome; my personal preference is to see extensions mapped 
into some form of repository configuration...

~harring

[gentoo-dev] Re: Implicit system dependencies

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

Il giorno mer, 24/08/2011 alle 19.49 +1000, Michael ha scritto:
> 
> Is /this/ considered to be a bug? Do I just write a patch to stop
> linking 
> against libs that aren't needed? 

Yes this is a bug in gpgme, it has to RDEPEND on libassuan rather than
just DEPEND on it, since it reports it to --libs.

Or it should simply not report it on --libs.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: Re: Implicit system dependencies

[Michael]

Mike Frysinger wrote:

> like Nathan said, only if the package directly uses/links against the X11
> API/libs in question.  if the package only uses gtk+ funcs, then it should
> not be declaring a dependency on the X11 libs.

Just to confirm (as I've had one such bug marked WONTFIX), should a package 
that depends on say, qt-gui (which in turn depends on qt-core), list an 
explicit dependency on qt-core if it uses symbols from it?

tinderbox ~ # readelf -d /usr/bin/lfhex | grep Qt
 0x0000000000000001 (NEEDED)             Shared library: [libQtGui.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libQtCore.so.4]

tinderbox ~ # equery g lfhex | grep qt
 [  1]  x11-libs/qt-gui-4.7.3-r1


Best regards,
Michael

Re: [gentoo-dev] Re: Re: Implicit system dependencies

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1358 bytes --]

On Sunday, August 28, 2011 14:38:31 Michael wrote:
> Mike Frysinger wrote:
> > like Nathan said, only if the package directly uses/links against the X11
> > API/libs in question.  if the package only uses gtk+ funcs, then it
> > should not be declaring a dependency on the X11 libs.
> 
> Just to confirm (as I've had one such bug marked WONTFIX), should a package
> that depends on say, qt-gui (which in turn depends on qt-core), list an
> explicit dependency on qt-core if it uses symbols from it?
> 
> tinderbox ~ # readelf -d /usr/bin/lfhex | grep Qt
>  0x0000000000000001 (NEEDED)             Shared library: [libQtGui.so.4]
>  0x0000000000000001 (NEEDED)             Shared library: [libQtCore.so.4]
> 
> tinderbox ~ # equery g lfhex | grep qt
>  [  1]  x11-libs/qt-gui-4.7.3-r1

i cant speak to this exact case, but the qt libs are a bit special.  upstream, 
it's really one package, but in Gentoo we've split them up into sep ones to 
make fixing/upgrading in the sub-libs easier/faster.  so by depending on any 
of the qt sublibs, they're hard bound to the qt-core as well.

also, your readelf doesnt exactly say that ifhex is using the qt-core API 
directly.  it could only be using qt-gui, but end up pulling in the others 
implicitly via the qt-gui APIs.  so in this specific case, qt-gui requirement 
only is OK.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]