[gentoo-dev] [RFC] More reliable hiding preserved libraries

[gentoo-dev] [RFC] More reliable hiding preserved libraries

[Maciej Mrozowski]

[-- Attachment #1: Type: Text/Plain, Size: 2010 bytes --]

Problem

..is known, let me summarize briefly.

Uninstalling packages providing libraries, without checking reverse runtime 
dependencies of those packages leaves their dependencies unsatisfied (packages 
with broken executables and/or shared libs).
Some package managers try their best not to remove said libraries, yet 
allowing packages to be removed.
Those orphaned libraries cause problems[1] as build systems of some other 
packages being (re)installed afterwards pick them up and abuse those orphaned 
libraries. (we don't like orphans abused, we prefer them... "gone").

Solution

Now, I suppose there are some ideas how to make orphaned libraries not go in a 
way. Basically then need to be available for system, but hidden for "emerge".

There is opt-out suggestion[2], unfortunately it does not provide any info how 
exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, 
maybe - as Brian Harring suggested - sandbox could be used to somehow "hide" 
preserved libraries or preserved library directory from ebuild environment 
(preserved library directory a'ka "purgatory" - libs could be moved there when 
considered orphaned).

Opt-in suggestion is as follows:
1. Use some library path (read by ld loader from environment) and export it 
globally to environment pointing it to preserved library directory.
2. During "emerge", unset environment variable corresponding to said preserved 
library directory - orphans are no longer located.
Attached patch for glibc (2.11, but should apply to any other glibc around) 
and uClibc (this one is not tested but should work as well) that makes ld 
loader aware of LD_GENTOO_PRESERVED_LIBRARY_PATH variable.

(LD_LIBRARY_PATH would work as well, but it's being used widely so cannot be 
safely mangled.. on the second though it can - one could filter out preserved 
library paths from it during "emerge").

Thoughts?

1. https://bugs.gentoo.org/show_bug.cgi?id=240323
2. https://bugs.gentoo.org/show_bug.cgi?id=307391

-- 
regards
MM

[-- Attachment #2: glibc-2.11_ld-gentoo-preserved-library-path.patch --]
[-- Type: text/x-patch, Size: 2436 bytes --]

diff -ru ../glibc-2.11/elf/rtld.c ./elf/rtld.c
--- ../glibc-2.11/elf/rtld.c	2009-10-30 18:17:08.000000000 +0100
+++ ./elf/rtld.c	2010-04-03 10:51:46.468906521 +0200
@@ -874,6 +874,8 @@
 
 /* The library search path.  */
 static const char *library_path attribute_relro;
+/* Gentoo preserved library path.  */
+static const char *preserved_library_path attribute_relro;
 /* The list preloaded objects.  */
 static const char *preloadlist attribute_relro;
 /* Nonzero if information about versions has to be printed.  */
@@ -1385,7 +1387,18 @@
 
   /* Initialize the data structures for the search paths for shared
      objects.  */
-  _dl_init_paths (library_path);
+  char *llp = alloca ((library_path != NULL ? strlen (library_path) : 0) +
+    (preserved_library_path != NULL ? strlen (preserved_library_path) : 0) + 2);
+  *llp = '\0';
+  if (library_path != NULL)
+    {
+      strcat (llp, library_path);
+      if (preserved_library_path != NULL)
+        strcat (llp, ":");
+    }
+  if (preserved_library_path != NULL)
+    strcat (llp, preserved_library_path);
+  _dl_init_paths (llp);
 
   /* Initialize _r_debug.  */
   struct r_debug *r = _dl_debug_initialize (GL(dl_rtld_map).l_addr,
@@ -2648,6 +2661,16 @@
 	    mode = trace;
 	  break;
 
+	case 29:
+	  /* Read Gentoo preserved library path from env here. Watch out for
+	     possible future 'case' additions in this switch as well as in
+	     EXTRA_LD_ENVVARS defined in sysdeps/<arch>/dl-librecon.h.
+	     LD_GENTOO_PRESERVED_LIBRARY_PATH is blacklisted for SUID
+	     programs in sysdeps/generic/unsecvars.h.  */
+	  if (memcmp (envline, "GENTOO_PRESERVED_LIBRARY_PATH", 29) == 0)
+	    preserved_library_path = &envline[30];
+	  break;
+
 	  /* We might have some extra environment variable to handle.  This
 	     is tricky due to the pre-processing of the length of the name
 	     in the switch statement here.  The code here assumes that added
diff -ru ../glibc-2.11/sysdeps/generic/unsecvars.h ./sysdeps/generic/unsecvars.h
--- ../glibc-2.11/sysdeps/generic/unsecvars.h	2009-10-30 18:17:08.000000000 +0100
+++ ./sysdeps/generic/unsecvars.h	2010-04-03 10:48:07.192280437 +0200
@@ -10,6 +10,7 @@
   "LD_DEBUG_OUTPUT\0"							      \
   "LD_DYNAMIC_WEAK\0"							      \
   "LD_LIBRARY_PATH\0"							      \
+  "LD_GENTOO_PRESERVED_LIBRARY_PATH\0"					      \
   "LD_ORIGIN_PATH\0"							      \
   "LD_PRELOAD\0"							      \
   "LD_PROFILE\0"							      \

[-- Attachment #3: uClibc-0.9.20.1_ld-gentoo-preserved-library-path.patch --]
[-- Type: text/x-patch, Size: 4397 bytes --]

diff -ru ../uClibc-0.9.30.1/ldso/include/dl-hash.h ./ldso/include/dl-hash.h
--- ../uClibc-0.9.30.1/ldso/include/dl-hash.h	2008-11-03 16:41:17.000000000 +0100
+++ ./ldso/include/dl-hash.h	2010-04-03 11:23:44.026291507 +0200
@@ -132,6 +132,7 @@
 extern int _dl_linux_dynamic_link(void);
 
 extern char * _dl_library_path;
+extern char * preserved_library_path;
 extern char * _dl_not_lazy;
 
 static __inline__ int _dl_symbol(char * name)
diff -ru ../uClibc-0.9.30.1/ldso/include/ldso.h ./ldso/include/ldso.h
--- ../uClibc-0.9.30.1/ldso/include/ldso.h	2008-05-30 16:35:31.000000000 +0200
+++ ./ldso/include/ldso.h	2010-04-03 11:24:24.256679689 +0200
@@ -47,6 +47,7 @@
 
 /* Global variables used within the shared library loader */
 extern char *_dl_library_path;         /* Where we look for libraries */
+extern char *preserved_library_path;   /* Gentoo preserved library path */
 extern char *_dl_preload;              /* Things to be loaded before the libs */
 extern char *_dl_ldsopath;             /* Where the shared lib loader was found */
 extern const char *_dl_progname;       /* The name of the executable being run */
diff -ru ../uClibc-0.9.30.1/ldso/include/unsecvars.h ./ldso/include/unsecvars.h
--- ../uClibc-0.9.30.1/ldso/include/unsecvars.h	2008-01-09 07:59:58.000000000 +0100
+++ ./ldso/include/unsecvars.h	2010-04-03 11:29:03.513616705 +0200
@@ -14,6 +14,7 @@
 #define UNSECURE_ENVVARS \
 	"LD_PRELOAD\0" \
 	"LD_LIBRARY_PATH\0" \
+	"LD_GENTOO_PRESERVED_LIBRARY_PATH\0" \
 	"LD_DEBUG\0" \
 	"LD_DEBUG_OUTPUT\0" \
 	"LD_TRACE_LOADED_OBJECTS\0" \
diff -ru ../uClibc-0.9.30.1/ldso/ldso/dl-elf.c ./ldso/ldso/dl-elf.c
--- ../uClibc-0.9.30.1/ldso/ldso/dl-elf.c	2008-11-18 15:01:35.000000000 +0100
+++ ./ldso/ldso/dl-elf.c	2010-04-03 11:32:37.518900167 +0200
@@ -244,6 +244,12 @@
 			return tpnt1;
 		}
 	}
+	/* Lookup Gentoo preserved library path */
+	if (preserved_library_path) {
+		_dl_if_debug_dprint("\tsearching LD_GENTOO_PRESERVED_LIBRARY_PATH='%s'\n", preserved_library_path);
+		if ((tpnt1 = search_for_named_library(libname, secure, preserved_library_path, rpnt)) != NULL)
+			return tpnt1;
+	}
 
 	/*
 	 * The ABI specifies that RUNPATH is searched after LD_LIBRARY_PATH.
diff -ru ../uClibc-0.9.30.1/ldso/ldso/ldso.c ./ldso/ldso/ldso.c
--- ../uClibc-0.9.30.1/ldso/ldso/ldso.c	2009-02-23 09:38:23.000000000 +0100
+++ ./ldso/ldso/ldso.c	2010-04-03 11:34:00.513996158 +0200
@@ -43,6 +43,7 @@
 
 /* Global variables used within the shared library loader */
 char *_dl_library_path         = 0;	/* Where we look for libraries */
+char *preserved_library_path   = 0;	/* Gentoo preserved library path */
 char *_dl_preload              = 0;	/* Things to be loaded before the libs */
 char *_dl_ldsopath             = 0;	/* Location of the shared lib loader */
 int _dl_secure                 = 1;	/* Are we dealing with setuid stuff? */
@@ -204,6 +205,7 @@
 		_dl_secure = 0;
 		_dl_preload = _dl_getenv("LD_PRELOAD", envp);
 		_dl_library_path = _dl_getenv("LD_LIBRARY_PATH", envp);
+		preserved_library_path = _dl_getenv("LD_GENTOO_PRESERVED_LIBRARY_PATH", envp);
 	} else {
 		static const char unsecure_envvars[] =
 #ifdef EXTRA_UNSECURE_ENVVARS
@@ -221,6 +223,7 @@
 		} while (*nextp != '\0');
 		_dl_preload = NULL;
 		_dl_library_path = NULL;
+		preserved_library_path = NULL;
 		/* SUID binaries can be exploited if they do LAZY relocation. */
 		unlazy = RTLD_NOW;
 	}
diff -ru ../uClibc-0.9.30.1/ldso/libdl/libdl.c ./ldso/libdl/libdl.c
--- ../uClibc-0.9.30.1/ldso/libdl/libdl.c	2008-10-29 14:34:35.000000000 +0100
+++ ./ldso/libdl/libdl.c	2010-04-03 11:23:18.998879057 +0200
@@ -90,6 +90,7 @@
 void *(*_dl_malloc_function)(size_t);
 void (*_dl_free_function) (void *p);
 char *_dl_library_path         = 0;         /* Where we look for libraries */
+char *preserved_library_path   = 0;         /* Gentoo preserved library path */
 char *_dl_ldsopath             = 0;         /* Location of the shared lib loader */
 int _dl_errno                  = 0;         /* We can't use the real errno in ldso */
 size_t _dl_pagesize            = PAGE_SIZE; /* Store the page size for use later */
@@ -232,6 +233,7 @@
 #ifndef SHARED
 	/* When statically linked, the _dl_library_path is not yet initialized */
 	_dl_library_path = getenv("LD_LIBRARY_PATH");
+	preserved_library_path = getenv("LD_GENTOO_PRESERVED_LIBRARY_PATH");
 #endif
 
 	/* Try to load the specified library */

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

On Sat, Apr 03, 2010 at 12:38:17PM +0200, Maciej Mrozowski wrote:
> exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, 
> maybe - as Brian Harring suggested - sandbox could be used to somehow "hide" 
> preserved libraries or preserved library directory from ebuild environment 
> (preserved library directory a'ka "purgatory" - libs could be moved there when 
> considered orphaned).

When I've tried experimenting w/ this I've had zero luck in 
intercepting lib loadup on that one.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Fabian Groffen]

On 03-04-2010 12:38:17 +0200, Maciej Mrozowski wrote:
> Problem
> 
> ..is known, let me summarize briefly.
> 
> Uninstalling packages providing libraries, without checking reverse
> runtime dependencies of those packages leaves their dependencies
> unsatisfied (packages with broken executables and/or shared libs).
> Some package managers try their best not to remove said libraries, yet
> allowing packages to be removed.  Those orphaned libraries cause
> problems[1] as build systems of some other packages being
> (re)installed afterwards pick them up and abuse those orphaned
> libraries. (we don't like orphans abused, we prefer them... "gone").

Is it known why this does happen exactly?  When a lib is kept because it
is still used, only its soname + what the soname points to should be
kept.  That would mean the lib can no longer be found during linking,
unless you add some trickery (or does GNU ld do something "handy" out of
the box right here?).  So for example:

 % ls
 usr/lib/libfoo.so -> libfoo.so.2.1
 usr/lib/libfoo.so.2 -> libfoo.so.2.1
 usr/lib/libfoo.so.2.1

 % scanelf --soname usr/lib/libfoo.so
 libfoo.so.2 usr/lib/libfoo.so

what should kept preserved is:

 usr/lib/libfoo.so.2 -> libfoo.so.2.1
 usr/lib/libfoo.so.2.1

because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should
(in theory and on some platforms at least) fail.

> Solution
> 
> Now, I suppose there are some ideas how to make orphaned libraries not
> go in a way. Basically then need to be available for system, but
> hidden for "emerge".
> 
> There is opt-out suggestion[2], unfortunately it does not provide any
> info how exactly it's supposed to be achieved. As far as
> portage/pkgcore is concerned, maybe - as Brian Harring suggested -
> sandbox could be used to somehow "hide" preserved libraries or
> preserved library directory from ebuild environment (preserved library
> directory a'ka "purgatory" - libs could be moved there when considered
> orphaned).
> 
> Opt-in suggestion is as follows:
> 1. Use some library path (read by ld loader from environment) and export it 
> globally to environment pointing it to preserved library directory.

you mean: move preserved libs to some special place and have that place
being found at runtime somehow?

> 2. During "emerge", unset environment variable corresponding to said
> preserved library directory - orphans are no longer located.  Attached
> patch for glibc (2.11, but should apply to any other glibc around) and
> uClibc (this one is not tested but should work as well) that makes ld
> loader aware of LD_GENTOO_PRESERVED_LIBRARY_PATH variable.
> 
> (LD_LIBRARY_PATH would work as well, but it's being used widely so
> cannot be safely mangled.. on the second though it can - one could
> filter out preserved library paths from it during "emerge").

In general, LD_LIBRARY_PATH is considered harmful, and I wouldn't like
to see it being used for normal operation.

Instead I'd like to know first why applications can find retained libs,
because from the Portage side, in theory they shouldn't.  Maybe patching
GNU ld if it turns out being too smart may solve problems in a nicer way.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

On Sat, 3 Apr 2010 12:38:17 +0200
Maciej Mrozowski <reavertm@gmail.com> wrote:

> 2. During "emerge", unset environment variable corresponding to said
> preserved library directory - orphans are no longer located.

Wouldn't that cause failure when the toolkit relies on a 'hidden'
preserved library?

-- 
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Gilles Dartiguelongue]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

Le samedi 03 avril 2010 à 12:38 +0200, Maciej Mrozowski a écrit :
> There is opt-out suggestion[2], unfortunately it does not provide any info how 
> exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, 
> maybe - as Brian Harring suggested - sandbox could be used to somehow "hide" 
> preserved libraries or preserved library directory from ebuild environment 
> (preserved library directory a'ka "purgatory" - libs could be moved there when 
> considered orphaned).

that sounds nice, it would allow us to more easily spot
packages/upstreams doing it wrong (maybe that would work for packages
linking to themselves too btw)

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Maciej Mrozowski]

On Saturday 03 of April 2010 12:56:04 Fabian Groffen wrote:
> Is it known why this does happen exactly?  When a lib is kept because it
> is still used, only its soname + what the soname points to should be
> kept.  That would mean the lib can no longer be found during linking,
> unless you add some trickery (or does GNU ld do something "handy" out of
> the box right here?).  So for example:
> 
>  % ls
>  usr/lib/libfoo.so -> libfoo.so.2.1
>  usr/lib/libfoo.so.2 -> libfoo.so.2.1
>  usr/lib/libfoo.so.2.1
> 
>  % scanelf --soname usr/lib/libfoo.so
>  libfoo.so.2 usr/lib/libfoo.so
> 
> what should kept preserved is:
> 
>  usr/lib/libfoo.so.2 -> libfoo.so.2.1
>  usr/lib/libfoo.so.2.1
> 
> because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should
> (in theory and on some platforms at least) fail.

It doesn't matter, as 'broken' build system may alphabetically find library by 
file name, and link to this library by full path.

On Saturday 03 of April 2010 13:13:00 Michał Górny wrote:
> > 2. During "emerge", unset environment variable corresponding to said
> > preserved library directory - orphans are no longer located.
> Wouldn't that cause failure when the toolkit relies on a 'hidden'
> preserved library?

It would indeed. Now when I think about it, moving stuff to preserved library 
dir could be just done - provided it's possible - along with fixing/setting 
DT_RPATH's in reverse runtime dependencies. This way no system-wide 
LIBRARY_PATH's would be necessary.
Is it possible? Mike?

On Saturday 03 of April 2010 13:33:16 Gilles Dartiguelongue wrote:
> > There is opt-out suggestion[2], unfortunately it does not provide any
> > info how exactly it's supposed to be achieved. As far as portage/pkgcore
> > is concerned, maybe - as Brian Harring suggested - sandbox could be used
> > to somehow "hide" preserved libraries or preserved library directory
> > from ebuild environment (preserved library directory a'ka "purgatory" -
> > libs could be moved there when considered orphaned).
> that sounds nice, it would allow us to more easily spot
> packages/upstreams doing it wrong (maybe that would work for packages
> linking to themselves too btw)

Keeping preserved libraries in separate location (in "purgatory" or dumping 
place) is just a method for making implementation possibly easier (or possible 
at all), with nice side effects though.

-- 
regards
MM

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Fabian Groffen]

On 03-04-2010 14:09:42 +0200, Maciej Mrozowski wrote:
> > because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should
> > (in theory and on some platforms at least) fail.
> 
> It doesn't matter, as 'broken' build system may alphabetically find
> library by file name, and link to this library by full path.

Shouldn't we fix that buildsystem then?  Do you have an example of a
package/buildsystem that does that?

> On Saturday 03 of April 2010 13:13:00 Michał Górny wrote:
> > > 2. During "emerge", unset environment variable corresponding to said
> > > preserved library directory - orphans are no longer located.
> > Wouldn't that cause failure when the toolkit relies on a 'hidden'
> > preserved library?
> 
> It would indeed. Now when I think about it, moving stuff to preserved library 
> dir could be just done - provided it's possible - along with fixing/setting 
> DT_RPATH's in reverse runtime dependencies. This way no system-wide 
> LIBRARY_PATH's would be necessary.
> Is it possible? Mike?

No, unless you somehow make sure you reserve space for this, by e.g.
setting a bogus rpath entry at buildtime.  If you want to go that route,
you probably want to look at the Prefix' binutils-config wrapper that
already calls the linker with added rpath arguments.  Afterwards you can
use chrpath to set it to the correct location.  Will get messy with the
vdb though, but if Portage's doing it, it can probably be dealt with.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 2665 bytes --]

Am Samstag, den 03.04.2010, 12:38 +0200 schrieb Maciej Mrozowski:
> Problem
> 
> ..is known, let me summarize briefly.
> 
> Uninstalling packages providing libraries, without checking reverse runtime 
> dependencies of those packages leaves their dependencies unsatisfied (packages 
> with broken executables and/or shared libs).
> Some package managers try their best not to remove said libraries, yet 
> allowing packages to be removed.
> Those orphaned libraries cause problems[1] as build systems of some other 
> packages being (re)installed afterwards pick them up and abuse those orphaned 
> libraries. (we don't like orphans abused, we prefer them... "gone").
> 
> Solution
> 
> Now, I suppose there are some ideas how to make orphaned libraries not go in a 
> way. Basically then need to be available for system, but hidden for "emerge".
> 
> There is opt-out suggestion[2], unfortunately it does not provide any info how 
> exactly it's supposed to be achieved. As far as portage/pkgcore is concerned, 
> maybe - as Brian Harring suggested - sandbox could be used to somehow "hide" 
> preserved libraries or preserved library directory from ebuild environment 
> (preserved library directory a'ka "purgatory" - libs could be moved there when 
> considered orphaned).
> 
> Opt-in suggestion is as follows:
> 1. Use some library path (read by ld loader from environment) and export it 
> globally to environment pointing it to preserved library directory.
> 2. During "emerge", unset environment variable corresponding to said preserved 
> library directory - orphans are no longer located.
> Attached patch for glibc (2.11, but should apply to any other glibc around) 
> and uClibc (this one is not tested but should work as well) that makes ld 
> loader aware of LD_GENTOO_PRESERVED_LIBRARY_PATH variable.
> 
> (LD_LIBRARY_PATH would work as well, but it's being used widely so cannot be 
> safely mangled.. on the second though it can - one could filter out preserved 
> library paths from it during "emerge").
> 
> Thoughts?

Don't fix the hack. Remove the preserve libs "feature", make the PMs
check for rdeps per default before unmerging things. Slot libraries
where needed, slot dep operators (EAPI 4) will help. And if that doesn't
work out we need a separate var to give the PM a hint when API/ABI
breakages happen (such that the PM knows when to re-install the rev
deps).


-- 
Tiziano Müller
Gentoo Linux Developer
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3551 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Maciej Mrozowski]

On Saturday 03 of April 2010 14:16:14 Fabian Groffen wrote:
> Shouldn't we fix that buildsystem then?  Do you have an example of a
> package/buildsystem that does that?
"We" already do, the thing is that maybe we don't have to.
https://bugs.gentoo.org/show_bug.cgi?id=240323
From top of my head: python having issues with sys-libs/db as well as some 
packages with readline.
 
> > It would indeed. Now when I think about it, moving stuff to preserved
> > library dir could be just done - provided it's possible - along with
> > fixing/setting DT_RPATH's in reverse runtime dependencies. This way no
> > system-wide LIBRARY_PATH's would be necessary.
> > Is it possible? Mike?
> No, unless you somehow make sure you reserve space for this, by e.g.
> setting a bogus rpath entry at buildtime.  If you want to go that route,
> you probably want to look at the Prefix' binutils-config wrapper that
> already calls the linker with added rpath arguments.  Afterwards you can
> use chrpath to set it to the correct location.  Will get messy with the
> vdb though, but if Portage's doing it, it can probably be dealt with.
Sounds messy indeed, what about hardened/SELinux/AppArmor/whatever - do they 
allow such DT_RPATH operations? It should be probably also restricted for 
binary-only packages.

On Saturday 03 of April 2010 20:51:43 Tiziano Müller wrote:
> Don't fix the hack. Remove the preserve libs "feature", make the PMs
> check for rdeps per default before unmerging things.
This will only prevent creating orphans of uninstalled libraries, what about 
upgraded ones when SOVERSION has been bumped (the most common case)? Besides I 
can already imagine PMS-related discussion regarding "make the PMs check for 
rdeps per default before unmerging things" - thx but no thx.

> Slot libraries where needed, slot dep operators (EAPI 4) will help.
Again, you suggest to SLOT every library that happened to bump SOVERSION. It's 
unrealistic. Besides library should be slotted when it's API changes, for 
source based distributions it's not needed for ABI changes - let's not confuse 
those two. Also excessive slotting increases probability of breaking library 
discovery mechanisms in various build systems (not everyone uses pkg-config).

> And if that doesn't work out we need a separate var to give the PM a hint 
when API/ABI breakages happen (such that the PM knows when to re-install the 
rev deps).
It needs PMS amended and thus GLEP. Please submit a GLEP item for this if you 
see it fit.
Anyway, as explained in OT, it's not a problem of package manager dependencies 
system but issue with broken/not smart build systems - no dependency tree 
magic will solve this issue.

-- 
regards
MM

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 4019 bytes --]

Am Samstag, den 03.04.2010, 23:05 +0200 schrieb Maciej Mrozowski:
> On Saturday 03 of April 2010 14:16:14 Fabian Groffen wrote:
> > Shouldn't we fix that buildsystem then?  Do you have an example of a
> > package/buildsystem that does that?
> "We" already do, the thing is that maybe we don't have to.
> https://bugs.gentoo.org/show_bug.cgi?id=240323
> From top of my head: python having issues with sys-libs/db as well as some 
> packages with readline.
>  
> > > It would indeed. Now when I think about it, moving stuff to preserved
> > > library dir could be just done - provided it's possible - along with
> > > fixing/setting DT_RPATH's in reverse runtime dependencies. This way no
> > > system-wide LIBRARY_PATH's would be necessary.
> > > Is it possible? Mike?
> > No, unless you somehow make sure you reserve space for this, by e.g.
> > setting a bogus rpath entry at buildtime.  If you want to go that route,
> > you probably want to look at the Prefix' binutils-config wrapper that
> > already calls the linker with added rpath arguments.  Afterwards you can
> > use chrpath to set it to the correct location.  Will get messy with the
> > vdb though, but if Portage's doing it, it can probably be dealt with.
> Sounds messy indeed, what about hardened/SELinux/AppArmor/whatever - do they 
> allow such DT_RPATH operations? It should be probably also restricted for 
> binary-only packages.
> 
> On Saturday 03 of April 2010 20:51:43 Tiziano Müller wrote:
> > Don't fix the hack. Remove the preserve libs "feature", make the PMs
> > check for rdeps per default before unmerging things.
> This will only prevent creating orphans of uninstalled libraries, what about 
> upgraded ones when SOVERSION has been bumped (the most common case)?
I addressed this in the next phrase.

>  Besides I 
> can already imagine PMS-related discussion regarding "make the PMs check for 
> rdeps per default before unmerging things" - thx but no thx.
This is not related to PMS. Paludis for example does it already with the
current EAPIs.

> 
> > Slot libraries where needed, slot dep operators (EAPI 4) will help.
> Again, you suggest to SLOT every library that happened to bump SOVERSION. It's 
> unrealistic. Besides library should be slotted when it's API changes, for 
> source based distributions it's not needed for ABI changes - let's not confuse 
> those two. Also excessive slotting increases probability of breaking library 
> discovery mechanisms in various build systems (not everyone uses pkg-config).
I know that slotting can cause problems. But forcing build systems to
use specific versions of libraries is much easier than "shadowing"
libraries present in library search dirs, especially when those libs are
not even tracked by the PM.

> 
> > And if that doesn't work out we need a separate var to give the PM a hint 
> when API/ABI breakages happen (such that the PM knows when to re-install the 
> rev deps).
> It needs PMS amended and thus GLEP.
Wrong, we don't have GLEPs for >90% of the PMS changes going into EAPI-3
or 4.

>  Please submit a GLEP item for this if you 
> see it fit.
> Anyway, as explained in OT, it's not a problem of package manager dependencies 
> system but issue with broken/not smart build systems - no dependency tree 
> magic will solve this issue.
It is a problem which can _only_ be solved at the PM level. You have to
tell the PM when API breakages happen. Either by slotting the lib or by
something else. Using guesswork to determine whether or not a library
should be removed may be a temporary solution. Remember: you're
partially ignoring a users request since he wanted the package to be
removed - and we all know how things like "protect the user from
himself" end up.




-- 
Tiziano Müller
Gentoo Linux Developer
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3551 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Maciej Mrozowski]

On Sunday 04 of April 2010 17:33:17 Tiziano Müller wrote:

>>  Besides I 
>> can already imagine PMS-related discussion regarding "make the PMs check 
for rdeps per default before unmerging things" - thx but no thx.
> This is not related to PMS. Paludis for example does it already with the
> current EAPIs.
So how does Paludis handle those issues?
(Read my comments about "emerge" atomicy below)

> It is a problem which can _only_ be solved at the PM level. You have to
> tell the PM when API breakages happen. Either by slotting the lib or by
> something else.
^^^^^^^^^^^^^^^
This is important - as slotting is not a practical solution. What is it then?

> Using guesswork to determine whether or not a library
> should be removed may be a temporary solution. Remember: you're
> partially ignoring a users request since he wanted the package to be
> removed - and we all know how things like "protect the user from
> himself" end up.

Unconditionally removing libraries (instead of preserving them) and making 
their reverse runtime dependencies reinstalled is unacceptable because 
"emerge" process involving multiple packages is not atomic. Simple as that.
Is this what you suggest? Correct me if I'm wrong:
1. Users wants to uninstall or reinstall package, we let him do it provided 
reverse runtime dependencies are satisfied afterwards. Let's say he wants to 
upgrade expat.
2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime 
reverse deps will still be satisfied when we upgrade.
3. Expat has been upgraded sucessfully,
4a. "emerge" discovers reverse runtime dependencies are broken and it starts 
to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not 
found. Because step 3 cannot be rolled back (no atomicy) - game over.
or
4b. "emerge does not discover those and does nothing. python is broken so 
emerge cannot be used anymore. Game over

-- 
regards
MM

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]

On Mon, Apr 05, 2010 at 08:16:42AM +0200, Maciej Mrozowski wrote:
> Unconditionally removing libraries (instead of preserving them) and making 
> their reverse runtime dependencies reinstalled is unacceptable because 
> "emerge" process involving multiple packages is not atomic. Simple as that.
> Is this what you suggest? Correct me if I'm wrong:
> 1. Users wants to uninstall or reinstall package, we let him do it provided 
> reverse runtime dependencies are satisfied afterwards. Let's say he wants to 
> upgrade expat.
> 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime 
> reverse deps will still be satisfied when we upgrade.
> 3. Expat has been upgraded sucessfully,
> 4a. "emerge" discovers reverse runtime dependencies are broken and it starts 
> to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not 
> found. Because step 3 cannot be rolled back (no atomicy) - game over.
> or
> 4b. "emerge does not discover those and does nothing. python is broken so 
> emerge cannot be used anymore. Game over

This is called 'nondeterministic resolution'- known issue also w/ 
proposals of that sort.

Pretty much everytime someone proposes it as a solution, it gets 
smacked down by most folk since an emerge -p invocation that is a 
single pkg upgade shouldn't be able to go rebuild your entire world.

The alternative is a slotted ABI var- basically a counter (although it 
doesn't have to be) w/in ebuilds themselves to indicate if they're 
carrying a new ABI from upstream for that slotting.  For example, 
you've got EXPAT merged w/ ABI=2, version 2.0.  version 2.0.1, for 
whatever reason, breaks ABI- thus v2.0.1 in the tree is ABI=2.0.1 (or 
3, as said it's an arbitrary value).

Via that, the resolver can see that a rebuild is necessary and plan a 
rebuild of all consumers (whether NEEDED based or revdep).  Note 
preserve-lib would be rather useful here- specifically holding onto 
the intermediate lib while doing rebuilding.  This however breaks down 
a bit when the ABI change is in reverse of normal versioning.

Usually whenever I've floated this idea, it's gotten smacked down by 
devs due to the extra work it may entail- someone doing an experiment 
on this would be rather useful (someone with a tinderbox 
specifically).

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 2748 bytes --]

Am Sonntag, den 04.04.2010, 23:44 -0700 schrieb Brian Harring:
> On Mon, Apr 05, 2010 at 08:16:42AM +0200, Maciej Mrozowski wrote:
> > Unconditionally removing libraries (instead of preserving them) and making 
> > their reverse runtime dependencies reinstalled is unacceptable because 
> > "emerge" process involving multiple packages is not atomic. Simple as that.
> > Is this what you suggest? Correct me if I'm wrong:
> > 1. Users wants to uninstall or reinstall package, we let him do it provided 
> > reverse runtime dependencies are satisfied afterwards. Let's say he wants to 
> > upgrade expat.
> > 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime 
> > reverse deps will still be satisfied when we upgrade.
> > 3. Expat has been upgraded sucessfully,
> > 4a. "emerge" discovers reverse runtime dependencies are broken and it starts 
> > to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not 
> > found. Because step 3 cannot be rolled back (no atomicy) - game over.
> > or
> > 4b. "emerge does not discover those and does nothing. python is broken so 
> > emerge cannot be used anymore. Game over
> 
> This is called 'nondeterministic resolution'- known issue also w/ 
> proposals of that sort.
> 
> Pretty much everytime someone proposes it as a solution, it gets 
> smacked down by most folk since an emerge -p invocation that is a 
> single pkg upgade shouldn't be able to go rebuild your entire world.
> 
> The alternative is a slotted ABI var- basically a counter (although it 
> doesn't have to be) w/in ebuilds themselves to indicate if they're 
> carrying a new ABI from upstream for that slotting.  For example, 
> you've got EXPAT merged w/ ABI=2, version 2.0.  version 2.0.1, for 
> whatever reason, breaks ABI- thus v2.0.1 in the tree is ABI=2.0.1 (or 
> 3, as said it's an arbitrary value).
> 
> Via that, the resolver can see that a rebuild is necessary and plan a 
> rebuild of all consumers (whether NEEDED based or revdep).  Note 
> preserve-lib would be rather useful here- specifically holding onto 
> the intermediate lib while doing rebuilding.
No, it doesn't help since you may have the same problems some people try
to solve in this thread.

>   This however breaks down 
> a bit when the ABI change is in reverse of normal versioning.
How so? Such a var should just specify the ABI and the PM only has to
check whether it changed from one PVR to the other. The "how" is
completely irrelevant.


-- 
Tiziano Müller
Gentoo Linux Developer
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3551 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 2678 bytes --]

Am Montag, den 05.04.2010, 08:16 +0200 schrieb Maciej Mrozowski:
> On Sunday 04 of April 2010 17:33:17 Tiziano Müller wrote:
> 
> >>  Besides I 
> >> can already imagine PMS-related discussion regarding "make the PMs check 
> for rdeps per default before unmerging things" - thx but no thx.
> > This is not related to PMS. Paludis for example does it already with the
> > current EAPIs.
> So how does Paludis handle those issues?
> (Read my comments about "emerge" atomicy below)
> 
> > It is a problem which can _only_ be solved at the PM level. You have to
> > tell the PM when API breakages happen. Either by slotting the lib or by
> > something else.
> ^^^^^^^^^^^^^^^
> This is important - as slotting is not a practical solution. What is it then?
> 
> > Using guesswork to determine whether or not a library
> > should be removed may be a temporary solution. Remember: you're
> > partially ignoring a users request since he wanted the package to be
> > removed - and we all know how things like "protect the user from
> > himself" end up.
> 
> Unconditionally removing libraries (instead of preserving them) and making 
> their reverse runtime dependencies reinstalled is unacceptable because 
> "emerge" process involving multiple packages is not atomic. Simple as that.
A side mark: preserving libs may work in a number of cases, but there
are a lot of (possible) examples where the rdeps (and/or the preserved
libs) are nevertheless broken or become useless (think of: ui-files,
plugins, dlopen'ed libs, etc.).

Please forget your atomicity. It's a really nice idea (and I'd like to
see it too) but not possible now or in the near future.


> Is this what you suggest? Correct me if I'm wrong:
> 1. Users wants to uninstall or reinstall package, we let him do it provided 
> reverse runtime dependencies are satisfied afterwards. Let's say he wants to 
> upgrade expat.
> 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime 
> reverse deps will still be satisfied when we upgrade.
> 3. Expat has been upgraded sucessfully,
> 4a. "emerge" discovers reverse runtime dependencies are broken and it starts 
> to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not 
> found. Because step 3 cannot be rolled back (no atomicy) - game over.
> or
In that case you most probably have a problem in your dep-tree (either
unspecified deps or a dep-resolver bug)

-- 
Tiziano Müller
Gentoo Linux Developer
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3551 bytes --]

Re: [gentoo-dev] [RFC] More reliable hiding preserved libraries

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On Mon, Apr 05, 2010 at 03:27:34PM +0200, Tiziano MMMller wrote:
> > Via that, the resolver can see that a rebuild is necessary and plan a 
> > rebuild of all consumers (whether NEEDED based or revdep).  Note 
> > preserve-lib would be rather useful here- specifically holding onto 
> > the intermediate lib while doing rebuilding.
>
> No, it doesn't help since you may have the same problems some people try
> to solve in this thread.

Might I suggest in the future purning down what you're responding to, 
to just that blurb?  At first read of your response I thought you were 
arguing against the ABI var itself (which didn't make a helluva lot of 
sense).

Meanwhile, yes, using preserve lib there still has some issues- the 
reason I mentioned it possibly being held onto is that if we're 
talking about something like openssl, having your system be horked for 
the intervening period isn't a great thing thus it may be useful as an 
option at the least.

> >   This however breaks down 
> > a bit when the ABI change is in reverse of normal versioning.
> How so? Such a var should just specify the ABI and the PM only has to
> check whether it changed from one PVR to the other. The "how" is
> completely irrelevant.

That comment is in reference to if preserve-lib is still enabled for 
the rebuild.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]