[gentoo-dev] GLEP: Making updates never break dependencies

[gentoo-dev] GLEP: Making updates never break dependencies

[Klavs Klavsen]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Hi guys,

I just wrote a GLEP, proposing a solution to the problem that updating
f.ex. openssl, requires recompiling almost world
(kde,mozilla,galeon,wget,irrsi,python etc.) - and breaks your system if
you don't complete all this, while you are running (since the old
openssl files are removed after the openssl upgrade -and programs that
haven't been recompiled will break).

You can see it here:
http://vsen.dk/files/GLEP-Making_updates_never_break_dependencies.txt

I hope you like the idea, and the proposed solution. Please comment if
you have any questions, as I would very much like to see this happen, so
I can run emerge -u world again :)

-- 
Regards,
Klavs Klavsen, GSEC - kl@vsen.dk - http://www.vsen.dk
PGP: 7E063C62/2873 188C 968E 600D D8F8  B8DA 3D3A 0B79 7E06 3C62
See my new managed CMS Hosting Service at http://www.VirkPaaNettet.dk

Working with Unix is like wrestling a worthy opponent. 
Working with windows is like attacking a small whining child 
who is carrying a .38.				

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GLEP: Making updates never break dependencies

[Jon Portnoy]

On Thu, Nov 06, 2003 at 08:56:18AM +0100, Klavs Klavsen wrote:
> Hi guys,
> 
> I just wrote a GLEP, proposing a solution to the problem that updating
> f.ex. openssl, requires recompiling almost world
> (kde,mozilla,galeon,wget,irrsi,python etc.) - and breaks your system if
> you don't complete all this, while you are running (since the old
> openssl files are removed after the openssl upgrade -and programs that
> haven't been recompiled will break).
> 
> You can see it here:
> http://vsen.dk/files/GLEP-Making_updates_never_break_dependencies.txt
> 
> I hope you like the idea, and the proposed solution. Please comment if
> you have any questions, as I would very much like to see this happen, so
> I can run emerge -u world again :)
> 

Is OpenSSL broken? The OpenSSL ebuild installs 0.9.6 libs for 
compatibility to prevent breakage (if 0.9.6 was installed previously).

-- 
Jon Portnoy
avenj/irc.freenode.net

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP: Making updates never break dependencies

[Klavs Klavsen]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On tor, 2003-11-06 at 09:02, Jon Portnoy wrote:
[SNIP]
> Is OpenSSL broken? The OpenSSL ebuild installs 0.9.6 libs for 
> compatibility to prevent breakage (if 0.9.6 was installed previously).

It didn't on my machine. I just upgrade to 0.9.7c (got tired of having
to upgrade 0.9.6 all the time) - and it broke my
mozilla,python,wget,some-kde-stuff,galeon,sylpheed-claws etc.

Besides, such a general issue, as handling dependencies IMHO shouldn't
have to be handled by the individual ebuild (as it will be missed for
many) - when it can be handled in general and for all, by portage.
-- 
Regards,
Klavs Klavsen, GSEC - kl@vsen.dk - http://www.vsen.dk
PGP: 7E063C62/2873 188C 968E 600D D8F8  B8DA 3D3A 0B79 7E06 3C62
See my new managed CMS Hosting Service at http://www.VirkPaaNettet.dk

Working with Unix is like wrestling a worthy opponent. 
Working with windows is like attacking a small whining child 
who is carrying a .38.				

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: GLEP: Making updates never break dependencies

[Sebastian Bergmann]

Klavs Klavsen wrote:
> It didn't on my machine.

  Same here. After upgrading from 0.9.7c to 0.9.7c-r1 the 0.9.6 libraries
  are gone.

-- 
Sebastian Bergmann
http://sebastian-bergmann.de/                   http://phpOpenTracker.de/

Das Buch zu PHP 5: http://professionelle-softwareentwicklung-mit-php5.de/


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: GLEP: Making updates never break dependencies

[Paul de Vrieze]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 06 November 2003 10:10, Sebastian Bergmann wrote:
> Klavs Klavsen wrote:
> > It didn't on my machine.
>
>   Same here. After upgrading from 0.9.7c to 0.9.7c-r1 the 0.9.6
> libraries are gone.

I've just submitted a bug about -r1. Apparently the 0.9.6l version that 
is used in -r1 is broken. It compiles the libs fine and then goes on to 
delete them at the make test stage. I also added ||die statements to 
make sure that it doesn't merge when this is the case.

Paul

- -- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/qiOFbKx5DBjWFdsRAgm+AKC8xcG/46ou+WehTqMGNEMpzku4eACdEYNi
rXGbx0+Zzyk7WLfleBWlKPk=
=TiKi
-----END PGP SIGNATURE-----


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: GLEP: Making updates never break dependencies

[Klavs Klavsen]

[-- Attachment #1: Type: text/plain, Size: 1456 bytes --]

On tor, 2003-11-06 at 11:33, Paul de Vrieze wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Thursday 06 November 2003 10:10, Sebastian Bergmann wrote:
> > Klavs Klavsen wrote:
> > > It didn't on my machine.
> >
> >   Same here. After upgrading from 0.9.7c to 0.9.7c-r1 the 0.9.6
> > libraries are gone.
> 
> I've just submitted a bug about -r1. Apparently the 0.9.6l version that 
> is used in -r1 is broken.

And besides the ebuild maintainer shouldn't have to maintain such a
general thing - could be done using standard scripts.
ie. IMHO we need a generic solution for this problem, so every ebuild
maintainer doesn't have to implement his solution for the problem.

What do you guys think of the proposed implementation?

I will gladly implement shell-scritps (I don't know python) that do what
the GLEP proposes, as a v0.1 system, which portage could call upon.

>  It compiles the libs fine and then goes on to 
> delete them at the make test stage. I also added ||die statements to 
> make sure that it doesn't merge when this is the case.
> 
:)
-- 
Regards,
Klavs Klavsen, GSEC - kl@vsen.dk - http://www.vsen.dk
PGP: 7E063C62/2873 188C 968E 600D D8F8  B8DA 3D3A 0B79 7E06 3C62
See my new managed CMS Hosting Service at http://www.VirkPaaNettet.dk

Working with Unix is like wrestling a worthy opponent. 
Working with windows is like attacking a small whining child 
who is carrying a .38.				

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: GLEP: Making updates never break dependencies

[Paul de Vrieze]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 06 November 2003 11:38, Klavs Klavsen wrote:
>
> And besides the ebuild maintainer shouldn't have to maintain such a
> general thing - could be done using standard scripts.
> ie. IMHO we need a generic solution for this problem, so every ebuild
> maintainer doesn't have to implement his solution for the problem.
>
> What do you guys think of the proposed implementation?
>
> I will gladly implement shell-scritps (I don't know python) that do
> what the GLEP proposes, as a v0.1 system, which portage could call
> upon.
>

There are basically only two ways to do rebuilding when necessary:
- - use a revdep like approach. This does not work perfectly and is a big
  resource drain as all libraries and binaries need to be checked for a
  dependency on the lib. Further this does not work for on-demand loaded
  libraries so is only a partial "solution". It is basically a shotgun
  solution that still manages to miss things.

- - Devise a way that emerge knows (either automatically or specified) that
  the libraries in a package have changed and rebuild all packages that
  depend on the package. Finding out the change automatically should not
  be too complicated. However finding out which packages need to be
  rebuild is. Especially since dependencies can be indirect. This
  approach is a lot cleaner although even here there can be no guarantee
  of correctness.

I like the second way. However it requires significant changes to portage 
so is unlikely to happen soon.

Paul

- -- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/qik+bKx5DBjWFdsRAkcvAJ4yLDxeBrShCL71PPPAiUE9oW9gYwCfXSvA
AEn1Y9ApcssLG1I14uXaE1M=
=OPu2
-----END PGP SIGNATURE-----


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: GLEP: Making updates never break dependencies

[Klavs Klavsen]

[-- Attachment #1: Type: text/plain, Size: 2637 bytes --]

On tor, 2003-11-06 at 11:58, Paul de Vrieze wrote:
[SNIP]
> There are basically only two ways to do rebuilding when necessary:
> - - use a revdep like approach. This does not work perfectly and is a big
>   resource drain as all libraries and binaries need to be checked for a
>   dependency on the lib. Further this does not work for on-demand loaded
>   libraries so is only a partial "solution". It is basically a shotgun
>   solution that still manages to miss things.
> 
Well - one way of detecting this, could be to only look through every
installed packages USE flags (as used on compile) and look for the use
flag for the certain package. This will handle openssl upgrades atleast.
In regards to gcc and glibc, we know they all depend on it - so we
needn't check that much - only need to set a timestamp, and until every
program on the machine, has been recompiled (so their timestamp is newer
than when gcc/glibc was upgraded) the gcc or glibc libs, has to be
available.

IMHO we could devise a logic like this, that would work. If we need more
finetuned stuff, portage should save this info, while compiling the
package (as it saves USE flags and other things), so we can just wade
through the package-info files. Using a db3 or db4 format for this,
would probably not be all bad :)

v0.1 doesn't have to be bullitproof - it should just catch the obvious
ones (which are the ones which have their own useflag IMHO),and ofcourse
be the right way, so it can be improved upon for next release.

> - - Devise a way that emerge knows (either automatically or specified) that
>   the libraries in a package have changed and rebuild all packages that
>   depend on the package. Finding out the change automatically should not
>   be too complicated. However finding out which packages need to be
>   rebuild is. Especially since dependencies can be indirect. This
>   approach is a lot cleaner although even here there can be no guarantee
>   of correctness.
> 
> I like the second way. However it requires significant changes to portage 
> so is unlikely to happen soon.
> 
I believe qpkg implemented some reverse dependency thing - where it can
show you what packages depend on it - rather quickly. It may not be
perfect though ? 

-- 
Regards,
Klavs Klavsen, GSEC - kl@vsen.dk - http://www.vsen.dk
PGP: 7E063C62/2873 188C 968E 600D D8F8  B8DA 3D3A 0B79 7E06 3C62
See my new managed CMS Hosting Service at http://www.VirkPaaNettet.dk

Working with Unix is like wrestling a worthy opponent. 
Working with windows is like attacking a small whining child 
who is carrying a .38.				

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GLEP: Making updates never break dependencies

[Bob Miller]

Klavs Klavsen wrote:

> I just wrote a GLEP, proposing a solution to the problem that updating
> f.ex. openssl, requires recompiling almost world
> (kde,mozilla,galeon,wget,irrsi,python etc.) - and breaks your system if
> you don't complete all this, while you are running (since the old
> openssl files are removed after the openssl upgrade -and programs that
> haven't been recompiled will break).

Isn't this the problem that SLOTs are supposed to solve?

Shouldn't openssl and other libraries just be made SLOT-aware?

-- 
Bob Miller                              K<bob>
kbobsoft software consulting
http://kbobsoft.com                     kbob@jogger-egg.com

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP: Making updates never break dependencies

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]

On Thu, Nov 06, 2003 at 12:48:58PM -0800, Bob Miller wrote:
> > I just wrote a GLEP, proposing a solution to the problem that updating
> > f.ex. openssl, requires recompiling almost world
> > (kde,mozilla,galeon,wget,irrsi,python etc.) - and breaks your system if
> > you don't complete all this, while you are running (since the old
> > openssl files are removed after the openssl upgrade -and programs that
> > haven't been recompiled will break).
> Isn't this the problem that SLOTs are supposed to solve?
> Shouldn't openssl and other libraries just be made SLOT-aware?
this isn't a simple as you make it sound, because of the following (best
explained thru an example).

libmysql links against libssl-0.9.6, and say you only have openssl-0.9.6
installed at the time.
now you upgrade a slotted openssl, and have both libssl-0.9.6 and
libssl-0.9.7.
now you compile some other program that wants to pull in BOTH libssl and
libmysql. it picks up libssl-0.9.7 as that is the latest, but libmysql
is still linked against libssl-0.9.6. at this point your program
probably won't compile properly (gcc will detect the problem in most
cases), but if it doesn't your program will just segfault when you try
to run it.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [gentoo-dev] GLEP: Making updates never break dependencies

[Klavs Klavsen]

[-- Attachment #1: Type: text/plain, Size: 1685 bytes --]

On tor, 2003-11-06 at 22:03, Robin H. Johnson wrote:
[SNIP]
> 
> libmysql links against libssl-0.9.6, and say you only have openssl-0.9.6
> installed at the time.
> now you upgrade a slotted openssl, and have both libssl-0.9.6 and
> libssl-0.9.7.
> now you compile some other program that wants to pull in BOTH libssl and
> libmysql. it picks up libssl-0.9.7 as that is the latest, but libmysql
> is still linked against libssl-0.9.6. at this point your program
> probably won't compile properly (gcc will detect the problem in most
> cases), but if it doesn't your program will just segfault when you try
> to run it.

Exactly what I write in the GLEP too (I use arts and kde as an example
:) - my proposed solution is to put all "old libs" in a seperate
directory that is not in the LD_PATH - and not accessible by portage, to
make sure it doesn't pickup old libraries.
And to fix this issue, a list of programs that was compiled against the
old libraries should be created when the library (f.ex. openssl) is
upgraded, and portage can then easily check if the program you want to
compile depends on any of the programs that have been compiled with old
libraries, and require them to be recompiled too. Once a program is
recompiled with the new libraries, portage should just remove it from
the list.
-- 
Regards,
Klavs Klavsen, GSEC - kl@vsen.dk - http://www.vsen.dk
PGP: 7E063C62/2873 188C 968E 600D D8F8  B8DA 3D3A 0B79 7E06 3C62
See my new managed CMS Hosting Service at http://www.VirkPaaNettet.dk

Working with Unix is like wrestling a worthy opponent. 
Working with windows is like attacking a small whining child 
who is carrying a .38.				

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: GLEP: Making updates never break dependencies

[Klavs Klavsen]

[-- Attachment #1: Type: text/plain, Size: 2108 bytes --]

I have tried to address the issues you mentioned, and rewritten the GLEP
to be a IMHO much more precise in exactly how I see this being
accomplished.

Please have a look, and tell me what you think.
The rewritten GLEP is still here:
http://vsen.dk/files/GLEP-Making_updates_never_break_dependencies.txt

On tor, 2003-11-06 at 17:13, Grant Goodyear wrote:
> >  I just wrote a GLEP, proposing a solution to the problem that updating
> >  f.ex. openssl, requires recompiling almost world
> >  (kde,mozilla,galeon,wget,irrsi,python etc.) - and breaks your system if
> >  you don't complete all this, while you are running (since the old
> >  openssl files are removed after the openssl upgrade -and programs that
> >  haven't been recompiled will break).
> 
> Dear Mr. Klavsen,
>   Thank you very much for submitting this GLEP. I think it may need a
> bit of work before we can accept it, however. A minor point is that we
> request that GLEPs be written in a formal manner (no abbreviations such
> as "f.ex", spelling should be checked, sentences should be complete,
> subject and predicate should agree in number, etcetera). The more
> substantive comment is that this GLEP raises a number of issues that I
> think the GLEP itself should address: who will be making the required
> portage changes, how would these changes impact developers, how
> extensive would those changes be, how would these changes impact
> performance, are there other alternative solutions to this problem that
> should be considered, would the introduction of "reverse-dependencies"
> solve this problem?  I've CC'd carpaski, the Gentoo Portage lead, in
> case he might have additional comments.
> 
> Sincerely,
> Grant Goodyear
> GLEP Editor
-- 
Regards,
Klavs Klavsen, GSEC - kl@vsen.dk - http://www.vsen.dk
PGP: 7E063C62/2873 188C 968E 600D D8F8  B8DA 3D3A 0B79 7E06 3C62
See my new managed CMS Hosting Service at http://www.VirkPaaNettet.dk

Working with Unix is like wrestling a worthy opponent. 
Working with windows is like attacking a small whining child 
who is carrying a .38.				

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GLEP: Making updates never break dependencies

[Paul de Vrieze]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 06 November 2003 22:03, Robin H. Johnson wrote:
> this isn't a simple as you make it sound, because of the following
> (best explained thru an example).
>
> libmysql links against libssl-0.9.6, and say you only have
> openssl-0.9.6 installed at the time.
> now you upgrade a slotted openssl, and have both libssl-0.9.6 and
> libssl-0.9.7.
> now you compile some other program that wants to pull in BOTH libssl
> and libmysql. it picks up libssl-0.9.7 as that is the latest, but
> libmysql is still linked against libssl-0.9.6. at this point your
> program probably won't compile properly (gcc will detect the problem
> in most cases), but if it doesn't your program will just segfault when
> you try to run it.

btw. This is exactly the problem with berkeley db, and it is quite common 
for berkeley db.

Paul

- -- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/q170bKx5DBjWFdsRAhWSAKDmhbzBsYVjBG4at8IjRgZlnfZ7ugCeJNkm
yxhl49zVmkOkqQx5/WO9g2A=
=gxYs
-----END PGP SIGNATURE-----


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP: Making updates never break dependencies

[Klavs Klavsen]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

On fre, 2003-11-07 at 09:59, Paul de Vrieze wrote:
[SNIP]
> >
> > libmysql links against libssl-0.9.6, and say you only have
> > openssl-0.9.6 installed at the time.
> > now you upgrade a slotted openssl, and have both libssl-0.9.6 and
> > libssl-0.9.7.
> > now you compile some other program that wants to pull in BOTH libssl
> > and libmysql. it picks up libssl-0.9.7 as that is the latest, but
> > libmysql is still linked against libssl-0.9.6. at this point your
> > program probably won't compile properly (gcc will detect the problem
> > in most cases), but if it doesn't your program will just segfault when
> > you try to run it.
> 
> btw. This is exactly the problem with berkeley db, and it is quite common 
> for berkeley db.
> 
And for a lot of other programs unfortunately. This is the thing that
for me is the biggest show-stopper, for server systems - as it requires
a lot of wellplaced thought and work, before you upgrade f.ex. OpenSSL
(which has been requireing upgrades quite often recently :( )

I hope you can find time to look at my rewritten GLEP and see how you
like it. I've detailed how I think it should be implemented, and it is
in now way a performance killer, but AFAIK it should catch every
occurence of this problem.
-- 
Regards,
Klavs Klavsen, GSEC - kl@vsen.dk - http://www.vsen.dk
PGP: 7E063C62/2873 188C 968E 600D D8F8  B8DA 3D3A 0B79 7E06 3C62
See my new managed CMS Hosting Service at http://www.VirkPaaNettet.dk

Working with Unix is like wrestling a worthy opponent. 
Working with windows is like attacking a small whining child 
who is carrying a .38.				

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]