From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id CD4A713877A for ; Fri, 1 Aug 2014 19:47:44 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id DA501E09D0; Fri, 1 Aug 2014 19:47:41 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 1F855E098E for ; Fri, 1 Aug 2014 19:47:41 +0000 (UTC) Received: from pomiot.lan (77-254-77-139.adsl.inetia.pl [77.254.77.139]) (using SSLv3 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: mgorny) by smtp.gentoo.org (Postfix) with ESMTPSA id 34A3B3400F1; Fri, 1 Aug 2014 19:47:39 +0000 (UTC) Date: Fri, 1 Aug 2014 21:47:51 +0200 From: =?ISO-8859-2?B?TWljaGGzIEfzcm55?= To: Michael Palimaka Cc: gentoo-project@lists.gentoo.org Subject: Re: [gentoo-project] Re: Call for agenda items - Council meeting 2014-08-12 Message-ID: <20140801214751.76a4e8d9@pomiot.lan> In-Reply-To: References: <21463.26330.847055.224071@a1i15.kph.uni-mainz.de> <20140731211239.0eb282cb@pomiot.lan> Organization: Gentoo X-Mailer: Claws Mail 3.10.1 (GTK+ 2.24.24; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Project discussion list X-BeenThere: gentoo-project@lists.gentoo.org Reply-To: gentoo-project@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/iCpzcB_a06//LPB505RwAZx"; protocol="application/pgp-signature" X-Archives-Salt: a112c378-f6f1-49ae-92bb-936f5659857e X-Archives-Hash: 3ee6be0ef093e89184d66a0e158a50e3 --Sig_/iCpzcB_a06//LPB505RwAZx Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Dnia 2014-08-01, o godz. 05:52:40 Michael Palimaka napisa=B3(a): > On 08/01/2014 05:12 AM, Micha=B3 G=F3rny wrote: > =20 > >> If we are to change our default dependency model, we need to do it > >> properly - we need wider consensus, more open discussion of what's > >> happening, and a proper plan of how to handle the pitfalls of the new > >> model. Otherwise, we're just trading one set of problems for another. = =20 > >=20 > > Yes, exactly. We need to get dynamic-deps right if they are ever > > supposed to become the default. That's one of the reasons that we want > > to revert the problematic changes and make Portage use the default > > model once again. =20 > What do you mean? Dynamic dependencies are the current default by > definition. =20 Unless I've missed something big, dynamic dependencies were enabled in Portage without prior discussion or much testing. The developers started to rely on it though that was never desired or agreed on. As far as I'm aware, the Council never voted on the matter but it rather spread from developer to developer. This also implies that it was never properly described, and most of the developers do not know the details or the pitfalls. That said, I'd dare say that the Gentoo contract still obliges us to follow PMS, and therefore the dependency model implied by PMS is the default one. > >> Specifically, I request the Council block the removal of dynamic > >> dependencies until the following issues are resolved: > >> > >> 1. Although there has been considerable discussion[2] regarding dynamic > >> dependencies in general, there has been no specific discussion regardi= ng > >> their actual removal. =20 > >=20 > > The thread pretty quickly turned into a bikeshed about that, so I don't > > understand what you're talking about. =20 > Perhaps I missed it due to the size of the thread, but the discussion > appeared to be more abstract rather than concerning any definite change. > I see relatively little comment from Portage team members in any case. > =20 > >> 2. The Portage team had made no announcement of their decision, nor do > >> they apparently intend to until 30 days prior to a new Portage release. > >> This is not adequate time for such a substantial change. =20 > >=20 > > I don't know what you mean by 'they apparently intend' but that sounds > > offensive. I'd really prefer if you sticked to the facts. =20 > How is that offensive? It is the apparent intention I've been able to > discern from the limited information the Portage team is providing. > =20 > > The Portage team is going to announce the decision when it's final > > and the code necessary for non-painful migration is in place. > > Otherwise, the announcement would only bring barren discussion that > > couldn't be supported by facts or numbers. =20 > This is not acceptable for such a significant change. =20 I think we misunderstand each other. The Portage team is quite small, and has a lot of work to do. As I mentioned, we are still collecting data and focusing on having the code necessary to collect it. In particular, today I've submitted the last patch necessary for @changed-deps set to work properly. By using this set, we can check how many packages had dependencies updated without revision bump and therefore estimate the scale of the problem. Furthermore, if dynamic dependencies are disabled in Portage, it will help users fixing the inconsistencies in their vdb which may prevent emerge from working properly afterwards. I simply meant that we haven't announced anything yet because we're nowhere close to doing this. We don't want to bother people too much before we know the scale of the problem, and before we can provide real data and suggestions. > >> 3. Few of the Portage team members were present[3] for the meeting, and > >> no vote was held to reach the decision. =20 > >=20 > > I don't understand how this is relevant. =20 > It's not acceptable for one or two people to make a decision that > affects the entire distribution, let alone if they don't have the > consensus of their own team. =20 While I don't want to diminish the role of the remaining team members, I would like to point out that those two were the team leads and the most active members. And those members didn't reach a strong conclusion -- as you pointed out, they never even voted. We just agreed on a goal we're going to work on reaching. > >> 4. While there is a good description of the theoretical issues affecti= ng > >> both dependency models[4], multiple requests for specific examples of > >> in-tree breakage have been ignored. This makes it difficult to assess > >> the actual breakage that removing dynamic dependencies is supposed to > >> address. =20 > >=20 > > The listing of theoretical issues is wrong and doesn't correspond to > > Portage code, and yes, that's my fault. However, it only proves that > > nobody on the thread bothered to look at the relevant Portage code, > > even the people who started providing patches... =20 > The burden of proof is on the person wanting to make the change, so it > would be nice if the Portage team would provide better information. > Despite repeated requests, we have little information to substantiate > claims like "it's fundamentally broken" and "dynamic dependencies is a > bug. We decided to fix this regression." =20 There are basically two sides to the issue: the issues with dynamic dependencies themselves and the issues caused by developers relying on them. An elaborate listing would likely require much more skill than I have. However, a short listing would include: 1. the dependency tree for installed packages is non-deterministic. It can suffer rapid changes due to seemingly irrelevant events, most importantly -- inability to find the original ebuild. This involves not only removal of ebuilds but also temporary changes to PORTDIR_OVERLAY, for example. Users don't expect that and the issues are very hard to debug -- why is A pulling in non-existing package B? it's nowhere in A's deps. 2. dynamic dependencies allow for dependencies stored in vdb to become inconsistent. I mean, since portage usually doesn't use them and the developers change dependencies in-place, the vdb dependency tree may contain completely impossible combinations of packages (depending on the time a particular package was installed). The problem is that only emerge does this, and the vdb is simply broken to any other package manager or tool -- including portage-utils (AFAICS 'qdepends -Q' gives meaningless output nowadays), eix, gentoolkit. 3. most of the developers don't understand when the revbump becomes necessary because of dynamic-deps (i.e. when you don't want them to apply). This occurs pretty rarely yet it's something I doubt we will ever be able to change. It is simply too confusing. 4. getting :=3D and dynamic-deps working together is pretty hard, and the code doing that in Portage is not very good. I suspect that the resulting dependency trees may be the cause of some issues people are reporting. If backtracking is involved, dynamic-deps are also likely to cause much longer dependency resolution. However, it's all very complex and hard to debug, so I have no definitive proof. I believe that fixing many of the issues in Portage will require severe changes to the code. Disabling the dynamic dependency support will likely make debugging easier for us, and therefore get us closer to having properly working dependency resolver. Additionally, disabled dynamic dependencies give users the ability to choose any package manager for Gentoo. With dynamic dependencies enabled, emerge breaks vdb pretty soon (as pointed out in (2)) which in turn breaks other package managers, and leaves user with no choice but to deepen the breakage. > >> 5. The removal plan doesn't consider the impact from increased number = of > >> rebuilds due to revbumps containing only dependency changes. Without > >> replacement functionality, widespread virtual or slot changes can cause > >> hundreds of packages to be rebuilt. =20 > >=20 > > Please support this claim with specific examples or numbers. Otherwise, > > it's just a 'theoretical issue' that you can't support. =20 > I'm happy to provide more examples beyond the 77 > useless-but-apparently-acceptable rebuilds you mentioned below if > necessary. It's not hard to check yourself, though. Almost 9000 ebuilds > depend on virtual/pkgconfig. Even if we assume that any future virtual > introduction will only affect an absolute maximum of 10% of that number > of packages, that's still a ridiculous number of unnecessary rebuilds. =20 If you assume that the immediate rebuild is actually a necessity. Please note that many of the changes can actually be carried out within a reasonable timeframe as part of package upgrade. Given your example, the change was necessary to support a different implementation of pkg-config. We may argue that choosing a non-default implementation is pretty rare, and requesting those people to rebuild a number of packages shouldn't really be a problem. > > There is a lot of FUD about unnecessary rebuilds. Sadly, most people > > seem to fight a holy war against them without realizing the real > > impact. In fact, more unnecessary rebuilds are caused by unnecessary > > USE flags than by dependency changes. =20 > Do you have some numbers to back up that statement? I can think of very > few instances when I have had to rebuild against my will due to USE flag > changes. =20 This is hard to count since it differs on per-case basis. I'm talking about the two common cases here: when user doesn't know/notice that he has to enable/disable a particular flag, and when user installs a new package that requires USE changes on other packages. > > Yet the same people believe in > > adding more flags to contain even more minor aspects of packages... =20 > Are you referring to flags for things like logrotate or systemd units? > When they were around, I either had them enabled or I didn't - I was > never forced to rebuild because of them. =20 Rather flags for various package install aspects -- switching very commonly used aspects, changing library install details. Think of USE=3Dtinfo on ncurses or attempt to add a flag to control library install to pypy. But that's really irrelevant to the topic. --=20 Best regards, Micha=B3 G=F3rny --Sig_/iCpzcB_a06//LPB505RwAZx Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJT2+7rXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2REJCMDdDQzRGMERBRDA2RUEwQUZFNDFC MDdBMUFFQUVGQjQ0NjRFAAoJELB6GurvtEZOqioP/ja1TQg+KzmhySycnrGhx7qK VTQaUluDf0X3zD1Bbkri7ZJRHAa+GPU01h0m5oHJ25L5Ksheni5sFvsY1wMpyJks q4ULNGV8FOGxx+O7b9gK+tXY/e8RRxgRXNYAwHldyi5WAZ27bfaXpyZ49/L8+VyO Q3xt8L54dDK5+TFv+FzuJ0Wafxs9zt6gCeqGhtfebbFF2VuYY/TYnMNk1xZchsf9 7wNL9cxvGqaIZXThSLIYNz6+dqcIh7flLEuqPjI5ShbFMkX3P6UR+by5v7UlxZ5D J896QqN640j5MNcV8colXlwZeRnW0WviH/ChYmI+uLWnDTTGmRrnY786whSjiEBo NBRJdYHFwLNyhSefHk4DyfJm7yhVxHleSKXLmE1dXbzTmZ5v8q6D8S0N0QFsFzTX 6P0w5AY6RDZMlPVfskFDh515OV6OGqI2sy+Q1J95OFDx7gw5lZD1iwk+t51AzVUS T3Md1LF9vCEonNRPvKex/8SN51wlRmSp3cZ/SPZOvw9q/P0lRnzLpqYg6dfPU2a7 Y9cCyd1ahwgQItNIAmzmpSuGVC7IUc4TwU/Wsf7zufb+5a+AqtPA/fHBQiNa1U+i BMG6tf0g4nl1SE8nCMZQITBeYNPHNn4cnXyGJ9MSbP9SJC0xZHx+LejuBw+9EuHF j4Bio9CIVTixu/uwrYIR =UdkU -----END PGP SIGNATURE----- --Sig_/iCpzcB_a06//LPB505RwAZx--