Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

[Zac Medico <zmedico@gentoo.org>]

[-- Attachment #1.1: Type: text/plain, Size: 6350 bytes --]

On 04/01/2018 11:29 AM, Michał Górny wrote:
> W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico
> napisał:
>> On 04/01/2018 03:57 AM, Michał Górny wrote:
>>> W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
>>> napisał:
>>>> Since key refresh is prone to failure, retry using exponential
>>>> backoff with random jitter. This adds the following sync-openpgp-*
>>>> configuration settings:
>>>>
>>>> sync-openpgp-key-refresh-retry-count = 40
>>>>
>>>>   Maximum number of times to retry key refresh if it fails.  Between
>>>>   each  key  refresh attempt, there is an exponential delay with a
>>>>   constant multiplier and a uniform random multiplier between 0 and 1.
>>>>
>>>> sync-openpgp-key-refresh-retry-delay-exp-base = 2
>>>>
>>>>   The base of the exponential expression. The exponent is the number
>>>>   of previous refresh attempts.
>>>>
>>>> sync-openpgp-key-refresh-retry-delay-max = 60
>>>>
>>>>   Maximum  delay between each retry attempt, in units of seconds. This
>>>>   places a limit on the length of the exponential delay.
>>>>
>>>> sync-openpgp-key-refresh-retry-delay-mult = 4
>>>>
>>>>   Multiplier for the exponential delay.
>>>>
>>>> sync-openpgp-key-refresh-retry-overall-timeout = 1200
>>>>
>>>>   Combined time limit for all refresh attempts, in units of seconds.
>>>>
>>>> Bug: https://bugs.gentoo.org/649276
>>>>
>>>> Zac Medico (4):
>>>>   Add ForkExecutor (bug 649588)
>>>>   Add ExponentialBackoff and RandomExponentialBackoff
>>>>   Add retry decorator (API inspired by tenacity)
>>>>   rsync: add key refresh retry (bug 649276)
>>>>
>>>>  cnf/repos.conf                                |   5 +
>>>>  man/portage.5                                 |  19 +++
>>>>  pym/portage/repository/config.py              |  22 ++++
>>>>  pym/portage/sync/modules/rsync/rsync.py       |  16 ++-
>>>>  pym/portage/sync/syncbase.py                  |  85 +++++++++++-
>>>>  pym/portage/tests/util/futures/test_retry.py  | 147 +++++++++++++++++++++
>>>>  pym/portage/util/_eventloop/EventLoop.py      |  45 ++++++-
>>>>  pym/portage/util/backoff.py                   |  48 +++++++
>>>>  pym/portage/util/futures/executor/__init__.py |   0
>>>>  pym/portage/util/futures/executor/fork.py     | 130 +++++++++++++++++++
>>>>  pym/portage/util/futures/futures.py           |   6 +
>>>>  pym/portage/util/futures/retry.py             | 178 ++++++++++++++++++++++++++
>>>>  12 files changed, 697 insertions(+), 4 deletions(-)
>>>>  create mode 100644 pym/portage/tests/util/futures/test_retry.py
>>>>  create mode 100644 pym/portage/util/backoff.py
>>>>  create mode 100644 pym/portage/util/futures/executor/__init__.py
>>>>  create mode 100644 pym/portage/util/futures/executor/fork.py
>>>>  create mode 100644 pym/portage/util/futures/retry.py
>>>>
>>>
>>> This essentially looks like ~700 lines of code to try to workaround
>>> broken networking. I would rather try to do that using 5 lines of code
>>> but that's just me, and my programs aren't enterprise quality. I just
>>> hope it actually solves as many problems as it's going to introduce.
>>
>> The vast majority of this code is generic and reusable, and I do intend
>> to reuse it. For example, the executor support will be an essential
>> piece for the asyncio.AbstractEventLoop for bug 649588.
> 
> Sure it is and sure you will. But tell me: who is going to maintain it
> all? Because as far as I can see, we're still dealing with a bus factor
> of one and all you're doing is making it worse. More code, more
> complexity, more creeping featurism and hacks.

In the worst case, people can disable the retry feature and all of the
associated code can be disabled simply by setting
sync-openpgp-key-refresh-retry-count = 0 in repos.conf.

In order to reduce the bus factor, I'm modeling the code on the standard
library's asyncio framework. This allows use to leverage people's
experience with asyncio, and also introduces people to asyncio concepts
if they are not yet familar.

> Last time you went away, you left us with a horribly unmaintainable
> package manager full of complexity, hacks and creeping featurism
> and a Portage team whose members had barely any knowledge of the code.
> Just when things started moving again, you came back and we're back to
> square one.

I'd love to help reduce the bus factor, and I'm working to do that, like
with the asyncio stuff.

> Today Portage once again is a one-developer project, full of more
> complexity, more hacks and more creeping featurism. And we once again
> have a bus factor of one -- one developer who apparently knows
> everything, does everything and tries to be nice to everyone, except he
> really ignores others, makes a lot of empty promises and consistently
> makes the health of the project go from bad to worse.

I disagree with your synopsis.

> So, please tell me: what happens when you leave again? How have you used
> your time in the project? What have you done to make sure that
> the project stays alive without you? Because as far as I can see, adding
> few thousand of lines of practically unreviewed code every month does
> not help with that.

Again, I'd love to help reduce the bus factor.

> I forked Portage because I didn't want to fight with you. When I forked
> it, I declared that I will merge mainline changes regularly for
> the benefit of my users. But after a week, I really start feeling like
> that's going to be a really bad idea. Like it's time to forget about
> mainline Portage as a completely dead end, and go our separate ways.

That's your decision to make, but I think you'll benefit from merging my
mainline changes.

> I'm seriously worried about the future of Gentoo. I'd really appreciate
> if you started focusing on that as well.

I feel like I'm doing my part, and I appreciate your role as well.
Thanks very much for your work.

> I get that all this stuff looks
> cool on paper but few months or years from now, someone will curse
> 'whoever wrote that code' while trying to fix some nasty bug. Or get
> things moving forward. Or implement EAPI 8.

Again, I'd love to help reduce the bus factor, and the asyncio stuff
represents movement toward that goal.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]