Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

[Zac Medico <zmedico@gentoo.org>]

[-- Attachment #1.1: Type: text/plain, Size: 3140 bytes --]

On 04/01/2018 03:57 AM, Michał Górny wrote:
> W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
> napisał:
>> Since key refresh is prone to failure, retry using exponential
>> backoff with random jitter. This adds the following sync-openpgp-*
>> configuration settings:
>>
>> sync-openpgp-key-refresh-retry-count = 40
>>
>>   Maximum number of times to retry key refresh if it fails.  Between
>>   each  key  refresh attempt, there is an exponential delay with a
>>   constant multiplier and a uniform random multiplier between 0 and 1.
>>
>> sync-openpgp-key-refresh-retry-delay-exp-base = 2
>>
>>   The base of the exponential expression. The exponent is the number
>>   of previous refresh attempts.
>>
>> sync-openpgp-key-refresh-retry-delay-max = 60
>>
>>   Maximum  delay between each retry attempt, in units of seconds. This
>>   places a limit on the length of the exponential delay.
>>
>> sync-openpgp-key-refresh-retry-delay-mult = 4
>>
>>   Multiplier for the exponential delay.
>>
>> sync-openpgp-key-refresh-retry-overall-timeout = 1200
>>
>>   Combined time limit for all refresh attempts, in units of seconds.
>>
>> Bug: https://bugs.gentoo.org/649276
>>
>> Zac Medico (4):
>>   Add ForkExecutor (bug 649588)
>>   Add ExponentialBackoff and RandomExponentialBackoff
>>   Add retry decorator (API inspired by tenacity)
>>   rsync: add key refresh retry (bug 649276)
>>
>>  cnf/repos.conf                                |   5 +
>>  man/portage.5                                 |  19 +++
>>  pym/portage/repository/config.py              |  22 ++++
>>  pym/portage/sync/modules/rsync/rsync.py       |  16 ++-
>>  pym/portage/sync/syncbase.py                  |  85 +++++++++++-
>>  pym/portage/tests/util/futures/test_retry.py  | 147 +++++++++++++++++++++
>>  pym/portage/util/_eventloop/EventLoop.py      |  45 ++++++-
>>  pym/portage/util/backoff.py                   |  48 +++++++
>>  pym/portage/util/futures/executor/__init__.py |   0
>>  pym/portage/util/futures/executor/fork.py     | 130 +++++++++++++++++++
>>  pym/portage/util/futures/futures.py           |   6 +
>>  pym/portage/util/futures/retry.py             | 178 ++++++++++++++++++++++++++
>>  12 files changed, 697 insertions(+), 4 deletions(-)
>>  create mode 100644 pym/portage/tests/util/futures/test_retry.py
>>  create mode 100644 pym/portage/util/backoff.py
>>  create mode 100644 pym/portage/util/futures/executor/__init__.py
>>  create mode 100644 pym/portage/util/futures/executor/fork.py
>>  create mode 100644 pym/portage/util/futures/retry.py
>>
> 
> This essentially looks like ~700 lines of code to try to workaround
> broken networking. I would rather try to do that using 5 lines of code
> but that's just me, and my programs aren't enterprise quality. I just
> hope it actually solves as many problems as it's going to introduce.

The vast majority of this code is generic and reusable, and I do intend
to reuse it. For example, the executor support will be an essential
piece for the asyncio.AbstractEventLoop for bug 649588.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]