From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 609951381F1 for ; Sun, 1 Apr 2018 15:59:41 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 5B387E0C32; Sun, 1 Apr 2018 15:59:40 +0000 (UTC) Received: from smtp.gentoo.org (dev.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 13BDBE0C31 for ; Sun, 1 Apr 2018 15:59:40 +0000 (UTC) Received: from [IPv6:2600:8802:605:7900:2e33:7aff:fef2:3005] (unknown [IPv6:2600:8802:605:7900:2e33:7aff:fef2:3005]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: zmedico) by smtp.gentoo.org (Postfix) with ESMTPSA id 924F5335D9C; Sun, 1 Apr 2018 15:59:38 +0000 (UTC) Subject: Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276) To: =?UTF-8?B?TWljaGHFgiBHw7Nybnk=?= , gentoo-portage-dev@lists.gentoo.org Cc: Zac Medico References: <20180401024631.31017-1-zmedico@gentoo.org> <1522580236.7711.6.camel@gentoo.org> From: Zac Medico Openpgp: preference=signencrypt Autocrypt: addr=zmedico@gentoo.org; keydata= xsDiBEOvrjkRBACHorAtOrGF7pqWEnm/DRr2CweqdBgrCsNRhYfzDNdwcQkz1RzDusAfnTF4 6d4+hZO2ijw6z1gEpU+eCePGWDaH2nYI3iDcBdFl+y1jo7hHOU+qv0lsA1A9kkxivGNIgjWZ Tjzg6GpUeCR1DmKUSureiPxLJDOE0a7rdrO539SepwCg9MSzVcYsw4yqeghc8ijU8KAtaPED /2tPOWvAVYFwBQO2NQAJlZMr9iAYQBDQStWWqkZyJB30ldLDwPFtfDTPd3S62eXC01UZ9054 ckZZ89DPFQCvu7dlqPxatc2SZg24vTn82IG7OA3d0L7FhZfO8+YiPE5G+7xpL/7D0GdyLMMo Ma9ZM+Jx15MeF9BagdGJKwM6Exv8A/93jNDjsK7Hg39VzuPlOq1a6nKjSV1OddqPzYO6RYa4 8K/XlXilGbs11bNyDpmUIdP90vfgU8ZKOlnJKJolP1dKSW8Ue2AD6uwxHixiEdOCIyZWckkO 3DwHPpTAfcGs65NG6Stn+EFJhfQdsD1LK9sPUeOL5gNgmXBVkegss/AYVM0jWmFjaGFyeSBN ZWRpY28gPHptZWRpY29AZ2VudG9vLm9yZz7CYAQTEQIAIAUCQ90+FgIbAwYLCQgHAwIEFQII AwQWAgMBAh4BAheAAAoJEP3o74WuVxmjGqoAniyHxOGuzezgV0gPcqkaFbgYo6yMAJ9K6rHq gShypIzVvF+AIQChDUBljM7ATQRDr646EAQA77th04RKDfiwNYGTb+FixHqMtZ5HIBL7WorF Q0K2AzWpQsYlpEyuGczDmTKR3QArmb+ibI4OptmAFHG2mE0P8mwlYnCmEEyP7m+pF8TraRr1 wET8sfN1p4OB1CTEqhDQR67fhz/5jkNoD0Quzl6MRiNKhzs4OY9qyjonh6NaPaMAAwUD/jBa +15ckMz9YIxXZFWK9cq78DHjyd6Lu3/smBKw1MSbdc3ApB/vhtdrinbp8oCgixEwHOJuhezE gTH8hLy1EUK1l81bfRmnF8fq+OcUgedNMcquy6e0GykkpInlYvTeqOxeTmZnQLFOjlbn/x9/ itHZIVV0zfTtbir6SPgKnpK5wkkEGBECAAkFAkOvrjoCGwwACgkQ/ejvha5XGaOWUgCgvMGs WMPRCB5tdq1DhQKEnahDZwkAoLrc8ffjyoGMpUpqqTCdMxfobblg Message-ID: Date: Sun, 1 Apr 2018 08:59:35 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 In-Reply-To: <1522580236.7711.6.camel@gentoo.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZTdHZ97JL8tbtPE1rhb5Rx8cUqDSbOXMW" X-Archives-Salt: a001a2eb-0462-4f97-a1ac-0b8bb8b0e45a X-Archives-Hash: a8a6fade86a258979519bc9839df7b28 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ZTdHZ97JL8tbtPE1rhb5Rx8cUqDSbOXMW Content-Type: multipart/mixed; boundary="i9OIP0Q8xwwhKCC2jjwyIzQCINKpvpUn8"; protected-headers="v1" From: Zac Medico To: =?UTF-8?B?TWljaGHFgiBHw7Nybnk=?= , gentoo-portage-dev@lists.gentoo.org Cc: Zac Medico Message-ID: Subject: Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276) References: <20180401024631.31017-1-zmedico@gentoo.org> <1522580236.7711.6.camel@gentoo.org> In-Reply-To: <1522580236.7711.6.camel@gentoo.org> --i9OIP0Q8xwwhKCC2jjwyIzQCINKpvpUn8 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 04/01/2018 03:57 AM, Micha=C5=82 G=C3=B3rny wrote: > W dniu sob, 31.03.2018 o godzinie 19=E2=88=B646=E2=80=89-0700, u=C5=BCy= tkownik Zac Medico > napisa=C5=82: >> Since key refresh is prone to failure, retry using exponential >> backoff with random jitter. This adds the following sync-openpgp-* >> configuration settings: >> >> sync-openpgp-key-refresh-retry-count =3D 40 >> >> Maximum number of times to retry key refresh if it fails. Between >> each key refresh attempt, there is an exponential delay with a >> constant multiplier and a uniform random multiplier between 0 and 1.= >> >> sync-openpgp-key-refresh-retry-delay-exp-base =3D 2 >> >> The base of the exponential expression. The exponent is the number >> of previous refresh attempts. >> >> sync-openpgp-key-refresh-retry-delay-max =3D 60 >> >> Maximum delay between each retry attempt, in units of seconds. This= >> places a limit on the length of the exponential delay. >> >> sync-openpgp-key-refresh-retry-delay-mult =3D 4 >> >> Multiplier for the exponential delay. >> >> sync-openpgp-key-refresh-retry-overall-timeout =3D 1200 >> >> Combined time limit for all refresh attempts, in units of seconds. >> >> Bug: https://bugs.gentoo.org/649276 >> >> Zac Medico (4): >> Add ForkExecutor (bug 649588) >> Add ExponentialBackoff and RandomExponentialBackoff >> Add retry decorator (API inspired by tenacity) >> rsync: add key refresh retry (bug 649276) >> >> cnf/repos.conf | 5 + >> man/portage.5 | 19 +++ >> pym/portage/repository/config.py | 22 ++++ >> pym/portage/sync/modules/rsync/rsync.py | 16 ++- >> pym/portage/sync/syncbase.py | 85 +++++++++++- >> pym/portage/tests/util/futures/test_retry.py | 147 +++++++++++++++++= ++++ >> pym/portage/util/_eventloop/EventLoop.py | 45 ++++++- >> pym/portage/util/backoff.py | 48 +++++++ >> pym/portage/util/futures/executor/__init__.py | 0 >> pym/portage/util/futures/executor/fork.py | 130 +++++++++++++++++= ++ >> pym/portage/util/futures/futures.py | 6 + >> pym/portage/util/futures/retry.py | 178 +++++++++++++++++= +++++++++ >> 12 files changed, 697 insertions(+), 4 deletions(-) >> create mode 100644 pym/portage/tests/util/futures/test_retry.py >> create mode 100644 pym/portage/util/backoff.py >> create mode 100644 pym/portage/util/futures/executor/__init__.py >> create mode 100644 pym/portage/util/futures/executor/fork.py >> create mode 100644 pym/portage/util/futures/retry.py >> >=20 > This essentially looks like ~700 lines of code to try to workaround > broken networking. I would rather try to do that using 5 lines of code > but that's just me, and my programs aren't enterprise quality. I just > hope it actually solves as many problems as it's going to introduce. The vast majority of this code is generic and reusable, and I do intend to reuse it. For example, the executor support will be an essential piece for the asyncio.AbstractEventLoop for bug 649588. --=20 Thanks, Zac --i9OIP0Q8xwwhKCC2jjwyIzQCINKpvpUn8-- --ZTdHZ97JL8tbtPE1rhb5Rx8cUqDSbOXMW Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQSG5RNTeMgVEruefzL96O+FrlcZowUCWsEB5wAKCRD96O+FrlcZ o+y6AJwJDtXqgaZxdnckyhSGJg9U23eucQCgqI9Woj7q2fo1FsXMa22rZWpZkS0= =q/J6 -----END PGP SIGNATURE----- --ZTdHZ97JL8tbtPE1rhb5Rx8cUqDSbOXMW--