From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 20BB5138334 for ; Thu, 10 Jan 2019 05:22:32 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 144C1E0A03; Thu, 10 Jan 2019 05:22:31 +0000 (UTC) Received: from smtp.gentoo.org (dev.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B9529E0A03 for ; Thu, 10 Jan 2019 05:22:30 +0000 (UTC) Received: from [10.128.13.179] (unknown [100.42.98.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: zmedico) by smtp.gentoo.org (Postfix) with ESMTPSA id A0CBB335C5A; Thu, 10 Jan 2019 05:22:28 +0000 (UTC) Subject: Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval To: gentoo-portage-dev@lists.gentoo.org, Fabian Groffen References: <20190108134207.15339-1-grobian@gentoo.org> <20190109083323.3590-1-grobian@gentoo.org> From: Zac Medico Openpgp: preference=signencrypt Autocrypt: addr=zmedico@gentoo.org; prefer-encrypt=mutual; keydata= mQINBFs7tmwBEADTzG+IcYtRfTfKryU7sUH7LlV1M+TdaCMfIkY4x6RyHXkaaqYuQ+U9HKn0 +m5FcZsZ1Ojik+We3Tz0F6kDbam6EWzBxmsLb/IHeUEsvsuLzuBQjiD9zzqGocZiPWr+uWJs AdbueS72R7FPXJPDUEPrJ9GdhGFyYARveY9cmdisOwcDOiSFfBjk3/89t4gROn4KUhezVuO9 VS14gVSns1561CJjlB47HkSBu4+FuzrfVygg4xitWAH119Ehw0vJcgkTw4Bqhk01Iw9us80m dFyU8JbJ0CVYe30gYKFFbnXoiT6xLLogKOkv0goPFxaXcMwWM9ei3SjAGVqgN6i8VnO7kquV LwkTe6ntEK0iY+l4qTKuyIOQLpCbWNI0eVwlx5b/pY2pt5TEGWAPMCZGjlidMx0aDcVX4oji 2/xegFAcxALrfOX3kj2FZ9kNAqLZu26AfqtslIqlBEAb5sZwPr351msBIdbaWX2UNw21I478 7eQ7UfohwXQHlXdhc/wop3VDkDzLBnvlK4ozSJI/9T5F/+9yEZvc6DKUWdEfD12o2El5hHan gCUQWDBKqZb1wcekK8KY2tmH8BBQi7k52IWYLJYfJdir/XpGm5SsDpf3zvDcIFXqFHAG7w7b fhriM+6oBOeIO9ew1Xj3swbRhDwdzRUhu7Uqayq1vdvKqGkgcQARAQABtCNaYWNoYXJ5IE1l ZGljbyA8em1lZGljb0BnZW50b28ub3JnPokCVAQTAQgAPgIbAwULCQgHAwUVCgkICwUWAwIB AAIeAQIXgBYhBEdYrNjamv1GpqYECtYQzzch54rNBQJbZpeGBQkEzWcaAAoJENYQzzch54rN iTEQALgfn8NqcY1P3VgE8n/ypYfCa8YhOhZcSlVOnIil9u4F0OPUP9/TFALaW2RRctUm7URP Oe6gMd8mzFauU1q8FKHW8vo6i48Oqb3RmkH4PNbH0EHfM1e3pAq+E6Bi08y7HzYUm5PKr+m3 oLNFvqZuG3RKgmKQTm8E56IpI8rODnVmBkLQEGkdi9hDB1Zkm9dlT+eRGglHAhnpb+AweOzh dvqdfu0SKEoyLjCvRIFltrtNIuKWpjqOVAaMU0hLfDlRVJR0oTThe7P0vhulKZtWS/L7wXJr e7NGIunM09JIAjYAXX5hmY9L0oKnp0WK5PksBKIu8W8TUwzVmTXNAs7qfEQQJ3LvAQo3xifa l2n0ixdoozltU8afxJohz8OKrWOAzE8kqWa+H+t1XZtQAy7yZA3x51vyp4hquyAMYrZbU8hC 0x/l6KM/qGI7RPohw5VxgneQCAbpu/G+3DLVH5QjPR+tKPtpFkEpxrbTNUhaOUqFipX92382 w2CCxDrNrBYbnFYKmVthZqVxrUuDfi4VIgeBkGzDKPwqjSkbdZ0/I5GAmjZh0aJWrRnzCum/ hZHHHcU+wOdXBwwGBhRf9SIaxCfaxw0PB+CZXsswvuaANah2b32EFTPcFCCgFvoGMNSSHFuo JR3PAo6qvwGhYdmTI/Na57P1FF+g3VojnAnK2eIEuQINBFs7tmwBEACfZb/EePObKC8tuVFl IMQaTRzm7q5THbwQvbdKdw/31cdYJaZZ7BFgnSBq2CBYDPxcm+TxvvjgNTam8kZGdEpCm3/v P1YdJSoiYbisJubV7JiAOoAmGtaDOVX5thpv28HL17wqK2d5jgqMTLFeT3hyaro5cjGAQlhT NOyfwVkdFMFLjOhGzq7aki4UZ9ieS/IedoyZvOblf79d5PV1xI0mf2w5yoHcfZRv7nwI9XMz nqlgC9/RQP7O+WfCl2pl9gyd0hc3uqUWl0ke9xHvVxlEdORH/f+OzeybYj9r80GkC6MeqiG8 qzuWO0IZZ5Gvzwkq0KgRAefr9Z/4vmyOiVhJrvk1gLp6VEdB54fUO3MG/PcnYiasRJOUwRQQ 1+q25w5i9ooZRk0LQ6oNnWbeVj+Pwn0mzL0/GjkdPsYclxzOLSlRAwkisepVl62L2m+XwXsZ j9DS3vIHEbbfCmKayBuGCXQEpJMmEnGqFLnOzv29cOutm3BnGXcWAg5aJjiV5PKEmr/4g9/E J8lphGnNCRQ2DvUzO1tlHpmT+JvDBwAgo5rfq5wjjeiS578kAgPoitOBzp/z5YM5FXX+shEd i5/ratErb9ysUfr3Z45YV3yM6MHgCpEg8+5k6fM6ey4tRnxXAYdJ4XzaSyML9fsnEg9aVaCP iRl+fuLSuF1QJf7LwwARAQABiQI8BBgBCAAmAhsMFiEER1is2Nqa/UampgQK1hDPNyHnis0F Als7trcFCQHhM8sACgkQ1hDPNyHnis0Wdw/9ERSLUHWjGW+oRJPC80RsSZO+9gbdo6AT8GP0 L3S1M60d/C4Pp9lT7sGvN31Q2h4/i1AGz2vsvjA4RUwuWI/M2F660ZmqbuLA4BbS/gjA+Wrf ZB8TDEoy7k0IO+ga0J6ENgxSOk0w1ZS+LYVrM8r7C73ZckAU2aknGOHvfIdRocpb464/JA+j kP8ENcXA0NVT17Te6K3OxpN3fBsHP7w9zbYaV3Ex3suXhLx0/+xxjLPTfwobEy3kTv5Qqumq Ph7TlXvc/9Ku4GwgAFUscLG5ZhsDULIWsRr0tyum0IYI3lpQnMykZiDCoiq7HETTwJeZPpJk PlVc7Qyy0eqcKyoiqd/ttr5tPwRFWk4OYmqzZFiZZ6sG7Zw68UXKHuhCV1MgXKPqS3M8Q6V+ nGL+ZwLE1GkvqwaPgn7uzJpfO+HzoWn3yxjNvh5plsZbM/kqCTMyByB0LQPZdjkO30GvV6dU wamblYkpytpZ5cmbIRITzsxPkOuWwXkLoz6y664JfH/Te3xVQ2rqZkuDCFrwy5nOqk92w7OY WGqyDOSL4tyCdcsXVoF5HXcHQLTv5WzCQ2MAQUW6zflyirYygTfJ3ds5mk5/J52UKEhIHPzJ Hc26Ubfvo38XuE0FJjb7xoZzCoSTpCVZ2H0f2cJOBLEVN9bnbelNlmp0m6mPAHfDrfsVR/u5 Ag0EWzvRagEQANK1C/HvZgnFVa+3tFmS4OVnCRO611C7WXubm5Y2xj/Lh2LOWv5TeTtTp5FR S3961b429TbJNv0q9N4mDi6XOGpZvWLkfiDw/VT9I+48B2eVXKx5N2H48S7t1Knwut6vuTEx 14MGiZToCs2Tu1fEUnaBv2Hg60ysVfplDAQadixzboLHM1DxLYn1W/cAUrhXAW5uNQlyE0Ze kiB10JxbnAurdpRSu4X81IrGJDK/oNAgAcWRieVX5J1N5LStPrloFKf+Dtl2z49WibXVTjwF Ir/BZYprkTTgNzeM3VRPnyRz7IA1pMkVX0r4C0O38mqHyDCpkM/TKsOpNMDqYcllD8Fa/Zyg S08RLymWvRXWn7Sz7MlZE9CCQ2aG+N20esYH3nwrYsdUdj87/nSwqYKhTWwBBIWtrrCHw56O ZMTXznX3OkLOBB0gXH92G5dKv4azBeAZzR6c+qvP6PxgVuAlvlnFbzgW5m4CrkTsEwSh/s2Y d9sJPctzkPkif6tDWk0qKy9lRwTKyOK0xqwZGcJBLdqInU05DLJlrf2QJKBS2SE1tEparQvT 8/+EBpzGk0omSxGmDxLW6EY1CTXV+LfbRYz42Y71f7aVNNqfF/CbJYLupe9DadjvceRM/ZFG WO6SzWoO7ed5uT2i8M33tC2EnK/BL1oZ5Wr7Kv5XED10JI+5ABEBAAGJBNIEGAEIACYWIQRH WKzY2pr9RqamBArWEM83IeeKzQUCWzvRagIbAgUJAeEzgAKgCRDWEM83IeeKzcHUIAQZAQgA fRYhBPDoF2pbVs6oErqQrtB1+4wQSj0gBQJbO9FqXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90 YXRpb25zLm9wZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGMEU4MTc2QTVCNTZDRUE4MTJCQTkw QUVEMDc1RkI4QzEwNEEzRDIwAAoJENB1+4wQSj0gJEMP/iP89Et7jH3Ai9w1bQ19T0ChOD4O 36IjisSTGTYsoeIAUgka/XFmuQSAgeTj5Tc73/Ny0V5jscjmtVKhUbFkLGetHuA0gHJpxEen j0zedAIQ36ToU0gia7Qsk7cpJJJRge3fdaUgb4aUcU5fTg2lkPdsMqW2B1/djlKz6T2cqX8y omvByiBdRSWVJlRmgqYQTJyQa6zLERWcknCG9T2oD9y9VJlVW3kcLT2QmB6a/3q/M0gCUBl8 qGHZYDGv+0su4SeppQ5BAOu9PXNV8pGdqTMsirQ+E20K5QprzmPg5kVG4x8qG1UpRpC/hJ9E 75wax9o4WrqHH4NmhvnL/Pfs2jK7PJLi/xRqvO65wGGO+4JW0kailhjvLoRYJi0B6ir5JhtK 44kctcqrn7QygO9ZB5r2/i1LyZrA+LWKLcpd7xxrraunFidoc63Mz5QadvWGcOduM6DOKS4b sLgr52V+413UkdtvBru/4hA9s1C8Dpc6DT0p0wFSJ7EBKE4dlu9VCnJ9+7X6HVMoq7aMj7ZW qJ1S1Jp/m6uJwzBDexcTokj8rTPBzbIo+3EfaGWMk47VoQgccee5B1jvxmLxhIbEd+LUc9ek jDQ0FETN7dOBuIIg5+Z6IdAycsLMHfNeSTP0jMHHZBKbptQwiaQ/hieMbSraN6fjozta4jw8 mUBoOJTgyjAP/2nbBGiOp/s8CkNEfbqFlLLVSnpN70PHPQ+FOKHe+FFg/bhyf6L0mGShJ8rH c8o7Re1Im/ztjj9hrGUfdicPFrU88Y+XgRGqb8DxjlyYSOVwIi6abyH5TgP1LNasBc23+cY/ nf2+n7Eq6/AOdgQRCKe5bIHXl94889HHVMQyfR+AvRZ9AiHNd7AGbhi2aMKsHgrnhuTcFO/d TR8NHolevl/zyOHmlcKaDUahjgTEK/SguUjmFHYDQYFDCrQj6U0o8uZtLA23wpiGgG1ajqpG w0VpKjkFzXcCNVeVme7XreRBxwstOda4Y00JUfOTGpR0+/bGRnJaQvBFV87WbBksVCa2McuY uzEqnkYwE4M/Hua9CRXReyvG31JRDbMubqXZjgisOtjIaNn7jvbweLoJhSzINkBB28tJiV6P ODhMx9XC5taaVLiPpABL/xzbqtHi6LCwXAYVlOS+ceiR2WIUigL866Ze607UePBJVQvsCuIS y7PV9kVbfN1/xKls9fiB87tv5XpyEUR9s+91Az6d8kterzLKqwxWSfp+HSepYuWOhlLsywUr BAJMnLR1yogNFuwyFkXIwKy/kNj4lIy+wLGNCQod1H84OLg1iHElIA+wxAWL7xnTYPxKX102 YsIUXy5A5ebVtwxpLuu8Yd6CnrTOvyLKqukoXSwP+16Mjo7C Message-ID: Date: Wed, 9 Jan 2019 21:22:26 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 In-Reply-To: <20190109083323.3590-1-grobian@gentoo.org> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="srtuUYEQFY1eHUwVe9gjX0x9Tdeo9p42U" X-Archives-Salt: 807237d2-7477-4f36-972d-1d7e9e06a686 X-Archives-Hash: 88987a7075d144eac50cc022ac01ca9a This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --srtuUYEQFY1eHUwVe9gjX0x9Tdeo9p42U Content-Type: multipart/mixed; boundary="Pna4XIRjQm9G3NkPtr6iufj49osHypmBW"; protected-headers="v1" From: Zac Medico To: gentoo-portage-dev@lists.gentoo.org, Fabian Groffen Message-ID: Subject: Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval References: <20190108134207.15339-1-grobian@gentoo.org> <20190109083323.3590-1-grobian@gentoo.org> In-Reply-To: <20190109083323.3590-1-grobian@gentoo.org> --Pna4XIRjQm9G3NkPtr6iufj49osHypmBW Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 1/9/19 12:33 AM, Fabian Groffen wrote: > The reporting of files remaining can look somewhat odd since the report= > interval is hardcoded to be per 1000 objects. Adjust this interval to > be time based. This means that modern (fast) machines likely will neve= r > see the countdown messages at all. On slow setups the message will be > informative that there is progress, albeit rather slowly. While at it,= > report percentage done. >=20 > Output before this patch: >=20 > * checking 6158 files for package collisions > 5158 files remaining ... > 4158 files remaining ... > 3158 files remaining ... > 2158 files remaining ... > 1158 files remaining ... > 158 files remaining ... >=20 > Possible output after this patch on a slower machine: >=20 > * checking 6158 files for package collisions > 48% done, 3145 files remaining ... > 96% done, 192 files remaining ... > 100% done >=20 > Signed-off-by: Fabian Groffen > --- > lib/portage/dbapi/vartree.py | 13 +++++++++++-- > 1 file changed, 11 insertions(+), 2 deletions(-) >=20 > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.p= y > index 4b91caea8..df192e6fc 100644 > --- a/lib/portage/dbapi/vartree.py > +++ b/lib/portage/dbapi/vartree.py > @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(), > 'portage.util.install_mask:install_mask_dir,InstallMask', > 'portage.util.listdir:dircache,listdir', > 'portage.util.movefile:movefile', > + 'portage.util.monotonic:monotonic', > 'portage.util.path:first_existing,iter_parents', > 'portage.util.writeable_check:get_ro_checker', > 'portage.util._xattr:xattr', > @@ -3475,13 +3476,19 @@ class dblink(object): > symlink_collisions =3D [] > destroot =3D self.settings['ROOT'] > totfiles =3D len(file_list) + len(symlink_list) > + tnow =3D monotonic() > + tinterv =3D 2 # seconds > + ninterv =3D tnow + tinterv > + falign =3D len("%d" % totfiles) > showMessage(_(" %s checking %d files for package collisions\n") % \= > (colorize("GOOD", "*"), totfiles)) > for i, (f, f_type) in enumerate(chain( > ((f, "reg") for f in file_list), > ((f, "sym") for f in symlink_list))): > - if i % 1000 =3D=3D 0 and i !=3D 0: > - showMessage(_("%d files remaining ...\n") % (totfiles - i)) > + if monotonic() > ninterv: > + showMessage(_("%3d%% done, %*d files remaining ...\n") % > + (falign, i * 100 / totfiles, totfiles - i)) > + ninterv =3D monotonic() + tinterv > =20 > dest_path =3D normalize_path( > os.path.join(destroot, f.lstrip(os.path.sep))) > @@ -3570,6 +3577,8 @@ class dblink(object): > break > if stopmerge: > collisions.append(f) > + if tnow + tinterv < ninterv: > + showMessage(_("100% done\n")) It took me a moment to understand the calculation here. How about if we do something like this instead: previous =3D monotonic() progress_shown =3D False for f in files: current =3D monotonic() if current - previous > tinterv: showMessage(...) previous =3D current progress_shown =3D True if progress_shown: showMessage(_("100% done\n")) > return collisions, dirs_ro, symlink_collisions, plib_collisions > =20 > def _lstat_inode_map(self, path_iter): >=20 --=20 Thanks, Zac --Pna4XIRjQm9G3NkPtr6iufj49osHypmBW-- --srtuUYEQFY1eHUwVe9gjX0x9Tdeo9p42U Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQKTBAEBCgB9FiEE8OgXaltWzqgSupCu0HX7jBBKPSAFAlw21pJfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEYw RTgxNzZBNUI1NkNFQTgxMkJBOTBBRUQwNzVGQjhDMTA0QTNEMjAACgkQ0HX7jBBK PSDp2g/8DlrvMoBomkD0VOaLoWp9y65pnzrrPL7faZtMxAv1C6JtcLlVt37ftyqK kG1XicKbpe4vpzpPIwhNdmAg2xK8R9ORezJ/CLW4CSHbHap1KW4ZddweEMZBW+gN 7wc9F5dfkeRP06Mi8ka5FGDTxuXJDCRET9iKGJ9Qr8OlW6ukknBvQFE0f2Ussqu9 l5geEP7ycIOCEKuMCPz0h9lbqm8Vw5zxPB4Msb3XccFo862TVHYwpQg9CiJQ1r+V IwZWNUiIlmwvpQ8DtEAnQBYMSlfQb6cV/AgiPNqLAe5ycpTlUGQNsWjBgrmqd2HN IDcsiHUTxHqUaiVwSwV8wyAcGgv6/14WrzNUEsbbpkdAAVsgwyNYL2u+hZQdyPun qtwCE1ovEqWdChEal1CYEfC3Ub8fgyWsKXfsa8SanJxF7HnR/lJ9BnoVHitg9hKT OVsmpl9JVvK2/feq2qI5TXg+zB+T4rbsA2haEh70ywVIK/4J4x8gKSf6R1AkzXrA NePKOABv6RmarYagNFRro3aErNFMifeTjE0Bo++z4oy/TQazBzqBQXZYCervZxOm AropihCVtYsle+2SlNUsm101Gvh4e+8sFVNxgq/zXHTuY0Y/dmuY3P+1oyoWFxHW CphBVxqWlWfJRHO9UXf74S4DH4JFtJCICu56KLZm4sagTrHTGI4= =ADbO -----END PGP SIGNATURE----- --srtuUYEQFY1eHUwVe9gjX0x9Tdeo9p42U--