From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id CEEBE138334 for ; Sun, 16 Dec 2018 07:38:10 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 53B30E0BA4; Sun, 16 Dec 2018 07:38:09 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 05441E0BA1 for ; Sun, 16 Dec 2018 07:38:08 +0000 (UTC) Received: from [IPv6:2600:8802:604:6600:b06e:5315:d0a7:5889] (unknown [IPv6:2600:8802:604:6600:b06e:5315:d0a7:5889]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: zmedico) by smtp.gentoo.org (Postfix) with ESMTPSA id 42724335C38; Sun, 16 Dec 2018 07:38:07 +0000 (UTC) Subject: Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method To: Alec Warner , gentoo-project , Zac Medico References: <1600658.mo6uCtQdLC@reaper> From: Zac Medico Openpgp: preference=signencrypt Autocrypt: addr=zmedico@gentoo.org; prefer-encrypt=mutual; keydata= mQINBFs7tmwBEADTzG+IcYtRfTfKryU7sUH7LlV1M+TdaCMfIkY4x6RyHXkaaqYuQ+U9HKn0 +m5FcZsZ1Ojik+We3Tz0F6kDbam6EWzBxmsLb/IHeUEsvsuLzuBQjiD9zzqGocZiPWr+uWJs AdbueS72R7FPXJPDUEPrJ9GdhGFyYARveY9cmdisOwcDOiSFfBjk3/89t4gROn4KUhezVuO9 VS14gVSns1561CJjlB47HkSBu4+FuzrfVygg4xitWAH119Ehw0vJcgkTw4Bqhk01Iw9us80m dFyU8JbJ0CVYe30gYKFFbnXoiT6xLLogKOkv0goPFxaXcMwWM9ei3SjAGVqgN6i8VnO7kquV LwkTe6ntEK0iY+l4qTKuyIOQLpCbWNI0eVwlx5b/pY2pt5TEGWAPMCZGjlidMx0aDcVX4oji 2/xegFAcxALrfOX3kj2FZ9kNAqLZu26AfqtslIqlBEAb5sZwPr351msBIdbaWX2UNw21I478 7eQ7UfohwXQHlXdhc/wop3VDkDzLBnvlK4ozSJI/9T5F/+9yEZvc6DKUWdEfD12o2El5hHan gCUQWDBKqZb1wcekK8KY2tmH8BBQi7k52IWYLJYfJdir/XpGm5SsDpf3zvDcIFXqFHAG7w7b fhriM+6oBOeIO9ew1Xj3swbRhDwdzRUhu7Uqayq1vdvKqGkgcQARAQABtCNaYWNoYXJ5IE1l ZGljbyA8em1lZGljb0BnZW50b28ub3JnPokCVAQTAQgAPgIbAwULCQgHAwUVCgkICwUWAwIB AAIeAQIXgBYhBEdYrNjamv1GpqYECtYQzzch54rNBQJbZpeGBQkEzWcaAAoJENYQzzch54rN iTEQALgfn8NqcY1P3VgE8n/ypYfCa8YhOhZcSlVOnIil9u4F0OPUP9/TFALaW2RRctUm7URP Oe6gMd8mzFauU1q8FKHW8vo6i48Oqb3RmkH4PNbH0EHfM1e3pAq+E6Bi08y7HzYUm5PKr+m3 oLNFvqZuG3RKgmKQTm8E56IpI8rODnVmBkLQEGkdi9hDB1Zkm9dlT+eRGglHAhnpb+AweOzh dvqdfu0SKEoyLjCvRIFltrtNIuKWpjqOVAaMU0hLfDlRVJR0oTThe7P0vhulKZtWS/L7wXJr e7NGIunM09JIAjYAXX5hmY9L0oKnp0WK5PksBKIu8W8TUwzVmTXNAs7qfEQQJ3LvAQo3xifa l2n0ixdoozltU8afxJohz8OKrWOAzE8kqWa+H+t1XZtQAy7yZA3x51vyp4hquyAMYrZbU8hC 0x/l6KM/qGI7RPohw5VxgneQCAbpu/G+3DLVH5QjPR+tKPtpFkEpxrbTNUhaOUqFipX92382 w2CCxDrNrBYbnFYKmVthZqVxrUuDfi4VIgeBkGzDKPwqjSkbdZ0/I5GAmjZh0aJWrRnzCum/ hZHHHcU+wOdXBwwGBhRf9SIaxCfaxw0PB+CZXsswvuaANah2b32EFTPcFCCgFvoGMNSSHFuo JR3PAo6qvwGhYdmTI/Na57P1FF+g3VojnAnK2eIEuQINBFs7tmwBEACfZb/EePObKC8tuVFl IMQaTRzm7q5THbwQvbdKdw/31cdYJaZZ7BFgnSBq2CBYDPxcm+TxvvjgNTam8kZGdEpCm3/v P1YdJSoiYbisJubV7JiAOoAmGtaDOVX5thpv28HL17wqK2d5jgqMTLFeT3hyaro5cjGAQlhT NOyfwVkdFMFLjOhGzq7aki4UZ9ieS/IedoyZvOblf79d5PV1xI0mf2w5yoHcfZRv7nwI9XMz nqlgC9/RQP7O+WfCl2pl9gyd0hc3uqUWl0ke9xHvVxlEdORH/f+OzeybYj9r80GkC6MeqiG8 qzuWO0IZZ5Gvzwkq0KgRAefr9Z/4vmyOiVhJrvk1gLp6VEdB54fUO3MG/PcnYiasRJOUwRQQ 1+q25w5i9ooZRk0LQ6oNnWbeVj+Pwn0mzL0/GjkdPsYclxzOLSlRAwkisepVl62L2m+XwXsZ j9DS3vIHEbbfCmKayBuGCXQEpJMmEnGqFLnOzv29cOutm3BnGXcWAg5aJjiV5PKEmr/4g9/E J8lphGnNCRQ2DvUzO1tlHpmT+JvDBwAgo5rfq5wjjeiS578kAgPoitOBzp/z5YM5FXX+shEd i5/ratErb9ysUfr3Z45YV3yM6MHgCpEg8+5k6fM6ey4tRnxXAYdJ4XzaSyML9fsnEg9aVaCP iRl+fuLSuF1QJf7LwwARAQABiQI8BBgBCAAmAhsMFiEER1is2Nqa/UampgQK1hDPNyHnis0F Als7trcFCQHhM8sACgkQ1hDPNyHnis0Wdw/9ERSLUHWjGW+oRJPC80RsSZO+9gbdo6AT8GP0 L3S1M60d/C4Pp9lT7sGvN31Q2h4/i1AGz2vsvjA4RUwuWI/M2F660ZmqbuLA4BbS/gjA+Wrf ZB8TDEoy7k0IO+ga0J6ENgxSOk0w1ZS+LYVrM8r7C73ZckAU2aknGOHvfIdRocpb464/JA+j kP8ENcXA0NVT17Te6K3OxpN3fBsHP7w9zbYaV3Ex3suXhLx0/+xxjLPTfwobEy3kTv5Qqumq Ph7TlXvc/9Ku4GwgAFUscLG5ZhsDULIWsRr0tyum0IYI3lpQnMykZiDCoiq7HETTwJeZPpJk PlVc7Qyy0eqcKyoiqd/ttr5tPwRFWk4OYmqzZFiZZ6sG7Zw68UXKHuhCV1MgXKPqS3M8Q6V+ nGL+ZwLE1GkvqwaPgn7uzJpfO+HzoWn3yxjNvh5plsZbM/kqCTMyByB0LQPZdjkO30GvV6dU wamblYkpytpZ5cmbIRITzsxPkOuWwXkLoz6y664JfH/Te3xVQ2rqZkuDCFrwy5nOqk92w7OY WGqyDOSL4tyCdcsXVoF5HXcHQLTv5WzCQ2MAQUW6zflyirYygTfJ3ds5mk5/J52UKEhIHPzJ Hc26Ubfvo38XuE0FJjb7xoZzCoSTpCVZ2H0f2cJOBLEVN9bnbelNlmp0m6mPAHfDrfsVR/u5 Ag0EWzvRagEQANK1C/HvZgnFVa+3tFmS4OVnCRO611C7WXubm5Y2xj/Lh2LOWv5TeTtTp5FR S3961b429TbJNv0q9N4mDi6XOGpZvWLkfiDw/VT9I+48B2eVXKx5N2H48S7t1Knwut6vuTEx 14MGiZToCs2Tu1fEUnaBv2Hg60ysVfplDAQadixzboLHM1DxLYn1W/cAUrhXAW5uNQlyE0Ze kiB10JxbnAurdpRSu4X81IrGJDK/oNAgAcWRieVX5J1N5LStPrloFKf+Dtl2z49WibXVTjwF Ir/BZYprkTTgNzeM3VRPnyRz7IA1pMkVX0r4C0O38mqHyDCpkM/TKsOpNMDqYcllD8Fa/Zyg S08RLymWvRXWn7Sz7MlZE9CCQ2aG+N20esYH3nwrYsdUdj87/nSwqYKhTWwBBIWtrrCHw56O ZMTXznX3OkLOBB0gXH92G5dKv4azBeAZzR6c+qvP6PxgVuAlvlnFbzgW5m4CrkTsEwSh/s2Y d9sJPctzkPkif6tDWk0qKy9lRwTKyOK0xqwZGcJBLdqInU05DLJlrf2QJKBS2SE1tEparQvT 8/+EBpzGk0omSxGmDxLW6EY1CTXV+LfbRYz42Y71f7aVNNqfF/CbJYLupe9DadjvceRM/ZFG WO6SzWoO7ed5uT2i8M33tC2EnK/BL1oZ5Wr7Kv5XED10JI+5ABEBAAGJBNIEGAEIACYWIQRH WKzY2pr9RqamBArWEM83IeeKzQUCWzvRagIbAgUJAeEzgAKgCRDWEM83IeeKzcHUIAQZAQgA fRYhBPDoF2pbVs6oErqQrtB1+4wQSj0gBQJbO9FqXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90 YXRpb25zLm9wZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGMEU4MTc2QTVCNTZDRUE4MTJCQTkw QUVEMDc1RkI4QzEwNEEzRDIwAAoJENB1+4wQSj0gJEMP/iP89Et7jH3Ai9w1bQ19T0ChOD4O 36IjisSTGTYsoeIAUgka/XFmuQSAgeTj5Tc73/Ny0V5jscjmtVKhUbFkLGetHuA0gHJpxEen j0zedAIQ36ToU0gia7Qsk7cpJJJRge3fdaUgb4aUcU5fTg2lkPdsMqW2B1/djlKz6T2cqX8y omvByiBdRSWVJlRmgqYQTJyQa6zLERWcknCG9T2oD9y9VJlVW3kcLT2QmB6a/3q/M0gCUBl8 qGHZYDGv+0su4SeppQ5BAOu9PXNV8pGdqTMsirQ+E20K5QprzmPg5kVG4x8qG1UpRpC/hJ9E 75wax9o4WrqHH4NmhvnL/Pfs2jK7PJLi/xRqvO65wGGO+4JW0kailhjvLoRYJi0B6ir5JhtK 44kctcqrn7QygO9ZB5r2/i1LyZrA+LWKLcpd7xxrraunFidoc63Mz5QadvWGcOduM6DOKS4b sLgr52V+413UkdtvBru/4hA9s1C8Dpc6DT0p0wFSJ7EBKE4dlu9VCnJ9+7X6HVMoq7aMj7ZW qJ1S1Jp/m6uJwzBDexcTokj8rTPBzbIo+3EfaGWMk47VoQgccee5B1jvxmLxhIbEd+LUc9ek jDQ0FETN7dOBuIIg5+Z6IdAycsLMHfNeSTP0jMHHZBKbptQwiaQ/hieMbSraN6fjozta4jw8 mUBoOJTgyjAP/2nbBGiOp/s8CkNEfbqFlLLVSnpN70PHPQ+FOKHe+FFg/bhyf6L0mGShJ8rH c8o7Re1Im/ztjj9hrGUfdicPFrU88Y+XgRGqb8DxjlyYSOVwIi6abyH5TgP1LNasBc23+cY/ nf2+n7Eq6/AOdgQRCKe5bIHXl94889HHVMQyfR+AvRZ9AiHNd7AGbhi2aMKsHgrnhuTcFO/d TR8NHolevl/zyOHmlcKaDUahjgTEK/SguUjmFHYDQYFDCrQj6U0o8uZtLA23wpiGgG1ajqpG w0VpKjkFzXcCNVeVme7XreRBxwstOda4Y00JUfOTGpR0+/bGRnJaQvBFV87WbBksVCa2McuY uzEqnkYwE4M/Hua9CRXReyvG31JRDbMubqXZjgisOtjIaNn7jvbweLoJhSzINkBB28tJiV6P ODhMx9XC5taaVLiPpABL/xzbqtHi6LCwXAYVlOS+ceiR2WIUigL866Ze607UePBJVQvsCuIS y7PV9kVbfN1/xKls9fiB87tv5XpyEUR9s+91Az6d8kterzLKqwxWSfp+HSepYuWOhlLsywUr BAJMnLR1yogNFuwyFkXIwKy/kNj4lIy+wLGNCQod1H84OLg1iHElIA+wxAWL7xnTYPxKX102 YsIUXy5A5ebVtwxpLuu8Yd6CnrTOvyLKqukoXSwP+16Mjo7C Message-ID: <1307e12b-2279-277e-a01e-0225bc88108c@gentoo.org> Date: Sat, 15 Dec 2018 23:38:03 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Project discussion list X-BeenThere: gentoo-project@lists.gentoo.org Reply-To: gentoo-project@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="9wt7VqG1uEdubVtNlJRX5Fa3fNHtGy2H0" X-Archives-Salt: 6fd9e956-fc44-4177-9b30-5d414b9abf26 X-Archives-Hash: 24baa2ca67ec4fa06fbb9283bf4512f4 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --9wt7VqG1uEdubVtNlJRX5Fa3fNHtGy2H0 Content-Type: multipart/mixed; boundary="YuFz9670wbJL6d0fmV1NeylFHuRy9uJO0"; protected-headers="v1" From: Zac Medico To: Alec Warner , gentoo-project , Zac Medico Message-ID: <1307e12b-2279-277e-a01e-0225bc88108c@gentoo.org> Subject: Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method References: <1600658.mo6uCtQdLC@reaper> In-Reply-To: --YuFz9670wbJL6d0fmV1NeylFHuRy9uJO0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 12/15/18 9:17 PM, Alec Warner wrote: >=20 >=20 > On Sun, Dec 16, 2018 at 12:13 AM Georgy Yakovlev > wrote: >=20 > On Saturday, December 15, 2018 8:40:38 PM PST Matt Turner wrote: > > On Sat, Dec 15, 2018 at 11:16 PM Alec Warner > wrote: > > > - Disk usage for git vs rsync > > > > This is why I have not switched. With git you pull down increasin= g > > amounts of history, whereas with rsync the data fits easily in a = <1GB > > partition. >=20 > Recent portage can use sync-depth =3D 1 > repo dir no longer grows as it used to and it's works fine unlike > initial > implementation that was giving trouble >=20 > https://bugs.gentoo.org/552814 >=20 > du -hs /var/db/repos/gentoo > 350M=C2=A0 =C2=A0 /var/db/repos/gentoo >=20 > example /etc/portage/repos.conf/gentoo.conf : > [DEFAULT] > main-repo =3D gentoo >=20 > [gentoo] > auto-sync =3D yes > location =3D /var/db/repos/gentoo > sync-type =3D git > sync-uri =3D https://github.com/gentoo-mirror/gentoo.git > sync-depth =3D 1 > sync-git-clone-extra-opts =3D -b master > sync-git-verify-commit-signature =3D true >=20 >=20 > sync is almost instantaneous compared to rsync, but some folks not > going to > like github as a mirror in this case. >=20 >=20 > I don't plan on using github for the mirror, so I'm not overly worried > about that portion. >=20 > +Zac Medico =C2=A0 >=20 > My recollection was that git doesn't ship with ebuild metadata by > default, so even if we make the first sync fast (by using depth=3D1 in = the > clone) do we have a good story for ebuild metadata? Is portage just > faster than in the past for ebuilds with missing metadata? Does emerge > --sync handle metadata regen for syncs with git origins? The metadata has to be included in the git repostory, and we've currently got "master" and "stable" branches which include everything that the rsync tree has: https://gitweb.gentoo.org/repo/sync/gentoo.git/log/?h=3Dmaster https://gitweb.gentoo.org/repo/sync/gentoo.git/log/?h=3Dstable Both branches are also mirrored on github: https://github.com/gentoo-mirror/gentoo/commits/master https://github.com/gentoo-mirror/gentoo/commits/stable It would be interesting to see some garbage collection stats for sync-deph =3D 1, people using it should post the output of this command: git count-objects -v > -A > =C2=A0 >=20 >=20 >=20 > --=20 > Georgy Yakovlev > Gentoo Linux Developer >=20 --=20 Thanks, Zac --YuFz9670wbJL6d0fmV1NeylFHuRy9uJO0-- --9wt7VqG1uEdubVtNlJRX5Fa3fNHtGy2H0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQKTBAEBCgB9FiEE8OgXaltWzqgSupCu0HX7jBBKPSAFAlwWANxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEYw RTgxNzZBNUI1NkNFQTgxMkJBOTBBRUQwNzVGQjhDMTA0QTNEMjAACgkQ0HX7jBBK PSBIMw//cHyOL3wpITN4oya8vs5M8XEYYme9PK+2ytTINKpRWiT7OJBuU2CFPxbG N64dc60wPITo+nNDlFwhp/M3iEUQFWhUEtKLDC8wP1V8ZzATb9fOofexXoQVv40p wWcRI/EWtp2CRttgW61c9brXpFv64vnkJkX9UQ03ABfXa/sCunjT1VSP+moua46K GQbu9U5DMikysB9coC0lU7mE3EGumunPFCr+/1mfLOPbDItooH+VlVTG7uPDKTFF Oa38DYe0boTiueCOFMIiRn1p2nTMZ5cq0Sqymt61u2jahplFfLv3khS6yYsOLh0e eF7+T+8pEjVIkmxxLCq1QKfTvZyPMwt1Vu7LC60ABldWZL2QRDWNM51U9GPi8Nt5 ZLbosCftw9vnBh9NCSDJR86T2lLGDI6QqkW5RgFlk0MgLk/NdLErZdUVHbewctJM M2PEjPK+guSNCAyHJJ6y6cHlLO7ni1lYqN59NlAj/+zstkfOe0elDDN9YUdX6Yj9 6xddh/rXzB7m8qHPXeJh97BNKPgNY89LJlQV/6F7eyySKBcaVyiW447NGAQgE2o4 R880G93xGrcR+fq2y8o3+PZQjSIzix0Zan8BMMMK8Mx34qoMaDsH56sfT6Jqt/ng U+sxui6RpIVGW99nXpi9T2W0qetaBKlLAv1Sv7V2DkPBwE6XHbc= =MnEk -----END PGP SIGNATURE----- --9wt7VqG1uEdubVtNlJRX5Fa3fNHtGy2H0--