From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id A8A23138334 for ; Sun, 16 Dec 2018 07:42:08 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 5EDDFE0BC3; Sun, 16 Dec 2018 07:42:07 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 02ADAE0BBC for ; Sun, 16 Dec 2018 07:42:07 +0000 (UTC) Received: from [IPv6:2600:8802:604:6600:b06e:5315:d0a7:5889] (unknown [IPv6:2600:8802:604:6600:b06e:5315:d0a7:5889]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: zmedico) by smtp.gentoo.org (Postfix) with ESMTPSA id DD5FF335C07; Sun, 16 Dec 2018 07:42:04 +0000 (UTC) Subject: Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method To: Alec Warner , gentoo-project , Zac Medico References: <1600658.mo6uCtQdLC@reaper> From: Zac Medico Openpgp: preference=signencrypt Autocrypt: addr=zmedico@gentoo.org; prefer-encrypt=mutual; keydata= mQINBFs7tmwBEADTzG+IcYtRfTfKryU7sUH7LlV1M+TdaCMfIkY4x6RyHXkaaqYuQ+U9HKn0 +m5FcZsZ1Ojik+We3Tz0F6kDbam6EWzBxmsLb/IHeUEsvsuLzuBQjiD9zzqGocZiPWr+uWJs AdbueS72R7FPXJPDUEPrJ9GdhGFyYARveY9cmdisOwcDOiSFfBjk3/89t4gROn4KUhezVuO9 VS14gVSns1561CJjlB47HkSBu4+FuzrfVygg4xitWAH119Ehw0vJcgkTw4Bqhk01Iw9us80m dFyU8JbJ0CVYe30gYKFFbnXoiT6xLLogKOkv0goPFxaXcMwWM9ei3SjAGVqgN6i8VnO7kquV LwkTe6ntEK0iY+l4qTKuyIOQLpCbWNI0eVwlx5b/pY2pt5TEGWAPMCZGjlidMx0aDcVX4oji 2/xegFAcxALrfOX3kj2FZ9kNAqLZu26AfqtslIqlBEAb5sZwPr351msBIdbaWX2UNw21I478 7eQ7UfohwXQHlXdhc/wop3VDkDzLBnvlK4ozSJI/9T5F/+9yEZvc6DKUWdEfD12o2El5hHan gCUQWDBKqZb1wcekK8KY2tmH8BBQi7k52IWYLJYfJdir/XpGm5SsDpf3zvDcIFXqFHAG7w7b fhriM+6oBOeIO9ew1Xj3swbRhDwdzRUhu7Uqayq1vdvKqGkgcQARAQABtCNaYWNoYXJ5IE1l ZGljbyA8em1lZGljb0BnZW50b28ub3JnPokCVAQTAQgAPgIbAwULCQgHAwUVCgkICwUWAwIB AAIeAQIXgBYhBEdYrNjamv1GpqYECtYQzzch54rNBQJbZpeGBQkEzWcaAAoJENYQzzch54rN iTEQALgfn8NqcY1P3VgE8n/ypYfCa8YhOhZcSlVOnIil9u4F0OPUP9/TFALaW2RRctUm7URP Oe6gMd8mzFauU1q8FKHW8vo6i48Oqb3RmkH4PNbH0EHfM1e3pAq+E6Bi08y7HzYUm5PKr+m3 oLNFvqZuG3RKgmKQTm8E56IpI8rODnVmBkLQEGkdi9hDB1Zkm9dlT+eRGglHAhnpb+AweOzh dvqdfu0SKEoyLjCvRIFltrtNIuKWpjqOVAaMU0hLfDlRVJR0oTThe7P0vhulKZtWS/L7wXJr e7NGIunM09JIAjYAXX5hmY9L0oKnp0WK5PksBKIu8W8TUwzVmTXNAs7qfEQQJ3LvAQo3xifa l2n0ixdoozltU8afxJohz8OKrWOAzE8kqWa+H+t1XZtQAy7yZA3x51vyp4hquyAMYrZbU8hC 0x/l6KM/qGI7RPohw5VxgneQCAbpu/G+3DLVH5QjPR+tKPtpFkEpxrbTNUhaOUqFipX92382 w2CCxDrNrBYbnFYKmVthZqVxrUuDfi4VIgeBkGzDKPwqjSkbdZ0/I5GAmjZh0aJWrRnzCum/ hZHHHcU+wOdXBwwGBhRf9SIaxCfaxw0PB+CZXsswvuaANah2b32EFTPcFCCgFvoGMNSSHFuo JR3PAo6qvwGhYdmTI/Na57P1FF+g3VojnAnK2eIEuQINBFs7tmwBEACfZb/EePObKC8tuVFl IMQaTRzm7q5THbwQvbdKdw/31cdYJaZZ7BFgnSBq2CBYDPxcm+TxvvjgNTam8kZGdEpCm3/v P1YdJSoiYbisJubV7JiAOoAmGtaDOVX5thpv28HL17wqK2d5jgqMTLFeT3hyaro5cjGAQlhT NOyfwVkdFMFLjOhGzq7aki4UZ9ieS/IedoyZvOblf79d5PV1xI0mf2w5yoHcfZRv7nwI9XMz nqlgC9/RQP7O+WfCl2pl9gyd0hc3uqUWl0ke9xHvVxlEdORH/f+OzeybYj9r80GkC6MeqiG8 qzuWO0IZZ5Gvzwkq0KgRAefr9Z/4vmyOiVhJrvk1gLp6VEdB54fUO3MG/PcnYiasRJOUwRQQ 1+q25w5i9ooZRk0LQ6oNnWbeVj+Pwn0mzL0/GjkdPsYclxzOLSlRAwkisepVl62L2m+XwXsZ j9DS3vIHEbbfCmKayBuGCXQEpJMmEnGqFLnOzv29cOutm3BnGXcWAg5aJjiV5PKEmr/4g9/E J8lphGnNCRQ2DvUzO1tlHpmT+JvDBwAgo5rfq5wjjeiS578kAgPoitOBzp/z5YM5FXX+shEd i5/ratErb9ysUfr3Z45YV3yM6MHgCpEg8+5k6fM6ey4tRnxXAYdJ4XzaSyML9fsnEg9aVaCP iRl+fuLSuF1QJf7LwwARAQABiQI8BBgBCAAmAhsMFiEER1is2Nqa/UampgQK1hDPNyHnis0F Als7trcFCQHhM8sACgkQ1hDPNyHnis0Wdw/9ERSLUHWjGW+oRJPC80RsSZO+9gbdo6AT8GP0 L3S1M60d/C4Pp9lT7sGvN31Q2h4/i1AGz2vsvjA4RUwuWI/M2F660ZmqbuLA4BbS/gjA+Wrf ZB8TDEoy7k0IO+ga0J6ENgxSOk0w1ZS+LYVrM8r7C73ZckAU2aknGOHvfIdRocpb464/JA+j kP8ENcXA0NVT17Te6K3OxpN3fBsHP7w9zbYaV3Ex3suXhLx0/+xxjLPTfwobEy3kTv5Qqumq Ph7TlXvc/9Ku4GwgAFUscLG5ZhsDULIWsRr0tyum0IYI3lpQnMykZiDCoiq7HETTwJeZPpJk PlVc7Qyy0eqcKyoiqd/ttr5tPwRFWk4OYmqzZFiZZ6sG7Zw68UXKHuhCV1MgXKPqS3M8Q6V+ nGL+ZwLE1GkvqwaPgn7uzJpfO+HzoWn3yxjNvh5plsZbM/kqCTMyByB0LQPZdjkO30GvV6dU wamblYkpytpZ5cmbIRITzsxPkOuWwXkLoz6y664JfH/Te3xVQ2rqZkuDCFrwy5nOqk92w7OY WGqyDOSL4tyCdcsXVoF5HXcHQLTv5WzCQ2MAQUW6zflyirYygTfJ3ds5mk5/J52UKEhIHPzJ Hc26Ubfvo38XuE0FJjb7xoZzCoSTpCVZ2H0f2cJOBLEVN9bnbelNlmp0m6mPAHfDrfsVR/u5 Ag0EWzvRagEQANK1C/HvZgnFVa+3tFmS4OVnCRO611C7WXubm5Y2xj/Lh2LOWv5TeTtTp5FR S3961b429TbJNv0q9N4mDi6XOGpZvWLkfiDw/VT9I+48B2eVXKx5N2H48S7t1Knwut6vuTEx 14MGiZToCs2Tu1fEUnaBv2Hg60ysVfplDAQadixzboLHM1DxLYn1W/cAUrhXAW5uNQlyE0Ze kiB10JxbnAurdpRSu4X81IrGJDK/oNAgAcWRieVX5J1N5LStPrloFKf+Dtl2z49WibXVTjwF Ir/BZYprkTTgNzeM3VRPnyRz7IA1pMkVX0r4C0O38mqHyDCpkM/TKsOpNMDqYcllD8Fa/Zyg S08RLymWvRXWn7Sz7MlZE9CCQ2aG+N20esYH3nwrYsdUdj87/nSwqYKhTWwBBIWtrrCHw56O ZMTXznX3OkLOBB0gXH92G5dKv4azBeAZzR6c+qvP6PxgVuAlvlnFbzgW5m4CrkTsEwSh/s2Y d9sJPctzkPkif6tDWk0qKy9lRwTKyOK0xqwZGcJBLdqInU05DLJlrf2QJKBS2SE1tEparQvT 8/+EBpzGk0omSxGmDxLW6EY1CTXV+LfbRYz42Y71f7aVNNqfF/CbJYLupe9DadjvceRM/ZFG WO6SzWoO7ed5uT2i8M33tC2EnK/BL1oZ5Wr7Kv5XED10JI+5ABEBAAGJBNIEGAEIACYWIQRH WKzY2pr9RqamBArWEM83IeeKzQUCWzvRagIbAgUJAeEzgAKgCRDWEM83IeeKzcHUIAQZAQgA fRYhBPDoF2pbVs6oErqQrtB1+4wQSj0gBQJbO9FqXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90 YXRpb25zLm9wZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGMEU4MTc2QTVCNTZDRUE4MTJCQTkw QUVEMDc1RkI4QzEwNEEzRDIwAAoJENB1+4wQSj0gJEMP/iP89Et7jH3Ai9w1bQ19T0ChOD4O 36IjisSTGTYsoeIAUgka/XFmuQSAgeTj5Tc73/Ny0V5jscjmtVKhUbFkLGetHuA0gHJpxEen j0zedAIQ36ToU0gia7Qsk7cpJJJRge3fdaUgb4aUcU5fTg2lkPdsMqW2B1/djlKz6T2cqX8y omvByiBdRSWVJlRmgqYQTJyQa6zLERWcknCG9T2oD9y9VJlVW3kcLT2QmB6a/3q/M0gCUBl8 qGHZYDGv+0su4SeppQ5BAOu9PXNV8pGdqTMsirQ+E20K5QprzmPg5kVG4x8qG1UpRpC/hJ9E 75wax9o4WrqHH4NmhvnL/Pfs2jK7PJLi/xRqvO65wGGO+4JW0kailhjvLoRYJi0B6ir5JhtK 44kctcqrn7QygO9ZB5r2/i1LyZrA+LWKLcpd7xxrraunFidoc63Mz5QadvWGcOduM6DOKS4b sLgr52V+413UkdtvBru/4hA9s1C8Dpc6DT0p0wFSJ7EBKE4dlu9VCnJ9+7X6HVMoq7aMj7ZW qJ1S1Jp/m6uJwzBDexcTokj8rTPBzbIo+3EfaGWMk47VoQgccee5B1jvxmLxhIbEd+LUc9ek jDQ0FETN7dOBuIIg5+Z6IdAycsLMHfNeSTP0jMHHZBKbptQwiaQ/hieMbSraN6fjozta4jw8 mUBoOJTgyjAP/2nbBGiOp/s8CkNEfbqFlLLVSnpN70PHPQ+FOKHe+FFg/bhyf6L0mGShJ8rH c8o7Re1Im/ztjj9hrGUfdicPFrU88Y+XgRGqb8DxjlyYSOVwIi6abyH5TgP1LNasBc23+cY/ nf2+n7Eq6/AOdgQRCKe5bIHXl94889HHVMQyfR+AvRZ9AiHNd7AGbhi2aMKsHgrnhuTcFO/d TR8NHolevl/zyOHmlcKaDUahjgTEK/SguUjmFHYDQYFDCrQj6U0o8uZtLA23wpiGgG1ajqpG w0VpKjkFzXcCNVeVme7XreRBxwstOda4Y00JUfOTGpR0+/bGRnJaQvBFV87WbBksVCa2McuY uzEqnkYwE4M/Hua9CRXReyvG31JRDbMubqXZjgisOtjIaNn7jvbweLoJhSzINkBB28tJiV6P ODhMx9XC5taaVLiPpABL/xzbqtHi6LCwXAYVlOS+ceiR2WIUigL866Ze607UePBJVQvsCuIS y7PV9kVbfN1/xKls9fiB87tv5XpyEUR9s+91Az6d8kterzLKqwxWSfp+HSepYuWOhlLsywUr BAJMnLR1yogNFuwyFkXIwKy/kNj4lIy+wLGNCQod1H84OLg1iHElIA+wxAWL7xnTYPxKX102 YsIUXy5A5ebVtwxpLuu8Yd6CnrTOvyLKqukoXSwP+16Mjo7C Message-ID: <9967b600-0048-0597-9b8b-6076e163179b@gentoo.org> Date: Sat, 15 Dec 2018 23:42:01 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Project discussion list X-BeenThere: gentoo-project@lists.gentoo.org Reply-To: gentoo-project@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="XEMyswiVK2awSeIRor063As6EM3fd0rnt" X-Archives-Salt: 02d0a046-5d1c-412f-9c35-5a7e019319f9 X-Archives-Hash: 41fad2d2f87d6071c93d7a0053d2219b This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --XEMyswiVK2awSeIRor063As6EM3fd0rnt Content-Type: multipart/mixed; boundary="S9qO0AN4MaX1OAoGHbrlrTYK6z694JCL5"; protected-headers="v1" From: Zac Medico To: Alec Warner , gentoo-project , Zac Medico Message-ID: <9967b600-0048-0597-9b8b-6076e163179b@gentoo.org> Subject: Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method References: <1600658.mo6uCtQdLC@reaper> In-Reply-To: --S9qO0AN4MaX1OAoGHbrlrTYK6z694JCL5 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 12/15/18 9:17 PM, Alec Warner wrote: >=20 >=20 > On Sun, Dec 16, 2018 at 12:13 AM Georgy Yakovlev > wrote: >=20 > On Saturday, December 15, 2018 8:40:38 PM PST Matt Turner wrote: > > On Sat, Dec 15, 2018 at 11:16 PM Alec Warner > wrote: > > > - Disk usage for git vs rsync > > > > This is why I have not switched. With git you pull down increasin= g > > amounts of history, whereas with rsync the data fits easily in a = <1GB > > partition. >=20 > Recent portage can use sync-depth =3D 1 > repo dir no longer grows as it used to and it's works fine unlike > initial > implementation that was giving trouble >=20 > https://bugs.gentoo.org/552814 >=20 > du -hs /var/db/repos/gentoo > 350M=C2=A0 =C2=A0 /var/db/repos/gentoo >=20 > example /etc/portage/repos.conf/gentoo.conf : > [DEFAULT] > main-repo =3D gentoo >=20 > [gentoo] > auto-sync =3D yes > location =3D /var/db/repos/gentoo > sync-type =3D git > sync-uri =3D https://github.com/gentoo-mirror/gentoo.git > sync-depth =3D 1 > sync-git-clone-extra-opts =3D -b master > sync-git-verify-commit-signature =3D true >=20 >=20 > sync is almost instantaneous compared to rsync, but some folks not > going to > like github as a mirror in this case. >=20 >=20 > I don't plan on using github for the mirror, so I'm not overly worried > about that portion. >=20 > +Zac Medico =C2=A0 >=20 > My recollection was that git doesn't ship with ebuild metadata by > default, so even if we make the first sync fast (by using depth=3D1 in = the > clone) do we have a good story for ebuild metadata? Is portage just > faster than in the past for ebuilds with missing metadata? Does emerge > --sync handle metadata regen for syncs with git origins? >=20 > -A The metadata has to be included in the git repostory, and we've currently got "master" and "stable" branches which include everything that the rsync tree has: https://gitweb.gentoo.org/repo/sync/gentoo.git/log/?h=3Dmaster https://gitweb.gentoo.org/repo/sync/gentoo.git/log/?h=3Dstable Both branches are also mirrored on github: https://github.com/gentoo-mirror/gentoo/commits/master https://github.com/gentoo-mirror/gentoo/commits/stable It would be interesting to see some garbage collection stats for sync-deph =3D 1, people using it should post the output of this command: git count-objects -v > =C2=A0 >=20 >=20 >=20 > --=20 > Georgy Yakovlev > Gentoo Linux Developer >=20 --=20 Thanks, Zac --S9qO0AN4MaX1OAoGHbrlrTYK6z694JCL5-- --XEMyswiVK2awSeIRor063As6EM3fd0rnt Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQKTBAEBCgB9FiEE8OgXaltWzqgSupCu0HX7jBBKPSAFAlwWAcpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEYw RTgxNzZBNUI1NkNFQTgxMkJBOTBBRUQwNzVGQjhDMTA0QTNEMjAACgkQ0HX7jBBK PSBrxw//a3FQoDCBJUBAPvkTAKeh3w9TvcR4WIEtuClEpsW6+v5veWD4AB6fPbx5 JGXiLCysvLPnp0JDo6XU4H3mS7/BQU1HDhCiUHmgpBbsYEMB43ZabbvqCGeB917n iz118gKZ2HnF798ENrkkp9z6XT+Ew/DAHnVB15qzyZaMnPwp3zYQ7CA4jCv1dxYl 69RL0gnhQXvzSZYkpKBNlgUjYhcfTd9m/8dBn/0pvDF4Rn46hd5+xJw4Y1sexap3 LoGsgdjisNI2NEydhDjVXYsVPr2nquP0xr9do/hDFA3Gu0yF3yWLWhQ0VRfmrqRZ OngYumOjqe3TVhr99cToCdLd7/AhsutW+DjW7/0cT6rZ5XDjjln1/WrvxcGAVQtp W++kfYUcyI/jCrf4CmbxeDndEobjB9gpQEuSbQRkgCxEJEJu7JDmxHCSgxICOidn +GjMUmWz6vOdNhX6TIVDFbTPwOO/IYFyzCbal3HTH2yW4McWcfrusk3J1DCF8/fc 3D/+G5aJB1gKbMmnX+FGkCTHnXHB2JLgenTe2Qu+K3IkEGcaiHuNWcsl+eeM5UAA 6txxdpemTFg2EE3Z0Wmgnngg8zQEmbX3GkF9bIcruC+6XCFnhcgoKM4iiK62zB1x sWJC3dQnSroEFa1Y+odgnWgExKiexICmGMWbjNY6Gz+LQoBEgB8= =jZhD -----END PGP SIGNATURE----- --XEMyswiVK2awSeIRor063As6EM3fd0rnt--