[gentoo-dev] Suggested sync method/Portage config for devs on ~arch?

[gentoo-dev] Suggested sync method/Portage config for devs on ~arch?

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 935 bytes --]

Ever since we switched to Git, I've tried to use gentoo.git (or a
mirror) to sync from. I later found a configuration that hasufell used
at the time. [1] It works well so far, but I wanted to ask the greater
developer community what method(s) they employ to sync from our
repositories and why it's a good fit for them. I'm hoping that this
discussion may lead to suggestions we can put in the devmanual or other
documentation to make the "transition" to a developer smoother for new
staff, or simply make it easier for our users to be close to the
bleeding edge, if that's what they desire.

So, for a developer/user using ~arch, what do you use and/or recommend
for Portage configuration?

Thanks for reading.

[1]: https://github.com/hasufell/portage-gentoo-git-config
-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Suggested sync method/Portage config for devs on ~arch?

[James Le Cuirot]

On Mon, 27 Feb 2017 23:29:20 -0800
Daniel Campbell <zlg@gentoo.org> wrote:

> Ever since we switched to Git, I've tried to use gentoo.git (or a
> mirror) to sync from. I later found a configuration that hasufell used
> at the time. [1] It works well so far, but I wanted to ask the greater
> developer community what method(s) they employ to sync from our
> repositories and why it's a good fit for them.

I use hasufell's repo too. I'm surprised we haven't made it more
official.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

Re: [gentoo-dev] Suggested sync method/Portage config for devs on ~arch?

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 486 bytes --]

On 2017-02-28 10:52, James Le Cuirot wrote:
> I use hasufell's repo too. I'm surprised we haven't made it more
> official.

The public Gentoo git mirror is

  https://github.com/gentoo-mirror/gentoo

This git mirror includes pre-generated metadata. No need for any
hack/additional step.

Devs maybe want to switch branch from stable (default branch) to master
branch (stable branch has CI coverage and will only sync if everything
is fine).


-- 
Regards,
Thomas



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] Suggested sync method/Portage config for devs on ~arch?

[Brian Evans]

[-- Attachment #1.1: Type: text/plain, Size: 4060 bytes --]

On 2/28/2017 5:14 AM, Thomas Deutschmann wrote:
> On 2017-02-28 10:52, James Le Cuirot wrote:
>> I use hasufell's repo too. I'm surprised we haven't made it more
>> official.
> 
> The public Gentoo git mirror is
> 
>   https://github.com/gentoo-mirror/gentoo
> 
> This git mirror includes pre-generated metadata. No need for any
> hack/additional step.
> 
> Devs maybe want to switch branch from stable (default branch) to master
> branch (stable branch has CI coverage and will only sync if everything
> is fine).
> 
> 

People, developers and users, may want to consider some facts when
cloning from that repository. (Please read the entire explanation below
before commenting.)

Git does a very poor job of data deduplication by default in this
repository and repacks are necessary to keep it sane.  Unfortunately,
there doesn't seem to be a way to trigger GitHub to do so.

This is not any developer's fault.  Simply a limitation or possible flaw
in git itself.

Shallow clones (--depth=1) are just fine.

Full clones are ridiculous in size because of needing a full repack
periodically.  GitHub even hung up on me for a full clone after about a
GiB downloaded. Infra's copy has been/is being repacked so the sizes are
not so bad.

If you don't need the full history, then this is very quick and small
footprint.
End users typically won't care but developers may.

Brian

Evidence:

Github...
> grknight@akame ~ $ git clone --depth=1 https://github.com/gentoo-mirror/gentoo.git
> Cloning into 'gentoo'...
> remote: Counting objects: 155625, done.
> remote: Compressing objects: 100% (128104/128104), done.
> remote: Total 155625 (delta 31631), reused 75529 (delta 26281), pack-reused 0
> Receiving objects: 100% (155625/155625), 80.36 MiB | 2.35 MiB/s, done.
> Resolving deltas: 100% (31631/31631), done.
> Checking out files: 100% (141011/141011), done.
> grknight@akame ~ $ du -sh gentoo/.git
> 101M    gentoo/.git
> grknight@akame ~ $ rm -fr gentoo
> grknight@akame ~ $ git clone https://github.com/gentoo-mirror/gentoo.git
> Cloning into 'gentoo'...
> remote: Counting objects: 3668662, done.
> remote: Compressing objects: 100% (677/677), done.
> error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
> fatal: The remote end hung up unexpectedly
> fatal: early EOF
> fatal: index-pack failed
> grknight@akame ~ $ rm -fr gentoo
> grknight@akame ~ $ git clone https://github.com/gentoo-mirror/gentoo.git
> Cloning into 'gentoo'...
> remote: Counting objects: 3668680, done.
> remote: Compressing objects: 100% (694/694), done.
> Receiving objects: 100% (3668680/3668680), 1.27 GiB | 2.19 MiB/s, done.
> remote: Total 3668680 (delta 326), reused 0 (delta 0), pack-reused 3667973
> Resolving deltas: 100% (3144716/3144716), done.
> Checking out files: 100% (141015/141015), done.
> grknight@akame ~ $ du -sh gentoo/.git
> 1.4G    gentoo/.git


Infra..
> grknight@akame ~ $ git clone --depth=1 https://anongit.gentoo.org/git/repo/sync/gentoo.git
> Cloning into 'gentoo'...
> remote: Counting objects: 155630, done.
> remote: Compressing objects: 100% (138925/138925), done.
> remote: Total 155630 (delta 44790), reused 68200 (delta 15465)
> Receiving objects: 100% (155630/155630), 75.28 MiB | 1.88 MiB/s, done.
> Resolving deltas: 100% (44790/44790), done.
> Checking out files: 100% (141015/141015), done.
> grknight@akame ~ $ du -sh gentoo/.git
> 96M     gentoo/.git
> grknight@akame ~ $ rm -fr gentoo
> grknight@akame ~ $ git clone https://anongit.gentoo.org/git/repo/sync/gentoo.git
> Cloning into 'gentoo'...
> remote: Counting objects: 3259105, done.
> remote: Compressing objects: 100% (472991/472991), done.
> remote: Total 3259105 (delta 2818449), reused 3193051 (delta 2758678)
> Receiving objects: 100% (3259105/3259105), 556.04 MiB | 1.02 MiB/s, done.
> Resolving deltas: 100% (2818449/2818449), done.
> Checking out files: 100% (141015/141015), done.
> grknight@akame ~ $ du -sh gentoo/.git
> 659M    gentoo/.git



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 834 bytes --]

Re: [gentoo-dev] Suggested sync method/Portage config for devs on ~arch?

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

On Tue, 28 Feb 2017 11:14:47 +0100 Thomas Deutschmann wrote:
> On 2017-02-28 10:52, James Le Cuirot wrote:
> > I use hasufell's repo too. I'm surprised we haven't made it more
> > official.
> 
> The public Gentoo git mirror is
> 
>   https://github.com/gentoo-mirror/gentoo

Please note that this is _mirror_. Gentoo infra provides official
synchronization friendly git repository:

https://gitweb.gentoo.org/repo/sync/gentoo.git/

It contains all caches and metadata pregenerated each ~20 min
interval and allows our users to easily use Gentoo without any
dependency on proprietary Github solution.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]