[gentoo-user] --sync

[gentoo-user] --sync

[n952162]

I've been running gentoo for years now, and every time I go to --sync,
it's really a painful process.

The process can take *very* before you find out if it succeeded or not.

I try different repos.conf servers - one works for a while, then
doesn't, then later, the new one doesn't work anymore and the old one
works again.

I've asked here about it multiple times and get the answers

- "I don't have a problem"

- "just change the server"

- "keep trying"

It can take several hours before it finally works

It seems like a time-out problem.  Or maybe a memory problem ... In any
case, it doesn't seem like it ought to be difficult to at least know
what the problem is.

Or?

Re: [gentoo-user] --sync

[Jack]

On 7/31/22 15:51, n952162 wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.
>
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.
>
> I've asked here about it multiple times and get the answers
>
> - "I don't have a problem"
>
> - "just change the server"
>
> - "keep trying"
>
> It can take several hours before it finally works
>
> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
> Or?

Well, since "it works for me" I can only suggest some things to think of 
and to try.  While a memory problem is possible, if it is timing out, 
I'd look at a possible network problem.

First, as you talk about repos.conf, is your issue with the main portage 
repo or some overlay?  Second, for the problem (or just pick one if it 
happens with several) server, can you ping that server without 
problems?  Third (maybe should have been first) what is your sync method 
and what is GENTOO_MIRRORS in your make.conf?  Related to that last one, 
have you run mirrorselect recently?

No solution, I know, but maybe something to trigger a useful thought.

Re: [gentoo-user] --sync

[Dale]

n952162 wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.
>
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.
>
> I've asked here about it multiple times and get the answers
>
> - "I don't have a problem"
>
> - "just change the server"
>
> - "keep trying"
>
> It can take several hours before it finally works
>
> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
> Or?
>
>
>

This wiki page and mirrorselect may help.

https://wiki.gentoo.org/wiki/Mirrorselect

I think this is the command I use to get the best server to sync with. 
You may want to make a backup of make.conf first, just in case.

mirrorselect -D -s4

One of those should help some unless you have a internet connection
problem.  If you have problems with multiple servers, it could be a
connection problem and you may figure that out.  It tests by downloading
a file.  If it fails on most or all servers, connection problem.  If it
works on most, just a bad server(s) most likely. 

Hope that helps.

Dale

:-)  :-) 

Re: [gentoo-user] --sync

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

On Sun, 31 Jul 2022 21:51:05 +0200, n952162 wrote:

> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
> 
> The process can take *very* before you find out if it succeeded or not.
> 
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.

I switched to using git for syncing, from github, and haven't looked
back. It is *much* faster, several times faster syncing from github than
using rsync to sync from a local mirror, and github is always there.

% cat /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
priority = 20
location = /var/portage
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo
auto-sync = yes


-- 
Neil Bothwick

Cereal Killer Strikes Again! Cap'n Crunch found dead...

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] --sync

[Stefan Schmiedl]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]


Sonntag, 31. Juli 2022 21:51:
 
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.

> The process can take *very* [long] before you find out if it succeeded or not.
...

> It can take several hours before it finally works
 
Use a tool like atop to get some info about the throughput
of your network connection. Something like "ss -ntp" might
show interesting info about the state of the connection 
between your machine and the server, too.
 
You might have problems with DNS name resolution or your
box patiently trying to establish an IPv6 connection when
your internet connection is IPv4 only.
 
But switching to git, like Neil said, does make things faster.
 
FWIW, my config file reads like this:
 
# cat /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /var/db/repos/gentoo
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo.git
auto-sync = yes
sync-git-verify-commit-signature = yes
sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
 
good luck,
s.

> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.

> Or?



[-- Attachment #2: Type: text/html, Size: 7010 bytes --]

Re: [gentoo-user] --sync

[Lee]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

Had a similar issue to the OP, and it turned out to be a flaky internet
connection. I switched ISP's and, while sync's still take longer than I'd
like, I don't have the failures, time outs I did before.

On Sun, Jul 31, 2022 at 1:54 PM Stefan Schmiedl <s@xss.de> wrote:

> Sonntag, 31. Juli 2022 21:51:
>
>
>
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
>
> The process can take *very* [long] before you find out if it succeeded or
> not.
>
> ...
>
>
> It can take several hours before it finally works
>
>
>
> Use a tool like atop to get some info about the throughput
>
> of your network connection. Something like "ss -ntp" might
>
> show interesting info about the state of the connection
>
> between your machine and the server, too.
>
>
>
> You might have problems with DNS name resolution or your
>
> box patiently trying to establish an IPv6 connection when
>
> your internet connection is IPv4 only.
>
>
>
> But switching to git, like Neil said, does make things faster.
>
>
>
> FWIW, my config file reads like this:
>
>
>
> # cat /etc/portage/repos.conf/gentoo.conf
>
> [DEFAULT]
>
> main-repo = gentoo
>
>
> [gentoo]
>
> location = /var/db/repos/gentoo
>
> sync-type = git
>
> sync-uri = https://github.com/gentoo-mirror/gentoo.git
>
> auto-sync = yes
>
> sync-git-verify-commit-signature = yes
>
> sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
>
>
>
> good luck,
>
> s.
>
>
> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
>
> Or?
>
>
>
>
>
>

-- 
Lee 😎
<ny6p01@gmail.com>

[-- Attachment #2: Type: text/html, Size: 3571 bytes --]

Re: [gentoo-user] --sync

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On Sun, 31 Jul 2022 15:55:51 -0700, Lee wrote:

> Had a similar issue to the OP, and it turned out to be a flaky internet
> connection. I switched ISP's and, while sync's still take longer than
> I'd like, I don't have the failures, time outs I did before.

I have no complaints about the time taken to sync

% time emerge --sync
>>> Syncing repository 'gentoo' into '/var/portage'...
/var/portage
/usr/bin/git fetch origin
[snip]
=== Sync completed for gentoo

Action: sync for repo: gentoo, returned code = 0


emerge --sync  1.58s user 0.51s system 75% cpu 2.750 total


-- 
Neil Bothwick

Do Roman paramedics refer to IV's as "4's"?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] --sync

[Peter Humphrey]

On Sunday, 31 July 2022 21:43:12 BST Neil Bothwick wrote:

> I switched to using git for syncing, from github, and haven't looked
> back. It is *much* faster, several times faster syncing from github than
> using rsync to sync from a local mirror, and github is always there.

One thing to note, if you do change to git syncing, is that you'll have to 
delete your entire portage tree: 'rm -r /var/db/repos/portage'. That may seem 
like a bad idea if you're having sync problems, but it isn't really. After 
that, the sync will take just seconds, as Neil said.

You'll never look back.

-- 
Regards,
Peter.

Re: [gentoo-user] --sync

[Lee]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Given the speed improvement in using git for --sync, I wonder why that
isn't the default in the portage section of the manual....

On Sun, Jul 31, 2022 at 4:49 PM Peter Humphrey <peter@prh.myzen.co.uk>
wrote:

> On Sunday, 31 July 2022 21:43:12 BST Neil Bothwick wrote:
>
> > I switched to using git for syncing, from github, and haven't looked
> > back. It is *much* faster, several times faster syncing from github than
> > using rsync to sync from a local mirror, and github is always there.
>
> One thing to note, if you do change to git syncing, is that you'll have to
> delete your entire portage tree: 'rm -r /var/db/repos/portage'. That may
> seem
> like a bad idea if you're having sync problems, but it isn't really. After
> that, the sync will take just seconds, as Neil said.
>
> You'll never look back.
>
> --
> Regards,
> Peter.
>
>
>
>
>

-- 
Lee 😎
<ny6p01@gmail.com>

[-- Attachment #2: Type: text/html, Size: 1503 bytes --]

Re: [gentoo-user] --sync

[n952162]

On 7/31/22 21:51, n952162 wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.
>
> I try different repos.conf servers - one works for a while, then
> doesn't, then later, the new one doesn't work anymore and the old one
> works again.
>
> I've asked here about it multiple times and get the answers
>
> - "I don't have a problem"
>
> - "just change the server"
>
> - "keep trying"
>
> It can take several hours before it finally works
>
> It seems like a time-out problem.  Or maybe a memory problem ... In any
> case, it doesn't seem like it ought to be difficult to at least know
> what the problem is.
>
> Or?
>
>

Thanks all, for the various suggestions, I'll try each.

Re: [gentoo-user] --sync

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

On Monday, 1 August 2022 06:09:07 BST n952162 wrote:
> On 7/31/22 21:51, n952162 wrote:
> > I've been running gentoo for years now, and every time I go to --sync,
> > it's really a painful process.
> > 
> > The process can take *very* before you find out if it succeeded or not.
> > 
> > I try different repos.conf servers - one works for a while, then
> > doesn't, then later, the new one doesn't work anymore and the old one
> > works again.
> > 
> > I've asked here about it multiple times and get the answers
> > 
> > - "I don't have a problem"
> > 
> > - "just change the server"
> > 
> > - "keep trying"
> > 
> > It can take several hours before it finally works
> > 
> > It seems like a time-out problem.  Or maybe a memory problem ... In any
> > case, it doesn't seem like it ought to be difficult to at least know
> > what the problem is.
> > 
> > Or?
> 
> Thanks all, for the various suggestions, I'll try each.

Two points to consider when choosing git to sync your portage tree:

1. It used to be the case the first time you run git it would try to download 
GB of commits history and take ages to do so on a slow connection.  The 
solution used to be to add "EGIT_CLONE_TYPE=shallow" in your make.conf, but 
I'm not sure if this is still needed (I don't use git).

2. These days rsync uses hashes and gpg to check the integrity of portage and 
will flag up a warning in case of file tampering, or corrupt data.  As far as I 
know such a solution doesn't exist with git.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] --sync

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

On Mon, 01 Aug 2022 07:58:05 +0100, Michael wrote:

> 1. It used to be the case the first time you run git it would try to
> download GB of commits history and take ages to do so on a slow
> connection.  The solution used to be to add "EGIT_CLONE_TYPE=shallow"
> in your make.conf, but I'm not sure if this is still needed (I don't
> use git).

It doesn't seem to be the case, running sync with an empty portage tree
gives

>>> Syncing repository 'gentoo' into '/var/portage'...
/usr/bin/git clone --depth 1 https://github.com/gentoo-mirror/gentoo .

% du /var/portage
608M    /var/portage

It did take 32s, but most of that was the Performing Global Updates stage.


-- 
Neil Bothwick

"A hundred years of forgetting and it all comes rushing back..."

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] --sync

[tastytea]

On 2022-08-01 07:58+0100 Michael <confabulate@kintzios.com> wrote:

> […]
> 
> 2. These days rsync uses hashes and gpg to check the integrity of
> portage and will flag up a warning in case of file tampering, or
> corrupt data.  As far as I know such a solution doesn't exist with
> git.

Verification can be turned on with
  sync-git-verify-commit-signature = yes
in repos.conf.[1] This does not seem to be enabled by default.[2]

[1] <https://wiki.gentoo.org/wiki/Project:Portage/Repository_verification#git>
[2] <https://wiki.gentoo.org/wiki/Portage_Security#git-mirror_repositories>

Re: [gentoo-user] --sync

[Michael]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

On Monday, 1 August 2022 10:43:55 BST tastytea wrote:
> On 2022-08-01 07:58+0100 Michael <confabulate@kintzios.com> wrote:
> > […]
> > 
> > 2. These days rsync uses hashes and gpg to check the integrity of
> > portage and will flag up a warning in case of file tampering, or
> > corrupt data.  As far as I know such a solution doesn't exist with
> > git.
> 
> Verification can be turned on with
>   sync-git-verify-commit-signature = yes
> in repos.conf.[1] This does not seem to be enabled by default.[2]
> 
> [1]
> <https://wiki.gentoo.org/wiki/Project:Portage/Repository_verification#git>
> [2] <https://wiki.gentoo.org/wiki/Portage_Security#git-mirror_repositories>

I see ... this is an improvement from what I recall it to be.  Thanks for 
pointing it out.  Perhaps I should start using git again.  :-)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: --sync

[Grant Edwards]

On 2022-07-31, n952162 <n952162@web.de> wrote:
> I've been running gentoo for years now, and every time I go to --sync,
> it's really a painful process.
>
> The process can take *very* before you find out if it succeeded or not.

In my experience, long --sync times have always been due to a slow
rsync server. Switching to a different server fixed it for me (though
sometimes it took a few tries to find a server that responded in a
timly manner).

That said, git is an order of magnitude faster than even the fastest
rsync server.

> It can take several hours before it finally works

That seems like something other than just a slow server. When dealing
with slow servers, it never took more than 15-30 minutes.

--
Grant

[ME TOO] Re: [gentoo-user] --sync

[Dr Rainer Woitok]

Neil,

On Sunday, 2022-07-31 21:43:12 +0100, you wrote:

> ...
> I switched to using git for syncing, from github, and haven't looked
> back. It is *much* faster, several times faster syncing from github than
> using rsync to sync from a local mirror, and github is always there.

The replies from you and some other people suggesting the use of Git for
syncing and providing howtos  were the straw that broke the camel's back
and overcame my lazyness.   So I too switched to Git and now am as happy
as can be :-D

Many thanks for this tip :-)

Sincerely,
  Rainer