[gentoo-user] emerge --sync keeps failing

[gentoo-user] emerge --sync keeps failing

[n952162]

  * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
...!!! Manifest verification failed:
  Manifest mismatch for metadata/news/Manifest

I've raised this question before and the only useful answer I got was to
keep trying

Today, I get this on two machines.  On one, I've run it 6 times and it
fails every time.  Each time, it takes about 26 minutes to run.  That's
almost 3 hours with no progress.

Am I doing something wrong?

Re: [gentoo-user] emerge --sync keeps failing

[Michael Orlitzky]

On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
>   * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
> ...!!! Manifest verification failed:
>   Manifest mismatch for metadata/news/Manifest
> 
> I've raised this question before and the only useful answer I got was to
> keep trying
> 

On the off chance that something is screwy on the remote end, you can
always use "emerge-webrsync" and delay the problem until next time.

Otherwise, I would say check "dmesg" for disk errors, but if it's
happening on two machines that's a lot less likely.

Re: [gentoo-user] emerge --sync keeps failing

[n952162]

On 8/1/21 8:32 PM, Michael Orlitzky wrote:
> On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
>>    * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
>> ...!!! Manifest verification failed:
>>    Manifest mismatch for metadata/news/Manifest
>>
>> I've raised this question before and the only useful answer I got was to
>> keep trying
>>
> On the off chance that something is screwy on the remote end, you can
> always use "emerge-webrsync" and delay the problem until next time.


Does that install binary images?


> Otherwise, I would say check "dmesg" for disk errors, but if it's
> happening on two machines that's a lot less likely.
>

There's plenty of space available.

Re: [gentoo-user] emerge --sync keeps failing

[n952162]

On 8/2/21 9:20 AM, n952162 wrote:
> On 8/1/21 8:32 PM, Michael Orlitzky wrote:
>> On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
>>>    * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
>>> ...!!! Manifest verification failed:
>>>    Manifest mismatch for metadata/news/Manifest
>>>
>>> I've raised this question before and the only useful answer I got
>>> was to
>>> keep trying
>>>
>> On the off chance that something is screwy on the remote end, you can
>> always use "emerge-webrsync" and delay the problem until next time.
>
>
> Does that install binary images?
>
>
>> Otherwise, I would say check "dmesg" for disk errors, but if it's
>> happening on two machines that's a lot less likely.
>>
>
> There's plenty of space available.
>
>
>

No movement in /var/log/messages during the attempt.  I've had two fails
already this morning.

Re: [gentoo-user] emerge --sync keeps failing

[Michael Orlitzky]

On 2021-08-02 09:20:19, n952162 wrote:
> On 8/1/21 8:32 PM, Michael Orlitzky wrote:
> > On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
> >>    * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
> >> ...!!! Manifest verification failed:
> >>    Manifest mismatch for metadata/news/Manifest
> >>
> >> I've raised this question before and the only useful answer I got was to
> >> keep trying
> >>
> > On the off chance that something is screwy on the remote end, you can
> > always use "emerge-webrsync" and delay the problem until next time.
> 
> 
> Does that install binary images?
>

No, it just tells portage to use HTTP instead of rsync. One big file
is downloaded and verified instead of many little ones.

Re: [gentoo-user] emerge --sync keeps failing

[n952162]

On 8/1/21 8:32 PM, Michael Orlitzky wrote:
> On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
>>    * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
>> ...!!! Manifest verification failed:
>>    Manifest mismatch for metadata/news/Manifest
>>
>> I've raised this question before and the only useful answer I got was to
>> keep trying
>>
> On the off chance that something is screwy on the remote end, you can
> always use "emerge-webrsync" and delay the problem until next time.
>
> Otherwise, I would say check "dmesg" for disk errors, but if it's
> happening on two machines that's a lot less likely.
>
>
>

I have this problem every month.  Why does it fail?  Is it just a
timeout because my network is slow?  Can that be tweaked?

Re: [gentoo-user] emerge --sync keeps failing

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

On Monday, 2 August 2021 12:10:12 BST n952162 wrote:
> On 8/1/21 8:32 PM, Michael Orlitzky wrote:
> > On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
> >>    * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
> >> 
> >> ...!!! Manifest verification failed:
> >>    Manifest mismatch for metadata/news/Manifest
> >> 
> >> I've raised this question before and the only useful answer I got was to
> >> keep trying
> > 
> > On the off chance that something is screwy on the remote end, you can
> > always use "emerge-webrsync" and delay the problem until next time.
> > 
> > Otherwise, I would say check "dmesg" for disk errors, but if it's
> > happening on two machines that's a lot less likely.
> 
> I have this problem every month.  Why does it fail?  Is it just a
> timeout because my network is slow?  Can that be tweaked?

I get this problem over here, but on rare occasions.  Leaving it for half a 
day usually fixes it.  Have you tried a different rsync mirror?  You can use 
'mirrorselect -i -r' for this task.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] emerge --sync keeps failing

[Dale]

Michael wrote:
> On Monday, 2 August 2021 12:10:12 BST n952162 wrote:
>> On 8/1/21 8:32 PM, Michael Orlitzky wrote:
>>> On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
>>>>    * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
>>>>
>>>> ...!!! Manifest verification failed:
>>>>    Manifest mismatch for metadata/news/Manifest
>>>>
>>>> I've raised this question before and the only useful answer I got was to
>>>> keep trying
>>> On the off chance that something is screwy on the remote end, you can
>>> always use "emerge-webrsync" and delay the problem until next time.
>>>
>>> Otherwise, I would say check "dmesg" for disk errors, but if it's
>>> happening on two machines that's a lot less likely.
>> I have this problem every month.  Why does it fail?  Is it just a
>> timeout because my network is slow?  Can that be tweaked?
> I get this problem over here, but on rare occasions.  Leaving it for half a 
> day usually fixes it.  Have you tried a different rsync mirror?  You can use 
> 'mirrorselect -i -r' for this task.


This is a very good idea.  I had to switch a few months ago.  The one I
was using was either really busy or had other issues that slowed it to a
crawl.  It reminded me of dial-up days.  I might add, I found some that
didn't appear to be set up to sync with anymore.  It appeared they
removed Gentoo files from the server.  Could be a glitch so I didn't
report it. 

I synced last night, about 12 hours ago, without any problems. 

Dale

:-)  :-) 

Re: [gentoo-user] emerge --sync keeps failing

[n952162]

On 8/2/21 2:01 PM, Michael wrote:
> On Monday, 2 August 2021 12:10:12 BST n952162 wrote:
>> On 8/1/21 8:32 PM, Michael Orlitzky wrote:
>>> On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
>>>>     * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
>>>>
>>>> ...!!! Manifest verification failed:
>>>>     Manifest mismatch for metadata/news/Manifest
>>>>
>>>> I've raised this question before and the only useful answer I got was to
>>>> keep trying
>>> On the off chance that something is screwy on the remote end, you can
>>> always use "emerge-webrsync" and delay the problem until next time.
>>>
>>> Otherwise, I would say check "dmesg" for disk errors, but if it's
>>> happening on two machines that's a lot less likely.
>> I have this problem every month.  Why does it fail?  Is it just a
>> timeout because my network is slow?  Can that be tweaked?
> I get this problem over here, but on rare occasions.  Leaving it for half a
> day usually fixes it.  Have you tried a different rsync mirror?  You can use
> 'mirrorselect -i -r' for this task.


Ah, good idea.

Re: [gentoo-user] emerge --sync keeps failing

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote:

> > I have this problem every month.  Why does it fail?  Is it just a
> > timeout because my network is slow?  Can that be tweaked?  
> 
> I get this problem over here, but on rare occasions.  Leaving it for
> half a day usually fixes it.  Have you tried a different rsync mirror?
> You can use 'mirrorselect -i -r' for this task.

I used to see this from time to time, but then I switched to using git
instead of rsync and haven't seen it again. There's the added bonus that
syncing is MUCH faster too.


-- 
Neil Bothwick

WinErr 019: User error - Not our fault. Is Not! Is Not!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] emerge --sync keeps failing

[Michael]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Monday, 2 August 2021 22:17:35 BST Neil Bothwick wrote:
> On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote:
> > > I have this problem every month.  Why does it fail?  Is it just a
> > > timeout because my network is slow?  Can that be tweaked?
> > 
> > I get this problem over here, but on rare occasions.  Leaving it for
> > half a day usually fixes it.  Have you tried a different rsync mirror?
> > You can use 'mirrorselect -i -r' for this task.
> 
> I used to see this from time to time, but then I switched to using git
> instead of rsync and haven't seen it again. There's the added bonus that
> syncing is MUCH faster too.

Last time I looked at git for portage it was not not using gpg to verify the 
downloaded data - has this feature been added since then?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] emerge --sync keeps failing

[John Covici]

On Mon, 02 Aug 2021 18:42:29 -0400,
Michael wrote:
> 
> [1  <text/plain; UTF-8 (7bit)>]
> On Monday, 2 August 2021 22:17:35 BST Neil Bothwick wrote:
> > On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote:
> > > > I have this problem every month.  Why does it fail?  Is it just a
> > > > timeout because my network is slow?  Can that be tweaked?
> > > 
> > > I get this problem over here, but on rare occasions.  Leaving it for
> > > half a day usually fixes it.  Have you tried a different rsync mirror?
> > > You can use 'mirrorselect -i -r' for this task.
> > 
> > I used to see this from time to time, but then I switched to using git
> > instead of rsync and haven't seen it again. There's the added bonus that
> > syncing is MUCH faster too.
> 
> Last time I looked at git for portage it was not not using gpg to verify the 
> downloaded data - has this feature been added since then?
I don't know when you last looked, but I see it checking the signature
when downloading from git.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici wb2una
         covici@ccs.covici.com

Re: [gentoo-user] emerge --sync keeps failing

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

On Tuesday, 3 August 2021 02:28:09 BST John Covici wrote:
> On Mon, 02 Aug 2021 18:42:29 -0400,
> 
> Michael wrote:
> > 
> > On Monday, 2 August 2021 22:17:35 BST Neil Bothwick wrote:
> > > On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote:
> > > > > I have this problem every month.  Why does it fail?  Is it just a
> > > > > timeout because my network is slow?  Can that be tweaked?
> > > > 
> > > > I get this problem over here, but on rare occasions.  Leaving it for
> > > > half a day usually fixes it.  Have you tried a different rsync mirror?
> > > > You can use 'mirrorselect -i -r' for this task.
> > > 
> > > I used to see this from time to time, but then I switched to using git
> > > instead of rsync and haven't seen it again. There's the added bonus that
> > > syncing is MUCH faster too.
> > 
> > Last time I looked at git for portage it was not not using gpg to verify
> > the downloaded data - has this feature been added since then?
> 
> I don't know when you last looked, but I see it checking the signature
> when downloading from git.

It was some inordinate time ago - can't recall when exactly.  It was after 
portage started using gpg with rsync and quarantined any fetched files after 
they were downloaded until their signatures were validated.  As I understood 
at the time the difference with using git was it would only check the top 
commit in the repository had a valid signature, but not what eventually landed 
at the local PC.  Also, even to do this check it required you have enabled 
"sync-git-verify-commit-signature = yes" - but according to 'man portage' it 
defaults to "no".

Anyhow, I recall using git to sync a live ebuild -9999 from some repo and 
portage started downloading gigabytes of cruft.  Presumably whole decades of 
old commits I didn't need or have space for.  I subsequently discovered I had 
to set "EGIT_CLONE_TYPE=shallow" in make.conf, but I can't find this variable 
in the man page now.  I suppose/hope portage using git will only download more 
recent commits?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] emerge --sync keeps failing

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On Tue, 03 Aug 2021 09:55:46 +0100, Michael wrote:

> Anyhow, I recall using git to sync a live ebuild -9999 from some repo
> and portage started downloading gigabytes of cruft.  Presumably whole
> decades of old commits I didn't need or have space for.  I subsequently
> discovered I had to set "EGIT_CLONE_TYPE=shallow" in make.conf, but I
> can't find this variable in the man page now.  I suppose/hope portage
> using git will only download more recent commits?

It does, you get just the current state of the tree by default, it's just
orders of magnitude faster, even compared with using rsync with a local
mirror. And I don't have to worry abut syncing too often and upsetting
infra as I'm syncing from github.


-- 
Neil Bothwick

Talk is cheap because supply exceeds demand.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] emerge --sync keeps failing

[Michael]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Tuesday, 3 August 2021 11:49:23 BST Neil Bothwick wrote:
> On Tue, 03 Aug 2021 09:55:46 +0100, Michael wrote:
> > Anyhow, I recall using git to sync a live ebuild -9999 from some repo
> > and portage started downloading gigabytes of cruft.  Presumably whole
> > decades of old commits I didn't need or have space for.  I subsequently
> > discovered I had to set "EGIT_CLONE_TYPE=shallow" in make.conf, but I
> > can't find this variable in the man page now.  I suppose/hope portage
> > using git will only download more recent commits?
> 
> It does, you get just the current state of the tree by default, it's just
> orders of magnitude faster, even compared with using rsync with a local
> mirror. And I don't have to worry abut syncing too often and upsetting
> infra as I'm syncing from github.

I see, thanks Neil.  Does it now check sigs of downloaded data, like rsync 
does?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] emerge --sync keeps failing

[Michael Orlitzky]

On Mon, 2021-08-02 at 13:10 +0200, n952162 wrote:
> > 
> 
> I have this problem every month.  Why does it fail?  Is it just a
> timeout because my network is slow?  Can that be tweaked?
> 

I'm not really sure. I've seen it fail in the past due to bad memory or
a dying hard drive, but it also "just happens." It might be the
upstream mirror? I'd have to investigate precisely what is failing..
and that would take longer than re-syncing a few times every couple of
months.

Re: [gentoo-user] emerge --sync keeps failing

[n952162]

On 8/2/21 2:01 PM, Michael wrote:
> On Monday, 2 August 2021 12:10:12 BST n952162 wrote:
>> On 8/1/21 8:32 PM, Michael Orlitzky wrote:
>>> On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote:
>>>>     * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine
>>>>
>>>> ...!!! Manifest verification failed:
>>>>     Manifest mismatch for metadata/news/Manifest
>>>>
>>>> I've raised this question before and the only useful answer I got was to
>>>> keep trying
>>> On the off chance that something is screwy on the remote end, you can
>>> always use "emerge-webrsync" and delay the problem until next time.
>>>
>>> Otherwise, I would say check "dmesg" for disk errors, but if it's
>>> happening on two machines that's a lot less likely.
>> I have this problem every month.  Why does it fail?  Is it just a
>> timeout because my network is slow?  Can that be tweaked?
> I get this problem over here, but on rare occasions.  Leaving it for half a
> day usually fixes it.  Have you tried a different rsync mirror?  You can use
> 'mirrorselect -i -r' for this task.


Unfortunately, mirrorselect doesn't seem to work anymore:

configparser.MissingSectionHeaderError: File contains no section headers.

It seems to have lost track of whether it's editing make.conf or repos.conf.

I ran it with the -o option, to a tempfile and tried to put that into
/etc/portage/repos.conf, I think, but that didn't work either.

Next, I manually editted /usr/share/portage/config/repos.conf and put
that line in where the old sync-uri was defined.  But that didn't work
either.

Now, I've created a new directory /etc/portage/repos.conf/ and moved the
file generate by the -o of mirrorselect into that as gentoo.conf and
added a "[gentoo]" section head.  It MIGHT be working now.