[gentoo-user] Missing digest for *** Tree looks messed up.

[gentoo-user] Missing digest for *** Tree looks messed up.

[Dale]

Howdy,

I'm sure I'm not alone on monitoring -dev to see upcoming changes.  I
noticed they switched to git or something and it seems to have caused a
bit of trouble.  At least I think it may have.  I did my usual 'eix-sync
&& emerge -uvaDN world'.  The sync took MUCH longer than usual.  I'm
talking a WHOLE LOT longer than usual.  My first thought, one time thing
because of the changes, maybe.  Then I got a screen full of this sort of
stuff.



 * Missing digest for
'/var/cache/portage/tree/media-libs/fontconfig/fontconfig-2.11.1-r2.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-plugins/frei0r-plugins/frei0r-plugins-1.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/sys-fs/mdadm/mdadm-3.3.1-r2.ebuild'
 * Missing digest for
'/var/cache/portage/tree/kde-base/kdm/kdm-4.11.19.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-plugins/gst-plugins-resindvd/gst-plugins-resindvd-0.10.23-r1.ebuild'
 * Missing digest for
'/var/cache/portage/tree/sys-fs/fuse/fuse-2.9.4.ebuild'
 * Missing digest for
'/var/cache/portage/tree/net-dialup/ppp/ppp-2.4.7.ebuild'
 - * Missing digest for
'/var/cache/portage/tree/net-misc/wget/wget-1.16.ebuild'
 | * Missing digest for
'/var/cache/portage/tree/sys-apps/dbus/dbus-1.8.16.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-video/transcode/transcode-1.1.7-r3.ebuild'
 - * Missing digest for
'/var/cache/portage/tree/kde-base/kwin/kwin-4.11.19.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-fonts/efont-unicode/efont-unicode-0.4.2-r1.ebuild'
 | * Missing digest for
'/var/cache/portage/tree/kde-apps/kolourpaint/kolourpaint-4.14.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/kde-apps/kbreakout/kbreakout-4.14.3.ebuild'
 - * Missing digest for
'/var/cache/portage/tree/dev-cpp/eigen/eigen-3.1.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/x11-proto/xcb-proto/xcb-proto-1.11.ebuild'
 | * Missing digest for
'/var/cache/portage/tree/app-portage/flaggie/flaggie-0.2.1.ebuild'
 * Missing digest for
'/var/cache/portage/tree/kde-apps/plasma-apps/plasma-apps-4.14.3.ebuild'
 / * Missing digest for
'/var/cache/portage/tree/x11-libs/libXi/libXi-1.7.4.ebuild'
 * Missing digest for
'/var/cache/portage/tree/kde-misc/krusader/krusader-2.4.0_beta3-r1.ebuild'
 \ * Manifest not found for
'/var/cache/portage/tree/virtual/acl/acl-0-r1.ebuild'
 - * Missing digest for
'/var/cache/portage/tree/dev-libs/libcdio/libcdio-0.92.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-libs/libass/libass-0.11.2.ebuild'
 | * Missing digest for
'/var/cache/portage/tree/kde-apps/pairs/pairs-4.14.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/x11-misc/obconf-qt/obconf-qt-0.1.0.ebuild'
 / * Missing digest for
'/var/cache/portage/tree/kde-base/kstartupconfig/kstartupconfig-4.11.19.ebuild'
 * Missing digest for
'/var/cache/portage/tree/dev-ruby/rdoc/rdoc-4.1.2-r1.ebuild'
 / * Missing digest for
'/var/cache/portage/tree/kde-apps/kalgebra/kalgebra-4.14.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/app-admin/webmin/webmin-1.730.ebuild'
 \ * Missing digest for
'/var/cache/portage/tree/x11-libs/libXau/libXau-1.0.8.ebuild'
 * Missing digest for
'/var/cache/portage/tree/dev-python/pillow/pillow-2.8.1.ebuild'
 / * Missing digest for
'/var/cache/portage/tree/kde-base/libkgreeter/libkgreeter-4.11.19.ebuild'
 * Missing digest for
'/var/cache/portage/tree/kde-apps/ksnapshot/ksnapshot-4.14.3.ebuild'
 | * Missing digest for
'/var/cache/portage/tree/net-im/pidgin/pidgin-2.10.11.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-sound/gsm/gsm-1.0.13-r1.ebuild'
 - * Manifest not found for
'/var/cache/portage/tree/virtual/ffmpeg/ffmpeg-9-r2.ebuild'
 * Missing digest for
'/var/cache/portage/tree/sys-libs/cracklib/cracklib-2.9.1-r1.ebuild'
 | * Missing digest for
'/var/cache/portage/tree/kde-apps/kiten/kiten-4.14.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/dev-perl/MailTools/MailTools-2.120.0-r1.ebuild'
 - * Missing digest for
'/var/cache/portage/tree/net-misc/netifrc/netifrc-0.2.2.ebuild'
 * Missing digest for
'/var/cache/portage/tree/x11-apps/iceauth/iceauth-1.0.7.ebuild'
 \ * Missing digest for
'/var/cache/portage/tree/kde-apps/kalzium/kalzium-4.14.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/dev-libs/liblinear/liblinear-196-r1.ebuild'
 / * Missing digest for
'/var/cache/portage/tree/app-misc/tmux/tmux-1.9a.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-libs/libraw/libraw-0.16.0-r1.ebuild'
 \ * Missing digest for
'/var/cache/portage/tree/kde-apps/kfmclient/kfmclient-4.14.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/app-admin/hddtemp/hddtemp-0.3_beta15-r7.ebuild'
 / * Missing digest for
'/var/cache/portage/tree/app-misc/screen/screen-4.0.3-r6.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-libs/raptor/raptor-2.0.9.ebuild'
 - * Missing digest for
'/var/cache/portage/tree/app-text/yelp-tools/yelp-tools-3.14.1.ebuild'
 * Missing digest for
'/var/cache/portage/tree/x11-misc/xkeyboard-config/xkeyboard-config-2.14.ebuild'
 | * Missing digest for
'/var/cache/portage/tree/media-video/lsdvd/lsdvd-0.17.ebuild'


That is just a very, very, very small copy and paste of it.  It goes on
for quite a while.  If I were betting, I'd think it is pretty much
everything that would usually be updated if not most of the tree.  So,
question.  Is this a bug and needs to be reported or will this clear up
sometime soon?   It looks broken to me.  o_O   Just saying.  Oh, nothing
will emerge now either.   This is really looking strange.  As a test:

root@fireball / # emerge -av seamonkey

These are the packages that would be merged, in order:

Calculating dependencies - * Missing digest for
'/var/cache/portage/tree/dev-libs/elfutils/elfutils-0.158.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-libs/fontconfig/fontconfig-2.11.1-r2.ebuild'
 * Missing digest for
'/var/cache/portage/tree/media-plugins/frei0r-plugins/frei0r-plugins-1.3.ebuild'
 * Missing digest for
'/var/cache/portage/tree/sys-fs/mdadm/mdadm-3.3.1-r2.ebuild'
 * Missing digest for
'/var/cache/portage/tree/kde-base/kdm/kdm-4.11.19.ebuild'

<<< The grandma of all snips >>>  <<< a whole lot of this >>>

* Missing digest for
'/var/cache/portage/tree/dev-libs/json-glib/json-glib-1.0.4.ebuild'
 \ * Missing digest for
'/var/cache/portage/tree/dev-perl/Net-SMTP-SSL/Net-SMTP-SSL-1.10.0-r1.ebuild'
 * Manifest not found for
'/var/cache/portage/tree/virtual/perl-Digest-MD5/perl-Digest-MD5-2.530.0-r2.ebuild'
 * Missing digest for
'/var/cache/portage/tree/kde-apps/bovo/bovo-4.14.3.ebuild'
... done!

!!! All ebuilds that could satisfy "seamonkey" have been masked.
!!! One of the following masked packages is required to complete your
request:
- www-client/seamonkey-2.33.1-r1::gentoo (masked by: corruption)

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

root@fireball / # equery list -p seamonkey
 * Searching for seamonkey ...
 * Missing digest for
'/var/cache/portage/tree/www-client/seamonkey/seamonkey-2.33.1-r1.ebuild'
[IP-] [  ] www-client/seamonkey-2.33.1-r1:0
root@fireball / #


This is the new part that I don't recall ever seeing before.  "masked
by: corruption"   What is that?  Just for giggles, I tried kdelibs as
well. 


!!! All ebuilds that could satisfy "kdelibs" have been masked.
!!! One of the following masked packages is required to complete your
request:
- kde-base/kdelibs-4.14.10::gentoo (masked by: corruption)
- kde-base/kdelibs-4.14.8::gentoo (masked by: corruption)
- kde-base/kdelibs-4.14.3-r1::gentoo (masked by: corruption)

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

root@fireball / #  


Can I take that "just saying" above back and say FUBAR instead?   :-( 

Thoughts?? 

Dale

:-)  :-) 

P. S.  I also notice this Gentoo page is not updating anymore.  Sort of
odd since they are posting about updates to packages on -dev but nothing
is showing up here: 

https://packages.gentoo.org/ 

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Rich Freeman]

On Sun, Aug 9, 2015 at 8:15 PM, Dale <rdalek1967@gmail.com> wrote:
>
> I'm sure I'm not alone on monitoring -dev to see upcoming changes.  I
> noticed they switched to git or something and it seems to have caused a
> bit of trouble.

There are a few issues they're working through, but nothing really
serious.  There were bound to be a few bumps with the change, and most
of the -dev posts are around trying to re-establish conventions,
especially around things like commit comments.  The desire is to make
the history easy to read, parse, etc, but it isn't really stopping
work from getting done and nobody is going to care if their history
looks a little different.

> At least I think it may have.  I did my usual 'eix-sync
> && emerge -uvaDN world'.  The sync took MUCH longer than usual.  I'm
> talking a WHOLE LOT longer than usual.  My first thought, one time thing
> because of the changes, maybe.  Then I got a screen full of this sort of
> stuff.

Interesting.  As I understand it rsync generation should be turned
off, but maybe some changes did make their way out.  I don't believe
commits are getting to the rsync servers right now, so you might see a
delay in package updates for a day or so.

Whether you've already seen it or will see it in the future, I would
expect the first rsync to be much longer.  The git migration probably
touched virtually every file in the tree in some way, which means that
rsync is going to be modifying just about everything.

I think it actually remains to be seen whether rsync is still the
fastest way to sync the new tree.  If you sync from anonymous git
(which is supported by repos.conf) that would probably be pretty
efficient, since by design git keeps track of what did and didn't
change and it doesn't need to read every inode in /usr/portage to
figure it out, unlike rsync.  I made a validator that finds what
changed between git revisions and you can do it VERY efficiently by
comparing the tree one directory level at a time (you can tell whether
anything in app-backup changed in a commit without having to read any
of the files, and if something changed you can repeat that one level
at a time until you read the actual files that changed - it is a O(log
base <dir-size>) tree search times the number of commits (O(n))).
Now, for enough time passing (many months most likely) rsync might be
more efficient since at some point you end up reading almost all the
files anyway and rsync doesn't care if a file changed three times or
if it changed once.

TL;DR - don't worry about it too much, but don't be surprised if
emerge --sync doesn't give you anything new for a day or two.  You can
of course clone any of the URLs under
https://gitweb.gentoo.org/repo/gentoo.git/ to get the latest changes
now straight from git.  We're up to 210 commits in git already today.

I'm sure the transition could have been less bumpy, but I'm glad we're
finally seeing it happen.  This has been in the works for a very long
time and sometimes you just have to pull the trigger.  If rsync is
down for a day it isn't the end of the world.

Oh, and a historical repo is posted at:
https://github.com/gentoo/gentoo-gitmig-20150809-draft
(that isn't official - I'm sure the official one will be
gentoo-hosted, but robbat has his hands plenty full already)

--
Rich

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Jc García]

2015-08-09 19:05 GMT-06:00 Rich Freeman <rich0@gentoo.org>:
> TL;DR - don't worry about it too much, but don't be surprised if
> emerge --sync doesn't give you anything new for a day or two
.
The rsync git mirror on github is up arlready I have been using this,
for some months, and has been working great.
I have this:
/etc/portage/repos.conf/gentoo.conf:
[gentoo]
location = /var/lib/portage/repos/gentoo
sync-type = git
sync-uri = https://github.com/gentoo/gentoo-portage-rsync-mirror
auto-sync = true

Will this be preferred for users or should we users consider change to
the same repo devs will commit to?
I know I care about history, but not always, and not in every computer
I have that uses gentoo, I'm fine with reading it from the copy of the
new git repo I already have in my directory dedicated to messing
around with overlays.

>  You can of course clone any of the URLs under
> https://gitweb.gentoo.org/repo/gentoo.git/ to get the latest changes
> now straight from git.  We're up to 210 commits in git already today.
>
> I'm sure the transition could have been less bumpy, but I'm glad we're
> finally seeing it happen.  This has been in the works for a very long
> time and sometimes you just have to pull the trigger.  If rsync is
> down for a day it isn't the end of the world.
>
Is good to see gentoo modernizing it's development infrastructure,
certainly, contribuiting will be much easier now. have you already
decided the way you will handle user constributions? (I have read you
will avoid merge commits, will devs be in charge of rebasing pull
request from users?)

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Jc García]

2015-08-09 22:53 GMT-06:00 Jc García <jyo.garcia@gmail.com>:

> Is good to see gentoo modernizing it's development infrastructure,
> certainly, contribuiting will be much easier now. have you already
> decided the way you will handle user constributions? (I have read you
> will avoid merge commits, will devs be in charge of rebasing pull
> request from users?)
...
BTW; gitg doesn't handle well the new git repo, trying to load the
first commit after I clicked on it took all my RAM and swap(8G total).
And vim + gitv coudln't handle the first commit either.

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Peter Humphrey]

On Sunday 09 August 2015 21:05:07 Rich Freeman wrote:

> Whether you've already seen it or will see it in the future, I would
> expect the first rsync to be much longer.  The git migration probably
> touched virtually every file in the tree in some way, which means that
> rsync is going to be modifying just about everything.

Yeah, 135,000 files transferred today.

-- 
Rgds
Peter

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Dale]

Rich Freeman wrote:
> On Sun, Aug 9, 2015 at 8:15 PM, Dale <rdalek1967@gmail.com> wrote:
>> I'm sure I'm not alone on monitoring -dev to see upcoming changes.  I
>> noticed they switched to git or something and it seems to have caused a
>> bit of trouble.
> There are a few issues they're working through, but nothing really
> serious.  There were bound to be a few bumps with the change, and most
> of the -dev posts are around trying to re-establish conventions,
> especially around things like commit comments.  The desire is to make
> the history easy to read, parse, etc, but it isn't really stopping
> work from getting done and nobody is going to care if their history
> looks a little different.


I was expecting a bump but not climbing a hill.  lol  Breaking things to
the point I can't emerge anything, that's a pretty big bump.  ;-)  Makes
me want to get out my tractor and box blade. 


>> At least I think it may have.  I did my usual 'eix-sync
>> && emerge -uvaDN world'.  The sync took MUCH longer than usual.  I'm
>> talking a WHOLE LOT longer than usual.  My first thought, one time thing
>> because of the changes, maybe.  Then I got a screen full of this sort of
>> stuff.
> Interesting.  As I understand it rsync generation should be turned
> off, but maybe some changes did make their way out.  I don't believe
> commits are getting to the rsync servers right now, so you might see a
> delay in package updates for a day or so.
>
> Whether you've already seen it or will see it in the future, I would
> expect the first rsync to be much longer.  The git migration probably
> touched virtually every file in the tree in some way, which means that
> rsync is going to be modifying just about everything.
>
> I think it actually remains to be seen whether rsync is still the
> fastest way to sync the new tree.  If you sync from anonymous git
> (which is supported by repos.conf) that would probably be pretty
> efficient, since by design git keeps track of what did and didn't
> change and it doesn't need to read every inode in /usr/portage to
> figure it out, unlike rsync.  I made a validator that finds what
> changed between git revisions and you can do it VERY efficiently by
> comparing the tree one directory level at a time (you can tell whether
> anything in app-backup changed in a commit without having to read any
> of the files, and if something changed you can repeat that one level
> at a time until you read the actual files that changed - it is a O(log
> base <dir-size>) tree search times the number of commits (O(n))).
> Now, for enough time passing (many months most likely) rsync might be
> more efficient since at some point you end up reading almost all the
> files anyway and rsync doesn't care if a file changed three times or
> if it changed once.
>
> TL;DR - don't worry about it too much, but don't be surprised if
> emerge --sync doesn't give you anything new for a day or two.  You can
> of course clone any of the URLs under
> https://gitweb.gentoo.org/repo/gentoo.git/ to get the latest changes
> now straight from git.  We're up to 210 commits in git already today.
>
> I'm sure the transition could have been less bumpy, but I'm glad we're
> finally seeing it happen.  This has been in the works for a very long
> time and sometimes you just have to pull the trigger.  If rsync is
> down for a day it isn't the end of the world.
>
> Oh, and a historical repo is posted at:
> https://github.com/gentoo/gentoo-gitmig-20150809-draft
> (that isn't official - I'm sure the official one will be
> gentoo-hosted, but robbat has his hands plenty full already)
>
> --
> Rich
>
>


It appears it was live even if it wasn't supposed to be.  I got the
updated version.  Maybe someone forgot to flip a switch until things got
settled??  Oh well. 

I've been reading -dev for ages.  I been using Gentoo since about 2003. 
I try to at least keep a eye on the changes that are coming.  I was
aware that the git change was coming and have read where many are well,
tickled pink about it.  I'm fine with it because it seems to be a
process that is heading toward something better.  I just wasn't
expecting this little hick-up.  At least it wasn't that other thing that
really gets on my nerve, that I don't want to mention.  ;-)  It broke my
rig.  That ain't good. 

I'm going to miss p.g.o.  Sometimes, I go there and look to see if
anything I'm interested in has been updated.  KDE, Firefox, GIMP, k3b
etc etc.  If they haven't, I generally wait a day or so, especially if I
know a update is coming and as soon as it is available, I want to sync
and update.  That saves a tiny fraction of load on servers and a little
of my time as well. 

I see that someone else filed a bug about this.  That was my reason for
asking.  I wanted to be sure it was a bug and not just a bad sync,
corrupted bad not just a bumpy road bad, before I filed one.  If I file
a bug, I'm pretty much positive it is a bug.  Anyway, here it is and it
seems to be fixed.

https://bugs.gentoo.org/show_bug.cgi?id=557184 

I might add, sync is taking a LONG time again.  Of course, my DSL is a
bit slower than some folks.  At least the bumpy road got smoothed out
tho.  It seems to be working again.   Maybe next sync will be back to
normal. 

Thanks much.

Dale

:-)  :-) 

Oh, keep us updated on when we can change our settings to the new and
improved stuff, after the bumpiness is gone of course.  :-D 

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Peter Humphrey]

On Monday 10 August 2015 09:13:01 Dale wrote:

> I might add, sync is taking a LONG time again.  Of course, my DSL is a
> bit slower than some folks.  At least the bumpy road got smoothed out
> tho.  It seems to be working again.   Maybe next sync will be back to
> normal.

I do one sync a day, of my LAN server, and sync anything else to that. I don't 
know how long it took today because I have it on a cron job at night.   :-)

-- 
Rgds
Peter

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Rich Freeman]

On Mon, Aug 10, 2015 at 11:12 AM, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
> On Monday 10 August 2015 09:13:01 Dale wrote:
>
>> I might add, sync is taking a LONG time again.  Of course, my DSL is a
>> bit slower than some folks.  At least the bumpy road got smoothed out
>> tho.  It seems to be working again.   Maybe next sync will be back to
>> normal.
>
> I do one sync a day, of my LAN server, and sync anything else to that. I don't
> know how long it took today because I have it on a cron job at night.   :-)
>

Well, if you synced with all broken manifest files, and re-synced with
all fixed manifest files, you'd expect one file to be updated for
every package in the repository, so that would be a long sync (but not
quite as long as the original one).

This was never intended to be a user-visible change (hence no news,
etc).  The digest issue was an oversight which was fixed.  At this
point rsync should work as it always did, minus one really long sync.

The main thing you'll probably see is that the headers of the ebuilds
all contain git hashes instead of cvs revisions.

-- 
Rich

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Dale]

Rich Freeman wrote:
> On Mon, Aug 10, 2015 at 11:12 AM, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>> On Monday 10 August 2015 09:13:01 Dale wrote:
>>
>>> I might add, sync is taking a LONG time again.  Of course, my DSL is a
>>> bit slower than some folks.  At least the bumpy road got smoothed out
>>> tho.  It seems to be working again.   Maybe next sync will be back to
>>> normal.
>> I do one sync a day, of my LAN server, and sync anything else to that. I don't
>> know how long it took today because I have it on a cron job at night.   :-)
>>
> Well, if you synced with all broken manifest files, and re-synced with
> all fixed manifest files, you'd expect one file to be updated for
> every package in the repository, so that would be a long sync (but not
> quite as long as the original one).
>
> This was never intended to be a user-visible change (hence no news,
> etc).  The digest issue was an oversight which was fixed.  At this
> point rsync should work as it always did, minus one really long sync.
>
> The main thing you'll probably see is that the headers of the ebuilds
> all contain git hashes instead of cvs revisions.
>


What I was expecting, one longer than usual sync maybe and even more
likely, a config update so that some change takes effect.  I expected
something to change but wasn't expecting what I got for sure.  That had
me wondering. 

We all good now tho.  < thumbs up >

Dale

:-)  :-) 

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Peter Humphrey]

On Monday 10 August 2015 11:16:08 Rich Freeman wrote:
> On Mon, Aug 10, 2015 at 11:12 AM, Peter Humphrey <peter@prh.myzen.co.uk> 
wrote:
> > On Monday 10 August 2015 09:13:01 Dale wrote:
> >> I might add, sync is taking a LONG time again.  Of course, my DSL is a
> >> bit slower than some folks.  At least the bumpy road got smoothed out
> >> tho.  It seems to be working again.   Maybe next sync will be back to
> >> normal.
> > 
> > I do one sync a day, of my LAN server, and sync anything else to that. I
> > don't know how long it took today because I have it on a cron job at
> > night.   :-)
> Well, if you synced with all broken manifest files, and re-synced with
> all fixed manifest files, you'd expect one file to be updated for
> every package in the repository, so that would be a long sync (but not
> quite as long as the original one).
> 
> This was never intended to be a user-visible change (hence no news,
> etc).  The digest issue was an oversight which was fixed.  At this
> point rsync should work as it always did, minus one really long sync.
> 
> The main thing you'll probably see is that the headers of the ebuilds
> all contain git hashes instead of cvs revisions.

Is syncing not yet sorted out? A fortnight later I'm still getting something 
like 24000 manifests synced every day, and about a week ago I got all the 
change-logs too.

When I tried syncing again just now I got all the manifests once more, and all 
the metadata.xml files too - 45,399 files in all. I hope this isn't the new 
steady state.

$ grep sync /etc/portage/repos.conf/gentoo.conf
sync-type = rsync
auto-sync = yes
sync-uri = rsync://rsync.uk.gentoo.org/gentoo-portage

-- 
Rgds
Peter

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Michael Orlitzky]

On 08/24/2015 08:08 AM, Peter Humphrey wrote:
> 
> Is syncing not yet sorted out? A fortnight later I'm still getting something 
> like 24000 manifests synced every day, and about a week ago I got all the 
> change-logs too.
> 

That was expected... one more time. Hopefully it shouldn't happen again.

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Peter Humphrey]

On Monday 24 August 2015 08:12:28 Michael Orlitzky wrote:
> On 08/24/2015 08:08 AM, Peter Humphrey wrote:
> > Is syncing not yet sorted out? A fortnight later I'm still getting
> > something like 24000 manifests synced every day, and about a week ago I
> > got all the change-logs too.
> 
> That was expected... one more time. Hopefully it shouldn't happen again.

Which, the change-logs or the manifests?

I can't help thinking that all those - billions? - of little files being 
transferred every day must be slugging the net somewhat.

-- 
Rgds
Peter

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Michael Orlitzky]

On 08/24/2015 09:47 AM, Peter Humphrey wrote:
> On Monday 24 August 2015 08:12:28 Michael Orlitzky wrote:
>> On 08/24/2015 08:08 AM, Peter Humphrey wrote:
>>> Is syncing not yet sorted out? A fortnight later I'm still getting
>>> something like 24000 manifests synced every day, and about a week ago I
>>> got all the change-logs too.
>>
>> That was expected... one more time. Hopefully it shouldn't happen again.
> 
> Which, the change-logs or the manifests?
> 
> I can't help thinking that all those - billions? - of little files being 
> transferred every day must be slugging the net somewhat.
> 

Both I think. I don't really know what was wrong, but some other people
do, and they say it should be fixed.

https://archives.gentoo.org/gentoo-dev/message/2c69bdebe4f7700bc939edd6780caa6c

https://archives.gentoo.org/gentoo-dev/message/c1a370104718167b2e7a53dbe14e7c17

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Peter Humphrey]

On Monday 24 August 2015 10:44:19 Michael Orlitzky wrote:
> On 08/24/2015 09:47 AM, Peter Humphrey wrote:
> > On Monday 24 August 2015 08:12:28 Michael Orlitzky wrote:
> >> On 08/24/2015 08:08 AM, Peter Humphrey wrote:
> >>> Is syncing not yet sorted out? A fortnight later I'm still getting
> >>> something like 24000 manifests synced every day, and about a week ago I
> >>> got all the change-logs too.
> >> 
> >> That was expected... one more time. Hopefully it shouldn't happen again.
> > 
> > Which, the change-logs or the manifests?
> > 
> > I can't help thinking that all those - billions? - of little files being
> > transferred every day must be slugging the net somewhat.
> 
> Both I think. I don't really know what was wrong, but some other people
> do, and they say it should be fixed.
 
As Fernando noted, it is now. Two syncs today: the first copied 41,787 files, 
the second 344.

> https://archives.gentoo.org/gentoo-dev/message/2c69bdebe4f7700bc939edd6780ca
> a6c
> 
> https://archives.gentoo.org/gentoo-dev/message/c1a370104718167b2e7a53dbe14e7
> c17

-- 
Rgds
Peter

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Dale]

Peter Humphrey wrote:
> On Monday 10 August 2015 09:13:01 Dale wrote:
>
>> I might add, sync is taking a LONG time again.  Of course, my DSL is a
>> bit slower than some folks.  At least the bumpy road got smoothed out
>> tho.  It seems to be working again.   Maybe next sync will be back to
>> normal.
> I do one sync a day, of my LAN server, and sync anything else to that. I don't 
> know how long it took today because I have it on a cron job at night.   :-)
>


I usually sync about twice a week, sometimes three if I see something I
want or need.  I just run my desktop here so updating isn't a huge
deal.  Sometimes I forget because I'm busy with other things and I might
go a week or more without a single sync up. 

When I used to run several rigs, I'd sync one rig and then sync the
others to my main rig.  I try not to sync to often. 

Dale

:-)  :-) 

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Rich Freeman]

On Mon, Aug 10, 2015 at 12:10 PM, Dale <rdalek1967@gmail.com> wrote:
> Peter Humphrey wrote:
>> On Monday 10 August 2015 09:13:01 Dale wrote:
>>
>>> I might add, sync is taking a LONG time again.  Of course, my DSL is a
>>> bit slower than some folks.  At least the bumpy road got smoothed out
>>> tho.  It seems to be working again.   Maybe next sync will be back to
>>> normal.
>> I do one sync a day, of my LAN server, and sync anything else to that. I don't
>> know how long it took today because I have it on a cron job at night.   :-)
>>
>
>
> I usually sync about twice a week, sometimes three if I see something I
> want or need.  I just run my desktop here so updating isn't a huge
> deal.  Sometimes I forget because I'm busy with other things and I might
> go a week or more without a single sync up.
>
> When I used to run several rigs, I'd sync one rig and then sync the
> others to my main rig.  I try not to sync to often.

I couldn't tell you who I ripped this off of but my cron routine is:

ionice -n 7 nice -n 20 emerge --sync              (alternatively
emerge-webrsync -k - preferred actually if you're using rsync)
ionice -n 7 nice -n 20 layman -S
emerge -puDv --changed-use world | col -bx | mutt -s "world update"
root@localhost
eix-update

Then I do this:
#!/bin/sh

LIST=$(mktemp);

emerge -puD --changed-use --color=n --columns --quiet=y --with-bdeps=n
world | awk '{print $2}' > ${LIST};

for PACKAGE in $(cat ${LIST});
do
  printf "Building binary package for ${PACKAGE}... "
  emerge -uN --quiet-build --quiet=y --buildpkgonly ${PACKAGE};
  if [[ $? -eq 0 ]];
  then
    echo "ok";
  else
    echo "failed";
  fi
done

That isn't optimally efficient, but works reasonably well.

The result is that the next morning I have an email containing a list
of the stuff to be merged, and a set of binary packages for most of it
(deps of modified packages cannot be pre-built in this way of course).
Then I just run an "emerge -uDNkv" world to install all of it, often
in minutes.

The script could be optimized - if libreoffice and chromium are on the
list and I don't install them for a few days, suffice it to say that
the heater won't be running as much those nights.

-- 
Rich

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Dale]

Rich Freeman wrote:
> On Mon, Aug 10, 2015 at 12:10 PM, Dale <rdalek1967@gmail.com> wrote:
>> Peter Humphrey wrote:
>>> On Monday 10 August 2015 09:13:01 Dale wrote:
>>>
>>>> I might add, sync is taking a LONG time again.  Of course, my DSL is a
>>>> bit slower than some folks.  At least the bumpy road got smoothed out
>>>> tho.  It seems to be working again.   Maybe next sync will be back to
>>>> normal.
>>> I do one sync a day, of my LAN server, and sync anything else to that. I don't
>>> know how long it took today because I have it on a cron job at night.   :-)
>>>
>>
>> I usually sync about twice a week, sometimes three if I see something I
>> want or need.  I just run my desktop here so updating isn't a huge
>> deal.  Sometimes I forget because I'm busy with other things and I might
>> go a week or more without a single sync up.
>>
>> When I used to run several rigs, I'd sync one rig and then sync the
>> others to my main rig.  I try not to sync to often.
> I couldn't tell you who I ripped this off of but my cron routine is:
>
> ionice -n 7 nice -n 20 emerge --sync              (alternatively
> emerge-webrsync -k - preferred actually if you're using rsync)
> ionice -n 7 nice -n 20 layman -S
> emerge -puDv --changed-use world | col -bx | mutt -s "world update"
> root@localhost
> eix-update
>
> Then I do this:
> #!/bin/sh
>
> LIST=$(mktemp);
>
> emerge -puD --changed-use --color=n --columns --quiet=y --with-bdeps=n
> world | awk '{print $2}' > ${LIST};
>
> for PACKAGE in $(cat ${LIST});
> do
>   printf "Building binary package for ${PACKAGE}... "
>   emerge -uN --quiet-build --quiet=y --buildpkgonly ${PACKAGE};
>   if [[ $? -eq 0 ]];
>   then
>     echo "ok";
>   else
>     echo "failed";
>   fi
> done
>
> That isn't optimally efficient, but works reasonably well.
>
> The result is that the next morning I have an email containing a list
> of the stuff to be merged, and a set of binary packages for most of it
> (deps of modified packages cannot be pre-built in this way of course).
> Then I just run an "emerge -uDNkv" world to install all of it, often
> in minutes.
>
> The script could be optimized - if libreoffice and chromium are on the
> list and I don't install them for a few days, suffice it to say that
> the heater won't be running as much those nights.
>

Well, I only have one rig right now.  May save this for when I build a
spare or something. 

Oh, I also ran across a link to this bug:

https://bugs.gentoo.org/show_bug.cgi?id=557192 

It seems there is still a few kinks to work out.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Rich Freeman]

On Sun, Aug 16, 2015 at 10:58 AM, Dale <rdalek1967@gmail.com> wrote:
> Rich Freeman wrote:
>> The result is that the next morning I have an email containing a list
>> of the stuff to be merged, and a set of binary packages for most of it
>> (deps of modified packages cannot be pre-built in this way of course).
>> Then I just run an "emerge -uDNkv" world to install all of it, often
>> in minutes.
>>
>
> Well, I only have one rig right now.  May save this for when I build a
> spare or something.
>

I only run this on a single box.  I don't use the binary packages
across multiple Gentoo installations.  They're just there so that I
can do my builds at night, and then review/install them at my leisure.

-- 
Rich

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Dale]

Rich Freeman wrote:
> On Sun, Aug 16, 2015 at 10:58 AM, Dale <rdalek1967@gmail.com> wrote:
>> Rich Freeman wrote:
>>> The result is that the next morning I have an email containing a list
>>> of the stuff to be merged, and a set of binary packages for most of it
>>> (deps of modified packages cannot be pre-built in this way of course).
>>> Then I just run an "emerge -uDNkv" world to install all of it, often
>>> in minutes.
>>>
>> Well, I only have one rig right now.  May save this for when I build a
>> spare or something.
>>
> I only run this on a single box.  I don't use the binary packages
> across multiple Gentoo installations.  They're just there so that I
> can do my builds at night, and then review/install them at my leisure.
>


I generally end up updating twice a week.  I generally wait to sync when
I see something updated that interests me.  I just keep a eye on it
until something gets changed.  Heck, on those slow times, I may not
update but once a week.  I don't think I've ever set it up as a cron job
or anything.  Be my luck, I'd sync and 20 minutes later, something I
want to update will pop up and I just missed it.  lol 

Anyway, looking forward to them getting this long sync time back to
normal.  I'm sure someone is back there with a hammer beating some sense
into it.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Fernando Rodriguez]

On Sunday, August 16, 2015 9:58:43 AM Dale wrote:
> Rich Freeman wrote:
> > On Mon, Aug 10, 2015 at 12:10 PM, Dale <rdalek1967@gmail.com> wrote:
> >> Peter Humphrey wrote:
> >>> On Monday 10 August 2015 09:13:01 Dale wrote:
> >>>
> >>>> I might add, sync is taking a LONG time again.  Of course, my DSL is a
> >>>> bit slower than some folks.  At least the bumpy road got smoothed out
> >>>> tho.  It seems to be working again.   Maybe next sync will be back to
> >>>> normal.
> >>> I do one sync a day, of my LAN server, and sync anything else to that. I 
don't
> >>> know how long it took today because I have it on a cron job at night.   
:-)
> >>>
> >>
> >> I usually sync about twice a week, sometimes three if I see something I
> >> want or need.  I just run my desktop here so updating isn't a huge
> >> deal.  Sometimes I forget because I'm busy with other things and I might
> >> go a week or more without a single sync up.
> >>
> >> When I used to run several rigs, I'd sync one rig and then sync the
> >> others to my main rig.  I try not to sync to often.
> > I couldn't tell you who I ripped this off of but my cron routine is:
> >
> > ionice -n 7 nice -n 20 emerge --sync              (alternatively
> > emerge-webrsync -k - preferred actually if you're using rsync)
> > ionice -n 7 nice -n 20 layman -S
> > emerge -puDv --changed-use world | col -bx | mutt -s "world update"
> > root@localhost
> > eix-update
> >
> > Then I do this:
> > #!/bin/sh
> >
> > LIST=$(mktemp);
> >
> > emerge -puD --changed-use --color=n --columns --quiet=y --with-bdeps=n
> > world | awk '{print $2}' > ${LIST};
> >
> > for PACKAGE in $(cat ${LIST});
> > do
> >   printf "Building binary package for ${PACKAGE}... "
> >   emerge -uN --quiet-build --quiet=y --buildpkgonly ${PACKAGE};
> >   if [[ $? -eq 0 ]];
> >   then
> >     echo "ok";
> >   else
> >     echo "failed";
> >   fi
> > done
> >
> > That isn't optimally efficient, but works reasonably well.
> >
> > The result is that the next morning I have an email containing a list
> > of the stuff to be merged, and a set of binary packages for most of it
> > (deps of modified packages cannot be pre-built in this way of course).
> > Then I just run an "emerge -uDNkv" world to install all of it, often
> > in minutes.
> >
> > The script could be optimized - if libreoffice and chromium are on the
> > list and I don't install them for a few days, suffice it to say that
> > the heater won't be running as much those nights.
> >
> 
> Well, I only have one rig right now.  May save this for when I build a
> spare or something. 
> 
> Oh, I also ran across a link to this bug:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=557192 
> 
> It seems there is still a few kinks to work out.  ;-)

It finally seems to be fixed. I synced twice in a row and the 2nd time
it only synced 3 files.
 
> Dale
> 
> :-)  :-) 
> 

-- 
Fernando Rodriguez

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Mickaël Bucas]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]

2015-08-10 2:15 GMT+02:00 Dale <rdalek1967@gmail.com>:
>
> Howdy,
>
> I'm sure I'm not alone on monitoring -dev to see upcoming changes.  I
> noticed they switched to git or something and it seems to have caused a
> bit of trouble.  At least I think it may have.  I did my usual 'eix-sync
> && emerge -uvaDN world'.  The sync took MUCH longer than usual.  I'm
> talking a WHOLE LOT longer than usual.  My first thought, one time thing
> because of the changes, maybe.  Then I got a screen full of this sort of
> stuff.
>
>
>
>  * Missing digest for
>
'/var/cache/portage/tree/media-libs/fontconfig/fontconfig-2.11.1-r2.ebuild'
[...]

I got a similar list about all my installed packages. The reason is that
the Manifest files have changed.
Before there was a digest for each ebuild in a category, as can be seen
here :
https://github.com/gentoo/gentoo-portage-rsync-mirror/blob/master/kde-base/knotes/Manifest
Now there is only the digests for source files
https://gitweb.gentoo.org/repo/gentoo.git/tree/kde-base/knotes/Manifest

>
> Can I take that "just saying" above back and say FUBAR instead?   :-(
>
> Thoughts??
>
> Dale

Either the Manifest generator encountered a problem, or the digest for
ebuilds is not necessary in new versions of portage, but the latter is less
likely in my opinion as it would lower overall security.
I didn't find a bug about this subject, but I don't really know what and
where to search...

Who could we ask for more insight into that strange behavior ?

Best regards
Mickaël Bucas

[-- Attachment #2: Type: text/html, Size: 2145 bytes --]

Re: [gentoo-user] Missing digest for *** Tree looks messed up.

[Mickaël Bucas]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]

2015-08-10 10:55 GMT+02:00 Mickaël Bucas <mbucas@gmail.com>:
>
>
> 2015-08-10 2:15 GMT+02:00 Dale <rdalek1967@gmail.com>:
> >
> > Howdy,
> >
> > I'm sure I'm not alone on monitoring -dev to see upcoming changes.  I
> > noticed they switched to git or something and it seems to have caused a
> > bit of trouble.  At least I think it may have.  I did my usual 'eix-sync
> > && emerge -uvaDN world'.  The sync took MUCH longer than usual.  I'm
> > talking a WHOLE LOT longer than usual.  My first thought, one time thing
> > because of the changes, maybe.  Then I got a screen full of this sort of
> > stuff.
> >
> >
> >
> >  * Missing digest for
> >
'/var/cache/portage/tree/media-libs/fontconfig/fontconfig-2.11.1-r2.ebuild'
> [...]
>
> I got a similar list about all my installed packages. The reason is that
the Manifest files have changed.
> Before there was a digest for each ebuild in a category, as can be seen
here :
>
https://github.com/gentoo/gentoo-portage-rsync-mirror/blob/master/kde-base/knotes/Manifest
> Now there is only the digests for source files
> https://gitweb.gentoo.org/repo/gentoo.git/tree/kde-base/knotes/Manifest
>
> >
> > Can I take that "just saying" above back and say FUBAR instead?   :-(
> >
> > Thoughts??
> >
> > Dale
>
> Either the Manifest generator encountered a problem, or the digest for
ebuilds is not necessary in new versions of portage, but the latter is less
likely in my opinion as it would lower overall security.
> I didn't find a bug about this subject, but I don't really know what and
where to search...

So the answer was in Git history, in the first commit :
https://gitweb.gentoo.org/repo/gentoo.git/log/kde-base/knotes/Manifest?showmsg=1

"3. Transform all Manifests to thin"

https://wiki.gentoo.org/wiki/Repository_format/package/Manifest
"A Thin Manifest is a Manifest file in which checksums are stored only for
distfiles (*DIST* type) and not for files inside the repository. The
motivation for that is whenever the repository is fetched through a VCS
which ensures local file integrity already."

I think I understand the problem a bit better : the CVS repository
contained classic Manifest files, and was exposed directly to rsync. The
Git repository contains only thin Manifest, but has been set as the source
for rsync at one point !

This has already been reported, and solved.
https://bugs.gentoo.org/show_bug.cgi?id=557184

Best regards
Mickaël Bucas

[-- Attachment #2: Type: text/html, Size: 3424 bytes --]