From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 028D11381F3 for ; Thu, 3 Oct 2013 18:58:40 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id CA0B5E0A60; Thu, 3 Oct 2013 18:58:33 +0000 (UTC) Received: from mail-we0-f170.google.com (mail-we0-f170.google.com [74.125.82.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 8B819E07EC for ; Thu, 3 Oct 2013 18:58:32 +0000 (UTC) Received: by mail-we0-f170.google.com with SMTP id u57so2060682wes.1 for ; Thu, 03 Oct 2013 11:58:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=MCREzN3XUOvTznzC4sDEMZUKAw14SBXOgQYTMdn9hq8=; b=Lye1/fE9HM4mI8gcq1oTdt3iXICjqQEdFqyJHT/brUeorzfsVz9E28K96af+ef77ni ToBNcvCfoM75mHMgyTuZtRJ95hcEbqSS73ZBEBXdypyrSaeBMGgDGypUpuuLRhi+yCBH HsQvtZdZ7HcqzMaWwsWYmLkqXbD8JcXkLx4xAtDF9FIOrYjvrWl52NhlXz4izDQxMlI3 4Mp9VatQK5ce4+s6IGm0BULafRcb2OlR82jVHVzSdAUeN2EhkMzAX3o6V/jKANmiAjL2 Yvfu16K4ASvbBRbG62M/aR+6jHvRfB9b2UZq1WDlmypKvueoUpi9LV5J6/VO/C98NOLn tFnQ== X-Received: by 10.194.10.193 with SMTP id k1mr3325343wjb.50.1380826711218; Thu, 03 Oct 2013 11:58:31 -0700 (PDT) Received: from [172.20.0.40] (196-210-102-121.dynamic.isadsl.co.za. [196.210.102.121]) by mx.google.com with ESMTPSA id b13sm104711wic.9.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 03 Oct 2013 11:58:30 -0700 (PDT) Message-ID: <524DBD40.8080705@gmail.com> Date: Thu, 03 Oct 2013 20:53:52 +0200 From: Alan McKinnon User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] emerge --sync issue on only one comp on LAN References: <20131001142006.GB23408@server> <524AEE09.5050707@gmail.com> <524C9566.60608@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Archives-Salt: bc2736fd-2e2b-4823-97fc-c2a5930dc7a9 X-Archives-Hash: 6bd84d6a97f8e489df58e66e64295308 On 03/10/2013 18:57, Paul Hartman wrote: > On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon wrote: >> On 02/10/2013 19:37, Paul Hartman wrote: >>> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon wrote: >>>> On 01/10/2013 17:17, Greg Turner wrote: [snip] >> You don't sync very often, right? > > I usually sync manually daily or every other day if I'm busy and don't > get a chance. I assumed there was some mass change to ebuild headers > or license text or something which caused everything in the tree to > get touched this week. > > My local portage tree is on a fast SSD in an 8-core box with 32GB of > RAM and a 100mbit internet connection, so the bottleneck hopefully is > not on my side of the transaction. ;) > > Let's do some more trials. Between yesterday and today, I have synced > with my normal mirror, but I'm syncing with your server again now: > > Number of files: 174410 > Number of files transferred: 17372 > Total file size: 306.28M bytes > Total transferred file size: 22.32M bytes > Literal data: 22.32M bytes > Matched data: 0 bytes > File list size: 4.31M > File list generation time: 379.920 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 382.35K > Total bytes received: 15.71M > > sent 382.35K bytes received 15.71M bytes 29.33K bytes/sec > total size is 306.28M speedup is 19.04 > > Now I'm immediately doing another sync, first deleting timestamp.chk > to force it to sync again. There should be zero files to transfer > (except the timestamp file). > > Number of files: 174410 > Number of files transferred: 1 > Total file size: 306.28M bytes > Total transferred file size: 32 bytes > Literal data: 32 bytes > Matched data: 0 bytes > File list size: 4.31M > File list generation time: 28.612 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 183 > Total bytes received: 4.31M > > sent 183 bytes received 4.31M bytes 128.75K bytes/sec > total size is 306.28M speedup is 71.01 > > > Now I'm switching back to my beloved mirror.steadfast.net and running > another sync. > > Number of files: 174409 > Number of files transferred: 17364 > Total file size: 306.30M bytes > Total transferred file size: 21.74M bytes > Literal data: 21.74M bytes > Matched data: 0 bytes > File list size: 4.39M > File list generation time: 0.001 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 355.23K > Total bytes received: 15.67M > > sent 355.23K bytes received 15.67M bytes 191.93K bytes/sec > total size is 306.30M speedup is 19.11 > > > Interestingly it transferred almost the same number of files as my > first sync with yours. Comparing timestamps, your server's latest > update is about 5 hours older than Steadfast's, so things must be > changing frequently in portage these days! 17k changes in 5 hours... My guess is the metadata. I'll have to do some checks on that mirror, IIRC it syncs every 6 hours, most likely steadfast syncs more often. One of the things about running a mirror is, it's very much set it up once and forget all about it evermore. Which is great and all, but users tend to spot problems long before the sysadmins do :-) > > My ping to your server is 300ms, my ping to steadfast is 18ms. I don't > know anything about how rsync works behind the curtain, if a higher > latency would cause the file list generation to be slower, or if that > is a measurement of server performance or something else. I don't think latency is much of a factor but let me re-read some FAQs before commenting further. 300ms is totally normal from here to eu, uk and us - we're in deepest darkest Africa where hyenas prowl the streets[1] - and all traffic goes over undersea cable with *lots* of repeaters > > Total sync times from my log: > > 1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage > 1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage > (first sync, 17k files updated, 552 seconds) > > 1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage > 1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage > (sync with no updates except timestamp.chk, 38 seconds) > > 1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage > 1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage > (re-sync with steadfast, 17k files updated, 83 seconds) > > 1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage > 1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage > (sync with no updates except timestamp.chk, 12 seconds) > > > HTH and thanks for the mirror :) > Paul > [1] Literally. True's bob, I kid you not. A 6 month old brown hyena this week wandered into the suburb where I live - it must have got separated from it's mother and walked 15 miles in the dark to get here -- Alan McKinnon alan.mckinnon@gmail.com