From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id B1EE61395E2 for ; Wed, 7 Dec 2016 02:54:00 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E5634E0D88; Wed, 7 Dec 2016 02:53:39 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id C5820E0D88 for ; Wed, 7 Dec 2016 02:53:29 +0000 (UTC) Received: from oystercatcher.gentoo.org (unknown [IPv6:2a01:4f8:202:4333:225:90ff:fed9:fc84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id EEFD53412CB for ; Wed, 7 Dec 2016 02:53:28 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id 509A4300 for ; Wed, 7 Dec 2016 02:53:27 +0000 (UTC) From: "Mart Raudsepp" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Mart Raudsepp" Message-ID: <1481079168.0522c4ccf0f4ca737572b8164cde6bb9c498ba7f.leio@gentoo> Subject: [gentoo-commits] proj/grumpy:master commit in: backend/lib/ X-VCS-Repository: proj/grumpy X-VCS-Files: backend/lib/sync.py X-VCS-Directories: backend/lib/ X-VCS-Committer: leio X-VCS-Committer-Name: Mart Raudsepp X-VCS-Revision: 0522c4ccf0f4ca737572b8164cde6bb9c498ba7f X-VCS-Branch: master Date: Wed, 7 Dec 2016 02:53:27 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: 7347d0a4-5b33-4515-ab19-c443ba2af04f X-Archives-Hash: 11f92d51c04a857503b5295010331ad1 commit: 0522c4ccf0f4ca737572b8164cde6bb9c498ba7f Author: Mart Raudsepp gentoo org> AuthorDate: Wed Dec 7 02:52:48 2016 +0000 Commit: Mart Raudsepp gentoo org> CommitDate: Wed Dec 7 02:52:48 2016 +0000 URL: https://gitweb.gentoo.org/proj/grumpy.git/commit/?id=0522c4cc sync: Increase the sync delta to 1 hour and print the sync count and oldest TS at start backend/lib/sync.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/backend/lib/sync.py b/backend/lib/sync.py index 8c687c6..7ba583d 100644 --- a/backend/lib/sync.py +++ b/backend/lib/sync.py @@ -5,7 +5,7 @@ from datetime import datetime from .. import app, db from .models import Category, Maintainer, Package, PackageVersion -SYNC_BUFFER_SECS = 30*60 +SYNC_BUFFER_SECS = 60*60 #1 hour proj_url = "https://api.gentoo.org/metastructure/projects.xml" pkg_url_base = "https://packages.gentoo.org/" http_session = requests.session() @@ -154,7 +154,10 @@ def sync_versions(): for maintainer in Maintainer.query.all(): existing_maintainers[maintainer.email] = maintainer - for package in Package.query.filter(Package.last_sync_ts < ts).order_by(Package.last_sync_ts).all(): + packages_to_sync = Package.query.filter(Package.last_sync_ts < ts).order_by(Package.last_sync_ts).all() + print("Going to sync %d packages%s" % (len(packages_to_sync), (" (oldest sync UTC timestamp: %s)" % packages_to_sync[0].last_sync_ts if len(packages_to_sync) else ""))) + + for package in packages_to_sync: cnt += 1 data = http_session.get(pkg_url_base + "packages/" + package.full_name + ".json") if not data: