From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 30F3E1381F3 for ; Tue, 23 Jul 2013 23:13:54 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 7D30DE0983; Tue, 23 Jul 2013 23:13:53 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 19617E0983 for ; Tue, 23 Jul 2013 23:13:52 +0000 (UTC) Received: from hornbill.gentoo.org (hornbill.gentoo.org [94.100.119.163]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 14F0933E9AE for ; Tue, 23 Jul 2013 23:13:52 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by hornbill.gentoo.org (Postfix) with ESMTP id 42852E5466 for ; Tue, 23 Jul 2013 23:13:49 +0000 (UTC) From: "Jauhien Piatlicki" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Jauhien Piatlicki" Message-ID: <1374621242.65e4510b7409a122a8744f1ced20286ca9ca0a3c.jauhien@gentoo> Subject: [gentoo-commits] proj/g-sorcery:master commit in: /, gs_pypi/ X-VCS-Repository: proj/g-sorcery X-VCS-Files: gs-pypi.json gs_pypi/pypi_db.py X-VCS-Directories: / gs_pypi/ X-VCS-Committer: jauhien X-VCS-Committer-Name: Jauhien Piatlicki X-VCS-Revision: 65e4510b7409a122a8744f1ced20286ca9ca0a3c X-VCS-Branch: master Date: Tue, 23 Jul 2013 23:13:49 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: 394ba9c3-1a97-457f-b939-95aa930d2975 X-Archives-Hash: 608303af6b3f03f2ce59e47d6ce06c13 commit: 65e4510b7409a122a8744f1ced20286ca9ca0a3c Author: Jauhien Piatlicki (jauhien) gmail com> AuthorDate: Tue Jul 23 23:14:02 2013 +0000 Commit: Jauhien Piatlicki gmail com> CommitDate: Tue Jul 23 23:14:02 2013 +0000 URL: http://git.overlays.gentoo.org/gitweb/?p=proj/g-sorcery.git;a=commit;h=65e4510b PyPI backend: experiments --- gs-pypi.json | 2 +- gs_pypi/pypi_db.py | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 55 insertions(+), 2 deletions(-) diff --git a/gs-pypi.json b/gs-pypi.json index eb26221..06a23a4 100644 --- a/gs-pypi.json +++ b/gs-pypi.json @@ -2,7 +2,7 @@ "package": "gs_pypi", "repositories": { "pypi": { - "repo_uri": "https://pypi.python.org/pypi" + "repo_uri": "http://pypi.python.org/pypi" } } } diff --git a/gs_pypi/pypi_db.py b/gs_pypi/pypi_db.py index 938f79c..71e5397 100644 --- a/gs_pypi/pypi_db.py +++ b/gs_pypi/pypi_db.py @@ -11,7 +11,60 @@ :license: GPL-2, see LICENSE for more details. """ +from g_sorcery.compatibility import py2k + +if py2k: + import xmlrpclib + import httplib + urlparse +else: + import xmlrpc.client as xmlrpclib + import http.client as httplib + import urllib.parse as urlparse + +import sys + +from g_sorcery.logger import Logger from g_sorcery.package_db import DBGenerator class PypiDBGenerator(DBGenerator): - pass + + def process_uri(self, uri, data): + url = uri["uri"] + client = xmlrpclib.ServerProxy(url) + logger = Logger() + logger.info("downloading packages data") + pkg_list = client.list_packages() + + number_of_packages = len(pkg_list) + downloaded_number = 0 + + connection = httplib.HTTPConnection(urlparse.urlparse(url).netloc) + + connection.request("GET", "/pypi/zmqpy/json") + response = connection.getresponse() + print(response.getheaders()) + + for pkg in pkg_list: + data[pkg] = {} + + chars = ['-','\\','|','/'] + show = chars[downloaded_number % 4] + percent = (downloaded_number * 100)//number_of_packages + length = 70 + progress = (percent * length)//100 + blank = length - progress + + sys.stdout.write("\r %s [%s%s] %s%%" % (show, "#" * progress, " " * blank, percent)) + sys.stdout.flush() + downloaded_number += 1 + + connection.close() + + sys.stdout.write("\r %s [%s] %s%%" % ("-", "#" * length, 100)) + sys.stdout.flush() + print("") + + def process_data(self, pkg_db, data, common_config, config): + #print(data) + pass