From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 26C2D1391DB for ; Sat, 15 Mar 2014 20:32:23 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id B3313E0A70; Sat, 15 Mar 2014 20:32:21 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 0A91FE0A6E for ; Sat, 15 Mar 2014 20:32:20 +0000 (UTC) Received: from big_daddy.dol-sen.ca (S010600222de111ff.vc.shawcable.net [96.49.5.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: dolsen) by smtp.gentoo.org (Postfix) with ESMTPSA id 0395033F9AF for ; Sat, 15 Mar 2014 20:32:19 +0000 (UTC) Date: Sat, 15 Mar 2014 13:32:10 -0700 From: Brian Dolbec To: gentoo-portage-dev@lists.gentoo.org Subject: [gentoo-portage-dev] plugin-sync progress report Message-ID: <20140315133210.32f79f55.dolsen@gentoo.org> Organization: Gentoo Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Archives-Salt: 4256c0f6-9839-48ca-8cb7-79cf8f3fb823 X-Archives-Hash: 5fba25653c51a6330210523539562902 I've started working on the repository/config.py changes needed for the plugin-sync system. First: The following class performed checks on the repos.conf entries for the sync variables regardless if they were being used or not. I would like to remove that code block completely from this loader/parser. I would prefer to have the sync module perform it's own checks and only when syncing or from a future emaint module with that specific 'check-config' option. I've sectioned off the code I'd like to remove below. Is there any reason they must be done here and during initial parsing? Advantages: - No need to import and initialize the sync module and the plugin system for other emerge/portage operations. - Should speed things up slightly with less to do. - Makes the sync module responsible the validation of any sync specific variables. Second: - Should all the repos.conf entires to be synced be validated prior to syncing and fail if there are any errors? - Or, just call each sync module's sync() and let each fail individually and continue syncing remaining repos? Third: - I would like to add a new config item for all repos. It is a variable to define which sources of sync operations which the repo will be allowed to be synced from. It could be a space separated list which could be used to prevent it from being synced via certain commands. i.e: [repo1] sync-from = emerge emaint [repo2] sync-from = emerge emaint layman [repo3] sync-from = None - Thoughts? - Opinions? ############ class RepoConfigLoader(object): """Loads and store config of several repositories, loaded from PORTDIR_OVERLAY or repos.conf""" @staticmethod def _parse(paths, prepos, ignored_map, ignored_location_map, local_config, portdir): """Parse files in paths to load config""" parser = SafeConfigParser() # use read_file/readfp in order to control decoding of unicode try: # Python >=3.2 read_file = parser.read_file source_kwarg = 'source' except AttributeError: read_file = parser.readfp source_kwarg = 'filename' recursive_paths = [] for p in paths: if isinstance(p, basestring): recursive_paths.extend(_recursive_file_list(p)) else: recursive_paths.append(p) for p in recursive_paths: if isinstance(p, basestring): f = None try: f = io.open(_unicode_encode(p, encoding=_encodings['fs'], errors='strict'), mode='r', encoding=_encodings['repo.content'], errors='replace') except EnvironmentError: pass else: # The 'source' keyword argument is needed since otherwise # ConfigParser in Python <3.3.3 may throw a TypeError # because it assumes that f.name is a native string rather # than binary when constructing error messages. kwargs = {source_kwarg: p} read_file(f, **portage._native_kwargs(kwargs)) finally: if f is not None: f.close() elif isinstance(p, io.StringIO): kwargs = {source_kwarg: ""} read_file(p, **portage._native_kwargs(kwargs)) else: raise TypeError("Unsupported type %r of element %r of 'paths' argument" % (type(p), p)) prepos['DEFAULT'] = RepoConfig("DEFAULT", parser.defaults(), local_config=local_config) for sname in parser.sections(): optdict = {} for oname in parser.options(sname): optdict[oname] = parser.get(sname, oname) repo = RepoConfig(sname, optdict, local_config=local_config) ################################### # remove these checks ################################### if repo.sync_type is not None and repo.sync_uri is None: writemsg_level("!!! %s\n" % _("Repository '%s' has sync-type attribute, but is missing sync-uri attribute") % sname, level=logging.ERROR, noiselevel=-1) continue if repo.sync_uri is not None and repo.sync_type is None: writemsg_level("!!! %s\n" % _("Repository '%s' has sync-uri attribute, but is missing sync-type attribute") % sname, level=logging.ERROR, noiselevel=-1) continue if repo.sync_type not in portage.sync.module_names + [None]: writemsg_level("!!! %s\n" % _("Repository '%s' has sync-type attribute set to unsupported value: '%s'") % (sname, repo.sync_type), level=logging.ERROR, noiselevel=-1) continue if repo.sync_type == "cvs" and repo.sync_cvs_repo is None: writemsg_level("!!! %s\n" % _("Repository '%s' has sync-type=cvs, but is missing sync-cvs-repo attribute") % sname, level=logging.ERROR, noiselevel=-1) continue ################################### # end remove ################################### # For backward compatibility with locations set via PORTDIR and # PORTDIR_OVERLAY, delay validation of the location and repo.name # until after PORTDIR and PORTDIR_OVERLAY have been processed. prepos[sname] = repo -- Brian Dolbec