* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2016-05-15 2:37 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2016-05-15 2:37 UTC (permalink / raw
To: gentoo-commits
commit: 642ac6486f6f0a22f9100c7318c4ba7e5b427bbb
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sun May 15 02:23:46 2016 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sun May 15 02:23:46 2016 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=642ac648
repoman/metadata.py: Add the local test path to metadata.xsd search
repoman/pym/repoman/metadata.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/repoman/pym/repoman/metadata.py b/repoman/pym/repoman/metadata.py
index a9ad3e8..f162b06 100644
--- a/repoman/pym/repoman/metadata.py
+++ b/repoman/pym/repoman/metadata.py
@@ -108,7 +108,10 @@ def get_metadata_xsd(repo_settings):
@returns: path to the metadata.xsd file
'''
metadata_xsd = None
- for path in reversed(repo_settings.repo_config.eclass_db.porttrees):
+ paths = reversed(repo_settings.repo_config.eclass_db.porttrees)
+ # add the test copy
+ paths.append("/usr/lib/portage/cnf/")
+ for path in paths:
path = os.path.join(path, 'metadata/xml-schema/metadata.xsd')
if os.path.exists(path):
metadata_xsd = path
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2016-05-15 4:13 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2016-05-15 4:13 UTC (permalink / raw
To: gentoo-commits
commit: ca7d515a91325e4c177e7169f86c76209ceb7891
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sun May 15 02:50:21 2016 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sun May 15 02:50:21 2016 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=ca7d515a
Fix reversed issue
repoman/pym/repoman/metadata.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/repoman/pym/repoman/metadata.py b/repoman/pym/repoman/metadata.py
index f162b06..77174f6 100644
--- a/repoman/pym/repoman/metadata.py
+++ b/repoman/pym/repoman/metadata.py
@@ -108,7 +108,8 @@ def get_metadata_xsd(repo_settings):
@returns: path to the metadata.xsd file
'''
metadata_xsd = None
- paths = reversed(repo_settings.repo_config.eclass_db.porttrees)
+ paths = list(repo_settings.repo_config.eclass_db.porttrees)
+ paths.reverse()
# add the test copy
paths.append("/usr/lib/portage/cnf/")
for path in paths:
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2016-05-15 8:40 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2016-05-15 8:40 UTC (permalink / raw
To: gentoo-commits
commit: a5b42d7c0846e25c880d92824186d5a556e4b116
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sun May 15 02:23:46 2016 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sun May 15 08:39:02 2016 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=a5b42d7c
repoman/metadata.py: Add the local test path to metadata.xsd search
repoman/pym/repoman/metadata.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/repoman/pym/repoman/metadata.py b/repoman/pym/repoman/metadata.py
index a9ad3e8..77174f6 100644
--- a/repoman/pym/repoman/metadata.py
+++ b/repoman/pym/repoman/metadata.py
@@ -108,7 +108,11 @@ def get_metadata_xsd(repo_settings):
@returns: path to the metadata.xsd file
'''
metadata_xsd = None
- for path in reversed(repo_settings.repo_config.eclass_db.porttrees):
+ paths = list(repo_settings.repo_config.eclass_db.porttrees)
+ paths.reverse()
+ # add the test copy
+ paths.append("/usr/lib/portage/cnf/")
+ for path in paths:
path = os.path.join(path, 'metadata/xml-schema/metadata.xsd')
if os.path.exists(path):
metadata_xsd = path
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
2016-05-15 23:51 [gentoo-commits] proj/portage:master " Brian Dolbec
@ 2016-05-15 18:34 ` Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2016-05-15 18:34 UTC (permalink / raw
To: gentoo-commits
commit: 2c45b9b37355ee3684a4a6cf732e08b16a5398cb
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sun May 15 02:23:46 2016 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sun May 15 18:27:18 2016 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=2c45b9b3
repoman/metadata.py: Add the local test path to metadata.xsd search
This fixes an issue in the travis testing where it tries to download
the metadata.xsd which fails.
repoman/pym/repoman/metadata.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/repoman/pym/repoman/metadata.py b/repoman/pym/repoman/metadata.py
index a9ad3e8..77174f6 100644
--- a/repoman/pym/repoman/metadata.py
+++ b/repoman/pym/repoman/metadata.py
@@ -108,7 +108,11 @@ def get_metadata_xsd(repo_settings):
@returns: path to the metadata.xsd file
'''
metadata_xsd = None
- for path in reversed(repo_settings.repo_config.eclass_db.porttrees):
+ paths = list(repo_settings.repo_config.eclass_db.porttrees)
+ paths.reverse()
+ # add the test copy
+ paths.append("/usr/lib/portage/cnf/")
+ for path in paths:
path = os.path.join(path, 'metadata/xml-schema/metadata.xsd')
if os.path.exists(path):
metadata_xsd = path
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 20:06 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 20:06 UTC (permalink / raw
To: gentoo-commits
commit: b7deb4dec0a6c3516050a87179aee37b2a6b70c8
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 18:08:50 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 19:51:21 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=b7deb4de
repoman: main.py: Remove unused InvalidStateError import
repoman/pym/repoman/main.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100644
new mode 100755
index 3b628de00..c1e3b99fe
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -24,7 +24,6 @@ from portage.output import ConsoleStyleFile, StyleWriter
from portage.util import formatter
from portage.util.futures.extendedfutures import (
ExtendedFuture,
- InvalidStateError,
)
from repoman.actions import Actions
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 20:06 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 20:06 UTC (permalink / raw
To: gentoo-commits
commit: da2a8745c7870c57c2b4025238c0b48acab09c5d
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:49:01 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 19:51:21 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=da2a8745
repoman: Move qa_data initialization and loading
Move the new QAData class instance init to repoman_main().
parse_args() remove unused qahelp parameter.
qa_tracker.py: Add default qacats and qawarnings parameters as None.
These will be assigned later due to circular init references.
repos.py: Perform the QAData class loading and complete intialization
assignments.
repoman/pym/repoman/argparser.py | 3 +--
repoman/pym/repoman/main.py | 26 ++++++++++++++------------
repoman/pym/repoman/qa_tracker.py | 10 +++++-----
repoman/pym/repoman/repos.py | 12 ++++++++++--
4 files changed, 30 insertions(+), 21 deletions(-)
diff --git a/repoman/pym/repoman/argparser.py b/repoman/pym/repoman/argparser.py
index 2d56a87e6..68701378f 100644
--- a/repoman/pym/repoman/argparser.py
+++ b/repoman/pym/repoman/argparser.py
@@ -15,11 +15,10 @@ from portage import _unicode_decode
from portage import util
-def parse_args(argv, qahelp, repoman_default_opts):
+def parse_args(argv, repoman_default_opts):
"""Use a customized optionParser to parse command line arguments for repoman
Args:
argv - a sequence of command line arguments
- qahelp - a dict of qa warning to help message
Returns:
(opts, args), just like a call to parser.parse_args()
"""
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100755
new mode 100644
index ccc735c7d..3b628de00
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -29,9 +29,8 @@ from portage.util.futures.extendedfutures import (
from repoman.actions import Actions
from repoman.argparser import parse_args
-from repoman.qa_data import (
- format_qa_output, format_qa_output_column, qahelp,
- qawarnings, qacats)
+from repoman.qa_data import QAData
+from repoman.qa_data import format_qa_output, format_qa_output_column
from repoman.repos import RepoSettings
from repoman.scanner import Scanner
from repoman import utilities
@@ -60,7 +59,7 @@ def repoman_main(argv):
nocolor()
options, arguments = parse_args(
- sys.argv, qahelp, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
+ sys.argv, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
if options.version:
print("Repoman", VERSION, "(portage-%s)" % portage.VERSION)
@@ -73,10 +72,6 @@ def repoman_main(argv):
else:
logger.setLevel(LOGLEVEL)
- if options.experimental_inherit == 'y':
- # This is experimental, so it's non-fatal.
- qawarnings.add("inherit.missing")
-
# Set this to False when an extraordinary issue (generally
# something other than a QA issue) makes it impossible to
# commit (like if Manifest generation fails).
@@ -91,14 +86,21 @@ def repoman_main(argv):
# avoid a circular parameter repo_settings
vcs_settings = VCSSettings(options, repoman_settings)
+ qadata = QAData()
+ logging.debug("repoman_main: RepoSettings init")
repo_settings = RepoSettings(
config_root, portdir, portdir_overlay,
- repoman_settings, vcs_settings, options, qawarnings)
+ repoman_settings, vcs_settings, options, qadata)
repoman_settings = repo_settings.repoman_settings
# Now set repo_settings
vcs_settings.repo_settings = repo_settings
+ # set QATracker qacats, qawarnings
+ vcs_settings.qatracker.qacats = repo_settings.qadata.qacats
+ vcs_settings.qatracker.qawarnings = repo_settings.qadata.qawarnings
+ logging.debug("repoman_main: vcs_settings done")
+ logging.debug("repoman_main: qadata: %s", repo_settings.qadata)
if 'digest' in repoman_settings.features and options.digest != 'n':
options.digest = 'y'
@@ -133,11 +135,11 @@ def repoman_main(argv):
if options.mode == "manifest":
sys.exit(result['fail'])
- for x in qacats:
+ for x in qadata.qacats:
if x not in vcs_settings.qatracker.fails:
continue
result['warn'] = 1
- if x not in qawarnings:
+ if x not in qadata.qawarnings:
result['fail'] = 1
if result['fail'] or \
@@ -174,7 +176,7 @@ def repoman_main(argv):
format_output = format_outputs.get(
options.output_style, format_outputs['default'])
format_output(f, vcs_settings.qatracker.fails, result['full'],
- result['fail'], options, qawarnings)
+ result['fail'], options, qadata.qawarnings)
style_file.flush()
del console_writer, f, style_file
diff --git a/repoman/pym/repoman/qa_tracker.py b/repoman/pym/repoman/qa_tracker.py
index 9bfe0e241..faaf8e398 100644
--- a/repoman/pym/repoman/qa_tracker.py
+++ b/repoman/pym/repoman/qa_tracker.py
@@ -2,15 +2,15 @@
import logging
import sys
-from repoman.qa_data import qacats, qawarnings
-
class QATracker(object):
'''Track all occurrances of Q/A problems detected'''
- def __init__(self):
+ def __init__(self, qacats=None, qawarnings=None):
self.fails = {}
self.warns = {}
+ self.qacats = qacats
+ self.qawarnings = qawarnings
def add_error(self, detected_qa, info):
'''Add the Q/A error to the database of detected problems
@@ -18,7 +18,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qacats list
@param info: string, details of the detected problem
'''
- if detected_qa not in qacats:
+ if detected_qa not in self.qacats:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_error(): %s, %s' % (detected_qa, info))
@@ -34,7 +34,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qawarnings list
@param info: string, details of the detected problem
'''
- if detected_qa not in qawarnings:
+ if detected_qa not in self.qawarnings:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_warning(): %s, %s' % (detected_qa, info))
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index 39f53c180..cd917828d 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -27,7 +27,7 @@ class RepoSettings(object):
def __init__(
self, config_root, portdir, portdir_overlay,
repoman_settings=None, vcs_settings=None, options=None,
- qawarnings=None):
+ qadata=None):
self.config_root = config_root
self.repoman_settings = repoman_settings
self.vcs_settings = vcs_settings
@@ -41,6 +41,14 @@ class RepoSettings(object):
except KeyError:
self._add_repo(config_root, portdir_overlay)
+ logging.debug("RepoSettings: init(); load qadata")
+ # load the repo specific configuration
+ self.qadata = qadata
+ if not self.qadata.load_repo_config(self.repodir, options):
+ logging.error("Aborting...")
+ sys.exit(1)
+ logging.debug("RepoSettings: qadata loaded", qadata.no_exec)
+
self.root = self.repoman_settings['EROOT']
self.trees = {
self.root: {'porttree': portage.portagetree(settings=self.repoman_settings)}
@@ -60,7 +68,7 @@ class RepoSettings(object):
del self.repositories[repo.name]
if self.repo_config.allow_provide_virtual:
- qawarnings.add("virtual.oldstyle")
+ qadata.qawarnings.add("virtual.oldstyle")
if self.repo_config.sign_commit and options.mode in ("commit", "fix", "manifest"):
if vcs_settings.vcs:
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 20:06 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 20:06 UTC (permalink / raw
To: gentoo-commits
commit: a04ef3863d39f84d70bf72bc5ed6d24aa28f7e92
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:38:48 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 19:51:07 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=a04ef386
qa_data.py: Initial move of all configurable data to the repo
All this data was ported to a metadata/repoman/qa_data.yml file.
repoman/pym/repoman/qa_data.py | 432 ++++++++---------------------------------
1 file changed, 77 insertions(+), 355 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index a59fed778..2bbf460df 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -1,368 +1,90 @@
# -*- coding:utf-8 -*-
import logging
+import os
+import yaml
from _emerge.Package import Package
# import our initialized portage instance
from repoman._portage import portage
-max_desc_len = 80
-allowed_filename_chars = "a-zA-Z0-9._-+:"
-qahelp = {
- "CVS/Entries.IO_error": (
- "Attempting to commit, and an IO error was encountered access the"
- " Entries file"),
- "ebuild.invalidname": (
- "Ebuild files with a non-parseable or syntactically incorrect name"
- " (or using 2.1 versioning extensions)"),
- "ebuild.namenomatch": (
- "Ebuild files that do not have the same name as their parent"
- " directory"),
- "changelog.ebuildadded": (
- "An ebuild was added but the ChangeLog was not modified"),
- "changelog.missing": (
- "Missing ChangeLog files"),
- "ebuild.notadded": (
- "Ebuilds that exist but have not been added to cvs"),
- "ebuild.patches": (
- "PATCHES variable should be a bash array to ensure white space safety"),
- "changelog.notadded": (
- "ChangeLogs that exist but have not been added to cvs"),
- "dependency.bad": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds)"),
- "dependency.badmasked": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds)"),
- "dependency.badindev": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds) in developing arch"),
- "dependency.badmaskedindev": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds) in developing arch"),
- "dependency.badtilde": (
- "Uses the ~ dep operator with a non-zero revision part,"
- " which is useless (the revision is ignored)"),
- "dependency.missingslot": (
- "RDEPEND matches more than one SLOT but does not specify a "
- "slot and/or use the := or :* slot operator"),
- "dependency.perlcore": (
- "This ebuild directly depends on a package in perl-core;"
- " it should use the corresponding virtual instead."),
- "dependency.syntax": (
- "Syntax error in dependency string"
- " (usually an extra/missing space/parenthesis)"),
- "dependency.unknown": (
- "Ebuild has a dependency that refers to an unknown package"
- " (which may be valid if it is a blocker for a renamed/removed package,"
- " or is an alternative choice provided by an overlay)"),
- "dependency.badslotop": (
- "RDEPEND contains ':=' slot operator under '||' dependency."),
- "file.executable": (
- "Ebuilds, digests, metadata.xml, Manifest, and ChangeLog do not need"
- " the executable bit"),
- "file.size": (
- "Files in the files directory must be under 20 KiB"),
- "file.size.fatal": (
- "Files in the files directory must be under 60 KiB"),
- "file.empty": (
- "Empty file in the files directory"),
- "file.name": (
- "File/dir name must be composed"
- " of only the following chars: %s " % allowed_filename_chars),
- "file.UTF8": (
- "File is not UTF8 compliant"),
- "inherit.deprecated": (
- "Ebuild inherits a deprecated eclass"),
- "inherit.missing": (
- "Ebuild uses functions from an eclass but does not inherit it"),
- "inherit.unused": (
- "Ebuild inherits an eclass but does not use it"),
- "java.eclassesnotused": (
- "With virtual/jdk in DEPEND you must inherit a java eclass"),
- "wxwidgets.eclassnotused": (
- "Ebuild DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass"),
- "KEYWORDS.dropped": (
- "Ebuilds that appear to have dropped KEYWORDS for some arch"),
- "KEYWORDS.missing": (
- "Ebuilds that have a missing or empty KEYWORDS variable"),
- "KEYWORDS.stable": (
- "Ebuilds that have been added directly with stable KEYWORDS"),
- "KEYWORDS.stupid": (
- "Ebuilds that use KEYWORDS=-* instead of package.mask"),
- "LICENSE.missing": (
- "Ebuilds that have a missing or empty LICENSE variable"),
- "LICENSE.virtual": (
- "Virtuals that have a non-empty LICENSE variable"),
- "DESCRIPTION.missing": (
- "Ebuilds that have a missing or empty DESCRIPTION variable"),
- "DESCRIPTION.toolong": (
- "DESCRIPTION is over %d characters" % max_desc_len),
- "EAPI.definition": (
- "EAPI definition does not conform to PMS section 7.3.1"
- " (first non-comment, non-blank line)"),
- "EAPI.deprecated": (
- "Ebuilds that use features that are deprecated in the current EAPI"),
- "EAPI.incompatible": (
- "Ebuilds that use features that are only available with a different"
- " EAPI"),
- "EAPI.unsupported": (
- "Ebuilds that have an unsupported EAPI version"
- " (you must upgrade portage)"),
- "SLOT.invalid": (
- "Ebuilds that have a missing or invalid SLOT variable value"),
- "HOMEPAGE.missing": (
- "Ebuilds that have a missing or empty HOMEPAGE variable"),
- "HOMEPAGE.virtual": (
- "Virtuals that have a non-empty HOMEPAGE variable"),
- "HOMEPAGE.missingurischeme": (
- "HOMEPAGE is missing an URI scheme"),
- "PDEPEND.suspect": (
- "PDEPEND contains a package that usually only belongs in DEPEND."),
- "LICENSE.syntax": (
- "Syntax error in LICENSE"
- " (usually an extra/missing space/parenthesis)"),
- "PROVIDE.syntax": (
- "Syntax error in PROVIDE"
- " (usually an extra/missing space/parenthesis)"),
- "PROPERTIES.syntax": (
- "Syntax error in PROPERTIES"
- " (usually an extra/missing space/parenthesis)"),
- "RESTRICT.syntax": (
- "Syntax error in RESTRICT"
- " (usually an extra/missing space/parenthesis)"),
- "REQUIRED_USE.syntax": (
- "Syntax error in REQUIRED_USE"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.syntax": (
- "Syntax error in SRC_URI"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.mirror": (
- "A uri listed in profiles/thirdpartymirrors is found in SRC_URI"),
- "ebuild.syntax": (
- "Error generating cache entry for ebuild;"
- " typically caused by ebuild syntax error"
- " or digest verification failure"),
- "ebuild.output": (
- "A simple sourcing of the ebuild produces output;"
- " this breaks ebuild policy."),
- "ebuild.nesteddie": (
- "Placing 'die' inside ( ) prints an error,"
- " but doesn't stop the ebuild."),
- "variable.invalidchar": (
- "A variable contains an invalid character"
- " that is not part of the ASCII character set"),
- "variable.readonly": (
- "Assigning a readonly variable"),
- "variable.usedwithhelpers": (
- "Ebuild uses D, ROOT, ED, EROOT or EPREFIX with helpers"),
- "LIVEVCS.stable": (
- "This ebuild is a live checkout from a VCS but has stable keywords."),
- "LIVEVCS.unmasked": (
- "This ebuild is a live checkout from a VCS but has keywords"
- " and is not masked in the global package.mask."),
- "IUSE.invalid": (
- "This ebuild has a variable in IUSE"
- " that is not in the use.desc or its metadata.xml file"),
- "IUSE.missing": (
- "This ebuild has a USE conditional"
- " which references a flag that is not listed in IUSE"),
- "IUSE.rubydeprecated": (
- "The ebuild has set a ruby interpreter in USE_RUBY,"
- " that is not available as a ruby target anymore"),
- "LICENSE.invalid": (
- "This ebuild is listing a license"
- " that doesnt exist in portages license/ dir."),
- "LICENSE.deprecated": (
- "This ebuild is listing a deprecated license."),
- "KEYWORDS.invalid": (
- "This ebuild contains KEYWORDS"
- " that are not listed in profiles/arch.list"
- " or for which no valid profile was found"),
- "RDEPEND.implicit": (
- "RDEPEND is unset in the ebuild"
- " which triggers implicit RDEPEND=$DEPEND assignment"
- " (prior to EAPI 4)"),
- "RDEPEND.suspect": (
- "RDEPEND contains a package that usually only belongs in DEPEND."),
- "RESTRICT.invalid": (
- "This ebuild contains invalid RESTRICT values."),
- "digest.assumed": (
- "Existing digest must be assumed correct (Package level only)"),
- "digest.missing": (
- "Some files listed in SRC_URI aren't referenced in the Manifest"),
- "digest.unused": (
- "Some files listed in the Manifest aren't referenced in SRC_URI"),
- "ebuild.absdosym": (
- "This ebuild uses absolute target to dosym where relative symlink"
- " could be used instead"),
- "ebuild.majorsyn": (
- "This ebuild has a major syntax error"
- " that may cause the ebuild to fail partially or fully"),
- "ebuild.minorsyn": (
- "This ebuild has a minor syntax error"
- " that contravenes gentoo coding style"),
- "ebuild.badheader": (
- "This ebuild has a malformed header"),
- "manifest.bad": (
- "Manifest has missing or incorrect digests"),
- "metadata.missing": (
- "Missing metadata.xml files"),
- "metadata.bad": (
- "Bad metadata.xml files"),
- "metadata.warning": (
- "Warnings in metadata.xml files"),
- "portage.internal": (
- "The ebuild uses an internal Portage function or variable"),
- "repo.eapi.banned": (
- "The ebuild uses an EAPI which is"
- " banned by the repository's metadata/layout.conf settings"),
- "repo.eapi.deprecated": (
- "The ebuild uses an EAPI which is"
- " deprecated by the repository's metadata/layout.conf settings"),
- "virtual.oldstyle": (
- "The ebuild PROVIDEs an old-style virtual (see GLEP 37)"),
- "virtual.suspect": (
- "Ebuild contains a package"
- " that usually should be pulled via virtual/, not directly."),
- "usage.obsolete": (
- "The ebuild makes use of an obsolete construct"),
- "upstream.workaround": (
- "The ebuild works around an upstream bug,"
- " an upstream bug should be filed and tracked in bugs.gentoo.org"),
- "uri.https": "URI uses http:// but should use https://",
-}
-
-qacats = list(qahelp)
-qacats.sort()
-
-qawarnings = set((
- "changelog.missing",
- "changelog.notadded",
- "dependency.unknown",
- "digest.assumed",
- "digest.unused",
- "ebuild.notadded",
- "ebuild.nesteddie",
- "dependency.badmasked",
- "dependency.badindev",
- "dependency.badmaskedindev",
- "dependency.badtilde",
- "dependency.missingslot",
- "dependency.perlcore",
- "DESCRIPTION.toolong",
- "EAPI.deprecated",
- "HOMEPAGE.virtual",
- "LICENSE.deprecated",
- "LICENSE.virtual",
- "KEYWORDS.dropped",
- "KEYWORDS.stupid",
- "KEYWORDS.missing",
- "PDEPEND.suspect",
- "RDEPEND.implicit",
- "RDEPEND.suspect",
- "virtual.suspect",
- "RESTRICT.invalid",
- "ebuild.absdosym",
- "ebuild.minorsyn",
- "ebuild.badheader",
- "ebuild.patches",
- "file.empty",
- "file.size",
- "inherit.unused",
- "inherit.deprecated",
- "java.eclassesnotused",
- "wxwidgets.eclassnotused",
- "metadata.warning",
- "portage.internal",
- "repo.eapi.deprecated",
- "usage.obsolete",
- "upstream.workaround",
- "IUSE.rubydeprecated",
- "uri.https",
-))
-
-
-missingvars = ["KEYWORDS", "LICENSE", "DESCRIPTION", "HOMEPAGE"]
-allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
-allvars.update(Package.metadata_keys)
-allvars = sorted(allvars)
-
-for x in missingvars:
- x += ".missing"
- if x not in qacats:
- logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
- qacats.append(x)
- qawarnings.add(x)
-
-valid_restrict = frozenset([
- "binchecks", "bindist", "fetch", "installsources", "mirror",
- "preserve-libs", "primaryuri", "splitdebug", "strip", "test", "userpriv"])
-
-
-suspect_rdepend = frozenset([
- "app-arch/cabextract",
- "app-arch/rpm2targz",
- "app-doc/doxygen",
- "dev-lang/nasm",
- "dev-lang/swig",
- "dev-lang/yasm",
- "dev-perl/extutils-pkgconfig",
- "dev-qt/linguist-tools",
- "dev-util/byacc",
- "dev-util/cmake",
- "dev-util/ftjam",
- "dev-util/gperf",
- "dev-util/gtk-doc",
- "dev-util/gtk-doc-am",
- "dev-util/intltool",
- "dev-util/jam",
- "dev-util/pkg-config-lite",
- "dev-util/pkgconf",
- "dev-util/pkgconfig",
- "dev-util/pkgconfig-openbsd",
- "dev-util/scons",
- "dev-util/unifdef",
- "dev-util/yacc",
- "media-gfx/ebdftopcf",
- "sys-apps/help2man",
- "sys-devel/autoconf",
- "sys-devel/automake",
- "sys-devel/bin86",
- "sys-devel/bison",
- "sys-devel/dev86",
- "sys-devel/flex",
- "sys-devel/m4",
- "sys-devel/pmake",
- "virtual/linux-sources",
- "virtual/linuxtv-dvb-headers",
- "virtual/os-headers",
- "virtual/pkgconfig",
- "x11-misc/bdftopcf",
- "x11-misc/imake",
-])
-
-suspect_virtual = {
- "dev-util/pkg-config-lite": "virtual/pkgconfig",
- "dev-util/pkgconf": "virtual/pkgconfig",
- "dev-util/pkgconfig": "virtual/pkgconfig",
- "dev-util/pkgconfig-openbsd": "virtual/pkgconfig",
- "dev-libs/libusb": "virtual/libusb",
- "dev-libs/libusbx": "virtual/libusb",
- "dev-libs/libusb-compat": "virtual/libusb",
-}
-
-ruby_deprecated = frozenset([
- "ruby_targets_ree18",
- "ruby_targets_ruby18",
- "ruby_targets_ruby19",
-])
-
-
-# file.executable
-no_exec = frozenset(["Manifest", "ChangeLog", "metadata.xml"])
+class QAData(object):
+
+ def __init__(self):
+ # Create the main exported data variables
+ self.max_desc_len = None
+ self.allowed_filename_chars = None
+ self.qahelp = None
+ self.qacats = None
+ self.qawarnings = None
+ self.missingvars = None
+ self.allvars = None
+ self.valid_restrict = None
+ self.suspect_rdepend = None
+ self.suspect_virtual = None
+ self.ruby_deprecated = None
+ self.no_exec = None
+
+
+ def load_repo_config(self, repopath, options):
+ '''Load the repository repoman qa_data.yml config
+
+ @param repopath: string, The path of the repository being scanned
+ This could be a parent repository using the
+ repoman_masters layout.conf variable
+ '''
+ filepath = os.path.join(repopath, 'metadata/repoman/qa_data.yml')
+ logging.debug("QAData: reading file", filepath)
+ try:
+ with open(filepath, 'r') as qadata_file:
+ qadata = yaml.safe_load(qadata_file.read())
+ except IOError as error:
+ logging.error("Failed to load 'qa_data.yml' file at path: %s", filepath)
+ logging.eception(error)
+ return False
+ self.max_desc_len = qadata['max_description_length']
+ self.allowed_filename_chars = qadata["allowed_filename_chars"]
+
+ self.qahelp = qadata['qahelp']
+
+ self.qacats = []
+ for x in sorted(self.qahelp):
+ for y in sorted(self.qahelp[x]):
+ self.qacats.append('.'.join([x, y]))
+ self.qacats.sort()
+
+ self.qawarnings = set(qadata['qawarnings'])
+ if options.experimental_inherit == 'y':
+ # This is experimental, so it's non-fatal.
+ self.qawarnings.add("inherit.missing")
+
+ self.missingvars = qadata["missingvars"]
+ logging.debug("QAData: missingvars", self.missingvars)
+ self.allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
+ self.allvars.update(Package.metadata_keys)
+ self.allvars = sorted(self.allvars)
+
+ for x in self.missingvars:
+ x += ".missing"
+ if x not in self.qacats:
+ logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
+ self.qacats.append(x)
+ self.qawarnings.add(x)
+
+ self.valid_restrict = frozenset(qadata["valid_restrict"])
+
+ self.suspect_rdepend = frozenset(qadata["suspect_rdepend"])
+
+ self.suspect_virtual = qadata["suspect_virtual"]
+
+ self.ruby_deprecated = frozenset(qadata["ruby_deprecated"])
+
+ # file.executable
+ self.no_exec = frozenset(qadata["no_exec_files"])
+ logging.debug("QAData: completed loading file: %s", filepath)
+ return True
def format_qa_output(
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 21:36 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 21:36 UTC (permalink / raw
To: gentoo-commits
commit: 5cc3bc2f3fbe8c4ca2c078eb5709d61b5c3f94c9
Author: Manuel Rüger <mrueg <AT> gentoo <DOT> org>
AuthorDate: Tue May 30 10:18:58 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 21:29:26 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=5cc3bc2f
repoman: Mark ruby-2.0 as deprecated
repoman/pym/repoman/qa_data.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index a59fed778..fed798f9f 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -358,6 +358,7 @@ ruby_deprecated = frozenset([
"ruby_targets_ree18",
"ruby_targets_ruby18",
"ruby_targets_ruby19",
+ "ruby_targets_ruby20",
])
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 21:36 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 21:36 UTC (permalink / raw
To: gentoo-commits
commit: 2c3728f246e5e38ea2d3611da4ca88efed34be78
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:38:48 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 21:32:53 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=2c3728f2
qa_data.py: Initial move of all configurable data to the repo
All this data was ported to a metadata/repoman/qa_data.yml file.
repoman/pym/repoman/qa_data.py | 433 ++++++++---------------------------------
1 file changed, 77 insertions(+), 356 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index fed798f9f..2bbf460df 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -1,369 +1,90 @@
# -*- coding:utf-8 -*-
import logging
+import os
+import yaml
from _emerge.Package import Package
# import our initialized portage instance
from repoman._portage import portage
-max_desc_len = 80
-allowed_filename_chars = "a-zA-Z0-9._-+:"
-qahelp = {
- "CVS/Entries.IO_error": (
- "Attempting to commit, and an IO error was encountered access the"
- " Entries file"),
- "ebuild.invalidname": (
- "Ebuild files with a non-parseable or syntactically incorrect name"
- " (or using 2.1 versioning extensions)"),
- "ebuild.namenomatch": (
- "Ebuild files that do not have the same name as their parent"
- " directory"),
- "changelog.ebuildadded": (
- "An ebuild was added but the ChangeLog was not modified"),
- "changelog.missing": (
- "Missing ChangeLog files"),
- "ebuild.notadded": (
- "Ebuilds that exist but have not been added to cvs"),
- "ebuild.patches": (
- "PATCHES variable should be a bash array to ensure white space safety"),
- "changelog.notadded": (
- "ChangeLogs that exist but have not been added to cvs"),
- "dependency.bad": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds)"),
- "dependency.badmasked": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds)"),
- "dependency.badindev": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds) in developing arch"),
- "dependency.badmaskedindev": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds) in developing arch"),
- "dependency.badtilde": (
- "Uses the ~ dep operator with a non-zero revision part,"
- " which is useless (the revision is ignored)"),
- "dependency.missingslot": (
- "RDEPEND matches more than one SLOT but does not specify a "
- "slot and/or use the := or :* slot operator"),
- "dependency.perlcore": (
- "This ebuild directly depends on a package in perl-core;"
- " it should use the corresponding virtual instead."),
- "dependency.syntax": (
- "Syntax error in dependency string"
- " (usually an extra/missing space/parenthesis)"),
- "dependency.unknown": (
- "Ebuild has a dependency that refers to an unknown package"
- " (which may be valid if it is a blocker for a renamed/removed package,"
- " or is an alternative choice provided by an overlay)"),
- "dependency.badslotop": (
- "RDEPEND contains ':=' slot operator under '||' dependency."),
- "file.executable": (
- "Ebuilds, digests, metadata.xml, Manifest, and ChangeLog do not need"
- " the executable bit"),
- "file.size": (
- "Files in the files directory must be under 20 KiB"),
- "file.size.fatal": (
- "Files in the files directory must be under 60 KiB"),
- "file.empty": (
- "Empty file in the files directory"),
- "file.name": (
- "File/dir name must be composed"
- " of only the following chars: %s " % allowed_filename_chars),
- "file.UTF8": (
- "File is not UTF8 compliant"),
- "inherit.deprecated": (
- "Ebuild inherits a deprecated eclass"),
- "inherit.missing": (
- "Ebuild uses functions from an eclass but does not inherit it"),
- "inherit.unused": (
- "Ebuild inherits an eclass but does not use it"),
- "java.eclassesnotused": (
- "With virtual/jdk in DEPEND you must inherit a java eclass"),
- "wxwidgets.eclassnotused": (
- "Ebuild DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass"),
- "KEYWORDS.dropped": (
- "Ebuilds that appear to have dropped KEYWORDS for some arch"),
- "KEYWORDS.missing": (
- "Ebuilds that have a missing or empty KEYWORDS variable"),
- "KEYWORDS.stable": (
- "Ebuilds that have been added directly with stable KEYWORDS"),
- "KEYWORDS.stupid": (
- "Ebuilds that use KEYWORDS=-* instead of package.mask"),
- "LICENSE.missing": (
- "Ebuilds that have a missing or empty LICENSE variable"),
- "LICENSE.virtual": (
- "Virtuals that have a non-empty LICENSE variable"),
- "DESCRIPTION.missing": (
- "Ebuilds that have a missing or empty DESCRIPTION variable"),
- "DESCRIPTION.toolong": (
- "DESCRIPTION is over %d characters" % max_desc_len),
- "EAPI.definition": (
- "EAPI definition does not conform to PMS section 7.3.1"
- " (first non-comment, non-blank line)"),
- "EAPI.deprecated": (
- "Ebuilds that use features that are deprecated in the current EAPI"),
- "EAPI.incompatible": (
- "Ebuilds that use features that are only available with a different"
- " EAPI"),
- "EAPI.unsupported": (
- "Ebuilds that have an unsupported EAPI version"
- " (you must upgrade portage)"),
- "SLOT.invalid": (
- "Ebuilds that have a missing or invalid SLOT variable value"),
- "HOMEPAGE.missing": (
- "Ebuilds that have a missing or empty HOMEPAGE variable"),
- "HOMEPAGE.virtual": (
- "Virtuals that have a non-empty HOMEPAGE variable"),
- "HOMEPAGE.missingurischeme": (
- "HOMEPAGE is missing an URI scheme"),
- "PDEPEND.suspect": (
- "PDEPEND contains a package that usually only belongs in DEPEND."),
- "LICENSE.syntax": (
- "Syntax error in LICENSE"
- " (usually an extra/missing space/parenthesis)"),
- "PROVIDE.syntax": (
- "Syntax error in PROVIDE"
- " (usually an extra/missing space/parenthesis)"),
- "PROPERTIES.syntax": (
- "Syntax error in PROPERTIES"
- " (usually an extra/missing space/parenthesis)"),
- "RESTRICT.syntax": (
- "Syntax error in RESTRICT"
- " (usually an extra/missing space/parenthesis)"),
- "REQUIRED_USE.syntax": (
- "Syntax error in REQUIRED_USE"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.syntax": (
- "Syntax error in SRC_URI"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.mirror": (
- "A uri listed in profiles/thirdpartymirrors is found in SRC_URI"),
- "ebuild.syntax": (
- "Error generating cache entry for ebuild;"
- " typically caused by ebuild syntax error"
- " or digest verification failure"),
- "ebuild.output": (
- "A simple sourcing of the ebuild produces output;"
- " this breaks ebuild policy."),
- "ebuild.nesteddie": (
- "Placing 'die' inside ( ) prints an error,"
- " but doesn't stop the ebuild."),
- "variable.invalidchar": (
- "A variable contains an invalid character"
- " that is not part of the ASCII character set"),
- "variable.readonly": (
- "Assigning a readonly variable"),
- "variable.usedwithhelpers": (
- "Ebuild uses D, ROOT, ED, EROOT or EPREFIX with helpers"),
- "LIVEVCS.stable": (
- "This ebuild is a live checkout from a VCS but has stable keywords."),
- "LIVEVCS.unmasked": (
- "This ebuild is a live checkout from a VCS but has keywords"
- " and is not masked in the global package.mask."),
- "IUSE.invalid": (
- "This ebuild has a variable in IUSE"
- " that is not in the use.desc or its metadata.xml file"),
- "IUSE.missing": (
- "This ebuild has a USE conditional"
- " which references a flag that is not listed in IUSE"),
- "IUSE.rubydeprecated": (
- "The ebuild has set a ruby interpreter in USE_RUBY,"
- " that is not available as a ruby target anymore"),
- "LICENSE.invalid": (
- "This ebuild is listing a license"
- " that doesnt exist in portages license/ dir."),
- "LICENSE.deprecated": (
- "This ebuild is listing a deprecated license."),
- "KEYWORDS.invalid": (
- "This ebuild contains KEYWORDS"
- " that are not listed in profiles/arch.list"
- " or for which no valid profile was found"),
- "RDEPEND.implicit": (
- "RDEPEND is unset in the ebuild"
- " which triggers implicit RDEPEND=$DEPEND assignment"
- " (prior to EAPI 4)"),
- "RDEPEND.suspect": (
- "RDEPEND contains a package that usually only belongs in DEPEND."),
- "RESTRICT.invalid": (
- "This ebuild contains invalid RESTRICT values."),
- "digest.assumed": (
- "Existing digest must be assumed correct (Package level only)"),
- "digest.missing": (
- "Some files listed in SRC_URI aren't referenced in the Manifest"),
- "digest.unused": (
- "Some files listed in the Manifest aren't referenced in SRC_URI"),
- "ebuild.absdosym": (
- "This ebuild uses absolute target to dosym where relative symlink"
- " could be used instead"),
- "ebuild.majorsyn": (
- "This ebuild has a major syntax error"
- " that may cause the ebuild to fail partially or fully"),
- "ebuild.minorsyn": (
- "This ebuild has a minor syntax error"
- " that contravenes gentoo coding style"),
- "ebuild.badheader": (
- "This ebuild has a malformed header"),
- "manifest.bad": (
- "Manifest has missing or incorrect digests"),
- "metadata.missing": (
- "Missing metadata.xml files"),
- "metadata.bad": (
- "Bad metadata.xml files"),
- "metadata.warning": (
- "Warnings in metadata.xml files"),
- "portage.internal": (
- "The ebuild uses an internal Portage function or variable"),
- "repo.eapi.banned": (
- "The ebuild uses an EAPI which is"
- " banned by the repository's metadata/layout.conf settings"),
- "repo.eapi.deprecated": (
- "The ebuild uses an EAPI which is"
- " deprecated by the repository's metadata/layout.conf settings"),
- "virtual.oldstyle": (
- "The ebuild PROVIDEs an old-style virtual (see GLEP 37)"),
- "virtual.suspect": (
- "Ebuild contains a package"
- " that usually should be pulled via virtual/, not directly."),
- "usage.obsolete": (
- "The ebuild makes use of an obsolete construct"),
- "upstream.workaround": (
- "The ebuild works around an upstream bug,"
- " an upstream bug should be filed and tracked in bugs.gentoo.org"),
- "uri.https": "URI uses http:// but should use https://",
-}
-
-qacats = list(qahelp)
-qacats.sort()
-
-qawarnings = set((
- "changelog.missing",
- "changelog.notadded",
- "dependency.unknown",
- "digest.assumed",
- "digest.unused",
- "ebuild.notadded",
- "ebuild.nesteddie",
- "dependency.badmasked",
- "dependency.badindev",
- "dependency.badmaskedindev",
- "dependency.badtilde",
- "dependency.missingslot",
- "dependency.perlcore",
- "DESCRIPTION.toolong",
- "EAPI.deprecated",
- "HOMEPAGE.virtual",
- "LICENSE.deprecated",
- "LICENSE.virtual",
- "KEYWORDS.dropped",
- "KEYWORDS.stupid",
- "KEYWORDS.missing",
- "PDEPEND.suspect",
- "RDEPEND.implicit",
- "RDEPEND.suspect",
- "virtual.suspect",
- "RESTRICT.invalid",
- "ebuild.absdosym",
- "ebuild.minorsyn",
- "ebuild.badheader",
- "ebuild.patches",
- "file.empty",
- "file.size",
- "inherit.unused",
- "inherit.deprecated",
- "java.eclassesnotused",
- "wxwidgets.eclassnotused",
- "metadata.warning",
- "portage.internal",
- "repo.eapi.deprecated",
- "usage.obsolete",
- "upstream.workaround",
- "IUSE.rubydeprecated",
- "uri.https",
-))
-
-
-missingvars = ["KEYWORDS", "LICENSE", "DESCRIPTION", "HOMEPAGE"]
-allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
-allvars.update(Package.metadata_keys)
-allvars = sorted(allvars)
-
-for x in missingvars:
- x += ".missing"
- if x not in qacats:
- logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
- qacats.append(x)
- qawarnings.add(x)
-
-valid_restrict = frozenset([
- "binchecks", "bindist", "fetch", "installsources", "mirror",
- "preserve-libs", "primaryuri", "splitdebug", "strip", "test", "userpriv"])
-
-
-suspect_rdepend = frozenset([
- "app-arch/cabextract",
- "app-arch/rpm2targz",
- "app-doc/doxygen",
- "dev-lang/nasm",
- "dev-lang/swig",
- "dev-lang/yasm",
- "dev-perl/extutils-pkgconfig",
- "dev-qt/linguist-tools",
- "dev-util/byacc",
- "dev-util/cmake",
- "dev-util/ftjam",
- "dev-util/gperf",
- "dev-util/gtk-doc",
- "dev-util/gtk-doc-am",
- "dev-util/intltool",
- "dev-util/jam",
- "dev-util/pkg-config-lite",
- "dev-util/pkgconf",
- "dev-util/pkgconfig",
- "dev-util/pkgconfig-openbsd",
- "dev-util/scons",
- "dev-util/unifdef",
- "dev-util/yacc",
- "media-gfx/ebdftopcf",
- "sys-apps/help2man",
- "sys-devel/autoconf",
- "sys-devel/automake",
- "sys-devel/bin86",
- "sys-devel/bison",
- "sys-devel/dev86",
- "sys-devel/flex",
- "sys-devel/m4",
- "sys-devel/pmake",
- "virtual/linux-sources",
- "virtual/linuxtv-dvb-headers",
- "virtual/os-headers",
- "virtual/pkgconfig",
- "x11-misc/bdftopcf",
- "x11-misc/imake",
-])
-
-suspect_virtual = {
- "dev-util/pkg-config-lite": "virtual/pkgconfig",
- "dev-util/pkgconf": "virtual/pkgconfig",
- "dev-util/pkgconfig": "virtual/pkgconfig",
- "dev-util/pkgconfig-openbsd": "virtual/pkgconfig",
- "dev-libs/libusb": "virtual/libusb",
- "dev-libs/libusbx": "virtual/libusb",
- "dev-libs/libusb-compat": "virtual/libusb",
-}
-
-ruby_deprecated = frozenset([
- "ruby_targets_ree18",
- "ruby_targets_ruby18",
- "ruby_targets_ruby19",
- "ruby_targets_ruby20",
-])
-
-
-# file.executable
-no_exec = frozenset(["Manifest", "ChangeLog", "metadata.xml"])
+class QAData(object):
+
+ def __init__(self):
+ # Create the main exported data variables
+ self.max_desc_len = None
+ self.allowed_filename_chars = None
+ self.qahelp = None
+ self.qacats = None
+ self.qawarnings = None
+ self.missingvars = None
+ self.allvars = None
+ self.valid_restrict = None
+ self.suspect_rdepend = None
+ self.suspect_virtual = None
+ self.ruby_deprecated = None
+ self.no_exec = None
+
+
+ def load_repo_config(self, repopath, options):
+ '''Load the repository repoman qa_data.yml config
+
+ @param repopath: string, The path of the repository being scanned
+ This could be a parent repository using the
+ repoman_masters layout.conf variable
+ '''
+ filepath = os.path.join(repopath, 'metadata/repoman/qa_data.yml')
+ logging.debug("QAData: reading file", filepath)
+ try:
+ with open(filepath, 'r') as qadata_file:
+ qadata = yaml.safe_load(qadata_file.read())
+ except IOError as error:
+ logging.error("Failed to load 'qa_data.yml' file at path: %s", filepath)
+ logging.eception(error)
+ return False
+ self.max_desc_len = qadata['max_description_length']
+ self.allowed_filename_chars = qadata["allowed_filename_chars"]
+
+ self.qahelp = qadata['qahelp']
+
+ self.qacats = []
+ for x in sorted(self.qahelp):
+ for y in sorted(self.qahelp[x]):
+ self.qacats.append('.'.join([x, y]))
+ self.qacats.sort()
+
+ self.qawarnings = set(qadata['qawarnings'])
+ if options.experimental_inherit == 'y':
+ # This is experimental, so it's non-fatal.
+ self.qawarnings.add("inherit.missing")
+
+ self.missingvars = qadata["missingvars"]
+ logging.debug("QAData: missingvars", self.missingvars)
+ self.allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
+ self.allvars.update(Package.metadata_keys)
+ self.allvars = sorted(self.allvars)
+
+ for x in self.missingvars:
+ x += ".missing"
+ if x not in self.qacats:
+ logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
+ self.qacats.append(x)
+ self.qawarnings.add(x)
+
+ self.valid_restrict = frozenset(qadata["valid_restrict"])
+
+ self.suspect_rdepend = frozenset(qadata["suspect_rdepend"])
+
+ self.suspect_virtual = qadata["suspect_virtual"]
+
+ self.ruby_deprecated = frozenset(qadata["ruby_deprecated"])
+
+ # file.executable
+ self.no_exec = frozenset(qadata["no_exec_files"])
+ logging.debug("QAData: completed loading file: %s", filepath)
+ return True
def format_qa_output(
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 21:36 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 21:36 UTC (permalink / raw
To: gentoo-commits
commit: e7da86559d42eb136b6c9caa13af26c64347c7ca
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:49:01 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 21:32:53 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=e7da8655
repoman: Move qa_data initialization and loading
Move the new QAData class instance init to repoman_main().
parse_args() remove unused qahelp parameter.
qa_tracker.py: Add default qacats and qawarnings parameters as None.
These will be assigned later due to circular init references.
repos.py: Perform the QAData class loading and complete intialization
assignments.
repoman/pym/repoman/argparser.py | 3 +--
repoman/pym/repoman/main.py | 26 ++++++++++++++------------
repoman/pym/repoman/qa_tracker.py | 10 +++++-----
repoman/pym/repoman/repos.py | 12 ++++++++++--
4 files changed, 30 insertions(+), 21 deletions(-)
diff --git a/repoman/pym/repoman/argparser.py b/repoman/pym/repoman/argparser.py
index 2d56a87e6..68701378f 100644
--- a/repoman/pym/repoman/argparser.py
+++ b/repoman/pym/repoman/argparser.py
@@ -15,11 +15,10 @@ from portage import _unicode_decode
from portage import util
-def parse_args(argv, qahelp, repoman_default_opts):
+def parse_args(argv, repoman_default_opts):
"""Use a customized optionParser to parse command line arguments for repoman
Args:
argv - a sequence of command line arguments
- qahelp - a dict of qa warning to help message
Returns:
(opts, args), just like a call to parser.parse_args()
"""
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100755
new mode 100644
index ccc735c7d..3b628de00
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -29,9 +29,8 @@ from portage.util.futures.extendedfutures import (
from repoman.actions import Actions
from repoman.argparser import parse_args
-from repoman.qa_data import (
- format_qa_output, format_qa_output_column, qahelp,
- qawarnings, qacats)
+from repoman.qa_data import QAData
+from repoman.qa_data import format_qa_output, format_qa_output_column
from repoman.repos import RepoSettings
from repoman.scanner import Scanner
from repoman import utilities
@@ -60,7 +59,7 @@ def repoman_main(argv):
nocolor()
options, arguments = parse_args(
- sys.argv, qahelp, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
+ sys.argv, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
if options.version:
print("Repoman", VERSION, "(portage-%s)" % portage.VERSION)
@@ -73,10 +72,6 @@ def repoman_main(argv):
else:
logger.setLevel(LOGLEVEL)
- if options.experimental_inherit == 'y':
- # This is experimental, so it's non-fatal.
- qawarnings.add("inherit.missing")
-
# Set this to False when an extraordinary issue (generally
# something other than a QA issue) makes it impossible to
# commit (like if Manifest generation fails).
@@ -91,14 +86,21 @@ def repoman_main(argv):
# avoid a circular parameter repo_settings
vcs_settings = VCSSettings(options, repoman_settings)
+ qadata = QAData()
+ logging.debug("repoman_main: RepoSettings init")
repo_settings = RepoSettings(
config_root, portdir, portdir_overlay,
- repoman_settings, vcs_settings, options, qawarnings)
+ repoman_settings, vcs_settings, options, qadata)
repoman_settings = repo_settings.repoman_settings
# Now set repo_settings
vcs_settings.repo_settings = repo_settings
+ # set QATracker qacats, qawarnings
+ vcs_settings.qatracker.qacats = repo_settings.qadata.qacats
+ vcs_settings.qatracker.qawarnings = repo_settings.qadata.qawarnings
+ logging.debug("repoman_main: vcs_settings done")
+ logging.debug("repoman_main: qadata: %s", repo_settings.qadata)
if 'digest' in repoman_settings.features and options.digest != 'n':
options.digest = 'y'
@@ -133,11 +135,11 @@ def repoman_main(argv):
if options.mode == "manifest":
sys.exit(result['fail'])
- for x in qacats:
+ for x in qadata.qacats:
if x not in vcs_settings.qatracker.fails:
continue
result['warn'] = 1
- if x not in qawarnings:
+ if x not in qadata.qawarnings:
result['fail'] = 1
if result['fail'] or \
@@ -174,7 +176,7 @@ def repoman_main(argv):
format_output = format_outputs.get(
options.output_style, format_outputs['default'])
format_output(f, vcs_settings.qatracker.fails, result['full'],
- result['fail'], options, qawarnings)
+ result['fail'], options, qadata.qawarnings)
style_file.flush()
del console_writer, f, style_file
diff --git a/repoman/pym/repoman/qa_tracker.py b/repoman/pym/repoman/qa_tracker.py
index 9bfe0e241..faaf8e398 100644
--- a/repoman/pym/repoman/qa_tracker.py
+++ b/repoman/pym/repoman/qa_tracker.py
@@ -2,15 +2,15 @@
import logging
import sys
-from repoman.qa_data import qacats, qawarnings
-
class QATracker(object):
'''Track all occurrances of Q/A problems detected'''
- def __init__(self):
+ def __init__(self, qacats=None, qawarnings=None):
self.fails = {}
self.warns = {}
+ self.qacats = qacats
+ self.qawarnings = qawarnings
def add_error(self, detected_qa, info):
'''Add the Q/A error to the database of detected problems
@@ -18,7 +18,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qacats list
@param info: string, details of the detected problem
'''
- if detected_qa not in qacats:
+ if detected_qa not in self.qacats:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_error(): %s, %s' % (detected_qa, info))
@@ -34,7 +34,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qawarnings list
@param info: string, details of the detected problem
'''
- if detected_qa not in qawarnings:
+ if detected_qa not in self.qawarnings:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_warning(): %s, %s' % (detected_qa, info))
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index 39f53c180..cd917828d 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -27,7 +27,7 @@ class RepoSettings(object):
def __init__(
self, config_root, portdir, portdir_overlay,
repoman_settings=None, vcs_settings=None, options=None,
- qawarnings=None):
+ qadata=None):
self.config_root = config_root
self.repoman_settings = repoman_settings
self.vcs_settings = vcs_settings
@@ -41,6 +41,14 @@ class RepoSettings(object):
except KeyError:
self._add_repo(config_root, portdir_overlay)
+ logging.debug("RepoSettings: init(); load qadata")
+ # load the repo specific configuration
+ self.qadata = qadata
+ if not self.qadata.load_repo_config(self.repodir, options):
+ logging.error("Aborting...")
+ sys.exit(1)
+ logging.debug("RepoSettings: qadata loaded", qadata.no_exec)
+
self.root = self.repoman_settings['EROOT']
self.trees = {
self.root: {'porttree': portage.portagetree(settings=self.repoman_settings)}
@@ -60,7 +68,7 @@ class RepoSettings(object):
del self.repositories[repo.name]
if self.repo_config.allow_provide_virtual:
- qawarnings.add("virtual.oldstyle")
+ qadata.qawarnings.add("virtual.oldstyle")
if self.repo_config.sign_commit and options.mode in ("commit", "fix", "manifest"):
if vcs_settings.vcs:
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 21:36 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 21:36 UTC (permalink / raw
To: gentoo-commits
commit: 7cc005b548feac3953e73a20d10df5bc89cb5b40
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 18:08:50 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 21:32:54 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=7cc005b5
repoman: main.py: Remove unused InvalidStateError import
repoman/pym/repoman/main.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100644
new mode 100755
index 3b628de00..c1e3b99fe
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -24,7 +24,6 @@ from portage.output import ConsoleStyleFile, StyleWriter
from portage.util import formatter
from portage.util.futures.extendedfutures import (
ExtendedFuture,
- InvalidStateError,
)
from repoman.actions import Actions
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 22:10 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 22:10 UTC (permalink / raw
To: gentoo-commits
commit: cd01ff0d8760de7f1f64d124a9164e8536a8a874
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 18:08:50 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 22:05:14 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=cd01ff0d
repoman: main.py: Remove unused InvalidStateError import
repoman/pym/repoman/main.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100644
new mode 100755
index 3b628de00..c1e3b99fe
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -24,7 +24,6 @@ from portage.output import ConsoleStyleFile, StyleWriter
from portage.util import formatter
from portage.util.futures.extendedfutures import (
ExtendedFuture,
- InvalidStateError,
)
from repoman.actions import Actions
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 22:10 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 22:10 UTC (permalink / raw
To: gentoo-commits
commit: b3f787a967e91188b92df55db0cd16bd8fc6bbd3
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:49:01 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 22:05:14 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=b3f787a9
repoman: Move qa_data initialization and loading
Move the new QAData class instance init to repoman_main().
parse_args() remove unused qahelp parameter.
qa_tracker.py: Add default qacats and qawarnings parameters as None.
These will be assigned later due to circular init references.
repos.py: Perform the QAData class loading and complete intialization
assignments.
repoman/pym/repoman/argparser.py | 3 +--
repoman/pym/repoman/main.py | 26 ++++++++++++++------------
repoman/pym/repoman/qa_tracker.py | 10 +++++-----
repoman/pym/repoman/repos.py | 12 ++++++++++--
4 files changed, 30 insertions(+), 21 deletions(-)
diff --git a/repoman/pym/repoman/argparser.py b/repoman/pym/repoman/argparser.py
index 2d56a87e6..68701378f 100644
--- a/repoman/pym/repoman/argparser.py
+++ b/repoman/pym/repoman/argparser.py
@@ -15,11 +15,10 @@ from portage import _unicode_decode
from portage import util
-def parse_args(argv, qahelp, repoman_default_opts):
+def parse_args(argv, repoman_default_opts):
"""Use a customized optionParser to parse command line arguments for repoman
Args:
argv - a sequence of command line arguments
- qahelp - a dict of qa warning to help message
Returns:
(opts, args), just like a call to parser.parse_args()
"""
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100755
new mode 100644
index ccc735c7d..3b628de00
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -29,9 +29,8 @@ from portage.util.futures.extendedfutures import (
from repoman.actions import Actions
from repoman.argparser import parse_args
-from repoman.qa_data import (
- format_qa_output, format_qa_output_column, qahelp,
- qawarnings, qacats)
+from repoman.qa_data import QAData
+from repoman.qa_data import format_qa_output, format_qa_output_column
from repoman.repos import RepoSettings
from repoman.scanner import Scanner
from repoman import utilities
@@ -60,7 +59,7 @@ def repoman_main(argv):
nocolor()
options, arguments = parse_args(
- sys.argv, qahelp, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
+ sys.argv, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
if options.version:
print("Repoman", VERSION, "(portage-%s)" % portage.VERSION)
@@ -73,10 +72,6 @@ def repoman_main(argv):
else:
logger.setLevel(LOGLEVEL)
- if options.experimental_inherit == 'y':
- # This is experimental, so it's non-fatal.
- qawarnings.add("inherit.missing")
-
# Set this to False when an extraordinary issue (generally
# something other than a QA issue) makes it impossible to
# commit (like if Manifest generation fails).
@@ -91,14 +86,21 @@ def repoman_main(argv):
# avoid a circular parameter repo_settings
vcs_settings = VCSSettings(options, repoman_settings)
+ qadata = QAData()
+ logging.debug("repoman_main: RepoSettings init")
repo_settings = RepoSettings(
config_root, portdir, portdir_overlay,
- repoman_settings, vcs_settings, options, qawarnings)
+ repoman_settings, vcs_settings, options, qadata)
repoman_settings = repo_settings.repoman_settings
# Now set repo_settings
vcs_settings.repo_settings = repo_settings
+ # set QATracker qacats, qawarnings
+ vcs_settings.qatracker.qacats = repo_settings.qadata.qacats
+ vcs_settings.qatracker.qawarnings = repo_settings.qadata.qawarnings
+ logging.debug("repoman_main: vcs_settings done")
+ logging.debug("repoman_main: qadata: %s", repo_settings.qadata)
if 'digest' in repoman_settings.features and options.digest != 'n':
options.digest = 'y'
@@ -133,11 +135,11 @@ def repoman_main(argv):
if options.mode == "manifest":
sys.exit(result['fail'])
- for x in qacats:
+ for x in qadata.qacats:
if x not in vcs_settings.qatracker.fails:
continue
result['warn'] = 1
- if x not in qawarnings:
+ if x not in qadata.qawarnings:
result['fail'] = 1
if result['fail'] or \
@@ -174,7 +176,7 @@ def repoman_main(argv):
format_output = format_outputs.get(
options.output_style, format_outputs['default'])
format_output(f, vcs_settings.qatracker.fails, result['full'],
- result['fail'], options, qawarnings)
+ result['fail'], options, qadata.qawarnings)
style_file.flush()
del console_writer, f, style_file
diff --git a/repoman/pym/repoman/qa_tracker.py b/repoman/pym/repoman/qa_tracker.py
index 9bfe0e241..faaf8e398 100644
--- a/repoman/pym/repoman/qa_tracker.py
+++ b/repoman/pym/repoman/qa_tracker.py
@@ -2,15 +2,15 @@
import logging
import sys
-from repoman.qa_data import qacats, qawarnings
-
class QATracker(object):
'''Track all occurrances of Q/A problems detected'''
- def __init__(self):
+ def __init__(self, qacats=None, qawarnings=None):
self.fails = {}
self.warns = {}
+ self.qacats = qacats
+ self.qawarnings = qawarnings
def add_error(self, detected_qa, info):
'''Add the Q/A error to the database of detected problems
@@ -18,7 +18,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qacats list
@param info: string, details of the detected problem
'''
- if detected_qa not in qacats:
+ if detected_qa not in self.qacats:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_error(): %s, %s' % (detected_qa, info))
@@ -34,7 +34,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qawarnings list
@param info: string, details of the detected problem
'''
- if detected_qa not in qawarnings:
+ if detected_qa not in self.qawarnings:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_warning(): %s, %s' % (detected_qa, info))
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index 39f53c180..cd917828d 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -27,7 +27,7 @@ class RepoSettings(object):
def __init__(
self, config_root, portdir, portdir_overlay,
repoman_settings=None, vcs_settings=None, options=None,
- qawarnings=None):
+ qadata=None):
self.config_root = config_root
self.repoman_settings = repoman_settings
self.vcs_settings = vcs_settings
@@ -41,6 +41,14 @@ class RepoSettings(object):
except KeyError:
self._add_repo(config_root, portdir_overlay)
+ logging.debug("RepoSettings: init(); load qadata")
+ # load the repo specific configuration
+ self.qadata = qadata
+ if not self.qadata.load_repo_config(self.repodir, options):
+ logging.error("Aborting...")
+ sys.exit(1)
+ logging.debug("RepoSettings: qadata loaded", qadata.no_exec)
+
self.root = self.repoman_settings['EROOT']
self.trees = {
self.root: {'porttree': portage.portagetree(settings=self.repoman_settings)}
@@ -60,7 +68,7 @@ class RepoSettings(object):
del self.repositories[repo.name]
if self.repo_config.allow_provide_virtual:
- qawarnings.add("virtual.oldstyle")
+ qadata.qawarnings.add("virtual.oldstyle")
if self.repo_config.sign_commit and options.mode in ("commit", "fix", "manifest"):
if vcs_settings.vcs:
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-06-27 22:10 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-06-27 22:10 UTC (permalink / raw
To: gentoo-commits
commit: 1c0dd942127f141b493513674a97089b69142b48
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:38:48 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Tue Jun 27 22:05:07 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=1c0dd942
qa_data.py: Initial move of all configurable data to the repo
All this data was ported to a metadata/repoman/qa_data.yml file.
repoman/pym/repoman/qa_data.py | 433 ++++++++---------------------------------
1 file changed, 77 insertions(+), 356 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index fed798f9f..c2721d94e 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -1,369 +1,90 @@
# -*- coding:utf-8 -*-
import logging
+import os
+import yaml
from _emerge.Package import Package
# import our initialized portage instance
from repoman._portage import portage
-max_desc_len = 80
-allowed_filename_chars = "a-zA-Z0-9._-+:"
-qahelp = {
- "CVS/Entries.IO_error": (
- "Attempting to commit, and an IO error was encountered access the"
- " Entries file"),
- "ebuild.invalidname": (
- "Ebuild files with a non-parseable or syntactically incorrect name"
- " (or using 2.1 versioning extensions)"),
- "ebuild.namenomatch": (
- "Ebuild files that do not have the same name as their parent"
- " directory"),
- "changelog.ebuildadded": (
- "An ebuild was added but the ChangeLog was not modified"),
- "changelog.missing": (
- "Missing ChangeLog files"),
- "ebuild.notadded": (
- "Ebuilds that exist but have not been added to cvs"),
- "ebuild.patches": (
- "PATCHES variable should be a bash array to ensure white space safety"),
- "changelog.notadded": (
- "ChangeLogs that exist but have not been added to cvs"),
- "dependency.bad": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds)"),
- "dependency.badmasked": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds)"),
- "dependency.badindev": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds) in developing arch"),
- "dependency.badmaskedindev": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds) in developing arch"),
- "dependency.badtilde": (
- "Uses the ~ dep operator with a non-zero revision part,"
- " which is useless (the revision is ignored)"),
- "dependency.missingslot": (
- "RDEPEND matches more than one SLOT but does not specify a "
- "slot and/or use the := or :* slot operator"),
- "dependency.perlcore": (
- "This ebuild directly depends on a package in perl-core;"
- " it should use the corresponding virtual instead."),
- "dependency.syntax": (
- "Syntax error in dependency string"
- " (usually an extra/missing space/parenthesis)"),
- "dependency.unknown": (
- "Ebuild has a dependency that refers to an unknown package"
- " (which may be valid if it is a blocker for a renamed/removed package,"
- " or is an alternative choice provided by an overlay)"),
- "dependency.badslotop": (
- "RDEPEND contains ':=' slot operator under '||' dependency."),
- "file.executable": (
- "Ebuilds, digests, metadata.xml, Manifest, and ChangeLog do not need"
- " the executable bit"),
- "file.size": (
- "Files in the files directory must be under 20 KiB"),
- "file.size.fatal": (
- "Files in the files directory must be under 60 KiB"),
- "file.empty": (
- "Empty file in the files directory"),
- "file.name": (
- "File/dir name must be composed"
- " of only the following chars: %s " % allowed_filename_chars),
- "file.UTF8": (
- "File is not UTF8 compliant"),
- "inherit.deprecated": (
- "Ebuild inherits a deprecated eclass"),
- "inherit.missing": (
- "Ebuild uses functions from an eclass but does not inherit it"),
- "inherit.unused": (
- "Ebuild inherits an eclass but does not use it"),
- "java.eclassesnotused": (
- "With virtual/jdk in DEPEND you must inherit a java eclass"),
- "wxwidgets.eclassnotused": (
- "Ebuild DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass"),
- "KEYWORDS.dropped": (
- "Ebuilds that appear to have dropped KEYWORDS for some arch"),
- "KEYWORDS.missing": (
- "Ebuilds that have a missing or empty KEYWORDS variable"),
- "KEYWORDS.stable": (
- "Ebuilds that have been added directly with stable KEYWORDS"),
- "KEYWORDS.stupid": (
- "Ebuilds that use KEYWORDS=-* instead of package.mask"),
- "LICENSE.missing": (
- "Ebuilds that have a missing or empty LICENSE variable"),
- "LICENSE.virtual": (
- "Virtuals that have a non-empty LICENSE variable"),
- "DESCRIPTION.missing": (
- "Ebuilds that have a missing or empty DESCRIPTION variable"),
- "DESCRIPTION.toolong": (
- "DESCRIPTION is over %d characters" % max_desc_len),
- "EAPI.definition": (
- "EAPI definition does not conform to PMS section 7.3.1"
- " (first non-comment, non-blank line)"),
- "EAPI.deprecated": (
- "Ebuilds that use features that are deprecated in the current EAPI"),
- "EAPI.incompatible": (
- "Ebuilds that use features that are only available with a different"
- " EAPI"),
- "EAPI.unsupported": (
- "Ebuilds that have an unsupported EAPI version"
- " (you must upgrade portage)"),
- "SLOT.invalid": (
- "Ebuilds that have a missing or invalid SLOT variable value"),
- "HOMEPAGE.missing": (
- "Ebuilds that have a missing or empty HOMEPAGE variable"),
- "HOMEPAGE.virtual": (
- "Virtuals that have a non-empty HOMEPAGE variable"),
- "HOMEPAGE.missingurischeme": (
- "HOMEPAGE is missing an URI scheme"),
- "PDEPEND.suspect": (
- "PDEPEND contains a package that usually only belongs in DEPEND."),
- "LICENSE.syntax": (
- "Syntax error in LICENSE"
- " (usually an extra/missing space/parenthesis)"),
- "PROVIDE.syntax": (
- "Syntax error in PROVIDE"
- " (usually an extra/missing space/parenthesis)"),
- "PROPERTIES.syntax": (
- "Syntax error in PROPERTIES"
- " (usually an extra/missing space/parenthesis)"),
- "RESTRICT.syntax": (
- "Syntax error in RESTRICT"
- " (usually an extra/missing space/parenthesis)"),
- "REQUIRED_USE.syntax": (
- "Syntax error in REQUIRED_USE"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.syntax": (
- "Syntax error in SRC_URI"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.mirror": (
- "A uri listed in profiles/thirdpartymirrors is found in SRC_URI"),
- "ebuild.syntax": (
- "Error generating cache entry for ebuild;"
- " typically caused by ebuild syntax error"
- " or digest verification failure"),
- "ebuild.output": (
- "A simple sourcing of the ebuild produces output;"
- " this breaks ebuild policy."),
- "ebuild.nesteddie": (
- "Placing 'die' inside ( ) prints an error,"
- " but doesn't stop the ebuild."),
- "variable.invalidchar": (
- "A variable contains an invalid character"
- " that is not part of the ASCII character set"),
- "variable.readonly": (
- "Assigning a readonly variable"),
- "variable.usedwithhelpers": (
- "Ebuild uses D, ROOT, ED, EROOT or EPREFIX with helpers"),
- "LIVEVCS.stable": (
- "This ebuild is a live checkout from a VCS but has stable keywords."),
- "LIVEVCS.unmasked": (
- "This ebuild is a live checkout from a VCS but has keywords"
- " and is not masked in the global package.mask."),
- "IUSE.invalid": (
- "This ebuild has a variable in IUSE"
- " that is not in the use.desc or its metadata.xml file"),
- "IUSE.missing": (
- "This ebuild has a USE conditional"
- " which references a flag that is not listed in IUSE"),
- "IUSE.rubydeprecated": (
- "The ebuild has set a ruby interpreter in USE_RUBY,"
- " that is not available as a ruby target anymore"),
- "LICENSE.invalid": (
- "This ebuild is listing a license"
- " that doesnt exist in portages license/ dir."),
- "LICENSE.deprecated": (
- "This ebuild is listing a deprecated license."),
- "KEYWORDS.invalid": (
- "This ebuild contains KEYWORDS"
- " that are not listed in profiles/arch.list"
- " or for which no valid profile was found"),
- "RDEPEND.implicit": (
- "RDEPEND is unset in the ebuild"
- " which triggers implicit RDEPEND=$DEPEND assignment"
- " (prior to EAPI 4)"),
- "RDEPEND.suspect": (
- "RDEPEND contains a package that usually only belongs in DEPEND."),
- "RESTRICT.invalid": (
- "This ebuild contains invalid RESTRICT values."),
- "digest.assumed": (
- "Existing digest must be assumed correct (Package level only)"),
- "digest.missing": (
- "Some files listed in SRC_URI aren't referenced in the Manifest"),
- "digest.unused": (
- "Some files listed in the Manifest aren't referenced in SRC_URI"),
- "ebuild.absdosym": (
- "This ebuild uses absolute target to dosym where relative symlink"
- " could be used instead"),
- "ebuild.majorsyn": (
- "This ebuild has a major syntax error"
- " that may cause the ebuild to fail partially or fully"),
- "ebuild.minorsyn": (
- "This ebuild has a minor syntax error"
- " that contravenes gentoo coding style"),
- "ebuild.badheader": (
- "This ebuild has a malformed header"),
- "manifest.bad": (
- "Manifest has missing or incorrect digests"),
- "metadata.missing": (
- "Missing metadata.xml files"),
- "metadata.bad": (
- "Bad metadata.xml files"),
- "metadata.warning": (
- "Warnings in metadata.xml files"),
- "portage.internal": (
- "The ebuild uses an internal Portage function or variable"),
- "repo.eapi.banned": (
- "The ebuild uses an EAPI which is"
- " banned by the repository's metadata/layout.conf settings"),
- "repo.eapi.deprecated": (
- "The ebuild uses an EAPI which is"
- " deprecated by the repository's metadata/layout.conf settings"),
- "virtual.oldstyle": (
- "The ebuild PROVIDEs an old-style virtual (see GLEP 37)"),
- "virtual.suspect": (
- "Ebuild contains a package"
- " that usually should be pulled via virtual/, not directly."),
- "usage.obsolete": (
- "The ebuild makes use of an obsolete construct"),
- "upstream.workaround": (
- "The ebuild works around an upstream bug,"
- " an upstream bug should be filed and tracked in bugs.gentoo.org"),
- "uri.https": "URI uses http:// but should use https://",
-}
-
-qacats = list(qahelp)
-qacats.sort()
-
-qawarnings = set((
- "changelog.missing",
- "changelog.notadded",
- "dependency.unknown",
- "digest.assumed",
- "digest.unused",
- "ebuild.notadded",
- "ebuild.nesteddie",
- "dependency.badmasked",
- "dependency.badindev",
- "dependency.badmaskedindev",
- "dependency.badtilde",
- "dependency.missingslot",
- "dependency.perlcore",
- "DESCRIPTION.toolong",
- "EAPI.deprecated",
- "HOMEPAGE.virtual",
- "LICENSE.deprecated",
- "LICENSE.virtual",
- "KEYWORDS.dropped",
- "KEYWORDS.stupid",
- "KEYWORDS.missing",
- "PDEPEND.suspect",
- "RDEPEND.implicit",
- "RDEPEND.suspect",
- "virtual.suspect",
- "RESTRICT.invalid",
- "ebuild.absdosym",
- "ebuild.minorsyn",
- "ebuild.badheader",
- "ebuild.patches",
- "file.empty",
- "file.size",
- "inherit.unused",
- "inherit.deprecated",
- "java.eclassesnotused",
- "wxwidgets.eclassnotused",
- "metadata.warning",
- "portage.internal",
- "repo.eapi.deprecated",
- "usage.obsolete",
- "upstream.workaround",
- "IUSE.rubydeprecated",
- "uri.https",
-))
-
-
-missingvars = ["KEYWORDS", "LICENSE", "DESCRIPTION", "HOMEPAGE"]
-allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
-allvars.update(Package.metadata_keys)
-allvars = sorted(allvars)
-
-for x in missingvars:
- x += ".missing"
- if x not in qacats:
- logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
- qacats.append(x)
- qawarnings.add(x)
-
-valid_restrict = frozenset([
- "binchecks", "bindist", "fetch", "installsources", "mirror",
- "preserve-libs", "primaryuri", "splitdebug", "strip", "test", "userpriv"])
-
-
-suspect_rdepend = frozenset([
- "app-arch/cabextract",
- "app-arch/rpm2targz",
- "app-doc/doxygen",
- "dev-lang/nasm",
- "dev-lang/swig",
- "dev-lang/yasm",
- "dev-perl/extutils-pkgconfig",
- "dev-qt/linguist-tools",
- "dev-util/byacc",
- "dev-util/cmake",
- "dev-util/ftjam",
- "dev-util/gperf",
- "dev-util/gtk-doc",
- "dev-util/gtk-doc-am",
- "dev-util/intltool",
- "dev-util/jam",
- "dev-util/pkg-config-lite",
- "dev-util/pkgconf",
- "dev-util/pkgconfig",
- "dev-util/pkgconfig-openbsd",
- "dev-util/scons",
- "dev-util/unifdef",
- "dev-util/yacc",
- "media-gfx/ebdftopcf",
- "sys-apps/help2man",
- "sys-devel/autoconf",
- "sys-devel/automake",
- "sys-devel/bin86",
- "sys-devel/bison",
- "sys-devel/dev86",
- "sys-devel/flex",
- "sys-devel/m4",
- "sys-devel/pmake",
- "virtual/linux-sources",
- "virtual/linuxtv-dvb-headers",
- "virtual/os-headers",
- "virtual/pkgconfig",
- "x11-misc/bdftopcf",
- "x11-misc/imake",
-])
-
-suspect_virtual = {
- "dev-util/pkg-config-lite": "virtual/pkgconfig",
- "dev-util/pkgconf": "virtual/pkgconfig",
- "dev-util/pkgconfig": "virtual/pkgconfig",
- "dev-util/pkgconfig-openbsd": "virtual/pkgconfig",
- "dev-libs/libusb": "virtual/libusb",
- "dev-libs/libusbx": "virtual/libusb",
- "dev-libs/libusb-compat": "virtual/libusb",
-}
-
-ruby_deprecated = frozenset([
- "ruby_targets_ree18",
- "ruby_targets_ruby18",
- "ruby_targets_ruby19",
- "ruby_targets_ruby20",
-])
-
-
-# file.executable
-no_exec = frozenset(["Manifest", "ChangeLog", "metadata.xml"])
+class QAData(object):
+
+ def __init__(self):
+ # Create the main exported data variables
+ self.max_desc_len = None
+ self.allowed_filename_chars = None
+ self.qahelp = None
+ self.qacats = None
+ self.qawarnings = None
+ self.missingvars = None
+ self.allvars = None
+ self.valid_restrict = None
+ self.suspect_rdepend = None
+ self.suspect_virtual = None
+ self.ruby_deprecated = None
+ self.no_exec = None
+
+
+ def load_repo_config(self, repopath, options):
+ '''Load the repository repoman qa_data.yml config
+
+ @param repopath: string, The path of the repository being scanned
+ This could be a parent repository using the
+ repoman_masters layout.conf variable
+ '''
+ filepath = os.path.join(repopath, 'metadata/repoman/qa_data.yml')
+ logging.debug("QAData: reading file", filepath)
+ try:
+ with open(filepath, 'r') as qadata_file:
+ qadata = yaml.safe_load(qadata_file.read())
+ except IOError as error:
+ logging.error("Failed to load 'qa_data.yml' file at path: %s", filepath)
+ logging.eception(error)
+ return False
+ self.max_desc_len = qadata.get('max_description_length', 80)
+ self.allowed_filename_chars = qadata.get("allowed_filename_chars", "a-zA-Z0-9._-+:")
+
+ self.qahelp = qadata.get('qahelp', {})
+
+ self.qacats = []
+ for x in sorted(self.qahelp):
+ for y in sorted(self.qahelp[x]):
+ self.qacats.append('.'.join([x, y]))
+ self.qacats.sort()
+
+ self.qawarnings = set(qadata.get('qawarnings', []))
+ if options.experimental_inherit == 'y':
+ # This is experimental, so it's non-fatal.
+ self.qawarnings.add("inherit.missing")
+
+ self.missingvars = qadata.get("missingvars", [])
+ logging.debug("QAData: missingvars", self.missingvars)
+ self.allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
+ self.allvars.update(Package.metadata_keys)
+ self.allvars = sorted(self.allvars)
+
+ for x in self.missingvars:
+ x += ".missing"
+ if x not in self.qacats:
+ logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
+ self.qacats.append(x)
+ self.qawarnings.add(x)
+
+ self.valid_restrict = frozenset(qadata.get("valid_restrict", []))
+
+ self.suspect_rdepend = frozenset(qadata.get("suspect_rdepend", []))
+
+ self.suspect_virtual = qadata.get("suspect_virtual", {})
+
+ self.ruby_deprecated = frozenset(qadata.get("ruby_deprecated", []))
+
+ # file.executable
+ self.no_exec = frozenset(qadata.get("no_exec_files", []))
+ logging.debug("QAData: completed loading file: %s", filepath)
+ return True
def format_qa_output(
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 17:52 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 17:52 UTC (permalink / raw
To: gentoo-commits
commit: 9bfffe319e3d199a77255813dde55b1dca68a587
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:38:48 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 17:48:03 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=9bfffe31
qa_data.py: Initial move of all configurable data to the repo
All this data was ported to a metadata/repoman/qa_data.yml file.
repoman/pym/repoman/qa_data.py | 433 ++++++++---------------------------------
1 file changed, 77 insertions(+), 356 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index fed798f9f..fed556628 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -1,369 +1,90 @@
# -*- coding:utf-8 -*-
import logging
+import os
+import yaml
from _emerge.Package import Package
# import our initialized portage instance
from repoman._portage import portage
-max_desc_len = 80
-allowed_filename_chars = "a-zA-Z0-9._-+:"
-qahelp = {
- "CVS/Entries.IO_error": (
- "Attempting to commit, and an IO error was encountered access the"
- " Entries file"),
- "ebuild.invalidname": (
- "Ebuild files with a non-parseable or syntactically incorrect name"
- " (or using 2.1 versioning extensions)"),
- "ebuild.namenomatch": (
- "Ebuild files that do not have the same name as their parent"
- " directory"),
- "changelog.ebuildadded": (
- "An ebuild was added but the ChangeLog was not modified"),
- "changelog.missing": (
- "Missing ChangeLog files"),
- "ebuild.notadded": (
- "Ebuilds that exist but have not been added to cvs"),
- "ebuild.patches": (
- "PATCHES variable should be a bash array to ensure white space safety"),
- "changelog.notadded": (
- "ChangeLogs that exist but have not been added to cvs"),
- "dependency.bad": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds)"),
- "dependency.badmasked": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds)"),
- "dependency.badindev": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds) in developing arch"),
- "dependency.badmaskedindev": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds) in developing arch"),
- "dependency.badtilde": (
- "Uses the ~ dep operator with a non-zero revision part,"
- " which is useless (the revision is ignored)"),
- "dependency.missingslot": (
- "RDEPEND matches more than one SLOT but does not specify a "
- "slot and/or use the := or :* slot operator"),
- "dependency.perlcore": (
- "This ebuild directly depends on a package in perl-core;"
- " it should use the corresponding virtual instead."),
- "dependency.syntax": (
- "Syntax error in dependency string"
- " (usually an extra/missing space/parenthesis)"),
- "dependency.unknown": (
- "Ebuild has a dependency that refers to an unknown package"
- " (which may be valid if it is a blocker for a renamed/removed package,"
- " or is an alternative choice provided by an overlay)"),
- "dependency.badslotop": (
- "RDEPEND contains ':=' slot operator under '||' dependency."),
- "file.executable": (
- "Ebuilds, digests, metadata.xml, Manifest, and ChangeLog do not need"
- " the executable bit"),
- "file.size": (
- "Files in the files directory must be under 20 KiB"),
- "file.size.fatal": (
- "Files in the files directory must be under 60 KiB"),
- "file.empty": (
- "Empty file in the files directory"),
- "file.name": (
- "File/dir name must be composed"
- " of only the following chars: %s " % allowed_filename_chars),
- "file.UTF8": (
- "File is not UTF8 compliant"),
- "inherit.deprecated": (
- "Ebuild inherits a deprecated eclass"),
- "inherit.missing": (
- "Ebuild uses functions from an eclass but does not inherit it"),
- "inherit.unused": (
- "Ebuild inherits an eclass but does not use it"),
- "java.eclassesnotused": (
- "With virtual/jdk in DEPEND you must inherit a java eclass"),
- "wxwidgets.eclassnotused": (
- "Ebuild DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass"),
- "KEYWORDS.dropped": (
- "Ebuilds that appear to have dropped KEYWORDS for some arch"),
- "KEYWORDS.missing": (
- "Ebuilds that have a missing or empty KEYWORDS variable"),
- "KEYWORDS.stable": (
- "Ebuilds that have been added directly with stable KEYWORDS"),
- "KEYWORDS.stupid": (
- "Ebuilds that use KEYWORDS=-* instead of package.mask"),
- "LICENSE.missing": (
- "Ebuilds that have a missing or empty LICENSE variable"),
- "LICENSE.virtual": (
- "Virtuals that have a non-empty LICENSE variable"),
- "DESCRIPTION.missing": (
- "Ebuilds that have a missing or empty DESCRIPTION variable"),
- "DESCRIPTION.toolong": (
- "DESCRIPTION is over %d characters" % max_desc_len),
- "EAPI.definition": (
- "EAPI definition does not conform to PMS section 7.3.1"
- " (first non-comment, non-blank line)"),
- "EAPI.deprecated": (
- "Ebuilds that use features that are deprecated in the current EAPI"),
- "EAPI.incompatible": (
- "Ebuilds that use features that are only available with a different"
- " EAPI"),
- "EAPI.unsupported": (
- "Ebuilds that have an unsupported EAPI version"
- " (you must upgrade portage)"),
- "SLOT.invalid": (
- "Ebuilds that have a missing or invalid SLOT variable value"),
- "HOMEPAGE.missing": (
- "Ebuilds that have a missing or empty HOMEPAGE variable"),
- "HOMEPAGE.virtual": (
- "Virtuals that have a non-empty HOMEPAGE variable"),
- "HOMEPAGE.missingurischeme": (
- "HOMEPAGE is missing an URI scheme"),
- "PDEPEND.suspect": (
- "PDEPEND contains a package that usually only belongs in DEPEND."),
- "LICENSE.syntax": (
- "Syntax error in LICENSE"
- " (usually an extra/missing space/parenthesis)"),
- "PROVIDE.syntax": (
- "Syntax error in PROVIDE"
- " (usually an extra/missing space/parenthesis)"),
- "PROPERTIES.syntax": (
- "Syntax error in PROPERTIES"
- " (usually an extra/missing space/parenthesis)"),
- "RESTRICT.syntax": (
- "Syntax error in RESTRICT"
- " (usually an extra/missing space/parenthesis)"),
- "REQUIRED_USE.syntax": (
- "Syntax error in REQUIRED_USE"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.syntax": (
- "Syntax error in SRC_URI"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.mirror": (
- "A uri listed in profiles/thirdpartymirrors is found in SRC_URI"),
- "ebuild.syntax": (
- "Error generating cache entry for ebuild;"
- " typically caused by ebuild syntax error"
- " or digest verification failure"),
- "ebuild.output": (
- "A simple sourcing of the ebuild produces output;"
- " this breaks ebuild policy."),
- "ebuild.nesteddie": (
- "Placing 'die' inside ( ) prints an error,"
- " but doesn't stop the ebuild."),
- "variable.invalidchar": (
- "A variable contains an invalid character"
- " that is not part of the ASCII character set"),
- "variable.readonly": (
- "Assigning a readonly variable"),
- "variable.usedwithhelpers": (
- "Ebuild uses D, ROOT, ED, EROOT or EPREFIX with helpers"),
- "LIVEVCS.stable": (
- "This ebuild is a live checkout from a VCS but has stable keywords."),
- "LIVEVCS.unmasked": (
- "This ebuild is a live checkout from a VCS but has keywords"
- " and is not masked in the global package.mask."),
- "IUSE.invalid": (
- "This ebuild has a variable in IUSE"
- " that is not in the use.desc or its metadata.xml file"),
- "IUSE.missing": (
- "This ebuild has a USE conditional"
- " which references a flag that is not listed in IUSE"),
- "IUSE.rubydeprecated": (
- "The ebuild has set a ruby interpreter in USE_RUBY,"
- " that is not available as a ruby target anymore"),
- "LICENSE.invalid": (
- "This ebuild is listing a license"
- " that doesnt exist in portages license/ dir."),
- "LICENSE.deprecated": (
- "This ebuild is listing a deprecated license."),
- "KEYWORDS.invalid": (
- "This ebuild contains KEYWORDS"
- " that are not listed in profiles/arch.list"
- " or for which no valid profile was found"),
- "RDEPEND.implicit": (
- "RDEPEND is unset in the ebuild"
- " which triggers implicit RDEPEND=$DEPEND assignment"
- " (prior to EAPI 4)"),
- "RDEPEND.suspect": (
- "RDEPEND contains a package that usually only belongs in DEPEND."),
- "RESTRICT.invalid": (
- "This ebuild contains invalid RESTRICT values."),
- "digest.assumed": (
- "Existing digest must be assumed correct (Package level only)"),
- "digest.missing": (
- "Some files listed in SRC_URI aren't referenced in the Manifest"),
- "digest.unused": (
- "Some files listed in the Manifest aren't referenced in SRC_URI"),
- "ebuild.absdosym": (
- "This ebuild uses absolute target to dosym where relative symlink"
- " could be used instead"),
- "ebuild.majorsyn": (
- "This ebuild has a major syntax error"
- " that may cause the ebuild to fail partially or fully"),
- "ebuild.minorsyn": (
- "This ebuild has a minor syntax error"
- " that contravenes gentoo coding style"),
- "ebuild.badheader": (
- "This ebuild has a malformed header"),
- "manifest.bad": (
- "Manifest has missing or incorrect digests"),
- "metadata.missing": (
- "Missing metadata.xml files"),
- "metadata.bad": (
- "Bad metadata.xml files"),
- "metadata.warning": (
- "Warnings in metadata.xml files"),
- "portage.internal": (
- "The ebuild uses an internal Portage function or variable"),
- "repo.eapi.banned": (
- "The ebuild uses an EAPI which is"
- " banned by the repository's metadata/layout.conf settings"),
- "repo.eapi.deprecated": (
- "The ebuild uses an EAPI which is"
- " deprecated by the repository's metadata/layout.conf settings"),
- "virtual.oldstyle": (
- "The ebuild PROVIDEs an old-style virtual (see GLEP 37)"),
- "virtual.suspect": (
- "Ebuild contains a package"
- " that usually should be pulled via virtual/, not directly."),
- "usage.obsolete": (
- "The ebuild makes use of an obsolete construct"),
- "upstream.workaround": (
- "The ebuild works around an upstream bug,"
- " an upstream bug should be filed and tracked in bugs.gentoo.org"),
- "uri.https": "URI uses http:// but should use https://",
-}
-
-qacats = list(qahelp)
-qacats.sort()
-
-qawarnings = set((
- "changelog.missing",
- "changelog.notadded",
- "dependency.unknown",
- "digest.assumed",
- "digest.unused",
- "ebuild.notadded",
- "ebuild.nesteddie",
- "dependency.badmasked",
- "dependency.badindev",
- "dependency.badmaskedindev",
- "dependency.badtilde",
- "dependency.missingslot",
- "dependency.perlcore",
- "DESCRIPTION.toolong",
- "EAPI.deprecated",
- "HOMEPAGE.virtual",
- "LICENSE.deprecated",
- "LICENSE.virtual",
- "KEYWORDS.dropped",
- "KEYWORDS.stupid",
- "KEYWORDS.missing",
- "PDEPEND.suspect",
- "RDEPEND.implicit",
- "RDEPEND.suspect",
- "virtual.suspect",
- "RESTRICT.invalid",
- "ebuild.absdosym",
- "ebuild.minorsyn",
- "ebuild.badheader",
- "ebuild.patches",
- "file.empty",
- "file.size",
- "inherit.unused",
- "inherit.deprecated",
- "java.eclassesnotused",
- "wxwidgets.eclassnotused",
- "metadata.warning",
- "portage.internal",
- "repo.eapi.deprecated",
- "usage.obsolete",
- "upstream.workaround",
- "IUSE.rubydeprecated",
- "uri.https",
-))
-
-
-missingvars = ["KEYWORDS", "LICENSE", "DESCRIPTION", "HOMEPAGE"]
-allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
-allvars.update(Package.metadata_keys)
-allvars = sorted(allvars)
-
-for x in missingvars:
- x += ".missing"
- if x not in qacats:
- logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
- qacats.append(x)
- qawarnings.add(x)
-
-valid_restrict = frozenset([
- "binchecks", "bindist", "fetch", "installsources", "mirror",
- "preserve-libs", "primaryuri", "splitdebug", "strip", "test", "userpriv"])
-
-
-suspect_rdepend = frozenset([
- "app-arch/cabextract",
- "app-arch/rpm2targz",
- "app-doc/doxygen",
- "dev-lang/nasm",
- "dev-lang/swig",
- "dev-lang/yasm",
- "dev-perl/extutils-pkgconfig",
- "dev-qt/linguist-tools",
- "dev-util/byacc",
- "dev-util/cmake",
- "dev-util/ftjam",
- "dev-util/gperf",
- "dev-util/gtk-doc",
- "dev-util/gtk-doc-am",
- "dev-util/intltool",
- "dev-util/jam",
- "dev-util/pkg-config-lite",
- "dev-util/pkgconf",
- "dev-util/pkgconfig",
- "dev-util/pkgconfig-openbsd",
- "dev-util/scons",
- "dev-util/unifdef",
- "dev-util/yacc",
- "media-gfx/ebdftopcf",
- "sys-apps/help2man",
- "sys-devel/autoconf",
- "sys-devel/automake",
- "sys-devel/bin86",
- "sys-devel/bison",
- "sys-devel/dev86",
- "sys-devel/flex",
- "sys-devel/m4",
- "sys-devel/pmake",
- "virtual/linux-sources",
- "virtual/linuxtv-dvb-headers",
- "virtual/os-headers",
- "virtual/pkgconfig",
- "x11-misc/bdftopcf",
- "x11-misc/imake",
-])
-
-suspect_virtual = {
- "dev-util/pkg-config-lite": "virtual/pkgconfig",
- "dev-util/pkgconf": "virtual/pkgconfig",
- "dev-util/pkgconfig": "virtual/pkgconfig",
- "dev-util/pkgconfig-openbsd": "virtual/pkgconfig",
- "dev-libs/libusb": "virtual/libusb",
- "dev-libs/libusbx": "virtual/libusb",
- "dev-libs/libusb-compat": "virtual/libusb",
-}
-
-ruby_deprecated = frozenset([
- "ruby_targets_ree18",
- "ruby_targets_ruby18",
- "ruby_targets_ruby19",
- "ruby_targets_ruby20",
-])
-
-
-# file.executable
-no_exec = frozenset(["Manifest", "ChangeLog", "metadata.xml"])
+class QAData(object):
+
+ def __init__(self):
+ # Create the main exported data variables
+ self.max_desc_len = None
+ self.allowed_filename_chars = None
+ self.qahelp = None
+ self.qacats = None
+ self.qawarnings = None
+ self.missingvars = None
+ self.allvars = None
+ self.valid_restrict = None
+ self.suspect_rdepend = None
+ self.suspect_virtual = None
+ self.ruby_deprecated = None
+ self.no_exec = None
+
+
+ def load_repo_config(self, repopath, options):
+ '''Load the repository repoman qa_data.yml config
+
+ @param repopath: string, The path of the repository being scanned
+ This could be a parent repository using the
+ repoman_masters layout.conf variable
+ '''
+ filepath = os.path.join(repopath, 'metadata/repoman/qa_data.yml')
+ logging.debug("QAData: reading file: %s", filepath)
+ try:
+ with open(filepath, 'r') as qadata_file:
+ qadata = yaml.safe_load(qadata_file.read())
+ except IOError as error:
+ logging.error("Failed to load 'qa_data.yml' file at path: %s", filepath)
+ logging.eception(error)
+ return False
+ self.max_desc_len = qadata.get('max_description_length', 80)
+ self.allowed_filename_chars = qadata.get("allowed_filename_chars", "a-zA-Z0-9._-+:")
+
+ self.qahelp = qadata.get('qahelp', {})
+
+ self.qacats = []
+ for x in sorted(self.qahelp):
+ for y in sorted(self.qahelp[x]):
+ self.qacats.append('.'.join([x, y]))
+ self.qacats.sort()
+
+ self.qawarnings = set(qadata.get('qawarnings', []))
+ if options.experimental_inherit == 'y':
+ # This is experimental, so it's non-fatal.
+ self.qawarnings.add("inherit.missing")
+
+ self.missingvars = qadata.get("missingvars", [])
+ logging.debug("QAData: missingvars: %s", self.missingvars)
+ self.allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
+ self.allvars.update(Package.metadata_keys)
+ self.allvars = sorted(self.allvars)
+
+ for x in self.missingvars:
+ x += ".missing"
+ if x not in self.qacats:
+ logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
+ self.qacats.append(x)
+ self.qawarnings.add(x)
+
+ self.valid_restrict = frozenset(qadata.get("valid_restrict", []))
+
+ self.suspect_rdepend = frozenset(qadata.get("suspect_rdepend", []))
+
+ self.suspect_virtual = qadata.get("suspect_virtual", {})
+
+ self.ruby_deprecated = frozenset(qadata.get("ruby_deprecated", []))
+
+ # file.executable
+ self.no_exec = frozenset(qadata.get("no_exec_files", []))
+ logging.debug("QAData: completed loading file: %s", filepath)
+ return True
def format_qa_output(
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 17:52 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 17:52 UTC (permalink / raw
To: gentoo-commits
commit: e89f107592c1a97647a77b60285e0420db926001
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:49:01 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 17:50:08 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=e89f1075
repoman: Move qa_data initialization and loading
Move the new QAData class instance init to repoman_main().
parse_args() remove unused qahelp parameter.
qa_tracker.py: Add default qacats and qawarnings parameters as None.
These will be assigned later due to circular init references.
repos.py: Perform the QAData class loading and complete intialization
assignments.
repoman/pym/repoman/argparser.py | 3 +--
repoman/pym/repoman/main.py | 26 ++++++++++++++------------
repoman/pym/repoman/qa_tracker.py | 10 +++++-----
repoman/pym/repoman/repos.py | 12 ++++++++++--
4 files changed, 30 insertions(+), 21 deletions(-)
diff --git a/repoman/pym/repoman/argparser.py b/repoman/pym/repoman/argparser.py
index 2d56a87e6..68701378f 100644
--- a/repoman/pym/repoman/argparser.py
+++ b/repoman/pym/repoman/argparser.py
@@ -15,11 +15,10 @@ from portage import _unicode_decode
from portage import util
-def parse_args(argv, qahelp, repoman_default_opts):
+def parse_args(argv, repoman_default_opts):
"""Use a customized optionParser to parse command line arguments for repoman
Args:
argv - a sequence of command line arguments
- qahelp - a dict of qa warning to help message
Returns:
(opts, args), just like a call to parser.parse_args()
"""
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100755
new mode 100644
index ccc735c7d..3b628de00
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -29,9 +29,8 @@ from portage.util.futures.extendedfutures import (
from repoman.actions import Actions
from repoman.argparser import parse_args
-from repoman.qa_data import (
- format_qa_output, format_qa_output_column, qahelp,
- qawarnings, qacats)
+from repoman.qa_data import QAData
+from repoman.qa_data import format_qa_output, format_qa_output_column
from repoman.repos import RepoSettings
from repoman.scanner import Scanner
from repoman import utilities
@@ -60,7 +59,7 @@ def repoman_main(argv):
nocolor()
options, arguments = parse_args(
- sys.argv, qahelp, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
+ sys.argv, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
if options.version:
print("Repoman", VERSION, "(portage-%s)" % portage.VERSION)
@@ -73,10 +72,6 @@ def repoman_main(argv):
else:
logger.setLevel(LOGLEVEL)
- if options.experimental_inherit == 'y':
- # This is experimental, so it's non-fatal.
- qawarnings.add("inherit.missing")
-
# Set this to False when an extraordinary issue (generally
# something other than a QA issue) makes it impossible to
# commit (like if Manifest generation fails).
@@ -91,14 +86,21 @@ def repoman_main(argv):
# avoid a circular parameter repo_settings
vcs_settings = VCSSettings(options, repoman_settings)
+ qadata = QAData()
+ logging.debug("repoman_main: RepoSettings init")
repo_settings = RepoSettings(
config_root, portdir, portdir_overlay,
- repoman_settings, vcs_settings, options, qawarnings)
+ repoman_settings, vcs_settings, options, qadata)
repoman_settings = repo_settings.repoman_settings
# Now set repo_settings
vcs_settings.repo_settings = repo_settings
+ # set QATracker qacats, qawarnings
+ vcs_settings.qatracker.qacats = repo_settings.qadata.qacats
+ vcs_settings.qatracker.qawarnings = repo_settings.qadata.qawarnings
+ logging.debug("repoman_main: vcs_settings done")
+ logging.debug("repoman_main: qadata: %s", repo_settings.qadata)
if 'digest' in repoman_settings.features and options.digest != 'n':
options.digest = 'y'
@@ -133,11 +135,11 @@ def repoman_main(argv):
if options.mode == "manifest":
sys.exit(result['fail'])
- for x in qacats:
+ for x in qadata.qacats:
if x not in vcs_settings.qatracker.fails:
continue
result['warn'] = 1
- if x not in qawarnings:
+ if x not in qadata.qawarnings:
result['fail'] = 1
if result['fail'] or \
@@ -174,7 +176,7 @@ def repoman_main(argv):
format_output = format_outputs.get(
options.output_style, format_outputs['default'])
format_output(f, vcs_settings.qatracker.fails, result['full'],
- result['fail'], options, qawarnings)
+ result['fail'], options, qadata.qawarnings)
style_file.flush()
del console_writer, f, style_file
diff --git a/repoman/pym/repoman/qa_tracker.py b/repoman/pym/repoman/qa_tracker.py
index 9bfe0e241..faaf8e398 100644
--- a/repoman/pym/repoman/qa_tracker.py
+++ b/repoman/pym/repoman/qa_tracker.py
@@ -2,15 +2,15 @@
import logging
import sys
-from repoman.qa_data import qacats, qawarnings
-
class QATracker(object):
'''Track all occurrances of Q/A problems detected'''
- def __init__(self):
+ def __init__(self, qacats=None, qawarnings=None):
self.fails = {}
self.warns = {}
+ self.qacats = qacats
+ self.qawarnings = qawarnings
def add_error(self, detected_qa, info):
'''Add the Q/A error to the database of detected problems
@@ -18,7 +18,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qacats list
@param info: string, details of the detected problem
'''
- if detected_qa not in qacats:
+ if detected_qa not in self.qacats:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_error(): %s, %s' % (detected_qa, info))
@@ -34,7 +34,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qawarnings list
@param info: string, details of the detected problem
'''
- if detected_qa not in qawarnings:
+ if detected_qa not in self.qawarnings:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_warning(): %s, %s' % (detected_qa, info))
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index 39f53c180..accbf1d9c 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -27,7 +27,7 @@ class RepoSettings(object):
def __init__(
self, config_root, portdir, portdir_overlay,
repoman_settings=None, vcs_settings=None, options=None,
- qawarnings=None):
+ qadata=None):
self.config_root = config_root
self.repoman_settings = repoman_settings
self.vcs_settings = vcs_settings
@@ -41,6 +41,14 @@ class RepoSettings(object):
except KeyError:
self._add_repo(config_root, portdir_overlay)
+ logging.debug("RepoSettings: init(); load qadata")
+ # load the repo specific configuration
+ self.qadata = qadata
+ if not self.qadata.load_repo_config(self.repodir, options):
+ logging.error("Aborting...")
+ sys.exit(1)
+ logging.debug("RepoSettings: qadata loaded: %s", qadata.no_exec)
+
self.root = self.repoman_settings['EROOT']
self.trees = {
self.root: {'porttree': portage.portagetree(settings=self.repoman_settings)}
@@ -60,7 +68,7 @@ class RepoSettings(object):
del self.repositories[repo.name]
if self.repo_config.allow_provide_virtual:
- qawarnings.add("virtual.oldstyle")
+ qadata.qawarnings.add("virtual.oldstyle")
if self.repo_config.sign_commit and options.mode in ("commit", "fix", "manifest"):
if vcs_settings.vcs:
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 17:52 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 17:52 UTC (permalink / raw
To: gentoo-commits
commit: 3ef1e0f29556276d2a4f1d1fef2d5fa1e3568f4d
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 18:08:50 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 17:50:19 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=3ef1e0f2
repoman: main.py: Remove unused InvalidStateError import
repoman/pym/repoman/main.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100644
new mode 100755
index 3b628de00..c1e3b99fe
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -24,7 +24,6 @@ from portage.output import ConsoleStyleFile, StyleWriter
from portage.util import formatter
from portage.util.futures.extendedfutures import (
ExtendedFuture,
- InvalidStateError,
)
from repoman.actions import Actions
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 22:31 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 22:31 UTC (permalink / raw
To: gentoo-commits
commit: 143f939ff920c7ae62dde0f63f63885321f46584
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:49:01 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 22:29:33 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=143f939f
repoman: Move qa_data initialization and loading
Move the new QAData class instance init to repoman_main().
parse_args() remove unused qahelp parameter.
qa_tracker.py: Add default qacats and qawarnings parameters as None.
These will be assigned later due to circular init references.
repos.py: Perform the QAData class loading and complete intialization
assignments.
repoman/pym/repoman/argparser.py | 3 +--
repoman/pym/repoman/main.py | 26 ++++++++++++++------------
repoman/pym/repoman/qa_tracker.py | 10 +++++-----
repoman/pym/repoman/repos.py | 12 ++++++++++--
4 files changed, 30 insertions(+), 21 deletions(-)
diff --git a/repoman/pym/repoman/argparser.py b/repoman/pym/repoman/argparser.py
index 2d56a87e6..68701378f 100644
--- a/repoman/pym/repoman/argparser.py
+++ b/repoman/pym/repoman/argparser.py
@@ -15,11 +15,10 @@ from portage import _unicode_decode
from portage import util
-def parse_args(argv, qahelp, repoman_default_opts):
+def parse_args(argv, repoman_default_opts):
"""Use a customized optionParser to parse command line arguments for repoman
Args:
argv - a sequence of command line arguments
- qahelp - a dict of qa warning to help message
Returns:
(opts, args), just like a call to parser.parse_args()
"""
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100755
new mode 100644
index ccc735c7d..3b628de00
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -29,9 +29,8 @@ from portage.util.futures.extendedfutures import (
from repoman.actions import Actions
from repoman.argparser import parse_args
-from repoman.qa_data import (
- format_qa_output, format_qa_output_column, qahelp,
- qawarnings, qacats)
+from repoman.qa_data import QAData
+from repoman.qa_data import format_qa_output, format_qa_output_column
from repoman.repos import RepoSettings
from repoman.scanner import Scanner
from repoman import utilities
@@ -60,7 +59,7 @@ def repoman_main(argv):
nocolor()
options, arguments = parse_args(
- sys.argv, qahelp, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
+ sys.argv, repoman_settings.get("REPOMAN_DEFAULT_OPTS", ""))
if options.version:
print("Repoman", VERSION, "(portage-%s)" % portage.VERSION)
@@ -73,10 +72,6 @@ def repoman_main(argv):
else:
logger.setLevel(LOGLEVEL)
- if options.experimental_inherit == 'y':
- # This is experimental, so it's non-fatal.
- qawarnings.add("inherit.missing")
-
# Set this to False when an extraordinary issue (generally
# something other than a QA issue) makes it impossible to
# commit (like if Manifest generation fails).
@@ -91,14 +86,21 @@ def repoman_main(argv):
# avoid a circular parameter repo_settings
vcs_settings = VCSSettings(options, repoman_settings)
+ qadata = QAData()
+ logging.debug("repoman_main: RepoSettings init")
repo_settings = RepoSettings(
config_root, portdir, portdir_overlay,
- repoman_settings, vcs_settings, options, qawarnings)
+ repoman_settings, vcs_settings, options, qadata)
repoman_settings = repo_settings.repoman_settings
# Now set repo_settings
vcs_settings.repo_settings = repo_settings
+ # set QATracker qacats, qawarnings
+ vcs_settings.qatracker.qacats = repo_settings.qadata.qacats
+ vcs_settings.qatracker.qawarnings = repo_settings.qadata.qawarnings
+ logging.debug("repoman_main: vcs_settings done")
+ logging.debug("repoman_main: qadata: %s", repo_settings.qadata)
if 'digest' in repoman_settings.features and options.digest != 'n':
options.digest = 'y'
@@ -133,11 +135,11 @@ def repoman_main(argv):
if options.mode == "manifest":
sys.exit(result['fail'])
- for x in qacats:
+ for x in qadata.qacats:
if x not in vcs_settings.qatracker.fails:
continue
result['warn'] = 1
- if x not in qawarnings:
+ if x not in qadata.qawarnings:
result['fail'] = 1
if result['fail'] or \
@@ -174,7 +176,7 @@ def repoman_main(argv):
format_output = format_outputs.get(
options.output_style, format_outputs['default'])
format_output(f, vcs_settings.qatracker.fails, result['full'],
- result['fail'], options, qawarnings)
+ result['fail'], options, qadata.qawarnings)
style_file.flush()
del console_writer, f, style_file
diff --git a/repoman/pym/repoman/qa_tracker.py b/repoman/pym/repoman/qa_tracker.py
index 9bfe0e241..faaf8e398 100644
--- a/repoman/pym/repoman/qa_tracker.py
+++ b/repoman/pym/repoman/qa_tracker.py
@@ -2,15 +2,15 @@
import logging
import sys
-from repoman.qa_data import qacats, qawarnings
-
class QATracker(object):
'''Track all occurrances of Q/A problems detected'''
- def __init__(self):
+ def __init__(self, qacats=None, qawarnings=None):
self.fails = {}
self.warns = {}
+ self.qacats = qacats
+ self.qawarnings = qawarnings
def add_error(self, detected_qa, info):
'''Add the Q/A error to the database of detected problems
@@ -18,7 +18,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qacats list
@param info: string, details of the detected problem
'''
- if detected_qa not in qacats:
+ if detected_qa not in self.qacats:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_error(): %s, %s' % (detected_qa, info))
@@ -34,7 +34,7 @@ class QATracker(object):
@param detected_qa: string, member of qa_data.qawarnings list
@param info: string, details of the detected problem
'''
- if detected_qa not in qawarnings:
+ if detected_qa not in self.qawarnings:
logging.error(
'QATracker: Exiting on error. Unknown detected_qa type passed '
'in to add_warning(): %s, %s' % (detected_qa, info))
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index 39f53c180..accbf1d9c 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -27,7 +27,7 @@ class RepoSettings(object):
def __init__(
self, config_root, portdir, portdir_overlay,
repoman_settings=None, vcs_settings=None, options=None,
- qawarnings=None):
+ qadata=None):
self.config_root = config_root
self.repoman_settings = repoman_settings
self.vcs_settings = vcs_settings
@@ -41,6 +41,14 @@ class RepoSettings(object):
except KeyError:
self._add_repo(config_root, portdir_overlay)
+ logging.debug("RepoSettings: init(); load qadata")
+ # load the repo specific configuration
+ self.qadata = qadata
+ if not self.qadata.load_repo_config(self.repodir, options):
+ logging.error("Aborting...")
+ sys.exit(1)
+ logging.debug("RepoSettings: qadata loaded: %s", qadata.no_exec)
+
self.root = self.repoman_settings['EROOT']
self.trees = {
self.root: {'porttree': portage.portagetree(settings=self.repoman_settings)}
@@ -60,7 +68,7 @@ class RepoSettings(object):
del self.repositories[repo.name]
if self.repo_config.allow_provide_virtual:
- qawarnings.add("virtual.oldstyle")
+ qadata.qawarnings.add("virtual.oldstyle")
if self.repo_config.sign_commit and options.mode in ("commit", "fix", "manifest"):
if vcs_settings.vcs:
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 22:31 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 22:31 UTC (permalink / raw
To: gentoo-commits
commit: 1d56c9fdfe1ca969b59f44b457235a06f8b0cdfa
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 17:38:48 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 22:29:29 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=1d56c9fd
qa_data.py: Initial move of all configurable data to the repo
All this data was ported to a metadata/repoman/qa_data.yml file.
repoman/pym/repoman/qa_data.py | 433 ++++++++---------------------------------
1 file changed, 77 insertions(+), 356 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index fed798f9f..fed556628 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -1,369 +1,90 @@
# -*- coding:utf-8 -*-
import logging
+import os
+import yaml
from _emerge.Package import Package
# import our initialized portage instance
from repoman._portage import portage
-max_desc_len = 80
-allowed_filename_chars = "a-zA-Z0-9._-+:"
-qahelp = {
- "CVS/Entries.IO_error": (
- "Attempting to commit, and an IO error was encountered access the"
- " Entries file"),
- "ebuild.invalidname": (
- "Ebuild files with a non-parseable or syntactically incorrect name"
- " (or using 2.1 versioning extensions)"),
- "ebuild.namenomatch": (
- "Ebuild files that do not have the same name as their parent"
- " directory"),
- "changelog.ebuildadded": (
- "An ebuild was added but the ChangeLog was not modified"),
- "changelog.missing": (
- "Missing ChangeLog files"),
- "ebuild.notadded": (
- "Ebuilds that exist but have not been added to cvs"),
- "ebuild.patches": (
- "PATCHES variable should be a bash array to ensure white space safety"),
- "changelog.notadded": (
- "ChangeLogs that exist but have not been added to cvs"),
- "dependency.bad": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds)"),
- "dependency.badmasked": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds)"),
- "dependency.badindev": (
- "User-visible ebuilds with unsatisfied dependencies"
- " (matched against *visible* ebuilds) in developing arch"),
- "dependency.badmaskedindev": (
- "Masked ebuilds with unsatisfied dependencies"
- " (matched against *all* ebuilds) in developing arch"),
- "dependency.badtilde": (
- "Uses the ~ dep operator with a non-zero revision part,"
- " which is useless (the revision is ignored)"),
- "dependency.missingslot": (
- "RDEPEND matches more than one SLOT but does not specify a "
- "slot and/or use the := or :* slot operator"),
- "dependency.perlcore": (
- "This ebuild directly depends on a package in perl-core;"
- " it should use the corresponding virtual instead."),
- "dependency.syntax": (
- "Syntax error in dependency string"
- " (usually an extra/missing space/parenthesis)"),
- "dependency.unknown": (
- "Ebuild has a dependency that refers to an unknown package"
- " (which may be valid if it is a blocker for a renamed/removed package,"
- " or is an alternative choice provided by an overlay)"),
- "dependency.badslotop": (
- "RDEPEND contains ':=' slot operator under '||' dependency."),
- "file.executable": (
- "Ebuilds, digests, metadata.xml, Manifest, and ChangeLog do not need"
- " the executable bit"),
- "file.size": (
- "Files in the files directory must be under 20 KiB"),
- "file.size.fatal": (
- "Files in the files directory must be under 60 KiB"),
- "file.empty": (
- "Empty file in the files directory"),
- "file.name": (
- "File/dir name must be composed"
- " of only the following chars: %s " % allowed_filename_chars),
- "file.UTF8": (
- "File is not UTF8 compliant"),
- "inherit.deprecated": (
- "Ebuild inherits a deprecated eclass"),
- "inherit.missing": (
- "Ebuild uses functions from an eclass but does not inherit it"),
- "inherit.unused": (
- "Ebuild inherits an eclass but does not use it"),
- "java.eclassesnotused": (
- "With virtual/jdk in DEPEND you must inherit a java eclass"),
- "wxwidgets.eclassnotused": (
- "Ebuild DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass"),
- "KEYWORDS.dropped": (
- "Ebuilds that appear to have dropped KEYWORDS for some arch"),
- "KEYWORDS.missing": (
- "Ebuilds that have a missing or empty KEYWORDS variable"),
- "KEYWORDS.stable": (
- "Ebuilds that have been added directly with stable KEYWORDS"),
- "KEYWORDS.stupid": (
- "Ebuilds that use KEYWORDS=-* instead of package.mask"),
- "LICENSE.missing": (
- "Ebuilds that have a missing or empty LICENSE variable"),
- "LICENSE.virtual": (
- "Virtuals that have a non-empty LICENSE variable"),
- "DESCRIPTION.missing": (
- "Ebuilds that have a missing or empty DESCRIPTION variable"),
- "DESCRIPTION.toolong": (
- "DESCRIPTION is over %d characters" % max_desc_len),
- "EAPI.definition": (
- "EAPI definition does not conform to PMS section 7.3.1"
- " (first non-comment, non-blank line)"),
- "EAPI.deprecated": (
- "Ebuilds that use features that are deprecated in the current EAPI"),
- "EAPI.incompatible": (
- "Ebuilds that use features that are only available with a different"
- " EAPI"),
- "EAPI.unsupported": (
- "Ebuilds that have an unsupported EAPI version"
- " (you must upgrade portage)"),
- "SLOT.invalid": (
- "Ebuilds that have a missing or invalid SLOT variable value"),
- "HOMEPAGE.missing": (
- "Ebuilds that have a missing or empty HOMEPAGE variable"),
- "HOMEPAGE.virtual": (
- "Virtuals that have a non-empty HOMEPAGE variable"),
- "HOMEPAGE.missingurischeme": (
- "HOMEPAGE is missing an URI scheme"),
- "PDEPEND.suspect": (
- "PDEPEND contains a package that usually only belongs in DEPEND."),
- "LICENSE.syntax": (
- "Syntax error in LICENSE"
- " (usually an extra/missing space/parenthesis)"),
- "PROVIDE.syntax": (
- "Syntax error in PROVIDE"
- " (usually an extra/missing space/parenthesis)"),
- "PROPERTIES.syntax": (
- "Syntax error in PROPERTIES"
- " (usually an extra/missing space/parenthesis)"),
- "RESTRICT.syntax": (
- "Syntax error in RESTRICT"
- " (usually an extra/missing space/parenthesis)"),
- "REQUIRED_USE.syntax": (
- "Syntax error in REQUIRED_USE"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.syntax": (
- "Syntax error in SRC_URI"
- " (usually an extra/missing space/parenthesis)"),
- "SRC_URI.mirror": (
- "A uri listed in profiles/thirdpartymirrors is found in SRC_URI"),
- "ebuild.syntax": (
- "Error generating cache entry for ebuild;"
- " typically caused by ebuild syntax error"
- " or digest verification failure"),
- "ebuild.output": (
- "A simple sourcing of the ebuild produces output;"
- " this breaks ebuild policy."),
- "ebuild.nesteddie": (
- "Placing 'die' inside ( ) prints an error,"
- " but doesn't stop the ebuild."),
- "variable.invalidchar": (
- "A variable contains an invalid character"
- " that is not part of the ASCII character set"),
- "variable.readonly": (
- "Assigning a readonly variable"),
- "variable.usedwithhelpers": (
- "Ebuild uses D, ROOT, ED, EROOT or EPREFIX with helpers"),
- "LIVEVCS.stable": (
- "This ebuild is a live checkout from a VCS but has stable keywords."),
- "LIVEVCS.unmasked": (
- "This ebuild is a live checkout from a VCS but has keywords"
- " and is not masked in the global package.mask."),
- "IUSE.invalid": (
- "This ebuild has a variable in IUSE"
- " that is not in the use.desc or its metadata.xml file"),
- "IUSE.missing": (
- "This ebuild has a USE conditional"
- " which references a flag that is not listed in IUSE"),
- "IUSE.rubydeprecated": (
- "The ebuild has set a ruby interpreter in USE_RUBY,"
- " that is not available as a ruby target anymore"),
- "LICENSE.invalid": (
- "This ebuild is listing a license"
- " that doesnt exist in portages license/ dir."),
- "LICENSE.deprecated": (
- "This ebuild is listing a deprecated license."),
- "KEYWORDS.invalid": (
- "This ebuild contains KEYWORDS"
- " that are not listed in profiles/arch.list"
- " or for which no valid profile was found"),
- "RDEPEND.implicit": (
- "RDEPEND is unset in the ebuild"
- " which triggers implicit RDEPEND=$DEPEND assignment"
- " (prior to EAPI 4)"),
- "RDEPEND.suspect": (
- "RDEPEND contains a package that usually only belongs in DEPEND."),
- "RESTRICT.invalid": (
- "This ebuild contains invalid RESTRICT values."),
- "digest.assumed": (
- "Existing digest must be assumed correct (Package level only)"),
- "digest.missing": (
- "Some files listed in SRC_URI aren't referenced in the Manifest"),
- "digest.unused": (
- "Some files listed in the Manifest aren't referenced in SRC_URI"),
- "ebuild.absdosym": (
- "This ebuild uses absolute target to dosym where relative symlink"
- " could be used instead"),
- "ebuild.majorsyn": (
- "This ebuild has a major syntax error"
- " that may cause the ebuild to fail partially or fully"),
- "ebuild.minorsyn": (
- "This ebuild has a minor syntax error"
- " that contravenes gentoo coding style"),
- "ebuild.badheader": (
- "This ebuild has a malformed header"),
- "manifest.bad": (
- "Manifest has missing or incorrect digests"),
- "metadata.missing": (
- "Missing metadata.xml files"),
- "metadata.bad": (
- "Bad metadata.xml files"),
- "metadata.warning": (
- "Warnings in metadata.xml files"),
- "portage.internal": (
- "The ebuild uses an internal Portage function or variable"),
- "repo.eapi.banned": (
- "The ebuild uses an EAPI which is"
- " banned by the repository's metadata/layout.conf settings"),
- "repo.eapi.deprecated": (
- "The ebuild uses an EAPI which is"
- " deprecated by the repository's metadata/layout.conf settings"),
- "virtual.oldstyle": (
- "The ebuild PROVIDEs an old-style virtual (see GLEP 37)"),
- "virtual.suspect": (
- "Ebuild contains a package"
- " that usually should be pulled via virtual/, not directly."),
- "usage.obsolete": (
- "The ebuild makes use of an obsolete construct"),
- "upstream.workaround": (
- "The ebuild works around an upstream bug,"
- " an upstream bug should be filed and tracked in bugs.gentoo.org"),
- "uri.https": "URI uses http:// but should use https://",
-}
-
-qacats = list(qahelp)
-qacats.sort()
-
-qawarnings = set((
- "changelog.missing",
- "changelog.notadded",
- "dependency.unknown",
- "digest.assumed",
- "digest.unused",
- "ebuild.notadded",
- "ebuild.nesteddie",
- "dependency.badmasked",
- "dependency.badindev",
- "dependency.badmaskedindev",
- "dependency.badtilde",
- "dependency.missingslot",
- "dependency.perlcore",
- "DESCRIPTION.toolong",
- "EAPI.deprecated",
- "HOMEPAGE.virtual",
- "LICENSE.deprecated",
- "LICENSE.virtual",
- "KEYWORDS.dropped",
- "KEYWORDS.stupid",
- "KEYWORDS.missing",
- "PDEPEND.suspect",
- "RDEPEND.implicit",
- "RDEPEND.suspect",
- "virtual.suspect",
- "RESTRICT.invalid",
- "ebuild.absdosym",
- "ebuild.minorsyn",
- "ebuild.badheader",
- "ebuild.patches",
- "file.empty",
- "file.size",
- "inherit.unused",
- "inherit.deprecated",
- "java.eclassesnotused",
- "wxwidgets.eclassnotused",
- "metadata.warning",
- "portage.internal",
- "repo.eapi.deprecated",
- "usage.obsolete",
- "upstream.workaround",
- "IUSE.rubydeprecated",
- "uri.https",
-))
-
-
-missingvars = ["KEYWORDS", "LICENSE", "DESCRIPTION", "HOMEPAGE"]
-allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
-allvars.update(Package.metadata_keys)
-allvars = sorted(allvars)
-
-for x in missingvars:
- x += ".missing"
- if x not in qacats:
- logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
- qacats.append(x)
- qawarnings.add(x)
-
-valid_restrict = frozenset([
- "binchecks", "bindist", "fetch", "installsources", "mirror",
- "preserve-libs", "primaryuri", "splitdebug", "strip", "test", "userpriv"])
-
-
-suspect_rdepend = frozenset([
- "app-arch/cabextract",
- "app-arch/rpm2targz",
- "app-doc/doxygen",
- "dev-lang/nasm",
- "dev-lang/swig",
- "dev-lang/yasm",
- "dev-perl/extutils-pkgconfig",
- "dev-qt/linguist-tools",
- "dev-util/byacc",
- "dev-util/cmake",
- "dev-util/ftjam",
- "dev-util/gperf",
- "dev-util/gtk-doc",
- "dev-util/gtk-doc-am",
- "dev-util/intltool",
- "dev-util/jam",
- "dev-util/pkg-config-lite",
- "dev-util/pkgconf",
- "dev-util/pkgconfig",
- "dev-util/pkgconfig-openbsd",
- "dev-util/scons",
- "dev-util/unifdef",
- "dev-util/yacc",
- "media-gfx/ebdftopcf",
- "sys-apps/help2man",
- "sys-devel/autoconf",
- "sys-devel/automake",
- "sys-devel/bin86",
- "sys-devel/bison",
- "sys-devel/dev86",
- "sys-devel/flex",
- "sys-devel/m4",
- "sys-devel/pmake",
- "virtual/linux-sources",
- "virtual/linuxtv-dvb-headers",
- "virtual/os-headers",
- "virtual/pkgconfig",
- "x11-misc/bdftopcf",
- "x11-misc/imake",
-])
-
-suspect_virtual = {
- "dev-util/pkg-config-lite": "virtual/pkgconfig",
- "dev-util/pkgconf": "virtual/pkgconfig",
- "dev-util/pkgconfig": "virtual/pkgconfig",
- "dev-util/pkgconfig-openbsd": "virtual/pkgconfig",
- "dev-libs/libusb": "virtual/libusb",
- "dev-libs/libusbx": "virtual/libusb",
- "dev-libs/libusb-compat": "virtual/libusb",
-}
-
-ruby_deprecated = frozenset([
- "ruby_targets_ree18",
- "ruby_targets_ruby18",
- "ruby_targets_ruby19",
- "ruby_targets_ruby20",
-])
-
-
-# file.executable
-no_exec = frozenset(["Manifest", "ChangeLog", "metadata.xml"])
+class QAData(object):
+
+ def __init__(self):
+ # Create the main exported data variables
+ self.max_desc_len = None
+ self.allowed_filename_chars = None
+ self.qahelp = None
+ self.qacats = None
+ self.qawarnings = None
+ self.missingvars = None
+ self.allvars = None
+ self.valid_restrict = None
+ self.suspect_rdepend = None
+ self.suspect_virtual = None
+ self.ruby_deprecated = None
+ self.no_exec = None
+
+
+ def load_repo_config(self, repopath, options):
+ '''Load the repository repoman qa_data.yml config
+
+ @param repopath: string, The path of the repository being scanned
+ This could be a parent repository using the
+ repoman_masters layout.conf variable
+ '''
+ filepath = os.path.join(repopath, 'metadata/repoman/qa_data.yml')
+ logging.debug("QAData: reading file: %s", filepath)
+ try:
+ with open(filepath, 'r') as qadata_file:
+ qadata = yaml.safe_load(qadata_file.read())
+ except IOError as error:
+ logging.error("Failed to load 'qa_data.yml' file at path: %s", filepath)
+ logging.eception(error)
+ return False
+ self.max_desc_len = qadata.get('max_description_length', 80)
+ self.allowed_filename_chars = qadata.get("allowed_filename_chars", "a-zA-Z0-9._-+:")
+
+ self.qahelp = qadata.get('qahelp', {})
+
+ self.qacats = []
+ for x in sorted(self.qahelp):
+ for y in sorted(self.qahelp[x]):
+ self.qacats.append('.'.join([x, y]))
+ self.qacats.sort()
+
+ self.qawarnings = set(qadata.get('qawarnings', []))
+ if options.experimental_inherit == 'y':
+ # This is experimental, so it's non-fatal.
+ self.qawarnings.add("inherit.missing")
+
+ self.missingvars = qadata.get("missingvars", [])
+ logging.debug("QAData: missingvars: %s", self.missingvars)
+ self.allvars = set(x for x in portage.auxdbkeys if not x.startswith("UNUSED_"))
+ self.allvars.update(Package.metadata_keys)
+ self.allvars = sorted(self.allvars)
+
+ for x in self.missingvars:
+ x += ".missing"
+ if x not in self.qacats:
+ logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
+ self.qacats.append(x)
+ self.qawarnings.add(x)
+
+ self.valid_restrict = frozenset(qadata.get("valid_restrict", []))
+
+ self.suspect_rdepend = frozenset(qadata.get("suspect_rdepend", []))
+
+ self.suspect_virtual = qadata.get("suspect_virtual", {})
+
+ self.ruby_deprecated = frozenset(qadata.get("ruby_deprecated", []))
+
+ # file.executable
+ self.no_exec = frozenset(qadata.get("no_exec_files", []))
+ logging.debug("QAData: completed loading file: %s", filepath)
+ return True
def format_qa_output(
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 22:31 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 22:31 UTC (permalink / raw
To: gentoo-commits
commit: fd099557cf568806baf3f67d57292ef66bbc590d
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Mon Jul 10 21:43:39 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 22:29:34 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=fd099557
repoman: repos.py: Add masters_list
Use masters_list for loading QAData config.
Signed-off-by: Brian Dolbec <dolsen <AT> gentoo.org>
repoman/pym/repoman/repos.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index accbf1d9c..605c7cffe 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -41,10 +41,18 @@ class RepoSettings(object):
except KeyError:
self._add_repo(config_root, portdir_overlay)
+ # Determine the master config loading list
+ self.masters_list = []
+ # get out repo masters value
+ masters = self.repositories.get_repo_for_location(self.repodir).masters
+ for repo in masters:
+ self.masters_list.append(os.path.join(repo.location, 'metadata', 'repoman'))
+ self.masters_list.append(os.path.join(self.repodir, 'metadata', 'repoman'))
+
logging.debug("RepoSettings: init(); load qadata")
# load the repo specific configuration
self.qadata = qadata
- if not self.qadata.load_repo_config(self.repodir, options):
+ if not self.qadata.load_repo_config(self.masters_list, options):
logging.error("Aborting...")
sys.exit(1)
logging.debug("RepoSettings: qadata loaded: %s", qadata.no_exec)
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 22:31 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 22:31 UTC (permalink / raw
To: gentoo-commits
commit: 794cd3ce0866e04b9b647b97e92f0845490abb69
Author: Brian Dolbec <bdolbec <AT> gaikai <DOT> com>
AuthorDate: Tue Jun 27 18:08:50 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 22:29:33 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=794cd3ce
repoman: main.py: Remove unused InvalidStateError import
repoman/pym/repoman/main.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/repoman/pym/repoman/main.py b/repoman/pym/repoman/main.py
old mode 100644
new mode 100755
index 3b628de00..c1e3b99fe
--- a/repoman/pym/repoman/main.py
+++ b/repoman/pym/repoman/main.py
@@ -24,7 +24,6 @@ from portage.output import ConsoleStyleFile, StyleWriter
from portage.util import formatter
from portage.util.futures.extendedfutures import (
ExtendedFuture,
- InvalidStateError,
)
from repoman.actions import Actions
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 22:31 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 22:31 UTC (permalink / raw
To: gentoo-commits
commit: 69dd9b44685bffb624dc1db5f2f3665f61f8b6ce
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Mon Jul 10 21:44:50 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 22:29:34 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=69dd9b44
repoman: scanner.py: Update module loading to use masters_list
Signed-off-by: Brian Dolbec <dolsen <AT> gentoo.org>
repoman/pym/repoman/scanner.py | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/repoman/pym/repoman/scanner.py b/repoman/pym/repoman/scanner.py
index ffce701be..b3d030570 100644
--- a/repoman/pym/repoman/scanner.py
+++ b/repoman/pym/repoman/scanner.py
@@ -116,9 +116,7 @@ class Scanner(object):
# Initialize the ModuleConfig class here
# TODO Add layout.conf masters repository.yml config to the list to load/stack
- self.moduleconfig = ModuleConfig([
- '/home/bdolbec/git/gentoo/metadata/repoman/repository.yml',
- ])
+ self.moduleconfig = ModuleConfig(self.repo_settings.masters_list)
checks = {}
# The --echangelog option causes automatic ChangeLog generation,
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 22:31 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 22:31 UTC (permalink / raw
To: gentoo-commits
commit: 398b8dc2b7b0297180ba38334526e24f5f85638e
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Mon Jul 10 21:40:40 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 22:29:34 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=398b8dc2
repoman: qa_data.py: Make it use masters stacking
Update logging message with 'QAData: ' prefix
Signed-off-by: Brian Dolbec <dolsen <AT> gentoo.org>
repoman/pym/repoman/qa_data.py | 32 +++++++++++++++++++++-----------
1 file changed, 21 insertions(+), 11 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index fed556628..50a1c764d 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -28,21 +28,31 @@ class QAData(object):
self.no_exec = None
- def load_repo_config(self, repopath, options):
+ def load_repo_config(self, repopaths, options):
'''Load the repository repoman qa_data.yml config
- @param repopath: string, The path of the repository being scanned
+ @param repopaths: list of strings, The path of the repository being scanned
This could be a parent repository using the
repoman_masters layout.conf variable
'''
- filepath = os.path.join(repopath, 'metadata/repoman/qa_data.yml')
- logging.debug("QAData: reading file: %s", filepath)
- try:
- with open(filepath, 'r') as qadata_file:
- qadata = yaml.safe_load(qadata_file.read())
- except IOError as error:
- logging.error("Failed to load 'qa_data.yml' file at path: %s", filepath)
- logging.eception(error)
+ qadata = {}
+ for path in repopaths:
+ filepath = os.path.join(path, 'qa_data.yml')
+ logging.debug("QAData: reading file: %s", filepath)
+ try:
+ with open(filepath, 'r') as qadata_file:
+ new_qadata = yaml.safe_load(qadata_file.read())
+ logging.debug("QAData: updating qadata with new values from: %s", filepath)
+ qadata.update(new_qadata)
+ except FileNotFoundError:
+ # skip a master that may not have our file
+ logging.debug("QAData: File not found at path: %s", filepath)
+ except IOError as error:
+ logging.error("QAData: Failed to load 'qa_data.yml' file at path: %s", filepath)
+ logging.exception(error)
+ return False
+ if qadata == {}:
+ logging.error("QAData: Failed to load a valid 'qa_data.yml' file at paths: %s", repopaths)
return False
self.max_desc_len = qadata.get('max_description_length', 80)
self.allowed_filename_chars = qadata.get("allowed_filename_chars", "a-zA-Z0-9._-+:")
@@ -69,7 +79,7 @@ class QAData(object):
for x in self.missingvars:
x += ".missing"
if x not in self.qacats:
- logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
+ logging.warning('QAData: * missingvars values need to be added to qahelp ("%s")' % x)
self.qacats.append(x)
self.qawarnings.add(x)
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 23:01 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 23:01 UTC (permalink / raw
To: gentoo-commits
commit: d39ec879d0dc5b25c8650d9c20e8914a51ceb886
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Mon Jul 10 21:40:40 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 23:00:50 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=d39ec879
repoman: qa_data.py: Make it use masters stacking
Update logging message with 'QAData: ' prefix
Signed-off-by: Brian Dolbec <dolsen <AT> gentoo.org>
repoman/pym/repoman/qa_data.py | 32 +++++++++++++++++++++-----------
1 file changed, 21 insertions(+), 11 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index fed556628..50a1c764d 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -28,21 +28,31 @@ class QAData(object):
self.no_exec = None
- def load_repo_config(self, repopath, options):
+ def load_repo_config(self, repopaths, options):
'''Load the repository repoman qa_data.yml config
- @param repopath: string, The path of the repository being scanned
+ @param repopaths: list of strings, The path of the repository being scanned
This could be a parent repository using the
repoman_masters layout.conf variable
'''
- filepath = os.path.join(repopath, 'metadata/repoman/qa_data.yml')
- logging.debug("QAData: reading file: %s", filepath)
- try:
- with open(filepath, 'r') as qadata_file:
- qadata = yaml.safe_load(qadata_file.read())
- except IOError as error:
- logging.error("Failed to load 'qa_data.yml' file at path: %s", filepath)
- logging.eception(error)
+ qadata = {}
+ for path in repopaths:
+ filepath = os.path.join(path, 'qa_data.yml')
+ logging.debug("QAData: reading file: %s", filepath)
+ try:
+ with open(filepath, 'r') as qadata_file:
+ new_qadata = yaml.safe_load(qadata_file.read())
+ logging.debug("QAData: updating qadata with new values from: %s", filepath)
+ qadata.update(new_qadata)
+ except FileNotFoundError:
+ # skip a master that may not have our file
+ logging.debug("QAData: File not found at path: %s", filepath)
+ except IOError as error:
+ logging.error("QAData: Failed to load 'qa_data.yml' file at path: %s", filepath)
+ logging.exception(error)
+ return False
+ if qadata == {}:
+ logging.error("QAData: Failed to load a valid 'qa_data.yml' file at paths: %s", repopaths)
return False
self.max_desc_len = qadata.get('max_description_length', 80)
self.allowed_filename_chars = qadata.get("allowed_filename_chars", "a-zA-Z0-9._-+:")
@@ -69,7 +79,7 @@ class QAData(object):
for x in self.missingvars:
x += ".missing"
if x not in self.qacats:
- logging.warning('* missingvars values need to be added to qahelp ("%s")' % x)
+ logging.warning('QAData: * missingvars values need to be added to qahelp ("%s")' % x)
self.qacats.append(x)
self.qawarnings.add(x)
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 23:01 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 23:01 UTC (permalink / raw
To: gentoo-commits
commit: 5c374e1aabb3c3ed825f8acf42712d150c172ca7
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Mon Jul 10 21:44:50 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 23:00:50 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=5c374e1a
repoman: scanner.py: Update module loading to use masters_list
Signed-off-by: Brian Dolbec <dolsen <AT> gentoo.org>
repoman/pym/repoman/scanner.py | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/repoman/pym/repoman/scanner.py b/repoman/pym/repoman/scanner.py
index ffce701be..b3d030570 100644
--- a/repoman/pym/repoman/scanner.py
+++ b/repoman/pym/repoman/scanner.py
@@ -116,9 +116,7 @@ class Scanner(object):
# Initialize the ModuleConfig class here
# TODO Add layout.conf masters repository.yml config to the list to load/stack
- self.moduleconfig = ModuleConfig([
- '/home/bdolbec/git/gentoo/metadata/repoman/repository.yml',
- ])
+ self.moduleconfig = ModuleConfig(self.repo_settings.masters_list)
checks = {}
# The --echangelog option causes automatic ChangeLog generation,
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-10 23:01 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-10 23:01 UTC (permalink / raw
To: gentoo-commits
commit: c7f39924f1c7e8579405bd42119e94d3afc9ff31
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Mon Jul 10 21:43:39 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Mon Jul 10 23:00:50 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=c7f39924
repoman: repos.py: Add masters_list
Use masters_list for loading QAData config.
Signed-off-by: Brian Dolbec <dolsen <AT> gentoo.org>
repoman/pym/repoman/repos.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index accbf1d9c..605c7cffe 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -41,10 +41,18 @@ class RepoSettings(object):
except KeyError:
self._add_repo(config_root, portdir_overlay)
+ # Determine the master config loading list
+ self.masters_list = []
+ # get out repo masters value
+ masters = self.repositories.get_repo_for_location(self.repodir).masters
+ for repo in masters:
+ self.masters_list.append(os.path.join(repo.location, 'metadata', 'repoman'))
+ self.masters_list.append(os.path.join(self.repodir, 'metadata', 'repoman'))
+
logging.debug("RepoSettings: init(); load qadata")
# load the repo specific configuration
self.qadata = qadata
- if not self.qadata.load_repo_config(self.repodir, options):
+ if not self.qadata.load_repo_config(self.masters_list, options):
logging.error("Aborting...")
sys.exit(1)
logging.debug("RepoSettings: qadata loaded: %s", qadata.no_exec)
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-15 2:08 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-15 2:08 UTC (permalink / raw
To: gentoo-commits
commit: 1769905581e8f4ea6f5486314c912de3f4385d39
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sat Jul 15 00:06:27 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sat Jul 15 02:08:27 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=17699055
repoman: Add a new config.py file with config loading utilities
These include recursively merging of multiple yaml files.
They are needed for masters stacking.
repoman/pym/repoman/config.py | 143 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 143 insertions(+)
diff --git a/repoman/pym/repoman/config.py b/repoman/pym/repoman/config.py
new file mode 100644
index 000000000..2825aee04
--- /dev/null
+++ b/repoman/pym/repoman/config.py
@@ -0,0 +1,143 @@
+# Copyright 2015-2016 Gaikai Inc, a Sony Computer Entertainment company
+
+import copy
+import itertools
+import json
+import os
+import stat
+
+import yaml
+
+
+class ConfigError(Exception):
+
+ """Raised when a config file fails to load"""
+ pass
+
+
+def merge_config(base, head):
+ """
+ Merge two JSON or YAML documents into a single object. Arrays are
+ merged by extension. If dissimilar types are encountered, then the
+ head value overwrites the base value.
+ """
+
+ if isinstance(head, dict):
+ if not isinstance(base, dict):
+ return copy.deepcopy(head)
+
+ result = {}
+ for k in itertools.chain(head, base):
+ try:
+ result[k] = merge_config(base[k], head[k])
+ except KeyError:
+ try:
+ result[k] = copy.deepcopy(head[k])
+ except KeyError:
+ result[k] = copy.deepcopy(base[k])
+
+ elif isinstance(head, list):
+ result = []
+ if not isinstance(base, list):
+ result.extend(copy.deepcopy(x) for x in head)
+ else:
+ if any(isinstance(x, (dict, list)) for x in itertools.chain(head, base)):
+ # merge items with identical indexes
+ for x, y in zip(base, head):
+ if isinstance(x, (dict, list)):
+ result.append(merge_config(x, y))
+ else:
+ # head overwrites base (preserving index)
+ result.append(copy.deepcopy(y))
+ # copy remaining items from the longer list
+ if len(base) != len(head):
+ if len(base) > len(head):
+ result.extend(copy.deepcopy(x) for x in base[len(head):])
+ else:
+ result.extend(copy.deepcopy(x) for x in head[len(base):])
+ else:
+ result.extend(copy.deepcopy(x) for x in base)
+ result.extend(copy.deepcopy(x) for x in head)
+
+ else:
+ result = copy.deepcopy(head)
+
+ return result
+
+def _yaml_load(filename):
+ """
+ Load filename as YAML and return a dict. Raise ConfigError if
+ it fails to load.
+ """
+ with open(filename, 'rt') as f:
+ try:
+ return yaml.safe_load(f)
+ except yaml.parser.ParserError as e:
+ raise ConfigError("{}: {}".format(filename, e))
+
+def _json_load(filename):
+ """
+ Load filename as JSON and return a dict. Raise ConfigError if
+ it fails to load.
+ """
+ with open(filename, 'rt') as f:
+ try:
+ return json.load(f) #nosec
+ except ValueError as e:
+ raise ConfigError("{}: {}".format(filename, e))
+
+def iter_files(files_dirs):
+ """
+ Iterate over nested file paths in lexical order.
+ """
+ stack = list(reversed(files_dirs))
+ while stack:
+ location = stack.pop()
+ try:
+ st = os.stat(location)
+ except FileNotFoundError:
+ continue
+
+ if stat.S_ISDIR(st.st_mode):
+ stack.extend(os.path.join(location, x)
+ for x in sorted(os.listdir(location), reverse=True))
+
+ elif stat.S_ISREG(st.st_mode):
+ yield location
+
+def load_config(conf_dirs, file_extensions=None):
+ """
+ Load JSON and/or YAML files from a directories, and merge them together
+ into a single object.
+ """
+
+ result = {}
+ for filename in iter_files(conf_dirs):
+ if file_extensions is not None and not filename.endswith(file_extensions):
+ continue
+
+ loaders = []
+ if filename.endswith('.json'):
+ loaders.append(_json_load)
+ elif filename.endswith('.yaml'):
+ loaders.append(_yaml_load)
+ else:
+ loaders.append(_yaml_load)
+ loaders.append(_json_load)
+
+ config = None
+ for loader in loaders:
+ try:
+ config = loader(filename) or {}
+ except ConfigError as e:
+ exception = e
+ else:
+ break
+
+ if config is None:
+ raise exception
+
+ if config:
+ result = merge_config(result, config)
+
+ return result
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-15 2:08 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-15 2:08 UTC (permalink / raw
To: gentoo-commits
commit: 7ccac26b5858b0359bc5fcb076c8af2d336a3a56
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sat Jul 15 00:09:11 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sat Jul 15 02:08:27 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=7ccac26b
repoman: Update qa_data.py to use the new load_config() utility
repoman/pym/repoman/qa_data.py | 20 +++-----------------
1 file changed, 3 insertions(+), 17 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index 2ccea1996..e361dd62e 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -8,6 +8,7 @@ from _emerge.Package import Package
# import our initialized portage instance
from repoman._portage import portage
+from repoman.config import load_config
class QAData(object):
@@ -35,22 +36,7 @@ class QAData(object):
This could be a parent repository using the
repoman_masters layout.conf variable
'''
- qadata = {}
- for path in repopaths:
- filepath = os.path.join(path, 'qa_data.yaml')
- logging.debug("QAData: reading file: %s", filepath)
- try:
- with open(filepath, 'r') as qadata_file:
- new_qadata = yaml.safe_load(qadata_file.read())
- logging.debug("QAData: updating qadata with new values from: %s", filepath)
- qadata.update(new_qadata)
- except FileNotFoundError:
- # skip a master that may not have our file
- logging.debug("QAData: File not found at path: %s", filepath)
- except IOError as error:
- logging.error("QAData: Failed to load 'qa_data.yaml' file at path: %s", filepath)
- logging.exception(error)
- return False
+ qadata = load_config([os.path.join(path,'qa_data.yaml') for path in repopaths], 'yaml')
if qadata == {}:
logging.error("QAData: Failed to load a valid 'qa_data.yaml' file at paths: %s", repopaths)
return False
@@ -93,7 +79,7 @@ class QAData(object):
# file.executable
self.no_exec = frozenset(qadata.get("no_exec_files", []))
- logging.debug("QAData: completed loading file: %s", filepath)
+ logging.debug("QAData: completed loading file: %s", repopaths)
return True
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-15 2:29 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-15 2:29 UTC (permalink / raw
To: gentoo-commits
commit: 848989db23ac37c1d5716d740cdb134142b3c305
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sat Jul 15 00:06:27 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sat Jul 15 02:25:44 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=848989db
repoman: Add a new config.py file with config loading utilities
These include recursively merging of multiple yaml files.
They are needed for masters stacking.
repoman/pym/repoman/config.py | 143 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 143 insertions(+)
diff --git a/repoman/pym/repoman/config.py b/repoman/pym/repoman/config.py
new file mode 100644
index 000000000..2825aee04
--- /dev/null
+++ b/repoman/pym/repoman/config.py
@@ -0,0 +1,143 @@
+# Copyright 2015-2016 Gaikai Inc, a Sony Computer Entertainment company
+
+import copy
+import itertools
+import json
+import os
+import stat
+
+import yaml
+
+
+class ConfigError(Exception):
+
+ """Raised when a config file fails to load"""
+ pass
+
+
+def merge_config(base, head):
+ """
+ Merge two JSON or YAML documents into a single object. Arrays are
+ merged by extension. If dissimilar types are encountered, then the
+ head value overwrites the base value.
+ """
+
+ if isinstance(head, dict):
+ if not isinstance(base, dict):
+ return copy.deepcopy(head)
+
+ result = {}
+ for k in itertools.chain(head, base):
+ try:
+ result[k] = merge_config(base[k], head[k])
+ except KeyError:
+ try:
+ result[k] = copy.deepcopy(head[k])
+ except KeyError:
+ result[k] = copy.deepcopy(base[k])
+
+ elif isinstance(head, list):
+ result = []
+ if not isinstance(base, list):
+ result.extend(copy.deepcopy(x) for x in head)
+ else:
+ if any(isinstance(x, (dict, list)) for x in itertools.chain(head, base)):
+ # merge items with identical indexes
+ for x, y in zip(base, head):
+ if isinstance(x, (dict, list)):
+ result.append(merge_config(x, y))
+ else:
+ # head overwrites base (preserving index)
+ result.append(copy.deepcopy(y))
+ # copy remaining items from the longer list
+ if len(base) != len(head):
+ if len(base) > len(head):
+ result.extend(copy.deepcopy(x) for x in base[len(head):])
+ else:
+ result.extend(copy.deepcopy(x) for x in head[len(base):])
+ else:
+ result.extend(copy.deepcopy(x) for x in base)
+ result.extend(copy.deepcopy(x) for x in head)
+
+ else:
+ result = copy.deepcopy(head)
+
+ return result
+
+def _yaml_load(filename):
+ """
+ Load filename as YAML and return a dict. Raise ConfigError if
+ it fails to load.
+ """
+ with open(filename, 'rt') as f:
+ try:
+ return yaml.safe_load(f)
+ except yaml.parser.ParserError as e:
+ raise ConfigError("{}: {}".format(filename, e))
+
+def _json_load(filename):
+ """
+ Load filename as JSON and return a dict. Raise ConfigError if
+ it fails to load.
+ """
+ with open(filename, 'rt') as f:
+ try:
+ return json.load(f) #nosec
+ except ValueError as e:
+ raise ConfigError("{}: {}".format(filename, e))
+
+def iter_files(files_dirs):
+ """
+ Iterate over nested file paths in lexical order.
+ """
+ stack = list(reversed(files_dirs))
+ while stack:
+ location = stack.pop()
+ try:
+ st = os.stat(location)
+ except FileNotFoundError:
+ continue
+
+ if stat.S_ISDIR(st.st_mode):
+ stack.extend(os.path.join(location, x)
+ for x in sorted(os.listdir(location), reverse=True))
+
+ elif stat.S_ISREG(st.st_mode):
+ yield location
+
+def load_config(conf_dirs, file_extensions=None):
+ """
+ Load JSON and/or YAML files from a directories, and merge them together
+ into a single object.
+ """
+
+ result = {}
+ for filename in iter_files(conf_dirs):
+ if file_extensions is not None and not filename.endswith(file_extensions):
+ continue
+
+ loaders = []
+ if filename.endswith('.json'):
+ loaders.append(_json_load)
+ elif filename.endswith('.yaml'):
+ loaders.append(_yaml_load)
+ else:
+ loaders.append(_yaml_load)
+ loaders.append(_json_load)
+
+ config = None
+ for loader in loaders:
+ try:
+ config = loader(filename) or {}
+ except ConfigError as e:
+ exception = e
+ else:
+ break
+
+ if config is None:
+ raise exception
+
+ if config:
+ result = merge_config(result, config)
+
+ return result
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-07-15 2:29 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-07-15 2:29 UTC (permalink / raw
To: gentoo-commits
commit: f41249f529180102bb1aeb54d4fc01174af85790
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Sat Jul 15 00:09:11 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Sat Jul 15 02:25:44 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=f41249f5
repoman: Update qa_data.py to use the new load_config() utility
repoman/pym/repoman/qa_data.py | 20 +++-----------------
1 file changed, 3 insertions(+), 17 deletions(-)
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index 2ccea1996..e361dd62e 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -8,6 +8,7 @@ from _emerge.Package import Package
# import our initialized portage instance
from repoman._portage import portage
+from repoman.config import load_config
class QAData(object):
@@ -35,22 +36,7 @@ class QAData(object):
This could be a parent repository using the
repoman_masters layout.conf variable
'''
- qadata = {}
- for path in repopaths:
- filepath = os.path.join(path, 'qa_data.yaml')
- logging.debug("QAData: reading file: %s", filepath)
- try:
- with open(filepath, 'r') as qadata_file:
- new_qadata = yaml.safe_load(qadata_file.read())
- logging.debug("QAData: updating qadata with new values from: %s", filepath)
- qadata.update(new_qadata)
- except FileNotFoundError:
- # skip a master that may not have our file
- logging.debug("QAData: File not found at path: %s", filepath)
- except IOError as error:
- logging.error("QAData: Failed to load 'qa_data.yaml' file at path: %s", filepath)
- logging.exception(error)
- return False
+ qadata = load_config([os.path.join(path,'qa_data.yaml') for path in repopaths], 'yaml')
if qadata == {}:
logging.error("QAData: Failed to load a valid 'qa_data.yaml' file at paths: %s", repopaths)
return False
@@ -93,7 +79,7 @@ class QAData(object):
# file.executable
self.no_exec = frozenset(qadata.get("no_exec_files", []))
- logging.debug("QAData: completed loading file: %s", filepath)
+ logging.debug("QAData: completed loading file: %s", repopaths)
return True
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-08-06 6:54 Michał Górny
0 siblings, 0 replies; 32+ messages in thread
From: Michał Górny @ 2017-08-06 6:54 UTC (permalink / raw
To: gentoo-commits
commit: a6c0a4d46a99b36cfef40e091d531ff7d0685918
Author: Michał Górny <mgorny <AT> gentoo <DOT> org>
AuthorDate: Thu Aug 3 13:52:53 2017 +0000
Commit: Michał Górny <mgorny <AT> gentoo <DOT> org>
CommitDate: Sun Aug 6 06:54:16 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=a6c0a4d4
repoman commit: Merge code generating common part of the footer (DCO)
The DCO Signed-off-by footer looks the same on the git branch, and on
the branch for other VCS-es. Therefore, move the code generating it
above the split branches. This also prepares the code for further footer
elements being added.
Reviewed-by: Zac Medico <zmedico <AT> gentoo.org>
repoman/pym/repoman/actions.py | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/repoman/pym/repoman/actions.py b/repoman/pym/repoman/actions.py
index 73b44c5f5..00bb5b2ca 100644
--- a/repoman/pym/repoman/actions.py
+++ b/repoman/pym/repoman/actions.py
@@ -342,25 +342,26 @@ class Actions(object):
sys.stderr.write("Failed to insert portage version in message!\n")
sys.stderr.flush()
portage_version = "Unknown"
+
+ # Common part of commit footer
+ commit_footer = "\n"
+ if dco_sob:
+ commit_footer += "Signed-off-by: %s\n" % (dco_sob, )
+
# Use new footer only for git (see bug #438364).
if self.vcs_settings.vcs in ["git"]:
- commit_footer = "\nPackage-Manager: Portage-%s, Repoman-%s" % (
+ commit_footer += "Package-Manager: Portage-%s, Repoman-%s" % (
portage.VERSION, VERSION)
if report_options:
commit_footer += "\nRepoMan-Options: " + " ".join(report_options)
if self.repo_settings.sign_manifests:
commit_footer += "\nManifest-Sign-Key: %s" % (gpg_key, )
- if dco_sob:
- commit_footer += "\nSigned-off-by: %s" % (dco_sob, )
else:
unameout = platform.system() + " "
if platform.system() in ["Darwin", "SunOS"]:
unameout += platform.processor()
else:
unameout += platform.machine()
- commit_footer = "\n"
- if dco_sob:
- commit_footer += "Signed-off-by: %s\n" % (dco_sob, )
commit_footer += "(Portage version: %s/%s/%s" % \
(portage_version, self.vcs_settings.vcs, unameout)
if report_options:
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/
@ 2017-08-17 1:59 Brian Dolbec
0 siblings, 0 replies; 32+ messages in thread
From: Brian Dolbec @ 2017-08-17 1:59 UTC (permalink / raw
To: gentoo-commits
commit: 2d7ed7a1fdac828dc5f6b26ec279b2d1e4ee83da
Author: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
AuthorDate: Wed Aug 16 23:23:09 2017 +0000
Commit: Brian Dolbec <dolsen <AT> gentoo <DOT> org>
CommitDate: Wed Aug 16 23:25:27 2017 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=2d7ed7a1
repoman: config.py: Fix whitespace
repoman/pym/repoman/config.py | 245 +++++++++++++++++++++---------------------
1 file changed, 124 insertions(+), 121 deletions(-)
diff --git a/repoman/pym/repoman/config.py b/repoman/pym/repoman/config.py
index 2825aee04..f41945c13 100644
--- a/repoman/pym/repoman/config.py
+++ b/repoman/pym/repoman/config.py
@@ -11,133 +11,136 @@ import yaml
class ConfigError(Exception):
- """Raised when a config file fails to load"""
- pass
+ """Raised when a config file fails to load"""
+ pass
def merge_config(base, head):
- """
- Merge two JSON or YAML documents into a single object. Arrays are
- merged by extension. If dissimilar types are encountered, then the
- head value overwrites the base value.
- """
-
- if isinstance(head, dict):
- if not isinstance(base, dict):
- return copy.deepcopy(head)
-
- result = {}
- for k in itertools.chain(head, base):
- try:
- result[k] = merge_config(base[k], head[k])
- except KeyError:
- try:
- result[k] = copy.deepcopy(head[k])
- except KeyError:
- result[k] = copy.deepcopy(base[k])
-
- elif isinstance(head, list):
- result = []
- if not isinstance(base, list):
- result.extend(copy.deepcopy(x) for x in head)
- else:
- if any(isinstance(x, (dict, list)) for x in itertools.chain(head, base)):
- # merge items with identical indexes
- for x, y in zip(base, head):
- if isinstance(x, (dict, list)):
- result.append(merge_config(x, y))
- else:
- # head overwrites base (preserving index)
- result.append(copy.deepcopy(y))
- # copy remaining items from the longer list
- if len(base) != len(head):
- if len(base) > len(head):
- result.extend(copy.deepcopy(x) for x in base[len(head):])
- else:
- result.extend(copy.deepcopy(x) for x in head[len(base):])
- else:
- result.extend(copy.deepcopy(x) for x in base)
- result.extend(copy.deepcopy(x) for x in head)
-
- else:
- result = copy.deepcopy(head)
-
- return result
+ """
+ Merge two JSON or YAML documents into a single object. Arrays are
+ merged by extension. If dissimilar types are encountered, then the
+ head value overwrites the base value.
+ """
+
+ if isinstance(head, dict):
+ if not isinstance(base, dict):
+ return copy.deepcopy(head)
+
+ result = {}
+ for k in itertools.chain(head, base):
+ try:
+ result[k] = merge_config(base[k], head[k])
+ except KeyError:
+ try:
+ result[k] = copy.deepcopy(head[k])
+ except KeyError:
+ result[k] = copy.deepcopy(base[k])
+
+ elif isinstance(head, list):
+ result = []
+ if not isinstance(base, list):
+ result.extend(copy.deepcopy(x) for x in head)
+ else:
+ if any(isinstance(x, (dict, list)) for x in itertools.chain(head, base)):
+ # merge items with identical indexes
+ for x, y in zip(base, head):
+ if isinstance(x, (dict, list)):
+ result.append(merge_config(x, y))
+ else:
+ # head overwrites base (preserving index)
+ result.append(copy.deepcopy(y))
+ # copy remaining items from the longer list
+ if len(base) != len(head):
+ if len(base) > len(head):
+ result.extend(copy.deepcopy(x) for x in base[len(head):])
+ else:
+ result.extend(copy.deepcopy(x) for x in head[len(base):])
+ else:
+ result.extend(copy.deepcopy(x) for x in base)
+ result.extend(copy.deepcopy(x) for x in head)
+
+ else:
+ result = copy.deepcopy(head)
+
+ return result
def _yaml_load(filename):
- """
- Load filename as YAML and return a dict. Raise ConfigError if
- it fails to load.
- """
- with open(filename, 'rt') as f:
- try:
- return yaml.safe_load(f)
- except yaml.parser.ParserError as e:
- raise ConfigError("{}: {}".format(filename, e))
+ """
+ Load filename as YAML and return a dict. Raise ConfigError if
+ it fails to load.
+ """
+ with open(filename, 'rt') as f:
+ try:
+ return yaml.safe_load(f)
+ except yaml.parser.ParserError as e:
+ raise ConfigError("{}: {}".format(filename, e))
def _json_load(filename):
- """
- Load filename as JSON and return a dict. Raise ConfigError if
- it fails to load.
- """
- with open(filename, 'rt') as f:
- try:
- return json.load(f) #nosec
- except ValueError as e:
- raise ConfigError("{}: {}".format(filename, e))
+ """
+ Load filename as JSON and return a dict. Raise ConfigError if
+ it fails to load.
+ """
+ with open(filename, 'rt') as f:
+ try:
+ return json.load(f) #nosec
+ except ValueError as e:
+ raise ConfigError("{}: {}".format(filename, e))
def iter_files(files_dirs):
- """
- Iterate over nested file paths in lexical order.
- """
- stack = list(reversed(files_dirs))
- while stack:
- location = stack.pop()
- try:
- st = os.stat(location)
- except FileNotFoundError:
- continue
-
- if stat.S_ISDIR(st.st_mode):
- stack.extend(os.path.join(location, x)
- for x in sorted(os.listdir(location), reverse=True))
-
- elif stat.S_ISREG(st.st_mode):
- yield location
-
-def load_config(conf_dirs, file_extensions=None):
- """
- Load JSON and/or YAML files from a directories, and merge them together
- into a single object.
- """
-
- result = {}
- for filename in iter_files(conf_dirs):
- if file_extensions is not None and not filename.endswith(file_extensions):
- continue
-
- loaders = []
- if filename.endswith('.json'):
- loaders.append(_json_load)
- elif filename.endswith('.yaml'):
- loaders.append(_yaml_load)
- else:
- loaders.append(_yaml_load)
- loaders.append(_json_load)
-
- config = None
- for loader in loaders:
- try:
- config = loader(filename) or {}
- except ConfigError as e:
- exception = e
- else:
- break
-
- if config is None:
- raise exception
-
- if config:
- result = merge_config(result, config)
-
- return result
+ """
+ Iterate over nested file paths in lexical order.
+ """
+ stack = list(reversed(files_dirs))
+ while stack:
+ location = stack.pop()
+ try:
+ st = os.stat(location)
+ except FileNotFoundError:
+ continue
+
+ if stat.S_ISDIR(st.st_mode):
+ stack.extend(os.path.join(location, x)
+ for x in sorted(os.listdir(location), reverse=True))
+
+ elif stat.S_ISREG(st.st_mode):
+ yield location
+
+def load_config(conf_dirs, file_extensions=None, valid_versions=None):
+ """
+ Load JSON and/or YAML files from a directories, and merge them together
+ into a single object.
+ """
+
+ result = {}
+ for filename in iter_files(conf_dirs):
+ if file_extensions is not None and not filename.endswith(file_extensions):
+ continue
+
+ loaders = []
+ if filename.endswith('.json'):
+ loaders.append(_json_load)
+ elif filename.endswith('.yaml'):
+ loaders.append(_yaml_load)
+ else:
+ loaders.append(_yaml_load)
+ loaders.append(_json_load)
+
+ config = None
+ for loader in loaders:
+ try:
+ config = loader(filename) or {}
+ except ConfigError as e:
+ exception = e
+ else:
+ break
+
+ if config is None:
+ raise exception
+
+ if config:
+ if config['version'] not in valid_versions:
+ raise ConfigError("Invalid file version: %s in: %s\nPlease upgrade repoman: current valid versions: %s"
+ % (config['version'], filename, valid_versions))
+ result = merge_config(result, config)
+
+ return result
^ permalink raw reply related [flat|nested] 32+ messages in thread
end of thread, other threads:[~2017-08-17 1:59 UTC | newest]
Thread overview: 32+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-06-27 20:06 [gentoo-commits] proj/portage:repoman commit in: repoman/pym/repoman/ Brian Dolbec
-- strict thread matches above, loose matches on Subject: below --
2017-08-17 1:59 Brian Dolbec
2017-08-06 6:54 Michał Górny
2017-07-15 2:29 Brian Dolbec
2017-07-15 2:29 Brian Dolbec
2017-07-15 2:08 Brian Dolbec
2017-07-15 2:08 Brian Dolbec
2017-07-10 23:01 Brian Dolbec
2017-07-10 23:01 Brian Dolbec
2017-07-10 23:01 Brian Dolbec
2017-07-10 22:31 Brian Dolbec
2017-07-10 22:31 Brian Dolbec
2017-07-10 22:31 Brian Dolbec
2017-07-10 22:31 Brian Dolbec
2017-07-10 22:31 Brian Dolbec
2017-07-10 22:31 Brian Dolbec
2017-07-10 17:52 Brian Dolbec
2017-07-10 17:52 Brian Dolbec
2017-07-10 17:52 Brian Dolbec
2017-06-27 22:10 Brian Dolbec
2017-06-27 22:10 Brian Dolbec
2017-06-27 22:10 Brian Dolbec
2017-06-27 21:36 Brian Dolbec
2017-06-27 21:36 Brian Dolbec
2017-06-27 21:36 Brian Dolbec
2017-06-27 21:36 Brian Dolbec
2017-06-27 20:06 Brian Dolbec
2017-06-27 20:06 Brian Dolbec
2016-05-15 23:51 [gentoo-commits] proj/portage:master " Brian Dolbec
2016-05-15 18:34 ` [gentoo-commits] proj/portage:repoman " Brian Dolbec
2016-05-15 8:40 Brian Dolbec
2016-05-15 4:13 Brian Dolbec
2016-05-15 2:37 Brian Dolbec
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox