public inbox for gentoo-portage-dev@lists.gentoo.org
 help / color / mirror / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download: 
* [gentoo-portage-dev] proposed emaint changes
@ 2010-03-28  9:26 99% Brian Dolbec
  0 siblings, 0 replies; 1+ results
From: Brian Dolbec @ 2010-03-28  9:26 UTC (permalink / raw
  To: gentoo-portage-dev


[-- Attachment #1.1: Type: text/plain, Size: 3461 bytes --]

I have recently been doing a re-write of eclean and one of the problems
to address was the need to run "emaint fix packages" when doing any
cleaning of the packages directory.  Since eclean and emaint are coded
in python, I looked into importing the emaint code and running it from
within eclean.  Currently emaint is an all inclusive script without any
easily import-able modules. Asking Zac, he stated that it would be fine
to change that to a cli script and library.

So, going through the code, I found it pretty much was already
modularized but just stuck together in the one file.  In there was also
a TODO message:


# TODO: Create a system that allows external modules to be added without
#       the need for hard coding.
modules = {
	"world" : WorldHandler,
	"binhost":BinhostHandler,
	"moveinst":MoveInstalled,
	"movebin":MoveBinary,
	"cleanresume":CleanResume
}


So having a bit of experience with porthole's plug-in system, I took it
upon myself to create a small plug-in system. I also designed it to load
as little code as possible. With this plug-in system emaint actually
runs slightly faster than the original script. The only changes
being done were to use the plug-in system and independent module files
containing the original unmodified module class.  I believe the
speed increase is due to python not having to load as much script, as
only the module's __init__.py for all modules is loaded initially and
then the requested operation module file is loaded.

I think you will find this plug-in system very straightforward, perhaps
a little overkill for emaint. But it does offer a number of advantages
including less code changes to add/remove modules, full module usability
by other apps, etc..  Fully re-useable plug-in system for other apps
(when complete).


I have a bzr branch of portage's svn trunk with the modified emaint.  It
is fully functional for your testing and review of the code. I do have a
little more work to do on it.  So far I have not modified the code to
make use of the 'functions' and 'func_desc' data provided by the modules
__init__.py module_spec dictionary.  It is currently relying on the
original hard coded ['check', 'fix'] functions in the different module
classes.  It has also been suggested to make the modules dictionary a
LazyLoad dictionary instead.


I have a bzr branch available: 

bzr co http://dev.gentooexperimental.org/~dol-sen/bzr/portage-trunk/

If I did it correct, I also have migrated it to a git branch too:

git remote add emaint http://dev.gentooexperimental.org/~dol-sen/git/portage.git/
git fetch emaint
git branch emaint

and test/review it.


I have attached 2 module __init__.py's and the module.py file for
your perusal without the need to checkout a copy of my repo.

For those of you getting a bzr checkout or git remote to test.  Try
moving modules in and out of the modules dir and then run emaint -h.
Also the vdbkeys module is old and no longer usable, so will not be
included in the final version should it be approved and implemented.  I
include it here so you can see how easily modules can be added/removed.

I have also coded up a dummy module demonstrating the ability to
automatically change the class returned for a module name according to
an environment variable setting. But there are a number of other ways
that it could use too.  
-- 
Brian Dolbec <brian.dolbec@gmail.com>




[-- Attachment #1.2: module.py --]
[-- Type: text/x-python, Size: 5263 bytes --]

#!/usr/bin/python -O
# vim: noet :
#
# Copyright 2010 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2 or later
#
# $Header$


from __future__ import print_function

from portage import os
from portage.exception import PortageException


class InvalidModuleName(PortageException):
	"""An invalid or unknown module name."""


class Module(object):
	"""Class to define and hold our plug-in modules

	@type name: string
	@param name: the module name
	@type path: the path to the new module
	"""

	def __init__(self, name, path):
		"""Some variables initialization"""
		self.name = name
		self.path = path
		self.kids_names = []
		self.initialized = self._initialize()

	def _initialize(self):
		"""Initialize the plug-in module

		@rtype: boolean
		"""
		try:
			os.chdir(self.path)
			mod_name = ".".join(["emaint", "modules", self.name])
			self._module = __import__(mod_name, [],[], ["not empty"])
			self.valid = True
		except ImportError, e:
			self.valid = False
			return False
		self.module_spec = self._module.module_spec
		#self._module.get_class = self.get_class
		self.kids = {}
		for submodule in self.module_spec['provides']:
			kid = self.module_spec['provides'][submodule]
			kidname = kid['name']
			kid['module_name'] = '.'.join([mod_name, self.name])
			#kid['parent'] = self._module
			kid['is_imported'] = False
			self.kids[kidname] = kid
			self.kids_names.append(kidname)
		return True

	def get_class(self, name):
		if not name or name not in self.kids_names:
			raise InvalidModuleName("Module name '%s' was invalid or not"
				%modname + "part of the module '%s'" %self.name)
		kid = self.kids[name]
		if kid['is_imported']:
			module = kid['instance']
		else:
			try:
				module = __import__(kid['module_name'], [],[], ["not empty"])
				kid['instance'] = module
				kid['is_imported'] = True
			except ImportError, e:
				raise
			mod_class = getattr(module, kid['class'])
		return mod_class


class Modules(object):
	"""Dynamic modules system for loading and retrieving any of the
	installed emaint modules and/or provided class's

	@param path: Optional path to the "modules" directory or
			defaults to the directory of this file + '/modules'
	"""

	def __init__(self, path=None):
		if path:
			self.module_path = path
		else:
			self.module_path = os.path.join((
				os.path.dirname(os.path.realpath(__file__))), "modules")
		self.modules = self._get_all_modules()
		self.module_names = list(self.modules)
		self.module_names.sort()

	def _get_all_modules(self):
		"""scans the emaint modules dir for loadable modules

		@rtype: dictionary of module_plugins
		"""
		module_dir =  self.module_path
		importables = []
		names = os.listdir(module_dir)
		for entry in names:
			if os.path.isdir(os.path.join(module_dir, entry)):
				try:
					statinfo = os.stat(os.path.join(module_dir, entry, '__init__.py'))
					if statinfo.st_nlink == 1:
						importables.append(entry)
				except EnvironmentError, er:
					pass
		os.chdir(module_dir)
		kids = {}
		for entry in importables:
			new = Module(entry, module_dir) #, self)
			for module_name in new.kids:
				kid = new.kids[module_name]
				kid['parent'] = new
				kids[kid['name']] = kid
		return kids

	def get_module_names(self):
		"""Convienence function to return the list of installed modules
		available

		@rtype: list
		@return: the installed module names available
		"""
		return self.module_names

	def get_class(self, modname):
		"""Retrieves a module class desired

		@type modname: string
		@param modname: the module class name
		"""
		if modname and modname in self.module_names:
			mod = self.modules[modname]['parent'].get_class(modname)
		else:
			raise InvalidModuleName("Module name '%s' was invalid or not"
				%modname + "found")
		return mod

	def get_description(self, modname):
		"""Retrieves the module class decription

		@type modname: string
		@param modname: the module class name
		@type string
		@return: the modules class decription
		"""
		if modname and modname in self.module_names:
			mod = self.modules[modname]['description']
		else:
			raise InvalidModuleName("Module name '%s' was invalid or not"
				%modname + "found")
		return mod

	def get_functions(self, modname):
		"""Retrieves the module class  exported functions

		@type modname: string
		@param modname: the module class name
		@type list
		@return: the modules class exported functions
		"""
		if modname and modname in self.module_names:
			mod = self.modules[modname]['functions']
		else:
			raise InvalidModuleName("Module name '%s' was invalid or not"
				%modname + "found")
		return mod

	def get_func_descriptions(self, modname):
		"""Retrieves the module class  exported functions descriptions

		@type modname: string
		@param modname: the module class name
		@type list
		@return: the modules class exported functions descriptions
		"""
		if modname and modname in self.module_names:
			desc = self.modules[modname]['func_desc']
		else:
			raise InvalidModuleName("Module name '%s' was invalid or not"
				%modname + "found")
		return desc

[-- Attachment #1.3: __init__.py --]
[-- Type: text/x-python, Size: 944 bytes --]

#!/usr/bin/python -O
# vim: noet :
#
# Copyright 2010 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2 or later
#
# $Header$

"""'This emaint module provides checks and maintenance for:
  1) "Performing package move updates for installed packages",
  2)"Perform package move updates for binary packages"
"""


module_spec = {
	'name': 'move',
	'description': "Provides functions to check for and move packages " +\
		"either installed or binary packages stored on this system",
	'provides':{
		'module1': {
			'name': "moveinst",
			'class': "MoveInstalled",
			'description': "Perform package move updates for installed packages",
			'options': ['check', 'fix']
			},
		'module2':{
			'name': "movebin",
			'class': "MoveBinary",
			'description': "Perform package move updates for binary packages",
			'functions': ['check', 'fix'],
			'func_desc': {}
			}
		}
	}

[-- Attachment #1.4: __init__.py --]
[-- Type: text/x-python, Size: 1092 bytes --]

#!/usr/bin/python -O
# vim: noet :
#
# Copyright 2010 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2 or later
#
# $Header$

"""'This emaint program module provides checks for a test of 
environment set changes which change the class exported via the plug-in system.
"""

import os

DEFAULT_CLASS = "EnvOne"
AVAILABLE_CLASSES = [ "EnvOne",  "EnvTwo",  "EnvThree"]
options = {"1": "EnvOne", "2": "EnvTwo", "3": "EnvThree"}


config_class = DEFAULT_CLASS
try:
	test_param = os.environ["TESTIT"]
	if test_param in options:
		config_class = options[test_param]
except KeyError:
	pass


module_spec = {
	'name': 'testdummy',
	'description': "Provides functions to scan, check and " + \
		"Generate an example auto config module",
	'provides':{
		'module1': {
			'name': "envset",
			'class': config_class,
			'description':  "Dummy module used to test environment settings " + \
					"to change the class being exported by the plug-in system",
			'functions': ['check', 'fix'],
			'func_desc': {}
			}
		}
	}

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

^ permalink raw reply	[relevance 99%]

Results 1-1 of 1 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2010-03-28  9:26 99% [gentoo-portage-dev] proposed emaint changes Brian Dolbec

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox