I'm writing a Python application that takes a command as an argument, for example:
$ python myapp.py command1
I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:
myapp/
__init__.py
commands/
__init__.py
command1.py
command2.py
foo.py
bar.py
So I want the application to find the available command modules at runtime and execute the appropriate one.
Python defines an __import__()
function, which takes a string for a module name:
__import__(name, globals=None, locals=None, fromlist=(), level=0) The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name. Source: https://docs.python.org/3/library/functions.html#__import__
So currently I have something like:
command = sys.argv[1]
try:
command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
# Display error message
command_module.run()
This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.
Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.
dir(__import__)
. The fromlist should be a list of names to emulate "from name import ...".
importlib
: stackoverflow.com/a/54956419/687896
With Python older than 2.7/3.1, that's pretty much how you do it.
For newer versions, see importlib.import_module
for Python 2 and Python 3.
You can use exec
if you want to as well.
Or using __import__
you can import a list of modules by doing this:
>>> moduleNames = ['sys', 'os', 're', 'unittest']
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)
Ripped straight from Dive Into Python.
The recommended way for Python 2.7 and 3.1 and later is to use importlib
module:
importlib.import_module(name, package=None) Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).
e.g.
my_module = importlib.import_module('os.path')
__import__
function in favor of above mentioned module.
os.path
; how about from os.path import *
?
globals().update(my_module.__dict)
Note: imp is deprecated since Python 3.4 in favor of importlib
As mentioned the imp module provides you loading functions:
imp.load_source(name, path)
imp.load_compiled(name, path)
I've used these before to perform something similar.
In my case I defined a specific class with defined methods that were required. Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:
import imp
import os
def load_from_file(filepath):
class_inst = None
expected_class = 'MyClass'
mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])
if file_ext.lower() == '.py':
py_mod = imp.load_source(mod_name, filepath)
elif file_ext.lower() == '.pyc':
py_mod = imp.load_compiled(mod_name, filepath)
if hasattr(py_mod, expected_class):
class_inst = getattr(py_mod, expected_class)()
return class_inst
expected_class
so you could do class_inst = getattr(py_mod,expected_name)()
instead.
Nowadays you should use importlib.
Import a source file
The docs actually provide a recipe for that, and it goes like:
import sys
import importlib.util
file_path = 'pluginX.py'
module_name = 'pluginX'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# check if it's all there..
def bla(mod):
print(dir(mod))
bla(module)
This way you can access the members (e.g, a function "hello
") from your module pluginX.py
-- in this snippet being called module
-- under its namespace; E.g, module.hello()
.
If you want to import the members (e.g, "hello
") you can include module
/pluginX
in the in-memory list of modules:
sys.modules[module_name] = module
from pluginX import hello
hello()
Import a package
Importing a package (e.g., pluginX/__init__.py
) under your current dir is actually straightforward:
import importlib
pkg = importlib.import_module('pluginX')
# check if it's all there..
def bla(mod):
print(dir(mod))
bla(pkg)
sys.modules[module_name] = module
to the end of your code if you want to be able to import module_name
afterwards
Use the imp module, or the more direct __import__()
function.
If you want it in your locals:
>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'
same would work with globals()
locals()
function explicitly warns that the "contents of this dictionary should not be modified".
You can use exec
:
exec("import myapp.commands.%s" % command)
as command_module
to the end of import statement and then do command_module.run()
Similar as @monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:
import os
import imp
def importFromURI(uri, absl):
mod = None
if not absl:
uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
path, fname = os.path.split(uri)
mname, ext = os.path.splitext(fname)
if os.path.exists(os.path.join(path,mname)+'.pyc'):
try:
return imp.load_compiled(mname, uri)
except:
pass
if os.path.exists(os.path.join(path,mname)+'.py'):
try:
return imp.load_source(mname, uri)
except:
pass
return mod
The below piece worked for me:
>>>import imp;
>>>fp, pathname, description = imp.find_module("/home/test_module");
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();
if you want to import in shell-script:
python -c '<above entire code in one line>'
The following worked for me:
import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
fl[i] = fl[i].split('/')[1]
fl[i] = fl[i][0:(len(fl[i])-3)]
modulist.append(getattr(__import__(fl[i]),fl[i]))
adapters.append(modulist[i]())
It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:
class modu1():
def __init__(self):
self.x=1
print self.x
The result is a list of dynamically loaded classes "adapters".
fl
? The for loop definition is overly complex. The path is hard coded (and for an example, irrelevant). It's using discouraged python (__import__
), what's all that fl[i]
stuff for? This is basically unreadable, and is unnecessarily complex for something that is not all that hard - see the top voted answer with its one-liner.
Success story sharing
__import__
. The 2.7 docs: "Because this function is meant for use by the Python interpreter and not for general use it is better to use importlib.import_module()..." When using python3, theimp
module solves this proble, as monkut mentions below.foreach
when you havefor element in
andmap()
though :)