Loading Python modules/plug-ins at runtime

Case 1: Loading installed python module

In case of SILPA I need to load pre-installed modules and here is the old code , that is a bit hacky code I copied from Active State Python recipies. I found a bit better way to do it using importlib module as shown below.

from __future__ import print_function
import sys
import importlib

def load_module(modulename):
    mod = None
    try:
        mod = importlib.import_module(modulename)
    except ImportError:
        print("Failed to load {module}".format(module=modulename),
                     file=sys.stderr)
    return mod

Here importlib itself takes care of checking if modulename is already loaded by checking sys.modules[modulename], if loaded it returns that value, otherwise it loads the module and sets it to sys.modules[modulename] before returning module itself.

Case 2: Loading python files from arbitrary location

In case of dictionary bot my requirement was bit different, I had some python files lying around in a directory, which I wanted to plug into the bot during run time and use them depending on some conditions. So basic structure which I was looking was as follows.

      pluginloader.py
      plugins
      |
      |__ aplugin.py
      |
      |__ bplugin.py

pluginloader.py is the file which needs to load python files under plugins directory. This was again done using importlib module as shown below.

import os
import sys
import re
import importlib

def load_plugins():
    pysearchre = re.compile('.py$', re.IGNORECASE)
    pluginfiles = filter(pysearchre.search,
                           os.listdir(os.path.join(os.path.dirname(__file__),
                                                 'plugins')))
    form_module = lambda fp: '.' + os.path.splitext(fp)[0]
    plugins = map(form_module, pluginfiles)
    # import parent module / namespace
    importlib.import_module('plugins')
    modules = []
    for plugin in plugins:
             if not plugin.startswith('__'):
                 modules.append(importlib.import_module(plugin, package="plugins"))

    return modules

Above code first searches for all python file under specified directory and creates a relative module name from it. For eg. file aplugin.py will become .aplugin. Before loading modules itself we will load the parent module in our case plugins, this is because relative imports in python expects parent module to be already loaded. Finally for relative imports to work with importlib.import_module we need to specify parent module name in package argument. Note that we ignore files begining with __, or specifically we don't want to import __init__.py, this will be done when we import parent module.

The above code was inspired from a answer on StackOverflow, which uses imp module, I avoided imp because its been deprecated from Python 3.4 in favor of importlib module.