Plugins using Eggs

Snake with egg
Presentation:Ian Bicking

Eggs

Q: What are Eggs?

Eggs

Q: What are Eggs?

A: They are a kind of package you download

Eggs

Q: What are Eggs?

A: No, I mean they are format for installation that pkg_resources knows how to read

So...

Q: What's this pkg_resources ?

So...

Q: What's this pkg_resources ?

A: It's a module for working with Eggs

So...

Q: What's this pkg_resources ?

A: It's in setuptools.

Getting to the point

Q: What's this setuptools ?

Getting to the point

Q: What's this setuptools ?

A: It builds Eggs.

Now you are annoying me

Q: What's this setuptools ?

A: It's really the package that does all of "this"

Almost there...

Q: What is "this"?

Almost there...

Q: What is "this"?

A: Plugins

Plugins

Q: What's a plugin?

Plugins

Q: What's a plugin?

A: Ok, let's stop now

So Many Names!

Scary snake

Some of the concepts:

More Names!

Less scary snake

...

But really it's not that bad. We Love Egg!

happy eggs

Anatomy of an Egg

From a source distribution::

MyPackage/
  MyPackage.egg-info/
    requires.txt
    top_level.txt
    entry_points.txt
  mypackage/
    __init__.py (etc)

Anatomy of an Egg

An Egg installed:

site-packages/
  MyPackage-1.0-py2.4.egg/
    EGG-INFO/
      requires.txt (etc)
    mypackage/
      __init__.py  (etc)

Importing

Your package is only importable when the Egg directory is on sys.path...

site-packages/
  MyPackage-1.0-py2.4.egg/
    mypackage/
      __init__.py  (etc)

Importing

site-packages/easy-install.pth adds entries to sys.path -- a list of installed Eggs that are activated by default:

/Users/ianb/co/Paste
/Library/Frameworks/Python.framework/Versions/2.4/lib/
...python2.4/site-packages/RuleDispatch-0.5a0.dev-
...py2.4-macosx-10.3-ppc.egg
/Library/Frameworks/Python.framework/Versions/2.4/lib/
...python2.4/site-packages/PyProtocols-1.0a0-py2.4-
...macosx-10.3-ppc.egg

Distributions

Metadata

Metadata files in .egg-info (or EGG-INFO):

PKG-INFO:
Readable package name, description, etc (what gets uploaded to the Cheese Shop/PyPI)
requires.txt:
Other distributions/packages that are required
top_level.txt:
The top-level Python packages
entry_points.txt:
A list of "entry points" (more later)

Users can add more metadata in .egg-info

Plugins

Back to the topic of plugins...

Plugins

Models:

  1. Extend an application:
    • The plugin is dependent on the application
    • The application owns the process/environment
  2. Cooperative packages:
    • Packages provide things to each other
    • Plugin system as neutral ground

Entry Points

A list of public objects in a distribution/Egg; example setup.py:

setup(...
    entry_points="""
    [console_scripts]
    paster = paste.script.command:run

    [paste.app_factory]
    test = paste.script.testapp:TestApplication
    """)

Entry Points

Objects sorted into "groups":

Entry Points

Entry Points

Loading an entry point:

import pkg_resources
ob = pkg_resources.load_entry_point(
    'Commentary', 'paste.app_factory', 'main')
wsgiapp = ob({}, storage='/tmp')
'Commentary':
Distribution name
'paste.app_factory':
The entry point group (the type of object we are expecting)
'main':
The name of the entry point

There are several other functions in pkg_resources for listing and inspecting entry points

Finding Plugins

Several strategies for finding plugins -

Explicit enumeration: a list of plugins to enable; specified with:

Finding Plugins

Scan for plugins:

Scanning for plugins

Entry Point Metadata

Entry points have metadata:

Avoiding dependencies

Avoiding Dependencies

Avoiding fragility

Manageable plugins

Plugins do not inject themselves

Injection example (this is bad!):

# Activate a plugin:
__import__(plugin_name)

With a bad plugin:

# install self:
import master_application
from plugin_name import listener
master_application.event_listeners.append(
    listener.Listener(master_application.app_instance))

Flexible plugins

Objects loaded as entry points should act, they should not be read

Example:

app_installer = load_entry_point(
    'AnApplication', 'paste.app_installer', 'main')
app_installer.install(cmd, output_dir, vars)

Not:

vars = load_entry_point(...)
copy_dir(vars.default_templates, output_dir + '/templates')
install_database(dbname, vars.initial_db_data)

Declarative

But I like declarative...

Abstract Base Class

Abstract base classes are one solution:

from appinstaller import AbstractInstaller

class MyInstaller(AbstractInstaller):
    default_templates = 'templates/'
    initial_db_data = [...]

Self-referencing entry points

For people who are even Too Lazy To Use An ABC:

[paste.app_install]
mypackage = paste.script.appinstall:Installer

Using that plugin:

ob = pkg_resources.load_entry_point(...)
ob = Installer(MyPackageDist, 'paste.app_install',
               'mypackage')

Not-abstract base class

What the ABC looks like...

class Installer(object): ...
    def config_content(self, command, vars):
        meta_name = 'paste_deploy_config.ini_tmpl'
        tmpl = Template(self.dist.get_metadata(meta_name),
                        searchList=[vars])
        return tmpl(**vars)

Another example

And another ABC technique...

class Installer(object): ...
    def packages(self):
        for line in self.dist.get_metadata_lines('top_level.txt'):
            line = line.strip()
            if not line and not line.startswith('#'):
                yield line
    def setup_config(self, ...):
        for package in self.packages():
            try:
                mod = import_module(mod_name)
            except ImportError:
                continue
            mod.setup_config(...)

Configurable pieces

"Paste Deploy" represents one pattern for creating objects, for web applications:

[app:someapp]
use = egg:SomePackage#somepoint
foo = bar

Means:

ob = pkg_resources.load_entry_point(
    'SomePackage', 'paste.app_factory', 'somepoint')
app = ob(global_conf, foo='bar')

That's probably it

I'm probably out of time; I didn't talk about:

Installation...

For your users to install plugins:

$ easy_install -f http://plugin-index... \
  -d plugin_dir \
  PluginName

Plugin Index

A plugin index is:

Installation...

easy_install:

Plugin Directory

Plugin Directory

OK...

Now I've petered out, hopefully we didn't get this far.