Application Components

The core of the extensible approach comes from the zope.component package. At the highest level of abstraction, this package provide a component model and an event/dispatch mechanism. The component model provides a mechanism for making declarations about what objects support and how they can be adapted, as well as a mechanism for registering adapters or subscribers for particular types of objects. The event/dispatch mechanism provides a basic way for objects to observe one another and notify their observers about changes. In practice the line between the two is very blurry.

The application architecture of ccPublisher 2 consists of a central framework package, P6, which provides the common component and event functionality that would likely be reused by derivative applications. The ccpublisher package contains ccPublisher-specific code, such as the Internet Archive backend implementation. In both the P6 and ccPublisher code, we view the application as a set of components which communicate with each other through minimal interface contracts or the event framework.

Examining how the application handles user entry of metadata provides a good overview of how the application is broken into components and how it leverages the Zope 3 packages to provide maximum flexibility. The P6 framework defines a window class which provides the wizard functionality, and a generic page which implements editting for a single metadata group (a set of fields). Focusing on the metadata group, we have the following set of actions taking place:

1. Retrieve the list of fields to edit.
2. Display the fields, along with any default or pre-defined values.
3. After the user finishes editting the fields, storing the new values.

Looking at these tasks, there are obviously other questions lying beneath the surface. For example, what portion of the application is responsible for saving the field values, and where do we get our list of defaults. A benefit of dividing the application into segmented components is that we can be willfully ignorant about these questions. As we will see, the metadata editting page simply deals with its own core functionality, leaving the other details to other portions of the framework.

The metadata editor takes a MetadataGroup object as a parameter to its constructor, and this group provides us with the list of fields to edit. We will see later how the MetadataGroup is instantiated and initialized. When the framework constructs the user interface object, it checks each field to find out if its persistance flag is set [2]. If the flag is set, the framework publishes a LoadMetadata event. This event is a notification to any metadata provider to load the particular field.

if field.persist:
    event = p6.metadata.events.LoadMetadataEvent(
               self.metagroup.appliesTo,
               self.metagroup, field,
            )

    zope.component.handle(event)

The event itself is not as interesting as the way the application handles it. Any component can register a subscriber for a particular event type and be notified when those events are published. So the application does not necessarily know where the persisted data is stored, or even if it is stored. It just provides the hook for the storage and retrieval to happen.

An interesting side-effect of this architecture is that an application extension may provide integration with other data sources simply by listening for and responding to events. For example, login credentials could potentially be integrated with an external user source by an extension which listened for the appropriate events and provided the correct credential information.

Once the user has editted the metadata fields, the framework performs the reverse process, storing the values back to a container. As with the initial load, the framework simply constructs an event and publishes it.

zope.component.handle(
    p6.metadata.events.UpdateMetadataEvent(self.metagroup.appliesTo,
                                           field,
                                           widget.GetValue()
                                           )
    )

This demonstrates how publishing events can dramatically decouple an application, but does not expose how objects are initially instantiated and registered. Our initial implementation placed the initialization code in the application class. While this was an improvement in that the code was self-contained, we again found that it duplicated effort between the "core" application functionality and "extension" functionality. In order to use the same type of solution for both, an additional Zope 3 package was included, configuration.

Assembling An Application

The zope.configuration package provides basic support for the ZCML configuration language. ZCML is an XML-ish language defined by the Zope 3 project for the purpose of configuring applications. Another way to look at this is that the code defines components, and the ZCML assembles them into the completed, functional application. ZCML can be extended with directives which are declared as a combination of interfaces and a Python implementation which is called when an instance of that directive is encountered.

P6 uses the core ZCML directives for registering adapters and subscribers for events, and also extends ZCML for some domain-specific configuration tasks. P6 uses a file called app.zcml for the core application instantiation. The app.zcml file always starts with a <configure> directive, which is the container for the entire configuration. The only additional requirement is that the file include the line

<include package="p6" />

before any application-specific directives. The include tag tells the ZCML parser to traverse into the Python package named and look for a file named configure.zcml, which is in turned parsed. With this single include statement the application loads the domain-specific directives and initializes the necessary data structures for loading the application.

When an application such as ccPublisher 2 is configured, there are three primary areas we're concerned with instantiating: storage, metadata, and user interface. Storage is the term given to backends such as the Internet Archive. An application must have at least one storage instance. A storage class is instantiated with a directive such as

<storage
   name="NoOp Storage"
   factory="p6.storage.basic.BasicStorage"
   />

The name attribute is purely for logging and reporting; it is never exposed to the end user. The factory attribute specifies a callable which returns an object implementing the IStorage interface. In this case the factory is actually a class definition, so the framework just calls the constructor when it encounters this directive. There are specific events which a storage provider should register for; new storage implementations can get this behavior by simply subclassing the BasicStorage class.

ccPublisher has metadata collection as a core piece of functionality. Recognizing that derivatives such as Ourmedia Publisher often want to collect different sets of metadata, the declaration for these fields was moved into the ZCML as well. Metadata fields are separated into groups, and each group is displayed as a page in the wizard. In addition to providing logical separation of fields, groups allows application architects to declare what sort of objects a metadata field is collected for (applies to). For example, ccPublisher collects some metadata for the entire work (which may include more than one file). Items such as author, title and year of copyright apply to the work as a whole. However, other pieces of metadata, such as file format, apply only to specific files. Looking at the metadatagroup declaration shows how this container is created.

<metadatagroup
   id="workinfo"
   title="Tell Us About Your Files"
   description="Metadata helps others to find your works."
   for="p6.storage.interfaces.IWork"
   >

In this declaration, the id must be unique among all metadatagroups; the title is used for the user interface, and the for attribute specifies the interface which the contained fields apply to. The specific field declarations take the following format.

<field id="title"
       label="Title of Work"
       type="p6.metadata.types.ITextField"
       validator="ccpublisher.validators.validateTitle"
       canonical="http://purl.org/dc/elements/1.1/title"
       />

This field declaration actually contains some optional elements. The id, label and type are all required; validator and canonical are optional. The type attribute specifies the interface which this field should conform to; in this case a text field. Interfaces current exist for choice fields and text fields; we specify an interface instead of a class, as the actual implementation is considered toolkit specific [3].

The validator attribute specifies a callable which, when passed the value for this field, returns True of False. The canonical attribute is a little more opaque. The canonical field provides a canonical URI (in this case the Dubline Core URI) for this type of field. This is primarily there for extension authors; since the field id and group id are arbitrary, if an extension wants the title of a work, they need some key to retrieve it by. The canonical attribute serves this purpose.

Finally, the P6 framework configures the application user interface through ZCML as well. Since the framework was designed for wizard-like applications, the container we use is the `pages` directive.

<pages appid="ccpublisher">

  <page factory="ccpublisher.ui.WelcomePage" />

  <fileselector />

  <metadatapages
      for="p6.storage.interfaces.IWork
           p6.storage.interfaces.IWorkItem
           p6.storage.interfaces.IStorage"  />

  <storepage  />

</pages>

This code sample is the entire user interface declaration for ccPublisher 2. It demonstrates the three types of pages: built in, metadata and custom. The `fileselector` and `storepage` directives are both built-in page types. The file selector page simply allows the user to select one or more files, and the store page provides the progress bar and user feedback during the upload process [4].

The `metadatapages` directive instructs P6 to generate the necessary pages for the metadata fields which apply to the specified interfaces (for). In another instance of willful ignorance, its interesting to note that the directive makes no assumptions regarding how many pages there are, or how many fields are defined for each page.

Finally, the simple `page` directive defines a custom page, in this case one specified by our application package (ccpublisher). A custom page factory returns an object which subclasses the XrcWizPage class provided by the ccwx package.

Packaging and Distribution

While separating an application into more independent components and stitching them together with ZCML has proven advantagous to productivity, it proves problematic from a packaging perspective. Previous release of ccPublisher used py2app on Mac OS X and py2exe on Windows platforms [5]. There are two primary problems, which are interrelated. The problems are bootstrapping an application which is not "assembled" until run time, and properly finding the necessary ZCML files for finalizing the configuration.

For developing ccPublisher we chose to punt on the first issue, and created a script which knows what the application-level class is and how to find the app.zcml file for further configuration. The application class constructor handles actually loading the file and the other necessary instantiation.

The second problem is more difficult to deal with. First, in regards to Windows and py2exe: py2exe bundles the Python byte code files specified by disutils into library.zip, and then loads them using the zipfile import mechanism at runtime. Unfortunately, specifying the ZCML files as plain distutils data files places them in library.zip. The ZCML file loading mechanism is unfortunately unable to properly traverse into zip files for loading ZCML.

To solve the problem on Windows platforms, we specify the required ZCML files as both package data and data files: package data is placed along side the packages in library.zip, and the data files are placed alongside the executable. The ZCML file loader is then monkey-patched so that if it can not find the file in the regular Python path, it looks in the "side by side" path before declaring the file missing.