Monthly Archives: September 2009

Nested Formsets with Django

I’ve published an updated post about nested formsets, along with an generic implementation and demo application on GitHub.

I spent Labor Day weekend in New York City working on a side project with Alex. The project is coming together (albeit slowly, sometimes), and there have been a few interesting technical challenges. Labor Day weekend I was building an interface for editing data on the site. The particular feature I’m working on uses a multi-level data model; an example of this kind of model would be modeling City Blocks, where each Block has one or more Buildings, and each Building has one or more Tenants. Using this as an example, I was building the City Block editor.

Django Formsets manage the complexity of multiple copies of a form in a view. They help you keep track of how many copies you started with, which ones have been changed, and which ones should be deleted. But what if you’re working with this hypothetical data model and want to allow people to edit the Buildings and Tenants for a Block, all on one page? In this case you want each form in the Building formset to have a complete Tenant formset, all its own. The Django Formset documentation is silent on this issue, possibly (probably?) because it’s an edge case and one that almost certainly requires some application-specific thought. I spent the better part of two days working on it — the first pretty much a throw away, the second wildly productive thanks to TDD — and this is what I came up with.

Formsets act as wrappers around Django forms, providing the accounting machinery and convenience methods needed for managing multiple copies of the form. My experience has been that, unlike forms where you have to write your form class (no matter how simple), you write a Formset class infrequently. Instead you use the factory functions which generate a default that’s suitable for most situations. As with regular Forms and Model Forms, Django offers Model Formsets, which simplify the task of creating a formset for a form that handles instances of a model. In addition to model formsets, Django also provides inline formsets, which make it easier to deal with a set of objects that share a common foreign key. So in the example data model, an instance of the inline formset might model all the Buildings on a Block, or all the Tenants in the Building. Even if you’re not interested in nested formsets, the inline formsets can be incredibly useful.

Let’s go ahead and define the models for our example:

models.py

class Block(models.Model):
    description = models.CharField(max_length=255)

class Building(models.Model):
    block = models.ForeignKey(Block)
    address = models.CharField(max_length=255)

class Tenant(models.Model):
    building = models.ForeignKey(Building)
    name = models.CharField(max_length=255)
    unit = models.CharField(max_length=255)

After we have our models in place we need to define the forms. The nested form is straight-forward — it’s just a normal inline formset.

forms.py

from django.forms.models import inlineformset_factory

TenantFormset = inlineformset_factory(models.Building, models.Tenant, extra=1)

Note that inlineformset_factory not only creates the Formset class, but it also create the ModelForm for the model (models.Tenant in this example).

The “host” formset which contains the nested one — BuildingFormset in our example — requires some additional work. There are a few cases that need to be handled:

  1. Validation — When validating an item in the formset, we also need to validate its sub-items (those on its nested formset.
  2. Saving existing data — When saving an item, changes to the items in the nested formset also need to be saved.
  3. Saving new parent objects — If the user adds “parent” data as well as sub-items (so adding a Building, along with Tenants), the nested form won’t have a reference back to the parent unless we add it ourselves.
  4. Finally, the very basic issue of creating the nested formset instance for each parent form.

Before delving into those issues, let’s look at the basic formset declaration.

forms.py

from django.forms.models import BaseInlineFormSet

class BaseBuildingFormset(BaseInlineFormSet):
    pass

BuildingFormset = inlineformset_factory(models.Block, models.Building,
                                formset=BaseBuildingFormset, extra=1)

Here we declare a sub-class of the BaseInlineFormSet and then pass it to the inlineformset_factory as the class we want to base our new formset on.

Let’s start with the most basic piece of functionality: associating the nested formsets with each form. The super class defines an add_fields method which is responsible for adding the fields (and their initial values since this is a model-based Form) to a specific form in the formset. This seemed as good a place as any to add our formset creation code.

forms.py

class BaseBuildingFormset(BaseInlineFormSet):

    def add_fields(self, form, index):
        # allow the super class to create the fields as usual
        super(BaseBuildingFormset, self).add_fields(form, index)

        # created the nested formset
        try:
            instance = self.get_queryset()[index]
            pk_value = instance.pk
        except IndexError:
            instance=None
            pk_value = hash(form.prefix)

        # store the formset in the .nested property
        form.nested = [
            TenantFormset(data=self.data,
                            instance = instance,
                            prefix = 'TENANTS_%s’ % pk_value)]

The heart of what we’re doing here is in the last statement: creating a form.nested property that contains a list of nested formsets — only one in our example and in the code I implemented; more than one would probably be a UI nightmare. In order to initialize the formset we need two pieces of information: the parent instance and a form prefix. If we’re creating fields for an existing instance we can use the get_queryset method to return the list of objects. If this is a form for a new instance (i.e., the form created by specifying extra=1), we need to specify None as the instance. We include the objects primary key in the form prefix to make sure the formsets are named uniquely; if this is an extra form we hash the parent form’s prefix (which will also be unique). The Django documentation has instructions on using multiple formsets in a single view that are relevant here.

Now that we have the nested formset created we can display it in the template.

views.py

def edit_block_buildings(request, block_id):
    """Edit buildings and their tenants on a given block."""

    block = get_object_or_404(models.Block, id=block_id)

    if request.method == 'POST’:
        formset = forms.BuildingFormset(request.POST, instance=block)

        if formset.is_valid():
            rooms = formset.save_all()

            return redirect('block_view’, block_id=block.id)

    else:
        formset = forms.BuildingFormset(instance=block)

    return render_to_response('rentals/edit_buildings.html’,
                              {'block’:block,
                               'buildings’:formset,
                               },
                              context_instance=RequestContext(request))

edit_buildings.html (fragment)

{{ buildings.management_form }}
{% for building in buildings.forms %}

  {{ building }}

  {% if building.nested %}
  {% for formset in building.nested %}
  {{ formset.as_table }}
  {% endfor %}
  {% endif %}

{% endfor %}

When the page is submitted, the idiom is to call formset.is_valid() to validate the forms. We override is_valid on our formset to add validation for the nested formsets as well.

forms.py

class BaseBuildingFormset(BaseInlineFormSet):
    ...

    def is_valid(self):
        result = super(BaseBuildingFormset, self).is_valid()

        for form in self.forms:
            if hasattr(form, 'nested’):
                for n in form.nested:
                    # make sure each nested formset is valid as well
                    result = result and n.is_valid()

        return result

Finally, assuming the form validates, we need to handle saving. As I mentioned earlier, there are two different situations here — saving existing data (and possibly adding new nested data) and saving completely new data.

For new data we need to override save_new and update the parent reference for any nested data after we save (well, instantiate) the parent.

forms.py

class BaseBuildingFormset(BaseInlineFormSet):
    ...

    def save_new(self, form, commit=True):
        """Saves and returns a new model instance for the given form."""

        instance = super(BaseBuildingFormset, self).save_new(form, commit=commit)

        # update the form’s instance reference
        form.instance = instance

        # update the instance reference on nested forms
        for nested in form.nested:
            nested.instance = instance

            # iterate over the cleaned_data of the nested formset and update the foreignkey reference
            for cd in nested.cleaned_data:
                cd[nested.fk.name] = instance

        return instance

Finally, we add a save_all method for saving the parent formset and all nested formsets.

forms.py

from django.forms.formsets import DELETION_FIELD_NAME

class BaseBuildingFormset(BaseInlineFormSet):
    ...

    def should_delete(self, form):
        """Convenience method for determining if the form’s object will
        be deleted; cribbed from BaseModelFormSet.save_existing_objects."""

        if self.can_delete:
            raw_delete_value = form._raw_value(DELETION_FIELD_NAME)
            should_delete = form.fields[DELETION_FIELD_NAME].clean(raw_delete_value)
            return should_delete

        return False

    def save_all(self, commit=True):
        """Save all formsets and along with their nested formsets."""

        # Save without committing (so self.saved_forms is populated)
        # — We need self.saved_forms so we can go back and access
        #    the nested formsets
        objects = self.save(commit=False)

        # Save each instance if commit=True
        if commit:
            for o in objects:
                o.save()

        # save many to many fields if needed
        if not commit:
            self.save_m2m()

        # save the nested formsets
        for form in set(self.initial_forms + self.saved_forms):
            if self.should_delete(form): continue

            for nested in form.nested:
                nested.save(commit=commit)

There are two methods defined here; the first, should_delete, is lifted almost directly from code in django.forms.models.BaseModelFormSet.save_existing_objects. It takes a form object in the formset and returns True if the object for that form is going to be deleted. We use this to short-circuit saving the nested formsets: no point in saving them if we’re going to delete their required ForeignKey.

The save_all method is responsible for saving (updating, creating, deleting) the forms in the formset, as well as all the nested formsets for each form. One thing to note is that regardless of whether we’re committing our save (commit=True), we initially save the forms with commit=False. When you save a model formset with commit=False, Django populates a saved_forms attribute with the list of all the forms saved — new and old. We need this list of saved forms to make sure we are able to save any nested formsets that are attached to newly created forms (ones that did not exist when the initial request was made). After we know saved_forms has been populated we can do another pass to commit if necessary.

There are certainly places this code could be improved, tightened up or generalized (for example, the nested formset prefix calculation and possibly save_all). It’s also entirely plausible that you could wrap much of this into a factory function. But this gets nested editing working and once you wrap your head around what needs to be done, it’s actually fairly straight forward.

date:2009-09-27 19:42:42
wordpress_id:1093
layout:post
slug:nested-formsets-with-django
comments:
category:development, koucou
tags:django, formsets, howto, orm, python

Pet Shop Boys at The Warfield, San Francisco

Wednesday evening I continued my ten year concert reunion with the Pet Shop Boys. Currently performing on their Pandemonium Tour (supporting Yes), James and I (along with several other friends) caught their second night at The Warfield in downtown San Francisco.

Pet Shop Boys (flickr)

Wednesday’s show exceeded my expectations. They opened with “Heart” and and barely paused for just over 90 minutes, touching on nearly every part of their catalog. Their performance of “Heart”, in particular, was really exciting and energetic. The set included several medleys; I tried to keep a set list as the night went on and ended up with the following (titles in square brackets indicate they just played snippets or a few lines of the song as a transition between numbers).

  • Heart
  • Did You See Me Coming
  • Pandemonium/Can You Forgive Her
  • Love, etc
  • [Integral, Building a Wall]
  • Go West
  • Two Divided by Zero
  • Why Don’t We Live Together? / Always On My Mind
  • New York City Boy
  • [Closer to Heaven]
  • Left to my Own Devices
  • Do I Have To?
  • King’s Cross
  • The way It Used To Be
  • Jealousy
  • Suburbia
  • All Around the World
  • Se A Vida É / Discoteca
  • Viva la Vida (Coldplay cover)
  • It’s a Sin

Encore:

  • Being Boring
  • West End Girls

I really enjoyed hearing songs that I didn’t at all expect to be played, but which feel like iconic PSB songs to me: “Two Divided by Zero”, “Suburbia”, “King’s Cross”. And of course, “It’s a Sin” and “Being Boring”, which we completely expected and yet still felt relevant and poignant; PSB is without question the band that marks time in my coming out story.

I last saw PSB on their Nightlife tour in 1999. That day, John and I drove three and a half hours from Fort Wayne to the Fox Theater in Detroit, watched the show from the balcony, and drove back the same night, arriving home at nearly four in the morning. My car was hit by a deer on the drive back that morning when we were less than fifteen minutes from home[1]_. In contrast, Wednesday night I carpooled ten minutes with friends from our pre-show pizza and could have walked home in under half an hour.

I somehow see the change in distance as metaphorical for how I’ve moved in my own life, as well. I remember thinking that night that there was no way I could live in Detroit; it seemed overwhelming and foreign. I suppose in the same way if I had been able to look forward ten years, my life today would have been similarly unrecognizable. As I think about it now, I guess I’m lucky; I was happy[2] with where I was then, and I’m happy to be where I am today. There are still things I’d like to do and improve but the present is pretty damn good.


[1]Yes, the deer hit me. I was at a dead stop. No, the insurance agent didn’t really believe me, either.
[2]Sure, I wanted some things to be different, but I was out, I had my own place, and I had my friends.
date:2009-09-25 16:05:12
wordpress_id:1120
layout:post
slug:pet-shop-boys-at-the-warfield-san-francisco
comments:
category:my life
tags:concerts, music, pet shop boys, psb

PyCon 2010 CFP: Five Days Left

The CFP for PyCon 2010 closes in five days. I’m on the program committee this year and it’s exciting to see good proposals come in. From the CFP:

Want to showcase your skills as a Python Hacker? Want to have hundreds of people see your talk on the subject of your choice? Have some hot button issue you think the community needs to address, or have some package, code or project you simply love talking about? Want to launch your master plan to take over the world with python?

PyCon is your platform for getting the word out and teaching something new to hundreds of people, face to face.

Previous PyCon conferences have had a broad range of presentations, from reports on academic and commercial projects, tutorials on a broad range of subjects and case studies. All conference speakers are volunteers and come from a myriad of backgrounds. Some are new speakers, some are old speakers. Everyone is welcome so bring your passion and your code! We’re looking to you to help us top the previous years of success PyCon has had.

PyCon 2010 is looking for proposals to fill the formal presentation tracks. The PyCon conference days will be February 19-22, 2010 in Atlanta, Georgia, preceded by the tutorial days (February 17-18), and followed by four days of development sprints (February 22-25).

Online proposal submission is open now! Proposals will be accepted through October 1st, with acceptance notifications coming out on November 15th. For the detailed call for proposals, please see:

http://us.pycon.org/2010/conference/proposals/

For videos of talks from previous years – check out:

http://pycon.blip.tv

We look forward to seeing you in Atlanta!

date:2009-09-25 15:07:50
wordpress_id:1113
layout:post
slug:pycon-2010-cfp-five-days-left
comments:
category:conf
tags:cfp, conference, pycon, python