Technical documentation with Tut

Long running branches are a personal pet peeve of mine: I’d much rather put the effort into managing feature flags, etc than I would into merging a branch that’s diverged significantly. So it’s with a mixture of embarrassment and relief that today I merged a branch that’s been hanging out for 3 years.

Four years ago I published Effective Django. As I was writing it, I knew I wanted to version control the text. As I worked with the code samples, however, it felt like I was having to work against my version control, rather than having it work for me: the code changes as the reader works through the content, and that’s not necessarily the same sequence of changes I’d make as I wrote. In effect, there were two timelines at work, and they didn’t match up. To make matters worse, it felt like I was repeating myself a lot: copying and pasting between source files when I wanted to start a new “chapter”, etc. Tut was my solution to this problem.

When I started updating Effective Django three years ago, I was aware there was still room for improvement: my Restructured Text source files were littered with specific line numbers from the source code, as well as prepend, pyobject, and end-before flags for my literal includes, all for the purpose of limiting the display to the changes. These felt brittle and inelegant: if I made a change in an early step of the tutorial (which tut made easy), all the line numbers could conceivably change; missing one was the sort of error that’s difficult to check without incredibly careful proofreading. It seemed like I should be able to get that information from the source control, so I set out to figure out how.


I think of the Effective Django tutorial, written using the first iteration of tut, as a form of “executable documentation”. It contains code samples, and the fact that those exist independently on disk means you can conceivably go to any point in time and run tests to make sure things are working. My first exposure to executable documentation was doctests; indeed, when I started thinking about tut, the place I started was reading the code for Sphinx’s doctest support.

As I thought about how to make my code inclusion more robust, I thought I wanted a “literate tut”: that is, a single document that contained both the text and source, from which I would weave the files if I wanted to execute them. I probably tried half a dozen approaches to make this work. Some “wove” the source files at Sphinx build time, another tried to marry Jupyter kernels with a Sphinx builder. In the end none of them worked perfectly, and — perhaps more importantly — none of them felt easy to author with. I knew I wasn’t going to use a tool for very long if it made life harder, no matter how elegant its approach.

Defeated, I decided to go back to where I started: using tut to manager source changes, but dropping the “tricks” I’d used to make inclusion work right. In other words, I stopped trying to be clever and just started writing.


In the years since I started the “literate tut” branch, I had made some improvements that seemed worth keeping around. I’d abstracted the git operations slightly and removed a lot of the global state. I’d also moved from trying to “infer” the order of steps based on history to maintaining an explicit list in a special configuration branch. These changes meant that as I started writing — and making progress — it was easier to try out a couple of ideas.

When Tut encounters a “checkpoint” it switches to a different git branch. This means that if the source tree is dirty, the build won’t succeed. After running into that a handful of times, I realized that I could use the same approach I used for fetching configuration (git show) to fetch a file from a specific branch, which potentially eliminates the need for playing chutes and ladders with head.

This got me thinking that I could grab two different versions of a file for side by side display, if I so desired. Combined with Python’s difflib module, however, it suddenly seemed like I had the basis for generating the sort of “documentation style” diffs I’d so tediously hand crafted before.


The result of this work is Tut 0.5, released today just over four years (!!) since the last release. This release includes a lot of changes, and is what I’m using as I work on an updated Effective Django, but I’m primarily releasing it to get the branch closed and focus myself on what else might be needed.

This release includes a new tut:literalinclude directive which operates exactly like the built-in one, with the exception that the content is read from git, not the filesystem. I’ve also added a tut:diff directive, which is able to format diffs for Python source files that do a reasonable(-ish) job of only showing the context that’s needed. The tut:diff directive can also show a link to the full file at that stage, so the entire contents are readily available.


I don’t think it’ll be another four years until the next release. As I work on the Effective Django update, I’m pretty sure I’ll need to improve diff generation and make it behave reasonably for non-Python files.

I thought I wanted “literate tut”: a single document that contained the diffs and text all in one. What I wound up with is something better: the ability to use the right tools to edit the text and code, and the ability to leverage the underlying framework to include exactly what I want as I write. You can find the new release of tut on PyPI; let me know what you think.

Effective Django at OSCON Post-Mortem

I’ve been on the road a little more than a week now, back to back conferences. On Tuesday I presented my Effective Django tutorial at OSCON. I’ve recently updated it to cover integrating static assets with your project, and re-organized some of the view information. The biggest challenge with presenting a tutorial like that is figuring out how fast (or slow) to go. I’ve practiced the tutorial and use Django daily, so of course I’m able to type and diagnose what’s going on more quickly. At the break on Tuesday a few people asked me to slow down, but as I walked around the room, two others told me (quite kindly) that they wouldn’t mind if I sped up. This was a little worse than at PyCon, I think, primarily because I didn’t have friends and colleagues assisting me. At PyCon they were able to wander the room and help backfill support for those who were moving a little slower.

I think the next time I deliver Effective Django — or any tutorial — I’ll probably try to mitigate with a few different ways:

  • Make sure I have assistants. I actually had one planned for OSCON, but he unfortunately fell ill. Next time I’ll try to have more than one scheduled (I had four at PyCon, three is probably fine).
  • Provide more guidance in the synopsis. If I’m going to assume no prior Django knowledge, I should probably say something like “We’ll start from zero and build a Django application together,” rather than just relying on the slightly ambiguous Novice tag in the program.
  • Provide some clear stopping points. I noticed that some attendees stayed with it for the first half, or first three quarters, but disengaged before the end. I tried to structure the tutorial so that you can stop at any time and still have learned something, but I could be more explicit about that. “And now we know how to write views that use the ORM,” rather than “We’ll see in a moment how to wire that into the …”
  • Make sure my handouts are ready to go. At OSCON I didn’t realize they were sitting in cardboard boxes off to the side until half way through. I noticed that in the second half, a couple of the people who had been struggling previously were keeping up slightly better once they had another source to refer to.

The people I spoke to afterward uniformly said they learned something (whether it was as much as they’d hoped is another question, I suppose), so this feels like an overall success.

author:Nathan Yergler
category:talks
tags:oscon, pyohio, effective django, python3
comments:

“Effective Django” at OSCON

I’m going to be presenting my introductory Django tutorial, Effective Django at OSCON later this month. If you’re going to be at OSCON and haven’t selected your tutorials yet, or just think a trip to Portland, Oregon sounds nifty, there’s still time to sign up. You can find the details on the OSCON tutorial page.

In preparation for that I’ve been continuing to work on the content. I presented the tutorial to some of the Eventbrite engineering team a couple weeks ago, and thier feedback was very useful. In response, I’ve made a few changes. Specifically, I split up the Views chapter with a brief interlude on static assets and template inheritance. It’s something that I didn’t cover the first time around, but based on the questions, I think some guidance is useful.

The revisions for OSCON also include updating the sample code repository that goes with the tutorial. I developed a tool, Tut, to help manage these stacked branches, and while making changes to early parts of the tutorial code, I realized it still requires significant work to really be a good workflow tool. One of the most important requirements for Tut is the ability to manage an ordered series of “checkpoints” and move between them.

When I started working on the sample code this time around, I was on a new laptop, so I had to start from a fresh clone. This was revealing and frustrating. I discovered Tut assumed all the branches were already local, which they obviously aren’t with a fresh clone. Worse, the git magic I was trying to use to get the branch list in the “right” order was pretty fragile, and broke when I tried to lean on it at all.

This screenshot shows a common editing case. My intention is to manage each step, or checkpoint, in the tutorial as a branch. Each step builds on the previous one, so if I make a change to something early in the tutorial, I just need to merge the branches “forward” until I get to master. In this example I’ve checked out the contact_form_test branch and added a new commit. In order for Tut to help me merge that forward, I need to be able to generate the list of steps.

The correct order here (last step first) is master, custom_form_rendering, contact_form_test, edit_addresses, address_model, confirm_contact_email, contact_detail_view. But you can’t get that out with either date or topo ordering. You really need to walk back from master, looking for branches (head refs), and at each step look for any head refs reachable (as children) that you haven’t already seen. I haven’t figured out how to do that yet with git plumbing commands, so for the time being I’m just using a text file to record the correct order. [1]

I’m really excited about presenting Effective Django at OSCON, and appreciate the feedback and suggestions from everyone.

[1]I contemplated just using a text file in the repo as the solution, but realized that this has its own issues: if it’s under version control as well, then what’s the “right” version to look at when you’re in a branch? That branch’s? Master’s? It’s not clear to me.
author:Nathan Yergler
category:Effective Django
tags:talks, effective django, python, tut,
comments:

“Effective Django” at PyCon 2013

PyCon (the US variation, at least), is about a month and half away, and once again I’m looking at the schedule of presentations and events and wondering how it is the community pulls it off every year. I’m also busy preparing my contribution to PyCon. This year I’m happy to be presenting a tutorial, Effective Django.

You may wonder what I mean by “Effective Django”. It’s an introduction to Django with a focus on good engineering practices. What I’ve noticed from my own experience over the years is that with all of its features and flexibility, Django makes it easy to get up and going really quickly. It also lets you write code that’s difficult to test, scale, and maintain. I have written plenty of code like that over the years, and the problem is that the real pain may come long after the initial implementation. From talking to engineers at Eventbrite and elsewhere I have learned that I’m not alone in this, so I’ve been working on documenting how to do leverage Django effectively. My goal is that attendees of the tutorial will leave feeling like they’re able to work on a Django application and identify things to do (and avoid) that will help them write code that’s cohesive, testable, and scalable.

I’m enjoying putting together the material for PyCon, and I hope that if you’re new to Django and interested in starting off on the right foot you’ll join me in Santa Clara for the afternoon.

If you’re totally new to Django and want to get a complete introduction to web apps and Django, Effective Django pairs well with Karen Rustad’s “Going from web pages to web apps“.

author:Nathan Yergler
category:pycon
tags:talks, effective django, python,
comments: