Technical documentation with Tut

Long running branches are a personal pet peeve of mine: I’d much rather put the effort into managing feature flags, etc than I would into merging a branch that’s diverged significantly. So it’s with a mixture of embarrassment and relief that today I merged a branch that’s been hanging out for 3 years.

Four years ago I published Effective Django. As I was writing it, I knew I wanted to version control the text. As I worked with the code samples, however, it felt like I was having to work against my version control, rather than having it work for me: the code changes as the reader works through the content, and that’s not necessarily the same sequence of changes I’d make as I wrote. In effect, there were two timelines at work, and they didn’t match up. To make matters worse, it felt like I was repeating myself a lot: copying and pasting between source files when I wanted to start a new “chapter”, etc. Tut was my solution to this problem.

When I started updating Effective Django three years ago, I was aware there was still room for improvement: my Restructured Text source files were littered with specific line numbers from the source code, as well as prepend, pyobject, and end-before flags for my literal includes, all for the purpose of limiting the display to the changes. These felt brittle and inelegant: if I made a change in an early step of the tutorial (which tut made easy), all the line numbers could conceivably change; missing one was the sort of error that’s difficult to check without incredibly careful proofreading. It seemed like I should be able to get that information from the source control, so I set out to figure out how.


I think of the Effective Django tutorial, written using the first iteration of tut, as a form of “executable documentation”. It contains code samples, and the fact that those exist independently on disk means you can conceivably go to any point in time and run tests to make sure things are working. My first exposure to executable documentation was doctests; indeed, when I started thinking about tut, the place I started was reading the code for Sphinx’s doctest support.

As I thought about how to make my code inclusion more robust, I thought I wanted a “literate tut”: that is, a single document that contained both the text and source, from which I would weave the files if I wanted to execute them. I probably tried half a dozen approaches to make this work. Some “wove” the source files at Sphinx build time, another tried to marry Jupyter kernels with a Sphinx builder. In the end none of them worked perfectly, and — perhaps more importantly — none of them felt easy to author with. I knew I wasn’t going to use a tool for very long if it made life harder, no matter how elegant its approach.

Defeated, I decided to go back to where I started: using tut to manager source changes, but dropping the “tricks” I’d used to make inclusion work right. In other words, I stopped trying to be clever and just started writing.


In the years since I started the “literate tut” branch, I had made some improvements that seemed worth keeping around. I’d abstracted the git operations slightly and removed a lot of the global state. I’d also moved from trying to “infer” the order of steps based on history to maintaining an explicit list in a special configuration branch. These changes meant that as I started writing — and making progress — it was easier to try out a couple of ideas.

When Tut encounters a “checkpoint” it switches to a different git branch. This means that if the source tree is dirty, the build won’t succeed. After running into that a handful of times, I realized that I could use the same approach I used for fetching configuration (git show) to fetch a file from a specific branch, which potentially eliminates the need for playing chutes and ladders with head.

This got me thinking that I could grab two different versions of a file for side by side display, if I so desired. Combined with Python’s difflib module, however, it suddenly seemed like I had the basis for generating the sort of “documentation style” diffs I’d so tediously hand crafted before.


The result of this work is Tut 0.5, released today just over four years (!!) since the last release. This release includes a lot of changes, and is what I’m using as I work on an updated Effective Django, but I’m primarily releasing it to get the branch closed and focus myself on what else might be needed.

This release includes a new tut:literalinclude directive which operates exactly like the built-in one, with the exception that the content is read from git, not the filesystem. I’ve also added a tut:diff directive, which is able to format diffs for Python source files that do a reasonable(-ish) job of only showing the context that’s needed. The tut:diff directive can also show a link to the full file at that stage, so the entire contents are readily available.


I don’t think it’ll be another four years until the next release. As I work on the Effective Django update, I’m pretty sure I’ll need to improve diff generation and make it behave reasonably for non-Python files.

I thought I wanted “literate tut”: a single document that contained the diffs and text all in one. What I wound up with is something better: the ability to use the right tools to edit the text and code, and the ability to leverage the underlying framework to include exactly what I want as I write. You can find the new release of tut on PyPI; let me know what you think.

“Effective Django” at OSCON

I’m going to be presenting my introductory Django tutorial, Effective Django at OSCON later this month. If you’re going to be at OSCON and haven’t selected your tutorials yet, or just think a trip to Portland, Oregon sounds nifty, there’s still time to sign up. You can find the details on the OSCON tutorial page.

In preparation for that I’ve been continuing to work on the content. I presented the tutorial to some of the Eventbrite engineering team a couple weeks ago, and thier feedback was very useful. In response, I’ve made a few changes. Specifically, I split up the Views chapter with a brief interlude on static assets and template inheritance. It’s something that I didn’t cover the first time around, but based on the questions, I think some guidance is useful.

The revisions for OSCON also include updating the sample code repository that goes with the tutorial. I developed a tool, Tut, to help manage these stacked branches, and while making changes to early parts of the tutorial code, I realized it still requires significant work to really be a good workflow tool. One of the most important requirements for Tut is the ability to manage an ordered series of “checkpoints” and move between them.

When I started working on the sample code this time around, I was on a new laptop, so I had to start from a fresh clone. This was revealing and frustrating. I discovered Tut assumed all the branches were already local, which they obviously aren’t with a fresh clone. Worse, the git magic I was trying to use to get the branch list in the “right” order was pretty fragile, and broke when I tried to lean on it at all.

This screenshot shows a common editing case. My intention is to manage each step, or checkpoint, in the tutorial as a branch. Each step builds on the previous one, so if I make a change to something early in the tutorial, I just need to merge the branches “forward” until I get to master. In this example I’ve checked out the contact_form_test branch and added a new commit. In order for Tut to help me merge that forward, I need to be able to generate the list of steps.

The correct order here (last step first) is master, custom_form_rendering, contact_form_test, edit_addresses, address_model, confirm_contact_email, contact_detail_view. But you can’t get that out with either date or topo ordering. You really need to walk back from master, looking for branches (head refs), and at each step look for any head refs reachable (as children) that you haven’t already seen. I haven’t figured out how to do that yet with git plumbing commands, so for the time being I’m just using a text file to record the correct order. [1]

I’m really excited about presenting Effective Django at OSCON, and appreciate the feedback and suggestions from everyone.

[1]I contemplated just using a text file in the repo as the solution, but realized that this has its own issues: if it’s under version control as well, then what’s the “right” version to look at when you’re in a branch? That branch’s? Master’s? It’s not clear to me.
author:Nathan Yergler
category:Effective Django
tags:talks, effective django, python, tut,
comments:

Tut: Easier Tutorial Documentation with Sphinx

The PyCon sessions wound down today, so I’m finally coming up for air. This year I presented Effective Django, which evolved out of last year’s PyCon presentation and my PyOhio 2012 talk. As I prepared my PyCon talk last year, I started building Hieroglyph, which makes it easy to build HTML-based slides using Sphinx. This year I was preparing a different kind of presentation: a tutorial. As I started putting it together, I realized that tutorial-style documents differ from the previous presentations and documentation I’ve written.

In the past I used Sphinx code blocks in my documents, and then used the doctest builder to verify that they were written correctly. With a tutorial, however, I was putting together a demo application, and wanted to include code directly from that. Sphinx has the literalinclude directive, but that wasn’t quite enough: I was using git to manage the sample source repository, so what I really wanted to do was include code from that repository at a particular point in time.

I didn’t want to just copy and paste the code from the Python source into the ReStructured Text files: I did that briefly, and found it difficult to keep in sync as my thinking about the code evolved. I’d copy and paste, write some more text, realize I needed to make a change to the sample code, and then need to go back and change it in two places.

To solve this problem, I wrote Tut. Tut is a Sphinx extension that provides a simple directive, checkpoint. The checkpoint directive switches a git repository to a particular point in time: a branch, tag, or SHA; basically anything you can git checkout.

I used tags for Effective Django, so Tut also includes a script to help manage those. The script installs a post-rewrite hook in your git repository, so that if you need to reorder your commits your tags will be moved to the new SHAs. This can be useful if you find a bug and want to change it (“back in time”), or decide to reorder parts of your tutorial source.

I think Tut is a pretty handy little extension: it allowed me to use Sphinx’s built in inclusion directives in my documents, and eliminated work that took my focus off of actually creating content. You can find it on PyPI, and the source is available on github.

author:Nathan Yergler
category:tut
tags:sphinx, rst, hieroglyph, tut
comments: