Technical documentation with Tut

Long running branches are a personal pet peeve of mine: I’d much rather put the effort into managing feature flags, etc than I would into merging a branch that’s diverged significantly. So it’s with a mixture of embarrassment and relief that today I merged a branch that’s been hanging out for 3 years.

Four years ago I published Effective Django. As I was writing it, I knew I wanted to version control the text. As I worked with the code samples, however, it felt like I was having to work against my version control, rather than having it work for me: the code changes as the reader works through the content, and that’s not necessarily the same sequence of changes I’d make as I wrote. In effect, there were two timelines at work, and they didn’t match up. To make matters worse, it felt like I was repeating myself a lot: copying and pasting between source files when I wanted to start a new “chapter”, etc. Tut was my solution to this problem.

When I started updating Effective Django three years ago, I was aware there was still room for improvement: my Restructured Text source files were littered with specific line numbers from the source code, as well as prepend, pyobject, and end-before flags for my literal includes, all for the purpose of limiting the display to the changes. These felt brittle and inelegant: if I made a change in an early step of the tutorial (which tut made easy), all the line numbers could conceivably change; missing one was the sort of error that’s difficult to check without incredibly careful proofreading. It seemed like I should be able to get that information from the source control, so I set out to figure out how.

I think of the Effective Django tutorial, written using the first iteration of tut, as a form of “executable documentation”. It contains code samples, and the fact that those exist independently on disk means you can conceivably go to any point in time and run tests to make sure things are working. My first exposure to executable documentation was doctests; indeed, when I started thinking about tut, the place I started was reading the code for Sphinx’s doctest support.

As I thought about how to make my code inclusion more robust, I thought I wanted a “literate tut”: that is, a single document that contained both the text and source, from which I would weave the files if I wanted to execute them. I probably tried half a dozen approaches to make this work. Some “wove” the source files at Sphinx build time, another tried to marry Jupyter kernels with a Sphinx builder. In the end none of them worked perfectly, and — perhaps more importantly — none of them felt easy to author with. I knew I wasn’t going to use a tool for very long if it made life harder, no matter how elegant its approach.

Defeated, I decided to go back to where I started: using tut to manager source changes, but dropping the “tricks” I’d used to make inclusion work right. In other words, I stopped trying to be clever and just started writing.

In the years since I started the “literate tut” branch, I had made some improvements that seemed worth keeping around. I’d abstracted the git operations slightly and removed a lot of the global state. I’d also moved from trying to “infer” the order of steps based on history to maintaining an explicit list in a special configuration branch. These changes meant that as I started writing — and making progress — it was easier to try out a couple of ideas.

When Tut encounters a “checkpoint” it switches to a different git branch. This means that if the source tree is dirty, the build won’t succeed. After running into that a handful of times, I realized that I could use the same approach I used for fetching configuration (git show) to fetch a file from a specific branch, which potentially eliminates the need for playing chutes and ladders with head.

This got me thinking that I could grab two different versions of a file for side by side display, if I so desired. Combined with Python’s difflib module, however, it suddenly seemed like I had the basis for generating the sort of “documentation style” diffs I’d so tediously hand crafted before.

The result of this work is Tut 0.5, released today just over four years (!!) since the last release. This release includes a lot of changes, and is what I’m using as I work on an updated Effective Django, but I’m primarily releasing it to get the branch closed and focus myself on what else might be needed.

This release includes a new tut:literalinclude directive which operates exactly like the built-in one, with the exception that the content is read from git, not the filesystem. I’ve also added a tut:diff directive, which is able to format diffs for Python source files that do a reasonable(-ish) job of only showing the context that’s needed. The tut:diff directive can also show a link to the full file at that stage, so the entire contents are readily available.

I don’t think it’ll be another four years until the next release. As I work on the Effective Django update, I’m pretty sure I’ll need to improve diff generation and make it behave reasonably for non-Python files.

I thought I wanted “literate tut”: a single document that contained the diffs and text all in one. What I wound up with is something better: the ability to use the right tools to edit the text and code, and the ability to leverage the underlying framework to include exactly what I want as I write. You can find the new release of tut on PyPI; let me know what you think.

Mapping file paths for pdbtrack

I saw python-mode 6.0.8 was release two weeks ago, and that reminded me to look and see if my change was included. Turns out it was included in 6.0.6, released in April:

– files inside a virtual machine made visible for pdbtrack

That’s my contribution.

python-mode provides support for developing with Python in Emacs. One of its most powerful (and often unknown) features is pdbtrack. pdbtrack recognizes when you’re interacting with a Python Debugger (PDB) session within Emacs, and opens the relevant files for you as you step through the program. When you’re digging into something very deep (lots of calls on the stack) or very broad (using lots of supporting libraries), this is invaluable for providing perspective on “where” your program is.

At work we develop using a Vagrant-managed virtual machine. The source checkout is stored “locally” and exported to the virtual machine using NFS. That’s great for development — edit locally, run from within the virtual machine — but broke pdbtrack. When Python reported it was at a certain line of a certain file, it was referring to a file inside the virtual machine. When pdbtrack tried to find that path, it didn’t exist because Emacs was running outside the virtual machine. When I want to use pdbtrack I’m usually pretty confused and in need of perspective, so it’s a pretty important tool for me. So I decided to fix this.

Starting with python-mode 6.0.6, you can customize the py-pdbtrack-filename-mapping variable. This is an alist which maps paths the Python interpreter sees to paths Emacs can see. If pdbtrack can’t find the buffer to open directly, it checks this mapping to see if the file exists in a different location. For example, my configuration has the following customization:

'(py-pdbtrack-filename-mapping (quote
   (("/home/vagrant/eventbritecore/" . "/Volumes/eb_home/work/eventbritecore/"))

Using Vagrant means we are able to develop on a configuration far closer to production than we would [easily] be able to otherwise. And starting with python-mode 6.0.6, I don’t have to give up pdbtrack to do that.

date:2012-06-07 07:04:03
tags:emacs, pdbtrack, python-mode

Updating el-get and getelget.el

Last week one of my Emacs using colleagues asked me how I managed my Emacs packages and configuration. Naturally I pointed him to el-get and my getelget.el bootstrap script. I’ve been happily managing my Emacs installation over the past five months using el-get and a private git repository for my configuration. However when I tried to square my .emacs.d/init.el with the current el-get documentation, I got a little confused; el-get is now better at bootstrapping itself from within your Emacs configuration. When my colleague read this and asked why he might want getelget.el, my response was… well, lackluster; this is an attempt to document that a littler better.)

Last night I decided to do a little clean-up on my Emacs configuration, and see if I could get rid of getelget.el. The documentation for el-get is great, so I started there. What I quickly realized is that the included el-get bootstrap mechanism is great if you want to ensure el-get is installed and then use el-get-install, el-get-remove, etc to manage your packages. But if you define your package list in you config file, it’s not quite enough. Specifically, when you first bootstrap your configuration, you want to defer calling (elget 'sync) until you’ve bootstrapped el-get. And on future runs, you want to go ahead and install any new packages that have been added to your list.

Luckily el-get has added support for hooks, which makes life a little easier. The new getelget.el (available here) looks something like this:

;; getelget – el-get boostrap script;;;; Checks to see if el-get has been checked out, and bootstraps it if;; it has not. After bootstrapping, calls el-get to load specified;; packages.;;;; el-get-packages should be defined before including this file. Any;; definitions from el-get-sources will be appended to el-get-packages.;;;; Written in 2011 by Nathan R. Yergler ;;;; To the extent possible under law, the person who associated CC0 with;; getelget has waived all copyright and related or neighboring rights;; to getelget.;;;; You should have received a copy of the CC0 legalcode along with this;; work.  If not, see .

    ;; add a hook listener for post-install el-get(defun post-install-hook (pkg)
  ;; after installing el-get, load the local package list
  (if (string-equal pkg “el-get”)
      (el-get 'sync
              (append el-get-packages
                      (mapcar 'el-get-source-name el-get-sources)))))(add-hook 'el-get-post-install-hooks 'post-install-hook)

    ;; add the el-get directory to the load path(add-to-list 'load-path
             (concat (file-name-as-directory user-emacs-directory)
                     (file-name-as-directory “el-get”)

    ;; try to require el-get(if (eq (require 'el-get nil t) nil)

    ;; urp, need to bootstrap
     (lambda (s)

    ;; successfully required el-get, load the packages!
    (post-install-hook “el-get”)

el-get also recommends splitting your package definitions from your local source recipes (which can themselves extend an included recipe). So getelget.el now expects you’ve defined two lists: el-get-packages, a list of packages to install from recipes, and el-get-sources, your local source list.

For example, I define a local recipe for magit that binds a key to magit-status and enables spell checking and fill mode for commit message editing:

(setq el-get-sources
      '((:name magit
               :after (lambda ()
                        (global-set-key “\C-x\r\r” 'magit-status)

                        ;; Enable spell checking, fill for log editing
                        (add-hook 'magit-log-edit-mode-hook
                                    (auto-fill-mode 1)
                                    (flyspell-mode 1)))))

And my el-get-packages list is just a list of packages I’m installing from the included el-get recipes.

(setq el-get-packages

Everything listed in both lists will be installed.


date:2011-09-26 09:59:24
category:development, tools
tags:emacs el-get getelget

Managing my Emacs packages with el-get

Update (20 April 2011): I’ve now tried this on my old MacBook running OS X 10.5. The bootstrap script initially threw an error, which I tracked down to an outdated version of git. Once I upgraded git and installed bzr (used by the python-mode recipe), I started Emacs and was rewarded with a fully functioning installation, complete with the extensions I want.

I’m on vacation for two weeks between jobs, so of course this means it’s time to sharpen the tools (because writing programs to help you write programs is almost always more fun than actually writing programs). I’ve been an Emacs user for many years, and of course I’ve customized my installation with additional modes and extensions. Previously I would check out code that I needed into a vendor directory, and then load it manually in init.el. And this worked fine, but that doesn’t mean I [STRIKEOUT:can’t] won’t spend a chunk of my day making it better.

A friend mentioned el-get to me, and I decided to give it a try. I like the combination of recipes for installing common things, and the fact that your list of packages is very explicit in init.el (so if I need to dig into one of them, I know exactly where to begin). Additionally, since I’ll have a new computer issued for the new job, I also wanted to get things into shape so that I could easily replicate my preferred editing environment. I wound up creating a small bootstrap file to help things along, getelget.el.

getelget.el checks to see if el-get has been previously bootstrapped, and if not, performs the lazy installation procedure. After it makes sure el-get is available, it loads and executes el-get. So if you need to get a new machine up and going with Emacs and any extensions, you can drop in your init.el and getelget.el, and Emacs will take care of the rest.

To use getelget, define your el-get-sources like you normally would in init.el:

(setq el-get-sources
         ;; etc...
       )  )

Then load getelget (the following assumes you have getelget.el in your user emacs directory along with init.el):

;; getelget -- bootstrap el-get if necessary and load the specified packages
   (concat (file-name-as-directory user-emacs-directory) "getelget.el"))

getelget will handle bootstrapping, loading, and executing el-get.

getelget is pretty trivial; you can download it here, and I’ve waived any rights I may hold on the code using the CC0 Public Domain Dedication.

date:2011-04-19 22:17:05
category:development, tools
tags:el-get, emacs