Technical documentation with Tut

Long running branches are a personal pet peeve of mine: I’d much rather put the effort into managing feature flags, etc than I would into merging a branch that’s diverged significantly. So it’s with a mixture of embarrassment and relief that today I merged a branch that’s been hanging out for 3 years.

Four years ago I published Effective Django. As I was writing it, I knew I wanted to version control the text. As I worked with the code samples, however, it felt like I was having to work against my version control, rather than having it work for me: the code changes as the reader works through the content, and that’s not necessarily the same sequence of changes I’d make as I wrote. In effect, there were two timelines at work, and they didn’t match up. To make matters worse, it felt like I was repeating myself a lot: copying and pasting between source files when I wanted to start a new “chapter”, etc. Tut was my solution to this problem.

When I started updating Effective Django three years ago, I was aware there was still room for improvement: my Restructured Text source files were littered with specific line numbers from the source code, as well as prepend, pyobject, and end-before flags for my literal includes, all for the purpose of limiting the display to the changes. These felt brittle and inelegant: if I made a change in an early step of the tutorial (which tut made easy), all the line numbers could conceivably change; missing one was the sort of error that’s difficult to check without incredibly careful proofreading. It seemed like I should be able to get that information from the source control, so I set out to figure out how.

I think of the Effective Django tutorial, written using the first iteration of tut, as a form of “executable documentation”. It contains code samples, and the fact that those exist independently on disk means you can conceivably go to any point in time and run tests to make sure things are working. My first exposure to executable documentation was doctests; indeed, when I started thinking about tut, the place I started was reading the code for Sphinx’s doctest support.

As I thought about how to make my code inclusion more robust, I thought I wanted a “literate tut”: that is, a single document that contained both the text and source, from which I would weave the files if I wanted to execute them. I probably tried half a dozen approaches to make this work. Some “wove” the source files at Sphinx build time, another tried to marry Jupyter kernels with a Sphinx builder. In the end none of them worked perfectly, and — perhaps more importantly — none of them felt easy to author with. I knew I wasn’t going to use a tool for very long if it made life harder, no matter how elegant its approach.

Defeated, I decided to go back to where I started: using tut to manager source changes, but dropping the “tricks” I’d used to make inclusion work right. In other words, I stopped trying to be clever and just started writing.

In the years since I started the “literate tut” branch, I had made some improvements that seemed worth keeping around. I’d abstracted the git operations slightly and removed a lot of the global state. I’d also moved from trying to “infer” the order of steps based on history to maintaining an explicit list in a special configuration branch. These changes meant that as I started writing — and making progress — it was easier to try out a couple of ideas.

When Tut encounters a “checkpoint” it switches to a different git branch. This means that if the source tree is dirty, the build won’t succeed. After running into that a handful of times, I realized that I could use the same approach I used for fetching configuration (git show) to fetch a file from a specific branch, which potentially eliminates the need for playing chutes and ladders with head.

This got me thinking that I could grab two different versions of a file for side by side display, if I so desired. Combined with Python’s difflib module, however, it suddenly seemed like I had the basis for generating the sort of “documentation style” diffs I’d so tediously hand crafted before.

The result of this work is Tut 0.5, released today just over four years (!!) since the last release. This release includes a lot of changes, and is what I’m using as I work on an updated Effective Django, but I’m primarily releasing it to get the branch closed and focus myself on what else might be needed.

This release includes a new tut:literalinclude directive which operates exactly like the built-in one, with the exception that the content is read from git, not the filesystem. I’ve also added a tut:diff directive, which is able to format diffs for Python source files that do a reasonable(-ish) job of only showing the context that’s needed. The tut:diff directive can also show a link to the full file at that stage, so the entire contents are readily available.

I don’t think it’ll be another four years until the next release. As I work on the Effective Django update, I’m pretty sure I’ll need to improve diff generation and make it behave reasonably for non-Python files.

I thought I wanted “literate tut”: a single document that contained the diffs and text all in one. What I wound up with is something better: the ability to use the right tools to edit the text and code, and the ability to leverage the underlying framework to include exactly what I want as I write. You can find the new release of tut on PyPI; let me know what you think.

Hieroglyph 0.6

I just uploaded Hieroglyph 0.6 to PyPI. This release contains a handful of new features, as well as fixes for a few bugs that people encountered. Some highlights:

  • Doug Hellmann contributed support for displaying presenter notes in the console using the note directive.
  • tjadevries contributed a fix for the stylesheet used when printing slides, which should prevent modern browsers from inserting a page break in the middle of a slide.
  • Slide numbering has been reimplemented, and received additional testing.
  • A hieroglyph-quickstart script has been added to make it easier to generate an empty project with hieroglyph enabled.

See the NEWS for the full details.

I’ve also started writing some automated tests for Hieroglyph. These are a little too involved to properly be called “unit tests”, but they’re being run using Travis CI now, which should help avoid regressions as I fix bugs in edge cases.

I spent a few days at OSCON about a week ago, and once again had the pleasure of attending Damian Conway’s “Presentation Aikido”. There are several things he talked about that I could be doing better with my talks. This release of Hieroglyph addresses one of them (quick fade or cut to the next slide, as opposed to the default slide left behavior). I’m working on what other changes I can make to Hieroglyph so that it’s dead simple to just write your slides, and maximize what your attendees take away.

author:Nathan Yergler
tags:rst, hieroglyph, sphinx

Hieroglyph 0.5.5

As I mentioned last month, there were a few improvements to Hieroglyph sitting in the master branch, awaiting a release. Now that PyCon is over, I’ve cut the Hieroglyph 0.5.5 release. This is primarily a bug fix release, and it feels good to fix a bunch of issues that I discovered as I used Hieroglyph to develop and present Effective Django.

In addition to fixing half a dozen bugs, I also reviewed and revised almost all of the documentation for this release. Asheesh used Hieroglyph for his talk on web scraping at PyCon last week, and it was really interesting to get feedback from him about what worked and what was confusing. As a result, there’s a new Getting Started guide. There’s still work to be done, in particular in the Advanced Usage document; hopefully that will get rewritten and expanded soon.

Asheesh’s feedback demonstrated to me how little I know about why people use (or don’t) use Hieroglyph, and what’s difficult or confusing to them that I take for granted. If you’ve used Hieroglyph, thought about it and rejected it, or tried it and been frustrated, I’d like to hear about your experience. You can email me at, or ping me on or Twitter: @nyergler

author:Nathan Yergler
tags:hieroglyph, sphinx, rst

Tut: Easier Tutorial Documentation with Sphinx

The PyCon sessions wound down today, so I’m finally coming up for air. This year I presented Effective Django, which evolved out of last year’s PyCon presentation and my PyOhio 2012 talk. As I prepared my PyCon talk last year, I started building Hieroglyph, which makes it easy to build HTML-based slides using Sphinx. This year I was preparing a different kind of presentation: a tutorial. As I started putting it together, I realized that tutorial-style documents differ from the previous presentations and documentation I’ve written.

In the past I used Sphinx code blocks in my documents, and then used the doctest builder to verify that they were written correctly. With a tutorial, however, I was putting together a demo application, and wanted to include code directly from that. Sphinx has the literalinclude directive, but that wasn’t quite enough: I was using git to manage the sample source repository, so what I really wanted to do was include code from that repository at a particular point in time.

I didn’t want to just copy and paste the code from the Python source into the ReStructured Text files: I did that briefly, and found it difficult to keep in sync as my thinking about the code evolved. I’d copy and paste, write some more text, realize I needed to make a change to the sample code, and then need to go back and change it in two places.

To solve this problem, I wrote Tut. Tut is a Sphinx extension that provides a simple directive, checkpoint. The checkpoint directive switches a git repository to a particular point in time: a branch, tag, or SHA; basically anything you can git checkout.

I used tags for Effective Django, so Tut also includes a script to help manage those. The script installs a post-rewrite hook in your git repository, so that if you need to reorder your commits your tags will be moved to the new SHAs. This can be useful if you find a bug and want to change it (“back in time”), or decide to reorder parts of your tutorial source.

I think Tut is a pretty handy little extension: it allowed me to use Sphinx’s built in inclusion directives in my documents, and eliminated work that took my focus off of actually creating content. You can find it on PyPI, and the source is available on github.

author:Nathan Yergler
tags:sphinx, rst, hieroglyph, tut

Hieroglyph Improvements

If you’re using Hieroglyph for generating slides with Sphinx, you may want to use the version in git rather than the release. A few things have landed there recently:

  • tjadevries contributed fixes for incremental slides in Chrome.

    It seems that a recent change in Chrome caused the incremental slide Javascript, which originated in the Google HTML5 Slides project, to stop working. master has a fix for that.

  • Proper pruning when autoslides are disabled.

    If you have autoslides turned off in a document, Hieroglyph will now properly prune the document when generating slides to only show explicit slide directives. This was broken in 0.5.

  • Fixed header sizing for slide directives.

    The slide directive allows you to specify a level attribute. This is supposed to be used for determining how to render the slide title (for example, a level of 2 should render the title as an <h2>). This was broken in 0.5, and is fixed now.

I’m using Hieroglyph for my PyCon tutorial, “Effective Django[1], but the way I’m using it is pretty different than it has been previously. The slides and HTML output differ more, so I’m not relying on automatic slide generation the way I was initially. I expect I’ll make a new release around PyCon (mid-March) when I’ve done an entire talk in this manner.

[1]I’ve started practicing my tutorial with the engineering team at Eventbrite in one hour chunks, once a week. I’ll probably post something about that experience once I have an opinion about how it works.
author:Nathan Yergler
tags:sphinx, hieroglyph, rst

Hieroglyph 0.5

During the last week of 2012 I pushed out a new release of Hieroglyph. I realized later that I haven’t really been talking about it here, so I wanted to mention some of the new features and functionality I’ve been working on. When I published the 0.3 release, I said I was planning to work on hooking analytics into the slide viewing so that creators could get a better sense of how their slide decks were being used. I have done some work to that end, but would up getting side tracking.

The 0.4 release consisted primarily of internal cleanups in preparation for adding analytics support. There were some places where the Javascript or styles were duplicated, so I put some effort into cleaning that up. I also split up the Javascript to separate slide control from user interaction. This was done to support both analytics and the initial implementation of the Presenter’s Console: a separate window that displays the previous, current, and next slides, and allows the presenter to control the primary display window. With Hieroglyph 0.4 and later you can open by pressing c within a slide document.

Hieroglyph 0.4 also added support for slide numbering and for applying a theme to individual documents inside of a project using the slideconf direction. For example:

.. slideconf::
   :theme: single-level

In December I put together some training for other engineers at Eventbrite around Python byte and unicode strings, and how to safely work with them both in our codebase. While I was putting that together, I realized that there was a lot of text and code samples in the documentation that I wanted to go through interactively, and not include in slides. I wound up going without slides, and found that presenting from the Python interactive interpreter was very effective for this sort of hands on training. That said, it would have been nice to have had a few key slides: an introduction, a summary of key points, and maybe where to go next. That’s the sort of content I’d like to keep with the main document so it doesn’t drift too far out of sync, but Hieroglyph didn’t really support that approach: it assumed that you wanted to generate a slide per section, unless you explicitly marked things as notslides.

Hieroglyph 0.5 adds a couple of features to support these different work modes. First, it adds an autoslides configuration parameter, that allows you to disable automatic slide generation of a project or document level. If autoslides is True (the default), you’ll see the previous Hieroglyph behavior, one slide per section. If it’s set to False, you’ll need to write slides using the second new feature, the slide directive.

The slide directive describes a single slide, including the title, level (which may be used for styling), and content. An example from the Hieroglyph smoketest document:

.. slide:: The ``slide`` Directive
   :level: 2

   In addition to headings, you can use the ``..slide::`` directive to
   define a slide.

Because the conditional slides directive differed by only one letter, Hieroglyph 0.5 also renamed the conditional directives to ifslides and ifnotslides. The previous names will continue to work (at least for a couple releases), but I’m using the new names myself to make it easier to understand what’s going on.

Thanks to bug reports filed in the Hieroglyph github project, the 0.5 release also includes fixes for styling nested lists and compatibility with the latex-pdf builder.

At this point Hieroglyph seems to be working pretty well for me and others. There are a few things I’d like to add yet, but for the next couple of months my focus is going to be more on using it rather than developing it. I’m working on more training for new engineers at Eventbrite, and I’ll be giving a tutorial of Effective Django at PyCon this year. I think both of those are going to be good opportunities use Hieroglyph in different contexts and hopefully get some more data on what’s working or not. If you’re using it (or just trying it out) and have comments or feedback, filing a bug is something I find really helpful.

author:Nathan Yergler
tags:hieroglpyh, sphinx, rst, slides


I’ve been contemplating leaving WordPress for a while. Part of it was a desire for something simpler, part of it was that as I worked on Hieroglyph, I found myself really appreciating the Sphinx build pipeline. Older blog posts on my site have also been in varying states of brokenness for the past few years: I used to use Textile for writing posts, but at some point the WordPress plugin I was using stopped working, so I just let it serve them with the raw markup. It felt like it’d be easier to fix those up into something sane if I had flat files to work with, instead of a relational database.

There are quite a few static site generators these days, but only one that I’ve found that leverages Sphinx. Tinkerer leverages Sphinx, adding an extension for aggregating post pages into archives, categories, and tag pages. Its use of Sphinx means that you can also generate non-blog pages with it, and you have quite a bit of flexibility to extend it if you need to.

Moving from my WordPress backend, I wanted keep the URLs as stable as possible. Tinkerer 1.0 is great, but it did a few things differently than I’d like. Specifically, it:

  • uses the Sphinx HTML builder, which adds .html extensions to the output pages
  • assumes that the blog’s root is the root of the domain (on my site it’s in the /blog/ path)
  • places every post in your RSS feed
  • doesn’t allow you to configure the output path for the feed or archives

I’ve forked Tinkerer on Bitbucket and addressed those issues. Making Tinkerer work with the dirhtml builder was most of the work, as there were several places that assumed you were dealing with HTML files (instead of directories with an index.html inside them). My experience with Sphinx from Hieroglyph came in handy here: the Sphinx HTML builders all have a .get_target_uri() method, which does the right thing. I’m sure there are some things I’ve missed or broken in the process, but my hope is that I can get my changes into shape and merged upstream. I’ve also added several configuration directives for customizing how Tinkerer works. I still need to update the documentation in my fork, but you can see those in the source.

As an aside, if you’re looking at migrating from WordPress to another platform, the exitwp script is a pretty sane place to start. I used it, along with pandoc, to transform my WordPress export into more uniform Restructured Text.

author:Nathan Yergler
tags:sphinx, tinkerer, rst

Hieroglyph 0.3.2: Slide Table and Interlinking

Since PyCon I’ve continued to think about how I can make slides from ReStructured Text documents and vice versa. I tend to write a lot of notes and text while I’m putting together a talk, and I like the idea of being able to keep slides and text output in sync. I’ve just a batch of changes to Hieroglyph, my tool for doing that. There’s some clean-up there — better handling of output paths when using things like blockdiag, code clean-up, etc — but there are two things I’m really excited about. First, two pull requests (one for Python 3 support, another for some documentation bugs), and second some new features that I think make Hieroglyph much more powerful.

Thinking about keeping slides and text (HTML) output in sync, it occurred to me there were probably times you’d want to easily switch between slides that provide an overview, and the HTML document for more details and context. Much of the work for 0.3.2 focused on enabling this interlinking. When enabled, Hieroglyph will add links to your HTML and Slide output that links to the other format. For HTML this can be enabled in the sidebar, as well as at the section level. For slides, the link is added next to each slide’s header, and shows up when you hover over the header. Check it out on the Hieroglyph documentation — just hover over any header and click the § link for the corresponding slides.

When I was working on my PyCon talk, I had anywhere from 50 to 70 slides in the deck at any given time (NB: yes, this is too many for a talk of that length). Navigating between them was challenging at times. The second feature I’ve added to Hieroglyph is designed to address this. When viewing a Hieroglyph presentation, you can now press the Escape key to see the Slide Table.

An example of the slide table in use for the Hieroglyph documentation (full size).

Press Escape again to return to the slide you were on, or click a slide to jump directly to it. You can try this with the Hieroglyph documentation slides.

Finally, what should really be considered the third new feature: expanded documentation. You can find expanded documentation on configuring Hieroglyph, styling your slides, etc in the docs online.

There are several additional things I’m working on for Hieroglyph. As Ilya points out in “All Presentation Software Is Broken”, web analytics are your “free lunch” if you use HTML-based slides. I plan to bake support for that directly into Hieroglyph. As I’m using Hieroglyph, I’m also realizing that slides don’t always correspond directly to sections in a document — sometimes (but not always) they’re a paragraph, list, or something else. Some way to indicate this may be helpful. If you find Hieroglyph useful (or interesting), let me know what you’d like to see.

date:2012-06-05 13:22:29
tags:hieroglyph, rst, slides

hieroglyph: Easy, Beautiful Slides with Restructured Text

I was happy to have my talk proposal accepted for PyCon this year, and happy with the feedback I received on my talk (Django Forms Deep Dive). But as I was putting my talk together the distracting question was not, “what should I say”, but “what should I say it with”. As a mentor once pointed out, “it’s more fun to write programs to help you write programs than it is to write programs.” The corollary I found over the past couple weeks: “it’s more fun to write programs to help you write slides than it is to write slides.”

I was putting together notes using reStructured Text and kept thinking that it’d be nice to generate both slides and longer written documentation from the same source. I’ve used docutils’ S5 generator in the past, but was looking for something a little more polished looking. Something like the HTML5 Slides.

So I wrote a Hieroglyph, a Sphinx builder for generating HTML5 Slides. I presented hieroglyph at the Sunday morning lightning talks at PyCon: you can see the slides, the reStructured Text source, as well as the HTML documentation generated from the same source.

I’m really happy with the output — it looks great in the browser, projects well, and because I’m using the html5slides CSS, looks great on mobile devices, too. I’m even happier that I’m able to work on my content in plain text. You can find the source on github.

date:2012-03-13 22:31:16
category:projects, hieroglyph
tags:python, rst, slides, sphinx