Emacs & Jedi

Brandon Rhodes delivered the keynote at PyOhio yesterday[1]. He talked about sine qua nons: features that without which, a language is nothing to him. One of the things he mentioned was Jedi, a framework for extending editors with Python autocompletion, documentation lookup, and source navigation. I say “editors”, vaguely, because Jedi consists of a Python server that the editor communicates with via an editor specific plugin. I’d seen Jedi before, but hadn’t managed to get it working with Emacs. After hearing Brandon speak of it so glowingly, I decided to give it another try. The actual installation was easy: using the master branch of el-get, the recipe installed the Jedi Emacs plugin and its dependencies seamlessly. And it seemed to just work for the standard library.

And once I’d enabled it for python-mode, I was indeed able to autocomplete things from the standard library, and jump to the source of members implemented in Python. But I found that I wasn’t able to navigate to the third party dependencies in my project, and eventually figured out there were three cases I needed to address.

  • Many things I work on use virtualenv. Jedi supports virtual environments if the VIRTUAL_ENV environment variable is set, but I tend to keep Emacs running and switch between several different projects, each with their own environment.
  • Some of my projects also use buildout. When I’m using buildout for a project, the dependencies end up in an eggs sub-directory, which Jedi (as far as I know) doesn’t actually know about.
  • Finally, the setup we use at Eventbrite requires some special handling, as well: we store our source checkouts on an encrypted disk image, which is then mounted into a Vagrant virtual machine, where the actual virtual environment lives. Since the virtual environment isn’t on the same “machine” that we’re editing on, I need to tell Jedi explicitly what directories to find source in.

The Jedi documentation includes an advanced example of customizing the server arguments on per buffer. It assumes static arguments, but it seemed like a solution was possible. I spent a couple hours this afternoon working on my Emacs Lisp skills to make Jedi work in all three of my cases.


kenobi.el (gist) is the result, and it does a few things:

  1. Fires hooks for each mode after the file or directory local variables have been set up. I found a StackOverflow post that confirmed what I had observed: any file or directory local variables weren’t set when the normal python-mode hook was fired. The mode-specific hooks are chained off of hack-local-variables-hook, which is fired after the local variables have been resolved.
  2. Walks up the directory hierarchy from the buffer file, looking for ./bin/activate at each level. If it finds one, it assumes this is the virtual env, and adds it to the list of virtual envs Jedi will look at.
  3. Walks up the directory hierarchy looking for an ./eggs/*.egg sub-directory at each level. If it finds one, it adds each of those .egg subdirectories to the sys.path Jedi will look at. This allows Jedi to work when you’re editing files in buildout-based projects.
  4. Looks to see if the aditional_paths variable has been set as a list of other paths to add.

The first three bits are sort of implementation details: you can usually just ignore them, and Jedi will just work. The final, though, needs a little explanation.

As I mentioned above, Eventbrite stores the source code on a disk image, which is mounted into a virtual machine where the actual virtualenv lives. That means I need to add specific paths to sys.path when I open a source file in that disk image. To get that to work, I create a .dir-locals.el in the root of the source tree, something like:

 (nil . ((additional_paths . (

I’m sure that my Emacs Lisp could be improved upon, but it felt pretty good to figure out enough to integration Jedi into the way I use Emacs. I haven’t worked with Jedi extensively, but so far it seems to work pretty well. The autocomplete features seem to be minimally invasive, and the show docstring and jump to definition both work great.

[1]The fact that the video is up less than 36 hours after the keynote is a testament to how great Next Day Video is. They do amazing work at Python (and other) conferences and make it possible to enjoy the hallway track without worrying about missing a presentation.
author:Nathan Yergler
tags:python, virtualenv, buildout

Effective Django at OSCON Post-Mortem

I’ve been on the road a little more than a week now, back to back conferences. On Tuesday I presented my Effective Django tutorial at OSCON. I’ve recently updated it to cover integrating static assets with your project, and re-organized some of the view information. The biggest challenge with presenting a tutorial like that is figuring out how fast (or slow) to go. I’ve practiced the tutorial and use Django daily, so of course I’m able to type and diagnose what’s going on more quickly. At the break on Tuesday a few people asked me to slow down, but as I walked around the room, two others told me (quite kindly) that they wouldn’t mind if I sped up. This was a little worse than at PyCon, I think, primarily because I didn’t have friends and colleagues assisting me. At PyCon they were able to wander the room and help backfill support for those who were moving a little slower.

I think the next time I deliver Effective Django — or any tutorial — I’ll probably try to mitigate with a few different ways:

  • Make sure I have assistants. I actually had one planned for OSCON, but he unfortunately fell ill. Next time I’ll try to have more than one scheduled (I had four at PyCon, three is probably fine).
  • Provide more guidance in the synopsis. If I’m going to assume no prior Django knowledge, I should probably say something like “We’ll start from zero and build a Django application together,” rather than just relying on the slightly ambiguous Novice tag in the program.
  • Provide some clear stopping points. I noticed that some attendees stayed with it for the first half, or first three quarters, but disengaged before the end. I tried to structure the tutorial so that you can stop at any time and still have learned something, but I could be more explicit about that. “And now we know how to write views that use the ORM,” rather than “We’ll see in a moment how to wire that into the …”
  • Make sure my handouts are ready to go. At OSCON I didn’t realize they were sitting in cardboard boxes off to the side until half way through. I noticed that in the second half, a couple of the people who had been struggling previously were keeping up slightly better once they had another source to refer to.

The people I spoke to afterward uniformly said they learned something (whether it was as much as they’d hoped is another question, I suppose), so this feels like an overall success.

author:Nathan Yergler
tags:oscon, pyohio, effective django, python3

Coffee Cup line study

Untitled (coffee cup line study), copyright 2013 Nathan Yergler

4” x 5” linocut print

I wanted to practice using lines to describe, rather than outline, shapes and surfaces, so I took a picture of a coffee cup on a sunny day and decided to try and make a print from it. I worked small and (relatively) simple to avoid investing too much time in what is (effectively) a practice piece.

It was pretty instructive to carve this in an afternon, and then print it on Wednesday. Because it was small project, I was able to remember what I expected when I was carving, and compare that to what came out. There were a few things that came out as expected, and a few that didn’t. That was sort of the point.

author:Nathan Yergler
tags:linocut, study, practice

New Work: “Candlestick Point”

Candlestick Point”, copyright 2013 Nathan Yergler

8” x 10” linocut print, printed on Magnani Pescia

A dear friend sent me a photograph she took from Candlestick Point shortly after I made a print of a canal in Amsterdam. She noted that I’d been working with images of water, and thought of me when she looked out over the bay. I’ve been thinking about the image for a while, and started working on the actual print a couple months ago. I was finally ready to print on Wednesday.

Candlestick Point” plate, midway through carving

In this case I made a drawing from the photograph, transferred it the lino plate, and then starting thinking about how to actually carve things away. I was about 90% of the way done when I scanned the plate, as an experiement. I think it’s pretty interesting to look at, too. The rocks, in particular, are something new for me. It’s taken me a while to ease into using parallel lines like that to indicate shading and shape. It’s a technique I’ve seen others use very effectively, and I’m excited to develop it for my own work.

author:Nathan Yergler

“Effective Django” at OSCON

I’m going to be presenting my introductory Django tutorial, Effective Django at OSCON later this month. If you’re going to be at OSCON and haven’t selected your tutorials yet, or just think a trip to Portland, Oregon sounds nifty, there’s still time to sign up. You can find the details on the OSCON tutorial page.

In preparation for that I’ve been continuing to work on the content. I presented the tutorial to some of the Eventbrite engineering team a couple weeks ago, and thier feedback was very useful. In response, I’ve made a few changes. Specifically, I split up the Views chapter with a brief interlude on static assets and template inheritance. It’s something that I didn’t cover the first time around, but based on the questions, I think some guidance is useful.

The revisions for OSCON also include updating the sample code repository that goes with the tutorial. I developed a tool, Tut, to help manage these stacked branches, and while making changes to early parts of the tutorial code, I realized it still requires significant work to really be a good workflow tool. One of the most important requirements for Tut is the ability to manage an ordered series of “checkpoints” and move between them.

When I started working on the sample code this time around, I was on a new laptop, so I had to start from a fresh clone. This was revealing and frustrating. I discovered Tut assumed all the branches were already local, which they obviously aren’t with a fresh clone. Worse, the git magic I was trying to use to get the branch list in the “right” order was pretty fragile, and broke when I tried to lean on it at all.

This screenshot shows a common editing case. My intention is to manage each step, or checkpoint, in the tutorial as a branch. Each step builds on the previous one, so if I make a change to something early in the tutorial, I just need to merge the branches “forward” until I get to master. In this example I’ve checked out the contact_form_test branch and added a new commit. In order for Tut to help me merge that forward, I need to be able to generate the list of steps.

The correct order here (last step first) is master, custom_form_rendering, contact_form_test, edit_addresses, address_model, confirm_contact_email, contact_detail_view. But you can’t get that out with either date or topo ordering. You really need to walk back from master, looking for branches (head refs), and at each step look for any head refs reachable (as children) that you haven’t already seen. I haven’t figured out how to do that yet with git plumbing commands, so for the time being I’m just using a text file to record the correct order. [1]

I’m really excited about presenting Effective Django at OSCON, and appreciate the feedback and suggestions from everyone.

[1]I contemplated just using a text file in the repo as the solution, but realized that this has its own issues: if it’s under version control as well, then what’s the “right” version to look at when you’re in a branch? That branch’s? Master’s? It’s not clear to me.
author:Nathan Yergler
category:Effective Django
tags:talks, effective django, python, tut,

New Work: “Welcome Home”

Welcome Home”, copyright 2013 Nathan Yergler

5” x 7” two plate linocut print, printed on Rives BFK

Richard and I just moved from South Beach back to the area I lived in when I first moved to San Francisco, and we first started dating. We’ve been in the new place for a couple weeks now, and it already feels like we’re more in the city than we were before: closer to friends, closer to things to do in the evenings, and more in a neighborhood. I was in the studio on a Wednesday evening, just days before we moved, and I wanted to commemorate our new home. This is what came out. It’s not a literal drawing of the front gate and building, but it sort of takes the elements and rearranges them a bit. It’s one of the first things I’ve done where I just drew directly on the plate before carving, instead of doing a drawing first. Of course, I forgot to reverse it, so I still wound up tracing what I’d drawn, cleaning off the plates, and then re-drawing it. Sigh.

author:Nathan Yergler
tags:linocut, multiplate

Hieroglyph 0.5.5

As I mentioned last month, there were a few improvements to Hieroglyph sitting in the master branch, awaiting a release. Now that PyCon is over, I’ve cut the Hieroglyph 0.5.5 release. This is primarily a bug fix release, and it feels good to fix a bunch of issues that I discovered as I used Hieroglyph to develop and present Effective Django.

In addition to fixing half a dozen bugs, I also reviewed and revised almost all of the documentation for this release. Asheesh used Hieroglyph for his talk on web scraping at PyCon last week, and it was really interesting to get feedback from him about what worked and what was confusing. As a result, there’s a new Getting Started guide. There’s still work to be done, in particular in the Advanced Usage document; hopefully that will get rewritten and expanded soon.

Asheesh’s feedback demonstrated to me how little I know about why people use (or don’t) use Hieroglyph, and what’s difficult or confusing to them that I take for granted. If you’ve used Hieroglyph, thought about it and rejected it, or tried it and been frustrated, I’d like to hear about your experience. You can email me at nathan@yergler.net, or ping me on Identi.ca or Twitter: @nyergler

author:Nathan Yergler
tags:hieroglyph, sphinx, rst

Tut: Easier Tutorial Documentation with Sphinx

The PyCon sessions wound down today, so I’m finally coming up for air. This year I presented Effective Django, which evolved out of last year’s PyCon presentation and my PyOhio 2012 talk. As I prepared my PyCon talk last year, I started building Hieroglyph, which makes it easy to build HTML-based slides using Sphinx. This year I was preparing a different kind of presentation: a tutorial. As I started putting it together, I realized that tutorial-style documents differ from the previous presentations and documentation I’ve written.

In the past I used Sphinx code blocks in my documents, and then used the doctest builder to verify that they were written correctly. With a tutorial, however, I was putting together a demo application, and wanted to include code directly from that. Sphinx has the literalinclude directive, but that wasn’t quite enough: I was using git to manage the sample source repository, so what I really wanted to do was include code from that repository at a particular point in time.

I didn’t want to just copy and paste the code from the Python source into the ReStructured Text files: I did that briefly, and found it difficult to keep in sync as my thinking about the code evolved. I’d copy and paste, write some more text, realize I needed to make a change to the sample code, and then need to go back and change it in two places.

To solve this problem, I wrote Tut. Tut is a Sphinx extension that provides a simple directive, checkpoint. The checkpoint directive switches a git repository to a particular point in time: a branch, tag, or SHA; basically anything you can git checkout.

I used tags for Effective Django, so Tut also includes a script to help manage those. The script installs a post-rewrite hook in your git repository, so that if you need to reorder your commits your tags will be moved to the new SHAs. This can be useful if you find a bug and want to change it (“back in time”), or decide to reorder parts of your tutorial source.

I think Tut is a pretty handy little extension: it allowed me to use Sphinx’s built in inclusion directives in my documents, and eliminated work that took my focus off of actually creating content. You can find it on PyPI, and the source is available on github.

author:Nathan Yergler
tags:sphinx, rst, hieroglyph, tut

flymake with Sphinx

Working on my PyCon tutorial (next week!), I’ve been spending a lot of time in Emacs editing reStructured Text documents. I use Sphinx and Hieroglyph to generate the HTML, the slides, and the PDF from a single source, which make it easy for me to keep everything in sync. flymake is an Emacs mode that’s designed to do syntax or spell checking as you work. The name reveals its roots: in its simplest form, it just runs make for your project.

I was tired of flipping over to the shell to re-build the Sphinx project, so I decided to enable flymake for .rst files and see what happened. The flymake-allowed-file-name-masks variable has a list of regular expressions to flymake commands, so I added the following element to the list:

("\\.rst\\'" flymake-simple-make-init)

That was enough to get flymake to invoke the Makefile, and then I just needed to add the target it looks for: check-syntax. I added the following target to my Sphinx project Makefile:

        $(SPHINXBUILD) -n -N -q -b html $(ALLSPHINXOPTS) $(BUILDDIR)/
        $(SPHINXBUILD) -n -N -q -b slides $(ALLSPHINXOPTS) $(BUILDDIR)/slides

In my case I’m building both HTML and Slides from the Sphinx project, and I wanted both to be updated when I changed a file in Emacs. That did it.

Now all I wanted was automagic execution of make, but to my pleasant surprise, Sphinx’s warning and error output is compatible with flymake by default. Suddenly Emacs highlighted missing targets and directives with missing arguments in red. With flymake-cursor enabled, moving my cursor over one of those red lines showed me the Sphinx error below the mode-line.

There you have it: Sphinx just works with Emacs and flymake, you just need to turn it on.

author:Nathan Yergler
category:python, sphinx, emacs

Hieroglyph Improvements

If you’re using Hieroglyph for generating slides with Sphinx, you may want to use the version in git rather than the release. A few things have landed there recently:

  • tjadevries contributed fixes for incremental slides in Chrome.

    It seems that a recent change in Chrome caused the incremental slide Javascript, which originated in the Google HTML5 Slides project, to stop working. master has a fix for that.

  • Proper pruning when autoslides are disabled.

    If you have autoslides turned off in a document, Hieroglyph will now properly prune the document when generating slides to only show explicit slide directives. This was broken in 0.5.

  • Fixed header sizing for slide directives.

    The slide directive allows you to specify a level attribute. This is supposed to be used for determining how to render the slide title (for example, a level of 2 should render the title as an <h2>). This was broken in 0.5, and is fixed now.

I’m using Hieroglyph for my PyCon tutorial, “Effective Django[1], but the way I’m using it is pretty different than it has been previously. The slides and HTML output differ more, so I’m not relying on automatic slide generation the way I was initially. I expect I’ll make a new release around PyCon (mid-March) when I’ve done an entire talk in this manner.

[1]I’ve started practicing my tutorial with the engineering team at Eventbrite in one hour chunks, once a week. I’ll probably post something about that experience once I have an opinion about how it works.
author:Nathan Yergler
tags:sphinx, hieroglyph, rst