Category: python

flymake with Sphinx

Working on my PyCon tutorial (next week!), I’ve been spending a lot of time in Emacs editing reStructured Text documents. I use Sphinx and Hieroglyph to generate the HTML, the slides, and the PDF from a single source, which make it easy for me to keep everything in sync. flymake is an Emacs mode that’s designed to do syntax or spell checking as you work. The name reveals its roots: in its simplest form, it just runs make for your project.

I was tired of flipping over to the shell to re-build the Sphinx project, so I decided to enable flymake for .rst files and see what happened. The flymake-allowed-file-name-masks variable has a list of regular expressions to flymake commands, so I added the following element to the list:

("\\.rst\\'" flymake-simple-make-init)

That was enough to get flymake to invoke the Makefile, and then I just needed to add the target it looks for: check-syntax. I added the following target to my Sphinx project Makefile:

check-syntax:
        $(SPHINXBUILD) -n -N -q -b html $(ALLSPHINXOPTS) $(BUILDDIR)/
        $(SPHINXBUILD) -n -N -q -b slides $(ALLSPHINXOPTS) $(BUILDDIR)/slides

In my case I’m building both HTML and Slides from the Sphinx project, and I wanted both to be updated when I changed a file in Emacs. That did it.

Now all I wanted was automagic execution of make, but to my pleasant surprise, Sphinx’s warning and error output is compatible with flymake by default. Suddenly Emacs highlighted missing targets and directives with missing arguments in red. With flymake-cursor enabled, moving my cursor over one of those red lines showed me the Sphinx error below the mode-line.

There you have it: Sphinx just works with Emacs and flymake, you just need to turn it on.

author:Nathan Yergler
category:python, sphinx, emacs
tags:none
comments:

Hate, hate, hate everywhere

Armin Ronacher is one of my favorite Python bloggers these days, and his post Python Packaging: Hate, hate, hate everywhere continues the trend of context and depth. You should read it. I immediately identified with the title: setuptools/easy_install isn’t (wasn’t?) perfect and it had maintenance problems, but when I get pull requests that reference propoganda labeling it as “craptastic”, I feel a little sad.

date:2012-06-23 16:04:54
wordpress_id:2170
layout:post
slug:hate-hate-hate-everywhere
comments:
category:python

super(self.__class__, self) # end of the line for subclassing

I’ve learned (and remembered) a lot in the past two months as I’ve gotten back to coding as my primary job. One thing that I guess I never quite internalized before is how super works. I have been bitten by code that looks something like the following a few times in the past month: 
class A(object):
    def init(self):
        super(self.__class__, self).init()

    class B(A):
    def init(self):
        super(B, self).init()
The surprise comes when I try to use my sub-class, B. Instantiating B() blows up the stack with: RuntimeError: maximum recursion depth exceeded while calling a Python object. What? According to the Python 2.7.2 standard library documentation, super “return[s] a proxy object that delegates method calls to a parent or sibling class of type.” So in the case of single inheritance, it delegates access to the super class, it does not return an instance of the super class. In the example above, this means that when you instantiate B, the follow happens:
  1. enter B.__init__()
  2. call super on B and call __init__ on the proxy object
  3. enter A.__init__()
  4. call super on self.__class__ and call __init__ on the proxy object
The problem is that when we get to step four, self still refers to our instance of B, so calling super points back to A again. In technical terms: Ka-bloom. TL;DR: super(self.__class__, self) may look like a neat trick, but it’s the end of the line for sub-classing. Further reading: Raymond Hettinger’s excellent blog post on super provides a great overview of super and shows off the improved Python 3 syntax, which removes the need to write the class name as part of the super statement. I was really pleased to find the Python standard library documentation links directly to it.
date: 2011-07-04 20:44:23
wordpress_id: 1990
layout: post
slug: super-self
comments:
category: development, python
tags: python, super

Making Fun of Objects

Oh, wait; not that sort of mocking. Fine. Following the talk on using the fixture package, Jeff Younker spoke on using the PyMock package for unit testing with mock objects. Younker spent the first few minutes of his talk talking about unit testing in general; what I initially thought was simply a meandaring exposition on the nature of testing was actually going somewhere: adding tests to existing code can be painful, because of how intertwined things can be. Creating mock objects can help you isolate pieces of the code from the remainder of the application.

PyMock makes this somewhat simpler by providing a way of “recording” what you want your mock objects to return, and the going into “play back” mode to run the test. It provides a way to mock methods, attributes and generators on Python objects.

Things I’m sure of following the talk:

  • The idea of mocking objects seems like a powerful way to isolate pieces of existing code for the purpose of testing.
  • Showing a slide full of code, with near-zero explanation, for only a few seconds before moving on, isn’t long enough for me to grok it; judging by the under-the-breath comments from those around me, I’m not alone in this.
  • Unit tests are hideously ugly when compared to the beauty of doctests.
date:2007-02-25 13:19:22
wordpress_id:499
layout:post
slug:making-fun-of-objects
comments:
category:pycon2007, python, testing

Testing with Fixtures

Following Jim’s talk on zc.ngi, I attended Kumar McMillan talk on using Fixtures to test your programs. According to McMillan, fixtures differ from mock objects in one important way: they use real objects with real data. I haven’t used fixtures or mock objects much, so that was a useful distinction.

McMillan is the developer of the fixture package, which provides a way of loading and referencing test data for your application. fixture also allows you to write tests against fixture objects using either a mixin or a decorator[1]_. Fixture supports loading data into SQLAlchemy and SQLObject; CSV and possible Django support are also under consideration.

Fixture is a relatively young project, but it does appear to be useful for testing data-driven applications.

[1]For testing with nose.
date:2007-02-25 12:48:58
wordpress_id:498
layout:post
slug:testing-with-fixtures
comments:
category:pycon2007, python, testing

Testing Network Applications with NGI

Following the keynote this morning, I attended Jim Fulton’s talk on testing network applications with the Network Gateway Interface. Jim advocates the use of doctests, in particular doctests which allow you to mix the documentation and narrative together (note to self: write a post about why this is compelling). When programming network applications, writing tests can be really, really hard: there can be sockets, time.sleep(), threads, etc.

In response to running into these problems with testing ZEO, Jim developed a library called the Network Gateway Interface (NGI). The goal of NGI is that application code doesn’t touch socket code. NGI splits the network application into two pieces: the application side and the implementation side. The application side doesn’t need to be thread aware (or even safe) and uses events to communicate with the implementation side. And the implementation side handles the actual socket or transport work.

Jim’s talk used the example of a simple application, z2monitoring, to demonstrate the NGI. The goal of z2monitoring was to monitor and get status from a running Zope 2 application at the time when status is needed most: when the application is bogged down, responding to requests. When testing with NGI, you create a special testing connection, such as a TextConnection (which just uses plain text). The connection conforms to the IConnection interface, which also provides a way to set the connection handler. The connection handler (IConnectionHandler, natch), is used to respond to events like input, connection close and exceptions.

In addition to the async, event-driven interface, zc.ngi provides a simple, blocking file-like API. This is convenient for developing simple, single-task applications.

Jim spent a short part of his talk discussing about the benefits of non-determinism, such as the existing ZEO implementation. One of these was the “opportunity for the unexpected.” Of course, Jim also noted (with an unclear amount of irony) that production is also a form of testing.

NGI 1.0 was released just before PyCon, and this will probably be the last release. [Un]fortunately, Jim recently found that Twisted provides facilities for this style of programming and testing. Given this, zc.ngi will probably only live on (if at all) as a simplified facade over Twisted’s facilities.

date:2007-02-25 12:48:06
wordpress_id:497
layout:post
slug:testing-network-applications-with-ngi
comments:
category:pycon2007, python, testing

Testing Tools Panel @ PyCon

Matt summarizes the PyCon’s Testing Tools Panel far better than I could here. I was also hoping for some more… depth, perhaps? I think that a problem with both the web frameworks and testing tools panels is that they had too many participants for the half-hour-ish time slot. Regardless it served as a good overview of the available tools (if nothing else giving me a list to start Googling from).

date:2007-02-24 18:06:19
wordpress_id:495
layout:post
slug:testing-tools-panel-pycon
comments:
category:pycon2007, python, testing

Scaling Python on the Web

First session of the day was on Scaling Python on the Web; rough notes which I may clean up later:

  • How fast is fast enough?
    • Don’t prematurely optimize
    • Know where the bottlenecks are, and optimize those specifically?
  • Orders of magnitude: static (httpd), dynamic (python), db-queried
  • Even 40 req/s in 3.4m pages/day
  • Hundreds to low thousands of dynamic page views is usually good enough
  • Scaling isn’t about the language, it’s about:
    • DRY: cache!
    • share nothing
  • built a sample photo-app, FlickrKillr, for demonstration purposes
    • preloaded with 100k’s users, 10-20 photos each
  • first iteration: CGI
    • roughly 23 requests/second
    • problems:
      • loading Python interpreter for each request
      • all resources initialized for each request (inc. db connection)
    • possible remedies:
      • run a Python web server (long-running process)
      • make one db connection per thread instead of request
    • other remedies:
      • fastcgi
      • snakelets, twisted.web, RhubarbTart
      • mod_python
  • second iteration: python app server (CherryPy used for this demo)
    • roughly 139 requests per second
    • problems
      • global interpreter lock — can only utilize one core on a dual core machine
      • sessions in the database — prefer an in-memory session store
    • remedies:
      • run multiple instances of CherryPy (overcode GIL)
      • but then we need to balance with something like nginx
    • other options
      • cherrypy in mod_python
  • version 3: load balancing with nginx
    • 217 requests/sec
    • outstanding problems
      • static files read from disk every time
      • and they’re being read/written from python
    • solutions:
      • memcached
      • combine with memcached w/ nginx
  • version 4: caching
    • 616 req/sec (benchmarking w/ homegrown tool)
    • 1750 req/sec (benchmarking w/ ab)
  • other notes:
    • don’t forget to index
    • without an index, the fourth iteration falls down to 28 requests/sec

http://www.polimetrix.com/pycon/demo.tar.gz

date:2007-02-24 12:35:29
wordpress_id:490
layout:post
slug:scaling-python-on-the-web
comments:
category:pycon2007, python

Keynote: Adele Goldberg

The morning keynote for day 2 of PyCon was given by Adele Goldberg. Unfortunately I overslept this morning and missed the first half of the talk[1]_. Goldberg’s topic was eLearning, and the state of computing in education today. My background working in education is very different from the schools she was talking about: she’s apparently been working with computer-based learning in public schools, and studying the aggregate information about the results. My experience was working for a private, K-12, well-funded (regardless of what administrators ever said, the truism was that if they wanted money to be there, money would be there) school. So a lot of the statistics about teachers complaining about computers being out of date didn’t apply.

But the other things, about what works and what doesn’t, just isn’t a suprise. I wish I could remember the examples through the lack-of-coffee haze, but I was suprised by the similarity — I would have thought “money changes everything” (to quote the Cindi). And two points that hit home: teachers (in the aggregate, I presume) haven’t had a pay raise of more than 1% since the 1990-1991 school year, and that the way to get students to learn is to get them to build things and explain what they built to other students.

I was exposed to the latter theory in my Software Engineering course at IPFW. Dr. Modesitt, on the first day of class, presented the idea that there are five levels of learning, with lecturing being the least effective, up to building as the second most effective, and teaching others being the most effective. Now Dr. Modesitt used this to set the stage for how the class would be structured, with most of the work occuring at those upper two levels. And that class was like drinking from the firehose, but I did learn a lot. So although it was in a completely different context, I can attest that it really works.

It was also interesting to hear that Goldberg had built a course/school system on Zope 2 (hmm, that sounds familiar), and that she’s currently working on rewriting it in Zope 3. And in building that from an “instructional” perspective versus “content management” perspective (which is what STOA definitely has). The new tool, CourseCloud, uses XML to represent exercises and instructional content, with the goal being to present the exercises in multiple contexts.

[1]What? I had two hours of sleep the night before, and wound up buying Age of Empires III for my MacBook at Fry’s and stayed up, well, later than planned.
date:2007-02-24 11:52:25
wordpress_id:489
layout:post
slug:keynote-adele-goldberg
comments:
category:education, pycon2007, python

Web, Web Afternoon

Lunch: tasty “Mexican” food, including those tasty cinnamon-sugar coated crunchy-chewy desserty things. Lots of those.

And after lunch, it’s fun with Python on the Web. The first afternoon talk was Ian Bicking’s Everything You Wanted to Know About WSGI but were Afraid to Ask. OK, that’s just my title; it’s really WSGI: An Introduction. WSGI is a specification for connecting pieces of web applications in Python. It’s something (like Eggs) that I can conceptually understand as being important. But (like Eggs) I have a hard time really groking it.

WSGI provides for this idea known as Middleware, which allows you to wrap your applications in, uh, other applications. Ian actually had a good litmus test for what is middleware: if you code is acting as both a client and a server, then by definition it is middleware. Ian also spent some time discussing the idea of “microframeworks”, pieces of software that don’t provide this full stack but can be combined in interesting ways to build something bigger. So yeah, WSGI: darn cool.

Following the WSGI talk was the Python Web Framework Panel, with representatives from Zope, CherryPy, Pylons, Django, TurboGears, Pyjamas, Spyce and Nevow[1]_. Nothing really ground-breaking, but Bob Brewer of CherryPy did make an amusing comment: “Reinventing the wheel isn’t so bad if you’re really interested in wheels.” What did sort of annoy me is the multitude of comments from creators about “looking at Zope and running away screaming.” Look, I agree that Zope isn’t for everyone. I also agree with Phillip: where Zope leads, Python follows. I don’t know when Zope became the Python web community’s collective bitch.

Now, off to my friend Vern’s talk about teaching with Python (and dumping that tired hag “hello, world”), then a nap, lightning talks and finally Shawn and I make our annual pilgrimage to Fry’s.

[1]Apologies if I missed one; this is from memory.
date:2007-02-23 17:10:01
wordpress_id:487
layout:post
slug:web-web-afternoon
comments:
category:pycon2007, python, wsgi