A Plan for GUI Testing

During my presentation on Developing Desktop Applications with Python at PyCon last month, one question in particular intrigued me: “How do you handle unit testing for user interfaces.” My answer, at the time, was the paragon of short and accurate: “I don’t.” But I’ve been thinking about it since then, and I think I have an answer. This post is my attempt to work out the details in my head before actually implementing anything. I ran across Twill the other day. Twill is an application which allows you to execute scripts against web applications for the purposes of testing. You can find a sample script in the Twill README. This idea of scripts got me thinking about testing GUI applications. The thoughts went something like this:

  • If you want to test a GUI, you often want to make sure things happen in a certain order, or that changes to widgets (buttons pressed, etc) cause the correct change to occur.
  • I think I could script at least 75% (and maybe more) of the tests I’d like to perform in a fairly straightforward manner. I think.
  • Integration of GUI testing with unit tests would make my code better. I’m certain of this.

Of course, anyone who’s done much GUI programming, particularly with wxPython, knows that there is the problem of main loops and threads. The main loop is what listens for GUI events, and fires the correct methods. It’s also blocking (i.e., as long as your app is running, it’s running). In wxPython, trying to interact with the GUI from another thread is a sure fire way to crash things in a dramatic and spectacular way. But I remembered seeing a presentation on matplotlib at PyCon, and in it the presenter used the IPython shell to modify a plot in place.

IPython is an enhanced interactive shell for Python (and is incidentally also used by the Twill project). I got to looking at IPython last night, and found that doing something like:

% ipython -wthread wxapp.py

launches IPython, runs wxapp.py in a separate thread and dumps you at the interactive prompt. From there you can introspect into loaded modules, etc. You can’t access instances from the application, but that’s not a problem with wxPython. In wxPython, calling wx.GetApp() will return the application object (which is a Singleton for a given process), and the App has it’s own cooresponding method GetTopLevelWindow(), which returns the parent of all widgets and subwindows for the running application. Armed with these two pieces of knowledge, you can easily introspect the state of widgets in a running wxPython application from the IPython shell.

So let’s consider a hypothetical test for the classic “currency-converter” application:

tester = wxTester()

        make sure the appropriate widgets exist
    assert_ (tester.widgetById(“txtCurrency”))assert_ (tester.widgetById(“txtRate”))assert_ (tester.widgetById(“cmdCalculate”))assert_ (tester.widgetById(“txtValue”))

    tester.widgetById(“txtCurrency”).SetValue(“10”)tester.widgetById(“txtRate”).SetValue(“10”)tester.click (tester.widgetById(“cmdCalculate”))assertEqual (tester.widgetById(“txtValue”).GetValue(), “100”)

So you can see the wxTester class provides a convenience method, widgetById, for finding a widget by it’s unique identifier. I imagine we’d also want to have methods for finding widgets by numeric ID and possibly label, as well. These methods return the actual wxPython object, so you can make calls like SetValue and GetValue to interact with the UI.

I imagine that the click method is actually a special case of a more generic sendMessage method. Since clicks in wxPython (and most other UI toolkits) are sent as messages, we generate a mouse click message, and send it to the appropriate widget. This lets us emulate mouse interactions.

So what are the unanswered questions? Well, I’d like to use the existing unittest framework for running tests. This would allow me to take advantage of it’s convenience methods and error reporting. I think this will be possible; the wxTester object will find the running application’s top level window when instantiated, so you should be able to simply instatiate the object within a particular testcase method. I may also want to have the initializer for wxTester to also run the application script, so that each test starts with a clean instance.

Additionally, there will need to be some experimentation to figure out how to (essentially) run two scripts simultaneously: the tester script and the tested script. IPython has a %run command which allows you to run an external Python script with a given set of command line arguments, and still introspect into it. If the %run command is available from our test script, we have our answer.

Finally, this will require the use of IPython for executing test scripts. This isn’t a deal killer, but doesn’t thrill me. A possible future improvement would be to use the IPython source to instantiate the threads and magic from within a standard Python interpreter. And while we’re on the subject of future improvements for vaporware, some sort of GUI toolkit abstraction would make the library useful for more than just wxPython applications. Someday.

date:2005-04-14 19:00:39
category:development, pycon2005

PyCon Wrap Up

PyCon is over, and I head back to balmy Indiana tomorrow. Overall PyCon was once again an excellent conference. I will, however, remember this as “The Year of Stairs.” For the first time ever the conference was split between the 1st and 3rd floors of the Marvin Center at GWU, and as a result I climbed lots of stairs. And my room is on the 8th floor of Hotel Lombardy, which has a slow, small elevator with an actual elevator operator. So as a result I’ve been using the stairs instead. Am I ever out of shape.

As with past years, there’s lots I want to look into post-PyCon. So in no particular order:

  • I had the opportunity to talk about Roundup with it’s principal developer, Richard Jones. I’m hoping to modify a Roundup tracker to help manage crash reports from CC applications.
  • Yarn looked darn cool, and I want to explore the possibility of building a “desktop information client” that uses Yarn to consume email, RSS, etc. I’m just not satisfied with my current email clients, and Yarn looks like a very capable tool.
  • I want to take another look at Zope 3’s component API, and how it might work outside of Z3.
  • Traits looks like a cool way to do type checking and validation in applications. When they actually ship code, it’ll be worth looking at.
  • I want to download Chandler 0.5 when it’s available and see what I can do with it.
  • I want to look again at rule based “inferrence” engines.
  • Lots of efficiency and implementation talks made me want to download the source to Python and start seeing how Python works under the hood.

But first, ccPublisher maintenance release and catching up at work.

date:2005-03-25 17:42:53

PyCon Day 3

Today was the final day of PyCon. The day started with a keynote by Greg Stein on “Python at Google.” Before the keynote started we were sternly warned that Greg had been requested (by Google legal, I and others around me assumed) not to publish his slides, so we should take notes. With that preface, I was anticipating a “sneak peak” into Google’s secret sauce of Python. Too bad that wasn’t the case. OK, so it was interesting to know that Google considers Python to be one of it’s top three “approved” languages (along with C++ and Java), and that Greg’s intuition is that Google has more Python code than Java in use, but really, how interested can I be in the fact they use SWIG. It’s interesting once, but really, move on, folks. Overall, interesting, but it could have been done in about a third of the time.

After the keynote, I went to Fred Drake’s “Acceptance of XML in the Python Community” followed by Abe Fettig’s “Yarn: Working with Messages in Diverse Formats and Protocols.” Fred’s talk, delayed by technical difficulties can be summarized as follows:

  • Python has several ways to handle XML
  • Most of them are un-Pythonic
  • As developers, its our fault that the documentation isn’t better. Or, as he put it, “I know you don’t like to write documentation. Get over it!”
  • There should be “one way to do it.”

OK, so his points are not completely without merit. But frankly, if it pisses you off, present a working alternative, not a parental lecture. But that was just my take.

Abe’s talk, on the other hand, was really cool. Yarn provides a standard interface for messages of all types. Abe demonstrated the “Yarn Browser,” which was able to easily use Yarn to retrieve message lists from IMAP, mbox, RSS and Atom feeds. He also demonstrated using Yarn to retrieve an RSS feed, modify it in place (and see the XML change as you modified class properties), and emit it as Atom. Darn cool.

The final talk of the morning was Evan Jones’ presentation on improving the Python memory allocator. I didn’t realize that Python currently doesn’t free memory when objects are garbaged collected. The memory is freed up for future allocations made from that Python process, but isn’t released back to the operating system until the task terminates. Ouch. Evan wrote a patch which implements a freeing algorithm that attempts to be a little friendlier to resource constrained platforms. Tim Peters said it will probably go in for Python 2.5 (which Guido, incidentally estimated for early 2006 in his keynote yesterday).

After lunch I hung out in the network block, which could probably be more accurately be called the Twisted block. The first talk of the afternoon, “Fast Networking with Python” was actually Yet Another Optimization Discussion (YAOD), during which Itamar Shtull-Trauring talked about things which you can optimize to improve your network application performance. All good suggestions, but I don’t do enough I/O bound networking to make it that worthwhile. Of particular interest was his discussion on the most memory efficient way to copy and concatenate strings, especially in light of Evan’s earlier talk on the memory allocator.

By far the most interesting talk of the afternoon was Thomas Vander Stichele’s demonstration of Flumotion, a Python app that binds to GStreamer and provides a distributed architecture. In his demonstration, he showed how you can perform the video capture on one machine, the encoding on another, and the actual streaming on yet another. And it really, really works. Oh, and did I mention that it contains support for CC licensing, displaying the CC logo superimposed on the stream? Very cool.

Tonite I’m having supper with Shawn (whom I met at Abe’s session this morning) and his wife, because they are truly kind people who didn’t want me to just hang out in my hotel room tonite.

date:2005-03-25 17:10:19

PyCon Notes

I just noticed that Ted Leung is once again hosting PyCon SubEthaEdit notes. You can find the directory here.

In other social editting software news, there was an open space session on “SubEmacsEdit: Developing Social Editting for Emacs” yesterday (seriously, I can’t make this shit up). I wonder what the outcome was.

UPDATE: Here are the notes from my desktop apps talk. From the looks of things, I guess it went OK.

date:2005-03-24 19:49:23

PyCon Day 2

Today was presentation day, which understandably occupied most of my thinking. So I’ll address that first. My desktop applications talk went well, and I had good questions. I think I’ll be doing some fine tuning for PyUK in April, but overall I think I hit the points that I wanted to. My RDF was less successful, in my mind. I never quite got into the rhythm of the talk, and as a result I don’t think I was very successful in connecting with my audience. Hopefully they’ll check out the wiki page and be inspired by some of the details there.

I had dinner with my former boss, Latin teacher and mentor, Vern, and he had some suggestions. His feedback was that I seemed to “come alive” when answering audience questions, and that if I could inject that energy into the primary portion of the talks, I’d be much further ahead. He suggested figuring out what “story” I was trying to tell with each slide before hand, so they could serve as backup instead of main exhibit. Vern was also encouraging about the RDF talk. While he agreed that I didn’t seem to get into the rhythm of speaking, he didn’t think it was a bomb, and thought there was some good information in it.

This morning before I spoke I had the opportunity to see Ted Leung and Katie Capps Parlante from OSAF present on building Chandler parcels. I’m sort of interested in that. I’m currently really unhappy with my email clients (Thunderbird and Evolution), not because they’re not doing their jobs, but because I’m having trouble keeping track of everything. vFolders help some in Evolution, but I like some of the Chandler features they demoed today. Hopefully it’ll be minimally functinal “real soon now”.

After lunch I attended the block of “database” talks and saw two database projects which approach the domain very differently. Patrick O’Brien presented on Schevo, a framework for creating object database applications. Schevo allows you to create an object schema, and provides a built in data “navigator” which essentially generates the basic selection/deletion/query user interface for you. A key feature that wasn’t demonstrated but alluded to is the ability to version your schema and “evolve” your database as needs change (or are corrected, depending on the client, I suppose).

Ed Leafe’s presentation on Dabo was from the polar opposite position. Ed is a former (current?) Visual FoxPro developer, and Dabo is a framework for developing 3 teir database applications in Python. The major difference of opinion between the two presenters is the role of SQL and RDBMS. O’Brien was really frank in his distaste for SQL (really, does anyone like it?) and the constraints relational databases place on development, while Leafe clearly longed for the days when Fox was king and Access was a toy.

Overall, though, it was good to see two different, well thought out view points and projects in a domain I don’t work with much lately. The one area of overlap between those projects, though, is automatic UI generation. It must be in the water this year, as David Morrill demonstrated a similar feature in his talk on Traits yesterday.

Following the afternoon break I attended what I suppose you could call the “rules and triples” block of talks, of which mine was last. Yarden Katz’s talk on PyChinko was interesting, although I found myself longing for a demo. Katz and his colleagues have obviously thought quite a bit more about RDF than I want to, and I’d be curious to have seen their inference engine in action. Between the two RDF talks was Phillip Eby’s talk on writing rule-driven software using PEAK’s Dispatch. I think he had an important message, but the talk seemed designed for a larger time slot, and I found it difficult to follow the reasoning. Again, I think a working example, followed by discussion of the details would probably have kicked ass.

date:2005-03-24 19:44:57

PyCon Day 1

So the first day of PyCon is over. After Jim’s exciting keynote about Python on .Net, I attended Bob Ippolito’s sessions on PyObjC (and introductory session and a hacking session). I’ve become really interested in Smalltalk lately, and (due to cosmetic similarities) Objective-C. I also continue to be interested in writing OS X applications a little closer to the actual OS.

Bob’s hacking talk was particularly interesting, as he demonstrated using PyObjC to inject code into running applications, or override implementation of particular methods. Pure evil, but darn cool.

After lunch I crunched email while watching John Hunter’s talk on matplotlib. Very cool, very straightforward. Makes me wish I had something to plot.

Other points of interest from day one:

  • David Morrill’s talk on Traits. Traits provide strong-typing (sort of) for Python, as well as a host of other facilities. For example, you can state that a variable should hold an integer from 0 to 10, and Traits will do validation for you. It also has some UI generation facilities, which seem to make it ideal for allowing non-geeks to extend applications.
  • Brett Cannon reported on his Master’s thesis, in which he modified Python 2.3.4 to do local type inferencing to see how much of a speed boost he’d get. The answer: not much. Sorta surprising, given last year’s talk on massive speed improvments with Starkiller.
  • Finally, the day closed (at 6:30 PM) with Jim Fulton’s talk on “A Layered Event System to Provide Method Extensibility”. He discussed events, and (tangentially) Zope 3’s component architecture. We’re dealing with some issues of 3rd party extensibility in ccPublisher 2, so I’ll have to look into whether or not this will help us out.

So yes, day one was long. Worth it, but long. After grabbing a burrito and heading back to my room, I spent the next two hours finishing my RDF talk and crashed. Slides are available here, and I’m actually more pleased with it than my Desktop Apps talk. We’ll see how they go.

date:2005-03-24 08:49:15

(re) Mixing Python

This morning’s keynote at PyCon was given by Jim Huginin, formerly of Jython fame, now of IronPython fame. Jim joined Microsoft [STRIKEOUT:8] 2 months ago, and was able to announce the release of IronPython 0.7 (website) today. Jim’s talk was interesting for a number of reasons, in no particular order:

  • He demonstrated interactively accessing Microsoft DLLs from the Python prompt. In particular he demonstrated instantiating the Microsoft Agent Peedy the Parrot and then calling Avalon. The Avalon portion was particularly interesting. He first showed declaratively creating the user interface, and then showed off loading the user interface from XAML. This reminded me a lot of both XML User interface Language”>XUL (from Mozilla) and (to a greater degree) XML Resource Control”>XRC (from wxWidgets). This was pretty interesting to me, since some of the “walk” methods that the framework provides would make life with XRC much easier.
  • He demonstrated delegating a C# method to a Python function, and then calling that Python function, through it’s C# parent class, from the interactive prompt. Yeah, he was sweating during that demo, too.
  • He talked “for 7 minutes” about the different ways in which the CPython virtual machine and the CLR virtual machine generate op codes, and further down the line, x86 machine code. I actually appreciated my class in assembly language from last year during that portion of the talk.

The only other impression I got from his talk is that Jim has had some less-than-savory run-ins with the Microsoft Legal Department, one of which led to his talk title “Implementation Running On .Net of Python” instead of just “IronPython”. Whatever.

date:2005-03-23 11:48:51

PyCon 2005

I’m in Washington, DC for the remainder of the week for PyCon 2005. This years’ schedule looks really interesting, and of course I’m presenting tomorrow. Twice. So I’m a little nervous about that.

I’ve posted updated slides for my Desktop Apps talk, and I’ll be putting slides for my RDF talk online later today (OK, probably tonite).

date:2005-03-23 08:48:09

Desktop Apps Talk Slides (1st Draft)

So I’m trying to get my slides put together for PyCon. It’s a busy week for me, and the hand isn’t making things any easier. Anyway, the first draft of my slides for the Desktop Apps talk are now online. Feedback is appreciated.

date:2005-03-16 12:28:01

Preparing for PyCon

PyCon 2005 is next week at GWU in Washington, DC. The schedule was posted online a couple weeks ago, and I’m very impressed with the selection of talks this year. While PyCon has always been informative and interesting, it looks like the quality of talks is going to be consistently good this year. And there’s still time to register at the “regular” rate (before it increases to the “at the door” rate on March 19).

I’m once again presenting, so I’m spending much of my time this week preparing my presentations. I’ve created a page in my wiki where I’ll be assembling talk related materials, links, etc. I’m using Eric Meyer’s S5 this year, which is pretty cool. Now if only I had a simple desktop app for assembling the XHTML.

date:2005-03-15 13:07:17