Newbie Nugget: Unit Testing with Mock

I presented my second “Newbie Nugget” at the October BayPIGgies meeting. Since I’ve been working with Mock quite a bit lately, I chose to present on that. (I also thought I could present on decorators, but I decided Mock would be easier since I’d have to do the least to prepare. I remember feeling confident on my knowledge of Mock itself, but I suspected there was still a lot I had to learn on unit testing. My suspicions proved correct.)

I didn’t make as much time to prepare as I would have liked. I ended up with 18 slides, and didn’t have time to trim them down. So I had to breeze through them very quickly. (In theory I only had 5 minutes. I don’t know how long I actually took—probably more like 10—it turned out there wasn’t any pressure this time to keep it brief.) I also would have liked to have time to come up with some real-world examples, but in my rush to finish the slides I ended up taking the code examples from Michael Foord’s documentation, except for the last two examples which I pulled out of my own code. I’ve uploaded the Keynote slides exported to PDF and to HTML. To save some of you some time, the references I had in my final slide are:

• The Mock documentation
• Michael Foord’s “Mocking, Patching, Stubbing: all that Stuff” blog post
• The Wikipedia article on mock objects
• Martin Fowler’s “Mocks Aren’t Stubs” essay

Note that I prepared this presentation based upon Mock version 0.3.1. Michael has since released version 0.4.0. TODO: I’d like to describe the details on what’s new in 0.4.0—but in the meantime, see the notes in the documentation or Michael’s blog post.

After the main presentation on PyGameSF, there was some extra time left and Jim Stockford asked if I’d be willing to take questions. I’m glad he asked, and I’m glad I did so. (How could I not?) I probably will not recall all the questions. Feel free to ask them again here, or on the BayPIGgies mailing list.

The first question I remember was whether I’ve found any bugs in Mock. I replied no, and elaborated that Mock was written using test-driven development and the tests themselves use Mock (and are included with Mock). So if you do find a bug, you can easily fix it yourself. (But looking at the changelog, it appears there have been few bugs fixes. Most changes have been new features. BTW, I’ve submitted a couple patches with new features myself, but I believe Michael decided he wasn’t ready for them or decided to implement them in a different way. TODO: I have a couple ideas for other new features to add.) And Mock’s unit tests are good examples of how to use Mock.

Then Alex Martelli commented (I probably won’t get this right) that my use of the @patch decorator in my tests is dangerous and won’t scale. (It’s not clear to me whether he meant it won’t scale to projects with large amounts of code, or large numbers of programmers, or large numbers of processes and/or threads, or something else entirely different.) He suggested we have a look at the video of him talking about this. Update: I thought that may be this YouTube video from Google Developer Day 2008 on “Python Design Patterns” (added June 4, 2007?), but Alex comments (below) that the video is not currently available, but we can find the slides at http://www.aleax.it/yt_pydi.pdf. And Alex described how the code would use “self” to refer to any dependency, which would allow that dependency to be changed by tests, or by other code that might use it.

I then asked a question (which I can no longer recall) and Alex stood up and came up to the microphone to elaborate. After sitting down, I (and others) asked further questions (which I also can’t recall) and Alex got some exercise sitting down and getting back up a few times until the two of us stayed at the podium and Alex gave me (and the audience) an impromptu (but very meaty) lesson on dependency injection. (I hope Google makes the video of this available—I’d like to watch it again myself. I haven’t found the video of my previous newbie nugget presentation yet though.) I do remember one (rather foolish) question I asked: I described how I had tried dependency injection using keyword arguments to functions with defaults, and how that gets complicated to test when testing a function which calls a function which calls a function with such an argument. Alex kindly repeated that dependency injection should be implemented using “self”, which I took to mean that one would use an object’s attributes to hold the dependencies. (This would certainly make the scenario I described much simpler to test.)

In a discussion after the meeting, I explained to one of several people I had very interesting follow-up conversations with (I didn’t get his name) on a white-board how this would work. The Google engineer who had graciously agreed at the last minute to host our meeting (I didn’t get is name either—but learned later that he was visiting from Australia) was watching (probably because we were over-staying our welcome) and as we were walking how he explained how the accepted technique is actually to use class attributes for the dependencies, since it’s less work to inject new dependencies into multiple objects, but one can still override dependencies for a particular object (since an object attribute will “replace” a class attribute with the same name).

One other question I recall (it may have been the last) was -jj’s. He asked (shrewdly) asked me to explain why one would want to use mocks. I decided to (figuratively) take a step back and first describe why one would want to test and use test-driven development (and what that is). I described the benefits those practices and then explained (or tried to) how mocks are used in testing to replace dependencies and allow unit tests to be truly “unit” tests and run fast to keep the test-implement-refactor cycle quick. TODO: Write this up in detail.

I concluded by thanking everyone for their questions and comments (especially Alex)—I stated I felt like I had learned more from everyone else than they probably did from me. When I sat down afterward, Alex turned to me and quoted Richard Bach: “You teach best what you most need to learn.”. Absolutely.

Next: Write up my experiences modifying my code and tests to use dependency injection, and contrast the new versions of the two examples from my slides.

Newbie Nugget: Introduction to the with statement

I’ve signed up to present the first BayPIGgies “Newbie Nugget” at tonight’s meeting, on the with statement. I know from past experience that the best way to learn something is to explain it—and one has to learn it very well to explain it to a large group of smart people.

What follows are the notes I compiled as I taught myself about the with statement. I’ll turn these into slides for tonight. Since I only have 10 minutes for the presentation, I’ll keep these notes brief. (Think of this as an introduction to the with statement.) You’ll find links to further reading below.

We’re all used to (or should be used to) writing code that looks like this:

f = open(filename)
try:
   print f.read()
finally:
   f.close()

The with statement (introduced in Python 2.5) simplifies that to:

from __future__ import with_statement
with open(filename) as f:
   print f.read()

(“with” and “as” are keywords in Python 2.6, and “from __future__ import…” is not needed.)

The specified syntax for the with statement is:

with EXPR as VAR:
BLOCK

The “as VAR” part is optional.

The EXPR must result in a context manager. Briefly, a context manager is an object with a __context__() method, which returns a context object. A context object has an __enter__() method, which is invoked as soon as a context object is returned. Its return value is assigned to VAR (if there is one). When the with statement terminates, the context object’s __exit__() method is called. (I won’t take the time to go into the arguments to __exit__().)

The contextlib module provides “utilities for common tasks involving the with statement.”

The contextlib.closing function returns a context manager that closes its argument. For example:

from __future__ import with_statement
from contextlib import closing
import urllib

with closing(urllib.urlopen('http://www.python.org')) as page:
   for line in page:
       print line

And the contextlib.contextmanager function “is a decorator that can be used to define a factory function for with statement context managers, without needing to create a class or separate enter() and exit() methods.” For example:

from __future__ import with_statement
from contextlib import contextmanager

@contextmanager
def opened(filename):
   f = open(filename)
   try:
       yield f
   finally:
       f.close()

with opened(filename) as f:
   print f.read()

There is plenty more detail in PEP 343, including plenty of fun examples. My favorites are:

• (2) The opened() context manager used above
• (3) transaction() context manager for committing or rolling back a database transaction
• (5) stdout_redirected() context manager for temporarily redirecting stdout
• (7) A context manager for blocking signals. (This one is actually left as an exercise to the reader.)

To learn about the with statement (before reading PEP 343 and the contextlib module documentation), I first turned to section 10.4 (Context Management) of Core Python Programming (2nd Edition) by Wesley J. Chun. It provided an excellent introduction.

Django multiple database API proposal

I finally wrote up the multiple database API I proposed at the Django sprint at PyCon 2008, in a post to the django-developers mailing list. Hopefully I'll soon be able to finish the coding I began then and finally create a working patch so the community can play with it and test it.

Google Developer Podcast: Google App Engine

If you have any interest in Google App Engine, I recommend listening to episode 15 of the Google Developer Podcast.

I haven't made time to play with App Engine yet, but I'm looking forward to it. I registered for the Google App Engine "Hack-a-thon" (in San Francisco on May 16th), so if I can't make time to play with it before then, at least I'll have the day to hack away. Hopefully they'll give out accounts to attendees (I wasn't one of the first 10,000 to sign up so I didn't get an account)--but if not, I can still download the SDK and start developing.

PyCon 2008 Notes

I just completed writing up my notes on PyCon 2008 for my employer (who paid my way), and thought I should also share them here. I apologize for using one long blog post; I decided the ability to refer to a single post outweighed the advantages of several smaller ones. (And you'll have to ignore the "TODOs" I've sprinkled in the text.)

I attended PyCon from March 13 through 20, 2008.

Tutorials

Thursday April 13 was a day of tutorials. I attended three (of a maximum of three).

Secrets of the Framework Creators Tutorial

The first tutorial I attended was "Secrets of the Framework Creators", presented by Feihong Hsu and Kumar McMillan. This was an excellent tutorial that opened my my to some much more sophisticated ways of using Python. (See the three previous posts I've written on this tutorial so far.)

You'll find the materials they prepared for the tutorial in the Google Group they created. They covered four topics:

• Frame Hacks
• Decorators
• Metaclasses
• Magic Methods

(Of these four, frame hacks and metaclasses were completely new to me. I already had some experience with decorators and magic methods.)

It would be a waste of my time to summarize any of the material presented, since the tutorial materials do such an excellent job introducing these subjects. If you want a quick overview, you can probably read through them in an hour or less. And if you really want to learn to use these techniques, you can spend a few hours on the exercises they've provided.

Here are the few brief notes I took during the tutorial:

• TODO - Learn about properties and descriptors
• TODO - Look into Trellis
• TODO - Look at functools
• TODO - Look at the inspect module
• The "automatic debugger methods" recipe in the metaclasses section is way cool

Generator Tricks for Systems Programmers

The second tutorial I attended was "Generator Tricks for Systems Programmers", presented by David Beazley. David has also made his tutorial materials (including excellent slides and plenty of code samples) publicly available: http://www.dabeaz.com/generators/.

At the start of the tutorial I had written iterators, and had a vague understanding of generators. After the tutorial, I now feel like my understanding of generators is much deeper. I very highly recommend you take the time to read through the slides (probably one to two hours) and look through his code samples.

TODO - add a summary here

My notes:

• I didn't know that "Conditional Expressions" (aka "the Ternary Operator") was added to Python in 2.5. See an example at the bottom of slide 64.
• There's a nifty trick in using the max function under the first bullet of slide 71. TODO - explain.
• I wrote on slide 135 that the find_404() function could be modified to take a receiver arg and call it instead of the print (and the print could be wrapped in a consumer function). TODO - explain
• TODO - I also gave myself a TODO on slide 135 to compare this to the previous use(s) of broadcast
• TODO - Someone mentioned the "etree" (I think they meant ElementTree), which either uses generators for parsing XML or can be used with generators.

Django Code Lab

The last tutorial I attended was the Django Code Lab. The presenters, Adrian Holovaty (one of the creators and core developers of Django), Jacob Kaplan-Moss (a core developer of Django) and James Bennett (release manager for Django) asked people to send in questions and code samples, and then reviewed them in the lab. (I did submit some code developed at the last minute--my main question was how best to integrate models that don't use the database with others that do--but mine was not chosen.)

The slides are posted at http://toys.jacobian.org/presentations/2008/pycon/codelab/. They may not be too useful out of context. But I refer to them in my notes (below):

The first up was Pim Van Heuven:

• he had a huge urls.py file - see the slides
• they also presented "Forms with extra parameters"

Next was Justin Lilly:

• Jacob ranted about TDD
• then he presented some useful information about testing Django
  • look at the model examples in the Django documentation, they're all working unit tests
  • see slides for example of using django.test.TestCase

Next was Richard House:

• he had a long list of BooleanFields in his model
  • they pointed out that one should use NullBooleanField rather than BooleanField(null=True)
  • the slides present a pattern for using "two pairs of models"
  • Adrian pointed out this is called EAV (Entity-Attribute-Value) and noted this has problems with searching and queries

Next, Peter Herndon:

• he suffered from slow queries (see the slides)
• someone recommended a Malcolm Tredinnick blog post (I think this one); though his blog is definitely recommended reading for anyone using Django
• see the optimization notes for your database - Jacob says the PostgreSQL optimization notes are especially good
• James recommends especially django-sphinx for searching (David Cramer used it for a very busy site with a single database server)
• Jacob mentioned Google Co-op several times

Next, J. Clifford (Cliff) Dyer:

• Handling previous/next links
• I didn't fully understand the discussion, and there were no code examples provided unfortunately (and no slides)

Next, Dave Lowe:

• the subject was when not to use the admin
  • the advice was that the admin should only be used by someone trusted with access to the entire database (even after the newforms-admin branch is merged back into the trunk)
• Adrian said that the key to mastering Django as a tool is to learn (through experience) when to use different parts of Django and when not to

Next, Bob Haugen:

• prepping for deployment
• make sure that django.views.static.serve is blocked out inside settings.DEBUG
• see the HOME trick with os.path.join() for TEMPLATES_DIR
• see the urlresolvers.reverse slide
• worth looking at mod_wsgi first (before mod_python)
• Jacob admitted (reluctantly) that MySQL is easier to setup than PostgreSQL
• see the slide 97, which summarizes "Develop" vs. "Deploy"
  • Jacob forgot to include creating 500.html & 404.html pages

Next, Wiley Kestner:

• see the slides for details on the Django dispatcher

Then they ran out of time, but the slides also contain some details on REST APIs,

Slides for other tutorials

I got the URL for the slides from the Introduction to SQLAlchemy tutorial from IRC: http://utahpython.org/jellis/. The slides are in "sa-intro.pdf". (But it looks like there's some other interesting stuff there too.)

Conference Sessions

Friday, April 14 through Sunday, April 16 were the Conference Days. Each day contained keynote talks, scheduled talks, lightning talks, and "open space". (See the previous link for details.)

I'll go into detail on the scheduled talks (from all three days) first, and then dump all my notes from the lightning talks.

The A/V team as begun posting recordings from PyCon 2008 to their YouTube channel: see this blog post. As of 2008-03-28 they've got 12 up, some from regular sessions and some from lightning talks. None are from tutorials--I don't know if they intend to post tutorial videos.

Guido van Rossum - "Python 3000 and You"

The first technical talk of the conference was fittingly Guido's talk on Python 3000. Guido has posted the slides and some advice when porting to Python 3.0 on his blog.

I didn't take any notes, but if you're interested in Python 3000, PEP 3000 is work reading. (In addition to the slides.)

Brett Cannon - How Import Works

This was useful information. You'll find the slides in http://us.pycon.org/2008/conference/schedule/event/12/.

See also http://www.python.org/dev/peps/pep-0328/ and http://www.python.org/dev/peps/pep-0302/.

Dr. Tim Couper - Python references and practical solutions to reference-related problems

This was an excellent talk. The video isn't up yet, but I hope it is soon. (Unfortunately no slides are attached to http://us.pycon.org/2008/conference/schedule/event/16/.)

Here are my (sketchy) notes:

• see the getrefcount() function
• weakref.ref()
  • can get the actual reference from a week reference, with "()"
• getweakrefcount()
• getweakref()
• garbage collector will clean up objects not deleted because of a circular reference
• can turn off GC if you're sure you'll have no circular references
• GC will not clean up objects with a __del__ magic method
  • but you can get a list of these objects
• then I got distracted and got lost when he talked about pickling and weak references as a solution for this?
  • something interesting about a pattern using __getstate__ and __setstate__ methods
  • also talked about finalizers

Noah Gift - Using Optparse, Subprocess, and Doctest To Make Agile Unix Utilities

The presentation and code samples are at http://code.noahgift.com/pycon2008/.

TODO - I need to take a look at the subprocess module.

Kevin Dangoor - Rich UI Webapps with TurboGears 2 and Dojo

This was a very slick, fast-moving presentation with excellent examples on how to use Dojo. There was also an example of (bleeding edge, I think) use of Comet to push data from the server to the client.

I'll want to see this one again when it's available on video. I haven't been able to find the slides, but Kevin promises to post a screencast version in this blog post.

Adrian Holovaty - The State of Django

I haven't been able to find the slides, and the video isn't up yet. But I recommend it (if you're interested in Django).

Here are my notes:

• What's new in the last year:
  • 0.96 - released March 23, 2007!
  • Unicode branch - is it still in a separate branch? (I think it was merged to trunk)
  • auto-escaping in templates
  • GeoDjango - still a branch
    • "hope to get it integrated into the trunk soon"
  • Sprints in Sept. & Nov.
  • 2432 checkins
• Community stuff:
  • http://www.djangosites.org/
  • http://www.djangosnippets.org/
  • http://djangogigs.com/
  • http://djangopeople.net/
• What's coming
  • mostly mature/stable
  • queryset-refactor to be merged to trunk
    • includes support for model subclassing
  • select_related - can now specify args
  • newforms-admin
    • admin options defined in separate class; register
    • has_change_position() method in ModelAdmin classes
    • can regulate which objects show up in the admin
    • can have multiple admin sites on the same website
  • want to add more sophisticated INSTALLED_APPS (using classes)
    • can define database prefixes & labels (?)
  • model validation
  • 1.0!

Marty Alchin - Django: Under the Hood

Plenty of useful information that I plan to review again (more slowly). His slides are at http://www.slideshare.net/Gulopine/django-under-the-hood/.

Some notes:

• the quick review of how metaclasses are used was useful as used a metaclass in the code I wrote during the Django sprint
• the signals discussion may be useful to users of Django too
• Utilities
  • functional utilties
    • several, including turn middleware into per-view decorators
  • text utilities
    • various
• "remember, Django is Python, just read the source"
  • but beware of query.py and related.py

See this Adam Gomaa blog post for links to the source files to look at for model metaclasses, the "signalling stuff" and "some neat functional utilities".

Chris McDonough & Mike Naberezny - Supervisor as a Platform

This looks like a very useful tool. I plan on using it very soon. The slides are in http://supervisord.org/wp-content/uploads/2008/03/ and are very good--I don't have anything to add. (Except I read a rumour somewhere that Guido wrote the original version of this--details are in http://supervisord.org/contributors/.)

Mike Bayer - SQLAlchemy 0.4 and Beyond

I missed this because it was at the same time as the Supervisor session. But I heard this was very good. You'll find the slides at http://techspot.zzzeek.org/?p=22.

Rodney Drenth - Decorated State Machines

I missed this one too (it was at the same time as the above two sessions). But I see the slides are at http://us.pycon.org/2008/conference/schedule/event/43/. TODO - I haven't looked at them yet.

Maciej Fijalkowski - The State of PyPy

This was interesting. I learned a bit about PyPy (a subject I know very little about). I can't find the slides, and the video isn't up yet.

My notes are very brief:

• Currently:
  • quite compliant 2.4/2.5
  • still missing some standard libs
  • new: ctypes!

I vaguely recall an interesting demo.

James Bennett - Developing Reusable Django Applications

This was a very good talk that would be worth reviewing by anyone using Django. The slides are in http://us.pycon.org/2008/conference/schedule/event/50/.

I'll repeat some of the slides that I noted:

• Four big ideas:
  • do one thing, and do it well (the UNIX philosophy)
  • don't be afraid of multiple apps
  • write for flexibility
  • build to distribute
• He wrote django-registration
  • people asked for profiles (in django-registration)--he said no
  • but then he wrote django-profiles (as a separate app)
• The Django mindset
  • application == some bit of functionality
  • site == several applications
  • tend to spin off new applications liberally
• What reusable apps look like
  • single module directly on Python path
  • related modules under a package
  • no project cruft whatsoever
• Good examples:
  • django-tagging (Jonathan Buchanan, http://code.google.com/p/django-tagging/)
  • django-atompub (James Tauber, http://code.google.com/p/django-atompub/)
• More information:
  • django-hotclub (http://groups.google.com/group/django-hotclub/)
  • Jannis Leidel’s django-packages (http://code.google.com/p/django-reusableapps/)

Alex Martelli - Don't call us, we'll call you: callback patterns and idioms in Python

This was at the same time as the Developing Reusable Django Applications talk, so I missed it. But the slides are in http://www.aleax.it/pyc08_cback.pdf. TODO - I haven't looked at them yet, but I've read good things about this talk.

Brandon Rhodes - Using Grok to Walk Like a Duck

You'll find the slides and example source code at http://rhodesmill.org/brandon/adapters/. (And this one is on YouTube. The title is misleading--this was really primarily about the Adapter pattern and the Zope Interface class. I left wishing he provided some examples of using Grok.

I got a little lost when he gets into zope.interface.Interface. TODO - The best way to understand this would be a play around with it.

Steven Wilcox - The Power of Django Admin (Even For Non-Django Projects)

The "slides" (actually a paper) are at http://devpicayune.com/pycon2008/django_admin.html. This doesn't look familiar, so I must have been busy with something else at the time--though I can't recall what. Perhaps that presentation wasn't as compelling as the material deserved. TODO - This does appear to be worth reading.

Brian Dorsey, Maciej Fijalkowski - py.test: Towards Interactive, Distributed and Rapid Testing

This was a very good presentation that I would recommend watching when it's available on YouTube.

See http://codespeak.net/py/dist/test.html

Here are my notes:

• tests for py.test are almost identical to nose
• tab completion in bash after -k flag slick
• I missed the test reporting page thing, but heard it was slick - and I believe that's one thing that distinguishes py.test from nose
• 1.0 soon
  • more plug-in architecture
• yes, there is a tool for converting unittest tests to py.test tests
  • you can convert tests, or just run them without (permanently) converting them
• what does py.test do that nose doesn't?
  • the introspection magic
• py.test.raises(Exception, stuff, arg1, arg2)
• TODO check out py.execnet (http://codespeak.net/py/dist/execnet.html - "A new view on distributed execution")

Jim Baker - More Iterators in Action

My notes say the slides are at http://zyasoft.com/, but that page strangely describes Zyasoft consulting services and software development but provides no links or contact information. The slides are attached to http://us.pycon.org/2008/conference/schedule/event/75/ though.

Here are my notes:

• LINQ - Language INtegrated Query - http://en.wikipedia.org/wiki/Language_Integrated_Query
• "The solution to integration in Python is naming (more or less)." [I may have got that wrong.]
• collections.namedtuple - new in 2.6
• see also cookbook recipe NamedTuple? - http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/303439
• TODO look up tee() - http://docs.python.org/lib/itertools-functions.html
• operator.itemgetter() - http://docs.python.org/lib/module-operator.html
• defaultdict - http://docs.python.org/lib/defaultdict-objects.html
• Recursive Generators
  • are they practical? sometimes, but they're fun
  • see cookbook recipe 190465 - http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/190465

Raymond Hettinger - Core Python Containers - Under the Hood

An excellent presentation that I will most definitely watch again when it's available on YouTube. I haven't been able to find the slides.

I didn't take very good notes:

• python/trunk/Include/listobject.h
• python/trunk/Objects/listobject.c
• collections.deque() is optimized for appends and pops at both ends, but it is slower for indexed access

Collin Winter - 2to3: Translating Python 2 to Python 3

I haven't been able to find the slides.

My notes are very brief:

• collinw@gmail.com
• workflow:
  • maintain in 2.x
  • fix 2.6 -3 warnings
  • run 2to3
  • test suite!
  • release 3.x version

Lightning Talks

There were three Lightning Talk sessions, after the scheduled talks on Friday, Saturday, and Sunday. Here are my notes. (I marked my favorites with *.)

Friday (April 14)

• *How to write a daemon in Python
• Trestle - a nose plugin for writing testable documentation for web apps (or shell commands)
• *Noonhat - "Changing the world through lunch"
• *Seattle Saturday House
• Thrift lightweight RPC mechanism
• *tinypy

Saturday (April 15)

• Bazaar - "if you can run Python 2.4, then you can run Bazaar"

Sunday (April 16)

• Wesley Chun - http://www.nearbynow.com/
  • Text a "mall code" to 632729 (NEARBY) to search products, brands and sales
• Zed Shaw
• *SNPedia - "SNP" is pronounced "SNIP"
• *python4ply - a Python parser for the Python language
• Jython - 2 use cases:
  • script Java libraries from Python
  • "maintain huge collapsing J2EE app"
• bug.gd - "...submit an error message you've seen and we'll try to find other people who've seen the same error..."
  • see also error_help() for Python hackers
• *geopy - a geocoding toolbox for Python
• from LJWorld.com:
  • Crimes reported in KU residences
  • Foreclosures: Sheriff's Sales
• Elixir - a declarative layer on top of the SQLAlchemy library

UPDATE (2008-04-03)

As I read through blog posts on PyCon, I'll add links here.

To start, I recommend -JJ's posts on:

• SQLAlchemy 0.4 and Beyond
• Managing Complexity (and Testing)
• Consuming HTML
• Core Python Containers--Under the Hood
  

Did I miss any? (You just may want to scan all his PyCon 2008 posts.)

UPDATE (2008-04-11)

Yesterday evening I presented my "PyCon 2008 Notes" to BayPIGgies, in slide form. The slides don't have any of the links above, and I stripped out much of my notes, but if you really want them you'll find them at http://www.spitzer.us/daryl/baypiggies/pycon_2008_notes/ (exported to HTML from Keynote) or http://www.spitzer.us/daryl/baypiggies/pycon_2008_notes/pycon_2008_notes.pdf.

String Interpolation

One of the prerequisites for recipe #1 ("Ruby-style string interpolation") from the Frame Hacks section of the "Secrets of the Framework Creators" PyCon tutorial is the string.Template class. Before I took the tutorial I was vaguely aware of the string.Template class, but I don't think I've ever used it.

I'm going to consider using it though. There are probably plenty of cases where it would be more readable than using the % operator--I can recall a couple times in code I wrote very recently.

In step 1 of Frame Hacks recipe #1, Feihong and Kumar how how one can write a function that wraps string.Template and the substitute method so the mapping argument (and optional keyword arguments) don't need to be provided.

SPOILER WARNING

def interpolate(templateStr):
  import sys
  from string import Template
  t = Template(templateStr)
  frame = sys._getframe(1)
  return t.substitute(**frame.f_locals)

name = 'Feihong'
place = 'Chicago'
print interpolate("My name is ${name}. I work in ${place}.")

Here's what it could look like without the sys._getframe() call:

def interpolate(templateStr, **kws):
    from string import Template
    return Template(templateStr).substitute(kws)

name = 'Feihong'
place = 'Chicago'
print interpolate("My name is ${name}. I work in ${place}.",
                  name=name, place=place)

I guess I could argue that the latter is more explicit, but perhaps the former is explicit enough. I guess I'll be the judge of that (for myself) the next time I write (or am ready to refactor) an ugly string using the % operator. I'll let you know if I decide to add the interpolate function to my utilities module.

The solution for step 2 of recipe #1 (which I showed a partial solution for when I wrote about Frame Hacks) allows for expressions in the template strings. I don't think I've run across a use case for that, but I'll write about it if I do.

I should also mention that Python 3.0 will include a new (third) method of string interpolation, the format method to be added to the built-in string class. (There's more to it than just that. You can read all about it in PEP 3101. And if--like me--you don't know just what a PEP is, I recommend reading the "PEPs" section hear the top of the description of Python's Development Process. And you'll find plenty more detail in PEP 1.)

The above example would look like this using the new format method:

name = 'Feihong'
place = 'Chicago'
print "My name is {name}. I work in {place}.".format(name=name, place=place)

There doesn't seem to be a spec for the format method yet, so it's not clear whether one could wrap it and use sys._getframe(). I don't know if that's an argument for avoiding using it now though. (I don't see any reason why the interpolate() function above wouldn't work in Python 3.0.)

TODO

I haven't installed the latest Python 3.0 alpha yet. I should make time to do so and play with the above.

Pygments

I thought I'd write a quick post on how I highlighted the Python code in my previous post. I used Pygments, which (though I've only had limited experience with it) appears to truly be a "top quality product". It was easy to install with easy_install.

Because I'm pasting the highlighted code into Blogger, I can't use the CSS it generates by default, so I use the "noclasses" option:

$ pygmentize -f html -O noclasses -o file.html file.py

And then I can paste the contents of the html file into my Blogger post's HTML.

Actually, on Mac OS X I can save a step using:

$ pygmentize -f html -O noclasses file.py | pbcopy

And then paste directly into Blogger.

Update: the following command-line allows me to pygmentize the Python code in the clipboard (without having to create a file for it):

$ pbpaste | pygmentize -f html -O noclasses -l python | pbcopy

Another Update: it occurred to me I could use the above (updated) command-line in TextExpander (see A TextExpander snippet to paste quoted text and A tr.im TextExpander snippet that works for me) to create a snippet that pastes a "Pygmentized" version of the Python code in the clipboard:

Frame Hacks

Someone (I forget who--I've listened to so many smart people here at PyCon) said yesterday that Python treats us like adults--if we want go "off-road" we can. (Other languages try to protect programmers from themselves and others. With Python we need to write tests to get that protection. Not that statically-typed languages with protected attributes and methods make testing unnecessary. But that's a subject for another time.) But until I took the "Secrets of the Framework Creators" tutorial at PyCon, I had no idea just how far into the wilderness one could go.

As Leihong and Kumar explain in the getting started chapter of the frame hacks section, one can call sys._getframe() to get the "frame" of the calling functions, all the way to the bottom of the call stack. This made immediate sense to me, since one of the first bits of real code I wrote when I started at Altera did a stack trace on the Nios (Classic) processor. (That was tricky, since the original Nios used register windows--but I'd better stop before I go down a rat hole.)

As I wrote previously, I'm working my way through the tutorial again on my own to give myself a chance to get to the point where I can use the techniques (rather than just understand them when I see them used).

I worked my way through the three steps of the first frame hacks recipe without a problem, though (to be honest), I've never used the string.Template class and the re.finditer function before (though they did seem familiar, so I guess I've read about them). I don't think I've used the yield statement either, though I'm eager to now that I've been immersed in generators and co-routines in the "Generator Tricks for Systems Programmers" tutorial (which was also excellent and which I'll write about later).

But I noticed that the code in the third step (step 2) of that recipe actually works without using sys._getframe():

SPOILER WARNING

import sys, re
from string import Template

def getchunks(s):
   matches = list(re.finditer(r"\$\{(.*?)\}", s))

   if matches:
       pos = 0
       for match in matches:
           yield s[pos : match.start()]
           yield [match.group(1)]
           pos = match.end()
       yield s[pos:]

def interpolate(templateStr):
   result = ''
   for chunk in getchunks(templateStr):
       if isinstance(chunk, list):
           result = ''.join((result, str(eval(chunk[0]))))
       else:
           result = ''.join((result, chunk))
   return result

name = 'Guido van Rossum'
places = 'Amsterdam', 'LA', 'New York', 'DC', 'Chicago',

s = """My name is ${'Mr. ' + name + ', Esquire'}.

I have visited the following cities:  ${', '.join(places)}.
"""

print interpolate(s)

That's because the eval in the interpolate function can find "name" and "places" in the global namespace. I can't recall if Leihong or Kumar noticed this in the tutorial, but I imagine they intended something like this:

import sys, re
from string import Template

def getchunks(s):
 matches = list(re.finditer(r"\$\{(.*?)\}", s))

 if matches:
     pos = 0
     for match in matches:
         yield s[pos : match.start()]
         yield [match.group(1)]
         pos = match.end()
     yield s[pos:]

def interpolate(templateStr):
 """Implement this function"""

def main():
 name = 'Guido van Rossum'
 places = 'Amsterdam', 'LA', 'New York', 'DC', 'Chicago',

 s = """My name is ${'Mr. ' + name + ', Esquire'}.

    I have visited the following cities:  ${', '.join(places)}.
    """
 print interpolate(s)

if __name__ == '__main__':
 main()

You'll need to use sys._getframe() in interpolate() to get this to work.

TODO

I should look into Trellis, Elixer and lxml, which were used in the tutorial to give use cases for frame hacks.

And before I move on to the next section of the "Secrets of the Framework Creators" tutorial, I should describe the string.Template class and the re.finditer function. Stay tuned.

Secrets of the "Secrets of the Framework Creators"

As I wrote in my personal blog, I'm at PyCon. There has already been plenty of good fodder for a handful of blog posts. But I'll start at the beginning and work my way through the conference chronologically.

Thursday was tutorial day. The first was "Secrets of the Framework Creators", presented by Feihong Hsu and Kumar McMillan. I've taken quite a few conference tutorials in the past (though this was my first at PyCon), but this must be the best-prepared. Leihong and Kumar put together a web-site and had the foresight to make it available for download instead of hosting it (which was good thinking since the conference Wi-Fi worked only periodically until yesterday). You can download it yourself from the Google Group they set up. (I asked their permission to blog about this. They also said they'd accept new members of the group that didn't take the tutorial.)

As you can see, they start (in index.html) by introducing themselves, and the tutorial. Then there are links to the four sections covered: Frame Hacks, Decorators, Metaclasses and Magic Methods. And finally on the main page is an introduction to PDB, should we need it. I didn't, since I have had to teach myself PDB and use it once or twice--because I was sloppy and didn't write unit tests. And I had already seen the breakpoint trick in one of -JJ's posts. But I didn't realize that when in PDB, you're in a "custom" Python interactive session. It's nice to know that anything I'm used to doing there should work in PDB.

In each of the four sections is a getting started section that introduces the concept, some (much appreciated) use cases that give examples of where it's been used, and a couple recipes to give you practice using each technique. They even think to list the prerequisite knowledge required for each recipe (with a link to documentation). Each recipe has hidden hints to help us along, and the solution if you're completely stuck. And they conclude each section with "caveats": warnings about the trade-offs and limitations of the technique.

It must have taken them quite a bit of time to put this together. (It's taking me a while just to describe it.)

They had time in the tutorial to go through one recipe in each section, and then we voted and used the time remaining to go through the second recipe in (if I recall correctly) Metaclasses and Frame Hacks.

I'm now working my way through the material again, so it (hopefully) sinks in. My next couple posts will be on what I learned in the frame hacks section.

I'm going to call it PyPap

I didn't intend for three months to go by before I finished my first "real" post to my new blog. And this isn't going to be a "real" post either. I just can't allow any more time to go by, and have any hope of this being a "real" blog.

The problem is I set my sights too high. I just don't have enough Python experience to write authoritative posts on Python Patterns and Practices. I aspire to get there someday, and perhaps this blog will help me get there faster. But in the meantime, I'm going to rename the blog from its URL: "PyPap". That feels much less pretentious to me, and should reduce my own expectations for future posts.

So for any of you out there that stumble upon this, hopefully what follows are semi-regular posts on bits of Python that are new to me. They may not be patterns, or practices, but hopefully some of them prove to be useful to you too.