String Interpolation

One of the prerequisites for recipe #1 ("Ruby-style string interpolation") from the Frame Hacks section of the "Secrets of the Framework Creators" PyCon tutorial is the string.Template class. Before I took the tutorial I was vaguely aware of the string.Template class, but I don't think I've ever used it.

I'm going to consider using it though. There are probably plenty of cases where it would be more readable than using the % operator--I can recall a couple times in code I wrote very recently.

In step 1 of Frame Hacks recipe #1, Feihong and Kumar how how one can write a function that wraps string.Template and the substitute method so the mapping argument (and optional keyword arguments) don't need to be provided.

SPOILER WARNING

def interpolate(templateStr):
  import sys
  from string import Template
  t = Template(templateStr)
  frame = sys._getframe(1)
  return t.substitute(**frame.f_locals)

name = 'Feihong'
place = 'Chicago'
print interpolate("My name is ${name}. I work in ${place}.")

Here's what it could look like without the sys._getframe() call:

def interpolate(templateStr, **kws):
    from string import Template
    return Template(templateStr).substitute(kws)

name = 'Feihong'
place = 'Chicago'
print interpolate("My name is ${name}. I work in ${place}.",
                  name=name, place=place)

I guess I could argue that the latter is more explicit, but perhaps the former is explicit enough. I guess I'll be the judge of that (for myself) the next time I write (or am ready to refactor) an ugly string using the % operator. I'll let you know if I decide to add the interpolate function to my utilities module.

The solution for step 2 of recipe #1 (which I showed a partial solution for when I wrote about Frame Hacks) allows for expressions in the template strings. I don't think I've run across a use case for that, but I'll write about it if I do.

I should also mention that Python 3.0 will include a new (third) method of string interpolation, the format method to be added to the built-in string class. (There's more to it than just that. You can read all about it in PEP 3101. And if--like me--you don't know just what a PEP is, I recommend reading the "PEPs" section hear the top of the description of Python's Development Process. And you'll find plenty more detail in PEP 1.)

The above example would look like this using the new format method:

name = 'Feihong'
place = 'Chicago'
print "My name is {name}. I work in {place}.".format(name=name, place=place)

There doesn't seem to be a spec for the format method yet, so it's not clear whether one could wrap it and use sys._getframe(). I don't know if that's an argument for avoiding using it now though. (I don't see any reason why the interpolate() function above wouldn't work in Python 3.0.)

TODO

I haven't installed the latest Python 3.0 alpha yet. I should make time to do so and play with the above.

Pygments

I thought I'd write a quick post on how I highlighted the Python code in my previous post. I used Pygments, which (though I've only had limited experience with it) appears to truly be a "top quality product". It was easy to install with easy_install.

Because I'm pasting the highlighted code into Blogger, I can't use the CSS it generates by default, so I use the "noclasses" option:

$ pygmentize -f html -O noclasses -o file.html file.py

And then I can paste the contents of the html file into my Blogger post's HTML.

Actually, on Mac OS X I can save a step using:

$ pygmentize -f html -O noclasses file.py | pbcopy

And then paste directly into Blogger.

Update: the following command-line allows me to pygmentize the Python code in the clipboard (without having to create a file for it):

$ pbpaste | pygmentize -f html -O noclasses -l python | pbcopy

Another Update: it occurred to me I could use the above (updated) command-line in TextExpander (see A TextExpander snippet to paste quoted text and A tr.im TextExpander snippet that works for me) to create a snippet that pastes a "Pygmentized" version of the Python code in the clipboard:

Frame Hacks

Someone (I forget who--I've listened to so many smart people here at PyCon) said yesterday that Python treats us like adults--if we want go "off-road" we can. (Other languages try to protect programmers from themselves and others. With Python we need to write tests to get that protection. Not that statically-typed languages with protected attributes and methods make testing unnecessary. But that's a subject for another time.) But until I took the "Secrets of the Framework Creators" tutorial at PyCon, I had no idea just how far into the wilderness one could go.

As Leihong and Kumar explain in the getting started chapter of the frame hacks section, one can call sys._getframe() to get the "frame" of the calling functions, all the way to the bottom of the call stack. This made immediate sense to me, since one of the first bits of real code I wrote when I started at Altera did a stack trace on the Nios (Classic) processor. (That was tricky, since the original Nios used register windows--but I'd better stop before I go down a rat hole.)

As I wrote previously, I'm working my way through the tutorial again on my own to give myself a chance to get to the point where I can use the techniques (rather than just understand them when I see them used).

I worked my way through the three steps of the first frame hacks recipe without a problem, though (to be honest), I've never used the string.Template class and the re.finditer function before (though they did seem familiar, so I guess I've read about them). I don't think I've used the yield statement either, though I'm eager to now that I've been immersed in generators and co-routines in the "Generator Tricks for Systems Programmers" tutorial (which was also excellent and which I'll write about later).

But I noticed that the code in the third step (step 2) of that recipe actually works without using sys._getframe():

SPOILER WARNING

import sys, re
from string import Template

def getchunks(s):
   matches = list(re.finditer(r"\$\{(.*?)\}", s))

   if matches:
       pos = 0
       for match in matches:
           yield s[pos : match.start()]
           yield [match.group(1)]
           pos = match.end()
       yield s[pos:]

def interpolate(templateStr):
   result = ''
   for chunk in getchunks(templateStr):
       if isinstance(chunk, list):
           result = ''.join((result, str(eval(chunk[0]))))
       else:
           result = ''.join((result, chunk))
   return result

name = 'Guido van Rossum'
places = 'Amsterdam', 'LA', 'New York', 'DC', 'Chicago',

s = """My name is ${'Mr. ' + name + ', Esquire'}.

I have visited the following cities:  ${', '.join(places)}.
"""

print interpolate(s)

That's because the eval in the interpolate function can find "name" and "places" in the global namespace. I can't recall if Leihong or Kumar noticed this in the tutorial, but I imagine they intended something like this:

import sys, re
from string import Template

def getchunks(s):
 matches = list(re.finditer(r"\$\{(.*?)\}", s))

 if matches:
     pos = 0
     for match in matches:
         yield s[pos : match.start()]
         yield [match.group(1)]
         pos = match.end()
     yield s[pos:]

def interpolate(templateStr):
 """Implement this function"""

def main():
 name = 'Guido van Rossum'
 places = 'Amsterdam', 'LA', 'New York', 'DC', 'Chicago',

 s = """My name is ${'Mr. ' + name + ', Esquire'}.

    I have visited the following cities:  ${', '.join(places)}.
    """
 print interpolate(s)

if __name__ == '__main__':
 main()

You'll need to use sys._getframe() in interpolate() to get this to work.

TODO

I should look into Trellis, Elixer and lxml, which were used in the tutorial to give use cases for frame hacks.

And before I move on to the next section of the "Secrets of the Framework Creators" tutorial, I should describe the string.Template class and the re.finditer function. Stay tuned.

Secrets of the "Secrets of the Framework Creators"

As I wrote in my personal blog, I'm at PyCon. There has already been plenty of good fodder for a handful of blog posts. But I'll start at the beginning and work my way through the conference chronologically.

Thursday was tutorial day. The first was "Secrets of the Framework Creators", presented by Feihong Hsu and Kumar McMillan. I've taken quite a few conference tutorials in the past (though this was my first at PyCon), but this must be the best-prepared. Leihong and Kumar put together a web-site and had the foresight to make it available for download instead of hosting it (which was good thinking since the conference Wi-Fi worked only periodically until yesterday). You can download it yourself from the Google Group they set up. (I asked their permission to blog about this. They also said they'd accept new members of the group that didn't take the tutorial.)

As you can see, they start (in index.html) by introducing themselves, and the tutorial. Then there are links to the four sections covered: Frame Hacks, Decorators, Metaclasses and Magic Methods. And finally on the main page is an introduction to PDB, should we need it. I didn't, since I have had to teach myself PDB and use it once or twice--because I was sloppy and didn't write unit tests. And I had already seen the breakpoint trick in one of -JJ's posts. But I didn't realize that when in PDB, you're in a "custom" Python interactive session. It's nice to know that anything I'm used to doing there should work in PDB.

In each of the four sections is a getting started section that introduces the concept, some (much appreciated) use cases that give examples of where it's been used, and a couple recipes to give you practice using each technique. They even think to list the prerequisite knowledge required for each recipe (with a link to documentation). Each recipe has hidden hints to help us along, and the solution if you're completely stuck. And they conclude each section with "caveats": warnings about the trade-offs and limitations of the technique.

It must have taken them quite a bit of time to put this together. (It's taking me a while just to describe it.)

They had time in the tutorial to go through one recipe in each section, and then we voted and used the time remaining to go through the second recipe in (if I recall correctly) Metaclasses and Frame Hacks.

I'm now working my way through the material again, so it (hopefully) sinks in. My next couple posts will be on what I learned in the frame hacks section.

I'm going to call it PyPap

I didn't intend for three months to go by before I finished my first "real" post to my new blog. And this isn't going to be a "real" post either. I just can't allow any more time to go by, and have any hope of this being a "real" blog.

The problem is I set my sights too high. I just don't have enough Python experience to write authoritative posts on Python Patterns and Practices. I aspire to get there someday, and perhaps this blog will help me get there faster. But in the meantime, I'm going to rename the blog from its URL: "PyPap". That feels much less pretentious to me, and should reduce my own expectations for future posts.

So for any of you out there that stumble upon this, hopefully what follows are semi-regular posts on bits of Python that are new to me. They may not be patterns, or practices, but hopefully some of them prove to be useful to you too.