Posts tagged python - Entropy and Ecstasy

Eventlet is Awesome!

Tue, 01 Feb 2011 21:22:54 GMT

Eventlet is an asynchronous networking library for Python which uses coroutines to allow you to write non-blocking code without needing to perform the normal mental gymnastics that usually go along with asynchronous programming. There are a bunch of async/event-driven networking framworks (eventmachine, node.js, tornado, and twisted to name a few), and their defining characteristic is that they use select/epolll/kqueue/etc. to do I/O asynchronously. Asynchronous I/O is cool because when done correctly it allows your server to handle a much greater number of concurrent clients than the one-OS-thread-per-connection approach. Furthermore, since you're using co-operative rather than preemptive multitasking, you can safely update shared data structures without locks, which makes it a lot easier to write correct code.

Anyway, eventlet is pretty cool. If you like Python and you're interested in async programming, you should check it out. After all, anything that reduces the incidence of Heisenbugs is worth a look ;)

Proxymachine in 55 Lines

Since they're so specialized, playing with this kind of library requires a specific kind of project. Consequently, I decided to put together a "Hello Word!" version of Proxymachine. Proxymachine is a proxy built on eventmachine that lets you configure it's routing logic using Ruby. Don't get me wrong, Proxymachine is awesome and way more production ready than this. That being said, it's still friggin' cool that I could put togther a pale imitation of Proxymachine in less than 100 lines thanks to eventlet:

import functools
import eventlet

CHUNK_SIZE = 32384

class Router(object):
    def route(self, addr, data):
        raise NotImplemented

def merge(*dicts):
    result = {}
    for d in dicts:
        result.update(d)
    return result

def forward(source, dest):
    while True:
        d = source.recv(CHUNK_SIZE)
        if d == '':
            break
        dest.sendall(d)

def route(router, client, addr):
    blocks = []
    while True:
        block = client.recv(CHUNK_SIZE)
        if block == '':
            raise Exception('Failed to route request: "{0}"'.format("".join(blocks)))

        blocks.append(block)

        route = router.route(addr, "".join(blocks))
        if route is not None:
            print "Forwarded connection from {0} to {1}".format(addr, route)

            server = eventlet.connect(route)
            for block in blocks:
                server.sendall(block)

            eventlet.spawn_n(forward, server, client)
            forward(client, server)
            return


def start(router, **kwargs):
    defaults = {
        "listen": ('localhost', 8080),
    }

    config = merge(defaults, kwargs)
    print "Listening on:", config['listen']

    eventlet.serve(
        eventlet.listen(config['listen']),
            functools.partial(route, router())
    )

Roulette: A router in ten lines

The logic here is laughable, route inbound connections to either localhost:9998 or localhost:9999 depending on whether the remote port is divisible by two, but the point is that the routing logic could be anything. We're writing Python here. We could look stuff up in a database, or check the phase of the moon or y'know, do something useful.

import roulette

class Router(roulette.Router):
    def route(self, addr, data):
        if addr[1] % 2 == 0:
            return ("localhost", 9999)
        return ("localhost", 9998)

roulette.start(Router,
    listen = ("localhost", 80)
)

Grayvatar 2.0

Sat, 11 Dec 2010 19:20:52 GMT

In my post about Grayvatar, I warned everybody away from actually using it for anything as it was a complete toy. The major reason I cited was that it did not make any attempt to respect cache headers. Enter Grayvatar 2.0, which is slightly less of a toy as it, at least, makes a half-hearted attempt to respect cache headers.

The key addition was httplib2, which maintains it's own cache, so that we're not hitting Gravatar's servers for every single request we receive. The other big change was to copy cache headers from the client to the server and vice-versa. This means that we'll pass a long the cache-control: max-age=300 that we get from Gravatar or the cache-control: max-age=0 we get from the client. All of this adds about 20 lines on top of the existing implementation. You still shouldn't use it for anything, it's still a toy, but at least it's slightly less naive one.

import StringIO
import tempfile
import urllib, urllib2

import cherrypy
import httplib2
import Image

from paver.easy import path

URL = "http://www.gravatar.com/avatar/%s?%s"
cache_headers = ("cache-control",)


class Grayvatar(object):
    def __init__(self):
        tmp_dir = tempfile.mkdtemp(".cache")
        self.http = http = httplib2.Http(tmp_dir)

    @cherrypy.expose
    def default(self, *bits, **kwargs):
        if bits:
            cherrypy.response.headers["Content-Type"] = "image/png"
            url = URL % (bits[0], urllib.urlencode(kwargs))
            headers, img = self.get_img(url)

            for header in cache_headers:
                if header in headers:
                    cherrypy.response.headers[header] = headers[header]

            return img_as_str(img.convert("L"))

        pwd = path(__file__).abspath().parent
        return (pwd / "how_it_works.html").text()

    def get_img(self, url):
        headers, body = self.http.request(url,
            headers = project(cherrypy.request.headers, cache_headers)
        )
        return headers, Image.open(StringIO.StringIO(body))


def project(d, keys):
    return dict((k, d[k]) for k in keys if k in d)

def img_as_str(img):
    sock = StringIO.StringIO()
    img.save(sock, "png")
    return sock.getvalue()

if __name__ == "__main__":
    cherrypy.quickstart(Grayvatar())

Grayvatar

Mon, 29 Nov 2010 22:35:40 GMT

The week before last I got to spend some time working on a prototype as part of a sekrit project at work. In order to give the prototype the air of not-quite-done, I did the whole thing in black and white. This worked great up until I added in Identicons to identify users, which added a bunch of brightly colored triangles and ruined the whole black-and-white aesthetic. I eventually ended up pulling a bunch of Identicons from Gravatar with wget and desaturated them with ImageMagik. It worked, but it was kind of gross.

Teh Codez

It occurred to me that it should be pretty straightforward to build a web app that implemented the Gravatar API and desaturate real Gravatars, so I took a crack at it:

import StringIO
import urllib, urllib2

import cherrypy
import Image

from paver.easy import path

URL = "http://www.gravatar.com/avatar/%s?%s"

class Grayvatar(object):
    @cherrypy.expose
    def default(self, *bits, **kwargs):
        if bits:
            cherrypy.response.headers["Content-Type"] = "image/png"
            url = URL % (bits[0], urllib.urlencode(kwargs))
            img = get_img(url).convert("L")
            return img_as_str(img)

        pwd = path(__file__).abspath().parent
        return (pwd / "how_it_works.html").text()

def get_img(url):
    sock = urllib2.urlopen(url)
    return Image.open(StringIO.StringIO(sock.read()))

def img_as_str(img):
    sock = StringIO.StringIO()
    img.save(sock, "png")
    return sock.getvalue()

if __name__ == "__main__":
    cherrypy.quickstart(Grayvatar())

I think it's awesome that, thanks to CherrPy and PIL, I could put together a web app that does all that 30 odd lines (including imports), but it's less awesome that so many of those lines are shuffling data in and out of a StringIO.

mod_wsgi

I set up an instance at: http://aaron.maenpaa.ca/grayvatar using Apache and mod_wsgi. I used a lightly adapted version of the the wsgi config from the CherryPy wiki.

import sys
sys.stdout = sys.stderr

import atexit
import threading
import cherrypy

sys.path = ['/home/aaron/code/grayvatar'] + sys.path

import grayvatar

cherrypy.config.update({
    'environment': 'embedded',
    'log.screen': True,
})

if cherrypy.__version__.startswith('3.0') and cherrypy.engine.state == 0:
    cherrypy.engine.start(blocking=False)
    atexit.register(cherrypy.engine.stop)

application = cherrypy.Application(grayvatar.Grayvatar(), script_name=None)

The most important change was adding 'log.screen': True to the config. This sends errors to stderr, which eventually makes it's way to /var/log/apache2/error.log which makes it possible to figure out what's going on when things don't go right.

How It Works

Just like Gravatar. The base URL is http://aaron.maenpaa.ca/grayvatar/. You calculate the MD5 hash of your email address, append it to the base URL and then add any query parameters that Gravatar supports. That gets you a URL like this: http://aaron.maenpaa.ca/grayvatar/634329486bd326c395c8b3539c035139?s=256&d=identicon&r=PG

It fetchs the appropriate Gravatar, desaturates and returns it. Keep in mind, this is a toy: It doesn't actually respect the cache headers Gravatar sends and worse, it doesn't even pass them along to the client.

Krypto and 24

Sat, 08 May 2010 19:28:34 GMT

Daily WTF inspired me to implement a Krypto and 24 solver:

import functools
import itertools
import operator

class Node:
        def evaluate(self): raise NotImplemted()
        def print(self): raise NotImplemted()

        def __add__(self, other): return Add(self, other)
        def __sub__(self, other): return Sub(self, other)
        def __mul__(self, other): return Mul(self, other)
        def __truediv__(self, other): return Div(self, other)

        def __str__(self):
                return "<Node: {}>".format(self.print())

        def __repr__(self):
                return str(self)

class Binary(Node):
        def __init__(self, left, right):
                self.left = left
                self.right = right

        def evaluate(self):
                try:
                        return self.op(
                                self.left.evaluate(),
                                self.right.evaluate(),
                        )
                except ZeroDivisionError:
                        return float("NaN")

        def print(self):
                return "({} {} {})".format(
                                self.left.print(),
                                self.symbol(),
                                self.right.print()
                )

        def op(self, left, rigth): raise NotImplemted()
        def symbol(self): raise NotImplemted()

class Unary(Node):
        def __init__(self, op, val):
                self.op = op
                self.val = val

        def evaluate(self):
                return self.op(self.val)

        def print(self):
                return str(self.evaluate())

def BinOp(op, symbol, commutative = True):
        class inner(Binary):
                def op(self, left, right):
                        return op(left, right)

                def symbol(self):
                        return symbol

                @classmethod
                def iscommutative(cls):
                        return commutative

        return inner

Add = BinOp(operator.add, "+")
Sub = BinOp(operator.sub, "-", commutative = False)
Mul = BinOp(operator.mul, "*")
Div = BinOp(operator.truediv, "/", commutative = False)
Lit = functools.partial(Unary, lambda x: x)

OPS = [Add, Sub, Mul, Div]

def generate_expressions(nodes):
        if len(nodes) == 1:
                yield first(nodes)

        for x, y, op in itertools.product(nodes, nodes, OPS):
                isnew = not op.iscommutative() or id(x) > id(y)
                if x is not y and isnew:
                        yield generate_expressions((nodes - set([x, y])) | set([op(x, y)]))

def flatten(xs):
        for x in xs:
                try:
                        for y in flatten(x):
                                yield y
                except TypeError:
                        yield x

def first(xs):
        for x in xs:
                return x

def krypto(nums, solution):
        def issolution(expr):
                return expr.evaluate() == solution

        nodes = set(map(Lit, nums))
        expr = first(filter(issolution, flatten(generate_expressions(nodes))))
        print(expr.print(), "=", expr.evaluate())

if __name__ == "__main__":
        import sys
        args = list(map(int, sys.argv[1:]))
        krypto(args[:-1], args[-1])

My favourite part is that the core of the algorithm is 6 lines:

def generate_expressions(nodes):
        if len(nodes) == 1:
                yield first(nodes)

        for x, y, op in itertools.product(nodes, nodes, OPS):
                isnew = not op.iscommutative() or id(x) > id(y)
                if x is not y and isnew:
                        yield generate_expressions((nodes - set([x, y])) | set([op(x, y)]))

The parametric classes are also a reminder of just how badass python is:

def BinOp(op, symbol, commutative = True):
        class inner(Binary):
                def op(self, left, right):
                        return op(left, right)

                def symbol(self):
                        return symbol

                @classmethod
                def iscommutative(cls):
                        return commutative

        return inner

Edit

2010-05-16: Apparently Python comes with an implementation of cross product.

Ford Circles

Sun, 20 Sep 2009 14:23:16 GMT

I picked up a copy of The Math Book after reading the review on John D. Cook's blog.

So far it has been pretty interesting and I've was plesently surprised with how much of the material I am familar with, despite the fact that I haven't studied math beyond the courses that are part of the engineering curriculum at my university.

The best part, though, has been the occasions when I find something that I can play with a little bit on my own. A good example is Ford Circles, which with a little time in NodeBox I had my own renderings:

Here's the code:

from __future__ import division

scale(1000, 1000)

def circle(x, y, r):
    oval(x * 1000, y * 1000, 2 * r, 2 * r)

def ford(h, k):
    r = 1 / (2 * k**2)
    x, y = h/k, r
    print x, y, r
    circle(x, y, r)


for h in range(0, 200):
    for k in range(1, 200):
        ford(h, k)

... and here's a bigger rendering, if you're interested.

Learning Python

Tue, 27 Jan 2009 21:40:47 GMT

Here's a collection of free resources for learning python and programming.

Python for Software Development

Python for Software Development which was initially a port of How to Think Like a Computer Scientist from Java to Python. It is forthcoming from Cambridge University Press and will be published in the first quarter of 2009. The manuscript is available for free as a PDF under the title Think Python.

This book introduces computer science from first pricipals and introduces:

Variables

Functions

Recursion

Fundamental Data structures (Lists and Maps)

The Python Tutorial

Python has an excellent official tutorial. It covers:

The Python Interpreter

Python's Flow Control Statements

Python's Built-in Data Structures (Lists, Sets, Dictionaries)

Python's Modules

I/O

Errors, Exceptions, Classes, etc.

Dive Into Python

Dive Into Python is an excellent book for learning Python provided you already know how to program. It is published by Apress, but is also freely available online. It covers more advanced topics such as:

Introspection
Regular Expressions
HTML and XML Processing
HTTP and SOAP Web Services
Unit Testing and Test First Programming
Refactoring
Functional Programming

What the Date?

Mon, 15 Dec 2008 14:30:00 GMT

Today's Code Snippet of the Day (CSOD) from The Daily WTF shows how not to validate a date. Inspired by boredom and the knowlege that I could do it shorter and better. I set about writing my own date parsing/validation routines as a form of Code Kata.

In Python, Take I

A first crack written in python:

date_pattern = re.compile(r'^(?P<day>\d\d)/(?P<month>\d\d)/(?P<year>\d\d\d\d)$')

def parse_date(input):
    if not date_pattern.match(input):
        raise ValueError("'%s' is not in DD/MM/YYYY format" % input)

    day, month, year = map(int, input.split('/'))
    d = datetime.date(year, month, day)
    if d > datetime.date.today():
        raise ValueError("'%s' is in the future" % input)

    return d

We validate the date against a regex so that we know what we're dealing with.
We split up the input string and construct the date.
We test that the date is not in the future, and return the result.

This implementation is better than the CSOD in a number of ways:

It uses a regex to validate the format of the input string with is so much more faster/expressive/productive that writing our own validation code.
We use the platform's built in Date object rather than storing and manipulating the year/month/day ourselves which helps to avoid all kinds of silly bugs.

Unfortunately, we still parse and construct the date ourselves, duplicating functionality present in the standard library.

In Python, Take II

A second attempt, this time we're going to rely on strptime rather than parsing the string ourselves:

def parse_date(input):
    d = datetime.datetime.strptime(input, "%d/%m/%Y").date()
    if d > datetime.date.today():
        raise ValueError("'%s' is in the future" % input)
    return d

This implementation is even better as it relyies on strptime to handle the parsing/validating and the only real code that we write is testing if the date is in the future which is our logic.

In JavaScript

An implementation in JavaScript because the CSOD was submitted in JS. This is essentially a transcription of the first Python implementation as none of the JS date parsing utilities seem to take a formate string:

date_pattern = new RegExp('^\\d\\d/\\d\\d/\\d\\d\\d\\d$');

function parse_date(input){
    if(!date_pattern.test(input)){
        alert("'" + input + "' does not conform to the dd/mm/yyyy format");
    }

    ordinals = input.split('/');
    d = new Date(ordinals[2], ordinals[1] - 1, ordinals[0]);
    if(d > new Date()){
        alert("'" + input + "' is in the future");
    }

    return d;
}

It would be preferable to try and achieve the simplicity of the second Python implementation but that would require writing (or including third party code) comparable to strptime.

Aside

It is very, very, strange that months are 0 indexed while day and year are not in the Date constructor:

js> new Date('12/02/2008')
Tue Dec 02 2008 00:00:00 GMT-0500 (EST)
js> new Date(2008, 12, 02)
Fri Jan 02 2009 00:00:00 GMT-0500 (EST)
js> new Date(2008, 11, 02)
Tue Dec 02 2008 00:00:00 GMT-0500 (EST)

Seriously, what's up with that?