Reflection and Introspection Over Modules and Packages In Python

Someone Twitter me if I’m missing something, but I couldn’t find a core way of doing reflection over packages in Python.
In this particular case, I wanted a way to load all the modules in a certain package (a directory with an __init__.py) and automatically add them into a running Twisted service.

To get this working, I created a small module called reflection:

import os
import sys
import re

'turns a lower case string with underscores into its camel case equivalent'
def camelize(string):
    return re.sub(r"(?:^|_)(.)", lambda x: x.group(0)[-1].upper(), string)

'returns a list of modules objects in the package identified by package_name'
def dir_modules(package_name):
    modules = []
    load_package(modules.append, package_name)
    return modules

'load the specified class from module, or return the default class derived from module.__name__ if no class_name is specified'
def load_class(module, class_name = None):
    return getattr(__import__(module) if module.__class__ == str else module, class_name or camelize(module.__name__.split('.')[-1]))

def load_package(function, package_name):
    os.path.walk(package_name, load_modules, function)

def load_modules(function, package_name, module_names):
    for module_name in module_names:
        if re.match(r"^(?!__)\w*\.py$", module_name):
            qualified_module_name = '%s.%s' % (package_name, module_name[0:-3])
            __import__(qualified_module_name)
            function(sys.modules[qualified_module_name])

My Twisted application then uses this code in its initializer:

xmlrpc.XMLRPC.__init__(self)
xmlrpc.addIntrospection(self)
for module in reflection.dir_modules('services'):
    klass = reflection.load_class(module)
    if issubclass(klass, xmlrpc.XMLRPC):
        print 'adding xmlrpc sub handler from %s' % module.__name__
        self.putSubHandler(module.__name__.split('.')[-1], klass())
    elif issubclass(klass, internet.TimerService):
        print 'initalizing timer service from %s' % module.__name__
        instance = klass(Application.Config[module.__name__.split('.')[-1]]['interval'])
        instance.setServiceParent(Service.Application)
        instance.startService()

Now you can drop new modules into the services directory and, provided they are named correctly, they will be auto-loaded into the application on restart.

Killing Python: Exiting Without Using SystemExit

Usually in python you exit a script programmatically by raising SystemExit or calling sys.exit. Both methods send an exception hurtling up the call stack, allowing every level of your program to execute finally statements and exit cleanly.

This behaviour changes if you throw SystemExit in a multithreaded application: it kills the calling thread instead. If the calling thread is not the main thread the application will just keep ticking along. Fine in most cases, but sometimes you just need to make the application quit.

To achieve this, you need to call:

os._exit()

with an error code of your choice.

Creating Static Methods In Python

Static methods can be a little confusing if you come to Python from other languages in which they are first class citizens. To create a static method you need to pass an existing method through staticmethod():

class Person:
  people = {}
  def __init__(self, name):
    self.name = name
    Person.people[self.name] = self

  def find_by_name(name):
    return Person.people.get(name)

  find_by_name = staticmethod(find_by_name)

This will do all the hokey Python magic to (I assume) bind the method correctly into the Class’s lookup __dict__.

Thankfully, newer versions of Python — including 2.5.1, the version which ships with Leopard — allow you to use the slightly more aesthetically pleasing decorator shortcut:

class Person:
  people = {}
  def __init__(self, name):
    self.name = name
    Person.people[self.name] = self

  @staticmethod
  def find_by_name(name):
    return Person.people.get(name)

Rails 2.2 Templates

TemplateHandlers have been significantly overhauled in Rails 2.2, and these changes are not backwards-compatible with Rails 2.1.

Instead of being responsible for rendering a template, TemplateHandlers should now provide a string of Ruby that will be eval’ed by ActionView further along the rendering chain.

So, where previously a TemplateHandler might have declared a render method:

class DotHandler < ActionView::TemplateHandlers::ERB

  def render(template, local_assigns = {})
    @view.controller.headers["Content-Type"] ||= 'image/png'

    input = super(template)

    output = IO.popen('dot -Tpng', 'r+') do |io|
      io.write(input)
      io.close_write
      io.read
    end

    output
  end
end

ActionView::Template::register_template_handler 'dot', DotHandler
ActionController::Base.exempt_from_layout :dot

Templates should now provide a compile method that will return ruby code for execution later:

class DotHandler < ActionView::TemplateHandler

  def compile(path)
    <<-EOS
      controller.response.content_type ||= Mime::PNG
      #{ActionView::Template.handler_class_for_extension('erb').call(path)}
      @output_buffer = IO.popen("dot -Tpng", 'r+') do |io|
      io.write(@output_buffer)
      io.close_write
      io.read
      end
    EOS
  end

end

ActionView::Template.register_template_handler 'dot', DotHandler
ActionView::Template.exempt_from_layout 'dot'

To chain template handlers, find the appropriate handler with ActionView::Template#handler_class_for_extension and then call it. The code it returns can then be injected into your compiled template.

Setting The Default Java Virtual Machine On Ubuntu

Select the default Ubuntu JVM with update-alternatives:

sudo update-alternatives --config java

Monit And My Sql

Usually Monit will be happy to watch MySQL with the following configuration:

check process mysqld with pidfile "/var/run/mysqld/mysqld.pid"
  group database
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"
    if cpu > 60% for 2 cycles then alert
    if cpu > 80% for 10 cycles then restart
    if failed port 3306 protocol mysql then restart

If you are getting connection refused errors, your MySQL database may not be bound to localhost. Check your bind address in /etc/mysql/my.cnf:

bind-address = 0.0.0.0

If the address MySQL is bound to is external, you will need to adjust your monitrc accordingly:

if failed host 0.0.0.0 port 3306 protocol mysql then restart

Dynamically Generated Classes

To create a class with a dynamic name in Ruby, create a new Class object and then set it as a constant within Object:

Object.const_set('ChunkyBacon', Class.new)
=> ChunkyBacon
>> ChunkyBacon.new
=> #<ChunkyBacon:0x5ef740>

You can use this to good effect with ActiveResource, creating arbitrary restful resources as you go:

Object.const_set(model, Class.new(ActiveResource::Base))
@class = model.constantize
@class.site = site
@class.new.save

Absolute Paths From Relative Paths In Bash

There doesn’t seem to be a nice, cross-platform way of deriving the absolute path from a relative one in Bash.
(`readline -f .` works in Linux, but doesn’t seem to work in OS X).

Here is a small program written in Python which does exactly that:

#!/usr/bin/env python
import os, sys
if len(sys.argv) < 2:
    print 'usage: %s <PATHS>' % sys.argv[0]
    raise SystemExit(1)
directory = os.path.dirname(sys.argv[1])
if len(sys.argv) > 2:
    directory = os.path.join(directory, *sys.argv[2:])
print os.path.abspath(directory)

Paste it into a file and chmod a+x it and you have yourself a handy utility:

~/Projects $ abspath .
/Users/simon/Projects

~/Projects $ abspath ../
/Users/simon

I use it in a script called server that starts various web-services. By starting the script with:

#!/bin/sh
directory=`abspath $0 ..`
cd $directory

...

you can now run it from anywhere on the file system. The script will cd itself into the web service’s working directory before kicking things off.

Custom 404 Pages With Passenger

On failing to match a route, Rails 2.1.2 appears to rescue the ActionController::RoutingError with its stock template (/action_controller/templates/rescues/layout.erb) and then returns an upstream 404 error code — even if you’ve defined a custom rescue_from in ActionController:

rescue_from ActionController::RoutingError, :with => :render_404

With your Mongrel/Thin and Apache/Nginx combo this is fine; you can intercept the error code and redirect your visitor to the appropriate custom error page. With Passenger (mod_rails) however, there is no such intermediary step. 404 error messages that do not match a controller will just generate a blank page.

To address this problem, the best solution I have found is to add a final catch-all route at the bottom of your routes.rb file that invokes your rescue_from method directly:

map.connect '*path', :controller => 'home', :action => 'render_404'

Custom Error Pages With Nginx and Thin

If you are using Nginx to proxy to another web server such as Thin or Mongrel, chances are you have had trouble getting your custom error pages to return the correct status codes.

If your application redirects to a cached error page Nginx will serve it as a normal file and will (correctly) tell the browser that everything was successful. Not quite the intended behaviour.

To fix this, add locations for your error pages and set them to be internal. Now the locations can only be reached when somethig goes wrong (as intended) and Nginx will never mistakenly serve them as valid files.

upstream thins {
  server 127.0.0.1:3000;
  server 127.0.0.1:3001;
}

access_log /var/www/example/log/access.log;
error_log /var/www/example/log/error.log;

root   /var/www/example/public/;

server {
  listen 80;
  server_name www.example.com;

  proxy_set_header  X-Real-IP  $remote_addr;
  proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_redirect off;

  error_page 404 /404.html;
  error_page 502 503 /503.html;

  location ~* ^/(404|502|503).html$ {
    if (!-f $request_filename) {
       proxy_pass http://thins; break;
    }
    internal; break;
  }
  location / {
    if (!-f $request_filename) {
      proxy_pass http://thins;
      break;
    }
  }
}