Simon Engledew's Blog

Programatically get the version of an egg

Wed, 12 Oct 2011 11:32:43 +0100

The documentation for pkg_resources does not make it 100% obvious how to get the current version of a Python egg. To do so:

pkg_resources.get_distribution(egg_name).version

Tee and exec at the same time

Fri, 13 May 2011 08:47:00 +0100

Print a dot for every four lines of output, also logging to output.log.

LOGFILE="output.log"
LOGPIPE="/tmp/$$.tmp"
trap "rm -f $LOGPIPE" EXIT
mknod "$LOGPIPE" p
sudo tee <"$LOGPIPE" -a "$LOGFILE" | awk '{ if (NR % 4 == 0) printf "."; system("") }' &
exec 1>>"$LOGPIPE" 2>&1

Ruby lambda in a different scope

Wed, 11 Aug 2010 11:10:00 +0100

When you create a ruby lambda it is bound the the scope it was defined in. Therefore, if you create it in the class scope, you cannot access instance variables or protected instance methods.

To execute the lambda in your instance’s scope, use the instance_eval method:

Git Post Commit Hook to Play "Hallelujah" After a Commit

Thu, 05 Aug 2010 13:53:00 +0100

Inspired by Brandon Keepers, what follows is a self contained script that will play a single Hallelujah after you commit.

To make it work, place it in your repository’s .git/hooks directory as post-commit and chmod it executable.

NSThread a selector with multiple arguments

Thu, 24 Jun 2010 10:03:00 +0100

One of the curious absences in the Mac SDK is the ability to invoke a thread that targets a method with multiple arguments.

If you want to use [+ NSThread detachNewThreadSelector:toTarget:withObject:] you either have to refactor your function to a) take a collection like an NSArray or NSDictionary or b) call an intermediary function which then makes the call on that thread. You would half-expect a method like [+ NSThread detachNewThreadSelector:toTarget:withObjects:] to exist.

In order to get round this, I created the SEThreadInvocation object which blends a NSThread and a NSInvocation together.

You can then call your method like so:

Duplicate NSManagedObject with NSFetchedResultsController

Fri, 30 Apr 2010 11:04:00 +0100

While working on WineLedger for the iPhone I spent a lot of time trying to work out why duplicate ‘ghost’ entries would sometimes appear when merging two NSManagedObjectContexts:

As I inserted a new entry, two copies would appear: One would remain stuck, while the second would correctly update when you edited it. As soon as the app was restarted the ghost entry would miraculously vanish. To add to the confusion, this behaviour would only occur when I used the camera to take a photo via UIImagePickerController.

After a lot of trial and error I finally discovered that the main view was being unloaded from memory as soon as you took a picture. Using the camera causes a huge memory spike as the phone stores the high resolution image.

This shouldn’t be a problem, but as the view loads back into memory the NSFetchedResultsController does a fetch. This initial fetch loads the full set of results (including the freshly inserted object), and then the delegate methods proceed to perform a nice slide animation when the two contexts are merged, inserting the object into the UITableView again.

The fix for this problem is to insert a guard to check to see if the NSFetchedResultsController already has some managed objects before it does a fetch on viewDidLoad.

if (self.fetchedResultsController.fetchedObjects == nil)
{
    NSError * error = nil;
	
    if (![self.fetchedResultsController performFetch:&error])
    {
        NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
    }
}

Running Resque / Rake Tasks with Monit

Wed, 03 Feb 2010 10:11:00 +0000

script/consumer:

#!/usr/bin/env ruby

def pidfile
  @pidfile ||= File.expand_path(File.join('..', '..', 'tmp', 'pids', 'consumer.pid'),  __FILE__)
end

case ARGV.first
  when 'start' then
    require 'rubygems'
    require 'rake'
    
    ENV['QUEUE'] ||= '*'
    ENV['RAILS_ENV'] ||= 'production'

    load(File.expand_path(File.join('..', '..', 'Rakefile'),  __FILE__))

    Process.detach(consumer = Process.fork do
      begin
        Rake::Task['environment'].invoke
        Rake::Task['resque:work'].invoke
      rescue Exception => e
        $stdout.puts([e, *e.backtrace].join($/))
      end
    end)

    %x(echo "#{consumer}" > #{pidfile})

  when 'stop' then
    %x(kill `cat #{pidfile}`)
    %x(rm #{pidfile})
    
  else
    
    puts 'usage: script/consumer start|stop'
end

Reinstalling git on Snow Leopard

Wed, 02 Sep 2009 21:50:00 +0100

If you are reinstalling git for Snow Leopard, don’t forget to nuke MacPorts. Leaving an old version on your machine will cause the git compile process to spit out various incorrect architecture errors:

ld: warning: in /opt/local/lib/libz.dylib, file is not of required architecture
ld: warning: in /opt/local/lib/libiconv.dylib, file is not of required architecture

Remove MacPorts with the terrifyingly brutal:

sudo rm -rf \
    /opt/local \
    /etc/manpaths.d/macports \
    /etc/paths.d/macports \
    /Applications/DarwinPorts \
    /Applications/MacPorts \
    /Library/LaunchDaemons/org.macports.* \
    /Library/Receipts/DarwinPorts*.pkg \
    /Library/Receipts/MacPorts*.pkg \
    /Library/StartupItems/DarwinPortsStartup \
    /Library/Tcl/darwinports1.0 \
    /Library/Tcl/macports1.0

Once you have removed MacPorts, the compile process should be as easy as:

make configure 
./configure --prefix=/usr/local
make prefix=/usr/local all
sudo make prefix=/usr/local install

git config --global user.name "[user.name]" 
git config --global user.email "[user.email]"

And when you are done, don’t forget this little gem for adding pretty colours to your git output:

git config --global color.ui "auto"

Reinstalling All Native Gems on Snow Leopard

Wed, 02 Sep 2009 21:38:00 +0100

After installing Snow Leopard all of my native code gems broke horribly, complaining that they were now on the wrong architecture. To re-install every gem automatically, I used:

sudo gem list | awk '{print $1}' | xargs sudo gem install

Flyweighting in Python Redux

Fri, 24 Jul 2009 01:56:00 +0100

I just expanded the Flyweighted Object class to support keyword arguments:

import weakref
import cPickle

class FlyweightedObject(object):
    _pool = weakref.WeakValueDictionary()

    def __new__(klass, *args, **kwargs):
        if not hasattr(klass.__init__, 'im_func'): raise 'cannot flyweight an object which has no python constructor'

        arguments = {}

        constructor = klass.__init__.im_func
        arguments_missing = constructor.func_code.co_argcount - len(args) - 1

        if arguments_missing > 0:
            args += constructor.func_defaults[-arguments_missing:]

        varnames = constructor.func_code.co_varnames[1:]

        for i in range(len(varnames)):
            varname = varnames[i]
            arguments[varname] = kwargs.get(varname, args[i])

        key = cPickle.dumps((klass, arguments))
        instance = klass._pool.get(key, None)

        if instance is None:
            instance = object.__new__(klass)
            klass._pool[key] = instance

        return instance

    def __getnewargs__(self):
        if hasattr(self.__class__.__init__, 'im_func'):
            constructor = self.__class__.__init__.im_func
            return tuple(getattr(self, attr) for attr in constructor.func_code.co_varnames[1:])
        return tuple()

Flyweighting in Python

Fri, 10 Jul 2009 02:55:00 +0100

If you are dealing with large static datasets in Python it can be useful to flyweight your objects. With flyweighting, every time you construct a new object you check to see if it already exists. If so, the original object will be returned instead of constructing a duplicate.

Recently I wrote a little bit of code to achieve this in the general case:

import weakref

class FlyweightedObject(object):
    _pool = weakref.WeakValueDictionary()
    
    def __new__(klass, *args):
        if hasattr(klass.__init__, 'im_func'):
            constructor = klass.__init__.im_func
            arguments_missing = constructor.func_code.co_argcount - len(args) - 1
            if arguments_missing > 0:
                args += constructor.func_defaults[-arguments_missing:]

        key = (klass,) + args
        instance = klass._pool.get(key, None)

        if instance is None:
            instance = object.__new__(klass)
            klass._pool[key] = instance
            
        return instance

Now when you inherit FlyweightedObject you get the flyweighting thrown in for free:

class Person(FlyweightedObject):
    def __init__(self, age, name = 'Simon'):
        self.age = age
        self.name = name

f = Person(1)
g = Person(1, 'Simon')

print id(f) == id(g)
# => True

One thing to watch out for is changing attributes after the object has been constructed. This will lead to the flyweight pool keys and the objects themselves going out of sync:

f = Person(1, 'Dave')
f.name = 'Simon'  

g = Person(1, 'Simon')

print id(f) == id(g)
# => False

One last thing. To get pickle working with flyweighted objects you’ll have to create a __getnewargs__ method which returns the tuple that will be passed to __new__ on unpickling:

class Person(FlyweightedObject):
    def __init__(self, age, name = 'Simon'):
        self.name = name
        self.age = age

    def __getnewargs__(self):
        return self.age, self.name

This can also be automated as long as the instance variables are named correctly:

def __getnewargs__(self):
    if hasattr(self.__class__.__init__, 'im_func'):
        constructor = self.__class__.__init__.im_func
        return tuple(getattr(self, attr) for attr in constructor.func_code.co_varnames[1:])
    return tuple()

Haml 2.2 Compile Error In Ugly Production Mode

Thu, 09 Jul 2009 12:09:00 +0100

If you were previously using the Haml syntax:

%p= 'string ', method, ' string'

you will need to change it to:

%p= ['string ', method, ' string']

to avoid compilation errors in production mode.

Python Slots

Tue, 07 Jul 2009 10:42:00 +0100

I just got clued in by Elf Sternberg’s Blog to a really useful feature in Python that I have never heard about before, slots:

class Foo(object):
    __slots__ = ['x']
    def __init__(self, n):
        self.x = n

From the Python reference manual:

“By default, instances of both old and new-style classes have a dictionary for attribute storage. This wastes space for objects having very few instance variables. The space consumption can become acute when creating large numbers of instances.

The default can be overridden by defining slots in a new-style class definition. The slots declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because dict is not created for each instance.”

At work we have a plans engine that loads millions of little Python objects into memory, so this is a great little optimisation for us.

Finally, to get a set of every slot attribute in the object hierarcy, you can add this method to your class:

def inherited_slots(self):
        return set(slot for klass in self.__class__.__mro__ if hasattr(klass, '__slots__') for slot in klass.__slots__)

JavaScript String format

Thu, 02 Jul 2009 13:11:00 +0100

Here’s a useful snippet of JavaScript I use for inserting arguments into a format string:

String.format = function()
{
  var replacements = arguments;
  return arguments[0].replace(/\{(\d+)\}/gm, function(string, match) {
    return replacements[parseInt(match) + 1];
  });
}

And here it is in action:

String.format('http://www.google.com/search?q={0}', escape(searchTerm))

Converting ASCII Into Unicode In Python

Mon, 15 Jun 2009 16:08:00 +0100

To convert the unicodeString José into Jose in python:

import unicodedata

unicodedata.normalize('NFKD', unicodeString).encode('ASCii', 'ignore')

IO Error From raw_input() in Jython

Wed, 20 May 2009 12:54:00 +0100

When I try to use raw_input on my Ubuntu machine in Jython 2.2.1 on Java 1.6.0_07 I get an empty IOError when executing a python file, but not when using a python shell:

-- raw_input.py --
raw_input()

>> jython raw_input.py 
Traceback (innermost last):
  File "raw_input.py", line 1, in ?
IOError:

-- console --
>>> raw_input()
hello
'hello'

To get round this I just use sys.stdin.readline().strip() to read a line from stdin and remove the trailing newline character.

Greedy and Reluctant Qualifiers

Fri, 15 May 2009 08:52:00 +0100

A common gotcha when using regular expressions occurs when using the default (greedy) qualifiers + ? and *. These qualifiers will attempt to make the longest match they possibly can.

The regular expression:

/'(.*)'/

will successfully match and group the word there in hello 'there', but will actually run on to the last single quote in the string hello 'there' how are 'you', matching there' how are 'you.

One solution is to restrict the set of characters you are searching for with the greedy qualifier, thus ensuring the match will finish before hitting the terminating character:

/'([^']*)'/

This works, but the more readable option is to turn the greedy qualifier into a reluctant one:

/'(.*?)'/

By adding a ? to the qualifier the expression will try to match the minimum string that satisfies the expression.

Making Java Processes Play Nice

Thu, 14 May 2009 14:50:06 +0100

Looking around, it seems that there is no easy way to stop Java from eating all your system resources when running a particularly heavy-going task.

Thankfully my lovely colleague Ben made me aware of a helpful UNIX command called nice.

By prefixing nice to any command you can ask the scheduler to be a bit more kind, running the process at a slightly lower priority to ensure it doesn’t starve other resources of CPU time:

nice java ExpensiveTask

Progressive Enhancement Using Comments

Tue, 28 Apr 2009 14:46:00 +0100

Most methods of progressive enhancement involve setting various DOM elements to display:none and then making them visible from JavaScript, or building new document nodes and inserting them.

In the former case you end up with lots of brittle snippets of code for traversing the DOM and toggling elements. In the later case you often end up having to write the same markup twice: Once in your web application, and once in the JavaScript which enhances your code.

I recently read a blog post by James Padolsey that suggested creating comment nodes with HTML inside and then promoting their contents to real nodes within their parent elements. Neat.

The comments for this article on Hacker News quickly moved to performance however: As you can’t natively scour the DOM for comments, it is necessary to iterate over every DOM element from JavaScript – checking its type and manipulating it if applicable. An expensive business.

If performance really is an issue, but you like the idea of baking your HTML directly into the page as comments, you can replace the DOM traversal with an atomic regular expression replace and an innerHTML assignment. Both of these operations occur in the underlying DOM implementation and are run at native speeds:

document.body.innerHTML = document.body.innerHTML.replace(/(?:<!--\[enhance\]>)|(?:<!\[enhance\]-->)/g, '');

Wrapped up into a tidy rails helper, you can use this to create blocked out elements which will only render if the user has JavaScript enabled:

def enhancement(&block)
  concat("<!--[enhance]>#{capture(&block)}<![enhance]-->")
end

<% enhancement do %>You have JavaScript enabled<% end %>

Obviously it’s not quite as flexible as the DOM-based solution — especially as James seems to be branching out into a fully-blown templating system called JSHTML — but I think it works as a speedy way to encapsulate JavaScript-only functionality.

MySQL Remote Database Transfers

Mon, 09 Mar 2009 13:53:00 +0000

One-liner to remote copy a MySQL database over SSH:

mysqldump [db] | ssh -C [host] 'mysql [db]'

For anything more complicated there is also taps.