February 3, 2010 at 10:11am
0 notes
Running Resque / Rake Tasks with Monit
script/consumer:
#!/usr/bin/env ruby
def pidfile
@pidfile ||= File.expand_path(File.join('..', '..', 'tmp', 'pids', 'consumer.pid'), __FILE__)
end
case ARGV.first
when 'start' then
require 'rubygems'
require 'rake'
ENV['QUEUE'] ||= '*'
ENV['RAILS_ENV'] ||= 'production'
load(File.expand_path(File.join('..', '..', 'Rakefile'), __FILE__))
Process.detach(consumer = Process.fork do
begin
Rake::Task['environment'].invoke
Rake::Task['resque:work'].invoke
rescue Exception => e
$stdout.puts([e, *e.backtrace].join($/))
end
end)
%x(echo "#{consumer}" > #{pidfile})
when 'stop' then
%x(kill `cat #{pidfile}`)
%x(rm #{pidfile})
else
puts 'usage: script/consumer start|stop'
end
September 2, 2009 at 9:50pm
0 notes
Reinstalling git on Snow Leopard
If you are reinstalling git for Snow Leopard, don’t forget to nuke MacPorts. Leaving an old version on your machine will cause the git compile process to spit out various incorrect architecture errors:
ld: warning: in /opt/local/lib/libz.dylib, file is not of required architecture
ld: warning: in /opt/local/lib/libiconv.dylib, file is not of required architecture
Remove MacPorts with the terrifyingly brutal:
sudo rm -rf \
/opt/local \
/etc/manpaths.d/macports \
/etc/paths.d/macports \
/Applications/DarwinPorts \
/Applications/MacPorts \
/Library/LaunchDaemons/org.macports.* \
/Library/Receipts/DarwinPorts*.pkg \
/Library/Receipts/MacPorts*.pkg \
/Library/StartupItems/DarwinPortsStartup \
/Library/Tcl/darwinports1.0 \
/Library/Tcl/macports1.0
Once you have removed MacPorts, the compile process should be as easy as:
make configure
./configure --prefix=/usr/local
make prefix=/usr/local all
sudo make prefix=/usr/local install
git config --global user.name "[user.name]"
git config --global user.email "[user.email]"
And when you are done, don’t forget this little gem for adding pretty colours to your git output:
git config --global color.ui "auto"
Reinstalling All Native Gems on Snow Leopard
After installing Snow Leopard all of my native code gems broke horribly, complaining that they were now on the wrong architecture. To re-install every gem automatically, I used:
sudo gem list | awk '{print $1}' | xargs sudo gem install
Flyweighting in Python Redux
I just expanded the Flyweighted Object class to support keyword arguments:
import weakref
import cPickle
class FlyweightedObject(object):
_pool = weakref.WeakValueDictionary()
def __new__(klass, *args, **kwargs):
if not hasattr(klass.__init__, 'im_func'): raise 'cannot flyweight an object which has no python constructor'
arguments = {}
constructor = klass.__init__.im_func
arguments_missing = constructor.func_code.co_argcount - len(args) - 1
if arguments_missing > 0:
args += constructor.func_defaults[-arguments_missing:]
varnames = constructor.func_code.co_varnames[1:]
for i in range(len(varnames)):
varname = varnames[i]
arguments[varname] = kwargs.get(varname, args[i])
key = cPickle.dumps((klass, arguments))
instance = klass._pool.get(key, None)
if instance is None:
instance = object.__new__(klass)
klass._pool[key] = instance
return instance
def __getnewargs__(self):
if hasattr(self.__class__.__init__, 'im_func'):
constructor = self.__class__.__init__.im_func
return tuple(getattr(self, attr) for attr in constructor.func_code.co_varnames[1:])
return tuple()
Flyweighting in Python
If you are dealing with large static datasets in Python it can be useful to flyweight your objects. With flyweighting, every time you construct a new object you check to see if it already exists. If so, the original object will be returned instead of constructing a duplicate.
Recently I wrote a little bit of code to achieve this in the general case:
import weakref
class FlyweightedObject(object):
_pool = weakref.WeakValueDictionary()
def __new__(klass, *args):
if hasattr(klass.__init__, 'im_func'):
constructor = klass.__init__.im_func
arguments_missing = constructor.func_code.co_argcount - len(args) - 1
if arguments_missing > 0:
args += constructor.func_defaults[-arguments_missing:]
key = (klass,) + args
instance = klass._pool.get(key, None)
if instance is None:
instance = object.__new__(klass)
klass._pool[key] = instance
return instance
Now when you inherit FlyweightedObject you get the flyweighting thrown in for free:
class Person(FlyweightedObject):
def __init__(self, age, name = 'Simon'):
self.age = age
self.name = name
f = Person(1)
g = Person(1, 'Simon')
print id(f) == id(g)
# => True
One thing to watch out for is changing attributes after the object has been constructed. This will lead to the flyweight pool keys and the objects themselves going out of sync:
f = Person(1, 'Dave')
f.name = 'Simon'
g = Person(1, 'Simon')
print id(f) == id(g)
# => False
One last thing. To get pickle working with flyweighted objects you’ll have to create a __getnewargs__ method which returns the tuple that will be passed to __new__ on unpickling:
class Person(FlyweightedObject):
def __init__(self, age, name = 'Simon'):
self.name = name
self.age = age
def __getnewargs__(self):
return self.age, self.name
This can also be automated as long as the instance variables are named correctly:
def __getnewargs__(self):
if hasattr(self.__class__.__init__, 'im_func'):
constructor = self.__class__.__init__.im_func
return tuple(getattr(self, attr) for attr in constructor.func_code.co_varnames[1:])
return tuple()
Haml 2.2 Compile Error In Ugly Production Mode
If you were previously using the Haml syntax:
%p= 'string ', method, ' string'
you will need to change it to:
%p= ['string ', method, ' string']
to avoid compilation errors in production mode.
Python Slots
I just got clued in by Elf Sternberg’s Blog to a really useful feature in Python that I have never heard about before, slots:
class Foo(object):
__slots__ = ['x']
def __init__(self, n):
self.x = n
From the Python reference manual:
“By default, instances of both old and new-style classes have a dictionary for attribute storage. This wastes space for objects having very few instance variables. The space consumption can become acute when creating large numbers of instances.
The default can be overridden by defining slots in a new-style class definition. The slots declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because dict is not created for each instance.”
At work we have a plans engine that loads millions of little Python objects into memory, so this is a great little optimisation for us.
Finally, to get a set of every slot attribute in the object hierarcy, you can add this method to your class:
def inherited_slots(self):
return set(slot for klass in self.__class__.__mro__ if hasattr(klass, '__slots__') for slot in klass.__slots__)
JavaScript String format
Here’s a useful snippet of JavaScript I use for inserting arguments into a format string:
String.format = function()
{
var replacements = arguments;
return arguments[0].replace(/\{(\d+)\}/gm, function(string, match) {
return replacements[parseInt(match) + 1];
});
}
And here it is in action:
String.format('http://www.google.com/search?q={0}', escape(searchTerm))
Converting ASCII Into Unicode In Python
To convert the unicodeString José into Jose in python:
import unicodedata
unicodedata.normalize('NFKD', unicodeString).encode('ASCii', 'ignore')
IO Error From raw_input() in Jython
When I try to use raw_input on my Ubuntu machine in Jython 2.2.1 on Java 1.6.0_07 I get an empty IOError when executing a python file, but not when using a python shell:
-- raw_input.py --
raw_input()
>> jython raw_input.py
Traceback (innermost last):
File "raw_input.py", line 1, in ?
IOError:
-- console --
>>> raw_input()
hello
'hello'
To get round this I just use sys.stdin.readline().strip() to read a line from stdin and remove the trailing newline character.
1.
