Category Archives: Programming

Programming topics. Threads, graphics, realtime, whatever.

Simple Batch Processing

See my new github project:

The goal of this is to be reasonable fast and powerful, while being dead simple to install and configure.

Installation and configuration should be as simple as:

  $ pip install simplebatch
  $ batchd &
  $ bsubmit -- my_first_job

The current status is pretty much there. There are rough edges, but it is as simple to get started as above.

Using more than working machine will initially require running batchd on the first one and batch_manager on additional machines, but eventually batchd will have a –slave mode.

My first use case for simplebatch will be running graphics magick over directories of images. Other places that it could be useful would be blender rendering, batch downloading, ffmpeg, R jobs, reporting, and many more.

Future goals:

Test large numbers of jobs in the queue.
Support postgres in addition to sqlite. Sqlite was initially chosen to make it easy to get going.

Support what Sun Grid Engine calls Array Jobs. This will make managing animation renders even easier. Currently you would presumably assign one job per frame, but with array jobs you would be able to submit a single job for an array of N frames, and simplebatch would track them as a group.


I realized that when searching for python and lzo, my github page is the first link. I’ve also noticed that people keep forking my github version and that there is no PyPI entry for python-lzo.

So, I decided it is fair to consider myself the maintainer for python-lzo and have registered it as a PyPI package. Find it here:

I guess the next steps will be to setup travisci, update the, add python3 compatibility and find out what changes the people forking it are making.


I liked’s irccat as an easy way to post events (task completion and errors) to an IRC channel. I was less crazy about the recommendation to use ant to execute it. When I moved it to a OpenVZ VPS that used venet, it stopped working reliably. I’ve actually had trouble with quite of few Java services on that sort of VPS. So, I wrote my own in python, and it can be found on github and PyPI.

An added benefit is that much less memory is now used as well.

Postgres Upgrade

I just upgraded from PostgreSQL 8.3 to 9.2.  In doing so, I also went from a Sun optimized build (since I installed their supplied version before) to a self built version where I didn’t fiddle with optimization at all (the documentation suggests -O5, but I let it go with the default of -O3).  Despite the lazy build, it seems a lot faster in terms of the queries some of my python code executes.

The Javacript Compiler I Want

There are a number of compilers that in something and output Javascript. A few of the more well known ones seem to be CoffeeScript, Dart, and emscripten.

What I really want though is a Javascript to Javascript compiler that will run in Javascript. Here is the practical application. Top in mind is that I want to write code for ECMAScript 5 or even use Harmony features, and have it compile to output that will run on IE7 (which only supports ECMAScript 3). A few features that come to mind are generators (the yield keyword), array comprehensions, getters and setters, and block scoping. A lot of this stuff has been around since Firefox 2! I think that getters/setters have been around even longer.

Once we have an AST, I can picture how to implement some features (getters/settings, maybe block scoping).

Two pieces that I think could be useful, if someone were to implement this are esprima and escodegen . When looking at esprima, be sure to check out the harmony branch.

Another related project is Narcissus ( Narcissus is a meta-cirular evaluator rather than a compiler, which I take to mean that there will be a performance hit likely to make it unacceptable for use with IE 7 & 8. However, performance on IE isn’t relevant anyway since this implementation relies on features only found in recent SpiderMonkey engines (and thus also won’t run in Chrome be default).

While on the topic, I should probably mention re-using some of the shims and polyfills out there (such as harmony-collections), as well as Traceur. Traceur is ECMAScript 6 input and ECMAScript 5 output.

One question I have is should the compiled output include polyfills, which can effect other libraries you use and would be unsafe in facebook apps, or should it all hide behind a function like Facebook’s ES5 library.  As you can see in Facebook’s blog post, the ES5 wrapped code is less pleasant to read and they mention a performance hit.  I suppose for extra work, such a compiler could support both.

Winter Banner

And now I have a new Winter themed banner. See the code and assets for it in my processing banners git repo here (
). Processing.js can be found here

This expands out into working with particles as objects, which I’ve done before, but is stretching further into the processing.js language further than I’ve been before.

I’m thinking for the next banner to get rid of the processing language and just write directly in Javascript against the processing.js API. As a general rule, I don’t like systems that force a class system on top of Javascript. I personally rather like prototypical object systems (like Javascript or Self).

How I like to build for Windows

I’ve seen a growing number of free software Windows programs where the recommended (or only) build procedure is to cross-build from Linux. I really like this idea.

The latest I noticed that did this is the Clementine music player (which I’m rather excited about). This is a fork of amaroK, with the KDE parts stripped to make in plain Qt, and thus portable. Their web site is Their Windows build directions are here, and involves a chroot on a Debian system. This, of course, lends itself to the idea of projects eventually distributing build VM appliances, or if LXC takes off, premade build containers.

virtualenv and pip

Sorry, this isn’t a very original post. There are many guides on using virtualenv and pip. Some of them will be more comprehensive. Still, I find value in posting this because to some extent I do use my web site as notes for myself.

First, install virtualenv. This is probably as easy as: easy_install virtualenv. Or apt-get install python-virtualenv, or something like that for your platform.

Then, to create a virtualenv, go to your chosen work directory and type:

virtualenv –no-site-packages my_new_project

This will create my_new_project, and inside that bin, lib, and include directories. You could also use ./ as the project directory.
To work in your virtualenv, cd to my_new_project, and type:

source bin/activate

Now, python stuff will work from that directory ignoring whatever is installed on the system in general.

Doing easy_install or the like inside the virtualenv will cause the installed packages to be done there instead of system wide. But, you probably shouldn’t keep using easy_install. Apparently the hot new tool is pip.

I don’t have strong opinions about pip. I am mainly using it because it seems to be the current best practice. On paper, pip offers compatibility with both distutils and setuptools based packages, as well as the ability to install unpackaged stuff from svn repositories.

Setting up a new project in the virtualenv then is just a matter of a few commands like:

pip install psycopg2
pip install PIL

And of course, the beauty is that the package you just installed are only in your virtualenv, and not polluting the entire system. Project A won’t use Project Bs libraries without your intentionally installing the same libraries in Project A. This is valuable when you go to install Project A on a new machine.

Even better, you can painlessly try new packages without polluting your system.

Anyway, I don’t know if I helped you, but at least I now know that my memory on how and why is some place more secure than my memory.