Simple Batch Processing

See my new github project:

The goal of this is to be reasonable fast and powerful, while being dead simple to install and configure.

Installation and configuration should be as simple as:

  $ pip install simplebatch
  $ batchd &
  $ bsubmit -- my_first_job

The current status is pretty much there. There are rough edges, but it is as simple to get started as above.

Using more than working machine will initially require running batchd on the first one and batch_manager on additional machines, but eventually batchd will have a –slave mode.

My first use case for simplebatch will be running graphics magick over directories of images. Other places that it could be useful would be blender rendering, batch downloading, ffmpeg, R jobs, reporting, and many more.

Future goals:

Test large numbers of jobs in the queue.
Support postgres in addition to sqlite. Sqlite was initially chosen to make it easy to get going.

Support what Sun Grid Engine calls Array Jobs. This will make managing animation renders even easier. Currently you would presumably assign one job per frame, but with array jobs you would be able to submit a single job for an array of N frames, and simplebatch would track them as a group.

Leave a Reply