Programming

Simple Batch Processing

See my new github project: https://github.com/jd-boyd/simplebatch

The goal of this is to be reasonable fast and powerful, while being dead simple to install and configure.

Installation and configuration should be as simple as:

 
  $ pip install simplebatch
  $ batchd &
  $ bsubmit -- my_first_job

The current status is pretty much there. There are rough edges, but it is as simple to get started as above.

Using more than working machine will initially require running batchd on the first one and batch_manager on additional machines, but eventually batchd will have a –slave mode.

My first use case for simplebatch will be running graphics magick over directories of images. Other places that it could be useful would be blender rendering, batch downloading, ffmpeg, R jobs, reporting, and many more.

Future goals:

Test large numbers of jobs in the queue.
Support postgres in addition to sqlite. Sqlite was initially chosen to make it easy to get going.

Support what Sun Grid Engine calls Array Jobs. This will make managing animation renders even easier. Currently you would presumably assign one job per frame, but with array jobs you would be able to submit a single job for an array of N frames, and simplebatch would track them as a group.

System Administration

Proxmox on DigitalOcean

I thought I’d try to install Proxmox 4 on DigitalOcean. How I struck on that idea was very convoluted, and I don’t have a specific purpose for doing so, but if you want LXC containers instead of Docker or rkt, and you want a full set of web admin tools, this might not be a bad way to go.

By the way, if you like this write up and don’t already use DigitalOcean, feel free to thank me by signing up with my referral link. That will give you a $10 getting started credit to experiment with and give me some referral credit to use for my future bill with them.

Thinking about it, I don’t expect DigitalOcean to support running KVM on their VMs (which are already KVM). I do expect the LXC containers to work. Also, there is likely to be some networking difficulty due to having only one external and one internal IP. However, that shouldn’t stop you from seeing it as a good idea. I’ve previously used LXC directly with only a single IP and before that Solaris Containers with only a single IP. Futher discussion on that point will come later.

DigitalOcean doesn’t let you install any distribution that you might want, so installing from the Proxmmox ISO was out of the question. However, Proxmox publishes directions for installing it on top of Debian Jessie, which is a supported distribution on DigitalOcean.

Here are those directions:
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie

They are painless to follow and just work.

Here is a shortened version of them:

  1. Open /etc/hosts and change the 127.0.1.1 line to replace that IP with your external IP.
  2. Run the following shell commands:

    3. echo "deb http://download.proxmox.com/debian jessie pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
    wget -O- "http://download.proxmox.com/debian/key.asc" | apt-key add -
    apt-get update && apt-get dist-upgrade
    apt-get install proxmox-ve ntp ssh postfix ksm-control-daemon open-iscsi openvswitch-switch

    Note: If you are following the guide on the proxmox site, I’ve added one additional package to be installed, openvswitch-switch. I will explain more about that later.

  3. You will be asked about configuring postfix. If you don’t know what to choose, just choose for it to be a local only postfix server.
  4. Reboot.

At that point, you can log into the web admin page. Use https://your-ip:8006/.

At this point you can log in, look around, and after configuring a local storage location, downloading templates works correctly. Unfortunately, Cceating a CT (lxc) doesn’t work initially. The dialog requires choosing a bridge and there are no bridges.

At this point I followed the guide for using OpenVSwitch here and installed OpenVSwitch then configured a switch that isn’t connected to a “physical” interface. From the command line,
I then created a vlan on the switch, so that the host could use the new switch.

Now, when creating the container, choose the vswitch bridge. Also, set the vlan tag of the containers interface to the number used on the host. Now you are good to go.

Using a vswitch not attached to a physical interface gets you private networking between your containers, but it still doesn’t solve what to do about only having a single IP. Previously my tactic was to have nginx in a container be the designated front end answering on port 80 and 443. This will mean setting up iptables on the host to forward those ports to the front end container. Then, in the nginx config for that container, I would proxy-pass by domain/hostname through to the various web service containers I was running (for instance, personal web site, project websites, and other web based software I might be using for monitoring, etc). This strategy can also be extended for other services like email, DNS, and so on.

Other

Just a Tree

Trying some inline processing.js.

Other

Make PostgreSQL default to UTF-8

From here :

UPDATE pg_database SET datistemplate=FALSE WHERE datname='template1';

DROP DATABASE template1;

CREATE DATABASE template1 WITH owner=postgres template=template0 encoding='UTF8';

UPDATE pg_database SET datistemplate=TRUE WHERE datname='template1';

Other
Aside

I placed an iPhone screen down on my Nexus 4. The Nexus got confused, took a picture, then rebooted to contemplate what had just happened.

Acurately predicted 20 out of the last 4 Zebra stampedes.