A new message queuing system

First, why a new one?  Because I haven’t found any that do what I need and look simple and well supported.  Besides, it seems like a reasonable learning experience.

The initial summary of what I need is a light weight method for PHP (in the form of scripts running in mod_php) to send messages to a back end program written in Python. However, in the future other languages may be used, so general cross language compatibility is important. That means either bindings exist for every conceivable language, or bindings are trivial to write.  Also, the solution must run on Solaris.

Comparison:

At work I use Apache QPID, which is an AMQP implementation. I can’t find any PHP client for AMQP though. I was able to find discussion that suggested that the AMQP protocol is too heavyweight for PHP being run in mod_php.

Looking at other solutions, I don’t want to run anything that requires Java for the server. That rules out Apache ActiveMQ and other JMS systems. I also believe that XMPP is too heavy weight to parse.  I also found some systems written in Perl, Ruby, and PHP, but they looked rather slap dash, and I don’t particularly want to use those languages.  The initial requirement for supporting PHP is only because I’m working on a PHP web app that I forked.  I do not want to add any new PHP code bases that I need to maintain.  Besides, once I start looking at fringe choices, it gets to be a lot easier to justify writing my own, particularly if I am going to use it as a learning project to get more familiar with, say, Erlang.

Summary:

It is to be a RESTful design.  I will be using JSON for the payloads, but I haven’t decided yet if it makes sense to force this, or if it makes sense to allow all ascii data.  Queues will be single read, meaning that if multiple end points need to get the same message, then there will need to be a separate queue for each end point.  Initially there will be no security model or persistence.  Commands will be standard HTTP verbs.  If possible I will try to make response codes valid HTTP response codes.

Goals:

Run on Solaris

Support Python, PHP, and Javascript as clients.

Limitations:

Initially this will not support persistence.

Also, this will not support any security.

Commands:

PUT queueName

Create a new queue. What the responses will be still need to be decided.

Responses:

201 created, entity required, probably just a confirmation message

409 already existed.

POST queueName

The body of the post will be the contents of the message.

403 queueName wasn’t found.

201 created, and perhaps the entity will be an id for the message.

GET

Gets that do not match the following patterns will be answered with a 404.

GET msg/queueName

Get the next message from the queue queueName.  Here I need some way to to return a message ID in addition to the message body.  It may make sense for the response to be JSON: {‘id’: integer, ‘content’: <valid JSON here>}

If the response is as proposed, the the contents of the POST must be valid JSON as well.

403 queueName wasn’t found.

200, the message

GET queues

Get a list of the created queues.

200

DELETE queueName/integer

Delete a message identified with integer from queue queueName.

204 deleted, no entity required

403, queuename or integer not found.

DELETE queueName

Delete a queue and all the messages in it.

204 Deleted, no entity in response

403 queueName wasn’t found

Thread Worker Pooling in Python

The worker pool pattern is a fairly common tool for writing multi-threaded programs.  You divide your work up into chunks of some size and you submit the to a work queue.  Then there is a pool of threads that watch that queue for tasks to execute, and when complete, they add the jobs into the finished queue.

Here is the file.

Thanks to Global Interpreter Lock, threads are of somewhat limited usefulness in Python.  I foresee myself mostly using this for network limited tasks, like downloaded a large quantity of RSS feeds.  My idea is that tasks put into the system shouldn’t modify global state, so if I actually needed this for computational tasks, it may be feasible to build it on forks instead, or perhaps the 2.6 multiprocessing system.  However, I still use a lot of systems with only python 2.3 installed, so I’m not likely to want to write 2.6 specific code anytime soon.

Many of the thread pool systems I seem have you specify a single function for the pool, then you just enqueue the inputs.  Mine is different in that each item in the queue can be a different function.  I haven’t actually used it this way though, so it is possible that the extra flexibility is generally wasted.

Python’s lambda seem rather limited.  It is limited to a single expression.  I suppose that this is what Lisp and Scheme do as well, but their expressions offer things like progn.  My first idea is that the task to execute would be a function with no arguments.  I was picturing using a lambda to wrap up whatever I wanted to do.

Now, I still offer that via addTask and assume it internally, but I also offer addTaskArgs, and it takes a function reference, and either an argument list (as a list) or a named argument list (as a dict) and wraps it in a lambda to enqueue.

I now find that my knowledge about how to unit test threaded code is rather limited, and the included unit tests are extremely thin.

Braided Cinnamon Bread

Preheat oven to 400 degrees.

Follow a basic milk bread recipe (flour, sugar, salt, yeast, water, milk, shortening) through the first kneading and rising stage.

Split dough into three equal portions

Roll out each portion, into a roughly 4:3 shape. Cover with a mixture of sugar, butter, and cinnamon, then roll into a tube.

Fan out three tubes, pinch ends together and braid.

Allow to rise again until double in size.

Bake until lightly golden.

Top with milk/powdered sugar glaze (roughly 1tbsp milk to 1/2 cup powdered suger, 2-4 multiples of that formula will be needed).

Serve. It took 4 hands and a lot of care to move that monster to the platter seen in the first picture.

Making Miro work with USB sound devices on Ubuntu

On Ubuntu (and possibly other linux distributions) Miro refuses to work with a secondary sound card, it will only work with the primary one despite what the ALSA default is set to, unlike most programs which offer some way to override the default.

Potentially, the second sound card in question could be a PCI card or something else, but based on other people’s experience (like my own) it is usually a USB sound card that is causing trouble. See here (note, the suggested fix didn’t work for me, just like it didn’t the original poster there) and here (they mention fixing it in the trunk, but that doesn’t help me until a new release comes out).

Some people actively want both the onboard sound and the USB or PCI device working, but if you are willing to sacrifice on-board sound, I found a work around. In my case, the on-board sound is worthless. It has some terrible humming/buzzing in the background so I never ever want to use it again.

The solution is to find what the module is that supplies your on-board sound. In my case, the on-board sound is a VT8233, so when I looked at the output from lsmod, it was obvious that the module for this sound device was the snd_via82xx module.

Then, open the /etc/modprobe.d/blacklist file to edit it:
sudo pico /etc/modprobe.d/blacklist
and add the line:
blacklist snd_via82xx
Then reboot.

Now, the USB audio device will be the first audio device.

Seeking

I am now looking for a new job and am no longer with Sigma Electronics.

My first preference would be a position writing software for post production or visual effects at either a software company or a post production or visual effects company.

Other than that, I am also interested in positions or contract work developing embedded systems, graphics applications, or web applications.

I just thought I would throw this out in case anyone can point me towards any leads.

Thank you.

A few Solaris 10 notes

Actually, these are primarily Solaris 11 notes, but they will probably all apply to Solaris 10 when the next release comes out, which I understand to be scheduled for sometime later this month.

First, recently a lot of SCSI hard drives I’ve gotten have been a little mysterious about being used by the Solaris installer and have looked a little odd in format. It turns out that they’ve been EFI labeled drives. Since Solaris understands EFI labelling, it doesn’t just suggest you relabel the drive and be done with it. However, despite Solaris understanding EFI, it refuses to boot or install from EFI on SPARC hardware. The trick has been to get a prompt, then use “format -e”. Then when you choose the label command, it will ask you about a SMI or a EFI label. Choose the SMI option. If you are going to choose to do a ZFS root, then the partitioning doesn’t matter.

After fixing the disk, you are ready to install. The ZFS boot option is only offered on very new copies of Solaris (2008/05 maybe, Solaris Express build 98 or maybe slightly older definately). However, you only get the choice from the text installer. If you are installing over the serial console, then no problem, you get this by default. However, from a graphical console, you will need to use a boot parameter. Thus, you boot command will look something like this: “boot cdrom – text” or “boot net – text”. Using – nowin instead may be faster.

When you get to the ZFS option, just choose it and away you go. You can choose to name the pool something other than rpool, but there is no need to.

If you want a mirrored root, it is easy to add the second disk later. First, when you install to a ZFS root, it repartitions the root drive and uses a slice (parition) instead of the whole disk (even though the slice fills the entire disk). You will need to partition the second disk identically. Just look at the partition map if the first disk in format, then copy it over to the second disk. Then from a root prompt, type something like “zpool attach rpool c0t0d0s0 c0t1d0s0”, assuming that c0t0d0 and c0t1d0 are the two disks in question (which is a good guess on a lot of two disk Sun systems). The mirror is now made, but it may take awhile to sync up in the background, and the machine may run slowly until it is done. Check the progress with “zpool status”.

To be able to do a fallback boot to the second disk will require rebooting and going back out to the OpenBoot ok prompt. But before that, you will need to make the second disk bootable with this command: “installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t1d0s0”
Finally, before you head to the OK prompt, you will want to find the openboot device paths for each disk. Do “ls -l /dev/dsk/c0t0d0s0 /dev/dsk/c0t1d0s0”. This will show you something like:

lrwxrwxrwx 1 root root 41 Oct 1 21:02 /dev/dsk/c0t0d0s0 -> ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
lrwxrwxrwx 1 root root 41 Oct 1 22:57 /dev/dsk/c0t1d0s0 -> ../../devices/pci@1f,4000/scsi@3/sd@1,0:a

Write down the target of the symlinks, the part after the ../../devices, changing the sd’s to disk’s, and get rid of the :a’s.

Now reboot and Stop-A to an ok prompt. If your second disks isn’t where the second disk normally will be, you will need to create a devalias for it. Assuming that you used the c0t0d0 and c0t1d0, then you can just do this:
setenv boot-device disk disk2

If you need to change the disk and disk2 aliases (or want to create new names), use the nvalias command from the ok prompt. See the man page for more detailed operation though.

Flash on Ubuntu 8.04 AMD64

I run Ubuntu 8.04 AM64 on a laptop at work.  I’ve been doing this since Ubuntu 6.10.  This has not been a smooth ride. Ubuntu 6.10 i386 on my old laptop (I only “upgraded” because the old one was stolen from the plane on a busines trip) worked flawlessly for me.  Things have gotten a bit better as upgrades came out, but I still can’t use the wireless  (BCM43 device of some sort, no native driver, ndiswrapper won’t play nice), for instance.

My first and biggest tip is to stay away from 64bit linux on the desktop or laptop, unless you know why you need it.  That is very unlikely to be the case on laptops.

Moving on, for the longest time Flash would not work.  When I tried to configure the nswrapper system, it would start (sometimes) then crash the plugin.  Maybe I could view one flash website before needing to restart, maybe no flash web sites.  I finally got Flash worked out, and that is the main point of this post.

The trick to make flash work was to first install the 32bit version of FF3 from the Mozilla web site.  Put it in a new location (I went with /usr/local/firefox), and put that location in your path before /usr/bin.  For this to run, you will need ia32-libs installed.

Step 2 then is to go to the Adobe web and download the Flash 9 .tar.gz.  Don’t try to use the autodiscovery/autoinstall thing that Firefox will offer to do.  Extract the Flash 9 installer to a temporary directory, then copy the file libflashplayer.so from the temporary directory to the plugins directory (/usr/local/firefox/plugins for me since that is where extracted the 32 bit firefox from the mozilla web site).  Now, when you restart firefox, you will be using the 32bit only Flash with a 32bit version of firefox, and everything will work happily.

I think that in general, Linux doesn’t handle the 64bit transition as well as Solaris or Irix did.  As far as I can tell, Flash is 32 bit only on all platforms.  However on Solaris and Irix, 32bit versions of firefox or Mozilla are supplied, even though they are running on 64bit hardware.  Also, there seems to be a lot more of defaulting to 32bit unless specified otherwise, which is often reasonable.  And the culter seems to do a better job with supplying both 32bit and 64bit versions of libraries.

Vanilla Extract

I made vanilla extract.  Since I just followed the directions, here is the link.  I can’t try it for 4 weeks, and it won’t really be done for six months.  I hope it doesn’t come out weak.  I dropped a bit of the seeds on the floor and decided to trash what was on the floor rather than risk tainting the rest.

I got the beans from my brother in law.  I used Smirnoff as pictured.

Pesto Tilapia on Tomato Risotto with Roast Asparagus

I made enough for 4 this time because I wanted left overs to pack for lunch.

The tilapia was 1/2 pound rubbed with pesto from a squeeze tube, then fried. I didn’t love the flavor, and if I were to try fish like this again, I’d pick a different white fish, possibly haddock. Otherwise I might go with pesto chicken.

The risotto was very well like though. The main difference (from last time) was what I used for a liquid. From memory:

  • 1 cup tomato sauce (1 can)
  • 1/2cup wine.
  • 2.5 cups chicken stock.

Also, before I put the rice in butter, I first put 1/2 onion and 2 cloves garlic, both  minced, in the pan and softened them in butter.  This would be a fairly typical thing to do, but when I posted about risotto last time, I had been out of onions and thus didn’t include it.

After coating the rice with butter, continue preparing normally.  That means add a little liquid, then stir till it is absorbed, and repeat.  Keep doing this until it has a texture you like, probably about 20 minutes, and you probably won’t use all the broth you made.  Near the end add 2 Tbps of tomato paste.  Then add parmesan to taste.

About ReferURL

ReferURL.net is a link shorting service I created. You paste in a long URL (say to an eBay auction or newpaper article) and it gives you a short URL to use (http://referurl.net/123). You also have the option of picking an alias for a referurl, something like http://referurl.net/r/xxx. Also, a common usage pattern is a bookmarklet that you drag from the page to the toolbar. Whenever you click on the bookmarklet, it runs a bit of javascript code that submits the page you are currently looking at to ReferURL.

URL shortening services are great for emailing URLs to friends. Recently they are even more important for posting URLs on twitter (with only 140 characters, every character saved counts).

A service that does similar things called TinyURL.com has been around for a long time. Personally, I do not like tinyurl.com. I think it is ugly. There is another reason I remember disliking them, but it is possible that I have two services confused, so I won’t mention it. They also didn’t offer aliases when I wrote ReferURL.

I used to use another service, but it broke repeatedly, then when it had several months of downtime I decided to write my own. That service also didn’t support aliases.

At this point when I look around the new services that are similar, I see three things that may be better than ReferURL.

  1. Some services are prettier (of course, extra graphics means slower load times).
  2. Some services put the new shorter url into your clipboard buffer so that you don’t have to copy it yourself. I would love to add this, but as far as I can tell it is implemented with Flash, which I don’t own.
  3. With twitter, every character counts. There are now some services with names much shorter than referurl.net. TinyURL.com is one character shorter. Bit.ly is six characters shorter. If any one has an good idea for a name that is shorter than referurl.net, I would love to steal it. In my own twitter usage, I haven’t had trouble with the length of ReferURL yet though.

Anyway, those are my comments on the creation of ReferURL.net. For the time being, I plan to keep looking for ways to improve it and will keep working on it.

Also, I will be releasing the code for people who want to run/write their own service in the future. I had previously released some code, but now that it is several months old, I took it down until I had time to clean the current code for re-release. If someone were to email me asking about that, it would probably get me to do it sooner.  It is a Python project build on mod_python and PostgreSQL.

Acurately predicted 20 out of the last 4 Zebra stampedes.