Joshua D. Boyd

TL;DR: nginx in a zone, replacing apache as the reverse proxy. Then, one web app per zone, PHP or Python.

Broadly, I am happy with Solaris Zones, used as I previously laid out in my first post on the topic. I’ve not made dramatic changes to the idea. However, I have been fine tuning things.

In the previous post I mentioned multiple web zones, but I didn’t really go into that in much depth. What I had been doing is one web zone for me (which also was in charge of proxies) and one web zone behind the proxy for other users. Recently, I’ve installed nginx in its own zone to use as the proxy and I’ve been splitting the applications on my one web-zone into one web zone per application.

While splitting applications apart, I’ve been re-evaluating how they are run. For instance, anything running in mod_python should be updated to run against a wsgi server, and perhaps Apache isn’t the right server for that. I wonder the same thing about the PHP programs, but I’m not as ready to touch them yet.

In the case of the feed reader I moved from mod_php to CherryPy. This did involve a rewrite of the backend code, but that rewrite had been creaping along anyway. Since the web stuff was a re-write, CherryPy was a fairly easy choice and a chance to learn more about it. What I learned is that I feel a bit limited by the default routing choices and I don’t like the logging. For applications already in Python, I want to go straight to wsgi with either my own routing or a third party routing plugin, and using Python logging directly. I haven’t moved the other services yet, but I’m working hard on choosing a server for that.

I’ve also started using virtualenv, which is a Python specific virtual environment tool. Part of my reason for using a lot of zones is not just security, but also having a minimum install for each service (which helps keep track of what the dependencies are). For Python projects, virtualenv makes this dead simple. You create the virtualenv environment, you start it, then you pip install your dependencies inside the environment and they get installed there instead of in system site packages.

With using virtualenv to contain the project’s dependencies, and using a higher port so that root isn’t required at all to start the service, I’ve been starting to wonder if zones might perhaps be overkill? What separation do I need between a non-root process and it’s environment? Without root, I’m not sure we need network separation, and also without root do I need as much process separation?

I thought that perhaps sticking the virtualenv in a chroot might provide all the separation and security I need. However, when I looked into trying I found trouble. chroots wreaks havoc on module loading. I don’t see anything in virtualenv that would fix this. It occurs to me that if I put a new Python install (and pip) into each chroot, then I could use the chroot instead of virtualenv. This would take more disk space, but over all I suspect it would hog resources less than a full blown zone does.

I have further thoughts on that, but I need to actually test it. I doubt I will go that way. This isn’t what I want to spend all my time doing. I just want to have useful tools and make this easy on myself so I can work on more interesting things.

Currently, I’m mostly focussing on the Python side of things, but I want to reconsider how PHP apps are served as well. I only run third party PHP programs like WordPress and RoundCube.

Update: This list is being moved to a script on github. See the zone_service_cleanup.sh script here.

A new zone copies the base system in a lot of areas, including services. Thus, new zones can often be found running CDE. To save RAM and CPU power, those might as well be turned off. User svcadm disable on the following services:
svc:/application/font/fc-cache:default
svc:/network/rpc/cde-calendar-manager:default
svc:/network/rpc/cde-ttdbserver:tcp
svc:/application/graphical-login/cde-login:default
svc:/application/cde-printinfo:default
svc:/network/dns/multicast:default
svc:/system/avahi-bridge-dsd:default

Also wbem is the Solaris web management system. I have never used this, nor wanted to use it and I can’t imagine it being very useful. And it also starts on every single zone and appears to be a fairly heavy Java app. So that goes as well:
svc:/application/management/wbem:default

I have a feeling that I’m missing other services that I like to turn off, but that is why I’m starting a list here. Does anyone have additional suggestions?

When I first started using zones on Solaris, I ran into networking difficulty. I didn’t want private zone traffic polluting the network at the ISP, and I didn’t need any of the zones to be directly exposed (all traffic could either be proxied or go through ipnat). There was no nice solution for doing this, so I had to do something that involved turning on routing and doing weird things with the arp cache. I forget the specifics, and really, the only reason to remember them would be to do something similar on a different platform if forced to.

About 2 years ago, Solaris added Crossbow network virtualization system to the feature set. This is very nice, and extremely simple. For a bit machine, it would be possible to create virtual networks that only some zones can use and not others. For instance, a customer with 10 zones could have their zones talk among themselves but not to another customers zones. It also makes it possible to control network profile of zones, for instance rate limiting and applying a quota to them.

For a small installation like mine, it just makes it easy to do the right thing. In the root zone:

dladmin create-etherstub etherstub0

Now, etherstub0 is your new private network. To attach devices to that network, do:

dladmin create-vnic -l etherstub0 vnic0

I use vnic0 in the root zone, then configure vnic1-N for the other containers. If you do this before creating the zone, then in the zone config you just do this:

set ip-type=exclusive
add net
set physical=vnic6
end

For a zone that is already setup, you have to alter the config while it is stopped. But you will also need to plumb that zones vnic interface and create /etc/hostname.vnicN. Like wise, in the root zone, if you have a vnic into the virtual network there, you will also need to plumb and create the hostname file for that vnic.

For my server, I have a single IP address. So, what I do is have ipnat on the root zone forward ports to specific zones on the virtual network. If you have multiple IPs, you could use routing from the root zone to other zones. However, it would probably be more efficient to use a virtual private network for interzone traffic, and also give each zone a vnic attached directly to the main interface with a command like:

dladm create-vnic -l bge0 vnic22

I’m sure there are many other possibilities that I’m not aware of.