Lessons

Fri, 21 Sep 2012 15:09:00 GMT

I have a problem. It's not an unusual problem, and I wish to tackle it precisely because of its ubiquity. The problem is, roughly, this: I have a large chunk of code which does some blocking work. I wish to augment this code so that it can be used with Twisted, and I wish to do so in a way that satisfies the following conditions:

A minimum of source code is changed.

There are no deep hacks which are not trivial to explain when taken one by one.

The caller may use either the Twisted or non-Twisted interfaces at their leisure.

The difference between the Twisted and non-Twisted results at all of the borders of the module should be undone by maybeDeferred.

These conditions should not be difficult to achieve and yet they seem to constantly stymie many would-be IRC bot authors. I'm going to see if I can improve on this with a couple lessons from Haskell.

So, first, let's consider why we cannot simply remove data from Twisted interfaces. It's elementary: Deferred computations cannot have their results accessed directly. Instead, actions have to be lifted up into a Deferred, which will run the action when it is ready.

To a Haskell programmer, this sounds quite a bit like how IO behaves, and indeed, Deferred is a Functor (and Monad) that cannot be unwrapped. So, with this in mind, let's look a bit at what kind of interface this would be in Haskell. First, let's consider the type of our computation:

computation :: a -> b

That is, we are taking some data of type a and returning some data of type b. The trick here, for those of you not well-versed in Haskell, is that computation may not do anything outside of these types; it cannot perform I/O or have any impure effects. (Okay, fine, I mean, you can perform horrid hacks to do those things, but remember our rules: No deep hackery here.)

Now, let's consider the type of the Twisted-style computation.

deferredComputation :: Deferred a -> Deferred b

That is, we're taking a Deferred containing data of type a, and returning a Deferred with data of type b.

Now, here's the fun part. We want to generate deferredComputation from computation, to achieve code reuse. How? Well, let's use fmap, since Deferred is a Functor!

computation :: a -> b

fmap :: Functor f => (a -> b) -> f a -> f b

deferredComputation :: Deferred a -> Deferred b
deferredComputation = fmap computation

Hey hey! Pretty cool, right? This might seem super-obvious to people with lots of Haskell experience, but I think it's still worth repeating since not everybody has done this sort of thing before.

And now we return to the land of Python. Python-land. It's time to construct this thing in Python, as well. So, how do we lift a function up into a Deferred in Python?

def computation(a):
    return b

def deferredComputation(deferred):
    deferred.addCallback(computation)
    return deferred

Think about this for a second. Remember, Deferred objects carry state around with them, so we need this "impure" sort of approach, which is really not actually impure but just object-at-a-time. If you're unsure of exactly what this snippet's doing, go through it one bit at a time.

Take a Deferred which will fire with a value of type a.

Append a callback which transforms a into b.

Return a Deferred which will fire with a value of type b.

Now, let's make this concrete with an example. Let's say that we've got a system that has two implementations of a client, one which is synchronous, and one which is asynchronous. We've isolated and split out these clients such that they have exactly the same setup functions, and they return exactly the same data, with one single difference: One client is blocking and returns the data, and the other client is non-blocking and returns a Deferred which will fire with the data. This is exactly the difference that maybeDeferred can paper over. We've got all of the code set up just the way we want it, according to those conditions I listed earlier.

But! These clients only make up a couple dozen lines of code. There are still thousands of lines of code that only work with the synchronous client. How do we make them work with Twisted without losing our synchronous abilities?

Let's create some helper which will apply the computation to the data. This helper will come with the client and will be tailored to the client's output. For unlifted non-Twisted data, this is simply the classic builtin apply, known to Haskellers as ($):

($) :: (a -> b) -> a -> b
f $ a = f a

def apply(f, a):
    return f(a)

Note that my apply is not the Python apply builtin function, which does a slightly different thing if its argument is iterable.

And for the Deferred-handling case, let's create a slightly more interesting applier which will continue to move data through the Deferred. We already wrote this above, actually, and in Haskell, it would be fmap:

fmap :: Functor f => (a -> b) -> f a -> f b

def deferredApply(f, deferred):
    deferred.addCallback(f)
    return deferred

And now we're ready to put everything together! Here's a small skeleton:

class SyncClient(object):
    @staticmethod
    def applier(f, value):
        return f(value)

    def request(self, s):
        return sync_library_call(s)

class AsyncClient(object):
    @staticmethod
    def applier(f, deferred):
        deferred.addCallback(f)
        return deferred

    def request(self, s):
        return async_library_call(s)

def computation(data):
    transform(data)
    poke(data)
    return data

def request_and_compute(client, resource):
    data = client.request(resource)
    return client.applier(computation, data)

Look at request_and_compute. It has no idea whether it's handling synchronous or asynchronous data, and it doesn't really care; it asks the client to actually apply the computation to the data. And the computation itself is totally unaware of things going on around it. It doesn't even have to be pure; it could do all kinds of side effects with that data. The only requirement for the computation is that it must remember to return the data so that subsequent computations can access it.

This is the approach I'm taking in a new library I'm hacking together for Ganeti, called Gentleman. I think it'll work out well.

Be Prepared

Wed, 15 Feb 2012 20:34:00 GMT

I like Flask. No, really, I do. Yesterday, during a lightning talk, I claimed that I love it, and if I don't love it, then at least I love the form and function of it. I wouldn't marry it, since I don't think being married to a microframework for Web applications would provide any tax benefits. Maybe I'm getting off-topic?

Flask is built on Werkzeug, and directly uses Werkzeug routes for view lookup and URL building. As a result, anything that can be done to Werkzeug can be done to Flask. A little-known ability of Werkzeug is the ability to add new URL converters. An example and explanation is provided in the Werkzeug documentation on converters. I decided to build some cool converters which would automate some of the work I have to do when working with certain objects.

Without further ado, I would like to present ModelConverter, a class which can convert a segment of a URL representing a text field on a model into an instance of that model, and vice versa.

from __future__ import with_statement

from werkzeug.routing import BaseConverter, ValidationError

from sqlalchemy.orm.exc import MultipleResultsFound, NoResultFound

class ModelConverter(BaseConverter):
    """
    Converts a URL segment to and from a SQLAlchemy model.

    Rather than use an initializer, this class should be subclassed and
    have the `model` and `field` class attributes filled in. `model` is
    the Flask-SQLAlchemy model to use for queries, and `field` is the
    field on the model to use for lookups.

    The field to use should be Unicode or bytes.
    """

    def to_python(self, value):
        try:
            with self.app.test_request_context():
                obj = self.model.query.filter_by(**{self.field: value}).one()
        except (MultipleResultsFound, NoResultFound):
            raise ValidationError()

        return obj

    def to_url(self, value):
        return getattr(value, self.field)

This particular flavor uses an inheritance-based approach in order to avoid clobbering BaseConverter's initializer, but a compositional approach works too. A make_model_converter convenience method can provide the glue needed to specialize the converter. To apply it to the Flask application, merely modify the URL map after application creation:

from converters import make_model_converter
from models import Character

app = Flask(__name__)

app.url_map.converters["character"] = make_model_converter(app, Character,
    "slug")

And now you can create cool things along the lines of:

@app.route("/<character:c>")
def character(c):
    return render_template("character.html", c=c)

There is one caveat with this technique: the model instances retrieved this way will be detached from SQLAlchemy and the current session will not know about them. If you need to look up any lazily-loaded data on the models, you will need to add them to the current session first. For example, assuming Character.friends is a lazily-loaded one-to-many mapping:

@app.route("/<character:c>/friends")
def character_and_friends(c):
    db.session.add(c)
    return render_template("friends.html", c=c, friends=c.friends)

Today's snippets are all real-world snippets from DCoN, and can be seen in the converters.py and views.py source files.

Japanese Standing Cat

Tue, 31 Jan 2012 11:42:00 GMT

I recently purchased and installed a standing desk. While I normally don't blog about things in my personal life, I figured that this was permissible since it directly affects my ability to write code.

This standing desk is a Fredrik "computer work station," but let's be honest, here: it's a standing desk. It's pretty sturdy and removes the need to hack together various Ikea desks to produce reasonably-scaled tabletops. Pictured here is my desk, in its natural habitat. My workstation and hacked-apart AGP box both have their own monitor, and there is plenty of tabletop room for anything I need to have at my hands.

For those in the audience who aren't yet aware of standing desks, and don't want to read Wikipedia on standing desks, here are the abridged notes:

Pros

Reduces back stress and pain after a few months

Reduces risk of DVT

When monitors are elevated, reduces neck stress and pain

Cons

Slightly more expensive than sitting desks

Hard to find

Causes foot and ankle stress for the first few weeks

Turns one into a hipster programmer

So, on the bright side, this desk has definitely improved my back pain. But, if I start writing Node.js code soon, we'll all know who to blame.

A Little Bit is Better Than Nada

Sun, 22 Jan 2012 12:00:00 GMT

It is a running joke in the X.Org community that X12 development will commence as soon as all of the X11 bugs in the FreeDesktop.org bug tracker have been closed. When I first heard this, my reaction was, of course, "Challenge accepted!" But where to start?

There were about 10,000 bugs open on the tracker at that point, a few years ago. We're definitely improving; there are about 9180 bugs open (at the time of writing) which could be considered relevant to any kind of desktop development. Many of those don't belong to anything in X11, but assuming they did...

Pretend that all of the card-carrying members of X.Org closed 92 bugs. That would be 1% per person. We could be done with X11! So, during XDC 2011, I pulled out my netbook, and started closing bugs. I picked, as my targets, things that nobody would miss, like Xprint.

I closed some Xprint-related bugs, before simply shuttering everything in the Xprint product. I also picked up a couple of stray bugs. Finally, with maintainer permission, I closed everything still assigned to xf86-video-nv.

The results:

24 Xprint-related bugs

37 bugs in Product: xprint

42 bugs in Component: driver/nv

37 bugs in Component: driver/nVidia (open)

#9455, #28657, #29832, #30129

A total of 144 bugs. We can do this, guys!

Secrets

Tue, 29 Nov 2011 17:03:00 GMT

After accidentally hitting yet another window manager key combination this morning, I finally was fed up, and went off to read chromeos-wm source code to discover all of the key bindings. Here's the complete list:

Ctrl-F5: Take fullscreen screenshot

Ctrl-Shift-F5, Print Screen: Take region screenshot

Alt-Tab: Cycle windows forward

Alt-Shift-Tab: Cycle windows backwards

Alt-1 through Alt-8: Go to window 1 through 8, 1-indexed

Alt-9: Go to the last window

Ctrl-Alt-t: Create a new shell window

Ctrl-Shift-W: Kill the current window

These are only available in overlapping-window mode, which appears to be the default on at least Samsung Series 5 Chromebooks:

F5: Toggle overlapping-window mode

Alt-Comma: Expand current window

Alt-Period: Shrink current window

Poison Was the Cure

Mon, 03 Oct 2011 10:16:59 GMT

glyph and I met for the first time a few weeks ago, and ever since, he's been asking me to do things. Write code, review patches, write blog posts, that kind of thing. Being the lazy person I am, I'm not really up-front or speedy about doing things outside of my habitual box. Thankfully, I got inspired by a troll.

Everybody's read Ted Dziuba's fantastic anti-Node troll, right? It's good food for thought, even if it is a little bit belligerent. I wanted to demonstrate that Twisted doesn't have the same problems as Node. I also wanted to write a small application for Heroku, because I participated in their Python beta and figured out how to run Twisted applications on their platform.

So, I wrote a tiny application that does a few things. First, it runs on Heroku. It includes the Procfile magic for running on Heroku correctly, and only depends on Twisted. Second, it calculates Fibonacci numbers. The idiom that I chose to implement involves not caching the numbers, and instead using threads to do background work. I was going to do something highly idiomatic and Twisted-like, such as setting a timer and coming back to do the work later, but I wanted to show off a feature of Twisted which is completely missing from Node: The ability to write threaded code without leaving your language.

That's right, Node.js does not have the ability to host multiple threads in JavaScript. (Correct me if I'm wrong, but after a half-hour of research and an assertion I made on Hacker News which went unchallenged, I'm pretty sure that this is the case.) Why? I don't know. Possibly because Node's tied to V8, which is very shiny but certainly not as cool as the dominant Python implementations, like CPython or PyPy. Now, make no mistake, many Pythons have GILs or fine-grained locks, which can slow down threaded code, but Node can't even attempt to thread off its CPU-bound long-running computations, and I think that's an important ability to have. And as a well-known anti-threading advocate, when I admit that threads have their uses, I hope people notice.

I also got very annoyed at everybody who felt that they were demonstrating the benefits of Node by writing equivalent servers in Python using the standard library. Python's built-in servers suck. They straight-up suck. Would you deploy a web server written in pure JavaScript with only the standard library shipped with all JavaScript implementations? Probably not! Just like one needs Node to write those servers, Python developers use Twisted to provide that functionality. Otherwise, you're deliberately crippling your examples in order to skew impressions of Python. I'm going to shame you here, because you should know better. You should know about Twisted, if for no other reason than that it is listed on the front page of Node.js's homepage as one of the inspirations for Node. No, really, go check. I'll wait here.

The actual Heroku side of things is kind of boring. Heroku's Python support is fantastic. Use the cedar stack when you create your app with heroku apps:create --stack cedar and put a line into your Procfile which passes the $PORT shell variable into your app somehow. If you're deploying stock Twisted Web, something like bin/python ./twistd -n web --port=$PORT is a good starter. You can, of course, easily deploy WSGI apps on top of this using --wsgi. You must not daemonize on Heroku. That's really about it; it's a very straightforward platform which is easy to use and I'm very impressed by their work.

I have an application running on Heroku right now. The link is currently right here although it might be taken down in a few months when I need to experiment more. I'm a free customer, you see, and I'm not willing to pay a monthly fee just to prove a point about people on the Internets. Anyway, the application does nothing except check how many threads are running on the system, calculate a Fibonacci number, and tell you how long the request took. It can do this for many concurrent users; you might notice, if there are many active requests, that the page will stop loading halfway through as your request to calculate a Fibonacci number sits in line in a threadpool. Your actual browser connection won't be dropped, though.

The code is a demonstration of Heroku's cedar stack and a few fancier things in Twisted Web, like the new templating system added in Twisted 11.0. You can find it on Github; the repo is called "cancer" for obvious reasons.

Oh, and as a closer: Some people have complained heavily about the idiom that was demonstrated at the end of Ted's rant. The reason that he posted that is that JavaScript programmers, unlike Python programmers, never really got in the habit of not using names which haven't been declared or defined. The idiom in Python looks like this:

if my_var is not None:
    ...

We don't check to see if my_var in locals() or anything stupid like that, because my_var should be easy to track and the line where it was defined should be obvious. If it's not, you're writing bad code.

Theme of DCoN

Sun, 04 Sep 2011 13:26:24 GMT

Some of you may remember the "Release Date: 2011" meme which we've been spreading for the past two years. We are proud to announce an actual piece of DCoN software, now available for your use.

Presenting DCoN, the Doubly-linked Comic ORM for Newrem. This is the new webcomic management system powering Newrem. It's young, but already it's powerful enough to fill the shoes it was designed to fill, and perhaps other people will find it useful as well.

We are releasing DCoN under the GPLv2, in accordance with obligations laid out by one of the component libraries. We hope it will be instructive for others looking to create or deploy webcomic management systems.

Please note that this software is, as people often say in the business, "brand-spankin'-new," so don't be surprised if there are still a few bugs left to hammer out.

Nobody Gonna Take My Car

Sat, 13 Aug 2011 15:22:29 GMT

Hi! Just a quick status update on that crazy Chromebook of mine.

To get VBA-M to work, use the GTK+ frontend by emerging with USE="gtk". Set it however you like; I highly advise against setting it globally, but if you have sgt-puzzles installed, you will already have GTK+. Afterwards, you will have a shiny new gvbam executable, ready to use.

While ZSNES might be very unhappy, Snes9x can correctly accept input. Well, to a point, at least; sound output and screen scaling appear to be broken. More research is needed.

Highway Star

Tue, 09 Aug 2011 02:03:29 GMT

I have a Chromebook. Specifically, a Samsung Series 5, courtesy of Google. It's pretty snazzy, but what's really great is that the "Chrome OS" which it comes with is actually a modified Gentoo Linux. Cool, right? Unfortunately, the Chromebook is a bit too hobbled for generic Gentoo usage, but I will outline the steps I have taken towards getting mine to a point where I'm happy with it.

When modding hardware, especially hardware over which one has limited control, it's good to set a goal, so I'll outline my goal in terms of what I usually do on traveling laptops: Games and music. Since there's usually a pretty heavy limit on disk space, but plenty of bandwidth, I want to be able to access some kind of music stream, and as far as games, I'll go with a selection of small puzzles as well as classic emulators. In particular, I want to have Pianobar (media-sound/pianobar), Simon Tatham's Puzzles (games-puzzle/sgt-puzzles), and either VBA-M (games-emulation/vbam) or ZSNES (games-emulation/zsnes).

First things first: Let's get access to the shell. Two keyboard commands to know (which have no mouse equivalents!) are the classic Alt+Tab, for switching between windows, as well as Alt+Shift+Tab for going through the window list backwards, and also the common Ctrl+Alt+T, which spawns a shell. On a Chromebook, the shell provided by default is "crosh", a remarkably useless shell mostly provided for diagnostics. Crosh has a secret command, "shell", which spawns a real shell, bash! However, we can't get to it until Developer Mode has been enabled. After Developer Mode is enabled, crosh will now permit us to get a real shell, and the real fun can begin.

As a user on the filesystem, we are always under the login "chronos". We can sudo freely and without limit; invocations like sudo -i to gain root shell access work as expected. Let's take a look at what's in the filesystem at the moment.

(Aside: If you just want to know how I got stuff working on my Chromebook, skip the next few paragraphs; I'm going to talk about my though processes and the current state of what I know about the machine. Interesting stuff, perhaps, but not really directly relevant to getting up and running.)

chronos@localhost ~ $ mount
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (ro,relatime)
devtmpfs on /dev type devtmpfs (rw,size=970408k,nr_inodes=242602,mode=755)
none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/tmp on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)
udev on /dev type tmpfs (rw,nosuid,noexec,relatime,mode=755)
shmfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
/dev/sda1 on /mnt/stateful_partition type ext3
(rw,nosuid,nodev,noexec,relatime,errors=continue,commit=600,data=ordered)
/dev/sda1 on /var type ext3
(rw,nosuid,nodev,noexec,relatime,errors=continue,commit=600,data=ordered)
/dev/sda1 on /home type ext3
(rw,nosuid,nodev,noexec,relatime,errors=continue,commit=600,data=ordered)
varrun on /var/run type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
varlock on /var/lock type tmpfs (rw,nosuid,nodev,noexec,relatime)
/media on /media type tmpfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
cgroup on /tmp/cgroup/cpu type cgroup (rw,relatime,cpu)
/home/.shadow/d72e888592600e3025a778270172192458d0a039/vault on
/home/chronos/user type ecryptfs
(rw,nosuid,nodev,relatime,ecryptfs_sig=6c1fcef0779bc58d,ecryptfs_fnek_sig=997328e527b573bc,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs)
chronos@localhost ~ $ df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                837M  548M  290M  66% /
/dev/root             837M  548M  290M  66% /
devtmpfs              948M  472K  948M   1% /dev
/tmp                  948M   24M  925M   3% /tmp
udev                  948M  472K  948M   1% /dev
shmfs                 948M  256K  948M   1% /dev/shm
/dev/sda1              11G  3.4G  6.7G  34% /mnt/stateful_partition
/dev/sda1              11G  3.4G  6.7G  34% /var
/dev/sda1              11G  3.4G  6.7G  34% /home
varrun                948M   68K  948M   1% /var/run
varlock               948M     0  948M   0% /var/lock
/media                948M     0  948M   0% /media
/home/.shadow/d72e888592600e3025a778270172192458d0a039/vault
                       11G  3.4G  6.7G  34% /home/chronos/user

What a list of mounts! I'll try to explain. The rootfs and kernel each live on partitions not mounted during normal operation, and are generally reasonably protected. Modifying them is beyond what I will discuss today, but rest assured that they are there and modifiable if developer-mode BIOS is enabled. (All the gory details are on the Chromium OS wiki, linked above.) If we want to avoid making changes which can be easily wiped out, then we will have to consider an alternative approach to modifying the rootfs. There is apparently a way to cook an entirely-new Chromium OS image, but I'm going to avoid that for a pair of reasons:

The build process takes a fair amount of space and time

The image has to be respun for every update

These two things wouldn't be that big of a deal, if I knew exactly what I wanted ahead of time and didn't plan to update more than every three months or so. However, knowing the speed of Gentoo, this isn't going to be reasonable. Moreover, there are a few things missing from the Chrome OS rootfs: Portage and a compiler, to say nothing of Python. There aren't even the tools for bootstrapping a stage3. Also, look at how much space is allocated to the default rootfs: 840MiB, nowhere near enough to get anywhere when it's already two-thirds full. An alternate approach sounds better.

Can't we just wipe the entire SSD and put a standard Gentoo install on it? Well, no. There's a couple obstacles. First, how does one get into a live Gentoo environment? We can't boot from USB or MMC/SD into a non-Chrome OS environment, thanks to a verified bootloader. We can't access the deepest bootloaders easily; while an EFI loader is shipped onboard, it appears to be unused, and documentation points to a signed kernel loader being used instead. This is frustrating, because it means that we have to use the same kind of partitioning as the current system, and we have to sign our kernels. While the tools are all there in the Chromium OS repositories, I didn't feel like going through the ardour of figuring out how to put it all together and flashing my device multiple times to get it all going. (I'm lazy like that; if some enterprising developer wants to do it, please, go ahead!) So, we'll stick to userspace modifications.

The next thing I noticed: chroot is available in the rootfs! This could get us into a Gentoo environment! But what would we do in there? We can't install things directly onto the rootfs or stateful partition, and we can't change the partition table. Either one would wreck the Chrome OS boot pretty thoroughly. But, we could always go the brutish, lazy route, and simply have a chroot in our home directory. Getting at our Portage and compiler would be as easy as invoking chroot, and we could still have the rest of the system (Chrome, WiFi manager, battery monitor, terminal emulator, window manager) at our disposal. Excellent.

How much disk space would this take? It sounds wasteful. As you may have noticed from the above filesystem readout, though, it only adds about 2.8GiB for an entire chroot with everything I like to have in my minimal Gentoo systems. I'm a bit of a pig, but even so, it's possible to have a very slimmed-down chroot.

So, let's get started. First, create your chroot directory in your home, calling it "chroot" or something like that. You can go ahead and be root for this. In fact, get comfortable being root; you will always be root in your chroot. I mean, seriously, why not? There's all the traditional reasons to not be root, but the Chrome OS is a brand-new world, a world where people can be root inside a chroot inside a cryptfs inside a single-user system inside the cloud.

You should have the Gentoo Handbook's Chroot Guide open while you do this, as what we are doing is quite similar. The one catch is that you must be root in order to create all the device nodes. Once your chroot is in place, the following script will enter the chroot: https://gist.github.com/1128864 Just put it in your home directory, chmod +x it, and we're almost ready to go!

Oops, wait, what's with this cryptic "bad interpreter" error? For security reasons, most filesystems are mounted noexec. Sadly, there's no good place to put a shell script to turn it off, so I instead encourage you to memorize mount -i -o remount,exec ~. It rolls right off the tongue! You only have to do this once, at boot, but you can't put it in the rootfs without alerting Chrome OS to our shenanigans. If Chrome OS gets ticked enough at our meddling, it will destroy the entire stateful partition, including the chroot. (If you want to keep your chroot on a USB drive or SD card, I would totally understand!)

There are a few more things required to get all of the things in the chroot working. Presumably you will eventually want to run applications which talk to the local X11 server; for that, you will need to add a single line to your .bashrc inside the chroot:

export XAUTHORITY="/root/.Xauthority"

So, now that we're inside our chroot, and have everything wired up, it's time to start experimenting. I've gone ahead and selected the following USE flags:

USE="X alsa ao mmx opengl sse sse2 sse3 -cups -xscreensaver"

And here's the package list in my world file (/var/lib/portage/world):

app-admin/localepurge
app-editors/vim
app-misc/screen
app-portage/eix
app-portage/genlop
app-portage/gentoolkit
app-portage/ufed
games-emulation/vbam
games-emulation/zsnes
games-puzzle/sgt-puzzles
media-sound/pianobar
net-im/pidgin
x11-apps/mesa-progs
x11-apps/xauth
x11-apps/xdpyinfo
x11-apps/xhost

You may notice that I emerged things I don't technically need. This is because I am a creature of habit, and several things nearly always make their way into my list, like vim, eix, genlop, gentoolkit, and ufed. I emerged all of the X and GL programs to help debug X issues; they are not necessary at all for you as long as you have set up .Xauthority as I have.

You may also notice that I grabbed localepurge. I highly recommend trimming the fat from your chroot with localepurge; that and trimming the categories from your Portage tree (do you really need sci-*?) are going to save around 500MiB if done zealously.

With the current setup, pianobar Just Works. sgt-puzzles installs its binaries as games, so you will have to make sure that, when chrooted, your user has the games group set. This is the purpose of that mystical --groups=35 on the chroot invocation -- 35 is the GID of the games group! This is definitely the cleanest solution. With that added, sgt-puzzles works. Two of four.

Summertime

Sun, 31 Jul 2011 23:47:12 GMT

Some people use summer vacation to do useful things. I have been using my summertime to demonstrate how awesome Twisted is.

Here's a fun example: Take this basic long-polling example, in gevent, and turn it into a Twisted-based server instead.

First, let's look at the chunk of code that will communicate with the web browser:

class Pusher(Resource):

    isLeaf = True

    def render_GET(self, request):
        d = cc.connectTCP("localhost", 6379)
        @d.addCallback
        def cb(protocol):
            protocol.request = request
            protocol.subscribe("messages")

        request.write(" " * 4096)
        request.write("<!DOCTYPE html><h1>Messages!</h1>\n")

        return NOT_DONE_YET

Pretty nifty, right? This is a straightforward Twisted Web resource. It will respond to GET requests by writing a small banner, "Messages!" and will leave the connection open via NOT_DONE_YET. It also opens a connection to some local server via the cc object, which I will explain momentarily.

Now, let's look at the Redis client. This is my first time ever using Redis, and also using txRedis, but I think I did alright.

class Puller(RedisSubscriber):

    request = None

    def messageReceived(self, channel, message):
        if self.request and not self.request.finished:
            self.request.write("<div>Message on %s: '%s'</div>\n"
                % (channel, message))
            if message == "quit":
                self.request.finish()

        if message == "quit":
            self.transport.loseConnection()

What does this do? I'm not super-sure, but it's pretty self-explanatory, thankfully. RedisSubscriber is a protocol which can subscribe to, and publish, messages on Redis. Only the messageReceived method needs to be overriden, and we just use it to send messages to that request object.

Here's the part where people might get lost. How are these two objects hooked up? Well, the answer (as some of you Twisted veterans may have guessed) is the venerable ClientCreator, which creates instances of Puller for every GET request made in the Pusher. Now, hopefully, the first half of render_GET() makes more sense: The ClientCreator is asked to connect to the local Redis server on port 6379, and will return a Puller in the callback. The Pusher then hands the request over, and the Puller will relay any Redis messages it picks up into that request. Everything else is plumbing or sanity stuff; of note are the 4KiB of empty space at the beginning of the request, which convinces browsers to start rendering the page, and also the logic in messageReceived for closing the request and transport properly.

Here's the entire thing, imports and all. Note that it's only one line longer than the gevent version, and comes with a bunch of nifty free features like PyPy compatibility.

from twisted.internet import reactor
from twisted.internet.protocol import ClientCreator
from twisted.web.resource import Resource
from twisted.web.server import Site, NOT_DONE_YET

from txredis.protocol import RedisSubscriber

class Puller(RedisSubscriber):

    request = None

    def messageReceived(self, channel, message):
        if self.request and not self.request.finished:
            self.request.write("<div>Message on %s: '%s'</div>\n"
                % (channel, message))
            if message == "quit":
                self.request.finish()

        # Ugh, Venn logic.
        if message == "quit":
            self.transport.loseConnection()

cc = ClientCreator(reactor, Puller)

class Pusher(Resource):

    isLeaf = True

    def render_GET(self, request):
        d = cc.connectTCP("localhost", 6379)
        @d.addCallback
        def cb(protocol):
            protocol.request = request
            protocol.subscribe("messages")

        request.write(" " * 4096)
        request.write("<!DOCTYPE html><h1>Messages!</h1>\n")

        return NOT_DONE_YET

reactor.listenTCP(1234, Site(Pusher()))
reactor.run()