Beware of the Train

Lazy lists, impatience and hubris

Fri, 16 May 2008 13:36:47 GMT

There was a recent mailing list post by Andrew Coppin (which received some discussion on programming.reddit) in which he bemoans the way that GHC, when presented with reasonable-looking code, can suddenly and mysteriously decide that it needs to allocate gigabytes of memory, slowing your system to a crawl for ages (I've had similar problems myself). Andrew's example was the following:

I offer up the following example:
mean xs = sum xs / length xs
Now try, say, mean [1.. 1e9], and watch GHC eat several GB of RAM. (!!)

This is not so mysterious when you remember that Haskell uses linked lists as its native list structure, and so the only way to calculate length xs is to traverse the list a second time: the runtime system "helpfully" keeps around the copy of xs that it generated while calculating sum xs, and you have a space leak. But what to do about it?

Andrew suggests the rather eye-watering

mean = (\(s,n) -> s / n) . foldr (\x (s,n) -> let s' = s+x; n' = n+1 
    in s' `seq` n' `seq` (s', n')) (0,0)

Stare at that for a while, and let your eyes de-focus: after a while, it becomes apparent that he's fused the two traversals of the list into one. All that seq business is, er, something to do with strictness, I dunno. Tilton's Law of Programming applies here. Anyway, Don Stewart's written a good article showing how to get to the fast version of the code from the slow version, including some handy tips on reading intermediate compiler output to see what's really going on. It's a bit disingenuous, though - he shows that allocating the list strictly will consume over 8GB, and concludes that "Strictness is not the solution here", whereas from what I can see, this is the only part of the program where strictness doesn't help! It appears that recent versions of GHC do a lot of strictness analysis when given the -O flag, and here that's enough. But there's a lot of good stuff in there, and I shall refer back to it next time my Haskell code performs surprisingly (although I must say, the last time I heard of anyone inspecting intermediate compiler output to get adequate performance was in the late 80s on early Ada compilers...).

Anyway, that's not what I wanted to talk about. As part of a very silly discussion prompted by Andrew's post, someone challenged me to "provide, in the strict language of [my] choice -- any language at all, as long as it is strict -- a function that calculates the mean of a list of doubles that correctly produces the mean of [1 .. 10e9] in constant space".

Now, one could argue that 8GB is a constant amount of space (albeit a large one), and write a version that simply allocated the list directly. But that would miss the point, because the challenge is actually very easy. Sure, if you want to consume a sensible amount of memory and yet retain genericity, you're going to need to generate the list lazily. But what my challenger had forgotten is that one can easily hand-roll lazy data-structures in any language with lexical closures or, come to that, objects. Which is to say, in any reasonably modern language. I chose Perl, because it's the strict language I know best:

    sub mean {
            my $getnext=shift; my ($next, $done, $sum, $count);
            while (!$done) {
                    ($next, $done) = $getnext->();
                    $sum += $next; $count++
            }
            return $sum/$count;
    }
    my $i = 1; print mean( sub { ($i++, $i>1000000000); } );

Now, that code's by no means fast: in fact, it took over half-an-hour to complete on my girlfriend's aging laptop. But it fulfilled the "constant space" part of the spec admirably, consuming about 1.5MB consistently (almost all of that for the Perl interpreter).

What's going on there? Well, instead of accepting a list, my mean function accepts an iterator: a closure that, when called, returns a pair containing the next element of the list and a boolean saying whether we've reached the end yet (the keyword sub, as well as declaring named functions, creates anonymous coderefs which can access variables in their surrounding lexical scope: in effect, it's Perl's lambda). This iterator may not look very listlike at first glance, but it does the same thing: conceptually, therefore, it's a list. Someone, probably Duncan Coutts, once suggested to me that the best way to think about Haskell's lazy lists is as forward-only iterators over collections: implementing a toList function for your new data structure is conceptually just like building an iterator, as the next part of the list is only built when it's asked for (which is equivalent to calling next() on an iterator in Java or C++). I've found this approach very helpful.

This (next_elt, done?) approach isn't the only way of making lazy lists, but it's a perfectly sensible one. One could generalise the way we've created the list (1..1e9):

sub make_stream {
    my $i = shift; my $upper = shift; my $step = shift || 1;
    $i -= $step;
    return sub { $i += step; ($i, $i >= $upper) };
}

or wrap an honest-to-God Perl list:

sub stream { my $i; my @xs = @_; sub { ($xs[$i++], $i > $#xs) }
mean(stream(1,2,3,7, -32.6 10**3.7))

The general form is

# set up lexicals here
my $stream = sub { return (
    # calculation to get next element; probably mutates state
    ,
    # are we done yet?
    )
}

As funktio suggested, you'd probably want to wrap the lexicals in either a do-block or a function like make_stream above.

What else could we have done? Another option would have been to create an object with next and end methods, and store any necessary state as a member variable of the object. The most standard solution would have been to create a coderef that returned (car, cdr) pairs, where car is the next element of the list, and cdr is another coderef which, when evaluated, returns the next (car, cdr) pair. This, incidentally, is how Haskell's laziness works under the hood: unevaluated values are stored as closures called "thunks", which, when called, return the value you want. If you never need the value, the thunk is never called, and so the calculation's never done. In Perl, this looks like:

sub range {
    my $lower = shift; my $upper = shift; my $step = shift || 1;
    my $test = ($lower >= $upper, 1, $lower <= $upper)[($step <=> 0) + 1];
    my $cdr = $test ? sub { range($lower + $step, $upper, $step) } : undef;
    return ($lower, $cdr);
}

sub range_to_list {
    my ($car, $cdr) = @_;
    my @list;
    while (defined($cdr)) {
        push @list, $car;
        ($car, $cdr) = $cdr->();
    }
    return @list;
}

In practice, you should use one of the lazy list modules on CPAN. I've never used any of them, but Tie::LazyList, which lets you use lazy lists like they were ordinary Perl lists, looks good.

We're almost done, but there's something else I'd like to look at. What would all this look like in Ruby? Well, one could easily do what we've just done: Ruby's anonymous blocks fulfil the same role as Perl's coderefs. But the more natural approach, I think, is the reverse of ours: You'd define a Range class which has an each method, and then define mean by passing a callback to each:

def mean(lst)
    sum = 0
    count = 0
    lst.each do |x|
        sum += x
        count += 1
    end
    sum / count
end

Then each could provide an iterator in any way appropriate, which probably wouldn't keep the list in memory. Even if we did two traversals of the list, we wouldn't run out of RAM. As a bonus, our mean function now works on any object which implements each, so it's approximately as general as the Haskell version. By the way, does anyone know how Ruby's ranges work internally?

So, the take-home messages:

While laziness is cool, it's not actually that magical, and you don't have to be using Haskell to take advantage of it: you can implement your own lazy data-structures in a few lines of any modern language. What Haskell brings to the table is pervasive laziness, so all the standard functions will work as-is with lazy structures, and dividing code into producer/consumer pairs is easier.
Perl has full lexical closures, and all that follows from them.
[If you haven't had a look at Mark Jason Dominus' book Higher-Order Perl yet, then now would be a good time. It's a free read online, and also available in dead tree format. And be sure to read the lovely story associated with the cover.]
A good way to think of lazy lists in Haskell is as forward-only iterators.
Callbacks Are Your Friends, and widely-supported APIs (like Ruby's Enumerable API) are also your friends.
If you're doing anything numerical with Haskell, you probably want to be giving the -O or -O2 flag to GHC!

Sage

Wed, 14 May 2008 12:25:24 GMT

Those of you who do scientific computation may be interested in this link:

Bye Matlab, hello Python, thanks Sage.

Short version: Sage is a bundle of all of the various numeric/scientific/graphing tools for Python, made easy to download and install (even if you don't have root access, apparently). According to the blogger linked above, it's now good enough to serve as a replacement for Matlab (and it has ambitions to be a replacement for Maple and Mathematica too, though I don't know how far along it is). It integrates with R, Gap, etc. And because it's Python, you get a real, high-level, well-designed programming language with a proper environment to do your coding in.

In other news, wormwood_pearl finished her Finals yesterday. As you can imagine, we're both pretty relieved. There's no tradition of meeting people outside Finals here like there is in Oxford, but I went along with a bottle of champagne anyway - and only then remembered that public drinking is against Glasgow bylaws. Drat it. We went out to lunch, the bottle went into the Department fridge, and we subsequently went round to her sisters' and drank it there while watching Layer Cake. Because we're that exciting.

Notation that sings

Fri, 09 May 2008 16:58:42 GMT

Some Actual Maths for a change. Some of it assumes some mathematical sophistication, but the core idea should be accessible to anybody. Have you ever been an arts student? I'm looking at you. Skate over the words you don't know, they're not important - they're just there to explain how the central result connects to more sophisticated and deeper areas of mathematics.

Over at the Comonad.Reader, Edward Kmett's been writing about so-called "abiding" pairs of binary operations: we say that * and + abide if (a+b)*(c+d) = (a*c)+(b*d) for all a, b, c and d. The term's a portmanteau of "above and beside", and comes from the following rather lovely notation:

a|c = (a*b)+(c*d)
-|-              
b|d              

a|c = (a+c)*(b+d)
---
b|d

We write * by stacking things vertically, and + by putting them side-by-side: then + and * abide if we can remove the lines without introducing ambiguity. By the way, + and * don't necessarily stand for ordinary addition and multiplication - in fact, they can't, because addition and multiplication don't abide! Just think of them as two machines: you feed two things into either of them, and get one thing out of the other end. We require the things we get back to satisfy the equation above (and yes, there are examples).

Why would we want to consider such a thing? Well, suppose (A,+,0) is a monoid: then one of the conditions we need for * to be a monoid homomorphism AxA -> A is that * and + abide. Generalise that example as far as you wish :-)

Though neither Edward's post nor the thesis he links to say so, this notation is related to the rather beautiful Eckmann-Hilton argument. I tried to say as much in a comment, but the proof works best in two dimensions, and his comment form strips <pre> tags. Fortunately, however, I have no such restriction here :-)

Theorem (Eckmann-Hilton): Let +, * be a pair of abiding binary operations on a set A, with units 0 and 1 respectively (so a+0 = a = 0+a and a*1 = a = 1*a for all a). Then 0 = 1, + = *, and + (and hence *) is commutative.
Proof:

0 = 0|0 = 1|0 = 1|0 = 1 = 1
          -|-   ---   -
          0|1   0|1   1

x|y = 1|y = 1|y = y = y|1 = y|1 = y|x = 1|x = 1|x = x
      -|-   ---   -   ---   -|-         -|-   ---   -
      x|1   x|1   x   1|x   1|x         y|1   y|1   y

QED.

You'll note that this chain of equations can be continued so that it loops round back to x+y again. If you do that, and arrange the stages in the proof in a circle, you get the "Eckmann-Hilton clock".

This, by the way, is the kind of thing I mean when I say I do "higher-dimensional algebra": some arguments naturally live in more than one dimension. In 2-d we're OK, because we can draw pictures on paper, but 3-d and higher rapidly gets hard to think about.

The Eckmann-Hilton argument originally arose in algebraic topology: it's how you prove that the higher homotopy groups π₂, π₃ must be commutative (the notation now has a geometric interpretation: we can literally lay out homotopies and morph them around each other as in the proof). In category theory, the argument is widespread, and it's related to the Stabilization Hypothesis (scroll down until you see the table, then read the paragraph or two above for the gist). A further example is the theorem that a monoid object in the category of monoids is a commutative monoid (because, as mentioned above, a monoid homomorphism AxA->A must abide with the binary operation on A).

I first encountered the Eckmann-Hilton argument in Joachim Kock's excellent book Frobenius algebras and 2D topological quantum field theories, which despite the title is a great introduction to the categorical way of thinking. It's aimed at beginning graduate students, is as easy to read as a maths textbook will ever be, and comes highly recommended. It's also interesting for the way the book's structured: it uses the bottom-up style currently in fashion in programming circles. The book introduces a series of "languages", each defined in terms of the previous one, which approach successively closer to the material in question. By the end of the book, we're able to replace literally pages of calculation with a few simple diagrams, and it's all perfectly rigorous. Better than that, the graphical language makes proofs extremely easy to construct. The main result of the book's pretty cute too :-)

LOLhacker?

Thu, 08 May 2008 13:24:55 GMT

I know I'm very late to the LOL$thing party, but what the hell:

moar allegedly funny pictures

moar allegedly funny pictures

Now, I reckon about half of you will recognise the guy in the pictures, and about half will understand the captions. I'm interested to know how large the intersection is...

Thesis sitrep

Wed, 07 May 2008 22:52:03 GMT

I've been collecting some more data on the thesis:

Wed May  7 23:45:49 BST 2008
4741 lines 23246 words 163656 characters
thesis.log:Output written on thesis.dvi (84 pages, 643096 bytes).
33 fixmes
80 definitions
141 term-definitions
24 theorems
21 lemmas
6 corollaries
30 examples
143 bullet-points

A "term-definition" is a call to the macro \defterm, which I use to emphasize the term being defined. The count's not the same as the number of definitions because I often group the definitions of related terms into the same \begin{defn}...\end{defn} block. The lemmas, theorems etc are mostly other people's :-(

I like bullet points. They reflect how I think, and bullet-pointed text takes up more space on the page.

In case you were wondering how I got this way...

Wed, 07 May 2008 22:43:36 GMT

The tail-end of a conversation with my Dad last night:

[A discussion of scary-sounding nonlinear dynamical techniques my Dad has been studying, with Poincaré's name attached for extra scariness]
Me: Sounds interesting. Have you come across the idea of considering the Poisson bracket as a symplectic form on the cotangent bundle of phase space?*
Dad: No, I don't think so...
Me: Something like that, anyway. I went to enough lectures on this stuff to pick up the jargon, but not enough to really get my head around it.
Dad: Yes, I know that feeling.
Me: But it's interesting how quickly physics becomes geometrical, isn't it?
Dad: Yes, certainly. You know, one of these days you and I should put our heads together and try to properly understand General Relativity.
Me: Actually, that was my plan for after I hand in my thesis. That, and learning to ride a unicycle.
Dad: [laughs] ...and learn a foreign language and a musical instrument.
Me: thinks: I wasn't going to tell him about that bit...

* My office-mate, who does The Physics, tells met that I meant "configuration space" - phase space is the cotangent bundle of configuration space!

Fine Structure: The Story So Far

Sun, 04 May 2008 18:04:37 GMT

There's a new Fine Structure story up, called The Story So Far. Go read it. Actually, there's another one that I never got round to posting about, called Failure Mode, which is... rather moving, actually. Great stuff.

Anyway, the missing story has now been revealed, and I was right! Ph34r my l33t story-prediction sk1llz!!!

Perhaps most interestingly, this (and Sam's announcement in the comments section of TSSF) mean that Being God is a Big Responsibility isn't part of FS. So we can rule out all the theories about the world being a quantum simulation. Though something of the layered universe concept seems to have survived.

I'm not going to write too much now, because I have a hangover. But I think there'd be some benefit to thinking harder about the fictional physics - I definitely should have spotted the "superlight" connection. I think there may be more insights to come from that source.

There's now a comments system on the Fine Structure pages themselves, which I suspect will largely replace the discussions here. Please continue to comment on my posts if you want, though! I'm hoping someone else will start blogging in depth about FS. Maybe we need a Fine Structure wiki? An E2 node?

Floreat gens togata

Sat, 03 May 2008 19:38:00 GMT

I see that Londoners have chosen the devil they don't know in place of the devil they do. I shall observe my fellow KS's future performance with considerable interest, and at a distance of several hundred miles.

In other news, my flatmates are looking to move to Oxford in the next few months (Nicola's starting an MSc) - can anyone offer some hints on Oxford flathunting? I was a wuss, and stayed in college accommodation for all four years, so can't be too much use.

Mountain photos

Wed, 30 Apr 2008 03:07:28 GMT

I've put a selection of photos from my last-but-one hiking trip on moblog: part 1, part 2, part 3. We went to Glen Etive (south and a bit west of Glen Coe), and climbed Ben Starav and Beinn nan Aighenan - the plan had been to do more, but we underestimated the time it would take and overestimated our fitness levels. It was an absolutely stunning day, and the scenery was just great. A sample:

By the way, my thesis page-count has finally drawn level with my Munro-bagging count:

Wed Apr 30 04:06:10 BST 2008
4552 lines 22350 words 156883 characters
thesis.log:Output written on thesis.dvi (82 pages, 628596 bytes).
34 fixmes

Yay!

This is a test

Tue, 29 Apr 2008 15:44:16 GMT

=pod

... do not be alarmed.

What happened was that I was talking on Reddit with Masklinn about Literate Programming, and s/he wanted to know if there were any systems other than Literate Haskell which allowed for blog posts which are valid code. "Hrmmm," I thought, "I don't see any obstacle to doing that with POD..."

[Yes, my inner monologue does contain hyperlinks. :-) ]

Literate Programming, by the way, is a workaround for the limitations of early Pascal compilers encountered by Donald Knuth when he was writing TeX. The bit everyone remembers is about intermingling code with a description of that code - a program called weave would then produce TeX source, which could be compiled into a beautifully-typeset manual, and another program called tangle would produce highly portable, lowest-common-denominator Pascal source, which could then be compiled into a runnable program. The original WEB system had many other features to enable higher-level programming without compromising portability: most of these are IMHO considerably less necessary with modern languages, and many of the rest are handled by systems such as Doxygen. Anyway, Literate Haskell handles the "intermingling code with docs" bit for Haskell, and POD does a similar job for Perl. Mark Dominus has written an article explaining where POD falls short of Knuth's WEB system (and of Knuth's conception of Literate Programming) - as far as I can see, most of it applies to Literate Haskell too. Knuth's original paper is, well, written by Donald Knuth - obviously it's worth reading! :-)

Anyway, having fulfilled Criterion 1 of Literate Programming (start with a long paragraph explaining what LP is and what it's for, and only then reveal that - shock, horror! The document you're reading is in fact a program!), I should get on with some actual coding.

=cut

print "Hello, Literate World!\n";

=pod

Did'ja see that? I totally embedded some code right in my blog post! Wow. Note that this was the classic "Hello, World" program - by tradition (which in hacker circles has a force greater than physical law and only slightly less than mathematical inevitability), this is the first program you must write with any new system.

Let's do something a bit more interesting:

=cut

while (<>) {
        s/!+/./g; print;
}

=pod

That accepts some text on standard input (and/or any files specified on the command line), and replaces all sequences of exclamation marks with a single full stop. The result is then placed on standard output. I used to have that as a filter on my incoming mail - we had a very excitable JCR committee, who used to send out lots of emails of the form "Massive event happening tonight!!!!!!! Everyone must come!!!!!".

But, uh-oh, there's a problem - see where it says while (<>)? That's actually while (<>), which isn't valid Perl. But if I didn't do that, your browser would think I'd included an empty tag, which would disappear silently when your browser tried to render it - not very helpful. The upshot is that you can't just save this HTML file as witteringmoron.pl and expect it to work - you'll have to cut-and-paste the body of the post (being sure to get the =pod at the top) and save it separately. Sorry about that. But actually, you needed to do that anyway - the interpreter expects to start off in Perl mode, not POD mode, and all the HTML header gubbins isn't valid Perl.

Anyway, let's try it. Copy... paste into a new text editor window... save as witteringmoron.pl... now, open a command-line and type perl witteringmoron.pl:

Hello, Literate World!

Excellent. Now, type some sample input:

This is, like, totally a test!!!!! ZOMG!!!!!!

This produces the output

This is, like, totally a test. ZOMG.

Which is correct, albeit somewhat deflating. You can stop the program from asking for more input by typing Ctrl-D.

So, yeah, you can write blog posts in POD, subject to the above limitations. Woo!

Right, back to work.

Why I like Perl

Sun, 27 Apr 2008 23:39:32 GMT

I wrote this in an email to a colleague of totherme's a while back, and have occasionally felt the need to refer to it since. Fortunately, I have a place where I can post my ill-informed ramblings for all the world to see, so here it is :-) It's an attempt to convince a computer scientist, who I believe comes from the Haskell/static typing camp, why Perl is worth learning.

[If you're not a computer scientist, the two chief reasons to learn Perl are (1) it's incredibly useful, (2) it's great fun. But I digress.]

In many ways, Perl occupies the opposite corner of design-space to Haskell. To a first approximation, Haskell proceeds from the question "what if everything we think we know about language design is right?", whereas Perl proceeds from the question "what if everything we think we know about language design is wrong?" Consequently, learning Perl is essential for any student of programming language design. It's not so obviously revolutionary now, because some of Perl's design decisions have since become accepted into the conventional wisdom, but there are still some surprises.

The original author, Larry Wall, was trained as a linguist; consequently, Perl takes inspiration from human languages more than it takes inspiration from mathematics (it's one of the few programming languages with pronouns, for instance). It also has no hesitation about borrowing good features from other languages: built-in hashes from awk, regular expressions from SNOBOL, Python's OO features, Lisp's closures, C's syntax, Ada's module system. This leads to a lack of consistency, but consistency is not rated very highly in the Perl community (There's More Than One Way To Do It). The default assumption is that the programmer (a) is competent, (b) uses the language regularly. Consequently, if there's a choice between maintaining consistency and providing some useful special case (particularly if it's a common special case), they'll often go for the latter. If you need a Perl-like language infrequently, Python occupies roughly the same niche but aims at consistency and simplicity. Python, though, lacks one of Perl's most interesting and useful features, which might be called implicit type coercion. From a Haskell perspective, if the Python interpreter encounters a type error involving basic types in your code, it will (safely) bug out at runtime with a helpful error message, but the Perl interpreter will attempt to recover by silently applying type coercion functions and continuing. If that sounds dangerous, it's because I've described it from the wrong viewpoint: from a Perl perspective, it means that you can largely ignore variable types when writing code, freeing up your mind to concentrate on your actual algorithm. I've written more about this point at http://pozorvlak.livejournal.com/47565.html .

The assumption that the programmer is competent manifests itself in other ways. One of Perl's mottos is "easy things should be easy, hard things should be possible", and so there are few limits on what you can do if you really try (though occasionally the mechanisms involved are a bit grubby and ad-hoc). From a Haskell perspective, this means that Perl code provides few guarantees. It's therefore essential that you read the documentation, and conversely that you provide good documentation for code that you write (this is considered part of the definition of competence). I read something by Cale Gibbard a while ago about how he'd made a big change to some Haskell module without ever reading the docs, simply by checking that everything compiled and the tests passed. In the Perl world, this would be somewhere between impossible, inconceivable and criminally negligent :-)

The downside of providing freedom for good coders to do wizardly things is that it makes it possible for bad coders to do stupid things. This is unfortunate. The solution is held to be self-discipline and education, and not hiring bad coders in the first place.

Perl is the Practical Extraction and Report Language, and a lot of its features are there to help with everyday scripting, data-munging and administration tasks, particularly in a Unix environment. In particular, there are a lot of features there to help with writing "one-liners". A lot of people say that this makes it unsuitable for writing large programs; this was true for versions 1-4, but version 5 improves the situation here a lot. People can and do write maintainable Perl 5 programs that are hundreds of thousands of lines long. It's also worth mentioning that you can do a lot more in 1000 lines of Perl than you can in 1000 lines of, say, Java. If people say negative things about Perl, it's always worth asking which version they used: if they say 4 or less, you can safely ignore them. Perl 6, a major redesign, has been on the cards for quite a few years now, and it should iron out some of the language's more irritating quirks while incorporating some nifty things from Haskell and Lisp. Unfortunately, they're learning the Mozilla Lesson that ground-up rewrites take a lot longer than you'd think. They seem to be making good progress now, though, and some useful Perl 6 features have been incorporated in version 5.10.

[ykokese from Reddit makes an excellent point: "I believe learning that you can shoot a problem with a one-liner is important too. I'm not trying to be witty, I truly think that Perl marked the beginning of an era by making it possible to solve problems an order of magnitude faster than before, and indeed introduced the stop-messing-around paradigm."]

I should certainly mention CPAN, the Comprehensive Perl Archive Network, which is without doubt the world's largest collection of open-source user-contributed add-on modules, widely mirrored and with a nifty tool for recursively downloading and installing modules. There is almost always a CPAN module to help with whatever task you are attempting. It might not be good, mind, but it's almost certainly going to be easier to start with some CPAN module than roll your own solution. And at their best, CPAN modules are stunningly good (the parser generator Parse::RecDescent springs to mind, here, but there are so many good ones - have a look at Mark Fowler's Perl Advent Calendars for some more recommendations). Perl's paradigm-agnosticism helps here, too: if it makes sense for a module to be OO, it can be OO; if it makes sense for it to be functional, it can be functional; if it makes sense for it to be a bunch of imperative procedures, it can be. All in all, it's the world's premier laboratory for add-on module design and implementation :-)

It's not all roses: Perl definitely has its irritating quirks and defaults-that-shouldn't-be. A while ago, I tried to think of five things I hated about the language and came up with ten. See http://pozorvlak.livejournal.com/49333.html for the list.

If you're interested in learning it, I've posted some advice on resources at http://pozorvlak.livejournal.com/58387.html . I'd also be happy to answer any questions. But since you're in London, you should seriously consider subscribing to the London Perl Mongers mailing list (london.pm.org), or going along to some of their meetings - there are some awesomely good programmers in that group, and they're a pretty friendly bunch. London.pm tech meetings are a great way of opening your eyes to how much is possible.

[Note: I specify London.pm mainly because I used to be a member and can personally vouch for their excellence. Your local Perl M[ou]ngers group may well be just as good, and is certainly worth looking at.]

Some of the stories in Memories of 20 years of Perl may also help convince you to take a look. I especially like the story in the second comment. Larry Wall's speech Perl, the first postmodern computer language is also compulsory reading.

I hope that helps!

Argmungers in universal algebra

Fri, 25 Apr 2008 23:49:07 GMT

Some of you might be interested to learn that argmungers are now explicitly included in my thesis: the mathematical concept, that is, not the code. One can recast classical universal algebra (which works syntactically, with words in a recursive language and denumerable sets of variables) in terms of planar trees of operators (abstract syntax trees, effectively) acted on by munging functions (and yes, I use the term "munging function"). Restricting the munging functions allowed is equivalent to imposing syntactic restrictions on the equations defining your theories: I'm writing up a proof of this at the moment. Interestingly, isolating the concept of munging functions makes the whole thing significantly cleaner: up until then, the theorem was frustratingly obvious, but trying to prove it (or even state it!) rigorously was like nailing jelly to the wall.

Maybe I should thank Hitesh in my acknowledgements...

Oh, and current state of play:

Sat Apr 26 00:47:12 BST 2008
4511 lines 22164 words 155551 characters
thesis.log:Output written on thesis.dvi (78 pages, 626444 bytes).
34 fixmes

I was up to 79 pages, but then I deleted some redundant stuff, so now I'm back down to 78.

Right, time to pack for the mountains tomorrow.

Attention, readers!

Fri, 25 Apr 2008 13:00:43 GMT

You have lost The Game.

That is all.

mi_guida: I have won the Pillaging Game.

The White Spider

Mon, 21 Apr 2008 10:00:56 GMT

I've just started reading The White Spider, Heinrich Harrer's account of his 1938 first ascent of the fearsome North Face of the Eiger. The climb was one of the pivotal moments in the history of mountaineering, and the book's considered one of the classics of mountain literature. I was particularly struck by the opening paragraphs:

"Writing a book about the North Face of the Eiger? Whatever for?"
The question was put to me by a man of some standing in Alpine circles. I was taken aback and slightly cross, so I gave him a somewhat off-hand answer: "For people to read, of course."
That started him off on a passionate tirade.
"Who's likely to read it? Don't you think the handful of climbers who are really interested in that crazy venture have had quite enough literature on the subject already? Or do you just want to join the sensation-mongers, from whose ranks a serious climber like yourself should keep as remote as possible?"
I answered him: "If all climbers shared your point of view, it wouldn't be surprising to find the newspaper reports overflowing with mis-statements and exaggerations. I believe the public has a right to authoritative information, especially when mountaineering problems become human ones. And I think it is a climber's duty to contribute to the formation of public opinion in such matters."
And with that I dropped the unpleasant argument.

It struck me as I read those words that one could replace "climbing" with "mathematics", "the Eiger" with the name of some serious mathematical problem, etc., and be left with a statement that's equally true, and equally important.

[By the way, there are some excellent, not to mention terrifyingly vertiginous, pictures of Sir Ranulph Fiennes on the Eiger's North Face here. Well worth a look.]

Public Service Announcement

Tue, 15 Apr 2008 15:05:19 GMT

I say this every year, but judging by my friends page and the state of my flatmates, it bears repeating.

Many of you are, or shortly will be, undergoing examinations - mostly Finals, a scattering of other things. To a man, woman, or small furry creature from Alpha Centauri, you all seem to be panicking, convinced that you haven't done any work, that you don't know anything, and that your only hope of redeeming your tattered academic credibility is to throw yourself into a routine of work so intense that your vocal cords atrophy from lack of use, your pen-holding fingers swell to the size of your arms, and your skin shrivels on contact with sunlight.

This is exactly wrong. First off, if you're reading this you're probably a friend of mine, and hence considerably smarter than the average bear: you have nothing to worry about. Secondly, mere lack of knowledge is not the problem. Mere lack of knowledge is generally a trivial problem. The real problem is Finals stress.

Allow me to expand on that. If you remain calm, keep alert, and generally keep your mind in a state where you can think as opposed to panicking, then your raw talent, coupled with the knowledge you currently have (a lot more than you realise, provided you haven't, say, been doing ten plays a term for the last three years), will get you a lot further than you now realise. Maybe not enough to get you as good a degree as you'd like, but enough to stop you embarrassing yourself. In addition, your mind will be better able to absorb more information, and you'll be more able to actually do some work. If, however, you allow yourself to be whipped up into a fervour of panic, then you will lose the all-important ability to think, and you will be lost. Worse than that: the panic will prevent you from getting any useful work done, and you won't take anything in.

So, your priorities should be:

Don't stress about how little work you're doing. That's not important, and stressing about it is actively unhelpful.
Do make sure you're sleeping, eating and exercising properly. Get out in the sun. Make sure you see your friends. Chill out. Relax.
Don't stress about your failure to achieve these goals, either. Get it right tomorrow. Repeat as necessary.

Get these right, and success will follow.

Diet update

Wed, 09 Apr 2008 09:58:44 GMT

This morning, I weighed 10st 3.2lb. That's 143.2lb for the USians, and 65kg in SI; either way, it means the following:

I am now the same weight as I was when I was sixteen and training 12-15 hours per week. Admittedly, I was more muscular then.
I have lost two stone (28lb/13kg) since starting the diet last July.
I have finally achieved my target weight, and can relax the calorie restrictions to "maintenance" levels.

So, success! It took a lot longer than expected, after a promising early start: I was down to 11 stone by the time we went hillwalking in Wales at the end of August, got down to 10st 5ish by the beginning of December, then the Christmas party season kicked in and it got a bit hard. I was 10st 10 after Christmas: it should only have taken a couple of months to lose the excess, but it turns out that it's really hard to convince yourself not to have that pint/sandwich/biscuit when you're hungry and/or wanting a drink and you're only a couple of pounds over.

Actually, it feels rather like cheating: I was 10st 6.4 on Monday, then apparently lost nearly 4lb in one day and became 10st 2.8 on Tuesday. I was treating this as a suspicious fluke, but today's measurement seems to confirm that there's something in it. Still, I shan't be too surprised if I'm 10st 5 again tomorrow. In general, my weight fluctuates by a pound or two from day to day.

Some things I've learned, in no particular order:

For me at least, calorie-counting really works. Basically, I've lost weight when I've been bothering to do the calorie-counting, and not when I haven't. I suppose there's an interesting question here: does calorie-counting work because it constrains you into only eating "healthy" things, or do "healthy thing" diets work because they reduce your calorie intake by stealth? It's not so easy to test, because so much of dieting is about psychology. That said, the physics is clear: calorie-counting will inevitably work if you can stick to it.
It's amazing how emotional people get about other people's diets. If I've learned anything from this, it's to ignore the opinions of others and stick to my guns.
For me, giving up alcohol is an unfortunate prerequisite to losing weight. As well as being calorific itself, alcohol makes it so much more tempting to eat and drink more.
I've had some serious advantages on this diet: chiefly, I've had a time in my recent past when I was fit and slim, and wanting to feel like that again has been a powerful motivator. I'm also male and do some exercise (both of which increase calories burned: more total calories burned give you more of a safety margin on any given day). I'm reasonably numerate, and I'm happy with making approximations - I never used to write anything down, just round everything I ate to the nearest fifty calories and added it up in my head as I went along. I drink coffee, and a hot drink is a wonderful distraction when you feel like eating something but you're not really hungry (or even when you are). I don't eat a lot of sweets. In general, my diet wasn't particularly bad before, but the occasional trip to the chip shop or the pub (or rather, the pub and then the chip shop) and the cheese-on-bagel snacks all added up over the course of years. Which is not to say that it's been easy: I like food and drink, and I like eating and drinking. I also strongly connect the idea of "having a good time" to the idea of "eating and drinking a lot" - I probably need to break that connection somehow.

All praise be unto The Hacker's Diet.

Pozorvlak's Conjectures

Sun, 30 Mar 2008 19:44:57 GMT

For the past couple of years (longer, actually), I've been thinking a lot about this business of static and dynamic typing in programming languages. Here's my current position on the matter.

Static and dynamic typing encourage and reward quite remarkably different approaches to the creation of software: everything from the nitty-gritty, (sit and think?)-edit-(compile?)-run-(test?)-(debug?)-(repeat?) cycle to the design (at all levels) of the software to be created, taking in every aspect of the tools and process used to support this design and implementation. Also, the choice of static or dynamic typing is best understood as part of a larger attitude towards development. Unfortunately, the use of a dynamic mindset with a static language not only means that you can't take advantage of the tools available (or even see why they'd be useful), it actively hinders success. The same is true for approaching a dynamic language with a static mindset.

I would therefore like to propose Pozorvlak's Conjectures:

If you find that a modern dynamic type system causes more problems than it solves, you're probably doing it wrong.
If you find that a modern static type system causes more problems than it solves, you're probably doing it wrong.

For instance, I find Haskell's type system causes me many more problems than it solves. Discussions with more expert Haskell users suggest that I am indeed doing it wrong. This guy, on the other hand, finds that "the dynamic languages stick out like sore thumbs because of the whole class of dynamic run time errors that show up that don't even appear in the statically typed languages". If you find this, I claim that it's because your code is not polymorphic enough¹. You can get away with having code which only accepts data of very restricted types when the compiler does all the checks for you statically. When it doesn't, you're better off writing code that behaves well for a wider class of inputs, and dynamic languages give you much more facility to do this. Once you learn to code like this, your problems largely go away. As a bonus, your code becomes more flexible, and whole classes of techniques become possible that would have been literally unthinkable before. Similar claims, it appears, can be made for static languages: at the higher levels of mastery, one can use a static type system as a theorem-prover to establish some quite nontrivial properties of your program automatically and across the entire codebase. This allows for some excitingly different approaches to program development.

A corollary is that if you ever finding yourself saying that your port of Feature X to Language Y is better than the original Feature X solely because it's (statically|dynamically) typed and the original Feature X was the other one, you should probably save your breath. It will probably also be worth your while to go back and determine what advantages the opposite choice of typing regimen gave to Feature X's users.

¹ A less interesting, but probably valid conjecture is that you're also not testing enough, or at least testing the wrong things. But this can't be the only answer. Dynamic programmers, in general, are not idiots; they are usually also Lazy, in the good sense. They're smart enough to work out that writing the equivalent isa_ok() test every time they would have written a type declaration in Java or whatever is no time-saver at all. Hence, they must need less type information overall for their code to be correct.

Better living through shell scripting

Fri, 28 Mar 2008 15:40:55 GMT

As I may have mentioned once or twice before, I'm writing up my thesis. This is an intensely slow and depressing process, and the temptation to slack off is overwhelming. Being a geek, I decided to write some code to help me. Writing code rather than writing thesis is still slacking off, obviously, but it's slacking off to a useful end. Writing blog posts about writing code that's meant to help me write my thesis is another matter...

I started with a simple script to make it as easy as possible to start work.

#!/bin/sh

cd ~/tex/thesis
evince thesis.dvi &
vi categorification.tex

That opens the directory containing my thesis, opens a viewer for the printer-ready version, and opens a text editor on one of the files containing source code. I stuck this in ~/bin/thesis, made it executable, then created a button on my Gnome toolbar to launch an xterm and run this file. This worked, to some extent, but there was still a problem: I could click the button, then open a browser and spend ages on Reddit or Livejournal before glancing at the clock and discovering to my horror that it was an hour later and I hadn't done any work. More drastic measures were needed. Version 2:

#!/bin/sh

cd ~/tex/thesis
evince thesis.dvi &
disallow firefox-bin gnobots2 ssh &
vi categorification.tex

disallow is a tiny script I wrote for just this purpose, which runs in the background and prevents you from running particular programs (here, Firefox, ssh (which I use for email) and the highly addictive Gnobots game). There's probably a standard way of doing this, but I don't know what it is, and it was trivial to write the script:

#!/bin/sh

while true
do
        killall $@ 2> /dev/null
        sleep 6
done

Talk about high-level programming: that practically is its specification. Loop continually, kill all the programs specified as command-line arguments, throwing away any error messages, go to sleep for six seconds, then do it again. The reason it's six seconds is because originally I had 60 seconds, deleted the 0 to make the testing cycle quicker, then thought "Actually, that's not such a bad idea..."

This sort of trivial scripting is pretty much the reason I use Unix in a nutshell. Unix has this deep hackability to it: writing your own utilities to do this kind of thing is so easy compared to most other systems.

Of course, disallow is an attempt to solve a social problem (my reluctance to do any work) by technological means, and comes with all the usual caveats. Two days in, it doesn't appear to be especially successful, but maybe it will have a more gradual effect.

The next script gathers data about the thesis, and tells me how long it is and how many incomplete sections there are.

#!/bin/sh

wc *.tex | grep total | total -l 'lines words characters'
grep pages *.log
grep -c XXX *.tex | total -l fixmes

The thesis is contained in a load of TeX files (with the extension .tex) which are compiled into an output file called thesis.dvi. The standard Unix utility wc can be used to find the number of lines, words and characters in all the .tex files - its notion of a "word" includes TeX formatting commands, but the idea of counting the words in a piece of mathematics is pretty wooly anyway. If you ask wc to count words in lots of files at once, it outputs a nice line at the bottom with the totals for all files: the grep total picks out this line and throws away the rest. The file thesis.log contains chatter from the compilation step, and in particular it contains a line saying how many pages were in the output. Incomplete sections are indicated by an "XXX" in the TeX file next to the relevant bit, because I can't be arsed to set up an actual bug database, and one would be overkill anyway.

The interesting parts of this script are the calls to the program total, which is also homebrewed. It's a program I've found myself needing a few times before, and of course it turned out to be quicker to write it than to work around its absence. Here's its code:

#!/usr/bin/perl
use Getopt::Std;

getopt('l');
my @totals;
while (<>) {
        my @newnums = /(\d+)/g;
        foreach (0 .. $#newnums) {
                $totals[$_] += $newnums[$_];
        }
}
my @labels = split /\s+/, $opt_l;
foreach (0 .. $#labels) {
        if (!defined($totals[$_])) { $totals[$_] = 0 }
        $totals[$_] .= " $labels[$_]";
}
print join(" ", @totals)."\n";

This code (of which I am in no way proud) takes each line in its input (in a grepy, "files specified on the command line or stdin if there are none", way), looks for numbers (specifically, positive integers), then adds them together columnwise. It then displays the totals, possibly with labels (specified using the -l option). It has many weaknesses: most egregiously, it only looks for positive integers (which would be easily fixable by using the CPAN module Regex::Common). It takes the contents of "column" n to be the nth number encountered on each line, so it can also get confused by missing data in early columns, or (as happened during testing) things like filenames with numbers in. This bug was closed by inspecting the relevant files, discovering that I didn't need them, and deleting them :-) But for now, total does what I need. And it paid off almost immediately: having written it, I realised that I could use it to label the output of wc to make it friendlier (though perhaps this functionality should be spun off into another
program, perhaps called label).

Here's the output of metrics:

5230 lines 24528 words 174762 characters
thesis.log:Output written on thesis.dvi (70 pages, 555828 bytes).
44 fixmes

Hurrah! As a bonus, I can now use this script on any directory containing TeX source :-)

[If you know the names of any of the standard Unix utilities I've undoubtedly reinvented, please let me know.]

Beinn Dubhcraig - photos

Tue, 25 Mar 2008 00:15:14 GMT

I've posted some photos of the Beinn Dubhcraig trip to my moblog. Larger versions exist, and I'll try to find a home for them: my Flickr account has sat unused for so long (prior to the Yahoo buyout) that it's not letting me log in any more :-(

Mountaineering experience

Mon, 24 Mar 2008 20:34:27 GMT

"What you need is combat experience. That's what keeps you alive in real combat."
"OK, but how can you tell the difference between combat experience and real combat?"
"Simple. If you're alive at the end, it was combat experience."
(From The Ballad of Halo Jones)

Saturday's trip to the hills started inauspiciously: I slept through all my alarms, and was only woken up by Jo texting to say "I'm parked outside". My bag was packed, but there was still a forty-minute interval of running around before I was showered, breakfasted, infused with tea, dressed and ready to go. Then we were off, at the decidedly late starting time of 9am.

Beauty

We'd planned to climb Beinn Dubhcraig, the easternmost of a group of four hills south-west of Tyndrum (the others being, from east to west, Ben Oss, Ben Lui, and Beinn a Chleibh). It's one of the most beautiful parts of the Highlands, but I've had a lot of bad luck there. Philipp and I once tried to do all four in a day, which resulted in us missing out two and having to run the last 7km to the station in an hour to avoid missing the last train - which doesn't sound like a lot, but you try doing it in hiking boots after you've been walking through foot-deep snow for eight hours. As for Beinn Dubhcraig, I've had maybe five really miserable days' walking in the last three years, and my first trip to Beinn Dubhcraig a few years ago was one of them - herringboning up steep slopes with hail battering into our eyes for hours on end, freezing cold and with soaking, freezing hands. We'd taken a grid reference for the summit at the bottom, but six-figure grid refs only specify an area the size of a football pitch, so we then had to spend three quarters of an hour wandering around the summit plain taking altitude readings until we found what appeared to be the summit - the visibility was far too bad for us to look from one high point to the other and compare.

Yesterday, however, the conditions were looking much better. The sun was out, and the forecast predicted an 80% chance of cloud-free tops and a mere -23C after wind chill - positively balmy after recent weeks. A quick hand stuck out the window suggested that I should probably wear the warm lined trousers anyway :-) It was Jo's first hillwalk in the winter, but I reassured her that on a day like today it really shouldn't be too big a deal, and that the mountains look stunning with snow on them. I packed the ice axe and the crampons anyway. Only one set of each, unfortunately, but since I was the only one with crampon-compatible boots, I thought I'd give her the axe and take the crampons for myself in the unlikely event that we needed them.

We got to the car-park at about 10.30, which is about the time we would have arrived if we'd got up much earlier and caught the train, so that was OK. We were presented with a choice of route - either we could go left, up a big winding path through the forest and eventually end up on the north-east ridge, or we could go right, cut across the flat land in front of the corrie and join the ridge from the right. We elected to do the latter, but this was probably the wrong choice: there was a small river between us and the mountain, and we'd forgotten that river crossings have an unfortunate tendency to take bloody ages. I got across some stepping stones fairly quickly, but the gap between two of them was too wide for Jo, so we wasted a good three-quarters of an hour walking up the river on opposite sides, trying to find a crossing that she was prepared to use. Eventually she just waded across a shallow part, getting her boots and socks wet. By this time, we were quite a bit off our planned route - in fact, we probably would have been quicker sticking to the path and going on for another kilometre or so to the bridge.

We were now pretty much equidistant between the north-east and north-west ridges, so we decided to change plans and head up the north-west ridge, which would afford better views of the rest of the range. We stopped for lunch in some trees at about 12.30, and it was at about this point that we saw the Easter Bunny (or rather, a hare still in its winter coat - though apparently the Easter Bunny was originally a hare, not a rabbit...). As we gained the ridge, more and more mountains came into view, and finally Ben Lui rose up above the ridge, looking as beautiful as I'd ever seen it. "From a find to a check, from a check to a view, from a view to a kill in the morning", I recited to myself.

It was at about this point that we reached the snowline, and took the opportunity to practice ice-axe arrests on a handy slope. The snow wasn't as good as I'd hoped - though there were patches of fresh powder, it was mostly old snow that had partially melted and re-frozen to form what skiers call "crud" (edit: and mountaineers call "névé"), making for a very slippery surface. As planned, I put my crampons on, and gave the axe to Jo, with a quick demo of how to use it to walk on snow slopes (hold it in the uphill hand, adze forward so it's in the right position for a self-arrest, and plant the spike in the snow to hold a slip while you step). The snow was patchier than I'd have liked, and I had to take my crampons off a couple of times as we went over grass again, but we were on the summit by just before three, to be greeted by one of the most gorgeous views I've ever seen. No difficulty finding the summit now - we could see all the way to the Paps of Jura. Loch Lomond and its surrounding hills, the mountains at Bridge of Orchy, the sites of any number of previous days out were visible to us. Absolutely stunning. We took a lot of pictures, but there wasn't a lot of time to hang about - the wind was starting to pick up, and, at nearly a thousand metres above sea level and exposed on all sides, it was getting chilly. Jo had wrung her socks out a few times, but her feet were still wet, and hence cold. We set off down the north-east ridge, partly because it was the more direct route to the car park, but mostly because it was the one we hadn't done.

Drama

Unfortunately, it wasn't as straightforward as the other one had been. About half an hour later, we encountered a big steep dip that we couldn't see over. Can't go under it, can't go over it, have to go round it. So we headed off to the right, hoping to circumnavigate it. Still a bit steep, but better - we could see the way ahead, at least. Shortly afterwards, we came to a snow slope, maybe fifty metres long.

"Slide down on our bums?" suggested Jo.

Sliding down snow slopes on your bum is technically known as "glissading", and if you look up "glissade" in the Mountain Leadership handbook's index you'll be presented with the pity advice "Don't". However, it has its upsides: most of all, it's a lot quicker than trudging down painstakingly (which carries its own dangers). "Hang on a bit, I just need to zip up my pockets so my railcard doesn't fall out," I assented.

Jo sat down and set off, and immediately I saw something was wrong. There's a standard technique for using the pick of your ice axe to control the speed and direction of your descent during a glissade, but I realised too late that I hadn't taught it to her, and she was Doing It Wrong, trying to use the spike as a rudder instead. That put her off-balance from the start, but the cruddy snow was her real undoing - the slippery, refrozen surface sped her up rather than slowing her down, and within moments she was hurtling down the slope, flailing wildly and totally out of control.

Shit, shit shit. Self-arrest, Jo, self-arrest...

The axe flew out of her hands as she spun around and she hurtled on without it. Now she wouldn't be able to use it to stop herself. Why wasn't it leashed to her wrist? But then, thank God, she hit a shallower part of the slope, and after another heartstopping second or two she came to a crumpled halt at the bottom of the slope.

I was at the top of the slope, she was all the way down at the bottom, and probably injured. I had to get down as quickly as possible to render aid. Nothing for it but to glissade myself. But the axe was a good forty metres below me, so I'd have to control my speed with my hands and feet. Not good.

I sat down, braced myself, and started to slide down. By scraping my boot edges along the snow (I'd taken my crampons off) I was able to control my speed and come down the slope under reasonable control. I veered slightly to the left to pick up the axe.

Red dots in the snow. Blood. She's bleeding. Fuck.

Get down to where Jo is. Her face is covered with blood, but she's moving. She's still got her glasses - that's a good thing.

"OK, let's have a look at you."
*peers*
"You've got a nasty cut in your eyebrow. Let's get the blood cleaned off and see how bad it is."
"There's a first aid kit in my bag."
"Which section?"
"The main one."
"Main section?"
"Yeah."

Rummage through bag, find first aid kit. Open it. Not much there.

"OK, I've got a wet-wipe, let's wipe the blood off and see what's wrong. Do you know what happened?"
"I think I must have hit my head on the shaft of the ice axe."
"Well, thank God it wasn't any of the sharp bits."

Take out wet-wipes. Fumble with them, drop them, watch them speed away down the slope. Bollocks. Kick shelf in snow for first aid kit. Take out antiseptic wipe, fumble, drop, watch it speed away. I've only got my thin inner gloves on, and my fingers are getting pretty painful. Successfully extract wipe from container, hand to Jo to wipe off relevant area. Take closer look at cut.

"I think we're going to have to go to A&E, I might need stitches."
"Yeah, possibly, it looks fairly deep. It's hard to tell though, it keeps welling blood."
"We'd better apply pressure to it."
"Yep. Hang on."

Fumble through first aid kit. Find dressing, extract from packet, hand to Jo. A bit thin, and nothing to hold it in place, but it'll do for a start.

"OK, hold this to the wound."
"Where exactly do I need to hold it?"
"Just... here. I can't see anything to hold it in place in here: I'll get out my first aid kit and have a look in that."

Take off rucksack, pin in place with axe, furkle through to bottom of bag to get first aid kit.

"OK, we've got some micropore tape and... aha, here's a dressing with a bandage."

Apply second dressing over first one, tie bandage underneath ponytail so it holds in place.

"Can you see?"
"Wait a second."

Jo shifted the dressing around, so she could see out of her right eye.

"Only out of my right eye".
"OK, we can have another look at it when we're on the flat."
"Shit! Look!"

I turned around, to see that a plastic bag had fallen out of Jo's rucksack and was sliding down the slope, picking up speed. It looked like it had her wallet in it and - shit, what if it had the car keys? I hurried down after it, trying not to slip and become a casualty myself. OK, wallet - grab, mobile phone - try to stop with boot edge, but there's a reason I never got picked for the football team at school... stomp down again, get it this time. Pick up phone, bag starting to slow, snatch at it. No sign of the car keys: they must have been in another bag. A couple of Alpen bars have fallen out of the bag and are making a separate break for freedom - what the hell, I've come this far down, and they're not moving too fast. Oh, great, they've stopped. OK, carefully pick them up and put them in the bag - ah, there's the missing antiseptic wipe. No sign of the wet-wipes, though. Pity, we could have used them to clean the dried blood off. Put recovered items in bag, trudge back up slope to Jo.

Jo was understandably shaken by the whole thing, but OK to walk. Her legs were shaking a bit, though, which wasn't ideal on the terrain. I thought it might be better if we tried giving her the crampons, and me keeping the axe: though her boots aren't crampon-rated, my crampons are pretty flexible. The ground was nasty and steep and strewn with rocks, so I decided to traverse round to the right: it looked like there would be a shallower descent along that way. So we traversed along the slope for a while, until we got to a really big, steep snow slope. At least 100 metres long, maybe 150. Maybe even 200. It's bloody hard to judge distances when you're standing on top of them, peering down and thinking "how the hell am I going to get down that? NB: all distances in this post should be assumed to have been calculated using FearCam(TM).

The snow was, needless to say, horrible: a coherent layer of thin ice lay on top of it, meaning that my boots would slip out from under me if I tried to stand on it. I might be able to kick steps in it, supporting myself with the axe. Or... it looked like there was a wide streak of powder snow a couple of metres out. It might just be covering more ice, of course, but it could be easier to descend.

"OK, Jo, it looks a bit easier to descend on that streak of powder. It might be easier for you to traverse this bit if you turn face-on to the slope and kick your toes in."

I turned back to face the slope and set off across the ice. Behind me, there was a yelp. I turned to see Jo sliding down the slope again. I don't remember this part, but apparently she was shouting "Help! Help!", even though she knew that I was the only one who could possibly hear her and there was nothing I could do.

I had the axe: she'd have to use her hands and feet to stop herself. On ice. Oh God no. I hope she knows not to dig crampons in... oh God, she's actually bouncing and rolling! Fuckfuckfuckfuckfuck. Stop, Jo, stop, please stop... but she carried on all the way to the bottom of the snow slope, rolling and crumpling like a rag doll. I hope she still has her glasses, I thought, because there's no way I'm going to find them otherwise.

I had to get down to her ASAP, which meant another glissade. Fortunately, I had the axe this time, and could use it to control my speed and direction. I tried sitting down, but couldn't get the axe in far enough, so I settled for turning side-on to the slope and sliding down on my thighs, holding onto the axe with both hands and dragging it down the ice. It was at this point that I really grew to love and appreciate my ice axe. No red spots this time, thank God, and Jo was kneeling up by the time I was half-way down - she knew she had to get up quickly to show me that she was still mobile.

My arms and shoulders were sore from the effort by the time I got down, but the important thing was that Jo was OK, albeit badly shaken. A quick check of her face showed no fresh blood, and she didn't have any other injuries - we'd been incredibly lucky. But we still had a big problem: we now only had an hour and a half to sunset, we were a long way down the side of the ridge, and, thanks to all our traversing round to find an easier way down, we were quite a long way off course. There was a big flat plain below us to the right, but if we descended down there in search of flatter ground, we'd have to re-gain all the height later in order to find the path back through the trees, and gaining height takes time and energy. We tried what looked like a flat bit slightly above us and further on, but it turned out to be no good - flatter than where we were, but still too steep to traverse across comfortably, especially with Jo's increasingly shaky legs. It didn't help that we were now in shadow; this, coupled with Jo's monocular vision, made it hard to determine the consistency of the snow. I spotted a long streak where the grass poked through the snow: reasoning that this would be less icy, we traversed over to there and then sidestepped down to the plateau. Plant axe, left foot, right foot, plant axe, left foot, right foot, repeat. We'd lost a lot of height, but now at least we were on ground we could walk on safely.

Another quarter of an hour or so later and we were back on the ridge, past the original obstruction. Looking back at it, it was clear that we should have tried to skirt it to the left, where it was much gentler. The slope we'd come down looked insane, but largely because the shallower sections we'd used were out of sight.

Daylight was still a worry, though, especially because we were still in snow. Looking across the valley, we could see how high the snowline was above the valley floor, and our reassuring claims that the snowline would be lower in north-facing corries rang a bit hollow. We pressed on - there was nothing else to do - with me casting the odd look at Jo's face to check that there wasn't any new blood. Eventually, the snow petered out, Jo took her crampons off, and I put the axe away. We came to a wire fence, with what looked like a stile on the other side of the stream, and a path (an actual path! Or at least a sheep track) beyond it. There were a few stepping stones across just above it, but Jo, only having one eye, needed to be guided across. Of course, once we were across the stile turned out not to be any use at all: all the planks were broken, and there were rusty nails poking out of it. In the end, we just climbed over the fence. We carried on down the path for a while, but it eventually became obvious that it wasn't the one we needed. It was now about two hours since the accident, and the light was definitely going. Out came the GPS set, which revealed that - mirabile dictu! - the path we needed was actually only about a hundred yards away on our right, but out of sight beyond a small rise. First, of course, there was another bloody river crossing to negotiate, but again we were lucky, and this one didn't take us too much time. Once over the hill, we found the track almost immediately - a wide Land Rover track, leading away into the forest, and not a moment too soon.

I'd spent much of the morning cursing myself for dismissing this track at the beginning, but actually avoiding it hadn't been such a bad idea - it took some long, winding detours, and it took us over two hours to walk back to the car park. We were both very glad I'd chased after the Alpen bars. At the end, it was completely dark, and we were negotiating obstacles with torches. Fortunately, there was a group of houses near the car park which had lights in their windows, so we had some measure of how far away we were. At last, at 8pm, about four hours after the first accident, we made it back to the car.

It was at this point that the pain in my testicle announced that it wasn't going to let me ignore it any more.

Some medical background: about eleven years ago, I suffered a testicular torsion, which was the most painful experience I hope ever to undergo. It was fixed surgically, and after a couple of weeks in bed and another couple having to climb staircases using the banisters and my arms, I was OK again; but it's left me susceptible to occasional bouts of infection, and the swelling and tenderness revealed by a quick check suggested that this was another such episode. The last one was a few years ago: obviously, they'd chosen this moment in order to be maximally annoying. Of course, it was the Saturday of Easter weekend, and I wasn't going to be able to see my doctor until Tuesday, so it looked like I would be going to Accident and Emergency as a patient in my own right. Jo wasn't up to driving, and so after a quick call to wormwood_pearl to let her know I wouldn't be home for a while, I drove off, trying to keep my legs as far apart as possible.

A&E and TMI

We reckoned that the nearest hospital with an A&E department was in Paisley, but by the time we'd driven all the way to Paisley we might as well have gone to Glasgow. It was now probably too late to avoid the Saturday night rush entirely, but Jo thought that the Southern was probably going to be the quietest of the various Glasgow hospitals, and besides, she knew the way there. First, of course, we had to drive to Glasgow, which meant the road alongside Loch Lomond. A few years ago, this was apparently voted the worst main road in Britain. They're resurfacing it piece-by-piece, but they've got a way to go yet. It's narrow, windy, bumpy, and unlit - just what you want when you've got sore baws :-(

We arrived in A&E at about 10pm, to find an almost empty waiting room and an LED sign reading "Patients will be seen in order of severity of symptoms. Approximate waiting time 2hrs". A bit surprising in view of how empty the place was, but still a lot better than any of my previous A&E trips. Jo registered with the (incredibly friendly) receptionist, who offered her a blanket, and then I did the same, slightly sheepishly. I didn't get a blanket, though. We sat for a while, not really feeling like reading the books we'd brought with us. The (black, male, presumably African) triage nurse called for Jo after a while, and when she came out he'd cleaned her up, and replaced the dressing with a plaster. I think we were both feeling pretty silly about wasting NHS time on what was obviously a very minor injury, but he'd said that the doctor would see her in a while, so it couldn't be completely trivial. Then he called me into the triage room, and asked me to describe my symptoms. I told him about the pain and swelling, and about the torsion; he said that because of the torsion he'd bump me up the priority list, and offered me some paracetamol for the pain (which I very gratefully accepted). He asked me why I'd let Jo fall. "It wasn't my fault," I said. "Yes it was, it was your fault," he replied. I dunno, maybe he was right. I gave a urine sample (into a cardboard sample bottle - I guess it saves on washing up...), took the paracetamol, and took the opportunity to drink several glasses of water from the cooler, as I was pretty thirsty.

I went back out and sat in the waiting room, and waited. Various people arrived, registered and were triaged; every so often, a doctor would come out and call a name to be seen. The paracetamol was kicking in, and the pain wasn't too bad, provided I didn't try to do anything stupid like put my legs together. After we'd been there for an hour, I was called through, along with another patient. I was surprised to be seen so quickly, especially as I was meant to be the one escorting the casualty! The doctor (female, dyed red hair, English accent, surprisingly young-looking - I must be getting old) led us to separate curtained areas, handed me a gown, a blanket, and a bag for my clothes, and told me to change. I changed, and then sat and waited for her to deal with her other patient. I heard the other patient say "Sorry for wasting your time", to be met with "Oh, you're not wasting my time!" Nice to know it's not just me, I thought, and decided not to say that to her.

The doctor came through, chatted briefly, examined me, and diagnosed me with Epididymitis, which was more-or-less what I expected. The most common cause of epididymitis in young men is sexually-transmitted urinary tract infections, but more or less everything else about my case argues against this explanation (chiefly that I haven't been sleeping around). Probably a random UTI, brought on by stress and dehydration. She gave me a mixture of painkillers and anti-inflammatories, and prescribed two weeks' worth of antibiotics (they didn't have any to give me at the time). I got back into my clothes, waited for the pills to arrive, and went back into the waiting room. It was only 11.15 - I'd been treated within an hour and a quarter of arriving, on a Saturday night. Not bad :-)

Jo had got into a conversation with a woman who'd been at an ice-hockey game and been hit by the puck. She had a big mess of blood-matted hair in the side of her head, which clashed rather with her highlights, but her team had won, so she was pretty cheerful. I tried the vending machine, but it was sold out of everything; then we remembered that there were some chocolate and peanuts left in the car, so I went out to get them so I could take some diclofenac. We sat around for a bit longer waiting for Jo to be seen, trying to decide whether it was worth sticking around. The wound had long since stopped bleeding, but Jo was having to keep her eyebrows immobile to stop it opening again. At about twenty past midnight the same doctor who'd seen me called for Jo, and a few minutes later Jo emerged, with her wound glued together and covered with steri-strips. We were out by half-past midnight, which was seriously impressive given the time of day and the minor nature of our injuries: I think they must have been clearing the decks for the post-pub rush. Jo was feeling up to driving by now, so she gave me a lift home. wormwood_pearl had gone to bed, but left me dinner in the oven: it had gone cold, but I was far too hungry (and it was far too tasty) for me to care :-)

All in all, not exactly Touching the Void, but it had some frightening moments.

Lessons Learned:

River crossings can take bloody ages, and can leave you with cold, wet feet, which are very much a Bad Thing. Jo wasn't sure, but reckoned that may have contributed to her second fall. Time to cross rivers should be factored in when planning routes.
Even if the weather's fine, bad snow conditions can make a walk dangerous. The weather on Saturday was some of the best I've ever had, but we still managed to get into trouble because of the poor snow.
Plan for the weakest member of the expedition, not the strongest. I've been out hiking in some truly atrocious weather with Jo, and she's a good person to hike with, but her experience in summer walking and skiing wasn't enough to make up for her inexperience walking on snow. And I'm not exactly Reinhold Messner myself - in fact, I'm wondering if "mountaineering inexperience" would have been a better title for this post. Still, I think we both gained a few XP that day.
Always keep the navigational Big Picture in mind. Trying to go around local difficulties by going the way that seemed most obvious, without any thought of our overall route, meant that we got quite badly off course, both when crossing the river and when descending the ridge.
I'm not sure what to think about safety gear. Having a full set of crampons and axes would certainly have helped, but then again, having the gear we did probably encouraged us to get out of our depth. And crampons and axes are expensive. Helmets, in retrospect, would probably have been a good idea, especially since we each had one sitting at home. I'm wondering now if we should have been roped together once we got onto the steep slope: of course, we didn't have ropes with us, so that wasn't an option. It's a tricky balance: you don't want to weigh yourself down with a load of unnecessary kit. Ropes, in particular, are heavy, and I've never needed one before.
We kept talking about changing Jo's dressing, but never actually did: this meant that she spent hours walking while only being able to see with one eye. We were never in an ideal situation to do it, but we should probably have prioritised it higher.
I should probably take a careful look through the first-aid kit, and check it's full of sensible stuff. I've already bought replacements for the bits I used.
Edit: Make sure everyone in the party knows how to use the gear properly. In particular, self-arrests should be practised (thanks to bronxelf_ag001).

Time to throw it open to the floor: what should I have done differently?

[Edit: some photos of the trip can be seen here.]

Nice probability question

Wed, 19 Mar 2008 17:08:15 GMT

Here's a nice question from a Cambridge entrance paper that some of you might enjoy:

You have a biased coin that comes up heads with probability p. With it, you play the following game: you toss the coin repeatedly, scoring 1 each time it comes up heads and 0 each time it comes up tails, and stop when you toss two tails in a row.
What is E, the expected number of tosses in a round of the game?
What is S, your expected score from one round (i.e. the expected number of heads tossed)?
Now suppose the rules are changed, so that you only stop the game after tossing r consecutive tails. What are the new values of E and S?

The question wasn't quite as stated: it actually included some extra information, which was useful but nonessential. In deference to the fact that many of you have much greater mathematical experience than the average Cambridge applicant, I've hidden it under a cut, and whited the text out - select to see it.

Hint for part 1:
E satisfies the equation

E = p(1+E) + p(1-p)(2+E) + (1-p)(1-p)2

The question asked candidates to first derive this equation, and then solve it for E.

Hint for part 3:
As before, candidates were given an expression for E, and asked to derive it:

E = (1-q^r)/pq^r

where q = 1 - p.

Wed, 12 Mar 2008 15:03:11 GMT

My paper was rejected. Not enough new material, and I'd failed to cite related work.

As well as being annoying, this is worrying: the paper contained almost all the new material that's in my thesis, strongly suggesting that my thesis doesn't contain enough new material either.

Bugger.

On the upside, the reviewer suggested some possible applications and further work: if I could do that, maybe that would be enough.

Minor annoyances

Mon, 10 Mar 2008 21:01:02 GMT

A guy on Reddit pointed me at this article today. It's the Wikipedia biography of the Welsh computer scientist Donald Davies. Like my father, he was from the Rhondda Valley; he was the co-inventor of packet-switched networks (like this big one you're using right now); and he once found a bug in Alan Turing's code, before the first computer had been built :-)

I think he may be my new hero.

In other news:

I've just finished Portal. It was great, but far too short :-( If you haven't played it yet, then you should go and buy/download a copy right now, and cover your ears and sing "La la la la la..." until it's loaded to avoid being spoilered like I was. The puzzles are still a lot of fun, but the plot and jokes would have been a lot better if I hadn't half-known they were coming.
[And I didn't find "The Cake is a Lie!", even though I was looking out for it...]
Spent a lot of time sitting at the computer today, but didn't get any thesis done. Bah.
After a similarly unproductive morning yesterday, I went to the climbing wall, and had quite a good session: I did two 6as (which is good, for me), led a widely-agreed-to-be-undergraded 5+, and finally knocked off a 5+ with a big scary overhang that had been tormenting me for months. Yay!
This may be the most awe-inspiring programming war story I've ever read.
Letter from Linacre: I didn't get the job. Never mind.
The winter mountaineering course I was booked on next weekend has been cancelled due to lack of interest. Just when the snow's started up again! I'm much more annoyed about that than about the rest, to be honest.

Ah well, tomorrow will be better. Anyway, have a look at this video, which is the trailer for one of the films I saw at the mountain film festival on Saturday:

That sea arch he's climbing up the underside of, by the way, is provisionally graded as a French 9b :-)

No hills this week :-(

Sun, 09 Mar 2008 13:19:44 GMT

Its hailing outside. The forecast for the hills this weekend was -13C after windchill, and gusting winds of up to 50mph - we've been out in much worse, but there was also the danger of blizzards, which would have been a problem. Plus Philipp's away, and Michael and Jo are busy this weekend, so the walk would have been me and someone else less experienced. I'm not going walking on my own in a blizzard, and while Bart, say, is a sensible guy with his head firmly screwed on and a few walks under his belt, I really didn't feel like taking him out in those conditions¹. So I'm sat here at home, failing to work on my thesis. Bah. My last hillwalk was two weeks ago now, and I'm starting to get the been-in-the-city-too-long shivers. Rock climbing's fun, but doesn't quite hit the same spot, particularly when it's indoors.

Going to the Adventure Film Festival yesterday didn't help, either...

firefliesinjune: I haven't forgotten about my promise to take you to the Highlands, but let's wait for the weather to improve, eh? :-)

¹ Maybe I should have left that choice up to him, I dunno.

Lockhart's Lament

Sun, 09 Mar 2008 12:41:12 GMT

Everyone needs to read this piece, especially those who don't consider themselves mathematicians.

A Mathematician's Lament, by Paul Lockhart.

There's a bit of biographical information on the author here. It's a cri de coeur about the state of mathematics education in the USA (and, by extension, in every industrialised country), which convinces generation after generation of kids that maths is boring and ritualistic, rather than the artful, elegant form of play that it really is. It's a bit long, but thoroughly, thoroughly excellent.

A musician wakes from a terrible nightmare. In his dream he finds himself in a society where music education has been made mandatory. “We are helping our students become more competitive in an increasingly sound-filled world.” Educators, school systems, and the state are put in charge of this vital project. Studies are commissioned, committees are formed, and decisions are made— all without the advice or participation of a single working musician or composer.

Since musicians are known to set down their ideas in the form of sheet music, these curious black dots and lines must constitute the “language of music.” It is imperative that students become fluent in this language if they are to attain any degree of musical competence; indeed, it would be ludicrous to expect a child to sing a song or play an instrument without having a thorough grounding in music notation and theory. Playing and listening to music, let alone composing an original piece, are considered very advanced topics and are generally put off until college, and more often graduate school.

As for the primary and secondary schools, their mission is to train students to use this language— to jiggle symbols around according to a fixed set of rules: “Music class is where we take out our staff paper, our teacher puts some notes on the board, and we copy them or transpose them into a different key. We have to make sure to get the clefs and key signatures right, and our teacher is very picky about making sure we fill in our quarter-notes completely. One time we had a chromatic scale problem and I did it right, but the teacher gave me no credit because I had the stems pointing the wrong way.”

In their wisdom, educators soon realize that even very young children can be given this kind of musical instruction. In fact it is considered quite shameful if one’s third-grader hasn’t completely memorized his circle of fifths. “I’ll have to get my son a music tutor. He simply won’t apply himself to his music homework. He says it’s boring. He just sits there staring out the window, humming tunes to himself and making up silly songs.”

In the higher grades the pressure is really on. After all, the students must be prepared for the standardized tests and college admissions exams. Students must take courses in Scales and Modes, Meter, Harmony, and Counterpoint. “It’s a lot for them to learn, but later in college when they finally get to hear all this stuff, they’ll really appreciate all the work they did in high school.” Of course, not many students actually go on to concentrate in music, so only a few will ever get to hear the sounds that the black dots represent.

icedragon1969: he has some hard words about high-school geometry, too.