pozorvlak (pozorvlak) wrote,
pozorvlak
pozorvlak

A quick introduction to redo

[This is a cleaned-up version of the notes from my Glasgow.pm tech talk last night. The central example is lifted wholesale from apenwarr's excellent README for redo; any errors are of course my own.]

Redo is a replacement for make designed by djb and written by Avery Pennarun.

You may have heard of djb - he's the author of the mail transport agent qmail and the DNS software djbdns. His code is famous for its uncompromising brilliance and very very low defect rate. Djb has been described as "the greatest programmer in the history of the world" - even if you disagree, how many people ever receive such an accolade?

Back to redo. Make has three major problems:

  1. It's Yet Another Goddamn Syntax to remember; one with its own set of stupid whitespace-handling and quoting issues. Worse, it's not even well-standardised, with every version of make expecting something slightly different.
  2. Recursive make sucks, for well-known reasons.
  3. Nonrecursive make sucks, because of make's poor support for modular coding and the need to parse a giant Makefile every time you build the smallest thing.

Most proposed replacements have focused on problem 1, by replacing make syntax with something more familiar but less well suited to the problem domain.

"Yes, we know all that," you cry, "get to the point!" Well, the point is that redo fixes all three of these problems:

  1. No new syntax - do-files are ordinary shell scripts.
  2. Recursive redo works properly.
  3. There is no nonrecursive redo.

Make has some more subtle problems:

  1. Your build also depends on the Makefile (suppose you change your CFLAGS? Now you have to rebuild all your programs), but how many people remember to declare that?
  2. Actually, each target only depends on some parts of the Makefile (suppose you change your CFLAGS: you don't now need to rebuild the documentation). Have fun tracking that.
  3. Building is not atomic - if the compiler crashes halfway through generating foobar.o, you'll be left with a broken object file in your tree.
  4. Complex Makefiles are a bitch to read and debug.

Redo fixes these too.

Time for an example!

I'm going to create a basic Hello World example. Here's hello.c:

#include <stdio.h>
#include "resources.h"
int main(char argc, char** argv)
{
    printf("%s\n", message);
}

resources.c:

char* message = "Hello, world!";

and resources.h:

extern char* message;

Note that hello.o depends on hello.c and resources.h, but resources.o only depends on resources.c.

Now to create our do-files. Redo wants each target in its own file, for reasons I'll explain later.

all.do:

redo-ifchange hello

redo-ifchange is an ordinary program that tells the redo system that the current target depends on the targets specified.

hello.do:

OBJS=hello.o resources.o
redo-ifchange $OBJS
gcc -o $3 $OBJS

$3 is the name of a temporary file created by redo: if the build succeeds, this is copied over the file we're building.

default.o.do:

redo-ifchange $1.c
gcc $CFLAGS -MD -MF $3.deps.tmp -o $3 -c $1.c
DEPS=$(sed -e "s/^$3://" -e 's/\\//g' <$3.deps.tmp)
rm -f $3.deps.tmp
redo-ifchange $DEPS

If redo can't find a specific do-file for the target it wants to build, it will look for a default one with the right extension. $1 is the name of the file being built, without its extension; the first line declares that the .o file depends on the corresponding .c file. The -MD and -MF flags to gcc tell gcc to output information about the dependencies of the file being compiled; we then do some sed magic to process that list and (in the final line) declare a dependency on what we found.

The key insight here is that the first time you build something it doesn't matter what its dependencies are: clearly it needs to be built, because it doesn't exist! Hence, we can use that first build run to capture dependencies. Redo allows you to add dependencies at any point of the build script, making this kind of thing easy; apparently it's possible with make, but it's a total pain.

clean.do:

rm *.o hello

Now, let's build the whole thing:

pozorvlak@delight:~/src/redo-talk
0 $ redo
redo  all
redo    hello
redo      hello.o
redo      resources.o

Isn't that pretty? The level of indentation shows you the structure of the recursive calls. Note, by the way, that the redos in the left margin mean that you can just copy-and-paste from the terminal to re-try a failing part of the build.

Redo's got some nice debugging options available:

pozorvlak@delight:~/src/redo-talk
0 $ redo clean && redo -x
redo  clean

redo  all
* sh -ex all.do all  all.redo2.tmp
+ redo-ifchange hello

redo    hello
* sh -ex hello.do hello  hello.redo2.tmp
+ OBJS=hello.o resources.o
+ redo-ifchange hello.o resources.o

redo      hello.o
* sh -ex default.o.do hello .o hello.o.redo2.tmp
+ redo-ifchange hello.c
+ gcc -MD -MF hello.o.redo2.tmp.deps.tmp -o hello.o.redo2.tmp -c hello.c
+ sed -e s/^hello.o.redo2.tmp:// -e s/\\//g
+ DEPS= hello.c /usr/include/stdio.h /usr/include/features.h 
 /usr/include/bits/predefs.h /usr/include/sys/cdefs.h 
 /usr/include/bits/wordsize.h /usr/include/gnu/stubs.h 
 /usr/include/gnu/stubs-32.h 
[snip]

Let's try updating resources.h and rebuilding:

pozorvlak@delight:~/src/redo-talk
0 $ touch resources.h && redo
redo  all
redo    hello
redo      hello.o

Exactly the right amount of work is done to get the build up-to-date. As one would hope :-)

I mentioned changing the Makefile earlier and the problems it can cause. Let's try that:

pozorvlak@delight:~/src/redo-talk
0 $ touch default.o.do && redo
redo  all
redo    hello
redo      hello.o
redo      resources.o

Every target has an implicit dependency on the do-file used to build it. This, by the way, is the reason why every target gets its own do-file: that way, you get fine-grained build-script dependencies for free. But where are these dependencies held?

pozorvlak@delight:~/src/redo-talk
0 $ cd .redo && ls
db.sqlite3          lock.18  lock.22  lock.27  lock.32  lock.38  lock.48
db.sqlite3-journal  lock.19  lock.23  lock.28  lock.33  lock.4   lock.7
lock.10             lock.2   lock.24  lock.29  lock.34  lock.40
lock.11             lock.20  lock.25  lock.30  lock.35  lock.42
lock.16             lock.21  lock.26  lock.31  lock.36  lock.45

Redo keeps all its dependency information in a SQLite database in your project's toplevel directory, which also contains information about when each target's dependencies were last checked. This means that

  1. Every invocation of redo has access to full dependency information.
  2. Redo doesn't have to parse a big text file every time it builds every tiny thing.
  3. You don't need to stat the same files over and over in a single build.

Hence, recursive redo is efficient and accurate.

The .redo directory also contains lockfiles to prevent parallel invocations from stomping on each other. Redo supports a -j option for parallel builds, and even supports jobserver's protocol, so make -j and redo -j can coexist.

Let's try the atomicity feature. Let's create a file called fred.do:

echo "Yabba-dabba-doo!"

Redo redirects standard output to the tempfile, so we don't need to pipe our output to $3.

Now let's try building it:

pozorvlak@delight:~/src/redo-talk
0 $ redo fred && cat fred
redo  fred
Yabba-dabba-doo!

OK, that worked. Now let's make fred.do fail:

echo "Yabba-dabba-don't!"
exit 1

pozorvlak@delight:~/src/redo-talk
0 $ redo fred; cat fred
redo  fred
redo  fred: exit code 1
Yabba-dabba-doo!

Hurrah!

How does redo handle subdirectories?

subdir/fred.c:

int main(char argc, char** argv)
{
    return 0;
}

pozorvlak@delight:~/src/redo-talk
0 $ redo subdir/fred.o
redo  no rule to make 'subdir/fred.o'

Redo requires the do-file to be in the same directory as the file you're building. There's a fork of the project which searches up through the tree looking for appropriate default.*.do files, but it's not clear whether that's the Right Thing: as it is, you can see what you can redo in a given directory with a simple ls. Compare the mess that you can create with included Makefiles!

Oh, one more thing:

barney.do:

#!/usr/bin/perl
use 5.010;
say "Whatever you say, Fred!"

pozorvlak@delight:~/src/redo-talk
1 $ redo barney && cat barney
redo  barney
Whatever you say, Fred!

You can write your do-files in any language that supports the #! convention :-)

So, in summary:

  1. Redo is a new build system
  2. It has a very simple design which solves many of the problems of make
  3. You should consider using it for your next project.
Tags: beware the geek, computers, perl, programming, redo, talks
Subscribe
  • Post a new comment

    Error

    default userpic

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 12 comments