Date: 2023-01-15
Git: https://gitlab.com/mort96/blog/blob/published/content/00000-home/00015-fast-interpreters.md

In this post, I hope to explore how interpreters are often implemented, what a "virtual machine" means in this context, and how to make them faster.

Note: This post will contain a lot of C source code. Most of it is fairly simple C which should be easy to follow, but some familiarity with the C language is suggested.

What is a (virtual) machine?

For our purposes, a "machine" is anything which can read some sequence of instructions ("code") and act upon them. A Turing machine reads instructions from the cells of a tape and changes its state accordingly. Your CPU is a machine which reads instructions in the form of binary data representing x86 or ARM machine code and modifies its state accordingly. A LISP machine reads instructions in the form of LISP code and modifies its state accordingly.

Your computer's CPU is a physical machine, with all the logic required to read and execute its native machine code implemented as circuitry in hardware. But we can also implement a "machine" to read and execute instructions in software. A software implementation of a machine is what we call a virtual machine. QEMU is an example of a project which implements common CPU instruction sets in software, so we can take native machine code for ARM64 and run it in a virtual ARM64 machine regardless of what architecture our physical CPU implements.

But we don't have to limit ourselves to virtual machines which emulate real CPU architectures. In the world of programming languages, a "virtual machine" is usually used to mean something which takes some language-specific code and executes it.

What is bytecode?

Many programming languages are separated into roughly two parts: the front-end, which parses your textual source code and emits some form of machine-readable code, and the virtual machine, which executes the instructions in this machine-readable code. This machine-readable code that's inteneded to be executed by a virtual machine is usually called "bytecode".

You're probably familiar with this from Java, where the Java compiler produces .class files containing Java bytecode, and the Java Virtual Machine (JVM) executes these .class files. (You may be more familiar with .jar files, which are essentially zip files with a bunch of .class files.)

Python is also an example of a programming language with these two parts. The only difference between Python's approach and Java's approach is that the Python compiler and the Python virtual machine are part of the same executable, and you're not meant to distribute the Python bytecode. But Python also generates bytecode files; the __pycache__ directories and .pyc files Python generates contains Python bytecode. This lets Python avoid compiling your source code to bytecode every time you run a Python script, speeding up startup times.

So how does this "bytecode" look like? Well, it usually has a concept of an "operation" (represented by some numeric "op-code") and "operands" (some fixed numeric argument which somehow modifies the behavior of the instruction). But other than that, it varies wildly between languages.

Note: Sometimes "bytecode" is used interchangeably with any form of code intended to be executed by a virtual machine. Other times, it's used to mean specifically code where an instruction is always encoded using exactly one byte for an "op-code".

Our own bytecode

In this post, we will invent our own bytecode with these characteristics:

• Each operation is a 1-byte "op-code", sometimes followed by a 4-byte operand that's interpreted as a 32-bit signed integer (little endian).
• The machine has a stack, where each value on the stack is a 32-bit signed integer.
• In the machine's model of the stack, stackptr[0] represents the value at the top of the stack, stackptr[1] the one before that, etc.

This is the set of instructions our bytecode language will have:

00000000: CONSTANT c:
Push 'c' onto the stack.
> push(c);

00000001: ADD:
Pop two values from the stack, push their
sum onto the stack.
> b = pop();
> a = pop();
> push(a + b);

00000010: PRINT:
Pop a value from the stack and print it.
> print(pop());

00000011: INPUT:
Read a value from some external input,
and push it onto the stack.
> push(input())

00000100: DISCARD:
Pop a value from the stack and discard it.
> pop();

00000101: GET offset:
Find the value at the 'offset' from the
top of the stack and push it onto the stack.
> val = stackptr[offset];
> push(val);

0000110: SET offset:
Pop a value from the stack, replace the value
at the 'offset' with the popped value.
> val = pop();
> stackptr[offset] = val;

00000110: CMP:
Compare two values on the stack, push -1 if
the first is smaller than the second, 1 if the
first is bigger than the second, and 0 otherwise.
> b = pop();
> a = pop();
> if (a > b) push(1);
> else if (a < b) push(-1);
> else push(0);

00000111: JGT offset:
Pop the stack, jump relative to the given 'offset'
if the popped value is positive.
> val = pop();
> if (val > 0) instrptr += offset;

00001000: HALT:
Stop execution

I'm sure you can imagine expanding this instruction set with more instructions. Maybe a SUB instruction, maybe more jump instructions, maybe more I/O. If you want, you can play along with this post and expand my code to implement your own custom instructions!

Throughout this blog post, I will be using an example program which multiplies two numbers together. Here's the program in pseudocode:

A = input()
B = input()

Accumulator = 0
do {
	Accumulator = Accumulator + A
	B = B - 1
} while (B > 0)

print(Accumulator)

(This program assumes B is greater than 0 for simplicity.)

Here's that program implemented in our bytecode language:

INPUT // A = input()
INPUT // B = input()

CONSTANT 0 // Accumulator = 0

// Loop body:

// Accumulator + A
GET 0
GET 3
ADD
// Accumulator = <result>
SET 0

// B - 1
GET 1
CONSTANT -1
ADD
// B = <result>
SET 1

// B CMP 0
GET 1
CONSTANT 0
CMP
// Jump to start of loop body if <result> > 0
// We get the value -43 by counting the bytes from
// the first instruction in the loop body.
// Operations are 1 byte, operands are 4 bytes.
JGT -43

// Accumulator
GET 0
// print(<result>)
PRINT

HALT

Note: If you're viewing this in a browser with JavaScript enabled, the above code should be interactive! Press the Step or Run buttons to execute it. The bar on the right represents the stack. The yellow box indicates the current stack pointer, a blinking green box means a value is being read, a blinking red box means a value is being written. The blue rectangle in the code area shows the instruction pointer. You can also edit the code; try your hand at writing your own program!

Here's a link which takes you directly to the interactive virtual machine.

You should take some moments to convince yourself that the bytecode truly reflects the pseudocode. Maybe you can even imagine how you could write a compiler which takes a syntax tree reflecting the source code and produces bytecode? (Hint: Every expression and sub-expression leaves exactly one thing on the stack.)

Implementing a bytecode interpreter

A bytecode interpreter can be basically just a loop with a switch statement. Here's my shot at implementing one in C for the bytecode language we invented:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

enum op {
	OP_CONSTANT, OP_ADD, OP_PRINT, OP_INPUT, OP_DISCARD,
	OP_GET, OP_SET, OP_CMP, OP_JGT, OP_HALT,
};

void interpret(unsigned char *bytecode, int32_t *input) {
	// Create a "stack" of 128 integers,
	// and a "stack pointer" which always points to the first free stack slot.
	// That means the value at the top of the stack is always 'stackptr[-1]'.
	int32_t stack[128];
	int32_t *stackptr = stack;

	// Create an instruction pointer which keeps track of where in the bytecode we are.
	unsigned char *instrptr = bytecode;

	// Some utility macros, to pop a value from the stack, push a value to the stack,
	// peek into the stack at an offset, and interpret the next 4 bytes as a 32-bit
	// signed integer to read an instruction's operand.
	#define POP() (*(--stackptr))
	#define PUSH(val) (*(stackptr++) = (val))
	#define STACK(offset) (*(stackptr - 1 - offset))
	#define OPERAND() ( \
		((int32_t)instrptr[1] << 0) | \
		((int32_t)instrptr[2] << 8) | \
		((int32_t)instrptr[3] << 16) | \
		((int32_t)instrptr[4] << 24))

	int32_t a, b;

	// This is where we just run one instruction at a time, using a switch statement
	// to figure out what to do in response to each op-code.
	while (1) {
		enum op op = (enum op)*instrptr;
		switch (op) {
		case OP_CONSTANT:
			PUSH(OPERAND());
			// We move past 5 bytes, 1 for the op-code, 4 for the 32-bit operand
			instrptr += 5; break;
		case OP_ADD:
			b = POP();
			a = POP();
			PUSH(a + b);
			// This instruction doesn't have an operand, so we move only 1 byte
			instrptr += 1; break;
		case OP_PRINT:
			a = POP();
			printf("%i\n", (int)a);
			instrptr += 1; break;
		case OP_INPUT:
			PUSH(*(input++));
			instrptr += 1; break;
		case OP_DISCARD:
			POP();
			instrptr += 1; break;
		case OP_GET:
			a = STACK(OPERAND());
			PUSH(a);
			instrptr += 5; break;
		case OP_SET:
			a = POP();
			STACK(OPERAND()) = a;
			instrptr += 5; break;
		case OP_CMP:
			b = POP();
			a = POP();
			if (a > b) PUSH(1);
			else if (a < b) PUSH(-1);
			else PUSH(0);
			instrptr += 1; break;
		case OP_JGT:
			a = POP();
			if (a > 0) instrptr += OPERAND();
			else instrptr += 5;
			break;
		case OP_HALT:
			return;
		}
	}
}

That's it. That's a complete virtual machine for our little bytecode language. Let's give it a spin! Here's a main function which exercises it:

int main(int argc, char **argv) {
	unsigned char program[] = {
		OP_INPUT, OP_INPUT,
		OP_CONSTANT, 0, 0, 0, 0,

		OP_GET, 0, 0, 0, 0,
		OP_GET, 3, 0, 0, 0,
		OP_ADD,
		OP_SET, 0, 0, 0, 0,

		OP_GET, 1, 0, 0, 0,
		OP_CONSTANT, 0xff, 0xff, 0xff, 0xff, // -1 32-bit little-endian (two's complement)
		OP_ADD,
		OP_SET, 1, 0, 0, 0,

		OP_GET, 1, 0, 0, 0,
		OP_CONSTANT, 0, 0, 0, 0,
		OP_CMP,
		OP_JGT, 0xd5, 0xff, 0xff, 0xff, // -43 in 32-bit little-endian (two's complement)

		OP_GET, 0, 0, 0, 0,
		OP_PRINT,

		OP_HALT,
	};
	int32_t input[] = {atoi(argv[1]), atoi(argv[2])};
	interpret(program, input);
}

Note: We use two's complement to represent negative numbers, because that's what the CPU does. A 32-bit number can represent the numbers between 0 and 4'294'967'295. Two's complement is a convention where the numbers between 0 and 2'147'483'647 are treated normally, and the numbers between 2'147'483'648 and 4'294'967'295 represent the numbers between -2'147'483'648 and -1.

Little-endian just means that order of the bytes are "swapped" compared to what you'd expect. For example, to express the number 35799 (10001011'11010111 in binary) as 2 bytes in little-endian, we put the last 8 bits first and the first 8 bits last: unsigned char bytes[] = {0b11010111, 0b10001011}. It's a bit counter-intuitive, but it's how most CPU architectures these days represent numbers larger than one byte.

When I compile and run the full C program with the inputs 3 and 5, it prints 15. Success!

If I instead ask it to calculate 1 * 100'000'000, my laptop (Apple M1 Pro, Apple Clang 14.0.0 with -O3) runs the program in 1.4 seconds. My desktop (AMD R9 5950x, GCC 12.2.0 with -O3) runs the same program in 1.1 seconds. The loop contains 12 instructions, and there are 6 instructions outside of the loop, so a complete run executes 100'000'000*12+6=1'200'000'006 instructions. That means my laptop runs 856 million bytecode instructions per second ("IPS") on average, and my desktop runs 1.1 billion instructions per second.

Note: The actual benchmarked code defines the program variable in a separate translation unit from the main function and interpret function, and link-time optimization is disabled. This prevents the compiler from optimizing based on the knowledge of the bytecode program.

Not bad, but can we do better?

Managing our own jump table

Looking at Godbolt, the assembly generated for our loop + switch is roughly like this:

loop:
	jmp jmp_table[*instrptr]

jmp_table:
	.quad case_op_constant
	.quad case_op_add
	.quad case_op_print
	.quad case_op_discard
	.quad case_op_get
	.quad case_op_set
	.quad case_op_cmp
	.quad case_op_jgt
	.quad case_op_halt

case_op_constant:
	; (code...)
	add instrptr, 5
	jmp loop

case_op_add:
	; (code...)
	add instrptr, 1
	jmp loop

; etc

Note: This isn't real x86 or ARM assembly, but it gives an idea of what's going on without getting into the weeds of assembly syntax.

We can see that the compiler generated a jump table; a table of memory addresses to jump to. At the beginning of each iteration of the loop, it looks up the target address in the jump table based on the opcode at the instruction pointer, then jumps to it. And at the end of executing each switch case, it jumps back to the beginning of the loop. This is fine, but it's a bit unnecessary to jump to the start of the loop just to immediately jump again based on the next op-code. We could just replace the jmp loop with jmp jmp_table[*instrptr] like this:

	jmp jmp_table[*instrptr]

jmp_table:
	.quad case_op_constant
	.quad case_op_add
	.quad case_op_print
	.quad case_op_discard
	.quad case_op_get
	.quad case_op_set
	.quad case_op_cmp
	.quad case_op_jgt
	.quad case_op_halt

case_op_constant:
	; code
	add instrptr, 5
	jmp jmp_table[*instrptr]

case_op_add:
	; code
	add instrptr, 1
	jmp jmp_table[*instrptr]

; etc

This has the advantage of using one less instruction per iteration, but that's negligible; completely predictable jumps such as our jmp loop are essentially free. However, there's a much bigger advantage: the CPU can exploit the inherent predictability of our bytecode instruction stream to improve its branch prediction. For example, a CMP instruction is usually going to be followed by the JGE instruction, so the CPU can start speculatively executing the JGE instruction before it's even done executing the CMP instruction. (At least that's what I believe is happeneing; figuring out why something is as fast or slow as it is, at an instruction-by-instruction level, is incredibly difficult on modern CPUs.)

Sadly, standard C doesn't let us express this style of jump table. But GNU C does! With GNU's Labels as Values extension, we can create our own jump table and indirect goto:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

enum op {
	OP_CONSTANT, OP_ADD, OP_PRINT, OP_INPUT, OP_DISCARD,
	OP_GET, OP_SET, OP_CMP, OP_JGT, OP_HALT,
};

void interpret(unsigned char *bytecode, int32_t *input) {
	int32_t stack[128];
	int32_t *stackptr = stack;
	unsigned char *instrptr = bytecode;

	#define POP() (*(--stackptr))
	#define PUSH(val) (*(stackptr++) = (val))
	#define STACK(offset) (*(stackptr - 1 - offset))
	#define OPERAND() ( \
		((int32_t)instrptr[1] << 0) | \
		((int32_t)instrptr[2] << 8) | \
		((int32_t)instrptr[3] << 16) | \
		((int32_t)instrptr[4] << 24))

	// Note: This jump table must be synchronized with the 'enum op',
	// so that `jmptable[op]` represents the label with the code for the instruction 'op'
	void *jmptable[] = {
		&&case_constant, &&case_add, &&case_print, &&case_input, &&case_discard,
		&&case_get, &&case_set, &&case_cmp, &&case_jgt, &&case_halt,
	};

	int32_t a, b;
	goto *jmptable[*instrptr];

case_constant:
	PUSH(OPERAND());
	instrptr += 5; goto *jmptable[*instrptr];
case_add:
	b = POP();
	a = POP();
	PUSH(a + b);
	instrptr += 1; goto *jmptable[*instrptr];
case_print:
	a = POP();
	printf("%i\n", (int)a);
	instrptr += 1; goto *jmptable[*instrptr];
case_input:
	PUSH(*(input++));
	instrptr += 1; goto *jmptable[*instrptr];
case_discard:
	POP();
	instrptr += 1; goto *jmptable[*instrptr];
case_get:
	a = STACK(OPERAND());
	PUSH(a);
	instrptr += 5; goto *jmptable[*instrptr];
case_set:
	a = POP();
	STACK(OPERAND()) = a;
	instrptr += 5; goto *jmptable[*instrptr];
case_cmp:
	b = POP();
	a = POP();
	if (a > b) PUSH(1);
	else if (a < b) PUSH(-1);
	else PUSH(0);
	instrptr += 1; goto *jmptable[*instrptr];
case_jgt:
	a = POP();
	if (a > 0) instrptr += OPERAND();
	else instrptr += 5;
	goto *jmptable[*instrptr];
case_halt:
	return;
}

With this interpreter loop, my laptop calculates 1 * 100'000'000 in 898ms, while my desktop does it in 1 second. It's interesting that Clang + M1 is significantly slower than GCC + AMD with the basic interpreter but significantly faster for this custom jump table approach. At least it's a speed-up in both cases.

Getting rid of the switch entirely with tail calls

Both of the implementations so far have essentially been of the form, "Look at the current instruction, and decide what code to run with some kind of jump table". But we don't actually need that. Instead of doing the jump table look-up every time, we could do the look-up once for every instruction before starting execution. Instead of an array of op codes, we could have an array of pointers to some machine code.

The easiest and most standard way to do this would be to have each instruction as its own function, and let that function tail-call the next function. Here's an implementation of that:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

union instr {
	void (*fn)(union instr *instrs, int32_t *stackptr, int32_t *input);
	int32_t operand;
};

#define POP() (*(--stackptr))
#define PUSH(val) (*(stackptr++) = (val))
#define STACK(offset) (*(stackptr - 1 - offset))
#define OPERAND() (instrs[1].operand)

static void op_constant(union instr *instrs, int32_t *stackptr, int32_t *input) {
	PUSH(OPERAND());
	instrs[2].fn(&instrs[2], stackptr, input);
}

static void op_add(union instr *instrs, int32_t *stackptr, int32_t *input) {
	int32_t b = POP();
	int32_t a = POP();
	PUSH(a + b);
	instrs[1].fn(&instrs[1], stackptr, input);
}

static void op_print(union instr *instrs, int32_t *stackptr, int32_t *input) {
	int32_t a = POP();
	printf("%i\n", (int)a);
	instrs[1].fn(&instrs[1], stackptr, input);
}

static void op_input(union instr *instrs, int32_t *stackptr, int32_t *input) {
	PUSH(*(input++));
	instrs[1].fn(&instrs[1], stackptr, input);
}

static void op_discard(union instr *instrs, int32_t *stackptr, int32_t *input) {
	POP();
	instrs[1].fn(&instrs[1], stackptr, input);
}

static void op_get(union instr *instrs, int32_t *stackptr, int32_t *input) {
	int32_t a = STACK(OPERAND());
	PUSH(a);
	instrs[2].fn(&instrs[2], stackptr, input);
}

static void op_set(union instr *instrs, int32_t *stackptr, int32_t *input) {
	int32_t a = POP();
	STACK(OPERAND()) = a;
	instrs[2].fn(&instrs[2], stackptr, input);
}

static void op_cmp(union instr *instrs, int32_t *stackptr, int32_t *input) {
	int32_t b = POP();
	int32_t a = POP();
	if (a > b) PUSH(1);
	else if (a < b) PUSH(-1);
	else PUSH(0);
	instrs[1].fn(&instrs[1], stackptr, input);
}

static void op_jgt(union instr *instrs, int32_t *stackptr, int32_t *input) {
	int32_t a = POP();
	if (a > 0) instrs += instrs[1].operand;
	else instrs += 2;
	instrs[0].fn(&instrs[0], stackptr, input);
}

static void op_halt(union instr *instrs, int32_t *stackptr, int32_t *input) {
	return;
}

This time, we can't just feed our interpreter an array of bytes as the bytecode, since there isn't really "an interpreter", there's just a collection of functions. We can manually create a program containing function pointers like this:

int main(int argc, char **argv) {
	union instr program[] = {
		{.fn = op_input}, {.fn = op_input},

		{.fn = op_constant}, {.operand = 0},

		{.fn = op_get}, {.operand = 0},
		{.fn = op_get}, {.operand = 3},
		{.fn = op_add},
		{.fn = op_set}, {.operand = 0},

		{.fn = op_get}, {.operand = 1},
		{.fn = op_constant}, {.operand = -1},
		{.fn = op_add},
		{.fn = op_set}, {.operand = 1},

		{.fn = op_get}, {.operand = 1},
		{.fn = op_constant}, {.operand = 0},
		{.fn = op_cmp},
		{.fn = op_jgt}, {.operand = -19},

		{.fn = op_get}, {.operand = 0},
		{.fn = op_print},

		{.fn = op_halt},
	};

	int32_t input[] = {atoi(argv[1]), atoi(argv[2])};
	int32_t stack[128];
	program[0].fn(program, stack, input);
}

And that works.

In a real use-case, you would probably want to have some code to automatically generate such an array of union instr based on bytecode, but we'll ignore that for now.

With this approach, my laptop calculates 1 * 100'000'000 in 841ms, while my desktop does it in only 553ms. It's not a huge improvement for the Clang + M1 case, but it's almost twice as fast with GCC + AMD! And compared to the previous approach, it's written in completely standard ISO C99, with the caveat that the compiler must perform tail call elimination. (Most compilers will do this at higher optimization levels, and most compilers let us specify per-function optimization levels with pragmas, so that's not a big issue in practice.)

Note: The timings from the benchmark includes the time it takes to convert the bytecode into this function pointer array form.

Final step: A compiler

All approaches so far have relied on finding ever faster ways to select which source code snippet to run next. As it turns out, the fastest way to do that is to simply put the right source code snippets after each other!

If we have the following bytecode:

CONSTANT 5
INPUT
ADD
PRINT

We can just generate C source code to do what we want:

PUSH(5);

PUSH(INPUT());

b = POP();
a = POP();
PUSH(a + b);

printf("%i\n", (int)POP());

We can then either shell out to GCC/Clang, or link with libclang to compile the generated C code. This also lets us take advantage of those projects's excellent optimizers.

Note: At this point, we don't have a "virtual machine" anymore.

One challenge is how to deal with jumps. The easiest solution from a code generation perspective is probably to wrap all the code in a switch statement in a loop:

int32_t index = 0;
while (1) {
	switch (index) {
	case 0:
		PUSH(5);

	case 5:
		PUSH(INPUT());

	case 6:
		a = POP();
		b = POP();
		PUSH(a + b);

	case 7:
		printf("%i\n", (int)POP());
	}
}

With this approach, a jump to instruction N becomes index = N; break;.

Note: Remember that in C, switch statement cases fall through to the next case unless you explicitly jump to the end with a break. So once the code for instruction 5 is done, we just fall through to instruction 6.

Here's my implementation:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

enum op {
	OP_CONSTANT, OP_ADD, OP_PRINT, OP_INPUT, OP_DISCARD,
	OP_GET, OP_SET, OP_CMP, OP_JGT, OP_HALT,
};

void write_operand(unsigned char *i32le, FILE *out) {
	fprintf(out, "    operand = %i;\n",
		(int)i32le[0] | (int)i32le[1] << 8 | (int)i32le[2] << 16 | (int)i32le[3] << 24);
}

void compile(unsigned char *bytecode, size_t size, FILE *out) {
	fputs(
		"#include <stdio.h>\n"
		"#include <stdint.h>\n"
		"#include <stdlib.h>\n"
		"\n"
		"int main(int argc, char **argv) {\n"
		"  int32_t stack[128];\n"
		"  int32_t *stackptr = stack;\n"
		"  char **inputptr = &argv[1];\n"
		"\n"
		"#define POP() (*(--stackptr))\n"
		"#define PUSH(val) (*(stackptr++) = (val))\n"
		"#define STACK(offset) (*(stackptr - 1 - offset))\n"
		"\n"
		"  int32_t a, b, operand;\n"
		"  int32_t index = 0;\n"
		"  while (1) switch (index) {\n",
		out);

	for (size_t i = 0; i < size;) {
		fprintf(out, "  case %zi:\n", i);

		enum op op = (enum op)bytecode[i];
		switch (op) {
		case OP_CONSTANT:
			write_operand(&bytecode[i + 1], out);
			fputs("    PUSH(operand);\n", out);
			i += 5; break;

		case OP_ADD:
			fputs(
				"    b = POP();\n"
				"    a = POP();\n"
				"    PUSH(a + b);\n",
				out);
			i += 1; break;

		case OP_PRINT:
			fputs(
				"    a = POP();\n"
				"    printf(\"%i\\n\", (int)a);\n",
				out);
			i += 1; break;

		case OP_INPUT:
			fputs("    PUSH(atoi(*(inputptr++)));\n", out);
			i += 1; break;

		case OP_DISCARD:
			fputs("    POP();\n", out);
			i += 1; break;

		case OP_GET:
			write_operand(&bytecode[i + 1], out);
			fputs(
				"    a = STACK(operand);\n"
				"    PUSH(a);\n",
				out);
			i += 5; break;

		case OP_SET:
			write_operand(&bytecode[i + 1], out);
			fputs(
				"    a = POP();\n"
				"    STACK(operand) = a;\n",
				out);
			i += 5; break;

		case OP_CMP:
			fputs(
				"    b = POP();\n"
				"    a = POP();\n"
				"    if (a > b) PUSH(1);\n"
				"    else if (a < b) PUSH(-1);\n"
				"    else PUSH(0);\n",
				out);
			i += 1; break;

		case OP_JGT:
			write_operand(&bytecode[i + 1], out);
			fprintf(out,
				"    a = POP();\n"
				"    if (a > 0) { index = %zi + operand; break; }\n",
				i);
			i += 5; break;

		case OP_HALT:
			fputs("    return 0;\n", out);
			i += 1; break;
		}
	}

	fputs(
		"  }\n"
		"\n"
		"  abort(); // If we get here, there's a missing HALT\n"
		"}",
		out);
}

If we run our compiler on the bytecode for our multiplication program, it outputs this C code:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

int main(int argc, char **argv) {
  int32_t stack[128];
  int32_t *stackptr = stack;
  char **inputptr = &argv[1];

  #define POP() (*(--stackptr))
  #define PUSH(val) (*(stackptr++) = (val))
  #define STACK(offset) (*(stackptr - 1 - offset))

  int32_t a, b, operand;
  int32_t index = 0;
  while (1) switch (index) {
  case 0:
    PUSH(atoi(*(inputptr++)));
  case 1:
    PUSH(atoi(*(inputptr++)));
  case 2:
    operand = 0;
    PUSH(operand);
  case 7:
    operand = 0;
    a = STACK(operand);
    PUSH(a);

  /* ... */

  case 49:
    b = POP();
    a = POP();
    if (a > b) PUSH(1);
    else if (a < b) PUSH(-1);
    else PUSH(0);
  case 50:
    operand = -43;
    a = POP();
    if (a > 0) { index = 50 + operand; break; }
  case 55:
    operand = 0;
    a = STACK(operand);
    PUSH(a);
  case 60:
    a = POP();
    printf("%i\n", (int)a);
  case 61:
    return 0;
  }

  abort(); // If we get here, there's a missing HALT
}

If we compile the generated C code with -O3, my laptop runs the 1 * 100'000'000 calculation in 204ms! That's over 4 times faster than the fastest interpreter we've had so far. That also means we're executing our 1'200'000'006 bytecode instructions at 5'882 million instructions per second! Its CPU only runs at 3'220 million CPU clock cycles per second, meaning it's spending significantly less than a clock cycle per bytecode instruction on average. My desktop with GCC is doing even better, executing all the code in 47ms, which means a whopping 25.7 billion instructions per second!

Note that in this particular case, the compiler is able to see that some instructions always happen after each other, which means it can optimize across bytecode instructions. For example, the bytecode contains a sequence GET 1; CONSTANT -1; ADD;, which the compiler is able to prove you won't ever jump into the middle of, so it optimizes out all the implied stack manipulation code; it's optimized into a single sub instruction which subtracts the constant 1 from one register and writes the result to another.

This is kind of an important point. The compiler can generate amazing code, if it can figure out which instructions (i.e switch cases) are potential jump targets. This is information you probably have access to in the source code, so it's worth thinking about how you can design your bytecode such that GCC or Clang can figure it out when looking at your compiler output. One approach could be to add "label" bytecode instructions, and only permit jumping to such a label. With our bytecode, the only jump instruction we have jumps to a known location, since the jump offset is an immediate operand to the instruction. If we added an instruction which reads the jump target from the stack instead, we might quickly get into situations where GCC/Clang has lost track of which instructions can be jump targets, and must therefore make sure not to optimize across instruction boundaries.

We can preventing the compiler from optimizing across instruction boundaries by inserting this code after the case 61: (the code for the HALT instruction):

if (argc > 100) { PUSH(argc); index = argc % 61; break; }

With this modification, every single instruction might be a branch target, so every instruction must make sense in its own right regardless of which instruction was executed before or how the stack looks.

This time, the 1 * 100'000'000 calculation happens in 550ms on my laptop with Clang, which is still not bad. It means we're executing 2'181 million bytecode instructions per second. My desktop is doing even better, at 168ms.

At this point, I got curious about whether it's the CPU or the compiler making the difference, so the next table contains all the benchmarks for both compilers on both systems.

I have no intelligent commentary on those numbers. They're all over the place. In the basic interpreter case for example, GCC is much faster than Clang on the AMD CPU, but Clang is much faster than GCC on the Apple CPU. It's the opposite in the custom jump table case, where GCC is much master than Clang on the Apple CPU, but Clang is much faster than GCC on the AMD CPU. The overall pattern we've been looking at holds though, for the most part: for any given CPU + compiler combination, every implementation I've introduced is faster than the one before it. The big exception is the tail call version, where the binary compiled by GCC performs horribly on the Apple CPU (even though it performs excellently on the AMD CPU!).

If anything though, this mess of numbers indicates the value of knowing about all the different possible approaches and choosing the right one for the situation. Which takes us to...

Bringing it all together

We have 4 different implementations of the same bytecode , all with different advantages and drawbacks. And even though every instruction does the same thing in every implementation, we have written 4 separate implementations of every instruction.

That seems unnecessary. After all, we know that ADD, in every implementation, will do some variant of this:

b = POP();
a = POP();
PUSH(a + b);
GO_TO_NEXT_INSTRUCTION();

What exactly it means to POP or to PUSH or to go to the next instruction might depend on the implementation, but the core functionality is the same for all of them. We can utilize that regularity to specify the instructions only once in a way that's re-usable across implementations using so-called X macros.

We create a file instructions.x which contains code to define all our instructions:

X(CONSTANT, 1, {
	PUSH(OPERAND());
	NEXT();
})

X(ADD, 0, {
	b = POP();
	a = POP();
	PUSH(a + b);
	NEXT();
})

// etc...

Let's say we want to create an instructions.h which contains an enum op with all the operation types and a const char *op_names[] which maps enum values to strings. We can implement that by doing something like this:

#ifndef INSTRUCTIONS_H
#define INSTRUCTIONS_H

enum op {
#define X(name, has_operand, code...) OP_ ## name,
#include "instructions.x"
#undef X
};

static const char *op_names[] = {
#define X(name, has_operand, code...) [OP_ ## name] = "OP_" #name,
#include "instructions.x"
#undef X
};

#endif

This code might look a bit confusing at first glance, but it makes sense: we have generic descriptions of instructions in the instructions.x file, and then we define a macro called X to extract information from those descriptions. It's basically a weird preprocessor-based application of the visitor pattern. In the above example, we use the instruction definitions twice: once to define the enum op, and once to define the const char *op_names[]. If we run the code through the preprocessor, we get something rouhly like this:

enum op {
OP_CONSTANT,
OP_ADD,
};

const char *op_names[] = {
[OP_CONSTANT] = "OP_CONSTANT",
[OP_ADD] = "OP_ADD",
};

Now let's say we want to write a function which executes an instruction. We could write that function like this:

void execute(enum op op) {
	switch (op) {
#define X(name, has_operand, code...) case OP_ ## name: code break;
#include "instructions.x"
#undef X
	}
}

Which expands to:

void execute(enum op op) {
	switch (op)
	case OP_CONSTANT:
		{
			PUSH(OPERAND());
			NEXT();
		} break;
	case OP_ADD:
		{
			b = POP();
			a = POP();
			PUSH(a + b);
			NEXT();
		} break;
	}
}

Note: We use a variadic argument for the code block because the C preprocessor has annoying splitting rules. Code such as X(FOO, 1, {int32_t a, b;}) would call the macro X with 4 arguments: FOO, 1, {int32_t a, and b;}. Using a variadic argument "fixes" this, because when we expand code in the macro body, the preprocessor will insert a comma between the arguments. You can read about more stupid preprocessor hacks here: https://mort.coffee/home/obscure-c-features/

This is starting to look reasonable, but it doesn't quite work. We haven't defined those PUSH/OPERAND/NEXT/POP macros, nor the a and b variables. We need to be a bit more rigorous about what exactly is expected by the instruction, and what's expected by the environment which the instruction's code is expanded into. So let's design a sort of "contract" between the instruction and the execution environment.

The environment must:

• Provide a POP() macro which pops the stack and evaluates to the result.
• Provide a PUSH(val) macro which push the value to the stack.
• Provide a STACK(offset) macro which evaluates to an lvalue for the stack value at offset.
• Provide an OPERAND() macro which evaluates to the current instruction's operand as a int32_t.
• Provide an INPUT() macro which reads external input and evaluates to the result.
• Provide a PRINT(val) macro which outputs the value somehow (such as by printing to stdout).
• Provide a GOTO_RELATIVE(offset) macro which jumps to currentInstruction + offset
• Provide a NEXT() macro which goes to the next instruction
• Provide a HALT() macro which halts execution.
• Provide the variables int32_t a and int32_t b as general-purpose variables. (This turns out to significantly speed up execution in some cases compared to defining the variables locally within the scope.)

As for the instruction:

• It must call X(name, has_operand, code...) with an identifier for name, a 0 or 1 for has_operand, and a brace-enclosed code block for code....
• The code block may only invoke OPERAND() if it has set has_operand to 1.
• The code block must only contain standard C code and calls to the macros we defined earlier.
• The code block must not try to directly access any other variables which may exist in the context in which it is expanded.
• The code block can assume that the following C headers are included: <stdio.h>, <stdlib.h>, <stdint.h>.
• The code must not change the stack pointer and dereference it in the same expression (essentially, no PUSH(STACK(1)), since there's no sequence point between the dereference and the increment).

With this, we can re-implement our basic bytecode interpreter:

#include "instructions.h"

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

void interpret(unsigned char *bytecode, int32_t *input) {
	int32_t stack[128];
	int32_t *stackptr = stack;
	unsigned char *instrptr = bytecode;

	int instrsize; // Will be initialized later

	#define POP() (*(--stackptr))
	#define PUSH(val) (*(stackptr++) = (val))
	#define STACK(offset) (*(stackptr - 1 - offset))
	#define OPERAND() ( \
		((int32_t)instrptr[1] << 0) | \
		((int32_t)instrptr[2] << 8) | \
		((int32_t)instrptr[3] << 16) | \
		((int32_t)instrptr[4] << 24))
	#define INPUT() (*(input++))
	#define PRINT(val) (printf("%i\n", (int)(val)))
	#define GOTO_RELATIVE(offset) (instrptr += (offset))
	#define NEXT() (instrptr += instrsize)
	#define HALT() return

	int32_t a, b;
	while (1) {
		switch ((enum op)*instrptr) {
#define X(name, has_operand, code...) \
		case OP_ ## name: \
			instrsize = has_operand ? 5 : 1; \
			code \
			break;
#include "instructions.x"
#undef X
		}
	}
}

And that's it! That's our whole generic basic bytecode interpreter, defined using the instruction definitions in instructions.x. And any time we add more bytecode instructions to instructions.x, the instructions are automatically added to the enum op and const char *op_names[] in instructions.h, and they're automatically supported by this new basic interpreter.

I won't deny that this style of code is a bit harder to follow than straight C code. However, I've seen VM with their own custom domain-specific languages and code generators to define instructions, and I find that much harder to follow than this preprocessor-based approach. Even though the C preprocessor is flawed in many ways, it has the huge advantage that C programmers already understand how it works for the most part, and they're used to following code which uses macros and includes. With decent comments in strategic places, I don't think this sort of "abuse" of the C preprocessor is wholly unreasonable. Your mileage may differ though, and my threshold for "too much preprocessor magic" might be set too high.

For completeness, let's amend instructions.x with all the instructions in the bytecode language I defined at the start of this post:

X(CONSTANT, 1, {
	PUSH(OPERAND());
	NEXT();
})

X(ADD, 0, {
	b = POP();
	a = POP();
	PUSH(a + b);
	NEXT();
})

X(PRINT, 0, {
	PRINT(POP());
	NEXT();
})

X(INPUT, 0, {
	PUSH(INPUT());
	NEXT();
})

X(DISCARD, 0, {
	(void)POP();
	NEXT();
})

X(GET, 1, {
	a = STACK(OPERAND());
	PUSH(a);
	NEXT();
})

X(SET, 1, {
	a = POP();
	STACK(OPERAND()) = a;
	NEXT();
})

X(CMP, 0, {
	b = POP();
	a = POP();
	if (a > b) PUSH(1);
	else if (a < b) PUSH(-1);
	else PUSH(0);
	NEXT();
})

X(JGT, 1, {
	a = POP();
	if (a > 0) { GOTO_RELATIVE(OPERAND()); }
	else { NEXT(); }
})

X(HALT, 0, {
	HALT();
})

Implementing the custom jump table variant and the tail-call variant using this X-macro system is left as an exercise to the reader. However, just to show that it's possible, here's the compiler variant implemented generically:

#include "instructions.h"

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

void compile(unsigned char *bytecode, size_t size, FILE *out) {
	fputs(
		"#include <stdio.h>\n"
		"#include <stdint.h>\n"
		"#include <stdlib.h>\n"
		"\n"
		"int main(int argc, char **argv) {\n"
		"  int32_t stack[128];\n"
		"  int32_t *stackptr = stack;\n"
		"  char **inputptr = &argv[1];\n"
		"\n"
		"#define POP() (*(--stackptr))\n"
		"#define PUSH(val) (*(stackptr++) = (val))\n"
		"#define STACK(offset) (*(stackptr - 1 - offset))\n"
		"#define OPERAND() operand\n"
		"#define INPUT() (atoi(*(inputptr++)))\n"
		"#define PRINT(val) printf(\"%i\\n\", (int)(val))\n"
		"#define GOTO_RELATIVE(offset) index += offset; break\n"
		"#define NEXT()\n"
		"#define HALT() return 0\n"
		"\n"
		"  int32_t a, b, operand;\n"
		"  int32_t index = 0;\n"
		"  while (1) switch (index) {\n",
		out);

	for (size_t i = 0; i < size;) {
		fprintf(out, "  case %zi:\n", i);

		enum op op = (enum op)bytecode[i];
		switch (op) {
#define X(name, has_operand, code...) \
		case OP_ ## name: \
			fprintf(out, "    index = %zi;\n", i); \
			i += 1; \
			if (has_operand) { \
				fprintf(out, "    operand = %i;\n", (int)( \
					((int32_t)bytecode[i + 0] << 0) | ((int32_t)bytecode[i + 1] << 8) | \
					((int32_t)bytecode[i + 2] << 16) | ((int32_t)bytecode[i + 3] << 24))); \
				i += 4; \
			} \
			fputs("    " #code "\n", out); \
			break;
#include "instructions.x"
#undef X
		}
	}

	fputs(
		"  }\n"
		"\n"
		"  abort(); // If we get here, there's a missing HALT\n"
		"}",
		out);
}

A word on real-world performance

I thought I should mention that the techniques described in this post won't magically make any interpreted language much faster. The main source of the performance differences we have explored here is due to the overhead involved in selecting which instruction to execute next; the code which runs between the instructions. By reducing this overhead, we're able to make our simple bytecode execute blazing fast. But that's really only because all our instructions are extremely simple.

In the case of something like Python, each instruction might be much more complex to execute. The BINARY_ADD operation, for example, pops two values from the stack, adds them together, and pushes the result onto the stack, much like how our bytecode's ADD operation does. However, our ADD operation knows that the two popped values are 32-bit signed integers. In Python, the popped values may be strings, they may be arrays, they may be numbers, they may be objects with a custom __add__ method, etc. This means that the time it takes to actually execute instructions in Python will dominate to the point that speeding up instruction dispatch is likely insignificant. Optimizing highly dynamic languages like Python kind of requires some form of tracing JIT to stamp out specialized functions which make assumptions about what types their arguments are, which is outside the scope of this post.

But that doesn't mean the speed-up I have shown here is unrealistic. If you're making a language with static types, you can have dedicated fast instructions for adding i32s, adding doubles, etc. And at that point, the optimizations shown in this post will give drastic speed-ups.

Further reading

• I watched this video about a year ago ago: Cheaply writing a fast interpreter - Neil Mitchell. I can't directly cite anything specific, but some ideas such as converting the instruction stream to an array of function pointers comes from that talk.
• Here's a discussion on how to do the custom jump table optimization in Zig: https://github.com/ziglang/zig/issues/8220
  • That thread links to this paper, which is also relevant: http://www.cs.toronto.edu/~matz/dissertation/matzDissertation-latex2html/node6.html

---

So those are my thoughts on speeding up virtual machine execution. If you want, you may check out my programming languages Gilia and osyris. Neither makes use of any of the techniques discussed in this post, but playing with Gilia's VM is what got me started down this path of exploring different techniques. If I ever get around to implementing these ideas into Gilia's VM, I'll add a link to the relevant parts of the source code here.

Date: 2022-07-23
Git: https://gitlab.com/mort96/blog/blob/published/content/00000-home/00014-tar.md

(If you're here from Google and just need help with tar being slow: If you trust the tar archive, extract with -P to make tar fast.)

A couple of days ago, I had a 518GiB tar.gz file (1.1 TiB uncompressed) that I had to extract. At first, GNU tar was doing a great job, chewing through the tar.gz at around 100MiB/s. But after a while, it slowed significantly; down to less than a kilobyte per second. pv's time estimate went from a bit over an hour, to multiple hours, to over a day, to almost a week. After giving it some time, and after failing to find anything helpful through Google, I decided that learning the tar file format and making my own tar extractor would probably be faster than waiting for tar. And I was right; before the day was over, I had a working tar extractor, and I had successfully extracted my 1.1TiB tarball.

I will explain why GNU tar is so slow later in this post, but first, let's take a look at:

The original tar file format

Tar is pretty unusual for an archive file format. There's no archive header, no index of files to fascilitate seeking, no magic bytes to help file and its ilk detect whether a file is a tar archive, no footer, no archive-wide metadata. The only kind of thing in a tar file is a file object.

So, how do these file objects look? Well, they start with a 512-byte file object header which looks like this:

struct file_header {
	char file_path[100];
	char file_mode[8];
	char owner_user_id[8];
	char owner_group_id[8];
	char file_size[12];
	char file_mtime[12];
	char header_checksum[8];
	char file_type;
	char link_path[100];

	char padding[255];
};

Followed by ceil(file_size / 512) 512-byte blocks of payload (i.e file contents).

We have most of the attributes we would expect a file object to have: the file path, the mode, the modification time (mtime), the user/group ID, the file size, and the file type. To support symlinks and hard links, there's also a link path.

The original tar file format defines these possible values for the file_type field:

• '0' (or sometimes '\0', the NUL character): Normal file
• '1': Hard link
• '2': Symbolic link

Future extensions to tar implements additional file types, among them '5', which represents a directory. Some old tar implementations apparently used a trailing slash '/' in a '0'-type file object to represent directories, at least according to Wikipedia.

You may think that the numeric values (file_mode, file_size, file_mtime, ...) would be encoded in base 10, or maybe in hex, or using plain binary numbers ("base 256"). But no, they're actually encoded as octal strings (with a NUL terminator, or sometimes a space terminator). Tar is the only file format I know of which uses base 8 to encode numbers. I don't quite understand why, since octal is neither space-efficient nor human-friendly. When representing numbers in this post, I will write them in decimal (base 10).

To encode a tar archive with one file called "hello.txt" and the content "Hello World", we need two 512-byte blocks:

1. Bytes 0-511: Header, type='0', file_path="./hello.txt", file_size=11
2. Bytes 512-1023: "Hello World", followed by 501 zero bytes

In addition, a tar file is supposed to end with 1024 zero-bytes to represent an end-of-file marker.

The two big limitations of the original tar format is that paths can't be longer than 100 characters, and files can't be larger than 8GiB (8^11 bytes). Otherwise though, I quite like the simplicity of the format. We'll discuss how various extensions address the limitations later, but first, let's try to implement an extractor:

(Feel free to skip this source code, but you should at least skim the comments)

// tarex.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/stat.h>
#include <string.h>

struct file_header {
	char file_path[100];
	char file_mode[8];
	char owner_user_id[8];
	char owner_group_id[8];
	char file_size[12];
	char file_mtime[12];
	char header_checksum[8];
	char file_type;
	char link_path[100];

	char padding[255];
};

// We don't bother with great error reporting, just abort on error
#define check(x) if (!(x)) abort()

// Utilities to abort on short read/write
#define xfread(ptr, size, f) check(fread(ptr, 1, size, f) == size)
#define xfwrite(ptr, size, f) check(fwrite(ptr, 1, size, f) == size)

// Tar represents all its numbers as octal
size_t parse_octal(char *str, size_t maxlen) {
	size_t num = 0;
	for (size_t i = 0; i < maxlen && str[i] >= '0' && str[i] <= '7'; ++i) {
		num *= 8;
		num += str[i] - '0';
	}

	return num;
}

// Extract one file from the archive.
// Returns 1 if it extracted something, or 0 if it reached the end.
int extract(FILE *f) {
	unsigned char header_block[512];
	xfread(header_block, sizeof(header_block), f);
	struct file_header *header = (struct file_header *)header_block;

	// The end of the archive is represented with blocks of all-zero content.
	// For simplicity, assume that if the file path is empty, the block is all zero
	// and we reached the end.
	if (header->file_path[0] == '\0') {
		return 0;
	}

	// The file path and link path fields aren't always 0-terminated, so we need to copy them
	// into our own buffers, otherwise we break on files with exactly 100 character paths.
	char file_path[101] = {0};
	memcpy(file_path, header->file_path, 100);
	char link_path[101] = {0};
	memcpy(link_path, header->link_path, 100);

	// We need these for later
	size_t file_size = parse_octal(header->file_size, sizeof(header->file_size));
	FILE *out_file = NULL;

	if (header->file_type == '0' || header->file_type == '\0') {
		// A type of '0' means that this is a plain file.
		// Some early implementations also use a NUL character ('\0') instead of an ASCII zero.

		fprintf(stderr, "Regular file: %s\n", file_path);
		out_file = fopen(file_path, "w");
		check(out_file != NULL);

	} else if (header->file_type == '1') {
		// A type of '1' means that this is a hard link.
		// That means we create a hard link at 'file_path' which links to the file at 'link_path'.

		fprintf(stderr, "Hard link: %s -> %s\n", file_path, link_path);
		check(link(link_path, file_path) >= 0);

	} else if (header->file_type == '2') {
		// A type of '2' means that this is a symbolic link.
		// That means we create a symlink at 'file_path' which links to the file at 'link_path'.

		fprintf(stderr, "Symbolic link: %s -> %s\n", file_path, link_path);
		check(symlink(link_path, file_path) >= 0);

	} else if (header->file_type == '5') {
		// A type of '5' means that this is a directory.

		fprintf(stderr, "Directory: %s\n", file_path);
		check(mkdir(file_path, 0777) >= 0);

		// Directories sometimes use the size field, but they don't contain data blocks.
		// Zero out file_size to avoid skipping entries.
		file_size = 0;

	} else {
		// There are other possible fields added by various tar implementations and standards,
		// but we'll ignore those for this implementation.
		fprintf(stderr, "Unsupported file type %c: %s\n", header->file_type, file_path);
	}

	// We have read the header block, now we need to read the payload.
	// If we're reading a file (i.e if 'outfile' is non-NULL) we will also write the body,
	// but otherwise we'll just skip it.
	char block[512];
	while (file_size > 0) {
		xfread(block, sizeof(block), f);
		size_t n = file_size > 512 ? 512 : file_size;

		file_size -= n;
		if (out_file != NULL) {
			xfwrite(block, n, out_file);
		}
	}

	if (out_file != NULL) {
		check(fclose(out_file) >= 0);
	}

	// Indicate that we have successfully extracted a file object, and are ready to read the next
	return 1;
}

int main() {
	while (extract(stdin));
}

Let's see it in action:

~/tarex $ ls
tarex.c testdir
~/tarex $ gcc -o tarex tarex.c
~/tarex $ tree
.
├── tarex.c
├── tarex
└── testdir
    ├── hello-symlink -> hello.txt
    ├── hello.txt
    └── subdir
        └── file.txt

~/tarex $ tar c testdir >testdir.tar
~/tarex $ mkdir extract && cd extract

~/tarex/extract $ ../tarex <../testdir.tar
Directory: testdir/
Symbolic link: testdir/hello-symlink -> hello.txt
Directory: testdir/subdir/
Regular file: testdir/hello.txt
Regular file: testdir/subdir/file.txt

~/tarex/extract $ tree
.
└── testdir
    ├── hello-symlink -> hello.txt
    ├── hello.txt
    └── subdir
        └── file.txt

The UStar file format

The first major extension to the tar file format we will look at is the UStar format, which increases the file length limit to 256 characters and adds some new file types. The header is expanded to this:

struct file_header {
	// Original tar header fields
	char file_path[100];
	char file_mode[8];
	char owner_user_id[8];
	char owner_group_id[8];
	char file_size[12];
	char file_mtime[12];
	char header_checksum[8];
	char file_type;
	char link_path[100];

	// New UStar fields
	char magic_bytes[6];
	char version[2];
	char owner_user_name[32];
	char owner_group_name[32];
	char device_major_number[8];
	char device_minor_number[8];
	char prefix[155];

	char padding[12];
};

We now have some magic bytes (defined to be "ustar\0" for the UStar format), as well as the owner user/group names. But most importantly, we have a prefix field, which allows up to 256 character file paths. With UStar, instead of just extracting the bytes from file_path and link_path like before, we must construct a file path like this:

void read_path(char dest[257], char path[100], char prefix[100]) {
	// If there's no prefix, use name directly
	if (prefix[0] == '\0') {
		memcpy(dest, path, 100);
		dest[100] = '\0';
		return;
	}

	// If there is a prefix, the path is: <prefix> '/' <path>
	size_t prefix_len = strnlen(prefix, 155);
	memcpy(dest, prefix, prefix_len);
	dest[prefix_len] = '/';
	memcpy(&dest[prefix_len + 1], path, 100);
	dest[256] = '\0';
}

int extract(FILE *f) {
	/* ... */

	char file_path[257];
	read_path(file_path, header->file_path, header->prefix);
	char link_path[257];
	read_path(link_path, header->link_path, header->prefix);

	/* ... */
}

The original tar format had the file types '0' (or '\0'), '1' and '2', for regular files, hard links and symlinks. UStar defines these additional file types:

• '3' and '4': Character devices and block devices. These are the reason for the new device_major_number and device_minor_number fields.
• '5': Directories.
• '6': FIFO files.
• '7': Contiguous files. This type isn't really used much these days, and most implementations just treat it as a regular file.

This is definitely an improvement, but we can still only encode up to 256 character long paths. And that 8GiB file size limit still exists. Which leads us to:

The pax file format

The POSIX.1-2001 standard introduced the pax command line tool, and with it, a new set of extensions to the tar file format. This format is identical to UStar, except that it adds two new file object types: 'x' and 'g'. Both of these types let us define "extended header records", as the spec calls it. Records set with 'x' apply to only the next file, while records set with 'g' apply to all following files.

With this new extended header, we can encode the access and modification times with more precision, user/group IDs above 8^7, file sizes over 8^11, file paths of arbitrary length, and a whole lot more. The records are in the payload of the extended headr file object, and use a simple length-prefixed key/value syntax. To represent our "hello.txt" example file with an access time attribute, we need these four 512-byte blocks:

1. Header, type='x', file_size=30
2. "30 atime=1658409251.551879906\n", followed by 482 zeroes
3. Header, type='0', file_path="hello.txt", file_size=11
4. "Hello World", followed by 501 zero bytes

Interestingly, these extended header records all seem to use decimal (base 10). On the one hand, using base 10 makes sense, but on the other hand, wouldn't it be nice to stick to one way of representing numbers?

Anyways, we can see that the file format has become quite complex now. Just the file path can be provided in any of four different ways:

• The full path might be in the file_path field.
• The path might be a combination of the prefix and the file_path fields.
• The previous file object might've been an 'x' type record with set a path property.
• There might've been some 'g' type file object earlier in the archive which set a path property.

The GNU tar file format

GNU tar has its own file format, called gnu, which is different from the pax format. Like pax, the gnu format is based on UStar, but it has a different way of encoding arbitrary length paths and large file sizes:

• It introduces the 'L' type, where the payload of the file object represents the file_path of the next file object.
• It introduces the 'K' type, where the payload of the file object represents the link_path of the next file object.
• A link with both a long file_path and a long link_path is preceeded by both an 'L' type file object and a 'K' type file object. The order isn't specified from what I can tell.
• If a file is over 8GiB, it will set the high bit of the first character in file_size, and the rest of the string is parsed as base 256 (i.e it's treated as a 95-bit integer, big endian).

In some ways, I prefer this approach over the pax approach, since it's much simpler; the pax format requires the extractor to parse the record grammar. On the other hand, the pax format is both more space efficient and vastly more flexible.

In any case, the result is that a tar extractor which wants to support both pax tar files and GNU tar files needs to support 5 different ways of reading the file path, 5 different ways of reading the link path, and 3 different ways of reading the file size.

Whatever happened to the nice and simple format we started out with?

Why GNU tar extracts in quadratic time

Our simple tar extraction implementation has what could be considered a quite serious security bug: It allows people to put files outside the directory we're extracting to. Nothing is stopping an evil arcive from containing a file object with file_path="../hello.txt". You might try to fix that by just disallowing file objects from using ".." as a path component, but it's not that simple. Consider the following sequence of file objects:

1. Symlink, file_path="./foo", link_path=".."
2. Normal file, file_path="./foo/hello.txt"

We want to allow symlinks which point to their parent directory, since there are completely legitimate use cases for that. We could try to figure out whether a symlink will end up pointing to somewhere outside of the extraction directory, but that gets complicated real fast when you have to consider symlinks to symlinks and hard links to symlinks. It might be possible to do correctly, but it's not the solution GNU tar goes for.

When GNU tar encounters a hard link or symlink with ".." as a path component in its link_path, tar will create a regular file in its place as a placeholder, and put a note about the delayed link in a linked list datastructure. When it's done extracting the entire archive, it will go through the whole list of delayed links and replace the placeholders with proper links. So far, so good.

The problem comes when trying to extract a hard link which doesn't contain ".." as a path component in its link_path. GNU tar wants to create such hard links immediately if it can. But it can't create a hard link if the target is occupied by a placeholder file. That means, every time GNU tar wants to create a hard link, it first has to walk the entire linked list of delayed links and see if the target is a delayed link. If the target is a delayed link, the new link must also be delayed.

Your time complexity alarm bells should be starting to ring now. For every hard link, we walk the list of all delayed links. But it actually gets worse; for reasons I don't quite understand yet, tar will actually go through the entire list of delayed links again if it found out that it can create the link immediately. So for all "normal" hard links, it has to go through the entire linked list of delayed links twice.

If you're a bit crafty, you can construct a tar archive which GNU tar extracts in precisely O(n^2) time; you just need to alternate between links whose link_path has ".." as a path component and thus get delayed, and "normal" hard links which don't get delayed. If you're a bit unlucky, you might have a totally benign tarball which nevertheless happens to contain a bunch of symlinks which refer to files in a parent directory, followed by a bunch of normal hard links. This is what had happened to me. My tarball happened to contain over 800 000 links with ".." as a path component. It also happened to contain over 5.4 million hard links. Every one of those hard links had to go through the entire list of every hitherto deferred link. No wonder tar got slow.

If you ever find yourself in this situation, pass the --absolute-paths (or -P) parameter to tar. Tar's documentation says this about --absolute-paths:

Preserve pathnames. By default, absolute pathnames (those that begin with a / character) have the leading slash removed both when creating archives and extracting from them. Also, tar will refuse to extract archive entries whose pathnames contain .. or whose target directory would be altered by a symlink. This option suppresses these behaviors.

You would never guess it from reading the documentation, but when you pass --absolute-paths during extraction, tar assumes that the archive is benign and the whole delayed linking mechanism is disabled. Make sure you trust the tar archive though! When extracted with --absolute-paths, a malicious archive will be able to put files anywhere it wants.

I'm absolutely certain that it's possible to make GNU tar extract in O(n) without --absolute-paths by replacing the linked list with a hash map. But that's an adventure for another time.

References

These are the documents I've drawn information from when researching for my tar extractor and this blog post:

• The Wikipedia article on tar: https://en.wikipedia.org/wiki/Tar_(computing)#File_format
• The POSIX spec for pax: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html#tag_20_92_13_01
• The GNU tar file format docs: https://www.gnu.org/software/tar/manual/html_node/Standard.html
• The GNU tar source code: https://git.savannah.gnu.org/cgit/tar.git/tree/ (in particular, extract.c)

If I have represented anything inaccurately in this post, please do correct me.

Date: 2020-08-03
Git: https://gitlab.com/mort96/blog/blob/published/content/00000-home/00013-fast-cpp-embeds.md
Tool: https://github.com/mortie/strliteral

It's really common to want to embed some static data into a binary. Game developers want to embed their shaders. Developers of graphical apps may want to embed sounds or icons. Developers of programming language interpreters may want to embed their language's standard library. I have many times built software whose GUI is in the form of a web app served from a built-in HTTP server, where I want to embed the HTML/JS/CSS into the binary.

Since neither C nor C++ currently has a built-in way to embed files, we use work-arounds. These usually fall into one of two categories: Either we use toolchain-specific features to generate object files with the data exposed as symbols, or we generate C code which we subsequently compile to object files. Since the toolchain-specific features are, well, toolchain-specific, people writing cross-platform software generally prefer code generation.

The most common tool I'm aware of to generate C code for embedding data is xxd, whose -i option will generate C code with an unsigned char array literal.

Given the following input text:

<html>
	<head>
		<title>Hello World</title>
	</head>
	<body>
		Hello World
	</body>
</html>

The command xxd -i index.html will produce this C code:

unsigned char index_html[] = {
  0x3c, 0x68, 0x74, 0x6d, 0x6c, 0x3e, 0x0a, 0x09, 0x3c, 0x68, 0x65, 0x61,
  0x64, 0x3e, 0x0a, 0x09, 0x09, 0x3c, 0x74, 0x69, 0x74, 0x6c, 0x65, 0x3e,
  0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64, 0x3c,
  0x2f, 0x74, 0x69, 0x74, 0x6c, 0x65, 0x3e, 0x0a, 0x09, 0x3c, 0x2f, 0x68,
  0x65, 0x61, 0x64, 0x3e, 0x0a, 0x09, 0x3c, 0x62, 0x6f, 0x64, 0x79, 0x3e,
  0x0a, 0x09, 0x09, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x57, 0x6f, 0x72,
  0x6c, 0x64, 0x0a, 0x09, 0x3c, 0x2f, 0x62, 0x6f, 0x64, 0x79, 0x3e, 0x0a,
  0x3c, 0x2f, 0x68, 0x74, 0x6d, 0x6c, 0x3e, 0x0a
};
unsigned int index_html_len = 92;

This works fairly well. Any C or C++ compiler can compile that code and produce an object file with our static data, which we can link against to embed that data into our binary. All in a cross-platform and cross-toolchain way.

There's just one problem: It's slow. Really slow. On my laptop, embedding a megabyte this way takes 2 seconds using g++. Embedding one decent quality MP3 at 8.4MB takes 23 seconds, using 2.5 gigabytes of RAM.

Whether or not we should embed files of that size into our binaries is a question I won't cover in this article, and the answer depends a lot on context. Regardless, processing data at just over 400kB per second is objectively terrible. We can do so much better.

The main reason it's so slow is that parsing arbitrary C++ expressions is actually really complicated. Every single byte is a separate expression, parsed using a complex general expression parser, presumably separately allocated as its own node in the syntax tree. If only we could generate code which combines lots of bytes of data into one token...

I wrote a small tool, called strliteral, which outputs data as a string literal rather than a character array. The command strliteral index.html will produce this C code:

const unsigned char index_html[] =
	"<html>\n\t<head>\n\t\t<title>Hello World</title>\n\t</head>\n\t<body>\n\t\tHello"
	" World\n\t</body>\n</html>\n";
const unsigned int index_html_len = 92;

It should come as no surprise that this is many times faster to parse than the character array approach. Instead of invoking a full expression parser for each and every byte, most of the time will just be spent in a tight loop which reads bytes and appends them to an array. The grammar for a string literal is ridiculously simple compared to the grammar for an array literal.

Compared to xxd's 23 seconds and 2.5GB of RAM usage for my 8.4MB file, my strliteral tool produces code which g++ can compile in 0.6 seconds, using only 138 megs of RAM. That's almost a 40x speed-up, and an 18x reduction in RAM usage. It's processing data at a rate of 15MB per second, compared to xxd's 0.4MB per second. As a bonus, my tool generates 26MB of C code, compared to xxd's 52MB.

Here's how that song looks, encoded with strliteral:

const unsigned char bippety_boppety_mp3[] =
	"\377\373\340D\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
	"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
	"\242\240,\2253]5\234\316\020\234\375\246\072D\307\027\203R\030\307\221\314`\243B\370\013\301\220\256"
	"\235\036\243E\246\331\216\026\004\341\362uU&\255\030@,\227\021q]1\231L\304\010E\311\231\005W\231\210"
	"j-\"\374|\210II0\221\026\045\021}qC\206\t9<\320\013\246w\350\263EmH`#\262\037\252\304\272\340\355`7\217"
	"\343*\016\236\320\345oa\217\204\361~k\224\255|\301cy\371\375\034\366K'\236\037\271\204\371\275\rV\267"
	"\252\020\245\322~\233\350\222\343\347\204\332\340~\236-\355S.W\045\365\301=\\+\236\270F\312\246g\266"
	"CX2\376\265V\242T0\337I\031\343\347\320\336\322\020\016\020H\250\007]\031\201\235\025\300h\2628d\000"
	/* 249707 lines snipped */
	"\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252"
	"\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252\252"
	"\252TAG\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
	"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
	"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
	"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000Created with LMMS\000"
	"\000\000\000\000\000\000\000\000\000\000\000\000\377";
unsigned int bippety_boppety_mp3_len = 8779359;

The difference is even bigger when processing mostly-ASCII text rather than binary data. Since xxd produces the same 6 bytes of source code for every byte of input (0x, two hex digits, comma, space), the data itself doesn't matter. However, strliteral produces 4 bytes of source code (\, then three octal digits) for every "weird" character, but just one byte of source code for every "regular" ASCII character.

Graphs

I wrote some benchmarking code to compare various aspects of xxd and strliteral. All times are measured using an Intel Core i7-8705G CPU in a Dell XPS 15 9575. g++ and xxd are from the Ubuntu 20.04 repositories. strliteral is compiled with gcc -O3 -o strliteral strliteral.c using GCC 9.3.0. The benchmarking source code can be found here: https://github.com/mortie/strliteral/tree/master/benchmark

Here's a graph which shows exactly how the two tools compare, across a range of input sizes, given either text or random binary data:

The 70x number in the title comes from this graph. The 60ms spent compiling strliteral-generated code is 72x faster than the 4324ms spent compiling xxd-generated code. Comparing random binary data instead of text would show a lower - though still respectable - speed-up of 25x.

Though most of the time spent when embedding data with xxd comes from the compiler, the xxd tool itself is actually fairly slow too:

Those ~200 milliseconds xxd takes to generate code for a 2MB file isn't very significant compared to the 4.3 second compile time, but if strliteral was equally slow, 75% of the time would've been spent generating code as opposed to compiling code. Luckily, strliteral runs through 2MB of text in 11ms.

Looking at the xxd source code, the reason it's so slow seems to be that it prints every single byte using a call to fprintf:

while ((length < 0 || p < length) && (c = getc(fp)) != EOF)
  {
    if (fprintf(fpo, (hexx == hexxa) ? "%s0x%02x" : "%s0X%02X",
                (p % cols) ? ", " : &",\n  "[2*!p],  c) < 0)
      die(3);
    p++;
  }

Finally, here's a graph over g++'s memory usage:

Caveats

Update: In the reddit discussion, someone pointed out that MSVC, Microsoft's compiler, has a fairly low maximum string length limit (the exact limit is fairly complicated). I had assumed that any modern compiler would just keep strings in a variable sized array. Maybe strliteral will eventually grow an MSVC-specific workaround, but until then, using a better compiler like Clang or GCC on Windows is an option.

Using string literals for arbitrary binary data is a bit more complicated than using an array with integer literals. Both xxd and strliteral might have trouble in certain edge cases, such as when cross-compiling if the host and target disagrees on the number of bits in a byte. Using string literals adds an extra complication due to the distinction between the "source character set" and the "execution character set". The C11 spec (5.2.1p2) states:

In a character constant or string literal, members of the execution character set shall be represented by corresponding members of the source character set or by escape sequences consisting of the backslash \ followed by one or more characters.

If you run strliteral on a file which contains the byte 97, it will output the code const unsigned char data[] = "a";. If that C code is compiled with a "source character set" of ASCII and an "execution character set" of EBCDIC, my understanding of the standard text is that the ASCII "a" (byte 97) will be translated to the EBCDIC "a" (byte 129). Whether that's even a bug or not depends on whether the intention is to embed binary data or textual data, but it's probably not what people expect from a tool to embed files.

This should only ever become an issue if you're compiling with different source and execution charsets, where the source charset and execution charset aren't based on ASCII. Compiling with a UTF-8 source charset and an EBCDIC execution charset will cause issues, but since all non-ASCII characters are printed as octal escape sequences, compiling with e.g a UTF-8 source charset and a LATIN-1 execution charset isn't an issue.

It seems extremely unlikely to me that someone will compile with a source charset and an execution charset which are both different and not based on ASCII, but I suppose it's something to keep in mind. If it does become an issue, the --always-escape option will cause strliteral to only generate octal escape sequences. That should work the same as xxd -i in all cases, just faster.

Some implementation notes

C is a weird language. For some reason, probably to better support systems where bytes are bigger than 8 bits, hex string escapes like "\x6c" can be an arbitrary number of characters. "\xfffff" represents a string with one character whose numeric value is 1048575. That obviously won't work on machines with 8-bit bytes, but it could conceivably be useful on a machine with 24-bit bytes, so it's allowed. Luckily, octal escapes are at most 3 numbers, so while "\xf8ash" won't work, "\370ash" will.

C also has a concept of trigraphs and digraphs, and they're expanded even within string literals. The string literals "??(" and "[" are identical (at least in C, and in C++ before C++17). Currently, strliteral just treats ?, : and % as "special" characters which are escaped, which means no digraphs or trigraphs will ever appear in the generated source code. I decided it's not worth the effort to add more "clever" logic which e.g escapes a ( if the two preceeding characters are question marks.

Date: 2020-01-27
Git: https://gitlab.com/mort96/blog/blob/published/content/00000-home/00012-clang-compiler-hacking.md

I happen to think that the current lambda syntax for C++ is kind of verbose. I'm not the only one to have thought that, and there has already been a paper discussing a possible abbreviated lambda syntax (though it was rejected).

In this blog post, I will detail my attempt to implement a sort of simplest possible version of an abbreviated lambda syntax. Basically, this:

[](auto &&a, auto &&b) => a.id() < b.id();

should mean precisely:

[](auto &&a, auto &&b) { return a.id() < b.id(); };

I will leave a discussion about whether that change is worth it or not to the end. Most of this article will just assume that we want that new syntax, and discuss how to actually implement it in Clang.

If you want to read more discussion on the topic, I wrote a somewhat controversial post on Reddit discussing why I think it might be a good idea.

Here's the implementation I will go through in this post: https://github.com/mortie/llvm/commit/e4726dc9d9d966978714fc3d85c6e9c335a38ab8 - 28 additions, including comments and whitespace, across 3 files.

Getting the Clang code

This wasn't my first time compiling Clang, but it was my first time downloading the source code with the intent to change it.

LLVM has a nice page which details getting and building Clang, but the tl;dr is:

git clone https://github.com/llvm/llvm-project.git
cd llvm-project && mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=`pwd`/inst -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE=Release ../llvm
make -j 8
make install

A few points to note:

• The build will take a long time. Clang is big.
• I prefer -DCMAKE_BUILD_TYPE=Release because it's way faster to build. Linking Clang with debug symbols and everything takes ages and will OOM your machine.
• This will install your built clang to inst (short for "install"). The clang binary itself will be in inst/bin/clang.

Now that we have a clang setup, we can have a look at how the project is laid out, and play with it.

Changing the Clang code

The feature I want to add is very simple: Basically, I want [] => 10 to mean the exact same thing as [] { return 10; }. In order to understand how one would achieve that, an extremely short introduction to how compilers work is necessary:

Our code is just a sequence of bytes, like [] => 10 + 20. In order for Clang to make sense of that, it will go through many steps. We can basically divide a compiler into two parts: the "front-end", which goes through many steps to build a thorough understanding of the code as a tree structure, and the "back-end" which goes through many steps to remove information, eventually ending up with a simple series of bytes again, but this time as machine code instead of ASCII.

We'll ignore the back-end for now. The front-end basically works like this:

1. Split the stream of bytes into a stream of tokens. This step turns [] => 10 + 20 into something like (open-bracket) (close-bracket) (fat-arrow) (number: 10) (plus) (number: 20).
2. Go through those tokens and construct a tree. This step turns the sequence of tokens into a tree: (lambda-expression (body (return-statement (add-expression (number 10) (number 20))))) (Yeah, this looks a lot like Lisp. There's a reason people say Lisp basically has no syntax; you're just writing out the syntax tree by hand.)
3. Add semantic information, such as types.

The first phase is usually called lexical analysis, or tokenization, or scanning. The second phase is what we call parsing. The third phase is usually called semantic analysis or type checking.

Well, the change I want to make involves adding a new token, the "fat arrow" token =>. That means we'll have to find out how the lexer (or tokenizer) is implemented; where it keeps its list of valid tokens types, and where it turns the input text into tokens. After some grepping, I found the file clang/include/clang/Basic/TokenKinds.def, which includes a bunch of token descriptions, such as PUNCTUATOR(arrow, "->"). This file seems to be a "supermacro"; a file which exists to be included by another file as a form of macro expansion.

I added PUNCTUATOR(fatarrow, "=>") right below the PUNCTUATOR(arrow, "->") line.

Now that we have defined our token, we need to get the lexer to actually generate it.

After some more grepping, I found clang/lib/Lex/Lexer.cpp, where the Lexer::LexTokenInternal function is what's actually looking at characters and deciding what tokens they represent. It has a case statement to deal with tokens which start with an = character:

case '=':
	Char = getCharAndSize(CurPtr, SizeTmp);
	if (Char == '=') {
		// If this is '====' and we're in a conflict marker, ignore it.
		if (CurPtr[1] == '=' && HandleEndOfConflictMarker(CurPtr-1))
			goto LexNextToken;

		Kind = tok::equalequal;
		CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);
	} else {
		Kind = tok::equal;
	}
	break;

Given that, the change to support my fatarrow token is really simple:

case '=':
	Char = getCharAndSize(CurPtr, SizeTmp);
	if (Char == '=') {
		// If this is '====' and we're in a conflict marker, ignore it.
		if (CurPtr[1] == '=' && HandleEndOfConflictMarker(CurPtr-1))
		goto LexNextToken;

		Kind = tok::equalequal;
		CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);

	// If the first character is a '=', and it's followed by a '>', it's a fat arrow
	} else if (Char == '>') {
		Kind = tok::fatarrow;
		CurPtr = ConsumeChar(CurPtr, SizeTmp, Result);

	} else {
		Kind = tok::equal;
	}
	break;

---

Now that we have a lexer which generates a tok::fatarrow any time it encounters a => in our code, we can start changing the parser to make use of it.

Since I want to change lambda parsing, the code which parses a lamba seems like a good place to start (duh). I found that in a file called clang/lib/Parse/ParseExprCXX.cpp, in the function ParseLambdaExpressionAfterIntroducer. Most of the function deals with things like the template parameter list and trailing return type, which I don't want to change, but the very end of the function contains this gem:

// Parse compound-statement.
if (!Tok.is(tok::l_brace)) {
	Diag(Tok, diag::err_expected_lambda_body);
	Actions.ActOnLambdaError(LambdaBeginLoc, getCurScope());
	return ExprError();
}

StmtResult Stmt(ParseCompoundStatementBody());
BodyScope.Exit();
TemplateParamScope.Exit();

if (!Stmt.isInvalid() && !TrailingReturnType.isInvalid())
	return Actions.ActOnLambdaExpr(LambdaBeginLoc, Stmt.get(), getCurScope());

Actions.ActOnLambdaError(LambdaBeginLoc, getCurScope());
return ExprError();

1. If the next token isn't an opening brace, error.
2. Parse a compound statement body (i.e consume a {, read statements until the }).
3. After some housekeeping, act on the now fully parsed lambda expression.

In principle, what we want to do is to check if the next token is a => instead of a {; if it is, we want to parse an expression instead of a compound statement, and then somehow pretend that the expression is a return statement. Through some trial, error and careful copy/pasting, I came up with this block of code which I put right before the if (!Tok.is(tok::l_brace)):

// If this is an arrow lambda, we just need to parse an expression.
// We parse the expression, then put that expression in a return statement,
// and use that return statement as our body.
if (Tok.is(tok::fatarrow)) {
	SourceLocation ReturnLoc(ConsumeToken());

	ExprResult Expr(ParseExpression());
	if (Expr.isInvalid()) {
		Actions.ActOnLambdaError(LambdaBeginLoc, getCurScope());
		return ExprError();
	}

	StmtResult Stmt = Actions.ActOnReturnStmt(ReturnLoc, Expr.get(), getCurScope());

	BodyScope.Exit();
	TemplateParamScope.Exit();

	if (!Stmt.isInvalid() && !TrailingReturnType.isInvalid())
		return Actions.ActOnLambdaExpr(LambdaBeginLoc, Stmt.get(), getCurScope());

	Actions.ActOnLambdaError(LambdaBeginLoc, getCurScope());
	return ExprError();
}

// Otherwise, just parse a compound statement as usual.
if (!Tok.is(tok::l_brace)) ...

This is really basic; if the token is a => instead of a {, parse an expression, then put that expression into a return statement, and then use that return statement as our lambda's body.

And it works! Lambda expressions with fat arrows are now successfully parsed as if they were regular lambdas whose body is a single return statement:

Was it worth it?

Implementing this feature into Clang was definitely worth it just to get more familiar with how the code base works. However, is the feature itself a good idea at all?

I think the best way to decide if a new syntax is better or not is to look at old code which could've made use of the new syntax, and decide if the new syntax makes a big difference. Therefore, and now that I have a working compiler, I have gone through all the single-expression lambdas and replaced them with my fancy new arrow lambdas in some projects I'm working on.

---

Before:

std::erase_if(active_chunks_, [](Chunk *chunk) { return !chunk->isActive(); });

After:

std::erase_if(active_chunks_, [](Chunk *chunk) => !chunk->isActive());

This code deletes a chunk from a game world if the chunk isn't currently active. In my opinion, the version with the arrow function is a bit clearer, but a better solution could be C++17's std::invoke. If I understand std::invoke correctly, if C++ was to adopt std::invoke for algorithms, this code could be written like this:

std::erase_if(active_chunks_, &Chunk::isInactive);

This looks nicer, but has the disadvantage that you need to add an extra method to the class. Having both isActive and its negation isInactive as member functions just because someone might want to use it as a predicate in an algorithm sounds unfortunate. I prefer lambdas' fleixibilty.

---

Before:

return map(begin(worldgens_), end(worldgens_), [](auto &ptr) { return ptr.get(); });

After:

return map(begin(worldgens_), end(worldgens_), [](auto &ptr) => ptr.get());

This code maps a vector of unique pointers to raw pointers. This is yet another case where I think the arrow syntax is slightly nicer than the C++11 alternative, but this time, we could actually use the member function invocation if I changed my map function to use std::invoke:

return map(begin(worldgens_), end(worldgens_), &std::unique_ptr<Swan::WorldGen::Factory>::get);

Well, this illustrates that invoking a member function doesn't really work with overly complex types. Imagine if the type was instead something more elaborate:

return map(begin(worldgens_), end(worldgens_),
	std::unique_ptr<Swan::WorldGen<int>::Factory, Swan::WorldGen<int>::Factory::Deleter>::get);

This also happens to be unspecified behavior, because taking the address of a function in the standard library is generally not legal. From https://en.cppreference.com/w/cpp/language/extending_std:

The behavior of a C++ program is unspecified (possibly ill-formed) if it explicitly or implicitly attempts to form a pointer, reference (for free functions and static member functions) or pointer-to-member (for non-static member functions) to a standard library function or an instantiation of a standard library function template, unless it is designated an addressable function.

---

Before:

bool needRender = dirty || std::any_of(widgets.begin(), widgets.end(),
	[](auto &w) { return w.needRender(); });

After:

bool needRender = dirty || std::any_of(widgets.begin(), widgets.end(),
	[](auto &w) => w.needRender());

Again, the short lambda version looks a bit better to me. However, here again, we could replace the lambda with a member reference if algorithms were changed to use std::invoke:

bool needRender = dirty || std::any_of(widgets.begin(), widgets.end(),
	&WidgetContainer::needRender);

---

Overall, I see a short lambda syntax as a modest improvement. The biggest readability win mostly stems from the lack of that awkward ; }); at the end of an expression; foo([] => bar()) instead of foo([] { return bar(); });. It certainly breaks down a bit when the argument list is long; neither of these two lines are particularly short:

foo([](auto const &&a, auto const &&b, auto const &&c) => a + b + c);
foo([](auto const &&a, auto const &&b, auto const &&c) { return a + b + c; });

I think, considering the minimal cost of implementing this short lambda syntax, the modest improvements outweigh the added complexity. However, there's also an opportunity cost associated with my version of the short lambda syntax: it makes a better, future short lambda syntax either impossible or more challenging. For example, accepting my syntax would mean we couldn't really adopt a ratified version of P0573R2's short lambda syntax in the future, even if the issues with it were otherwise fixed.

Therefore, I will argue strongly that my syntax makes code easier to read, but I can't say anything about whether it should be standardized or not.

Aside: Corner cases

If we were to standardize this syntax, we would have to consider all kinds of corner cases, not just accept what clang with my changes happens to to do. However, I'm still curious about what exactly clang happens to do with my changes.

How does this interact with the comma operator?

The comma operator in C++ (and most C-based languages) is kind of strange. For example, what does foo(a, b) do? We know it calls foo with the arguments a and b, but you could technically decide to parse it as foo(a.operator,(b)).

My arrow syntax parses foo([] => 10, 20) as calling foo with one argument; a function with the body 10, 20 (where the comma operator means the 10 does nothing, and 20 is returned). I would probably want that to be changed, so that foo is called with two arguments; a lambda and an int.

This turns out to be fairly easy to fix, because Clang already has ways of dealing with expressions which can't include top-level commas. After all, there's precedence here; since clang parses foo(10, 20) without interpreting the , a top-level comma as a comma operator, we can use the same infrastructure for arrow lambdas.

In clang/lib/Parser/ParseExpr.cpp, Clang defines a function ParseAssignmentExpression, which has this comment:

Parse an expr that doesn't include (top-level) commas.

Calling ParseAssignmentExpression is also the first thing the ParseExpression function does. It seems like it's just the general function for parsing an expression without a top-level comma operator, even though the name is somewhat misleading. This patch changes arrow lambdas to use ParseAssignmentExpression instead of ParseExpression: https://github.com/mortie/llvm/commit/c653318c0056d06a512dfce0799b66032edbed4c

How do immediately invoked lambdas work?

With C++11 lambdas, you can write an immediately invoked lambda in the obvious way; just do [] { return 10; }(), and that expression will return 10. With my arrow lambda syntax, it's not quite as obvious. Would [] => foo() be interpreted as immediately invoking the lambda [] => foo, or would it be interpreted as creating a lambda whose body is foo()?

In my opinion, the only sane way for arrow lambdas to work would be that [] => foo() creates a lambda with the body foo(), and that creating an immediately invoked lambda would require extra parens; ([] => foo())(). That's also how my implementation happens to work.

How does this interact with with explicit return types, specifiers, etc?

Since literally all the code before the arrow/opening brace is shared between arrow lambdas and C++11 lambdas, everything should work exactly the same. That means that all of these statements should work:

auto l1 = []() -> long => 10;
auto l2 = [foo]() mutable => foo++;
auto l3 = [](auto a, auto b) noexcept(noexcept(a + b)) => a + b;

And so would any other combination of captures, template params, params, specifiers, attributes, constraints, etc, except that the body has to be a single expression.

Date: 2018-07-26
Git: https://gitlab.com/mort96/blog/blob/published/content/00000-home/00011-c-compiler-quirks.md

In a previous blog post, I wrote about some weird features of C, the C preprocessor, and GNU extensions to C that I used in my testing library, Snow.

This post will be about some of the weird compiler and language quirks, limitations, and annoyances I've come across. I don't mean to bash compilers or the specification; most of these quirks have good technical or practical reasons.

Compilers lie about what version of the standard they support

There's a handy macro, called __STDC_VERSION__, which describes the version of the C standard your C implementation conforms to. We can check #if (__STDC_VERSION__ >= 201112L) to check if our C implementaion confirms to C11 or higher (C11 was published in December 2011, hence 2011 12). That's really useful if, say, you're a library author and have a macro which uses _Generics, but also have alternative ways of doing the same and want to warn people when they use the C11-only macro in an older compiler.

In theory, this should always work; any implementation of C which conforms to all of C11 will define __STDC_VERSION__ as 201112L, while any implementation which doesn't conform to C11, but conforms to some earlier version, will define __STDC_VERSION__ to be less than 201112L. Therefore, unless the _Generic feature gets removed in a future version of the standard, __STDC_VERSION__ >= 201112L means that we can safely use _Generic.

Sadly, the real world is not that clean. You could already in GCC 4.7 enable C11 by passing in -std=c11, which would set __STDC_VERSION__ to 201112L, but the first release to actually implement all non-optional features of C11 was GCC 4.9. That means, if we just check the value of __STDC_VERSION__, users on GCC 4.7 and GCC 4.8 who use -std=c11 will see really confusing error messages instead of our nice error message. Annoyingly, GCC 4.7 and 4.8 happens to still be extremely widespread versions of GCC. (Relevant: GCC Wiki's C11Status page)

The solution still seems relatively simple; just don't use -std=c11. More recent compilers default to C11 anyways, and there's no widely used compiler that I know of which will default to setting __STDC_VERSION__ to C11 without actually supporting all of C11. That works well enough, but there's one problem: GCC 4.9 supports all of C11 just fine, but only if we give it -std=c11. GCC 4.9 also seems to be one of those annoyingly widespread versions of GCC, so we'd prefer to encourage users to set -std=c11 and make the macros which rely on _Generic work in GCC 4.9.

Again, the solution seems obvious enough, if a bit ugly: if the compiler is GCC, we only use _Genric if the GCC version is 4.9 or greater and __STDC_VERSION__ is C11. If the compiler is not GCC, we just trust it if it says it supports C11. This should in theory work perfectly:

#if (__STDC_VERSION__ >= 201112L)
# ifdef __GNUC__
#  if (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 9))
#   define IS_C11
#  endif
# else
#  define IS_C11
# endif
#endif

Our new IS_C11 macro should now always be defined if we can use _Generic and always not be defined when we can't use _Generic, right?

Wrong. It turns out that in their quest to support code written for GCC, Clang also defines the __GNUC__, __GNUC_MINOR__, and __GNUC_PATCHLEVEL__ macros, specifically to fool code which checks for GCC into thinking Clang is GCC. However, it doesn't really go far enough; it defines the __GNUC_* variables to correspond to the the version of clang, not the version of GCC which Clang claims to imitate. Clang gained support for C11 in 3.6, but using our code, we would conclude that it doesn't support C11 because __GNUC__ is 3 and __GNUC_MINOR__ is 6. Update: it turns out that Clang always pretends to be GCC 4.2, but the same issue still applies; __GNUC__ is 4, and __GNUC_MINOR__ is 2, so it fails our version check. We can solve this by adding a special case for when __clang__ is defined:

#if (__STDC_VERSION__ >= 201112L)
# if defined(__GNUC__) && !defined(__clang__)
#  if (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 9))
#   define IS_C11
#  endif
# else
#  define IS_C11
# endif
#endif

Now our code works with both Clang and with GCC, and should work with all other compilers which don't try to immitate GCC - but for every compiler which does immitate GCC, we would have to add a new special case. This is starting to smell a lot like user agent strings.

The Intel compiler is at least nice enough to define __GNUC__ and __GNUC_MINOR__ according to be the version of GCC installed on the system; so even though our version check is completely irrelevant in the Intel compiler, at least it will only prevent an otherwise C11-compliant Intel compiler from using _Generic if the user has an older version of GCC installed.

User: Hi, I'm using the Intel compiler, and your library claims my compiler doesn't support C11, even though it does.

You: Upgrading GCC should solve the issue. What version of GCC do you have installed?

User: ...but I'm using the Intel compiler, not GCC.

You: Still, what version of GCC do you have?

User: 4.8, but I really don't see how that's relevant...

You: Try upgrading GCC to at least version 4.9.

(Relevant: Intel's Additional Predefined Macros page)

_Pragma in macro arguments

C has had pragma directives for a long time. It's a useful way to tell our compiler something implementation-specific; something which there's no way to say using only standard C. For example, using GCC, we could use a pragma directive to tell our compiler to ignore a warning for a couple of lines, without changing warning settings globally:

#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wfloat-equal"
// my_float being 0 indicates a horrible failure case.
if (my_float == 0)
	abort();
#pragma GCC diagnostic pop

We might also want to define a macro which outputs the above code, so C99 introduced the _Pragma operator, which works like #pragma, but can be used in macros. Once this code goes through the preprocessor, it will do exactly the same as the above code:

#define abort_if_zero(x) \
	_Pragma("GCC diagnostic push") \
	_Pragma("GCC diagnostic ignored \"-Wfloat-equal\"") \
	if (x == 0) \
		abort(); \
	_Pragma("GCC diagnostic pop")

abort_if_zero(my_float);

Now, imagine that we want a macro to trace certain lines; a macro which takes a line of code, and prints that line of code while executing the line. This code looks completely reasonable, right?

#define trace(x) \
	fprintf(stderr, "TRACE: %s\n", #x); \
	x

trace(abort_if_zero(my_float));

However, if we run that code through GCC's preprocessor, we see this mess:

#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wfloat-equal"
#pragma GCC diagnostic pop
fprintf(stderr, "TRACE: %s\n", "abort_if_zero(my_float)"); if (my_float == 0) abort();

The pragmas all got bunched up at the top! From what I've heard, this isn't against the C standard, because the standard not entirely clear on what happens when you send in _Pragma operators as macro arguments, but it sure surprised me when I encountered it nonetheless.

For the Snow library, this means that there are certain warnings which I would have loved to only disable for a few lines, but which I have to disable for all code following the #include <snow/snow.h> line.

Side note: Clang's preprocessor does exactly what one would expect, and produces this output:

fprintf(stderr, "TRACE: %s\n", "abort_if_zero(my_float)");
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wfloat-equal"
 if (my_float == 0) abort();
#pragma GCC diagnostic pop

Line numbers in macro arguments

Until now, the quirks I've shown have been issues you could potentially encounter in decent, real-world code. If this quirk has caused issues for you however, it might be a sign that you're slightly over-using macros.

All testing code in Snow happens within macro arguments. This allows for what I think is a really nice looking API, and allows all testing code to be disabled just by changing one macro definition. This is a small example of a Snow test suite:

#include <stdio.h>
#include <snow/snow.h>

describe(files, {
	it("writes to files", {
		FILE *f = fopen("testfile", "w");
		assertneq(f, NULL);
		defer(remove("testfile"));
		defer(fclose(f));

		char str[] = "hello there";
		asserteq(fwrite(str, 1, sizeof(str), f), sizeof(str));
	});
});

snow_main();

If that assertneq or asserteq fails, we would like and expect to see a line number. Unfortunately, after the code goes through the preprocessor, the entire nested macro expansion ends up on a single line. All line number information is lost. __LINE__ just returns the number of the last line of the macro expansion, which is 14 in this case. All __LINE__ expressions inside the block we pass to describe will return the same number. I have googled around a bunch for a solution to this issue, but none of the solutions I've looked at actually solve the issue. The only actual solution I can think of is to write my own preprocessor.

Some warnings can't be disabled with pragma

Like the above example, this is probably an issue you shouldn't have come across in production code.

First, some background. In Snow, both the code which is being tested and the test cases can be in the same file. This is to make it possible to test static functions and other functionality which isn't part of the component's public API. The idea is that at the bottom of the file, after all non-testing code, one should include <snow/snow.h> and write the test cases. In a non-testing build, all the testing code will be removed by the preprocessor, because the describe(...) macro expands to nothing unless SNOW_ENABLED is defined.

My personal philosophy is that your regular builds should not have -Werror, and that your testing builds should have as strict warnings as possible and be compiled with -Werror. Your users may be using a different compiler version from you, and that compiler might produce some warnings which you haven't fixed yet. Being a user of a rolling release distro, with a very recent of GCC, I have way too often had to edit someone else's Makefile and remove -Werror just to make their code compile. Compiling the test suite with -Werror and regular builds without -Werror has none of the drawbacks of using -Werror for regular builds, and most or all of the advantages (at least if you don't accept contributions which break your test suite).

This all means that I want to be able to compile all files with at least -Wall -Wextra -Wpedantic -Werror, even if the code includes <snow/snow.h>. However, Snow contains code which produces warnings (and therefore errors) with those settings; among other things, it uses some GNU extensions which aren't actually part of the C standard.

I would like to let users of Snow compile their code with at least -Wall -Wextra -Wpedantic -Werror, but Snow has to disable at least -Wpedantic for all code after the inclusion of the library. In theory, that shouldn't be an issue, right? We just include #pragma GCC diagnostic ignored "-Wpedantic" somewhere.

Well, as it turns out, disabling -Wpedantic with a pragma doesn't disable all the warnings enabled by -Wpedantic; there are some warnings which are impossible to disable once they're enabled. One such warning is about using directives (like #ifdef) inside macro arguments. As I explained earlier, everything in Snow happens inside of macro arguments. That means that when compiling with -Wpedantic, this code produces a warning which it's impossible to disable without removing -Wpedantic from the compiler's arguments:

describe(some_component, {
#ifndef __MINGW32__
	it("does something which can't be tested on mingw", {
		/* ... */
	});
#endif
});

That's annoying, because it's perfectly legal in GNU's dialect of C. The only reason we can't do it is that it just so happens to be impossible to disable that particular warning with a pragma.

To be completely honest, this issue makes complete sense. I imagine the preprocessor stage, which is where macros are expanded, doesn't care much about pragmas. It feels unnecessary to implement pragma parsing for the preprocessor just in order to let people compile files with -Wpedantic but still selectively disable this particular warning. That doesn't make it less annoying though.

Funnily enough, I encountered this issue while writing Snow's test suite. My solution was to just define a macro called NO_MINGW which is empty if __MINGW32__ is defined, and expands to the contents of its arguments otherwise.