2026-11-16 16:01:35
Six months ago, I wrote an SQLite-backed cache to speed up the build system for this site. A static site generator is an ideal candidate for caching, because there are lots of steps that have deterministic behaviour and no side effects. On a site with hundreds of pages, even a few milliseconds per step adds up to a meaningful improvement.
That first cache was fairly basic. It supported getting and setting strings, with a namespace for different functions, and not much else:
from mosaic.cache import SQLiteCache
cache = SQLiteCache("example.db")
cache.set(namespace="fibonacci", key="10", value="55")
print(cache.get(namespace="fibonacci", key="10"))
This showed the value of a cache, but it provided cache storage, not cache logic. If I wanted to cache a function, I had to write per-function logic for saving values to the cache and retrieving them later. A lot of that code was tricky to test, and broken in subtle ways. I was so unsure of this code that I deleted the cache whenever I built the site for publication.
I wanted something better, so last week I wrote funkycache, a new library that encapsulates all of my caching logic, and is much easier to use and test. It can store arbitrary Python objects as both inputs and outputs, and the interface is much simpler for the caller.
To cache a function, you create a FunkyCache instance and add a decorator to your function:
from funkycache import FunkyCache
cache = FunkyCache(storage="sqlite")
@cache.register
def expensive_function(arg1: str, arg2: int, arg3: date) -> tuple[str, date]:
...
This is modelled after functools.cache in the standard library.
The output of the expensive function is written to an SQLite database, so it’s available after the Python process exits.
The cache tries to be smart about invalidations.
If the source file containing expensive_function changes, the cache for that function is invalidated and will be rebuilt on the next call.
You can also specify a dependency on other files and static values, changes to which will also trigger a cache invalidation:
from funkycache import FunkyCache
cache = FunkyCache(storage="sqlite")
@cache.register(
files=["data/input.yml", "templates/example.html"],
values=[pygments.__version__],
)
def expensive_function(arg1: str, arg2: int, arg3: datetime) -> tuple[str, datetime]:
...
When writing funkycache, I wanted to spend a week thinking hard about caching logic, test it thoroughly, then not think about it again for months. You should be able to use funkycache without understanding how it works – but if you’d like to, that’s what I’ll explain in the rest of this post.
If you’d like to try it yourself, funkycache is on PyPI, and the source code is on this website.
The cache is meant for functions which always produce the same output for a given input, so the inputs are the cache key. When we decorate a function, we can see the arguments passed on each call:
from functools import wraps
def my_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
print({"args": args, "kwargs": kwargs})
return func(*args, **kwargs)
return wrapper
This doesn’t account for different styles of calling the same function – for example, positional versus keyword arguments, or omitting or overriding default arguments.
To get a complete, consistent set of arguments, I use the inspect module:
import inspect
sig = inspect.signature(func)
bound = sig.bind(*args, **kwargs)
bound.apply_defaults()
cache_key = bound.arguments # {"arg1": val1, "arg2": val2, ...}
This gives me a Python dictionary of the complete set of arguments to the function, which is the same regardless of the calling style.
Python dictionaries aren’t hashable, so they can’t be used as the keys in dictionaries, nor can they be used as keys in SQLite databases. We need to serialise them as a string or byte sequence.
I considered two options for this serialisation: pickle and JSON. In my limited testing, JSON was faster than pickle, so that’s what I chose. Using JSON also gave me more flexibility for speeding up the serialisation.
My initial serialisation was a naive json.dumps call, with a couple of options to ensure consistent ordering and handle types that the standard library doesn’t know how to encode:
json_key = json.dumps(cache_key, sort_keys=True, default=str)
When I started using funkycache, this encoding step quickly emerged as the slowest part, so I started “reducing” the arguments to make this step faster. This reduction included:
These are all lossy transformations, but that’s fine for a cache key – we never need to retrieve the original value. It’s theoretically possible for different values to get a colliding hash key, but at my scale it’s unlikely. (A clean build of my site only uses 32,000 keys.)
The functions in my static site generator are processing things like the content of articles, so individual function arguments might be kilobytes in size. You can use multi-kilobytes keys with SQLite, but you probably shouldn’t – you take a performance hit for larger keys.
I wrote a small benchmark to compare three approaches: storing the JSON-encoded arguments directly, hashing them with MD5, and hashing them with XXH64. The results are a wash for small key sizes, but for larger keys XXH64 is significantly faster, and MD5 is slower than storing the keys directly in SQLite:
Based on this benchmark, funkycache hashes the JSON cache key with XXH64 before inserting it into SQLite.
The xxHash class of hashing algorithms was a pleasant discovery for me during this project. They are explicitly designed to be fast, not cryptographically secure. Previously I’d used MD5 for this purpose, because it’s widely available and I conflated “weak” with “fast”. Actually running a benchmark revealed my mistake.
Serialising cache keys can be a one-way operation, but cached values need to be retrievable.
For this, I’m using the inspect module to get the return type from the cached function, then handing the serialisation to Pydantic.
import inspect
import pydantic
# Get the return type of the function.
sig = inspect.signature(func)
return_type = sig.ret_annotation
# Create a Pydantic type adapter
adapter = TypeAdapter(return_type)
# Serialise as JSON
json_str = adapter.dumper_json(value)
# Deserialise from JSON
retrieved_value = adapter.validate_json(stored_value)
I use type hints and type annotations for all of my Python, so this information is guaranteed to be present. This wouldn’t work with untyped Python, but it’s good enough for me.
The alternative is using pickle, which can store arbitrary Python objects. I wouldn’t have to worry about the security risks because I’d only ever be pickling and unpickling trusted data, but I continued with JSON because I’m already using it for cache keys, and it’s easier to debug.
One of the functions I wanted to cache is a lazy generator, which was a little trickier.
Using functools.cache will cache the generator object, so every call gets the same generator instance, and they all step together.
This strikes me as potentially confusing:
import functools
@functools.cache
def even_numbers():
for i in range(10):
yield i * 2
n1 = even_numbers()
n2 = even_numbers()
print(next(n1)) # 0
print(next(n1)) # 2
print(next(n2)) # 4
funkycache takes a different approach: it materialises the generator into a list, and stores that list.
When it’s retrieved, the list is converted back into a generator by wrapping it with iter.
To protect against unbounded memory use from a large or infinite generator, there’s a soft memory budget. If the materialised list exceeds this budget, I stop gathering the output in the list and don’t save anything to the cache. The list and any subsequently generated items are passed directly from the original function to the caller.
Some of the functions in my static site generator depend on the contents of template files. When those files change, I want to invalidate the cache and rebuild the pages that use those templates.
When you register a function with funkycache, you can specify a list of files or folders that it depends on:
@cache.register(files=["mosaic/text", "templates"])
def render_html(content: str, content_vars: dict[str, Any],) -> str:
...
funkycache will read the contents of each of those files, and include them in the cache key. When a file changes, it re-evaluates the list of files and decides whether to keep or invalidate the cache.
I decided not to put any file change tracking in funkycache itself. I already have code for that, and I wanted to keep this project small and focused.
A function’s cache can also be based on static values. I use this to include the version of important libraries, so the cache gets invalidated when I update the library. Here’s a short example:
import pygments
@cache.register(values=[pygments.__version__])
def apply_syntax_highlighting(src: str, lang: str) -> str:
...
This project started because I was thinking about reactive programming.
Reactive programming is a bit like using a spreadsheet. You have input cells where you enter data, and other cells that read that data and automatically update when the inputs change. Those cells can be inputs to other cells, and so on.
Imagine building a static site: your Markdown files act as input cells, and your rendered HTML pages are the dependent cells. Your tasks form a directed acyclic graph (DAG), and updates propagate through the graph.
For example, in the diagram above, when article1.md changes, the system knows it needs to rebuild article1.html and ripples that update to list_of_articles.html.
But because there’s been no change in their inputs, the tasks to render articles 2 and 3 don’t run at all.
When done correctly, this can be a very efficient style of programming, because steps only ever run if there’s been a change in their inputs.
There are tools that will do this for you, like Apache Airflow, but I wanted to build my own. My website is a sandbox where I like building things myself as a learning exercise, rather than using somebody else’s library.
I spent several weeks trying to design an API, but I couldn’t come up with something satisfactory. However, I experimented with a decorator-based API, and that led me to build funkycache, which delivers a lot of the benefits with less fo the complexity.
Building my website with a warm cache now takes about half a second, and a significant chunk of that time is spent traversing the file system for Markdown files. I have ideas for speeding up that step as well, but that’s a task for another day.
funkycache is only a few hundred lines of code, including tests. If this problem sounds interesting, I encourage you to read it, even if it’s not a library you expect to use.
[If the formatting of this post looks odd in your feed reader, visit the original article]
2026-11-02 03:38:34
I’ve recently joined a new team that’s focused on performance, stability, and scalability – a change from my previous projects, where traffic was low enough that performance was rarely a concern.
This codebase has already gone through multiple rounds of optimisation, so there’s no low-hanging fruit left. Any proposed change is inherently complex, and it’s rarely obvious whether a tweak will actually speed things up or slow them down. To measure the impact of our work, we’re leaning on the benchmarking tools in Go’s standard library.
On one of my recent pull requests, I wrote a benchmark, and I pasted the output from two before/after runs. Paul improved this with a short shell script that ran multiple samples, and compared the results with the benchstat command.
$ for N in {1..6}; do for C in 613000c fc12d4c; do echo $N: $C; git checkout -q $C && ./tool/go test -run=^X -bench=BenchmarkPostureAnalyzer ./control/posture/analysis/ | tee -a bench-$C.txt; done; done
$ benchstat bench-fc12d4c.txt bench-613000c.txt
benchstat does statistical analysis on the output from multiple go test -bench=... runs, which is more reliable and easier to read than the raw output.
In particular, it puts all the data in a single table, and shows the percentage change against the baseline.
I like this script, but it’s a little fiddly. I have to identify the two commits to compare, it leaves text files in my working directory, and it leaves my repo in a detached head state when it’s done. I wanted something more ergonomic.
I’ve written a new tool gobench to give me a smoother experience.
I check out my feature branch, run gobench with the arguments I’d pass to go test, and it prints the final analysis from benchstat.
Under the hood, it creates a Git work tree for the base branch (leaving my working directory as-is), captures the benchmark output in memory (so there are no extra text files), and does an alternating sample of benchmarks.
The alternating samples is a feature I really like from Paul’s original script. I run benchmarks on my laptop rather than a dedicated workstation, and the benchmarks could be sensitive to thermal throttling, background processes, or other things happening on my computer. Alternating between the two branches reduces the effect of this noise.
Let’s look at an example. Suppose my main branch has a recursive implementation of Fibonacci numbers, while my feature branch has an iterative approach. Unsurprisingly, the iterative approach is faster, and gobench makes it easy to quantify the improvement.
$ go run ./cmd/gobench ./example/fib/ -bench=BenchmarkFib
Setting up temporary worktree for main (8f2351d)...
Running 6 alternating samples...
100% |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (12/12, 42 it/min)
== Benchmark results ==
goos: darwin
goarch: arm64
pkg: alexwlchan.net/projects/gotools/example/fib
cpu: Apple M4 Pro
│ main (8f2351d) │ faster-fib (0969b03) │
│ sec/op │ sec/op vs base │
Fib-12 14572.500n ± 6% 5.455n ± 2% -99.96% (p=0.002 n=6)
My first version of gobench was an expansion of Paul’s script, but as I started adding features, it became clear that Bash was the wrong tool.
I rewrote it as a Go tool that shells out to git, go test, and benchstat.
If you’d like to try gobench yourself, the code is available in my gotools repository, or you can install it with:
$ go install alexwlchan.net/gotools/cmd/gobench@latest
gobench has already saved me from shipping code that looked fast on paper but regressed in practice. As I keep working on performance-critical Go code, having low-friction benchmarks will make it easier to test my ideas early and often.
[If the formatting of this post looks odd in your feed reader, visit the original article]
2026-11-01 22:11:41
To protect against link rot and disappearing content, I save local copies of web pages I like into my web archive.
Ideally, this archive is completely self-contained. I want a local copy of each page’s HTML and all the assets it depends on (CSS, images, scripts, and so on). I don’t want my archived pages fetching resources from the original website, which could disappear at any time.
Initially, I was verifying self-containedness by hand. I’d open each saved page in a web browser, and check the browser dev tools. I’d check the Console tab for errors and failed requests, and watch the Network tab for requests going out to the internet. That approach was tedious and error-prone, and I wanted something better.
I now automate the process using Playwright, a testing framework that allows you to control web browsers. I open each page using Playwright, and check if it’s making external requests.
When you open a page in Playwright, you can subscribe to request events which tell you when the page issues a request.
In this Python snippet, we start Playwright, start a WebKit browser, and open a new page.
We tell the page to append external requests to a list, and then load example.com.
We see that example.com makes two requests on page load: one for https://example.com, and one for a JavaScript file.
from contextlib import closing
from playwright.sync_api import sync_playwright
with (
sync_playwright() as p,
closing(p.webkit.launch()) as browser,
closing(browser.new_page()) as page,
):
requests = []
page.on("request", lambda msg: requests.append(msg))
page.goto("https://example.com")
print("requests:", requests)
# requests: [<Request url='https://example.com/' method='GET'>,
# <Request url='https://example.com/s.js' method='GET'>]
In a more complex scenario, we might use Playwright to interact with the page after loading – for example, clicking a button or filling in a form, and then we could see the requests triggered by each action.
Once we have a list of requests made by the page, we can inspect the url property to check if any of them were fetching from an external website.
I wrote a function is_external_request to carry out this check:
from playwright.sync_api import Request
import urllib.parse
def is_external_request(req: Request, original_url: str) -> bool:
"""
Check if a request has a different origin than the original URL.
A request is considered internal if:
- it matches the original URL exactly
- it has the same scheme, host, and port as the original URL
- it uses a `blob:` scheme (for example, in-memory media representations)
"""
if req.url == original_url:
return False
# Compare the scheme/host of the request to that of the original URL.
org_url = urllib.parse.urlsplit(original_url)
req_url = urllib.parse.urlsplit(req.url)
if org_url.scheme == req_url.scheme and org_url.netloc == req_url.netloc:
return False
# Ignore blob: URLs, which are in-memory representation of some data,
# and not an external request.
#
# Playwright detects blob: URLs when loading media files; for example
# using the <audio> or <video> tag.
if req_url.scheme == "blob":
return False
# Fail open: if we can't decide if a request was internal or external,
# assume it was external.
return True
assert_page_is_self_contained(ctx, url)I’ve wrapped these two steps in a function assert_page_is_self_contained(ctx, url), which is part of my chives utility library.
The function takes a Playwright Browser (or BrowserContext) and a URL, then:
It throws an exception if there are any:
The function tries to trigger important network requests that might be skipped on the initial page load.
For example, it evaluates the body’s offsetHeight to force a synchronous reflow, which causes the browser to request any lazily loaded images or fonts that might otherwise be missed.
Here’s an example which verifies a file example.html from the working directory:
from chives.browser_fixtures import assert_page_is_self_contained, file_uri
from contextlib import closing
from playwright.sync_api import sync_playwright
with (
sync_playwright() as p,
closing(p.webkit.launch()) as browser,
):
assert_page_is_self_contained(browser, url=file_uri("example.html"))
I originally passed a Page as the first argument, but changing the function to take a Browser or BrowserContext lets it manage its own pages.
It gets a fresh page for every verification, and closes it when it’s done.
(The first time I ran this, I created a new page for every verification, and never closed them – my computer ground to a crawl as my script opened hundreds of browser tabs.)
If you create a new context, you can emulate different browser settings, like location, locale, and window size. Here’s an example where we use contexts to test the same page in light/dark mode:
with (
sync_playwright() as p,
closing(p.webkit.launch()) as browser,
):
with closing(browser.new_context(color_scheme="light")) as light_ctx:
assert_page_is_self_contained(light_ctx, url=file_uri("example.html"))
with closing(browser.new_context(color_scheme="dark")) as dark_ctx:
assert_page_is_self_contained(dark_ctx, url=file_uri("example.html"))
In my web archive, I check every page in light/dark mode, and at 1×/2× pixel densities. It takes at least a second to run each verification, so I cache the results. As I save new pages, they’re added to a verification queue, and I’m warned of any missing assets.
Unsurprisingly, the first run flagged many missing fonts and images, which I added to my local archive. Every saved page now passes cleanly.
Even with automated testing, it’s difficult to guarantee a completely self-contained archive. Different instances of a browser might load different assets, and a single page load might miss different variations:
You can mitigate some of this by testing with multiple contexts or adding post-load interactions, but every step makes the process take longer. Fortunately, most of the pages I save are simple, blog-style pages where I’ve stripped out the JavaScript. For my specific use case, initial load verification is more than good enough.
There are other projects where it could be useful to intercept and audit the requests made by a web page:
I don’t do much front-end web development, but Playwright has become a key tool in my toolbox. I read a lot about people using AI agents and MCPs to interact with websites, but I prefer Playwright. It’s more fiddly and requires explicit instructions, but it’s constrained, deterministic, and I never have to worry about it doing something unexpected.
[If the formatting of this post looks odd in your feed reader, visit the original article]
2026-10-27 02:38:45
At work, we use Prometheus to monitor our services.
Every few seconds, Prometheus scrapes a /metrics endpoint and records the data in a time series database.
The metrics are then used to trigger alerts when services aren’t behaving correctly.
Recently we had a flapping alert where an overloaded service was intermittently failing. We wanted to silence/adjust the alert until we could fix the service, to avoid alert fatigue, but it would be easy to forget to revert the adjustment once the service was fixed.
We solved this using a “positive alert” that would tell us when the service was consistently passing the original alert – that is, when it was no longer flapping.
Let’s walk through an example.
One of our alerts checks that Prometheus can successfully scrape our services, which you can check with the automatically generated up time series.
A successful scrape gets a 1, a failed scrape gets a 0.
Here’s an alert that warns if more than 5% of scrapes fail within a half hour window:
groups:
- name: services
rules:
- alert: PrometheusTargetIntermittentlyDown
expr: avg_over_time(up[30m]) < 0.95
for: 1h
labels:
severity: warning
annotations:
summary: Prometheus is struggling to scrape metrics from this service.
One of our services was struggling to serve the /metrics endpoint that Prometheus scrapes.
When the service had a lot of work in its queue, the worker thread would starve the web server and the metrics request would fail.
The scrape rate would dip below 95%, the alert would fire, then the service would empty its queue and the rate would recover, the alert would clear, and so we went round and round.
Dozens of notifications which were nothing but noise.
We use Alertmanager Silences when we want to pause an alert for a short period, but it wasn’t clear if or when we’d be able to fix this service. If the silence expired before the problem was fixed, the noise would return. If the problem was fixed before the silence expired, we could unknowingly regress the fix.
Instead, we modified the alert to exclude the overloaded service, then added a “positive alert” to tell us if the service had more than 95% uptime for a fortnight:
groups:
- name: services
rules:
- alert: PrometheusTargetIntermittentlyDown
expr: avg_over_time(up{instance!="overloaded_service"}[30m]) < 0.95
for: 1h
labels:
severity: warning
annotations:
summary: Prometheus is struggling to scrape metrics from this service.
- alert: PrometheusTargetUpAvg
expr: avg_over_time(up{instance="overloaded_service"}[30m]) >= 0.95
for: 14d
labels:
severity: info
annotations:
summary: This service is no longer overloaded; remove the exception.
This worked exactly as intended. A few weeks ago, one of my colleagues made changes to improve the overloaded service, and Prometheus was able to crawl it consistently. The alert fired at the weekend as it ticked over a fortnight of continuous uptime, and we removed the exception.
[If the formatting of this post looks odd in your feed reader, visit the original article]
2026-10-13 14:24:07
I have a lot of digital cross-stitch patterns and until recently they were a mess. My “collection” was a series of several hundred PDFs, disorganised and spread across my disk.
Recently, I was looking for a pattern and it took me several minutes to find it. I had a brief moment of panic, thinking that I’d lost it, and the website this pattern came from is long since gone. Fortunately it did turn up, but it was clear I had to organise my pattern collection.
Some people would use apps like Pattern Keeper or Flosshaven, but I just created another static website on my media drive. This is what it looks like:

The patterns are arranged in a masonry layout with a preview thumbnail for each pattern, which is normally the first page of the PDF. If a pattern has multiple files – for example, different files for colour and black-and-white printing – there’s a list of links below the thumbnail. I can also add a source link to where I bought the pattern, if I want to find more patterns by the same author.
Internally, it’s structured like my other static websites. The metadata for the patterns is stored in a JavaScript file, and then I have an HTML page which renders that metadata in this visual design. My pattern collection doesn’t change very often, so I’m adding new files by hand.
There are no sorting or filtering options, because my collection is small enough that they’re unnecessary. I can scroll through the entire list in less than a minute and find whatever I need.
I’d like to start making my own clothes soon, and I expect to copy this design to store clothing patterns.
This collection now sits alongside my saved bookmarks, podcasts, TV shows, and more. I can keep all my patterns organised, and I won’t lose them in the depths of my hard drive.
There are a couple of technical pieces I’m quite pleased with in this process:
Masonry layout. I love the cascading grid style of a masonry layout, and I think they’re great when you have variable-sized content – like my pattern cards. I’ve been using them for over a decade; screenshots of my early web apps show my experiments with David Desandro’s Masonry.js library.
Masonry layouts are now a native feature of CSS called “Grid Lanes”, which is simpler and more performant than the JavaScript approach. I’m excited to use these layouts in even more places.
Thumbnails of PDFs. Most digital postage patterns are shipped as PDFs, and I’m creating the thumbnail images with my create_thumbnail tool.
I had to extend my tool to support PDFs, which turned out to be straightforward with the pdf_oxide crate. I’d not come across Oxide before, but it’s a cross-language library for working with PDFs, and I can imagine using it in future.
A coloured stripe for the palette. In the metadata for each project, I have the complete list of DMC thread colours and the pattern dimensions. I have a lookup table that converts DMC thread colours to hex codes, and then I’m able to create a linear gradient in CSS to render a coloured stripe of the pattern colours.
Here’s an example from a pattern with four colours:
border-bottom: 12px solid transparent;
border-image: linear-gradient(
to right,
#7a451f 0%,
#7a451f 25%,
#985e33 25%,
#985e33 50%,
#e4bb8e 50%,
#e4bb8e 75%,
#361f0e 75%,
#361f0e 100%
) 1;
Building the HTML, CSS, and JavaScript was very quick – I have notes on how I build my media sites, and I’ve done it plenty of times before. I threw together the initial prototype in less than 15 minutes. The longer part was finding all the PDFs scattered across my disk – but now that’s done, and I know exactly where to find my patterns.
[If the formatting of this post looks odd in your feed reader, visit the original article]
2026-09-26 16:43:00
I’ve been revisiting some of my small Rust utilities this week, and I wanted to improve the test suite. There’s a lot of repetition, and I wanted to switch to parametrised tests, a pattern I use a lot in Python and Go. (These are also called “table-driven tests”.)
Parameterised tests allow you to write the test logic once, then run it with many different inputs and expected results. This reduces repetition, highlights variation, and simplifies adding new cases.
For example, in my thumbnail creator, I want to create a thumbnail with various image formats, and check the thumbnail has the correct filename and dimensions. Rather than copy-pasting test functions, I wanted to declare test cases with named fields, like this:
animated_gif: {
img_path: "src/tests/animated_squares.gif",
target: TargetDimension::MaxWidth(16),
expected_filename: "animated_squares.mp4",
expected_dims: (16, 16),
},
Supporting this syntax required me to create a custom Rust macro.
I found three existing approaches to writing this type of test in Rust:
Use a for loop over the test cases.
This is the simplest approach: define our test cases in an array, then iterate over them one-by-one.
#[test]
fn test_fibonacci_numbers() {
let test_cases = [(1, 1), (6, 8), (10, 55)];
for (input, expected) in test_cases {
assert_eq!(fibonacci(input), expected);
}
}
This is easy to write and understand, but Rust’s standard assertions don’t give us test case isolation. If one case fails, the test stops immediately and skips the rest.
(Go’s testing library has t.Error, which marks a test as failed but allows it to continue running.
When a test fails, you often have more information than if the test had failed immediately.
I now miss this when working in other languages.)
Define a macro with positional arguments. Many guides, such as over-code’s article, define custom macros with positional arguments.
fib_test!(my_tests,
basic: 1, 1,
sixth: 6, 8,
tenth: 10, 55,
);
Named test cases are nice, but positional arguments hurt readability if you have more than a few parameters.
Use a third-party testing library. Crates like parameterized, rstest, and yare use attributes to declare test cases.
use rstest::rstest;
#[rstest]
#[case(1, 1)]
#[case(6, 8)]
#[case(10, 55)]
fn fibonacci_test(#[case] input: u32,#[case] expected: u32) {
assert_eq!(expected, fibonacci(input))
}
If I was working in a larger codebase, I’d compare these libraries and pick my favourite.
I deliberately skipped them this time, because I only use Rust for tiny personal tools, and I write it as a learning exercise. I’d rather write a basic version myself and learn something new, than use somebody else’s library and miss out on the lesson.
Since I couldn’t find a macro that gave me named parameters, I decided to write my own.
Here’s the complete macro invocation for my image thumbnailer, with three test cases:
image_thumbnail_tests! {
animated_gif: {
img_path: "src/tests/animated_squares.gif",
target: TargetDimension::MaxWidth(16),
expected_filename: "animated_squares.mp4",
expected_dims: (16, 16),
},
static_gif: {
img_path: "src/tests/yellow.gif",
target: TargetDimension::MaxWidth(16),
expected_filename: "yellow.gif",
expected_dims: (16, 8),
},
jpeg: {
img_path: "src/tests/noise.jpg",
target: TargetDimension::MaxWidth(16),
expected_filename: "noise.jpg",
expected_dims: (16, 32),
},
}
The goal is to have clear, self-documented test cases. You can see what the test does and how to add a new test case without reading the body of the macro.
Here’s the macro definition, with extra whitespace to make the structure more obvious:
macro_rules! image_thumbnail_tests {
(
$(
$name:ident: {
img_path: $img_path:expr,
target: $target:expr,
expected_filename: $expected_filename:expr,
expected_dims: $expected_dims:expr,
},
)*
)
=>
{
$(
#[test]
fn $name() {
let path = PathBuf::from($img_path);
let out_dir = test_dir();
let thumbnail_path = create_thumbnail(&path, &out_dir, $target).unwrap();
assert_eq!(thumbnail_path, out_dir.join($expected_filename));
assert!(thumbnail_path.exists());
assert_eq!(get_dimensions(&thumbnail_path), $expected_dims);
}
)*
}
}
If we zoom out, the structure of a declarative macro looks like a match expression:
macro_rules! image_thumbnail_tests {
( matcher ) => { transcriber }
}
I think of the two halves are input/output, but they’re really called matcher/transcriber:
The left-hand side (matcher). The matcher defines a pattern of Rust tokens the input code must match precisely, including commas and brackets (but ignoring whitespace).
$(…)* works like a regular expression, matching zero or more instances of the enclosed pattern.$name:ident and $img_path:expr match a Rust identifier and expression, respectively, and bind them to metavariables (similar to named capture groups in regexes).I deliberately designed my pattern to resemble Rust’s struct syntax. Macro inputs don’t have to be valid Rust code, but I wanted something that would look familiar in a Rust file.
The right-hand side (transcriber). The transcriber is a template for Rust code, which gets substituted in at compile time, replacing metavariables with values captured by the matcher.
Here, the $(…)* iterates over the matches and creates a corresponding function with a #[test] attribute.
This attribute is why we need to use a macro – there’s no way to dynamically generate new test functions in Rust without macros.
When compiled, the macro gets replaced with all the generated code from the transcriber, as if I’d written out the individual tests in full:
#[test]
fn animated_gif() {
let path = PathBuf::from("src/tests/animated_squares.gif");
let out_dir = test_dir();
let thumbnail_path = create_thumbnail(&path, &out_dir, TargetDimension::MaxWidth(16)).unwrap();
assert_eq!(thumbnail_path, out_dir.join("animated_squares.mp4"));
assert!(thumbnail_path.exists());
assert_eq!(get_dimensions(&thumbnail_path), (16, 16));
}
#[test]
fn static_gif() {
let path = PathBuf::from("src/tests/yellow.gif");
…
This means all of my test cases run as separate tests, so I get all the benefits of individual test functions without having to copy-paste lots of boilerplate throughout my codebase.
I’ve seen custom macros pop up in the few Rust codebases I’ve read, and it feels good to have a better understanding of how they work. I used to copy-paste the same snippets thoughtlessly from project to project, but now I know what the different brackets and asterisks mean. The next time I write a macro or read somebody else’s, I’ll know what’s going on.
[If the formatting of this post looks odd in your feed reader, visit the original article]