Xe IasoModify

2026-08-15 08:00:00

While I was developing tooling for my job this weekend, I found a few blogposts that I don't remember writing:

• Advice for staying in the hospital for a week
• The Unbound Scepter
• Using Clankers to Help Me Process Surgery

Checking the date range, they coincide with when I was in the hospital earlier this year. I have removed them from the blog index and plan to rewrite them from scratch in my own words when I have the time.

I apologize for any inconvenience this may have caused. I apparently did not plan for the effect that hospital drugs (including fairly powerful blood thinners) would affect my mental state. In light of this, I will take steps to set up a sandbox environment for me to post in if I have future short term hospitalizations.

As my evaluation of my local tooling improves, I will update this post with more removed posts as they are discovered. I would request erring on the side of leaving me alone as I complete this process.

2026-08-11 08:00:00

Tigris has a pretty advanced replication scheme for writes. What happens when you actually need to delete things? Turns out deleting things is hard in distributed systems. Especially when you have a geo-replicated active-active database like Tigris does. We can (and do) use tombstones to mark where data once was, but how do you let people undo an accidental delete?

Tigris wants to turn storage inside out, so our implementation of soft deletion is by giving users the Recycle Bin for objects and buckets. Today we're going to dig into how this works, why it works, and what this gives you in terms of using object storage today.

Recycle bins and you

In Windows and macOS, the Recycle Bin (or Trash can) is a form of purgatory where deleted files wait for their storage to be deallocated by the user. This allows users to hit "delete" fearlessly because if they made a mistake they can just drag it back out and go on with life.

This works great in your local filesystem because there's only one writer in one region. This kinda falls apart when you have multiple regions in your database and any one of them could be writing to it. How do you name things in the recycle bin? How do you handle the conflict of an update happening in one region before the deletion was fully replicated out from another region?

This is the fun of distributed systems, which is the kind of problem space that Tigris lives in.

One way to think about how the Recycle Bin works is that the file metadata gets moved there when the user hits delete. No data bytes move around on the disk, but the file doesn't show up in My Documents anymore. In a distributed systems context you can't just move the metadata around, you have to leave a tombstone behind to record where that metadata once was. This prevents other regions from being confused when actions happen really close to each other in time.

Soft deletes in some universes

At a high level, a soft-delete is when a DELETE action doesn't actually remove the data. When data is soft-deleted, it's still there but just not visible in the main usage flow. This lets you get the data back when a delete is made by accident.

Your database becomes your API

One of the interesting side effects of designing any API is that you end up leaking the internals of how your database works to your users. Many object storage systems were designed with overwriting or deleting data as one of the primary operations, and as such have had to bolt versioning onto the side. For the most part this does work; but once you get into advanced versioning schemes everything starts to fall apart. Tigris doesn't suffer from the same problems because we built immutability into the core from day one, and in immutable systems you have to append new data on the end instead of overwriting data.

At the least, actually storing the data en masse is a boring problem. You put the data somewhere, maybe name it after the checksum of its contents, and then have a daemon make sure it's copied three places. That daemon also handles cases when drives go offline and new ones are added to make sure data is shuffled around the cluster. This is largely a solved problem with projects like Ceph, Longhorn, or other distributed storage systems.

S3 uses delete markers

Some object storage systems like S3 expose platform internals to make soft deletion work. In S3 deleting an object creates a delete marker (tombstone). A delete marker is an explicit marker that the object is deleted and should not be returned in normal operation. Here's what that looks like in practice:

  ┌───────────────────┐  ┌───────────────────┐  ┌───────────────────┐ ◀── DeleteObject
  │ v1                │  │ v2                │  │ v3        current │
  │ report.pdf        │  │ report.pdf        │  │ report.pdf        │
  │ …198086           │  │ …198088           │  │ delete marker     │
  └─────────┬─────────┘  └─────────┬─────────┘  └───────────────────┘
            │                      │              no data
            ▼                      ▼
  ┌───────────────────────────────────────────────────────────────────┐
  │ sea of data                                                       │
  │    ┌──────────┐         ┌──────────┐                              │
  │    │ v1 bytes │         │ v2 bytes │                              │
  │    └──────────┘         └──────────┘                              │
  └───────────────────────────────────────────────────────────────────┘
 
  // the bytes stay. only the newest record says the object is gone.

I don't know how I feel about this flow. Based on reading between the lines in the delete marker documentation it really feels like this is a leaked internal implementation detail of how S3's eventually consistent database works instead of a full fledged feature of the storage system. If I had to choose between leaking internal database details in the API and implementing a higher level API for something complicated like soft deletion, I'd want to implement the higher level API.

Tigris' soft deletes are external references

Let's rethink what soft deletes really are. What if they were like the Recycle Bin in Windows?

Soft-deletes are external references to buckets or objects that live in a different namespace from normal buckets or objects. We implemented them as external references instead of tombstones because this is effectively moving object metadata to the recycle bin. Tombstones mark the data as not being there, but soft-delete markers are a copy of the data that was there. This makes it easy to put the object back in place if you deleted it by mistake.

Garbage collection roots

One way to think about objects and buckets is that they are garbage collection roots for points in the endless sea of data. Any data in the sea without a root anchoring it down is eligible to be deleted. Uploading multiple versions of an object with a forkable bucket creates multiple metadata entries at their different timestamped version numbers. You can then fork a bucket from any one of those timestamps to see what the bucket was like at that point:

  every write appends a new version entry
 
  v1              v2              ╎   v3              v4
  ┌────────────┐  ┌────────────┐  ╎   ┌────────────┐  ┌────────────┐
  │ …198086    │  │ …431907    │  ╎   │ …764522    │  │ …055310    │
  │ 4.1 MB     │  │ 4.3 MB     │  ╎   │ 4.4 MB     │  │ 5.0 MB     │
  └─────┬──────┘  └─────┬──────┘  ╎   └─────┬──────┘  └─────┬──────┘
  ──────┴───────────────┴─────────╎─────────┴───────────────┴───────▶
  earlier                         ╎ fork point                later
  the fork inherits these         ╎   written later — the fork never sees them
                                  ▼
                      ┌──────────────────────────────┐
                      │ uploads/report.pdf           │
                      │ current version v2           │
                      │ as of 1775929812004431907    │
                      └──────────────────────────────┘
 
  // appending metadata instead of overwriting keeps any past instant addressable.

This would solve the soft-delete problem, but our existing database schema using FoundationDB requires us to enable forking and snapshots at bucket creation time. In essence, we need something that's halfway between what we have (each bucket being a globally mutable namespace) and the bucket forking land of every action being appending metadata onto the end.

To do that, we basically implemented most of that appending metadata on the end trick but to a different place: the soft deletion corner. When you enable soft-deletion and delete an object, its metadata gets moved to the trashcan so you can pluck it back into place:

  main table · live keyspace                soft-delete keyspace · newest first
  ┌────────────────────────────┐            ┌──────────────────────────────────┐
  │ uploads/report.pdf         │            │ uploads/report.pdf   3rd delete  │
  │ live metadata record       │───────▶    │ deleted …768707198086            │
  │ in ListObjectsV2 output    │◀╌╌╌╌╌╌     └──────────────────────────────────┘
                                            ┌──────────────────────────────────┐
                                            │ uploads/report.pdf   2nd delete  │
                                            │ deleted …412888100731            │
  ┌────────────────────────────┐            └──────────────────────────────────┘
  │ uploads/notes.md           │            ┌──────────────────────────────────┐
  │ untouched by the delete    │            │ uploads/report.pdf   1st delete  │
  └────────────────────────────┘            │ deleted …104233715492            │
                                            └──────────────────────────────────┘
 
  ───────▶  DeleteObject moves the record out — one entry per delete
  ◀╌╌╌╌╌╌   RestoreObject moves the same metadata back
 
  // the bin entry is a copy of the metadata, not a marker. restoring is a move.

It's the same basic idea as the recycle bin on your desktop. Any buckets or objects left in the recycle bin for long enough become eligible to be deleted, which then makes the backend go and securely erase things. Effectively, any bits of metadata in the soft deletion corner are still considered garbage collection roots, they're just not shown when you do a normal ListObjectsV2 call.

Distributed systems are fun*

The real fun comes into play when you remember that Tigris has a globally replicated active-active database where any region can change any object at any time. Most of the time things work out and objects are replicated without too much strife. The annoying part comes when two events are ordered weirdly. Imagine a scenario where one agent in one datacentre deletes an object after another agent in another datacentre:

  ORD (Chicago)                               IAD (Ashburn)
  ┌──────────────────────────┐
  │ PutObject  · agent A     │
  │ uploads/report.pdf       │  ──── replicates PUT ────▶
  │ t = …768707198086        │
  └──────────────────────────┘
 
                                              ┌──────────────────────────┐
                                              │ DeleteObject  · agent B  │
  ◀── replicates DELETE ──                    │ uploads/report.pdf       │
                                              │ t = …768984210773        │
                                              └──────────────────────────┘
 
  ORD applies   PUT  ▸  DELETE    deleted, as expected
  IAD applies   DELETE  ▸  PUT    the put looks brand new
 
  // two writers, one key. the regions disagree about whether it exists.

How would this replicate out? Well for one each change is timestamped by when it's done in terms of Unix nanoseconds, so the replication messages kinda look like this:

  produced first                          produced 277 ms later
  ┌──────────────────────────────────┐    ┌──────────────────────────────────┐
  │ uploads/report.pdf               │    │ uploads/report.pdf               │
  │ op: PUT                          │    │ op: DELETE                       │
  │ LastModified  1775929768707198086│    │ LastModified  1775929768984210773│
  │ block 0x3f2ac701 · origin ORD    │    │ block 0x3f2ac701 · origin IAD    │
  └──────────────────────────────────┘    └──────────────────────────────────┘
 
  last write wins
  984210773   is greater than  707198086
 
  // the delete carries the newer LastModified, so the delete survives

This means that in theory, a user could DELETE an object before an update is processed by another region, and that would make the regions disagree about if the object exists or not. This is a horrible state to be in and usually requires support intervention or to recreate/re-delete the object.

The root cause boils down to deleting objects actually deleting metadata from the database doesn't scale past a single region. Updates to metadata include the entire metadata object, so if you delete it locally and a new version is pushed remotely, the object will gain the remote state.

We don't want users to have to deal with that, so we added the concept of anti-resurrection to Tigris. Any write to an object must prove it is newer than the deletion.

  ORD · agent PutObject                     IAD · user DeleteObject
  ┌──────────────────────────┐              ┌──────────────────────────┐
  │ uploads/report.pdf       │              │ uploads/report.pdf       │
  │ new version · v2         │◀─────────────│ row deleted · marker kept│
  │ t = 15                   │   delete     │ tombstone t = 25         │
  └────────────┬─────────────┘              └────────────┬─────────────┘
               └────────────────────┬────────────────────┘
                                    ▼
    ┌────────────────────────────────────────────────────────────┐
    │ at IAD: is the write newer than the marker?                │
    │ write t = 15  ·  marker t = 25  ·  is 15 > 25 ?            │
    │ no — not strictly newer, so the write is dropped           │
    │ equal timestamps lose too · the guard runs for every bucket│
    └────────────────────────────────────────────────────────────┘
 
  // without the marker, an empty slot looks exactly like a key that never existed.

In this circumstance, a user sent a DeleteObject request to the IAD datacentre at time 25, but an agent sent a new version of the object with PutObject to the ORD datacentre at time 15. The user's delete is newer than the agent's put, so the new version is rejected and the delete gets sent back to ORD.

Using soft deletes

Tigris extends the S3 API by having users add headers to their requests. For example, to create a bucket with soft deletion enabled:

import (
        	"context"
        
        	"github.com/aws/aws-sdk-go-v2/aws"
        	"github.com/aws/aws-sdk-go-v2/service/s3"
        	"github.com/tigrisdata/storage-go"
        )
        
        client, err := storage.New(ctx,
        	storage.WithGlobalEndpoint(),
        	storage.WithAccessKeypair(
        		os.Getenv("TIGRIS_STORAGE_ACCESS_KEY_ID"),
        		os.Getenv("TIGRIS_STORAGE_SECRET_ACCESS_KEY"),
        	),
        )
        
        _, err := client.CreateBucketWithSoftDelete(ctx, &storage.CreateBucketWithSoftDeleteInput{
        	CreateBucketInput: &s3.CreateBucketInput{Bucket: aws.String("my-bucket")},
        	RetentionDays:     30, // 0 uses the 7-day default
        })
        

Or to list soft-deleted objects:

out, err := client.ListSoftDeletedObjects(ctx, &storage.ListSoftDeletedObjectsInput{
        	Bucket: "my-bucket",
        	Prefix: "uploads/",
        })
        if err != nil {
        	return err
        }
        for _, o := range out.Objects {
        	log.Printf("%s v=%s %d bytes deleted=%s", o.Key, o.VersionID, o.Size, o.LastModified)
        }
        

Or to permanently delete one soft-deleted version:

_, err := client.PermanentlyDeleteObject(ctx,
        	"my-bucket", "uploads/report.pdf", "1775929768707198086")
        

When you have a soft-delete enabled bucket, you can also forcibly delete an entire bucket:

_, err := client.ForceDeleteBucket(ctx, &s3.DeleteBucketInput{
        	Bucket: aws.String("my-bucket"),
        })
        

And then bring it back from the dead:

trash, err := client.ListSoftDeletedBuckets(ctx, nil)
        if err != nil {
        	return err
        }
        for _, b := range trash.Buckets {
        	log.Printf("%s (%d day retention)", b.Name, b.RetentionDays)
        	if _, err := client.RestoreBucket(ctx, &storage.RestoreBucketInput{Bucket: b.Name}); err != nil {
        		return err
        	}
        }
        

Now what?

Object storage entered our stacks as an unlimited FTP server we all used for backups. A distressing amount of the world's most important data lives in object storage buckets because it's the best place to put it. This is why having an "undo" button matters, it's what makes it safe to trust your backups in the cloud. To err is human, and mistakes are a "when" to plan for, not an "if" that you hopefully never have happen. The blast radius of one overly wide --recursive flag is measured in years of people's lives.

One of the biggest usecases that comes to mind is ransomware prevention. Imagine a case where an attacker downloads everything in your bucket, deletes it, and asks for a ransom to send you the files back. With Tigris, soft deletes means that the ransom can be ignored, you can un-delete your data, and be on your merry way with incident response. The other big usecase is for agents, where they somehow get the idea that deleting production data is the right way to solve a problem. Both cases mean you need a quick and fast way to go back to before things went wrong.

If you want true isolation instead of recovery, that's why we have bucket forking. Bucket forking needs to be enabled before a bucket is created, but you can enable soft deletion on any bucket in the dashboard whenever you want.

Every storage system is going to make you choose between ones that hide how the platform works and ones that expose the gorey internals to users. I think that hiding the internals and exposing the high level operations built on top of them is the right way to go, if only because the higher level operations are much easier to make safe in our globally distributed future.

Enable soft delete on any Tigris bucket, new or existing, and every delete becomes recoverable for up to 90 days. Restoring a whole bucket is one call. Read the soft delete docs.

2026-08-08 08:00:00

Anubis v1.27.0 (Moenbryda Wilfsunnwyn) is now available via Docker and direct download from GitHub releases. This release adds Windows Server support, automatically renames cookies based on settings to avoid infinite challenge loops, adds two new localizations, and more.

Breaking change: cookie names are dynamically created based on cookie settings

Anubis tries to avoid breaking changes as much as possible, but sometimes we have to make them for the sake of the users. This is technically a breaking change in something that is not part of the public API of Anubis; but some administrators rely heavily on cookie names in advanced configurations.

It seems that browsers store cookies disambiguated with their options. This means you can have multiple cookies named the same but with different options. Browsers will send these cookies to the server without the list of options. This means that changing any cookie settings requires you to change COOKIE_PREFIX, creating a new "cookie epoch" that will set things properly.

In order to be more robust, Anubis will automatically change cookie names based on the cookie settings. For example, the default configuration creates cookies named techaro.lol-anubis-auth-347ddb4a.

Without this change, changing any cookie setting without every client clearing their cookies causes challenges to become an infinite loop of thrashing, making it appear that Anubis "blocked" them.

If this becomes onerous in practice for administrators of HAProxy and other advanced setups that rely on cookie names, we will add an escape hatch in the policy file.

Windows Server support (beta)

Anubis now publishes .msi packages, allowing administrators to install and run Anubis on Windows Server. Please read the Windows Server page for more information.

This support is beta-grade as the Anubis team does not have a lot of experience with developing software for Windows Server. Feedback is more than welcome.

Please let us know how it works for you!

Pre-release docker images no longer populate the latest tag

Due to a misconfiguration of the GitHub Action docker/metadata-action, pre-release Docker images previously populated the :latest tag. This means that administrators that expected the :latest tag to result in a stable release of Anubis got a prerelease version suddenly when they ran automatic updates.

If administrators want to opt-in to the prerelease build track of Anubis for more frequent access to new features, they can use the :pre tag:

image: ghcr.io/techarohq/anubis:pre
        

Features

• Allow importing bot policy rules using wildcard matching (#1815).
• Add support for JSON sources to iplist2rule (#1833)
• Make the honeypot feature log detected addresses to the disk every minute when honeypot.ip_log_file is set. See the IP address logging section for more information.
• Add standard library rules for making Dillo less threatening to Anubis and document how to enable them.

Crawlers

• Allow Arquivo.pt, the Portuguese web archive, by default via its crawling network.
• Add (data)/bots/lyrenth.yaml snippet that denies Lyrenth's AIWebIndex crawler and AIWebIndex-Agent on-demand fetcher by user agent and by their published IP ranges. This is imported by (data)/bots/_deny-pathological.yaml.
• Updates Alibaba cloud IP list (#1813)
• Updates Googlebot IP list (#1812)
• Updates IP list for DuckDuckBot (#1810)
• Update Huawei Cloud IP list (#1814)

Fixes

• Fix bot policy imports to not require pedantically correct YAML formatting when using wildcard matching.
• JavaScript served by the fast challenge is loaded using defer instead of async (#1782).
• Amend default Lightpanda rule to match current behaviour, add smoke test to ensure it keeps working (#1822).
• Fix a panic when a request asks for the undetermined language tag, such as Accept-Language: und (#1776).
• Allow user agents that start with capital-G Git in (data)/clients/git.yaml.
• Enabled the Partitioned flag on cookies by default (#1701).
• Fix Windows MSI builds on prerelease tags such as v1.27.0-pre1.
• Bump AI-robots.txt to version 1.47.

i18n

• Add Basque (eu) localization.
• Update Bulgarian locale (#1708)
• Add Croatian (hr) localization.

2026-08-06 08:00:00

SigV4 looks simple: sign a request, check the signature. Then you implement canonicalization, clock skew, and a cache that isn't allowed to hold your key.

Tigris is a drop-in replacement for AWS S3 (or GCS, anything S3API compatible). As such, we need to be fully compatible with both the mechanisms and semantics of S3 including the SigV4 authentication protocol. This is the lingua franca of authentication in the object storage landscape; even Google Cloud Storage has a way to enable SigV4 support so you can use existing applications against its object storage service.

At first I thought that SigV4 was fairly simple. Clients sign requests, servers do the same work and make sure the result matches. The main sticking point is that the cryptography involved is symmetric cryptography, the kind where both parties need to have the same secrets. This makes some scaling issues weird, but we'll get into that in the future.

SigV4 in a shellnut

At a high level when a client signs a request with SigV4 you get an access key ID and secret access key. The access key ID is functionally a username and the secret access key is functionally a password. Admins can identify keypairs by the access key ID (without special training or tools) and services use the owner of the access key or policies delegated to that access key to determine what actions that client may take.

SigV4 uses HMAC (hash-based Message Authentication Code) and SHA-256 (SHA-2 with a 256 bit hash width) to do authentication by creating salted hashes based on request metadata.

In order to send a SigV4 request, clients take the outgoing request, reduce it to a canonicalized form, and sign it with a symmetric key derived from the secret access key, the current date, region of the service, and service name, kinda like this Go code:

func HMAC(key, data []byte) []byte {
        	h := hmac.New(sha256.New, key)
        	h.Write(data)
        	return h.Sum(nil)
        }
        
        var (
        	kDate    = HMAC("AWS4"+secretAccessKey, nowDate)
        	kRegion  = HMAC(kDate, region)
        	kService = HMAC(kRegion, service)
        	kSigning = HMAC(kService, "aws4_request")
        )
        

As an example, let's see what a signed GET request to a HTTP debugging endpoint looks like on the wire with and without the signature:

$ curl http://localhost:3000 -v
        
        GET /
        User-Agent: curl/8.7.1
        Accept: */*
        

And when you add the signature with --aws-sigv4:

$ curl \
          --user tid_YOISC719YLXSONFU:tsec_DiYqeH8t0IKjKUKfqhzTsqrCCUl9Wm0m+6MXNhhi1fU \
          --aws-sigv4 aws:amz:auto:s3 \
          -v \
          http://localhost:3000
        
        GET /
        User-Agent: curl/8.7.1
        Accept: */*
        Authorization:
          AWS4-HMAC-SHA256
          Credential=tid_YOISC719YLXSONFU/20260720/auto/s3/aws4_request,
          SignedHeaders=host;x-amz-date,
          Signature=879bcdd43749cfc9782b876d9ceb3ff153d79ab1482290cca7ab915bb7f8785d
        X-Amz-Date: 20260720T153748Z
        

Breaking it down we have two extra headers in the request:

• Authorization: The fixed string AWS4-HMAC-SHA256 to signal to the server which authentication mechanism is in use. The rest of the string is information about the request signature so the server can properly canonicalize the request.
• X-Amz-Date: The date and time (UTC) of the request so the server knows when the request was signed. Servers will use this request date in order to reject old requests to prevent replay attacks.

Request canonicalization and signing

On the wire, HTTP/1.1 requests look kinda like this:

GET /api/list?page=0&count=30
        User-Agent: curl/8.7.1
        Accept: */*
        Host: myawesomesite.example
        

However the headers could be sent in any order, and changing the order of request headers doesn't result in different requests. Additionally any query string parameters could be formatted in any way a client (or server) could imagine, including the use of semicolons to separate values. All attempts to canonicalise HTTP requests MUST deal with this ambiguity and define their own rules.

SigV4 canonical requests are made up of a few parts:

• The HTTP method (GET, PUT, POST, DELETE, etc.)
• The URI path of the request (/api/list, etc.)
• The sorted canonical query string (you must exactly match the server-side canonicalization logic)
• The signed headers terminated with two newlines
• The sorted list of signed headers joined by semicolons
• The SHA256 checksum of the request body

For that example /api/list request, the canonical form would look like this:

GET
        /api/list
        count=30&page=0
        host:myawesomesite.example
        x-amz-date:20260715T204745Z
        
        host;x-amz-date
        e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        

As the request has no body, the empty sha256 checksum e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is put as the body checksum.

I'm reading the blogpost at <link> and Xe mentioned AWS SigV4's
        STREAMING-AWS4-HMAC-SHA256-PAYLOAD method. I would like to learn more about how
        this works. Please research how this works and give me code and request body
        samples.
        

To make the signature, you take the sha256 checksum of the canonical request and then HMAC it against that derived signing key:

finalRequestSignature := HMAC(kSigning, reqSig.Bytes())
        

And construct the Authorization header based on your access key ID, service region, and service name:

req.Header.Set("Authorization", fmt.Sprintf(
        	"AWS4-HMAC-SHA256 Credential=%s/%s/%s/%s/aws4_request, SignedHeaders=%s, Signature=%x",
        	accessKeyID, nowDate, region, service,
        	strings.Join(signedHeaders, ";"),
        	finalRequestSignature,
        ))
        

What about SigV4a?

AWS has made an extension to SigV4 that uses asymmetric cryptography called SigV4a (the "a" means asymmetric). Instead of using symmetric cryptography on both the client and server in ways that means the server needs to either know the client's secret access key (or a value derived from the secret access key), SigV4a uses key derivation functions to derive a cryptographic keypair. Servers authenticating requests fetch the public key from IAM. Only the client and IAM know what the private key is, and that private key is what signs outgoing requests.

I'd love to use SigV4a more because it makes adding additional services to the mix (such as a git service) a lot safer as you can have those additional services exist in different trust domains than the core product. This is the core of how microservices end up happening. However, it's not super widely used even within AWS. The only SigV4a use I can find in Amazon is S3 Express Zones, however they may end up using it in other services I'm just not aware of.

When I did my own experimentation with SigV4a (where I was implementing my own IAM server so that I really understood this all at a low level), I had to copy a lot of internal AWS SDK code into my repo in order to get it working.

I'll talk about SigV4a some more another time.

Replay attacks and you: a young coder's illustrated primer

One of the weaknesses of using signatures for API authentication like this is the problem of replay attacks. When you make a naïve signature of a value, there's no real way to tell when that signature was created. If you sign a request to create a compute instance at time instance t0, it's still technically valid at any other time instance tN. This is why the canonical form of SigV4 requests includes the current date and time:

Authorization: [...] SignedHeaders=host;x-amz-date, [...]
        X-Amz-Date: 20260715T205432Z
        

This means that the request was signed on July 15, 2026 at 20:54:32 UTC. Time changes constantly (at least at the rate of one second per second!) and the client has to have a working clock in order for TLS to work. Servers can trivially read the contents of X-Amz-Date and reject old requests. This means that you don't need to add or store nonce (number used once) values with each request because that doesn't scale.

As a side effect of this, you need to set a temporal skew window for validating requests. This window needs to be generous enough to accommodate slow clients, sloppy timekeeping on the client side, highly latent clients, leap seconds, or other exceptional temporal phenomena. In general time synchronization is a surprisingly hard problem, so it's best to just be tolerant of clients in order to make things more robust in practice. AWS uses a temporal skew window of 15 minutes for validating requests. I'm going to use a window of 5 minutes for my API because 300 seconds is a nice round number and I don't have to deal with the same amount of legacy code that AWS does.

How TAG changes the game

So all of this SigV4 business had been working really well for Tigris. Then we worked with a few customers who needed a local cache to fully saturate their hungry GPUs. To be fair, Tigris is plenty fast, but the real thing that kills AI training is latency and something that runs locally will always be faster than the cloud.

In order to provide that sweet middle spot between making everything rely on the cloud and having everything local, we made TAG, the Tigris Acceleration Gateway. This effectively gives you most of a Tigris region in your own infrastructure.

When you connect to TAG, your code uses its existing access keypairs, buckets, and code. You point your code to TAG, you point TAG to Tigris, and then everything is cached for you. But how does TAG authenticate with your code? TAG doesn't have access to all your existing API keys (and to be honest it shouldn't), but it's still able to authenticate them with SigV4 authentication.

TAG and the IAM server both implement a signing key proxying feature that lets a client and TAG both prove their identity to Tigris. Once that proof is sent, then TAG gets the intermediate derived signing key and uses that for locally validating requests, kinda like this:

sequenceDiagram
           participant Client
           participant TAG
           participant Tigris
        
           Client->>TAG: ListBuckets<br/>(signed)
           TAG->>Tigris: ListBuckets<br/>(signed) + proxy hdrs
           Note right of Tigris: 2xx, keys returned
           Tigris-->>TAG: ListBucketsResponse<br/>+ keys (encrypted)
           Note right of TAG: decrypt, cache
           TAG-->>Client: ListBucketsResponse
        
           Client->>TAG: ListBuckets<br/>(signed)
           Note right of TAG: verify locally,<br/>cache hit
           TAG-->>Client: 200 OK
        

The actual implementation in TAG involves some derived AES logic so that the derived signing keys are very much limited to the client that requested it (namely: the AES key is the SHA256 encoded form of the proxy secret access key). One of the weird parts is that the canonical form of the proxied requests differ from the normal SigV4 canonicalization process, namely looking like this:

tag.default.svc.cluster.local # Host header from the client
        1784577479                    # Unix timestamp of the request (X-Tigris-Proxy-Timestamp)
        GET                           # HTTP method of the client
        /                             # HTTP path of the client
        

This is signed using the same SigV4 signature process as before but added differently to the request:

• X-Tigris-Forwarded-Host: the HTTP Host of client requests (EG: tag.default.svc.cluster.local)
• X-Tigris-Proxy-Access-Key: the Tigris keypair used to authenticate TAG itself (must be in the same organization as the client)
• X-Tigris-Proxy-Timestamp: the time of the request in unix timestamp format
• X-Tigris-Proxy-Signature: the hex output of signing the canonical form of the request against TAG's secret access key

And then TAG reads the response from Tigris, caches those derived signing keys, and then uses those in the standard SigV4 process to authenticate clients: no round trip to the cloud required.

I was wrong about the simple part

The happy path is exactly what I thought it was. Reduce a request to a canonical form, run four HMACs, compare the result. That part fits in an afternoon.

Everything expensive lives in the questions around it. Which bytes count as the request? Whose clock decides that a signature is still good? Who gets to hold the key that proves any of it? Each question has an obvious answer, and each obvious answer is wrong in some specific way you only find by implementing it.

That last question is the one that surprised me. I read symmetric cryptography as a hard limit: if the verifier needs your secret, the verifier has to be Tigris. It isn't. SigV4 derives its signing key through a chain of four HMACs, each one scoped tighter than the last: date, then region, then service. Those intermediate values can travel without the secret behind them. TAG rides that. The key it holds stops working when the UTC date rolls over. It covers one region and one service. You can't walk it backwards into a secret access key.

We also didn't write any of this, which is its own kind of relief. SigV4 is old, widely deployed, and hammered on by every S3 client in existence. Any compatibility bugs here are ours. The protocol's bugs are everyone's.

The place a protocol bends is usually some intermediate value that somebody already designed to be thrown away.

If you want a Tigris region in your own datacentre, the Tigris Acceleration Gateway caches your buckets locally and authenticates your existing keypairs with the same SigV4 dance your SDK already speaks.

2026-07-14 08:00:00

The scraping problem is worse than anyone can imagine and thanks to my friends at Sourceware we have some real data to prove it.

I've been working more on Anubis' reputation database and I've run into a really weird discovery: 80-90% of the hits created by the honeypot feature are from IP addresses that do not belong to any existing threat monitoring lists.

Here's a breakdown of the honeypot hits Sourceware has gotten in the last few months:

Assessment of ./data/manually-submitted/sourceware/202607141625.txt against ./var/reputationdb.mmdb

In case this interests you, I have put the full tables in Appendix A: Full tables for the reputation database input.

Flags (of flagged addresses)

Categories (6 distinct, of flagged addresses)

Providers (126 distinct, of flagged addresses)

Mara

Methodology note: "provider" here means one of two things:

1. The company or organization associated with the IP address.
2. The place the list was gotten from.

For example, scaleway is based off of Scaleway's publicly posted IP address ranges, firehol-level1 is based on a daily snapshot of FireHOL's Level 1 IP list, and fdo is based on data contributed by the administrators of freedesktop.org.

(remainder snipped for brevity)

Countries (229 distinct, of all addresses)

This doesn't list data from 204 additional countries. Given that the ISO 3166-1 standard comprises 249 countries (193 of which are UN members), it's safe to say this is a global problem.

ASNs (21116 distinct, of all addresses)

There are 18069 more ASNs not listed.

How Anubis' honeypot works

In order to collect data on how widespread the scraper problem is, I added a honeypot feature to Anubis. On every challenge page it adds semantically invalid HTML akin to the following:

<script type="ignore">
          <a href="/.within.website/x/cmd/anubis/api/honeypot/<uuidv4>/init">Don't click me</a>
        </script>
        

Visiting that page gets you cheap to generate vacuous anti-content that has two links to other pages. This is intended to get badly written scrapers caught in the honeypot so they scrape that instead of the protected website. I made it on a whim but thought it would be great for collecting data on how widespread this problem actually is.

This is a global problem

Based on the data I've seen, this is a global problem. If I had to guess where most of this traffic is coming from, it's from compromised smart appliances contributing traffic to proxy networks. I don't think there's any way to make a real impact on this problem without concerted simultaneous global action.

TL;DR: the scraping problem is actually widespread enough that web application firewalls like Anubis make sense.

2026-07-14 08:00:00

A presigned URL is a replay attack you did on purpose.

Replayable auth tokens are the textbook way to create vulnerable systems, but Tigris ships them as a first-class feature with presigned URLs and so does every other object storage system on the planet. However this isn't an oversight because presigned URLs turn a weakness into a feature.

Replay attacks are a real problem and the classic fix is miserable

When you authenticate a request with Amazon's SigV4 protocol for Tigris, your client boils down the request to a canonical form: a SHA256 hash of the request's method, path, query parameters, signed headers and a SHA256 hash of the payload. It runs the result of that through HMAC with a signing key derived from your secret access key. Nothing secret ever crosses the wire. The server derives the same key as the client, does the same canonical form transformation, and compares the result.

Being able to make a valid signature proves that the request came from someone holding the secret access key, but it proves nothing about when that request was made. A signature that was made a year ago would still be valid today or any other time you send it, so in theory an attacker could warehouse your signed requests only to replay them en masse later. Imagine sitting on a pile of signed "create EC2 instance" calls only to spam them all out at a later date. You would be a twirling moustache villain able to spawn dozens of servers at a moment's notice.

Traditionally the fix is to bake a nonce (number used once) into the signature (sorry to any British readers in the audience). This makes every signature differ because that nonce differs.

However with great power comes great responsibility and making sure that something used once is only used once is a surprisingly hard distributed systems problem. You can't verify that something is only used once locally. Say you store them all for a 15 minute smear window at a low request rate like 10,000 Bq. That's 9 million live nonces, and every frontend node needs to have a consistent view of the whole set as it churns.

You have made your fast authentication check slow from having to ensure things are only used once.

What you want instead is something that changes constantly without coordination and invalidates those old signatures for free. For an added bonus you want this to also be in the standard library of every programming language.

Sign the clock

There's exactly one value that changes constantly, (mostly) monotonically, and is already actively coordinated across all elements of the stack: the clock. Your OS already keeps time in sync with the public NTP pool (or a private NTP pool if you are cool enough to have radioactive PCI cards laying around). Without an accurate view of time you can't make TLS connections, which means you can't make API calls to Tigris at all, so the auth layer gets to assume a working clock exists.

SigV4 signs the current time into the request. If an attacker gets their greasy hacker paws on a signature, they have about 15 minutes to use it before it becomes a digital paperweight. If time is an input to the signature and the time changes enough to invalidate the signature, the signature is null and void. Sure in theory a sufficiently funded attacker could create a black hole in your datacentre and disrupt temporal flow, but at that point the planet is probably toast which makes the attack profile moot. Commit mass object storage fraud with this one neat trick! The department of temporal investigations will have hated it!

This makes your verification stay stateless. Everything gets checked against the system clock the server already needs and you can give clients a 15 minute signature smear window as a grace period for old or delayed clients (exponential backoff is a good thing and Tigris will reward you for doing it).

Of course the real thing keeping the signatures safe on the wire is TLS (HTTPS). If that is broken we have bigger problems and object storage fraud is the least of our problems.

Time is the only nonce you need because both sides already agree on it anyways.

Some thorns have roses

Presigned URLs take the replay tolerance that SigV4 spends all this effort nerfing and then buffs it into the feature. The entire auth dance gets flattened into URL parameters that any HTTP client can use, be it a browser, curl, Go's net/http, or something you made by bit-banging HTTP over a socket. Here's a real presigned URL I sundered into visibility:

https://xe-sophia-base.t3.tigrisfiles.io/moby-dick.txt
        ?X-Amz-Algorithm=AWS4-HMAC-SHA256
        &X-Amz-Credential=tid_ubYBNEYAmTciLVwszw_QrUXDmtcyQisryryGfxgznDsCnOvNqh/20260714/auto/s3/aws4_request
        &X-Amz-Date=20260714T043308Z
        &X-Amz-Expires=3600
        &X-Amz-SignedHeaders=host
        &X-Amz-Signature=0dcaf4972911527a7582ff36ea457e9760a8efccb6655a178685aaa281637a36
        

Here are the parts (forgive the AI looking listicle because this is genuinely the best way to format this):

• X-Amz-Algorithm: the signature scheme. Effectively always AWS4-HMAC-SHA256.
• X-Amz-Credential: the access key ID plus the credential scope — date, region, service, and the literal terminator aws4_request. The signing key is derived by chaining HMAC through exactly those parts, so a signature is only ever valid for that day, that region, that service.
• X-Amz-Date: the second the URL was born, in UTC.
• X-Amz-Expires: how many seconds it gets to live, chosen by the signer.
• X-Amz-SignedHeaders: which HTTP headers are folded into the signature. Usually just host, because you can't force whoever you hand a URL to into sending exotic headers.
• X-Amz-Signature: 64 hex characters of HMAC-SHA256 over the canonical request — the method, the path, every parameter above, the signed headers, and the payload hash. Change any of them and the math stops agreeing.

All of these are normally HTTP headers in standard SigV4 requests.

GET /moby-dick.txt HTTP/1.1
        Host: xe-sophia-base.t3.tigrisfiles.io
        X-Amz-Date: 20260714T043308Z
        X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        Authorization: AWS4-HMAC-SHA256
        Credential=tid_ubYBNEYAmTciLVwszw_QrUXDmtcyQisryryGfxgznDsCnOvNqh/20260714/auto/s3/aws4_request
        SignedHeaders=host;x-amz-content-sha256;x-amz-date
        Signature=0dcaf4972911527a7582ff36ea457e9760a8efccb6655a178685aaa281637a36
        

Note that this request is not a legal request, it's an example to illustrate the point, here be dragons, etc etc etc.

It's best to think about this presigned URL as a capability grant. Whoever holds it gets to make exactly one (1) kind of API call with one (1) HTTP method against one (1) object in one (1) bucket. They can do this as many times as they want until the presigned URL expires. The signature covers the method, the path, and the signed headers so a user can't take a presigned request for GETting a copy of Moby Dick from a development environment and weaponize it into a way to delete everything in your production bucket.

Possession is authorization until the clock says no.

What it costs you

Capability grants like this can have some sharp edges. There is no real way to revoke any individual presigned URL short of killing the access key it was signed with. When that key dies, everything it signed dies too. This includes any URLs you may have wanted. This cuts both ways and it kinda has to unless you make a new keypair per presigned request, which is probably out of scope.

Expiry has fine print too. A presigned request can live anywhere from one (1) second to one (1) week (seven (7) periods of twenty-four (24) hours).

There's no limit to the number of times a client can use a presigned request. If you give a mouse permission to GET one cookie, they can GET that same cookie over and over. You end up having to pay for the GetObject calls in the end, so keep that in mind.

URLs also leak, but these URLs are born to die. Presigned URLs will end up in API responses, chat messages, GitHub comments, and your browser history. The tradeoff is acceptable because all the links self-destruct, but it's a tradeoff you need to keep in mind when you design your services, not a panacea for access control.

Presigned URLs sound like a great way to prevent hotlinking. At some level they are (a few of my services use them as such), but what they actually do is put a lifetime on hotlinking. This makes things annoying enough that it usually gets people to stop.

The hole in the fence is the gate

SigV4 makes a lot of API authentication challenges so much easier. It spent most of its innovation budget on making signatures die quickly because replay attacks are the classic way that signed requests go wrong. Presigned URLs looked at that property, shrugged, flipped it on its head, and made it into a feature.

The thing that looked like a problem becomes a fundamental construct to build your apps upon.

Want to hand out links that expire themselves? Tigris supports presigned URLs out of the box with the same SigV4 dance you already know, on globally distributed, S3-compatible object storage. Read the docs.