2025-06-25 13:54:58
I've long had a list of "magic numbers" which show up in a bunch of places, and even made a post about it back in November of 2020. You ever wonder about certain permutations, like 497 days, or 19.6 years, or 5184 hours, and what they actually mean?
I've been doing that stuff by hand in a calculator and finally decided to just do it in Javascript and put it online for anyone to try.
So, here's my latest waste of CPU cycles:
I still haven't figured out the Crucial SSD 5184 hour thing, so it's not in there. 5124 hours, sure, I can understand that one, but 5184? 60 more?
Anyway, have fun while the world burns.
2025-06-01 04:39:23
rsync is one of those tools which is rather useful. It saves you from spending the time and effort on copying data which you already have. It's the backbone of many a mirror site, and it also gets used for any number of backup solutions.
There's just one problem: in the name of efficiency, it can miss certain changes. rsync normally looks at the size and modification time of a candidate file, and if they are the same at both ends, that's the end of any consideration. It won't get any further attention and it moves on to something else.
"So what", you might think. "All files change at least their mtime when someone writes to them. That's the whole point of a mtime."
And yet... I'm writing this post, and here we are.
The keen-eyed observers out there are probably already thinking "ooh, bit rot" and other things where one of the files has actually become corrupted while "at rest" for whatever reason. Those observers are right! That's totally a problem that you have to worry about, especially if you're using SSDs to hold your bits and those SSDs aren't always being powered.
But no, this is something you have to worry about *beyond* that. This is about a "sneak path" that you probably didn't consider. I didn't.
Here, let's run a little experiment. If you have a x86_64 Debian box that's relatively current and you've been backing up the whole thing via rsync for a year or two, go do something for me.
Go run your favorite file-hasher tool on /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0 for me. Give it a sha256sum or whatever, or even md5sum if you're feeling brash. Then note the modification time on the file.
Now mount one of your backups and do the same thing on the version of the file that's on the backup device. See anything ... odd? Unusual?
Identical mtimes, identical sizes... and different hashes, right? I spotted this on a bunch of my machines after going "hmmm..." about the whole SSD-data-loss thing.
Clearly, something unusual happened somewhere, and it's been escaping the notice of your rsync runs ever since. I haven't gone digging into the package history for this thing to find out just when and where it happened, and (more importantly) how. It's rather unusual.
If you're freaking out right now, there is some hope. rsync has both -I and -c which promise to not use the quick method and instead will run a checksum on the files. It's slower so you won't want to do this normally, but it's not a bad idea to add this to the mix of things that you do every so many rotations.
I should point out that the first time you do a forced-checksum run, --dry-run will let you see the changes before it blows anything away, so you can make the call as to which version is the right one! In theory, your *source* files can get corrupted, and if you just copy one of those across, you have now corrupted your backup.
Isn't entropy FUN?
2025-05-25 09:26:29
At the start of 2023, I wrote a post talking about why I still had an "old-school cert" on my https site. Well, things have shifted, and it's time to talk about why.
I've been aware of the ACME protocol for a while. I have tech notes going back as far as 2018, and every time I looked at it, I recoiled in horror. The whole thing amounts to "throw in every little bit of webshit tech that we can", and it makes for a real problem to try to implement this in a safe and thorough way.
Many of the existing clients are also scary code, and I was not about to run any of them on my machines. They haven't earned the right to run with privileges for my private keys and/or ability to frob the web server (as root!) with their careless ways.
That meant I was stuck: unwilling to bring myself to deal with the protocol while simultaneously unwilling to budge on allowing the cruft code of existing projects into my life.
Well, time passed, and I managed to crack some of my own barriers. It wasn't by using the other projects, though. I started ripping into them to figure out just how the spec really worked, and started biting off really really small pieces of the problem. It took a particular forcing function to get me off my butt and into motion.
About six months ago, I realized that it was probably time to get away from Gandi as a registrar and also SSL provider (reseller). They had been eaten by private equity some years before, and the rot has been setting in. Their "no bullshit" tagline is gone, and their prices have been creeping up. I happened to renew my domains for multiple years and have been insulated for a while, but it was going to be a problem in 2025.
Giving them the "yeet" was no big deal, but the damn rbtb certificate was going to be a problem. Was I going to start paying even more for the stupid thing every year, or was I going to finally suck it up and deal with ACME?
That still left the problem of overcoming my inherent disgust for the entire protocol and having to deal with all of these encodings they force upon you. My first steps towards the solution involved writing really small and stupid utility functions and libraries that would come in handy later. I'm talking about wrapping jansson (a C library that handles JSON) so that it made sense in my C++ world and I could import JSON (something I use as little as possible). That kind of thing.
This also meant going down some dead-ends, like noticing that libraries existed which would allegedly create certain things (like a JWK) for you, and then realizing that they were not going to make my life any easier. I'd poke at it, reach my limit, and then swear and walk away for another couple of days.
This went on for some time. I have a series of notes where I'd grab a piece of the problem, wrangle it around for a while, get grossed out, and then set it down and go do something else. This just kept happening but I slowly made progress with small pieces that would Do Stuff, and then they'd connect to each other, and so on like this.
One positive development during all of this was discovering this "pebble" test server I could run on an isolated fake system. It would act as an ACME server and would let me harass it with my feeble attempts at implementing a client instead of bothering the real CAs. Even "staging" servers deserve better treatment than active development, after all.
And, well, after a whole lot of mangling and dead-ends and rewrites and other terrible crap, I had an awful little tool that would take a CSR, do all of the idiot dances and would plop out a certificate. I pointed it at Let's Encrypt staging, and it worked. Then I pointed it at their prod site, and _that_ worked. So I did it for the real thing ([www.]rachelbythebay.com), and *that* worked, and I dropped it into place.
Thus, for the past couple of weeks, if you've been hitting the https version of my site, you've been doing it across the new setup.
Now, I took notes about this, and I wanted to share some of my original off-the-cuff thoughts about implementing this for anyone who's similarly broken in the head and wants to see how bad it can be. I will note that I wrote this based on the first thing that worked, and it does not necessarily reflect the implementation I'm on a few weeks later.
...
Make an RSA key for your web site. Then make a CSR for it, setting the CN and adding a matching altname as an extension. No other fields matter. Nobody looks at those, anyway, and none of them will influence your final certificate no matter how prosaic or precise you get in there.
Make an RSA key of 4096 bits. Call it your personal key.
Write something that'll read a CSR. It needs to extract the CN and the SANs - the DNS: ones, at least. Ensure there's actually a CN [*] and actually SANs, and that the CN occurs within the SANs. So, yes, you have to have at least one SAN.
[* - I now know you can run with just SANs. I did not at the time.]
Write something that'll do a HTTP GET to <directory URL> which is given to you by the ACME service operator. It then needs to parse the body as JSON (or die) and extract some strings from the top-level object: "newNonce", "newAccount" and "newOrder" at the very least.
Write something that'll read an RSA key file on disk. It needs to extract the publicExponent (probably 65537, but you never know...) and the modulus. Make it read your personal key from earlier.
If you end up using "openssl rsa -in foo -noout -text" to do this, the modulus is a bunch of printed hex digits, like "00:ff:11:ab:cd:ef:22:33". It hard-wraps the output and also indents the lines, so you get to clean all of that up first.
Skip the first 00 for some inexplicable reason. Take the other bytes of the modulus and turn them into the actual character values, so a literal 0xff, 0x11, 0xab, 0xcd and so on down the line in the same order you find them in the file. Hang on to these values for later.
Write something that'll turn an ordinary integer into its equivalent big-endian bag of bytes, but don't pad it out to any particular alignment. You need to take that "65537" from your publicExponent and turn it into the equivalent bytes, so 0x01, 0x00, 0x01. Yes, it's a number you just turned back into a binary representation.
Write something that will do base64 *style* encoding, but not quite. The last two characters in the encoding set are usually + and /, but that won't do for webshit, so you need to make it use - and _ instead.
Take that publicExponent (65537), pump it through your big-endian bag-of-bytes encoder to get 0x01 0x00 0x01, then put it through your "base64web" encoder to get "AQAB".
Start a new JSON object. Add a string called "e" and set it to the output of the above step. So, yes, instead of saying that "e" equals "65537", you're saying that "e" equals "AQAB". Aren't you glad you did those extra steps?
Add another string called "kty" and set it to "RSA".
Add another string called "n" and set it to the "base64web" version of your modulus bag-of-bytes from earlier.
Take this JSON object and make it into a sorted compact string representation. This means it goes "e, kty, n" and it also has all of the usual padding (spaces) squished out. Call this a JWK string and save it for later.
Create a second JSON object. Add a boolean to it named "termsOfServiceAgreed" that's set to true. (Guess you'd better agree...)
Look up the URL for "newNonce" from the directory JSON you got earlier.
Make a HTTP HEAD request to that URL. Dig around in the headers (not the body, since there is no body on a HEAD) until you find "Replay-Nonce". Extract the value of that header. Hang onto it for later.
Look up the URL for "newAccount" from that directory JSON for before.
Create a third JSON object. Add a string to it called "url". Set it to that (newAccount) URL. Add a string called "alg". Set it to "RS256". Add a string called "nonce" and set it to the value from the *header* in that last HTTP HEAD request.
Add an object to this third object called "jwk". Within it, add "e", "kty" and "n" in a manner that matches what you did earlier (you know, from the "JWK string" you're still holding for later).
Dump this third JSON object to a sorted compact string representation. Call it "protected".
Dump the second JSON object to a sorted compact string representation. Call it "payload".
Create a string where you literally concatenate those two prior strings, such that it's the value of protected, then an actual period (as in 0x2e, a full stop, whatever), then the value of payload.
Write something that'll create a SHA256 digest of an arbitrary string and will sign it with an arbitrary RSA key (like "openssl dgst -sha256 -sign <key>"). The key in question is your personal key.
Pump that "<protected>.<payload>" string through the digest function. Then run it through your "base64web" encoder. Call this the signature.
Create a fourth JSON object. Add a string called "protected" and set it to whatever you built a few steps earlier. Add another string called "payload" and set it likewise. Then add one called "signature" and set it, too.
Take this fourth JSON object and dump it to a sorted compact string representation. Call this your post body.
Make a HTTP POST request to the "newAccount" URL from the directory. Set the content-type to "application/jose+json". Set the post data to the post body string from the previous step.
Dig around in the headers of the response, looking for one named "Location". Don't follow it like a redirection. Why would you ever follow a Location header in a HTTP header, right? Nope, that's your user account's identifier! Yes, you are a URL now. Hang on to that URL for later.
You now have an account.
You still have much to do.
...
That's about where I stopped writing my take on the protocol.
Again, my program no longer works quite like this, but this is where it started after having observed a bunch of other stuff that already existed.
So far, we have (at least): RSA keys, SHA256 digests, RSA signing, base64 but not really base64, string concatenation, JSON inside JSON, Location headers used as identities instead of a target with a 301 response, HEAD requests to get a single value buried as a header, making one request (nonce) to make ANY OTHER request, and there's more to come.
We haven't even scratched the surface of creating an order, dealing with authorizations and challenges, the whole "key thumbprint" thing, what actually goes into those TXT records, and all of that other fun stuff.
...
Random side note: while looking at existing ACME clients, I found that at least one of them screws up their encoding of the publicExponent and ends up interpreting it as hex instead of decimal. That is, instead of 65537, aka 0x10001, it reads it as 0x65537, aka 415031!
Somehow, this anomaly exists and apparently doesn't break anything? I haven't actually run the client in question, but I imagine people are using it since it's in apt.
...
This complexity must be job security for somebody. Maybe multiple somebodies.
2025-05-02 09:20:22
During the lockdown of 2020 and subsequent Twilight Zone times that followed, I learned a bit about how a local restaurant's web site operates. That web site had become the lifeline for the business since everything had turned into takeout or delivery, and it wasn't always right, so I brought it up with the owner. He said he had "a guy who takes care of it". This makes sense since his thing was running a restaurant and keeping that going. He doesn't sit around dealing with domain names and web sites every day.
At one point "his guy" became unreachable for several months, and the restaurant owner asked me what his options were since he knew I was "one of these computer people". I had to lay out the top to bottom of how this stuff works in terms of "this content gets displayed with this name on it", and figured it might benefit others if I put some variant of it online.
Let's say you're the restaurant owner. You're paying some third party to run a web site and list your menu, hours, and that kind of stuff. They might also have some kind of thing rigged up to send orders through online, or an agreement with yet another vendor to do it through them.
You need to get something changed on the site. Maybe your menu changed and you need to warn people about the increased price of egg-based products. Maybe you have new hours of operation. Whatever. What steps can make that happen?
In no particular order, an incomplete list:
"Ask them to change it" - contact your person (people) and ask them to change it. That's what you're paying them for. You are paying them, right?
"Change the actual files" - find out how the serving of content works, gain access to it, and then make changes to the file(s). This requires account access to whatever they happen to be using to run the site. If it's just a bunch of flat files on a disk somewhere, it might be easy. If it's just one entry in a much bigger system, it might not. Also you have to know how to do this and/or have new people to do this for you (as with most of the entries in this list).
"Replace the document root" - figure out how the web server works, then swing your-restaurant.example.com around from their directory to your directory, or do the equivalent database wrangling if it's some dynamic thing. This requires some admin powers on the box/system itself.
"Take over the server" - gain admin powers on the box through security holes or good old-fashioned physical access and single-user mode. Then go in and change the document root, edit the files, or do whatever else you feel like doing. Requires security skills and/or physical access to the server.
"Replace the server" - find out where the server is (assuming it's just one machine) and then physically replace it with one that you control but is otherwise configured for the right IP address(es) and whatever else is required to serve the site. This might mean "unplug the old one and plug a new one in on the same spot". Requires physical access to the hosting arrangements, and if it's in the clown, yeah, forget about it.
"Change the DNS" - get access to whatever controls the DNS zone for that domain and repoint it to new hosting arrangements with new content installed. This means changing things like A and AAAA records and/or CNAMEs. Requires ability to log in to whatever's running the show.
"Change the primary nameservers" - get into the account at the registrar for the domain and repoint the primary nameservers to ones where you can set the data for the domain. Continue as above. Requires ability to log into the registrar account.
"Move the domain to another registrar" - maybe you can't get into the existing registrar, but maybe you can go through the transfer process to pull it to another one where you can change the nameservers. Then you proceed as in that option (above). Requires ability to transfer a domain, which typically involves a one-time key from the "losing" registrar, which in turn usually involves access to the account. There may be legal options for *yoinking* the domain without such access, assuming you can prove ownership, or otherwise bludgeon the companies involved into doing what you want.
Basically, if you actually own some of the items in question, you have options. If you are the owner of the registrar account and it's just some rando tech person who's a contact on it that does the work, you authenticate yourself to the registrar out of band, have them removed, get yourself (or a new tech rando) installed, and continue from there. The same applies for the DNS serving, or the actual web hosting.
This pattern pretty much plays out everywhere: is it your account? Or, did you pay them to "take care of everything" and your only "interface" to it is through them?
I don't think people really know the implications of some of these setups.
2025-03-27 03:08:29
Okay, first off, everybody breathe. Everyone is freaking out. This is not the way to do this.
Right, so, let me try a half-assed analogy here. There's a day care center and a bunch of people send their kids to it. Then one day, someone finds out that they built the playground out of the sharpest materials available. It's not that the playground is from the '70s or something, either. It's relatively new.
Saying "maybe you don't want to send your kids there" is what I did.
Now, so, this is a bad analogy, because I didn't go into details. In this case, I'd rather the world have a chance to turn this stuff off before it becomes known just what all is possible. The nature of this sort of thing means it's a race and I'm on the side of protecting machines for some reason, so I asked people to stop running it.
Now, first off, I don't know exactly how to exploit this sort of thing. I was there in the 90s when this overflow stuff started popping off, and I'm pretty sure that if you can do this, you can do much worse.
This is what I'm talking about:
Terminal 1:
user1$ random-tool
Terminal 2:
user2$ atop malloc(): corrupted top size Aborted
... that's pretty bad, right?
So you twiddle the tool a little bit and go again and...
user2$ atop Segmentation fault
That's not great, either!
Or... how about...
user2$ atop Fatal glibc error: malloc assertion failure in sysmalloc: (old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)
Again, this is not my world. I've never written a heap exploit, but reading about it briefly makes me think that there's meat on these bones.
user1 does something... and gets user2 to blow up. If you can make that do something useful, then you get user2 to run stuff on your behalf.
What if user2 is root? Then you own the box.
Okay? So maybe stop running it as root, but also just stop running it.
2025-03-26 06:29:27
My life as a mercenary sysadmin can be interesting. Sometimes I find things, and sometimes I hear things. Now and then I say things.
Right now, I think it's probably best if you uninstall atop. I don't mean just stopping it, but actually keep it from being executed.
I'm not talking about the OG top, or htop, iftop, or anything else with a "top" name. Just atop.
I can go into why another time.