2025-05-02 09:20:22
During the lockdown of 2020 and subsequent Twilight Zone times that followed, I learned a bit about how a local restaurant's web site operates. That web site had become the lifeline for the business since everything had turned into takeout or delivery, and it wasn't always right, so I brought it up with the owner. He said he had "a guy who takes care of it". This makes sense since his thing was running a restaurant and keeping that going. He doesn't sit around dealing with domain names and web sites every day.
At one point "his guy" became unreachable for several months, and the restaurant owner asked me what his options were since he knew I was "one of these computer people". I had to lay out the top to bottom of how this stuff works in terms of "this content gets displayed with this name on it", and figured it might benefit others if I put some variant of it online.
Let's say you're the restaurant owner. You're paying some third party to run a web site and list your menu, hours, and that kind of stuff. They might also have some kind of thing rigged up to send orders through online, or an agreement with yet another vendor to do it through them.
You need to get something changed on the site. Maybe your menu changed and you need to warn people about the increased price of egg-based products. Maybe you have new hours of operation. Whatever. What steps can make that happen?
In no particular order, an incomplete list:
"Ask them to change it" - contact your person (people) and ask them to change it. That's what you're paying them for. You are paying them, right?
"Change the actual files" - find out how the serving of content works, gain access to it, and then make changes to the file(s). This requires account access to whatever they happen to be using to run the site. If it's just a bunch of flat files on a disk somewhere, it might be easy. If it's just one entry in a much bigger system, it might not. Also you have to know how to do this and/or have new people to do this for you (as with most of the entries in this list).
"Replace the document root" - figure out how the web server works, then swing your-restaurant.example.com around from their directory to your directory, or do the equivalent database wrangling if it's some dynamic thing. This requires some admin powers on the box/system itself.
"Take over the server" - gain admin powers on the box through security holes or good old-fashioned physical access and single-user mode. Then go in and change the document root, edit the files, or do whatever else you feel like doing. Requires security skills and/or physical access to the server.
"Replace the server" - find out where the server is (assuming it's just one machine) and then physically replace it with one that you control but is otherwise configured for the right IP address(es) and whatever else is required to serve the site. This might mean "unplug the old one and plug a new one in on the same spot". Requires physical access to the hosting arrangements, and if it's in the clown, yeah, forget about it.
"Change the DNS" - get access to whatever controls the DNS zone for that domain and repoint it to new hosting arrangements with new content installed. This means changing things like A and AAAA records and/or CNAMEs. Requires ability to log in to whatever's running the show.
"Change the primary nameservers" - get into the account at the registrar for the domain and repoint the primary nameservers to ones where you can set the data for the domain. Continue as above. Requires ability to log into the registrar account.
"Move the domain to another registrar" - maybe you can't get into the existing registrar, but maybe you can go through the transfer process to pull it to another one where you can change the nameservers. Then you proceed as in that option (above). Requires ability to transfer a domain, which typically involves a one-time key from the "losing" registrar, which in turn usually involves access to the account. There may be legal options for *yoinking* the domain without such access, assuming you can prove ownership, or otherwise bludgeon the companies involved into doing what you want.
Basically, if you actually own some of the items in question, you have options. If you are the owner of the registrar account and it's just some rando tech person who's a contact on it that does the work, you authenticate yourself to the registrar out of band, have them removed, get yourself (or a new tech rando) installed, and continue from there. The same applies for the DNS serving, or the actual web hosting.
This pattern pretty much plays out everywhere: is it your account? Or, did you pay them to "take care of everything" and your only "interface" to it is through them?
I don't think people really know the implications of some of these setups.
2025-03-27 03:08:29
Okay, first off, everybody breathe. Everyone is freaking out. This is not the way to do this.
Right, so, let me try a half-assed analogy here. There's a day care center and a bunch of people send their kids to it. Then one day, someone finds out that they built the playground out of the sharpest materials available. It's not that the playground is from the '70s or something, either. It's relatively new.
Saying "maybe you don't want to send your kids there" is what I did.
Now, so, this is a bad analogy, because I didn't go into details. In this case, I'd rather the world have a chance to turn this stuff off before it becomes known just what all is possible. The nature of this sort of thing means it's a race and I'm on the side of protecting machines for some reason, so I asked people to stop running it.
Now, first off, I don't know exactly how to exploit this sort of thing. I was there in the 90s when this overflow stuff started popping off, and I'm pretty sure that if you can do this, you can do much worse.
This is what I'm talking about:
Terminal 1:
user1$ random-tool
Terminal 2:
user2$ atop malloc(): corrupted top size Aborted
... that's pretty bad, right?
So you twiddle the tool a little bit and go again and...
user2$ atop Segmentation fault
That's not great, either!
Or... how about...
user2$ atop Fatal glibc error: malloc assertion failure in sysmalloc: (old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)
Again, this is not my world. I've never written a heap exploit, but reading about it briefly makes me think that there's meat on these bones.
user1 does something... and gets user2 to blow up. If you can make that do something useful, then you get user2 to run stuff on your behalf.
What if user2 is root? Then you own the box.
Okay? So maybe stop running it as root, but also just stop running it.
2025-03-26 06:29:27
My life as a mercenary sysadmin can be interesting. Sometimes I find things, and sometimes I hear things. Now and then I say things.
Right now, I think it's probably best if you uninstall atop. I don't mean just stopping it, but actually keep it from being executed.
I'm not talking about the OG top, or htop, iftop, or anything else with a "top" name. Just atop.
I can go into why another time.
2025-03-12 04:49:49
Look out, I'm talking about feed stuff again.
Since last week's post with the latest report on feed reader behaviors, a fair number of things have happened.
First up, I reported that two FreshRSS instances were sending unconditional requests. This turned out to be a pair of interacting bugs, and the developers hopped right on that and stamped it out. Their dev instances shifted right away. That's great.
Then, something funny happened. I wrote that post about the modem noises and it included a MP3 file. The way I do this in my stuff here is that I have some commands baked into my page generator that knows how to create the <audio> container and then put one or more <source> elements into it. So, I did that for the (admittedly crappy) recording and shipped it off.
I didn't know this at the time, but this immediately triggered strange behavior in a number of feed readers, FreshRSS included. They all started sending the wrong data. That is, they'd get the latest version of the feed with the "2025/03/05/1670" post in it, but on their next poll they'd still be sending the If-Modified-Since and/or If-None-Match values from the previous version.
This sent me down the road of trying to flush out this behavior on the feed score project side of things, but nothing I did would reproduce it. Again, I didn't know that something about that specific post was responsible, so I was groping around in the dark.
Finally, I reported this to the FreshRSS devs, and they came right back and showed me my problem: my feed says it's xhtml, but it was doing something that's not valid: <audio controls>. That's not valid XML, so it's not valid XHTML by extension. It's just fine in a regular web page, naturally.
I hate the web.
How did this happen? Easy: In 2023, I rewrote my generator for these web pages and the feeds. There's now a bunch of code that *tries* to crap out reasonable (and validating) results instead of doing "HTML by printf" like it had been before.
Along the way, I obviously didn't internalize the rule that "in XHTML, attribute minimization is forbidden". It sounds like the kind of thing I would have read once back in the 90s when I was trying to get pages to validate as XHTML 1.0 for some cursed reason, and then promptly forgot. So, it would do a bare "controls" attribute and thought it was fine.
In practical terms, it means you can't do this:
<audio controls loop>
But instead you have to do this:
<audio controls="controls" loop="loop">
So yeah, I wrote something terrible. Again, I hate the web.
Anyway, what was happening is that a single post within the feed was failing validation as a self-proclaimed xhtml document, so the feed readers were rejecting the _entire poll_ and were not updating their caches. The next time they polled, it was as if they had never sent the request in the first place (as far as they were concerned).
Once I shipped a fixed version of the feeds, they went back to normal.
I'd snark about something here, but clearly, I'm no better at this.
What's different now? Well, my bits of code which are responsible for attributes ("controls", "loop", "href", ...) now refuse to generate a minimized attribute when being used for XML, and that includes the feed stuff. They'll blow up before they ship garbage like that again.
My thanks to the FreshRSS team for being responsive and helping me out.
Now, where did I put my clown nose...
2025-03-07 23:34:51
I recently wrote about getting a pair of VIC-20s and attached 1670 modems to connect over a length of line cord with nothing else in the middle. At the end, I tossed out a side note about the odd noises these things make in dial mode.
Some readers have been asking if it could be a relay, like for pulse dialing, and so I figured I should try to clarify as much as I can.
First, here's a recording from the other day's shenanigans, complete with me bopping the keys on the less-reliable computer to make it Do Stuff.
For those who can't hear it for whatever reason: there's some background humming in the whole recording from other devices in the room. But, then there's a source of white noise that's much closer and louder - that's the modem joining in. It "dips" every now and then and is replaced with the "pa-tink" noise I mentioned and then it comes back up to the previous volume level. This happens a few times, and I smack a key or two, and then the recording ends.
Looking in Audacity, it hits at about 1.2s, 2.4s, 3.6s, and 4.8s. That's curious, right? About 1.2 seconds each time?
To be crystal clear: this is not pulse dialing. That sounds completely different, and besides, I didn't tell the modem to dial anything. I sent it "ATD" which basically means "go into originate mode without actually trying to generate any pulse or tone signals - just grab the line and start listening".
Another rejected hypothesis: this is not just something broken in my originating test unit here. Both of them do it when they're in that role. It's also not a case of "they're almost 40 years old and they both degraded into this state", because the original I had way back in 1988 definitely did this, too. That's how it got stuck in my head in the first place!
One thing I don't know is whether this is solely a function of the local speaker, or if it's actually putting that clicking onto the line. I'm guessing it's just the speaker, but I'm also not feeling like setting up the test rig and somehow finding a way to listen to one side and not the other.
For whatever reason, the 1670 makes those noises when it's sitting there waiting for an answer tone from the other side, and I've never understood why. Obviously, at this point, it doesn't matter, but it's just one of life's little nerd mysteries.
2025-03-04 05:55:56
In mid-January, I made an announcement that the feedscore system would be resetting the logs for a fresh start, and was bumping the URL along to cast off forgotten readers. This had the desired effect of focusing on those people who are actively involved and let the others start getting DNS failures without bothering my servers. Win-win!
Anyway, this means we now have about 45 days of data to look at, and it's possible to build a report based on the latest observed behaviors.
The usual caveats apply: if one reader instance is being odd, that might be one user smacking the refresh button over and over and/or using terrible settings. If multiple instances of the same program are behaving the same way, it's probably the program itself.
Let's start with the ones that seem to have their conditional request situations figured out. This is clearly the biggest part of the list, and it's great to see things lining up like this.
First up, there's one that's not published yet, and they're doing fine. I hope their efforts are paying off and they have a successful launch when they are ready.
There are a bunch of newer Minifluxes doing their thing, with no problems to report. It seems rather popular.
One of the Miniflux users has a different reader program using the same key and it's screwing up the reporting. You should retire that key and get two more from me. This advice goes for anyone: don't point more than one instance at any given key. It renders the data unusable.
There are several NetNewsWire instances doing their thing with no trouble. I haven't been able to trip them up with pathological (and yet totally valid) bumps of the Last-Modified and ETag values yet. This is good! Some of them have bizarre polling intervals... guessing that's user-driven?
There are a couple of SpaceCowboys Android RSS reader instances which seem to be doing fine. One of them has a bunch of oddly-timed unconditional requests which I assume is the user smacking that refresh button. Smack that button with care!
A couple of Vienna instances are doing fine.
Audrey2 RSS reader is fine.
A couple of Emacs Elfeed instances are fine.
"unnamed-feed-reader" (yes, really, that's what it sends) is doing fine.
There's something which doesn't send a User-Agent at all (okay...), and it has some bizarre anomaly of sending a "" INM at startup, but then has been behaving. (Hint: just drop the header if you don't have a value for it!)
Someone put a lot of work into making a curl instance do the right thing, and it's running great.
Roy and Rianne's Righteously Royalty-free RSS Reader is fine.
github.com/er0k/feeds is fine.
walrss is fine.
feedmail.org/0 is fine.
MANOS is fine. (Is that the feed reader of fate?)
Bloggulus is fine.
There's some unidentified browser extension that's doing fine.
feedbase-fetcher is doing fine.
inforss is doing fine.
feedparser/6.0.2 is *so close*. It just needs the 59/60 minute timing adjustment thing (assuming it's going for 1 poll per hour).
...
There are a few which are still "double-tapping" the server when the feed is first added. This would trip a 429 on the real feed and generally indicates a problem with the data model. Another reason for the URL change for this project was to get readers to run their "new feed" flow again.
Two Feedbin instances did this same thing when they were added.
Yarr/1.0 did a second poll about 300 milliseconds later, and a third poll another 27 seconds after that. This is the wrong way to do things. The other timing also seems to not follow any real pattern.
...
Now let's talk about readers with some cache bugginess:
A couple of Newsboat instances still can be tripped up and will keep sending old values in their If-None-Match headers.
There's a BazQux instance which has the same INM bugginess. I wonder if they're using the same library for their HTTP comms.
...
Then there are some weird ones:
rss2email randomly goes unconditional at times. No idea why.
NextCloud-News/25.x.x still sent at least one 1800 IMS. I don't get this. If you weren't sent a value, don't return it in a header. On a positive note, it's no longer beating the hell out of the server trying to get a favicon this week (which isn't in the reports, but I can see it on my side).
...
Finally, everything else:
Something claims to be a browser (Chrome 77? Sure...) and sends 100% unconditionals. This is bad. It also sends HEAD requests which aren't visible in the report, but I can see them, and they're broken and pointless.
One key either has two separate Thunderbird instances banging away (bad! get separate keys!) or has one terrible implementation that always polls twice. Either way, it's not usable data.
There are also two FreshRSS instances which are both sending 100% unconditionals. What happened here? Over and over, it's just this sort of request:
Host: <redacted> User-Agent: FreshRSS/1.26.1-dev (Linux; https://freshrss.org) Accept: */* Accept-Encoding: deflate, gzip, br, zstd
I just don't understand.