(Long story short - It was Backblaze!)

Unwelcome little surprise tonight just as I planned to go to sleep for the evening… A new user messaged me that they couldn’t set a profile or banner pic.

It seemed odd… I haven’t had any “media upload” issues since I self-inflicted some, twice.

First was really early days, in the first move of media out of the server directories to a ZFS partition, and then again in January 2023 when I moved them wholly off-site to Backblaze B2, using the S3-compatible functions. However, once I got that working, it has been completely problem-free. Just long enough to need to revisit some old notes of how that chain is setup.

The thing is…it’s quite straightforward! Mastodon supports S3-compatible storage and Backblaze B2 provides the API. There’s really not much to break here, and my small instance sure shouldn’t be pushing the boundaries of, well, anything.

I do all the usual checks, all the services are up. Out of laziness, I first do some API calls from my home system instead of the server, they work fine.

…Scratches head… 🤔

So I do the same call from the server CLI, and failure. But, the failure is weird… Initially, I saw it couldn’t find the file and I’m here wondering if there was a path change I wasn’t aware of, but I noticed I overlooked the obvious.

No path to host

What do you mean, “no path to host”? I have a functional social media site. I’ve been using it all day. How can my object storage be down? It didn’t click with me at first. The whole design of the object storage is that my users access it directly, it doesn’t need to go through the site. So, if there are files already up there, me and the rest of the users see it just fine. But the instance can’t see the files (nor upload any further). Well, that’s a problem.

But, why can’t my server talk to Backblaze? That seems really odd, that Linode (who loves to remind you they are now Akamai) can’t talk to Backblaze. These folks know as much about networking as anyone! So I start doing some pings. Now we get somewhere!

So, my goodies are on us-west-000. Which my server can’t reach, but I can reach from any other network I own. Another valid server is us-west-001. I can reach that, but my goodies aren’t there 😢

I don’t want to post too many OPSEC-failing pictures here, but the traceroute gives similar results. us-west-000 gets lost in a maze of rdns.backblaze.com bounces (like…20 straight), whereas us-west-001 is clean.

Despite my musical title to this post, I think the likely culprit here is Backblaze. I am left to wonder if somehow my IP got blacklisted by them (depsite 5 months as a paying customer for this site, plus 10 years on other projects!)

But, Linode’s migration to Akamai also concerns me. I don’t think Akamai is interested in small-midsize sites, and that was always the magic of Linode. Unless there is a genuine routing issue (way outside my control…), I just don’t think Linode would bother to block my port 443 egress traffic.

In any case, I documented this up and sent it to Backblaze. It’s the first time I’ve ever needed support from their side, so we’ll see how that goes.

I am now very sleepy so we’ll update this one tomorrow. I thank the users of my instance for their patience!

UPDATE!

Yeah, it actually was Backblaze which was borked. They replied to the support ticket the next morning stating that they had failures with S3 communications.

Search terms for instance owners: 503 , status=503 , mastodon , linode , akamai , backblaze , media uploads