Having a little monster limits the amount of time I can spend watching movies. And the pandemic certainly slowed the production line of content down. But last night, I managed to squeeze in a watch of Clerks 3.
As a caveat, I’m a fan of Kevin Smith and his movies. I didn’t discover Clerks until well after its release, but loved it. Mallrats, Chasing Amy, Dogma, Jay and Silent Bob Strike Back, and Clerks 2 are all favorites as well. So it isn’t a surprise I also enjoyed Clerks 3, which is a two-hour nostalgia bomb of a movie. Kevin Smith, like all of us, is much older now, with more life experience under his belt. And this shows throughout the movie, which reminds the viewer to cherish the time we have, even as it dedicates large portions to fan service. The movie wraps up the Clerks trilogy nicely in the end, and it is worth watching through the credits to hear Kevin Smith mention some dialog he omitted in the end.
The only downside is this does feel like the end of the View Askew universe. But for the sake of Jay and Silent Bob, I hope not.
My recourse? Adding a credit card and switching my account from a truly free tier to one where Oracle can bill me if/when I exceed the free usage limitations. Ok, that’s fine, but I also don’t particularly like entities holding my credit card information when I’m just trying to use their free service. And as you the reader might appreciate, avoiding Oracle’s definition of idling requires something to chew up more CPU or network utilization than this small little WordPress blog and the publicly facing personal services I run. Certainly WordPress is a large and complicated CMS in this day and age, but the amount of web traffic required to meet either of the CPU or network utilization thresholds vastly exceeds what I’ve ever seen. Shocking, right?
On one hand, this policy is understandable. Reclaiming the instances in this manner allows for reallocation of my rounding-error of a workload throughout whatever data center they live in and there’s great reasons for this. And on occasion, Oracle has moved my instances onto different hardware. Downtime is acceptable at this price so this never caused any concern. Indeed, that’s just how hosting works sometimes. This is one reason I didn’t start with self-hosting. Hardware continues to break despite our best efforts. And since my self-hosting needs are so light, it didn’t make sense to use my hardware.
The wrinkle was that not all instances were equally easy to obtain. Oracle’s Ampere A1 instances (VM.Standard.A1.Flex) were substantially faster than the AMD instances (VM.Standard.E2.1.Micro. Substantially faster in that I perceived the speed difference just serving this website! Being superior, these A1 instances are difficult to obtain. Indeed, I couldn’t get one when I first signed up in November 2021 and didn’t have one when I blogged about it. Only in March 2022 did I luck into one when just playing around with Oracle’s console. After just playing around some and noticing the speed difference just moving around the system, I migrated most of my compatible services there. My speed indicators jumped immediately, boosting my PageRank and bringing the riches of readership that I now enjoy.
Even using the much slower AMD instances, I can’t hit Oracle’s thresholds. I’m using a well liked and likely very efficient software stack to serve traffic, and have taken precautions to filter traffic through Cloudflare. Even if I removed their services, the traffic wouldn’t cause even a poorly thought out and inefficient software stack to meet Oracle’s thresholds. And to be frank, tinkering on the AMD instances wasn’t fun. They were substantially slower than anything I’ve used in recent memory, including an HP ProLiant Micro Server N40L.
The answer, after some noodling, was simply to move those services back onto my own hardware. Yes, extreme self-hosting. I already host some personal stuff at home using a Celeron G3900 server living inside a Fractal Design Node 804 case. Here too, the CPU load isn’t too high since it mostly maintains the spinning rust the case houses. But the trick is that my ISP most likely filters or blocks the traffic on ports 80 and 443 to discourage hosting. And these publicly facing services are web ones, so those are key.
Enter Cloudflare’s Tunnels which have become all the rage in some subreddits. No publicly routable IP address is necessary and instead I run a daemon on the system to create an outbound connection to Cloudflare’s infrastructure. Here’s a picture from Cloudflare themselves illustrating this configuration.
Cloudflare’s helpful diagram.
Yes, that means I’m i) executing someone else’s code on my system at the system level to ii) explicitly introduce a middleman between myself and my traffic. But this isn’t different from trusting any other software on my server (e.g., Docker in general, my ad-filtering DNS server), and for my public-facing infrastructure, I’ve already employed Cloudflare to filter out malicious traffic. So employing Cloudflare’s tunnel only slightly changes the security profile while giving the benefit of self-hosting these small, and apparently, unwanted workloads.
So over the course of a few days, I look into what’s necessary to run Cloudflare’s daemon on my hardware. I added their Docker image to my compose file, made some tweaks, and fired things up. After fiddling with a few settings, everything started right up like it did before. Caddy’s configuration didn’t change at all because of the migration. Indeed, none of the containers needed tweaking aside from shifting everything from the default network to a specific network associated with the Cloudflare daemon. And by employing Docker, at least I have some certainty that should someone find a way into the system (e.g., through a supply chain attack on Cloudflare for example), access is restricted by the Docker engine. And I’m still running other Docker containers separately to host services that aren’t public facing.
So instead of relying on a cloud provider to host the infrastructure, I’ve brought it in house. Decentralization is the theme for 2023 it seems, and self-hosting is the way forward.
A saying so old that it has its own website and is heavily mentioned in forums filled similarly interested individuals. And it has even been meme’d.
(not my work, just one of many immediate results)
So last month, as I was processing more pictures from the latest play date, I noticed several pictures in the main archive weren’t loading. Strange, but I didn’t pay any mind. See, my current Lightroom setup is a repurposed Thinkpad T470 from several years ago that just happens to have a ton of RAM. But the little i5 inside? It is a poor little 6200U. Skylake, but low voltage and from 2015. And the thermals on the Lenovo aren’t made for Lightroom—the poor little thing overheats if I run it with the lid closed. This all happened because my desktop started glitching badly and refusing to boot. It is either the memory or the video card and I don’t have the spare hardware to isolate it. And my 2014 vintage Mac Mini is similarly limited processing power wise, and even more so RAM wise.
The crazy part is of all these old computers, it was my desktop with a separate GPU that couldn’t power the Dell U3818DW at its native resolution, necessitating a GPU upgrade a few years ago. But to be honest, you can tell the laptop is hurting (mostly through graphics artifacts/glitching). The Mac Mini, however, somehow pumps out the video for both the 38″ Dell and my 24″ Asus, enabling a gigantic dual monitor setup that I hope to continue once I upgrade.
But back to the RAID issue.
As seen a few years ago, I migrated to a BTRFS setup. One big motivation was to avoid bitrot. XFS certainly didn’t have that capability, and over the years I have noticed a few glitches in my data. So even though I didn’t invest in ECC, my thinking was at least the filesystem should know about such things, and if possible, fix them.
So as the poor little Lenovo tries to render 1:1 previews for the latest picture load, I notice a few of the older pictures weren’t loading. I move most of the pictures over to the NAS to minimize the pictures stored on the laptop alone, so sometimes it is just the WiFi. But not this time.
Indeed, later on, I look in the NAS logs and I see crc errors. Lots of them. Some digging around in the btrfs device stats showed one of the drives in my RAID5 array giving me a ton of corruption errors. So I immediately run a scrub on the array to try and fix the data. Since this btrfs volume is setup with RAID5 data and RAID1C3 metadata, I wasn’t too concerned about the array crashing. But I wanted to get to the bottom of this.
Eventually, the scrub hits the bad file and fixes up the file. But as the scrub keeps running, it finds more and more errors. Which is a bit unsettling and now making me wonder how stable the array really is. A few more scrubs later and I notice the errors are transient as well. Sometimes the drive spits out tons of errors and sometimes it is humming along just fine. So because it is configured with RAID5 data, I ran a scrub on each of the individual devices. In general, RAID5 and btrfs comes with significant warnings because of a variety of issues documented on the development page. Scrubbing the individual devices in the RAID5 is one of the guidelines from the developers. And who am I to question them. But if you too are venturing down the btrfs RAID5 path, read through that email from the developers because it sets the ground rules, and foundational expectations.
Hrmm, more errors in the array, but all concentrated on one drive. Well, that’s good I guess. But then, weirdly, another one of the arrays in the NAS starts glitching. That’s odd too. By now, I’m having flashbacks to my desktop system glitching out with weird rebooting loops. So I dig the server out from its home at the bottom of the shelf and give it a good vacuum before I go in and take a look to see what’s going on. Since the errors are concentrated on one drive, I pull the problematic drive out and immediately see the problem—a loose piece of tape on the SATA connector.
Huh? Tape? What?
You see, the 10TB drives I rebuilt the array with were shucked. What’s that? Well, when you’re a hard drive manufacturer, you want to charge extra for those dorks that are trying to build a home NAS. You slap some extra warranty on the drive, and then you juice the price. But you still have to sell to the unwashed masses. So you take some drives and you throw them in a plastic case and sell those. But the unwashed masses have highly elastic demand, so you gotta cut your price to meet this month’s sales targets. So you shuck some external drives, take their internals, and use them in your NAS. But WD, particularly for their white label drives, decided to implement something slightly different.
When I pulled off the SATA controller, the tape was wrinkled and not really blocking the third SATA pin properly. After a few minutes, I reapplied the tape and slapped the drive back into the system. And while I was in there, I checked all the cables on all the drives. The SFF-8087 to SATA cables I got when I built the server were barely long enough, so the tension is a little tight.
After closing up and booting back up, all the drives came online which was a good sign. And afterwards, I ran a series of scrubs on all arrays. No more errors! And Lightroom is happily pulling up pictures from the NAS. Just slowly, very slowly.
It’s been about a month now since that crop of errors and things have worked smoothly since. Knock on wood. Glad these old drives are still working well and have plenty of space for more play date pictures!
Too bad about the rain this weekend. It would’ve been nice to go down to the DC Funk Parade and let the little one revel in the music and sights. But another weekend maybe.
The computing field is always in need of new cliches.
Alan Perlis
My self-hosting journey is an odd one. Once upon the time in college, my computer was simultaneously my media center, my workstation, and a server. Self-hosting was how you did it. Back then, I also hosted websites on Pair.com. After my stint as an IT guy, I lost interest in that tinkering so my skills withered for the better part of a decade.
So when I first got this old fashioned blog back online I went with an old reliable host: Dreamhost. They’ve served me well before and made things easy. Simple shared hosting. Dreamhost gives more access than many other shared hosts (e.g., SSH access), but you didn’t have full control of the system. They were far better than some of the hosts I used earlier in this century. Remember iPaska? Yea those guys were terrible.
But that was just how things were done in the early 2000s. They were simpler times where you didn’t have full access. Instead, everyone had their own control panel of some sort, and they made it easy to install common applications like WordPress. Dreamhost was a competent shop and provided a reliable service (unlike iPaska). And although people complained about the speed of the service, I never had a problem. I also didn’t have that much traffic but that’s another issue for another day.
As with all things, times changed. Dreamhost is still here, providing the shared hosting experience. They sell a good service and continue to run it competently. But the big boys (Google, Microsoft, Amazon) now sell you cloud services and also offer free levels for people to use. Sure it isn’t a 12-core processor with gobs of memory, but it is more than enough to host a few web apps. And it isn’t like I get that kind of traffic anyway. All I need is a decently fast system that I can SSH into and have root on. What I would’ve given for this level of access back when I was younger.
The big boys are appealing, but there is a dark horse in the cloud race: Oracle Cloud Free Tier. They give you two AMD compute VMs, and you can get up to Arm Ampere instances, all for free forever. The AMD compute VMs are easier to get, depending on which region you’re interested in your instance living. And they let you use standard Linux distributions including Ubuntu. They’re not a big name in the space, but boy it is hard to argue with two free AMD VMs. It isn’t the fanciest (1 GB memory each) but it is more than enough to handle a few web apps.
With a free system like that online, I’ve spent a little time here and there over the past month to get everything setup. Lately I’ve started using Docker more at home to manage some of the applications hosted on the server. That has helped simplify deployment even though it isn’t as efficient as installing everything on bare metal. But hey the AMD VMs have the resources. WordPress has its official Docker image, so I used one of those variants as my base. The good people at linuxserver.io provided the database, and I tried out a reverse proxy of a more recent vintage with Caddy. The end result is a self-hosted WordPress instance that has a valid SSL certificate that autorenews. Not bad for the price of free, with a little bit of tinkering time over the past few weekends.
Now I’ve expanded on my self-hosted journey. I’ve created a Wallabag instance for my read-it-later service. I don’t commute anymore thanks to remote life, so I don’t have the same amount of idle time every day to read through the day’s articles on Pocket. But I want to guarantee that my articles are always there for me. Even though Pocket is owned by Mozilla now, I wanted to self-host if possible. And Oracle’s AMD VMs are more than enough to meet the task.
Now, not only am I back to self-hosting this blog and some useful tools, I’m back to tinkering. It feels nice after so many years away.
Almost the entirety of 2021 almost passed without me throwing another post up. There went my hopes for consist posting. What can I say except
But at the same time, my life has changed greatly.
My daughter is a main focus of my life now. I guess that means I really am a parent. My daily goal is making sure she’s happy, well fed, and learning a little more each and every day. Night is for catching up on work and errands. She’s used to us being around all the time now so on one hand, the pandemic was amazing for our family. But on the other hand, I do worry if she has enough social interaction for a child of her age. She has some interactions with kids in the playground, and we’ve had success with her in a small play group. But it isn’t the same as before the pandemic certainly. And we’re still being very cautious about introducing her to daycare until we have more certainty about the availability of vaccines for children of her age.
My photography has switched over to documenting my daughter’s life largely. She’s a very difficult subject but over time has become accustomed to the cameras in her life. She even sometimes reviews the pictures on the screen after a shot and laughs! I just hope when she grows up she can appreciate some of these pictures. Not that she’ll remember anything of course.
I’m still with the job I started at the beginning of the year. Work life on a Mac is not bad at all. Using Dropbox as our main document archive is usually quite good, but there are times I miss the exclusive locking capability of the various document management systems I’ve used over the years. That way the one true copy is always available. With Dropbox, you have to hope your coworkers don’t overwrite your work. Or that your coworkers don’t save a new copy right over yours, overwriting your changes. One thing I didn’t expect from the pandemic is meeting coworkers for the first time and being surprised by their height. Because no one sees how tall everyone is when we’re all just boxes on the screen.
Life has otherwise slowly started to return to some form of normalcy. Last weekend there was a raucous party down the hall that reminded me of house parties of days long past. This time, I was the one just hoping the next crowd entering/leaving wouldn’t slam the door too loud to wake up my daughter. What a change from just a few years ago.
Soon I’ll take my small little family down for the holidays. For the first time, there’ll be someone new who will be amazed. I hope we can give her a nice holiday tradition to grow up with.
Life is an adventure to say the least. Let’s see if I can post more consistently for the remainder of the year, and into the next.
