How things have been

life always has other plans

When I wrote last, I talked about how I had some new plans after my self hosting infrastructure woes were resolved. That was back in March, so only about nine months later am I circling back to report on how, none of that really happened. Not that I need to justify to anyone why plans never quite started, but back in around April I started the process of buying my first home. That naturally, consumed quite a lot of my time and brain bandwidth. Thankfully this has culminated in now being sale agreed on a new build apartment, which naturally I’m over the moon about! I’ll be writing another sort of standalone blog on that entire process, as I find that there is not a whole lot of information on new build apartment buying in Ireland. So, maybe what I’ve experienced could be of help to someone. The process is far from being done, at the time of writing I’m still waiting on a few things before everything is formalised. But hopefully that gets sorted soon so I can start to feel a lot more excited about the process.

On the self hosting world, honestly I’m struggling to think of what I did, if anything, in the intervening months. Back in May too, my favourite podcast and only one I listened to, Self-Hosted, decided to wrap up. Totally understood the reasoning behind it, but yeah it was a sad moment for me. Chris and The Badger were the voices that narrated all my solo walking trips, so while I won’t have new episodes, I fortunately kept up the walking and just re-listen to old episodes currently.

While that was just a say, small sabotage in my motivations, the big one was in September when my local kubernetes cluster for lack of a better word, exploded. Note there was no real explosions. While I will document this in a subsequent post, in short from what I could tell, I suffered a bit of a catastrophic etcd failure. I did have five(!) nodes acting as my control plane, all Raspberry Pi 4s with SATA SSDs on the USB bus. Based on my back and forth with Claude (also more on that later) it seems that I started to experience network saturation as the etcd nodes all tried to resync. Which, slowly built up a backlog of messages on multiple nodes, steadily building up system load until they crashed. On top of this, my attempt at restoring the cluster completely failed. This was a bit of a gut punch as it seemed the cluster was all but lost and I sort of just wiped everything and ignored all things self hosting for a bit.

I didn’t last long, I bought new hardware, a singular NUC to replace the Raspberry Pi’s and their screaming fans. Which was a problem when I relocated my network rack into my bedroom again along with my desk. So I’ve greatly simplified the cluster and very slowly over the last two or so months, I’ve been extremely methodical in testing backups. I know now I can for example fully recover my Synology server in two different paths. I’m also actively testing my Longhorn volumes and their snapshots, so I know that I can have in cluster recovery. With those volume backups now also being sent to the aforementioned Synology server.

I sort of “stealth published” a blog post a week or two ago about why I self host. I did that as I want that to be one of the first posts that I narrate in my upcoming companion podcast to these blog posts. And because its over 4,000 words long. But, this is me now signalling that I’m getting back into the groove of things both with my blog and hopefully my life in general haha. And the path is clear. Self hosting has never been more important, it might be really hard at times, but the journey has to continue. Hopefully, you’ll continue to read along too!