etcd fail, backup fail, Claude win?
Back in September, I performed some routine updates of my infrastructure. I have this nice, Saturday morning routine where I get my coffee, play some chill music and go about running updates across all my hosts, any pending software updates on tools like Home Assistant and also update any client workloads. It’s quite nice, I could automate a lot of this with say, unattended-upgrades. But, I like the routine and I also like being around for an upgrade in case it goes south. With my Kubernetes cluster, there’s obviously the host operating system itself, then the cluster software. In my case, K3s.
Return on investment while moving to Grants in Tailscale
One of my favourite blog posts and generally, favourite thing I did, was use test driven development with my Tailscale ACL. I’m pretty much a noob when it comes to software engineering, even though it was the basis of my degree, but I read a lot about it and have great colleagues who have decades of experience in it. One thing that always came up, was test driven development. If you’re not familiar, the very very basic idea is that when writing code, you write your test first. That test, will fail and you subsequently write code that will have the test pass. You would then refactor this code as time goes on, following the pattern of red, green and refactor. Or rephrased slightly; failing test, passing test, refactor.
life always has other plans
When I wrote last, I talked about how I had some new plans after my self hosting infrastructure woes were resolved. That was back in March, so only about nine months later am I circling back to report on how, none of that really happened. Not that I need to justify to anyone why plans never quite started, but back in around April I started the process of buying my first home. That naturally, consumed quite a lot of my time and brain bandwidth. Thankfully this has culminated in now being sale agreed on a new build apartment, which naturally I’m over the moon about! I’ll be writing another sort of standalone blog on that entire process, as I find that there is not a whole lot of information on new build apartment buying in Ireland. So, maybe what I’ve experienced could be of help to someone. The process is far from being done, at the time of writing I’m still waiting on a few things before everything is formalised. But hopefully that gets sorted soon so I can start to feel a lot more excited about the process.
A letter to myself on why I do this
This is a bit of a different blog post and also feels weird to write. Almost like I am trying to justify this hobby to myself, because it is a hobby, right? Something you gain enjoyment from? My running theory is that I am actually a masochist because who would continue to persist with something after it causing so much pain? This does allude to some future blog posts that I hope to get out soon, which continue my previous writing spells theme of “things break, I lose data, I get sad, but its fine”.
Using methods from Slow Productivity to better plan work
Now that I feel I can take my gaze away from my infrastructure, what do I want to work on now? I am going to do a follow up post, perhaps a standalone piece on my website where I really deep dive into the why’s of my self hosting journey. I touch on it at points throughout my posts and I feel like having a condensed piece that goes over everything, will be good for me to just get it all written down and to have a resource to refer to when I get asked the question “well why are you a lunatic?” haha
Trying my best to tell myself it is indeed solved
The dust has very much settled at this point. I’ve sort of sat and stared for a while as I can’t quite believe the stability I’ve experienced after so much instability. It’s been about two or so weeks since I took action once again on the hardware front, and it’s been all quiet on this front. I mentioned previously acquiring new hardware, but as it turns out that was not the issue. But it did lead to discovery of what seemed to be either the root cause or at least partner in crime of my woe.
Probably not, but I’m just letting the dust settle
While I took a break from my current endeavours for a while, I jumped back in quite quickly. I’ve since taken a more deliberate break. This weekend gone, besides some routine patching I did literally nothing self hosting related and it was honestly glorious. But it’s time to go over what steps I took after the realisation of having to lose an entire Kubernetes cluster, yet again, and why I feel / hope that the end is approaching in this story arc.
“a victory that is not worth winning because the winner has lost so much in winning it”
It must be said that emotions on a self hosting follow a sine wave pattern of “its so over” for dips, to “we are so back” for highs. We’ve had several problems that I was able to solve, (the high) and then feeling like everything was done, encountered new ones that have derailed things entirely (the low). The title of this blog post is I think the perfect way of describing how my weekend went with trying to fix my local cluster once and for all.
Things just mostly started working again?
As I wrote about yesterday, the plan was to get home and start troubleshooting the issue with the one NUC. Absolutely horrible drive home in terms of weather, though certainly better than ice and cold. I was expecting to encounter some kind of issue that would be new to me and therefore would need an evening of research. At a bare minimum I was expecting some form of operating system corruption perhaps, hopefully not hardware failures beyond say storage and memory. Once I got home it was a case of unplugging things so that I could plug them into the NUC, a further reminder of needing to invest in a KVM.
