Changing Cloud Providers

The lift and shift involved even on a small scale, changing clouds

I’m writing this in the present, but this change happened quite a while ago, maybe around November 2020. I shifted the primary production workloads I maintain, away from Amazon Web Services and towards DigitalOcean. I had multiple reasons for this and I still have experimental items in place and the foundational production configuration if I ever change my mind again. But lets get into it!

The first reason for changing, was due to a some what surprising outage I experienced on my EKS cluster. I did not spend much time to devote to identifying the root cause since it was mid week. But it looked as if my worker plane just completely disconnected and failed to reconnect to the control plane of the cluster. In my head, what the control plane should of done in this case was see that there was no worker capacity and just replaced the machines. Note that I didn’t say ‘auto scale the cluster’. This wasn’t a we’re reaching the limits on our capacity, this was a “oh we’ve no compute at all” situation. Maybe my expectations were set at the wrong bar, but to have three machines, across three availability zones, just disconnect and not get replaced, left a sour taste in my mouth.

The next reason, appears to relate to a technical limitation of my size of compute nodes. I had three, T3.Medium instances spread across the three availability zones of the Ireland region in AWS. I felt that they were an adequate size, 2 vCPU and 4GB of RAM. Some may argue to not run burstable instances in production, but they also have the benefit of being cheap enough. What I was finding, was that I was sooner running into limitations of the onboard NICs of the instance than any resource limits. If I recall correctly, the instance size only supports up to 32 IP addresses total. So when you factor in however many Pods run in kube-system, what you’re left with may be in the mid to low twenties of usable IP addresses. Based on my resource requests and limits, I would sooner reach the IP limit rather than any resource constraint. This may have changed, but I think I could only go down to a T3.Small if I wanted to and even then, the IP address limit was lower and my overall resource usage was higher than a T3.Small, but lower than the max of a T3.Medium. Either way, I felt it wasn’t the most efficient use of my resources or money.

Finally, the cost of this hobby was what came back to it all too. Yes, I reduced my cost with some Reserved Instances (haven’t hopped on the Savings Plan train yet) but there is still the cost of the EKS control plane, which I believe works out at around 75 USD per month before tax. Sure, it’s probably running three control nodes across three AZs which is lovely. But it was always the bulk of my bill cost and after a while, with everything outlined above, I felt it was no longer worthwhile. The aim became to reduce the monthly cloud bill, while still retaining the features that I quite enjoy about having my own Kubernetes cluster. So, enter DigitalOcean!

I used DO in the past, what brought me back was a new feature to me, of VPC networking. Having this level of control now on how the network is structured, made it really nice to come back to them while not taking on the complexity of building a new network on say, GCP. DOK8s has gotten some love too in the mean time. While I’m sure that it’s probably only one control plane VM running in the background, they do have automated upgrades unlike EKS. My cluster just this weekend did a patch on 1.19 and next weekend, it’ll upgrade to 1.20. Finally there’s the simplicity in the costs of compute on DO, they’re cheaper than a T3.Medium instance for their equivalent size Droplet for one. When I factor in the total cost of my Kubernetes footprint on DO, it’s almost equivalent to the cost of the control plane alone on EKS. Not to mention I can have a full, 110 Pods on my Nodes, so I’d sooner be hitting a resource ceiling than a networking one.

The actual lifting and shifting from AWS to DO was pretty seamless. Like with most managed Kubernetes offerings, they’re CNCF certified. So provided you’re not using cloud provider specific features, a kubectl apply of your yaml on the new cluster should get you most of the way there. I also took the opportunity to initially try out DigitalOcean Container Registry, but to do everything I wanted from my container registry, it’d be an extra 20 USD per month which seems ridiculous to me, even in a world where Dockerhub are changing their strategy on image pulling and image storage. For now, I went with the Github Container Registry, which I believe will eventually supersede their Docker package implementation that I disliked a lot. I think a lot of what I do will fall within my Github Pro plan limits, but I think any overusages will be very reasonable, if they’re due. I use Github Actions already extensively, so it’s one ecosystem that I don’t mind growing my usage in.

Sufficed to say, I’m pretty pleased! DO is cheaper than AWS for me, is probably also cheaper than a GCP config and I’m still getting plenty of the feature set. I really feel that AWS could probably eat into DOs market share with either a free tier EKS control plane offer, or a single AZ option for the control plane for cheap or for free. But, I figure something like that would probably fall in their Lightsail brand rather than AWS itself. Oh, did I mention all the free bandwidth I get too? Highly recommend DigitalOcean for those looking to start out either with Kubernetes, or just cloud in general.