Managing Lets Encrypt certificates has been one of the hardest things to do in Kubernetes. To be fair, it wasn't easy before Kubernetes either. For a while I was using Traefik, but then I ran into some issues related to scalability. Then I moved to nginx-ingress + cert-manager and things were good for a while.
A few weeks ago I received an automated email from Lets Encrypt informing me that one of my certs was going to expire soon. This was strange since cert-manager is supposed to automatically renew my certs. It took some digging, but it looks like this is an issue with Kubernetes DNS. You can read a bit more about this issue here.
I couldn't find a solution to my problem, and it wasn't clear when this bug would get fixed. So I finally decided to learn how to create certs manually. I enjoyed this exercise, and it feels good to know that if the magic automation fails, I know how to work without it.
I've created a repository with the code and instructions to do this: https://github.com/nabsul/k8s-letsencrypt