# Scenario

1. etcdadm managed etcd cluster
2. kubeadm managed kubernetes cluster
3. all etcd certificates have been expired (self-signed CA is okay)
4. all kubernetes certificates have been expired

# Recovery

> Note: Do not forget to make backup before changing any files.

1. stop all relevant systemd units on all nodes:
   ```terminal
   # systemctl stop etcd.service
   # systemctl stop kubelet.service
   ```
2. renew etcd certificates on all nodes:
   ```terminal
   # for crt in /etc/etcd/pki/*.crt ; do echo "$crt:" ; openssl x509 -noout -dates -in $crt ; echo ; done
   # find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -exec echo {} \;
   # find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -delete
   # etcdadm join phase certificates https://lan.ip.goes.here:2379
   # systemctl start etcd.service
   # systemctl status etcd.service
   # source /etc/etcd/etcdctl.env
   # etcdctl member list
   ```
3. renew kubernetes certificates (control plane nodes):
   ```terminal
   # kubeadm certs check-expiration --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # rm -f /var/lib/kubelet/pki/kubelet.crt
   # rm -f /var/lib/kubelet/pki/kubelet.key
   # rm -f /var/lib/kubelet/pki/kubelet-client*
   # rm -f /etc/kubernetes/admin.conf
   # rm -f /etc/kubernetes/controller-manager.conf
   # rm -f /etc/kubernetes/kubelet.conf
   # rm -f /etc/kubernetes/scheduler.conf
   # kubeadm init phase kubelet-finalize all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # kubeadm certs renew all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
   # kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > /etc/kubernetes/kubelet.conf
   # systemctl start kubelet.service
   # systemctl status kubelet.service
   # kubeadm token create --print-join-command
   ```
4. renew kubernetes certificates (worker nodes):
   ```terminal
   # kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock
   # kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock --ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt
   # systemctl start kubelet.service
   # systemctl status kubelet.service
   ```
5. check status:
   ```terminal
   $ kubectl get nodes
   NAME                 STATUS   ROLES           AGE     VERSION
   inst-biuce-vmp-prv   Ready    <none>          2y88d   v1.24.10
   inst-elb5m-vmp-pub   Ready    control-plane   2y95d   v1.24.10
   inst-tjvsi-vmp-pub   Ready    control-plane   2y95d   v1.24.10
   ```