Skip to content

Instantly share code, notes, and snippets.

@avoidik

avoidik/README.md

Last active May 26, 2025 18:22
Show Gist options
  • Save avoidik/7aac98b2fbc25524b0e06f285f37b5bd to your computer and use it in GitHub Desktop.
Save avoidik/7aac98b2fbc25524b0e06f285f37b5bd to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. avoidik revised this gist May 25, 2025. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -27,7 +27,7 @@
    ```
    3. renew kubernetes certificates (control plane nodes):
    ```terminal
    # kubeadm certs check-expiration
    # kubeadm certs check-expiration --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # rm -f /var/lib/kubelet/pki/kubelet.crt
    # rm -f /var/lib/kubelet/pki/kubelet.key
    # rm -f /var/lib/kubelet/pki/kubelet-client*
    @@ -36,7 +36,7 @@
    # rm -f /etc/kubernetes/kubelet.conf
    # rm -f /etc/kubernetes/scheduler.conf
    # kubeadm init phase kubelet-finalize all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm certs renew all
    # kubeadm certs renew all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > /etc/kubernetes/kubelet.conf
    # systemctl start kubelet.service
  2. avoidik revised this gist May 25, 2025. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -31,7 +31,10 @@
    # rm -f /var/lib/kubelet/pki/kubelet.crt
    # rm -f /var/lib/kubelet/pki/kubelet.key
    # rm -f /var/lib/kubelet/pki/kubelet-client*
    # rm -f /etc/kubernetes/admin.conf
    # rm -f /etc/kubernetes/controller-manager.conf
    # rm -f /etc/kubernetes/kubelet.conf
    # rm -f /etc/kubernetes/scheduler.conf
    # kubeadm init phase kubelet-finalize all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm certs renew all
    # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
  3. avoidik revised this gist May 25, 2025. 1 changed file with 6 additions and 2 deletions.
    8 changes: 6 additions & 2 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -7,6 +7,8 @@

    # Recovery

    > Note: Do not forget to make backup before changing any files.
    1. stop all relevant systemd units on all nodes:
    ```terminal
    # systemctl stop etcd.service
    @@ -26,8 +28,10 @@
    3. renew kubernetes certificates (control plane nodes):
    ```terminal
    # kubeadm certs check-expiration
    # rm -rf /var/lib/kubelet/pki/kubelet.crt
    # rm -rf /var/lib/kubelet/pki/kubelet.key
    # rm -f /var/lib/kubelet/pki/kubelet.crt
    # rm -f /var/lib/kubelet/pki/kubelet.key
    # rm -f /var/lib/kubelet/pki/kubelet-client*
    # rm -f /etc/kubernetes/kubelet.conf
    # kubeadm init phase kubelet-finalize all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm certs renew all
    # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
  4. avoidik revised this gist May 25, 2025. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -28,7 +28,7 @@
    # kubeadm certs check-expiration
    # rm -rf /var/lib/kubelet/pki/kubelet.crt
    # rm -rf /var/lib/kubelet/pki/kubelet.key
    # kubeadm init phase kubelet-finalize all
    # kubeadm init phase kubelet-finalize all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm certs renew all
    # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > /etc/kubernetes/kubelet.conf
  5. avoidik revised this gist May 25, 2025. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -26,6 +26,8 @@
    3. renew kubernetes certificates (control plane nodes):
    ```terminal
    # kubeadm certs check-expiration
    # rm -rf /var/lib/kubelet/pki/kubelet.crt
    # rm -rf /var/lib/kubelet/pki/kubelet.key
    # kubeadm init phase kubelet-finalize all
    # kubeadm certs renew all
    # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
  6. avoidik revised this gist May 25, 2025. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -26,6 +26,7 @@
    3. renew kubernetes certificates (control plane nodes):
    ```terminal
    # kubeadm certs check-expiration
    # kubeadm init phase kubelet-finalize all
    # kubeadm certs renew all
    # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > /etc/kubernetes/kubelet.conf
  7. avoidik revised this gist May 25, 2025. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -35,7 +35,7 @@
    ```
    4. renew kubernetes certificates (worker nodes):
    ```terminal
    # rm -f /etc/kubernetes/kubelet.conf
    # kubeadm reset -f --cri-socket unix:///var/run/cri-dockerd.sock
    # kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock --ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt
    # systemctl start kubelet.service
    # systemctl status kubelet.service
  8. avoidik revised this gist May 25, 2025. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -35,7 +35,8 @@
    ```
    4. renew kubernetes certificates (worker nodes):
    ```terminal
    # kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock
    # rm -f /etc/kubernetes/kubelet.conf
    # kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock --ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt
    # systemctl start kubelet.service
    # systemctl status kubelet.service
    ```
  9. avoidik revised this gist May 25, 2025. 1 changed file with 32 additions and 24 deletions.
    56 changes: 32 additions & 24 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -8,34 +8,42 @@
    # Recovery

    1. stop all relevant systemd units on all nodes:
    ```
    systemctl stop etcd.service
    systemctl stop kubelet.service
    ```terminal
    # systemctl stop etcd.service
    # systemctl stop kubelet.service
    ```
    2. renew etcd certificates on all nodes:
    ```
    for crt in /etc/etcd/pki/*.crt ; do echo "$crt:" ; openssl x509 -noout -dates -in $crt ; echo ; done
    find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -exec echo {} \;
    find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -delete
    etcdadm join phase certificates https://lan.ip.goes.here:2379
    systemctl start etcd.service
    systemctl status etcd.service
    source /etc/etcd/etcdctl.env
    etcdctl member list
    ```terminal
    # for crt in /etc/etcd/pki/*.crt ; do echo "$crt:" ; openssl x509 -noout -dates -in $crt ; echo ; done
    # find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -exec echo {} \;
    # find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -delete
    # etcdadm join phase certificates https://lan.ip.goes.here:2379
    # systemctl start etcd.service
    # systemctl status etcd.service
    # source /etc/etcd/etcdctl.env
    # etcdctl member list
    ```
    3. renew kubernetes certificates (control plane nodes):
    ```
    kubeadm certs check-expiration
    kubeadm certs renew all
    kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > kubelet.conf
    systemctl start kubelet.service
    systemctl status kubelet.service
    kubeadm token create --print-join-command
    ```terminal
    # kubeadm certs check-expiration
    # kubeadm certs renew all
    # kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    # kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > /etc/kubernetes/kubelet.conf
    # systemctl start kubelet.service
    # systemctl status kubelet.service
    # kubeadm token create --print-join-command
    ```
    4. renew kubernetes certificates (worker nodes):
    ```
    kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock
    systemctl start kubelet.service
    systemctl status kubelet.service
    ```terminal
    # kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock
    # systemctl start kubelet.service
    # systemctl status kubelet.service
    ```
    5. check status:
    ```terminal
    $ kubectl get nodes
    NAME STATUS ROLES AGE VERSION
    inst-biuce-vmp-prv Ready <none> 2y88d v1.24.10
    inst-elb5m-vmp-pub Ready control-plane 2y95d v1.24.10
    inst-tjvsi-vmp-pub Ready control-plane 2y95d v1.24.10
    ```
  10. avoidik created this gist May 25, 2025.
    41 changes: 41 additions & 0 deletions README.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,41 @@
    # Scenario

    1. etcdadm managed etcd cluster
    2. kubeadm managed kubernetes cluster
    3. all etcd certificates have been expired (self-signed CA is okay)
    4. all kubernetes certificates have been expired

    # Recovery

    1. stop all relevant systemd units on all nodes:
    ```
    systemctl stop etcd.service
    systemctl stop kubelet.service
    ```
    2. renew etcd certificates on all nodes:
    ```
    for crt in /etc/etcd/pki/*.crt ; do echo "$crt:" ; openssl x509 -noout -dates -in $crt ; echo ; done
    find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -exec echo {} \;
    find /etc/etcd/pki -type f ! -iname 'ca.crt' ! -iname 'ca.key' -delete
    etcdadm join phase certificates https://lan.ip.goes.here:2379
    systemctl start etcd.service
    systemctl status etcd.service
    source /etc/etcd/etcdctl.env
    etcdctl member list
    ```
    3. renew kubernetes certificates (control plane nodes):
    ```
    kubeadm certs check-expiration
    kubeadm certs renew all
    kubeadm init phase kubeconfig all --config /home/ubuntu/projects/kubeadmcfg-external.yaml
    kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config /home/ubuntu/projects/kubeadmcfg-external.yaml > kubelet.conf
    systemctl start kubelet.service
    systemctl status kubelet.service
    kubeadm token create --print-join-command
    ```
    4. renew kubernetes certificates (worker nodes):
    ```
    kubeadm join api.server.hostname:6443 --token join.token.goes.here --discovery-token-ca-cert-hash sha256:certificate.hash.goes.here --cri-socket unix:///var/run/cri-dockerd.sock
    systemctl start kubelet.service
    systemctl status kubelet.service
    ```