Skip to content

Instantly share code, notes, and snippets.

@leahnp

leahnp/Kuberneteslnp

Last active September 8, 2022 21:21
Show Gist options
  • Save leahnp/521e9a7f76bd9d3062edace700d09d5d to your computer and use it in GitHub Desktop.
Save leahnp/521e9a7f76bd9d3062edace700d09d5d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Kuberneteslnp
https://github.com/kelseyhightower/kubernetes-the-hard-way/tree/master/docs
systemd: https://www.digitalocean.com/community/tutorials/systemd-essentials-working-with-services-units-and-the-journal
https://www.linuxtrainingacademy.com/systemd-cheat-sheet/
kubectl: https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
worker nodes:
- kube proxy: $ ps ax | grep kubeproxy
- kubelet: systemctl list-units grep hyperkube | kubelet.service
- kubelet: sudo systemctl status kubelet
Create the kubelet systemd unit: https://coreos.com/blog/introducing-the-kubelet-in-coreos.html
sudo systemctl daemon-reload sudo systemctl start kubelet
sudo systemctl enable kubelet
master nodes:
- scheduler
- controller
- API server: systemctl/journalctl | grep api/apiserver
DNS: https://gist.github.com/leahnp/1a31816909b8ef2d6882fec1a7824a1c
start pod: kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
Nslookup kubernetes
namespace services: nslookup kubernetes.kube-system
Lay of the land:
kubectl get all --all-namespaces
systemctl list-units
Debug:
ps -aux
$ journalctl -u krakenevents-ssl.service | grep -Ei 'failed|error'
kubectl logs/describe
kubectl get events
https://www.computerhope.com/unix/udf.htm
resource limits: $ kubectl describe pod frontend | grep -A 3 Events
unit files: $ cd /etc/systemd/system
Troubleshooting (is node there, can I ssh to it, are things running, restart, is node healthy (system resources DF (out of disc space (is /dev/disk1 at 100%), memory, cpu) kubectl describe node, reboot
Testing:
https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md
./test 2>&1 | tee test.out
Writing things into /opt:
stuff > temp_file.txt/yaml
sudo mv temp_file.txt /opt/folder/foobar.txt
completions:
```apiVersion: batch/v1
kind: Job
metadata:
name: printer
spec:
completions: 100
parallelism: 4
template:
metadata:
name: printer
spec:
containers:
- name: printer
image: busybox
command: ["sh", "-c", "echo 'hello'"]
restartPolicy: Never```
get pods by label:
`kubectl get pod -l key=value`
get pvs by storage capacity
`kubectl get pv --sort-by=.spec.capacity.storage`
daemonset:
```aapiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: frontend
spec:
template:
metadata:
labels:
app: frontend-webserver
spec:
# nodeSelector:
# app: frontend-node
containers:
- name: webserver
image: nginx
ports:
- containerPort: 80```
init container:
```apiVersion: v1
kind: Pod
metadata:
name: init-demo
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
# These containers are run during pod initialization
initContainers:
- name: install
image: busybox
command:
- wget
- "-O"
- "/work-dir/index.html"
- http://kubernetes.io
volumeMounts:
- name: workdir
mountPath: "/work-dir"
dnsPolicy: Default
volumes:
- name: workdir
emptyDir: {}```
rollback record
```apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.11.10
ports:
- containerPort: 80```
create deployment (which also makes rs and pods)
add rolling update info to spec:
```apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
strategy:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
spec:
containers:
- name: nginx
image: nginx:1.11.10
ports:
- containerPort: 80```
$ kubectl apply -f nginx.yaml --record
rollback: (remember to set image)
https://tachingchen.com/blog/Kubernetes-Rolling-Update-with-Deployment/
├─ Deployment: <name>
│ └─ Replica Set: <name>-<rs>
│ └─ Pod: <name>-<rs>-<randomString>
secrets
kubectl create secret generic my-secret --from-literal=key1=supersecret
create pod and mount to secret:
```apiVersion: v1
kind: Pod
metadata:
name: pod-secrets-via-file
namespace: default
spec:
containers:
- image: redis
name: redis
volumeMounts:
- mountPath: /secrets
name: foo
volumes:
- name: foo
secret:
secretName: my-secret
```
exec onto container and see secret in /secrets/file
ingress nginx:
```apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
labels:
k8s-app: nginx-ingress-controller
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: nginx-ingress-controller
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
# hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
# however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
# that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
# like with kubeadm
# hostNetwork: true
terminationGracePeriodSeconds: 60
containers:
- image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.12
name: nginx-ingress-controller
readinessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
ports:
- containerPort: 80
hostPort: 80
- containerPort: 443
hostPort: 443
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend```
pod consuming most CPU
kubectl top pod --namespace=C --selector=A=B
create a service and make DNS nslookup
make a deployment:
`kubectl run nginx2 --image=nginx --replicas=2 --port=80`
back them by a service:
` kubectl expose deployment nginx2`
describe service
`$ kubectl describe svc nginx2`
make a pod, to exec onto it and run nslookup
`kubectl run busybox --image=busybox --rm --restart=OnFailure -ti -- /bin/nslookup nginx2.default`
network policy to deny all ingress access to pods in a certain namespace
create a namespace with a label set:
```{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "secert",
"labels": {
"name": "secret"
}
}
}```
create a network policy:
```kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
name: access-nginx
namespace: secret
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- namespaceSelector:
matchExpressions:
- key: Namespace
operator: NotIn
values:
- secret
ports:
- protocol: TCP
port: 80
```
deploy nginx-worker in secret namespace
```apiVersion: v1
kind: Pod
metadata:
name: nginx-worker
namespace: secret
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80```
test: you want 2 busybox pods… one in default and one in secret namespace. then test the access from both. (if you are trying to show that your network policy is working)
Backup etcd
Ssh onto etcd node
Ps aux | grep etcd ( then grab pid of etcd from there)
sudo lsof -p <PID> | grep -v TCP ( look for path of db - up to the word member )
etcdctl backup --data-dir <data directory> --backup-dir <target in the instructions>
``` etcdctl backup \
--data-dir %data_dir% \
[--wal-dir %wal_dir%] \
--backup-dir %backup_data_dir%
[--backup-wal-dir %backup_wal_dir%]``` https://coreos.com/etcd/docs/latest/v2/admin_guide.html
to get the data_dir: ETCD_DATA_DIR
you need to find the startup exec wrapper script, env var gets set int he wrapper script that starts etcd
juju directory setup
/var/lib/juju $ tree . -L 2
‘pods won’t start’ or ‘deployments have no pods’
Kube controller-manager is not started
It tells you where to go
Systemctl to figure out what is busted
Systemctl enable
Systemctl start whichever_is_busted
etcd location via juju
$ systemctl cat snap.etcd.etcd.service
# /etc/systemd/system/snap.etcd.etcd.service
or in env var in:
/var/lib/juju/etcd/current/ or /var/snap or
# cat /proc/$(pgrep etcd)/environ | xargs -0 -n1 echo | grep ETCD_DATA
ETCD_DATA_DIR=/var/snap/etcd/current/etcd0.etcd
systemd managed service /etc/kubernetes/manifests
ssh to node N
sudo
Edit sudo vim /etc/systemd/system/kubelet.service
Add --pod-manifest-path /etc/kubernetes/manifests
mkdir -p /etc/kubernetes/manifests exists on node N
Add the pod manifest file (this is the yaml file) to /etc/kubernetes/manifests
kind: Pod
metadata:
name: manifest-nginx
spec:
containers:
- name: manifest-nginx
image: nginx
$ systemctl daemon-reload
$ systemctl restart kubelet.service
annotations :
kubectl edit pod <pod_name>
```apiVersion: v1
kind: Pod
metadata:
name: kubernetes-downwardapi-volume-example
labels:
zone: us-est-coast
cluster: test-cluster1
rack: rack-22
annotations:
build: two
builder: john-doe```
redis memcached:
apiVersion: v1
kind: Pod
metadata:
name: manifest-thingy
spec:
containers:
- name: nginx
image: nginx
- name: redis
image: redis
- name: memcached
image: memcached
- name: consul
image: consul
$ kubectl get rc production
$ kubectl scale rc production --replicas=6
$ kubectl get rc production
divert
kubectl drain $node --delete-local-data=true --force &
KUBECTL:
systemctl stuff
##### use systemctl to get list units that are active, add --all for everything
systemctl list-units
##### use systemctl to list all units with specific state, inactive, active, enabled, running, exited
systemctl list-units --all --state=inactive
##### use systemctl to list all unit files
systemctl list-unit-files
##### see log items from the most recent boot
journalctl -b
##### to see only kernal messages, add -b for at the most recent boot
journalctl -k
##### to get the status of a service
systemctl status nginx.service
##### to get the log entries for a service since boot
journalctl -b -u nginx.service
##### check out the service file
systemctl cat nginx.service
##### list dependencies of a service, add --all to expand dependencies recursively
systemctl list-dependencies nginx.service
##### to see low level details of a service settings on the system
systemctl show nginx.service
##### reload or restart service
sudo systemctl reload-or-restart nginx.service
##### to check if the service is active, enabled or failed
systemctl is-active | is-enabled | is-failed nginx.service
systemctl list-unit-files | grep enabled
top stuff
##### set time to 1 second update and order by mem
top -d 1 -o %MEM
##### see processes running
ps aux
kubectl stuff
##### create a namespace
kubectl create namespace [namespace]
##### create a pod in a specific namespace, using a .yaml file
kubectl create -f [yaml file] --namespace [namespace]
##### get nodes on a cluster
kubectl get nodes
see terminated pods
kubectl get pods --show-all
##### get contexts to use
kubectl config get-contexts
##### use a different context for kubectl
kubectl config use-context [context name]
##### get kubernetes status of components
kubectl get componentstatus
##### get container names from a pod
kubectl get pods -n testing [pod name] -o jsonpath={.spec.containers[*].name}
##### describe a pod
kubectl describe pod -n testing [pod name]
##### get logs for a pod
kubectl logs -f -n testing [pod name]
##### get logs for a container in a pod
kubectl logs -f -n testing [pod name] -c [container name]
##### get top information for a pod
kubectl top pod [pod name] -n prod
##### get top information for containers in a pod
kubectl top pod [pod name] --containers -n prod
##### get top information for a node
kubectl top node [node name]
##### create a secret and update it if it does not exist yet
kubectl create secret generic leah-secret --from-file=newusername.txt --dry-run -o yaml | kubectl apply -f -
kubectl create secret generic my-secret --from-literal=key1=supersecret - secret from literal
##### setup simple kubernetes pod with an ubuntu container, expose port 8080 on the node
kubectl run ubuntu-pod --image=gcr.io/google_containers/ubuntu:14.04 --port=8080
kubectl expose deployment ubuntu-pod --type=NodePort
##### watch for changes to a pod
kubectl get --watch pod [pod name] -n testing
##### get events on the cluster
kubectl get events
##### set an image to a new version on a deployment, which will trigger an update
kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
##### checkout the rollout status
kubectl rollout status deployment nginx-deployment
##### attach label to resource
kubectl label pods --all person=leah
##### remove label from resource
kubectl label pods --all person-
##### get resources by label
kubectl get pods -l person=leah
## filter kubernetes response with kubectl
### make file with all results , then filter into new file
`kubectl get nodes > all_nodes.txt`
`cat all_nodes.txt | while read line ; do if [[ $line == *"sobusy"* ]]; then echo $line; fi; done > filtered_nodes.txt`
### filter results by search criteria
`kubectl logs <pod> <search_criteria> > some_file.txt`
ex `kubectl logs busybox | grep fail > logs.txt`
### Get a different version of kubectl (eg: v1.6.2)
```
mkdir ~/bin
cd ~/bin
wget https://dl.k8s.io/v1.6.2/kubernetes.tar.gz
tar xzf kubernetes.tar.gz
./kubernetes/cluster/get-kube-binaries.sh
export PATH=$HOME/bin/kubernetes/cluster/bin:$PATH
```
#### see processes running
`ps aux`
## kubectl stuff
`kubectl cluster-info` - get cluster info
### create
`kubectl create namespace [namespace]` - create a namespace
`kubectl create -f [yaml file] --namespace [nmamespace]` - create a pod in a specific namespace using yaml file
### config
`kubectl config get-contexts` - get contexts
`kubectl config use-context [context name]` - use different context
### get / describe
`kubectl get componentstatus` - status of all components
`kubectl get pods -n testing [pod name] -o jsonpath={.spec.containers[*].name}` - get container names from a pod
`kubectl get --watch pod [pod name] -n testing` - watch for changes to a pod
`kubectl get events` - get events on the cluster (events are namespaced)
`kubectl get pods -l person=kevin` - get resources by label
`kubectl get node <node> -o wide` - get more info from resource
`kubectl describe pod -n testing [pod name]` - describe pod
### logs
`kubectl logs -f -n testing [pod name]` - get logs from a pod
`kubectl logs <pod_name> | grep <search_term>` - filter logs from a pod
`kubectl logs -f -n testing [pod name] -c [container name]` - get logs from a container
#### usage
`kubectl top pods` - get usage info for pods
`kubectl top pod [pod name] --containers -n prod` - get usage info for containers in a pod
`kubectl top node [node name]` - get top info for a node
### secrets
#### create a secret and update it if it does not exist yet
`kubectl create secret generic kevin-secret --from-file=my_secret.txt --dry-run -o yaml | kubectl apply -f -`
`kubectl create secret generic my-secret --from-literal=key1=supersecret` - secret from literal
#### setup simple kubernetes pod with an ubuntu container, expose port 8080 on the node
`kubectl run mynginx --image=nginx --replicas=2 --port=80` - make simple deployment with two replicas
`kubectl run ubuntu-pod --image=gcr.io/google_containers/ubuntu:14.04 --port=8080`
`kubectl expose deployment ubuntu-pod --type=NodePort` - create service for existing service
### rolling update
##### set an image to a new version on a deployment, which will trigger an update
`kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1 --record`
`kubectl rollout history deployment/nginx-deployment` - check history
`kubectl rollout status deployment nginx-deployment` - check rollout status
`kubectl label pods --all person=leah` - attach label to resource
`kubectl label pods --all person- ` - remove label from resource
k8s yamls https://gist.github.com/DStorck/ba30fc08174a8935a97c87d1070bebf4
setup a node :
scp hk8s-node-1:/etc/systemd/system/kubelet.service .
scp hk8s-node-1:/etc/systemd/system/kube-proxy.service .
scp kubelet.service ik8s-node-0:
scp kube-proxy.service ik8s-node-0:
ssh ik8s-node-0 # manually edit over there, put into /etc/systemd/system, use systemctl, etc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment