Skip to content

Instantly share code, notes, and snippets.

@naviat
Forked from tuannvm/argo.md
Last active March 10, 2020 03:39
Show Gist options
  • Save naviat/a0ad7fa7e8be1f378c49992d46f7c3d1 to your computer and use it in GitHub Desktop.
Save naviat/a0ad7fa7e8be1f378c49992d46f7c3d1 to your computer and use it in GitHub Desktop.
#Helm #Kubernetes #cheatsheet, happy helming!

Helm cheatsheet

Get started


Struture

.
├── Chart.yaml --> metadata info
├── README.md
├── requirements.yaml --> define dependencies
├── templates
│   ├── spark-master-deployment.yaml --> configuration with template supported
│   ├── spark-worker-deployment.yaml
│   └── spark-zeppelin-deployment.yaml
│   └── NOTES.txt --> display when run "helm chart"
│   └── _helpers.tpl --> template handler
└── values.yaml --> variable list, will be interpolated on templates file during deployment
│
└── charts
    ├── apache/
        ├── Chart.yaml
  • Chart.yaml
  name: The name of the chart (required)
  version: A SemVer 2 version (required)
  description: A single-sentence description of this project (optional)
  keywords:
    - A list of keywords about this project (optional)
  home: The URL of this project's home page (optional)
  sources:
    - A list of URLs to source code for this project (optional)
  maintainers: # (optional)
    - name: The maintainer's name (required for each maintainer)
      email: The maintainer's email (optional for each maintainer)
  engine: gotpl # The name of the template engine (optional, defaults to gotpl)
  icon: A URL to an SVG or PNG image to be used as an icon (optional).
  appVersion: The version of the app that this contains (optional). This needn't be SemVer.
  deprecated: Whether or not this chart is deprecated (optional, boolean)
  tillerVersion: The version of Tiller that this chart requires. This should be expressed as a SemVer range: ">2.0.0" (optional)
  • requirements.yaml

Adding an alias for a dependency chart would put a chart in dependencies using alias as name of new dependency. Condition - The condition field holds one or more YAML paths (delimited by commas). If this path exists in the top parent's values and resolves to a boolean value, the chart will be enabled or disabled based on that boolean value. Only the first valid path found in the list is evaluated and if no paths exist then the condition has no effect. Tags - The tags field is a YAML list of labels to associate with this chart. In the top parent's values, all charts with tags can be enabled or disabled by specifying the tag and a boolean value. Conditions (when set in values) always override tags

  dependencies:
  - name: apache
    version: 1.2.3
    repository: http://example.com/charts
    alias: new-subchart-1
    condition: subchart1.enabled, global.subchart1.enabled
        tags:
          - front-end
          - subchart1

  - name: mysql
    version: 3.2.1
    repository: http://another.example.com/charts
    alias: new-subchart-2
    condition: subchart2.enabled,global.subchart2.enabled
        tags:
          - back-end
          - subchart1

General Usage

  helm list --all
  helm repo (list|add|update)
  helm search
  helm inspect <chart-name>
  hem install --set a=b -f config.yaml <chart-name> -n <release-name> # --set take precedented, merge into -f
  helm status <deployment-name>
  helm delete <deployment-name>
  helm inspect values <chart-name>
  helm upgrade -f config.yaml <deployment-name> <chart-name>
  helm rollback <deployment-name> <version>

  helm create <chart-name>
  helm package <chart-name>
  helm lint <chart-name>

  helm dep up <chart-name> # update dependency
  helm get manifest <deployment-name> # prints out all of the Kubernetes resources that were uploaded to the server
  helm install --debug --dry-run <deployment-name> # it will return the rendered template to you so you can see the output
  • --set outer.inner=value is translated into this:
  outer:
  inner: value
  • --set servers[0].port=80,servers[0].host=example:
  servers:
  - port: 80
    host: example
  • --set name={a, b, c} translates to:
  name:
  - a
  - b
  - c
  • --set name=value1,value2:
  name: "value1,value2"
  • --set nodeSelector."kubernetes.io/role"=master
  nodeSelector:
  kubernetes.io/role: master
  • --set livenessProbe.exec.command=[cat,docroot/CHANGELOG.txt] --set livenessProbe.httpGet=null
livenessProbe:
-  httpGet:
-    path: /user/login
-    port: http
  initialDelaySeconds: 120
+  exec:
+    command:
+    - cat
+    - docroot/CHANGELOG.txt
  • --timeout
  • --wait
  • --no-hooks
  • --recreate-pods

Template

Values that are supplied via a values.yaml file (or via the --set flag) are accessible from the .Values object in a template

Release.Name:
Release.Time:
Release.Namespace: The namespace the chart was released to.
Release.Service: The service that conducted the release. Usually this is Tiller.
Release.IsUpgrade: This is set to true if the current operation is an upgrade or rollback.
Release.IsInstall: This is set to true if the current operation is an install.
Release.Revision: The revision number. It begins at 1, and increments with each helm upgrade.
Chart: The contents of the Chart.yaml. Thus, the chart version is obtainable as "Chart.Version" and the maintainers are in "Chart.Maintainers".
Files: Files can be accessed using {{index .Files "file.name"}} or using the "{{.Files.Get name}}" or "{{.Files.GetString name}}" functions. You can also access the contents of the file as []byte using "{{.Files.GetBytes}}"
Capabilities: "({{.Capabilities.KubeVersion}}", Tiller "({{.Capabilities.TillerVersion}}", and the supported Kubernetes API versions "({{.Capabilities.APIVersions.Has "batch/v1")"

{{.Files.Get config.ini}}
{{.Files.GetBytes}} useful for things like images

{{.Template.Name}}
{{.Template.BasePath}}
  • default value
{{default "minio" .Values.storage}}

//same
{{ .Values.storage | default "minio" }}
  • put a quote outside
heritage: {{.Release.Service | quote }}

# same result
heritage: {{ quote .Release.Service }}
  • global variable
global:
  app: MyWordPress

// could be access as "{{.Values.global.app}}"
  • Includes a template called mytpl.tpl, then lowercases the result, then wraps that in double quotes
value: {{include "mytpl.tpl" . | lower | quote}}
  • required function declares an entry for .Values.who is required, and will print an error message when that entry is missing
value: {{required "A valid .Values.who entry required!" .Values.who }}
  • The sha256sum function can be used together with the include function to ensure a deployments template section is updated if another spec changes
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
[...]
  • The annotation "helm.sh/resource-policy": keep instructs Tiller to skip this resource during a helm delete operation

  • In the templates/ directory, any file that begins with an underscore(_) is not expected to output a Kubernetes manifest file. So by convention, helper templates and partials are placed in a _helpers.tpl file.

Hooks

Read more

  • include these annotation inside hook yaml file, for e.g templates/post-install-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    # This is what defines this resource as a hook. Without this line, the
    # job is considered part of the release.
    "helm.sh/hook": post-install, post-upgrade
    "helm.sh/hook-weight": "-5"

Chart repository

Read more

Signing

Read more

Test

Read more

Flow Control


If/Else

{{ if PIPELINE }}
  # Do something
{{ else if OTHER PIPELINE }}
  # Do something else
{{ else }}
  # Default case
{{ end }}

data:
  myvalue: "Hello World"
  drink: {{ .Values.favorite.drink | default "tea" | quote }}
  food: {{ .Values.favorite.food | upper | quote }}
  {{- if eq .Values.favorite.drink "lemonade" }}
  mug: true
  {{- end }} # notice the "-" in the left, if will help eliminate newline before variable

With

with can allow you to set the current scope (.) to a particular object

data:
  myvalue: "Hello World"
  {{- with .Values.favorite }}
  drink: {{ .drink | default "tea" | quote }}
  food: {{ .food | upper | quote }}
  {{- end }} # instead of writing ".Values.favorite.drink"

Inside of the restricted scope, you will not be able to access the other objects from the parent scope

Range

# predefined variable
pizzaToppings:
  - mushrooms
  - cheese
  - peppers
  - onions

toppings: |-
    {{- range $i, $val := .Values.pizzaTopping }}
    - {{ . | title | quote }}  # upper first character, then quote
    {{- end }}

sizes: |-
    {{- range tuple "small" "medium" "large" }}
    - {{ . }}
    {{- end }} # make a quick list

Variables

It follows the form $name. Variables are assigned with a special assignment operator: :=

data:
  myvalue: "Hello World"
  {{- $relname := .Release.Name -}}
  {{- with .Values.favorite }}
  drink: {{ .drink | default "tea" | quote }}
  food: {{ .food | upper | quote }}
  release: {{ $relname }}
  {{- end }}

# use variable in range
 toppings: |-
    {{- range $index, $topping := .Values.pizzaToppings }}
      {{ $index }}: {{ $topping }}
    {{- end }}

#toppings: |-
#      0: mushrooms
#      1: cheese
#      2: peppers
#      3: onions

{{- range $key,$value := .Values.favorite }}
  {{ $key }}: {{ $value }}
  {{- end }} # instead of specify the key, we can actually loop through the values.yaml file and print values

There is one variable that is always global - $ - this variable will always point to the root context

...
labels:
    # Many helm templates would use `.` below, but that will not work,
    # however `$` will work here
    app: {{ template "fullname" $ }}
    # I cannot reference .Chart.Name, but I can do $.Chart.Name
    chart: "{{ $.Chart.Name }}-{{ $.Chart.Version }}"
    release: "{{ $.Release.Name }}"
    heritage: "{{ $.Release.Service }}"
...

Named Templates

template names are global

# _helpers.tpl
{{/* Generate basic labels */}}
{{- define "my_labels" }}
  labels:
    generator: helm
    date: {{ now | htmlDate }}
    version: {{ .Chart.Version }}
    name: {{ .Chart.Name }}
{{- end }}

When a named template (created with define) is rendered, it will receive the scope passed in by the template call.

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
  {{- template "my_labels" . }} # Notice the final dot, it will pass the global scope inside template file. Without it version & name will not be �generated.
  {{- include "my_labels" . | indent 2 }} # similar to "template" directive, have the ability to control indentation

referable to use include over template. Because template is an action, and not a function, there is no way to pass the output of a template call to other functions; the data is simply inserted inline.

Files inside Templates

# file located at parent folder
# config1.toml: |-
#   message = config 1 here
# config2.toml: |-
#   message = config 2 here
# config3.toml: |-
#   message = config 3 here

data:
  {{- $file := .Files }} # set variable
  {{- range tuple "config1.toml" "config2.toml" "config3.toml" }} # create list
  {{ . }}: |- # config file name
    {{ $file.Get . }} # get file's content
  {{- end }}

Glob-patterns & encoding

apiVersion: v1
kind: ConfigMap
metadata:
  name: conf
data:
+{{ (.Files.Glob "foo/*").AsConfig | indent 2 }}
---
apiVersion: v1
kind: Secret
metadata:
  name: very-secret
type: Opaque
data:
+{{ (.Files.Glob "bar/*").AsSecrets | indent 2 }}

+token: |-
+  {{ .Files.Get "config1.toml" | b64enc }}

YAML reference

# Force type
age: !!str 21
port: !!int "80"

# Fake first line to preserve integrity
coffee: | # �no strip
  # Commented first line
         Latte
  Cappuccino
  Espresso

coffee: |- # strip off trailing newline
  Latte
  Cappuccino
  Espresso

coffee: |+ # preserve trailing newline
  Latte
  Cappuccino
  Espresso


another: value

myfile: | # insert static file
{{ .Files.Get "myfile.txt" | indent 2 }}

coffee: > �# treat as one long line
  Latte
  Cappuccino
  Espresso

Kubernetes cheatsheet

Getting Started

  • Fault tolerance
  • Rollback
  • Auto-healing
  • Auto-scaling
  • Load-balancing
  • Isolation (sandbox)

Sample yaml

apiVersion: <>
kind: <>
metadata:
  name: <>
  labels:
    ...
  annotations:
    ...
spec:
  containers:
    ...
  initContainers:
    ...
  priorityClassName: <>

Workflow

  • (kube-scheduler, controller-manager, etcd) --443--> API Server

  • API Server --10055--> kubelet

    • non-verified certificate
    • MITM
    • Solution:
      • set kubelet-certificate-authority
      • ssh tunneling
  • API server --> (nodes, pods, services)

    • Plain HTTP (unsafe)

Physical components

Master

  • API Server (443)
  • kube-scheduler
  • controller-manager
    • cloud-controller-manager
    • kube-controller-manager
  • etcd

Other components talk to API server, no direct communication

Node

  • Kubelet

  • Container Engine

    • CRI
      • The protocol which used to connect between Kubelet & container engine
  • Kube-proxy

Everything is an object - persistent entities

  • maintained in etcd, identified using

    • names: client-given
    • UIDs: system-generated
  • Both need to be unique

  • three management methods

    • Imperative commands (kubectl)
    • Imperative object configuration (kubectl + yaml)
      • repeatable
      • observable
      • auditable
    • Declarative object configuration (yaml + config files)
      • Live object configuration
      • Current object configuration file
      • Last-applied object configuration file

Namespaces

  • Three pre-defined

    • default
    • kube-system
    • kube-public: auto-readable by all users
  • Objects without namespaces

    • Nodes
    • PersistentVolumes
    • Namespaces

Labels

  • key / value
  • loose coupling via selectors
  • need not be unique

ClusterIP

  • Independent of lifespan of any backend pod
  • Service object has a static port assigned to it

Controller manager

  • ReplicaSet, deployment, daemonset, statefulSet
  • Actual state <-> desired state
  • reconciliation loop

Kube-scheduler

  • nodeSelector
  • Affinity & Anti-Affinity
    • Node
      • Steer pod to node
    • Pod
      • Steer pod towards or away from pods
  • Taints & tolerations (anti-affinity between node and pod!)
    • Base on predefined configuration (env=dev:NoSchedule)
      ...
      tolerations:
      - key: "dev"
        operator: "equal"
        value: "env"
        effect: NoSchedule
      ...
    • Base on node condition (alpha in v1.8)
      • taints added by node controller

Pod

kubectl run name --image=<image>

What's available inside the container?

  • File system
    • Image
    • Associated Volumes
      • ordinary
      • persistent
    • Container
      • Hostname
    • Pod
      • Pod name
      • User-defined envs
    • Services
      • List of all services

Access with:

  • Symlink (important):

    • /etc/podinfo/labels
    • /etc/podinfo/annotations
  • Or:

volumes:
  - name: podinfo
    downwardAPI:
      items:
        - path: "labels"
          fieldRef:
            fieldPath: metadata.labels
        - path: "annotations"
          fieldRef:
            fieldPath: metadata.annotations

Status

  • Pending
  • Running
  • Succeeded
  • Failed
  • Unknown

Probe

  • Liveness
    • Failed? Restart policy applied
  • Readiness
    • Failed? Removed from service

Pod priorities

  • available since 1.8
  • PriorityClass object
  • Affect scheduling order
    • High priority pods could jump the queue
  • Preemption
    • Low priority pods could be pre-empted to make way for higher one (if no node is available for high priority)
    • These preempted pods would have a graceful termination period

Multi-Container Pods

  • Share access to memory space
  • Connect to each other using localhost
  • Share access to the same volume
  • entire pod is host on the same node
  • all in or nothing
  • no auto healing or scaling

Init containers

  • run before app containers
  • always run to completion
  • run serially

Lifecycle hooks

  • PostStart
  • PreStop (blocking)

Handlers:

  • Exec
  • HTTP
...
spec:
  containers:
    lifecycle:
      postStart:
        exec:
          command: <>
      preStop:
        http:
...

Could invoke multiple times

ReplicaSet

Features:

  • Scaling and healing
  • Pod template
  • number of replicas

Components:

  • Pod template

  • Pod selector (could use matchExpressions)

  • Label of replicaSet

  • Number of replica

  • Could delete replicaSet without its pods using --cascade =false

  • Isolating pods from replicaSet by changing its labels

Deployments

  • versioning and rollback

  • Contains spec of replicaSet within it

  • advanced deployment

  • blue-green

  • canary

  • Update containers --> new replicaSet & new pods created --> old RS still exists --> reduced to zero

  • Every change is tracked

  • Append --record in kubectl to keep history

  • Update strategy

    • Recreate
      • Old pods would be killed before new pods come up
    • RollingUpdate
      • progressDeadlineSeconds
      • minReadySeconds
      • rollbackTo
      • revisionHistoryLimit
      • paused
        • spec.Paused
  • kubectl rollout undo deployment/<> --to-revision=<>

  • kubectl rollout statua deployment/<>

  • kubectl set image deployment/<> <>=<>:<>

  • kubectl rollout resume/pause <>

ReplicationController

  • RC = ( RS + deployment ) before
  • Obsolete

DaemonSet

  • Ensure all nodes run a copy of pod
  • Cluster storage, log collection, node monitor ...

StatefulSet

  • Maintains a sticky identity
  • Not interchangeable
  • Identifier maintains across any rescheduling

Limitation

  • volumes must be pre-provisioned
  • Deleting / Scaling will not delete associated volumes

Flow

  • Deployed 0 --> (n-1)
  • Deleted (n-1) --> 0 (successor must be completely shutdown before proceed)
  • Must be all ready and running before scaling happens

Job (batch/v1)

  • Non-parallel jobs
  • Parallel jobs
    • Fixed completion count
      • job completes when number of completions reaches target
    • With work queue
      • requires coordination
  • Use spec.activeDeadlineSeconds to prevent infinite loop

Cronjob

  • Job should be idempotent

Horizontal pod autoscaler

  • Targets: replicaControllers, deployments, replicaSets
  • CPU or custom metrics
  • Won't work with non-scaling objects: daemonSets
  • Prevent thrashing (upscale/downscale-delay)

Services

  • Logical set of backend pods + frontend

  • Frontend: static IP + port + dns name

  • Backend: set of backend pods (via selector)

  • Static IP and networking.

  • Kube-proxy route traffic to VIP.

  • Automatically create endpoint based on selector.

  • CluterIP

  • NodePort

    • external --> NodeIP + NodePort --> kube-proxy --> ClusterIP
  • LoadBalancer

    • Need to have cloud-controller-manager
      • Node controller
      • Route controller
      • Service controller
      • Volume controller
    • external --> LB --> NodeIP + NodePort --> kube-proxy --> ClusterIP
  • ExternalName

    • Can only resolve with kube-dns
    • No selector

Service discovery

  • SRV record for named port
    • port-name.port-protocol.service-name.namespace.svc.cluster.local
  • Pod domain
    • pod-ip-address.namespace.pod.cluster.local
    • hostname is metadata.name

spec.dnsPolicy

  • default
    • inherit node's name resolution
  • ClusterFirst
    • Any DNS query that does not match the configured cluster domain suffix, such as “www.kubernetes.io”, is forwarded to the upstream nameserver inherited from the node
  • ClusterFirstWithHostNet
    • if host network = true
  • None (since k8s 1.9)
    • Allow custom dns server usage

Headless service

  • with selector? --> associate with pods in cluster
  • without selector? --> forward to externalName

Could specify externalIP to service

Volumes

Lifetime longer than any containers inside a pod.

4 types:

  • configMap

  • emptyDir

    • share space / state across containers in same pod
    • containers can mount at different times
    • pod crash --> data lost
    • container crash --> ok
  • gitRepo

  • secret

    • store on RAM
  • hostPath

Persistent volumes

Role-Based Access Control (RBAC)

  • Role
    • Apply on namespace resources
  • ClusterRole
    • cluster-scoped resources (nodes,...)
    • non-resources endpoint (/healthz)
    • namespace resources across all namespaces

Basic commands

# show current context
kubectl config current-context

# get specific resource
kubectl get (pod|svc|deployment|ingress) <resource-name>

# Get pod logs
kubectl logs -f <pod-name>

# Get nodes list
kubectl get no -o custom-columns=NAME:.metadata.name,AWS-INSTANCE:.spec.externalID,AGE:.metadata.creationTimestamp

# Run specific command | Drop to shell
kubectl exec -it <pod-name> <command>

# Describe specific resource
kubectl describe (pod|svc|deployment|ingress) <resource-name>

# Set context
kubectl config set-context $(kubectl config current-context) --namespace=<namespace-name>

# Run a test pod
kubectl run -it --rm --generator=run-pod/v1 --image=alpine:3.6 tuan-shell -- sh
  • from @so0k link

  • access dashboard

# bash
kubectl -n kube-system port-forward $(kubectl get pods -n kube-system -o wide | grep dashboard | awk '{print $1}') 9090

# fish
kubectl -n kube-system port-forward (kubectl get pods -n kube-system -o wide | grep dashboard | awk '{print $1}') 9090

Chapter 13. Integrating storage solutions and Kubernetes

  • External service without selector (access with external-database.svc.default.cluster endpoint)
kind: Service
apiVersion: v1
metadata:
  name: external-database
spec:
  type: ExternalName
  externalName: "database.company.com
  • external service with IP only
kind: Service
apiVersion: v1
metadata:
  name: external-ip-database
---
kind: Endpoints
apiVersion: v1
metadata:
  name: external-ip-database
subsets:
  - addresses:
    - ip: 192.168.0.1
    ports:
    - port: 3306
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment