Ten manifests: Kubernetes from a single pod to a load-balanced service

Most Kubernetes introductions either hand you a two-hundred-line Helm chart or stop at kubectl run nginx. For a Techmunch meetup talk I built neither: ten small manifests that climb the object ladder one rung at a time: Pod, then Deployment, then Service, then Ingress, then a chaos Job, with a demo app small enough to read in one sitting and, crucially, an app that tells you which pod you’re talking to. Every rung is something you watch happen, not something you take on faith.

There is no README in the repository. The narration lived in my head on the night. This post is that narration, written down four years late, plus a caveat the manifests themselves already earned: 04-ingress.yaml was dead on arrival. Its API was removed from Kubernetes a month before I gave this talk, and I never noticed.

An app that introduces itself

The demo is app.py, a Flask app just over a hundred lines long, backed by names.py, a flat list of 199 first names. On boot, each instance either reads a name that survived from a previous run or picks a new one at random (app.py, lines 18–26):

if os.path.isfile('/etc/namesvol/file'):
    my_name = open('/etc/namesvol/file', 'r').read()
else:
    my_name = NAMES[random.randint(0, len(NAMES)-1)]
    try:
        with open('/etc/namesvol/file', 'w') as fp:
            fp.write(my_name)
    except Exception as e:
        print(f"Could not save my name: {e}")

That’s the whole trick, and it’s the reason this repository is worth a post: “pod 7d4f9b8f9-x2kpl” teaches nothing about what Kubernetes is doing, but “Hello, my name is Deborah” is something you can watch change, or watch not change, and draw the right conclusion from. / prints the name alongside every other piece of configuration the pod can see at once (app.py, lines 56–58):

@app.route("/")
def hello_world():
    return f"<h1>Hello, World!</h1><h2>My name is: {my_name}</h2><h2>My config is: {my_app_config}</h2><h2>My secret is: {my_app_secret_config}</h2><h2>My files hold</h2><p>{my_app_file_contents}</p>"

One page, one glance, and you’ve seen an environment variable, a mounted file, and (as post 54 covers) a ConfigMap and a Secret. Two more routes matter for this post: /whoami returns the bare name for machine-to-machine calls, and /neighbour (used below once a Service exists) asks the internal Service who else is out there.

Containerising it

The Dockerfile is thirteen lines with comments worth keeping:

# docker build -t sjnarmstrong/techmunch-sholto:20210717 .
# docker run  -p 8888 sjnarmstrong/techmunch-sholto:20210717
# see best practices here https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
# alpine is small and small ~ secure. Distroless
FROM python:3.7-alpine3.14
# Copy first as it will never change
COPY requirements.txt /tmp/
RUN python -m pip install -r /tmp/requirements.txt

COPY . /app
WORKDIR /app
EXPOSE 8888
ENTRYPOINT ["python"]
CMD ["app.py"]

requirements.txt is copied and installed before the application source, so an edit to app.py invalidates only the last two layers, not the dependency install: the single most common Docker layer-ordering trick, and it’s already here. .dockerignore keeps deployments/, .git and the Dockerfile itself out of the build context, which is why the manifests below can kubectl apply a directory that never once ends up inside the image.

Play with the cluster

Everything from here down has a live version. No real cluster, no backend, it’s a small state machine over fake pods, built to reconcile the way the real Deployment and StatefulSet controllers do: pick a manifest, hit Apply, and watch the panel of pods react. Kill one and see what comes back (or doesn’t). Scale the replica count. Click GET / a few times and watch the Service spread the load.

00: a bare Pod

deployments/00-pod.yaml, with its opening paragraph (a Pod definition close enough to the Kubernetes docs’ own wording that it’s rewritten below rather than quoted) trimmed:

# kubectl apply -f deployments/00-pod.yaml
# kubectl delete -f deployments/00-pod.yaml
# kubectl port-forward pod/techmunch 8888
apiVersion: v1
kind: Pod
metadata:
  name: techmunch
  labels:
    app: techmunch
spec:
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    fsGroup: 0
  containers:
    - name: techmunch
      image: sjnarmstrong/techmunch-sholto:20210717
      imagePullPolicy: Always
      ports:
        - containerPort: 8888
          protocol: TCP
          name: http
      resources:
        requests:
          memory: "500Mi"
          cpu: "1000m"
        limits:
          memory: "500Mi"
          cpu: "1000m"

A Pod is the smallest thing you can ask Kubernetes to run: one or more containers, sharing storage and network, scheduled together as a unit. Nothing here is managed: requests is what the scheduler reserves when picking a node, limits is the hard ceiling the kubelet enforces, and setting them equal (as this Pod does) is Kubernetes’ Guaranteed quality-of-service class. There’s no Service yet, so kubectl port-forward pod/techmunch 8888 is the only way in.

Apply 00 in the widget above, then delete the pod. Nothing comes back. That’s not a bug: nothing is watching this Pod to notice it’s gone. It’s the entire motivation for everything below.

01: a Deployment

deployments/01-deployment.yaml, opening paragraph trimmed for the same reason:

# kubectl apply -f deployments/01-deployment.yaml
# kubectl delete -f deployments/01-deployment.yaml

apiVersion: apps/v1
kind: Deployment
# Metadata of deployment
metadata:
  name: techmunch
  labels:
    app: techmunch
spec:
  replicas: 3
  # metadata deployment uses to manage the pods
  selector:
    matchLabels:
      app: techmunch
  template:
    # Metadata of pods
    metadata:
      labels:
        app: techmunch
    spec:
      containers:
        - name: techmunch
          image: sjnarmstrong/techmunch-sholto:20210717
          imagePullPolicy: Always
          ports:
            - containerPort: 8888
              protocol: TCP
              name: http
          resources:
            requests:
              memory: "500Mi"
              cpu: "1000m"
            limits:
              memory: "500Mi"
              cpu: "1000m"

A Deployment describes a desired state, “three of these, please”, and a controller continuously reconciles the actual state toward it. The field that trips up almost everyone learning this for the first time is that there are two separate app: techmunch blocks in this one file: spec.selector.matchLabels, which is how the Deployment decides which pods belong to it, and spec.template.metadata.labels, which is what actually gets stamped onto each pod it creates. They have to agree. Get them out of sync and the Deployment either adopts pods it shouldn’t or spins up new ones forever, chasing a label nothing has.

Apply 01, then kill a pod in the widget. A replacement appears, with a different name. Identity was never preserved; the Deployment only promised a count.

02: a Service

deployments/02-service.yaml. The long inline comment the original attaches to type: ClusterIP is, again, close to verbatim Kubernetes docs wording, so it’s shortened below rather than quoted:

# kubectl apply -f deployments/02-service.yaml
# kubectl delete -f deployments/02-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: techmunch
spec:
  # type: ClusterIP   # cluster-internal only, the default
  # type: NodePort    # + a fixed port on every node
  type: LoadBalancer  # + a cloud load balancer in front of that
  selector:
    app: techmunch
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8888

All three Service types live in this one file as commented alternatives, which is a nice way to see the ladder within the ladder: ClusterIP is internal-only and the default; NodePort opens a fixed port on every node; LoadBalancer, the one actually active here, asks the cloud provider for an external address in front of that. The selector is the same idea as the Deployment’s, one level up: it’s how the Service finds which pods to send port 80 traffic to, forwarded to port 8888, the port Flask listens on.

03: talking to your neighbour

deployments/03-service-internal.yaml, same trim:

# kubectl apply -f deployments/03-service-internal.yaml
# kubectl delete -f deployments/03-service-internal.yaml
apiVersion: v1
kind: Service
metadata:
  name: techmunch-internal
spec:
  type: ClusterIP   # cluster-internal only, the default
  # type: NodePort
  selector:
    app: techmunch
  ports:
    - protocol: TCP
      port: 8888
      targetPort: 8888
      # nodePort: 32000

A second Service, same selector, no external exposure: this is the one /neighbour calls by DNS name from inside the cluster (app.py, lines 60–72):

@app.route("/neighbour")
def neighbour():
    try:
        resp = requests.get(f"http://techmunch-internal:{app_port}/whoami")
    except requests.exceptions.ConnectionError:
        return "<h1>I have no neighbors</h1>"

    neighbors = [resp.text]

    resp = ["<h1>My neighbour is:</h1>", "<ul>"]
    resp.extend((f"<li>{i}</li>" for i in neighbors))
    resp.append('</ul>')
    return ''.join(resp)

kube-proxy spreads connections across every ready pod behind techmunch-internal. Apply 03 in the widget, make sure the Deployment has more than one replica, and click GET / a dozen or so times, and the tally chart fills in as the answer rotates between pods, which is load-balancing made visible rather than asserted.

04: an Ingress that was already dead

deployments/04-ingress.yaml, quoted whole (it has no comment header to trim):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  labels:
    app: techmunch
  name: techmunch
spec:
  rules:
  - host: techmunch.matrix.iotnxt.io
    http:
      paths:
      - backend:
          serviceName: techmunch
          servicePort: http
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - techmunch.matrix.iotnxt.io
    secretName: techmunch-cert

This is the only manifest in the repository with no comment header at all, and there’s a reason it’s an awkward one to write commentary for: apiVersion: extensions/v1beta1 was removed from Kubernetes in version 1.22, released in August 2021, a month before I gave this talk. It never worked against a cluster newer than that, and I didn’t notice on the night because I never actually applied it against a live ingress controller; it existed to show the shape of name-based routing and TLS via cert-manager, not to be run. It also hardcodes a private host (techmunch.matrix.iotnxt.io) and a letsencrypt-prod issuer that won’t exist in anyone else’s cluster, so it was never going to work as-is regardless.

The shape that still works is networking.k8s.io/v1, which nests the backend as service.name / service.port.name instead of the flat serviceName / servicePort fields above: the modernised manifest is in the widget’s “Ingress” step, under the fold, next to the original.

06: a chaos Job

deployments/06-job.yaml, opening paragraphs (a Job definition, again close to the docs) trimmed:

# kubectl apply -f deployments/06-job.yaml
# kubectl delete -f deployments/06-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: destroy-techmunch
spec:
  template:
    spec:
      containers:
        - name: techmunch
          image: sjnarmstrong/techmunch-sholto:20210717
          imagePullPolicy: Always
          command: ["python3", "app.py", "-j"]
      restartPolicy: Never
  backoffLimit: 4

A Job runs a Pod to completion and retries it if it fails: the general shape covers anything from a database migration to a batch export. This one is neither: it’s the same container image, started with -j instead of the Flask server, which walks every member of the StatefulSet and calls /kill on each in turn (app.py, lines 39–50; a commented-out alternate URL on line 45 is trimmed):

if args.job:
    i=-1
    while True:
        i+=1
        try:
            resp = requests.get(f"http://techmunch-{i}.techmunch-headless.techmunch-sholto.svc.cluster.local:{app_port}/kill")
        except requests.exceptions.ConnectionError:
            break
        if resp.status_code != 200:
            break
    quit()

The route it’s hammering, /kill (app.py, lines 97–103):

@app.route("/kill")
def kill():
    func = request.environ.get('werkzeug.server.shutdown')
    if func is None:
        raise RuntimeError('Not running with the Werkzeug Server')
    func()
    return "<p>I am no longer :(</p>"

The job as written targets the StatefulSet from post 54 by its headless-Service DNS name: it’s a chaos monkey shipped as a kind: Job, and the “kill everything, watch it heal” lesson is the same one whether the pods underneath are a Deployment’s or a StatefulSet’s. The widget’s chaos-job button applies that lesson to whatever’s currently applied: kill every pod at once and watch the controller (if there is one) replace them.

What’s still missing

Five manifests aren’t here: 05-statefulset.yaml, 07-persistant-volumes.yaml, 08-env-variables.yaml, 09-configmap.yaml and 09-secrets.yaml: identity, storage and configuration, the half of the ladder that matters once you leave the tutorial. That’s post 54, built on the same simulated-cluster engine as this one (src/widgets/_shared/cluster-sim/), with a StatefulSet running next to this Deployment and a “detach the volume” toggle that answers, in one interaction, exactly what a PersistentVolumeClaim was buying you.