Skip to content
MINH VO A working notebook
by an engineer in Vietnam
Foundation9 min read

Watch a Deployment replace a Pod without changing its intent

Run a local Kubernetes deletion experiment and compare owner UIDs, desired replicas, Ready status and minReadySeconds before deleting the test cluster.

Miniature conveyor line with three stations and a curved side track

Deleting one Pod does not change a Deployment’s request for three replicas. The missing Pod is replaced because the controller still has work to do. The replacement has a new identity, and its existence does not mean it is ready to serve traffic.

This article is for a developer learning to operate workloads in Kubernetes. It assumes basic container and command-line knowledge. We run a real single-node local cluster, delete one controlled Pod, inspect its replacement and then change the desired replica count. The experiment deliberately separates created, ready and available replicas using a file-based readiness probe. It is a control-plane exercise, not an application load test or a cloud availability measurement.

Follow the owner chain

A Deployment manages ReplicaSets. A ReplicaSet creates or deletes Pods to approach its requested count, using a selector and a Pod template. The kubelet on a node manages containers inside each Pod. Those responsibilities explain why deleting a Pod and restarting a container are different events.

Look at the actual owner references rather than inferring ownership from a similar name. A controlled Pod points to a ReplicaSet UID; that ReplicaSet points to the Deployment UID. The local experiment checks both links. It also checks that the ReplicaSet UID remains unchanged after deletion. No new application template is introduced, so this replacement is not an application rollout.

Actual local owner-reference UIDs connect one Deployment to the same ReplicaSet before and after deletion; the removed Pod and its replacement have distinct UIDs.View full-size image ↗

Kubernetes defines a Pod’s identity by its UID. A replacement is a different Pod, even if a workload can reuse a name. Here the names differ too, but the proof uses UIDs. The replacement’s container filesystem does not inherit the readiness marker written into the deleted container. This observation concerns that temporary file; it says nothing about recovery from a separately provisioned persistent volume.

Ask four different questions

The desired count comes from the Deployment specification. The other counts are controller-reported status. For the tested Kubernetes 1.34 API, the versioned field definitions distinguish targeted non-terminating Pods, those with a Ready condition, and those ready long enough to be available.

Current replicas therefore do not mean healthy application workers. A Pod can exist while waiting for scheduling or startup. This experiment additionally waits for its selected Pods to reach the Running phase so that the executable probe can be controlled. That extra condition is a property of the test, not the definition of the Deployment replica count.

Ready and available differ because the manifest sets fifteen minimum ready seconds. A newly ready Pod must satisfy that interval without a container crash before it counts as available. The default is zero, which can hide the distinction in a quick demonstration. Neither count establishes successful business transactions; this probe only checks for a file.

Create a deliberately gated workload

Save this as deployment.yaml. It requests three sleeping containers and marks each unready until /tmp/ready exists. There is no liveness probe. A failed readiness check therefore does not instruct the kubelet to restart the container. The memory limit and small request keep this local example bounded; they are not production sizing advice.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reconcile-lab
spec:
  replicas: 3
  minReadySeconds: 15
  selector:
    matchLabels:
      app: reconcile-lab
  template:
    metadata:
      labels:
        app: reconcile-lab
    spec:
      terminationGracePeriodSeconds: 1
      containers:
        - name: sleeper
          image: busybox:1.37.0
          imagePullPolicy: Never
          command: ["sleep", "3600"]
          readinessProbe:
            exec:
              command: ["test", "-f", "/tmp/ready"]
            periodSeconds: 1
            failureThreshold: 1
          resources:
            requests:
              cpu: 10m
              memory: 8Mi
            limits:
              memory: 32Mi

The label in the selector matches the label in the Pod template. Keep this experiment’s selector separate from other controllers. Kubernetes documentation warns that overlapping controller selectors can produce conflicts; a matching label alone is not a reason to attach an unrelated Pod to this workload.

The example uses Docker, kind 0.30.0 and kubectl 1.34.1 on Linux. The kind release supplies the pinned Kubernetes 1.34.0 node image used by the script. Install those tools, then prepare the exact BusyBox image and run the experiment in a directory containing the two files:

docker pull busybox:1.37.0@sha256:9db7b59979c38555a39def84a31fb98b5296952f9e3afd4f6f11f05b07adfab0
docker tag busybox@sha256:9db7b59979c38555a39def84a31fb98b5296952f9e3afd4f6f11f05b07adfab0 busybox:1.37.0
python3 -I experiment.py

The tag is used only after verifying its expected repository digest. The workload sets image pulling to Never because the script explicitly loads the image into the local node. An omitted load should fail visibly; it should not silently fetch a different image during the observation.

Run the controlled sequence

Save the following as experiment.py. It creates a uniquely named kind cluster and an isolated kubeconfig. Every kubectl command names that file and the generated context. The cleanup uses the same kubeconfig and removes the owned cluster even when an assertion fails. The only local output file is observations.json, containing the raw status records and identities.

import json
import os
from pathlib import Path
import subprocess
import tempfile
import time
import uuid

NODE = 'kindest/node:v1.34.0@sha256:7416a61b42b1662ca6ca89f02028ac133a309a2a30ba309614e8ec94d976dc5a'
BUSYBOX_DIGEST = 'busybox@sha256:9db7b59979c38555a39def84a31fb98b5296952f9e3afd4f6f11f05b07adfab0'
kind = os.environ.get('DEVOPS_KIND', 'kind')
cluster = 'reconcile-' + uuid.uuid4().hex[:10]
manifest = Path(__file__).with_name('deployment.yaml')
output = Path(__file__).with_name('observations.json')
records = []

def run(args, timeout=45):
    result = subprocess.run(args, text=True, capture_output=True, timeout=timeout)
    if result.returncode:
        raise RuntimeError(f'{args!r}\n{result.stdout}\n{result.stderr}')
    return result.stdout

image = json.loads(run(['docker', 'image', 'inspect', 'busybox:1.37.0']))[0]
assert BUSYBOX_DIGEST in image['RepoDigests'], 'Pull the stated BusyBox image first'
with tempfile.TemporaryDirectory(prefix='devops-reconcile-') as directory:
    kubeconfig = str(Path(directory) / 'config')
    base = ['kubectl', '--kubeconfig', kubeconfig, '--context', 'kind-' + cluster]
    def k(*args):
        return run(base + list(args))
    def get(*args):
        return json.loads(k('get', *args, '-o', 'json'))
    def wait_state(label, desired, ready, available):
        deadline = time.monotonic() + 120
        while time.monotonic() < deadline:
            d = get('deployment', 'reconcile-lab')
            pods = get('pods', '-l', 'app=reconcile-lab')
            active = [p for p in pods['items'] if not p['metadata'].get('deletionTimestamp')]
            s = d.get('status', {})
            observed = (d['spec']['replicas'], s.get('replicas', 0), s.get('readyReplicas', 0), s.get('availableReplicas', 0))
            running = all(p.get('status', {}).get('phase') == 'Running' for p in active)
            if (observed == (desired, desired, ready, available) and
                    len(active) == desired and running and
                    s.get('observedGeneration') == d['metadata']['generation']):
                row = {'label': label, 'elapsedSeconds': round(time.monotonic() - observe_start, 3),
                       'counts': observed, 'deployment': d,
                       'pods': pods, 'replicasets': get('replicasets', '-l', 'app=reconcile-lab')}
                records.append(row)
                print(label, 'desired/current/ready/available', *observed, flush=True)
                return row
            time.sleep(0.3)
        raise TimeoutError(f'{label}: expected state not observed')
    try:
        run([kind, 'create', 'cluster', '--name', cluster, '--kubeconfig', kubeconfig,
             '--image', NODE, '--wait', '120s'], timeout=210)
        run([kind, 'load', 'docker-image', 'busybox:1.37.0', '--name', cluster], timeout=90)
        versions = json.loads(k('version', '-o', 'json'))
        observe_start = time.monotonic()
        k('apply', '-f', str(manifest))
        blocked = wait_state('created-not-ready', 3, 0, 0)
        for p in blocked['pods']['items']:
            k('exec', p['metadata']['name'], '--', 'touch', '/tmp/ready')
        before = wait_state('baseline', 3, 3, 3)
        old = before['pods']['items']
        old_uids = {p['metadata']['uid'] for p in old}
        victim = sorted(old, key=lambda p: p['metadata']['name'])[0]
        k('delete', 'pod', victim['metadata']['name'], '--wait=true', '--timeout=45s')
        replaced = wait_state('replacement-not-ready', 3, 2, 2)
        new = [p for p in replaced['pods']['items'] if p['metadata']['uid'] not in old_uids]
        assert len(new) == 1
        replacement = new[0]
        assert victim['metadata']['uid'] not in {p['metadata']['uid'] for p in replaced['pods']['items']}
        old_rs = before['replicasets']['items'][0]
        new_rs = replaced['replicasets']['items'][0]
        assert len(before['replicasets']['items']) == len(replaced['replicasets']['items']) == 1
        assert old_rs['metadata']['uid'] == new_rs['metadata']['uid']
        assert replacement['metadata']['ownerReferences'][0]['uid'] == old_rs['metadata']['uid']
        assert old_rs['metadata']['ownerReferences'][0]['uid'] == before['deployment']['metadata']['uid']
        assert before['deployment']['metadata']['generation'] == replaced['deployment']['metadata']['generation']
        k('exec', replacement['metadata']['name'], '--', 'test', '!', '-e', '/tmp/ready')
        k('exec', replacement['metadata']['name'], '--', 'touch', '/tmp/ready')
        wait_state('ready-before-available', 3, 3, 2)
        wait_state('recovered', 3, 3, 3)
        k('scale', 'deployment/reconcile-lab', '--replicas=0')
        scaled = wait_state('desired-zero', 0, 0, 0)
        assert scaled['deployment']['metadata']['generation'] > before['deployment']['metadata']['generation']
        assert scaled['replicasets']['items'][0]['metadata']['uid'] == old_rs['metadata']['uid']
        evidence = {'scope': 'Actual single-node local kind control-plane experiment; no cloud deployment or service-traffic measurement.',
                    'versions': versions, 'kindVersion': run([kind, 'version']).strip(),
                    'nodeImage': NODE, 'busyboxImageId': image['Id'],
                    'victimUid': victim['metadata']['uid'], 'replacementUid': replacement['metadata']['uid'],
                    'replicaSetUid': old_rs['metadata']['uid'], 'replacementReadyFileMissing': True,
                    'records': records}
    finally:
        run([kind, 'delete', 'cluster', '--name', cluster, '--kubeconfig', kubeconfig], timeout=90)
        remaining = run(['docker', 'ps', '-a', '--filter', 'name=' + cluster, '--format', '{{.ID}}']).strip()
        assert not remaining, 'Owned kind node still exists'
    evidence['ownedClusterDeleted'] = True
    output.write_text(json.dumps(evidence, indent=2) + '\n')
    print('same ReplicaSet; different Pod UID; owned cluster deleted', flush=True)

The polling limits are test deadlines, not Kubernetes guarantees. The script waits for specific status combinations and checks that the controller has observed the current generation. Deployment, Pod and ReplicaSet reads are separate API calls, so the stored records are not one atomic snapshot across resources. The stable, deliberately gated stages make these checks inspectable without claiming to record every transient state.

Six observed local stages plot current, ready and available replica counts against the sample times recorded by the harness; the ready-before-available sample separates the last two counts.View full-size image ↗

After initial creation, all three Pods exist and run, but none is ready. Touching each marker allows the probe to succeed; after the minimum interval, all three become available. The script then deletes exactly one Pod from the owned workload. It does not lower the desired count or change the template.

At the next selected stage, three Pods again exist, while only the two surviving Pods are ready and available. The controller may replace the object quickly enough that polling never records a current count of two. The experiment does not invent that missing observation. The replacement has a new UID and no marker, while the Deployment generation and ReplicaSet UID stay the same.

Creating the replacement’s marker produces a stage with three ready replicas but only two available. The following stage returns to three available. The plotted times include local startup, probe and API polling effects; they are observations from one run, not a recovery-time benchmark or a promise that another machine reaches each stage at the same second.

Change intent when you want a different steady state

Deleting a controlled Pod is useful here as a bounded experiment, but it is not a way to permanently reduce replica count. The final command changes the Deployment specification to zero replicas. The controller then converges toward zero instead of creating another replacement. The generation increases, while the same ReplicaSet is retained in this unchanged-template example. In both recorded runs, the zero-count sample still listed three Pods with deletion timestamps. They were terminating, so the current count excluded them. A zero status count was not evidence that every old process had already exited.

Literal output from the executed local Kubernetes experiment records six desired/current/ready/available combinations and confirms the same ReplicaSet, a different Pod UID and cluster cleanup.View full-size image ↗

A rollout has a different trigger: changing the Deployment’s Pod template creates a new rollout. Scaling alone does not. Conversely, the selected rollout strategy is not an “at most three processes” lock during arbitrary Pod deletion. Kubernetes documents that replacement can start while the old Pod is still terminating. Our one-second termination grace and selected samples do not establish a universal ordering of old-process exit and new-process start.

Reconciliation also has prerequisites. A missing image, insufficient scheduling capacity, denied API operation or unavailable node can keep current, ready and available counts below the request. A controller can retry without making the application healthy. For that reason, a desired count is a target to diagnose against, not evidence that all replicas exist or that users are receiving service.

For a small follow-up, change minReadySeconds to zero and run a fresh local cluster. Adapt the harness to wait for three ready and three available together instead of requiring the intermediate three-ready/two-available sample; at zero, that sample is no longer a required state. Keep the UID and owner-reference checks unchanged. A second exercise is to remove the marker-creation action and observe the failed readiness condition until a bounded timeout. Explain why the three unready Pods do not, by themselves, request three additional replicas. Stop the owned cluster after either test, just as the supplied harness does.

Sources & further reading

  1. Kubernetes: Deployments
  2. Kubernetes: ReplicaSets
  3. Kubernetes: Pod lifecycle
  4. Kubernetes 1.34: Deployment status fields
  5. kind 0.30.0 release
← Back to the journal
All notes

Illustration

100%