When developers first start containerizing applications for Kubernetes, it is tempting to treat every workload the same way:

apiVersion: apps/v1
kind: Deployment # ⚠️ The hidden trap for databases
metadata:
  name: mongodb

It works fine on day one. You deploy MongoDB, attach a PersistentVolumeClaim (PVC), and your backend connects happily.

Then, you trigger your first rolling update—maybe bumping the image version from mongo:8.0 to mongo:9.0.

Suddenly, your deployment hangs. Your pods enter an endless ContainerCreating crash loop, and your application goes completely offline.

What happened? You just hit the dreaded Multi-Attach Volume Error.

The Autopsy: The Multi-Attach Disaster

Kubernetes Deployments were engineered for stateless applications (like a React frontend or an Express API). In a stateless deployment:

1. Pods are completely disposable and have random hash names (e.g., backend-58f9ddc96-bwsfc).
2. During a rolling update, Kubernetes uses maxSurge by default: it starts the new pod before terminating the old pod to achieve zero-downtime.

Here is why that breaks stateful databases:

[Old Pod: mongodb-7c89f] ──(Holds Disk Lock)──▶ [PersistentVolume (RWO)]
                                                        ▲
[New Pod: mongodb-992ab] ──(Tries to Mount)─────────────┘
  ❌ FATAL ERROR: Multi-Attach error for volume. 
     Volume is already exclusively attached to one node.

Cloud storage volumes (like AWS EBS or digital block storage) almost always have an access mode of ReadWriteOnce (RWO). This means the disk can only be attached to one node at a time.

Because Kubernetes started the new pod before terminating the old one, both pods attempted to mount the exact same disk simultaneously. The storage driver rejected the mount, leaving the new pod stuck and freezing your database.

The Cure: Upgrading to a StatefulSet

To safely run stateful storage workloads like MongoDB, PostgreSQL, or Redis in Kubernetes, you must use a StatefulSet.

Here is the hardened MongoDB StatefulSet manifest I implemented in DealVault:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
  namespace: deal-aggregator
spec:
  serviceName: mongodb # Links to our headless service
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 999 # Non-root mongodb user
        fsGroup: 999
      containers:
        - name: mongodb
          image: mongo:9.0.2
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: data
              mountPath: /data/db
  # The magic happens here:
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

3 Reasons Why StatefulSets Save Your Database

1. Predictable, Stable Identity

Unlike Deployments with randomized pod names, StatefulSet pods get sticky, ordered names: mongodb-0, mongodb-1, etc. When mongodb-0 restarts, it comes back as mongodb-0. It never changes identity.

2. Sequential, Safe Updates

A StatefulSet never starts a second pod while the first one is still attached to storage. When updating, it executes an orderly, sequential sequence:

1. Sends a SIGTERM to mongodb-0.
2. Flushes database writes cleanly to disk.
3. Unmounts the volume.
4. Starts the updated container and re-attaches the existing volume. Zero volume lock collisions.

3. Dedicated volumeClaimTemplates

Instead of having a separate PVC YAML file that multiple pods might accidentally fight over, the volumeClaimTemplates block dynamically provisions a unique, permanently bonded disk for that specific pod index (data-mongodb-0). Even if the pod is rescheduled onto a different server, Kubernetes re-binds that exact disk volume.

Don't Forget the Headless Service (clusterIP: None)

A standard Kubernetes Service acts as a round-robin load balancer via a virtual cluster IP. But with a database, you don't want a load balancer proxying traffic randomly—you need direct communication with the exact database pod.

We use a Headless Service:

apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: deal-aggregator
spec:
  clusterIP: None # 👈 "Headless" (No virtual IP)
  selector:
    app: mongodb
  ports:
    - port: 27017
      targetPort: 27017

By setting clusterIP: None, Kubernetes DNS bypasses the proxy and returns the direct pod IP, giving our database a permanent internal DNS address: mongodb-0.mongodb.deal-aggregator.svc.cluster.local (or simply mongodb within the namespace).

The Verdict

If your container writes data to disk that needs to survive restarts, do not use a Deployment. Use a StatefulSet with volumeClaimTemplates and a headless service to ensure your data stays intact through every deployment cycle.

This architecture was implemented and verified as part of DealVault, an open-source DevSecOps portfolio project featuring automated CI testing on Kubernetes (KinD).

I wanted that check to happen where the change is made: in the pull request. So I built Cloud Policy Guard, a Policy-as-Code security gate for Terraform.

Scan the plan, not the account

The tool reads the JSON form of a Terraform plan, which describes exactly what Terraform is about to create:

terraform plan -out=tfplan
terraform show -json tfplan > plan.json
cloud-policy-guard scan --terraform-plan plan.json

Because it works on the plan file, it runs fully offline and needs no AWS credentials. Nothing has to be deployed, and no secrets have to be handed to the CI job.

The rules

The rules are context-aware: they evaluate the combination of actions and resources to set the severity, instead of flagging every wildcard the same way.

Turning findings into a pass/fail decision

Each finding adds to a deterministic risk score: Critical = 10, High = 7, Medium = 4, Low = 1. A pipeline fails if any finding reaches the fail_on severity, or if the total score passes max_risk_score:

# .cpg.yaml
fail_on: critical       # fail if any finding is critical or worse
max_risk_score: 50      # fail if the total risk score exceeds 50
exclude_rules:
  - LP005               # rules can be disabled per repo
format: text            # text for humans, json for pipelines

The exit code does the rest: 0 for PASS, 1 for FAIL. Any CI system (GitHub Actions, GitLab CI, Jenkins) blocks the merge without extra integration work.

In a GitHub Actions workflow

- name: Terraform Plan
  run: |
    terraform init
    terraform plan -out=tfplan
    terraform show -json tfplan > plan.json

- name: Scan IAM least privilege
  # exit code 1 fails the job and blocks the PR
  run: cloud-policy-guard scan --terraform-plan plan.json

Testing it honestly

The repo ships with a deliberately vulnerable Terraform example and a secure one. The vulnerable plan must FAIL and the secure plan must PASS, and a pytest suite keeps the rules from regressing. The tool is also containerised with Docker, so it behaves the same on my laptop and in CI.

What I took away

• Shifting left only works if the check is fast, offline and gives a clear yes/no.

• A severity model plus a score threshold lets teams adopt a gate gradually instead of all at once.

• Two outputs matter: coloured text that developers will actually read, and JSON that pipelines can parse.