Container Security in DevOps - Build to Runtime

Master container security in DevOps: build-time scanning, registry hygiene, admission control, runtime detection, and automated policy enforcement.

Container security spans the entire software development lifecycle from code through build to runtime, requiring security controls at each stage. Security engineers enforce image integrity, least privilege, and runtime protection through automated policies that prevent insecure containers from reaching production.

Containers introduce unique security challenges including image vulnerabilities, privilege escalation through container escapes, and lateral movement between containers. According to NIST SP 800-190, container security requires defense-in-depth across build, registry, admission, and runtime stages.

Container Security Stages Overview

The following table summarizes security controls across each container lifecycle stage:

Build-Time Security

Build-time security establishes the foundation for secure container deployments by ensuring images are minimal, hardened, and verifiable.

Minimal Base Images

Minimal base images reduce attack surface by eliminating unnecessary packages and utilities:

• Distroless images: Google Distroless images contain only the application and runtime dependencies, removing shells and package managers
• Alpine Linux: Alpine provides a minimal Linux distribution (~5MB) with musl libc and BusyBox
• Scratch images: For statically compiled binaries, scratch images contain no operating system components
• Chainguard Images: Chainguard provides hardened, minimal images with daily vulnerability scanning

Version Pinning and Updates

Pinned base image versions prevent unexpected changes from upstream updates:

# Pin to specific digest for reproducibility
FROM python:3.11-slim@sha256:abc123...

# Multi-stage build separates build from runtime
FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server

FROM gcr.io/distroless/static-debian12
COPY --from=builder /app/server /server
USER nonroot:nonroot
ENTRYPOINT ["/server"]

Version pinning should be combined with automated updates using tools like Dependabot or Renovate to incorporate security patches.

Non-Root Users

Containers should run as non-root users to limit impact from container escapes:

• USER directive: Specify non-root users in Dockerfiles with explicit UID/GID
• User namespaces: Map container root to unprivileged host users for additional isolation
• Read-only root filesystem: Prevent container modification with readOnlyRootFilesystem: true

# Create non-root user with explicit UID
RUN addgroup --gid 10001 appgroup && \
    adduser --uid 10001 --gid 10001 --disabled-password appuser
USER 10001:10001

Software Bill of Materials (SBOM)

SBOMs document all packages and dependencies in container images, enabling vulnerability tracking and license compliance:

• SBOM generation: Use Syft, Trivy, or Docker SBOM to generate SBOMs during builds
• Standard formats: SPDX and CycloneDX provide standardized dependency documentation
• Attestation storage: Store SBOMs as in-toto attestations alongside images for verification

# Generate SBOM with Syft
syft packages myimage:latest -o spdx-json > sbom.spdx.json

# Attach SBOM attestation with Cosign
cosign attest --predicate sbom.spdx.json --type spdxjson myimage:latest

Image Signing

Cryptographic image signing provides provable image integrity and provenance:

• Sigstore Cosign: Cosign provides keyless signing using OIDC identity
• Docker Content Trust: DCT uses Notary for image signing
• SLSA provenance: SLSA framework attestations document build provenance

# Sign image with Cosign (keyless)
cosign sign --yes myregistry.io/myimage:v1.0.0

# Verify signature before deployment
cosign verify myregistry.io/myimage:v1.0.0 \
  --certificate-identity=builder@example.com \
  --certificate-oidc-issuer=https://accounts.google.com

Registry Security

Registry security controls protect container images from unauthorized access, tampering, and ensure only scanned images are deployed.

Private Registries

Private container registries provide access control and audit logging for container images:

• Enterprise registries: Harbor, JFrog Artifactory, and cloud-native options (Amazon ECR, Google Artifact Registry, Azure Container Registry)
• Least privilege access: Separate credentials for push and pull operations with minimal required permissions
• Audit logging: Enable comprehensive logging for compliance and incident investigation

Content Trust and Verification

Content trust ensures that only signed images can be pushed to or pulled from registries:

• Notary v2: Notary Project provides OCI-native signing and verification
• Docker Content Trust: DCT enforces signature verification on pull
• Cosign verification: Integrate Cosign verification into CI/CD pipelines

Vulnerability Scanning

Vulnerability scanning identifies known CVEs before images reach production:

• Scan on push: Trivy, Grype, Clair, or registry-native scanners
• Blocking policies: Block pushes of images with critical or high-severity vulnerabilities
• Periodic rescanning: Rescan stored images to identify newly disclosed vulnerabilities
• CVSS thresholds: Configure severity thresholds per environment (stricter for production)

# Example: Trivy scanning in CI pipeline
- name: Scan image for vulnerabilities
  run: |
    trivy image --exit-code 1 --severity CRITICAL,HIGH \
      --ignore-unfixed myregistry.io/myimage:${{ github.sha }}

Admission Control and Policy Enforcement

Admission controllers act as the last line of defense before workloads run in the cluster, enforcing security policies at deployment time.

Signature and Provenance Verification

Admission controllers should verify image signatures before allowing pod creation:

• Sigstore Policy Controller: Policy Controller verifies Cosign signatures and attestations
• Kyverno image verification: Kyverno supports Cosign and Notary verification
• SLSA provenance: Verify SLSA attestations to ensure images were built by trusted systems

# Kyverno policy: require signed images
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "myregistry.io/*"
          attestors:
            - entries:
                - keyless:
                    issuer: "https://accounts.google.com"
                    subject: "builder@example.com"

Pod Security Standards

Pod Security Standards define three security levels for Kubernetes workloads:

Pod Security Admission enforces these standards at the namespace level:

# Enforce restricted security standard on namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Policy-as-Code

Policy-as-code enables version-controlled, testable security policies:

• OPA Gatekeeper: Gatekeeper uses Rego policies for Kubernetes admission control
• Kyverno: Kyverno provides Kubernetes-native policies in YAML
• Kubewarden: Kubewarden uses WebAssembly policies for portability

# OPA Gatekeeper: deny containers without resource limits
package k8srequiredresources

violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  not container.resources.limits.cpu
  msg := sprintf("Container '%v' must have CPU limits", [container.name])
}

Capability Restrictions

Linux capabilities should be explicitly managed to prevent privilege escalation:

• Drop all capabilities: Start with drop: ["ALL"] and add only required capabilities
• Avoid dangerous capabilities: Never grant CAP_SYS_ADMIN, CAP_NET_ADMIN, or CAP_SYS_PTRACE unless absolutely necessary
• Read-only root filesystem: Prevent container modification with readOnlyRootFilesystem: true

# Secure security context
securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL

Runtime Security

Runtime security provides the last layer of defense, detecting and preventing attacks against running containers.

System Call Filtering

Seccomp profiles restrict system calls available to containers:

• Default profiles: Kubernetes applies a default seccomp profile blocking dangerous syscalls
• Custom profiles: Generate application-specific profiles using Inspektor Gadget or Security Profiles Operator
• RuntimeDefault: Use RuntimeDefault seccomp profile as minimum baseline

Mandatory Access Control

AppArmor and SELinux provide mandatory access control:

• AppArmor profiles: Define allowed file access, network operations, and capabilities
• SELinux contexts: Enforce type enforcement policies for container isolation
• Profile enforcement: Use enforce mode in production, complain mode for testing

Runtime Detection

eBPF-based runtime detection identifies suspicious container behavior:

• Falco: Falco detects unexpected process execution, file access, and network connections
• Tetragon: Tetragon provides eBPF-based security observability and enforcement
• Sysdig Secure: Commercial runtime security with threat detection and response

# Falco rule: detect shell spawned in container
- rule: Terminal shell in container
  desc: Detect shell spawned in a container
  condition: >
    spawned_process and container and
    shell_procs and proc.tty != 0
  output: >
    Shell spawned in container (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING

Network Policies

Kubernetes Network Policies control pod-to-pod communication:

• Default deny: Start with deny-all policies and explicitly allow required traffic
• Namespace isolation: Prevent cross-namespace communication except where required
• Egress controls: Restrict outbound traffic to prevent data exfiltration

# Default deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Service Mesh Security

Service meshes provide additional network security:

• Mutual TLS: Istio, Linkerd, and Cilium encrypt all service-to-service traffic
• Authorization policies: Fine-grained access control based on service identity
• Traffic observability: Visibility into encrypted traffic for security monitoring

Feedback Loops and Continuous Improvement

Continuous improvement requires automated feedback loops that identify issues early and drive remediation.

Deployment Blocking

Configure vulnerability thresholds to block insecure deployments:

Exceptions should be time-bounded with documented compensating controls and automatic expiration.

Automated Remediation

Automation accelerates vulnerability remediation:

• Dependabot/Renovate: Dependabot and Renovate create automated PRs for base image updates
• Scheduled rebuilds: Rebuild images on a schedule (daily/weekly) to incorporate latest patches
• Auto-merge for patches: Automatically merge security patches after tests pass

Policy Testing

Policy test suites validate that policies work correctly:

• Unit tests: Test policies with known-good and known-bad inputs
• Integration tests: Validate policies against real workload manifests
• Staging validation: Test policy changes in non-production before production deployment

# Test Kyverno policies with kyverno CLI
kyverno apply policies/ --resource test-resources/

# Test OPA policies with conftest
conftest test deployment.yaml --policy policies/

Container Security Metrics

Track key metrics to measure container security program effectiveness:

Conclusion

Container security in DevOps requires defense-in-depth across build, registry, admission, and runtime stages. Security engineers design container security programs that integrate with DevOps workflows, providing automated security guardrails without blocking developer productivity.

Key success factors:

• Minimal, hardened base images with automated updates and vulnerability scanning
• Image signing and verification throughout the supply chain
• Policy-as-code enforcement at admission with Pod Security Standards
• Runtime detection and network policies for defense-in-depth
• Automated feedback loops with clear metrics and remediation SLAs

Organizations that invest in container security fundamentals build secure containerized applications while maintaining rapid deployment velocity. The CIS Kubernetes Benchmark and NIST SP 800-190 provide comprehensive guidance for implementing these controls.

Related Articles

• Container and Kubernetes Security - Runtime container security
• DevSecOps Pipeline Security - CI/CD pipeline security
• Software Supply Chain Security - Image supply chain
• Infrastructure as Code Security - Container IaC security
• Security Testing Automation - Automated container scanning

References

• NIST SP 800-190: Application Container Security Guide - Federal guidance on container security
• CIS Kubernetes Benchmark - Industry-standard Kubernetes hardening guide
• Kubernetes Security Best Practices - Official Kubernetes security documentation
• Sigstore - Keyless signing and verification for containers
• SLSA Framework - Supply chain security levels and attestations
• OWASP Docker Security Cheat Sheet - Container security best practices
• Falco - Cloud-native runtime security
• Kyverno - Kubernetes-native policy management
• OPA Gatekeeper - Policy-as-code for Kubernetes
• Trivy - Comprehensive vulnerability scanner for containers