Posted on 2025-05-15 :: Tags: kubernetes , image-volumes , observability , oci , containers , platform-engineering , devops , kind :: Source Code DRAFT

Kubernetes Image Volumes: Mounting OCI Images as Volumes

Kubernetes v1.33 introduces Image Volumes as a beta feature, allowing you to mount container images directly as read-only volumes in pods. This opens up new possibilities for delivering observability agents, configuration, and other dependencies without modifying application containers.

📁 Complete Demo: All code is available in the k8s-oci-volume-source-demo repository.

What Are Image Volumes?

Image Volumes let you reference OCI images as volume sources in Kubernetes pods:

volumes:
  - name: otel-agent
    image:
      reference: localhost:5001/opentelemetry-javaagent:v2.15.0

The feature graduated to beta in Kubernetes v1.33, adding subPath support and kubelet metrics (kubelet_image_volume_*). While beta, it's still disabled by default and requires containerd v2.1.0+ support.

Use Case: Auto-Instrumenting Java Apps

We'll demonstrate mounting an OpenTelemetry Java agent from an OCI image to auto-instrument a Spring Boot application without touching the application container.

Setting Up the Environment

1. Kind Cluster with containerd v2.1.0

Since standard Kind clusters use older containerd versions, we need a custom build:

# Build custom Kind image with containerd v2.1.0
git clone https://github.com/kubernetes-sigs/kind.git
cd kind && git checkout v0.27.0
cd images/base
make quick EXTRA_BUILD_OPT="--build-arg CONTAINERD_VERSION=v2.1.0" TAG=oci-source-demo

# Build node image with Kubernetes v1.33
curl -L "https://dl.k8s.io/v1.33.0/kubernetes-server-linux-amd64.tar.gz" -o k8s.tar.gz
kind build node-image ./k8s.tar.gz --base-image gcr.io/k8s-staging-kind/base:oci-source-demo

2. Cluster with Local Registry

Create a cluster with the ImageVolume feature gate enabled:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  "ImageVolume": true

Creating OCI Artifacts: The Challenge

The biggest technical challenge is creating proper OCI images. Standard tools like Docker create images with multiple layers and complex configurations. For Image Volumes, you need minimal, single-layer images.

The config.json Problem

OCI images require a config.json with rootfs.diff_ids matching the uncompressed layer SHA256:

{
  "architecture": "amd64",
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": ["sha256:UNCOMPRESSED_LAYER_SHA256"]
  }
}

ORAS Tool Limitations

ORAS simplifies OCI artifact creation but doesn't handle the config complexity automatically. You must:

Create a tar layer from your files
Calculate the SHA256 of the uncompressed tar
Compress the tar (gzip)
Generate config.json with the uncompressed SHA256
Push using ORAS

Example script excerpt:

# Create reproducible tar layer
tar c -f layer.tar -C agent-dir --sort=name --format=posix \
  --owner=0 --group=0 --numeric-owner --mode=0444 \
  --clamp-mtime --mtime=0 opentelemetry-javaagent.jar

# Calculate diff_id (uncompressed SHA256)
DIFF_ID="$(sha256sum layer.tar | head -c 64)"

# Compress layer
gzip --best --no-name layer.tar

# Create config with diff_id
printf '{"architecture":"amd64","os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:%s"]}}' \
  "${DIFF_ID}" > config.json

# Push with ORAS
oras push --disable-path-validation \
  --config "config.json:application/vnd.oci.image.config.v1+json" \
  --oci-layout "layout:latest" \
  "layer.tar.gz:application/vnd.oci.image.layer.v1.tar+gzip"

Deployment Configuration

The Kubernetes deployment mounts the OCI image as a volume:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-hello-world
spec:
  template:
    spec:
      containers:
        - name: app
          image: springio/hello-world:0.0.1-SNAPSHOT
          env:
            - name: JAVA_TOOL_OPTIONS
              value: -javaagent:/mnt/javaagent/opentelemetry-javaagent.jar
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://aspire-dashboard-service.aspire-dashboard:4317
          volumeMounts:
            - name: otel-agent
              mountPath: /mnt/javaagent
              readOnly: true
      volumes:
        - name: otel-agent
          image:
            reference: localhost:5001/opentelemetry-javaagent:v2.15.0

Local Testing with Kind

The complete setup involves:

Custom Kind build - Ensuring containerd v2.1.0 support
Local registry - For hosting OCI artifacts locally
Feature gate - Enabling ImageVolume=true
Registry configuration - Configuring containerd to use the local registry

Bonus: Aspire Dashboard

Deploy the .NET Aspire Dashboard to visualize telemetry:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aspire-dashboard
spec:
  template:
    spec:
      containers:
        - name: aspire-dashboard
          image: mcr.microsoft.com/dotnet/aspire-dashboard:9.0
          ports:
            - containerPort: 18888  # Dashboard UI
            - containerPort: 4317   # OTLP gRPC