Posted on :: Tags: , , , , , , DRAFT

Want to see Kubernetes Image Volumes in action? This post walks through a complete working demo that shows how to mount OpenTelemetry agents from container registries instead of baking them into application images.

What You'll Build

By the end of this demo, you'll have:

  • A Kind cluster with Image Volumes enabled (Kubernetes v1.33 beta feature)
  • OpenTelemetry Java agent packaged as an OCI artifact
  • Spring Boot app with auto-injected observability
  • Aspire Dashboard visualizing metrics and traces

All without modifying a single application Dockerfile!

Prerequisites

# Required tools
- Docker
- kind
- kubectl
- curl
- oras (for OCI artifacts)

Step-by-Step Setup

1. Clone and Setup

git clone https://github.com/dol/k8s-oci-volume-source-demo
cd k8s-oci-volume-source-demo

2. Build Custom Kind Environment

Since Image Volumes require containerd v2.1.0+, we need a custom Kind image:

# This builds a Kind node with the right containerd version
./01-build-custom-kind-image.sh

What it does:

  • Clones Kind v0.27.0 source
  • Builds custom base image with containerd v2.1.0-rc.0
  • Downloads Kubernetes v1.33.0 binaries
  • Creates Kind node image with Image Volume support

3. Create Cluster with Registry

# Creates Kind cluster + local registry
./02-kind-with-registry.sh

Key features enabled:

  • ImageVolume feature gate activated (disabled by default in beta)
  • Local registry at localhost:5001
  • Proper containerd configuration for Image Volumes

4. Package OpenTelemetry Agent

Here's where the magic happens - converting a JAR file into a mountable OCI image:

# Downloads OpenTelemetry agent and packages it as OCI artifact
./03-artifact-javaagent-upload.sh

The process:

  1. Downloads OpenTelemetry Java agent v2.15.0
  2. Creates reproducible tar layer with proper timestamps
  3. Builds OCI image with config.js containing rootfs and diff_ids
  4. Pushes to local registry using ORAS

Key challenge solved: ORAS tools struggle with single-file OCI creation, so the script manually constructs the OCI layout with proper metadata.

5. Deploy Spring Boot Application

# Deploys app with OCI volume-mounted agent
./04-deploy-spring-hello-world.sh

The deployment manifest shows the key pattern:

volumeMounts:
- name: otel-agent
  mountPath: /opt/opentelemetry
  readOnly: true
volumes:
- name: otel-agent
  image:
    reference: localhost:5001/opentelemetry-javaagent:v2.15.0

The Java process starts with: -javaagent:/opt/opentelemetry/opentelemetry-javaagent.jar

6. Add Observability Dashboard

# Deploys Aspire Dashboard for telemetry visualization
./04a-deploy-aspire-dashboard.sh

Bonus feature: The demo includes Microsoft's Aspire Dashboard, which provides an excellent view of OpenTelemetry data without requiring complex observability stack setup.

What Makes This Work

New Beta Features

Kubernetes v1.33 introduces several enhancements for Image Volumes:

  • subPath Support: Mount specific subdirectories from Image Volumes
  • Enhanced Metrics: New kubelet metrics for monitoring volume operations:
    • kubelet_image_volume_requested_total: Number of requested Image Volumes
    • kubelet_image_volume_mounted_succeed_total: Successful mounts
    • kubelet_image_volume_mounted_errors_total: Failed mounts

containerd Modifications

The key technical requirement is containerd v2.1.0+ with Image Volume support. Standard Kind images use older containerd versions, hence the custom build.

Kubernetes 1.33 Beta API

The ImageVolume feature gate enables the new volume type (still disabled by default in beta):

featureGates:
  "ImageVolume": true

Kind Local Testing

Kind provides the perfect environment for testing this beta feature:

  • Isolated from production clusters
  • Easy registry integration
  • Fast iteration cycles

ORAS Tool Challenges

The demo works around current ORAS limitations when creating single-file OCI images. The script manually creates the OCI layout with proper config.js structure including rootfs and diff_ids.

Testing the Results

After running all scripts, you'll see output like:

Spring Hello World application is accessible at: http://172.18.0.3:30080
Aspire Dashboard is accessible at: http://172.18.0.3:30888

Visit both URLs to see:

  • Your Spring Boot app running with OpenTelemetry
  • Real-time metrics, traces, and logs in Aspire Dashboard

Key Benefits Demonstrated

  1. Clean Separation: Application image contains only application code
  2. Version Independence: Update observability without rebuilding apps
  3. Consistent Instrumentation: Same agent version across all services
  4. Registry Integration: Standard OCI tools for agent distribution

Cleanup

# Removes everything created by the demo
./05-cleanup.sh

This removes:

  • Kind clusters and registry
  • Downloaded artifacts
  • Docker images
  • Temporary files

Technical Deep Dive

OCI Image Structure

The created OpenTelemetry artifact follows OCI image specification:

{
  "architecture": "amd64",
  "os": "linux", 
  "rootfs": {
    "type": "layers",
    "diff_ids": ["sha256:..."]
  }
}

Volume Mount Behavior

When Kubernetes sees an image volume, it:

  1. Pulls the OCI image to the node
  2. Extracts layers to a temporary directory
  3. Mounts the directory as a read-only volume
  4. Makes content available to the container

Reproducible Builds

The artifact creation uses reproducible build techniques:

  • Sorted tar entries
  • Clamped timestamps
  • Consistent ownership and permissions
  • Deterministic compression

Next Steps

This demo provides a foundation for exploring Image Volumes with:

  • Different types of agents and tools
  • Production registry integration
  • GitOps workflows
  • Multi-cluster deployments

The feature is now in beta status in Kubernetes v1.33, making it an excellent time to experiment and prepare for production adoption.


Complete source code and detailed scripts available at [{{ repository }}]({{ repository }})