Off-Cluster QEMU Provisioner

The qemu-ssh.jumpstarter.dev provisioner runs QEMU virtual targets on remote lab hosts outside the Kubernetes cluster. It connects to hosts via SSH and deploys containers using Podman quadlets, giving you the same container-based exporter + runtime pattern as in-cluster QEMU but on dedicated hardware with direct KVM access.

When to use this provisioner:

  • Your lab hosts have KVM-capable hardware or GPU passthrough not available in the cluster

  • You need bare-metal performance for emulation (e.g., automotive SoC targets)

  • You want to scale virtual target pools across multiple lab machines while keeping orchestration centralized in Kubernetes

Prerequisites

Remote lab hosts

Each remote host must have:

  • Podman installed (for running containers)

  • systemd (for quadlet-based container lifecycle)

  • SSH access with key-based authentication

  • (Optional) KVM support (/dev/kvm present) for hardware-accelerated emulation

Kubernetes cluster

The cluster must have:

  • Jumpstarter operator installed

  • The qemu-ssh.jumpstarter.dev provisioner enabled in the Jumpstarter CR

Step 1: Enable the provisioner

Add qemu-ssh.jumpstarter.dev to the exporterSets.provisioners list in your Jumpstarter CR:

apiVersion: operator.jumpstarter.dev/v1alpha1
kind: Jumpstarter
metadata:
  name: jumpstarter
  namespace: jumpstarter
spec:
  exporterSets:
    image: quay.io/jumpstarter-dev/exporter-set-controller:latest
    provisioners:
      - name: qemu.jumpstarter.dev
        enabled: true
      - name: qemu-ssh.jumpstarter.dev
        enabled: true

The operator creates a Deployment for the qemu-ssh controller automatically.

Step 2: Create SSH credentials

Create a Kubernetes Secret containing the SSH private key used to connect to your lab hosts:

kubectl create secret generic lab-ssh-key \
  --from-file=ssh-privatekey=$HOME/.ssh/id_ed25519 \
  -n jumpstarter

The Secret uses the standard kubernetes.io/ssh-auth type. The key must be in the ssh-privatekey field. The SSH username is configured in the VirtualTargetClass parameters (not in the Secret).

Step 3: Create a VirtualTargetClass

The VirtualTargetClass defines the pool profile — which host to use, SSH configuration, and default resource allocations:

apiVersion: virtualtarget.jumpstarter.dev/v1alpha1
kind: VirtualTargetClass
metadata:
  name: qemu-ssh-aarch64
  namespace: jumpstarter
spec:
  provisioner: qemu-ssh.jumpstarter.dev
  credentialsSecretRef:
    name: lab-ssh-key
  bindingMode: Immediate
  reclaimPolicy: Delete
  parameters:
    ssh:
      user: root
      port: 22
    host:
      name: lab-host-01.example.com
    runtime:
      kvm: true
    arch: aarch64
    resources:
      cpu: 4
      memory: 4Gi
    storage:
      size: 16Gi

Parameters reference

Parameter

Description

ssh.user

Default SSH username (fallback: root)

ssh.port

Default SSH port (fallback: 22)

host.name

FQDN or IP of the remote host

host.user

Per-host SSH user override

host.port

Per-host SSH port override

runtime.kvm

Pass /dev/kvm to the runtime container

runtime.devices

Additional devices to pass through (list of paths)

arch

Default QEMU architecture

resources.cpu

Default vCPU count

resources.memory

Default memory (e.g., 4Gi)

storage.size

Default disk size (e.g., 16Gi)

Step 4: Create an ExporterSet

Each ExporterSet manages instances on one host. The maxReplicas field controls how many concurrent instances run on that host. To use multiple hosts, create additional ExporterSets that reference the same VirtualTargetClass but override parameters.host:

apiVersion: virtualtarget.jumpstarter.dev/v1alpha1
kind: ExporterSet
metadata:
  name: aarch64-ssh-pool
  namespace: jumpstarter
spec:
  minReplicas: 0
  maxReplicas: 4
  minAvailableReplicas: 1
  scaleDownCooldown: 5m
  recycleStrategy: ExitAndReplace
  virtualTargetClassName: qemu-ssh-aarch64
  selector:
    matchLabels:
      board: aarch64-qemu
      virtual: "true"
  template:
    metadata:
      labels:
        board: aarch64-qemu
        arch: aarch64
        virtual: "true"
    spec:
      drivers:
        - name: qemu
          type: jumpstarter_driver_qemu.driver.Qemu

The tcp driver is auto-injected by the provisioner (SSH host-forwarding on port 2222).

How it works

When the ExporterSet scales up:

  1. The controller creates an Exporter CR (cluster-side)

  2. Once credentials are ready, it connects to the remote host via SSH

  3. Writes the exporter configuration YAML to /etc/jumpstarter/exporters/

  4. Creates Podman quadlet .container files under /etc/containers/systemd/ for both the runtime (QEMU) and exporter containers

  5. Creates a shared Podman volume for inter-container communication (QEMU sockets)

  6. Reloads systemd and starts the containers

  7. Annotates the Exporter CR with the host assignment

When scaling down or cleaning up:

  1. Stops and disables the systemd services

  2. Removes the quadlet files and exporter config

  3. Removes the shared Podman volume

  4. Reloads systemd

  5. Deletes the Exporter CR

Container layout on the remote host

Each exporter instance creates two containers:

  • <name>-runtime: Runs the QEMU emulator with access to /dev/kvm (if enabled) and the shared volume for communication sockets

  • <name>-exporter: Runs the Jumpstarter exporter, connecting back to the cluster controller and mounting the shared volume

Both containers communicate via Unix sockets on the shared Podman volume (/shared/launcher.sock).

Leasing targets

Users lease off-cluster targets the same way as any other target — the placement is transparent:

jmp lease -l board=aarch64-qemu,virtual=true

Troubleshooting

Check exporter-set controller logs

kubectl logs -l component=exporterset-controller,provisioner=qemu-ssh-jumpstarter-dev \
  -n jumpstarter -f

Check remote host containers

SSH into the lab host and inspect the Podman containers:

podman ps -a --filter "name=-(runtime|exporter)$"
systemctl status *-runtime *-exporter
journalctl -u <name>-runtime -u <name>-exporter

Check quadlet files

ls /etc/containers/systemd/*.container
cat /etc/jumpstarter/exporters/*.yaml

Common issues

  • SSH connection failures: Verify the SSH key in the Secret matches authorized_keys on the remote hosts

  • KVM not available: Ensure /dev/kvm exists on the host and the runtime.kvm parameter is set

  • Containers not starting: Check podman logs <name>-runtime and podman logs <name>-exporter on the remote host

  • Capacity full: Increase maxReplicas in the ExporterSet or create additional ExporterSets for more hosts