5 min read

DCGM Exporter Metrics on EKS, GKE or AKS: GPU Utilization and Memory in Skyportal

Skyportal
DCGM Exporter Metrics on EKS, GKE or AKS: GPU Utilization and Memory in Skyportal

DCGM exporter metrics already say what every GPU on your EKS, GKE or AKS cluster is doing: how busy it is, how much memory is taken, how hot it runs and which pod holds it. Reading them is the awkward part. You run Prometheus and Grafana for it, or you kubectl port-forward to the exporter pod on the right node and scroll through Prometheus text.

If the cluster is connected to Skyportal with a read-only token, Skyportal finds the exporter that the NVIDIA GPU Operator runs and reads each exporter pod through the Kubernetes API server every 20 seconds. The numbers show up on the cluster's Monitor Dashboard, and SARA, the agent in Skyportal's chat, answers from them. You don't install anything else in the cluster, and you don't need a Prometheus server or cloud account keys.

What Skyportal reads from the DCGM exporter

From each exporter pod, Skyportal keeps five series per GPU:

  • DCGM_FI_DEV_GPU_UTIL, GPU utilization in percent.
  • DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE, used and free memory in MiB. Skyportal adds the two to get the total.
  • DCGM_FI_DEV_GPU_TEMP, the temperature in Celsius.
  • DCGM_FI_DEV_POWER_USAGE, the power draw in watts.

It also keeps the labels that say which GPU a line belongs to: the GPU index, the model name, the node, and the pod and namespace using the GPU. The GPU Operator turns that pod mapping on in the exporter, so Skyportal can tell you which pod holds a GPU. A GPU with no pod on it shows its node instead.

Those are the names dcgm-exporter uses up to version 4.8.4, the one GPU Operator v26.7.2 ships. Version 4.9.0, released on November 6, 2026, renames them and changes units: utilization becomes a 0 to 1 ratio in DCGM_FI_DEV_GPU_UTIL_RATIO, and memory is reported in bytes. Skyportal doesn't read the new names yet. If you pick the exporter version yourself, stay on 4.8.4 for now.

How Skyportal finds the exporter

Skyportal looks for a Service labeled app=nvidia-dcgm-exporter in the gpu-operator namespace, then in kube-system. That is the Service the GPU Operator creates, in the namespace NVIDIA's install command uses. To check that yours matches:

kubectl get svc,pods -n gpu-operator -l app=nvidia-dcgm-exporter

You should see one Service and one exporter pod for each GPU node. Skyportal reads /metrics on port 9400 of every one of those pods, all at the same time. It goes pod by pod because the Service would hand each request to a single pod, and you would only ever see one node's GPUs.

A DCGM exporter installed another way isn't found today. The standalone dcgm-exporter Helm chart, for example, labels its pods app.kubernetes.io/name=dcgm-exporter, and Skyportal doesn't look for that label.

The role the token needs

Finding the Service and listing its pods is covered by the view role from the read-only access post. Reading a pod's /metrics through the API server takes get on pods/proxy. If you added that post's GPU role, the token has it already. Otherwise, add it:

kubectl create clusterrole skyportal-gpu-metrics --verb=get \
  --resource=pods/proxy
kubectl create clusterrolebinding skyportal-gpu-metrics \
  --clusterrole=skyportal-gpu-metrics \
  --serviceaccount=skyportal:skyportal

Know what this grants. pods/proxy lets the token send GET requests to any HTTP port of any pod in the cluster, not only the exporter's. It can't read Secrets, but a debug page or an internal dashboard with no login of its own is within reach. Skyportal's own readings only use it on the exporter pods, so you can keep it to their namespace with a Role instead:

kubectl create role skyportal-gpu-metrics -n gpu-operator --verb=get \
  --resource=pods/proxy
kubectl create rolebinding skyportal-gpu-metrics -n gpu-operator \
  --role=skyportal-gpu-metrics \
  --serviceaccount=skyportal:skyportal

What the dashboard shows for each GPU

Open the cluster's Monitor Dashboard. The Infrastructure view has one row per GPU: node, index, model, memory used and free, utilization, power, temperature and the pod using it. The tiles above the table give the peak GPU memory, the average and highest utilization, how many GPUs are idle (under 5%) and the smallest amount of free memory on any GPU.

The Workloads view lists your workloads with their GPUs, GPU memory, utilization and power. It marks a workload CPU-fallback when its pods sit on a GPU node with no GPU assigned to them, or when its GPUs stay under 5% while its CPU is at 60% or more. The label is meant for the job that was supposed to use the GPU and is running on the CPU instead.

The table and the tiles come from a reading at most 15 seconds old. The charts next to them only gain a point when the dashboard is open or SARA takes a live reading, so they are often thin. The full history is stored separately, and SARA reads it.

Ask SARA about your GPUs

Select the cluster in the chat and ask:

  • "Which GPUs are busy right now, and which pods are using them?"
  • "How much GPU memory is free on this cluster?"
  • "How busy were the GPUs over the last 24 hours?"

For right now, SARA reads the cluster again, through a cache of up to 15 seconds, and gets one line per GPU, up to eight: the index, the model, the pod or node, utilization, memory used out of the total, watts and temperature. For a window of time, it reads the stored readings instead, up to the last 7 days, and reports the average, highest and lowest utilization across all GPUs. That history isn't split per GPU. None of this waits for your approval, because SARA only reads.

One number is off today. SARA shows a GPU at exactly 1% as 100%, both in the live answer and in the history. The dashboard has the right value, so check the table when an idle GPU looks busy in the chat.

A busy GPU doesn't tell you how the model server on it is coping. If you serve with vLLM, the vLLM metrics post covers KV cache usage, the queue and preemptions.

When SARA finds no GPU metrics

SARA says GPU metrics are not available for several different problems, and Skyportal can't tell them apart: no exporter matched, the token lacks pods/proxy, or the exporter is on 4.9.0 with the new names. To see which one you have, make the same read as the service account, from your own admin kubeconfig:

POD=$(kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter \
  -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw \
  "/api/v1/namespaces/gpu-operator/pods/$POD:9400/proxy/metrics" \
  --as=system:serviceaccount:skyportal:skyportal \
  | grep -m 4 '^DCGM_FI_DEV_GPU_UTIL'

An error from the first command means no exporter pod is where Skyportal looks. Forbidden means the role is missing. Lines that start with DCGM_FI_DEV_GPU_UTIL{ are the ones Skyportal reads. Lines that start with DCGM_FI_DEV_GPU_UTIL_RATIO mean the exporter is on 4.9.0 or later.

Get started

Connect the cluster with a read-only token first. The read-only access post has every command for EKS, GKE and AKS. If the GPU Operator already runs there, add the pods/proxy role above, open the Monitor Dashboard and ask SARA which GPUs are busy. The Free plan is enough to try it. Sign up here, and see the Kubernetes guide in the docs for the rest of what Skyportal does with a cluster.