You serve a model with vLLM on EKS, GKE or AKS, and one afternoon the responses get slow. vLLM already knows why. Its /metrics endpoint says how full the KV cache is, how many requests are waiting for a slot, and whether it has started preempting requests to make room. Getting at those numbers usually means running Prometheus and Grafana next to the model, or a kubectl port-forward and a long scroll through Prometheus text.
If the cluster is connected to Skyportal with a read-only token, Skyportal finds your vLLM Services and reads their /metrics through the Kubernetes API server every 20 seconds. SARA, the agent in Skyportal's chat, answers from those readings. You don't install anything in the cluster, and you don't need a Prometheus server or cloud account keys.
What Skyportal reads from vLLM
From each vLLM Service it finds, Skyportal keeps these series:
vllm:kv_cache_usage_perc, the share of the KV cache in use. Close to 100%, new requests have to wait or push running ones out.vllm:num_requests_runningandvllm:num_requests_waiting, the batch vLLM is working on and the queue behind it.vllm:num_preemptions_total, how many times vLLM took a running request off the GPU because the KV cache ran out. A preempted request waits again, and vLLM redoes the work it lost.
It also stores the prompt and generation token counters, vllm:request_success_total, and the sums and counts of the time-to-first-token, inter-token and end-to-end latency histograms. It skips the histogram buckets, so there are no p95 or p99 numbers, and SARA's answers don't cover latency.
Older servers work too. Skyportal reads the vllm_ prefix some builds use, and the names from before two renames: gpu_cache_usage_perc became kv_cache_usage_perc in vLLM 0.9.2, and time_per_output_token_seconds became inter_token_latency_seconds in 0.10.2.
How Skyportal finds the vLLM Service
Skyportal lists the Services in every namespace and takes the ones with one of these labels: app=vllm, app.kubernetes.io/name=vllm, component=vllm or app.kubernetes.io/component=vllm-openai. A Service without them still counts if "vllm" is part of its name or its namespace. If yours matches neither, label it:
kubectl label service my-model -n serving app=vllmFor the port, Skyportal takes the one named metrics, then one named http, then 8000, then the first port listed. vLLM serves /metrics on the same port as its API, 8000 by default, so a plain Service in front of it works as it is.
Skyportal sends one request per Service each time. With three replicas behind one Service, each reading comes from whichever replica answers, so the numbers describe one replica, not all three.
The role the token needs
Reading a Service's /metrics through the API server takes get on services/proxy. If you set up the token with the read-only access post and added its GPU role, the token has it already. Otherwise, add it:
kubectl create clusterrole skyportal-vllm-metrics --verb=get \
--resource=services/proxy
kubectl create clusterrolebinding skyportal-vllm-metrics \
--clusterrole=skyportal-vllm-metrics \
--serviceaccount=skyportal:skyportalKnow what this grants. services/proxy lets the token send GET requests to any Service in the cluster, not only vLLM's. It can't read Secrets, but an internal dashboard with no login of its own is within reach. To keep it to the namespace where vLLM runs, create a Role there instead:
kubectl create role skyportal-vllm-metrics -n serving --verb=get \
--resource=services/proxy
kubectl create rolebinding skyportal-vllm-metrics -n serving \
--role=skyportal-vllm-metrics \
--serviceaccount=skyportal:skyportalFinding the Service needs nothing more. The view role from that post already lets the token list Services.
Ask SARA about vLLM
Select the cluster in the chat and ask:
- "How full is the KV cache on the vLLM server right now?"
- "Are requests queuing in vLLM?"
- "Did vLLM preempt anything in the last 24 hours?"
For right now, SARA reads the cluster again, through a cache of up to 15 seconds, and gets one line per vLLM Service, up to four: the namespace and Service, the model name, KV cache usage, preemptions so far, and the requests running and waiting. For a window of time, it reads the stored readings instead, up to the last 7 days, and reports the peak KV cache usage and how much the preemption count grew.
Those two numbers together tell you what kind of slow you have. A full KV cache with no new preemptions means the server is busy and coping. Preemptions going up means vLLM is evicting requests and redoing their work, and the latency your users see goes up with them. None of this waits for your approval, because SARA only reads.
When SARA finds no vLLM metrics
SARA reports that it found no vLLM metrics for several different problems, and Skyportal can't tell them apart: no Service matched, the token lacks services/proxy, or /metrics asks for a login or only speaks HTTPS. To see which one you have, make the same read as the service account, from your own admin kubeconfig:
kubectl get --raw \
/api/v1/namespaces/serving/services/my-model:8000/proxy/metrics \
--as=system:serviceaccount:skyportal:skyportal | grep -m 5 '^vllm:'Forbidden means the role is missing. Lines starting with vllm: mean the read works, and the Service needs one of the labels above or "vllm" in its name.
If only VictoriaMetrics scrapes your vLLM pods, Skyportal can ask it instead. When no vLLM Service answers, it looks for the VictoriaMetrics query service (vmsingle or vmselect), preferring port 8428, and asks it for every vLLM series through the same proxy, so the token needs services/proxy in that namespace too. Those numbers add the replicas together, except KV cache usage, which is averaged across them.
Get started
Connect the cluster with a read-only token first. The read-only access post has every command for EKS, GKE and AKS. Then add the services/proxy role above, select the cluster in the chat and ask about vLLM. The Free plan is enough to try it. Sign up here, and see the Kubernetes guide in the docs for the rest of what Skyportal does with a cluster.