Nothing Runs Until You Say So: Inside SkyPortal's Agent Guardrails
An AI agent that can read your infrastructure is a convenience. An agent that can change your infrastructure is a decision you have to be able to defend on Monday morning.
That distinction has driven most of what we shipped since the SkyPortal SDK and CLI launch. The hard problem in AI ops is not getting a model to suggest systemctl restart nginx. It is building the machinery around that suggestion so a platform team can hand an agent real credentials and still answer four questions: what is it allowed to do, who said yes, how do we know it actually worked, and what did it see along the way.
Here is how SkyPortal answers each one.
Approval is a gate, not a suggestion
Every command the agent wants to run passes through an ordered chain of checks before it reaches a host. The order runs from cheapest to strictest: a repo deny list, then read-only classification, then sensitive-file rules, then your permit list of pre-cleared commands, then any checkpoint the current plan already established, and only then does the agent stop and ask for a fresh approval.
The ordering exists so routine investigation stays fast. Asking the agent to check disk usage, tail a log, or describe a pod produces no approval prompt, because none of it changes anything. The moment a command is classified as destructive, the workflow pauses and waits for a human.
Two properties of that chain matter more than the chain itself. Every rule lives in one place, so the batch preflight and single-command enforcement can never disagree — there is a test whose only job is to pin those two paths together. And a permit-list entry does not override everything above it: a kubectl exec, a secrets read, or an RBAC change forces a fresh approval even when the command is on your allow list, because the filesystem-path detectors that catch ~/.ssh are structurally blind to kubectl.
Approve from wherever you already work
Approvals are not trapped in the web UI. The CLI exposes the whole lifecycle, which is what makes the agent usable from a terminal, a runbook, or CI:
$ skyportalai chat send "nginx is 502ing on web-01, fix it" \
--server 42 --wait
$ skyportalai chat status 118
status: awaiting_approval
pending_approvals:
- approval_id: a41f9c
type: bash_command
command: systemctl restart nginx
$ skyportalai chat approve 118 a41f9c --type bash_command
Approval a41f9c: approved
--type takes bash_command for a single command or plan for a whole proposed plan, so you can review at whichever altitude you care about. skyportalai chat reject records a --reason that goes back to the agent instead of leaving it at a dead end, and skyportalai chat cancel stops an in-flight workflow outright.
Automating approvals without losing the audit trail
Plenty of teams want an agent that runs unattended over a known-safe class of work. The naive way to build that is a flag that skips the approval system. We deliberately did not build that.
Setting the permission mode to autoapprove still submits every concrete approval through the same audited endpoint, one at a time:
from skyportalai import Skyportal
with Skyportal(api_key="sk-...") as client:
client.set_permission_mode("autoapprove")
chat = client.chat.create_chat(
"What changed before GPU utilization dropped?",
server_id=12,
)
result = chat.wait()
print(result.status)
Autoapprove decides who answers the prompt, not whether the prompt happens. Each approval still lands in the audit trail with its checkpoint, and it does not bypass read-only environments, server scope, repository deny lists, or any other backend policy. Those remain separate hard gates, re-validated server-side at the moment of decision — if autoapproval has since been turned off for the account, a client still holding a stale marker gets a 409, not an execution.
If you want per-action logic instead of a blanket mode, pass a callback. It takes precedence over the account setting, and returning None leaves that approval pending for a human:
result = chat.wait(on_approval=lambda approval: approval.type == "bash_command")
Three properties of that wait loop only matter under load, which is exactly why they are worth stating. If the permission setting cannot be read, the wait fails closed and hands back the awaiting-approval status instead of assuming yes. An approval ID already submitted during a wait is never submitted twice, even when a status response is stale. And approvals are handled strictly in checkpoint order, because resuming a plan out of order is its own category of outage.
"Done" means verified, not claimed
A plan that reports success because the model believes it succeeded is not worth much. Plans in SkyPortal can carry an acceptance condition — one checkable sentence, like "kubectl get nodes reports 2 nodes, all Ready" — and when they do, finishing the last step is not the end of the plan.
Instead the agent opens a final verification step whose rules are deliberately narrow. It must gather fresh evidence using read-only checks (status, list, get, describe), and it is explicitly forbidden from relying on earlier steps' output. If the fresh evidence shows the condition is unmet, that is a terminal result for the check, reported as exactly what is unmet.
The important half of that constraint is the second one: the verification step is not allowed to change state to make its own condition true. An agent that can fix what it is measuring cannot be trusted to measure it. So the verifier only ever looks, and a separate judge rules on whether the fresh output actually satisfies the condition.
Which leaves the obvious attack on your own system: what stops the agent from simply saying it verified? Three independent signals decide a step is done — a zero exit code, the judge's verdict, and a check that a command was actually dispatched during that turn, read from the turn's own execution record. The model's narration of its own success is deliberately not one of them. A check marked complete with no command behind it gets rejected with a message saying so, and the step goes back to running.
Failure here pauses the plan rather than failing it. You get told which condition could not be confirmed and why, and you can fix the underlying problem and ask for a re-check — the re-check is explicitly instructed to re-run the same read-only command and judge it on the fresh output alone, which is the one case where repeating yourself is correct.
There is a related guard for a different failure mode. If the last ten observations in a turn are identical, the agent is spinning — same tool, same error — and the run is cut off rather than left to burn its budget rediscovering the same wall.
Secrets do not reach the model, and do not reach the transcript
Command output is the leakiest surface in an ops agent. A single cat of the wrong config file puts a private key into the conversation, and from there into the model's context and your chat history permanently.
So redaction happens at the funnel where a tool result becomes a model observation, before that text is ever handed to a provider. The stdout, stderr, and error fields are scrubbed — a PEM block becomes [REDACTED PRIVATE KEY], password=SuperSecret99x becomes password=[REDACTED], kubeconfig token and client-key fields keep their names and lose their values — while control-plane fields like the command itself are left intact, so redaction never corrupts the agent's own bookkeeping.
The ordering there is load-bearing in a way that is easy to get wrong: redaction runs before truncation. A PEM block that has already been truncated no longer matches BEGIN...END, so doing it the other way around would quietly let the interesting half of a private key through. There is a test that builds an over-long key specifically to pin that order.
The outbound direction is screened too. Assistant messages pass an egress gate before they are persisted or broadcast, combining a deterministic secret and system-prompt-leak scan with an LLM judge for the ambiguous middle. The fail postures are intentionally asymmetric: the deterministic tier fails closed, because shipping an unscanned response is the failure worth preventing, while a judge timeout degrades to the deterministic verdict so a slow model never withholds a legitimate answer. And when the gate redacts, it re-scans its own output — if a secret survived redaction, the response is withheld instead of shipped.
Two things are deliberately not redacted. IP addresses, because the agent needs real addressing to reason about hosts and networking. And the live terminal stream, because that is a human looking at their own server.
Cancel means cancelled
Cancellation gets treated as a safety property rather than a convenience. The executor checks for cancellation at every iteration boundary, from an in-memory flag and a Redis key, and if the Redis client has gone away during teardown the workflow treats itself as cancelled and exits gracefully rather than continuing blind.
The subtler work was scoping. Each turn mints a token at start and persists it in the same write that marks the turn as processing; a cancel request stamps that token as the flag's value. A leftover flag from an earlier turn therefore cannot be read as a cancel of a later one — the specific bug being that a healthy, successful turn would report itself cancelled because a previous turn's marker had never been cleared. Conversely, a real cancel is no longer erased by the very turn it was cancelling.
Cancelling also unblocks every pending approval, closes out pending checkpoints so none are orphaned in the database, and writes one durable line into the conversation:
----Agent stopped at your request----
Scope is an allowlist
When a chat spans several hosts, the server list is a boundary, not a hint:
chat = client.chat.create_chat(
"Compare GPU health on all selected hosts",
server_ids=[12, 18],
active_server_id=12,
selected_namespaces={18: ["default", "vllm"]},
)
The active server handles anything ambiguous, and the agent fans out across the full set only when the prompt explicitly targets all selected hosts. An agent given two hosts cannot wander onto a third, and a Kubernetes scope of default and vllm means those namespaces and no others.
A fan-out that needs approval produces one consolidated request covering every target, and any rejection blocks all of them — there is no partial-yes that leaves your fleet in mixed states. Concurrent batches are capped at sixteen commands, and a per-host permit check that raises an exception counts as "needs approval" rather than "probably fine".
This matters more than it sounds. The most common way ops automation causes an incident is not a wrong command — it is a right command aimed at the wrong machine.
The same rules on SSH and Kubernetes
SkyPortal drives plain SSH hosts and Kubernetes clusters through one funnel. The tool, the scope check, and the security gates all sit above the point where the two paths diverge; only genuinely transport-specific work — SSH credentials versus a kubeconfig, a remote shell versus a local kubectl — happens below it. Even a single-host command aimed at a non-active server is routed through the fan-out path on purpose, so it cannot skip a gate the broadcast path owns.
On the Kubernetes side that funnel ends somewhere stricter than a shell. Commands are tokenized and the binary must be exactly kubectl or helm, which is what makes sudo, pipelines, and shell wrapping unrepresentable rather than merely denied. Namespace scope is enforced by injecting -n into the argument vector before the approval gate runs, so the command you approve is the namespaced command that executes. And --all-namespaces requires a live namespace listing rather than a cached one, failing closed to a refusal if that read fails — a cached list can undercount, and undercounting here means over-granting.
That last set of choices is architecture in service of a safety property. If approval logic were implemented once per transport, the two copies would drift, and a gate that holds on your SSH fleet would quietly not hold on your clusters. We have fixed bugs of exactly that shape, which is why keeping shared logic above the fork is now a reviewed rule rather than a convention.
Get started
The CLI, the interactive terminal, and the Python SDK are open source and need Python 3.11 or newer:
git clone https://github.com/SkyportalAi/skyportalai.git
cd skyportalai
./run.sh
Run /login once, list your fleet with /servers, select hosts with /server <id>, and check where your approvals stand with /permission. Source and issues live at github.com/SkyportalAi/skyportalai.
Guardrails are not what you bolt on once the agent works. They are the reason it gets to touch production at all.