Agentic DevOps Extravaganza — K8sGPT × Robusta × LLM

The stack

Two agents.
One broken cluster.
The LLM of your choice.

We stood up a mock Kubernetes API server in Python — real REST API, real discovery, real read/write verbs over TLS, serving a deliberately broken payment-prod namespace — and pointed the real binaries at it. K8sGPT runs its analyzers; a small reference agent handles the alert-driven loop that Robusta performs. Explanations go through llm_proxy.py, which bridges k8sgpt's customrest protocol to any OpenAI-compatible backend. Real Robusta itself runs in the separate kind/ path, against a real cluster.

Why these two?

Of the nine products in the brief, only K8sGPT and Robusta are open source and self-hostable. The rest are commercial SaaS (Datadog, PagerDuty, Cleric, Snyk, Torq, Tines, Workato, GitHub Copilot Autofix). They were skipped per the brief's "if not open source, skip" rule.

Prometheus + Alertmanager

observability

Emits PaymentAPIHighErrorRate — a JSON fixture in this path; a real PrometheusRule in kind/

↓ webhook

alert_triage_agent.py

reference impl of Robusta's loop

Receives alert → queries cluster → calls LLM → renders Slack card (posting is opt-in)

↓ /api/v1/* (kubectl)

Kubernetes API (mock)

payment-prod ns

2 nodes (1 DiskPressure) · 2 broken Deployments · 2 failing Pods · 1 dangling Ingress · 1 endpoint-less Service · 1 Pending PVC

↓ query state

K8sGPT

SRE agent

Runs its Go analyzers → emits 8 findings → optional LLM explain

↓ customrest → OpenAI-compat

llm_proxy.py → your backend

ollama · openai · openrouter · zai · custom · replay

Translates k8sgpt prompts into chat completions, returns explanations

K8s

K8sGPT

v0.4.36

An SRE agent that runs as a controller inside your cluster. It queries the Kubernetes API, runs structured analyzers in Go, and uses an LLM only for the natural-language explanation — making it deterministic and safe for production.

License Apache 2.0

Repo github.com/k8sgpt-ai/k8sgpt

Stars 8.4k+

Language Go

LLM Backend customrest → llm_proxy.py

Analyzers 14 active

Demo 1 · Cluster triage

Eight findings, no LLM involved.

The real k8sgpt v0.4.36 binary scans the payment-prod namespace over the Kubernetes REST API — served here by scripts/mock_k8s_server.py, a simulator the binary treats like any other kube-apiserver. Every error below comes from a real Go analyzer; the LLM is not involved at this stage.

z@devops-extravaganza: ~/demo REAL OUTPUT

k8sgpt analyze — text output (captured) CLI

terminal output

$ k8sgpt analyze --kubecontext mock-context --no-cache -n payment-prod
AI Provider: AI not used; --explain not set

0: Node worker-3()
- Error: worker-3 has condition of type DiskPressure, reason KubeletHasNoDiskSpace: kubelet has disk pressure

1: PersistentVolumeClaim payment-prod/payment-data-pvc()
- Error: storageclass.storage.k8s.io "standard" not found

2: Service payment-prod/payment-api-svc()
- Error: Service has no endpoints, expected label app=payment-api-frontend

3: Ingress payment-prod/payment-ingress()
- Error: Ingress uses the ingress class nginx which does not exist.
- Error: Ingress uses the service payment-prod/payment-frontend which does not exist.

4: Pod payment-prod/payment-api-7c4f5b-x9qkl(Deployment/payment-api)
- Error: the last termination reason is Error container=api pod=payment-api-7c4f5b-x9qkl

5: Pod payment-prod/payment-worker-6d8b2c-p3mnr(Deployment/payment-worker)
- Error: the last termination reason is OOMKilled container=worker pod=payment-worker-6d8b2c-p3mnr

6: Deployment payment-prod/payment-api()
- Error: Deployment payment-prod/payment-api has 1 replicas but 0 are available with status running

7: Deployment payment-prod/payment-worker()
- Error: Deployment payment-prod/payment-worker has 1 replicas but 0 are available with status running

Demo 2 · AI root-cause analysis

Same scan, with an LLM writing the diagnosis.

Passing --explain sends each finding to whichever backend you configured through llm_proxy.py — ollama (the credential-free default), openai, openrouter, zai, any custom OpenAI-compatible endpoint, or replay, which answers offline from responses captured earlier in captured/llm_cache.json. The model produces a step-by-step remediation plan; the findings it explains still come from the deterministic analyzers.

z@devops-extravaganza: ~/demo LLM-EXPLAINED

k8sgpt analyze --output json — captured excerpt JSON

raw JSON from k8sgpt (2 of 8 results shown)

{
  "provider": "",
  "errors": null,
  "status": "ProblemDetected",
  "results": [
    {
      "kind": "Deployment",
      "name": "payment-prod/payment-api",
      "error": [
        {
          "Text": "Deployment payment-prod/payment-api has 1 replicas but 0 are available with status running",
          "KubernetesDoc": "",
          "Sensitive": [
            { "Unmasked": "payment-prod", "Masked": "YSNVVSQjbDVwUHM3" },
            { "Unmasked": "payment-api",  "Masked": "NCZ5Tn1hK3ZFWzc=" }
          ]
        }
      ]
    },
    {
      "kind": "Pod",
      "name": "payment-prod/payment-worker-6d8b2c-p3mnr",
      "error": [
        {
          "Text": "the last termination reason is OOMKilled container=worker pod=payment-worker-6d8b2c-p3mnr"
        }
      ]
    },
    // ... 6 more results (Node, PersistentVolumeClaim, Service, Ingress, second Pod, second Deployment)
  ]
}

Why it's production-grade

Deterministic core. LLM only for sugar.

14 structured analyzers

Each analyzer (Pod, Deployment, Service, Ingress, PVC, Node, ReplicaSet, CronJob, etc.) is plain Go code that knows exactly what to look for. No hallucination possible in the detection phase.

Pluggable LLM backends

14 backends including OpenAI, Azure, Ollama, LocalAI, Amazon Bedrock, and customrest — which is what we wired to llm_proxy.py. Sensitive fields are auto-anonymized (see the Masked entries above).

Runs in-cluster or CLI

Deploy as a controller for continuous monitoring, or run k8sgpt analyze as a one-off CLI in CI / incident response. We used the CLI mode here.

JSON for pipelines

--output json emits a stable schema (kind, name, error array) that you can grep, route to Slack, or feed into a remediation playbook.

Robusta

v0.20+

A Kubernetes troubleshooting and automation platform. When a Prometheus alert fires, Robusta autonomously investigates — pulls logs, queries cluster state, calls an LLM, and posts a root-cause card to Slack.

License Apache 2.0

Repo github.com/robusta-dev/robusta

Stars 2.4k+

Language Python

Trigger Prometheus / Alertmanager

Sinks Slack · PagerDuty · MS Teams

The autonomous loop

Alert in, root-cause card out.

Robusta receives the Alertmanager webhook, decides what to investigate, gathers the evidence, asks the LLM, and posts a Slack card with remediation suggestions.

What runs where. The demo on this page does not run Robusta. It runs scripts/alert_triage_agent.py, a ~250-line reference implementation of the same alert → cluster-context → LLM → Slack-card loop, so you can read the whole thing in one sitting. Slack posting is opt-in: nothing leaves your machine unless you set SLACK_WEBHOOK_URL and pass --post-slack. Real Robusta — the actual robusta-dev Helm chart — runs in the separate kind/ path: make kind-up && make kind-broken && make kind-robusta && make kind-fire-alert creates a real kind cluster, deploys genuinely broken workloads, helm-installs Robusta with the Prometheus stack, and fires a real Prometheus alert into it.

STEP 01

Alert fires

An Alertmanager PaymentAPIHighErrorRate webhook arrives (a fixture here; a real Prometheus alert in kind/)

→

STEP 02

Gather context

The agent queries pod, deployment, events and node state via the Kubernetes API

→

STEP 03

LLM analysis

Prompt = alert + cluster state. The configured LLM backend produces RCA + remediation

→

STEP 04

{ }

Render card

Markdown rendered into a Slack-style card with graph & runbook links

→

STEP 05

Post (opt-in)

With SLACK_WEBHOOK_URL set, the card is posted to Slack; otherwise it is only rendered locally

Demo 3 · Alert triage

Watch the reference agent work.

alert_triage_agent.py, recorded live. Not Robusta — Robusta's flow.

z@devops-extravaganza: ~/demo REFERENCE IMPL

The output

The card the agent renders

A captured LLM root-cause analysis, rendered in the Slack card layout the agent emits. Posting it to a real channel is opt-in — this page is showing the local render, and the alert it responds to is the fixture payload below.

#sre-incidents

2026-08-20 03:42:18 UTC

alert-triage-agent APP 12 minutes ago

🔥 Root Cause

The payment-api pod is in a ContainersNotReady state, likely due to a crash loop or failed startup, causing the high 5xx error rate.

Evidence

Pod status is False for Ready and ContainersReady since 03:34:02Z.
Container status message: "Containers with unready status: [api]".
Alert started 28 minutes ago, coinciding with the deployment of payment-api rev 3.

Impact

Critical: Payment API is unavailable, with 12.4% of requests failing. All payment processing is likely impacted.

Recommended Action

Check the pod's logs for the api container: kubectl logs payment-api-7c4f5b-x9qkl -c api --previous. If the container is crashing, roll back the deployment: kubectl rollout undo deployment/payment-api.

📊 prometheus.internal.acme.io/graph?g0.expr=sum(rate(http_requests_total...))

📋 runbooks.internal.acme.io/payment-api-high-error-rate

🔗 grafana.internal.acme.io/d/payment-api

alerts/payment-api-high-error-rate.json — Alertmanager webhook fixture INPUT

webhook body fed to the agent

{
  "version": "4",
  "status": "firing",
  "receiver": "robusta",
  "commonLabels": {
    "alertname": "PaymentAPIHighErrorRate",
    "severity": "critical",
    "namespace": "payment-prod",
    "service": "payment-api"
  },
  "commonAnnotations": {
    "summary": "Payment API error rate above 5% for 5 minutes",
    "description": "Payment API is returning 5xx errors to 12.4% of requests. Threshold: 5%. The service has been erroring since the last deployment (payment-api rev 3) 28 minutes ago."
  },
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "PaymentAPIHighErrorRate",
        "severity": "critical",
        "pod": "payment-api-7c4f5b-x9qkl"
      },
      "startsAt": "2026-08-20T03:42:00Z",
      "fingerprint": "f1a2b3c4d5e6f7a8"
    }
  ]
}

Why it's different from K8sGPT

Reactive, not proactive. Alert-driven.

Alert-driven loop

K8sGPT scans the whole cluster on demand. Robusta reacts to specific alerts — so its investigations are scoped to the symptoms your monitoring already flagged.

Enriched context

Each alert pulls in related logs, pod yaml, events, deployment history, node metrics — all forwarded to the LLM as evidence. Not just the alert text.

Human-in-the-loop playbooks

Robusta ships Python playbooks you can extend. Auto-remediation is opt-in per playbook — the AI suggests, the human approves by default.

Sink-native output

Posts to Slack / PagerDuty / MS Teams in the format each tool expects (block kit, incident notes, adaptive cards). No formatting work needed.

Components

Six pieces, every one replaceable

Each component is a thin Python service or static asset. Swap any of them for your preferred alternative — a real kube-apiserver, a Slack webhook, your own model host — without touching the others.

K8s

mock_k8s_server.py

:8443 · HTTPS

A mock kube-apiserver serving a deliberately broken payment-prod namespace. Full discovery plus full read/write verbs (GET/POST/PUT/PATCH/DELETE) across /api/v1, /apis/apps/v1, /apis/networking.k8s.io/v1, … — faithful enough that the real kubectl and k8sgpt can't tell the difference. A reconciler applies writes, so remediation actually changes state. TLS certs are generated on first boot.

Simulates:your kube-apiserver

K8G

k8sgpt (binary)

v0.4.36 · Apache 2.0

Structured Go analyzers (Pod, Deployment, Service, Ingress, PVC, Node, ReplicaSet, …). Rules-based and deterministic — returns 8 findings against this cluster with no LLM involved. Add --explain to route each finding to your configured model.

Replaces:your SRE's first 30 minutes

LLM

llm_proxy.py

:8081 · HTTP

Translates k8sgpt's customrest request shape ({model, prompt, options.message}) into an OpenAI-compatible /chat/completions call and injects whatever auth the provider needs. Provider-agnostic: ollama (default, credential-free), openai, openrouter, zai, custom, or replay.

Fronts:any OpenAI-compatible API

MDL

the model (your choice)

LLM_BACKEND=ollama by default

Whatever backend you point the proxy at writes the root-cause analyses for the --explain pass and the triage card. No vendor is required. LLM_BACKEND=replay serves previously captured responses from captured/llm_cache.json with no network at all, which is what keeps the recorded demos reproducible.

Any of:ollama / openai / openrouter / zai / custom

RBS

alert_triage_agent.py

: one-shot · ~250 lines

A readable reference implementation of Robusta's loop: receives an Alertmanager webhook, gathers cluster context via kubectl, builds an LLM prompt from alert + state, calls the model, renders a Slack-style incident card. Posting to Slack is opt-in via SLACK_WEBHOOK_URL.

Not Robusta:real Robusta runs in kind/

SHW

index.html + GIFs

:80 / Vercel edge

Single-file dark-themed landing page with linked demo GIFs, JSON outputs, and a Slack-style RCA card. Self-contained — no JS framework, no build step, no external assets except Google Fonts.

Replaces:your project's README

Acceptance suite

18 checks you can run yourself

No score is baked into this page. The suite reports pass / fail / skip per check — skipping anything whose prerequisite binary or service is missing rather than pretending it passed — and writes a machine-readable summary to outputs/uat_results.json. Run it: ./run.sh uat

Check	What it verifies
`A1`	the mock API answers the Kubernetes discovery endpoints
`A2`	payment-api Pod is in CrashLoopBackOff
`A3`	payment-worker Pod was OOMKilled (exit 137)
`A4`	worker-3 Node reports DiskPressure=True
`A5`	payment-ingress backend Service does not exist
`A6`	payment-data-pvc is Pending with no StorageClass
`A7`	payment-api-svc selector matches no Pod
`B1`	kubectl reads the cluster
`B2`	kubectl writes are accepted (a patch round-trips through the API)
`B3`	k8sgpt finds every broken resource (8 findings, 6 kinds)
`C1`	remediate.sh drives the finding count to zero with real kubectl writes
`C2`	reset restores the broken state
`D1`	the LLM proxy reports its backend honestly (including replay mode)
`D2`	the proxy answers k8sgpt's customrest request shape
`D3`	the replay cache is keyed on the prompt, not the model name
`E1`	no absolute developer paths anywhere in the repo
`E2`	no credentials committed
`E3`	every file referenced by run.sh exists

18 checks · results are whatever your run produces, not a number typed into this page. Checks B* and C* need the kubectl and k8sgpt binaries and the mock API running; D* needs the LLM proxy. Anything missing is reported as skip.

Four ways to run it

Pick your poison

Vercel for the landing page · local Python for the full demo stack against the mock API · Docker for the containerized version · make kind-* for a real kind cluster with real Robusta.

▲ VERCEL static site · 30s deploy

# one-click deploy (clones to your GitHub + deploys) # → https://vercel.com/new/clone?repository-url= # https://github.com/adventurewave-labs/agentic-devops-extravaganza # OR via the Vercel CLI: npm i -g vercel git clone https://github.com/adventurewave-labs/agentic-devops-extravaganza.git cd agentic-devops-extravaganza vercel # preview deploy vercel --prod # production deploy # vercel.json tells Vercel to serve the repo root as a # static site with security headers + GIF caching.

SANDBOX no Docker · Python only · stdlib

# clone + start the full demo stack git clone https://github.com/adventurewave-labs/agentic-devops-extravaganza.git cd agentic-devops-extravaganza # bring up: mock K8s :8443 + LLM proxy :8081 + site :8080 ./run.sh demo # scan the mock cluster with the real k8sgpt binary ./run.sh scan # 8 findings, no LLM # real kubectl writes, then the same scan again: 8 -> 0 ./run.sh remediate ./run.sh reset # put the breakage back # the alert -> context -> LLM -> Slack-card agent ./run.sh triage # acceptance suite + stop everything ./run.sh uat ./run.sh stop

DOCKER site + mock K8s + LLM proxy

# choose an LLM backend (default: ollama, no credentials) export LLM_BACKEND=ollama # or: openai / openrouter / zai / custom (+ that # provider's API key), or replay for no network at all # bring up the full stack (3 services) docker compose up demo # endpoints: # showcase: http://localhost:8080 # mock K8s: https://localhost:8443 (self-signed) # LLM proxy: http://localhost:8081 # one-shot: the alert-triage reference agent docker compose run robusta

RECORD re-record GIFs · needs agg binary

# re-record the demos in a real pty and rebuild the GIFs ./run.sh record # record_demos.py runs the real commands live, then agg # turns each cast into a GIF: # scan.gif k8sgpt triage (no LLM) # explain.gif k8sgpt --explain through the LLM proxy # remediate.gif real kubectl fixes, 8 findings -> 0 # triage.gif alert -> context -> LLM -> Slack card # install agg (asciinema GIF generator): curl -L https://github.com/asciinema/agg/releases/download/v1.9.0/agg-x86_64-unknown-linux-musl \ -o /usr/local/bin/agg chmod +x /usr/local/bin/agg

	K8sGPT	Robusta
License	Apache 2.0	Apache 2.0
Language	Go	Python
Trigger model	Manual / scheduled / controller	Alert-driven (Alertmanager webhook)
Output format	Text + JSON (CLI)	Slack card, PagerDuty, MS Teams
LLM role	Optional --explain	Always on for alert enrichment
Can auto-remediate?	Experimental	Yes, via opt-in playbooks
Best for	Cluster-wide audits, CI gates	Single-alert deep dives, on-call rotation
Demo in this page	Real binary: 8 findings → 0 after remediation	Reference implementation of the loop (real Robusta runs in kind/)

Reproducibility

How the demo was built — and what is simulated.

Everything above was produced by real binaries — k8sgpt v0.4.36 and kubectl — talking to a mock Kubernetes API server over the real Kubernetes REST protocol. The cluster is simulated; the binaries, the API calls, the writes and the findings are not. Here are the building blocks.

1 · mock_k8s_server.py

A single-file Python HTTPS server implementing the Kubernetes REST API — discovery plus GET/POST/PUT/PATCH/DELETE across /api/v1, /apis/apps/v1, /apis/networking.k8s.io/v1 and friends — well enough that kubectl and k8sgpt behave exactly as they would against a kube-apiserver. It serves broken payment-prod state and reconciles the writes remediation makes. Self-signed certs are generated on first boot.

2 · k8sgpt analyze

The actual k8sgpt v0.4.36 binary from GitHub releases, configured with --kubeconfig pointing at the mock. The analyzers run unmodified and return 8 findings. --output json emits the schema shown above.

3 · llm_proxy.py

k8sgpt's customrest backend POSTs to a single URL with a non-OpenAI request shape. This proxy translates that shape into a standard chat-completions call and injects whatever auth the chosen provider needs, so the same demo works on ollama, openai, openrouter, zai, any custom OpenAI-compatible endpoint, or the offline replay cache.

4 · the model

No specific model is required and none is bundled. Point LLM_BACKEND at whatever you have — ollama needs no credentials at all. Responses captured from earlier runs live in captured/llm_cache.json; LLM_BACKEND=replay serves those verbatim with no network, and labels every answer as a replay.

5 · record_demos.py

Runs each demo for real in a pty and records the session to an asciinema cast — the commands actually execute, and what you see scrolling past is the binaries' live output rather than a script's idea of it. (Earlier versions of this repo hand-authored the cast files; that is no longer how the GIFs are made.)

6 · agg → GIF

The agg static binary from the asciinema project converts each .cast file to a GIF. Four GIFs total: scan.gif, explain.gif, remediate.gif, triage.gif.

Files included in this delivery

tree

agentic-devops-extravaganza/
├── index.html                    # this page
├── gifs/
│   ├── scan.gif                 # Demo 1: cluster triage
│   ├── explain.gif              # Demo 2: LLM explanations
│   ├── remediate.gif            # real kubectl fixes, 8 -> 0
│   └── triage.gif               # alert -> LLM -> Slack card
├── recordings/                   # the asciinema casts behind the GIFs
├── scripts/
│   ├── mock_k8s_server.py       # the simulated cluster
│   ├── remediate.sh             # the real kubectl fixes
│   ├── llm_proxy.py             # customrest -> any OpenAI-compat API
│   ├── alert_triage_agent.py    # reference impl of Robusta's loop
│   ├── record_demos.py          # records the demos in a pty
│   └── run_uat.py               # the acceptance suite
├── kind/                         # real cluster + real Robusta (helm)
└── outputs/
    ├── k8sgpt_analyze.json      # raw JSON from k8sgpt analyze
    ├── k8sgpt_analyze.txt       # raw text from k8sgpt analyze
    ├── k8sgpt_analyze_after.txt # the same scan after remediation
    ├── alert_triage.json        # what the triage agent produced
    └── uat_results.json         # last acceptance-suite run

Agentic DevOps,
in production.

8 findings.
Then 0.

Two agents.
One broken cluster.
The LLM of your choice.

K8sGPT

Eight findings, no LLM involved.

Same scan, with an LLM writing the diagnosis.

Deterministic core. LLM only for sugar.

Robusta

Alert in, root-cause card out.

Watch the reference agent work.

The card the agent renders

🔥 Root Cause

Evidence

Impact

Recommended Action

Reactive, not proactive. Alert-driven.

Six pieces, every one replaceable

mock_k8s_server.py

k8sgpt (binary)

llm_proxy.py

the model (your choice)

alert_triage_agent.py

index.html + GIFs

18 checks you can run yourself

Pick your poison

When to reach for which.

How the demo was built — and what is simulated.

1 · mock_k8s_server.py

2 · k8sgpt analyze

3 · llm_proxy.py

4 · the model

5 · record_demos.py

6 · agg → GIF

Files included in this delivery

Get the code

Agentic DevOps, in production.

8 findings. Then 0.

Two agents.One broken cluster.The LLM of your choice.

K8sGPT

Eight findings, no LLM involved.

Same scan, with an LLM writing the diagnosis.

Deterministic core. LLM only for sugar.

Robusta

Alert in, root-cause card out.

Watch the reference agent work.

The card the agent renders

🔥 Root Cause

Evidence

Impact

Recommended Action

Reactive, not proactive. Alert-driven.

Six pieces, every one replaceable

mock_k8s_server.py

k8sgpt (binary)

llm_proxy.py

the model (your choice)

alert_triage_agent.py

index.html + GIFs

18 checks you can run yourself

Pick your poison

When to reach for which.

How the demo was built — and what is simulated.

1 · mock_k8s_server.py

2 · k8sgpt analyze

3 · llm_proxy.py

4 · the model

5 · record_demos.py

6 · agg → GIF

Files included in this delivery

Get the code

Agentic DevOps,
in production.

8 findings.
Then 0.

Two agents.
One broken cluster.
The LLM of your choice.