Incident Copilot

Know why the pod is failing before you open a terminal.

Incident Copilot sits in your cluster and notices broken pods. It collects the events, logs and spec you would have pulled by hand, then a model explains the cause and the fix: one running next to it, or Claude, OpenAI or Bedrock. The answer lands in Slack or Teams.

Installs into any Kubernetes cluster, minikube on a laptop included. It only reads, and with the default model nothing leaves your cluster.

The routine it takes off your hands

Every failed pod starts with the same commands, and the same scrolling through output to find the one line that matters.

  • kubectl get pods -n payments
  • kubectl describe pod payment-api-7d9f8c6b5-x2kqp
  • kubectl logs payment-api-7d9f8c6b5-x2kqp --previous
  • kubectl get events --field-selector involvedObject.name=...
  • search the error message, read three forum threads

Incident Copilot runs those reads for you every 15 seconds and puts the evidence and the explanation in one place. You still decide what to change. It only reads.

What it catches

These are the nine failures that ship as samples. make break deploys all of them so you can watch it work.

Critical: the workload is down or can't start. Warning: it runs but isn't healthy.

What happens when a pod breaks

One small Python service and one model, both inside the cluster.

  1. It noticesA poll every 15 seconds checks container states, conditions and restart counts in the namespaces you choose.
  2. It gathers evidenceThe pod spec, failing conditions, the last events, logs from the crashed run, and any volume claims. Secret values are never read, only their names.
  3. It explainsA rule-based read shows at once. Then the model writes the root cause, the lines that prove it, and the commands to fix it. Hosted models first investigate with read-only tools, following the trail to the Service, the rollout, the node or the quota.
  4. It tells the right peopleSlack, Teams, Discord, Google Chat, email or any webhook, filtered by severity and namespace.
  5. It closes the loopWhen the pod is healthy or gone, the incident is marked resolved and a short all-clear goes out.

Choose what it watches

Everything is set in one file. The same config.env works when running locally and becomes a ConfigMap in the cluster.

config.env

# names, or * for every namespace
WATCH_NAMESPACES=demo,checkout
# plus any namespace a team labels
WATCH_NAMESPACE_SELECTOR=incident-copilot/watch=true
IGNORE_NAMESPACES=kube-system,kube-public,kube-node-lease
ALERT_SEVERITY=critical
OLLAMA_MODEL=qwen2.5:1.5b

Teams opt in without touching the copilot

kubectl label namespace payments incident-copilot/watch=true
kubectl annotate namespace payments \
  incident-copilot/owner=payments-team \
  incident-copilot/alert-severity=warning
On the namespaceWhat it does
incident-copilot/watch=true labelStart watching it
incident-copilot/ignore=true labelNever watch it, even with *
incident-copilot/owner annotationShow the owning team in the UI and in alerts
incident-copilot/alerts=off annotationShow incidents, but send no alerts
incident-copilot/alert-severity annotationcritical or warning for this namespace only

Run the model locally, or bring your own

The default is a small model inside the cluster, so nothing leaves it. If you would rather use Claude, OpenAI or Bedrock, set one line and the copilot runs as a single small pod, with no model server to host.

LLM_PROVIDERWhat runs in the cluster
ollamaThe copilot plus Ollama (about 2 GB). Nothing leaves the cluster.
anthropic, openai, azure, geminiThe copilot only. API key in secrets.env.
bedrockThe copilot only. On EKS, no keys at all: Pod Identity or IRSA.
openai-compatibleThe copilot only. OpenRouter, Groq, vLLM, LM Studio.

Before anything goes to a hosted model, passwords, tokens, keys, email addresses and card numbers are stripped out. A namespace annotated incident-copilot/llm=local never leaves the cluster at all.

Alerts go where your team already talks

Add a webhook URL to secrets.env and that channel is on. By default an alert waits up to three minutes for the model's diagnosis, so the first message already says what to do.

  • Slack
  • Microsoft Teams
  • Discord
  • Google Chat
  • Email
  • Webhook
incident-copilot10:31

Critical: ImagePullBackOff in demo/frontend

Pod frontend-6c9d7b8f4-lmv2t on minikube.
Why: the image tag nginx:1.27-alpien does not exist. It is a typo for 1.27-alpine.

Fix:
kubectl -n demo set image deploy/frontend web=nginx:1.27-alpine

10:38 Resolved: demo/frontend is healthy again after 7 minutes.

Install it on your cluster

It goes into whatever cluster your kubectl context points at, from EKS to minikube. When you are done, one command removes all of it and leaves the cluster as it was.

git clone https://github.com/yousafkhamza/k8s-incident-copilot
cd k8s-incident-copilot
cp config.env.example config.env
cp secrets.env.example secrets.env  # for alerts
make up      # into the current kube context
make break   # deploy the broken workloads
make ui      # http://localhost:8080

No make? Use kustomize

kubectl apply -k .
kubectl apply -f k8s/20-ollama.yaml

Clean up

make uninstall   # remove only this project
make local-down  # if you used make local-up

Safe to point at a real cluster