The routine it takes off your hands
Every failed pod starts with the same commands, and the same scrolling through output to find the one line that matters.
kubectl get pods -n paymentskubectl describe pod payment-api-7d9f8c6b5-x2kqpkubectl logs payment-api-7d9f8c6b5-x2kqp --previouskubectl get events --field-selector involvedObject.name=...search the error message, read three forum threads
Incident Copilot runs those reads for you every 15 seconds and puts the evidence and the explanation in one place. You still decide what to change. It only reads.
What it catches
These are the nine failures that ship as samples. make break deploys all of them so you can watch it work.
- CrashLoopBackOffThe app can't reach its database and exits.
- OOMKilledThe process uses 150 MB under a 64 Mi limit.
- ImagePullBackOffA typo in the image tag.
- CreateContainerConfigErrorThe ConfigMap the pod reads doesn't exist.
- UnschedulableIt asks for 16 CPUs on a laptop.
- UnschedulableIts volume claim names a StorageClass that isn't there.
- RunContainerErrorThe entrypoint script isn't in the image.
- Restart loopThe liveness probe checks the wrong port.
- Not readyThe readiness path returns 404.
Critical: the workload is down or can't start. Warning: it runs but isn't healthy.
What happens when a pod breaks
One small Python service and one model, both inside the cluster.
- It noticesA poll every 15 seconds checks container states, conditions and restart counts in the namespaces you choose.
- It gathers evidenceThe pod spec, failing conditions, the last events, logs from the crashed run, and any volume claims. Secret values are never read, only their names.
- It explainsA rule-based read shows at once. Then the model writes the root cause, the lines that prove it, and the commands to fix it. Hosted models first investigate with read-only tools, following the trail to the Service, the rollout, the node or the quota.
- It tells the right peopleSlack, Teams, Discord, Google Chat, email or any webhook, filtered by severity and namespace.
- It closes the loopWhen the pod is healthy or gone, the incident is marked resolved and a short all-clear goes out.
Choose what it watches
Everything is set in one file. The same config.env works when running locally and becomes a ConfigMap in the cluster.
config.env
# names, or * for every namespace WATCH_NAMESPACES=demo,checkout # plus any namespace a team labels WATCH_NAMESPACE_SELECTOR=incident-copilot/watch=true IGNORE_NAMESPACES=kube-system,kube-public,kube-node-lease ALERT_SEVERITY=critical OLLAMA_MODEL=qwen2.5:1.5b
Teams opt in without touching the copilot
kubectl label namespace payments incident-copilot/watch=true kubectl annotate namespace payments \ incident-copilot/owner=payments-team \ incident-copilot/alert-severity=warning
| On the namespace | What it does |
|---|---|
incident-copilot/watch=true label | Start watching it |
incident-copilot/ignore=true label | Never watch it, even with * |
incident-copilot/owner annotation | Show the owning team in the UI and in alerts |
incident-copilot/alerts=off annotation | Show incidents, but send no alerts |
incident-copilot/alert-severity annotation | critical or warning for this namespace only |
Run the model locally, or bring your own
The default is a small model inside the cluster, so nothing leaves it. If you would rather use Claude, OpenAI or Bedrock, set one line and the copilot runs as a single small pod, with no model server to host.
| LLM_PROVIDER | What runs in the cluster |
|---|---|
ollama | The copilot plus Ollama (about 2 GB). Nothing leaves the cluster. |
anthropic, openai, azure, gemini | The copilot only. API key in secrets.env. |
bedrock | The copilot only. On EKS, no keys at all: Pod Identity or IRSA. |
openai-compatible | The copilot only. OpenRouter, Groq, vLLM, LM Studio. |
Before anything goes to a hosted model, passwords, tokens, keys, email addresses and card numbers are stripped out. A namespace annotated incident-copilot/llm=local never leaves the cluster at all.
Alerts go where your team already talks
Add a webhook URL to secrets.env and that channel is on. By default an alert waits up to three minutes for the model's diagnosis, so the first message already says what to do.
- Slack
- Microsoft Teams
- Discord
- Google Chat
- Webhook
Critical: ImagePullBackOff in demo/frontend
Pod frontend-6c9d7b8f4-lmv2t on minikube.
Why: the image tag nginx:1.27-alpien does not exist. It is a typo for 1.27-alpine.
Fix:kubectl -n demo set image deploy/frontend web=nginx:1.27-alpine
10:38 Resolved: demo/frontend is healthy again after 7 minutes.
Install it on your cluster
It goes into whatever cluster your kubectl context points at, from EKS to minikube. When you are done, one command removes all of it and leaves the cluster as it was.
git clone https://github.com/yousafkhamza/k8s-incident-copilot cd k8s-incident-copilot cp config.env.example config.env cp secrets.env.example secrets.env # for alerts make up # into the current kube context make break # deploy the broken workloads make ui # http://localhost:8080
No make? Use kustomize
kubectl apply -k . kubectl apply -f k8s/20-ollama.yaml
Clean up
make uninstall # remove only this project make local-down # if you used make local-up
Safe to point at a real cluster
- Read-only accessIts ClusterRole can only get, list and watch. It has no access to Secrets, and the agent's tools are reads too.
- Your data stays putBy default the model runs in the cluster through Ollama. With a hosted model, data is redacted first, and sensitive namespaces can stay local.
- No secret valuesEnvironment variables that come from Secrets are sent to the model by name only.
- Locked-down containerNon-root, read-only filesystem, all capabilities dropped, a 256 Mi memory limit.