Ask a language model, “Why is the checkout service down?” while forty alerts are firing, and it will give you a story.
It will be well-structured, confident, and plausible. It might even be right.
But plausible is a property of the prose. Correct is a property of the system. And the more the model has to guess about, the further those two drift apart.
The instinct is to reach for a better model. We think that’s the wrong lever.
The better approach is to give the model less to guess about – answer every question that has a deterministic answer before probabilistic reasoning starts.
Two structures do most of that work: a topology map of what depends on what right now, and a knowledge graph of what happened before.
Deterministic Where It Can Be, Probabilistic Where It Must Be
Consider the questions an engineer actually asks during an incident:
- What does this service depend on? – deterministic. It’s a graph lookup.
- What else is affected if this fails? – deterministic. Same graph, opposite direction.
- Did anything deploy in the last hour? – deterministic. It’s an event query.
- Has this happened before, and what fixed it? – deterministic retrieval, then judgement.
- Given all of that, what’s the most likely root cause? – probabilistic. This is where the model is useful.
Most AI-driven RCA hands the model all five questions and the whole cluster’s telemetry and asks it to sort things out.
We’d rather answer the first four with a graph and a database, then give the model a small, bounded, cited set of candidates for the fifth.
That makes the AI investigation more focused – and easier to verify.
The Topology Map: Knowing What Depends on What
The dependency map is a graph of entities – clusters, nodes, namespaces, workloads, pods, services, databases, and external endpoints – connected by typed edges such as contains, hosts, routes to, calls, and depends on.
It is per-environment, tenancy-isolated, and deliberately small: hundreds to low thousands of nodes for a typical client cluster.
Everything follows one direction convention, fixed once and enforced everywhere:
A ──DEPENDS_ON──▶ B
A needs B to work.
For an alarm on A:
- Root-cause candidates → follow outgoing edges
- Blast radius → follow incoming edges
That’s enough to make root-cause candidates and impact a bounded graph traversal.
The graph is deliberately small enough to run as a recursive query against the relational database the platform already has – no graph database, no second source of truth, and nothing new to operate.
Most Alert Storms Are Vertical, Not Horizontal
One observation shaped much of this design: many alert storms are containment, not cascade.
A node fails. That produces pod alerts, workload alerts, and namespace alerts. They’re the same event appearing at different levels of the Kubernetes object hierarchy.
You don’t necessarily need a service-call graph to collapse those alerts. You need to know which pod belongs to which workload and which node hosts it.
That information is already being scraped.
kube-state-metrics publishes the Kubernetes ownership graph as Prometheus timeseries – pod-to-owner, ReplicaSet-to-Deployment, pod-to-node, ingress-to-service.
Chaining those relationships gives you pod → deployment, which matters because alerts often identify pods while operators think in terms of workloads.
So the baseline topology costs nothing new: no new agent, no additional privileges, and no separate data collection mechanism. It arrives through the same remote-write channel the platform’s Prometheus agent already uses.
Building the Rest of the Dependency Map
Cross-service cascades – for example, a payments API becoming slow because its database is slow – require call-level relationships.
Different data sources provide different levels of visibility:
| Source | What it tells us | Confidence |
|---|---|---|
| kube-state-metrics | Ownership and containment | Fact |
| CoreDNS dnstap | A resolved B, including external dependencies | Existence |
| OpenTelemetry | Actual service calls, rates, and errors | Measured |
| eBPF | Kernel-level calls, volume, and errors | Measured |
Each has a different operational cost and a different level of confidence.
DNS is particularly useful because it can reveal dependencies across the cluster boundary. Trace-derived graphs only show external dependencies when the caller is instrumented, while the Kubernetes control plane doesn’t see them at all.
Cluster DNS sees resolutions from every pod.
It can’t tell you how much traffic flows or whether calls fail, so DNS edges don’t feed blast-radius weighting. But they can establish that a dependency exists – including dependencies that nobody has documented.
Every edge carries its provenance and confidence score.
A DNS-derived edge doesn’t look like a measured trace-derived edge, and a control-plane fact doesn’t look like an inference. If two sources agree, both are visible.
An Observed Call Is Not Necessarily a Dependency
This is an important distinction.
eBPF and traces can tell you that A called B. They don’t necessarily tell you that A fails when B does.
A service might call an analytics endpoint on every request and simply ignore the response if it times out. There is an edge, but it’s a soft dependency.
If you calculate blast radius as though every edge were hard, you’ll overstate the impact of many incidents.
So edge criticality – hard or soft – is a separate attribute that the map cannot infer. An operator sets it.
The graph tells you the structure. It doesn’t pretend to know what matters.
The Unglamorous Part That Decides Everything
None of this works unless an incoming alert can be resolved to a node in the graph.
Alert labels are not consistent. One exporter says pod, another says pod_name, another says kubernetes_pod_name. instance might be an IP and port in one system and a hostname in another.
Label normalization is, honestly, the whole design.
It’s handled by a data-driven extraction ruleset per technology, with per-client overrides – and it’s measured.
The platform exposes one number:
What percentage of alerts resolve to a workload-level entity?
If that number is low, every downstream classification becomes less reliable. We’d rather expose that metric than hide it.
The Knowledge Graph: What Happened Before
Topology tells you structure.
It doesn’t tell you history.
The second layer adds operational knowledge to the same entities:
- Deployment events from the client’s GitOps pipeline – because “what changed?” is often the first question during an incident.
- Past RCA episodes – completed investigations stored as incident, root cause, evidence, and remediation, then retrieved by semantic similarity when a new incident resembles an old one.
- Related Jira tickets – found through free-text search over the client’s connected project.
- Runbooks and SOPs – already associated with alert types and operational procedures.
Consider the difference between raw history and useful memory.
“Connection-pool exhaustion after a config deploy lowered max-connections; rolled back, cleared in three minutes.”
That is far more useful to the next investigation than a database full of disconnected telemetry.
Each knowledge source is per-environment and tenancy-isolated by construction – a separate index per client, rather than a runtime filter that could be forgotten.
How Topology and Knowledge Graphs Change AI-Powered RCA
Put the two structures together and the RCA agent’s job changes significantly.
Triage gets sharper.
Blast-radius signals – how many namespaces are affected, whether shared infrastructure is involved, and how many downstream alerts are firing – become computable from the topology instead of guessed.
Related alerts can also be grouped by graph proximity rather than only by time window.
This is the topology-aware part of the triage pipeline we discussed in the previous post.
The candidate set gets smaller.
Instead of receiving the whole cluster, the model gets:
- the alarm’s entity;
- upstream dependencies within a few hops;
- confidence and criticality for those dependencies;
- recent deployments affecting them; and
- the closest past RCA episodes.
That creates a bounded, cited investigation scope.
The reasoning becomes easier to check.
Because the inputs are retrievable facts, claims in the RCA can cite the tool call that produced them. A separate verifier can then check those citations against the recorded execution trace.
The layers have clear responsibilities:
The deterministic layer narrows.
The probabilistic layer explains.
The verification layer audits.
The model is still doing the hard part.
It’s just no longer doing the easy parts badly.
What the Topology Map Won’t Tell You
A dependency map is not a complete inventory.
The control-plane tier only maps what the cluster reports. DNS only sees what resolves through cluster DNS, so direct-IP traffic and long-lived pooled connections are invisible. Traces only cover instrumented services.
A dependency map should never be presented as:
“Here is everything.”
It should be:
“Here is what we can prove, with the confidence we can prove it at.”
The same principle applies to correlation.
If a correlation rule could hide a real incident by attaching a symptom to an existing incident instead of opening a new one, it runs in shadow mode first.
It can run for weeks, producing verdicts that are compared against what operators actually did, before it changes a single ticket.
Where This Fits in the ITGix AIOps Platform
Topology and knowledge graphs are not a standalone feature. They are part of the broader incident-resolution workflow we’re building within the ITGix AIOps Platform.
The topology layer provides the current state of the environment. The knowledge layer provides operational history. Together, they give the AI SRE agent a smaller and more reliable context for investigation.
That also connects directly to the alert-triage and RCA capabilities covered in the previous posts:
alerts → triage → correlation → topology context → AI investigation → verified RCA → remediation
The goal isn’t to make the AI guess better.
It’s to make sure there is less for it to guess about in the first place.
Where We Are
The topology work is being built in stages.
Entity extraction and the control-plane map come first because they provide an inspectable graph with zero risk to alerting. Streaming-pipeline changes follow, then topology-aware triage in shadow mode.
DNS, trace, and opt-in eBPF enrichment come after that.
RCA memory – including episode recall and Jira similar-issue lookup – is on the same roadmap.
We’ll say when each piece is live.
In the meantime, the principle stands:
Answer every question that has a deterministic answer first, and the AI gets dramatically better at the one that doesn’t.
Keep an eye on the ITGix blogs for the next chapter.
