A week later, finance asks why a payments worker role can read an analytics bucket it was never scoped for. You open the agent transcript to see what it actually did that night. The transcript says exactly what the summary said. It is the only record you have of the agent's reasoning, and the agent wrote it.

You can read that transcript a hundred times. It will never tell you what the agent did, because it was generated by the agent. The thing you are trying to verify produced the only evidence you have. That is the problem this post is about, and the fix has been sitting in your account the whole time.

The transcript is self-attestation

When an agent reports that it scaled a deployment, attached a policy, and merged a fix, it is narrating its own behavior. The report is a generated artifact from the same model whose actions are in question. A run that did exactly what it claimed and a run that quietly did something else produce transcripts that look identical: confident, well-formatted, plausible. Confidence is not a property of having done the right thing. It is a property of the decoder.

This is the same failure mode as a 200 OK on a wrong answer. The HTTP status confirms the call returned, not that the answer was correct, and an AI system can return a fabricated return policy with a clean 200 and 340ms latency (covered in [200 OK, Wrong Answer](https://stxkxs.io/blog/ai-observability-200-ok)). The agent transcript is the same shape one layer up. It confirms the agent said it did the work. It does not confirm the work.

The gap is small when a human reviews and merges every change. The human merge is the checkpoint, and the transcript is just notes. The gap becomes load-bearing the moment an agent holds a credential and acts on infrastructure on its own schedule. At that point the transcript is the entire audit trail, and it was written by the party under audit.

The system you are trying to verify produced the only evidence you have.

Builds already learned this

Software builds went through this exact reckoning. For years "the build succeeded" was the contract, and the build log was the proof. Then SolarWinds and a decade of dependency compromises made it clear that a build agent's own report is not evidence of what it produced. The answer was provenance: an independent attestation of what a build step did, signed, bound to the identity that did it.

SLSA defines the provenance format and the levels of assurance. in-toto links each step in the chain to the identity that performed it. Sigstore signs the attestation so it cannot be forged after the fact. The build no longer vouches for itself. A separate record vouches for the build, and consumers verify that record instead of trusting the log. This is the same supply-chain logic that turns transitive dependencies from a blind spot into something you can inspect (covered in [Your Dependencies Have Dependencies](https://stxkxs.io/blog/supply-chain-security-ai-era)).

An agent acting on production is a build step. It mutates IAM, scales workloads, opens and merges pull requests, touches data. It is a step in your delivery chain with real blast radius and no attestation. We learned not to trust a build agent's self-report, then wired up the same trust for agents that change live infrastructure and called it autonomy.

INFO: Provenance is independence
The point of build provenance is not that it is signed. It is that the signer is not the thing being attested. An agent signing its own transcript reproduces the original problem with a cryptographic flourish. The evidence has to come from a system the agent does not control.

The record already exists

Whatever the agent did to your infrastructure, it did through an API. The agent cannot scale a deployment without going through the Kubernetes API server, and if audit logging is enabled, that call writes an audit event. It cannot attach an IAM policy without an AWS API call, which CloudTrail records by default. It cannot merge a pull request without a GitHub event. None of these logs were written by the agent. CloudTrail management events are on by default; the Kubernetes audit log exists the moment you turn it on.

CloudTrail records the identity, the action, the request parameters, the source, and the result for management API calls, and delivers them within an average of about 5 minutes of the call, though AWS does not guarantee that window. The Kubernetes audit log captures every request to the API server across four stages, from RequestReceived to ResponseComplete, at whatever fidelity your audit policy sets. GitHub's audit log and events API record repository and organization activity. These are exactly the independent records the agent transcript is not.

• CloudTrail delivery: ~5 min avg — Management events are delivered within an average of about 5 minutes of the API call, not guaranteed, with identity, parameters, and result ([AWS CloudTrail documentation](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/get-and-view-cloudtrail-log-files.html)).
• K8s audit stages: 4 — The API server records RequestReceived, ResponseStarted, ResponseComplete, and Panic for each request, at the level set by audit policy ([Kubernetes documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)).
• GitHub git events: 7 days — The enterprise audit log retains git.push and git.clone events for 7 days; other events for 180 days ([GitHub Enterprise Cloud documentation](https://docs.github.com/en/enterprise-cloud@latest/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/accessing-the-audit-log-for-your-enterprise)).
• Machine vs human identities: 109:1 — Machine identities outnumber human ones 109 to 1, with AI agents now the majority of machine identities ([2026 Identity Security Landscape](https://www.paloaltonetworks.com/idira/identity-security-landscape-report), Palo Alto Networks/CyberArk, 2,930 respondents).

A Kubernetes audit event for the scale action from the incident is concrete and unambiguous. It names the user, the verb, the resource, the object, and the response code. It is the ground truth the agent summary stands in for.

{
  "kind": "Event",
  "stage": "ResponseComplete",
  "requestURI": "/apis/apps/v1/namespaces/payments/deployments/worker/scale",
  "verb": "update",
  "user": {
    "username": "system:serviceaccount:agents:remediator",
    "groups": ["system:serviceaccounts:agents"]
  },
  "objectRef": {
    "resource": "deployments",
    "namespace": "payments",
    "name": "worker",
    "subresource": "scale"
  },
  "responseStatus": { "code": 200 },
  "requestReceivedTimestamp": "2026-06-15T02:39:51Z",
  "stageTimestamp": "2026-06-15T02:39:51Z"
}

Nobody reconciles this against what the agent claimed. The audit log goes to a SIEM bucket for compliance retention and is read after something has already gone wrong. The agent transcript goes to a trace viewer. They are never compared, so the one question that matters at 02:40 stays unanswered: did the agent do what it said, and only that.

Identity collapses to a credential

Even with the logs in hand, attribution is thin. The audit event above says system:serviceaccount:agents:remediator did the scale. It does not say which agent, running which task, on whose behalf, under what reasoning. The infrastructure sees a credential, not an actor. With machine identities now outnumbering humans 109 to 1 and AI agents the majority of that count (2026 Identity Security Landscape, Palo Alto Networks/CyberArk), a single service account or assumed role can stand in for a whole fleet of distinct decisions.

Git makes this sharper. A commit carries an author and a committer, and both are self-asserted strings the agent sets. An agent can author a commit as anyone. The only fields with independent weight are the push event in the audit log and a verified signature, because those come from GitHub and from a key the committer cannot fabricate after the fact. The commit message is part of the agent's story. The push event is part of the record.

WARNING: One credential, many actors
Shared agent service accounts make the audit trail technically complete and practically useless. Every action traces to the same principal, so "who did this" resolves to "the agent platform" and stops there. Per-task identity is what makes reconciliation able to point at a specific decision.

Reconcile claims against logs

The missing piece is a read-only step that treats the agent's report as a set of claims and checks each one against the independent record. The agent says it scaled the worker deployment to six replicas at 02:39. The reconciliation reads the audit log, finds the matching update, and confirms the principal, the verb, the object, and the result line up with the claim. Where they diverge, it flags. It writes nothing and changes nothing. It only compares.

interface AgentClaim {
  action: 'scale' | 'attach-policy' | 'merge'
  resource: string
  expectedPrincipal: string
  expectedScope?: string // the resource ARN or policy document the agent claims it touched
  at: string // ISO timestamp the agent reported
}

// Claim to log: does the record support what the agent said it did.
function reconcile(claim: AgentClaim, events: AuditEvent[]): Finding {
  const match = events.find(
    (e) => e.objectRef === claim.resource && near(e.timestamp, claim.at),
  )

  if (!match) return { claim, verdict: 'unsupported' } // agent said it; the log did not see it
  if (match.principal !== claim.expectedPrincipal)
    return { claim, verdict: 'wrong-actor', actual: match.principal }
  if (claim.expectedScope && match.requestParameters.scope !== claim.expectedScope)
    return { claim, verdict: 'scope-mismatch', actual: match.requestParameters.scope }
  if (match.responseCode >= 400)
    return { claim, verdict: 'failed-but-claimed-success', code: match.responseCode }

  return { claim, verdict: 'confirmed' }
}

// Log to claim: does every action the agent's principal took have a matching claim.
// This direction catches what the claim-to-log pass structurally cannot: the
// agent that reports three actions and quietly took a fourth.
function findUnclaimed(principal: string, window: TimeWindow, claims: AgentClaim[], events: AuditEvent[]): Finding[] {
  return events
    .filter((e) => e.principal === principal && inWindow(e.timestamp, window))
    .filter((e) => !claims.some((c) => e.objectRef === c.resource && near(e.timestamp, c.at)))
    .map((e) => ({ claim: null, verdict: 'unclaimed', actual: e }))
}

The verdicts are the whole point. A confirmed claim is the boring, common case. An unsupported claim means the agent reported an action the infrastructure never saw, which is the report drifting from reality. A wrong-actor verdict means the action happened under a principal the agent did not expect. A scope-mismatch verdict means the principal and the action line up but the resource the log recorded is broader than the resource the agent claimed, which is how the policy attachment to the analytics bucket would have surfaced the same night instead of a week later in a finance ticket.

Reconcile as shown only runs claim to log: for every claim, find the matching event. The reverse direction is the one that matters more. Walk the audit events under the agent's principal for the task window and require a matching claim; anything left over is unclaimed, an action the agent took and never reported. An agent that lies by omission passes every claim-to-log check while doing exactly that, and the log-to-claim pass is the only one of the two that can see it. It also depends on per-task identity: run it against a shared service account and "unclaimed" collapses into "some other task in the same account," not a finding.

Reconciliation is a third practice next to monitoring and evaluation. Monitoring asks whether the system is up. Evaluation asks whether the output is good. Reconciliation asks whether the actor's account of its actions matches the record those actions left behind. It is the agentic version of double-entry bookkeeping, and like provenance for builds, its value comes entirely from the second entry being written by someone else. The closed agentic loop, where agents deploy and remediate without a human in the path, is exactly where this becomes non-optional (covered in [The Agentic DevOps Loop](https://stxkxs.io/blog/agentic-devops-loop)).

Key Point: You already collect the ground truth. CloudTrail, the Kubernetes audit log, and GitHub events record what the agent did to systems it does not control. The work is comparing the agent's account against that record, not gathering more of it.

When reconciliation is overkill

The value scales with autonomy and blast radius, and plenty of deployments have neither. An agent that only opens pull requests a human reads and merges does not need reconciliation, because the human merge is already the independent check and the diff is already the evidence. An agent scoped to read-only credentials can be wrong in its summary, but it cannot have mutated anything, so there is little to reconcile.

There is also a real chance the autonomy that justifies this never arrives for a given team. Gartner projects that more than 40% of agentic AI projects will be canceled by the end of 2027, citing cost, unclear value, and inadequate risk controls. Building a reconciliation layer for an agent program that gets shelved is wasted effort. The honest trigger is the first time an agent holds a credential that can change production state without a human in the path. Before that, the merge button is doing the job.

And reconciliation is not free. It needs per-task identity to be useful, which means the shared service account has to go. It needs the audit sources actually enabled, with a Kubernetes audit policy richer than the default and CloudTrail data events where they matter. A reconciliation layer reading thin logs produces confident green checks that mean nothing, which is worse than no check at all.

Autonomy widens the gap

The trajectory only points one way. Gartner projects that 15% of day-to-day work decisions will be made autonomously through agentic AI by 2028, up from none in 2024, and that a third of enterprise software will embed agentic AI over the same window. Every point of that curve is another action taken by a system that writes its own account of what it did.

Agentic AI penetration by 2028

• Enterprise apps with agentic AI: 33%
• Autonomous work decisions: 15%

The build world saw this curve about five years ago, and it took SolarWinds to force the answer. Provenance became a default only once the breach had already happened. Agent platforms have a chance the build world did not: wiring up reconciliation ahead of their own equivalent breach instead of after. The logs are already there. The reconciliation between what the agent claims and what the record shows is the piece nobody has wired up.

Ask the question from the incident again. The worker role could read a bucket it should never have touched. Can you prove the agent did that, and prove it did nothing else you have not noticed yet. If your agents share a service account and nobody reads the audit log against the transcript, the honest answer is no. The evidence is not missing. Nobody is reading it against the agent's word. That is a strange place to have arrived, given the logs were sitting in the account the entire time.

Resources & Further Reading

• SLSA (Supply-chain Levels for Software Artifacts): https://slsa.dev/ - Provenance format and assurance levels for build artifacts
• in-toto: https://in-toto.io/ - Framework for binding each step of a supply chain to the identity that performed it
• Sigstore: https://www.sigstore.dev/ - Signing and verification for software artifacts and attestations
• AWS CloudTrail documentation: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-events.html - Management and data event logging for AWS API calls
• Kubernetes Auditing: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/ - API server audit policy, stages, and levels
• GitHub enterprise audit log: https://docs.github.com/en/enterprise-cloud@latest/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/accessing-the-audit-log-for-your-enterprise - Repository and organization activity records and retention
• 2026 Identity Security Landscape: https://www.paloaltonetworks.com/idira/identity-security-landscape-report - Machine-to-human identity ratio and growth drivers (Palo Alto Networks/CyberArk)
• Gartner agentic AI predictions: https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027 - Autonomy projections and project cancellation forecast
• 200 OK, Wrong Answer: https://stxkxs.io/blog/ai-observability-200-ok - Why a clean status code is not evidence of a correct AI output
• The Agentic DevOps Loop: https://stxkxs.io/blog/agentic-devops-loop - The closed loop where agents deploy and remediate without a human in the path
• Your Dependencies Have Dependencies: https://stxkxs.io/blog/supply-chain-security-ai-era - Supply-chain provenance and the limits of trusting a build's own report

KubeCon Europe ran March 23–26 in Amsterdam with [nearly 13,500 attendees](https://www.cncf.io/reports/kubecon-cloudnativecon-europe-2026/), per CNCF's post-event report; vendor recaps from OVHcloud and Sokube called it the largest KubeCon Europe to date, a claim CNCF's own report does not make. Dynamic Resource Allocation had reached general availability in upstream Kubernetes v1.34 the prior August (Aug 27, 2025).

On March 24, 2026, CNCF's [v1.35 Kubernetes AI Requirements](https://www.cncf.io/announcements/2026/03/24/cncf-nearly-doubles-certified-kubernetes-ai-platforms/) expanded the AI Conformance Program (co-led by Google's Janet Kuo; launched at KubeCon NA, November 2025) to validate agentic workloads: multi-step, tool-calling agents, alongside training and inference. One requirement set covers all three (DRA support, Gateway API interoperability, OCI-format model artifacts, observability integration). There is no separate certification tier for agents.

The CNCF sandbox absorbed a wave of AI-infra projects spotlighted at the show: llm-d (inference disaggregation, from IBM Research, Red Hat, and Google Cloud; accepted March 12, 2026), HolmesGPT (an AI SRE agent from Robusta.dev with major contributions from Microsoft; accepted October 8, 2025), and KAI Scheduler (GPU-aware scheduling; accepted December 21, 2025). NVIDIA donated its DRA driver to the Kubernetes project for full community ownership at the show, seven months after DRA GA. Solo.io donated agentregistry to the CNCF and introduced agentevals as open source. On May 7, Solo.io added NemoClaw support to kagent, moving NVIDIA's sandbox-first OpenClaw reference stack from a single-host deployment to a Kubernetes-native fleet.

Kubernetes is becoming the agent operating system. The CRDs and gateways are open-source projects with sandbox status, vendor-supported distributions, and conformance criteria. Two years ago this was a vendor pitch. Today it is a CNCF program. The [CNCF Annual Cloud Native Survey](https://www.cncf.io/announcements/2026/01/20/kubernetes-established-as-the-de-facto-operating-system-for-ai-as-production-use-hits-82-in-2025-cncf-annual-cloud-native-survey/) (January 2026) found that 66% of organizations hosting generative AI models use Kubernetes for some or all inference workloads. Agents are the next workload class on the same infrastructure. Platform teams inherit them by default, not by choice.

Key Point: Agents land on Kubernetes because inference already grew the parts they need: fractional GPU scheduling, protocol mediation, identity, observability, scale-to-zero. Agents arrive late and reuse what is there.

• GenAI orgs on K8s: 66% — Share of organizations hosting generative AI models that use Kubernetes for some or all inference workloads (CNCF Annual Cloud Native Survey, Jan 2026)
• DRA Status: GA, Aug 2025 (v1.34) — Dynamic Resource Allocation reached general availability in upstream Kubernetes v1.34 (Aug 27, 2025); Red Hat OpenShift 4.21 shipped DRA GA downstream at KubeCon EU 2026 in Amsterdam
• Enterprise Agent Adoption: 70% target — Gartner forecast for 2029 (70% of enterprises will deploy agentic AI as part of IT infrastructure operations); up from less than 5% in 2025
• Inference Cost Trajectory: 90% lower by 2030 — Gartner projects inference cost for a 1T-parameter LLM will fall more than 90% from 2025 levels by 2030

Kubernetes ate ML training

This pattern has played out before. Java EE produced WebLogic, JBoss, and WebSphere. Kubernetes ate those by treating apps as containers and letting the orchestration layer absorb the lifecycle. The app-server vendors lost because the platform layer moved beneath them.

ML training was the second round. Kubeflow, KServe, vLLM, and the GPU operator stack pushed training and serving into Kubernetes between 2018 and 2024. By 2025 model serving on K8s was the default: the cluster already provided scheduling, GPU device plugins, autoscaling, ingress, secrets, and policy. Adding model serving was a smaller step than building all of that elsewhere. Agents follow the same arc.

Agents need fractional GPU access for embedding and reranking, topology-aware scheduling for multi-step inference, scale-to-zero for sporadic workloads, and protocol mediation for MCP, A2A, and LLM APIs. None of those are new. Scale-to-zero shipped with KFServing in 2019 (renamed KServe in 2021). NVIDIA's GPU operator shipped topology awareness. Knative demonstrated request-driven scaling years before agents were a category. The agent runtime is assembled from pieces that already shipped.

The agent runtime is assembled from pieces Kubernetes already shipped. The few that were missing landed this spring.

DRA fixes GPU scheduling

For a decade, GPU scheduling on Kubernetes meant `nvidia.com/gpu: 1`. The device plugin treated accelerators as opaque integer counts: a whole GPU or none. There was no way to ask for a 20GB MIG slice on an H100, NVLink to a neighboring pod, or a particular firmware revision. Training tolerated that model because training jobs want whole GPUs. Inference and agent workloads want the opposite: thin slices, short bursts, fan-out across many small calls.

Dynamic Resource Allocation introduces four built-in API types in the `resource.k8s.io` group: `ResourceClaim`, `ResourceClaimTemplate`, `DeviceClass`, and `ResourceSlice`. Vendor drivers register what they can offer. Pods declare what they need. The scheduler matches them against structured constraints. Same move Kubernetes made for storage with `PersistentVolumeClaim` and `StorageClass`: a vendor-specific pile of devices becomes a typed, claimable resource.

NVIDIA donated its DRA driver to the Kubernetes project at KubeCon EU 2026, seven months after DRA GA. AMD's GPU DRA driver reached v1.0.0 on May 20, 2026. Intel ships its own resource drivers for accelerators. The abstraction is multi-vendor by construction. A cluster can host H100s, MI300Xs, Gaudi 3s, and TPU v5e's in the same node pool, and a pod can claim "8GB of FP16 inference capacity, latency-class A" without pinning to a specific SKU.

INFO: DRA vs the device plugin
Device plugins exposed integer counts of opaque resources. DRA exposes typed claims with vendor-specific selectors. Fractional GPU, MIG slices, multi-vendor accelerators, NUMA-aware placement, and topology hints become scheduler policy instead of webhooks and custom controllers.

# DRA claim for an agent that needs a 20GB H100 MIG slice
# with NVLink topology awareness. The driver-specific selector
# decouples the workload from the vendor SKU.
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: research-agent-gpu
spec:
  devices:
    requests:
      - name: inference-slice
        deviceClassName: nvidia.com/h100-mig
        selectors:
          - cel:
              expression: |
                device.attributes["memory.gb"] >= 20 &&
                device.attributes["topology.nvlink"] == true
---
apiVersion: v1
kind: Pod
metadata:
  name: research-agent
spec:
  resourceClaims:
    - name: gpu
      resourceClaimName: research-agent-gpu
  containers:
    - name: agent
      image: registry.example.com/agents/research:v1
      resources:
        claims:
          - name: gpu

The bin-packing problem on shared GPUs is brutal. Inference is bursty and short. A naive one-GPU-per-pod policy burns most accelerator hours on idle. DRA does not solve bin-packing. It gives the scheduler enough information to place workloads well and gives downstream tools (kagent, autoscalers, FinOps gates) a structured object to reason about. The previous model exposed none of that.

Agents become pods

kagent was sandboxed in May 2025 by Solo.io. It models agents, tools, and model bindings as CRDs. An `Agent` object specifies a model, a tool list, a system prompt, and resource claims. The controller reconciles each one into a pod or set of pods that calls whichever model server runs in the cluster.

Agent frameworks are not scarce. What kagent contributes is alignment with the rest of the cluster. The framework reuses identity (service accounts), secrets (Secrets API), telemetry (OpenTelemetry), scheduling (now DRA-aware), and policy (admission webhooks). The agent is a pod, so those come along for free.

apiVersion: kagent.dev/v1alpha1
kind: Agent
metadata:
  name: incident-summarizer
  namespace: sre
spec:
  model:
    provider: anthropic
    name: claude-sonnet-4-6
    routing: agentgateway-prod
  systemPrompt: |
    You summarize PagerDuty incidents into 5-bullet briefs.
    Always cite the source incident ID.
  tools:
    - name: pagerduty
      mcpRef: pagerduty-mcp-server
    - name: github-search
      mcpRef: github-mcp-server
  resources:
    claims:
      - name: gpu
        resourceClaimTemplateName: small-inference-slice
  rbac:
    serviceAccountName: incident-agent
  scaling:
    minReplicas: 0
    maxReplicas: 8
    activationTarget: 1

kagent does not call the model provider directly. It hands the call to a gateway, the way a service mesh hands HTTP traffic to a sidecar. The `routing: agentgateway-prod` field is the seam. That decoupling is the second half of the stack.

EXAMPLE: NemoClaw lands inside kagent
NVIDIA introduced NemoClaw in March 2026 as a reference stack for safely running OpenClaw agents, with sandbox-first isolation as the core idea. On May 7, 2026, Solo.io added NemoClaw support to kagent, bringing that pattern from a single-host reference into a fleet-scale Kubernetes platform with lifecycle controls, identity, and policy.

The gateway hides the providers

Solo.io contributed agentgateway to the Linux Foundation in August 2025 (announced Aug 25, 2025). The data plane is written in Rust. It mediates four protocol families that did not share a sentence three years ago: inference (vLLM, TGI, Triton style), OpenAI-compatible HTTP, MCP (Anthropic's tool-call protocol), and A2A (Google's agent-to-agent handoff). One process, four protocols.

The agent process never sees which model provider is on the other end, what its rate limit is, what authentication MCP servers expect, or which A2A peer picks up the next handoff. The gateway resolves all of that. Calls go to a local endpoint and return structured responses. The pattern is Envoy plus Istio for HTTP, with stateful, token-billed, silent-failure-prone protocols substituted in.

apiVersion: agentgateway.io/v1
kind: ModelRoute
metadata:
  name: claude-sonnet-route
spec:
  match:
    model: claude-sonnet-4-6
  upstream:
    provider: anthropic
    region: us-west
    failover:
      - provider: anthropic
        region: us-east
  rateLimit:
    tokensPerMinute: 200000
  costGuard:
    maxUsdPerHour: 50
  observability:
    tracing: otel
    promptLog: redacted
---
apiVersion: agentgateway.io/v1
kind: MCPRoute
metadata:
  name: pagerduty-mcp-route
spec:
  serverRef: pagerduty-mcp-server
  authPolicy:
    method: oauth-on-behalf-of
  scopeAllowlist:
    - incidents:read
    - incidents:annotate

The cost guard is a first-class field. Token spend has become the new bandwidth bill, and it has to be enforced as policy at the gateway, not flagged on a dashboard a day later. MCP routes carry an `authPolicy` with on-behalf-of semantics. MCP launched without a robust authorization story; the gateway is where the gap closes in production.

Conformance drew the line

The AI Conformance Program defines what "AI-ready" means for a Kubernetes platform: DRA, Gateway API interoperability, OCI-format model artifacts, and observability integration, applied as one requirement set. A conformance program for a runtime is what a spec is for a protocol; it forces vendors to either pass the tests or explain why not. Co-led by Google (Janet Kuo) with Microsoft, Kubermatic, and Red Hat, it launched at KubeCon NA 2025 (Atlanta, Nov 11; beta at KubeCon Japan in June).

The March 24, 2026 expansion to agentic workloads answers a question platform teams had been asking per-vendor: does this distribution actually run our AI workloads, or are we about to spend a quarter rewriting YAML? Conformance answers it once across a broader set of workload shapes. The program covers the runtime layer. It does not cover agent lifecycle semantics, prompt evaluation, billing attribution, or safety policy. Those remain where vendors compete and where the platform team decides locally. The line in 2026 is drawn at the kernel, not the application.

TIP: Demand conformance
When evaluating managed Kubernetes for agent workloads, ask whether the vendor targets the Conformance program and whether that certification covers agentic workloads. A "proprietary AI mode" outside conformance is the OpenStack-era pattern of incompatible distributions. The DRA, kagent, and agentgateway stack is open and multi-vendor by construction. Pick distributions that honor that.

Don't build it yet

This stack is not the right answer for every agent workload. For some, kagent, agentgateway, and DRA is over-engineering. Building the platform before the workloads exist is a common failure mode.

Single-agent applications with no scale-out story do not need it. One agent, one product, one model, predictable load: run it on Lambda or Cloud Run. The managed-function cost is rounding error against the platform time a Kubernetes-native runtime requires. kagent earns its weight when multiple agents share tools, identity, or budget.

Sub-second tool-augmented chat is another miss. When the agent sits in the request path of a streaming chat interface, a single inference endpoint with client-side tool dispatch beats MCP-over-mesh. [Istio's own benchmarks](https://istio.io/latest/blog/2019/performance-best-practices/) put mesh overhead at roughly 3ms p50 and 10ms p99 through two sidecar proxies at 1000 RPS. That pays off for autonomous, fan-out, asynchronous agents. It does not when a user is watching tokens stream.

Single-team prototypes are premature for platform tooling until at least three teams run agents. Below that, two teams can share an OpenAI-compatible gateway and a shared GPU pool with no CRDs, cheaper for a year than a kagent install. Adopt the platform when the integration tax across teams exceeds the platform tax.

Teams that never leave one cloud should weigh Bedrock AgentCore, Vertex AI Agent Engine, and Azure AI Foundry first. For a single-cloud team with no on-prem footprint, those are fewer moving parts than kagent, DRA, and agentgateway combined. The Kubernetes path wins when the estate is multi-vendor, spans on-prem or regulated infrastructure, or already runs a cluster the agents can join. It loses when the workload never leaves one cloud's control plane.

WARNING: The premature platform trap
Platform teams fail in a recognizable way: building infrastructure before the workloads exist. The K8s, DRA, kagent, and agentgateway stack is genuinely useful, but adopting it before agents are in production designs for use cases that will not match what arrives. Ship one or two agent workloads on the simplest setup first. Build the platform when the second team's agents collide with the first team's.

The platform team inherits

The platform team that runs your cluster will run your agents, whether they signed up for it or not. The workloads land on infrastructure already underneath them. The pieces are there. They are not yet good enough.

Above the runtime, prompt evaluation, budget attribution at the agent level, tool registry curation, MCP authentication at scale, and A2A trust boundaries are not standardized. Each falls to the platform team. The runtime layer is becoming a commodity. Governance and economics on top are not.

For the next quarter: do not fork kagent, do not write a DRA driver, do not implement an agent gateway from scratch. Use the open versions. Build the layers above. Agent identity mapped to existing IAM. Cost gates wired into the FinOps stack. Evaluation harnesses that gate promotion through environments. Observability that makes a misbehaving agent legible to an on-call engineer who has never read its prompt. That work survives the next round of platform consolidation. The forks do not.

[nanohype's eks-agent-platform](https://github.com/nanohype/eks-agent-platform) is one shape of that work: a `Platform` custom resource provisions per-tenant IRSA and KMS grants, a `BudgetPolicy` resource ties an hourly CUR/Athena cost check to an EventBridge kill-switch at 120% of budget, and an `EvalSuite` resource runs Argo Workflows against the fleet before promotion.

Key Point: The forcing function is the next conformance revision. When agentic-workload validation criteria harden further, the platform teams that already built governance, economics, and identity on top of the runtime will meet it on schedule. The ones that did not will meet it on a deadline.

Resources & Further Reading

• CNCF Kubernetes AI Conformance Program expansion: https://www.cncf.io/announcements/2026/03/24/cncf-nearly-doubles-certified-kubernetes-ai-platforms/ - March 24, 2026: v1.35 Kubernetes AI Requirements add validation for agentic workloads alongside training and inference
• CNCF Annual Cloud Native Survey (Jan 2026): https://www.cncf.io/announcements/2026/01/20/kubernetes-established-as-the-de-facto-operating-system-for-ai-as-production-use-hits-82-in-2025-cncf-annual-cloud-native-survey/ - 66% of orgs hosting GenAI models use Kubernetes for some or all inference; 82% of container users run K8s in production
• CNCF KubeCon + CloudNativeCon Europe 2026 post-event report: https://www.cncf.io/reports/kubecon-cloudnativecon-europe-2026/ - Official attendance figures and event recap
• Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/ - ResourceClaim, ResourceClaimTemplate, and DeviceClass; GA in upstream Kubernetes v1.34 (Aug 2025)
• kagent (CNCF Project): https://www.cncf.io/projects/kagent/ - Kubernetes-native agent runtime; agents, tools, and model bindings as CRDs
• kagent on GitHub: https://github.com/kagent-dev/kagent - Source, examples, and CRD definitions
• agentgateway: https://agentgateway.dev/ - Rust-built data plane for inference, LLM, MCP, and A2A traffic; contributed to the Linux Foundation in August 2025
• Istio performance best practices: https://istio.io/latest/blog/2019/performance-best-practices/ - Sidecar-proxy latency benchmarks (~3ms p50, ~10ms p99 at 1000 RPS)
• Solo.io brings NemoClaw to kagent (May 7, 2026): https://www.globenewswire.com/news-release/2026/05/07/3290085/0/en/solo-io-brings-nemoclaw-to-production-agentic-runtime-for-kubernetes.html - NemoClaw support in kagent; NVIDIA reference stack on Kubernetes-native runtime
• NVIDIA NemoClaw: https://github.com/NVIDIA/NemoClaw - Reference stack for running OpenClaw agents in a sandbox-first runtime
• NVIDIA DRA Driver: https://github.com/NVIDIA/k8s-dra-driver - Vendor DRA for NVIDIA accelerators; donated to the Kubernetes project at KubeCon EU 2026
• AMD GPU DRA Driver: https://github.com/ROCm/k8s-gpu-dra-driver - DRA-compliant driver for ROCm-based AMD accelerators (v1.0.0, May 20, 2026)
• Intel Resource Drivers for Kubernetes: https://github.com/intel/intel-resource-drivers-for-kubernetes - DRA-based resource drivers for Intel accelerators
• KubeCon + CloudNativeCon Europe 2026 highlights (Solo.io): https://www.solo.io/blog/highlights-from-kubecon-cloudnativecon-europe-2026 - Agent-runtime track: agentgateway, kagent, agentregistry
• eks-agent-platform (nanohype): https://github.com/nanohype/eks-agent-platform - Per-tenant IAM, agentgateway, kagent, and Argo Workflows eval pipeline via Tenant/Platform custom resources
• Self-Hosted AI Agents for Incident Response: https://stxkxs.io/blog/openclaw-self-hosted-ai-agents - Background on OpenClaw, the agent NVIDIA NemoClaw is built to run securely
• MCP Is Now a Linux Foundation Standard: https://stxkxs.io/blog/mcp-linux-foundation - Protocol context for MCP, the tool-call protocol mediated by agentgateway
• Gartner inference economics forecast (March 2026): https://www.gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025 - Quantitative grounding for the inference cost trajectory

The FinOps Foundation has 95,000+ members (finops.org: 95k+ community). 97 of the Fortune 100 are represented in that community. The cloud FinOps market hit $13.5 billion in 2024. And cloud waste, which Flexera defines as the percentage of cloud spend that produces no value, rose to 29% in 2026, the first increase in five years, with AI workloads cited as the driver (Flexera 2026 State of the Cloud). The cloud infrastructure market grew roughly 45% over the same period, from $228 billion in 2022 to $330 billion in 2024 (Synergy Research). FinOps got adopted at scale. Waste did not fall to match.

The explanation is structural. FinOps was adopted as a procurement and finance initiative: dashboards for leadership, monthly cost review meetings, chargeback reports that nobody outside finance reads. The people making provisioning decisions are engineers, and most of them never see the dashboard. Those decisions happen in code, in a PR, in a CI pipeline, long before any monthly report reaches them.

Cost needs to be a first-class engineering signal: in CI, in IDPs, in IaC, in autoscaling. The historical precedent is reliability. The transformation that SRE drove for uptime needs to happen for cost. AI infrastructure costs are growing faster than any other line item, so the window to get this right is closing.

• FinOps Adoption: 95K+ members — FinOps Foundation community size; 97 of Fortune 100 represented (finops.org)
• Cloud Waste: 29% — Percentage of cloud spend that is wasted; rose in 2026 for the first time in five years, driven by AI workloads (Flexera 2026)
• FinOps Market: $13.5B — Cloud FinOps market ~$13.5–13.6B in 2024; MarketsandMarkets later projects ~$14.9B in 2025 to $26.91B by 2030 at 12.6% CAGR

Reliability had this problem first

Before Google published the SRE book in 2016, reliability was a separate team's problem. Developers wrote code. Operations kept it running. Monitoring was a dashboard that ops checked. Uptime was someone else's KPI. The organizational structure existed: NOCs, operations teams, incident managers, change advisory boards. The processes existed: runbooks, escalation procedures, post-mortems. Systems were still unreliable, because the people writing the code that caused outages were structurally disconnected from the signals about reliability.

SRE fixed this by making reliability an engineering concern with engineering primitives. SLOs gave developers a budget they could spend. Error budgets created a feedback loop between deployment velocity and production stability. On-call rotations put developers in the blast radius of their own decisions. Everyone already agreed that reliability mattered. The insight was that reliability improves when the signal reaches the decision point: the engineer writing the code, not the ops team cleaning up afterward.

FinOps is stuck in the pre-SRE phase. The organizational adoption happened. The reporting layer exists. The monthly cost review meetings are on the calendar. But cost signals do not reach engineers at decision time. A developer provisioning an oversized instance does not see the cost delta in their PR. A team spinning up a GPU cluster does not get a cost estimate before deployment. The FinOps team publishes a report; engineers do not read it. The bottleneck is signal delivery.

EXAMPLE: The early movers
Spotify's open-source [Backstage Cost Insights plugin](https://backstage.io/blog/2020/10/22/cost-insights-plugin/) alerts teams when a project's cost growth exceeds a chosen threshold, surfacing that signal inside the IDP itself instead of in a separate report. The pattern works. It stays niche.

The dashboard isn't reaching engineers

The State of FinOps 2026 report surveyed 1,192 respondents managing over $83 billion in annual cloud spend. 78% of FinOps practices now report into the CTO or CIO, up 18 percentage points versus 2023. That structural shift toward engineering leadership has happened. Most of those practices still run as a reporting function inside an engineering org chart, not as an engineering workflow.

The gap between organizational adoption and operational maturity reveals the core problem. FinOps was designed as a practice: cross-functional teams, chargeback models, optimization recommendations reviewed in meetings. That works for procurement-scale decisions like reserved instance purchases, enterprise discount programs, and right-sizing recommendations reviewed quarterly. It does not work for the hundreds of daily provisioning decisions engineers make: instance types in Terraform modules, container resource limits in Kubernetes manifests, GPU allocations for inference workloads. Those decisions happen in code, in PRs, in CI pipelines. A monthly report cannot influence them.

The multi-cloud dimension makes it worse. Flexera 2025 reports organizations average 2.4 public cloud providers and 70% embrace hybrid strategies; multi-cloud presence is the norm even when designs are not intentional. Cost governance rarely keeps pace across all of them. Each cloud has a different billing model, different discount structures, different cost allocation taxonomy. The FOCUS spec (the FinOps Open Cost and Usage Specification, v1.3 ratified December 2025) is trying to normalize this. 68% of large spenders ($100M+ annually) are using or experimenting with FOCUS-formatted data (State of FinOps 2026). Normalization solves the data problem. Delivery is a separate problem. Unified billing data in a dashboard is still a dashboard.

In a 2023 Spot by NetApp survey of 310 US IT decision-makers, 96% of tech executives agreed FinOps is important to their cloud strategy, yet only 9% had a mature practice. The bottleneck is execution architecture. The data and the tools are both in place. What is missing is cost feedback that reaches engineers at the moment they make provisioning decisions.

Key Point: The FinOps maturity gap is a signal-delivery problem. The data and dashboards already exist. What is missing is cost feedback reaching engineers when they make provisioning decisions: the PR comment, the CI gate, the IDP budget widget.

AI makes this urgent

Everything above applies to traditional cloud spend. AI infrastructure makes it existential. Average GPU utilization across enterprise Kubernetes clusters is 5%, per Cast AI's 2026 State of Kubernetes Optimization Report (23,000 clusters analyzed): 95% of provisioned GPU capacity sits idle. Idle GPU-hours bill at the same rate as busy ones, so most of an AI infrastructure budget pays for capacity that produces nothing. Inference, not training, is the ongoing bill for most AI workloads once a model ships, and waste on AI workloads tracks higher than waste on traditional cloud.

Token-based and GPU-based pricing does not map to traditional billing frameworks. A single LLM inference request can cost a fraction of a cent against a small model with a short prompt, or several dollars against a frontier model at maximum context with no cached tokens. Traditional cost allocation assumes relatively stable per-unit pricing. AI workloads break that assumption. The top tooling request in the State of FinOps 2026 report is granular monitoring of AI spend (tokens, LLM requests, GPU utilization), and no vendor yet covers token-, request-, and GPU-level attribution in a single product.

Two years ago, 31% of FinOps practices managed AI spend. Today it is 98%. FinOps for AI is now the number one forward-looking priority, and AI value management is the top skillset teams are seeking to add. The FinOps Foundation updated its mission from "advancing the people who manage the value of Cloud" to "advancing the people who manage the Value of Technology." The ambition grew faster than the execution model of dashboards and meetings.

FinOps scope expansion (2026)

• AI Spend: 98%
• SaaS: 90%
• Licensing: 64%
• Private Cloud: 57%
• Data Center: 48%

WARNING: GPU waste is different
GPU instances cost roughly 5-6x more per hour than comparable large CPU instances: a p5.48xlarge on AWS costs $55.04/hour on demand in us-east-1, versus about $9.68/hour for a similarly sized m7i.48xlarge CPU instance. At 5% utilization, the p5.48xlarge alone wastes roughly $52/hour. The cloud-wide waste rate applied to GPU workloads translates to dollar amounts that make traditional cloud waste look trivial. GPU optimization requires understanding model serving patterns, batching strategies, and inference scheduling. Those are engineering decisions.

Shifting cost into engineering

The fix is architectural. Cost needs to be a signal in the systems engineers already use: CI pipelines, [infrastructure-as-code](https://stxkxs.io/blog/iac-landscape-2026), internal developer platforms, and autoscaling policies. Three tools cover the leading edge: Infracost for cost estimates in pull requests, OpenCost for real-time Kubernetes cost allocation, and CDK Budgets for deploying cost guardrails as infrastructure. Each addresses a different point in the engineering decision chain.

Infracost: cost in the PR

Infracost generates cloud cost estimates for Terraform and OpenTofu changes and posts them as PR comments. The engineer sees the cost impact of their infrastructure change before it merges, inside the same code review workflow they already use. The integration is a CI step, not a separate tool to learn.

name: Infracost
on: [pull_request]

jobs:
  infracost:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      - name: Setup Infracost
        uses: infracost/actions/setup@v4
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Generate cost estimate
        run: |
          infracost breakdown --path=. \
            --format=json \
            --out-file=/tmp/infracost.json

      - name: Post PR comment
        run: |
          infracost comment github \
            --path=/tmp/infracost.json \
            --repo=${{ github.repository }} \
            --pull-request=${{ github.event.pull_request.number }} \
            --github-token=${{ secrets.GITHUB_TOKEN }} \
            --behavior=update

That is the entire integration. A CI step that runs on every PR touching infrastructure files. The engineer sees "$427/month -> $1,240/month" in their PR comment. The cost delta is visible at decision time. No dashboard required. This is the SRE equivalent of a latency regression test: a change that makes things more expensive shows up before merge, not after the invoice arrives.

OpenCost: runtime cost allocation

OpenCost is a CNCF Incubating project (promoted from Sandbox on Oct 25, 2024) that provides a cloud-agnostic API for real-time Kubernetes cost metrics. It allocates costs to namespaces, deployments, pods, and labels using actual cloud billing rates. Where Infracost catches cost problems before deployment, OpenCost catches them in production. It identifies overprovisioned workloads, idle resources, and cost anomalies at the namespace level.

# OpenCost deployment with Prometheus integration
opencost:
  exporter:
    defaultClusterId: "production"
    extraEnv:
      CLOUD_PROVIDER_API_KEY: "<cloud-billing-api-key>"
      EMIT_KSM_V1_METRICS: "false"

  prometheus:
    internal:
      enabled: true
    external:
      enabled: false

  ui:
    enabled: true
    ingress:
      enabled: true
      hosts:
        - host: costs.internal.company.com

OpenCost exposes a /allocation API that returns cost-per-namespace, cost-per-deployment, and cost-per-label in real time. Platform teams wire this into Backstage or Grafana views. The more important wiring is into alerting. A namespace exceeding its cost budget triggers a Slack notification to the owning team. The signal reaches the team that can act on it, at the time they can act on it.

CDK: budgets as infrastructure

For AWS-native teams, cost guardrails belong in the infrastructure stack itself. AWS Budgets and Cost Anomaly Detection can be provisioned as CDK constructs alongside the resources they monitor. The budget becomes an infrastructure resource deployed with the stack it governs.

import * as cdk from 'aws-cdk-lib'
import * as budgets from 'aws-cdk-lib/aws-budgets'
import * as sns from 'aws-cdk-lib/aws-sns'
import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions'
import { Construct } from 'constructs'

export class CostGuardrails extends Construct {
  constructor(scope: Construct, id: string, props: {
    monthlyBudget: number
    teamEmail: string
    environment: string
  }) {
    super(scope, id)

    const topic = new sns.Topic(this, 'CostAlertTopic')
    topic.addSubscription(
      new subscriptions.EmailSubscription(props.teamEmail)
    )

    new budgets.CfnBudget(this, 'MonthlyBudget', {
      budget: {
        budgetName: `${props.environment}-monthly`,
        budgetType: 'COST',
        timeUnit: 'MONTHLY',
        budgetLimit: { amount: props.monthlyBudget, unit: 'USD' },
      },
      notificationsWithSubscribers: [{
        notification: {
          notificationType: 'ACTUAL',
          comparisonOperator: 'GREATER_THAN',
          threshold: 80,
        },
        subscribers: [{
          subscriptionType: 'SNS',
          address: topic.topicArn,
        }],
      }],
    })
  }
}

The budget is co-located with the infrastructure, owned by the same team, deployed by the same pipeline, and version-controlled in the same repo. When the team changes the infrastructure, the budget travels with it. This is the CDK equivalent of deploying an SLO alarm alongside the service it monitors. The cost constraint becomes part of the infrastructure definition, not an afterthought managed in a different console.

TIP: Start with Infracost
The single highest-impact change from this post is adding Infracost to a CI pipeline. It takes 15 minutes, requires no infrastructure changes, and gives every engineer cost visibility at PR time. OpenCost and CDK budgets require more platform investment but close the loop in production. Layer them in that order: PR visibility first, runtime allocation second, infrastructure guardrails third.

FOCUS fixes the data layer

The FinOps Open Cost and Usage Specification (FOCUS) v1.3 was ratified in December 2025. AWS, Azure, GCP, Oracle Cloud, and Tencent Cloud have adopted it. For the first time, there is a common schema for billing data across major cloud providers: normalized resource identifiers, consistent pricing units, standardized commitment discount representation. This solves the data normalization problem that has plagued multi-cloud cost governance since organizations started splitting workloads across providers.

Vantage is the most interesting commercial tool in this space. It has 20+ native integrations spanning AWS, Azure, GCP, Kubernetes, Snowflake, Datadog, OpenAI, Anthropic, MongoDB Atlas, and Databricks. It aggregates cost data from infrastructure and SaaS providers into a single view. More importantly, it ships an MCP server and an automated FinOps agent that takes action on cost anomalies (removing unattached EBS volumes and obsolete snapshots based on configurable policies). Cost management starts looking less like reporting and more like autonomous remediation.

FOCUS and aggregation tools solve the data normalization problem. They do not change engineer behavior. The value of FOCUS is as a data layer underneath engineering tools, feeding normalized cost data into Infracost estimates, OpenCost allocations, IDP budget widgets, and CI cost gates. The standard matters because it makes the plumbing reliable. The plumbing matters because it delivers cost signals to engineers. If the data stays in a dashboard that only the FinOps team checks, the standard changes nothing.

INFO: FOCUS adoption status
Among organizations spending $100M+ annually on cloud, 68% are using or experimenting with FOCUS-formatted billing data. Another 18% plan to adopt it. The remaining 14% are not planning to use it. AWS, Azure, GCP, Oracle, and Tencent already export FOCUS-formatted data natively. The bottleneck is practitioner tooling that consumes FOCUS data and routes it to engineering workflows. Provider support is no longer the gating problem.

When shifting left fails

Shifting cost left is the right move for organizations making hundreds of provisioning decisions in code every week. It is the wrong move for several cases, and the tools carry real limits that a PR comment hides.

Infracost's free CLI estimates are list-price approximations by default. They price resources off public on-demand rates and do not see committed-use discounts, reserved-instance coverage, or savings plans unless a team pays for Infracost Cloud, which can apply enterprise discount programs (AWS EDP, Azure Prepayment, Google Commitment Agreement) and custom price books. A team running the free CLI against a 40% EDP commitment reads a PR comment that overstates real cost by a wide margin, and a "$1,240/month" delta that lands inside negotiated pricing reads as alarming when it is not. The estimate is useful for relative comparison between two changes. Treated as an absolute bill without the paid tier's discount configuration, it misleads.

OpenCost is not free to run. It adds a deployment to every cluster, depends on Prometheus for metrics storage, and the cost data is only as accurate as the cloud billing rates fed into it. A platform team without an existing Prometheus stack inherits the operational burden of running and scaling one, plus the long-term storage cost of the cost metrics themselves. The allocation API answers cost-per-namespace questions, but someone has to own the deployment, the upgrades, and the alerting wiring that makes the data actionable.

Cost gates that block merges create developer friction. A gate that fails a PR above a cost threshold will fire on legitimate changes: a new environment, a deliberate capacity increase, a migration that is expensive on paper and cheaper in practice. Every false positive trains engineers to treat the gate as noise and reach for the override. A cost gate works as a non-blocking comment that informs the reviewer. As a hard merge block, it competes with shipping, and shipping wins.

Below a certain org size, none of this is worth maintaining. A team running a handful of services on a single account, where one person can read the monthly bill and recognize every line, does not need a cost gate, a Kubernetes allocation API, or a FOCUS pipeline. The maintenance cost of the tooling exceeds the waste it would catch. The argument for shifting cost left scales with the number of independent provisioning decisions, not with the existence of a cloud bill. Adopt it when no single person can hold the spend in their head.

Key Point: Infracost's free CLI prices off public on-demand rates; committed-use and EDP discounts need Infracost Cloud price books, so free-tier numbers are comparative, not absolute. OpenCost carries Prometheus and platform operating cost. Hard cost gates generate false positives that erode trust. Below the org size where provisioning decisions outrun any one person's attention, the tooling costs more than the waste it prevents.

Cost becomes an SLO

FinOps reporting is moving from finance to engineering leadership: 78% under CTO/CIO, up 18 points in three years. The FOCUS spec is normalizing the data layer across providers. Infracost, OpenCost, and Vantage are building the tooling that routes cost signals into engineering workflows. AI infrastructure costs are creating urgency that traditional cloud spend never did. The missing piece is organizational adoption of cost-as-engineering-signal at the same scale that SRE drove reliability-as-engineering-signal.

Within 18 months, cost gates in CI pipelines will become as common as security scans. Infracost or equivalent tools will integrate into standard PR workflows with configurable thresholds that block merges above a cost delta. Second, internal developer platforms (see "[Platform Engineering in the AI Era](https://stxkxs.io/blog/platform-engineering-ai-era)" on this blog) will surface cost per service as a first-class metric alongside latency and error rate. Backstage, Cortex, and Port already have the plugin architecture for it. Third, AI cost attribution will force the issue. A single p5.48xlarge inference endpoint running around the clock costs roughly $40,000 a month at $55.04/hour; at single-digit utilization, the team running it will demand real-time cost visibility regardless of whether the organization has a FinOps practice.

The engineer provisioning the next GPU cluster sees the cost before merge or after the invoice arrives.

Key Point: FinOps adopted as a procurement practice produced dashboards. FinOps adopted as an engineering practice produces CI cost gates, IDP budget widgets, IaC cost constructs, and autoscaling policies with cost constraints. The data says adoption is massive and waste is unchanged. Deliver cost feedback where engineers already work: the pull request, the CI pipeline, and the deployment.

Resources & Further Reading

• State of FinOps 2026 Report: https://data.finops.org/ - Sixth annual survey covering 1,192 respondents representing $83B+ in managed cloud spend
• FOCUS Specification (v1.3): https://focus.finops.org/ - FinOps Open Cost and Usage Specification for normalized billing data across providers
• Infracost: https://www.infracost.io/ - Cloud cost estimates for Terraform in pull requests
• Infracost GitHub: https://github.com/infracost/infracost - Open-source CLI and CI integrations for infrastructure cost estimation
• OpenCost: https://www.opencost.io/ - CNCF Incubating project (promoted from Sandbox on October 25, 2024) for real-time Kubernetes cost monitoring and allocation
• OpenCost GitHub: https://github.com/opencost/opencost - Cloud-agnostic cost allocation API for Kubernetes workloads
• Vantage: https://www.vantage.sh/ - Multi-cloud cost platform with 20+ native integrations including AI providers
• AWS Budgets CDK Documentation: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_budgets-readme.html - CDK constructs for AWS Budgets and cost management
• Flexera 2026 State of the Cloud Report: https://www.flexera.com/blog/finops/flexera-2026-state-of-the-cloud-report-the-convergence-of-cloud-and-value/ - Annual cloud report covering spend, waste, and FinOps maturity
• FinOps Foundation FOCUS Adoption Guide: https://www.finops.org/wg/adopting-focus-the-finops-open-cost-and-usage-specification/ - Working group guide for adopting the FOCUS billing data standard
• Google SRE Book: https://sre.google/sre-book/table-of-contents/ - The foundational text on making reliability an engineering concern

The five stages of the DevOps loop (Code, Deploy, Observe, Diagnose, Remediate) have each been independently augmented by AI for years. The connection between stages has not. An AI agent that writes a fix does not trigger the deployment. The deployment system does not feed outcomes back to the observability layer in a format agents can act on. The incident diagnosis does not automatically generate a pull request. Humans copy-paste context between tools, translate formats, decide when to proceed, and supply the connective tissue the toolchain lacks.

Every stage already has a shipping agent. Claude Code and GitHub Copilot coding agent write code. Harness AIDA and Argo CD deploy it. Dynatrace Davis and Datadog Watchdog observe production. Rootly and PagerDuty AIOps diagnose incidents. Resolve AI and Rundeck automate remediation. Each capability is real. Each lives in a separate product, with a human bridging every gap.

[AI Creates Software Faster Than Ops Can Handle](https://stxkxs.io/blog/second-order-explosion) asked what breaks when AI writes more code. This post asks what happens when AI also deploys, monitors, and fixes it, and which seams must exist before that cycle finishes closing. Harness, GitHub, and Dynatrace are already wiring adjacent stages. The missing product is identity, protocols, and audit at the handoffs. Another agent inside a stage does not close those gaps.

Five stages, four gaps

The human still lives in the gaps. That is also where platform infrastructure has to land. The stage map is short on purpose: the work is the handoff, not the agent inside each box.

Code stops at merge

Claude Code, GitHub Copilot coding agent, and Cursor implement features, write tests, and open PRs. GitHub reported in February 2023 that Copilot wrote an average of 46% of code in files where it was enabled; that figure describes completion, not the autonomous agent, and GitHub has not published an agent equivalent. Generation is no longer the bottleneck. The gap into Deploy is PR approval and merge: remove it and bugs ship at machine speed; keep it and the reviewer caps the cycle.

Deploy lacks change semantics

Harness AIDA scores deployment risk. Argo CD and Flux deploy whatever reaches main. Flagger automates canary analysis. GitOps already removes a human from the deploy step. The gap into Observe is deployment-aware context: monitors see a new container version, not "we changed retry logic in payments." Without that semantic payload, correlating anomalies to specific code changes stays a human investigation.

Observe lacks incident context

As covered in [200 OK, Wrong Answer](https://stxkxs.io/blog/ai-observability-200-ok), AI observability is consolidating into platform features after roughly six acquisitions or shutdowns over eighteen months. Dynatrace Davis, Datadog Watchdog, and New Relic AI detect anomalies and surface likely causes. For AI-generated code under AI-augmented deploys, the system under watch is itself non-deterministic, and golden signals lie. The gap into Diagnose is structured incident context: change intent, architecture, and history of similar failures. Telemetry alone does not carry those.

Diagnose stops at hypothesis

Rootly AI summarizes timelines. PagerDuty AIOps correlates alerts. [OpenClaw](https://stxkxs.io/blog/openclaw-self-hosted-ai-agents) can query infrastructure and investigate through natural language. These accelerate diagnosis and still hand confirmation to a human. The gap into Remediate is fix generation: turning a timeout-regression hypothesis into a code change, config patch, or rollback is still engineering judgment.

Remediate reopens trust

Auto-scaling, restarts, and circuit breakers have been production staples for a decade. Rundeck runs predefined sequences. Resolve AI investigates and opens remediation PRs grounded in a found root cause, not a fixed runbook. An auto-scaler adding replicas is bounded. An agent writing and shipping a fix on its own diagnosis is unbounded. Diagnose is the most dangerous stage to fully automate: a wrong hypothesis feeds every downstream action, and the system gets harder to reason about with each confident wrong step.

Vendors stop at adjacent

No single vendor owns the full cycle. **Harness** expands from delivery: AIDA correlates deploys with production health (Deploy through Diagnose on one platform); code generation and automated remediation still need integrations. **GitHub** owns the repository and the most contentious gate, PR review and merge; Observe through Remediate still depend on marketplace glue. **Dynatrace** starts at Observe: Davis into Diagnose, Workflows into remediation. That right-side chain is the most complete in market; generating a code fix and shipping it through CI/CD still needs external tools. Each vendor closes adjacent stages. The complete cycle is more likely through protocol-level integration (MCP and A2A) than through one product owning all five.

Closure creates new failures

Connecting stages enables machine-speed iteration. It also creates failure modes that do not exist when humans bridge the gaps.

Cascading autonomy failures

The dangerous pattern is a cascade where each stage makes a locally reasonable, slightly wrong decision and the errors compound. A slow database query triggers an alert; diagnosis misclassifies it as a network partition; remediation restarts services; restarts release load the query had been throttling; fresh alerts fire. Each agent performed correctly on local context. No single agent sees the full cycle, and the connections do not carry enough shared state to stop the spiral. Without a cycle-level circuit breaker, this pattern can run multiple iterations before a human notices.

Approval becomes the bottleneck

Human gates defend against cascades and reintroduce the speed limit closure was meant to remove. If every AI-generated fix needs review before deploy, the cycle runs at human speed. The workable resolution is approval tiers by blast radius, confidence, and reversibility: auto for reversible config on non-critical services, human at every stage for production schema migrations. Platform work is technical enforcement of the right tier, not a wiki page about it.

Blast radius amplifies

Humans bridging stages limit blast radius through judgment and speed. Agents at machine speed do not make those calls unless the platform encodes them. A human-bridged cycle might complete once per hour during an incident; an autonomous cycle can complete once per minute. If each pass makes things slightly worse, damage rate scales with iteration rate.

Automation needs its own telemetry

Current observability watches the application. Once stages connect, the platform also needs signals about the automation: cycles completed, whether they converge or diverge, false-positive rate, human intervention rate. [200 OK, Wrong Answer](https://stxkxs.io/blog/ai-observability-200-ok) covered observing AI systems where a successful response can contain wrong content. A closed cycle compounds that: agents watching AI-deployed code written by AI. Meta-observability becomes infrastructure.

Humans between stages are circuit breakers and context carriers. Removing them requires infrastructure that replaces those functions.

Protocols leave trust open

The gaps between stages are protocol gaps before they are model gaps. Each tool speaks its own API. Bridging today means custom integrations for every pair, usually held together by a human pasting context. Model Context Protocol (MCP), now a Linux Foundation standard, is the connectivity layer for agent-to-tool access. As covered when [Anthropic donated MCP to the Linux Foundation](https://stxkxs.io/blog/mcp-linux-foundation), an MCP server wraps a deployment platform, observability backend, or CI system and exposes it to any compatible agent. Agent-to-Agent (A2A), created by Google and now also a Linux Foundation project, complements MCP for stage handoffs: MCP connects an agent to a tool; A2A connects an agent to another agent.

Connectivity is not trust. MCP and A2A solve mechanical context loss. They do not decide blast radius, assign a durable identity to the actor, or prove that the agent did only what it claimed. Identity is the load-bearing seam most MCP configs ignore: if every stage agent shares one service account, the audit trail collapses to a single principal. Per-task or per-agent identity, scoped credentials, and short-lived tokens are prerequisites for governance. [The Agent Said It Worked](https://stxkxs.io/blog/agent-said-it-worked) covers the parallel failure mode: the agent transcript is self-attestation. Independent evidence still has to come from API audit trails (CloudTrail, Kubernetes audit, GitHub events), reconciled against claims.

{
  "mcpServers": {
    "github":    { "command": "mcp-server-github",    "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" }, "description": "Stage 1: Code" },
    "argocd":    { "command": "mcp-server-argocd",    "env": { "ARGOCD_SERVER": "${ARGOCD_SERVER}" }, "description": "Stage 2: Deploy" },
    "datadog":   { "command": "mcp-server-datadog",   "env": { "DD_API_KEY": "${DD_API_KEY}" }, "description": "Stage 3: Observe" },
    "pagerduty": { "command": "mcp-server-pagerduty", "env": { "PD_API_KEY": "${PD_API_KEY}" }, "description": "Stage 4: Diagnose" },
    "resolve":   { "command": "mcp-server-resolve",   "env": { "RESOLVE_TOKEN": "${RESOLVE_TOKEN}" }, "description": "Stage 5: Remediate" }
  }
}

TIP: Invest in MCP servers now
Even without closing the cycle today, MCP servers for internal tools create the integration surface later automation will use. Authorization is the hard part. Each server connection needs its own RBAC policy. Build that governance layer alongside the servers, not after the first autonomous merge.

Seams need platform work

The closed cycle will emerge from connections between existing tools, mediated by MCP and A2A, and governed by platform infrastructure. Four systems are the minimum viable governance layer.

Tiered approval gates

A tiered system classifies changes by blast radius, reversibility, and confidence, then applies the matching gate. Enforcement has to be technical. Agents at machine speed will not respect a checklist.

• Tier 1: Full Auto: No human — Low blast radius, fully reversible, high confidence. Config on non-critical services, scaling, feature flags.
• Tier 2: Notify: Inform human — Moderate blast radius, reversible, high confidence. Code fixes with tests, canaries, standard runbooks.
• Tier 3: Approve: Human approves — High blast radius or low reversibility. Migrations, API contract changes, cross-service modifications.
• Tier 4: Human Execute: Human does it — Critical blast radius, irreversible, or novel. Production data mods, security-sensitive or first-time remediations.

Agent audit trails

Every agent decision in the cycle needs full context logged: data observed, options considered, choice made, and rationale. When an autonomous cycle makes things worse, the post-mortem has to reconstruct the decision graph, not only the action sequence. The trail still is not independent evidence of infrastructure mutations. Those proofs live in CloudTrail, Kubernetes audit, and GitHub events, reconciled against claims as argued in [The Agent Said It Worked](https://stxkxs.io/blog/agent-said-it-worked). [nanohype/fab](https://github.com/nanohype/fab)'s merge gate applies evidence-bound approval to a narrower autonomous action: four gate roles (pr-reviewer, qa-security, build-verifier, artifact-auditor) vote APPROVE, REJECT, or REQUEST_CHANGES, and a verdict with no cited evidence is automatically downgraded to REJECT. The gate does not trust the agent's conclusion; it requires the transcript that produced it.

Blast radius controls

Progressive rollout puts AI-generated fixes on a small traffic slice first, with automated rollback on metric degradation. Scope limits restrict which services an agent may modify. Rate limits cap cycles per hour so cascades are time-bounded. The critical control is the automatic halt on divergence: if metrics degrade despite remediation, stop and escalate. That convergence check must be a platform primitive, not something each agent reimplements with its own optimistic definition of success.

Watch the automation

Standard observability watches the application. Loop-aware observability watches the automation: completion rate without human intervention, convergence rate on the target metric, human intervention rate, mean time to escalation, and agent accuracy against root causes established in post-mortems. Without those numbers, tier promotions are vibes.

• Start at Tier 4 (human execute); baseline agent accuracy before reducing oversight.
• Ship agent audit trails and independent claim reconciliation before any tier promotion.
• Add progressive rollout, scope limits, rate limits, and convergence-based circuit breakers.
• Instrument completion, convergence, intervention, escalation, and diagnosis accuracy against post-mortems.
• Promote only on measured accuracy and override rates; reserve Tier 1 for bounded reversible actions with automatic rollback.

Sometimes leave it open

Closing stages pays off when human handoff speed is the measured constraint on delivery. For a large fraction of teams it is not, and the integration work buys a larger surface to govern without moving the real bottleneck. Four situations make full closure the wrong call; none is a maturity problem that more adoption time fixes.

Low deploy frequency wastes it

DORA State of DevOps research measures four delivery metrics; deployment frequency is the one stage-closure targets. If a team ships weekly or monthly, time spent bridging tools is a rounding error against release windows and product decisions. Machine-speed iteration solves a bottleneck that does not exist. Tiered gates, audit trails, and loop-aware observability are real engineering cost paid against a constraint that was never handoff speed.

Regulation keeps the human

Under change-control regimes, a human approving a production change is the correct permanent state, not a stepping stone to autonomy. Medical devices, payment processing, aviation, and clinical systems require an accountable person who signed off. Agent audit trails make the approver faster and better informed. They do not replace the approver. Treating Tier 4 as a stage to graduate out of misreads why the gate exists.

Small teams pay more

A team of a few engineers already has the full cycle in one head. The person who wrote the code watches the deploy, reads the dashboard, diagnoses the incident, and ships the fix, with zero translation loss because there is no handoff to translate. Building MCP servers, a tiered approval engine, and meta-observability is months of platform work better spent on the product. Below that threshold, the human bridge is cheaper than the infrastructure that replaces it.

Diagnose stays human longest

Even teams that close Code→Deploy and automate bounded remediations should keep Diagnose on a short leash. A wrong hypothesis turns every subsequent automated step into damage. Prefer agents that draft timelines, rank causes, and propose fixes while a human confirms root cause before remediation runs, especially for novel incidents and hard-to-reverse changes. Full diagnostic autonomy is the last promotion, not the first.

WARNING: Closure is not the default
The right end state is usually a partially automated cycle with a permanent human gate where regulation or blast radius demands it. Closing end to end is one valid target among several, justified only when human handoff speed is the measured constraint. Build governance because handoffs are slowing real delivery, not because full autonomy is assumed to be the destination.

Build breakers first

The closed cycle is arriving on a schedule of integration, not invention. Harness, GitHub, and Dynatrace each build toward it from a different stage. MCP covers agent-to-tool access; A2A covers agent-to-agent handoffs. Adjacent stages are already connected in production products. The open variables are how fast the remaining seams get protocolized, and whether identity and audit land before the automation does.

[AI Creates Software Faster Than Ops Can Handle](https://stxkxs.io/blog/second-order-explosion) argued that platform engineers need operational infrastructure before the flood of AI-generated code arrives. The mandate extends: governance has to exist before autonomous handoffs connect. Tiered approval gates, agent audit trails reconciled against independent logs, blast radius controls, loop-aware observability. Those decide whether a cycle amplifies capability or amplifies failure. Organizations that build governance now close stages on their terms. Those that wait close reactively, after an autonomous cycle makes the case for guardrails more persuasively than any architecture document. Build the circuit breakers first.

The loop will close. Circuit breakers decide whether that is progress. Platform engineers build those breakers.

Resources & Further Reading

• AI Creates Software Faster Than Ops Can Handle: https://stxkxs.io/blog/second-order-explosion - The prequel: what breaks when AI writes more code
• 200 OK, Wrong Answer: https://stxkxs.io/blog/ai-observability-200-ok - AI observability and golden-signal failure for non-deterministic systems
• The Agent Said It Worked: https://stxkxs.io/blog/agent-said-it-worked - Why agent transcripts are not independent evidence of what ran
• Self-Hosted AI Agents for Incident Response: https://stxkxs.io/blog/openclaw-self-hosted-ai-agents - OpenClaw as a Stage 4 (Diagnose) tool
• MCP Is Now a Linux Foundation Standard: https://stxkxs.io/blog/mcp-linux-foundation - Protocol context for MCP and A2A as stage connectivity
• nanohype/fab: https://github.com/nanohype/fab - Evidence-bound merge gate with multi-role APPROVE/REJECT votes
• Harness AIDA: https://www.harness.io/products/aida - AI assistant for CI/CD analysis and deployment risk
• GitHub Copilot Coding Agent: https://github.blog/news-insights/product-news/github-copilot-meet-the-new-coding-agent/ - Autonomous coding agent from GitHub Issues
• Dynatrace Davis AI: https://www.dynatrace.com/platform/artificial-intelligence/ - Causal AI for observability, diagnosis, and remediation
• Resolve AI: https://www.resolve.ai/ - Autonomous investigation that opens remediation PRs from found root cause
• DORA State of DevOps Report: https://dora.dev/research/ - Deployment frequency, lead time, MTTR, change failure rate
• Google A2A Protocol: https://github.com/a2aproject/A2A - Agent-to-Agent communication protocol specification

This arrives before anyone budgets for it. The stack you spent years building (golden signals, distributed traces, SLO burn rate alerts) assumes a contract: same input, same output. AI systems have no such contract. Every response is generated. Two identical prompts can produce different outputs. Correctness is a semantic judgment about what the output means, and the HTTP status code never carries that judgment.

The fix starts with instrumenting different signals, and it starts this week.

Your observability stack is broken

Wire AI workloads into existing monitoring and everything looks fine until it is not. The dashboards answer questions that no longer decide whether users got a usable answer. Four assumptions break under AI traffic.

Metrics measure the wrong things

Latency, error rate, throughput, and saturation stay necessary and stop being sufficient. A 200ms response can cost $0.002 or $0.20 depending on model, context size, and whether cached tokens were used. A 0% error rate is quiet while the dangerous failures return 200 OK. Throughput in requests per second ignores that one request may burn 100,000 tokens and another 500. Token cost per request, time to first token, semantic correctness, cache hit rate, and reasoning-token overhead are the signals that matter, and traditional stacks never modeled them. Datadog and Grafana are bolting them onto request-response architectures. The data model is wrong.

Traces assume DAGs

Distributed tracing assumes a request enters, hops through services, and exits. Spans nest into a waterfall. Agents break that shape immediately: call a tool, judge the result insufficient, change approach, call another tool, loop until a quality threshold. That is a cycle with conditional branches, dynamic tool selection, and variable-depth recursion. In Jaeger you get a wall of spans where why the agent looped, what it retried, and how many iterations it took is buried in attributes or gone.

Cost and latency decouple

In traditional systems, slow roughly tracks expensive. In AI systems that correlation collapses. A fast response from a large model with a long context can cost 100x a slow response from a small model: cached tokens are cheaper than fresh tokens, reasoning tokens add cost without lengthening the user-visible output, and input and output tokens price differently. Trimming a redundant system prompt or warming a cold cache can move cost per request with no latency change. Without cost per request as a first-class metric, the fastest-growing infrastructure line item is invisible.

Failure is semantic

A 500 means the server errored. A timeout means a deadline was exceeded. Those signals are machine-readable and page automatically. AI's most dangerous failures return 200 OK with well-formed, confidently wrong answers. Catching them means comparing output against ground truth, checking factual consistency, and looking for contradictions with source material. That is evaluation work, and the stack was never built for it.

Key Point: Golden signals report whether the system is up. For AI workloads, users care whether the system is right.

Instrument the right signals

Semantic evaluation, agent tracing, cost tracking, and quality scores all at once stalls before anything ships. Layer signals by difficulty. Each layer produces data that makes the next one cheaper to build.

Start with token economics

Token usage is four differently priced quantities: input, output, cached input, and reasoning. A prompt-cache hit can cost about 90% less than the same request cold. A model using [extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) can consume 10x the tokens of a direct response, billed at the model's standard output-token rate. OpenTelemetry GenAI semantic conventions standardize the attributes (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`) so the same instrumentation works across backends. Week one usually surfaces oversized context windows, redundant system prompts, and cold caches.

Add semantic evaluation

Semantic quality stacks three techniques that complement each other. Heuristics (regex, keyword detection, length checks) are fast and cheap enough for every response; use them to catch obvious garbage. Embedding similarity against reference answers catches semantic drift and struggles with novel correct answers. LLM-as-judge is the most accurate and too expensive and slow for every request. Run heuristics on the hot path, embeddings on critical paths with references, and reserve LLM-as-judge for offline batches and CI quality gates.

Purpose-built evaluation models close the gap between heuristic speed and judge accuracy. [Galileo's Luna-2](https://docs.galileo.ai/concepts/luna/luna) runs sub-200ms at roughly $0.02 per million tokens. Against GPT-4o at $2.50 per million tokens as judge, that is roughly 30-125x cheaper. At 10 million completions per day and about 1,000 tokens each, Luna-2 is roughly $200/day; a frontier model as judge is roughly $25,000/day. Real-time monitoring is finally crossing the price where sampling 0.1% is no longer the only survivable option.

Instrument agent traces

Only if you actually run agent workflows. Standard span trees fail on recursive agent cycles: planning, tool calls, intermediate evaluations, retries, and synthesis each need spans grouped into one execution that preserves decision-making, not a pure request DAG. Trace context through async loops is the hard part; parent-child relationships break when sub-agents spawn tools that spawn more agents. LangSmith uses nested runs, Braintrust scores experiment traces at each step, and Arize Phoenix follows OpenInference. The implementations differ. The requirement is the same: capture cycles.

Instrument against OTel

Instrument against OTel GenAI semantic conventions now, whatever backend you ship to. The conventions are still experimental and have changed multiple times since mid-2025, so wrap them in a thin abstraction that absorbs renames without touching every callsite. The alternative is proprietary SDKs from vendors that keep getting acquired. Over roughly eighteen months the vendor landscape consolidated hard: CoreWeave acquired Weights & Biases for $1.7 billion; WhyLabs ceased commercial operations and was [acqui-hired by Apple](https://www.geekwire.com/2025/founders-at-seattle-startup-whylabs-join-apple-following-under-the-radar-acquisition/) (founders joined Apple; whylogs and langkit were open-sourced); Anthropic acqui-hired Humanloop in August 2025; ClickHouse acquired Langfuse in January 2026; Mintlify acquired Helicone in March 2026; Cisco [announced intent to acquire Galileo](https://blogs.cisco.com/news/cisco-announces-the-intent-to-acquire-galileo) (AI observability and evaluation) in April 2026 and closed in May 2026. That Galileo is unrelated to the generative UI tool Alphabet folded into Google Stitch. AI observability is becoming a feature of larger platforms. Proprietary instrumentation means re-instrumenting after each deal. OTel means swapping the exporter.

import { trace, SpanKind, SpanStatusCode, metrics } from '@opentelemetry/api'

const tracer = trace.getTracer('ai-service', '1.0.0')
const meter = metrics.getMeter('ai-service', '1.0.0')

// Duration, tokens, and TTFT are histograms, not span attributes
const operationDuration = meter.createHistogram('gen_ai.client.operation.duration', {
  unit: 's',
  description: 'Duration of GenAI client operations',
})
const tokenUsage = meter.createHistogram('gen_ai.client.token.usage', {
  unit: '{token}',
  description: 'Number of tokens used per GenAI operation',
})
// Client TTFT; server-side instrument is gen_ai.server.time_to_first_token
const timeToFirstToken = meter.createHistogram(
  'gen_ai.client.operation.time_to_first_chunk',
  {
    unit: 's',
    description: 'Client-observed time to first chunk for streaming GenAI responses',
  }
)

interface ChatCompletionResult {
  content: string
  model: string
  inputTokens: number
  outputTokens: number
  cachedTokens: number
  finishReason: string
  ttftMs: number
}

export async function tracedChatCompletion(
  prompt: string,
  model: string,
  onComplete: (prompt: string, model: string) => Promise<ChatCompletionResult>
): Promise<ChatCompletionResult> {
  return tracer.startActiveSpan(
    'chat',
    { kind: SpanKind.CLIENT },
    async (span) => {
      try {
        span.setAttribute('gen_ai.provider.name', 'anthropic')
        span.setAttribute('gen_ai.request.model', model)
        span.setAttribute('gen_ai.operation.name', 'chat')
        span.setAttribute('gen_ai.request.max_tokens', 4096)
        span.setAttribute('gen_ai.request.temperature', 0.7)

        const startTime = performance.now()
        const result = await onComplete(prompt, model)
        const durationMs = performance.now() - startTime

        span.setAttribute('gen_ai.usage.input_tokens', result.inputTokens)
        span.setAttribute('gen_ai.usage.output_tokens', result.outputTokens)
        span.setAttribute(
          'gen_ai.usage.cache_read.input_tokens',
          result.cachedTokens
        )
        span.setAttribute('gen_ai.response.model', result.model)
        span.setAttribute('gen_ai.response.finish_reasons', [result.finishReason])

        const metricAttrs = {
          'gen_ai.response.model': result.model,
          'gen_ai.operation.name': 'chat',
        }
        operationDuration.record(durationMs / 1000, metricAttrs)
        tokenUsage.record(result.inputTokens + result.outputTokens, metricAttrs)
        timeToFirstToken.record(result.ttftMs / 1000, metricAttrs)

        span.setStatus({ code: SpanStatusCode.OK })
        return result
      } catch (error) {
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: error instanceof Error ? error.message : 'Unknown error',
        })
        throw error
      } finally {
        span.end()
      }
    }
  )
}

Usage counts ride as span attributes under `gen_ai.*`. Operation duration, token usage totals, and time to first token record as histograms per the GenAI metrics convention, not as span attributes. For streaming responses, time to first token and total duration are different numbers; optimizing one does not fix the other. Provider extensions cover Anthropic, OpenAI, AWS Bedrock, and Azure OpenAI so features like extended thinking and function calling get stable names.

Adoption is already past the tipping point. Datadog, Langfuse, Splunk, and Grafana consume OTel GenAI conventions natively. OpenLLMetry (7.3K GitHub stars) auto-instruments major LLM SDKs. Opik (19.4K stars, 40 million traces per day) supports OTel export. Gateway volume is on the same curve: Portkey reported 500B+ tokens per day at its Series A in February 2026 and [1 trillion tokens per day by March 2026](https://portkey.ai/blog/1-trillion-tokens-and-the-death-of-the-chatbot). The conventions are still under revision; adoption at this scale is already ahead of a formal stable release.

Open Source AI Observability by GitHub Stars

• Langfuse: 28140
• Opik: 19396
• Phoenix: 9885
• OpenLLMetry: 7300

Use the platform you have

If Datadog or Grafana already runs the rest of the estate, evaluate their AI monitoring before buying a second stack. Datadog LLM Observability puts token usage and quality evaluation next to existing APM waterfalls. Grafana AI Observability is OTel-native, so GenAI spans land in the same alerting and on-call paths. Splunk has agent-level span visibility. Dynatrace ties GPU utilization through to semantic quality. Seeing AI telemetry where on-call already looks is worth more than a prettier standalone UI. Every extra vendor adds a pipeline, a dashboard surface, an alert config, a bill, and a pager context switch.

Dedicated tools still win at two jobs: deep agent tracing (LangSmith, Braintrust, Phoenix all had to extend or replace standard tracing to make agent workflows legible) and real-time semantic evaluation (Fiddler AI ships sub-100ms guardrails and raised $30 million in January 2026 to scale them). If either is a hard requirement, evaluate them. Otherwise start inside the platform already paid for. Eighty percent coverage in the tools the pager lives in beats one hundred percent in a tool nobody opens at 2am.

Most Recent Funding Round by Company

• LangChain: 125$M
• Braintrust: 80$M
• Arize AI: 70$M
• Fiddler AI: 30$M
• Portkey: 15$M

WARNING: Check release notes first
The failure mode: adopt a dedicated AI observability tool, spend weeks integrating it, then discover the existing platform shipped the same feature natively. Before adding a vendor, read six months of your current platform's release notes. This space moves fast enough that the gap may already be closed.

The playbook this month

Token-cost tracking alone is enough for low request volumes, human-reviewed outputs, or non-customer-facing prototypes. Semantic evaluation and agent tracing earn their cost once volume or blast radius outgrows spot-checks. Skip phases that do not apply.

• Phase 1: Token + Cost: Week 1-2 — Instrument token usage with OTel GenAI attributes. Calculate cost per request from provider pricing. Alert on cost anomalies.
• Phase 2: Semantic Quality: Week 3-4 — Heuristic checks on the hot path. Embedding similarity on critical paths. LLM-as-judge for CI/CD gates only.
• Phase 3: Agent Tracing: Month 2 — Only if you run agent workflows. Evaluate LangSmith, Braintrust, or Phoenix for cycle-aware tracing.
• Phase 4: Optimize: Ongoing — Right-size models, tune prompt caching, shorten context windows. Token data is where the double-digit cost cuts live.

Use OTel GenAI conventions for the instrumentation layer either way. The adoption cost is a handful of attribute names and histogram instruments. The alternative is re-instrumenting after an acquisition, a price hike, or a proprietary SDK break. Vendor-neutral instrumentation is the hedge while backends consolidate under you.

Monitoring becomes evaluation

Observability was rebuilt for non-deterministic systems once already. By 2020, tabular ML monitoring (concept drift, data drift, output distributions) was production practice: Evidently open-sourced drift detection in November 2020, and Fiddler, founded in 2018, was monitoring tabular and NLP models. Image monitoring followed. LLMs changed the output modality. Judging whether free-form text is correct is harder than watching a classification score drift, so the evaluation layer is where the new work lives. Six acquisitions and shutdowns in eighteen months put AI observability on the same path as every other specialized telemetry category: platform feature, stable interface, competing backends. Instrument against that interface now.

Resources & Further Reading

• OTel GenAI Semantic Conventions: https://github.com/open-telemetry/semantic-conventions-genai - Vendor-neutral schema for AI telemetry instrumentation
• Braintrust: https://www.braintrust.dev/ - AI product evaluation, tracing, and prompt management platform
• Arize AI / Phoenix: https://github.com/Arize-ai/phoenix - Open-source AI observability with OTel-compatible tracing
• LangSmith: https://smith.langchain.com/ - LangChain's tracing and evaluation platform for LLM applications
• Langfuse: https://langfuse.com/ - Open-source LLM engineering platform (acquired by ClickHouse, Jan 2026)
• Portkey: https://portkey.ai/ - AI gateway with token-level observability and cost tracking
• Fiddler AI: https://www.fiddler.ai/ - Real-time AI model monitoring with sub-100ms guardrails
• OpenLLMetry: https://github.com/traceloop/openllmetry - OTel-native auto-instrumentation for LLM applications
• Opik: https://github.com/comet-ml/opik - Open-source LLM evaluation and tracing (40M traces/day)
• Datadog LLM Observability: https://docs.datadoghq.com/llm_observability/ - End-to-end LLM tracing integrated with Datadog APM
• Grafana AI Observability: https://grafana.com/products/cloud/ai-observability/ - AI monitoring built on the Grafana + OTel stack

Roughly $425M in disclosed venture capital went into purpose-built vector databases: Pinecone at $138M, Zilliz at $113M, Qdrant at $87.8M, Weaviate at $67.6M, and Chroma at $18M. That capital validated the use cases and pushed the technology forward. The capability they pioneered is now a native feature of every major database teams already run. The window where purpose-built was the only option has closed.

A team standing up RAG already has PostgreSQL holding documents, users, permissions, and audit logs. They need vector search. Someone opens a Pinecone tab. Before long the team is designing a sync pipeline, adding a service to on-call, and evaluating five vendors, to store a column of floats next to the data that generated them.

Vectors are a data type, not a database category. Reaching for a dedicated vector store usually buys complexity the workload never needed. The same arc played out with JSON, geospatial, and full-text search. [Platform teams wiring AI infrastructure](https://stxkxs.io/blog/platform-engineering-ai-era) face the same stack question: extend what already runs the data plane before adding a specialized product.

• VC Funding: $425M — Disclosed VC across Pinecone ($138M), Zilliz ($113M), Qdrant ($87.8M), Weaviate ($67.6M), and Chroma ($18M)
• pgvector Stars: 22K+ — GitHub stars for pgvector, the PostgreSQL extension for vector similarity search
• DBs with Vectors: 10+ — Major general-purpose databases that have added native vector support since 2021

How many RAG pipelines actually run under 10 million vectors is a number no public survey has measured. This post's working estimate is that most do, drawn from production deployments this author has seen and from free and starter tiers most vector vendors size in the low millions of vectors before pricing steps up.

INFO: What is a vector embedding?
A vector embedding is a numerical representation of data (text, images, audio, code) as a list of floating-point numbers, typically 384 to 3072 dimensions. Models place semantically similar items close together in that space. "How do I reset my password?" and "I forgot my login credentials" share no keywords yet produce nearby vectors. Nearest-neighbor search enables semantic search, recommendations, RAG, and anomaly detection.

History keeps repeating

A specialized data type shows up. Startups build purpose-built databases around it. Enterprises adopt. The databases everyone already runs absorb the capability. Purpose-built consolidates to the high end.

MongoDB launched in 2009 on the thesis that relational databases were a poor fit for documents. PostgreSQL shipped jsonb in 9.4 (2014) with GIN indexes and containment operators; MySQL, SQL Server, and Oracle followed. MongoDB still serves document-heavy workloads where its sharding and tooling win. The claim that JSON alone requires a separate database became much harder to defend.

Geospatial repeated the arc: PostGIS, then spatial types in MySQL, SQL Server, and Oracle. Full-text search did too: Sphinx and Elasticsearch, then tsvector, FULLTEXT, and built-in search in general-purpose stores for many workloads.

The best database for vectors is the one that already has your data.

Vectors follow the same market pattern with a wider technical gap. ANN search needs specialized indexes (HNSW, DiskANN, ScaNN) and a recall tradeoff general-purpose planners were not built around. That gap is closing: pgvector HNSW is production-grade for most RAG scale. It has not fully closed at the extreme end.

pgvector is the default

pgvector is the quiet production default. Amazon RDS, Supabase, Neon, Azure Database for PostgreSQL, Google Cloud SQL, and AlloyDB all support it. One CREATE EXTENSION adds vector search to the store that already holds documents and permissions. No new service, connection pool, or monitoring plane.

Before 0.5, pgvector was IVFFlat-only and limited. 0.5 (August 2023) added HNSW; 0.8 is production-grade. The memory arithmetic is the part that gets underestimated. Ten million vectors at 1536 dimensions in float32 are about 61GB raw, but HNSW stores a second full copy of every vector, so the index alone lands near 77GB, and the heap plus TOAST adds another 60-odd on top. A fully cached working set is closer to 140GB than to the 61GB the raw math suggests. CREATE INDEX blocks writes; CREATE INDEX CONCURRENTLY avoids the lock at roughly double the build time and the risk of an invalid index on failure. Plan for rebuilds when the embedding model changes.

HNSW delivers 95%+ recall with single-digit millisecond latency at millions of vectors. Quantization is where the memory goes back. `halfvec` (fp16, added in 0.7) halves the index to roughly 39GB at near-identical recall, and is the safe default. Binary quantization cuts it to about 5GB, but the full vectors stay on disk for reranking and recall turns model-dependent: [Jonathan Katz measured](https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/) 91.6% recall on dbpedia-1536 with reranking, and 0% on gist-960. Test it against the actual embeddings before banking the saving. Under 10M vectors, the range this post estimates covers most production RAG, that performance is enough.

The operational costs are the ones a benchmark never shows. Vacuum comes first: pgvector's own README warns that vacuuming an HNSW index "can take a while" and recommends REINDEX CONCURRENTLY ahead of VACUUM, and the path is still being repaired, with 0.8.3 fixing index corruption during HNSW vacuum and 0.8.4 fixing an unrepaired-graph error. Write amplification comes second: unlogged tables build HNSW indexes [23 to 31 times faster](https://supabase.com/blog/pgvector-fast-builds), and that gap is the WAL cost of vector writes, paid again at replication and point-in-time recovery. Build time comes third: a 40M-vector index at 768 dimensions [stalled at 29.2% after 19 hours](https://github.com/pgvector/pgvector/issues/822) once the graph outgrew maintenance_work_mem. Connection limits get blamed on pgvector and belong to PostgreSQL's process-per-connection model, which is a different argument.

-- Enable pgvector
CREATE EXTENSION IF NOT EXISTS vector;

-- Documents with embeddings alongside relational data
CREATE TABLE documents (
  id BIGSERIAL PRIMARY KEY,
  title TEXT NOT NULL,
  content TEXT NOT NULL,
  embedding VECTOR(1536),  -- OpenAI text-embedding-3-small
  team_id INT REFERENCES teams(id),
  created_at TIMESTAMPTZ DEFAULT NOW(),
  is_active BOOLEAN DEFAULT TRUE
);

-- HNSW index for fast approximate nearest neighbor search
CREATE INDEX ON documents
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

-- Vectors + relational filters in one query
SELECT d.id, d.title, d.content,
       1 - (d.embedding <=> $1::vector) AS similarity
FROM documents d
JOIN teams t ON d.team_id = t.id
WHERE d.is_active = TRUE
  AND t.org_id = $2
  AND d.created_at > NOW() - INTERVAL '90 days'
ORDER BY d.embedding <=> $1::vector
LIMIT 10;

That query is the architectural argument in SQL. Similarity JOINs relational tables, filters by flags and dates, and enforces tenant isolation in one statement. A purpose-built store means search in one system, hydrate from another, and keep both consistent. Colocation removes that class of bugs and reuses monitoring, backups, pooling, and failover already in place.

MongoDB Atlas Vector Search, Redis vector similarity, Elasticsearch dense_vector, Cassandra vector types, sqlite-vec, SingleStore, CockroachDB, and Snowflake followed the same absorption. If the primary store already holds the documents, put the index next to them.

TIP: pgvectorscale for larger sets
Timescale's pgvectorscale adds StreamingDiskANN indexes for datasets larger than RAM, and statistical binary quantization for large memory cuts with limited recall loss. Between roughly 10M and 100M vectors, it extends PostgreSQL without a second database category.

The sync tax

The monthly bill is the smaller cost of a purpose-built vector database. The real cost is the synchronization pipeline maintained forever.

Every document lives in the primary database. A copy of its embedding lives in the vector store. Updates regenerate embeddings. Deletes remove vectors. Permission changes update metadata. That becomes queues, workers, retries, dead letters, and drift monitoring. Engineering investment runs in the hundreds of hours per year.

• Pinecone (10M vectors): ~$70/mo — Pinecone serverless, 1536 dimensions, ~60-70GB storage (~$0.33/GB), $50/mo Standard minimum, moderate query volume
• pgvector on RDS: ~$0/mo — Marginal cost when added to an existing RDS PostgreSQL instance with available capacity

Build and operate a reliable sync path (queues, retries, backfills on model change, on-call for drift) and the total cost premium versus pgvector on infrastructure already paid for lands roughly 3-10x by this author's estimate of the work involved (not a published TCO study). Pinecone serverless remains a legitimate zero-ops path for teams with no database to extend and a need to ship this week. For teams that already operate Postgres or MongoDB, the comparison is trajectory and total cost of ownership.

WARNING: The sync problem
Search returns a runbook rewritten months ago because the old embedding still points at old content. A deleted document keeps its vector. Partial permission updates leak results users should no longer see. These failures exist because vectors live apart from the data they represent. Colocating vectors with source data removes the failure class.

When purpose-built wins

Colocation has limits. Some workloads need a purpose-built vector database.

Raw scale is the first case. At 5M vectors, pgvector is strong. At 50M, HNSW parameters and partitioning start to matter. At 500M, PostgreSQL is the wrong chassis. Milvus's disaggregated architecture scales storage, indexing, and query nodes independently for bulk load versus steady-state query. Billion-vector systems need that lever.

Filtered search at scale is the second. Production queries look like "top 10 similar, this tenant, after this date, these tags." Qdrant co-designs payload indexing with HNSW. At moderate scale pgvector is fine; filtering to 0.1% of a 100M collection at sub-10ms is where purpose-built filtering wins. Unfiltered ANN benchmarks miss that pattern.

Multi-modal and multi-vector search is the third: separate embeddings for title, body, and image with different weights. Qdrant named vectors and Weaviate modules are first-class. Multiple pgvector columns plus application score fusion rebuilds that product on PostgreSQL.

Greenfield ML services with no relational store to extend are the fourth. With nothing to colocate against, a managed vector product is often the fastest path to production.

• Billion+ vectors with ANN algorithms (DiskANN, ScaNN) general-purpose stores have not fully matched
• Filtered search at 100M+ with selective payload filters and sub-10ms latency
• Multi-modal search across text, image, and audio with per-vector-type weighting
• Multi-tenant SaaS that needs native per-tenant index isolation
• Greenfield or standalone ML services with no existing relational data to colocate

Containers were "just a process type" Linux could run natively; running them well at scale still justified dedicated orchestration. Purpose-built vector stores play a similar role at the high end and are unnecessary for most mid-scale RAG. Tutorial defaults push dedicated products long before workloads demand them.

Key Point: Most production workloads do not need a separate database for vectors at all.

Vector count decides

Vector count decides this. The thresholds are lower than vendor pitch decks imply.

Under 10M: use Postgres

Use what already runs. pgvector, Atlas Vector Search, or Redis vector similarity. HNSW on pgvector delivers single-digit millisecond latency at this scale. A 2M-vector index at 1536 dimensions and m=16 needs on the order of 16GB RAM, fits a db.r6g.xlarge (32 GiB), and builds in single-digit minutes. Operational simplicity outweighs any purpose-built edge for the vast majority of RAG, semantic search, and recommendation systems.

10M-100M: tune first

Optimize before replacing. pgvectorscale StreamingDiskANN for larger-than-RAM sets. Raise m and ef_construction for recall at memory cost. Quantize (binary, scalar, product) for 50-75% memory cuts. Partition by tenant or time. Evaluate purpose-built only after those levers are exhausted.

100M+: purpose-built earns it

Benchmark purpose-built against an optimized existing store. Milvus disaggregation, Qdrant on-disk quantization, or Weaviate distributed deployments may win on latency, throughput, or cost. Verify on real data and query patterns. Synthetic ANN leaderboards rarely match production filters and multi-tenancy.

Team shape matters as much as vector count. Full control and near-zero marginal cost favor pgvector for teams that already operate PostgreSQL. Zero-ops and no ANN tuning favor managed Pinecone for teams without that expertise, at the price of control, cost at scale, and a sync path. Metadata filters on managed stores are also less expressive than SQL WHERE clauses; debugging moves from EXPLAIN plans to vendor status pages.

• Under 10M: Use existing DB — pgvector, Atlas Vector Search, or Redis; single-digit ms latency, zero additional infrastructure
• 10M-100M: Optimize first — pgvectorscale, HNSW tuning, quantization, partitioning; extend before replacing
• 100M+: Benchmark both — Purpose-built may win on latency/throughput; verify against actual workload, not synthetic benchmarks
• Any scale: Avoid sync tax — Every separate vector DB adds sync pipelines, consistency bugs, and operational overhead

Graph favors colocation

Retrieval is moving toward vector similarity plus graph traversal. Microsoft GraphRAG (arXiv 2404.16130) reported graph-enhanced retrieval winning 72-83% of head-to-head comparisons on comprehensiveness and 62-82% on diversity for global sensemaking questions over million-token corpora, with naive vector search still stronger on directness.

That direction favors colocation. Graph-enhanced RAG wants similarity, traversal, and relational filters in one place. Vectors in Pinecone, a graph in Neo4j, and rows in PostgreSQL means three round-trips and application-level merges. Vectors via pgvector, relationships via recursive CTEs or Apache AGE, and relational data natively can run in a single statement.

-- Hybrid retrieval: vector similarity + relational links
WITH semantic_matches AS (
  SELECT id, title, content, embedding,
         1 - (embedding <=> $1::vector) AS similarity
  FROM documents
  WHERE team_id = $2
    AND is_active = TRUE
  ORDER BY embedding <=> $1::vector
  LIMIT 20
),
related_docs AS (
  SELECT DISTINCT d.id, d.title, d.content, d.embedding,
         0.5 AS similarity
  FROM semantic_matches sm
  JOIN document_links dl ON sm.id = dl.source_id
  JOIN documents d ON dl.target_id = d.id
  WHERE d.is_active = TRUE
    AND d.id NOT IN (SELECT id FROM semantic_matches)
)
SELECT id, title, content, MAX(similarity) AS score
FROM (
  SELECT * FROM semantic_matches
  UNION ALL
  SELECT * FROM related_docs
) combined
GROUP BY id, title, content
ORDER BY score DESC
LIMIT 10;

EXAMPLE: Microsoft GraphRAG
GraphRAG builds a knowledge graph from documents via LLM-extracted entities and relationships, then combines community summaries with vector similarity. Vector search finds semantic neighbors; graph traversal finds structural connections. Complex questions often need both (arXiv 2404.16130).

Vectors as a data type

Purpose-built vector databases will not disappear. Milvus will serve billion-scale deployments. Qdrant will serve latency-critical filtered search. Weaviate will serve multi-tenant hybrid retrieval. Those niches are real. For most production workloads (RAG under this post's ~10M-vector estimate, semantic search inside SaaS products, recommendations that need vectors next to relational data), vectors belong in the database already on-call.

Vectors are following the same path as JSON, geospatial, and full-text search. The hype cycle created a database category. Maturity is dissolving it back into a data type.

Key Point: The $425M bet on purpose-built vector databases was early, not wrong. Those products proved the market and pushed the incumbents to ship. The clearest read on where it lands: Pinecone, the best-funded pure play, [engaged bankers to explore a sale](https://www.calcalistech.com/ctechnews/article/rz31q82b5) in August 2025. The right vector database is usually the one already running.

Resources & Further Reading

• pgvector: https://github.com/pgvector/pgvector - Open-source vector similarity search for PostgreSQL
• pgvectorscale: https://github.com/timescale/pgvectorscale - StreamingDiskANN for datasets larger than RAM
• Pinecone: https://www.pinecone.io/ - Fully managed vector database, serverless architecture
• Weaviate: https://weaviate.io/ - Open-source AI-native vector database with hybrid search
• Qdrant: https://qdrant.tech/ - High-performance vector search engine written in Rust
• Milvus: https://milvus.io/ - Open-source vector database for billion-scale deployments
• Chroma: https://www.trychroma.com/ - Embeddable open-source vector database for AI applications
• MongoDB Atlas Vector Search: https://www.mongodb.com/products/platform/atlas-vector-search - Vector search in the aggregation pipeline
• Microsoft GraphRAG: https://github.com/microsoft/graphrag - Graph-enhanced retrieval augmented generation
• ANN Benchmarks: https://ann-benchmarks.com/ - Benchmarking approximate nearest neighbor algorithms
• Platform Engineering in the AI Era: https://stxkxs.io/blog/platform-engineering-ai-era - Stack choices for inference, vectors, and gateways
• VectorDBBench: https://github.com/zilliztech/VectorDBBench - Open-source vector database benchmark tool

An attacker obtains a compromised npm publish token for a popular open-source AI coding assistant, the class of credential that let the ua-parser-js (2021) and event-stream (2018) attacks succeed against ordinary utility libraries. The malicious release adds one line to package.json: a postinstall script that silently installs a second, attacker-controlled package on every machine that updates. Because the script runs automatically and the injected package is scoped globally, it reaches CI runners and developer laptops before anyone reviews the diff. This scenario is illustrative. It describes a known attack pattern, hijacking an AI coding tool's dependency chain to distribute a second payload, and does not correspond to a specific dated incident.

Two trends make a scenario like this increasingly plausible. AI-assisted development has pushed open source consumption and the volume of AI-generated code shipped with known security flaws sharply upward, the same dynamic covered in [AI Creates Software Faster Than Ops Can Handle](https://stxkxs.io/blog/second-order-explosion). Sonatype's 2026 State of the Software Supply Chain report counted 454,648 new malicious packages last year, a 75% jump over the prior year. A compromise of this shape sits at the convergence of those two trends.

AI inflates the attack surface

Black Duck's 2026 OSSRA report analyzed 947 codebases and found mean vulnerabilities per codebase jumped 107% in a single year. Component counts up 30%, file counts up 74%. The report correlates this growth with increased AI-assisted development; correlation during a period of rapid AI adoption is not conclusive causation. Codebases are accumulating components and files faster than security review can keep up, and AI-generated code is a significant contributor to that velocity.

• Malicious Packages: 1.23M+ — Cumulative OSS malware packages detected (Sonatype 2026)
• Vuln Growth: +107% — Mean vulnerabilities per codebase YoY (Black Duck OSSRA)
• OSS Downloads: 9.8T — Annual downloads across top 4 registries, up 67% YoY (Sonatype 2026)
• AI Code Insecure: 62% — AI-generated code with design flaws or known vulns (study cited by CSA, 2025)

AI-generated code security by the numbers

• AI code with flaws: 62%
• Orgs without full security evaluation: 76%
• Orgs doing full evaluation: 24%
• Codebases with license conflicts: 68%

A study cited by the Cloud Security Alliance found 62% of AI-generated code solutions contain design flaws or known security vulnerabilities; that figure is not from Black Duck OSSRA. Models train on public repositories that mix secure and insecure implementations without distinguishing them, and they have no visibility into an application's threat model or internal standards. The result is missing controls, logic flaws, and inconsistent patterns that static analysis struggles to catch. Only 24% of organizations perform comprehensive IP, license, security, and quality evaluations for AI-generated code (Black Duck OSSRA 2026); the remaining 76% spot-check for security only or perform no evaluation. Open source licensing conflicts hit an all-time high at 68% of audited codebases, a 12-point jump in a single year, as AI tools pull in dependencies without license awareness.

AI coding assistants also suggest packages they were trained on, pulling in transitive trees that developers never manually evaluated. A single `npm install` can add hundreds of packages. When the model suggests the install, the developer's usual heuristic (have I heard of this package, does it look maintained) is bypassed in favor of implicit trust in the completion.

Attacks are now industrialized

Supply chain attacks have moved from opportunistic typosquatting to multi-stage operations. Sonatype's 2026 report documents the shift: threats have moved from spam and stunts to sustained, industrialized campaigns, many state-sponsored. Over 99% of detected open source malware targets npm.

Typosquatting (publishing packages with names similar to popular ones, like `lodahs` instead of `lodash`) still accounts for a large share of malicious detections. Dependency confusion exploits the gap between public and private registries: if an organization uses `@company/auth-utils` privately and the public registry has no package by that name, an attacker publishes a higher version publicly and build tools that check public registries first pull it. Both vectors are amplified by AI coding tools that may surface the typosquatted or confused name in completions.

Token compromise scales the same pattern. The postinstall scenario turns on a stolen npm publish token. ua-parser-js (2021) and event-stream (2018) used similar vectors at smaller scale. Sonatype reports exposed development secrets grew 11% across major repositories last year. When an attacker obtains a publish token for a popular package, the blast radius is every downstream consumer that runs `npm install` or `npm update` before the compromise is detected.

Shai-Hulud, discovered in September 2025, was the first known self-replicating npm malware. Using stolen tokens, the worm enumerated the compromised maintainer's packages, injected a malicious postinstall (bundle.js), and republished new versions. ReversingLabs identified patient zero as rxnt-authentication@0.0.3 (published September 14, 2025); the leading suspected vector was credential-harvesting phishing spoofing npm MFA-update notices, not a @testing-library typosquat. Shai-Hulud did not need to compromise a popular package directly. It turned every infected developer into a distribution node through packages they published normally.

State-linked actors have industrialized the same pipeline. Group-IB's 2026 High-Tech Crime Trends report documents Lazarus Group campaigns that begin with a malicious npm package or compromised GitHub Action, harvest credentials from environment variables and CI secrets, then persist via modified build scripts that survive dependency updates. Palo Alto Unit 42's Contagious Interview campaign targeted developers through fake job interviews that installed backdoored packages. Group-IB identifies supply chain attacks as the top global cyber threat, with state-linked actors scaling the approach.

The stack is ready

Three complementary frameworks have matured at once. SBOMs inventory what is inside an artifact. SLSA attests how and where it was built. Sigstore proves the artifact has not been tampered with since signing. Each covers a distinct layer. Deploying one without the others leaves gaps: an SBOM without provenance can be fabricated, provenance without signatures can be forged, and signatures without an inventory say nothing about what was signed.

SBOMs need ownership

A Software Bill of Materials is a machine-readable inventory of every component in an artifact: direct and transitive dependencies, versions, licenses, and known vulnerabilities. SPDX (Linux Foundation) and CycloneDX (OWASP) dominate. CISA's updated guidance requires machine-readable formats, and leading ecosystems are integrating SBOM generation into build tools. Generation is largely solved. Syft, Trivy, and cdxgen produce SBOMs from images, filesystems, and manifests. Quality is the hard problem: full transitive coverage, current vulnerability mappings, build-time dependencies that never ship but can compromise the build, and for AI-generated code, which suggested packages a human evaluated versus accepted without review.

Running cdxgen against a mature production codebase (three years of accumulated dependencies, not a demo) typically surfaces hundreds of packages maintained by anonymous handles, some with no commits in two years. The inventory rarely matches the team's prior sense that dependencies were under control.

SLSA proves the build

Supply-chain Levels for Software Artifacts (SLSA) is a framework for artifact integrity. Version 1.2 from the Linux Foundation (late 2025) defines Build Track levels L0 through L3. L0 is absence of SLSA. L1 requires a consistent build process with provenance describing how the artifact was built. L2 requires a hosted build platform on dedicated infrastructure with digitally signed provenance. L3 requires a hardened platform that isolates builds and keeps signing secrets inaccessible to user-defined steps, producing non-falsifiable provenance. There is no Level 4 in the current spec; two-person review and hermetic reproducible builds belonged to the obsolete SLSA v0.1 model.

In the postinstall scenario, the compromised package had no build provenance; it was published from a local machine with a stolen token. A SLSA Level 2+ requirement would reject that pattern: the build must originate from CI, and the provenance record must show the source commit, builder identity, and pipeline. Mismatch with the expected pipeline is a reject.

Sigstore verifies identity

Sigstore is free, open-source code signing without long-lived key management. Cosign signs and verifies images and blobs. Fulcio issues short-lived certificates tied to OIDC identity. Rekor is an immutable transparency log of signing events. Keyless signing ties the event to an identity provider (GitHub, Google, Microsoft) with certificates that expire in minutes, and records the event in Rekor.

# Sign a container image (keyless - uses OIDC identity)
cosign sign ghcr.io/myorg/myapp:v1.2.3

# Verify the signature matches expected identity
cosign verify ghcr.io/myorg/myapp:v1.2.3 \
  --certificate-identity="https://github.com/myorg/myapp/.github/workflows/release.yml@refs/tags/v1.2.3" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com"

# Verify SLSA provenance attestation
cosign verify-attestation ghcr.io/myorg/myapp:v1.2.3 \
  --type slsaprovenance \
  --certificate-identity-regexp="^https://github.com/slsa-framework/slsa-github-generator/" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com"

Lock the pipeline first

Frameworks matter only when they land in existing CI. Order by impact-to-effort for teams that have not yet invested.

Lock everything down

Commit lockfiles (`package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, `Cargo.lock`, `go.sum`). Use `npm ci` in CI, not `npm install`: it installs exactly what the lockfile specifies and fails on mismatch. Pin GitHub Actions to full commit SHAs instead of tags (`uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29` instead of `@v4`). Tags are mutable; an attacker who compromises the action can move the tag to a malicious commit. Verify the SHA against the tagged release on the action repository before pinning it. The SHAs below illustrate the pattern only.

name: CI
on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read  # Principle of least privilege
    steps:
      # Pin actions to full SHA, not tags (verify the SHA against the tag's release page before use)
      - uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
        with:
          node-version: '22'
          cache: 'npm'

      # Use npm ci, not npm install
      - run: npm ci --ignore-scripts  # Skip postinstall scripts
      - run: npm audit --audit-level=high
      - run: npm run build
      - run: npm test

Audit dependencies continuously

Enable Dependabot or Renovate for automated updates with security alerts. Fail CI on high-severity findings from `npm audit` or `yarn audit`. Use OpenSSF Scorecard on critical dependencies before adoption; it scans over 1 million projects weekly for branch protection, code review, CI configuration, signed releases, and vulnerability disclosure. For high-value projects, Socket.dev or Snyk can flag malicious packages before production.

The first week after wiring `npm audit --audit-level=high` into CI on a mature project is brutal: years of transitive high-severity findings surface at once, many buried six levels deep in something nobody touches. The temptation is to allowlist everything. The useful move is ruthless triage. Most of the noise sits in dev dependencies that never reach production; the findings that do matter need a fix, not a suppression. Once the backlog clears, the gate is the strongest early signal for new risk.

Sign and verify artifacts

Sign container images with Cosign in CI. Configure Kyverno or OPA Gatekeeper to reject unsigned images in Kubernetes. For npm packages, `npm publish --provenance` in GitHub Actions generates a SLSA Level 2 provenance attestation and publishes it alongside the package. Consumers verify on npmjs.com or via `npm audit signatures`. For package maintainers, that single flag is the highest-impact step available: verifiable build attestation without a custom signing stack.

name: Publish
on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write  # Required for npm provenance
    steps:
      - uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29
      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
        with:
          node-version: '22'
          registry-url: 'https://registry.npmjs.org'
      - run: npm ci
      - run: npm test
      # Publish with SLSA provenance attestation
      - run: npm publish --provenance --access public
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Generate queryable SBOMs

Add SBOM generation to the release pipeline with Syft (images) or cdxgen (source). Store SBOMs alongside artifacts: GitHub releases, OCI registries via Cosign, or a dedicated repository. Scan with Grype or Trivy against vulnerability databases. The goal is a queryable inventory that answers "are we running anything affected by CVE-2026-XXXXX?" within minutes of a disclosure, not a compliance file nobody reads.

EXAMPLE: Minimum viable supply chain security
If nothing else lands this quarter: (1) commit lockfiles and use `npm ci` in CI, (2) pin GitHub Actions to SHAs, (3) enable Dependabot or Renovate, (4) run `npm audit` in CI. These four actions are free, take less than an hour, and close the most common vectors. Sigstore, SLSA, and SBOMs build on that floor.

Review is the binding constraint

AI-generated code introduces vulnerabilities at a rate and volume traditional security review was not designed to handle. Manual review catches issues at roughly the rate a human can read code. As a growing share of new code becomes AI-generated, the review bottleneck becomes the binding constraint on security. The parallel failure mode for agents that act on infrastructure is self-attestation: the transcript is not independent evidence of what ran (covered in [The Agent Said It Worked](https://stxkxs.io/blog/agent-said-it-worked)). Dependencies suggested by a model have the same shape. Trust requires a check the model does not control.

AI-aware static analysis can flag generation-specific mistakes: hallucinated imports, insecure defaults, missing input validation, packages that do not exist or have been deprecated. IDE integrations catch insecure patterns at generation time, before commit. LLM-based review can surface logic flaws and missing controls that rule-based scanners miss. Enforce checks at generation (IDE), commit (pre-commit hooks), CI (build pipeline), and deploy (admission control). Earlier is cheaper. The IDE check is easiest to bypass and admission control is hardest, so each stage backstops the ones before it.

When this is overkill

A solo internal tool that builds from a private repo, ships to one cluster, and pulls a dozen well-known dependencies does not need SLSA Level 3, admission control, and a signed SBOM per release. The threat model that justifies this stack is a public package consumed by people you will never meet, or a build pipeline holding credentials worth stealing. For a script three people run, the full apparatus costs more attention than the risk it removes. Match the controls to the blast radius.

SBOMs decay without ownership

Generating an SBOM is a one-flag command. Keeping it accurate is a standing job. Every dependency bump, base-image change, and transitive shift makes yesterday's SBOM wrong, and a wrong inventory is worse than none because it answers the CVE-disclosure question confidently and incorrectly. If no one owns regeneration and validation, the SBOM is a compliance artifact that quietly drifts out of sync with what actually runs.

Admission control adds friction

Configuring Kyverno or OPA Gatekeeper to reject unsigned images stops a real attack and also stops a 2 AM hotfix built outside the normal pipeline. Break-glass paths get added, then used routinely, then become the default, and the control is theater. A signing gate everyone has learned to route around provides the audit log of a security control without the security.

AI scanners cry wolf

AI-aware static analysis is young. Immature scanners flag hallucinated-import patterns and missing-validation heuristics on code that is fine. A high false-positive rate trains developers to dismiss output, which is exactly the failure mode that lets a real finding through. Until a tool earns trust on a given codebase, treat its alerts as candidates for review, not merge blockers, and measure precision before wiring it into CI.

WARNING: The compliance-theater objection
Most of this stack becomes paperwork without disciplined triage. An SBOM nobody queries, a signature nobody verifies on consumption, and an audit gate that allowlists every finding to stay green all produce the documentation of security with none of the substance. Black Duck found only 24% of organizations do comprehensive evaluation of AI-generated code; the other 76% are not all skipping the work because they lack tools. Some bought the tools and never built the triage muscle. Controls without follow-through are worse than honest gaps, because they manufacture the false confidence that the work is handled.

The line between security and theater is whether a human acts on the output. An SBOM matters when a disclosure triggers a query and a patch. A signature matters when verification runs at deploy and fails closed. An audit gate matters when high-severity findings get fixed instead of allowlisted. Adopt a control only when the discipline to act on its output already exists.

Adoption is the gap

SLSA 1.2, Sigstore, SBOMs, OpenSSF Scorecard, and npm provenance are mature enough to run in production pipelines today. Provenance attestation, artifact signing, and systematic dependency evaluation remain uncommon. The default pipeline still ships without any of them. CISA's updated SBOM guidance and the EU Cyber Resilience Act create compliance pressure independent of engineering conviction. US federal policy has moved the other way: Executive Order 14306 (June 2025) scaled back the attestation mandates of EO 14144, and OMB memorandum M-26-05 (January 2026) rescinded the government-wide software attestation requirement in favor of an optional, agency-led risk-based approach. Organizations that implement generation, provenance, and signing now will be ahead of requirements that return under different names.

When code generation is cheap, code volume explodes, and every additional dependency is a potential entry point. AI developer tools are themselves supply chain targets, and the developers who use them typically hold broad repository access, CI credentials, and deployment permissions. The tools to write code are attack surfaces. The binding work is adoption and review of AI-suggested dependencies, not inventing another framework.

The frameworks are ready. The gap is pipelines that still ship without them, and AI-suggested dependencies that still land without review.

Resources & Further Reading

• Sonatype 2026 State of the Software Supply Chain: https://www.sonatype.com/state-of-the-software-supply-chain/introduction - OSS malware counts, consumption trends, and supply chain risk data
• Black Duck 2026 OSSRA Report: https://www.blackduck.com/resources/analyst-reports/open-source-security-risk-analysis.html - Open source risk analysis across 947 codebases
• SLSA Framework: https://slsa.dev - Supply-chain Levels for Software Artifacts specification
• Sigstore Documentation: https://docs.sigstore.dev - Keyless signing, Cosign, Fulcio, and Rekor
• OpenSSF Scorecard: https://scorecard.dev - Automated security health metrics for open source projects
• CISA SBOM Resources: https://www.cisa.gov/sbom - Federal guidance on SBOM generation, formats, and consumption
• CycloneDX Specification: https://cyclonedx.org - OWASP SBOM format and tooling
• npm Provenance: https://docs.npmjs.com/generating-provenance-statements - Publish packages with SLSA provenance attestation
• StepSecurity Harden-Runner: https://github.com/step-security/harden-runner - GitHub Actions agent for detecting compromised dependencies
• ReversingLabs 2026 SSCS Report: https://www.reversinglabs.com/sscs-report - Supply chain security guidance timeline and threat analysis
• AI Creates Software Faster Than Ops Can Handle: https://stxkxs.io/blog/second-order-explosion - Velocity of AI-generated code outrunning operational capacity
• The Agent Said It Worked: https://stxkxs.io/blog/agent-said-it-worked - Why self-attestation is not independent evidence of what ran

If you started a new infrastructure project today, what would you reach for? Five years ago the answer was obviously Terraform. Two years ago it was probably still Terraform, with a footnote about the license change. Today the answer depends on who you ask: platform teams land on different tools for defensible reasons, and none of those reasons are wrong. The IaC landscape has fragmented, and the old default answer is gone.

Pulumi is the better tool for new infrastructure today. The 62% already on Terraform have good reason to stay. Those are not contradictory positions. They reflect the reality that "which IaC tool" is now four separate questions masquerading as one, and the answer depends on language preferences, cloud strategy, and honestly, how much organizational inertia is worth fighting.

What follows is the read on where each tool actually sits as of early 2026, from someone who has shipped infrastructure with most of these tools and has opinions about all of them.

Four paradigms grow at once

The market is real and growing: roughly $1.3 billion in 2025, projected to reach about $9.4 billion by 2034 at a 24% CAGR (Precedence Research). The size matters less than the shape: four distinct paradigms are growing simultaneously without cannibalizing each other. HCL declarative (Terraform, OpenTofu), general-purpose languages (Pulumi, CDK), Kubernetes-native (Crossplane), and infrastructure-from-code (Nitric, Encore). Each serves a different engineering culture. Each has legitimate strengths. The question is which culture is yours.

GitHub stars, Feb 2026 (Wayback)

• Ansible: 67999
• Terraform: 47600
• OpenTofu: 27871
• SST: 25386
• Pulumi: 24692
• AWS CDK: 12700
• Crossplane: 11397
• Encore: 11412
• Bicep: 3500

Stars are a vanity metric, but the shape of this chart is still informative. OpenTofu hitting 27.9K in about two and a half years is remarkable velocity for a fork. SST at 25.4K is notable, and misleading, because the project is in maintenance mode. Pulumi at 24.7K understates its real footprint given 100M+ SDK downloads. CDK at 12.7K understates its usage even more: 3.5 million weekly npm downloads is quietly enormous. No single metric survives contact with the others.

• Market Size: $1.3B — Global IaC market size in 2025 (Precedence Research)
• Terraform Adoption: 62% — Current organizational adoption rate (Firefly 2025)
• OpenTofu Downloads: 10M+ — Total downloads across all releases
• Pulumi Downloads: 100M+ — Total SDK downloads across package managers

The number that matters

Forget market size. The most important number in the IaC landscape is 15. That is the gap between Terraform's current organizational adoption (62%, per Firefly's State of IaC 2025) and the percentage of respondents who commit to it as their future primary tool (47%). Nearly one in four current Terraform users is actively looking at alternatives. 62% is dominant by any measure, but the grip is loosening.

Current adoption vs future commitment

• Terraform: 62%
• OpenTofu: 12%

OpenTofu: future commitment (27%) is more than double current usage (12%). That is the shape of a tool in its adoption upswing. Firefly's survey does not break out Pulumi, Crossplane, or CDK the same way, so any percentage for those tools beyond the ones cited elsewhere in this post is unsourced and left out here.

Key Point: The IaC market is fragmenting by use case: HCL declarative, general-purpose languages, Kubernetes-native, and infrastructure-from-code. All four segments are growing simultaneously. There is no single right answer anymore. For teams already invested in Terraform, the switching cost is often the deciding factor. Migrating 50,000 lines of HCL to Pulumi is a multi-quarter project, regardless of which tool is "better." Factor migration cost into every comparison.

The fork that stuck

Most open-source forks die quietly. OpenTofu did not. When HashiCorp switched Terraform to BSL 1.1 in August 2023, the fork landed at the Linux Foundation within weeks. IBM acquired HashiCorp for $6.4 billion, closing in early 2025. The fork now has real feature divergence, which is the only thing that makes a fork matter long-term.

Terraform: the incumbent

Terraform is still the default: 62% of infrastructure teams are on it. The Terraform Registry now lists ~6,600 total providers (34 official, 391 partner, ~6,175 community); HashiCorp's last official integration milestone was "3,000+" in March 2023. It has the deepest CI/CD integration story, a decade of accumulated blog posts and Stack Overflow answers and consulting expertise. When you Google an infrastructure problem, you get a Terraform answer. That ecosystem gravity is the actual reason Terraform keeps its 62%: the switching cost is not worth the fight, and that is a different thing from having evaluated the alternatives.

The risks are equally straightforward. IBM ownership means strategic uncertainty. The BSL license makes procurement teams at open-source-first shops nervous. Feature velocity has slowed relative to OpenTofu. State encryption, early variable evaluation, provider for_each: OpenTofu shipped all of these first. For a project with Terraform's market position, being outpaced on features by its own fork is not a great look.

OpenTofu: earning it

OpenTofu entered the CNCF Sandbox in April 2025. It has 160+ contributors, 10 million downloads, and its registry handles roughly 10 million requests per day. More importantly, it has shipped features Terraform does not have: native state encryption with AES-GCM and KMS provider support, early variable evaluation in backend and module sources, provider for_each, and OCI-compliant module sources.

terraform {
  encryption {
    method "aes_gcm" "default" {
      keys = key_provider.aws_kms.main
    }

    key_provider "aws_kms" "main" {
      kms_key_id = "arn:aws:kms:us-west-2:123456789:key/my-key-id"
      key_spec   = "AES_256"
      region     = "us-west-2"
    }

    state {
      method   = method.aes_gcm.default
      enforced = true
    }

    plan {
      method   = method.aes_gcm.default
      enforced = true
    }
  }
}

Native state encryption is the feature to point to if someone asks "why does OpenTofu matter?" Terraform state files contain secrets in plaintext. Everyone knows this. Everyone has worked around it with backend encryption, remote state, access controls. OpenTofu just encrypts the state. The fact that this took a fork to happen tells you something about where Terraform's priorities were.

TIP: Migration is boring (in a good way)
State files are compatible. Provider binaries are compatible. The migration is mechanical: swap `terraform` for `tofu` in your CI, optionally rename blocks, point registry references to `registry.opentofu.org`. The hard part is organizational: updating runbooks, retraining muscle memory, and getting buy-in from teams that have "Terraform" in their job titles.

OpenTofu migration status (2025)

• No plans: 65%
• Evaluating: 24%
• Planning: 6%
• Completed: 5%

Why Pulumi is better

On a new infrastructure project, Pulumi is the better choice over Terraform. Writing infrastructure in the same language you write your application in eliminates an entire class of problems. HCL served its purpose, but the gap is real. You get real conditionals, not HCL's count-based hacks. You get actual loops, not for_each with maps. You get the same IDE, the same type system, the same test framework, the same package manager. 100 million SDK downloads and $99 million in funding say this is not a niche opinion.

When you wire this up for the first time (defining an S3 bucket as a TypeScript class with full autocompletion, then writing a unit test for your infrastructure the same way you would test application logic), the HCL workaround era feels immediately anachronistic. Pulumi has also been working toward HCL support, to let teams import existing Terraform modules without a rewrite. That is a smart bridge strategy: meet people where they are, then show them why the other side is better.

I understand exactly why people stick with Terraform, and it is not irrational. HCL's constraints are a feature for teams that need them. When every infrastructure file looks the same (same syntax, same structure, same limited set of operations), cross-team readability and auditing get dramatically easier. Pulumi's flexibility means two teams might write completely different patterns for the same infrastructure. If your organization does not already have strong code review culture and shared libraries for application code, Pulumi's flexibility will amplify that problem, not solve it.

WARNING: The flexibility tax
The provider ecosystem is still smaller than Terraform's ~6,600 providers, and community knowledge (blog posts, tutorials, Stack Overflow answers) is an order of magnitude thinner. You will hit more "I am the first person to try this" moments. Budget for the linting, code review, and shared libraries that keep that flexibility from turning into drift.

The debugging story is also worth being honest about. When an infrastructure deployment fails in Terraform, you are reading HCL and provider logs. In Pulumi, you are debugging through layers of language runtime, Pulumi engine, and cloud API. The failure modes are more complex. For experienced engineers, that trade-off is worth it. For teams where infrastructure is a side responsibility, it might not be.

CDK and the single-cloud bet

stxkxs.io itself deploys from TypeScript CDK: the site, its APIs, and the Lambda functions behind them all come from the same stack. That is a deliberate lock-in choice for an AWS-committed setup, trading portability for productivity. CDK gives AWS-only shops the highest-fidelity access to AWS services of any IaC tool, because AWS itself maintains it.

A single ApplicationLoadBalancedFargateService construct replaces dozens of raw CloudFormation resources. The type system catches misconfigurations at compile time. When AWS releases a new service, CDK support typically lands faster than any third-party provider.

CDK has 12.7K stars but 3.5 million weekly npm downloads, which makes it one of the largest footprints in the IaC space. CDK does not have the community buzz of tools that are fighting for mindshare. AWS shops just use it. It is the path of least resistance when you have already made the cloud commitment.

Azure Bicep occupies the same niche for Microsoft's cloud. 3.5K stars and a clever design choice: Bicep has no state files because ARM itself is the state store, eliminating an entire category of state management problems. For Azure-committed shops, it is the right answer for the same reasons CDK is the right answer for AWS shops.

INFO: The lock-in calculus
The standard objection is lock-in, and it deserves a straight answer. 89% of enterprises report multi-cloud strategies (Flexera 2024; up from 87% the prior year), and the average organization uses 2.4 cloud providers (Flexera 2025). "Multi-cloud strategy" often means "we have some workloads on AWS and some on Azure," not "we need to move workloads between clouds." For a genuinely single-cloud estate, and the Flexera average of 2.4 providers hides plenty of them, CDK or Bicep is a rational, even optimal choice. If you might need to move clouds, you need Terraform, OpenTofu, or Pulumi. Be honest about which camp you are in.

CloudFormation itself is worth a brief note. AWS's own [CDK best-practices guidance](https://docs.aws.amazon.com/cdk/v2/guide/best-practices.html) treats CloudFormation as the compilation target rather than the primary authoring surface for new work. Teams still writing raw CloudFormation YAML are mostly maintaining legacy stacks, not starting new projects. For those teams, CDK is the obvious migration path: same deployment model, same underlying engine, dramatically better developer experience.

Crossplane: right tool, narrow lane

Crossplane graduated from the CNCF in November 2025, one of 37 graduated projects out of 230+ CNCF-hosted projects. With 11.4K stars and 3,000+ contributors, it represents a fundamentally different model: infrastructure as Kubernetes Custom Resource Definitions, managed by controllers that continuously reconcile desired state with actual state. You do not run apply. The cluster runs it for you, constantly.

If you are building an Internal Developer Platform on Kubernetes, Crossplane is the provisioning engine you want. Cloud resources get managed with kubectl apply, the same workflow as pods and deployments. GitOps tools like Argo CD and Flux work natively. Drift detection is automatic because the reconciliation loop never stops. Platform teams can build self-service abstractions that hide cloud complexity from application developers. It is elegant for this specific use case.

If you are not already running Kubernetes as your platform substrate, Crossplane is the wrong answer. It requires a running cluster, which is meaningful overhead. Provider maturity is uneven: AWS and GCP are strong, niche providers lag far behind Terraform's ecosystem. Debugging requires understanding Kubernetes controller semantics, CRD schemas, and provider-specific resource models. For teams that are not deeply invested in the Kubernetes ecosystem, Crossplane adds complexity without proportional benefit. Evaluate it as a platform engineering tool on top of K8s, never as a general-purpose IaC replacement for non-K8s shops.

EXAMPLE: Who should look at Crossplane
Platform teams already running Kubernetes as an Internal Developer Platform. If you are building a self-service infrastructure layer on top of K8s (where application teams request resources through a portal or GitOps workflow), Crossplane fits naturally as the provisioning engine. If you are not running K8s, do not adopt K8s in order to use Crossplane. That is the tail wagging the dog.

Great DX, dead business

The fourth paradigm inverts the model entirely. Instead of declaring infrastructure and referencing it from application code, you write application code and the framework infers what infrastructure you need. Import a storage SDK, use it in your handler, and the framework provisions the bucket at deploy time. No separate IaC files, no state management, no resource graph. The developer experience is compelling. The track record is concerning.

SST (25.4K stars, the highest-profile infrastructure-from-code tool) entered maintenance mode in mid-2025 when the team pivoted to building OpenCode, an AI coding agent. Winglang, created by Elad Ben-Israel (the original creator of AWS CDK), raised $20 million in seed funding to build a purpose-built cloud programming language. Wing Cloud shut down in April 2025. Winglang remains open source under MIT, but nobody took over maintenance: barely a commit has landed since, and winglang.io no longer resolves. Ben-Israel told [Calcalist](https://www.calcalistech.com/ctechnews/article/bj90wnmrjl) that nobody picked up the baton. The signals are different (SST's team chose to leave for a bigger opportunity, Winglang could not sustain the business), but the conclusion is the same: standalone IfC frameworks have not found a sustainable business model.

Infrastructure-from-code is not a bad idea. The developer experience SST offered was real. I know people who shipped faster with it than with any other tool. The teams behind SST and Winglang were talented and well-funded. They still could not make it work as standalone products.

IfC survives as a layer

Nitric and Encore represent the more durable model: IfC as a layer on top of existing IaC engines, not a replacement for them. Nitric is an open-source, multi-cloud SDK. Write an API handler that reads from a bucket, and it generates Pulumi or Terraform to provision the right resource on AWS, GCP, or Azure. The multi-cloud capability is real, not theoretical. Encore takes a type-safe approach for TypeScript and Go, automatically provisioning cloud resources from typed infrastructure primitives.

import { api, bucket } from "@nitric/sdk";

const uploads = bucket("uploads").allow("read", "write");

const mainApi = api("main");

mainApi.get("/files/:name", async (ctx) => {
  const file = uploads.file(ctx.req.params.name);
  const url = await file.getDownloadUrl();
  ctx.res.json({ url });
});

The key difference from SST and Winglang is the fallback story. Nitric and Encore generate standard Pulumi or Terraform underneath. If the framework goes away, and that possibility is worth pricing into any IfC tool, the ejection path leads to the underlying IaC and the project keeps going. That ejection path is what makes the abstraction acceptable. Without it, the foundation is unmaintainable.

WARNING: The abstraction risk
Infrastructure-from-code trades control for velocity. While the abstraction holds, delivery is faster. When it breaks, and at scale it will, the generated IaC underneath becomes the thing that has to be understood. Two questions decide whether an IfC tool is safe to adopt: whether the project can eject to raw IaC if the framework is abandoned, and whether the generated infrastructure can be inspected and modified. SST's maintenance mode and Winglang's shutdown are not abstract cautionary tales. They happened in 2025.

The fragmentation persists

Terraform will remain the most-used IaC tool for years. 62% adoption does not evaporate, and the switching costs are real. That 15-point commitment gap tells you the direction. IBM's stewardship is the deciding factor: invest in the community and the position is defensible; prioritize commercial extraction and the erosion accelerates. Gradual erosion is the likely bet either way.

OpenTofu is the fastest-growing tool in the HCL lane, and the CNCF trajectory matters. If graduation follows (as it did for Crossplane), OpenTofu could become the default recommendation for new HCL projects within two years. The migration data supports this: 5% completed, 24% evaluating. The adoption wave is still ahead, not behind.

No public forecast breaks the $9.4 billion 2034 projection down by paradigm with real numbers, so this is qualitative, not a chart: HCL declarative keeps the largest current base by sheer install count, while general-purpose languages and Kubernetes-native tooling are growing off a smaller number, which is why their percentage gains look steeper. Four engineering cultures, four different growth rates, no convergence in sight.

Pulumi is the tool to recommend most often for new projects, with a caveat: you need the engineering maturity to use it well. Real language flexibility without real conventions produces real chaos. Pulumi's push toward HCL support is smart, if it lands: it would let teams migrate incrementally instead of rewriting everything, which is how most migrations actually succeed in practice.

Crossplane's growth is tied directly to Kubernetes adoption. As more organizations build internal platforms on K8s, Crossplane becomes the natural provisioning layer. It will never be a general-purpose IaC tool and it should not try to be. Its growth will track the platform engineering segment specifically.

Infrastructure-from-code is the youngest and most uncertain lane. The developer experience is real. The business model is unproven. If the paradigm survives (and it likely will, in some form), it will be as a layer that generates IaC rather than a framework that replaces it. Nitric and Encore are the bets worth watching. Do not build critical production infrastructure on any IfC tool without a clear ejection plan.

Pick the paradigm that matches how your team actually builds and operates.

Key Point: The right tool follows the team's actual constraints: governance requirements point to OpenTofu, vendor support needs point to Terraform, engineering maturity points to Pulumi, Kubernetes-native platforms point to Crossplane, single-cloud commitments point to CDK or Bicep, and a credible ejection plan is what makes Nitric or Encore defensible.

Resources & Further Reading

• Firefly State of IaC 2025: https://www.firefly.ai/state-of-iac-2025 - Adoption data covering Terraform, OpenTofu, and broader IaC trends
• Stack Overflow Developer Survey 2025: https://survey.stackoverflow.co/2025/ - Developer usage data across IaC and infrastructure tools
• OpenTofu Documentation: https://opentofu.org/docs/ - Official docs covering all OpenTofu-specific features
• OpenTofu GitHub: https://github.com/opentofu/opentofu - Source code, releases, and community discussions
• Pulumi Documentation: https://www.pulumi.com/docs/ - Getting started guides and provider references
• Crossplane Documentation: https://docs.crossplane.io/ - Architecture, providers, and composition guides
• AWS CDK Documentation: https://docs.aws.amazon.com/cdk/ - Construct library, API reference, and examples
• Azure Bicep Documentation: https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/ - Language reference and module registry
• Nitric Documentation: https://nitric.io/docs - Multi-cloud infrastructure-from-code SDK for TypeScript, Python, Go, and Dart
• Encore Documentation: https://encore.dev/docs - Type-safe backend framework with automatic infrastructure provisioning
• CNCF Landscape: Provisioning: https://landscape.cncf.io/guide#provisioning - Where IaC tools fit in the cloud-native ecosystem
• MarketsandMarkets IaC Report: https://www.marketsandmarkets.com/Market-Reports/infrastructure-as-code-market-180538080.html - Market sizing and growth projections through 2027

Inference accounts for roughly two-thirds of AI compute in 2026, up from a third in 2023 ([Deloitte 2026 TMT Predictions](https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html)). Training still owns the headlines: frontier model launches, multi-billion-dollar clusters, quarterly CapEx guidance. Once a model ships, every API call, RAG query, and agent step is inference. Training is a capital event. Inference is an operating expense that scales with usage and does not stop.

Model selection and prompt craft dominate discourse. The operational questions get less airtime: who schedules the GPUs, who owns the serving runtime, who pays when token volume doubles on a Thursday. That layer is platform work. Resource quotas, multi-tenancy, autoscaling, cost attribution, on-call. The unit price is higher and the failure modes leave less margin.

Inference is the bottleneck platform engineers own. Self-hosting the stack pays only when sustained volume, latency, or data residency force it. Short of that, a thin control plane over hosted APIs is the cheaper mandate.

• Inference share: ~2/3 — Share of AI compute spent on inference in 2026, up from ~1/3 in 2023 (Deloitte 2026 TMT Predictions)
• Inference market: $106B — AI inference market size on a path toward $255B by 2030 at 19.2% CAGR (MarketsandMarkets)
• AI CapEx 2026: $660-700B — Projected capital expenditure from the top five US cloud providers; roughly $450B tied to AI infrastructure (Goldman Sachs/CNBC)
• Enterprise GenAI: 80%+ — Enterprises that will have used GenAI APIs/models and/or deployed GenAI-enabled apps by 2026, up from under 5% in 2023 (Gartner, Oct 2023)

INFO: Scope
Model serving, GPU cost and scheduling, vector storage choices, and LLM gateways for teams that may self-host or hybridize inference. Not prompt engineering. Not a catalog of every MLOps product. The threshold for self-hosting is covered in Don't self-host yet.

Inference vs training compute share

• 2023: 33%
• 2025: 50%
• 2026: 67%

Serving composes two pieces

Production inference is two problems that get conflated. The engine turns a model artifact into tokens under latency and throughput constraints. The orchestrator turns that engine into a Kubernetes workload with rollouts, traffic split, and scale signals. [vLLM](https://github.com/vllm-project/vllm) owns most of the engine conversation for open LLM serving. [KServe](https://kserve.github.io/website/) owns the Kubernetes lifecycle for teams already living in InferenceService CRDs.

vLLM's adoption came from memory and batching work that made dense GPU packing practical. [PagedAttention](https://arxiv.org/abs/2309.06180) treats the KV cache like virtual memory pages instead of one contiguous allocation per request. The [vLLM launch post](https://vllm.ai/blog/2023-06-20-vllm) put the waste it removes at 60% to 80% of KV cache memory, lost to fragmentation and over-reservation; the paper states the same result from the other side, measuring only 20.4% to 38.2% of that cache actually holding token state. Continuous batching then fills free slots as requests finish instead of waiting for a static batch boundary. Launch benchmarks reported up to 24x throughput over naive Hugging Face Transformers serving. Governance under the PyTorch Foundation reduced the "single-company project" risk that blocks enterprise adoption.

KServe reached CNCF incubating status in September 2025. The InferenceService CRD separates predictor, transformer, and explainer so preprocessing can change without redeploying weights, and so the runtime behind the predictor can swap without rewriting the request path. Version 0.15 added first-class LLM paths and KEDA-oriented scaling on signals closer to token work than raw HTTP request count. A single generation request can consume orders of magnitude more GPU time than a classifier call; scaling only on QPS misreads the load. [Bloomberg runs KServe](https://www.bloomberg.com/company/stories/the-journey-to-build-bloombergs-ml-inference-platform-using-kserve-formerly-kfserving/) as the multi-tenant inference platform behind dozens of engineering teams, which is the shape most platform groups are actually buying: one serving layer, many owners.

The composition that shows up in production is vLLM as the runtime inside a KServe InferenceService. Engine performance and cluster lifecycle stop being separate science projects.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-prod
spec:
  predictor:
    model:
      modelFormat:
        name: vLLM
      storageUri: s3://models/llama-3-70b/
      resources:
        requests:
          cpu: "4"
          memory: 64Gi
          nvidia.com/gpu: "1"
        limits:
          nvidia.com/gpu: "1"
      # Scale on work that tracks tokens, not only RPS
      # (wire KEDA/PromQ against your runtime metrics)

NVIDIA Triton and TensorRT-LLM still matter for mixed-framework fleets and for shops already standardized on NVIDIA's optimization path. TensorRT-LLM's gains are real and NVIDIA-bound; vLLM's multi-vendor support is the hedge when AMD or Inferentia is on the table. For most platform teams standing up text generation first, vLLM plus KServe is enough surface area to operate well.

Key Point: Treat serving as engine plus orchestrator. Optimize the engine for tokens per dollar. Own the orchestrator for deploys, tenancy, and scale signals that match GPU work.

Idle GPUs own the bill

An NVIDIA H100 runs roughly $31,000 to $35,000 per card at retail. Renting is where the number gets interesting: an 8-GPU H100 node goes for about $24 to $32 an hour on the neoclouds (RunPod, Lambda) and $55 to $98 an hour on the hyperscalers, with AWS cutting p5.48xlarge roughly 45% to $55.04 in June 2025. That is a fourfold spread for identical silicon, which means the provider decision moves the bill as much as the scheduling one does. The problem is utilization of hardware priced like that, not merely "place the pod somewhere with a device plugin." Static team-level reservations produce the familiar failure mode: GPUs idle off-peak while training jobs queue, because quotas and preemption were never designed for devices this expensive and this indivisible.

The [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/) covers the boring-critical path: drivers, device plugin, node labeling, health. Dynamic Resource Allocation went GA in Kubernetes v1.34 in August 2025, and vendor DRA drivers are where finer claims over accelerators are heading; that path is the long-term replacement for "whole GPU or nothing" thinking. Multi-Instance GPU (MIG) gives hardware-isolated slices on supported cards. Time-slicing and fractional projects trade isolation for packing density. None of this is as mature as CPU sharing, and operating it is still specialist work.

Cost control has to be a control plane concern, not a quarterly spreadsheet. [nanohype's eks-agent-platform](https://github.com/nanohype/eks-agent-platform) wires that shape on AWS: a `BudgetPolicy` custom resource ties an hourly CUR/Athena cost check to an EventBridge kill-switch at 120% of budget, while `EvalSuite` gates promotion with Argo Workflows before a fleet change lands. The useful pattern is independent of that repo: budget and eval are platform resources, not wiki pages. Pooling and preemption beat buying another rack when utilization is the real gap; more H100s to paper over static allocation is how CapEx stories outrun product value.

WARNING: Utilization is organizational
NVIDIA puts the gain from fractioning and bin packing at roughly [2x GPU utilization](https://developer.nvidia.com/blog/maximizing-gpu-utilization-with-nvidia-runai-and-nvidia-nim/) (February 2026). The scheduler that does it is available: NVIDIA open-sourced the Run:ai scheduler as [KAI Scheduler](https://developer.nvidia.com/blog/nvidia-open-sources-runai-scheduler-to-foster-community-collaboration/) under Apache 2.0 in April 2025, now a CNCF sandbox project. Treat wider claims with suspicion; the often-quoted jump from 25% to 80% traces to a single customer in a 2020 vendor white paper, and the 80% appears in no source. The tooling is real and the trade it demands is organizational: teams give up dedicated cards for an SLO on a shared pool. Without that trade, the hardware stays peak-reserved and off-peak idle.

Vectors need less stack

RAG needs nearest-neighbor search over embeddings. That requirement does not automatically justify a new operational database category. Purpose-built vector stores raised large rounds and shipped real systems; general-purpose databases absorbed vector types and indexes on a familiar timeline. For a full absorption argument, funding totals, and when a specialized store still wins, see [Vectors Are a Data Type](https://stxkxs.io/blog/vectors-are-a-data-type).

The platform question is narrower. If the documents, permissions, and audit trail already live in PostgreSQL (or the warehouse that owns the business data), adding a sync pipeline and a second on-call surface to store float arrays is usually the wrong first move. Start with the store you already operate. Introduce a dedicated vector system when scale, hybrid search features, or isolation requirements outgrow that base, with eyes open about the operational tax.

Key Point: Vector search is a capability on the data path. Treating it as an automatic new platform product line is usually wrong.

Gateways replace scattered middleware

Every multi-provider integration eventually grows the same middleware: retries, fallbacks, key management, per-team spend caps, and request logging. LLM gateways productize that layer. Gartner's Market Guide for AI Gateways (October 2025) projected that by 2028, 70% of software engineering teams building multimodel applications will use AI gateways for reliability and cost control, up from 25% in 2025.

[LiteLLM](https://github.com/BerriAI/litellm) is the common open default: one OpenAI-shaped API over many providers, YAML-configured routes, budgets, and fallbacks that change without redeploying app code. Commercial options (Portkey and peers) compete on SLA, guardrails, and enterprise packaging. Observability-first proxies fill the gap when routing already exists and spend visibility does not. Envoy AI Gateway and similar mesh-adjacent projects matter when the traffic plane is already Envoy and token-aware routing belongs next to the rest of edge policy.

model_list:
  - model_name: chat-primary
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: chat-fallback
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  fallbacks:
    - chat-primary: [chat-fallback]

litellm_settings:
  max_budget: 5000
  budget_duration: 30d
  # Per-team virtual keys enforce attribution without a GPU fleet

For a team whose "AI platform" is still mostly hosted APIs, the gateway is the platform. Cost allocation, rate limits, and vendor failover land here long before vLLM is justified. Building retry and budget logic inside every service recreates the gateway as an unowned distributed system.

INFO: Gateway before serving
Stand up routing, budgets, and attribution first. Add self-hosted serving when the invoice and the latency budget both demand it. Reversing that order buys GPUs for traffic a proxy and a metered API would have covered.

Don't self-host yet

Self-hosting earns its keep at high, sustained inference volume where the GPU bill clearly exceeds the fully loaded cost of the engineers who run the fleet, or where latency, data residency, or model control requirements make a hosted API unacceptable. Below that threshold the math inverts. A small team that stands up vLLM, a GPU operator stack, a vector product, and an MLOps plane to serve traffic a hosted endpoint would handle on a meter is funding idle capacity and on-call for a control problem they do not have.

If the strategy is "call a frontier API," the platform mandate is keys, quotas, attribution, fallbacks, and vendor negotiation. One gateway covers that path. Managed end-to-end platforms (Bedrock, Vertex AI, SageMaker, or a GPU cloud that bundles serving) beat a six-layer open-source assembly when the team lacks Kubernetes operator depth, when load is spiky enough that reserved GPUs sit cold, or when paying a margin is cheaper than carrying the pager.

Pull layers in-house when the savings clear the operating cost. Serving and GPU scheduling are usually the first candidates because that is where the bill concentrates. Vector storage and experiment tracking are usually the last, because general-purpose data stores and managed tracking already exist. Start managed, measure token and dollar volume with honest attribution, and promote a layer only when the spreadsheet and the SRE rotation both agree.

WARNING: Assembly is a cost
Six self-hosted layers buy control, neutrality, and residency. They cost operators, incident load, and idle silicon. Interesting architecture is not a threshold. Sustained utilization and a constraint hosted APIs cannot meet are.

Key Point: The default is hosted inference behind a governed gateway. Self-host when forced by cost at scale, hard latency, or residency. Until then, platform value is attribution and reliability on top of someone else's GPUs.

Platform owns the OpEx

Inference is the durable cost center of production AI. Platform engineers already own scheduling, multi-tenancy, and cost signals for cheaper resources; the same instincts apply when the unit is a GPU-hour and the failure mode is a runaway agent loop. The second-order pressure is real: cheaper software production increases software volume, which increases inference load ([AI Creates Software Faster Than Ops Can Handle](https://stxkxs.io/blog/second-order-explosion)). Headcount no longer tracks the bill.

Own the bottleneck you can actually govern: serving quality when you must run weights, GPU utilization when you own the hardware, and gateway policy when you do not. Everything else is inventory until a constraint makes it load-bearing.

Inference is the OpEx that never stops. Self-host only when the meter and the mandate force the hardware home.

Key Point: Inference is the bottleneck platform engineers own. Self-host only when forced.

Resources & Further Reading

• Deloitte 2026 TMT Predictions (compute power for AI): https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html - Inference vs training compute share cited in the hook and chart
• vLLM: https://github.com/vllm-project/vllm - Open-source LLM serving engine; PagedAttention and continuous batching
• PagedAttention paper: https://arxiv.org/abs/2309.06180 - KV-cache paging design behind vLLM's memory efficiency
• KServe: https://kserve.github.io/website/ - CNCF incubating Kubernetes-native model serving (InferenceService CRD)
• KEDA: https://keda.sh/ - Event-driven autoscaling often paired with token-aware LLM scale signals
• NVIDIA GPU Operator: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/ - Drivers, device plugin, and GPU node lifecycle on Kubernetes
• Kubernetes Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/ - ResourceClaim-based accelerator scheduling primitives
• NVIDIA MIG User Guide: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ - Hardware-isolated GPU instances on supported accelerators
• LiteLLM: https://github.com/BerriAI/litellm - Open-source multi-provider LLM gateway with budgets and fallbacks
• Envoy AI Gateway: https://gateway.envoyproxy.io/docs/tasks/ai-gateway/ - LLM-aware routing and policy on Envoy
• eks-agent-platform (nanohype): https://github.com/nanohype/eks-agent-platform - BudgetPolicy, per-tenant platform CRs, and EvalSuite promotion gates on EKS
• Vectors Are a Data Type: https://stxkxs.io/blog/vectors-are-a-data-type - Why purpose-built vector databases are usually the wrong first buy
• AI Creates Software Faster Than Ops Can Handle: https://stxkxs.io/blog/second-order-explosion - Why inference load scales with software volume, not headcount
• FinOps as First-Class Engineering: https://stxkxs.io/blog/finops-first-class-engineering - Cost as a decision-time signal for infrastructure changes

Cross-cloud connectivity used to take weeks or months when built the colo-and-BGP way ([AWS](https://aws.amazon.com/about-aws/whats-new/2025/11/preview-aws-interconnect-multicloud/)). Colo ticket, cross-connect, IP negotiation, BGP, MACsec ownership fights, then weeks proving end-to-end encryption to security. By the time traffic flows, the project has already shipped a hand-maintained VPN over the public internet with no real failover plan.

That is why most multi-cloud architectures are accidental. Teams inherit an Azure tenant from a merger, need a GCP service AWS does not offer, or absorb a vendor that deploys into a cloud they did not pick. Networking between those clouds is always an afterthought, so architects design around the path instead of through it: duplicated workloads, copied data, worse local substitutes for services that already exist elsewhere because the network is too fragile to depend on.

That constraint is breaking. AWS announced Interconnect in preview at re:Invent 2025 (November 30, 2025): a managed, MACsec-encrypted, quad-redundant service connecting AWS directly to Google Cloud. It reached general availability on April 14, 2026 for five AWS-to-Google Cloud region pairs, with Oracle Cloud connectivity in preview since May 2026 and Azure support still later in the year ([AWS, April 2026](https://aws.amazon.com/blogs/aws/aws-interconnect-is-now-generally-available-with-a-new-option-to-simplify-last-mile-connectivity/)). Google shipped Cross-Cloud Interconnect two years earlier. With both sides managed, provisioning moves from infrastructure project to configuration change: target cloud, region pair, bandwidth. Minutes. Cross-cloud connectivity you provision from an API is a new architectural primitive.

Key Point: Cross-cloud networking used to be a project. Now it is a configuration change. When connecting to another cloud is as easy as connecting to another region, intentional multi-cloud stops being a networking bet and becomes an architecture choice.

Data gravity decides ownership

Data gravity is the load-bearing concept multi-cloud networking discussions skip. Data is heavy in dependencies, not only in bytes. A 50TB dataset in S3 has Lambda readers, Glue jobs, Athena queries, and downstream services hanging off those. Moving it to GCS is not a 50TB copy. It is rebuilding every integration that touches it.

Move compute to the data, not the data to the compute. Managed interconnect makes that practical. Instead of replicating a dataset to the cloud with the best processing engine, run that engine's compute against the data over private interconnect. The data stays where its dependency graph lives. The compute travels on a 100 Gbps encrypted link. Ownership follows the dependency graph; the transfer bill is secondary.

Wired correctly, each cloud owns the data its ecosystem depends on. Cross-cloud traffic is compute-to-data queries and model artifact transfers, not bulk replication. S3 stays on AWS. BigQuery tables stay on GCP. Azure Data Lake stays on Azure. What crosses the interconnect is inference requests, query results, and trained model weights: high-value, relatively low-volume traffic compared to moving the underlying datasets.

WARNING: The costly mistake
The most expensive multi-cloud architecture fights data gravity. Designs that start with "first, replicate everything to a central lake" spend more on transfer and synchronization than they save on best-of-breed services. Interconnect makes moving compute cheap. Moving data is never free.

No single cloud dominates every analytics workload. Event streaming tends to run wherever the application already emits events. Warehousing follows whichever platform a team standardized on first. BI follows the office suite already licensed. The estate is usually stuck with wherever the data landed first, not choosing best-of-breed per workload. Managed interconnect turns that inheritance into something that can be orchestrated instead of duplicated.

Intentional multi-cloud works

Split training and inference

Application stack, data pipelines, MLOps tooling, and feature store on AWS; training economics better elsewhere. Google's TPU v5p pods deliver roughly 2.8x faster LLM training and about 2.1x better performance-per-dollar than TPU v4 for large transformer workloads ([Google Cloud](https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-tpu-v5p-and-ai-hypercomputer)); Trillium (v6e) and Ironwood have since taken over as Google's fastest TPU generations. Before managed interconnect the options were duplicate the training pipeline on GCP or accept worse economics on AWS. Both are common. Both are wasteful.

Clean shape: training data stays in S3. A cross-cloud pipeline streams batches over interconnect to TPU pods on GCP. When training completes, the model artifact (tens to hundreds of gigabytes) transfers back over the 100 Gbps dedicated link to AWS. Inference runs on SageMaker or Inferentia next to the application stack that calls it. The artifact transfer that bottlenecked on public internet now takes minutes. One MLOps pipeline, not two parallel stacks.

Cross-cloud inference chains

Foundation models have differentiated strengths. Claude on Bedrock for nuanced reasoning and long-context analysis. GPT on Azure OpenAI for structured output and function calling. Gemini on Vertex AI for native multimodal grounding. Teams that standardize on one model accept its limits. Teams that use several often route over the public internet, adding latency per hop and exposing data in transit.

A private inference chain keeps each hop on managed interconnect. Document in: Claude for reasoning and extraction, GPT for schema-validated domain types, Gemini for images and diagrams when present. Each hop on a predictable private link. Data never touches the public internet. Each model is used for what it is best at because the network path is finally dependable enough to chain them.

• Model Transfer Speed: 100 Gbps — Maximum bandwidth per port on AWS Interconnect - multicloud and Google Cross-Cloud Interconnect

Federated streaming analytics

A common ugly pipeline: event ingestion on AWS (Kinesis, MSK, EventBridge) because that is where the app emits events; analysis in BigQuery for serverless, petabyte-scale SQL; features back to SageMaker for real-time inference. The old path dumps Kinesis Firehose into S3, copies to GCS, ingests into BigQuery with minutes-to-hours delay, exports features back to S3 and a SageMaker feature store. Round-trip latency kills anything real-time, so teams stand up a parallel Flink stack on AWS and duplicate logic BigQuery ML could run in SQL.

With interconnect: events stream from Kinesis to GCP Dataflow, which has exactly-once semantics via the Storage Write API and native BigQuery output. BigQuery ML runs feature engineering in SQL beside the analytical queries. Computed features flow back over interconnect to SageMaker endpoints. Round-trip adds milliseconds, not minutes. One pipeline, not two stacks with duplicated logic.

TIP: Start with the workaround
The clearest cross-cloud opportunity is wherever a team already has a workaround: scheduled data copies between clouds, public API routing when a private link would be faster, or a worse local service because the better one lives elsewhere. Start there. Greenfield multi-cloud architecture is how resume-driven designs get funded.

Interconnect becomes IaC

Managed interconnect differs from colo-and-BGP in how it fits infrastructure-as-code. AWS Interconnect - multicloud is provisioned through the Direct Connect console and API rather than a purpose-built CloudFormation resource, which is enough surface to wrap behind a CDK custom resource. Cross-cloud networking then goes through the same PR review, drift detection, and CI/CD pipeline as the rest of the infrastructure. No side-channel tickets to the networking team.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

// Illustrative only. AWS Interconnect - multicloud (GA April 2026) is
// provisioned through the Direct Connect console/API, not a native
// CloudFormation resource type. A CDK construct wraps that API behind
// a custom resource.
export class CrossCloudInterconnect extends Construct {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    const interconnect = new cdk.CfnResource(this, 'GcpInterconnect', {
      type: 'Custom::CrossCloudInterconnect',
      properties: {
        targetProvider: 'GCP',
        targetRegion: 'us-central1',
        awsRegion: 'us-west-2',
        bandwidthGbps: 10,
        encryption: 'MACSEC',
        redundancy: 'QUAD',
      },
    });

    new cdk.CfnResource(this, 'CrossCloudRoute', {
      type: 'AWS::EC2::TransitGatewayRoute',
      properties: {
        destinationCidrBlock: '10.128.0.0/16', // GCP VPC range
        transitGatewayAttachmentId: interconnect.ref,
      },
    });
  }
}

Compared to Terraform modules and manual runbooks for a colo cross-connect, the wrapped custom resource is a versioned, reviewable, deployable piece of the stack. Spin up a dev interconnect, tear it down, parameterize bandwidth and region pair per environment. Cross-cloud connectivity becomes a build-time decision, not a procurement process.

AWS published the interconnect specification as an open spec on GitHub. Other providers can implement compatible endpoints. Whether they will is open, but an open spec moves the conversation from proprietary AWS service toward a potential industry standard, which matters for teams justifying multi-cloud connectivity without signing another lock-in argument.

Agents need real interconnect

Connectivity is the foundation. Cross-cloud data pipelines are the plumbing. The orchestration layer that ties them together is less discussed than it should be: Bedrock AgentCore, generally available since October 2025. It is a managed runtime for AI agents built to run multiple frameworks and multiple models across multiple clouds. That multi-cloud support is the architectural decision, not a checkbox.

AgentCore runs agents built with CrewAI, LangGraph, LlamaIndex, Google's Agent Development Kit, the OpenAI Agents SDK, and Anthropic's Claude SDK. It supports Claude, GPT, Gemini, Llama, and any model accessible via API. It implements Google's A2A protocol for agent-to-agent communication and Anthropic's MCP through a native MCP Gateway. A LangGraph agent using Claude can discover and talk to a CrewAI agent using GPT, with tools gated through one governance layer.

// Bedrock AgentCore: multi-framework agent deployment
// Based on announced API surface at re:Invent 2025

import {
  BedrockAgentCoreClient,
  CreateAgentRuntimeCommand,
} from '@aws-sdk/client-bedrock-agentcore';

const client = new BedrockAgentCoreClient({ region: 'us-west-2' });

const langGraphAgent = await client.send(
  new CreateAgentRuntimeCommand({
    agentName: 'research-agent',
    framework: 'LANGGRAPH',
    modelId: 'anthropic.claude-sonnet-4-5-20250929-v1:0',
    mcpServers: [
      {
        name: 'company-docs',
        uri: 'https://mcp.internal.company.com/docs',
      },
    ],
    memoryConfig: {
      type: 'SEMANTIC',
      retentionDays: 90,
    },
    guardrails: {
      contentFilters: ['HARMFUL_CONTENT', 'PII_DETECTION'],
      maxTokensPerTurn: 4096,
    },
  })
);

const crewAgent = await client.send(
  new CreateAgentRuntimeCommand({
    agentName: 'analysis-crew',
    framework: 'CREWAI',
    modelId: 'openai.gpt-oss-20b',
    a2aConfig: {
      enabled: true,
      discoverable: true,
    },
  })
);

An agent on AgentCore can orchestrate a pipeline that calls Bedrock for reasoning, routes to Vertex AI for multimodal grounding, pulls structured data from Azure Cognitive Services, and coordinates with agents on other clouds via A2A, all over private interconnect. Without that private path, cross-cloud agent orchestration is public-internet glue with latency and exposure nobody would accept for a data plane. Combined with managed interconnect, agents become the control plane between cloud-specific services that previously needed custom integration code.

WARNING: Agent sprawl is Lambda sprawl
A new abstraction makes creation cheap. Organizations create prolifically. Operational complexity follows. Agent sprawl will mirror Lambda sprawl: unclear ownership, undocumented tool access, unpredictable cross-agent interactions. AgentCore's MCP Gateway centralizes tool governance, but ownership requirements, discovery, and deprecation pathways matter more than any platform feature. Teams that lived through "everyone deploys Lambdas, nobody owns them" already know the playbook.

Resume-driven multi-cloud fails

Do not do this. Managed interconnect will tempt organizations into multi-cloud architectures they do not need and cannot operate. Multi-cloud has a clear value when differentiated capabilities from multiple providers justify the overhead. It has none when clouds are added for resume padding, merger residue left un-reconciled, or a strategy slide that never named a workload. Cross-cloud pipelines that could have been one SageMaker endpoint are a recurring failure mode: organizational indecision presented as technical sophistication.

Flexera's 2025 State of the Cloud report found 70% of organizations embrace hybrid strategies and an average of 2.4 public cloud providers per organization. Presence on multiple clouds is already the norm. That does not make every multi-cloud design intentional. The test is whether each cloud earns its place with a specific technical advantage large enough to pay for identity federation, unified observability, consistent security policy, and operators who understand more than one platform.

Training on TPUs where price-performance is documented: worth it. BigQuery when Redshift already covers the query patterns with marginal difference: usually not. Three foundation models in an inference chain because each excels at a specific task: worth it. Three clouds because three teams each picked a favorite is an org-chart problem, not an architecture decision.

Multi-cloud is a tool, not a strategy. The strategy is using the best capability for each workload. Sometimes that means two clouds. Sometimes it means one.

• Avg Cloud Providers: 2.4 — Average number of public cloud providers per organization; 70% embrace hybrid strategies (Flexera 2025)

AWS wants the control plane

Look at what AWS built in aggregate. Cross-cloud connections originate from AWS. Agent orchestration runs on AgentCore. The MCP Gateway centralizes tool access on AWS. The hub is always AWS. This is the EKS playbook applied to multi-cloud: treat multi-cloud adoption as inevitable, build the management layer for it, and make sure that layer runs on your cloud.

The strategy is sound, and teams should enter with open eyes. Google shipped Cross-Cloud Interconnect two full years before AWS; it has been generally available since 2023 with production support for direct connections to AWS, Azure, and Oracle Cloud. Azure's enterprise footprint gives it a natural governance seat: Entra ID and Microsoft 365 already sit inside most large enterprises' identity and compliance stack. All three building managed interconnect and multi-model orchestration says the stack is real. Who owns the control plane is still an open fight.

The multi-cloud stack has four layers: private network fabric, differentiated AI services, specialized data and analytics, and agent orchestration as control plane. Each layer exists because no single cloud is best at everything. The multi-cloud decision was, in the Flexera data, already made years ago. What remains is which strengths to combine, what data flow looks like between them, and whether the operational overhead of spanning clouds is justified by a real capability delta.

Start with one pipeline

• Map cross-cloud workarounds: VPN tunnels, public API calls that should be private, scheduled dataset copies. Workarounds mark where the architecture already wants a private path.
• Pick one pipeline with a real limitation (model economics, analytics engine, public-internet routing). Build that one on managed interconnect and learn the operational model before expanding.
• Design around data gravity: which cloud's compute moves to the data, not which data moves to the compute. Bulk-copy-first designs are fighting the cost structure.
• Fix identity before networking. Long-lived cross-cloud credentials with trivial connectivity is worse security than a painful VPN. Workload identity federation first. Deploy OpenTelemetry so a request can be traced across clouds; CloudWatch-plus-Cloud-Monitoring alone cannot.
• Model port and transfer costs before committing. As of July 2026, [Google Cross-Cloud Interconnect](https://cloud.google.com/network-connectivity/docs/interconnect/pricing) port fees run ~$4,032/mo for 10 Gbps and ~$21,600/mo for 100 Gbps; [AWS Interconnect - multicloud](https://aws.amazon.com/interconnect/multicloud/pricing/) runs ~$9,001/mo for a 10 Gbps Tier-1 connection. Two 100 Gbps links alone are roughly $43K/month before per-GB egress. Let the cost model validate the architecture.

INFO: The pattern that works
Successful multi-cloud programs start with one pipeline that solves a real limitation, build identity, observability, and cost tracking for that pipeline, and expand only when the next workload has a clear capability justification. Programs that start from a "multi-cloud strategy" and go hunting for workloads reverse the order and fail for the same reason resume-driven multi-cloud always fails.

Key Point: Managed interconnect makes intentional multi-cloud cheap to provision. Resume-driven multi-cloud still fails. Move compute to data; agents can orchestrate across clouds only with real interconnect and clear control-plane ownership.

Resources & Further Reading

• AWS Interconnect Preview Announcement: https://aws.amazon.com/about-aws/whats-new/2025/11/preview-aws-interconnect-multicloud/ - Preview announcement for managed cross-cloud networking, re:Invent 2025
• AWS Interconnect General Availability: https://aws.amazon.com/blogs/aws/aws-interconnect-is-now-generally-available-with-a-new-option-to-simplify-last-mile-connectivity/ - GA announcement, April 2026, AWS-to-Google Cloud (five region pairs)
• AWS Interconnect Architecture: https://aws.amazon.com/blogs/networking-and-content-delivery/build-resilient-and-scalable-multicloud-connectivity-architectures-with-aws-interconnect-multicloud/ - Technical blog on multicloud connectivity architectures
• Interconnect Open Specification: https://github.com/aws/Interconnect - OpenAPI 3.0 spec for the Connection Coordinator API
• Bedrock AgentCore: https://aws.amazon.com/bedrock/agentcore/ - Managed platform for building, deploying, and operating AI agents at scale
• Google Cross-Cloud Interconnect: https://cloud.google.com/network-connectivity/docs/interconnect/concepts/cci-overview - Google Cloud cross-cloud connectivity service
• Google Cross-Cloud Interconnect Pricing: https://cloud.google.com/network-connectivity/docs/interconnect/pricing - Port and transfer pricing for Cross-Cloud Interconnect
• Flexera 2025 State of the Cloud: https://www.flexera.com/blog/finops/the-latest-cloud-computing-trends-flexera-2025-state-of-the-cloud-report/ - 70% hybrid strategies, 2.4 average public cloud providers
• GCP BigQuery ML: https://cloud.google.com/bigquery/docs/bqml-introduction - SQL-native machine learning in BigQuery

A 3am PagerDuty page for a payment-service outage still costs the responder a laptop, VPN, cluster auth, and log pulls before the first finding lands in the incident channel. An always-on ChatOps agent in Slack collapses that ceremony: one message runs kubectl, parses logs, and posts a structured summary where the whole team can see it. That value is real. It is also the wrong place to start.

[OpenClaw](https://github.com/openclaw/openclaw) is a self-hosted LLM agent that sits in Slack, Teams, WhatsApp, or Telegram and runs tools under a service account the operator controls. Session state and workspace files stay on the host by default. Prompts and context still leave for the configured model provider, and every message still leaves for the chat platform. Self-hosted does not mean air-gapped.

A self-hosted ChatOps agent earns its place only when sandboxing is on, the service account is scoped, and egress is described honestly. The default install fails that bar. Sandbox mode starts at off. Until those three controls are deliberate, the agent is a new attack surface holding cluster credentials, not a production tool.

INFO: Clawdbot, Moltbot, OpenClaw
Renamed twice in January 2026 (Clawdbot → Moltbot on the 27th, OpenClaw on the 30th). Legitimate sources: [github.com/openclaw/openclaw](https://github.com/openclaw/openclaw), npm package `openclaw`, and [openclaw.ai](https://openclaw.ai). As of July 2026 the repo has roughly 380K GitHub stars. Creator Peter Steinberger joined OpenAI in February 2026 with a plan to move the project under an open-source foundation.

Local process, remote data

OpenClaw runs as a Node.js process on the operator's infrastructure: a dedicated VM is the usual ChatOps shape, with a persistent Slack socket and pre-authenticated sessions to clusters. That is a different trust model from a SaaS assistant that phones home with infrastructure context. Security review can reason about the host, the service account, and the audit trail the same way it does for any other production binary.

Chat-driven operations is not new. Hubot ran commands from chat in 2011, and StackStorm automated runbooks for a decade after that, neither with a model in the loop. What changed is the thing choosing the command. A script maps a phrase to a fixed action. A model decides what the phrase meant, and that is why the blast-radius question lands differently this time.

The affirmative case for self-hosting is compliance rather than preference. FedRAMP authorization, HIPAA business associate agreements, and EU data-residency rules constrain where infrastructure context and prompts are allowed to travel, and a SaaS assistant that phones home to a third-party cloud often cannot satisfy them. Self-hosting the agent and the model closes the provider path. The chat platform still sees every message, so the residency argument only holds when the chat platform is in scope too.

Model choice is orthogonal to host ownership. The agent is model-agnostic: cloud providers, Azure and Bedrock, and local models via Ollama all work. Pairing a self-hosted model with a self-hosted agent closes one egress path. It does not close chat-platform egress, and it does not replace sandbox or RBAC.

Permission inheritance is the load-bearing property. When the agent runs kubectl or terraform, it uses exactly the RBAC of the service account that launched it: no ambient escalation, no hidden admin role. Existing audit infrastructure already records what that account does. The same identity discipline applies whether a human or an agent holds the credential; [independent evidence of what the agent did still has to come from the API audit trail, not the agent transcript](https://stxkxs.io/blog/agent-said-it-worked).

Sandbox off is default

OpenClaw exposes three sandbox modes: `off`, `non-main`, and `all`. Execution backends (docker, ssh, openshell) are chosen separately. Sandboxing is off by default. A first production change is turning it on: start at `all`, and relax to `non-main` only with a written reason. Network disabled inside the Docker sandbox keeps a successful prompt injection inside a disposable container with no egress.

• Sandbox modes: 3 — off, non-main, all; sandboxing is off by default ([OpenClaw security docs](https://docs.openclaw.ai/gateway/security))
• dmPolicy modes: 4 — open, allowlist, pairing, disabled; pairing is the security-first default for unknown senders
• Security audit: CLI — `openclaw security audit` flags risky configuration before deploy ([docs.openclaw.ai/gateway/security](https://docs.openclaw.ai/gateway/security))

A common failure mode is unrestricted mode left on after a test. An ambiguous Slack message becomes a kubectl delete. If the service account lacks delete, nothing catastrophic happens. If it has delete, the mistake reaches production. Sandbox configuration and RBAC are independent layers; production needs both.

OpenClaw has no built-in explain-then-confirm step before tool execution. Its `dmPolicy` modes are open, allowlist, pairing, and disabled. Pairing gates unknown senders behind a one-time code; it does not gate each command. A confirming owner-approval mode was proposed in issue #6262 and closed as not planned on February 1, 2026. Until something equivalent ships, confirmation for destructive work has to live in the runbook pattern, channel scoping, sandbox mode, and service-account RBAC.

WARNING: Production security baseline
Docker sandbox mode `all` with network disabled; channel allowlist (`allowFrom` + `groupPolicy: allowlist`); `dmPolicy: pairing` for DMs; least-privilege service account; command audit logs shipped to a SIEM. Review open security issues on the GitHub repo before depending on the agent for on-call.

Scope channels and sandbox

Install needs Node.js 22+, then `npm install -g openclaw`. Provider auth uses the models CLI: `openclaw models auth login --provider openai`, then `openclaw models set ollama/llama3.2` for a local default. Slack needs a socket-mode app with the usual bot scopes (`chat:write`, `app_mentions:read`, `channels:history`). Access and sandbox configuration matter more than the install one-liner.

{
  "channels": {
    "slack": {
      "botToken": "$SLACK_BOT_TOKEN",
      "appToken": "$SLACK_APP_TOKEN"
    }
  },
  "access": {
    "dmPolicy": "pairing",
    "allowFrom": ["#ops", "#incidents", "#deployments"],
    "groupPolicy": "allowlist"
  },
  "agents": {
    "defaults": {
      "sandbox": {
        "mode": "all",
        "docker": {
          "network": false
        }
      }
    }
  }
}

Channel allowlisting means the agent ignores #random. Pairing means unknown DMs never reach tools until a human issues a code. Sandbox mode `all` with Docker network disabled means tool execution is contained even when a prompt is hostile. Two layers (access policy and sandbox) have to fail before the blast radius reaches the cluster, and the service account is still the third.

Two patterns earn cost

ChatOps agents earn their keep by removing context switches during incidents. The workflows that justify the attack surface are the ones that already burn minutes of auth and copy-paste. Two patterns clear that bar often enough to matter.

Incident triage in channel

The first fifteen minutes of many incidents are auth and discovery: which namespace, which pods, which recent events. A single channel message that runs get/describe/logs/events and posts a correlated summary puts findings in the thread the incident commander already watches. The agent still needs verification before anyone acts on a recommendation. The win is shared context in seconds instead of four private terminals.

On-call triage from a phone is the same pattern with higher stakes. False positives die without opening a laptop. Escalation decisions get the lag graph, the deployment history, and the blast-radius note before the responder is fully awake. That only stays safe if the agent cannot mutate production without a human checkpoint or without RBAC that already forbids the mutation.

Runbooks with human gates

Destructive runbooks (failover, credential rotation, rollback) are sequences of commands with go/no-go pauses. At 3am those pauses are what prevent the wrong environment or a skipped sync check. OpenClaw has no native confirm-before-execute mode, so the pause has to be designed into the workflow: the agent posts the next step and waits for an explicit yes in thread before continuing.

!claw run postgres failover runbook

**Step 3/5**: Wait for replica sync
> replica-1: lag 0s
> replica-2: lag 0s
Proceed to step 4 (promote replica-1)? (yes/no)

**yes**

**Step 4/5**: Promote postgres-replica-1
> pg_ctl promote executed
> New primary accepting writes

The Slack thread is the audit trail for free. Other operators can intervene. The human still owns every go/no-go. Pair that pattern with a service account that cannot promote databases outside the intended path, and the agent is an executor of a reviewed procedure rather than an unsupervised operator. How this sits inside a broader [agentic DevOps loop](https://stxkxs.io/blog/agentic-devops-loop) is a separate design problem; the security bar does not change.

Key Point: If the agent can only read cluster state, channel scoping and sandbox still matter (prompt injection and data exfiltration remain real). If it can write, human gates and least-privilege RBAC are non-negotiable.

MCP widens the surface

Without MCP, OpenClaw is a sandboxed shell runner. With MCP, the same agent issues structured tool calls against observability, incident, cloud, and docs systems. That is useful during incidents: typed metrics beat raw curl JSON. It is also a larger trust boundary. Every MCP server added is another set of credentials and capabilities the agent can reach if a prompt goes wrong.

Use official or well-maintained servers, not invented paths under `modelcontextprotocol/servers`. Datadog publishes a hosted MCP server in its own docs. Kubernetes operations have a Go-native server in the containers org, not a phantom path in the reference servers repo. Treat MCP addition as a security change: least privilege on the MCP credentials, same sandbox, same channel policy. The protocol itself is now under the Linux Foundation ([MCP is a Linux Foundation standard](https://stxkxs.io/blog/mcp-linux-foundation)); that does not make every server safe to wire into production RBAC.

Rare toil does not qualify

An always-on agent holding cluster credentials is justified by recurring toil. Teams without that volume should keep a documented runbook and a kubectl alias.

The project still churns

Two renames in four days moved the package, the org, and the docs domain. Pinned configs and Slack manifests reference surfaces that have not been stable for a full quarter. Whoever deploys it owns breakage when the next API or naming shift lands. The closed-as-not-planned owner-approval mode (issue #6262) is a reminder that the maintainer roadmap will not always match a security review.

No one to contain it

Scoping the service account, writing the sandbox policy, and owning both over time is the security story. A team without that role will ship default unsandboxed mode and discover the gap when the service account has delete. If nobody can own containment, OpenClaw is a liability waiting for the wrong prompt.

Nothing is watching it

Command logs only help if something reads them. Without a SIEM or equivalent alerting on out-of-policy actions, the log is forensics after damage. That is a worse posture than a human typing the same commands, because the human is the detection layer.

Too few incidents

The product is compressing time between page and understanding. For a team paged twice a quarter, the ceremony is rare and the new surface sits exposed every day. Below some incident volume, documented runbooks clear the same path without an always-on process holding cluster credentials. Occasional annoyance does not justify it.

Security is the product

ChatOps is a strong interface for infrastructure agents: the team already coordinates there, the audit trail is free, and triage from a phone is real. None of that excuses default install. Sandbox on, identity scoped, egress stated honestly, logs shipped somewhere that pages. Evaluate OpenClaw in that order. Stars and growth curves do not pass a security review.

Self-hosted ChatOps agents earn their place only with production security. The default install is not that configuration.

Resources & Further Reading

• OpenClaw GitHub: https://github.com/openclaw/openclaw - Source, issues, and security discussions
• OpenClaw security docs: https://docs.openclaw.ai/gateway/security - Sandbox modes and `openclaw security audit`
• OpenClaw site: https://openclaw.ai - Downloads and project home
• npm openclaw: https://www.npmjs.com/package/openclaw - Install via `npm install -g openclaw`
• The Agent Said It Worked: https://stxkxs.io/blog/agent-said-it-worked - Why agent transcripts are not evidence of what ran
• The Agentic DevOps Loop: https://stxkxs.io/blog/agentic-devops-loop - How agents fit the operations workflow
• MCP Is Now a Linux Foundation Standard: https://stxkxs.io/blog/mcp-linux-foundation - Protocol governance behind tool integrations
• Model Context Protocol: https://modelcontextprotocol.io/ - Spec and registry entry points
• Datadog MCP Server: https://docs.datadoghq.com/mcp_server/ - Official Datadog-hosted MCP for metrics and logs
• Kubernetes MCP Server: https://github.com/containers/kubernetes-mcp-server - Go-native MCP for Kubernetes and OpenShift
• Slack app config: https://api.slack.com/apps - Socket-mode apps for ChatOps bots
• ZeroLeaks OpenClaw analysis: https://zeroleaks.ai/reports/openclaw-analysis.pdf - Independent security analysis
• Steinberger joins OpenAI: https://techcrunch.com/2026/02/15/openclaw-creator-peter-steinberger-joins-openai/ - Coverage of the February 2026 hire

Addy Osmani's "The Efficiency Paradox" and Aaron Levie's work on Jevons Paradox for knowledge work established the foundational insight: making software dramatically cheaper to build leads organizations to build dramatically more of it. This piece takes that as given and asks the operational follow-up: which systems prevent the predictable failures?

Strictly speaking, Jevons Paradox describes resource consumption (coal efficiency raised coal usage because demand was elastic), and the software analogy is imperfect. More code is the visible effect. Integration complexity between that new code grows quadratically, and that mechanism is worse than the one Jevons described. The historical pattern across every major transition (assembly to high-level languages, bare metal to cloud, manual deploy to CI/CD) is the same: efficiency gains expand output rather than reduce effort.

The audience is platform engineers, DevOps teams, and technical leadership who accept the premise and need countermeasures. The focus is narrow: failure modes that follow from the premise, and the systems that bound them.

Why linear scaling fails

Platform teams have always supported growing service portfolios. AI collapses implementation cost, which changes both the growth rate and what gets built.

A platform team supporting N services pays some baseline operational cost C per service (monitoring, runbooks, dependency updates, incident capacity). Total load is approximately N × C, scaled historically by hiring. When implementation becomes 10x cheaper, organizations do not build 10x the same services. The backlog of "not quite worth building" internal tools becomes viable. Bespoke integrations replace manual workflows. Custom dashboards proliferate. One-off automation graduates to production. Service count rises and composition shifts toward smaller, more numerous, less-documented, less-standardized systems.

Linear growth vs quadratic complexity

• 25: 25
• 50: 50
• 75: 75
• 100: 100
• 150: 150
• 200: 200
• 250: 250

The integration layer is where quadratic scaling becomes dangerous. N services connect, share data, and trigger workflows. Potential integration points approach N²; even a fraction active grows the operational surface faster than linearly. A team that handled 50 services struggles at 150 even with triple headcount, because integration complexity grew 9x while capacity grew 3x.

This piece's working forecast is 3-5x service growth at organizations with aggressive AI adoption. That range is an author estimate, directional rather than measured, not a published survey figure. Documentation coverage and ownership clarity move the way the mechanism predicts: creation outpaces documentation, systems change hands without formal transfer, more end up orphaned. The rest of this piece treats that forecast as premise, not established fact.

Failure arrives in sequence

At organizations with aggressive AI adoption, these failure modes arrive in roughly the same order the microservices wave produced them, at smaller scale. Order matters: earlier interventions prevent cascading failures downstream.

The discovery collapse

Discovery fails first: the ability to find what already exists. It precedes the rest because it causes duplication, which accelerates every later problem. When finding an existing solution takes longer than building a new one, rational actors build new ones. Three teams solve the same problem three ways in one quarter, each unaware of the others. Nobody was careless. The existing solution genuinely took longer to find than to rebuild. Symptoms: Slack threads with no answer or conflicting answers; post-mortems that hit unknown systems; new services that duplicate available function; architecture reviews that surface integrations nobody knew about. Knowledge systems designed for 50 services do not scale to 200.

TIP: Platform team action: service catalog infrastructure
Populate a catalog from CI/CD, runtime, and monitoring without manual registration. Make it authoritative and invest in search quality. It should answer "does something like this exist?" within 30 seconds. If it cannot, engineers will bypass it.

The ownership vacuum

Software built quickly often lacks clear ownership. The engineer who wrote it owns it implicitly until they transfer, get promoted, or leave. The service keeps running. Dependencies keep depending. When it breaks, needs a security patch, or requires an update, no one steps forward. It is an orphan. Each orphan is a maintenance liability that falls to whoever draws the short straw in incident response. Platform teams absorb these by default and spend capacity on systems they did not build and do not understand, instead of on platform improvement.

WARNING: The orphan audit
Quarterly: for every production service, identify the specific person (not a team or rotation) who will answer pages at 3 AM. Services without clear pager assignment are orphans regardless of documentation. Track orphan count as a platform health metric. If it trends up, ownership systems are failing.

Key Point: Ownership is a pager assignment, not a documentation exercise. If no individual will be woken at 3 AM for a service, that service should not reach production.

The security bypass

Security review was designed for slow software. Implementation time rate-limited the queue. When implementation accelerates 10x and security capacity stays flat, the queue backs up. Engineers waiting weeks for review find workarounds. Shadow development appears: production systems that never hit established gates. Predictable outcomes include internal tools with production database access that skipped review, integrations that expose customer data on unsecured endpoints, and admin interfaces shipped with default credentials. The security team is overwhelmed by a volume it was never resourced to handle.

TIP: Platform team action: security as code in the platform
Shift security from review to guardrails: default-deny network policies, mandatory secrets management, automated dependency scanning, required authentication for new services. Make the secure path the easy path. Reserve human review for novel risk categories. Reduce attack surface by default rather than reviewing every service by hand.

The dependency cascade

More services with more integrations deepen dependency graphs. A change to a foundational service (API contract, deprecation, behavior shift) propagates through unknown consumers. A minor update becomes a cascading incident. Cascades violate the mental model of isolated failure. Engineers believe they understand the blast radius; the actual radius includes services they have never heard of, built by teams they do not know. Post-mortem graphs routinely look nothing like the expected graph, with a large share of connections undocumented integrations spun up in an afternoon and forgotten.

TIP: Platform team action: dependency mapping and contract testing
Track runtime service-to-service communication in addition to declared dependencies. Keep a continuously updated real dependency graph. For critical services, require contract tests that fail upstream changes when downstream consumers would break. The platform should answer "what breaks if I change this?" before the change ships.

The maintenance cliff

Every deployed service incurs ongoing maintenance: dependency updates, security patches, compatibility fixes. Those costs are relatively fixed per service and do not fall when the service was built quickly. A 5x service count is a 5x maintenance burden without matching engineering capacity. The cliff arrives gradually, then suddenly. Deferred updates accumulate. Security patches sit in backlog. Deprecated dependencies keep running. An external forcing function (critical CVE, cloud provider deadline, compliance requirement) brings the debt due at once. Keeping existing systems alive then consumes the capacity needed for new work.

WARNING: The maintenance budget
Before approving new services, calculate the maintenance budget. Every service needs an explicit allocation of ongoing engineering time: scheduled, not aspirational. If the budget is fully allocated, new services require additional headcount or deprecation of existing ones. Treat maintenance capacity as a finite resource that can be exhausted.

Four systems replace rate limits

These failure modes emerge from systems that were implicitly rate-limited by implementation difficulty. When that rate limit disappears, explicit systems must replace it. Minimum viable operational infrastructure for a fast-growing inventory is four pieces: a service registry, guardrails, deprecation tooling, and maintenance capacity planning.

Service registry with teeth

An unused registry creates false confidence, which is worse than no registry (see premature governance below). What separates a registry engineers keep current from one that goes stale:

• Automatic registration: services appear when deployed, not when someone remembers to add them
• Required ownership: deployment fails without a valid owner that passes validation
• Lifecycle tracking: creation date, last deployment, activity metrics, maintenance status
• Dependency mapping: runtime traffic analysis populates actual dependencies, not only declared ones
• Search that works: find by function, data touched, team, or technology in seconds

Guardrails over gates

Security and architecture review as gates (approval required before proceed) does not scale when implementation velocity rises 10x. Guardrails encode constraints in the platform so compliant behavior is the default and non-compliant behavior is difficult or impossible. Effective guardrails are invisible when followed and obvious when violated: network policies default deny; secrets management is the only credential path; container images fail promotion without vulnerability scanning; authentication is required for new services by platform default. Those constraints eliminate whole review categories because the platform prevents the problems before they appear.

INFO: The guardrail principle
For every issue security or architecture review currently catches, ask whether platform design can prevent it. Shift investment from review capacity to guardrail implementation. Fewer things requiring review beats faster review.

Deprecation needs automation

Software organizations excel at creation and struggle with removal. When creation velocity rises, that asymmetry becomes critical. Without active deprecation, service count only grows, maintenance only increases, and operational capacity ends up keeping legacy systems alive. Effective deprecation needs infrastructure: usage metrics that flag candidates (nothing called this in 90 days), dependency analysis for what would break on removal, automated notification to consumers, and eventual automated decommissioning after sunset completes. Make deprecation as procedurally clear as deployment; it needs to happen just as often.

Track maintenance capacity

Capacity planning usually tracks compute, memory, and storage. Engineering attention is the scarcer resource in an expanded portfolio. Every service needs patches, dependency updates, compatibility fixes, and incident response. Track maintenance hours per service (estimated at creation, refined with actuals), total portfolio load, engineering capacity dedicated to maintenance (not borrowed from feature work), and the delta between required and available capacity. When that capacity is exhausted, new services require headcount or deprecation. The tradeoffs become explicit rather than hidden.

Incentives must catch up

Technical systems alone are insufficient. Failure modes also come from org structures and incentives designed for constrained implementation capacity. Those have to evolve with the infrastructure.

Ownership as prerequisite

Every service needs an owner before production: a specific person responsible for availability, maintenance, and incident response. Ownership needs teeth. Backstage's software catalog requires a valid owner field in catalog-info.yaml before a component is accepted; a deployment pipeline can refuse promotion without the same. Without validation, the most available person gets named, not the right person. If no individual accepts the responsibility, the service should not be built. This is the highest-impact single intervention available.

Modify deployment pipelines to require valid owner assignment. Define ownership transfer when people change roles. Define ownership operationally (pager, maintenance commitment, deprecation authority). Track ownership health as an org metric. nanohype's [platform-tenant contract](https://stxkxs.io/blog/platform-engineering-ai-era) already enforces a version of this: it renders a tenant's Helm chart and Platform CR server-side and refuses the GitOps commit unless required fields are present. The same render-time refusal generalizes to ownership: reject the artifact before it exists rather than review it after.

Discovery before creation

Before approving a new service, require documented evidence that existing solutions were evaluated: what exists, why it is insufficient, cost of extending versus building new. Lightweight implementation is a required proposal field linking registry searches performed and results summarized. The build-new decision should be made with awareness of what already exists, because building economics now make that step easy to skip.

Match deprecation to creation

Track and report service deprecation alongside creation. A healthy organization deprecates at a meaningful fraction of creation rate. If creation outpaces deprecation indefinitely, the fleet grows without bound and maintenance capacity exhausts. Make deprecation visible and recognized equivalently to shipping new capabilities: engineers who sunset services free capacity for higher-value work.

TIP: The portfolio health dashboard
Surface total production services, created/deprecated this quarter, orphans (no clear owner), overdue maintenance, and maintenance capacity utilization. Review at the same cadence as feature delivery metrics. Portfolio health is as important as feature velocity.

Start with ownership

Teams that cannot tackle everything at once should sequence by impact and dependency. Ownership first: it gates everything else.

• Ownership enforcement: modify deployment pipelines to require owner assignment. Highest-impact single change; enables everything else.
• Service registry automation: automatic registration from deployment systems. Manual registries fail at scale; automated ones provide the foundation for discovery.
• Dependency mapping: runtime dependency tracking. The actual graph is prerequisite to managing cascade risk.
• Security guardrails: shift controls from review gates to platform defaults. Start with secrets management, network policies, and authentication.
• Deprecation tooling: identify candidates and manage sunset. Critical as portfolio size increases.
• Maintenance capacity tracking: explicit load versus available capacity. Makes tradeoffs visible before crises.

Each stage builds on the previous. Ownership without a registry leaves orphans invisible. Dependency mapping without ownership shows what depends on what without who can fix problems. Sequence matters.

Most teams should wait

The strongest objection is that building governance before the explosion wastes engineering capacity. Every system here costs real time to build and maintain. A team of six running a dozen services does not have an N² problem worth solving. Integration surface fits in one person's head, discovery lives in a Slack channel, and ownership is obvious because everyone knows who wrote what. A registry with teeth for that team solves a problem they do not have, at the cost of the work they were hired to do.

Quadratic math only bites once N is large enough that a fraction of N² exceeds informal tracking. Below that threshold the curve in Figure 1 is indistinguishable from linear, and informal systems still work. Premature governance has its own failure mode: a registry nobody needs becomes a registry nobody updates, and a stale registry is worse than none because it creates false confidence. The same applies to ownership enforcement that blocks deploys before there are enough orphans to justify the friction.

The 3-5x service growth forecast is an author estimate, directional not precise, and it will not hold everywhere. Where AI mostly speeds changes to existing services rather than spawning new ones, the portfolio explosion does not arrive. Maintenance grows with code volume, not service count, and the systems that matter are code review and test coverage, not service registries. The case for this infrastructure rests on service count climbing fast. Where it is flat, the case does not apply.

Key Point: Build the governance when the service count is climbing fast and ownership has started to blur, not before. The trigger is observed growth in the portfolio, not the existence of an AI coding tool.

The case for building early is narrower than every team doing this now. Retrofitting governance onto an already-exploded inventory costs far more than instrumenting while the fleet is still small. That tradeoff favors early investment only for organizations that can see the growth coming. Ownership enforcement is the exception that earns its place first regardless: a pipeline check that requires an owner costs little even at a dozen services and prevents the orphan accumulation everything downstream depends on.

Governance is the moat

AI-assisted development is making software dramatically cheaper to build, and organizations are building more of it. The efficiency paradox Osmani and Levie describe is already in motion. Platform teams that treat operational infrastructure as optional will face the failure sequence above at a rate the microservices wave of 2015-2018 only previewed. Microservices debt took years. AI-accelerated development can create the same debt in months.

The patterns that eventually emerged for microservices are the patterns needed now, deployed earlier: service catalogs, ownership registries, contract testing. Spotify built Backstage for this reason. Where agents also close the [agentic DevOps loop](https://stxkxs.io/blog/agentic-devops-loop) (deploy and remediate without a human in the path), creation velocity and operational mutation compound. Ownership, discovery, and dependency mapping become load-bearing substrate for that loop.

Platform engineering's mandate is to ensure that what gets built remains comprehensible, maintainable, and valuable over time. That is harder than making building easy. It is also more important.

Service registries, ownership systems, deprecation tooling, and maintenance capacity planning are unglamorous. The teams that ship them before the portfolio explodes will treat governance as infrastructure they own, and they will out-build the teams still drowning in debt they let accumulate.

Resources & Further Reading

• The Efficiency Paradox (Addy Osmani): https://addyosmani.com/blog/the-efficiency-paradox/ - Foundational essay on Jevons Paradox and AI-assisted development
• Aaron Levie on Jevons Paradox: https://x.com/levie/status/2004654686629163154 - Original observations on efficiency paradoxes in knowledge work
• The Agentic DevOps Loop: https://stxkxs.io/blog/agentic-devops-loop - Closed loop where agents deploy and remediate without a human in the path
• Backstage by Spotify: https://backstage.io/ - Open-source platform for developer portals and service catalogs
• Platform Engineering Maturity Model: https://tag-app-delivery.cncf.io/whitepapers/platform-eng-maturity-model/ - CNCF framework for assessing platform capabilities
• Team Topologies: https://teamtopologies.com/ - Frameworks for managing cognitive load in technology organizations
• The Staff Engineer's Path (Tanya Reilly): https://www.oreilly.com/library/view/the-staff-engineers/9781098118723/ - Scope, judgment, and organizational impact

AI coding assistants now write a meaningful share of new production code, though no single public metric reliably measures how much. What shifted is the ratio of time spent designing versus translating design into syntax: less of the second, more of the first. That ratio change reweights which language properties matter.

The [stxkxs.io](https://stxkxs.io) stack runs TypeScript for the SPA, CDK infrastructure, and API handlers, and three Rust Lambdas for SES events, email forwarding, and Cognito OTP (`apps/web`, `apps/infra`, `apps/api`, `apps/lambdas`). Neither choice was academic. Each was a bet on how AI-assisted development would go, and the split held: the compiler is a second pair of eyes on every line an agent writes.

Key Point: AI raises the value of languages with strong type systems and tight feedback loops. Training-data popularity alone is not enough.

Types catch AI's mistakes

When an agent adds a feature to the CDK stack here, the TypeScript compiler runs immediately after generation. Most of the time it passes. When it does not, the error is specific: wrong type on a construct prop, a missing required field, a return type mismatch. That is the category of error AI produces most often: structural mistakes where the logic is plausible but the data shape is wrong.

// AI generates this — looks correct, compiles fine
const handler = new NodeLambda(this, 'ApiHandler', {
  entry: resolve(__dirname, '../../api/dist/analytics/index.mjs'),
  environment: {
    ANALYTICS_TABLE: table.tableName,
    ALLOWED_ORIGINS: config.web.domain,
  },
  timeout: Duration.seconds(30),
  memorySize: 256,
})

// AI generates this — wrong type on authorizationType
// TypeScript catches it BEFORE deploy
const integration = new HttpLambdaIntegration('handler', handler.fn)
api.addRoutes({
  path: '/api/analytics',
  methods: [HttpMethod.POST],
  integration,
  authorizationType: 'JWT', // TS Error: not assignable to HttpRouteAuthorizationType
})

In `apps/api`, the same boundary applies. The `json()` helper returns a structured API Gateway response. The DynamoDB client expects typed inputs. Every seam between components is a type boundary, and every type boundary is a place where AI-generated code gets checked before deploy. Each of those failures would be a runtime error in JavaScript: the kind that surfaces when a handler throws because `undefined` is not a valid ARN.

INFO: The real AI error pattern
AI coding assistants are strong at syntax and idiom. Their consistent weakness is type correctness: ensuring data flows through a program with the right shape at every boundary. Static type systems verify exactly that. The compiler is the QA layer for AI output.

TypeScript won mechanically

In August 2025, TypeScript overtook both Python and JavaScript to become the most-used language on GitHub by monthly contributors: about 2.6 million contributors, 66% year-over-year growth ([GitHub Octoverse 2025](https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-second-as-ai-leads-typescript-to-1/)). TypeScript was already growing before AI coding tools, driven by React/Next.js, Deno, Bun, and Node modernization. AI accelerated a rise already underway. The acceleration tracks the failure mode AI exposes in untyped code.

Programming language growth on GitHub

• TypeScript: 66%
• Python: 49%
• JavaScript: 25%

The explanation is mechanical. JavaScript and TypeScript share syntax. TypeScript adds annotations that developers historically treated as overhead. When AI generates substantial portions of a codebase, that overhead moves off the human. The agent writes the annotations; the type checker verifies them. New projects start typed. Existing ones migrate. The historical argument that types slow you down lost its force once the AI paid the annotation cost.

Python remains dominant for AI/ML workloads. It peaked at a historic 26.98% on the TIOBE index in July 2025, the highest rating any language has ever reached, and has slipped since while still ranking #1 by a wide margin. That is the honest tension: the stack that builds AI (PyTorch, transformers, LangChain) runs on a dynamic language. The resolution is partial, not absolute. PyTorch runs mypy in CI; type hints are common across major ML libraries. Direction of travel is more type information everywhere, including Python, because AI made annotations cheap.

Rust enforces guarantees

stxkxs.io runs Rust for three Lambda functions: an SES event handler, an email forwarder, and a Cognito OTP flow (`apps/lambdas`). They process every inbound email and every authentication event on ARM64 Graviton2 with the PROVIDED_AL2023 runtime. Cold starts are fast relative to interpreted runtimes, and memory stays negligible at this scale. The stronger reason they are Rust is the guarantee model.

These handlers touch email content, authentication tokens, and user data. A null pointer in the forwarder means lost mail. A memory error in the auth handler is a security vulnerability. Rust's ownership system makes those failure modes structurally impossible at compile time, whether the code came from a human or an agent.

// The compiler enforces that every possible SES event variant is handled.
// AI can generate the match arms, but it cannot skip one —
// Rust's exhaustive pattern matching won't compile if a case is missing.

async fn handle_event(event: SesEvent) -> Result<Response, Error> {
    let records = event.records;

    for record in &records {
        let action = &record.ses.receipt.action;
        let from = &record.ses.mail.common_headers.from;
        let subject = &record.ses.mail.common_headers.subject;

        match action.action_type {
            ActionType::Lambda => process_lambda_action(&record.ses).await?,
            ActionType::S3 => process_s3_action(&record.ses).await?,
            ActionType::Bounce => process_bounce(&record.ses).await?,
            ActionType::Stop => log_stopped_processing(&record.ses),
            // Forgetting a variant here is a compile error, not a runtime bug
        }
    }

    Ok(build_response(records.len()))
}

Write the same handler in Python or JavaScript and the agent generates the same match structure, with nothing enforcing exhaustiveness. A new SES event type arrives; in Rust the build breaks, in Python the code falls through until a support ticket about missing mail. Rust will not replace Python for data science or TypeScript for the web. It is winning where correctness is non-negotiable: AWS built Firecracker (the microVM under Lambda and Fargate) in Rust; the Linux kernel has accepted Rust since 6.1; Cloudflare's Pingora proxy replaced NGINX at the edge.

• Most Admired Language: 10 Years — Consecutive years atop Stack Overflow survey

WARNING: The borrow checker still bites
AI compressed Rust's learning curve; it did not erase it. Concepts that took months can now be grasped in weeks with immediate feedback on ownership errors. The mental models still need building: AI can explain a borrow error and suggest a fix, but designing programs that work with ownership still requires understanding it.

Experts still hit friction

A randomized controlled trial by METR found that experienced developers working on familiar codebases were 19% slower with AI assistance. METR notes the sample skewed toward experienced OSS contributors and that the tasks may not represent typical professional work. The directional finding matches practice: it draws the line between where assistance helps and where it gets in the way, without arguing against AI tools wholesale.

Deep familiarity changes the calculus. A developer who has maintained a Rust module for months already knows the ownership patterns and error-handling strategy. AI suggestions in that context are friction: plausible, wrong for the codebase's patterns, and costlier to review than writing the fix. The opposite holds for unfamiliar territory: a new CDK stack, DynamoDB GSIs, CloudFront cache behaviors. The architecture is clear; the API surface is not memorized. Describing the intent, reviewing the construct calls, and letting tsc verify types turns an hour of docs into minutes.

AI helps most when you know what to build but not how to spell it. It helps least when you already have the incantation memorized.

Language choice amplifies that split. In TypeScript, wrong generation fails at tsc and the loop is tight. In untyped Python, the same mistake may wait for integration tests or production. Expert slowdown on familiar code is a ceiling effect. The real gain is safer exploration of unfamiliar territory, and typed languages widen that gain.

Training data is incomplete

AI does not produce uniform quality across languages. Three factors matter: training data volume, syntactic clarity, and whether a static type system can reject bad output. Massive corpora without a verifier leave the human as the only filter. A smaller corpus with a strict compiler can still yield safer generation, because the models that trained on that corpus learned from code that already survived the filter.

TypeScript and Python sit high on training volume and documentation. TypeScript adds compile-time verification. Rust has less absolute training data, but the code that exists is higher quality on average: the compiler filtered entire bug classes before commit, so models trained on Rust learn from already-checked examples. That is why training-data popularity alone is not the ranking. Feedback loops change the value of whatever the model emits.

AI assistance by language

• Tier 1. TypeScript, Python, JavaScript: Massive training data, clear syntax, well-documented patterns. TypeScript adds compile-time verification that catches the structural errors AI produces most often.
• Tier 2. Go, Rust, Java, C#: Strong type systems verify generated code. Go trades simplicity for speed of feedback; Rust trades learning curve for correctness guarantees; Java and C# bring large enterprise corpora.
• Tier 3. C++, Ruby, PHP, Swift: Adequate training data, but syntax complexity or paradigm variation reduces completion quality.
• Tier 4. Perl, Haskell, Lisp dialects: Smaller corpora plus unusual paradigms that current models handle inconsistently.

The distribution is self-reinforcing. Better AI support attracts developers, which produces more training data, which improves support further. TypeScript is deep in that cycle. Rust is entering it as enterprise adoption grows. Languages without either volume or a strong verifier face the inverse.

Most admired languages

• Rust: 72.4%
• Gleam: 70.8%
• Elixir: 66%
• Zig: 64.2%

The language developers most want to keep using is Rust (Stack Overflow Developer Survey 2025). The cohort around it (Gleam, Elixir, Zig) skews toward compiler-caught mistakes rather than runtime discovery, the same property that makes AI-generated code safer to ship.

TypeScript wide, Rust narrow

The stxkxs.io split is TypeScript for the web and infrastructure layer, Rust for security-critical Lambdas. A year into that split under heavy AI assistance, the bet still holds.

TypeScript for the web stack

React, CDK, API handlers: one language across the web-facing stack. CDK construct types flow into API handler types into frontend client types. A change to a response shape triggers errors everywhere that shape is consumed. The agent can generate the change; the type checker verifies every downstream consumer. That cross-layer check does not exist across language boundaries.

// Same language, same types, across every layer.
// AI generates the handler — TypeScript verifies it matches
// the DynamoDB schema AND the frontend contract.

import { json, error } from '../lib/response'
import { getClient, getTableName } from '../lib/dynamo'
import { PutCommand } from '@aws-sdk/lib-dynamodb'

export default async function handler(event: APIGatewayProxyEventV2) {
  const origin = event.headers?.origin ?? ''

  if (!isAllowedOrigin(origin)) {
    return error(403, 'Forbidden')
  }

  const body = JSON.parse(event.body ?? '{}')

  await getClient().send(new PutCommand({
    TableName: getTableName(),
    Item: {
      pk: `PAGE#${body.path}`,
      sk: `TS#${Date.now()}`,
      path: body.path,
      referrer: body.referrer ?? 'direct',
      timestamp: new Date().toISOString(),
    },
  }))

  return json({ tracked: true })
}

Rust where correctness matters

The email forwarder and Cognito handler are small, critical, rarely-changed functions. The compiler enforces invariants that would otherwise need re-checking on every deploy, with fast cold starts and low memory on Graviton2 as a bonus. AI writes these handlers capably; when ownership is wrong, the build rejects the change before it reaches a diff. They could run on TypeScript. The Node.js Lambda runtime is fine. Fine is different from "no null pointer, no data race, no use-after-free." For email content and auth tokens, the guarantee is worth more than the convention.

TIP: The practical stack decision
Pick TypeScript when AI should move fast across a full web stack with type safety at every boundary. Add Rust for the few functions where correctness guarantees matter more than ramp time. Skip Rust when the math does not clear: the first months of borrow-checker fluency are still hard, the hiring pool is smaller, and onboarding costs real time. If no workload needs memory safety or hard performance, that cost buys nothing. Go is a strong alternative with a gentler curve and solid AI support.

Dynamic languages still win

Typed languages are not the right answer for every workload. Python still owns AI/ML research and most model training pipelines because the ecosystem is the product: notebooks, PyTorch, the scientific stack. Shipping a typed rewrite to get better agent completions is the wrong trade when the library surface only exists in Python. Short-lived scripts, data exploration, and one-off glue code also stay cheaper in dynamic languages; the feedback loop is the REPL and the test, not tsc, and the blast radius is small.

JavaScript remains the runtime of the browser. TypeScript is a compile-time layer on top of that fact, not a replacement for it. Teams with deep Python or Ruby production skill and mature test suites can stay dynamic and still use AI productively; the cost is that more of the verification load sits on tests and review instead of the compiler. Dynamic languages are not dying. When AI raises generation volume, the languages that catch mistakes before runtime simply become worth more, and popularity measured without that filter is an incomplete number.

Compilers beat popularity

For decades, language choice optimized for human writing speed. Python and JavaScript won that race. Types were overhead; compile steps were friction. As AI-generated code becomes a larger share of what ships, verification reliability matters more than how fast a human types. Annotations are free when the agent writes them. Compile-time checks are instant. The overhead that made dynamic languages attractive for greenfield work shrank; the safety gap between "caught at compile time" and "caught in production" did not.

TIOBE Index language rankings

• Python: 23.28%
• C++: 10.29%
• Java: 10.15%
• C: 8.86%
• C#: 4.45%
• JavaScript: 4.2%
• Go: 2.61%
• Rust: 1.16%

Domains are converging on typed defaults without erasing the exceptions above. Web development moved from JavaScript to TypeScript. Systems work is moving from C and C++ toward Rust where memory safety matters. Cloud-native services often pick Go for simplicity plus a fast compiler. Python stays #1 on TIOBE while adding mypy and Pyright in serious codebases. The evaluation question for new projects shifted from "which language minimizes writing friction?" to "which language maximizes the reliability of code AI helps generate?" Languages that score on type systems, informative compiler feedback, and adequate training data (TypeScript, Rust, Go) are the primary beneficiaries.

The compiler became the product.

Follow the compilers, not the hype cycles. TypeScript for anything that touches the web. Rust where correctness is the feature. Go for services that want simplicity and fast feedback. Python for AI/ML, with type hints on. The learning curves still exist; AI compressed them enough that the safety tradeoff is no longer close for production systems where agents write a large share of the diff. Pick the language that catches AI mistakes at compile time.

Resources & Further Reading

• GitHub Octoverse 2025: https://github.blog/news-insights/octoverse/ - TypeScript overtakes Python and JavaScript as the most-used language on GitHub
• GitHub Blog. Why AI is pushing developers toward typed languages: https://github.blog/ai-and-ml/llms/why-ai-is-pushing-developers-toward-typed-languages/ - Analysis of the feedback loop between AI code generation and type systems
• Stack Overflow Developer Survey 2025: https://survey.stackoverflow.co/2025/technology - Rust maintains most admired status (72%) for tenth consecutive year
• METR. Measuring the Impact of AI on Developer Productivity: https://metr.org/ - Randomized controlled trial showing 19% slowdown for experienced developers with AI assistance
• JetBrains State of Developer Ecosystem 2025: https://blog.jetbrains.com/research/2025/10/state-of-developer-ecosystem-2025/ - Comprehensive survey of developer tool usage, language trends, and AI adoption
• TIOBE Index: https://www.tiobe.com/tiobe-index/ - Python reached a historic 26.98% rating in July 2025; Rust hit an all-time high of #13 in January 2026
• corrode Rust Consulting. Flattening Rust's Learning Curve: https://corrode.dev/blog/flattening-rusts-learning-curve/ - How AI tools are compressing the time to Rust proficiency

In December 2025, Anthropic donated Model Context Protocol (MCP) to the Linux Foundation's Agentic AI Foundation. Anthropic, Block, and OpenAI co-founded the foundation. AWS, Google, Microsoft, Bloomberg, and Cloudflare joined as platinum members. The donation is a governance move: tool integration as a shared wire format (auth, discovery, transport) owned by no single model vendor.

INFO: Adoption beat standardization
Google donated Kubernetes to the CNCF in 2015 after years of production use. SOAP, CORBA, and a graveyard of foundation projects had vendor logos too and still failed. MCP is closer to the Kubernetes path: 97 million monthly SDK downloads and 10,000+ active public servers before the foundation donation ([Anthropic / MCP foundation materials, Dec 2025](https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation)). Standardization follows ships-in-production adoption.

Fragmentation multiplies cost

AI tooling still looks like cloud infrastructure circa 2014. Claude has tool use, OpenAI has function calling, Gemini has its own function-calling shape, Copilot has agent APIs. A non-trivial integration against one of those surfaces does not move to another without a rewrite.

That rewrite tax becomes accidental lock-in: teams stay on a model when the tools will not port, even after a bake-off would prefer another model on quality alone. Tool authors pick one host ecosystem, so capability splits across Claude-shaped and OpenAI-shaped catalogs. Prompt templates, server configs, and host wiring stay stuck to the original stack when a new model is evaluated. Portability dies at the integration layer even when the models are interchangeable for the task.

Three tiers, one wire

MCP is a three-tier split. The host does not hardcode every tool. The server does not care which model is reasoning. Discovery and invocation sit on the wire between them.

• Host: user-facing application (VS Code, Claude Desktop, ChatGPT, Cursor). Owns UI, session state, and client instantiation.
• Client: LLM-powered decision layer. Receives requests, chooses tools, orchestrates invocations. Agency lives here.
• Server: integration surface. A GitHub server exposes repo ops, a Postgres server exposes queries, a Slack server exposes messaging. Each publishes a schema for its tools and resources.

Capability discovery is dynamic. Point a host at a new server and connected clients learn the tool list at runtime. No per-tool redeploy of every client, no parallel schema registry outside the protocol.

// MCP Server for GitHub integration
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';

const server = new Server({
  name: 'github-mcp-server',
  version: '1.0.0',
}, {
  capabilities: {
    tools: {},
  },
});

server.setRequestHandler(ListToolsRequestSchema, async () => ({
  tools: [
    {
      name: 'create_pull_request',
      description: 'Create a new pull request on GitHub',
      inputSchema: {
        type: 'object',
        properties: {
          repo: { type: 'string', description: 'Repository name (owner/repo)' },
          title: { type: 'string', description: 'PR title' },
          body: { type: 'string', description: 'PR description' },
          head: { type: 'string', description: 'Branch to merge from' },
          base: { type: 'string', description: 'Branch to merge into' },
        },
        required: ['repo', 'title', 'head', 'base'],
      },
    },
  ],
}));

const transport = new StdioServerTransport();
await server.connect(transport);

That server is usable from any MCP-compatible host once the host speaks the protocol. Write the integration once; swap the model or the IDE without rewriting the GitHub surface. That is the whole value proposition in under 50 lines.

JSON-RPC and two transports

The wire is JSON-RPC 2.0. Language-agnostic, transport-flexible. Production transports are stdio for local tools and Streamable HTTP for remote or cloud deployments. Standalone SSE as a transport was deprecated in the March 2025 spec revision; Streamable HTTP may still use SSE optionally for response streaming. Plan new remote servers on Streamable HTTP, not on SSE-only clients and servers that will age out of the ecosystem.

The protocol defines resources (readable structured data), prompts (named templates), tools (callable actions), and sampling (servers requesting LLM completions for chaining). Those four are the MCP capability surface.

Tool Search and Programmatic Tool Calling are Anthropic Claude API features (beta `advanced-tool-use-2025-11-20`, November 2025). They are not additions to the MCP 2025-11-25 specification. The 2025-11-25 revision added OIDC discovery, icons metadata, sampling tool-calling, and experimental Tasks. When thousands of tools are available, loading every definition into context is wasteful; Anthropic's Tool Search beta lets Claude clients query for relevant tools on demand. That behavior lives in the model product. It is outside the shared MCP wire format. Multi-host architectures that assume Tool Search is portable MCP will break on every non-Claude host.

Adoption preceded the foundation

Download counts are easy to inflate. The useful signal for MCP is who integrated, how fast, and whether competitors treated the protocol as obligatory.

• Monthly SDK Downloads: 97M+ — Python and TypeScript implementations combined ([MCP one-year anniversary](https://blog.modelcontextprotocol.io/posts/2025-11-25-first-mcp-anniversary/), Nov 2025)
• Active MCP Servers: 10,000+ — Public servers across GitHub, integrations, and enterprise tools (same anniversary post)
• First-Class Clients: 6 Named — ChatGPT, Claude, Cursor, VS Code, Gemini, Copilot (Anthropic donation post, Dec 2025)
• Competitor adoption lag: ~4 months — OpenAI announced MCP support in March 2025, four months after Anthropic's November 2024 release

OpenAI announced MCP support in March 2025. In its November 2025 anniversary post, OpenAI's Srinivas Narayanan wrote that MCP was "a key part of how we build at OpenAI, integrated across ChatGPT and our developer platform." Google wired MCP into Gemini tool use. Microsoft put it in Copilot and Semantic Kernel. AWS, Google Cloud, and Azure offered managed MCP server hosting. Block, a foundation co-founder, built its goose agent framework on MCP for internal tool access. When direct competitors adopt the same wire format and say so publicly, competition moves to model quality and operations. Proprietary tool schemas stop being a durable moat.

Vendor cooperation looks odd until the moat is named correctly. Frontier labs compete on model quality and product surface. Exclusive tool catalogs shrink total available integrations and tax every enterprise buyer who refuses single-vendor function-calling APIs for critical workflows. Shared tools grow the pie. Lock-in, where it remains, sits on model quality and product UX: the surface vendors already want to defend.

A2A covers horizontal handoffs

MCP is vertical: agent to tool. Once agents can query databases and call APIs reliably, coordination between agents is a separate problem. Google released Agent-to-Agent (A2A) in April 2025 as an explicit complement: a research agent hands findings to an analysis agent, an orchestrator delegates to specialists, a support agent escalates with structured context. MCP leaves that path alone. A2A leaves tool schemas alone.

• MCP: agent creates a GitHub PR, queries Postgres, posts to Slack. Vertical, agent-to-capability.
• A2A: research agent shares findings with analysis agent; orchestrator delegates; support agent escalates. Horizontal, agent-to-agent.

Google contributed A2A to the Linux Foundation as its own Agent2Agent Protocol Project in June 2025, six months before the Agentic AI Foundation formed around MCP, goose, and AGENTS.md. Both protocols sit under the Linux Foundation umbrella with separate projects and separate governance. Complementarity is by design; stewardship is not shared. In a multi-stage automation loop, MCP is how a diagnosis agent talks to PagerDuty or Datadog; A2A is how it hands structured root-cause context to a remediation agent (the stage-connectivity view is covered in [The Agentic DevOps Loop](https://stxkxs.io/blog/agentic-devops-loop)).

A concrete handoff looks like this: a customer-service agent receives an order inquiry and requests status from an order-tracking agent over A2A. The order-tracking agent uses MCP against a Postgres server. On delay, it requests a delivery estimate from a logistics agent over A2A; that agent uses MCP against a shipping API. Each specialist keeps its own tool surface. No agent needs every credential. The horizontal protocol carries task and context; the vertical protocol carries side effects.

Orchestration frameworks (LangGraph, CrewAI, AutoGen and peers) sit above both protocols. They own workflow graphs, role assignment, and retries. Frameworks that speak open protocols keep model and tool choice portable. Frameworks that re-encode proprietary tool surfaces reintroduce the fragmentation tax one layer up. LangGraph, CrewAI, and AutoGen are integrating MCP. Dependency direction is protocol first, product orchestration second.

At cluster scale, the same split shows up as runtime infrastructure. Kubernetes-native agent controllers and gateways mediate MCP and A2A the way a service mesh mediates HTTP: discovery, auth policy, and route-level budgets land at the gateway rather than inside every agent process (see [Kubernetes as an Agent OS](https://stxkxs.io/blog/kubernetes-agent-os)). The mesh and the gateway are product categories. MCP and A2A remain the wire formats those products implement.

INFO: HTTP parallel, one level up
HTTP became the standard for service-to-service calls. Service meshes added observability, routing, and resilience while individual services still talked to databases and queues. The same layering is forming for agents: A2A as the inter-agent call path, meshes and gateways as the operational plane, MCP tools as the backing capabilities.

Gateways capture production value

MCP and A2A are free. Production value accrues where free protocols leave hard work: auth, rate limits, observability, policy, and cost control. Kubernetes is free; Red Hat sold to IBM for $34 billion largely on OpenShift. The same shape applies here. Gateways and proxies sit between agents and servers. Domain-specific servers package compliance, finance, or internal APIs as reusable capabilities. Orchestration control planes and agent observability products sell the operational plane the specs omit.

Heterogeneous model routing is the pattern that makes the wire format earn its keep. A frontier model plans. Smaller or cheaper models execute routine tool steps. Orchestrators delegate over A2A; specialists hit tools over MCP. Without a shared protocol, every model-tool pair needs custom glue. With MCP and A2A, the plan-and-execute split is configuration and policy rather than a combinatorial rewrite.

• Agents in production (Q3 2025): 42% — Organizations with AI agents deployed in production ([KPMG AI Quarterly Pulse, Q3 2025](https://kpmg.com/us/en/media/news/q3-ai-pulse.html))
• Agents in production (Q4 2025): 26% — Same survey series; Q4 2025 figure, report published January 2026 ([KPMG AI Quarterly Pulse, Q4 2025](https://kpmg.com/us/en/media/news/q4-ai-pulse.html))
• Piloting agents (Q3 2025): 55% — Organizations piloting AI agents ([KPMG AI Quarterly Pulse, Q3 2025](https://kpmg.com/us/en/media/news/q3-ai-pulse.html))
• Gartner cancel forecast: >40% by 2027 — Gartner prediction: over 40% of agentic AI projects canceled by end of 2027 on cost, scaling complexity, or risk

The Q3-to-Q4 drop in KPMG's production rate (42% to 26%) is a reminder to date the figures carefully and avoid treating a single survey wave as a permanent adoption curve. Pilots are common; durable production is harder. Protocol maturity and operational maturity move on different clocks.

Auth still lags the wire

The Agentic AI Foundation is new. Spec change process, prioritization, and neutrality under unequal contribution volume are open questions. Kubernetes worked those issues out slowly inside the CNCF. MCP needs the same clarity while adoption is still accelerating.

Scale of discovery remains open after the 2025-11-25 additions. OIDC discovery, icons, sampling tool-calling, and experimental Tasks help hosting and metadata. They leave clients without a standard way to search across thousands of tools on hundreds of servers. Anthropic's Tool Search beta answers that for Claude API consumers outside the shared spec. Package registries and search indexes are the historical precedent. The foundation still has to standardize an equivalent if multi-host discovery is to stay portable.

Security is the harder gap. MCP servers can delete repos, query production databases, and send mail. The March 2025 revision added OAuth 2.1 at the transport and server level; 2025-11-25 added OIDC discovery on top. There is still no built-in tool-level permission model. SEP-1880 proposed per-tool OAuth scopes and was closed as not planned, so RBAC stays with the implementer. Any server can declare any tool, and the host trusts the advertisement unless something outside the core protocol constrains it.

Production practice today: limit MCP to trusted, internally maintained servers; put fine-grained auth at a gateway; treat third-party server install as a security change with least-privilege credentials (the same posture [OpenClaw](https://stxkxs.io/blog/openclaw-self-hosted-ai-agents) applies when wiring MCP into a self-hosted agent host). Server signing, audit trails for every tool call, and sandboxing for untrusted servers are still ecosystem work rather than settled spec.

Debugging is still early. Failures through JSON-RPC often surface as opaque errors. Stdio traces are raw JSON without a standardized logging profile. Budget debugging infrastructure alongside the first production servers. "Works in a demo" and "traceable at 2am" are different bars.

Skip MCP when single-model

MCP has adoption cost. Shipping against one model with a handful of stable tools is often cheaper with that model's native function calling than with a server, client, and host wired together for tools that will not change and will never leave that provider.

If third-party MCP server code cannot be audited before it receives API keys or database access, wait until the trust model or the gateway layer is ready. MCP pays when the product of models times tools is large enough that writing the integration once amortizes: more than one model in production or evaluation, or more than a small fixed tool set. Below that threshold, the protocol is machinery without a portability return. The wire format can be solid while operational maturity is not; protocol adoption does not invent production discipline.

Protocol work, not product

Treat MCP as infrastructure protocol work: transport choice, OAuth and identity at the server boundary, discovery at scale, audit of tool calls, and gateways that enforce policy the core spec leaves out. "MCP support" on a product page does not substitute for those systems. The commercial layer around free protocols (gateways, domain servers, observability) is real for the same reason OpenShift grew around Kubernetes: production needs what the wire format deliberately omits.

Build servers for tools that will outlive a single model contract. Prefer Streamable HTTP for remote deployments. Keep Tool Search and other host-vendor betas out of the portable contract. Pair MCP with A2A only where agent-to-agent handoffs exist. The Kubernetes moment for AI tooling is this protocol layer becoming boring infrastructure while proprietary agent frameworks compete above it.

Key Point: MCP under the Linux Foundation is the wire format for tool integration. Invest in auth, discovery, and transport. Leave model competition to the models.

Resources & Further Reading

• MCP Official Specification: https://modelcontextprotocol.io/specification/2025-11-25 - November 2025 (2025-11-25) MCP specification: experimental tasks, sampling tool calls, URL elicitation, OIDC discovery
• MCP GitHub Organization: https://github.com/modelcontextprotocol - SDKs, reference implementations, and server examples
• Anthropic - Donating MCP to Agentic AI Foundation: https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation - December 2025 donation announcement
• Linux Foundation - Agentic AI Foundation Launch: https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation - Foundation charter and founding members
• MCP One Year Anniversary: https://blog.modelcontextprotocol.io/posts/2025-11-25-first-mcp-anniversary/ - Adoption numbers one year after launch
• Google A2A Protocol: https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/ - Agent-to-Agent protocol announcement
• OpenAI - Joining Agentic AI Foundation: https://openai.com/index/agentic-ai-foundation/ - OpenAI on joining the foundation
• GitHub Blog - MCP Joins Linux Foundation: https://github.blog/open-source/maintainers/mcp-joins-the-linux-foundation-what-this-means-for-developers-building-the-next-era-of-ai-tools-and-agents/ - What the donation means for developers
• The Agentic DevOps Loop: https://stxkxs.io/blog/agentic-devops-loop - MCP and A2A as stage connectivity with trust still open
• Kubernetes as an Agent OS: https://stxkxs.io/blog/kubernetes-agent-os - Gateways mediating MCP and A2A in-cluster
• OpenClaw Self-Hosted Agents: https://stxkxs.io/blog/openclaw-self-hosted-ai-agents - Treating MCP wiring as a security change
• KPMG AI Quarterly Pulse Q4 2025: https://kpmg.com/us/en/media/news/q4-ai-pulse.html - Production agent rates (published Jan 2026)

GitHub CLI reached 1.0 in September 2020, giving developers one tool for pull requests, issues, and releases without leaving the terminal. Platform teams rarely have an equivalent for their own infrastructure. A backend engineer deploying a config change runs kubectl for current state, ssh to a bastion for logs, an internal job for the deploy, curl for smoke checks, and a metrics UI to confirm the change landed. Each tool brings its own auth, flags, and mental model.

Flag conventions alone split the stack: AWS-style `--flag=value`, GNU-style `--flag value`, and single-letter shortcuts. Auth is worse. SSO for one tool, cloud credentials for another, a still-live API key in a third. The cost stays invisible until a new hire burns an afternoon just getting credentials to work across the set.

The usual response is another wrapper script. That helps one team for a quarter, then forks. The serious alternative is a unified internal CLI that earns its maintenance cost, or an honest decision that Unix composition already covers the work.

INFO: The hidden cost of sprawl
Context switching across many CLIs in one workflow compounds into hours per week. Navigation and auth dominate the cost; the underlying work is usually short.

Product CLIs set the bar

Platform vendors treat the CLI as a product surface with its own roadmap, docs, and release cadence. Stripe's CLI set the reference pattern for API development tools. Heroku defined the Git-shaped deploy experience later platforms copied. Railway, Vercel, and GitHub CLI ship the same idea: the terminal is a first-class interface. The patterns below come from those tools; all of them install and study cleanly.

Stripe ships local loops

Stripe CLI solved webhook testing without a staging deploy. Before the CLI, validating a payment webhook meant shipping to staging and waiting on a real event. `stripe listen --forward-to localhost:3000/webhooks` plus `stripe trigger payment_intent.succeeded` exercises the full payment path on a laptop. Commands follow `stripe [resource] [action]`. Run `stripe` alone for structured help; `stripe completion` installs shell autocomplete for bash, zsh, or fish.

stripe login
stripe listen --forward-to localhost:3000/webhooks
# other terminal
stripe trigger payment_intent.succeeded
stripe events list --limit 10

Heroku made deploys Git

`heroku create` adds a Git remote. Deploy with `git push heroku main`. Tail logs with `heroku logs --tail`. The CLI extended a model developers already had, so deploy did not require a second mental model. That reduction in ceremony is the bar later PaaS CLIs chased.

Heroku also pioneered CLI plugins: third parties shipped commands without forking the core. GitHub CLI continues that with `gh` extensions. Shopify CLI and Salesforce CLI push the same idea further on oclif, the framework Heroku popularized. Plugin economics showed up in product CLIs long before most internal platform teams treated extensibility as a design constraint.

Zero-config still has defaults

Railway and Vercel push further toward zero-config deploy. `railway init` links a project; `railway up` deploys. Nixpacks (and Railpack after it) detect the framework at build time. Vercel does the same for frontend stacks: `vercel` in a Next.js tree just works. Interactive prompts carry smart defaults. Need a database on Railway? Pick a type from a menu; connection strings land as env vars. The developer stays in the terminal for the whole loop.

None of these tools is an internal platform CLI. They still set the UX bar customers already accept from vendors. An in-house tool that still requires a wiki page of flags is competing with that memory, whether the platform team acknowledges it or not.

Context inference is mandatory

Stripe, Heroku, GitHub, Railway, and Vercel share the traits that separate a useful internal CLI from a thin wrapper. The trait that pays for the rest is context: detect Git branch, working directory, recent operations, and linked project so the default action matches what the engineer meant.

Defaults should match intent

Run `railway up` in a directory already linked to a Railway project and it deploys that project. No project ID, environment, or service name required. Context lives in local files (`.railway.json`, `.vercel.json`), environment variables, or a config dir under `~/.config`. Default to the right thing most of the time; keep explicit flags for the rest.

An internal CLI should do the same with repo metadata. `devctl deploy` in the payment-service repo on `main` should deploy payment-service to prod without a flag picnic. Branch conventions (main → prod, develop → staging, feature → dev) are enough to start. Service name can come from package.json, a Chart.yaml, or a small project manifest the platform owns. Every keystroke the tool can eliminate is one the engineer does not have to remember under load.

Context inference fails open when it is wrong. If the repo maps to multiple services, prompt. If the branch convention is ambiguous, require `--env`. Silent wrong defaults destroy trust faster than missing defaults; engineers will stop using the short form and go back to long flag lists.

Prompt when args are missing

Great CLIs do not require memorized syntax. When a value is missing or a reference is ambiguous, drop into a type-to-filter picker instead of printing a usage error. GitHub CLI does this inside commands like `gh pr merge`. Running `gh` with no arguments prints static help; prompting is scoped to the moment a command needs input. Libraries such as fzf, inquirer, and Charm's bubbletea cover the UI pieces.

One command grammar only

Stripe uses `stripe [resource] [action]`: `stripe customers list`, `stripe invoices create`. Once the pattern is learned, unguessed commands become guessable. Docker uses `docker [object] [action]` (`docker container run`). Kubernetes uses `kubectl [verb] [object]` (`kubectl get pods`). Pick one grammar and refuse exceptions. Inconsistent surfaces (`list-services` next to `db:create` next to `get-logs --service=`) tax every new command.

Noun-verb versus verb-noun is a team choice. Mixing both inside one binary is the failure. Document the chosen grammar in the help root, mirror it in autocomplete, and reject PRs that invent a third shape for convenience.

Errors should teach next steps

Weak CLIs print status codes and stack traces. Strong ones name the failure, the cause, and the fix. Cargo surfaces Rustc diagnostics with problem, cause, suggestion, and doc links. Stripe does the same on auth failure: invalid API key, where to find keys in the dashboard, link to auth docs. Catch the common cases (missing credentials, wrong environment, network failure) with specific guidance and color-coded severity.

# Bad
Error: 401 Unauthorized

# Good
Authentication failed: API key is invalid or expired.
Find keys: Dashboard → Developers → API keys
Docs: https://docs.stripe.com/keys
Then: stripe login

A useful CLI removes the round-trip between the engineer and the docs by inferring context and choosing the right default.

Dual mode is non-negotiable

Interactive mode earns adoption for humans. Structured output earns a place in CI. A unified CLI that only prompts and pretty-prints loses the Unix argument on contact. Support both: interactive when arguments are missing, non-interactive when every argument is present. Every interactive command needs a flag equivalent.

GitHub CLI exposes `--json` with field selection so scripts pull only what they need. Stripe CLI and kubectl both treat machine-readable output as a first-class mode, not an afterthought bolted on for one subcommand. That dual mode is the convention worth copying for any internal platform CLI. Without it, the tool is a dashboard with a terminal skin, and the strongest case against unified CLIs (composability) stands unanswered.

# Interactive: discovery and common workflows
devctl deploy
# → service picker, environment picker, confirm

# Non-interactive: CI and scripts
devctl deploy payment-service --env prod --skip-confirm --json

# Hybrid: some args fixed, rest prompted
devctl deploy payment-service
# → only asks for environment

Progress feedback matters for the human path. Operations longer than about a second need a spinner or stream; Vercel shows stage messages during deploy, Railway streams build logs live. Sub-second operations should acknowledge completion immediately. The dual-mode rule still holds: progress chrome stays out of `--json` output so pipelines do not have to strip spinners from logs.

Exit codes are part of structured output. Non-zero on failure, zero on success, and stable meanings across commands so a shell script can branch without parsing prose. Pretty tables belong on the human path; machines get objects and status.

Plugins justify the cost

For a platform team supporting dozens of engineers across multiple services, a unified CLI can repay its build cost. It only stays repaid if teams can extend it without queuing behind the platform group. Context inference and structured output get a first version used. A plugin story keeps it from becoming the bottleneck it was meant to remove.

Pick one framework

Commit to a foundation and stop shopping. Go: Cobra (kubectl, GitHub CLI). Python: Click or Typer. Node.js: Commander or oclif (Heroku, Salesforce). Rust: clap. Argument parsing, subcommands, and help text are solved problems. Spend the engineering time on workflows and extension points. Reimplementing argv is a trap that delays the product surface.

Auth belongs in the foundation, not in each command. One login that covers the internal APIs the CLI calls (SSO device flow, short-lived tokens, or whatever the org already uses) is usually the first feature people notice. Stripe's browser pairing and `gh auth login` both treat credentials as a product problem. Internal CLIs that leave each engineer to wire kubeconfigs, cloud profiles, and vault tokens by hand recreate the sprawl tax under a single binary name.

Extensions without a bottleneck

Different teams need different commands. Data wants job submission. ML wants model deploy. Infra wants cluster operations. A plugin registry that lazy-loads team-owned packages lets those groups ship on their own schedule. Shopify's CLI versions plugins as npm packages resolved at runtime. Core stays small and versioned by the platform team; team commands version independently.

The hard questions are operational. How do teams publish a command without a platform PR? How do plugins version separately from the core binary? How do you isolate dependency conflicts when two plugins pin different major versions of the same library? A runtime-resolved registry answers the first two; isolation (separate processes, language-level module boundaries, or package managers that allow nested deps) answers the third. Without that split, every new command is a platform PR, and release freezes reappear inside the tool that was supposed to collapse them.

If the org cannot staff a plugin model, keep separate tools and shared conventions instead of forcing a single binary. A monolithic internal CLI owned by one team is just a new form of ticket queue with nicer help text. Plugins are what turn unification from a centralization project into a platform.

Ship the pain-killers first

Start with the workflows engineers actually hit every day, usually deploy, logs, and rollback. A wide command surface with few used commands only makes help noisy. Wrap kubectl, docker, and the cloud CLI with sensible defaults for your environments; expand after the core three feel inevitable. The first release should make one painful day shorter, not catalogue every operation the platform team can imagine.

WARNING: The configuration trap
Avoid long setup before first success. Stripe CLI clears this bar: `stripe login` issues a pairing code, opens a browser to confirm, and stores a generated key under `~/.config/stripe/config.toml`. No key paste, no hand-edited config. First useful command under a minute.

Treat the CLI as a product. It needs docs, versioning, release notes, and a feedback loop. Track which commands run and where people drop off. A CLI that ships once and never measures usage decays into folklore scripts with a binary entrypoint. Version the CLI the way you version APIs: breaking changes need migration notes, and silent flag renames break CI overnight.

Composable tools often win

The Unix position still holds for a large share of real work: small tools, each good at one job, composed with pipes. A unified CLI is convenient until a workflow needs something the wrapper never modeled. Then the engineer is worse off than with raw tools and a pipe. That failure mode is common enough that it should be the default objection in design review.

Below roughly twenty engineers, building a platform CLI is usually a poor trade. A handful of scripts plus kubectl, docker, and the cloud CLI cover the day. The wrapper becomes another thing to learn, document, and keep current while everyone already knows the three commands they run. Build cost is real; so is the ongoing tax of tracking every upstream flag change. Shared aliases and a short runbook often beat a binary at that size.

A single-tool shop has nothing to unify. If daily work is almost entirely kubectl, a second CLI over kubectl is pure indirection, and the wrapper lags kubectl's own flags and releases. The same applies to teams that live inside Terraform or one cloud provider CLI. Wrapping a tool the team already masters adds a layer that only the platform team fully understands.

CI pipelines, data jobs, and any flow where one command's stdout feeds the next want parseable text and exit codes. Menus and pretty tables fight that grain. If structured output and pipeability are the primary requirement and interactive discovery is rare, separate single-purpose tools remain the better default. Even a dual-mode CLI is the wrong investment when nobody needs the interactive half.

Unix composition also wins when the organization values thin, reviewable scripts over a single binary's release process. A shell pipeline in a checked-in Makefile is reviewable by anyone. A plugin-loaded CLI is reviewable by fewer people and harder to bisect when a deploy path changes.

A unified CLI earns its keep only when context inference, dual-mode output, and a plugin path are all real commitments rather than backlog items. Miss the first and people still memorize flags. Miss the second and CI stays on raw tools. Miss the third and the platform team becomes a release gate. Any one miss is a reason to stay with composable tools.

Earn the abstraction

Vercel and Railway cover full deploy loops from the terminal. Stripe did not need a CLI to sell payments; shipping one met developers where they already worked and raised the cost of competing on dashboard-only surfaces. Internal platforms face the same bar their vendors already cleared.

A unified internal CLI is justified when it infers context, exposes structured output beside interactive flows, and lets teams extend commands without blocking each other. Miss any of those three and composable single-purpose tools remain the cheaper, more honest answer.

The foundation libraries are solved. Cobra, Click, oclif, clap, and the TUI kits already exist. The product decision is whether the org will fund the three properties that make one binary worth maintaining. Fund them and the CLI absorbs sprawl. Skip them and the wrapper becomes another tool in the pile it was meant to replace.

Resources & Further Reading

• Stripe CLI: https://docs.stripe.com/stripe-cli - Local webhook forwarding and resource/action command grammar
• GitHub CLI: https://cli.github.com/ - Open source reference for prompts, extensions, and --json
• Heroku CLI: https://github.com/heroku/cli - Plugin-based CLI architecture on oclif lineage
• Cobra (Go): https://github.com/spf13/cobra - Framework behind kubectl and GitHub CLI
• Click (Python): https://click.palletsprojects.com/ - Subcommands and options without ceremony
• oclif (Node.js): https://oclif.io/ - Framework used by Heroku and Salesforce CLIs
• CLI Guidelines: https://clig.dev/ - Cross-ecosystem practices for flags, help, and output
• Charm: https://charm.land/ - Terminal UI components (bubbletea and related)

Surge pricing is a hard real-time analytics problem: demand across thousands of city zones, recomputed every few seconds, to balance supply with rider requests. The system has to answer high-cardinality questions under a tight freshness budget: how many ride requests hit a zone in the last two minutes, what is the supply-to-demand ratio across the city, which neighborhoods are trending and should pull drivers.

A warehouse on Hadoop and Hive can return accurate answers. It returns them in tens of seconds to minutes. Demand moves inside that window, so the calculation finishes on market conditions that no longer exist. Ride-hailing companies built dedicated real-time analytics for this reason. Uber's AresDB and later adoption of Apache Pinot are documented examples of purpose-built infrastructure replacing a warehouse that could not keep up.

The same wall shows up anywhere an operational decision is gated on an analytical query inside a user-facing latency budget: pricing engines, fraud review, personalization. Batch warehouses were not designed for that combination of volume, concurrency, and sub-second response. When the product decision waits on the query, multi-second OLAP is a product bug with a dashboard-shaped symptom.

INFO: OLTP, OLAP, real-time OLAP
OLTP databases (PostgreSQL, MySQL) handle transactional reads and writes quickly and struggle with large analytical aggregations. Classic OLAP (Redshift, BigQuery) handles analytical scans well, often in seconds and sometimes tens of seconds on large or complex queries. Real-time OLAP (Druid, ClickHouse, Pinot) targets sub-second analytical queries on continuously arriving event data.

Batch misses the window

Batch analytics starts with overnight ETL: extract from production databases, load Redshift or BigQuery off-peak, query yesterday's snapshot in the morning. That matched decisions that ran on daily or weekly cycles. When competitors react to the same signals within the hour, yesterday's snapshot is already stale.

Running the batch job hourly, then every 15 minutes, multiplies fixed per-run overhead (scheduling, extraction coordination, load sequencing) without shrinking it. Even optimized ETL on dedicated infrastructure rarely lands reliably under 5–10 minutes of end-to-end latency. The micro-batch treadmill burns ops time and still fails a sub-second product SLO.

Streaming inverted the flow. Applications emit events to Kafka as they occur. Processors like Flink maintain derived state and trigger actions as events arrive. Freshness moved from hours to milliseconds for the paths that were pre-modeled as continuous jobs. That solved whether the data is current for those pipelines. It left open whether an analyst or a pricing service can ask an unanticipated group-by against recent history and get an answer before the UI times out.

Stream processors excel at forward-only, pre-defined aggregations and alerts. Questions like hourly signups by country over 30 days, or p95 API latency by endpoint over arbitrary windows, need random access to history plus flexible group-bys. Flink can maintain specific windows; it is a poor general ad-hoc query engine for business users exploring dimensions that were not declared when the job was written.

Dumping the stream into Redshift or BigQuery keeps query latency in the multi-second range and fights batch-oriented write paths under continuous load. High-frequency inserts and concurrent dashboard fan-out are the wrong stress profile for systems optimized for large scans of largely static tables. The missing piece is a store built for analytical queries on time-series event streams.

Druid fits streaming first

Apache Druid came out of Metamarkets (later acquired by Snap Inc.) for sub-second queries on billions of advertising impression events while data was still arriving. The design choices map to that job: columnar segments for analytical scans, time partitioning with retention policies, optional pre-aggregation at ingest, horizontal scale for both ingestion and queries, and native Kafka (and Kafka-compatible) ingestion with supervisor-based exactly-once semantics.

Netflix's engineering write-up ("How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience," Netflix Tech Blog, March 2020) describes reading playback and device events from Kafka to monitor streaming quality and member experience, with metrics such as error rates and engagement becoming queryable within seconds of the stream. That is the architecture Druid was built for: Kafka in, sub-second OLAP out.

Airbnb used a similar path for pricing recommendations, search ranking, and fraud detection. By 2018 their Druid cluster was ingesting about 10TB of data daily across hundreds of sources (Airbnb Engineering). PayPal appears on Druid's Powered By page as a production user. The pattern is consistent: high-volume event streams, operational decisions or dashboards that cannot wait for batch, and a team willing to run a specialized OLAP cluster.

A later Netflix Tech Blog post on interval-aware caching for Druid (April 2026) describes growth past the 2020 architecture note: over 10 trillion rows and ingest rates up to about 15 million events per second, with dashboards, canary analysis, and live-event monitoring driving repetitive query load. The category is stable even as the numbers move: continuous events, concurrent analytical access, latency that cannot wait for a warehouse batch.

Druid is one system in a class that also includes Pinot and ClickHouse at neighboring design points. When query latency and concurrency on a continuous event stream force real-time OLAP, a Kafka-plus-Druid-class path is the right category. Batch warehouses remain the wrong category for that SLO, regardless of brand.

Those production write-ups share a workload shape: continuous event arrival, analytical aggregations under concurrent load, and a freshness budget measured in seconds rather than hours. If that shape is missing, the citations are interesting history rather than a procurement argument.

ClickHouse wins elsewhere

ClickHouse, Druid, and Apache Pinot dominate real-time OLAP discussions, with different defaults. Netflix and Airbnb run Druid for streaming-first analytics. Cloudflare and Lyft run ClickHouse: Cloudflare for high-throughput analytics on request-scale data, Lyft for ride analytics and cost modeling. LinkedIn built Pinot for user-facing analytics and runs it at very large event scale.

Streaming-first, Kafka-native workloads still favor Druid's supervisor model. Batch-loaded analytics, log-style workloads, and small teams usually favor ClickHouse: fewer process roles, simpler ops, stronger day-to-day SQL ergonomics. ClickHouse has closed much of the ingestion gap with its Kafka table engine and ClickHouse Cloud ClickPipes for managed Kafka and Kinesis. The decision is mostly about operational surface for the streaming path, once both systems can read Kafka at all.

SQL surface area still differs in practice. ClickHouse tends to feel closer to warehouse SQL for ad-hoc analysts. Druid's SQL layer is strong for the event-analytics subset it targets, with native time and rollup concepts that reward a schema designed at ingest. Teams that want "one SQL dialect for every BI question" and mostly batch loads should stop the evaluation early and pick ClickHouse or stay on the warehouse.

EXAMPLE: Confluent chose Druid
Confluent evaluated Druid, Pinot, and ClickHouse for its own cloud analytics platform and chose Druid. One stated reason for skipping ClickHouse at the time was the need for C++ plugins to read custom Kafka formats; Druid's supervisor path fit their stream shapes with less custom glue. Throughput figures for that deployment circulate without a primary source, so they are left out here.

Druid pays for specialized node roles (Coordinator, Overlord, Broker, Historical, MiddleManager), a metadata store, and historically ZooKeeper (newer deployments can use other coordination paths depending on version and operator). That surface is real. Without infrastructure engineers who own it, the cluster becomes the product. With that team, and with continuous ingest plus tight query SLOs, the distribution model is the point of the system.

Pinot sits closest to user-facing product analytics: LinkedIn's origin story, heavy filtering, upsert-shaped serving needs, and very low p99 targets. Teams evaluating the three should start from query shape and ops budget, then pick the default that matches. A generic "real-time analytics" label flattens differences that show up in on-call load within the first production quarter.

• Druid: Kafka/Kinesis-fed operational analytics, sub-second freshness on high-cardinality event dimensions, multi-tenant query patterns that need a mature streaming supervisor
• ClickHouse: batch-loaded OLAP, log and metrics-style workloads, teams that optimize for single-binary or few-node ops over specialized streaming roles
• Pinot: user-facing product analytics (LinkedIn's origin story), very low p99 latency targets, workloads that need upserts and heavy filtering at the serving path

Kafka Druid S3 split

Production layouts converge on the same split. Applications emit domain topics to Kafka (user events, transactions, system metrics) with retention measured in days. Druid supervisors ingest from those topics with tuned task counts, flush intervals, and rollup. Brokers and SQL gateways serve dashboards and APIs. Deep storage on S3 holds immutable segments for retention and recovery.

The split lets each layer scale on its own axis. Ingestion capacity grows with Kafka brokers and Druid MiddleManagers. Query fan-out grows with Historicals. Cold retention grows with object storage. A team can raise query concurrency without replaying the retention policy, and can lengthen retention without resizing the hot query fleet.

{
  "type": "kafka",
  "spec": {
    "dataSchema": {
      "dataSource": "user-events",
      "timestampSpec": { "column": "timestamp", "format": "iso" },
      "dimensionsSpec": {
        "dimensions": ["user_id", "event_type", "country", "platform"]
      },
      "metricsSpec": [
        { "type": "count", "name": "count" },
        { "type": "longSum", "name": "session_duration", "fieldName": "duration" },
        { "type": "hyperUnique", "name": "unique_users", "fieldName": "user_id" }
      ],
      "granularitySpec": {
        "segmentGranularity": "hour",
        "queryGranularity": "minute",
        "rollup": true
      }
    },
    "ioConfig": {
      "topic": "user-events",
      "consumerProperties": { "bootstrap.servers": "kafka:9092" },
      "taskCount": 4,
      "replicas": 2,
      "taskDuration": "PT1H"
    },
    "tuningConfig": {
      "type": "kafka",
      "maxRowsPerSegment": 5000000,
      "maxRowsInMemory": 100000
    }
  }
}

TIP: Rollup buys the latency
With rollup enabled, Druid pre-aggregates at ingest. Billions of raw events collapse into far fewer rows at the chosen query granularity. Queries hit summarized segments instead of replaying every event. Metrics and dimensions must be chosen up front; a dimension omitted at ingest cannot be recovered without reprocessing history.

Schema is sticky for that reason. Over-dimension early when storage cost is acceptable. High-cardinality fields such as raw user_id or session_id expand storage and slow filters; keep them only when product queries need them, and put true entity lookups in an OLTP store. Most operational dashboards over-query recent windows, so tiered Historicals (SSD for hot segments, S3 for cold) should follow measured access patterns rather than a uniform keep-everything-hot default.

Ingest lag is the operational metric that couples Kafka and Druid. If MiddleManagers fall behind topic throughput, "real-time" becomes a marketing label on a delayed segment. Task count, replicas, and maxRowsInMemory are the first knobs; cluster role splits come after a single supervisor path is stable under production peak.

Segments buy the speed

An aggregation that sits in the multi-second range on a batch warehouse can return in hundreds of milliseconds on Druid when the data model fits. Time-partitioned immutable segments, columnar layout with aggressive compression, and broker scatter-gather across Historicals that already hold the relevant segments do most of the work.

Segments cover a fixed time range (hour or day is common). A "last 24 hours" query prunes everything outside those segments. Columnar storage reads only the dimensions and metrics named in the query. A country rollup never touches unused columns. Cardinality of a column drives compression: low-cardinality dimensions compress hard; high-cardinality IDs do not.

The Broker maps a query to segment sets and fans work to Historicals; each node scans local data and returns partial results for merge. Parallelism tracks how well the time range and filters map onto independent segments. Pre-aggregation plus partition pruning plus column pruning is why the same logical group-by can be tractable on streaming event volumes that overwhelm a row-oriented warehouse path.

INFO: Scan less, answer faster
Rollup reduces rows before query time. Time partitions drop irrelevant segments. Columnar layout drops unused fields. Together they shrink scanned bytes relative to replaying raw events in a data lake. Treat that as an architectural claim about work avoided. Published latency numbers vary by cluster, schema, and concurrency; copy them only with a named source.

Immutable segments also constrain the write model. Appends and late data handling are first-class; in-place row updates are not. That is why transactional systems and frequently corrected dimensions stay in OLTP, with events or periodic snapshots feeding Druid rather than treating Druid as a system of record.

Single node first

The production pattern is Kafka first, Druid second, cluster roles last. Capture high-value application events with a stable schema (timestamp, entity keys, event type, a small dimension set) before arguing about Historical sizing. Without the stream, there is nothing for real-time OLAP to index, and retrofitting event emission across services is slower than standing up a database.

Prove the path on a single Druid process (or a managed equivalent) reading one priority topic. Validate ingest lag, query correctness, and latency on the actual dashboard or decision path. Only then split into the three servers Druid actually defines: Master (Coordinator plus Overlord), Query (Broker plus Router), and Data, where Historical and MiddleManager are colocated by design rather than pulled onto separate boxes.

Deep storage on S3 from day one is cheap insurance even on a small footprint: segments leave local disk and retention policy becomes an object-lifecycle problem. Multi-region Historical fleets, complex multi-stage rollup, and heavy multi-tenancy isolation are forced by concurrency and SLOs. They are optional until a metric says otherwise.

Operational maturity is boring and load-bearing: alert on ingest lag and query latency, set retention that finance and compliance both accept, and expose a SQL or API layer so product teams stop filing tickets for every group-by. The Kafka-to-Druid skeleton absorbs those layers without a redesign if the first supervisor path was kept simple.

When the split does come, the hardware follows what each server actually does, and two of the three are routinely mismatched. The Broker is memory-hungry rather than CPU-bound, because it holds a heap plus off-heap merge buffers, so a general-purpose box ([Druid's own example is an m5.2xlarge](https://druid.apache.org/docs/latest/tutorials/cluster/)) beats a compute-optimized one with half the RAM. The MiddleManager is the opposite: task slots default to one per CPU minus one, and tasks stage segment data on local disk, so it wants cores and NVMe, not a memory-optimized box with neither. Data servers want free RAM on top of that, since Historicals page-cache their segments. In current families that lands on m7i or m8g for Master and Query, and i7ie, i8g, or im4gn for Data.

Capacity planning follows ingest rate and concurrent query fan-out, not a generic "enterprise cluster" template. Druid publishes six sized single-server configs and no events-per-day ceiling, which is the honest position: throughput is dominated by dimension cardinality and rollup, so the widely repeated "single node handles 500 million events a day" has no source behind it. Resize when lag or p99 latency says so; pre-building a 50-node topology for a single product funnel wastes the budget the rollup was meant to save.

Most stacks skip Druid

Druid is specialized real-time OLAP for append-heavy event analytics. Workloads that need frequent updates and deletes belong in PostgreSQL or MySQL. Full-text document search belongs in Elasticsearch or an equivalent search engine. Graph traversal belongs in a graph store. Using Druid for those jobs fights the storage model.

If data lands in hourly or daily batches and the team is small, start with ClickHouse (or stay on BigQuery/Redshift if latency already meets the product bar). Druid's operational tax is justified when native streaming ingest and sub-second queries on data that arrived seconds ago are requirements. If dashboards can wait five seconds, or event volume stays in the low millions per day, PostgreSQL with materialized views or TimescaleDB usually delivers with less machinery. Single-node first is still the right Druid entry when the stream and the SLO are real but the team has not yet earned a multi-role cluster.

User-facing product analytics with extreme p99 targets may fit Pinot better than Druid. Log aggregation and warehouse-style SQL on batch loads may fit ClickHouse better. Picking Druid because a peer company runs it, without matching their stream volume and latency budget, is how teams inherit five node roles and none of the payoff.

Internal BI that refreshes every few minutes is the common false positive. A warehouse, a scheduled transform, or ClickHouse on batch load is enough for that refresh rate. Adopt Druid when product latency and concurrent analytical load make the warehouse the bottleneck. Fashionable real-time language is not a requirements doc.

WARNING: Don't buy dashboard latency
If the SLO is "dashboard updates within a few minutes," skip Druid. Spend the engineering time on event instrumentation quality and a simpler OLAP path. Bring Druid in when measured sub-second concurrency on a live stream fails under the warehouse or single-node SQL you already run.

Latency must force it

Sub-second analytics on high-event streams needs a real-time OLAP path: events into Kafka (or equivalent), a Druid-class system for ingest-time indexing and concurrent analytical queries, object storage for durable segments. Batch warehouses remain the right tool for large historical jobs that can wait. Mixing those two jobs into one system is how teams relearn the surge-pricing lesson at dashboard scale.

Druid only pays when query latency and concurrency force it. Instrument streams first. Measure the warehouse (or Postgres) path under the real dashboard and decision load. Reach for specialized OLAP when those measurements fail the product SLO and the event path is already continuous. Below that line, simpler stores win on ops cost.

The companies that made the Kafka-to-Druid pattern famous did not adopt it for prettier charts. They adopted it because a pricing, quality, or fraud decision was gated on an aggregation that the warehouse could not return in time. Match that condition first. The cluster topology is secondary.

Key Point: Kafka plus a Druid-class OLAP path is the category for sub-second analytics on high-event streams. Batch warehouses are the wrong tool for that SLO. Skip Druid until latency and concurrency make the operational tax worth it.

Resources & Further Reading

• Apache Druid Documentation: https://druid.apache.org/docs/latest/design/ - Architecture and operational guides
• Apache Druid Comparisons: https://druid.apache.org/docs/latest/comparisons/ - Project comparison docs (no official Druid-vs-ClickHouse page)
• How Netflix uses Druid for Real-time Insights: https://netflixtechblog.com/how-netflix-uses-druid-for-real-time-insights-to-ensure-a-high-quality-experience-19e1e8568d06 - Netflix Tech Blog, March 2020
• Interval-Aware Caching for Druid at Netflix Scale: https://netflixtechblog.com/stop-answering-the-same-question-twice-interval-aware-caching-for-druid-at-netflix-scale-22fadc9b840e - Netflix Tech Blog, April 2026 (scale and query-load notes)
• Airbnb Engineering - Druid: https://medium.com/airbnb-engineering/druid-airbnb-data-platform-601c312f2a4c - Airbnb's Druid deployment notes
• Apache Druid Powered By: https://druid.apache.org/druid-powered - Production users including PayPal
• AWS MSK: https://aws.amazon.com/msk/ - Managed Kafka on AWS
• Apache Druid Quick Start: https://druid.apache.org/docs/latest/tutorials/index.html - Getting started tutorials
• Confluent / Kafka: https://www.confluent.io/ - Commercial Kafka platform with cloud and self-hosted options