The Internet Learned to Route Packets.
 Now It Must Route Intelligence.

Imagine asking an AI assistant to review a 200-page contract.

Two data centers can handle the request. The first is five milliseconds away, but its accelerators are heavily loaded and dozens of requests are already waiting. The second is twenty-five milliseconds away, but it has available capacity, the correct model already loaded, and part of the document context cached from an earlier interaction.

A conventional proximity-based system may prefer the first location.

The second location could still produce the answer sooner.

That gap, between the nearest service and the best service, is becoming one of the most important networking problems of the AI era.

For decades, the Internet has been remarkably good at answering a basic question: How do I reach a destination? Anycast, content delivery networks, and global load balancing improved the answer by directing users toward one of several service locations. Yet network proximity has never guaranteed balanced server load. RFC 4786, published in 2006, noted that balancing traffic among anycast nodes is typically difficult and that traffic distribution can be uneven.

Key idea: The best AI endpoint is the site that can produce the required outcome under latency, capacity, state, policy, and cost constraints. It may not be the nearest site.

The nearest AI may not be the fastest AI

A traditional content-delivery system usually chooses where to retrieve an existing object, such as an image, video, software package, or webpage.

An AI service must produce something that does not yet exist.

That difference changes the performance equation. The user experience depends on whether the destination has the required model, enough accelerator memory, a manageable queue, available storage, and sufficient capacity to process the request. It may also depend on whether the destination already holds useful session state.

A more honest latency equation: End-to-end AI response time equals network time + queueing time + prefill time + token-generation time + any state-transfer time.

Large language model inference is not one uniform operation. During prefill, the system processes the input prompt and builds the internal state needed to begin the answer. During decode, it produces subsequent tokens incrementally. The two stages place different demands on compute, memory, and networking resources. The IETF Computing-Aware Traffic Steering use-case document now treats distributed AI inference, including prefill and decode, as an explicit traffic-steering scenario.

Reducing network time while ignoring the other components can produce a technically shorter path and a noticeably slower service.

Research on LLM serving already shows how large the placement effect can be. DistServe separates prefill and decode work across different GPUs and accounts for network bandwidth when placing those resources. In the authors' evaluations, it served up to 7.4 times more requests or supported 12.6 times tighter service-level objectives than the comparison systems, while keeping more than 90 percent of requests within the stated latency constraints.

Those figures are workload-specific research results, not a universal prediction. They still make the architectural point clearly: where each stage runs can matter as much as raw accelerator speed.

Routing can no longer stop at the IP address

The IP layer works because it does not need to understand what a packet means. A router does not need to know whether it is carrying a medical image, a banking transaction, or a video stream. It forwards according to reachability and policy.

That separation should not disappear.

Fast-changing GPU statistics, prompt details, and model inventories do not belong in global BGP. Mixing second-by-second compute state into the global routing table would create an unstable, unscalable, and potentially dangerous control system.

What should not happen: Compute-aware networking does not mean turning every router into an AI scheduler or placing raw GPU telemetry into global BGP. The practical design is a controlled service-steering layer, usually within one provider or administrative domain.

The more realistic change is a service-steering layer that coordinates network conditions with compute conditions.

This idea is already taking shape at the IETF through Computing-Aware Traffic Steering, or CATS. The CATS framework describes an overlay in which service and network metric agents provide information to a path-selection function. That selector can consider the state of the network and the available service sites before directing a request. Its core framework is deliberately scoped to a single service provider.

As of August 4, 2026, the core CATS framework and use-case documents were in the RFC Editor queue as Informational documents. A separate working-group draft was defining a hierarchical metric model for compute and communication conditions. That draft emphasizes abstraction rather than distributing every raw infrastructure counter. The goal is to make the information useful enough for selection without making the control plane depend on an unbounded flood of low-level measurements.

Standards snapshot

CATS Framework: Defines functional components and workflows for compute-aware steering within one service provider. Status on August 4, 2026: RFC Editor queue, Informational.

CATS Use Cases: Defines the problem statement, requirements, and scenarios that include distributed AI inference. Status on August 4, 2026: RFC Editor queue, Informational.

CATS Metrics: Defines hierarchical abstractions for compute and communication metrics used in service selection. Status on August 4, 2026: active working-group draft.

Internet-Drafts are works in progress. Their status and wording can change before publication as RFCs.

From destination routing to outcome routing

I think of the larger idea as outcome routing.

Destination routing asks where a packet should go.

Outcome routing asks where a particular result can be produced under a set of constraints.

An AI request require all of the following:

1. Capability: Model family, version, context length, tools, precision, and safety profile.
2. Compute: Available accelerator memory, queue depth, expected time to first token, and token rate.
3. Network: Latency, loss, congestion, bandwidth, resilience, and path health.
4. State: Session locality, reusable prompt prefixes, KV cache availability, and transfer cost.
5. Policy: Data residency, security boundary, tenant isolation, and approved providers.
6. Economics: Price, energy mix, service tier, and contractual objectives.

The best destination is the location that can satisfy the complete request, not merely the one with the shortest path.

This is a subtle but important shift. In the traditional model, the application chooses an endpoint and the network finds a path to it. In an outcome-oriented model, endpoint selection and path selection influence each other.

The application knows the required capability. The compute platform knows where that capability is available. The network knows whether the path is congested or impaired. The policy system knows where the data is allowed to go.

No single layer has the entire answer.

The hardest problem is stale information

Collecting compute metrics is relatively straightforward. Using them safely is much harder.

Network routes usually do not change every time a server becomes slightly busier. Compute conditions can change from one request to the next. A previously idle accelerator can fill quickly. A long prompt can consume a large amount of memory. A queue can grow in seconds.

Now imagine thousands of clients receiving the same report that one site is lightly loaded. They all send their next requests there. By the time the traffic arrives, the site is no longer lightly loaded.

The controller reacts and sends traffic elsewhere. The new site fills next, and the system begins chasing its own measurements.

The CATS requirements explicitly call for staleness handling and for mechanisms that avoid routing loops or path oscillation. They also identify the basic tradeoff: frequent metric distribution improves freshness but raises signaling overhead, while slower updates reduce overhead but can leave decision-makers acting on old information.

A workable system will need more than raw utilization percentages. It will need familiar networking controls such as thresholds, hysteresis, dampening, and bounded policy, combined with admission control and predictive scheduling at the application layer.

It will probably also separate decisions by timescale.

The underlying network path can remain comparatively stable. A service gateway can make faster request-level decisions. A cluster scheduler can react even more quickly inside a data center. Not every change in queue depth needs to become a routing update.

The goal is coordination among layers, not the collapse of every layer into one enormous control loop.

Compute telemetry becomes control-plane information

There is another uncomfortable question: What happens when the metrics are wrong?

A malfunctioning service site could report spare capacity that it does not have. A compromised site could do so deliberately.

In conventional routing, a false announcement can attract traffic. In compute-aware steering, a false announcement could attract traffic, prompts, proprietary documents, and valuable AI workloads.

An individual IETF security draft has already catalogued threats such as false compute metrics, fake service-instance registration, path manipulation, topology poisoning, and leakage of sensitive operational data. It discusses safeguards that include authenticated identities, protected metric channels, signed capability information, access control, and limiting the detail exposed to decision-makers.

The draft is exploratory rather than a finalized standard, but the security principle is already clear: compute advertisements must be treated as control-plane claims.

Those claims need a verifiable origin, an explicit validity period, and a narrowly defined authority. The right to advertise a model capability should be separate from the right to claim that capacity is currently available.

Detailed telemetry also cannot be shared indiscriminately. GPU inventories, load patterns, queue depths, model availability, and request volume can reveal commercially sensitive information or help an attacker identify attractive targets.

Providers will need to expose enough information to make a useful decision without publishing a live map of their infrastructure.

Normalized scores help, but they create a governance problem of their own. Who defines what a score of 80 means? Can scores from two providers be compared? Can a provider tune its formula to make its services appear healthier than a competitor's?

The metric is not simply a number. It is a claim that somebody must be able to trust.

The best endpoint are the one that remembers you

AI inference introduces a form of locality that ordinary routing does not understand: model and session state.

When a language model processes a prompt, it creates intermediate key-value information, commonly called the KV cache, that can be reused while generating later tokens. Reusing relevant state can avoid repeated work. Moving that state between servers, however, consumes time and network capacity.

Mooncake, a KVCache-centric serving architecture presented at FAST 2025, separates prefill and decode resources and uses CPU memory, storage, and network-interface capacity to create a distributed cache. Its scheduler treats cached state as a first-class serving resource rather than an incidental implementation detail.

A July 2026 individual Internet-Draft goes further by describing KV cache as a network-distributed resource. It proposes exposing whether reusable state exists at a service instance, the size of the cache entry, and the expected retrieval cost so that those signals can be combined with compute and network metrics.

It also warns that cache information and cache content can reveal details about the inputs being processed. This draft has no formal standing in the IETF standards process, but it captures a real architectural pressure.

Suppose one site is less busy, but another already holds the context for a long-running AI session. Moving the request reduce queueing and increase state-transfer time. Recomputing the state avoid the transfer but waste accelerator capacity. Keeping the session in place preserves locality but it overloads the site.

The system must decide whether to move the request, move its memory, or move neither.

In practical terms, the route depend on where the AI currently remembers the conversation.

Fast is not the same as acceptable

Performance will not be the only constraint.

An enterprise allow a general research request to use any approved region while requiring a request containing customer records to remain inside a specific jurisdiction. A security-sensitive workload is allowed to use only a privately operated model. An industrial system prefer a smaller edge model because the cloud connection is not reliable enough for the control loop.

Cost and energy can also enter the decision. The CATS use-case document considers steering based on energy price or energy mix when multiple sites can provide similar response times. That makes the best endpoint a policy choice as much as a performance choice.

The preferred destination will be:

• Fastest site,
• Cheapest site,
• Cite using the desired energy mix,
• Compliant site,
• Site with the correct model,
• OR the site that already holds the required state.

Sometimes these will be the same place. Often they will not.

Who should make the decision?

The application knows the user's intent, but it should not need a detailed view of the network.

The network understands path conditions, but routers should not inspect prompts to decide whether a request involves legal analysis or image generation.

The inference platform understands models and accelerators, but it will have limited visibility into congestion outside its own data center.

The likely answer is a controlled exchange of limited information.

An application or AI gateway can describe the capability and policy requirement without revealing the full prompt. A compute platform can advertise bounded availability signals without exposing its complete topology. A network controller can contribute path conditions. A policy engine can remove destinations that are not eligible. A steering function can then select among the remaining choices.

The CATS framework permits the selection function to be integrated with a forwarding element or implemented as part of a more centralized controller. Its single-provider boundary is sensible. Sharing dynamic compute information across independent providers introduces harder questions of trust, comparability, confidentiality, commercial incentives, and accountability.

Cross-provider intelligence delivery eventually emerge, but it will require more than a routing extension. It will require contracts for capability, identity, metrics, policy enforcement, settlement, and responsibility when the decision is wrong.

The next networking abstraction is a promise

The Internet was built around destinations. An address told the network where traffic needed to go, and the routing system determined how to reach it.

AI services increasingly begin with a capability request:

Find a system that can perform this task, with this model class, within this latency target, under these security and geographic restrictions, at an acceptable cost.

That does not make IP addresses disappear. It makes the address the output of a richer decision.

The application expresses an intended outcome. Compute systems advertise what they can provide. The network contributes path conditions. Policy narrows the acceptable choices. The steering layer binds the request to a site and a path.

This change will not arrive through one protocol, one standards body, or one vendor platform. Some decisions will remain inside data centers. Others will occur at cloud gateways, telecom edges, enterprise networks, or regional inference exchanges.

But the direction is becoming clear.

The network of the AI era will not merely transport prompts to preselected machines. It will increasingly participate in deciding where intelligence is produced.

The Internet's next routing question is not only "Where is the server?" It is "Where should this particular piece of intelligence be produced, under which constraints, and with what evidence that the decision was safe?"

Research notes and sources

Standards status was checked on August 4, 2026. Internet-Drafts can be updated, replaced, or withdrawn. Research performance numbers reflect the authors' test environments and should not be treated as universal production benchmarks.

1. J. Abley and K. Lindqvist, RFC 4786: Operation of Anycast Services, December 2006.
2. K. Yao et al., Computing-Aware Traffic Steering Problem Statement, Use Cases, and Requirements, draft-ietf-cats-usecases-requirements-14, February 3, 2026.
3. C. Li et al., A Framework for Computing-Aware Traffic Steering, draft-ietf-cats-framework-24, April 2, 2026.
4. K. Kehan et al., CATS Metrics Definition, draft-ietf-cats-metric-definition-10, June 22, 2026.
5. Y. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, USENIX OSDI 2024.
6. H. Shi et al., Security Considerations for Computing-Aware Traffic Steering, draft-wang-cats-security-considerations-04, March 2, 2026.
7. R. Qin et al., Mooncake: A KVCache-centric Architecture for Serving LLM Chatbot, USENIX FAST 2025.
8. Z. Li et al., KV Cache Distribution for Distributed LLM Inference, draft-li-cats-kv-cache-distribution-00, July 4, 2026.
9. IETF Computing-Aware Traffic Steering Working Group documents.

Editorial note: The term "outcome routing" is used here as a conceptual framing. It is not presented as the name of an adopted IETF standard.