CUDA 13.4 and Shared GPU Inference: What MPS v3 Changes

Shared GPUs need explicit resource boundaries

Start with a service objective. The hypothesis worth testing is whether finer resource controls allow more work on one GPU while preserving interactive response times. Online conversation and offline embedding generation make useful experimental workload classes: one values first-token latency and continuous output, while the other may prioritize batch completion. A single average utilization figure collapses those objectives into one number and cannot establish that both workloads benefit.

Separate capacity, bandwidth and isolation. A process allocation cap answers part of a capacity-budget question; it does not by itself establish a fixed bandwidth share or response-time guarantee. The MIG guide describes separate compute and memory-system resources on supported hardware.[5] Our recommendation is to document capacity boundaries, interference tolerance and recovery requirements separately, then test the appropriate mechanism. An MPS configuration label is insufficient evidence that it substitutes for a MIG deployment.

Design the memory budget with its failure behavior

Examine how limits change failure. The MPS v3 memory-partitioning documentation requires Linux, cgroup v2 and a non-MIG device. Hard-limit violations fail allocation; soft-limit borrowers can become eviction candidates under real pressure, and managed memory has additional constraints.[4] The operational inference is to budget for interrupted requests. Evaluate interruptible background work and interactive inference separately rather than accepting successful model loading as a sufficient readiness check.

Budget across the request lifecycle. We propose measuring memory high-water marks for an idle service, model loading, warm-up, short contexts, long contexts and peak concurrency. Record resident allocations, allocations that grow with active requests, temporary workspace and recovery headroom separately. Use observations from the actual serving stack instead of a universal spare-memory percentage. Two backends or batching policies need not behave equivalently simply because their configured capacity limits match.

A common mistake is to measure memory after model loading and allocate all remaining capacity to other processes. The true peak may arrive with a long request, a larger batch or a temporary workspace allocation. Record steady-state and peak behavior for each task class and reserve recovery headroom. Limits should reflect the service lifecycle, not a single monitoring snapshot. Otherwise, sharing merely postpones a capacity collision until production.

Decide which work may be interrupted. An offline job can often be rerun, whereas an interactive request has a person waiting for it. Equal memory consumption does not imply an equal pressure-handling policy. Classify work as retryable, deferrable or continuity-sensitive before assigning budgets and queue priorities. This translates low-level controls into a product promise rather than a set of precise-looking configuration values.

Measure a baseline before sharing and limiting

Build an attributable experiment. Hold hardware, driver, weights, precision, serving image and request samples constant. First measure an isolated deployment, then shared execution, then introduce controls individually. Preserve request arrival times while independently varying long-context frequency, output length and background intensity. Repeat runs and keep full results. This design helps distinguish the effects of a toolchain upgrade, a different workload mix and the resource policy itself.

Count requests that meet the objective. We propose defining useful throughput as complete, successful requests per unit time that satisfy both the chosen first-token and output-interval targets. Set those targets from product requirements before examining results. Report P50, P95 and P99 latency, errors, admission rejections and completions separately for each workload class. Higher aggregate tokens per second can still be an unacceptable outcome if interactive tail latency violates the service objective.

Control how requests arrive. A test that sends each request only after the previous one completes can hide queueing under simultaneous users. Replay the same anonymized arrival sequence across configurations, then inspect waiting-time distributions for interactive and background work separately. Report the workload conditions rather than presenting the best low-concurrency run as sustainable capacity. This is an experiment proposal, not a hardware result measured by Token4AI.

Recovery and upgrades determine operational viability

Include recovery in the experiment. MPS documents limits to error containment: some clients sharing affected GPUs can be disrupted by a fatal GPU fault.[6] In an isolated test environment, examine allocation exhaustion, abnormal client termination and service restart. Measure recovery duration and check that retries neither duplicate results nor keep amplifying offered load. For streaming requests, define what happens after partial output has already reached a client; a final HTTP status alone does not describe that experience.

Validate an upgrade separately from a new capability. The release notes distinguish CUDA 13.x minor-version compatibility from support for new 13.4 capabilities and platforms, which require an R615 or newer driver that supports them. Rubin support remains a developer preview.[2] Our proposed rollout starts with a fixed software combination on a test node, verifies model correctness and backend compatibility, and only then measures sharing behavior against the existing service objectives.

Connect GPU observations to service outcomes

Translate controls into an explicit deployment policy. Our assessment is that complementary workloads, meaningful task classes and adequate observability are strong starting conditions for an evaluation. Produce a reviewable decision record: which interference is acceptable, which jobs may be interrupted, which objectives must hold and how recovery works. Better management interfaces make such policies easier to implement. Actual savings still depend on completed work that meets those objectives and the full cost of operating the service.

For teams using the Token Cloud API in Token AI Cloud, directly observable signals remain request latency, failures and token usage. These can reveal a change in service experience, but cannot establish whether the underlying system uses MPS, MIG or another partitioning strategy. This article explains how a deployment team should evaluate resource controls; it does not disclose a particular architecture used by our platform.

The cover is an original Token4AI diagram of shared compute budgets, memory limits and request quality. It is not a physical GPU partition map and does not depict MPS controls as MIG hardware isolation.