Data-center AI cluster, rack simulator

Sizing an AI data center that stays relevant for 5-10 years requires navigating aggressive hardware roadmaps, evolving model architectures, and uncertain workload mixes. Decisions on GPUs, fabrics, and rack counts are costly to reverse, so teams need a quick way to stress-test assumptions before committing capital.

To support that planning, we built a rack-scale throughput estimatora nd initial data-center simulator that models how large language models run across multiple data-center racks. The simulator sweeps tensor, pipeline, and expert parallelism, explores 4/8/16-bit quantisation, and reports sustained decode throughput along with the limiting subsystem (compute, HBM, or fabric).

Out of the box we include eight-rack presets for:

• NVIDIA GB200 NVL72 cabinets
• NVIDIA GB300 NVL72 cabinets (projected)
• AMD MI300X OAM racks
• AMD MI355X OAM racks (projected)

The default workload is Qwen3-235B, a state-of-the-art open-source large-language-model (LLM) model suitable for high-scale inference.

Everything is open for inspection and extension. Clone the repository, run the simulator, and contribute improvements at github.com/culurciello/dc-simulator.

Why this models?

Many startup companies and incumbents in the field of AI applications use large language models (LLMs) to run inference at scale. However, data center providers such as Azure, AWS, and Vultr often abstract scaling parameters and infrastructure so heavily that it becomes difficult to optimize for inference performance or accurately estimate monthly operational costs.

In addition, this abstraction limits visibility into key factors such as GPU utilization efficiency, network latency, and storage throughput—parameters that directly affect both cost and responsiveness. Without fine-grained control or transparency, teams struggle to design cost-effective scaling strategies, especially for workloads that experience variable demand or require low-latency responses.

We decided to investigate the scaling behavior of inference on the latest AI LLM models, such as Qwen3, to better understand the performance–cost trade-offs across different deployment environments. Our goal is to measure how inference latency, throughput, and GPU utilization scale with model size, batch size, and concurrent user load.

By running controlled experiments across multiple infrastructure providers and hardware configurations, we aim to quantify how efficiently these models can be served under real-world conditions. This includes evaluating containerized deployments, quantized model variants, and distributed inference frameworks. The results should highlight practical limits of current scaling abstractions and inform strategies for cost optimization and hardware selection in production environments.

Simulation detailed results

The optimiser landed on batch 12 for every preset; tokens/GPU flatten beyond that point because each rack is compute-bound.

Batch-12 optima by quant level

The brace notation highlights the expert-parallel degree chosen for each quant level (1 for 4-bit, 2 for 8-bit, 8 for 16-bit).

Interpretation

Computation on each GPU was the primary limiting factor in overall inference performance. Interestingly, communication bandwidth across the interconnect fabric was largely underutilized. The measured Fabric Load per Instance (GB/s) for all three cases—intra-node, inter-node, and inter-rack—was effectively zero.

This suggests that the inference workload was dominated by local GPU computation rather than data exchange between GPUs or across racks. In other words, scaling inefficiencies were not caused by communication bottlenecks but rather by the computational intensity of the model itself. This finding implies that optimizing kernel efficiency, tensor parallelism, or GPU scheduling could yield larger performance gains than network-level optimizations in this setup.

The communication infrastructure is underutilized because it is primarily designed to sustain training workloads, which require extensive data and gradient exchange between GPUs. In contrast, inference workloads are far less communication-intensive, as most computations are localized to individual GPUs once the model weights are loaded. As a result, high-bandwidth interconnects such as NVLink or InfiniBand remain largely idle during inference runs.

This highlights a key mismatch between hardware design and inference requirements: data center clusters optimized for large-scale distributed training may not provide proportional efficiency when repurposed for inference at scale. Tailoring infrastructure for inference—through lighter interconnect fabrics, higher memory density per GPU, or improved batch scheduling—could significantly improve cost-performance ratios.

Insights

Inter-rack and inter-node communication can be optimized to become more adaptive and resource-aware, enabling the system to dynamically allocate compute and communication resources based on real-time workload demands. When computation becomes the bottleneck, the infrastructure should be able to source additional GPU or accelerator capacity with minimal communication overhead. Conversely, when communication is underutilized—as observed in our measurements—the system could reconfigure the network topology or downscale link utilization to operate more efficiently using lower-cost components and switches.

This flexibility would allow data centers to strike a better balance between performance and cost-efficiency. By decoupling inference communication requirements from training-oriented infrastructure design, future AI clusters could achieve higher utilization rates and reduced total cost of ownership (TCO) without compromising latency or throughput.

Detailed results

NVIDIA GB200 NVL72 (8 racks, 576 GPUs)

• 4-bit quantisation keeps each instance on a single GPU (TP=PP=EP=1), reaching 182 k tokens/s aggregate.
• Higher-precision runs increase expert parallelism to stay under the 192 GB memory cap while holding the rack compute bound.

NVIDIA GB300 NVL72 (8 racks, 576 GPUs)

• Higher sustained FLOPs (llm_cluster_simulator.py:123) lift tokens/GPU to 422 at batch 12 with INT4 weights.
• Even at large batches, the intra-rack fabric remains idle, so scaling hinges purely on compute throughput.

AMD MI300X OAM Rack (8 racks, 512 GPUs)

• Lower sustained FLOPs (llm_cluster_simulator.py:105) yield ~152 tokens/GPU at batch 12 with INT4 weights.
• Expert parallelism scales from 1->8 as precision grows, keeping per-GPU memory under the 192 GB cap.

AMD MI355X OAM Rack (8 racks, 512 GPUs)

• MI355X inherits the MI300X topology but with stronger blocks: sustained FLOPs jump to 14.5 TF/s, lifting total throughput to ~98 k tokens/s at batch 12 INT4.
• Fabric load remains low; further tuning should focus on kernel efficiency or additional racks rather than networking.

Metric glossary

• Tok/GPU(avg): cluster-average tokens/s per active GPU (total rack TPS divided by total active GPUs); ideal for hardware-to-hardware comparisons.
• Inst TPS: sustained tokens/s for a single model instance under the chosen parallel plan.
• Total TPS: aggregate throughput across all eight racks of the preset.
• Limit: subsystem that capped throughput (compute, hbm, comm).
• Bounds tok/GPU(avg): theoretical ceilings derived from compute, HBM, and communication throughput before efficiency scaling.
• Fabric load: effective bandwidth consumption per instance on the intra-server, inter-server, and inter-rack links.

Because the simulator models steady-state decode, it omits prefill latency; extend the model if you need prompt-heavy workloads.

Modeling notes

Utilisation ceilings

The marketing peak numbers (e.g. 2,500 TFLOPs for GB200) assume perfect FP8 matmuls. Real decode workloads rarely exceed 20-30 % utilisation because they thrash the KV cache and wait on collectives. The sustained FLOPs in GPU_PRESETS (llm_cluster_simulator.py:105-128) reflect decode-mode telemetry from public benchmarks. Replace them with your own nsys/ncu traces if you have better data.

The compute ceiling is estimated with:

where compute_scale comes from QUANT_PRESETS, and flops_per_token is derived from the active parameter count in the MoE (llm_cluster_simulator.py:70-101). If observed tokens/GPU exceed the ceiling, tighten those inputs.

Parameter provenance

• sustained_flops: decode-mode throughput after utilisation losses (20-35 % below peak). Update with measurements if available.
• hbm_bw: effective HBM bandwidth assuming 60-70 % utilisation on decode-heavy kernels.
• compute_scale: empirical uplift from lower-precision kernels; tune to your INT4/INT8 stack.
• kv_bytes_per_elem: defaults assume FP16 KV for 16-bit, FP8/INT8 for 8-bit, and compressed FP8 for 4-bit variants.
• Interconnect bandwidths (intra_server_bw, inter_server_bw, inter_rack_bw): sourced from vendor topology guides; swap in NCCL microbenchmark results for accuracy.

Qwen3 mixture-of-experts assumptions

• Default: top-8-of-128 MoE with 88 % of parameters in experts and 12 % shared; adjust expert_param_fraction, shared_param_fraction, and active_experts to match your checkpoint.
• Expert traffic is modelled as an all-to-all exchange of FP16 activations with a 2x dispatch factor (moe_dispatch_factor); raise it if packing overhead is higher.
• Expert-parallel degrees (ep_candidates) multiply the tensor x pipeline mesh; shared weights replicate, expert weights shard.

Why GB200 outpaces MI355X

• Per-GPU sustained FLOPs: 24 TF/s for GB200 versus 14.5 TF/s for MI355X (llm_cluster_simulator.py:112-119), a 1.65x gap that dominates when runs are compute bound.
• Rack population: NVL72 racks host 72 GPUs each (9 servers x 8 GPUs) whereas the MI355X racks hold 64 (llm_cluster_simulator.py:132-171), adding ~12.5 % more devices to the GB200 sweep.
• Aggregate compute wins: multiplying the factors above gives ~1.9x more usable FLOPs for GB200, so total TPS lands just under 2x higher despite identical batch sizes and quant modes.

References

• SemiAnalysis: AMD Advancing AI - MI350X and beyond
• Qwen3 architecture primer: Sebastian Raschka’s deep dive
• NVIDIA GB racks architecture
• AMD racks

Want me to work with you on this?

Do you want to work together to optimize your data center, racks, nodes, networking, storage, operations? Contact me!

about the author

I have more than 20 years of experience in neural networks in both hardware and software (a rare combination). About me: Medium, webpage, Scholar, LinkedIn.

If you found this article useful, please consider a donation to support more tutorials and blogs. Any contribution can make a difference!