Back previous/next report: k j

NVIDIA NVIDIA GB300 NVL72 x8 (TRT-LLM + Dynamo)

Submitted by NVIDIAon 2026-11-13 Published on 2026-11-13

Result statement

NVIDIA ran GPT-OSS 120B on NVIDIA GB300 NVL72 x8 (TRT-LLM + Dynamo) and achieved a provisional 164,188.7 tokens/second System TPS at 19,504.9ms TTFT P90 in the Standardized division of the Available category on November 13, 2026. Pending MLCommons Association verification, results are subject to change pending peer review. .

[1] MLPerf Endpoints v1.0 GPT-OSS 120B, published November 13, 2026; retrieved from results.mlcommons.org on November 18, 2026. Result —, run 11: concurrency = 12,288 queries, TTFT P90 = 19,504.9ms, System TPS = 164,188.7 tokens/sec, interactivity = 13.4 tokens/sec per user and total provisioned system power = 15 kW. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Submission Info
NVIDIA
053005da-b0ad-495d-9721-e39e9b144bd8
Submission Status REVIEW_PENDING
2026-11-13
2026-11-13
Version v1.0
Division Standardized
Deployment Client on Premises
Model
Model Name GPT-OSS 120B
Division Standardized
Model Precision —
Model Link —
Calibration Link —
Model Notes —
Datasets
Dataset Name perf
Dataset Type performance
Average Input Tokens —
Dataset Link —

System TPS vs. Tokens/s per User

050k100k150kSystem TPS0100200300400500Tokens/s per User

System TPS vs. Concurrency

050k100k150kSystem TPS05,00010,000# of queries

Time to First Token P90 vs. Concurrency

05.0k10k15k20kTTFT P90 (ms)05,00010,000# of queries

Tokens/s per User vs. Concurrency

0100200300400500Tokens/s per User05,00010,000# of queries
SUT Summary
NVIDIA
NVIDIA GB300 NVL72 x8 (TRT-LLM + Dynamo)
Available
System Category — · —
8 accelerators
1
2
Max Supported Concurrency 12,288
Node Type ID 0
Node Count 2
Software
Ubuntu
Framework TensorRT-LLM 1.3.0rc29 (GPT-OSS branch feat/gpt-oss_gb300_26.08) + Dynamo 1.4.2
TensorRT-LLM 1.3.0rc29 (GPT-OSS branch feat/gpt-oss_gb300_26.08) + Dynamo 1.4.2
TensorRT-LLM 1.3.0rc29, GPT-OSS branch feat/gpt-oss_gb300_26.08, commit f19ee5a1aa889c996b67478d0d0a4885060324cc
580.173.10
Link to Container built from the public TensorRT-LLM branch feat/gpt-oss_gb300_26.08, commit f19ee5a1aa889c996b67478d0d0a4885060324cc
Operating System Ubuntu 24.04.5 LTS (host), kernel 6.17.0-1032-nvidia-64k; engine container Ubuntu 24.04.4 LTS
Filesystem Lustre
Other Software Dynamo 1.4.2 (nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.4.2); FlashInfer 0.6.18 + 2 commits (branch gpt-oss-gb300-endpoints-2026-10, commit 76a20ecb); CUDA 13.4.1; PyTorch 2.14.0a0+4fdf77b; NCCL 2.30.7; client ghcr.io/mlcommons/endpoints:d3c57f2; nv-sflow b8afb5b8
Software Notes TensorRT-LLM 1.3.0rc29 with the GPT-OSS branch feat/gpt-oss_gb300_26.08 at commit f19ee5a1aa889c996b67478d0d0a4885060324cc, served through Dynamo 1.4.2, with FlashInfer 0.6.18 + 2 commits (76a20ecb). The branch's runtime features are off unless a point sets their flag.
Hardware

Processor

NVIDIA Grace (ARM Neoverse-V2)
2
72
VCPUs Per Processor 72

Accelerator

—
0
—
—
—
—

Host / Storage

955.6 GiB per node (MemTotal 1,002,051,392 kB)
Memory Configuration ~478 GiB + ~477 GiB per node, one NUMA node per Grace CPU (489,770 MiB and 488,795 MiB per numactl; 958.5 GiB of memory blocks online per lsmem)
Storage Capacity 15.4 TB per node (4 x 3.84 TB + 1 x 1.92 TB NVMe)
Storage Type NVMe SSD

Networking

5 per node: 4 ConnectX-8 (InfiniBand, 800 Gb/s XDR each) and 1 BlueField-3 (one 400 Gb/s NDR InfiniBand port and one 200 Gb/s Ethernet port)
NVLink + InfiniBand + Ethernet
Network Topology —

Other

Other Hardware —
Cooling Liquid-cooled
Hardware Notes Two nodes of one GB300 NVL72 rack, 4 GPUs each, joined by the rack's NVLink fabric; every point allocates both nodes and serves on all 8 GPUs. Each node has 144 cores (2 x 72-core Grace, Neoverse-V2; no SMT, so 144 vCPUs); host_processor_core_count and host_processor_vcpu_count count per processor. docs/software_disclosure.md lists the client placement and the baseline latency measurement.
Run Data
Field NameRun 1Run 2Run 3Run 4Run 5Run 6Run 7Run 8Run 9Run 10Run 11
Run Date11/13/202611/13/202611/13/202611/13/202611/13/202611/13/202611/13/202611/13/202611/13/202611/13/202611/13/2026
Concurrency163248641282565121,0243,0726,14412,288
System Tokens/Second7,559.512,157.415,810.218,940.331,412.157,091.080,035.193,697.0131,352.7153,553.6164,188.7
Tokens/Second per User472.5379.9329.4295.9245.4223.0156.391.542.825.013.4
TTFT P90 (ms)143.889.195.1212.8259.3389.72306.67852.8981.31627.019504.9
Utilization4.6%7.4%9.6%11.5%19.1%34.8%48.7%57.1%80.0%93.5%100.0%
Configuration Summarygb300-x8-trtllm-dynamo-gptoss-c16 · submissiongb300-x8-trtllm-dynamo-gptoss-c32 · submissiongb300-x8-trtllm-dynamo-gptoss-c48 · submissiongb300-x8-trtllm-dynamo-gptoss-c64 · submissiongb300-x8-trtllm-dynamo-gptoss-c128 · submissiongb300-x8-trtllm-dynamo-gptoss-c256 · submissiongb300-x8-trtllm-dynamo-gptoss-c512 · submissiongb300-x8-trtllm-dynamo-gptoss-c1024 · submissiongb300-x8-trtllm-dynamo-gptoss-c3072 · submissiongb300-x8-trtllm-dynamo-gptoss-c6144 · submissiongb300-x8-trtllm-dynamo-gptoss-c12288 · submission
Config Typesubmissionsubmissionsubmissionsubmissionsubmissionsubmissionsubmissionsubmissionsubmissionsubmissionsubmission
Config Region———————————
Target QPS———————————
Result Statecompletecompletecompletecompletecompletecompletecompletecompletecompletecompletecomplete
Measured Accuracy Score———————————
Benchmark Versionunknownunknownunknownunknownunknownunknownunknownunknownunknownunknownunknown
Result Version0.1.00.1.00.1.00.1.00.1.00.1.00.1.00.1.00.1.00.1.00.1.0
Archive URIgs://mlperf-endpoints-blob/runs/6427ac25-8560-4962-9cfe-db6e75dd4875/archive.tar.gzgs://mlperf-endpoints-blob/runs/39c97b6b-0f19-455f-b2f5-49e5e14de007/archive.tar.gzgs://mlperf-endpoints-blob/runs/feba013e-0604-4b0f-a46c-da59a6d07572/archive.tar.gzgs://mlperf-endpoints-blob/runs/bf1e7a19-b7a2-4808-85c7-f4f059def75a/archive.tar.gzgs://mlperf-endpoints-blob/runs/e9a2a8ac-0a29-4d94-8434-24e52623b1b0/archive.tar.gzgs://mlperf-endpoints-blob/runs/5a67533f-6748-4d4f-83b6-5281a3d27ae9/archive.tar.gzgs://mlperf-endpoints-blob/runs/3eb23e77-cde8-43da-9b17-f619fcdf1a83/archive.tar.gzgs://mlperf-endpoints-blob/runs/f81dc1a8-0016-4731-93ec-165b19d1ce99/archive.tar.gzgs://mlperf-endpoints-blob/runs/5791fcfc-0ba5-48cd-9553-98a8dd8a1cf7/archive.tar.gzgs://mlperf-endpoints-blob/runs/f7838493-d160-4a95-9e4d-69f09be2140e/archive.tar.gzgs://mlperf-endpoints-blob/runs/120ab122-722e-4e09-84bd-700875217e8e/archive.tar.gz
QPS5.65659.099611.815614.165123.945143.536061.035171.8053100.1644117.0377125.1890
Total Output Tokens42,738,71142,726,51742,791,82151,313,15983,905,135142,585,987218,063,320250,379,700343,888,769411,186,638444,591,759
Average Output Tokens1,336.421,336.041,338.081,337.121,311.841,311.351,311.31,304.881,311.371,3121,311.53
Run Duration (s)5,653.673,514.452,706.592,709.202,671.112,497.522,724.602,672.232,618.062,677.802,707.81
Total Queries31,98031,98031,98038,37663,960108,732166,296191,880262,236313,404338,988
Total Queries Issued31,98031,98031,98038,37663,960108,732166,296191,880262,236313,404338,988
Total Queries Failed00000000000
Time to First Token (TTFT) (ms)
Minimum101.750.151.3127.5194.0198.6659.2635.7488.9729.57604.9
Average119.265.571.7172.6231.9304.01973.27451.2953.21738.018942.0
P50110.257.260.7167.9224.4285.41922.27448.1819.11386.218359.2
P90143.889.195.1212.8259.3389.72306.67852.8981.31627.019504.9
P95150.393.6102.9237.7272.9428.72360.77979.61040.81735.320107.2
P99180.4122.9135.6276.6303.7524.22638.58207.910022.819862.740255.3
P999230.1847.71274.41786.81233.12849.56275.412158.013160.225021.378451.6
Maximum1136.21331.01281.91834.71486.34862.19393.015139.813355.925964.282171.7
Time Per Output Token (TPOT) (ms)
Minimum1.561.661.922.083.183.143.513.697.5811.413.3
Average2.072.592.993.323.994.355.005.3123.039.361.0
P502.072.582.983.323.984.344.995.3023.139.561.6
P902.202.743.173.534.274.595.355.6124.041.164.5
P952.242.793.223.594.354.665.465.7124.241.665.4
P992.322.893.333.714.504.825.645.9124.742.567.0
P9992.403.023.443.844.665.295.866.9025.443.869.1
Maximum2.515.613.544.084.945.856.477.9229.249.993.8
Query Latency (ms)
Minimum956.11251.41349.01599.81698.41973.23532.54017.83683.414011.032781.3
Average2827.83514.74058.94515.25338.15870.88370.914219.730559.852258.497398.8
P502790.03458.73996.94444.25253.95778.88260.114101.930105.351507.896231.8
P903571.94437.05136.45715.06809.77436.810245.116217.838697.866449.6119908.1
P953833.24770.25509.26127.67340.58005.510914.416925.841594.971489.6128134.1
P994456.15530.46352.47088.68500.49266.612411.418577.947839.982406.7148162.2
P9995443.36701.47639.58547.810376.811300.915024.021202.857792.498440.8179564.6
Maximum6807.19279.010011.111613.814518.715989.221066.026505.885072.4158436.3256408.6