Back previous/next report: k j

ND MI300X-v5

Submitted by Azureon 2026-10-28 Published on 2026-11-15

Result statement

Azure ran GPT-OSS 120B on ND MI300X-v5 and achieved a provisional 36,723.9 tokens/second System TPS at 11,851.1ms TTFT P90 in the Standardized division of the Available category on November 15, 2026. Pending MLCommons Association verification, results are subject to change pending peer review. .

[1] MLPerf Endpoints v1.0 GPT-OSS 120B, published November 15, 2026; retrieved from results.mlcommons.org on November 18, 2026. Result —, run 7: concurrency = 4,096 queries, TTFT P90 = 11,851.1ms, System TPS = 36,723.9 tokens/sec, interactivity = 9.0 tokens/sec per user and total provisioned system power = 10.05 kW (MLC Estimated Power). The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Submission Info
Azure
d85bf912-34c1-406b-a00d-010df722c645
Submission Status REVIEW_PENDING
2026-10-28
2026-11-15
Version v1.0
Division Standardized
Deployment Client on Premises
Model
Model Name GPT-OSS 120B
Division Standardized
Model Precision —
Model Link —
Calibration Link —
Model Notes —
Datasets
Dataset Name perf
Dataset Type performance
Average Input Tokens —
Dataset Link —

System TPS vs. Tokens/s per User

10k20k30kSystem TPS050100Tokens/s per User

System TPS vs. Concurrency

10k20k30kSystem TPS02,0004,0006,000# of queries

Time to First Token P90 vs. Concurrency

0500k1.0MTTFT P90 (ms)02,0004,0006,000# of queries

Tokens/s per User vs. Concurrency

050100Tokens/s per User02,0004,0006,000# of queries
SUT Summary
Intel
ND MI300X-v5
Available
System Category — · —
8 accelerators
1
1
Max Supported Concurrency 4,096
Node Type ID 0
Node Count 1
Software
Ubuntu
Framework vLLM 0.31.0 (image vllm/vllm-openai-rocm@sha256:749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f)
vLLM 0.31.0 (image vllm/vllm-openai-rocm@sha256:749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f)
ROCm 7.2.3 (in the vLLM image)
amdgpu 6.8.5
Link to Container vllm/vllm-openai-rocm@sha256:749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f
Operating System Ubuntu 22.04.5 LTS (kernel 5.15.0-1073-azure)
Filesystem xfs
Other Software —
Software Notes Same-host localhost TCP via Linux loopback, zero physical network hops. No benchmark-time baseline was captured. Supplemental standalone TCP echo RTT measured 2026-10-30T21:02:54.319555+00:00: mean 0.034468 ms, p90 0.038971 ms. Post-upgrade measurement, not historical vLLM HTTP latency; see docs/loopback_baseline.json and docs/software_disclosure.md. Serving recipe and optimization code provided by MangoBoost, adapted for Azure ND MI300X-v5. See docs/software_disclosure.md for attribution and software-stack details.
Hardware

Processor

Intel(R) Xeon(R) Platinum 8480C
2
48
VCPUs Per Processor —

Accelerator

—
0
—
—
—
—

Host / Storage

1.8 TiB
Memory Configuration 1,850 GiB VM memory per Azure specification; physical host DIMM count, capacity, type and speed are undisclosed by the VM.
Storage Capacity 32.0 TB in 10 drives
Storage Type NVMe SSD

Networking

8: 8x Mellanox ConnectX Family mlx5Gen Virtual Function (1 port)
Ethernet (client runs on the same node)
Network Topology —

Other

Other Hardware —
Cooling air-cooled
Hardware Notes —
Run Data
Field NameRun 1Run 2Run 3Run 4Run 5Run 6Run 7Run 8
Run Date10/28/202610/28/202610/28/202610/28/202610/28/202610/28/202610/28/202610/28/2026
Concurrency32481282561,0242,0484,0966,396
System Tokens/Second4,026.44,832.28,158.611,553.122,543.628,690.836,723.936,354.4
Tokens/Second per User125.8100.763.745.122.014.09.05.7
TTFT P90 (ms)133.1219.6231.1244.0297.8373.511851.11260278.8
Utilization11.0%13.2%22.2%31.5%61.4%78.1%100.0%99.0%
Configuration Summarygpt-oss-120b-conc32-accuracy · online · c32gpt-oss-120b-conc48-accuracy · online · c48gpt-oss-120b-conc128-accuracy · online · c128gpt-oss-120b-conc256 · online · c256gpt-oss-120b-conc1024 · online · c1024gpt-oss-120b-conc2048 · online · c2048gpt-oss-120b-conc4096-accuracy · online · c4096gpt-oss-120b-offline-accuracy · offline
Config Typeonlineonlineonlineonlineonlineonlineonlineoffline
Config Region————————
Target QPS————————
Result Statecompletecompletecompletecompletecompletecompletecompletecomplete
Measured Accuracy Score————————
Benchmark Versiond3c57f2d3c57f2d3c57f2d3c57f2d3c57f2d3c57f2d3c57f2d3c57f2
Result Version0.1.00.1.00.1.00.1.00.1.00.1.00.1.00.1.0
Archive URIgs://mlperf-endpoints-blob/runs/1b83b92d-ff6b-46f7-bacb-d38950676035/archive.tar.gzgs://mlperf-endpoints-blob/runs/38a2ae09-37c2-49c6-843f-9f6857449894/archive.tar.gzgs://mlperf-endpoints-blob/runs/f3aeb58e-41b8-4454-8875-9b24fd9fea1a/archive.tar.gzgs://mlperf-endpoints-blob/runs/18556bdd-ee87-4a20-8a97-1e35bd072359/archive.tar.gzgs://mlperf-endpoints-blob/runs/04125f73-9b03-4a5c-a5cc-425b8539bd7c/archive.tar.gzgs://mlperf-endpoints-blob/runs/10b962a5-f605-469b-9761-a9a8114c3831/archive.tar.gzgs://mlperf-endpoints-blob/runs/a3632f9c-1c1a-4605-848f-e86983201c53/archive.tar.gzgs://mlperf-endpoints-blob/runs/39b2657f-872f-4d63-bfe4-e60dbe9fbc12/archive.tar.gz
QPS2.89733.39765.73278.123215.857720.172825.787325.5324
Total Output Tokens44,443,37445,483,61345,512,64845,482,79845,463,27654,580,24263,760,23554,641,771
Average Output Tokens1,389.721,422.251,423.161,422.231,421.621,422.251,424.111,423.85
Run Duration (s)11,037.869,412.575,578.523,936.862,016.681,902.361,736.211,503.03
Total Queries31,98031,98031,98031,98031,98038,37644,77238,376
Total Queries Issued31,98031,98031,98031,98031,98038,37644,77238,376
Total Queries Failed00000000
Time to First Token (TTFT) (ms)
Minimum35.257.965.773.1102.1138.8190.02924.9
Average98.1162.2183.7215.8508.71081.34176.8675676.1
P5086.6152.1163.5174.2211.8259.4401.1679154.5
P90133.1219.6231.1244.0297.8373.511851.11260278.8
P95146.7256.5268.7284.0359.74621.825437.01334529.7
P99182.4340.5361.3409.411466.322932.647322.01390857.8
P999533.72022.04107.96632.216044.328211.253644.51406583.8
Maximum2718.23743.54562.97263.216628.829715.955158.71415293.2
Time Per Output Token (TPOT) (ms)
Minimum4.935.858.4213.118.125.532.430.9
Average7.889.8015.522.044.669.5105.8110.6
P507.889.8015.522.044.870.2108.1112.7
P908.0510.516.222.946.773.1112.9116.8
P958.1110.716.423.247.274.2114.0122.3
P998.2211.116.923.847.976.2118.6151.3
P9998.4411.517.424.349.880.7124.2196.6
Maximum9.5213.518.826.553.485.3134.2240.3
Query Latency (ms)
Minimum4224.95022.27071.111459.424752.734889.445309.0106061.3
Average11040.214119.222285.831435.263873.299829.9154745.9832932.1
P5010879.613864.021938.130975.962870.898410.9152465.2837713.2
P9013833.318046.528284.539797.081432.8127624.6200322.21414748.5
P9514822.519506.330433.642964.787444.7137023.2215946.11453365.0
P9917116.722637.235287.749810.9100875.9157664.9246988.51476983.4
P99920203.227441.941710.560180.5119407.9186266.9288480.91488641.9
Maximum26157.640138.453951.377059.7164035.0266252.0346492.21498337.6