Back previous/next report: k j
HPE Cray XD670 (8x H200-SXM-141GB, vLLM)
Submitted by Kraion 2026-11-10 Published on 2026-11-10
Result statement
Krai ran deepseek-v4_1-flash on HPE Cray XD670 (8x H200-SXM-141GB, vLLM) and achieved a provisional 12,696.3 tokens/second System TPS at 2,038.1ms TTFT P90 in the Standardized division of the Available category on November 10, 2026. Pending MLCommons Association verification, results are subject to change pending peer review. .
[1] MLPerf Endpoints v1.0 deepseek-v4_1-flash, published November 10, 2026; retrieved from endpoints.mlcommons.org on November 13, 2026. Result —, run 7: concurrency = 1,024 queries, TTFT P90 = 2,038.1ms, System TPS = 12,696.3 tokens/sec, interactivity = 12.4 tokens/sec per user and total provisioned system power = 9.73 kW (MLC Estimated Power). The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.
Krai
647657c4-6b54-4085-aaae-105b273580bf
Submission Status REVIEW_PENDING
2026-11-10
2026-11-10
Version v1.0
Division Standardized
Deployment Client on Premises
Model Name deepseek-v4_1-flash
Division Standardized
Model Precision —
Model Link —
Calibration Link —
Model Notes —
Dataset Name agentic_combined
Dataset Type performance
Average Input Tokens —
Dataset Link —
System TPS vs. Tokens/s per User
System TPS vs. Concurrency
Time to First Token P90 vs. Concurrency
Tokens/s per User vs. Concurrency
HPE Cray XD670 (8x H200-SXM-141GB, vLLM)
Available
System Category Datacenter · —
8x H200-SXM-141GB
1
1
Max Supported Concurrency 1,024
Node Type ID 0
Node Count 1
Framework vLLM v0.32.0 (5df63a72) with a custom patch (shared src: src/benchmarking/docker/patches/v0.32.0)
vLLM v0.32.0 (5df63a72) with a custom patch (shared src: src/benchmarking/docker/patches/v0.32.0)
CUDA 13.0, PyTorch 2.13.0, NCCL 2.30.7, FlashInfer 0.7.0, FlashMLA, DeepGEMM; Humming MoE GEMM (indexed, FP8 E4M3 activations)
NVIDIA 595.58.03 (CUDA 13.2)
Link to Container https://hub.docker.com/r/wongcannan/vllm-v0.32.0-patched-nsys-withsrc (v1.0@sha256:e554ced380fe37cc4fa9be6f6ffa7a0a8f79f675f0e21f1ecb8e1b095a8e9adf), https://hub.docker.com/r/wongcannan/mlperf-endpoints (v1.1@sha256:514e6360b981f2f59300a796f84585edaa4c6401dad352b062d4038dd9b32ea4), https://hub.docker.com/_/nginx (1.27-alpine)
Operating System Red Hat Enterprise Linux 9.4 (kernel 5.14.0-427.13.1.el9_4.x86_64)
Filesystem XFS, Lustre (HPE Cray ClusterStor E1000)
Other Software Docker 27.4.1, NVIDIA Container Toolkit 1.17.3, MLPerf Endpoints client d3c57f2, nginx 1.27.5
Software Notes Powered by Krai/Gentic, next-gen automation
Processor
Intel(R) Xeon(R) Platinum 8562Y+
2
32
VCPUs Per Processor 64
Accelerator
—
0
—
—
—
—
Host / Storage
2 TiB
Memory Configuration 32x HPE Cray Supercomputing XD 64GB (1x64GB) Dual Rank x4 DDR5-5600 Registered Standard Memory
Storage Capacity 15.36 TB SSD (8x 1.92 TB)
Storage Type HPE Cray XD670 1.92 TB NVMe Read Intensive SFF U.3 7450 SSD
Networking
8x NVIDIA ConnectX-7 400Gb InfiniBand NDR, 2x dual-port NVIDIA ConnectX-6 Ethernet (2x 100Gb bonded, 2x 40Gb), 1x Intel X710 10GBASE-T
InfiniBand, Ethernet
Network Topology —
Other
Other Hardware —
Cooling Air-cooled
Hardware Notes 6x Delta Titanium 3,600-Watt power supplies with N+2 redundancy; GPUs at the default 700 W power limit.
| Field Name | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 |
|---|---|---|---|---|---|---|---|
| Run Date | 11/10/2026 | 11/10/2026 | 11/10/2026 | 11/10/2026 | 11/10/2026 | 11/10/2026 | 11/10/2026 |
| Concurrency | 32 | 40 | 64 | 128 | 256 | 512 | 1,024 |
| System Tokens/Second | 3,307.9 | 3,780.6 | 4,599.0 | 5,588.7 | 7,653.7 | 10,076.9 | 12,696.3 |
| Tokens/Second per User | 103.4 | 94.5 | 71.9 | 43.7 | 29.9 | 19.7 | 12.4 |
| TTFT P90 (ms) | 251.2 | 264.6 | 307.3 | 694.4 | 951.6 | 1330.0 | 2038.1 |
| Utilization | 26.1% | 29.8% | 36.2% | 44.0% | 60.3% | 79.4% | 100.0% |
| Configuration Summary | multireplica-v1.0.1__M_dsv41f__BS_long__SC_endpoints__P_nvidia__C_32__vanilla__tag_v032-opt__param_eval/experiment_0-0001 · online · c32 | multireplica-v1.0.1__M_dsv41f__BS_long__SC_endpoints__P_nvidia__C_40__vanilla__tag_v032-opt__param_eval/experiment_0-0001 · online · c40 | multireplica-v1.0.1__M_dsv41f__BS_long__SC_endpoints__P_nvidia__C_64__vanilla__tag_v032-opt__param_eval/experiment_0-0001 · online · c64 | multireplica-v1.0.1__M_dsv41f__BS_long__SC_endpoints__P_nvidia__C_128__vanilla__tag_v032-opt__param_eval/experiment_0-0001 · online · c128 | multireplica-v1.0.1__M_dsv41f__BS_long__SC_endpoints__P_nvidia__C_256__vanilla__tag_v032-opt__param_eval/experiment_0-0001 · online · c256 | multireplica-v1.0.1__M_dsv41f__BS_long__SC_endpoints__P_nvidia__C_512__vanilla__tag_v032-opt-g2048__param_eval/experiment_0-0001 · online · c512 | multireplica-v1.0.1__M_dsv41f__BS_long__SC_endpoints__P_nvidia__C_1024__vanilla__tag_v032-opt__param_eval/experiment_0-0001 · online · c1024 |
| Config Type | online | online | online | online | online | online | online |
| Config Region | — | — | — | — | — | — | — |
| Target QPS | — | — | — | — | — | — | — |
| Result State | complete | complete | complete | complete | complete | complete | complete |
| Measured Accuracy Score | — | — | — | — | — | — | — |
| Benchmark Version | d3c57f2 | d3c57f2 | d3c57f2 | d3c57f2 | d3c57f2 | d3c57f2 | d3c57f2 |
| Result Version | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 |
| Archive URI | gs://mlperf-endpoints-blob/runs/dd82f8b2-640b-4015-aae6-2aa3cfbfbb59/archive.tar.gz | gs://mlperf-endpoints-blob/runs/dc26c176-41cd-4ef2-81fc-0d93823825a8/archive.tar.gz | gs://mlperf-endpoints-blob/runs/53e70ba8-dcdf-4f3d-a295-7c0e3936e687/archive.tar.gz | gs://mlperf-endpoints-blob/runs/c3dc8fe3-a91e-4f9b-a9a0-b3c5ed9c1bf4/archive.tar.gz | gs://mlperf-endpoints-blob/runs/34bf1001-b5a8-4494-8d81-88fdba0ac4ff/archive.tar.gz | gs://mlperf-endpoints-blob/runs/b3c071ec-a1b9-44a4-8692-4d114f5b27b3/archive.tar.gz | gs://mlperf-endpoints-blob/runs/925fd103-b184-4e20-b9a8-c1adea6cf89f/archive.tar.gz |
| QPS | 3.7417 | 4.1818 | 5.2252 | 6.1312 | 8.4103 | 11.2692 | 14.0398 |
| Total Output Tokens | 16,681,644 | 16,772,313 | 15,796,118 | 14,711,039 | 28,817,152 | 58,154,831 | 116,304,451 |
| Average Output Tokens | 884.08 | 904.07 | 880.15 | 911.52 | 910.03 | 894.19 | 904.3 |
| Run Duration (s) | 5,042.90 | 4,436.41 | 3,434.72 | 2,632.26 | 3,765.12 | 5,771.11 | 9,160.52 |
| Total Queries | 18,869 | 18,552 | 17,947 | 16,139 | 31,666 | 65,036 | 128,612 |
| Total Queries Issued | 18,869 | 18,552 | 17,947 | 16,139 | 31,666 | 65,036 | 128,612 |
| Total Queries Failed | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Time to First Token (TTFT) (ms) | |||||||
| Minimum | 64.2 | 71.5 | 86.9 | 80.6 | 109.9 | 146.5 | 227.1 |
| Average | 171.6 | 185.2 | 226.9 | 441.3 | 586.1 | 854.8 | 1379.0 |
| P50 | 130.0 | 138.0 | 160.8 | 263.6 | 328.5 | 445.1 | 660.9 |
| P90 | 251.2 | 264.6 | 307.3 | 694.4 | 951.6 | 1330.0 | 2038.1 |
| P95 | 370.3 | 403.2 | 538.9 | 1093.3 | 1574.3 | 2367.8 | 3795.8 |
| P99 | 620.1 | 689.3 | 1075.6 | 3982.4 | 5888.4 | 9459.5 | 16808.7 |
| P999 | 4608.6 | 5395.6 | 5702.4 | 12563.4 | 18921.5 | 32092.3 | 58846.1 |
| Maximum | 9544.2 | 11282.8 | 12146.1 | 42692.9 | 45950.8 | 114337.4 | 247260.8 |
| Time Per Output Token (TPOT) (ms) | |||||||
| Minimum | -0.02 | -0.02 | -0.02 | 0.01 | 0.03 | 0.03 | 0.02 |
| Average | 6.27 | 7.04 | 9.64 | 17.4 | 26.4 | 40.6 | 65.2 |
| P50 | 6.34 | 7.12 | 9.74 | 17.8 | 26.9 | 41.8 | 67.5 |
| P90 | 8.69 | 9.69 | 13.3 | 22.9 | 34.3 | 52.3 | 83.5 |
| P95 | 9.36 | 10.5 | 14.3 | 25.3 | 38.2 | 57.8 | 91.4 |
| P99 | 11.1 | 12.4 | 17.0 | 34.1 | 57.5 | 86.4 | 123.3 |
| P999 | 14.6 | 15.5 | 23.4 | 94.9 | 126.1 | 121.8 | 147.8 |
| Maximum | 62.4 | 68.1 | 70.1 | 183.1 | 191.2 | 176.4 | 331.7 |
| Query Latency (ms) | |||||||
| Minimum | 180.9 | 191.7 | 260.9 | 341.8 | 440.5 | 636.0 | 900.6 |
| Average | 6816.1 | 7797.8 | 10357.5 | 18291.7 | 27453.4 | 41741.5 | 68019.2 |
| P50 | 1145.6 | 1292.8 | 1734.0 | 3079.2 | 4586.7 | 6936.9 | 11167.8 |
| P90 | 15800.5 | 19030.7 | 24337.0 | 45530.3 | 69372.1 | 104091.5 | 171476.8 |
| P95 | 49861.4 | 57123.2 | 72842.9 | 141977.8 | 218198.9 | 329108.9 | 543016.2 |
| P99 | 72476.7 | 80791.7 | 110513.5 | 181365.8 | 269878.3 | 415835.0 | 672203.0 |
| P999 | 77270.0 | 87171.2 | 117562.9 | 200683.2 | 287361.4 | 440045.6 | 711408.1 |
| Maximum | 82318.8 | 93274.7 | 121959.1 | 214906.8 | 318418.2 | 489908.8 | 806821.2 |