Serving Nemotron 3.5 Super VL 120B-A12B on Litespark Inference
Litespark Inference serves NVIDIA’s Nemotron 3.5 Super VL on two H200 GPUs, reaching 962 tokens per second per user in warm-start benchmarks.
Powerful open weights models only matter if people can serve them quickly and efficiently. For coding, multi-modal reasoning and long-horizon agentic workloads, model quality is only half of the experience. The other half is how soon the first token appears, how quickly the rest follow, and how much hardware it takes to get there.
NVIDIA’s Nemotron 3.5 Super VL 120B-A12B is a vision-language model that accepts text, image, and video inputs while producing text outputs. Built with BF16 precision, it uses Mixture of Experts (MoE) architecture with 120B total parameters of which 12B are active at a time.
Mindbeam AI is now serving Nemotron 3.5 Super with Litespark Inference. On two NVIDIA H200 GPUs, Litespark delivers 962 tokens per second per user and 947 tokens per second of output throughput in our warm-start benchmark.
The interesting part is not one unusually fast kernel. It is what happens when the entire path from request to token is treated as one performance problem.
How Litespark Inference achieves faster performance
No single optimization explains the result. Litespark improves the places where a serving stack can quietly lose time because of repeated setup, padded work, data movement, and recovery from speculative tokens that are not accepted.
Generation repeats much of the same GPU execution pattern for every token. Litespark keeps compatible recurring work on a streamlined path, reusing prepared execution and memory while refreshing the request state that changes. Requests outside an optimized path continue through normal execution, so specialization does not become a restriction.
Mixture of experts models activate only part of the network for a given token. A general purpose path can still spend time arranging, padding, or moving work that contributes little to the answer. Litespark keeps routing close to GPU execution, organizes useful expert work efficiently, and reduces unnecessary movement around those layers.
Speculative generation evaluates several likely next tokens together. The challenge is recovering cleanly when only part of a proposal is correct. Litespark commits verified progress while ensuring rejected proposals never become part of the accepted context.
Together, these choices reduce the small delays that accumulate across every generated token. That is why the gains appear not only in throughput, but also in TTFT, ITL, and end-to-end latency.
Benchmark Results: Litespark vs. vLLM and TensorRT-LLM
We compared Litespark with vLLM and TensorRT-LLM using the same model precision, KV format, workload, concurrency, and generation policy.
Cold start
Cold start measures from process launch through the first completed response, including initialization, model loading, kernel preparation, and graph setup. The one time model download is excluded.
Table 1: Cold start metrics
The cold start run tells the same story: Litespark leads across throughput, TTFT, ITL, and end-to-end latency.
Per-user throughput reflects the speed experienced by an individual request. Output throughput measures the total rate delivered by the server. TTFT, ITL, and E2E latency show how that performance feels from the first token through the completed response.
Warm start
Warm start measures steady state serving after model loading, preparation, and warm up.
Table 2: Warm start metrics
At warm start, Litespark delivers nearly 6.9× the per-user throughput of vLLM and 3.4× that of TensorRT-LLM with MTP.
Conclusion
Faster inference changes both the user experience and the economics of serving a model. Responses begin sooner. Tokens arrive more smoothly. The same workload requires fewer GPUs, leaving more room to scale users or control infrastructure cost.
Nemotron 3.5 Super VL shows how far open weights models have progressed in coding, multi-modal reasoning, and agentic work. Serving it with Litespark shows what becomes possible when the inference engine is just as ambitious.
