# Multi-GPU Benchmarks Report 1\. Introduction We conducted a series of **vLLM** inference benchmarks to evaluate performance under different **GPU presets**, **model sizes**, and **parallelization strategies** (pipeline vs. tensor). We focused on: * **Prompt tokens/s** and **generation tokens/s** (throughput). * **Requests** in different states (running, swapped, waiting). * **KV-cache usage** on **GPU** and **CPU**. * **Average request latencies**. * **Error counts** (e.g., OOM or network-related failures). These tests spanned a range of **context lengths** (from **2048** up to **128k** tokens) and **model sizes** (e.g., 1.5B, 8B, 32B). We also **compared pipeline parallel** with **tensor parallel** deployments to see how different parallelization strategies impacted throughput. ### 2. Environment & Setup #### 2.1 GPU Presets We used the following GPU presets, each with distinct CPU, memory, and GPU counts/types: | **Preset** | **vCPUs** | **Memory** | **VRAM** | **GPU Count** | **GPU Type** | | ---------- | --------- | ---------- | -------- | ------------- | ----------------------- | | gpu-small | 30.0 | 63.0 GB | 16 GB | 1 | NVIDIA V100 (PCIe 16GB) | | gpu-medium | 60.0 | 126.0 GB | 32 GB | 2 | NVIDIA V100 (PCIe 16GB) | | gpu-large | 120.0 | 252.0 GB | 64 GB | 4 | NVIDIA V100 (PCIe 16GB) | | gpu-xlarge | 120.0 | 504.0 GB | 128 GB | 8 | NVIDIA V100 (PCIe 16GB) | | mi210x1 | 15 | 65.0 GB | 64 GB | 1 | AMD MI210 | | mi210x2 | 30 | 130.0 GB | 128 GB | 2 | AMD MI210 | | H100X1 | 63.0 | 265.0 GB | 80 GB | 1 | NVIDIA H100 (PCIe 80GB) | | H100X2 | 126.0 | 530.0 GB | 160 GB | 2 | NVIDIA H100 (PCIe 80GB) | Each preset was deployed via a Helm-based “apolo” flow. #### 2.2 Models We tested multiple Hugging Face models: * **`deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`**` `` ``(smaller ~1.5B params) ` * **`meta-llama/Llama-3.2-3B-Instruct`**` `` ``(~3B params) ` * **`meta-llama/Llama-3.1-8B-Instruct`**` `` ``(~8B params) ` * **`deepseek-ai/DeepSeek-R1-Distill-Qwen-32B`**` `` ``(~32B params) ` We then varied **context length** from 2048 up to 128k tokens to measure how throughput, memory usage, and latencies scaled with input size. #### 2.3 Benchmark Script & Methodology A custom Python script: 1. **Deploys** each (preset, model) combination,\ With the following arguments:\ `--host=0.0.0.0 --port=8000 --model=meta-llama/Llama-3.1-8B-Instruct --code-revision=main --tokenizer= --tensor-parallel-size= --dtype=half --max-model-len= --enforce-eager --trust-remote-code`\ For context length we use a different length for each run and for tensor-parallel-size we use the number of GPUs available on the preset 2. **Waits** until the endpoint is ready (`/v1/models` returning 200), 3. **Sends** load (e.g., 100 requests) at concurrency=1, * 100 requests @10req at a time to the `/v1/completions` endpoint * Uses a fixed prompt: "Let's explore some architecture patterns for microservices" * Configures max\_tokens=512/2048 and temperature=0.7 4. **Polls** `/metrics` every second for: * `prompt_tokens_total` & `generation_tokens_total` => used to compute tokens/s * `num_requests_running`, `num_requests_swapped`, `num_requests_waiting` * `gpu_cache_usage_perc`, `cpu_cache_usage_perc` 5. **Tracks** per-request latencies and errors, 6. **Continues** polling until all requests are finalized (no pending/running/swapped), 7. **Writes** aggregated metrics (averages) into a CSV file, 8. **Generates** bar charts for each metric. We repeated these steps for **pipeline parallel** vs. **tensor parallel** where relevant. ### 3. Overview of Parallel Strategies 1. **Pipeline Parallel**: Splits the model layers into stages across multiple GPUs so that each GPU processes a portion of layers in sequence. 2. **Tensor Parallel**: Splits *tensors* (e.g., weight matrices) across multiple GPUs in a more fine-grained way so the same layers are effectively distributed among GPUs. **In general**, **tensor parallel** is often more efficient for large or similarly sized GPUs, whereas **pipeline parallel** can help in some multi-GPU cases but can introduce significant inter-stage waiting time and memory overheads, especially if each pipeline stage has different computational loads. Below are **side-by-side tables** comparing **pipeline parallel** vs. **tensor parallel** for **Llama-3B at 2048 context length** —where we have overlapping data. Each cell shows **(Prompt TPS, Generation TPS)**. After each table, we list **observations** for that model. #### **Llama-3.2-3B-Instruct - 2048 context length @10req concurrency for 100 total requests**
GPU PresetPipeline (Prompt TPS)Tensor (Prompt TPS)Pipeline (Gen TPS)Tensor (Gen TPS)Prompt Speedup (%)Gen Speedup (%)
H100X147.6447.35564.79573.80-0.6%1.6%
H100X236.1440.98449.28500.4313.4%11.4%
gpu-medium34.1743.03416.18521.9225.9%25.4%
gpu-small47.2247.80581.37576.271.2%-0.9%
mi210x147.7946.29581.30574.77-3.1%-1.1%
mi210x234.2040.59425.94489.3018.7%14.9%
**Observations (Llama-3B, 2048 ctx, Pipeline vs. Tensor)** * **H100X1**: \~-0.6% prompt speedup, \~1.6% gen speedup. * **H100X2**: \~13.4% prompt speedup, \~11.4% gen speedup. * **gpu-medium**: \~25.9% prompt speedup, \~25.4% gen speedup. * **gpu-small**: \~1.2% prompt speedup, \~-0.9% gen speedup. * **mi210x1**: \~-3.1% prompt speedup, \~-1.1% gen speedup. * **mi210x2**: \~18.7% prompt speedup, \~14.9% gen speedup. Overall, we notice that: * For small models, if they fit on a single GPU, splitting them across multiple GPUs doesn't help. It actually slows down inference. * The tensor parallel size split is faster than the pipeline parallel size split on multi-gpu setups, which is expected. ### 4. Results & Observations Below we break down **context length** runs and highlight models performance on specific configurations #### 4.1 2048-Token Context Benchmarks Below we have three models: **Qwen-1.5B**, **Llama-3B**, **Llama-8B**, and **Qwen-32B**. **4.1.1 Qwen-1.5B - 2048 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X145.74649.786.9571.645.100
H100X237.44521.108.4757.644.330
gpu-large38.09535.918.3558.934.430
gpu-medium39.73552.697.9661.434.680
gpu-small46.63676.106.7974.315.210
mi210x143.73618.487.2468.465.060
mi210x29.35125.168.8455.094.1310
**Observations (Qwen-1.5B, 2048 ctx)**: * Highest **Prompt TPS**: `gpu-small` with 46.63 tokens/s * Highest **Gen TPS**: `gpu-small` with 676.10 tokens/s * Notable errors on: mi210x2 (10 errs) **4.1.2 Llama-3B - 2048 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X19.12105.127.9564.265.3610
H100X240.23500.807.8355.764.550
gpu-large41.73510.797.3758.004.840
gpu-medium43.03532.967.2459.754.910
gpu-small47.22576.566.4965.875.510
gpu-xlarge42.42510.527.3157.834.910
mi210x145.35570.086.7365.085.310
mi210x238.17475.528.3052.584.300
**Observations (Llama-3B, 2048 ctx)** * Highest **Prompt TPS**: `gpu-small` with 47.22 tokens/s * Highest **Gen TPS**: `gpu-small` with 576.56 tokens/s * Notable errors on: H100X1 (10 errs) **4.1.3 Llama-8B - 2048 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X128.07399.0711.6042.913.040
H100X25.0866.2611.0145.723.1920
gpu-large34.24492.479.5252.813.690
gpu-medium34.56491.249.1754.033.841
gpu-xlarge34.27488.519.4152.933.750
mi210x128.09407.7211.6643.223.020
mi210x231.46444.9310.1748.533.470
**Observations (Llama-8B, 2048 ctx)**: * Highest **Prompt TPS**: `gpu-medium` with 34.56 tokens/s * Highest **Gen TPS**: `gpu-large` with 492.47 tokens/s * Notable errors on: gpu-medium (1 errs), H100X2 (20 errs) **4.1.4 Qwen-32B - 2048 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X114.44123.7222.2713.421.950
H100X220.39175.2515.8818.872.700
gpu-xlarge8.2073.9912.0627.643.5310
**Observations (Qwen-32B, 2048 ctx)**: * Highest **Prompt TPS**: `H100X2` with 20.39 tokens/s * Highest **Gen TPS**: `H100X2` with 175.25 tokens/s * Notable errors on: gpu-xlarge (10 errs) #### 4.2 8192-Token Context Benchmarks #### 4.2.1 DeepSeek-R1-Distill-Qwen-1.5B **- 8192 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X123.80605.5512.7969.473.210
H100X220.07517.6815.3358.552.890
gpu-large20.76531.3915.2458.742.610
gpu-medium22.98551.0313.7761.232.970
gpu-small27.97666.1811.5671.973.870
mi210x124.30602.9512.6768.553.410
mi210x219.05469.0117.0250.652.560
**Observations** * Highest **Prompt TPS**: `gpu-small` with 27.97 tokens/s * Highest **Gen TPS**: `gpu-small` with 666.18 tokens/s #### 4.2.2 Llama-3.2-3B-Instruct **- 8192 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X137.67522.518.2858.674.700
H100X238.42496.428.2355.054.630
gpu-large40.86523.267.8257.394.850
gpu-medium38.88531.078.2358.164.700
gpu-small38.51521.737.2265.375.230
gpu-xlarge36.69512.538.5556.854.640
mi210x137.78545.317.6664.915.361
mi210x229.98421.539.1153.834.290
**Observations** * Highest **Prompt TPS**: `gpu-large` with 40.86 tokens/s * Highest **Gen TPS**: `mi210x1` with 545.31 tokens/s * Notable errors on: mi210x1 (1 errs) #### 4.2.3 Llama-8B **- 8192 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X18.27369.3323.5639.562.1863
H100X27.83382.4626.8241.912.0564
gpu-large10.82493.5530.1252.871.780
gpu-medium10.18503.8732.1553.541.530
gpu-xlarge10.79494.3030.4452.601.740
mi210x18.10397.2140.7941.651.250
mi210x210.57455.3430.6348.881.810
**Observations (Llama-8B, 8192 ctx)** * Highest **Prompt TPS**: `gpu-large` with 10.82 tokens/s * Highest **Gen TPS**: `gpu-medium` with 503.87 tokens/s * Notable errors on: H100X1 (63 errs), H100X2 (64 errs) #### 4.2.4 Qwen-32B **- 8192 context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X111.49116.3422.6013.391.9114
H100X217.82160.9316.1718.632.892
gpu-xlarge25.81229.4111.2827.483.970
**Observations** * Highest **Prompt TPS**: `gpu-xlarge` with 25.81 tokens/s * Highest **Gen TPS**: `gpu-xlarge` with 229.41 tokens/s * Notable errors on: H100X1 (14 errs), H100X2 (2 errs) #### 4.3 64k-Token Context Benchmarks #### 4.3.1 Qwen-1.5B **-** 64k **context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X124.50621.0613.0468.123.350
H100X27.58181.1614.9657.852.8510
mi210x124.49609.7812.6868.153.290
mi210x218.45502.8817.1254.942.480
**Observations** * Highest **Prompt TPS**: `H100X1` with 24.50 tokens/s * Highest **Gen TPS**: `H100X1` with 621.06 tokens/s * Notable errors on: H100X2 (10 errs) #### 4.3.2 Llama-3.2-3B **-** 64k **context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X145.85570.006.7464.605.400
H100X240.34501.897.8055.934.710
mi210x140.51595.077.8565.215.220
mi210x230.43419.559.0453.194.300
**Observations** * Highest **Prompt TPS**: `H100X1` with 45.85 tokens/s * Highest **Gen TPS**: `mi210x1` with 595.07 tokens/s #### 4.3.3 Llama-3.1-8B **-** 64k **context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X18.35367.0519.5939.502.5365
H100X28.09386.2634.0441.591.6441
mi210x28.65451.6138.2447.711.260
**Observations** * Highest **Prompt TPS**: `mi210x2` with 8.65 tokens/s * Highest **Gen TPS**: `mi210x2` with 451.61 tokens/s * Notable errors on: H100X1 (65 errs), H100X2 (41 errs) #### 4.3.4 DeepSeek-R1-Distill-Qwen-32B **-** 64k **context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X219.32176.1015.9919.062.912
**Observations** * Highest **Prompt TPS**: `H100X2` with 19.32 tokens/s * Highest **Gen TPS**: `H100X2` with 176.10 tokens/s * Notable errors on: H100X2 (2 errs) #### 4.4 128k-Token Context Benchmarks #### 4.4.1 Qwen-1.5B **- 128k context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X126.45642.5411.9870.883.450
H100X221.76536.2714.7758.372.870
mi210x124.13618.1713.1068.263.220
mi210x217.80494.4117.4954.642.400
**Observations** * Highest **Prompt TPS**: `H100X1` with 26.45 tokens/s * Highest **Gen TPS**: `H100X1` with 642.54 tokens/s #### 4.4.2 Llama-3.2-3B **- 128k context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X145.53574.226.8464.605.300
H100X230.05451.149.5055.154.410
mi210x144.17574.077.0864.165.300
mi210x232.06444.009.4251.154.190
**Observations** * Highest **Prompt TPS**: `H100X1` with 45.53 tokens/s * Highest **Gen TPS**: `H100X1` with 574.22 tokens/s #### 4.4.3 Llama-3.1-8B **- 128k context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X17.10360.4027.7540.102.0080
H100X28.38381.7128.3941.351.9549
mi210x210.08467.4332.7049.461.620
**Observations** * Highest **Prompt TPS**: `mi210x2` with 10.08 tokens/s * Highest **Gen TPS**: `mi210x2` with 467.43 tokens/s * Notable errors on: H100X1 (80 errs), H100X2 (49 errs) #### 4.4.4 DeepSeek-R1-Distill-Qwen-32B **- 128k context length @10req at a time for 100 total requests**
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSErrors
H100X218.96168.0516.0218.982.901
**Observations** * Highest **Prompt TPS**: `H100X2` with 18.96 tokens/s * Highest **Gen TPS**: `H100X2` with 168.05 tokens/s * Notable errors on: H100X2 (1 errs) ### 5. Overview of concurrency strategies Concurrency significantly impacts overall system throughput. You will observe that the request-level tokens per second (TPS) decreases as concurrency rises, while the total system throughput increases due to the vLLMs Paged Attention algorithm and other enhancements. This aspect can be adjusted for various trade-offs to boost system throughput, accommodate a reasonable number of request level tokens per second, and reduce errors.
PresetPrompt TPSGen TPSAvg Latency (s)Request Generation Level TPSRequest Prompt Level TPSConcurrencyErrors
H100X218.96168.0516.0218.982.90101
H100X23.6229.3610.202729.453.6210

Concurrency of 10req at a time

Concurrency of 1req at a time

### **Important notes** 1. The script uses a short, fixed prompt, which can lead to high generation throughput. 2. It employs sequential concurrent requests, maximizing GPU utilization. 3. The powerful hardware capabilities align with these performance figures. 4. You can squeeze a lot more from these presets with multiple model instances on same GPUs.