vLLM Inference details

Under the Hood

When you specify a multi-GPU preset (like a preset with multiple Nvidia and AMD GPUs), LLM Inference:

1. Determines GPU Provider & Count

   • AMD MI210 → gpuProvider=amd, 2 GPUs.

   • NVIDIA → gpuProvider=nvidia, 2 GPUs.

2. Sets GPU Visibility

   • On AMD: HIP_VISIBLE_DEVICES=0,1, ROCR_VISIBLE_DEVICES=0,1 (if 2 GPUs).

   • On NVIDIA: CUDA_VISIBLE_DEVICES=0,1.

3. Applies a Default Parallel arguments (e.g. --tensor-parallel-size=2) if the user hasn’t already done so.

Environment Variables for AMD

By default, if you select a preset with AMD GPU cards, the chart’s logic sets:

• HIP_VISIBLE_DEVICES=0,1... and ROCR_VISIBLE_DEVICES=0,1... depending on number of available GPUs: Tells ROCm which GPUs are accessible.

• TORCH_USE_HIP_DSA=1: Enables direct storage access for HIP.

• HSA_FORCE_FINE_GRAIN_PCIE=1 & HSA_ENABLE_SDMA=1: Improves GPU ↔ Host & GPU ↔ GPU memory transfers.

• ROCM_DISABLE_CU_MASK=0: All compute units remain active.

• VLLM_WORKER_MULTIPROC_METHOD=spawn: Avoids “fork” issues on AMD.

• NCCL_P2P_DISABLE=0: By default, we assume your cluster has correct kernel parameters for GPU–GPU direct memory access. If not, you can pass --set envAmd.NCCL_P2P_DISABLE=1 to forcibly disable P2P.

Final Notes

• NCCL / RCCL logs will appear in the vLLM container logs. Look for lines referencing peer-to-peer if you see a hang.

• On AMD, if you do have persistent hangs, append --set "envAmd.NCCL_P2P_DISABLE=1" to your Apolo command to force fallback GPU communication.

• For the best performance, we keep ROCm version 6.2+ or 6.3+ in sync with your Docker image (rocm/vllm-ci)

By combining the right Apolo preset, environment variables, you can reliably run vLLM on multiple GPUs—be they AMD or NVIDIA —and get high token throughput for large LLMs.