This tutorial walks through running LLM inference with vLLM on a Snellius by efficiently batching and streaming multiple model requests concurrently, maximizing GPU utilization and your budget.
Code can be found here: https://github.com/SURF-ML/vllm-inference-slurm
1. Retrieve container
The inference code utilizes the vLLM container from NVIDIA. As Docker is not supported by Snellius, we will utilize Apptainer. For more information on Apptainer, please refer to our Apptainer tutorial here.
This container includes packages like PyTorch, Triton, Transformers(engine), Flash Attention, Flash Infer XFormers and vLLM. In addition, the container recipe and prebuilt container also adds datasets and weights and biases.
Use prebuilt container
On Snellius there is a prebuilt container located at path /projects/2/managed_datasets/containers/vllm/vllm.sif
Set CONTAINER=/projects/2/managed_datasets/containers/vllm/vllm.sif as noted below.
The container has the following major packages and can be checked with
apptainer exec /projects/2/managed_datasets/containers/vllm/vllm.sif pip list
Some LLMs do not work well with this official NVIDIA vLLM container. Consider using a less-optimized but more robust CUDA container with manual vLLM installation:
CONTAINER=/projects/2/managed_datasets/containers/vllm/cuda-13.0-vllm.sif
Build and extend container
Although the vLLM container by NVIDIA provides all code to run various generative models including LLMs, VLMS, LM for embedding etc, you might want to extend the container by adding more packages which were not in the list of packages below.
For that, a build script has been provided here. Please add your packages in that script like:
%post
pip install datasets wandb <add your packages here>
# you can also add apt-get install -y <packages> like ffmpeg, git, etc.
...
After adding the packages in the apptainer build script, build the container as follows:
sbatch jobs/build_vllm_apptainer.job
2. Set up task variables
Environment variables
In jobs/run_vllm_serve.job, the task and experiments are set and passed to src/vllm_serve.py.
First set up the containers and directories:
# Path to .sif apptainer. Use your own container or the prebuilt container below CONTAINER_PATH=/projects/2/managed_datasets/containers/vllm/vllm_25.09.sif # In case data is on project space define this such that apptainer binds the project space PROJECT_SPACE=
To run a particular LLM or VLM, please specify the variables below. The defaults here are for machine translating the math dataset GSM8K with the LLM-based machine translation model Tower Plus 2B
MODEL_CHECKPOINT=Unbabel/Tower-Plus-2B DATASET=openai/gsm8k TEMPLATE_PRESET=gsm8k PORT=8000 DATA_SPLIT=test[:100] # only load 100 samples and run inference on them VLLM_BASE_URL=http://localhost:$PORT/v1 TEMPERATURE=0.7 MAX_TOKENS=256 MAX_CONCURRENT=64 OUTPUT_JSON=predictions.json
Run inference
To start the SLURM job, please specify the correct job details at jobs/run_vllm_serve.job
#!/bin/bash #SBATCH --job-name=vllm_inference #SBATCH --partition=gpu_a100 # or gpu_h100 #SBATCH --nodes=1 #SBATCH --ntasks=1 # equal to gpus-per-node #SBATCH --gpus-per-node=1 # 1-4 GPUs per node #SBATCH --time=02:00:00
Start the job in the queue with:
sbatch jobs/run_vllm_serve.job
Or interactively:
# salloc a GPU... chmod +x jobs/run_vllm_serve.job ./jobs/run_vllm_serve.job
3. Understanding the vLLM inference script
Templates
The script supports multiple dataset templates via the TEMPLATES dictionary:
For machine translation, tower is used. Please add your own instruction-based template
TEMPLATES = {
"gsm8k": "Solve this math problem step by step:\n\n{question}",
"alpaca": "{instruction}",
"squad": "Answer the following question based on the context.\n\nContext: {context}\n\nQuestion: {question}",
"mmlu": "Answer the following multiple choice question:\n\n{question}",
"tower": "Translate the following {source_lang} source text to {target_lang}:\n{source_lang}: {text}\n{target_lang}: ",
"default": "{text}",
}
Each template defines how dataset fields are formatted into a model prompt.
You can either:
Pass a custom template with
--instruction_template, orSelect one of the presets with
--template_preset tower,--template_preset gsm8k, etc.
Async efficient inference
vLLM supports handling multiple requests at the same time. By spawning the vLLM server first in jobs/run_vllm_serve.job:
apptainer exec --nv \
-B "${BIND_DIRS}" \
"${CONTAINER_PATH}" \
vllm serve $MODEL_CHECKPOINT \
--tensor-parallel-size $SLURM_GPUS_ON_NODE \
--download-dir $DOWNLOAD_DIR \
--uvicorn-log-level warning \
--port $PORT &
The vLLM server is started within the Apptainer but with the hostname being visible from the host (your terminal). The vLLM server is persistent such that
Loads the model once into GPU memory.
Handles incoming generation requests over HTTP (using the OpenAI API format).
Manages batching, scheduling, and token streaming efficiently across requests.
Then, the Python script connects to this server using the AsyncOpenAI client and sends requests asynchronously to:
Avoid reloading the model for each inference call, saving huge startup time.
Allow concurrency: many samples can be processed in parallel via
asyncio.Enable distributed workflows: multiple clients or nodes can query the same model server.
async def generate_predictions(...):
semaphore = asyncio.Semaphore(max_concurrent)
async def bounded_process(item, idx):
async with semaphore:
return await process_item(item, idx)
tasks = [bounded_process(item, idx) for idx, item in enumerate(dataset)]
results = await asyncio.gather(*tasks)
This uses:
asyncioto run multiple inference requests concurrently.A semaphore to cap concurrency (
--max_concurrent), preventing overload while keeping all GPUs busy.Non-blocking I/O with the
AsyncOpenAIclient, ideal for large datasets or parallel evaluation.
The result: you achieve much higher throughput than sequential requests, fully utilizing the GPU and your budget