Artifact for A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines (Faizan Anwar and Sina Keshvadi, Thompson Rivers University).

The paper compares vLLM 0.6.3 and SGLang 0.4.1 serving Llama-3.1-8B-Instruct on one NVIDIA L4, under a multi-agent workload with shared prompt prefixes. Under each engine's defaults, vLLM's median time to first token is up to 64× SGLang's. Most of that gap comes from one default: SGLang chunks long prefills on this GPU and vLLM does not. Matching the chunk size removes 78% of the capacity gap and 30–67% of the latency gap. A 2.4–6.4× tail-latency advantage for SGLang remains.

This repository holds everything behind those numbers: the testbed, the workload generator, the campaign runners, the raw logs of all 99 runs, and the analysis that turns them into every table, figure and quoted number.

git clone https://github.com/FaizanAnwar01/llm-serving-config-eval
cd llm-serving-config-eval
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
make

make parses the 99 run logs into data/processed/, runs the paired statistics, and then runs analysis/check_claims.py, which recomputes the numbers quoted in the paper (107 checks, including every significance marker) and compares each with the printed value. It ends with All checks passed.

The pinned versions in requirements.txt produced the paper. The figures are pixel-identical to the paper's with these versions; matplotlib 3.11 draws them with small rendering differences but the same content.

Each row is a section of check_claims.py's output.

infra/terraform/       GKE cluster and platform (KServe, Prometheus, KEDA), two stages
infra/manifests/       engine InferenceServices for both arms, model cache, metrics monitor
wlgen/                 workload generator: trace compiler, validator, async dispatcher,
                       sweep and restart Job generators, metrics and event collectors
runners/               campaign runners, exactly as they ran, and stage_campaign.sh
analysis/              log parser, paired statistics, restart analysis, claims check
figures/               figure and table generators, shared style
data/                  raw logs, Prometheus extracts, restart events, ledgers (see data/README.md)
results/               generated figures, tables and restart summary
docs/PROVENANCE.md     where each file came from and the configuration evidence

Everything else is identical across arms: bf16, context 16,384, --gpu-memory-utilization=0.88 (vLLM), --mem-fraction-static=0.85 (SGLang), prefix caching on. docs/PROVENANCE.md gives the evidence for each setting.

This needs a Google Cloud project with quota for one NVIDIA L4, access to the gated meta-llama/Llama-3.1-8B-Instruct on Hugging Face, and Terraform, gcloud and kubectl. The full campaign is about 33 hours of measured load; the three conditions behind the decomposition take about 8 hours. A run costs GPU time, so read runners/run_sweep.sh before starting.

1. Provision. Copy infra/terraform/00-cluster/terraform.tfvars.example to terraform.tfvars, set project_id, then apply 00-cluster and 01-platform in that order. Fetch credentials with gcloud container clusters get-credentials llm-serving-eval --zone us-central1-a --project <project>.
2. Build the generator image into the registry the first stage created: gcloud builds submit wlgen/ --tag us-central1-docker.pkg.dev/<project>/llm-eval-images/wlgen:v5.
3. Add the Hugging Face token to both namespaces that pull the model or its tokenizer: kubectl -n inference create secret generic hf-token --from-literal=token=<token> and the same for -n traffic-gen. The manifests read it from the secret; no token belongs in any file.
4. Stage an arm. runners/stage_campaign.sh defaults <workdir> (or matched) lays out the runners, generator and manifests the way they ran. In <workdir>, apply manifests/03-engine-config.yaml, 03-model-cache-pvc.yaml and 04-servicemonitor.yaml.
5. Generate the Jobs. python3 wlgen/gen_sweep.py --image <image> --model meta-llama/Llama-3.1-8B-Instruct --reps 3 --which rate --outdir sweeps (also sigma, prefix for the defaults arm; ratecp, prefix-cp for the matched arm). With the paper's image these regenerate the campaign's Job specifications byte for byte, seeds included.
6. Deploy one engine with kubectl apply -f manifests/03-inferenceservice-<engine>.yaml and confirm the resolved chunked-prefill setting in its start-up log before measuring.
7. Run. ./run_sweep.sh <engine> <sweep> runs one Job at a time, waits 30 s between runs, retries a run below 95% success up to twice, and resumes after an interruption. The matched arm ran as ./run_campaign_phase.sh <engine>, which also swaps the explicit SGLang manifest in for the 8,192-token runs and measures restarts with ./run_coldstart.sh. Prometheus keeps its series only as long as the cluster exists, so run ./collect_campaign_metrics.sh before tearing down.
8. Analyse by copying the new logs/ into data/logs/ and running make. check_claims.py will then report how your numbers compare with the paper's.
9. Destroy both Terraform stages to stop billing.

SGLang chooses its default chunk size from GPU memory, so a different GPU compares a different pair of configurations. On an L4 at a 16,384-token context, expect each engine's latencies within the 0.5–11.7% we saw between our own two deployments, and the decomposition's ratios within their confidence intervals.

See CITATION.cff. The preprint link will be added here when it is posted.

Code: Apache 2.0 (LICENSE). Data and generated results in data/ and results/: CC BY 4.0 (LICENSE-DATA). See NOTICE.

We thank the Mitacs Globalink Research Internship program and Thompson Rivers University for supporting this research.