Standard aerial search-and-rescue (SAR) benchmarks are collected in calm, clear conditions. Real SAR missions are not: they happen during floods, wildfires, storms, and at night, exactly when the imagery least resembles what the detector was trained and tested on. AegisBench measures that gap directly.

We apply nine physically motivated visual corruptions, spanning four disaster families and three calibrated severities each, to two established aerial person-detection datasets, and evaluate three architecturally distinct detectors under one shared, deployment-realistic protocol.

Headline finding. Under low-light conditions, detection does not degrade, it collapses. Recall falls to exactly 0.000 (bootstrap 95% CI: [0.000, 0.000]) on both datasets and for all three detector architectures. See Results below.

_{One HERIDAL frame under every corruption. Rows: clean, then the nine corruptions. Columns: severities 1–3.}

• Why this benchmark
• The nine corruptions
• Evaluation protocol
• Results
• Repository layout
• Quickstart
• Full pipeline
• Datasets
• Reproducibility
• Citation
• License

Detectors are usually reported as a single accuracy number measured on imagery collected under good conditions. That number says nothing about what happens when the same detector is flown into the disaster it is meant to help with. AegisBench closes that gap with three design commitments:

1. Physically motivated corruptions, not generic noise. Every corruption models a documented optical phenomenon, not an arbitrary filter. Haze uses the Koschmieder atmospheric scattering model. Low light is modeled in a linear photometric domain with signal-dependent shot noise. Dust is distinguished from wildfire smoke by airlight chromaticity, not by a label alone.
2. Calibrated, not eyeballed, severities. Each corruption's three severity levels are defined by a measurable image statistic, declared in configs/corruptions.yaml and machine-verified for monotonicity. What "severity 2" means is falsifiable, not a guess.
3. A deployment-realistic evaluation protocol. Every detector's confidence threshold is selected once, on clean validation data, and frozen across all 28 test conditions, exactly as a real deployed system would work. One shared evaluator scores every architecture so results are directly comparable.

Full design rationale and physical grounding: docs/CORRUPTIONS.md.

Every pixel-unit parameter is specified per 1000 px of image size and scaled at runtime, so a given severity means the same thing on a 4000x3000 HERIDAL frame and a 640x640 SARD frame. All corruptions are appearance-only (they recolor, darken, or occlude without moving content), so ground-truth boxes drawn on the clean image remain valid on the corrupted one.

Full sweep: 3 detectors (Faster R-CNN, YOLOv11, RT-DETR) x (1 clean + 9 corruptions x 3 severities) x 2 datasets = 168 evaluation conditions, all inference-only against operating points frozen on clean validation data.

Recall at each model's frozen operating point, mean over 1000 bootstrap resamples with 95% confidence interval.

HERIDAL (n = 101 test images):

SARD (n = 862 test images), point estimates, all three models:

A [0.000, 0.000] interval means every one of the 1000 bootstrap resamples produced zero recall: not "usually fails", but fails on every resampling of the evaluation data. Total collapse by severity 3 replicates across both datasets and all three architecturally unrelated detectors, which is what makes it a property of the task rather than a quirk of one model. The one architecture-specific claim that does not replicate: RT-DETR is the most fragile of the three on HERIDAL even at the mildest severity, but on SARD it retains more capability than YOLOv11 at that same severity, the ordering flips. What holds on both datasets is that Faster R-CNN is consistently the most robust of the three, that is a safe general claim; which of the other two is second-most fragile depends on the dataset.

SARD, YOLOv11, mean bootstrap recall by corruption and severity:

At matched severity 3, heavy rain costs about four points of recall while low light costs all of it. That spread, produced by corruptions calibrated on comparable statistical ladders, is what makes the benchmark diagnostic rather than merely difficult.

Aggregated by disaster family (mean relative recall drop, HERIDAL): storm is worst for every model (0.78–0.87, dominated by low_light), flood is mildest for every model (0.23–0.30). The same ordering mostly holds on SARD, except earthquake (dust_haze) edges out storm as the worst family there. Full per-model, per-family table: docs/PAPER_GUIDE.md.

We separately measure localization stability: for instances detected in both the clean and corrupted conditions, how much does the predicted box drift? Even at severity 3, surviving detections retain 0.78–0.91 IoU against their own clean-condition box, while the count of surviving detections collapses in lockstep with recall. In this domain, corruption does not make detectors imprecise, it makes them stop seeing the person at all. Full methodology and numbers: src/aegisbench/evaluation/localization.py.

# 1. Environment (GPU machine; see docs/RUNBOOK.md for the Blackwell note)
bash scripts/setup_env_sm120.sh
python scripts/phase0_check_gpu.py            # must print PHASE 0: PASS

# 2. Unit tests + a GPU-free demo of the corruption engine and tiling
pip install -e ".[dev]" && pytest
python scripts/make_synthetic_samples.py --out data/synthetic --n 3
python scripts/phase3_generate_grid.py \
    --image data/synthetic/synthetic_000.jpg --out results/phase3/grid.jpg

Each phase in docs/RUNBOOK.md ends with an artifact to inspect before the next phase is allowed to run; later phases assume earlier checkpoints were verified.

AegisBench evaluates on two datasets spanning the two dominant real-world capture regimes:

• HERIDAL: high-altitude orthophoto search imagery (~4000x3000), 101 official test images.
• SARD: lower-altitude drone video frames (a Roboflow re-export resized to 640x640), split group-aware by filename token to prevent near-duplicate leakage.

Neither dataset is redistributed by this repository; both require manual download under their own licenses. See docs/DATA.md for sources, expected layout, and split policy.

• Every stochastic element of every corruption is seeded by a stable SHA-256 hash of (image_id, corruption, severity, global_seed), independent of PYTHONHASHSEED and platform. The corrupted test sets are fixed datasets, bit-identical on any machine.
• Training configs (configs/train_*.yaml) carry fixed seeds and are copied verbatim into each run directory.
• The sweep CSV logs git SHA, corruption-config hash, seed, and timestamp per row, and every long-running phase (training, sweeping, bootstrap CI, localization) is resumable from an interruption without recomputation of already-finished work.
• One evaluator scores every model; operating points are selected on clean validation data once and frozen across all corruption conditions.
• Raw predictions are archived per condition, so downstream analyses (confidence intervals, localization stability) never require re-running inference.

The paper is currently under review. A full citation will be added here upon publication.

@inproceedings{aegisbench,
  title     = {AegisBench: A Disaster-Grounded Robustness Benchmark for
               Aerial Search-and-Rescue Person Detection},
  author    = {TODO},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027},
  note      = {under review}
}

Released under the MIT License. Neither HERIDAL nor SARD is redistributed here; obtain each under its own license per docs/DATA.md.

_{Before exporting any double-blind submission archive: run python -m aegisbench.anonymize --root . and export with git archive so .git history never ships. See src/aegisbench/anonymize.py.}