What the measurements show
Generated 2026-10-09 05:09 · code ec1196f · thresholds frozen at 9325b7e before test and judge sim.
What the results support
- When no object of the requested class is on the camera, the system says so. The frame-similarity baseline and the random baseline always return something and never answer 'nothing there'.
- Parsing the question into class, camera and time constraints is what makes the system work at all: replacing the parsed plan with raw-text search collapses both object hits and negatives in the ablation.
What they do not show
- Object retrieval better than chance on the main set: a random moment inside the same camera and window (the null system) scores as well as or better than our object Hit@1 on every split. The frame-similarity baseline is above it on the lenient score; see the strict comparison in the limits.
- Better timestamp localisation than chance or the frame baseline: after the time-window boundary fix the differences are within noise, and the random baseline is not worse.
- A benefit of track-centric indexing: frame-level retrieval is better than the full system in the ablation. The track layer is limited by detector recall on small, distant people.
- Any effect of attribute scoring, verification or clarify-once memory: no evaluated query exercises them.
- Says “nothing there” when nothing is there1.00n 16, all splits · Frame baseline (B0) 0.00
- Median distance from the true moment5.3 sn 9, test split · Frame baseline (B0) 8.6 s
- Right clip ranked first, object questions0.56n 18, all splits · Frame baseline (B0) 1.00
Results
Same footage, same parsed time window and camera filter for every system; only retrieval differs. n is the number of questions a metric applies to. Hit@1 counts a returned window that overlaps the labelled one; the strict Hit@1 also needs its peak moment inside it.
| System | Split | n | Hit@1 | Hit@1 strict | Hit@5 | MRR | Camera accuracy | Timestamp error | Negative precision | Re-asks | First answer p50 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| EVORA | test | 19 | 0.31 | 0.31 | 0.69 | 0.44 | 0.69 | 5.3 s | 0.67 | 1 | 19 ms |
| Frame baseline (B0) | test | 19 | 0.62 | 0.38 | 0.69 | 0.68 | 0.77 | 8.6 s | 0.00 | 0 | 48 ms |
| null | test | 19 | 0.46 | 0.46 | 0.77 | 0.59 | 0.85 | 7.4 s | 0.00 | 0 | 1 ms |
| EVORA | dev | 28 | 0.39 | 0.39 | 0.56 | 0.47 | 0.67 | 15.3 s | 0.90 | 0 | 11 ms |
| Frame baseline (B0) | dev | 28 | 0.56 | 0.28 | 0.56 | 0.56 | 0.78 | 19.6 s | 0.00 | 0 | 54 ms |
| null | dev | 28 | 0.22 | 0.22 | 0.56 | 0.35 | 0.56 | 14.7 s | 0.00 | 0 | 1 ms |
| EVORA | judge sim | 9 | 0.25 | 0.25 | 0.75 | 0.44 | 0.50 | 146.8 s | 0.80 | 1 | 12 ms |
| Frame baseline (B0) | judge sim | 9 | 0.25 | 0.25 | 0.50 | 0.35 | 0.50 | 7.4 s | 0.00 | 0 | 46 ms |
| null | judge sim | 9 | 0.00 | 0.00 | 0.50 | 0.21 | 0.50 | 62.9 s | 0.00 | 0 | 1 ms |
Extra checks
Harder or narrower question sets run on the same frozen system. They sit next to the main results above and do not replace them.
Object questions where a random moment is less likely to be right
The same kind of object question, kept only for 1-minute windows where MEVA's annotated actors fill at most half of the window, so a random moment is less likely to be right. Cameras keep their split. These windows overlap the main set: this re-weights it toward harder chance, it is not new footage.
| System | Hit@1 | Hit@1 strict | Hit@5 | MRR |
|---|---|---|---|---|
| EVORA | 0.56 n 18 | 0.56 n 18 | 0.83 n 18 | 0.69 n 18 |
| Random moment | 0.33 n 18 | 0.28 n 18 | 0.94 n 18 | 0.57 n 18 |
| Frame baseline (B0) | 1.00 n 18 | 0.50 n 18 | 1.00 n 18 | 1.00 n 18 |
| Split | EVORA | Random moment | Frame baseline (B0) |
|---|---|---|---|
| dev | 0.9 s n 11 | 12.3 s n 11 | 5.3 s n 11 |
| test | 7.4 s n 6 | 18.1 s n 6 | 3.1 s n 6 |
| judge sim | 4.4 s n 1 | 19.8 s n 1 | 3.2 s n 1 |
Asking once about a place, then remembering it
Questions that name an invented place instead of a camera. The first mention must trigger one clarification, later ones (including three fixed rewordings of the name) must not, and answers must come from the camera the place was bound to.
| Measure | EVORA |
|---|---|
| First mentions that were asked about | 1.00 n 6 |
| Questions asked that were needed | 1.00 n 6 |
| Asked again after it was answered | 0 n 30 |
Asking for a colour, scored against human labels
Asking for a colour on one camera, scored against human colour labels over the labelled tracks only: does the first labelled track returned have the colour? Compared with attribute grounding switched off and with a random order of the same tracks. The colour model was tuned on part of these labels, and the labels cover a sample of tracks, so read this as an upper-leaning estimate on small n.
| Tracks | n | Colour used, Hit@1 | Colour ignored, Hit@1 | Random order, Hit@1 |
|---|---|---|---|---|
| All labelled tracks | 39 | 0.64 | 0.67 | 0.30 |
| Persons | 27 | 0.59 | 0.63 | 0.33 |
| Vehicles | 12 | 0.75 | 0.75 | 0.25 |
The second look at colour, against human labels
The visual check (a small local vision model) against human colour labels, through the full router. Accuracy counts only labelled tracks the check looked at; the effect compares the labelled evidence before and after candidates the check said no to are set aside. Same caveats as the colour retrieval set: a sample of labelled tracks, small n, one footage set.
| Measure | Value | n |
|---|---|---|
| Said yes when the colour was right | 0.69 | 109 |
| Said no when the colour was wrong | 0.83 | 127 |
| Hit@1 before and after the check | 0.67 to 0.72 | 36 |
| Share of right colours among the evidence, before and after | 0.49 to 0.60 | 36 |
| Questions where the check set aside every labelled track | 3 | 39 |
By capability
| Capability | System | Split | Hit@1 | Camera accuracy | Timestamp error | Negative precision |
|---|---|---|---|---|---|---|
| Finding an object class | EVORA | test | 0.50 (n 8) | 1.00 (n 8) | 4.2 s (n 8) | – |
| Saying “nothing there” | EVORA | test | – | – | – | 1.00 (n 4) |
| Finding an object class | Frame baseline (B0) | test | 1.00 (n 8) | 1.00 (n 8) | 5.9 s (n 8) | – |
| Saying “nothing there” | Frame baseline (B0) | test | – | – | – | 0.00 (n 4) |
| Finding an object class | null | test | 0.75 (n 8) | 1.00 (n 8) | 2.9 s (n 8) | – |
| Saying “nothing there” | null | test | – | – | – | 0.00 (n 4) |
| Finding an object class | EVORA | dev | 0.56 (n 9) | 1.00 (n 9) | 14.7 s (n 9) | – |
| Saying “nothing there” | EVORA | dev | – | – | – | 1.00 (n 8) |
| Finding an object class | Frame baseline (B0) | dev | 1.00 (n 9) | 1.00 (n 9) | 12.0 s (n 9) | – |
| Saying “nothing there” | Frame baseline (B0) | dev | – | – | – | 0.00 (n 8) |
| Finding an object class | null | dev | 0.44 (n 9) | 1.00 (n 9) | 9.5 s (n 9) | – |
| Saying “nothing there” | null | dev | – | – | – | 0.00 (n 8) |
| Finding an object class | EVORA | judge sim | 1.00 (n 1) | 1.00 (n 1) | 4.4 s (n 1) | – |
| Saying “nothing there” | EVORA | judge sim | – | – | – | 1.00 (n 4) |
| Finding an object class | Frame baseline (B0) | judge sim | 1.00 (n 1) | 1.00 (n 1) | 3.2 s (n 1) | – |
| Saying “nothing there” | Frame baseline (B0) | judge sim | – | – | – | 0.00 (n 4) |
| Finding an object class | null | judge sim | 0.00 (n 1) | 1.00 (n 1) | 19.8 s (n 1) | – |
| Saying “nothing there” | null | judge sim | – | – | – | 0.00 (n 4) |
Not evaluated
- Counting
- MEVA labels only actors in annotated activities (parked cars and bystanders are unlabelled), so annotated counts are lower bounds and exact counts cannot be scored
- Colour
- no colour labels in MEVA; needs the blind human labels from scripts/colour_label_tool.py
- Carrying
- no bag-carrying ground truth: MEVA's carried objects are class 'other', one bag actor in 6 cameras
- Paths across cameras
- no cross-camera identity ground truth (MEVA actor ids are per clip and camera)
Against the baseline, test split
- Hit@1
- Hit@5
- MRR
- Camera accuracy
- Negative precision
What each part contributes
Each row turns one contribution off and reruns the frozen test split. A shorter bar than the full system means that part was helping; equal bars mean no effect could be measured on these questions.
Hit@1
- Full system0.38
- without Track-centric multi-granular retrieval (C1)0.46
- without Attribute grounding (C2)0.38
- without Plan-then-verify: LLM planner (C3a)0.15
- without Plan-then-verify: verification (C3b)0.38
- without Clarify-once referent memory (C4)0.38
- without Topology-aware cross-camera linking (C5)Not run: needs the footage re-indexed with the switch off
- without Motion-gated adaptive sampling (C6)Not run: needs the footage re-indexed with the switch off
Negative precision
- Full system0.83
- without Track-centric multi-granular retrieval (C1)0.83
- without Attribute grounding (C2)0.83
- without Plan-then-verify: LLM planner (C3a)0.00
- without Plan-then-verify: verification (C3b)0.83
- without Clarify-once referent memory (C4)0.83
- without Topology-aware cross-camera linking (C5)Not run: needs the footage re-indexed with the switch off
- without Motion-gated adaptive sampling (C6)Not run: needs the footage re-indexed with the switch off
Time to the first answer
Dot is the median, the line runs to the 95th percentile, measured on the same machine in the same session.
- EVORA test19 ms · 3.37 s
- Frame baseline (B0) test48 ms · 62 ms
- null test1 ms · 2 ms
- EVORA dev11 ms · 350 ms
- Frame baseline (B0) dev54 ms · 61 ms
- null dev1 ms · 1 ms
- EVORA judge sim12 ms · 4.01 s
- Frame baseline (B0) judge sim46 ms · 50 ms
- null judge sim1 ms · 2 ms
Limits
- Ground truth is MEVA's: it labels only actors that take part in annotated activities, so a correct but unlabelled object counts as a miss. Object scores are conservative; negatives are built only for classes absent from a camera's whole annotation.
- Sample sizes are small (one 5-minute window, 8 cameras). Read every number together with its n.
- The judge_sim split shares the recording day with dev and test: it checks unseen cameras, not unseen footage.
- Thresholds were frozen before the test and judge_sim runs; nothing was tuned on them.
- Chance level: a random moment inside the same camera and window already scores object Hit@1 0.56 (n=18), because MEVA's annotated actors are dense in time. Read every object result next to this row.
- Hit@1 is reported two ways (pooled, n=18): ours 0.56 to 0.56; frame baseline 1.00 to 0.61; random moment 0.56 to 0.56. The lenient score counts any overlap between the returned window and the labelled one, so a system that returns long windows is right almost by construction; the strict score needs the returned peak moment inside the labelled window widened by 2 s. Compare systems on the strict score.