18/20
brief events found by agentic
agentic +3 events
The short version
Across five valid pairs, agentic processing found 18 of 20 brief events; static found 15. Agentic also improved edit-decision classification, but static retrieved broad moments slightly better, localized boundaries more tightly, answered more frozen evidence checks, while agentic took 81% longer.
15/20
brief events found by static
same model
81.32%
more planning latency for agentic
static faster
1 of 6
agentic outputs rejected by the frozen JSON contract
no repair or retry
Automated scorecard
| Metric | Agentic | Static | Observed edge |
|---|---|---|---|
| Moment-retrieval F1 | 0.2667 | 0.3000 | Static +0.0333 |
| Short-event recall | 0.9000 | 0.7500 | Agentic +0.1500 |
| Boundary error, median | 1.20 s | 1.00 s | Static by 0.20 s |
| Boundary error, p90 | 4.00 s | 2.80 s | Static by 1.20 s |
| Frozen-rule evidence accuracy | 0.8333 | 0.9000 | Static +0.0667 |
| Edit-decision macro F1 | 0.6807 | 0.5481 | Agentic +0.1326 |
| Automated brief pass rate | 4/5 | 4/5 | Tie |
| Total accounted tokens | 392,902 | 289,108 | Static used 26.42% fewer |
| Paid-equivalent API cost | $0.330778 | $0.254673 | Static cost 23.01% less |
| Total planning latency | 243.608 s | 134.352 s | Static was 44.85% faster |
Percentages in the table describe the reduction from the agentic total to the static total. The paired aggregate can also be stated as agentic using 35.90% more tokens, costing 29.88% more, and taking 81.32% longer.
What the ten metrics mean
1. Moment retrieval
Whether the response found the expected broad highlights in each video.
2. Short-event recall
Whether it caught two-second-or-shorter visual and audio events that are easy to miss.
3. Boundary accuracy
How closely predicted start and end times matched the frozen ground truth.
4. Evidence accuracy
A deterministic concept-and-time-overlap check—not a human semantic judgment.
5. Edit-decision quality
Macro F1 across the frozen edit labels expected for each candidate moment.
6. Story coherence
Not measured. Requires rendered cuts and blinded editor ratings.
7. Brief compliance
Whether the proposed edit satisfied deterministic duration and structure constraints.
8. Cut-seam defects
Not measured. Requires shared renders, automatic QA, and blinded review.
9. Compute efficiency
Accounted tokens and paid-equivalent API cost for the five valid pairs.
10. Planning latency
End-to-end model planning time recorded for each valid response.
Per-video results
| Fixture | Mode | Moment F1 | Short recall | Evidence | Edit F1 | Brief | Latency |
|---|---|---|---|---|---|---|---|
| screen-01 | Agentic | 0.1667 | 1.00 | 0.8333 | 0.6741 | pass | 45.016 s |
| screen-01 | Static | 0.1667 | 0.75 | 0.8333 | 0.3333 | pass | 33.286 s |
| screen-02 | Agentic | 0.1667 | 1.00 | 0.8333 | 0.2222 | pass | 51.779 s |
| screen-02 | Static | 0.3333 | 0.75 | 0.9167 | 0.3333 | pass | 26.724 s |
| solo-01 | Agentic | 0.1667 | 1.00 | 0.9167 | 0.7407 | pass | 48.165 s |
| solo-01 | Static | 0.1667 | 1.00 | 1.0000 | 0.4074 | pass | 23.481 s |
| podcast-01 | Agentic | 0.5000 | 0.50 | 0.7500 | 0.8333 | pass | 39.201 s |
| podcast-01 | Static | 0.6667 | 0.50 | 0.8333 | 1.0000 | pass | 25.869 s |
| podcast-02 | Agentic | 0.3333 | 1.00 | 0.8333 | 0.9333 | fail: 126 s | 59.447 s |
| podcast-02 | Static | 0.1667 | 0.75 | 0.9167 | 0.6667 | fail: 130 s | 24.992 s |
| solo-02 | Agentic | invalid JSON-only output; excluded without repair or retry | 102.614 s | ||||
| solo-02 | Static | not run by the frozen conditional gate | |||||
How we ran it
- Generated six 600-second, 960×540 videos locally from programmatic graphics, system-synthesized voices, and generated tones.
- Hashed every video and froze its ground truth, evidence rules, prompt, schema, and editing brief before the request.
- Used Gemini 3.7 Flash with low thinking, an 8,192-token output ceiling,
store: false, and either agentic or static video processing. - Required a provider processing call to verify the agentic arm rather than trusting a label in the response.
- Rejected malformed output without repair or unequal retry. Solo-02 failed this gate, so its static arm did not run.
- Scored the ten valid outputs with committed deterministic scripts. Earlier protocol versions were excluded from quality aggregation.
Limits and open questions
Small, synthetic sample
Five valid pairs are not enough for significance testing and may not represent real creator footage.
One-sided attrition
One agentic result was malformed. The protocol correctly excluded it, but the comparison ends with only five pairs.
No human viewing panel
Story coherence and cut quality cannot be inferred from JSON planning outputs alone.
No PaperEdits arm
A future comparison needs a frozen PaperEdits commit and configuration applied to the identical fixtures.
Total paid-equivalent spend across every protocol version and invalid attempt was $0.826280. The final paired economic totals above include only the five valid pairs. This evaluation was conducted independently by the PaperEdits team and was not sponsored or endorsed by Google.
Inspect the evidence
The protocol, prompts, ground truth, model responses, deterministic scorers, failure record, and aggregate are committed publicly. Reproducibility matters more than a polished headline.
Original protocolPlanned design and ten benchmark metrics
Frozen promptThe exact v0.4 model instruction
Ground truthExpected events and editing decisions
Raw results and comparisonsValid outputs, failure record, and pair reports
Final automated aggregateThe source for every number on this page
Scoring toolsDeterministic benchmark scripts