midReportedcurated
A video summariser samples one frame every three seconds. A customer summary says an export may not have finished, but the recording shows an Export ready toast. What went wrong, and how do you fix it?
30-second answer
- The toast was visible for about 1.2 seconds inside a 3-second sampling gap, so no sampled frame contained it and the summariser answered from the last state it could see.
- The rule is that the sampling interval must be shorter than the shortest event you must catch: 1 second would guarantee a frame here and costs 90 frames instead of 30.
- Before paying that everywhere, add frames on scene changes or rely on the transcript for events the narrator states aloud, then measure each option on a golden set containing the recording and the question about whether the export finished.
- Denser sampling multiplies image tokens and cost, and each frame competes for context and prefill time, so the interval is a budget decision rather than a quality setting.
- It does not fix a summary that contradicts the audio: conflicting evidence still needs an explicit rule for which source wins, plus an abstain path when the frames and transcript disagree.
Likely follow-ups
Open a question to rehearse the answer.
How would you decide between denser sampling and scene-change triggering?
- Compare the shortest event you must catch with the scene-change rate; scene triggering wins when state changes produce visible transitions and fails when the change is a small toast.
- Measure event recall on the golden set at each strategy, not average frame count, because a cheap strategy that misses the toast is not cheaper.
- Price the token cost per minute of video, since a fixed 1-second interval is predictable while scene-change bursts are not.
What else in that pipeline could have produced a confident wrong answer?
- The frame may have contained the toast but preprocessing resized or compressed it until the small text was illegible.
- The prompt may have implied an answer instead of offering an abstain, so the model described the last visible state as final.
- The transcript and the frames may disagree, and the pipeline had no rule for resolving the conflict or flagging it.
Red flags
- Suggests a better prompt without touching the sampling rate
- Says to send every frame, ignoring token cost and context limits
Concepts this tests
Tags
Source: forwarddeployedprep.com
More in Multimodal, vision & voice
- Argue both sides: cascaded ASR plus LLM plus TTS versus a native speech-to-speech model for a production voice agent.
- Budget the latency for a real-time voice agent: VAD, ASR, LLM, TTS, network. Where does the time go, and what do you optimise first?
- When would you use a multimodal model instead of OCR plus an extraction pipeline, and what are its failure modes?