Quality methodology

How we know the output is good.

AI output is statistical. Anyone shipping an AI product who can’t tell you their pass rates and how they measure them is shipping in the dark. Here is exactly what we measure, how, and where we currently sit.

Approach

Cutline runs a 12-stage pipeline. We evaluate quality stage-by-stage rather than only at the final-MP4 level - a single end-to-end pass/fail tells you nothing about where the failure was. Our eval harness scores each stage independently against a held-out set of prompts and a blend of automated checks and human spot-review.

We deliberately don’t run evals against the same model we use in production. Stage-level checks use cheaper deterministic validators where possible (length, structure, schema conformance), and human review where structure isn’t enough to judge quality (script tone, voice naturalness, image relevance).

Quality rubric

Each pipeline stage has a fixed pass/fail rubric defined in code. We don’t grade on a curve; the bar is binary per attempt.

Intent

Audience inferred, goal classified into one of {explain, persuade, tease, demo, narrate}, tone classified, durationSeconds within bounds.

Narrative

3-5 beats, each beat has a purpose, no two beats with identical purpose, total estimated duration within ±10% of target.

Shots

8-12 shots produced, every shot has a purpose / duration / text-density / motion-hint, durations sum within ±5% of target.

Script

Per-shot text exists or is explicitly null (silence). No empty strings. Words-per-second estimate within natural-speech range (2.0-3.5).

Subtitles (draft)

Every spoken segment chunked into ≤7-word lines. Estimated timing per chunk monotonic.

TTS

Audio file written for every non-silent segment. Duration within ±15% of script-estimated duration. Word timings present where supported.

Subtitle refine

≥95% of subtitle chunks aligned to actual word timings (post-TTS). No chunk overlaps the next chunk's start.

Motion

MotionSpec exists per shot. Scale, pan, zoom values within physical bounds (no NaN, no >5x scale, no overflow).

Asset analysis

Skipped if no assets uploaded. Otherwise: dominant colors extracted (≥3), per-asset role classified, no unhandled vision errors.

Visuals

VisualSpec validated against Zod schema. Required fields present.

Image sourcing

Every shot has a non-placeholder image URL after fallback chain (Unsplash → DALL·E → Pexels → simplified query). Placeholder is logged but not allowed in production paths.

Render

MP4 produced. File size within plausible bounds (≥100KB, ≤MAX_VIDEO_OUTPUT_MB). Duration ±2% of target. ffprobe shows valid H.264 codec.

Automated checks

Run on every job in production and on every CI build against the regression suite:

Schema validation - every stage’s output is parsed by a Zod schema before the next stage runs. A malformed payload is treated as a failure of the producing stage, not a downstream silent corruption.
Quality gates - explicit checks in src/lib/pipeline/qualityGate.ts for shot count, duration totals, words-per-second, and subtitle alignment ratio.
Strict script mapping - strictScriptMap.ts verifies that every shot in the shot list has corresponding script text (or explicit silence).
Render validation - post-render ffprobe checks codec, duration, file size. A render that produces a 0-byte or wrong-codec file fails the job, not the user.
Model fallback telemetry - when the primary OpenRouter model fails, we record the fallback hit. Stages with elevated fallback rates trigger investigation.

Human spot-checks

Automated checks catch structural failures. They do not catch a script that’s grammatical but boring, a voice that’s correctly synced but unpleasant, or an image that’s technically relevant but tone-deaf. We run a weekly human spot-check on a 50-job sample stratified across personas (creators, marketers, educators, e-commerce, social, agencies). Each job is reviewed against:

Script naturalness - does it sound like a human wrote it for a 30-second video?
Voice clarity - pacing, pronunciation, emotional fit.
Image relevance - does each shot’s image match what the script is saying at that moment?
Caption legibility on mobile - burned-in subtitle styling on real phone screens.
Edit pacing - too slow, too fast, or right.

Findings get filed as issues. Repeat patterns become regression-suite entries.

Regression suite

We maintain a fixed corpus of ~120 prompts spanning short (10s) and long (60s), all six personas, with and without uploaded assets. The suite runs against every release candidate before deploy. Any prompt that previously passed and now fails blocks the release until it passes again or the rubric is consciously updated.

Current pass rates

Stage-level pass rates over the last 30 days of production traffic. Refreshed manually per release.

Intent classification

98.4%

audience / tone / goal correctly extracted

n = 612 jobs · last 30 days

Narrative arc validity

94.8%

3-5 beats, no orphan beats

n = 612 jobs · last 30 days

Shot list well-formed

96.2%

8-12 shots, durations sum within ±5% of target

n = 612 jobs · last 30 days

Script length match

91.6%

spoken text fits estimated TTS duration ±10%

n = 612 jobs · last 30 days

Subtitle word alignment

89.1%

post-TTS refinement aligns ≥95% of words

n = 612 jobs · last 30 days

Image source per shot

99.7%

every shot has a non-placeholder image (Unsplash → DALL·E → Pexels)

n = 612 jobs · last 30 days

End-to-end render success

97.9%

MP4 produced without unhandled error

n = 612 jobs · last 30 days

Sample size n = 612 reflects production jobs over the trailing 30 days excluding cancelled jobs. Pass rates are stage-level - a job can fail one stage and still recover via fallback before the final MP4. End-to-end render success (97.9%) is the user-visible metric.

What we don’t measure

Honest disclosure of the gaps:

Factual accuracy of the script. If your prompt says “explain why coffee makes you feel awake”, we don’t currently fact-check the script’s claims about adenosine receptors. The LLM is responsible. Review your script.
Image rights / model release. Stock photos come with their own license; AI-generated frames inherit the provider’s policy. We pass through; we don’t audit.
Subjective taste. Two viewers can disagree about whether a video is “good.” Our human reviewers grade the rubric, not personal preference.
Cross-job consistency. Two similar prompts from different users may receive different videos. We don’t promise idempotency on creative output.

Report a failure

Hit something the rubric should have caught? Email parbhat@parbhat.work with the job ID. We add it to the regression suite if it reproduces.

Quality methodology

How we know the output is good.

Approach

Quality rubric

Each pipeline stage has a fixed pass/fail rubric defined in code. We don’t grade on a curve; the bar is binary per attempt.

Intent

Audience inferred, goal classified into one of {explain, persuade, tease, demo, narrate}, tone classified, durationSeconds within bounds.

Narrative

3-5 beats, each beat has a purpose, no two beats with identical purpose, total estimated duration within ±10% of target.

Shots

8-12 shots produced, every shot has a purpose / duration / text-density / motion-hint, durations sum within ±5% of target.

Script

Per-shot text exists or is explicitly null (silence). No empty strings. Words-per-second estimate within natural-speech range (2.0-3.5).

Subtitles (draft)

Every spoken segment chunked into ≤7-word lines. Estimated timing per chunk monotonic.

TTS

Audio file written for every non-silent segment. Duration within ±15% of script-estimated duration. Word timings present where supported.

Subtitle refine

≥95% of subtitle chunks aligned to actual word timings (post-TTS). No chunk overlaps the next chunk's start.

Motion

MotionSpec exists per shot. Scale, pan, zoom values within physical bounds (no NaN, no >5x scale, no overflow).

Asset analysis

Skipped if no assets uploaded. Otherwise: dominant colors extracted (≥3), per-asset role classified, no unhandled vision errors.

Visuals

VisualSpec validated against Zod schema. Required fields present.

Image sourcing

Every shot has a non-placeholder image URL after fallback chain (Unsplash → DALL·E → Pexels → simplified query). Placeholder is logged but not allowed in production paths.

Render

MP4 produced. File size within plausible bounds (≥100KB, ≤MAX_VIDEO_OUTPUT_MB). Duration ±2% of target. ffprobe shows valid H.264 codec.

Automated checks

Run on every job in production and on every CI build against the regression suite:

Schema validation - every stage’s output is parsed by a Zod schema before the next stage runs. A malformed payload is treated as a failure of the producing stage, not a downstream silent corruption.
Quality gates - explicit checks in src/lib/pipeline/qualityGate.ts for shot count, duration totals, words-per-second, and subtitle alignment ratio.
Strict script mapping - strictScriptMap.ts verifies that every shot in the shot list has corresponding script text (or explicit silence).
Render validation - post-render ffprobe checks codec, duration, file size. A render that produces a 0-byte or wrong-codec file fails the job, not the user.
Model fallback telemetry - when the primary OpenRouter model fails, we record the fallback hit. Stages with elevated fallback rates trigger investigation.

Human spot-checks

Script naturalness - does it sound like a human wrote it for a 30-second video?
Voice clarity - pacing, pronunciation, emotional fit.
Image relevance - does each shot’s image match what the script is saying at that moment?
Caption legibility on mobile - burned-in subtitle styling on real phone screens.
Edit pacing - too slow, too fast, or right.

Findings get filed as issues. Repeat patterns become regression-suite entries.

Regression suite

Current pass rates

Stage-level pass rates over the last 30 days of production traffic. Refreshed manually per release.

Intent classification

98.4%

audience / tone / goal correctly extracted

n = 612 jobs · last 30 days

Narrative arc validity

94.8%

3-5 beats, no orphan beats

n = 612 jobs · last 30 days

Shot list well-formed

96.2%

8-12 shots, durations sum within ±5% of target

n = 612 jobs · last 30 days

Script length match

91.6%

spoken text fits estimated TTS duration ±10%

n = 612 jobs · last 30 days

Subtitle word alignment

89.1%

post-TTS refinement aligns ≥95% of words

n = 612 jobs · last 30 days

Image source per shot

99.7%

every shot has a non-placeholder image (Unsplash → DALL·E → Pexels)

n = 612 jobs · last 30 days

End-to-end render success

97.9%

MP4 produced without unhandled error

n = 612 jobs · last 30 days

What we don’t measure

Honest disclosure of the gaps:

Factual accuracy of the script. If your prompt says “explain why coffee makes you feel awake”, we don’t currently fact-check the script’s claims about adenosine receptors. The LLM is responsible. Review your script.
Image rights / model release. Stock photos come with their own license; AI-generated frames inherit the provider’s policy. We pass through; we don’t audit.
Subjective taste. Two viewers can disagree about whether a video is “good.” Our human reviewers grade the rubric, not personal preference.
Cross-job consistency. Two similar prompts from different users may receive different videos. We don’t promise idempotency on creative output.

Report a failure

Hit something the rubric should have caught? Email parbhat@parbhat.work with the job ID. We add it to the regression suite if it reproduces.