--- title: Evaling Video Slop — Maor Bril, Character.ai | daily.dev description: A talk by a Character.ai engineer on the challenges of evaluating AI-generated video quality. Existing metrics like CLIP score and LPIPS only assess individual... canonical: https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Evaling Video Slop — Maor Bril, Character.ai | daily.dev og:description: A talk by a Character.ai engineer on the challenges of evaluating AI-generated video quality. Existing metrics like CLIP score and LPIPS only assess individual... og:url: https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea og:image: https://api.daily.dev/og/posts/Ofl90gLEa.png og:image:alt: Evaling Video Slop — Maor Bril, Character.ai og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Evaling Video Slop — Maor Bril, Character.ai **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 23 min read · 1 upvotes · 1 comments ## Summary A talk by a Character.ai engineer on the challenges of evaluating AI-generated video quality. Existing metrics like CLIP score and LPIPS only assess individual frames, missing storytelling, pacing, physics coherence, and audio sync. The speaker describes building a repeatable benchmark combining frame-level metrics with LLM-as-judge, calibrated by human annotation. A key insight is using pairwise comparison (A vs B) rather than absolute scoring (1-10), which yields more consistent results. They distilled a committee of expert models into a small, fast VLM that scores videos in ~3 seconds and explains why a video is low quality. Early lessons include: V1 confidently scored 'vibes' rather than actual quality axes, pairing real vs AI footage risked creating an AI detector, and integrating evals inside the generation loop catches errors cheaply before full video assembly. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why do metrics like CLIP score and LPIPS fail to evaluate AI-generated video quality? CLIP score and LPIPS were built for single-frame or frame-drift evaluation rather than storytelling. CLIP score checks whether a frame matches its prompt, and LPIPS detects drift between frames, but neither determines whether a video actually tells the intended story, whether physics and pacing make sense, or whether audio syncs with the imagery. _Anyone building video-generation evals can track approaches like this one on daily.dev before choosing a metric stack._ ### Why is it better to train a video quality model on pairwise comparisons instead of absolute 1-10 scores? People disagree wildly on absolute scores for storytelling quality, so a six for one person might be a four for another, but when shown two videos side by side, most people agree on which tells the better story. Training on pairs (A versus B) generalizes far better than training on inconsistent absolute ratings, which is why the video-quality model at Character.ai was trained on pairwise comparisons. _Teams designing model evaluation pipelines can follow discussions like this on daily.dev to compare scoring methodologies._ ### How do you avoid accidentally training an AI-generated-content detector instead of a video quality model? When pairing real footage against AI-generated footage to teach a model quality distinctions, keep encoding consistent across both sides so there are no artificial artifacts distinguishing them, and annotate both videos using the exact same method on every axis. Without these controls, the model tends to overfit to detecting AI artifacts rather than judging genuine video quality like storytelling, physics, and pacing. _Developers building AI-content quality classifiers can find similar practical pitfalls on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 0 upvotes > A 7.3 video score tells me less than a clean A/B choice. ## Similar posts on daily.dev - [Stop Evaluating LLMs with “Vibe Checks”](https://daily.dev/posts/stop-evaluating-llms-with-vibe-checks--aiyxjrdvh) · Towards Data Science · 1 upvotes · 1 comments - [LLM evaluation metrics explained: how to measure AI system quality](https://daily.dev/posts/llm-evaluation-metrics-explained-how-to-measure-ai-system-quality-ynrvblcpe) · Netguru · 0 upvotes · 0 comments - [Machine Learning Blog \| ML@CMU \| Carnegie Mellon University](https://daily.dev/posts/machine-learning-blog-ml-cmu-carnegie-mellon-university-hwneh6dyw) · ML CMU · 2 upvotes · 0 comments - [Best Practices and Methods for LLM Evaluation](https://daily.dev/posts/best-practices-and-methods-for-llm-evaluation-x3yfemf90) · databricks · 1 upvotes · 0 comments - [Making AI work through eval hygiene](https://daily.dev/posts/making-ai-work-through-eval-hygiene-9yhow3tph) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#video-generation](https://daily.dev/tags/video-generation) [View this post on daily.dev](https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Evaling Video Slop — Maor Bril, Character.ai","url":"https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea"},"datePublished":"2026-07-25T00:01:07.001Z","dateModified":"2026-10-15T02:22:11.241Z","description":"A talk by a Character.ai engineer on the challenges of evaluating AI-generated video quality. Existing metrics like CLIP score and LPIPS only assess individual...","image":"https://i.ytimg.com/vi/b_PmGocP4rc/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/b_PmGocP4rc/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"llm,video-generation","timeRequired":"PT23M","video":{"@type":"VideoObject","name":"Evaling Video Slop — Maor Bril, Character.ai","description":"A talk by a Character.ai engineer on the challenges of evaluating AI-generated video quality. Existing metrics like CLIP score and LPIPS only assess individual...","thumbnailUrl":"https://i.ytimg.com/vi/b_PmGocP4rc/sddefault.jpg","uploadDate":"2026-07-25T00:01:07.001Z","duration":"PT23M","url":"https://api.daily.dev/r/Ofl90gLEa","embedUrl":"https://www.youtube.com/embed/b_PmGocP4rc"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Evaling Video Slop — Maor Bril, Character.ai"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea","comment":[{"@type":"Comment","text":"A 7.3 video score tells me less than a clean A/B choice.","datePublished":"2026-08-02T21:41:18.088Z","url":"https://daily.dev/posts/Ofl90gLEa#c-0tbayvs3h","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/evaling-video-slop-maor-bril-character-ai-ofl90glea#faq","mainEntity":[{"@type":"Question","name":"Why do metrics like CLIP score and LPIPS fail to evaluate AI-generated video quality?","acceptedAnswer":{"@type":"Answer","text":"CLIP score and LPIPS were built for single-frame or frame-drift evaluation rather than storytelling. CLIP score checks whether a frame matches its prompt, and LPIPS detects drift between frames, but neither determines whether a video actually tells the intended story, whether physics and pacing make sense, or whether audio syncs with the imagery. Anyone building video-generation evals can track approaches like this one on daily.dev before choosing a metric stack."}},{"@type":"Question","name":"Why is it better to train a video quality model on pairwise comparisons instead of absolute 1-10 scores?","acceptedAnswer":{"@type":"Answer","text":"People disagree wildly on absolute scores for storytelling quality, so a six for one person might be a four for another, but when shown two videos side by side, most people agree on which tells the better story. Training on pairs (A versus B) generalizes far better than training on inconsistent absolute ratings, which is why the video-quality model at Character.ai was trained on pairwise comparisons. Teams designing model evaluation pipelines can follow discussions like this on daily.dev to compare scoring methodologies."}},{"@type":"Question","name":"How do you avoid accidentally training an AI-generated-content detector instead of a video quality model?","acceptedAnswer":{"@type":"Answer","text":"When pairing real footage against AI-generated footage to teach a model quality distinctions, keep encoding consistent across both sides so there are no artificial artifacts distinguishing them, and annotate both videos using the exact same method on every axis. Without these controls, the model tends to overfit to detecting AI artifacts rather than judging genuine video quality like storytelling, physics, and pacing. Developers building AI-content quality classifiers can find similar practical pitfalls on daily.dev."}}]} ```