Netflix built a three-stage pipeline to enable semantic video search across millions of hours of raw production footage. The system runs an ensemble of specialized AI models (character recognition, scene classification, dialogue transcription, object detection), each producing different output formats and time intervals. Stage 1 ingests raw annotations into Apache Cassandra. Stage 2 performs offline temporal bucketing — normalizing all model outputs into one-second intervals and fusing overlapping annotations into unified records. Stage 3 indexes these enriched buckets in Elasticsearch for hybrid keyword and vector similarity search. The query layer supports exact and approximate nearest-neighbor search, configurable distance metrics, confidence thresholds, fuzzy dialogue matching, and union/intersection result modes. Key tradeoffs include offline fusion latency vs. ingestion throughput, and ensemble accuracy vs. architectural complexity. Netflix is also exploring a unified foundation model (MediaFM) as an alternative to the ensemble approach.
