Case study 01
MOSAIC
Ask questions about any video and jump straight to the moment that answers it.
Overview
MOSAIC is a multimodal video understanding system. It indexes what is said, what is shown and what is written on screen, so a long video becomes something you can search and talk to.
Upload one or more videos and ask in plain language. MOSAIC finds the relevant moments, cuts the clips with timestamps, and answers with grounded references back to the footage.
What it does
Transcript search
Speech is transcribed with Whisper and embedded, so you can search a video by what was said.
Visual search
Frames are embedded with CLIP, so a query like "red car at night" finds the scene even if nobody says it.
Caption understanding
A vision LLM describes key frames, adding a third, descriptive layer to retrieval.
Clip extraction
Matching moments are cut with FFmpeg and returned with exact timestamps.
Video Q&A and summaries
A ReAct agent plans which search tools to call and answers with sources from the video.
Multi-video and MCP
Search across a library of videos, and expose the tools to any MCP client.
How it works
- 01Ingest: FFmpeg splits audio and samples frames from each upload.
- 02Understand: Whisper transcribes speech, CLIP embeds frames, a vision LLM captions key frames.
- 03Index: embeddings go into ChromaDB and FAISS for fast hybrid retrieval.
- 04Reason: a LangChain ReAct agent picks the right search tools for the question.
- 05Answer: the agent replies with timestamps and extracted clips as evidence.
Stack
- Python
- FastAPI
- LangChain
- CLIP
- Whisper
- ChromaDB
- FAISS
- Groq
- Mistral
- FFmpeg
- PyTorch
- Next.js
- Docker
- MCP