Universal local-first media intelligence for AI agents.
OcularAudio lets MCP-compatible AI systems watch, listen to, search, inspect, compare, and reason over media. It turns URLs, streams, and local media into reusable timestamped evidence: metadata, transcripts, frames, OCR, retrieval results, multimodal moments, and cross-video intelligence.
v1.3.0 · MIT
MCP server for video transcripts, screenshots, and OCR on YouTube and web videos.
2026-11-09
ดาว
—
เครื่องมือ
—
อัปไทม์
—
Product
What it does
Universal media intake: web URLs, direct media, local files, live/stream sources, and a broad set of social, video, audio and archive platforms.
Speech understanding: timestamped captions with local Whisper fallback when captions are unavailable.
On-screen text: optional Tesseract OCR produces searchable visual text evidence.
Evidence search: timestamp-aware lexical, semantic and hybrid retrieval over stable, provenance-preserving canonical evidence records.
Multimodal analysis: aligns transcript, visual and OCR evidence around the same moments.
Batch intelligence: process multiple sources, search across them, and generate cross-video comparisons with failure isolation.
Agentic execution: turn analysis requests into bounded plans with retries, timeouts, audit records and health reporting.
Capability modes: Lite, Intelligence, Deep and Auto control processing depth and response richness.
Local caching: reuse processed evidence and manage cache state through CLI or MCP.
Operational awareness: capabilities, health, readiness, security state, metrics and the full protocol contract are machine-readable.
MCP interoperability: tools, resources and prompts over stdio or Streamable HTTP.
Production deployment: bearer-token protection, host/origin controls, Docker deployment, persistent cache and readiness checks.
Best use cases
Research and lecture analysis · product demonstrations · competitor analysis · interviews · technical talks · UI walkthroughs · multi-video comparison · evidence-grounded AI investigation.
Capability matrix
Capability
User outcome
Universal sources
Analyze URLs, files and streams through one interface
Transcript evidence
Find exactly when something was said
Visual evidence
Inspect what was shown at a timestamp
OCR
Search text visible inside video frames
Hybrid retrieval
Combine speech and visual evidence
Multimodal moments
Understand speech and visuals together
Batch analysis
Investigate collections of videos
Cross-video comparison
Compare evidence across sources
Agentic workflows
Execute bounded evidence investigations
Capability modes
Auto-select or enforce Lite, Intelligence, and Deep execution budgets
Cache
Make repeated analysis faster
MCP prompts
Start common investigation workflows
MCP resources
Give hosts operational context
HTTP deployment
Run as a protected network service
Canonical evidence architecture
All source artifacts are normalized into one local manifest per source. Evidence records have stable content-derived IDs, modality labels, timestamp bounds, content hashes, provenance pointers, and metadata. Transcript search, timelines, moment inspection, and hybrid retrieval share the same canonical IDs; OCR and visual-frame records are indexed alongside transcript evidence. Manifests are written atomically, validated on read, deduplicated by ID, and refreshed from the existing transcript and visual caches without requiring a migration or a database. Clearing a source removes its transcript, canonical manifest, visual index, and frame artifacts together. Manifest metadata intentionally stores a bounded summary rather than the source URL or credentials.
Four-level intelligence engine
Every request is resolved through one deterministic policy engine. Auto resolves to Lite, Intelligence, or Deep based on query intent, requested features, source count, media duration, and an optional time budget. The decision is returned as machine-readable metadata with explicit limits, disabled features, capability gaps, and adjustments.
Mode
Evidence
Visual frames
Default sampling
Runtime ceiling
Sources
Behavior
Lite
8
8
30 s
45 s
1
Transcript/metadata first; OCR, multimodal and batch are disabled
Intelligence
24
24
10 s
180 s
10
Balanced retrieval, visual/OCR and multimodal analysis
Deep
64
64
3 s
600 s
32
Expanded evidence and multi-step investigation
Auto
Policy-selected
Policy-selected
Policy-selected
Policy-selected
Policy-selected
Resolves to one of the three execution modes
Modes enforce evidence caps, frame limits, source limits, retry counts, concurrency, sampling intervals, and runtime ceilings. Missing OpenCV, FFmpeg, or Tesseract is reported as a capability gap; optional unavailable features are disabled rather than represented as successful. Explicit source-limit violations fail with a clear error instead of silently dropping sources. Source coverage remains shared across modes.
The CLI's existing --mode flag now reaches the Python runtime. Use --budget-seconds to impose a caller-specified runtime ceiling (0.001-900 seconds). MCP planning, agentic execution, hybrid search, visual indexing, multimodal analysis, and batch analysis expose the same decision contract. Use --mode deep for comprehensive evidence or --mode lite for a fast overview.
Quick start
NPM / MCP
Release status: this repository contains source version 1.3.0, but that version is not yet published to npm. The latest published package is 1.2.1, so the npx examples below install 1.2.1 until a separate 1.3.0 release is authorized and published. To test the current checkout, clone the repository, install the Python dependencies from requirements.txt, then run node bin/cli.js from the checkout.
npx -y ocular-audio-mcp
Local MCP hosts can launch the same command. Example:
The MCP endpoint is /mcp. For network exposure, configure authentication, real issuer/resource URLs, allowed hosts and allowed origins. Do not expose an unauthenticated instance to an untrusted network.
Docker
cp .env.example .env
docker compose -f docker/docker-compose.yml up --build -d
The supplied deployment uses Python 3.11, FFmpeg, a persistent cache volume, a read-only container filesystem, dropped capabilities, no-new-privileges, restart policy and a healthcheck.
For internet-facing deployments, use TLS and an organization identity provider. The built-in static bearer verifier is intended for controlled deployments and can be replaced by JWT validation or RFC 7662 introspection.
Runtime requirements
Python 3.11+ recommended for the release environment.
Node.js 18+ for the NPM CLI.
FFmpeg for media processing.
OpenCV for visual processing.
faster-whisper for local transcription fallback.
Tesseract is optional for OCR.
Network access is required for remote sources.
The NPM package does not hide native Python/system prerequisites; OcularAudio is a local media-processing server rather than a hosted API.
Cookies and restricted media
Operator-supplied cookies.txt can be used for sources that require an authenticated browser session. Never commit cookies, access tokens or credentials.
Release v1.3.0
npm install
npm run check
npm pack
This creates ocular-audio-mcp-1.3.0.tgz. Inspect it before publishing:
npm publish ocular-audio-mcp-1.3.0.tgz --access public
See RELEASE_v1.3.0.md. A manual GitHub Actions workflow can also build and upload the exact tarball as an artifact.