Framefetch

FrameFetch turns one YouTube, YouTube Shorts, TikTok, Instagram Reels, Pinterest, or Reddit video URL into a single JSON response: a direct answer to a question about the video, metadata, engagement insights, a transcript (captions or Whisper), an LLM digest (text or spoken mp3), structured JSON (chapters/entities/products/claims), comments + sentiment, parametrically-sampled frames (every Nth / 1-per-second / a time range, at any width), and the on-screen text burned into those frames (OCR — captions, price tags, signage). Plus keyword search when you don't have a URL yet, and batch for up to 10 URLs in one call. Built API-first and MCP-first for AI agents.

This repo is the open-source client + docs. The service itself runs at framefetch.net — you bring a free API key (or pay per call with x402); the backend stays hosted.

Why

An LLM can't watch a video. To reason about one it needs the video turned into text and images first — an answer, a transcript, metadata, a few frames. FrameFetch returns all of that from a URL, across six platforms, through one schema.

Install

npm install framefetch

Node 18+ (uses built-in fetch). Get a free key: framefetch.net.

Version note. This repo is at 0.4.0. The newest version currently on npm is 0.3.0 — npm install framefetch still gives you that one, and it has only extract/metadata/transcript/frames/platforms/status/demo/createKey. Everything else documented below is live on the API today and available from this repo; from npm 0.3.0 you can reach the same data through extract({ fields: [...] }).

Ask a question — get an answer, not a transcript dump

A direct question about a video returns a short, grounded answer with timestamped quotes, instead of you parsing a 25,000-token transcript yourself.

import { FrameFetch } from 'framefetch';

const ff = new FrameFetch({ apiKey: process.env.FRAMEFETCH_API_KEY });

const { ask } = await ff.ask(
  'https://www.youtube.com/watch?v=jNQXAC9IVRw',
  'What does the presenter say to do first?',
);

console.log(ask.answer);          // short, direct answer
console.log(ask.confidence);      // 'high' | 'medium' | 'low'
for (const q of ask.quotes) {     // verbatim, timestamped supporting quotes
  console.log(`[${q.t_sec}s] ${q.text}`);
}
console.log(ask.coverage);        // which part of the transcript was analyzed

Charged only when an answer is actually produced. A repeat question about an already-extracted video reuses the cached transcript, so it answers fast without a re-download or re-transcription — but the answer itself is always freshly generated, never cache-served.

Frames-based answers: when a video has no transcript (e.g. Pinterest, or transcription failed), the answer is grounded in sampled keyframe images instead. Then coverage.mode is "frames", quotes is [] (no transcript text to quote), and confidence is capped at "medium".

Quick start

import { FrameFetch } from 'framefetch';

const ff = new FrameFetch({ apiKey: process.env.FRAMEFETCH_API_KEY });

const r = await ff.extract({
  url: 'https://www.youtube.com/watch?v=jNQXAC9IVRw',
  fields: ['metadata', 'transcript', 'frames', 'text_overlay'],
  frames: { mode: 'fps', fps: 1, width: 480 },
});

console.log(r.metadata.title);         // "Me at the zoo"
console.log(r.transcript.text);        // "All right, so here we are, in front of the elephants…"
console.log(r.frames.length);          // 19 — frames is an array
console.log(r.textOverlay?.[0]?.text); // on-screen text detected in the first frame, if any

Note the two spellings: text_overlay is the request field name, textOverlay is the response key.

Scoped helpers

await ff.metadata(url);          // title, author, duration, views, likes…
await ff.transcript(url);        // captions, else Whisper
await ff.frames(url, { mode: 'fps', fps: 1, width: 512 });
await ff.ask(url, 'What product is being reviewed?'); // grounded Q&A, see above
await ff.digest(url);            // LLM summary of the transcript
await ff.audioDigest(url, { voice: 'nova' }); // spoken mp3 briefing (signed URL, 24h)
await ff.structured(url);        // chapters/entities/products/claims/key_moments
await ff.comments(url, { comments_cap: 50 }); // top-level comments
await ff.commentSentiment(url);  // aggregated audience-mood rollup (+ the comments)
await ff.platforms();            // capability matrix (no key)
await ff.status();               // live service health (no key)

// on-screen text (OCR) — requires "frames" alongside it, use extract() directly:
await ff.extract({ url, fields: ['frames', 'text_overlay'], frames: { mode: 'fps', fps: 1 } });

Every helper above is a thin wrapper over extract(), so anything extract() accepts (translate, format, extra fields, …) can be passed as the last argument and is forwarded unchanged.

Search for videos, extract many at once

// Find something to extract when you don't have a URL yet
const s = await ff.search('how to make sourdough', { limit: 5 });
for (const hit of s.results) {
  console.log(hit.title, hit.url, hit.durationSec);
}

// Then extract up to 10 of them in ONE call. Shared options apply to every url.
const b = await ff.batch(s.results.slice(0, 3).map((r) => r.url), {
  fields: ['metadata', 'digest'],
});
for (const item of b.results) {
  if (!item.ok) { console.error(item.url, item.error?.code); continue; }
  console.log(item.metadata.title, '→', item.digest.gist);
}

One failing URL never fails the batch — each entry carries its own ok flag and, when ok is false, an error with code/message/hint. Per-URL frames specs are not accepted in a batch; use extract() for those.

Translate the transcript, export subtitles

// translate the transcript into 1 of 25 languages (surfaced as transcript_translated)
const r = await ff.transcript(url, { translate: 'ja' });
console.log(r.transcript_translated.text);

// export subtitles directly — format is sent as a query param, response comes back as a string
const srt = await ff.transcript(url, { format: 'srt' });         // source-language subtitles
const vttJa = await ff.transcript(url, { translate: 'ja', format: 'vtt' }); // translated subtitles

No signup

const ff = new FrameFetch();                              // no key
await ff.demo('https://youtu.be/jNQXAC9IVRw');            // instant metadata, rate-limited
const { key } = await ff.createKey('[email protected]');    // self-serve key + free credit

Use it from an MCP agent

FrameFetch ships an MCP server (Streamable HTTP) with four tools: framefetch_extract, framefetch_platform_capabilities, framefetch_search and framefetch_account. Add it to Claude, Cursor, or any MCP client:

{
  "mcpServers": {
    "framefetch": {
      "url": "https://framefetch.net/mcp",
      "headers": { "Authorization": "<YOUR_FRAMEFETCH_KEY>" }
    }
  }
}

Or one line:

claude mcp add --transport http framefetch https://framefetch.net/mcp \
  --header "Authorization: <YOUR_FRAMEFETCH_KEY>"

MCP lives at https://framefetch.net/mcp and speaks JSON-RPC over Streamable HTTP. REST lives under /v1/* and takes plain JSON ({"url": "…"}). Crossing the two is the single most common first-call mistake, so both directions answer clearly: a REST body POSTed to /mcp comes back as a JSON-RPC parse error, and a JSON-RPC body POSTed to /v1/extract comes back as 400 WRONG_ENDPOINT naming the right URL for your client.

Local stdio bridge

Prefer a local stdio server (Claude Desktop, sandboxes, no inbound HTTP)? This package ships framefetch-mcp, a zero-dependency stdio↔HTTP bridge that exposes the same tools and forwards calls to framefetch.net:

{
  "mcpServers": {
    "framefetch": {
      "command": "npx",
      "args": ["-y", "framefetch-mcp"],
      "env": { "FRAMEFETCH_API_KEY": "<YOUR_FRAMEFETCH_KEY>" }
    }
  }
}

tools/list works with no key; tool calls use FRAMEFETCH_API_KEY (or x402). Override the endpoint with FRAMEFETCH_MCP_URL.

Pay without an account (x402)

Autonomous agents can pay per call in USDC via x402 on Base — no signup, no human in the loop. Discoverable in the x402 Bazaar and at /.well-known/x402.json. Humans can use a free tier, prepaid credits, or a Stripe card.

Errors

Failed calls throw FrameFetchError with .status, .code, and .hint:

import { FrameFetchError } from 'framefetch';
try {
  await ff.transcript(url);
} catch (e) {
  if (e instanceof FrameFetchError && e.status === 402) {
    // out of credit — top up at framefetch.net or via x402
  }
}

API surface

Full extract() request shape

ff.extract({
  url: string,
  fields?: Field[],            // 'metadata' | 'insights' | 'transcript' | 'frames' | 'text_overlay'
                               // | 'digest' | 'audio_digest' | 'structured' | 'comments'
                               // | 'comment_sentiment' | 'delta'
  frames?: { mode, n, fps, from, to, format, width },
  translate?: string,          // ISO-639-1 target language (25 supported)
  voice?: string,              // TTS voice for audio_digest
  comments_cap?: number,       // 1-200, default 100
  ask?: string,                // 3-500 char question — see ff.ask() above
  publish?: boolean,           // opt in to a public per-video SEO page
  format?: 'md' | 'markdown' | 'srt' | 'vtt', // alternate egress rendering (returns a string, not JSON)
});

See index.d.ts for the complete typed response shape (ExtractResult, Ask, VideoStructured, VideoComments, CommentSentiment, AudioDigest, SearchResult, BatchResult, …).

Full OpenAPI: framefetch.net/openapi.json · Docs: framefetch.net/docs

Links

Website · Docs · Pricing · Status · Guide: giving an agent video data · Compare vs alternatives

License

MIT