MarketTwits News + Russian Market Data (x402)

x402-markettwits-news

x402-paid HTTP API serving English translations of MarketTwits (t.me/markettwits) market news. Built with Express and the official x402 v2 packages (@x402/express, @x402/core, @x402/evm).

Endpoints

The six analytics routes have fixed prices (independent of NEWS_PRICE) and use the same q matching and pre-payment validation as /news/search; a non-numeric hours/limit also returns 400 before payment. The summary is deterministic (no LLM): a lead sentence (N matching items since <time>; X high-importance.) plus the de-duplicated first sentence/headline of the matching items, high-importance first, then newest. /product/info (in product.mjs): Wildberries tries the public card API card.wb.ru/cards/v4|v2|v1/detail (price in kopecks, rating, feedbacks, stock), then the page JSON-LD, then the static CDN (basket-NN.wbbasket.ru card.json + feedbacks + weekly price-history.json; then price.as_of is set, availability is unknown, prices older than 60 days are refused). Ozon tries the page JSON-LD, then composer-api.bx/page/json/v2. Both marketplaces run anti-bot protection that often blocks datacenter IPs; then the route answers 502. x402 v2 (exact on EVM = "authorization" flow) verifies the payment before the handler and settles only if the handler's status is < 400, so a 4xx/5xx is never charged. RU data and Telegram routes (ru-data.mjs, tg.mjs, upstream.mjs). Each upstream call goes direct first (8 s; Rosstat 15 s) and, if that fails (network error, timeout or a non-2xx status), once more through PROXY_URL (disable with DATA_PROXY_FALLBACK=0); a host that only worked through the proxy goes proxy-first for an hour. Responses are cached in memory (CBR 30 min/7 d, key rate 3 h, MOEX 30-60 s, Rosstat 6 h, Telegram 2 min) with a stale copy served if the upstream is down; failures with no cached copy → 502 upstream_unavailable (not settled). rosstat.gov.ru uses the Russian Trusted Root CA, which is not in Node's trust store: the public Root + Sub CA certificates are bundled in ru-ca.mjs and used only for Rosstat requests (verification stays on). MOEX ISS data is public and delayed ~15 min; Telegram routes read only public web previews. Marketplace proxy (optional). Wildberries/Ozon block datacenter IPs, so /product/info (and the admin probe) can go through an HTTP(S)/SOCKS5 proxy in Russia. Only these marketplace fetches use it as the first path; the RU data / Telegram routes below use it only as a fallback. PROXY_URL (e.g. http://user:[email protected]:3128, also https://, socks5://) takes priority over PROXY_HOST/PROXY_PORT/PROXY_USER/PROXY_PASS. PROXY_SCOPE (default wb,ozon) selects the marketplaces; PROXY_FALLBACK_DIRECT (default true) retries directly when the proxy path fails (proxy phase ≤ 9 s, direct ≤ 6 s). With the proxy the live paths are tried first (card.wb.ru; Ozon page/composer); the WB static CDN stays the last resort. Responses carry via: "proxy"|"direct"; /health shows only proxy_configured: true|false; credentials and the proxy host are masked in errors and never logged. Without a proxy the behaviour is unchanged (via: "direct"). Proxy on a Russian VPS (Ubuntu/Debian, as root; squid with basic auth, CONNECT :443 only, systemd, ufw/iptables): curl -fsSL https://raw.githubusercontent.com/Croomee/x402-markettwits-news/main/scripts/beget-proxy-setup.sh | bash (| bash -s -- 8080 --reset for another port / new credentials). It prints PROXY_URL=...; set that on Render.

Logic lives in news-logic.mjs; npm test runs its unit tests.

Default network: Base mainnet eip155:8453 (USDC 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913, resolved automatically by @x402/evm from the $ price), via the Coinbase CDP facilitator (https://api.cdp.coinbase.com/platform/v2/x402, authenticated with CDP_API_KEY_ID / CDP_API_KEY_SECRET through createFacilitatorConfig() from @coinbase/x402, which signs a per-request CDP JWT).

Facilitator selection: CDP keys set → CDP; otherwise FACILITATOR_URL (default https://x402.org/facilitator, testnet only). On mainnet without CDP keys and with the x402.org facilitator the server refuses to start. Testnet: X402_NETWORK=eip155:84532 (Base Sepolia, USDC 0x036CbD53842c5426634e7929541eC2318f3dCF7e) without CDP keys.

Item schema (also the JSONL line format)

{"id":389638,"source_url":"https://t.me/markettwits/389638","published_at":"2026-10-20T12:20:00Z",
 "importance":"high","text_en":"...","posted_to_x":true,"buffer_post_id":"6ad7...","stored_at":"2026-10-20T12:23:03Z"}

id, published_at, importance (high|medium|low) and text_en are required; the rest is filled in. The server keeps the newest 500 items in memory and appends new ones to NEWS_FILE (default ./data/news.jsonl) best-effort; on startup and whenever the file changes it is re-read.

Configuration (env)

Headless-browser parser (WB / Ozon anti-bot)

Plain HTTP requests to Wildberries/Ozon are blocked even from Russian residential IPs (WB wbaas 403/498 token challenge, Ozon ?__rr=1 / abt challenge). parser/ is a small Playwright + Chromium service for a VPS that loads the real page through the residential proxy, lets the site's own JavaScript pass the challenge, and then reads the site's own JSON (WB __internal/u-card/cards/v4/detail; Ozon entrypoint-api / ld+json).

• GET /parse?url=<WB or Ozon product URL> (or POST {"url": ...}), header Authorization: Bearer $PARSER_TOKEN. 200 → {ok, source, product_id, url, title, price, old_price, currency, rating, reviews_count, in_stock, availability, brand, seller, data_source, fetched_at, ms, cached}. 400 invalid url, 401 bad token, 404 product not found, 502 blocked (anti-bot, e.g. Ozon captcha), 504 timeout, 503 queue full. GET /health (no auth).
• One shared browser/context: cookies and the WB token are reused (and saved to STATE_DIR), so after the first cold request (~10–20 s incl. challenge) WB items take well under a second. Concurrency 2, 25 s per request, 10 min cache; images/fonts/media are not downloaded (saves residential-proxy traffic).
• Chromium cannot use a SOCKS5 proxy with a password, so a socks5:// / socks5h:// PROXY_URL is served through a built-in loopback HTTP-CONNECT bridge (parser/bridge.mjs). http://USER:PASS@host:port proxies are passed to Chromium directly.
• Memory: Chromium with a warm WB tab ≈ 500–800 MB. On < 1.5 GB RAM the setup script sets CONCURRENCY=1 and LOW_MEMORY=1 (fewer renderer processes, smaller JS heap); it adds a 2 GB swapfile when RAM < 2 GB and there is no swap. The unit has MemoryMax=1200M and restarts daily. The SOCKS bridge refuses Chromium's own Google background hosts (BRIDGE_DENY) so they don't consume proxy traffic.
• Setup script: logs every step to /var/log/x402-parser-setup.log and prints the failing step + log tail; --debug for set -x. Node 20 goes to /usr/local from the official tarball (nodejs.org → npmmirror.com), leaving an apt nodejs (e.g. Ubuntu 24.04's v18) untouched; NodeSource is only a fallback. npm falls back to registry.npmmirror.com, Chromium to npmmirror.com/mirrors/playwright, then the distro chromium package (Debian; not Ubuntu's snap).

Install / update on the VPS (root; re-run to update, keeps the token):

PROXY_URL='socks5h://USER:[email protected]:5000' \
  bash <(curl -fsSL https://raw.githubusercontent.com/Croomee/x402-markettwits-news/main/scripts/beget-parser-setup.sh)

It prints PARSER_URL and PARSER_TOKEN; set both on Render. /product/info then calls the parser first (timeout PARSER_TIMEOUT_MS, default 30 s), returns via: "browser", and on any parser failure falls back to the existing proxy/direct chain. 400-before-payment / 402 / settle-only-on-2xx are unchanged; a 502 is still never charged. Security: the parser is protected only by the bearer token over plain HTTP (Render has no static egress IPs to allow-list). Keep the token secret, rotate it with --reset-token, and put it behind HTTPS (e.g. Caddy) when possible.

MCP server (mcp/)

mcp/ is a separate, publish-ready npm package: x402-markettwits-mcp. It is a stdio MCP server with one tool per paid endpoint (31); the tools are generated from the live /openapi.json at startup, falling back to a bundled snapshot. Calls are auto-paid with @x402/fetch + @x402/evm from X402_PRIVATE_KEY, capped at X402_MAX_PAYMENT_USD (default $0.02) per call. See mcp/README.md. It is not part of the Render build (the root npm ci ignores it).

Paid monitor (scripts/paid-monitor.mjs)

Each run makes one real paid call, rotating hourly through cheap endpoints:

• /ru/fx $0.001
• /ru/key-rate $0.001
• /news/since $0.001
• /moex/index $0.0015
• /tg/channel/latest $0.002

It checks:

• HTTP 200
• PAYMENT-RESPONSE with success and a tx hash
• payTo is ours
• a sane body (it also alerts if the news store is empty)
• the tx receipt on Base
• the wallet's USDC balance

On failure it prints a JSON line and ALERT … on stderr, appends the line to MONITOR_LOG (default ~/.x402-monitor/paid-monitor.jsonl), exits 1 and sends a Telegram message when TG_BOT_TOKEN + TG_CHAT_ID are set. It also sends a "recovered" message after a failure, and a low-balance warning at most once a day. --dry-run pays nothing: it checks /health and the 402 challenge (amount, payTo, network, Bazaar) of every group. Exit codes: 0 ok, 1 failure, 2 config error.

cd mcp && npm ci --omit=dev && cd ..          # the monitor uses the MCP package's x402 client
node scripts/paid-monitor.mjs --dry-run
X402_PRIVATE_KEY_FILE=~/.x402-monitor/key node scripts/paid-monitor.mjs [--group cbr-fx] [--no-telegram]

Hourly on a VPS (Beget) with a systemd timer (env file from scripts/paid-monitor.env.example, chmod 600):

# /etc/systemd/system/x402-monitor.service
[Unit]
Description=x402 paid monitor
After=network-online.target
[Service]
Type=oneshot
User=USER
EnvironmentFile=/etc/x402-monitor.env
ExecStart=/usr/bin/node /opt/x402-markettwits-news/scripts/paid-monitor.mjs

# /etc/systemd/system/x402-monitor.timer
[Unit]
Description=x402 paid monitor, hourly
[Timer]
OnCalendar=hourly
RandomizedDelaySec=120
Persistent=true
[Install]
WantedBy=timers.target

systemctl daemon-reload && systemctl enable --now x402-monitor.timer; logs: journalctl -u x402-monitor. Or cron: 7 * * * * set -a; . /etc/x402-monitor.env; set +a; node /opt/x402-markettwits-news/scripts/paid-monitor.mjs >/dev/null 2>&1.

Cost: one paid call an hour is about 720 calls a month, averaging $0.0013, so about $0.94 of USDC a month moves from the hot wallet to our own payTo (net zero, but top up the hot wallet from payTo now and then). The real cost is the facilitator: the CDP x402 facilitator gives 1,000 free settlements a month, then $0.001 each, so $0–$0.72 a month depending on real customer volume. Buyers pay no gas (EIP-3009).

Run locally

npm ci
cp .env.example .env   # fill in INGEST_TOKEN
node --env-file=.env server.mjs
curl -i localhost:4021/news/latest          # 402 + PAYMENT-REQUIRED header (base64 JSON)
curl -s -D- -o /dev/null localhost:4021/news/latest | grep -i payment-required | cut -d' ' -f2 | base64 -d

Deploy on Render

1. Render dashboard → New → Blueprint, pick this GitHub repo (uses render.yaml), or New → Web Service: runtime Node, build npm ci, start node server.mjs, health check /health.
2. Set PAY_TO_ADDRESS, CDP_API_KEY_ID, CDP_API_KEY_SECRET (prompted; sync: false). INGEST_TOKEN is auto-generated by the blueprint — copy it from the service's Environment tab.
3. After deploy: curl https://<service>.onrender.com/health → 200; /news/latest → 402.
4. Feeding data: on the machine that produces translations, put the service URL in api_url.txt and the token in ingest_token.txt (chmod 600) and run push_to_api.sh (POSTs the newest 50 items; server dedupes).

Notes: Render free instances sleep after ~15 min idle (first request is slow) and have an ephemeral disk — the in-memory/file store is lost on restart/redeploy, and is refilled by the next push (newest 50). For durable history use a paid instance with a persistent disk or an external DB.

Paying client

Use @x402/fetch or @x402/axios with a Base wallet holding USDC (mainnet), or testnet USDC on Base Sepolia (https://faucet.circle.com) when running with X402_NETWORK=eip155:84532.

Discovery (x402scan)

GET /openapi.json follows the agentcash/x402scan discovery spec v1: info.x-guidance, info.contact.email, and on the paid operation responses.402 plus structured x-payment-info: {"price":{"mode":"fixed","currency":"USD","amount":"0.0015"},"protocols":[{"x402":{...}}]}. The runtime 402 challenge also carries the Bazaar extension (extensions.bazaar, input/output schema). Check with: npx @agentcash/discovery discover <origin> -v and npx @agentcash/discovery check <origin>/news/latest. Register at https://www.x402scan.com (Add Server → origin URL).