Alchemylake Databricks

AlchemyLake for Databricks

Install a governed creative surface directly inside your Databricks workspace. AlchemyLake turns the data your lakehouse already trusts into finished deliverables Genie doesn't make: board-ready PowerPoint decks with a read-aloud script and Q&A prep under every slide, enterprise PDF dossiers with a statistical appendix and an Excel evidence workbook, deep-research dossiers (a planned multi-step investigation — Genie sources get governed SQL follow-ups in one continuing conversation), designed infographics, animated video briefings with narration (six formats), two-host data podcasts (five formats), and sonified scores (six genres) whose tempo follows your growth.

Get started free — no card required

1. Sign up — 50 free credits land in your account immediately, nothing to enter but an email.
2. Grab a developer key — sign in at app.alchemylake.com → Studio → Developer · MCP & keys → Forge a new key (starts with alk_…).
3. Clone this repo and deploy it into your Databricks workspace:

   git clone https://github.com/zorost/alchemylake-databricks.git
   cd alchemylake-databricks
   # edit the workspace host in databricks.yml (see Path 1 below), then:
   databricks bundle deploy -t prod
   databricks bundle run alchemylake_app -t prod
   

   Tracking main gets you the latest build; git checkout v0.4.0 first pins the deploy to a tagged release instead, if you'd rather not move with main.
4. Open the App URL the CLI prints, paste your key, then Load governed sources → pick one → Transmute.

That's a governed PDF, deck, infographic, or video briefing — sealed to your own data, running inside your own workspace — for free, in about five minutes. No Databricks Apps on your workspace? Skip straight to Path 2 — register one URL and Genie, Claude, or Cursor get the same 50 free credits to work with, no bundle required.

Every lane binds to a governed source, and the numbers discipline holds throughout: the platform computes an analyst-grade statistical dossier from the bound rows by code — trend with fit quality, outliers, correlations, segment shares, concentration, pivots — the model never authors a figure, a verifier checks every claim after generation, and the result is sealed (source · row count · data sha256 · verification score) with the seal embedded in the file itself. Genie answers bind as sources too, and one-click recipes ship a board deck or a whole campaign pack from one table. Every credit accounted.

Where Genie stops, AlchemyLake starts. Genie answers questions about your data inside the workspace — tables and charts. AlchemyLake takes the same governed rows the last mile: "Ask Genie for Q2 revenue by region, then render an 8-slide deck titled Q2 Momentum" leaves the workspace as a .pptx the CFO can present cold — every figure verified against the rows, provenance sealed into the file.

This repository is a self-contained Databricks Asset Bundle. One command deploys a Databricks App (SSO-authenticated, running on your workspace) that calls the AlchemyLake platform over MCP. Nothing bypasses governance: the same credit ledger, provenance seals, and role checks that protect the web app protect every call made from inside Databricks. Your data never leaves the lakehouse except as the exact rows you choose to bind.

.
├── databricks.yml        # the Asset Bundle (App resource + targets)
├── app/
│   ├── app.py            # the Streamlit App (thin MCP client)
│   ├── app.yaml          # App runtime config (command + env)
│   └── requirements.txt
└── sql/
    └── ai_render.sql     # optional UC function: call AlchemyLake from SQL/Genie

Three ways to use AlchemyLake with Databricks

What agents can ship from a table (things a Genie answer alone cannot):

• render_deep_research — a planned multi-step investigation: the brief is decomposed into sub-questions, each answered with real evidence (Genie sources get governed SQL follow-ups in one continuing conversation), then synthesized into a sealed PDF dossier + Excel evidence workbook.
• render_presentation — a .pptx board deck, 5–20 slides, speaker script + Q&A in every notes pane, AI cover art, real charts from the rows.
• render_report — an enterprise PDF dossier (KPI band, chart sections, statistical appendix, citations, methodology) plus an Excel evidence workbook: raw rows, facts, statistics, pivot, correlations.
• render_infographic — a designed KPI poster with the exact figures rendered in-image and a branded provenance strip.
• render_video_briefing — an animated video briefing: platform-drawn charts, spoken narration, motion and crossfades. Six formats (style param): consultant walkthrough, newsroom segment, executive stand-up, documentary deep-dive, field report, social recap.
• render_podcast — a two-host audio briefing with a sealed transcript. Five formats: two-host interview, skeptic's debate, executive stand-up, narrative deep-dive, plain-language walkthrough.
• render_music — a sonified score (tempo ↔ momentum, mode ↔ trend) plus the literal data-motif WAV of the rows. Six genres: cinematic score, corporate uplift, ambient data fields, electronic pulse, orchestral arc, lo-fi data study.

Prefer plain REST? The same keys work on the public API: https://app.alchemylake.com/api/public/v1 (OpenAPI at /api/public/v1/openapi.json). Prefer a terminal? npx alchemylake is a zero-dependency CLI over that API:

export ALCHEMYLAKE_API_KEY=alk_YOUR_KEY
npx alchemylake upload ./q3-actuals.xlsx       # CSV/Excel/PDF/Word/text → governed source
npx alchemylake render report --prompt "Board brief" --source up.1a2b3c
npx alchemylake render deep_research --prompt "Why did Q3 dip?" --source genie:dbx1

How it fits together

AlchemyLake never runs inside this bundle — the App is a thin, SSO'd client that calls the governed platform over HTTPS/MCP. Your tables stay in your lakehouse; only the exact rows you bind for a given render ever leave the workspace, and only to produce that one sealed deliverable.

flowchart LR
    subgraph WS["Your Databricks workspace"]
        UC[("Unity Catalog<br/>tables & volumes")]
        Genie["Genie space"]
        CSV["Uploaded CSV / Excel"]
        App["AlchemyLake App<br/>(this bundle · Streamlit · SSO'd)"]
        Agents["Genie / Agent Bricks<br/>Claude · Cursor"]
    end

    AL(["AlchemyLake<br/>governed render platform"])

    UC -- "bind rows" --> App
    Genie -- "bind an answer<br/>(conversation continues)" --> App
    CSV -- "bind (BYO-data)" --> App
    App <-- "MCP: render, verify, seal" --> AL
    Agents <-- "13 MCP tools, same contract" --> AL
    AL -. "no-egress text tier" .-> FM["Your Databricks<br/>Foundation Model endpoint"]

    AL --> Out(["Sealed deliverable<br/>PDF · PPTX · image · video · audio"])

Every render — from the App, from an agent, or from ai_render() in SQL — goes through the same discipline before it comes back:

flowchart LR
    A["Bind<br/><sub>pick a governed source</sub>"] --> B["Analyze<br/><sub>compute the facts</sub>"]
    B --> C["Design<br/><sub>plan the deliverable</sub>"]
    C --> D["Transmute<br/><sub>generate</sub>"]
    D --> E["Verify<br/><sub>check every claim</sub>"]
    E --> F["Seal<br/><sub>embed provenance</sub>"]

The model is never the source of a number — every figure is computed from your rows first, and whatever the render says is checked against those figures afterward. The seal (source · row count · data hash · verification score) rides embedded in the file itself, in every format, so it survives being forwarded, downloaded, or printed.

Beyond a Genie answer

What people actually ask it for

Every lane binds to a governed source, so the figures in the output are the figures in your table — not a paraphrase. A sample of real workflows across the eight lanes:

Free to try: sign up for 50 credits, no card required, and point this bundle at your own key.

Prerequisites

1. Databricks CLI ≥ 0.230 and authentication to your workspace:

   databricks auth login --host https://<your-workspace>.cloud.databricks.com
   

2. Databricks Apps enabled on the workspace (Premium/Enterprise).
3. An AlchemyLake developer key (alk_…): sign up free (50 credits, no card required) or sign in at https://app.alchemylake.com → Studio → Developer · MCP & keys → Forge a new key.

Path 1 — Deploy the App (one command)

First set your workspace host in databricks.yml — replace the your-workspace… placeholder under both targets.dev and targets.prod (run databricks auth profiles if you need to find it). Then:

databricks bundle deploy -t prod                 # upload source + create the App
databricks bundle run alchemylake_app -t prod    # start it; prints the App URL

CLI note (Terraform pin). Some Databricks CLI builds download a Terraform whose signing key has expired, which breaks bundle deploy. If you hit that, point the CLI at a local Terraform binary and pin the version to match it:

export DATABRICKS_TF_EXEC_PATH="$(command -v terraform)"
export DATABRICKS_TF_VERSION="1.9.8"   # set to your local `terraform version`

This is the exact, verified path AlchemyLake's own reference App is deployed with — it runs live in-workspace as a Streamlit App under SSO.

Open the App URL that the CLI prints. Paste your alk_… key in the sidebar (or bind it from a secret — see below), then Load governed sources → pick one → Transmute. Every result carries its provenance seal.

The pasted key is remembered in that browser (client-side only, until cleared or replaced) so it survives a refresh — a workspace-bound secret still always takes priority. The App also carries a self-contained Docs tab and an expanded About / Install tab (how to use it, why the verification model matters, where the render compute actually runs, and the fastest path to a key) so it's useful even where the public docs site isn't reachable.

Bind the key from a secret (recommended for teams)

So analysts never paste a key:

databricks secrets create-scope alchemylake
databricks secrets put-secret alchemylake api_key      # paste alk_...

Then uncomment the ALCHEMYLAKE_API_KEY / valueFrom block in app/app.yaml and databricks.yml, and redeploy.

Path 2 — Register AlchemyLake as an MCP server (Genie / Agent Bricks)

No bundle required. In your agent/MCP client configuration add:

{
  "mcpServers": {
    "alchemylake": {
      "url": "https://app.alchemylake.com/api/mcp",
      "headers": { "Authorization": "Bearer alk_YOUR_KEY" }
    }
  }
}

Genie / Agent Bricks, Claude, and Cursor will discover the thirteen tools automatically. Ask, for example: “Using AlchemyLake, write a sealed 3-bullet executive summary of gold_ridership_national_monthly” — or go bigger: “Render an 8-slide presentation of that table titled Ridership Momentum” (a .pptx with a presenter script under every slide), “Run deep research on why ridership dipped this quarter” (render_deep_research returns a sealed dossier + evidence workbook), or “Run the campaign-pack recipe” (run_recipe renders the poster, the report, and the copy in one call, all verified and sealed).

Path 3 — ai_render() in SQL / Genie

Edit sql/ai_render.sql (set your catalog/schema) and run it in a SQL editor on serverless compute with external network access enabled. Then:

SELECT ai_render(
  'One-sentence executive read of this quarter''s ridership',
  'ntd_demo.ntd.gold_ridership_national_monthly'
) AS narrative;

See the header of sql/ai_render.sql for the network-egress and secret prerequisites, and the MCP fallback if egress cannot be enabled.

Validate before deploying

databricks bundle validate -t prod

FAQ

Does my data leave the lakehouse? Only the exact rows you bind for a given render are sent to produce that render — nothing else is read, and bound data is never used to train models. A no-egress text tier can route even that narrative step to your own Databricks Foundation Model endpoint instead; see the residency docs.

Do I need Databricks Apps enabled for this? Only for Path 1 (the App). Path 2 (MCP) and Path 3 (ai_render()) need nothing beyond CLI/SQL access and a developer key — no App, no extra compute resource.

Can agents use this without a human in the loop? Yes — that's Path 2. Register /api/mcp once and Genie, Agent Bricks, Claude, and Cursor all gain the same thirteen governed tools, metered and sealed exactly like the UI.

What does a Genie question cost? Genie compute bills inside your Databricks workspace (serverless DBUs — roughly 1.5 DBU ≈ $0.11 per question; Databricks gives every user 150 free Genie DBUs a month). AlchemyLake credits meter only the render itself, and the Studio shows the DBU estimate next to any Genie source.

I don't have Databricks — can I still use everything? Yes. Upload CSV, Excel, PDF, Word, text or Markdown (in the web Studio, this App's Sources tab, the API, the CLI, or the upload_source MCP tool) and every lane — including Deep Research — runs with the same verification and seals.

Is AlchemyLake a Databricks product? No. AlchemyLake is an independent platform by Zorost Intelligence that plugs into Databricks; this repository is Apache-2.0 licensed and is how it plugs in. The hosted platform it talks to is a separate, metered service (50 free credits to start, no card required).

What does this bundle actually contain? Only a Streamlit client (app/app.py), its App runtime config, and an optional SQL function — everything under sql/ and app/. It holds no model weights, no proprietary rendering code, and no secrets; it is a thin, auditable client over the same MCP/REST contract documented at app.alchemylake.com/docs.

Something's broken. Open an issue in this repository, or email [email protected] — see SECURITY.md for vulnerability reports specifically.

Documentation

• docs/DEVELOPER_GUIDE.md — CLI, REST API & MCP: get a key, every command, VS Code/Cursor/Claude setup, use cases, Python/Node examples, GitHub Actions.
• docs/GENIE_GUIDE.md — Genie integration: connect a space, health score, certification, all eight lanes, sharing model.
• Platform docs: app.alchemylake.com/docs (Studio walkthrough, credits, security).

Uninstall

databricks bundle destroy -t prod

AlchemyLake is a product of Zorost Intelligence. Platform docs: https://app.alchemylake.com/docs. Support: [email protected].