SpriteGameGen

AI Sprite & Game Asset Tool

A local-first character animation workbench that turns a text prompt (and optional reference image) into reusable animation clips and versioned engine-ready bundles. Azure GPT Image, Gemini, or an independently operated loopback ComfyUI server can do the generative work; a deterministic Python pipeline handles repair, palettes, sprite-sheet packing, frame ZIPs, bundle checksums, and Godot resources. Side-scroller walks also provide Gemini with deterministic eight-phase pose guides, so frames change limb geometry instead of merely restyling the same stance.

Licensed under Apache-2.0. See CONTRIBUTING.md, CODE_OF_CONDUCT.md, and SECURITY.md before contributing or reporting a vulnerability.

Layout

backend/    FastAPI + deterministic image pipeline (Python 3.11+)
frontend/   React + Vite + TypeScript thin client
projects/   per-project output folders (git-ignored)

Backend setup

cd backend
uv sync --locked --extra dev
cp .env.example .env        # then fill in the values below
uv run python ../scripts/doctor.py
uv run pytest -q
uv run uvicorn app.main:app --reload

Environment / auth

This project authenticates to Gemini through Vertex AI (Google Agent Platform) using a service-account JSON key — not a GEMINI_API_KEY. Set in .env:

The service-account key must stay local — it is git-ignored (project-*.json) and must never be pushed.

Region note: Gemini 3.x image models (gemini-3.1-flash-image, gemini-3-pro-image) are served from the global endpoint, not a regional one like us-central1 — a regional endpoint returns a 404 "model not found in region".

First run: rembg downloads its ~170 MB background-removal model on first use, so the first generate/animate call is slow. The onnxruntime inference backend is a declared dependency (installed via pip install -e ".[dev]").

Frontend setup

cd frontend
npm ci
npm run dev     # dev server on http://localhost:5173
npm run build

Testing

• Pipeline (backend/tests/): pure unit tests against committed fixtures — fast, free, deterministic.

• Gemini client: tested against a mocked SDK; no real API calls in CI.

• Frontend (frontend/): npm test — Vitest covers API request shaping, project browser resume/delete behavior, Zustand hydration, FrameStrip regenerate/curation actions, and the AnimationPlayer loop timing math.

• Live smoke test: makes real Gemini calls, manual only (kept out of pytest). Run it from the backend venv after configuring .env:

  cd backend
  # Windows: .venv\Scripts\activate    macOS/Linux: source .venv/bin/activate
  python ../scripts/smoke_generate.py "a knight with a sword"
  

  It generates a base sprite, edits it into one walk frame, runs the pipeline, and writes the results to scripts/smoke_out/ (git-ignored) for inspection. Use this to confirm the live model IDs and SDK signatures still match after any dependency bump.

• Live model validation matrix: probes the configured prompt, generation, and edit models by region; records availability, latency, optional safety-block behavior, and raw/processed artifacts for a scored manual quality review. It is billable, opt-in, outside CI, and writes to a disposable git-ignored directory. See docs/live-model-validation.md for the supported model/region table and acceptance rubric.

  cd backend
  .venv\Scripts\python.exe ..\scripts\validate_live_models.py --repeats 3 --include-block-probe
  

Using the app

With both the backend (uvicorn) and frontend (npm run dev) running, open http://localhost:5173. The saved-project browser loads local projects first; open a healthy project to restore its prompt, sprite, animation frames, and export workflow. Then work through the character workspace:

1. Generate — describe the sprite, pick pixel/hi-res and a game camera, then choose an allowed direction. Prompt enhancement is optional: request a visible preview, edit it, and explicitly accept it before generation. Choose Auto, Azure, or Gemini; Auto prefers Azure, then Gemini, then configured ComfyUI. Hyperagent is shown as Experimental but remains disabled until its agent-mediated image path is validated. You can also attach a reference.
2. Animate — create or replace a named clip from an action pack or custom motion, then preview loop ranges and per-frame durations. Side-scrollers allow left/right; top-down/2.5D projects allow all eight directions.
3. Repair — rerender from retained sources, share a palette, flip, nudge, enable/disable, reset, or regenerate without compounding transforms.
4. Export — download the sheet, atlas, individual-frame ZIP, or deterministic character bundle. The Godot profile includes Godot 4.7 SpriteFrames and scene files.

See bundle and Godot documentation, action packs and recipes, and the ComfyUI security boundary.

Local MCP server

The backend installs sprite-mcp, a local stdio MCP server backed by the same synchronous SpriteService, ProviderRegistry, and project store as FastAPI. Startup is storage-only safe: initialization, capability discovery, project reads, and resource reads do not require cloud credentials. A creative tool fails with a safe tool error if its required provider is not configured.

After installing the backend, register it in an MCP client using the console script:

{
  "mcpServers": {
    "sprite-game": {
      "command": "/absolute/path/to/backend/.venv/bin/sprite-mcp",
      "env": {
        "SPRITE_ENV_FILE": "/absolute/path/to/backend/.env",
        "PROJECTS_DIR": "/absolute/path/to/projects"
      }
    }
  }
}

SPRITE_ENV_FILE must be absolute. PROJECTS_DIR should also be absolute in an MCP client configuration so behavior is explicit even when the client launches the server from a foreign working directory.

Direct MCP contract

The exact direct tool inventory is:

auto prefers Azure when configured and otherwise uses Gemini, but only during initial generation. The resolved concrete provider is persisted. animate and regenerate_frame always use that stored provider and return an error if it is no longer configured; they never silently switch providers.

get_capabilities reports application version 0.1.0, provider availability, action presets, camera/direction combinations, and all prompt, upload, decoded-image, export, sheet, and frame-error limits. FastMCP 1.28.1 does not expose a public FastMCP constructor argument for the application version, so the application version is reported here rather than in the MCP initialization server-version field.

Project outputs are MCP-specific DTOs. They include revision, concrete provider, operation outcome, frame status/errors, absolute local paths, and sprite:// resource URIs, but never expose the persisted/browser-only Frame.url field. The server publishes two read-only resource templates:

sprite://projects/{project_id}/manifest
sprite://projects/{project_id}/assets/{filename}

Both resolve through the canonical PROJECTS_DIR store and reject unsafe IDs, filenames, symlink escapes, and traversal. The server does not accept arbitrary filesystem paths or reference-image paths.

FastMCP 1.28.1 generates useful input schemas from Annotated/Pydantic constraints, but its public tool decorator does not provide a strict-extra option. Unknown top-level tool arguments are therefore ignored by the SDK's generated argument model. This limitation is covered by a contract test; the server does not patch private SDK internals.

Capability boundaries

The tools above are exported directly by the local MCP server. Additional operations may exist internally in the framework-neutral runtime; absence from tools/list only means an operation is not a direct MCP tool.

Hyperagent remains an experimental remote-agent capability. It appears in provider capability metadata as unavailable. Its authenticated, agent-mediated image path still requires separate validation; do not infer remote-agent capability from the direct server's tool inventory.

Run the credential-free protocol smoke test from the repository root:

backend\.venv\Scripts\python.exe scripts\smoke_mcp.py

The smoke script launches the installed sprite-mcp console entrypoint from a temporary foreign CWD with absolute temporary SPRITE_ENV_FILE and PROJECTS_DIR values, removes cloud credentials, initializes stdio, asserts the exact tool inventory, calls get_capabilities, and prints only its parent-process success line.

Agent skill

A portable Agent Skills-compatible guide is available at skills/sprite-game-gen/. Install that complete directory in your agent host's skill path and connect the sprite-mcp server. The skill teaches agents to discover live capabilities, distinguish billable and destructive operations, preserve partial animation results, verify project revisions, and use sprite:// resources safely. Its setup reference includes a client-neutral stdio configuration example.

Versioning and releases

The package, project manifest, bundle, action-pack, recipe, and batch-state versions evolve independently. See version contracts, the release process, and CHANGELOG.md.