SQL migration assistance, AWS Glue job template generation, and Spark code
optimization — as an MCP server.
Not the live-Spark pyspark-mcp package. This project is SQL → PySpark /
Glue source generation, published as pyspark-tools.
introspects a
running SparkSession. A deprecated console script remains here so
old configs keep working; it prints a warning, then starts this server.
SQL migration, AWS Glue job generation, and PySpark optimization as an MCP server.
SQLGlot already transpiles dialects. This MCP adds three things around that kernel: DataFrame-API pretty-printing with join/window/cast mappings that the conversion tests lock, Glue job boilerplate strings (bookmarks, DynamicFrames, catalog tables) so an agent can emit a file instead of assembling one, and a 14-tool FastMCP surface so an LLM picks convert / mode=sql instead of wiring sqlglot itself. If you only need sqlglot.transpile(...), use sqlglot.
Quick Start
pip install pyspark-tools
pyspark-tools
Zero-clone alternative: uvx pyspark-tools. run_server.py is a development convenience that inserts sys.path and prints startup banners. Prefer pyspark-tools in configs and production.
Try it
pip install pyspark-tools
python -c "from pathlib import Path; from pyspark_tools.sql_converter import SQLToPySparkConverter as C; from pyspark_tools.consolidated_tools import glue_job; c,s,o=C(),Path('examples'),Path('examples/out'); [(o/f'{n}.py').write_text(c.convert_sql_to_pyspark((s/f'{n}.sql').read_text(), dialect=d).pyspark_code) for n,d in [('postgres_orders','postgres'),('oracle_decode','oracle')]]; (o/'orders_etl_glue.py').write_text(glue_job(mode='template', job_name='orders_etl', sql_query=(s/'postgres_orders.sql').read_text())['template'])"
Writes the same files as examples/out/. MCP stdio CLI: pyspark-tools.
Example: SQL → PySpark
SELECT o.customer_id, c.name, SUM(o.amount) AS total
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.status = 'paid'
GROUP BY o.customer_id, c.name
Call convert with mode=sql. Captured converter output (dialect=spark):
Exact output depends on dialect detection and fallbacks; conversion tests in tests/test_sql_conversion_fixes.py pin the important constructs. Notebook-style import * / show() is opt-in via style="notebook" on the converter.
mcp:
servers:
pyspark:
command: pyspark-tools
enabled_tools: all
Docker
The image is stdio only (FastMCP over stdin/stdout). There is no HTTP server
on port 8000. docker compose up is for local tests, not a health-checkable
web service.
docker compose --profile test run --rm pyspark-tools-test
Tools
Three primary tools. The other eleven routers stay registered this minor
version but are deprecated — prefer convert, glue_job, and review.
Legacy / deprecated:analyze, optimize, glue_schema, glue_s3,
glue_data, refactor, search, context, batch_status, s3_source,
analytics. Still callable; do not advertise to new agents.
Security
This MCP can read local files (SQL, TXT, PDF) and, if the [aws] extra is installed, list/read S3 with the host's default AWS credentials. File tools only allow paths under the process working directory (or an explicit base_path / FileHandler(base_directory=...)). That is not a sandbox.
Run the server under a restricted OS account. Do not point it at secrets directories. Do not attach AWS credentials with write access unless you intend S3 reads via s3_source / glue_s3. Optional extras:
pip install "pyspark-tools[aws]" # boto3 for S3/Glue catalog helpers
pip install "pyspark-tools[spark]" # pyspark — not required at runtime; generated code only
Development
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
# Test
pytest tests/ -v --cov=pyspark_tools
# Format
black pyspark_tools tests
isort pyspark_tools tests
# Lint
flake8 pyspark_tools tests