SQL migration assistance, AWS Glue job template generation, and Spark code
optimization — as an MCP server.
Not the live-Spark pyspark-mcp package. This project is SQL → PySpark /
Glue source generation, published as .
introspects a
running SparkSession. A deprecated console script remains here so
old configs keep working; it prints a warning, then starts this server.
SQL migration, AWS Glue job generation, and PySpark optimization as an MCP server.
SQLGlot already transpiles dialects. This MCP adds three things around that kernel: DataFrame-API pretty-printing with join/window/cast mappings that the conversion tests lock, Glue job boilerplate strings (bookmarks, DynamicFrames, catalog tables) so an agent can emit a file instead of assembling one, and a 14-tool FastMCP surface so an LLM picks convert / mode=sql instead of wiring sqlglot itself. If you only need sqlglot.transpile(...), use sqlglot.
Quick Start
pip install pyspark-tools
pyspark-tools
Zero-clone alternative: uvx pyspark-tools. run_server.py is a development convenience that inserts sys.path and prints startup banners. Prefer pyspark-tools in configs and production.
Try it
pip install pyspark-tools
python -c "from pathlib import Path; from pyspark_tools.sql_converter import SQLToPySparkConverter as C; from pyspark_tools.consolidated_tools import glue_job; c,s,o=C(),Path('examples'),Path('examples/out'); [(o/f'{n}.py').write_text(c.convert_sql_to_pyspark((s/f'{n}.sql').read_text(), dialect=d).pyspark_code) for n,d in [('postgres_orders','postgres'),('oracle_decode','oracle')]]; (o/'orders_etl_glue.py').write_text(glue_job(mode='template', job_name='orders_etl', sql_query=(s/'postgres_orders.sql').read_text())['template'])"
Writes the same files as examples/out/. MCP stdio CLI: pyspark-tools.
Example: SQL → PySpark
SELECT o.customer_id, c.name, SUM(o.amount) AS total
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.status = 'paid'
GROUP BY o.customer_id, c.name
Call convert with mode=sql. Captured converter output (dialect=spark):
Exact output depends on dialect detection and fallbacks; conversion tests in tests/test_sql_conversion_fixes.py pin the important constructs. Notebook-style import * / show() is opt-in via style="notebook" on the converter.
mcp:
servers:
pyspark:
command: pyspark-tools
enabled_tools: all
Docker
The image is stdio only (FastMCP over stdin/stdout). There is no HTTP server
on port 8000. docker compose up is for local tests, not a health-checkable
web service.
docker compose --profile test run --rm pyspark-tools-test
Tools
Three primary tools. The other eleven routers stay registered this minor
version but are deprecated — prefer convert, glue_job, and review.
Legacy / deprecated:analyze, optimize, glue_schema, glue_s3,
glue_data, refactor, search, context, batch_status, s3_source,
analytics. Still callable; do not advertise to new agents.
Security
This MCP can read local files (SQL, TXT, PDF) and, if the [aws] extra is installed, list/read S3 with the host's default AWS credentials. File tools only allow paths under the process working directory (or an explicit base_path / FileHandler(base_directory=...)). That is not a sandbox.
Run the server under a restricted OS account. Do not point it at secrets directories. Do not attach AWS credentials with write access unless you intend S3 reads via s3_source / glue_s3. Optional extras:
pip install "pyspark-tools[aws]" # boto3 for S3/Glue catalog helpers
pip install "pyspark-tools[spark]" # pyspark — not required at runtime; generated code only
Development
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
# Test
pytest tests/ -v --cov=pyspark_tools
# Format
black pyspark_tools tests
isort pyspark_tools tests
# Lint
flake8 pyspark_tools tests
अभी टूल सूची नहीं है। रिमोट सर्वर को पहले हेल्थ चेक के बाद सूची मिलती है।
FAQ
सवाल
PySpark MCP कैसे इंस्टॉल करें?
इस पेज पर Cursor, Claude Desktop, Claude Code या VS Code के लिए stdio कॉन्फ़िग कॉपी करें। यह पब्लिशर का इंस्टॉल इस्तेमाल करता है, PlugMCP प्रॉक्सी नहीं।
क्या PySpark MCP के लिए API key चाहिए?
इस रिकॉर्ड में कोई ज़रूरी एनवायरनमेंट वेरिएबल दर्ज नहीं है।
PySpark MCP कौन सा ट्रांसपोर्ट इस्तेमाल करता है?
PySpark MCP stdio पर लोकल चलता है।
PySpark MCP की पिछली जाँच कब हुई थी?
पिछली जाँच 2026-11-08 को हुई। SQL migration, AWS Glue job generation, and PySpark optimization as an MCP server.