Athena MCP RAG Server

Athena MCP RAG Server

Production-ready Pinecone-based RAG server for Claude Desktop with large-scale PDF ingestion, evaluation pipelines, and AI teaching tools

Quick Setup

1. Clone & Install Dependencies

git clone https://github.com/HarshRohilla04/Athena-MCP-RAG-server.git
cd Athena-MCP-RAG-server
uv sync

uv pip install -r requirements.txt
# OR
uv sync  # uses pyproject.toml

2. Pinecone Setup

-Create account at https://www.pinecone.io/

-Go to API Keys → copy your api key

-Create index:

Name: rag-storage   # name your rag storage 
Modality: Text  
Vector type: Dense  
Dimension: 1024  
Metric: cosine 

3. Create .env

PINECONE_API_KEY = "your api key"

4. Main.py Changes

# ----------------------------------------------------------------------
# Global Configuration
# ----------------------------------------------------------------------
pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))
index = pc.Index("name of the index")

5. Install to Claude Desktop

uv run fastmcp install claude-desktop main.py

6. Configure Claude Desktop

{
  "SERVER_NAME": {
    "command": "C:/Users/YOUR_USERNAME/Athena-MCP-RAG-server/.venv/Scripts/python.exe",
    "args": ["C:/Users/YOUR_USERNAME/Athena-MCP-RAG-server/main.py"],
    "env": {
      "PINECONE_API_KEY": "your api key"
    },
    "transport": "stdio",
    "cwd": "C:/Users/YOUR_USERNAME/Athena-MCP-RAG-server",
    "timeout": 600
  }
}

7. Launch

1. Save the JSON file

2. Close Claude Desktop completely

3. Open Task Manager → end all Claude processes

4. Restart Claude Desktop

Configuration

Athena uses environment variables for all api keys, file paths, credentials, and model configuration. This makes the project portable across machines and operating systems.

Create a .env file using .env.example as reference.

Required Environment Variables

# SAMPLE .ENV

PINECONE_API_KEY=your_key_here
PINECONE_INDEX_NAME=athena-rag
PINECONE_NAMESPACE=ml-theory-algorithms-text
OPENAI_API_KEY=your_api_key_here

# Documents
BOOK_PDF=/absolute/path/to/your/book.pdf
BOOK_NAME=Understanding Machine Learning: Theory and Algorithms

# Evaluation
TESTSET_CSV_PATH=/absolute/path/to/testset.csv

# Models
EMBEDDING_MODEL_NAME=BAAI/bge-large-en-v1.5
LLM_MODEL_NAME=gpt-4o-mini

Project Structure

Athena-MCP-RAG-server/
├─ main.py                 # MCP server for Claude Desktop
├─ pinecone_ingestion.py   # Offline PDF ingestion
├─ testset_generator.py    # Synthetic QA generation (RAGAS)
├─ evaluate_rag.py         # RAG evaluation pipeline
├─ .env.example            # Environment variable template
├─ requirements.txt
├─ testset_small(10Q).csv
├─ testset_large(100Q).csv
└─ README.md

Pinecone Ingestion

This script ingests a textbook-scale PDF into Pinecone using a production-quality semantic ingestion pipeline.

Key features:

1. Page-aware PDF parsing using pdfplumber
2. Text cleaning to remove common PDF artifacts
3. Semantic chunking (512 tokens, 100 overlap)
4. Context-enriched chunks (book title + page number)
5. High-quality embeddings using BAAI/bge-large-en-v1.5
6. Batched upserts into Pinecone

Required configuration:

1. BOOK_PDF – path to the textbook PDF
2. BOOK_NAME – title of the document
3. PINECONE_INDEX_NAME
4. PINECONE_NAMESPACE
5. PINECONE_API_KEY

Run ingestion:

python pinecone_ingestion.py

Note: Due to context window and transport limits, large PDFs are best ingested offline via pinecone_ingestion.py, while smaller documents can be safely uploaded through the Claude Desktop MCP interface.

RAG Evaluation

Athena includes a complete RAG evaluation pipeline built using the RAGAS framework. This allows users to quantitatively assess retrieval and generation quality rather than relying on subjective judgment.

Metrics Evaluated

The evaluation pipeline reports the following metrics:

1. Context Precision – How relevant the retrieved chunks are.
2. Context Recall – How well the retriever covers the required information.
3. Faithfulness – Whether the answer is grounded in retrieved context.
4. Answer Relevancy – How well the answer addresses the question.

The evaluation logic is implemented in:

evaluate_rag.py

Evaluation process:

1. Load questions from a CSV testset
2. Retrieve context from Pinecone
3. Generate answers using the LLM
4. Compute RAGAS metrics

Users can run the evaluation directly once documents are ingested into Pinecone.

TestSet Generation

This script generates synthetic question–answer pairs from the source PDF using the RAGAS testset generator.

Generation process:

1. Loads the source PDF
2. Splits the text into semantic chunks
3. Generates high-quality QA pairs using an LLM
4. Exports the dataset as a CSV file

Required configuration:

1. TESTSET_CSV_PATH
2. PINECONE_INDEX_NAME
3. PINECONE_NAMESPACE
4. EMBEDDING_MODEL_NAME
5. LLM_MODEL_NAME

Run testset generation:

python testset_generator.py

Output:

testset.csv

Note: MiniLM embeddings are used only during testset generation. Retrieval and evaluation use BGE-large embeddings

Evaluation Testsets

To make evaluation reproducible and easy to run, this repository includes two pre-generated testsets derived from the textbook Understanding Machine Learning.

Available Testsets

1. testset_small(10Q).csv 10-question lightweight testset for quick sanity checks
2. testset_large(100Q).csv 100-question comprehensive testset for robust evaluation

Each testset contains:

1. High-quality synthetic questions
2. Ground-truth reference answers
3. Designed to test definitions, theoretical concepts, and multi-hop reasoning

These CSV files allow users to evaluate Athena’s RAG performance immediately without regenerating test data.

Reference Textbook

This project is built around the book:

Understanding Machine Learning: From Theory to Algorithms Shai Shalev-Shwartz and Shai Ben-David Download Book

The book is used for:

1. Document ingestion
2. Testset generation
3. Quantitative RAG evaluation

ATHENA is ready for use!