The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.
If you have access to more than one Okareo account, simply Clear Authentication and then re-authenticate with the account you want to work in. You can always ask which account your are in and have acccess to.
Local install (alternative)
Run the MCP server on your own machine. Useful for offline / airgapped environments and for development.
Prefer the Remote MCP section above unless you have a specific reason to install locally (airgapped, custom build, development on this repo). The remote endpoint requires no install and stays current automatically.
For multi-org users, the remote endpoint also exposes tenant management tools (list_tenants, switch_tenant).
Step 1: Set Your API Key
export OKAREO_API_KEY="your-api-key"
Add this to your ~/.zshrc or ~/.bash_profile for persistence.
Cursor inherits OKAREO_API_KEY from your shell environment.
Alternative: pip install
If you don't have uv installed:
pip install okareo-mcp
Then use "command": "okareo-mcp" instead of "command": "uvx" with "args": ["okareo-mcp"].
Configuration Reference
Variable
Default
Description
OKAREO_API_KEY
(required)
Your Okareo API key
OKAREO_BASE_URL
https://api.okareo.com
Override for on-prem Okareo backend
TRANSPORT
stdio
Transport: stdio (local) or sse (Docker)
PORT
8000
Port for SSE transport
Available Tools
Scenarios
Tool
Description
save_scenario
Save a named scenario from rows of input/result data (idempotent)
list_scenarios
List all scenarios in the project with names, IDs, and row counts
get_scenario
Retrieve a scenario's metadata and all data rows by name or ID
create_scenario_version
Create a new version of an existing scenario with updated data
preview_delete_scenario
Preview what will be deleted before removing a scenario
delete_scenario
Permanently delete a scenario and all related test data
Generation Models
Tool
Description
list_available_llms
Browse available LLMs from the Okareo registry
register_generation_model
Register a generation model for testing by selecting an LLM from the registry
list_generation_models
List all registered generation models in the project
get_generation_model
Read detailed information about a registered generation model
update_generation_model
Change the LLM a registered generation model points to
delete_generation_model
Remove a registered generation model and all its related test data
Tests & Checks
Tool
Description
list_checks
List available quality checks (built-in and custom) for evaluating model outputs
run_test
Run a quality test that evaluates a model against a scenario using specified checks
list_test_runs
Find past test runs, most recent first. detail_level="summary" (default) or "detailed" for aggregate metrics
get_test_run_results
Load one page of a run's results by ID or name. detail_level="summary" (default) / "detailed" (per-check outcomes + explanations) / "full" (transcripts). Defaults to 20 conversations. Carries a rerun block — the call and config to re-run this simulation
get_conversation_transcript
Retrieve the full conversation transcript for a single data point
reevaluate_test_run
Re-score a completed test run against a (possibly different) set of checks
create_or_update_check
Create or update a quality check by name — model-based, code-based, or audio (upsert)
generate_check
Generate a check from a natural-language description, then save it
get_check
Retrieve a check's full configuration, including its prompt template or code
delete_check
Permanently delete a check by name
calibrate_check
Dry-run a draft check against a finished test run — per-row verdicts and the arguments the check received, saving nothing
Simulations (Multi-Turn)
Tool
Description
create_or_update_target
Create or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio)
get_target
Retrieve a Target's configuration by name (all types)
list_targets
Find simulation targets (voice and custom_endpoint). Bounded to 20 by default; name_contains filters
delete_target
Remove a simulation target and all its related test data
create_or_update_driver
Define a simulated user persona that will interact with your target
get_driver
Retrieve a Driver's full configuration including the persona prompt
list_drivers
Find Driver personas. Bounded to 20 by default; name_contains filters
list_driver_voices
Discover the voices, voice profiles, and languages available for voice drivers
run_simulation
Run a multi-turn conversation evaluation. With based_on_run_id it inherits the original run's full configuration; any argument you pass overrides that one field
list_simulations
List past simulation runs with optional filters (target, scenario, limit)
Voice Monitoring
Tool
Description
ingest_conversations
Submit completed voice conversations to Okareo for monitoring
connect_voice_integration
Connect a voice provider so its traffic flows into Okareo monitoring
list_voice_integrations
List the voice provider integrations in your project
get_voice_integration
Retrieve a voice provider integration by ID, including its status
update_voice_integration
Update a voice provider integration's metadata
rotate_voice_integration_secret
Rotate a voice provider integration's secrets
delete_voice_integration
Delete a voice provider integration by ID
get_voice_webhook_url
Get the inbound webhook endpoint for a voice provider
Projects
Tool
Description
list_projects
List every project you can work in — id, name, tags, archive state — with the active one marked
select_project
Choose the project to work in for this conversation
create_project
Create a new project
update_project
Rename a project, or replace its tags
archive_project
Hide a project from the project picker. Reversible; nothing is deleted
unarchive_project
Restore an archived project to the picker
clone_project
Copy an existing project's scenarios into a new project (dry-run first)
Okareo has no project delete — in the app, the API, or here. Archiving is
the removal, and it only hides the project from the picker: its scenarios,
runs, and dashboards stay intact and every tool still works against it.
Analytics & Dashboards
Tool
Description
query_analytics
Query Okareo's product analytics to understand evaluation trends
list_dashboards
List the analytics dashboards in your project
get_dashboard
Retrieve a dashboard's full configuration by name
save_dashboard
Create or update an analytics dashboard by name (upsert)
reorder_dashboards
Set the display order of dashboards
delete_dashboard
Delete a dashboard by name
Tenant Management (remote MCP only)
Tool
Description
list_tenants
List every Okareo organization you have access to in this MCP session
switch_tenant
Doesn't change the active organization; returns how to reconnect and pick another one at sign-in
Documentation & Templates
Tool
Description
get_docs
Query the Okareo documentation system for conceptual or user-legible explanations
get_templates
Retrieve prompt templates for common Okareo patterns (works offline)
get_reps_baseline
Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release
Troubleshooting
Symptom
Cause
Fix
okareo-mcp: command not found
Not installed or not in PATH
Run pip install -e . (dev) or use uvx okareo-mcp (user)
Server exits with API key error
OKAREO_API_KEY not set
Export it: export OKAREO_API_KEY="..."
pip install fails on Python 3.13+
Okareo SDK requires Python <3.13
Use Python 3.10–3.12
Copilot can't connect (Docker)
Wrong URL
Ensure URL ends with /sse and port matches
Cursor doesn't pick up API key
Cursor launched from Dock, not terminal
Launch Cursor from terminal: cursor .
Contributing
This repository is a curated public mirror; the canonical source is maintained by Okareo. We welcome issues and consider community pull requests — see CONTRIBUTING.md for how proposed changes are reviewed and ported.
"Okareo", the Okareo logo, and Okareo product names are trademarks of Okareo, Inc. and are not covered by the Apache 2.0 license. See TRADEMARK.md for permitted use.