Warning
This project is in active development and intended for security testing, research, and educational purposes only. It is not production-ready. Do not deploy in production environments. APIs, tool interfaces, and data formats may change without notice. Use at your own risk.
CrowdSentinel transforms traditional SIEM querying into intelligent, framework-driven investigations using natural language. It serves as a unified security intelligence layer that connects large language models to enterprise security data sources, enabling:
Natural Language Threat Hunting — Query Elasticsearch using plain English
AI-Guided Investigation Workflows — Built-in prompts guide agents through proper IR methodology
Persistent Investigation State — Memory-managed IoC tracking, forensic timelines, and cross-query correlation that survives across sessions (8GB FIFO storage)
Cross-Tool IoC Correlation — IoCs discovered in one tool are automatically available to all others
Multi-Source Analysis — Elasticsearch, EVTX logs (Chainsaw), PCAP files (Wireshark), live endpoint forensics (Velociraptor), local host forensics (osquery)
Velociraptor Endpoint Forensics — 25 MCP tools for live artefact collection (processes, network, persistence, execution evidence, NTFS MFT, SRUM, remote EVTX) with automatic IoC extraction
Encoded-Command Deobfuscation — Automatically peels base64/UTF-16LE/gzip/hex layers off encoded commands (e.g. PowerShell -EncodedCommand) so C2 addresses hidden inside them surface as IoCs
Adaptive Schema Intelligence — Data-driven schema registry (ECS, Sysmon, Windows Security, auditd, Zeek, Packetbeat, AWS CloudTrail) adapts hunts to each index's field conventions
DFIR Knowledge Resources — 9 MCP resources exposing investigation playbooks, Pyramid of Pain reference, and cross-correlation guidance directly to connected AI agents
Standalone CLI — Full threat hunting from the terminal without an MCP client
Installation
Install from PyPI (recommended)
# Install with pip
pip install crowdsentinel-mcp-server
# Or install with uv
uv pip install crowdsentinel-mcp-server
# Download detection rules, Chainsaw, and Sigma rules (one-time)
crowdsentinel setup
# Validate connector configuration and readiness (offline, no external calls)
crowdsentinel doctor
Optional extras:
# Velociraptor live endpoint forensics (adds 25 MCP tools)
pip install 'crowdsentinel-mcp-server[velociraptor]'
# Long-tail deobfuscation coverage via chepy (XOR, exotic encodings).
# The stdlib decoder core works without this — chepy only extends it.
pip install 'crowdsentinel-mcp-server[deobf]'
Detection rules (6,060 Lucene + EQL + ES|QL) are bundled with the package — no download needed. The setup command downloads additional tools:
Chainsaw binary for EVTX analysis
3,000+ Sigma rules for Chainsaw
Downloaded tools are stored in ~/.crowdsentinel/ and persist across package upgrades.
System dependency for PCAP analysis:
# Required for network traffic analysis and cross-tool IoC correlation
sudo apt install tshark # Debian/Ubuntu/Kali
sudo dnf install wireshark-cli # Fedora/RHEL
brew install wireshark # macOS
git clone https://github.com/thomasxm/CrowdSentinels-AI-MCP.git
cd CrowdSentinels-AI-MCP
chmod +x setup.sh && ./setup.sh
# Or install everything, including all optional extras (deobf/chepy, velociraptor)
./setup.sh --full
The setup script will:
Install dependencies (pipx, uv, Claude Code CLI if needed)
Bundle 6,060 detection rules and download Chainsaw binary
Prompt for Elasticsearch credentials (never hardcoded)
Configure the MCP server with Claude Code
Validate your connection
With --full, every optional extra is installed too, so all 139 MCP tools have their dependencies available — the recommended mode for a production-ready server.
Installed Size
CrowdSentinel bundles 6,060 detection rules and integrates with external analysis tools. Below is the full disk space breakdown so you can plan accordingly.
Core package (via pip or uvx):
Component
Size
Notes
CrowdSentinel package
49 MB
The server itself
— Bundled Sigma rules (src/rules/)
30 MB
6,060 pre-converted detection rules
— Elastic TOML rules (src/detection-rules/)
17 MB
Original TOML format rules + hunting queries
— Python code (clients, tools, etc.)
2 MB
Actual application code
Dependencies
64 MB
All transitive deps
— cryptography
14 MB
Largest dependency (TLS)
— elasticsearch
8.3 MB
ES Python client
— pygments
5.2 MB
Syntax highlighting
— pydantic_core
5 MB
Validation engine
— opensearchpy
3.6 MB
OpenSearch client
— Others (27 packages)
~28 MB
mcp, fastmcp, httpx, anthropic, etc.
Core total
113 MB
pip install crowdsentinel-mcp-server
Additional tools (via crowdsentinel setup):
Component
Download
Installed
Notes
Chainsaw binary (v2.13.1)
~3 MB
~15 MB
EVTX log analysis engine
Sigma rules (SigmaHQ)
~3 MB
~30 MB
3,000+ Sigma rules for Chainsaw
Chainsaw mappings
—
<1 MB
Event log source mappings
Setup total
~6 MB
~46 MB
Stored in ~/.crowdsentinel/
System dependency (via package manager):
Component
Installed
Install Command
Notes
tshark + Wireshark libs
~132 MB
sudo apt install tshark
PCAP network analysis — required for cross-tool IoC correlation
Full installation summary:
Scenario
Total Disk Space
Core only (pip install)
~113 MB
Core + setup (crowdsentinel setup)
~159 MB
Full platform (+ tshark)
~291 MB
Note: PyPI download size is only 8.9 MB (wheel) thanks to compression of the bundled detection rules.
Quick Start
1. Set environment variables
export ELASTICSEARCH_HOSTS="https://localhost:9200"
export ELASTICSEARCH_API_KEY="your_api_key"
# Or use username/password:
# export ELASTICSEARCH_USERNAME="elastic"
# export ELASTICSEARCH_PASSWORD="your_password"
export VERIFY_CERTS="false"
2. Connect to an MCP Client
CrowdSentinel works with any MCP-compatible AI agent. Choose your client below:
The analyse --mcp flag replaces deterministic analysis with an AI agent that autonomously uses all 139 MCP tools to investigate. The agent follows the 4-phase IR methodology: hunt, analyse, correlate, report.
Authentication:
# Option 1: Browser sign-in (ChatGPT subscription — no API billing)
crowdsentinel auth login
# Option 2: Anthropic (setup-token or API key)
crowdsentinel auth login --provider anthropic
# Option 3: Environment variable
export ANTHROPIC_API_KEY="sk-ant-..." # or OPENAI_API_KEY
# Option 4: Local models (Ollama, vLLM — free)
crowdsentinel analyse --mcp --model-url http://localhost:11434/v1 --model llama3.1
# Check auth status
crowdsentinel auth status
Threat hunting, detection rules, forensics, endpoint collection, network analysis, cross-correlation, command deobfuscation, and IoC enrichment — all accessible via natural language
Next-step guidance, progressive disclosure, system status, storage cleanup
Velociraptor tools register when VELOCIRAPTOR_API_CONFIG is set; osquery tools register when a local osqueryi binary is present; the 10 high-risk write tools are hidden when DISABLE_HIGH_RISK_OPERATIONS=true.
Security Frameworks
Framework
Purpose
Cyber Kill Chain
Hunt by attack stage (7 stages), predict adversary's next move
Pyramid of Pain
Prioritise IoCs by difficulty for attackers to change (6 levels)
Log Sources: PowerShell, Sysmon, Security Events, Process Creation, Audit logs
Encoded-Command Deobfuscation
Real intrusions hide C2 addresses and URLs inside encodings the SIEM stores verbatim — most commonly PowerShell -EncodedCommand (base64 of UTF-16LE), but also FromBase64String, base64→gzip, bare base64, and hex blobs. CrowdSentinel decodes these layers automatically during IoC extraction and via the dedicated decode_command tool, so the inner indicators surface instead of staying buried:
Zero extra dependencies — the decoder core uses only the Python standard library (base64/zlib/codecs/binascii)
Optional long tail — install the deobf extra (pip install 'crowdsentinel-mcp-server[deobf]') to add chepy for XOR and exotic encodings; its absence degrades gracefully
Provenance — every decode records its recipe (e.g. base64->utf-16le) for the IR report
Safe by design — decode-only (nothing is executed), with depth and output-size bounds so a decompression bomb cannot hang the server
Configuration
Environment Variables
# Connection (required — choose one)
ELASTICSEARCH_HOSTS="https://localhost:9200" # Self-hosted
# OR
ELASTICSEARCH_CLOUD_ID="deployment:base64..." # Elastic Cloud
# Authentication — choose one (in priority order):
ELASTICSEARCH_BEARER_TOKEN="service_token_here" # Service/bearer token
ELASTICSEARCH_API_KEY="your_api_key" # API key (recommended)
ELASTICSEARCH_USERNAME="elastic" # Basic auth
ELASTICSEARCH_PASSWORD="your_password"
# TLS / Certificate verification
VERIFY_CERTS="true" # Verify against system CA bundle
# VERIFY_CERTS="/path/to/ca.crt" # Verify against custom CA certificate
# ELASTICSEARCH_CA_CERT="/path/to/ca.crt" # Explicit CA certificate path
# ELASTICSEARCH_CLIENT_CERT="/path/to/client.crt" # Client certificate (mTLS)
# ELASTICSEARCH_CLIENT_KEY="/path/to/client.key" # Client private key (mTLS)
# Options
REQUEST_TIMEOUT="30" # Request timeout in seconds
DISABLE_HIGH_RISK_OPERATIONS="true" # Block all write operations
# HTTP/SSE transport security (stdio unaffected)
# MCP_ALLOWED_ORIGINS="https://app.example.com" # Extra allowed Origins (comma-separated); loopback always allowed
# Velociraptor endpoint forensics (optional — enables 25 extra tools)
# VELOCIRAPTOR_API_CONFIG="/path/to/api_client.yaml"
# Threat Intelligence (optional — Shodan InternetDB works without any key)
VIRUSTOTAL_API_KEY="your_vt_key" # Free: 500 lookups/day
ABUSEIPDB_API_KEY="your_abuse_key" # Free: 1,000 lookups/day
THREATFOX_API_KEY="your_tf_key" # Free: unlimited
# MISP Integration (optional — offline JSON export works without a server)
MISP_URL="https://misp.example.org" # MISP instance URL
MISP_API_KEY="your_misp_key" # MISP API key (40-char hex)
# MISP_SSL_VERIFY="true" # Set "false" for self-signed (dev only)
Security Warning: Never use VERIFY_CERTS="false" or plain-text passwords in production. Use API keys or service tokens with TLS certificate verification enabled. For self-signed certificates, set ELASTICSEARCH_CA_CERT to your CA certificate path.
This configuration is not suitable for production. Use API keys with TLS in production environments.
Transport Modes
Mode
Use Case
Command
stdio (default)
MCP clients (Claude Code, etc.)
Auto-configured by setup.sh
SSE
Web/streaming clients
--transport sse --port 8000
HTTP
REST API access
--transport streamable-http
The HTTP transports are hardened at the transport layer:
Origin validation (anti DNS-rebinding) — browser requests are rejected unless their Origin is loopback or listed in MCP_ALLOWED_ORIGINS (comma-separated; * alone disables the check). Requests without an Origin header — every non-browser MCP client — are unaffected.
Audit logging — every HTTP exchange is logged at the transport boundary (JSON-RPC method, tool name, body size, path), including requests the tool layer never sees, such as unknown tools or protocol errors. Logs are metadata-only, memory-bounded, and sanitised against log forgery.
Disabling Write Operations
For read-only access (recommended for production):
You: Hunt for brute force attacks in the last hour
CrowdSentinel: Searching for failed authentication attempts...
[Calls hunt_by_timeframe with attack_types=["brute_force"]]
Found 47 failed login attempts from 3 source IPs targeting the "admin" account.
Top sources:
- 192.168.1.105: 28 attempts
- 10.0.0.42: 15 attempts
- 172.16.0.8: 4 attempts
Example 2: Investigate a Suspicious Host
You: Show me all activity on host DESKTOP-ABC123 in the last 24 hours
CrowdSentinel: Generating a forensic timeline for that host...
[Calls get_host_activity_timeline]
Timeline shows:
- 09:15: User "jsmith" logged in (Event 4624)
- 09:17: PowerShell executed encoded command
- 09:18: New scheduled task created "WindowsUpdate"
- 09:22: Outbound connection to 203.0.113.42:443
- 09:25: LSASS memory access detected
Example 3: Kill Chain Analysis
You: We found C2 beaconing. What should we hunt for next?
CrowdSentinel: Hunting adjacent kill chain stages...
[Calls hunt_adjacent_stages with current_stage="COMMAND_AND_CONTROL"]
Looking backwards (Installation stage):
- Found scheduled task persistence on 3 hosts
- Registry Run key modifications detected
Looking forwards (Actions on Objectives):
- Lateral movement to 2 additional hosts via SMB
- Credential dumping attempts detected
Example 4: Network Traffic Analysis
You: Analyse this PCAP for C2 beaconing
CrowdSentinel: Checking for regular connection intervals...
[Calls detect_beaconing on the PCAP file]
Detected beaconing pattern:
- Host: 192.168.1.50 → 203.0.113.100:443
- Interval: 60 seconds (±2s jitter)
- Duration: 4 hours
- Confidence: HIGH
Recommend blocking 203.0.113.100 and investigating 192.168.1.50.
Example 5: CLI Threat Hunt
# Hunt for encoded PowerShell
crowdsentinel hunt "powershell -enc" -i winlogbeat-* --timeframe 1440 -o json
# Pipe results to analysis
crowdsentinel hunt "event.code:4625" -i winlogbeat-* -o json | \
crowdsentinel analyse -c "Failed authentication investigation"
# Search detection rules for lateral movement
crowdsentinel rules --tactic lateral_movement -p windows
The test suite is a bottom-up pyramid (see tests/README.md): hermetic unit tests, then live tiers that spin up a real Elasticsearch (via testcontainers) and the real MCP server.
make test-unit # fast hermetic loop (default developer command)
make coverage # unit tier + coverage policy gate
make check-tests # policy: every new function vs origin/main has a test
make test-integration # real ES via testcontainers (Docker required)
make test-api # + real MCP server subprocess over streamable-http
make test-e2e # + real MCP client journey over the wire
make test-suite # the whole pyramid, bottom-up, with a summary table
CrowdSentinel integrates with Velociraptor for live endpoint forensic artefact collection. When configured, 25 additional MCP tools and 5 cross-correlation tools become available.
Setup
# Install optional dependencies
pip install crowdsentinel-mcp-server[velociraptor]
# Set the Velociraptor API config path
export VELOCIRAPTOR_API_CONFIG="/path/to/api_client.yaml"
The API client config is generated from the Velociraptor server:
Validate SIEM findings on endpoints, pivot from endpoint IoCs to fleet-wide SIEM searches, unified timelines
Cross-Source IoC Sharing
IoCs extracted from Velociraptor artefacts are automatically captured into the active investigation alongside SIEM, Chainsaw, and Wireshark findings. Use get_shared_iocs to retrieve the combined indicator set for cross-source correlation.
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
Licence
GNU General Public Licence v3.0 — See LICENSE for details.
Cyber Range
CrowdSentinel ships with a companion crowdsentinel-range — a fully ephemeral AWS lab that generates ground-truth-labeled attack telemetry so you can investigate real detections rather than synthetic data.
Current range (v1) — 2-host AWS lab
┌──────────────────────────── AWS VPC (private subnet) ────────────────────────────────┐
│ │
│ ┌──────────────────────────────┐ ┌──────────────────────────────────────────┐ │
│ │ Linux SIEM (Ubuntu 22.04) │ │ Windows Range Box (Server 2022) │ │
│ │ Elasticsearch 8.x (TLS) │◄─────│ Sysmon (olafhartong) + Winlogbeat 9 │ │
│ │ Auditbeat + Filebeat │ │ Atomic Red Team + PurpleSharp │ │
│ └──────────────────────────────┘ └──────────────────────────────────────────┘ │
│ SSM VPC endpoints — no inbound ports open │
└───────────────────────────────────────┬────────────────────────────────────────────--┘
│ SSM port-forward :19200
┌──────────▼──────────────────────┐
│ ./range-mcp.sh → CrowdSentinel │
│ MCP Server → Claude Code │
└─────────────────────────────────┘
3 commands to go from zero to hunting:
cd crowdsentinel-range && make up && make attack-windows
cd .. && ./range-mcp.sh
# then in Claude Code: /mcp → "investigate potential intrusions on the windows machine"
Demo: 73,328 events triaged in ~40 seconds — 2 shellcode payloads decoded, C2 IP extracted from raw bytes, full IR report with 19 MITRE techniques.
Future range (v2) — enterprise AD environment
The next range generation targets full Active Directory kill-chain coverage: initial access → lateral movement → domain compromise, with a network security monitor layer, runtime security instrumentation, centralised vulnerability management, and deception infrastructure. Inspired by DetectionLab, Ludus, and SimuLand.
