HeyClaw 🦞
A lightweight, open-source, voice-first personal AI agent.
HeyClaw listens locally for a wake word, opens a real-time voice session, and connects speech to a small, readable Python agent powered by your choice of Gemini, OpenAI, or Anthropic, plus Mem0, MCP tools, and workspace skills. It keeps the device-facing audio path separate from the agent backend, so the intelligence can run remotely while each room only needs a lightweight satellite.
Why HeyClaw
- Voice first: wake word, microphone input, streamed responses, and interruption-aware speech are the primary interface.
- Small, readable core: the agent loop is intentionally compact and avoids a large orchestration framework.
- Persistent memory: Mem0 retrieves relevant user context and stores durable details across conversations.
- Tool ready: MCP servers provide external capabilities, while workspace skills define when and how the agent may use them.
- Self-hosted orchestration: the backend, prompts, workspace, tools, and configuration remain under your control.
- Satellite architecture: audio capture and wake-word detection are isolated from the backend and ready to move onto dedicated hardware.
What HeyClaw can do
HeyClaw currently provides:
- local wake-word detection with openWakeWord;
- real-time speech-to-text and text-to-speech through ElevenLabs Speech Engine;
- configurable Gemini, OpenAI, or Anthropic language models through DSPy;
- a DSPy ReAct agent with dynamically discovered MCP tools;
- long-term, user-scoped semantic memory through Mem0;
- runtime instructions, identity, user profile, and skills loaded from a local workspace;
- a FastAPI service for health and readiness checks;
- separate backend and satellite Python packages.
🏗️ How it fits together
flowchart TD
User([User]) <-->|Voice| Satellite[Satellite<br/>Wake word, microphone, speaker]
Satellite <-->|Real-time audio| ElevenLabs[ElevenLabs Speech Engine<br/>STT, turn taking, TTS]
ElevenLabs <-->|Transcript and streamed response| Backend[HeyClaw backend]
Backend <--> LLM[Gemini, OpenAI, or Anthropic]
Backend <--> Mem0[Mem0 long-term memory]
Backend <--> MCP[MCP tools]
Backend --> Workspace[Workspace instructions and skills]
The repository contains two independent Python projects:
heyclaw/contains FastAPI, the ElevenLabs Speech Engine server, DSPy, Gemini/OpenAI/Anthropic integrations, Mem0, MCP integration, and the runtime workspace.satellite/contains local audio, echo handling, openWakeWord detection, and the ElevenLabs conversation client.
ElevenLabs handles the speech layer. When a user finishes speaking, it sends the transcript to HeyClaw's public WebSocket endpoint. HeyClaw retrieves relevant memories, lets the model use applicable skills and MCP tools, and streams the final text back for speech synthesis.
Requirements
Both platforms require Python 3.12, uv, GNU Make, ngrok, and working microphone and speaker devices.
- Linux and WSL additionally require
lsof. - Native Windows requires PowerShell 7 with
pwsh.exeavailable inPATH.
Linux and WSL use Makefile. Native Windows uses the separate Makefile.windows, which delegates process management to heyclaw/scripts/dev.ps1.
ngrok CLI
HeyClaw runs its Speech Engine WebSocket server locally on port 3001, but ElevenLabs must be able to connect to it from the public Internet. ngrok provides the public wss:// endpoint that forwards traffic to the local server without router configuration.
- Create or sign in to an ngrok account.
- Open the official ngrok CLI setup page.
- Select your operating system and follow the displayed installation instructions.
- Add the authentication token shown by the dashboard to the CLI configuration.
- Ensure
ngrokis available inPATH. The platform-specificbackendtarget starts and supervises the configured tunnel automatically.
The Makefile reads gateway.publicWsUrl from heyclaw/config.json, validates that it is a wss:// URL ending in /ws, and gives its hostname to ngrok.
sequenceDiagram
participant S as Satellite
participant E as ElevenLabs
participant N as ngrok public endpoint
participant H as HeyClaw on port 3001
S->>E: Start voice session
E->>N: Open Speech Engine WebSocket and send transcript
N->>H: Forward connection and transcript to /ws
H-->>N: Stream agent response
N-->>E: Forward response through the tunnel
E-->>S: Synthesize and stream speech
This connection model follows the official ElevenLabs Speech Engine documentation: ElevenLabs performs speech recognition and synthesis, while HeyClaw supplies the LLM, memory, skills, and tool logic.
🔑 Configuration
After installing the dependencies, create or refresh both JSON configuration files with the satellite CLI:
uv run --project satellite heyclaw-satellite onboard
onboard never replaces configured values. It creates missing files from their
examples, adds fields introduced by newer versions, and fills unset audio device
indices only when PortAudio can make a conservative recommendation. Existing API
keys, IDs, model choices, and audio selections are preserved.
After connecting a different microphone or output device, refresh only the audio
selection while preserving every other setting. In an interactive terminal,
onboard presents the connected input and output devices as selectable lists:
uv run --project satellite heyclaw-satellite onboard --update-audio
If Windows keeps the old defaults after you connect new hardware, list the current PortAudio indices and select the new endpoints explicitly:
uv run --project satellite heyclaw-satellite devices
uv run --project satellite heyclaw-satellite onboard --update-audio --input-device-index 2 --output-device-index 4
Echo suppression modes
defaults.agent.echoSuppressionMode controls how satellite playback is kept out
of the microphone stream:
offkeeps full-duplex audio and performs no software suppression. Use it with headphones or a speakerphone that exposes a matching hardware echo-cancelled input/output pair. With ordinary speakers, playback can be transcribed as a new user turn and trigger loops.gatecloses the microphone while the assistant is speaking and forechoGuardMsafterward. It is the safest default for separate speakers, at the cost of preventing the user from interrupting playback.aecruns software acoustic echo cancellation using the playback stream as its reference. It keeps interruption possible, but results depend on device and driver timing; use it only after a real speaker test.
agc (automatic gain control) is a different audio process and is not a supported
echoSuppressionMode value.
Prepare the logging environment files separately:
heyclaw/.envfromheyclaw/.env.example;satellite/.envfromsatellite/.env.example;
The .env files contain logging settings only. Provider credentials, agent defaults, audio settings, memory, and MCP servers belong in the corresponding config.json files.
ElevenLabs
- Sign in to ElevenLabs and open the API Keys page.
- Create an API key.
- Set
providers.elevenlabs.elevenlabsApiKeyin bothheyclaw/config.jsonandsatellite/config.json. - Set
gateway.publicWsUrlinheyclaw/config.jsonto your public secure WebSocket endpoint ending in/ws. - Create an ElevenLabs Speech Engine connected to that WebSocket URL by following the official Speech Engine quickstart.
- Copy the resulting
seng_...identifier intoproviders.elevenlabs.elevenlabsSpeechEngineIdin both configuration files.
The API key and Speech Engine ID must match across the backend and satellite.
Language model provider
Select the hosted LLM with defaults.agent.llmProvider, set its model in defaults.agent.llmModel, and add only the matching API key under providers.
Gemini remains the default. Create credentials in Google AI Studio, the OpenAI API platform, or the Anthropic Console. Model names may be written with or without their matching DSPy provider prefix, for example gpt-5.6-luna or openai/gpt-5.6-luna.
For real-time voice use, the current recommended choices are:
- OpenAI:
gpt-5.6-lunafor the lowest latency and cost,gpt-5.6-terrafor a balanced option, orgpt-5.6-solfor maximum capability; - Anthropic:
claude-sonnet-5for the best general balance,claude-opus-5for maximum capability, orclaude-haiku-4-5-20251001when latency and cost matter most.
Some current reasoning models accept only their default temperature. HeyClaw detects these model families and omits llmTemperature automatically while continuing to use the configured value for models that support it.
To switch to OpenAI, for example:
{
"defaults": {
"agent": {
"llmProvider": "openai",
"llmModel": "gpt-5.6-luna"
}
},
"providers": {
"openai": {
"openaiApiKey": "your_key_here"
}
}
}
Mem0
- Sign in to the Mem0 API Keys dashboard.
- Create a platform API key.
- Store it as
defaults.memory.mem0.apiKeyinheyclaw/config.json.
Mem0 is the agent's persistent memory layer. HeyClaw searches user-scoped memories before answering and stores only durable user information after a conversation.
Perplexity web search
- Sign in to the Perplexity API Console.
- Create an API group if your account does not already have one.
- Generate an API key and save it when it is displayed; Perplexity does not show the complete key again.
- Replace
your_key_hereattools.mcpServers.perplexity.env.PERPLEXITY_API_KEYinheyclaw/config.json.
The configured Perplexity MCP server gives the web-search skill access to current information.
🚀 Quick start
After starting the backend and ngrok tunnel for the first time, create the ElevenLabs Speech Engine from a second terminal:
uv run --project heyclaw heyclaw-create-engine
Run this command again after changing ElevenLabs accounts. It creates the Speech
Engine with the required first_message override enabled. Copy the generated
seng_... ID into both config.json files before starting the satellite.
Linux and WSL
Install Python dependencies for both components from the repository root:
make setup
Start the backend, Speech Engine, and ngrok tunnel:
make backend
In another terminal, start the voice satellite:
make satellite
Say the configured wake word, then speak normally. Press Ctrl+C to end the active process.
Native Windows
Open PowerShell 7 in the repository root and install dependencies for both components:
make -f Makefile.windows setup
Start the backend, Speech Engine, and ngrok tunnel:
make -f Makefile.windows backend
In another PowerShell 7 terminal, start the voice satellite:
make -f Makefile.windows satellite
The Windows Makefile must always be selected explicitly with -f Makefile.windows. Its PowerShell helper reads the ngrok hostname from heyclaw/config.json, supervises the backend and tunnel, and stops both when either exits or the command is interrupted.
Active Makefile commands
The two Makefiles expose the same active targets. Use make <target> on Linux/WSL and make -f Makefile.windows <target> on native Windows.
Always specify a target: neither Makefile defines an active default help target.
🎙️ Changing the wake word
HeyClaw uses openWakeWord locally, before opening a voice session. andromeda is the bundled default, and veronica is also included.
To use another model:
- Browse the Home Assistant Wake Word Collection or the pretrained models published by openWakeWord.
- Download an openWakeWord-compatible
.tflitemodel and its license or accompanying attribution. - Place the files under
satellite/app/audio/models/. - Set
defaults.agent.wakeWordModelinsatellite/config.jsonto its path relative tosatellite/, for example:
{
"defaults": {
"agent": {
"wakeWordEnabled": true,
"wakeWordModel": "app/audio/models/my_wake_word.tflite",
"wakeWordThreshold": 0.5
}
}
}
You can also use the name of a model distributed directly by openWakeWord, such as hey_jarvis; HeyClaw downloads supported named models when needed. Restart the satellite after changing the model.
The default threshold is 0.5. Raise it to reduce false activations or lower it to make detection more sensitive, then test it with the actual microphone, room acoustics, distance, and accents used in deployment.
[!IMPORTANT] A
.tfliteextension alone does not guarantee compatibility. Use a model built for openWakeWord's audio preprocessing and inference pipeline, and preserve the model author's license alongside it.
Workspace and skills
The runtime workspace lives in heyclaw/workspace/:
AGENTS.mddefines operating rules;SOUL.mddefines the assistant's identity and speaking style;USER.mdprovides explicit user context;TOOLS.mdcontains general tool guidance;skills/*/SKILL.mddescribes specialized procedures and their required MCP tools.
Skills are discovered at startup and loaded only when relevant. This keeps the base context small while making tool use explicit and auditable.
🦞 Roadmap
The next major step is dedicated satellite firmware for the reSpeaker XMOS XVF3800 with XIAO ESP32S3. The target design keeps wake-word detection and audio handling on the device while securely streaming active conversations to a remote HeyClaw backend, with device identity, reconnection, mute and LED states, and OTA updates.
Credits
HeyClaw's small-core philosophy and workspace-oriented agent design were inspired by:
- HKUDS/nanobot, an ultra-lightweight Python personal agent;
- openclaw/openclaw, the personal assistant project that also inspired nanobot.
Wake-word support is made possible by:
- fwartner/home-assistant-wakewords-collection, the community collection that provides the bundled Andromeda and Veronica models and many alternatives;
- dscripka/openWakeWord, the local wake-word detection framework used by the satellite.
HeyClaw also builds on the work of ElevenLabs, Google Gemini, OpenAI, Anthropic, DSPy, Mem0, the Model Context Protocol ecosystem, and their open-source communities.
