Agent Mail Gateway

Agent Mail Gateway

Give every AI agent its own email address — without giving it the keys to the mailbox.

Agent Mail Gateway is a small self-hosted service (one Docker container) that sits between your AI agents and ordinary mailboxes on your existing mail server. Each agent gets one API key for exactly one mailbox. Through the gateway it can read and answer mail, send attachments and manage calendar invitations — but only with the people you allow, and every outgoing message is checked before it leaves.

Agents talk to it over MCP (Model Context Protocol) or a plain REST API. Mail arrives as Markdown instead of HTML, which saves a lot of tokens.

Why

Giving an agent the IMAP/SMTP password of a mailbox means it can read everything, write to anyone, and a single prompt injection in an incoming email can make it leak data or spam people. The gateway keeps the password to itself and enforces your rules on every request:

Features

Mailbox access

• List, search, read, mark, delete; attachments in and out — as text, base64 or straight from a received message, so agents never have to juggle base64 for a CSV.
• PDFs and images as text for the agent (get_attachment with format: "text"): PDF text layer built in, images via a vision model or local OCR (Tesseract).
• Reply (incl. reply-all) and forward — the gateway fills in recipients, Re:/Fwd: and threading.
• HTML → Markdown for reading, Markdown → HTML for sending.

Calendar

• Send, update and cancel invitations; they update cleanly in Outlook, Gmail and Apple Calendar.
• See who accepted or declined your invitations.
• Read invitations you receive and accept, decline or tentatively accept them.

Control and safety

• Allow lists per mailbox for who the agent may receive mail from and write to (name@domain or *@domain). Everything else is invisible to the agent and moved to Trash (or kept).
• Forged senders are rejected (SPF/DKIM/DMARC as checked by your mail server).
• Attachment and content checks on incoming mail, each set to hide, flag or off and skippable per sender: dangerous files (programs, macros, disguised file types — on by default), virus scan with ClamAV, and a prompt injection check of text, PDFs and images (details below).
• Send limit per hour, audit log of every send, rejection and deletion.
• No fast retries on wrong passwords, so your server's fail2ban never blocks the gateway.

Review before sending (details below)

• Built-in rules catch empty mails, attachment-only mails, "see attached" without attachment, leftover placeholders and accidental duplicates — on by default.
• Optional LLM review (e.g. a local model in Ollama) for "is this mail complete and sensible?".
• Optional policies: your own rules in plain language, for everyone or for specific recipients — e.g. "never share financial information", "never mention gifts to Alex".

Integration

• MCP over Streamable HTTP, a stdio bridge for stdio-only clients, and a REST API with OpenAPI docs at /docs.
• Signed webhooks when new mail arrives (compatible with Hermes Agent webhook routes).
• Hermes Agent plugin with a skill that teaches the agent how to use its mailbox.
• Works with any IMAP/SMTP server: Plesk, IONOS, Outlook, Postfix/Dovecot, …

How it works

 Agent ──MCP / REST + API key──▶ ┌────────────────────────────────────────┐
                                 │ Auth       key → exactly one mailbox   │
 Agent ◀──signed webhook──────── │ Policy     who may write / be written  │
                                 │ Review     rules · LLM · your policies │
                                 │ Inbound    file types · virus · inject.│
                                 │ Converter  HTML ⇄ Markdown             │
                                 │ Calendar   invitations and replies     │
                                 │ Mailbox    IMAP + instant new-mail push│──IMAP──▶ your mail server
                                 │ Sender     SMTP + copy to "Sent"       │──SMTP──▶
                                 └────────────────────────────────────────┘
                                   config.yaml + .env (read-only)

Mail stays on your mail server; the gateway only keeps a small SQLite file with its own state.

Quick start

Use a dedicated mailbox for each agent (for example [email protected]), not your personal one: by default, mail from senders that are not on the allow list is moved to Trash.

1. Get the files:

   mkdir agent-mail-gateway && cd agent-mail-gateway
   curl -LO https://raw.githubusercontent.com/dominikamann/agent-mail-gateway/main/docker-compose.yml
   curl -L -o config.yaml https://raw.githubusercontent.com/dominikamann/agent-mail-gateway/main/config.example.yaml
   curl -L -o .env https://raw.githubusercontent.com/dominikamann/agent-mail-gateway/main/.env.example
   

2. Edit config.yaml: one entry per agent with its mail server, login and allow lists.
3. Put the secrets into .env (openssl rand -hex 32 makes a good API key).
4. Start it and check:

   docker compose up -d
   curl http://localhost:8080/health   # {"status":"ok"}; with a key also that mailbox's state
   curl -H "Authorization: Bearer <the AGENT_API_KEY from .env>" http://localhost:8080/v1/mailbox
   

A minimal mailbox entry:

mailboxes:
  - name: assistant
    address: [email protected]
    api_key: ${AGENT_API_KEY}
    imap: { host: mail.yourmailserver.eu, port: 993, security: tls }
    smtp: { host: mail.yourmailserver.eu, port: 465, security: tls }
    username: [email protected]
    password: ${AGENT_MAIL_PASSWORD}
    allow_receive_from: [[email protected], "*@yourcompany.eu"]
    allow_send_to: [[email protected]]

Every option is explained in docs/configuration.md.

Review before sending

Agents are sometimes sloppy — an email with only an attachment, "please find attached" without a file, Hello {name}. The gateway checks every outgoing email and invitation before it is sent. If something is wrong, nothing is sent and the agent gets a clear error with the reasons (review_rejected), so it can fix the message and try again. Rejected attempts don't count towards the send limit.

There are three layers; you choose per mailbox:

Rules and the LLM review can block the message, only warn (send anyway and report it), or be switched off; each policy rule either blocks or warns. The check covers new mails, replies, forwards, invitations and their changes, and comments in invitation answers.

    review:
      rules: block                       # block | warn | off
      llm:                               # any OpenAI-compatible endpoint, e.g. a local Ollama
        url: http://ollama:11434/v1
        model: llama3.1:8b
        mode: warn                       # quality check: warn | block | off
      policies:
        mode: block                      # what a violation does: block | warn
        rules:
          - rule: Never share financial information such as revenue, prices, invoices, bank details or salaries.
          - rule: Never mention gifts, presents or surprise plans.
            recipients: [[email protected]]
          - rule: Never send calendar invitations.
            recipients: [[email protected]]

Policies also read attachments: text files, PDFs, and images when attachments.read_images is set. A rule with recipients applies only when one of those people receives the message (To, Cc, Bcc or invitation attendee). With Ollama the review runs entirely on your own machine; no mail content leaves your server. Long messages are checked in parts — nothing is cut off; part size and limits are configurable to fit your model's context window. The built-in review prompt can be replaced or extended — see docs/configuration.md.

Incoming mail checks

Allow lists decide who can reach your agent; these checks look at what arrives. For each check you choose what a finding does: hide the message (like a sender that is not allowed), flag it (the agent sees it with a warning and cannot download dangerous files), or off. With skip_senders a check is skipped for trusted senders — so you can still send your agent instructions by mail.

    attachments:
      clamav: { host: clamav, port: 3310 }          # optional, see docker-compose.yml
      read_images: { engine: ocr, languages: [eng, deu] }   # or a vision model
    inbound_checks:
      prompt_injection:
        action: flag
        skip_senders: [[email protected]]       # your own instructions are not flagged
      dangerous_attachment: { action: hide }

Images are read by a vision model (e.g. qwen2.5vl in Ollama) or by OCR inside the gateway (Tesseract); PDFs by pdf.js. Everything runs on your own machine if you want it to. Details: docs/configuration.md.

Using it

MCP tools

Point any MCP client at http://<host>:8080/mcp with the header Authorization: Bearer <api key>, or use the stdio bridge.

REST

# send a message
curl -X POST http://localhost:8080/v1/messages \
  -H "Authorization: Bearer $AGENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"to":["[email protected]"],"subject":"Daily report","body_markdown":"All **green** today."}'

# unread mail
curl -H "Authorization: Bearer $AGENT_API_KEY" "http://localhost:8080/v1/messages?unread=true"

All endpoints, error codes and the webhook format: docs/api.md.

Hermes Agent

Connect the MCP server in ~/.hermes/config.yaml and install the plugin that teaches your agents to use their mailbox safely:

hermes plugins install dominikamann/agent-mail-gateway/integrations/hermes/agent-mail-gateway --enable

Step by step, including waking the agent on new mail: docs/hermes.md.

Documentation

• Configuration — every option, allow lists, review, ports, sender authentication
• REST API, webhooks and MCP tools
• Hermes Agent integration and plugin
• stdio clients
• Security model
• Contributing · Changelog

Security

Run the gateway on a private network or behind a reverse proxy with TLS — API keys travel in the Authorization header. Report vulnerabilities privately as described in SECURITY.md.

License

MIT