myKG — The Knowledge Graph Engine
myKG automatically generates a confidence-scored knowledge graph from a set of mixed documents — Markdown, plain text, PDF, Word, PowerPoint, Excel, HTML, and images — grounded in an induced RDFS/OWL ontology.
Updates
-
v0.4.10 — Schema drift detection and pruning:
mykg detect-schema-driftreports concepts and properties the graph no longer uses, andmykg prune-schemaremoves them (with--preview, an editable plan file, and a backup). See Schema Drift & Pruning. -
v0.4.9 — The
nodes.jsonl,edges.jsonlandknowledge_graph.ttlexports now stream to disk record by record instead of building the whole file in memory first, so peak memory stays flat as the graph grows. -
v0.4.8 —
mykg_search_nodesandmykg_query_graph's seed selection now use BM25 instead of a linear substring scan, ~2,800x faster at 100K nodes (0.5ms vs. 1.4s per query) and no longer degrading to minutes per query as the graph grows.
Contents
- Features
- Quick Start
- Command line
- Articles & Tutorials
- Configuration
- Extract Pipeline
- Outputs
- MCP (Model Context Protocol) Server
- Using mykg with Claude Code
- Advanced Options
Features
MyKG builds trustworthy knowledge graphs through a self-evolving ontology that continuously adapts, maintains consistency, and assigns confidence scores to knowledge, keeping information grounded and reliable as it grows.
Ontology-Guided Extraction
-
Schema-guided knowledge graph generation — the extracted graph is always grounded in a formal RDFS/OWL schema: concept types, property names, domain/range constraints, and the is-a hierarchy are explicit and inspectable before any entity is extracted
-
AI coding assistant friendly — designed for smooth use alongside AI coding assistants such as Claude Code; Use the Claude Code Skill to run extractions, inspect outputs, and iterate on your knowledge graph without leaving your coding environment; see Using mykg with Claude Code
-
Second brain for AI coding assistants — the Obsidian vault output turns your extracted knowledge graph into a directory of wikilinked Markdown notes that any AI coding assistant can read as project context; point Claude Code, Cursor, or Copilot at
output/obsidian_vault/and ask questions, trace relationships, and get answers grounded in your own documents -
MCP server for desktop AI apps — run
mykg mcp-serveto expose your knowledge graph via the Model Context Protocol; integrates with Claude Desktop, Cherry Studio, and any MCP-compatible client — 16 query tools let LLMs search entities, explore relationships, find paths, traverse the graph, and read wiki notes directly from your extracted knowledge; see MCP Server -
Incremental updates — append new files to an existing session, extracting only what changed. Optionally grow the schema from new documents while preserving existing concepts and properties
-
Resumable pipeline — every stage persists intermediate state; re-enter at any step after a crash or edit
-
Session isolation — each run is fully self-contained; inputs, intermediate state, outputs, and logs co-located
-
Cross-session merge — combine two independently-produced graphs into one unified knowledge graph
-
Bring your own ontology — supply a
--base-schemaTTL file (RDFS or OWL) to lock in classes and properties from an existing formal ontology; the LLM expands it with domain-specific concepts but will not rename, remove, or contradict your authoritative vocabulary. Please note that this mechanism is controlled by the LLM, and may not be strictly enforced. Add to skip LLM schema induction entirely and extract from the documents strictly against your ontology verbatim — no surprise types, no invented properties
Input
-
Mixed-format corpora — point
mykg extract-graphat any directory; supported extensions are converted to Markdown automatically before ingest:Anything outside the allowlist (e.g.
.svg,.css,.phpassets next to an HTML bundle) is logged and skipped, never silently dropped. The allowlist is configurable viapreprocess.extensionsinmykg_config.yaml.Incremental conversion — unchanged source files are skipped on re-run. Adding one PDF to a corpus and re-running only re-converts that PDF. Force a full re-conversion with
mykg extract-graph --from-step preprocess.
Graph & Output
- Provider-agnostic — works with Anthropic (Claude), OpenAI (GPT), Google Gemini, Ollama (local), OpenRouter, or the
claudeCLI - Five output families — JSONL for Neo4j/NetworkX/RAG, Turtle RDF for OWL toolchains, NetworkX multi-format for graph analysis, Obsidian vault for linked personal knowledge management, and an optional Neo4j LOAD CSV bundle (plain-header CSVs + paste-and-run Cypher script for Neo4j Browser /
cypher-shell) - Obsidian vault — second brain for AI coding assistants — every extracted entity becomes a wikilinked Markdown note in
output/obsidian_vault/; open it in Obsidian to navigate the graph with backlinks and Graph View, or point your AI coding assistant (Claude Code, Cursor, Copilot) at the vault folder so it can answer questions, trace relationships, and reason over your knowledge base in natural language - Interactive HTML graph — node/edge filtering, search, hover popups; opens directly in a browser
- Confidence scoring — every extracted attribute, node, and edge carries a
0.0–1.0confidence score
Quick Start
Requires Python 3.11+ (developed on macOS; automated CI runs the test suite on Ubuntu and Windows), and one of: an Anthropic/OpenAI/Gemini/OpenRouter API key, Ollama running locally, or the claude CLI.
Install from PyPI
Install mykg, then run the interactive setup wizard — it asks for your provider, model, and API key and writes mykg_config.yaml and .env.mykg in one step.
pip install mykg
mykg init
Then extract a knowledge graph from your notes:
mykg extract-graph my_notes/
Open mykg_sessions/<timestamp>/output/knowledge_graph.html in your browser to explore the result.
Install from source
Install uv, clone the repo, sync dependencies, run the setup wizard, then extract.
git clone https://github.com/SenolIsci/mykg && cd mykg
uv sync && uv run mykg init --force
Then extract a knowledge graph from your notes:
uv run mykg extract-graph my_notes/
For Ollama (local inference, no API key needed), pull a model and select the ollama-local profile when mykg init prompts you.
ollama pull llama3.3
mykg init
mykg extract-graph my_notes/
Command line
All Commands
mykg extract-graph my_notes/ # any directory: .md, .txt, .pdf, .docx, .html, images
It uses a two-pass LLM pipeline: Pass 1 induces a global RDFS/OWL schema from your document corpus; Pass 2 extracts typed entity and relationship instances per file against that schema. Non-Markdown inputs (.txt .pdf .docx .doc .pptx .xlsx .png .jpg .jpeg .html .htm) are converted to Markdown automatically before extraction. The result is exported to multiple formats: JSONL for property-graph consumers such as Neo4j, Turtle RDF for OWL toolchains, seven NetworkX formats for graph analysis and visualization, an Obsidian vault — a second brain of wikilinked Markdown notes your AI coding assistant (Claude Code, Cursor, Copilot) can read and reason over directly — and optionally a Neo4j LOAD CSV bundle with a paste-and-run Cypher script for one-step import into Neo4j Browser or cypher-shell.
Articles & Tutorials
Walkthroughs and case studies on Medium:
- How I Turned My Website Into a Knowledge Graph with myKG
- How to Build a Second Brain You Can Actually Trust
- Build an LLM Wiki for Your AI Agents
- From Documents to a Living Knowledge Graph: Introducing myKG
Configuration
All configuration lives in a single mykg_config.yaml file discovered automatically from the working directory (or any parent). There are no hardcoded defaults in the code — the YAML is the sole source of truth.
mykg init # interactive: choose provider, model, paste API key
# writes mykg_config.yaml and .env.mykg in one step
mykg init --force # overwrite an existing config
mykg init --profile openrouter-free --model google/llama-4-maverick --api-key sk-or-... # non-interactive
The wizard walks you through three prompts:
- Profile — choose your LLM provider (OpenRouter, Anthropic, OpenAI, Gemini, Ollama, Claude CLI, or Agent / Claude Code skill)
- Model — accept the default or type any model slug for that provider (skipped in agent mode — the host Claude Code session is the LLM)
- API key — paste your key (skipped for Ollama, Claude CLI, and agent mode)
LLM Providers
Switch provider by setting profile: at the top of mykg_config.yaml.
API Keys
myKG reads API keys from environment variables. Set them by exporting directly or by creating a .env.mykg file in your project directory (loaded automatically on startup).
Option A — export in your shell:
export ANTHROPIC_API_KEY=sk-ant-...
Option B — create a .env.mykg file:
# .env.mykg
ANTHROPIC_API_KEY=sk-ant-...
For source installs you can also copy sample.env.mykg to .env.mykg as a starting template.
Extract Pipeline
Reads a directory of mixed format files and produces a typed knowledge graph in three output formats. The pipeline runs 12 sequential steps; all intermediate state is persisted so any step can be re-entered without repeating upstream work.
Running
mykg extract-graph <input_dir> [OPTIONS]
# source installs: uv run mykg extract-graph <input_dir> [OPTIONS]
<input_dir> is any directory containing your source files. Subdirectories are included recursively. Only files matching the configured extensions are copied into the session:
.md— always included (the pipeline's native format)- All extensions listed under
preprocess.extensionsinmykg_config.yaml(.pdf,.docx,.doc,.pptx,.xlsx,.png,.jpg,.jpeg,.html,.htm,.txtby default)
Everything else (.py, .json, .yaml, lock files, etc.) is ignored. Hidden directories (.venv, .git, etc.) and the sessions folder are also excluded automatically, so you can safely point extract-graph at the project root or any parent directory.
Options
Examples
# New run — auto-creates a timestamped session
mykg extract-graph my_notes/
# Resume a session with 4 parallel Pass 2 workers
mykg extract-graph my_notes/ --session 2026-05-17T18-31-07 --workers 4
# Run once against a different profile/model without editing mykg_config.yaml
mykg extract-graph my_notes/ --profile openrouter-free --model google/llama-4-maverick
# Pause for schema review after Pass 1
mykg extract-graph my_notes/ --review
# → edit mykg_sessions/<name>/intermediate/schema.json
mykg approve-schema --session 2026-05-17T18-31-07
mykg extract-graph my_notes/ --session 2026-05-17T18-31-07 --review
# Re-run from assembly onward (reuses existing extractions)
mykg extract-graph my_notes/ --session 2026-05-17T18-31-07 --from-step assemble
# Lock a base ontology so the LLM won't rename its classes
mykg extract-graph my_notes/ --base-schema ontology/core.ttl
# Induce and inspect the schema first, extract in a second invocation
mykg extract-graph my_notes/ --pass1-schema-induction-only
# → review/edit mykg_sessions/<name>/intermediate/schema.json
mykg extract-graph my_notes/ --session <name> --pass2-kg-extraction-only
# Mix models across phases: a strong model for schema induction,
# a smaller/cheaper one for the laborious Pass 2 extraction
mykg extract-graph my_notes/ --profile openrouter-free --model google/gemma-4-26b-a4b-it:free --pass1-schema-induction-only
Per-run --profile / --model for phase-specific model selection. Because
--profile and --model override the run without touching mykg_config.yaml, you
can pick a different model for each phase of a two-phase extraction. Schema induction
(Pass 1) benefits from a strong reasoning model, while the laborious per-file Pass 2
extraction can run on a smaller, cheaper model — just change --profile/--model
between the --pass1-schema-induction-only and --pass2-kg-extraction-only runs.
Sessions
Every run automatically creates an isolated session folder:
mykg_sessions/
2026-05-17T18-31-07/
input/ ← archived copy of all input Markdown files
intermediate/ ← all intermediate pipeline state
output/ ← final outputs (JSONL, TTL, HTML, NetworkX)
run.log ← log file
walkthrough.md ← post-run report
Sessions are the primary unit of resumability. Pass --session <name> to resume from the last completed step. Pass --from-step <step> to force-restart from a specific point.
The sessions root is configurable via pipeline.paths.sessions_dir (default: mykg_sessions/ in the current directory).
Pipeline Steps
The pipeline runs 12 steps in sequence. All intermediate state is written to disk so any step can be re-entered without repeating upstream work.
Pass 1 internally runs four sequential stages: parallel batch induction → algorithmic merge → harmonization LLM call → quality review LLM call. pass1.max_schema_proposals (default 50) caps the number of batches dispatched to the LLM; set to -1 to dispatch all batches on large corpora.
Outputs
Property Graph (JSONL)
nodes.jsonl — one JSON line per entity:
{
"id": "person-alice",
"type": "Person",
"confidence": 0.94,
"source_files": ["team.md"],
"attributes": {
"name": {"value": "Alice", "confidence": 1.0},
"email": {"value": "[email protected]", "confidence": 0.88}
},
"aliases": ["Alice Smith", "A. Smith"]
}
edges.jsonl — one JSON line per relationship:
{
"id": "works_at-abc123",
"type": "works_at",
"from": "person-alice",
"to": "org-acme-corp",
"confidence": 0.96,
"method": "llm_extraction",
"attributes": {
"role": {"value": "Engineer", "confidence": 0.91},
"start_date": {"value": null, "confidence": 0.0}
}
}
Missing attributes are never dropped — they are represented as {"value": null, "confidence": 0.0}.
The method field distinguishes edges extracted by Pass 2 (llm_extraction) from edges inferred by the orphan pass (orphan_inferred).
RDF / OWL (Turtle)
knowledge_graph.ttl — pure RDFS/OWL triples, no edge metadata:
@prefix ex: <http://mykg.local/schema/> .
@prefix : <http://mykg.local/data/> .
ex:Person a rdfs:Class .
ex:works_at rdfs:domain ex:Person ; rdfs:range ex:Organization .
:person-alice a ex:Person ; rdfs:label "Alice" .
:person-alice ex:works_at :org-acme-corp .
Load in Protégé, query with SPARQL (Fuseki, GraphDB), or reason with HermiT/Pellet.
Interactive HTML
knowledge_graph.html — self-contained D3.js force-directed graph. Open in any browser, no server required. Supports:
- Filter nodes and edges by type
- Filter by confidence threshold
- Search by name
- Hover popups with full attribute values
- Resizable sidebar
Not written when the graph exceeds export.html_max_nodes nodes (default 5000; -1 disables the ceiling) — rendering a graph that large would overwhelm vis.js in the browser. Every other output format is still written in full. The key is read from the active profile's pipeline.export: block but is not present in the shipped config; add it there to override.
NetworkX Formats (networkx_output/)
Node/edge attributes are exported as attr_<name>_value / attr_<name>_confidence scalar pairs for GML compatibility.
Obsidian Vault (obsidian_vault/)
One .md note per extracted entity, grouped into subdirectories by concept type. Each note has YAML frontmatter (id, type, confidence, sources), an attributes section, outgoing and incoming wikilink relationship sections, and a source files list. An index.md at the vault root summarizes node counts per type with links to every entity.
Open output/obsidian_vault/ as a vault in Obsidian to get Graph View, backlink navigation, and full-text search across the extracted entities.
Neo4j LOAD CSV Bundle (neo4j_csv/)
Optional, off by default. Enable with --neo4j-csv on the command line, or set pipeline.export.neo4j_csv_enabled: true in mykg_config.yaml.
When enabled, step_validate_graph writes a self-contained Neo4j import bundle next to the other outputs:
output/neo4j_csv/
nodes_<Label>.csv ← one per concept type (Person, Organization, …)
relationships_<TYPE>.csv ← one per property (WORKS_AT, KNOWS, …)
import_browser.cypher ← paste-and-run for Neo4j Browser
import_shell.cypher ← for `cypher-shell -f`
README.md ← bundle-local quick-reference
Plain CSV headers (id,name,name_confidence,_node_confidence,_parents,_source_files,...) — no :ID / :LABEL decorations, so the same files work for both Neo4j and any other CSV-aware tool.
Two ways to import the bundle:
# Flow A — Neo4j Browser
# 1. Copy *.csv into your DBMS's import/ directory
# 2. Paste the contents of import_browser.cypher and press play
# Flow B — cypher-shell (set dbms.security.allow_csv_import_from_file_urls=true first)
cypher-shell -u neo4j -p <pw> -f output/neo4j_csv/import_shell.cypher
Both scripts use idempotent MERGE against a _MykgNode uniqueness constraint, so re-running updates the graph in place. Requires Neo4j 5+. No Python driver, no plugin, no APOC — the scripts use only core Cypher.
See Neo4j Load CSV Export below for configuration details and the standalone CLI fallback.
MCP (Model Context Protocol) Server
Serve any completed session as an MCP server for LLM-powered Q&A:
# Serve the latest session (stdio transport — for Claude Desktop)
mykg mcp-serve
# Serve a specific session
mykg mcp-serve --session 2026-06-25T19-16-18
# Serve via streamable HTTP for web clients (Cherry Studio, etc.)
mykg mcp-serve --transport streamable_http --port 3100
# Serve a specific session via streamable HTTP
mykg mcp-serve --session 2026-06-25T19-16-18 --transport streamable_http --port 3100
Claude Desktop (stdio) — the client launches mykg as a subprocess, no manual server start needed. Add to claude_desktop_config.json. Replace /path/to/your/project with the absolute path to your project folder:
{
"mcpServers": {
"mykg": {
"command": "uv",
"args": [
"--directory",
"/path/to/your/project",
"run",
"mykg",
"mcp-serve"
]
}
}
}
To serve a specific session:
{
"mcpServers": {
"mykg": {
"command": "uv",
"args": [
"--directory",
"/path/to/your/project",
"run",
"mykg",
"mcp-serve",
"--session",
"2026-06-25T19-16-18"
]
}
}
}
Claude Code (stdio) — add a .mcp.json file to your project root so Claude Code launches the MCP server automatically at session start:
{
"mcpServers": {
"mykg": {
"command": "uv",
"args": [
"--directory",
"/path/to/your/project",
"run",
"mykg",
"mcp-serve"
]
}
}
}
To serve a specific session:
{
"mcpServers": {
"mykg": {
"command": "uv",
"args": [
"--directory",
"/path/to/your/project",
"run",
"mykg",
"mcp-serve",
"--session",
"2026-06-25T19-16-18"
]
}
}
}
Restart required. Claude Code reads
.mcp.jsonat session start. After creating or editing this file, restart your Claude Code session for the server to connect. You will be prompted to approve themykgMCP server on first use.
After a pipeline run (extract-graph, --append, etc.), the in-memory graph becomes stale. Call the mykg_reload tool to refresh it without restarting — or restart the Claude Code session to reload from scratch.
If you also have the /mykg skill installed, the skill will prefer MCP tools for read-only queries when the server is connected, and fall back to direct file reads when it is not.
Streamable HTTP (Cherry Studio, MCP Inspector, or any HTTP-based MCP client) — start the server first with mykg mcp-serve --transport streamable_http --port 3100, then connect your client:
{
"mcpServers": {
"mykg": {
"type": "streamableHttp",
"url": "http://localhost:3100/mcp"
}
}
}
The MCP server exposes 16 tools: mykg_search_nodes, mykg_get_node, mykg_get_neighbors, mykg_find_path, mykg_get_schema, mykg_list_node_types, mykg_query_subgraph, mykg_get_stats, mykg_query_graph (BFS/DFS traversal), mykg_hub_nodes, mykg_orphan_nodes, mykg_schema_drift (read-only schema drift report), mykg_get_source (source chunks behind a node), mykg_read_note (Obsidian vault LLM wiki notes), mykg_list_sessions, and mykg_reload (refresh the in-memory graph after pipeline runs).
Configuration — default transport, host, and port are set per profile in mykg_config.yaml:
mcp:
host: localhost # bind address for streamable HTTP (ignored for stdio)
port: 3100 # port for streamable HTTP (ignored for stdio)
transport: stdio # default transport: stdio | streamable_http
To default to streamable HTTP, change transport: streamable_http in your active profile. CLI flags (--transport, --host, --port) override the config.
Terminal query (mykg query)
mykg query "<question>" [--session NAME] [--mode bfs|dfs] [--depth N] [--token-budget N]
Query the knowledge graph directly from the shell — no MCP server required. mykg query finds up to 3 seed nodes matching your question by name, alias, or attribute value, traverses outward through the graph to a bounded depth, and prints a text context window of the visited nodes and the relationships between them. It is the terminal-callable equivalent of the MCP mykg_query_graph tool, and it is read-only: it operates on the latest completed session unless --session is passed.
question(positional, required) — the free-text query.--session NAME— session undermykg_sessions/; defaults to the latest completed session (newest dir withoutput/nodes.jsonl).--mode bfs|dfs— traversal strategy (defaultbfs).--depth N— traversal depth limit (default2).--token-budget N— approximate token budget bounding the returned context (default2000).
mykg query "who is Alice" --depth 2
# Knowledge Graph Context: who is Alice
Seeds: <ids> | Mode: bfs | Depth: 2
Nodes visited: N | Edges found: M
## Nodes
- [Type] Name (node-id) conf=0.95 (attr=val, ...)
## Relationships
- from-id --[edge_type]--> to-id (conf=0.90)
If nothing matches, it prints No nodes found matching '<question>'. Try mykg_search_nodes for more flexible search. and exits 0.
Using mykg with Claude Code
myKG ships with two complementary integrations for running extractions from inside Claude Code:
- Agent mode (
agent-claude-codeprofile + bundled skill) — the pipeline writes LLM tasks to a session-local inbox folder and a Claude Code skill dispatches subagents to answer them. Parallel by default. claude-cliprofile — the pipeline shells out to theclaude -pbinary for each LLM step. Serial only.
Pick the claude-cli for a drop-in claude-as-LLM experience; pick agent-claude-code when you want agent skill and parallel subagent dispatch.
Agent mode (Claude Code skill)
Agent mode is a different way to run myKG inside Claude Code: instead of claude -p subprocesses, the pipeline writes LLM tasks to a session-local inbox folder and a Claude Code skill dispatches subagents to answer them. Pick agent mode over claude-cli when you want parallel subagent dispatch from inside an active Claude Code session.
Why pick agent mode
- No API key needed. Uses your existing Claude Pro/Max plan via the skill subagents — same as
claude-cli, but without invoking theclaude -pbinary. - Inspectable LLM I/O. Every prompt lands as
intermediate/agent_inbox/<id>.task.jsonand every answer asintermediate/agent_outbox/<id>.answer.json. Replay or edit any step by hand. - Parallel by default. The skill dispatches up to
pass2.max_workerssubagents per wave in a single message — not serial likeclaude-cli. Pass-2 chunks complete in parallel waves.
Install and configure
pip install mykg # or: uv tool install mykg
mykg init --profile agent-claude-code
# ...then restart Claude Code so the skill loader picks up the new entry.
mykg init --profile agent-claude-code writes mykg_config.yaml, copies the bundled skill into ~/.claude/skills/mykg (honoring $CLAUDE_CONFIG_DIR if set), and adds a managed ... block to the project's CLAUDE.md. A .mykg_skill_version stamp file is written next to the skill so future runs can detect drift; the CLAUDE.md block tells Claude Code where the wiki lives, how to find the most-recent session, and how to extend the graph with new documents (no separate setup required).
Upgrade after pip install -U mykg:
mykg init --reinstall-skill --reinstall-claude-md
This atomically refreshes the bundled skill (copy to .tmp → os.replace) without touching your mykg_config.yaml, and replaces the content between the CLAUDE.md markers with the version shipped in the current package — any user content outside the markers is preserved. Either flag can be used alone (--reinstall-skill only / --reinstall-claude-md only). The copy-based skill install is deliberately not a symlink — symlinks fail on Windows without Developer Mode, dangle if mykg is uninstalled, and don't sync through OneDrive. The cost (live edits don't auto-propagate) only matters for mykg developers, who pass --reinstall-skill between edits.
The agent: block in the generated mykg_config.yaml configures the inbox/outbox paths and poll interval:
profile: agent-claude-code
profiles:
agent-claude-code:
provider: agent
agent:
inbox_dir: agent_inbox # relative to <session>/intermediate/
outbox_dir: agent_outbox
poll_interval_seconds: 2
pipeline:
pass2:
max_workers: 8 # how many subagents the skill dispatches per wave
Invoke from inside Claude Code
The skill exposes one slash command — /mykg — that accepts free-form intent. You describe what you want; the skill figures out which mykg CLI command to run, reads the live --help to validate flags, confirms expensive actions, and (for extract-graph) drains the LLM inbox in parallel waves.
Examples:
Any flag mykg accepts on the CLI works here too — the skill reads --help rather than maintaining its own list, so --from-step orphan_connect, --workers 8, --obsidian-vault, etc. all flow through.
mykg init and mykg merge-graphs are intentionally not wrapped: init is interactive (run from a shell once per machine), and merge-graphs has additional design questions and will be added in a follow-up.
Full design and contract: docs/agent-mode.md. Skill source: src/mykg/data/skills/mykg/SKILL.md.
claude-cli profile
myKG ships with a claude-cli profile that runs extractions through the locally-installed claude CLI.
Setup
Install the claude CLI, then install mykg and run the setup wizard — select [5] Claude CLI when prompted.
npm install -g @anthropic-ai/claude-code
pip install mykg && mykg init
mykg extract-graph my_notes/
How it works
The claude-cli provider calls claude -p as a subprocess for every LLM step (Pass 1 schema induction, Pass 2 extraction, orphan connection, name normalization). All pipeline features — session isolation, resumability, orphan recovery, cross-session merge — work identically to API-based providers.
Key constraints of the claude-cli profile:
max_workersmust be1— theclaudeCLI is serial by design; parallel workers will queue- The
effortandmodelfields inmykg_config.yamlmap directly to--effortand--modelflags passed toclaude -p
Using myKG from inside Claude Code Session
You can run myKG extractions as a tool call from within a Claude Code session. This is useful for building knowledge graphs from notes or documentation while you work:
# From any Claude Code session terminal:
mykg extract-graph ./docs/ --session my-docs-kg
# Then reference the output in your session:
# mykg_sessions/my-docs-kg/output/nodes.jsonl
# mykg_sessions/my-docs-kg/output/knowledge_graph.ttl
Claude Code can then read nodes.jsonl or edges.jsonl as well as the Obsidian vault directly to answer questions about the extracted graph, or load knowledge_graph.ttl into a SPARQL tool for structured queries.
Advanced Options
Re-running from a Specific Step
Use --from-step to delete a step's outputs and all downstream outputs, then re-run from that point.
SESSION=2026-05-17T18-31-07
# Re-run from Pass 2 (reuse the existing schema)
mykg extract-graph my_notes/ --session $SESSION --from-step pass2
# Re-run only assembly + export (reuse raw extractions)
mykg extract-graph my_notes/ --session $SESSION --from-step assemble
# Re-run both orphan stages
mykg extract-graph my_notes/ --session $SESSION --from-step orphan_score
# Orphan LLM pass only — full clean sweep
mykg extract-graph my_notes/ --session $SESSION --from-step orphan_connect_fullsweep
# Orphan LLM pass only — additive (preserves prior confirmed edges)
mykg extract-graph my_notes/ --session $SESSION --from-step orphan_connect_incremental
# Re-run only merge/harmonize/quality-review (reuse existing Pass 1 batch proposals)
mykg extract-graph my_notes/ --session $SESSION --from-step merge_proposals
Five re-entry patterns:
Orphan-Connection Pass
After assembly, nodes with zero edges are "orphans" — present in the graph but unreachable by traversal. The orphan pass reconnects them in two stages:
Stage 1 — orphan_score (no LLM): Uses chunk_node_index.json to find nodes that co-occur in the same source chunk as each orphan. Candidates are scored by co-occurrence frequency and filtered by schema type compatibility. Written to orphan_candidates.json.
Stage 2 — orphan_connect (LLM): One LLM call per source chunk. The prompt includes the full chunk text, all orphan IDs from that chunk, co-occurring connected nodes, and all schema properties. Confirmed edges carry "method": "orphan_inferred" and are merged directly into edge_metadata.json.
Unconnectable orphans (no resolvable source chunk) are logged as orphan_unconnectable advisory events in orphan_log.json.
Configure via pipeline.orphan_pass.* in mykg_config.yaml. Disable entirely with pipeline.orphan_pass.enabled: false.
Human Review Gate (--review)
Pause after Pass 1 to inspect and edit the induced schema before Pass 2 runs:
mykg extract-graph my_notes/ --review
# → pipeline halts; edit mykg_sessions/<name>/intermediate/schema.json
mykg approve-schema --session <name>
mykg extract-graph my_notes/ --session <name> --review # resumes from Pass 2
Locked Base Schema (--base-schema)
Lock certain classes and properties so the LLM cannot rename, remove, or restructure them:
mykg extract-graph my_notes/ --base-schema ontology/base.ttl
Both RDFS and OWL ontologies are accepted — owl:Class, owl:ObjectProperty, and owl:DatatypeProperty are recognized alongside their RDFS equivalents, so you can use an OWL ontology exported from Protégé directly without conversion. Locked entries can still receive additional attributes proposed by the LLM. Near-duplicate LLM proposals are collapsed into the locked entry with a warning.
Frozen Schema (--freeze-schema)
Use the base schema verbatim — skip Pass 1 LLM induction entirely. The LLM extracts instances against exactly the classes and properties you provide, with no additions:
mykg extract-graph my_notes/ --base-schema ontology/base.ttl --freeze-schema
This is useful when you have a complete ontology and want strict extraction — no LLM-invented types, no surprise properties. Only the concepts and relationships in your TTL will appear in the output. Requires --base-schema. Mutually exclusive with --append and --append-with-grow-schema.
Comparison with --base-schema alone:
SKOS Thesaurus (--thesaurus)
Resolve near-duplicate concept names during schema merge using a SKOS vocabulary:
mykg extract-graph my_notes/ --thesaurus ontology/terms.skos.ttl
skos:exactMatch→ silent collapseskos:closeMatch→ collapse with warning inmerge_log.jsonskos:broader→ breaks ties when two proposals give a concept different parents; the declared broader term wins regardless of which proposal came firstskos:narrower→ advisory hints only
Name matching alone can't tell that two proposals disagree about the structure underneath a shared name, so those disagreements are recorded rather than silently dropped. merge_log.json (and merge_manifest.json for merge-graphs) gains a parent_conflict, parent_conflict_resolved, domain_range_conflict, or attribute_synonym entry whenever two proposals collapse onto one name but differ on a parent, a domain/range, or an attribute spelling. The first-seen value still wins — the log tells you what was discarded.
Website / Repo Fetching (mykg fetch-web)
Crawl a website, or shallow-clone a GitHub repo, into a folder that's a ready-made extract-graph input:
# Crawl a website (same-domain, robots.txt-respecting)
mykg fetch-web https://example.com
mykg extract-graph ./mykg_web_fetch/example.com/
# Shallow-clone a GitHub repo (git clone --depth 1, no Crawlee/venv)
mykg fetch-web https://github.com/SenolIsci/mykg
mykg extract-graph ./mykg_web_fetch/github.com_SenolIsci_mykg/input/
# Fetch multiple seeds (one URL per line; mix of sites and GitHub repos)
mykg fetch-web --url-list urls.txt --output ./mykg_web_fetch/batch/
- GitHub URLs (
https://github.com/<owner>/<repo>) are detected automatically and routed togit clone— no Crawlee, no venv. - Everything else is crawled with Crawlee inside an ephemeral
uvvenv (same pattern as the MinerU venv used bypreprocess), respectingrobots.txt,--max-pages,--max-depth, and a configurable request delay/concurrency. - Resumable —
fetch_manifest.jsonrecords a SHA-256 per page; re-running skips unchanged pages.--forcere-fetches everything. - Output dir defaults to
./<fetch.output_dir>/<seed-domain>/(configurable viafetch.output_dirinmykg_config.yaml, defaultmykg_web_fetch);--outputoverrides it and is required with--url-list. - Downloaded assets (PDFs, images, etc.) are filtered through the same
preprocess.extensionsallowlist used byextract-graph, so anything Crawlee saves is somethingpreprocessalready knows how to convert.
All knobs live under fetch: in mykg_config.yaml — see docs/architecture.md for the full crawl/clone sequence diagrams. Run mykg fetch-web --help for the complete flag list.
From Claude Code, the /mykg skill handles fetch requests in plain English — no flags to remember:
/mykg fetch https://example.com and extract
/mykg download the repo: https://github.com/SenolIsci/mykg
/mykg fetch these urls: <url1> <url2> ... and extract
All work: the skill picks the right fetch-web invocation (single page, GitHub clone, or --url-list batch with an auto-generated temp file for inline URLs), runs it, and — for the "and extract" intents — chains straight into extract-graph on the fetched output (one fresh session per seed for multi-seed fetches), confirming with you before the LLM-bearing extraction step.
Standalone Document Conversion (mykg parse-docs)
extract-graph already converts non-Markdown inputs (PDF, DOCX, images, …) to Markdown automatically via the preprocess step — on both the initial run and on --append. You only need parse-docs when you want to convert documents on their own, without running a pipeline or creating a session: inspecting MinerU output, building a Markdown corpus to commit, or feeding another tool.
# Convert a single file
mykg parse-docs --input report.pdf --output ./md/
# Convert every non-.md file under a directory (recursive; structure preserved)
mykg parse-docs --input raw_docs/ --output ./md/
# Convert only specific files (relative to --input; repeatable)
mykg parse-docs --input raw_docs/ --output ./md/ --file a.pdf --file sub/b.docx
- MinerU in an ephemeral venv — conversion runs MinerU inside a throwaway
uv-managed venv that is built per invocation and deleted on exit; nothing is installed into mykg's own interpreter. The multi-GB install is paid once per call and reused across every file in that call. - Extension allowlist — candidate files are filtered through
preprocess.extensionsfrommykg_config.yaml(the same allowlistextract-graphuses)..html/.htmare always hard-skipped (MinerU cannot convert HTML — useextract-graph, which routes HTML throughmarkdownify). Pass--no-filterto send every non-.mdfile to MinerU regardless of suffix. - Large corpora — use
--file-list <path>(one rel-path per line) instead of repeated--fileflags to avoid the OS argv-size limit.--fileand--file-listare mutually exclusive. - No session —
parse-docsis a pure file-to-file utility: it does not create a session or touchmykg_sessions/. Per-file failures are logged and the run continues, exiting non-zero at the end if any file failed.
Run mykg parse-docs --help for the complete flag list. From Claude Code, /mykg convert pdfs in ./inbox to ./md maps to the right invocation automatically.
Append Mode
Re-run the pipeline on new or modified files without re-running Pass 1:
mykg extract-graph my_notes/ --session <name> --append
The input directory may contain PDF, DOCX, HTML, TXT, and image files alongside .md — newly-added non-Markdown files are converted automatically during the append run (the same incremental preprocess step the initial run uses, subject to preprocess.enabled). No separate mykg parse-docs step is needed. Only new or changed source files are converted; unchanged ones are skipped by content hash.
Reconciling Edits and Deletions (--sync)
--append adds; --sync reconciles. Plain --append extracts new files
only — it detects modified and deleted files and warns about them, but leaves
the graph untouched. Add --sync to act on them:
mykg extract-graph my_notes/ --session <name> --append --sync
All four combinations, and what each does with the three kinds of change:
The three flags are orthogonal axes: --append decides which files are
processed, --sync decides whether existing entries are reconciled, and
-with-grow-schema decides whether the schema may grow.
--sync requires --append. --append-with-grow-schema already implies
--append, so the last row needs only the two flags shown. --update is
shorthand for --append --sync:
mykg extract-graph my_notes/ --session <name> --update
Reach for the fourth row when the new vocabulary lives in a document you
edited rather than one you added — without --sync, the locked Pass 1 only
sees new files, so a concept introduced by an edit never reaches the LLM.
⚠️ Behaviour change: before the release 0.4.3, plain
--appendre-extracted modified files. It no longer does — add--sync. If your edits stop showing up in the graph, that is why; the run will have logged a warning naming the files.
Deleting a file removes the nodes and edges that came only from it; nodes shared
with a surviving file are recomputed, so their confidence and attribute
values no longer reflect the removed document. Running --sync with nothing to
reconcile costs nothing — it degrades to exactly plain --append, so it is safe
to leave in a script.
Exact deletion needs pass2.prep_mode: per_file or batch_chunks with
pass2.batch_per_file: true. On the shipped default one LLM call covers several
files, so a co-batched sibling may retain some of the deleted file's nodes.
Multiple source folders. Each folder you append to a session is registered
with its own subtree of session/input/, so two fetch-web crawls that both
produce index.md no longer overwrite each other. --sync is scoped to the
folder you name — reconciling one folder can never delete another's files, which
is why no confirmation prompt or size cap is needed.
Incremental Schema Growth (--append-with-grow-schema)
Plain --append freezes the schema — Pass 1 is skipped, so new entity types and relationships are never induced. Use --append-with-grow-schema when you add documents that introduce concepts the current schema doesn't cover:
mykg extract-graph my_notes/ --session <name> --append-with-grow-schema
This runs a locked Pass 1 over only the changed files: the LLM may add new concepts and properties but cannot rename, remove, or restructure existing ones. When the schema grows, a surgical back-fill re-extracts the old chunks most likely to contain instances of the new types (configurable via append.grow_schema_backfill_top_k_chunks_per_type, default 10; set 0 to disable). When the new documents don't introduce new types, the run collapses to a plain --append at no extra cost — and with nothing to reconcile at all, the locked Pass 1 is skipped entirely (zero LLM calls). Note that "changed files" means new files unless you also pass --sync.
The flag implies --append (no need to pass both) and is mutually exclusive with --from-step and --base-schema (the session's existing schema.ttl is auto-loaded as the locked base).
Mixed-format inputs: Both
--appendand--append-with-grow-schemaautomatically preprocess newly-added non-Markdown files (PDF, DOCX, HTML, TXT, images). Just drop the new files into the input directory and append — thepreprocessstep runs incrementally, converting only the new or changed sources (unchanged files are skipped by content hash) and feeding the converted Markdown straight into extraction. No separatemykg parse-docsstep is required.
Merging Sessions
Combine two independently-produced sessions into a unified knowledge graph:
mykg merge-graphs <session-A> <session-B> [OPTIONS]
# Example
mykg merge-graphs 2026-05-01T10-00-00 2026-05-15T14-30-00
# Resume a merge (last incomplete step auto-detected)
mykg merge-graphs A B --output-session <merged-name>
Options:
What happens:
- Both schemas are merged via the same three-stage chain as Pass 1 (algorithmic union → LLM harmonization → LLM quality review)
- All file-keyed structures are namespaced (
session_a/<filename>,session_b/<filename>) before merging - Nodes are deduplicated across sessions: same type + canonical name → single node, regardless of source session
- Re-extraction strategy (
none/surgical/full) handles properties absent from one session's schema source_map.jsonrecords full file provenance;merge_manifest.jsonrecords schema deltas and strategy usedwalkthrough.mdincludes a Merge Provenance section with before/after counts and node/edge breakdowns
Configure the re-extraction strategy:
merge_graphs:
reextraction_strategy: surgical # none | surgical | full
Obsidian Vault Export
Every run writes a linked Markdown vault to output/obsidian_vault/ by default. Open that folder in Obsidian to explore the extracted knowledge graph with Graph View and backlinks.
Vault structure:
output/obsidian_vault/
index.md ← overview: node count per type, links to every entity
Person/
person-alice-smith.md ← one note per entity
person-bob-jones.md
Organization/
organization-acme-corp.md
...
Each entity note contains:
---
id: person-alice-smith
type: Person
confidence: 0.94
sources:
- team.md
---
# Alice Smith
## Attributes
- **role**: Engineer (0.91)
- **email**: [email protected] (1.0)
## Relationships
### Outgoing
- [[Acme Corp]] — works_at (0.96)
### Incoming
- [[Bob Jones]] — manages (0.88)
## Source Files
- team.md
Wikilinks ([[...]]) are Obsidian-native — clicking them in the app navigates to the linked entity note, and the Graph View shows the full relationship network automatically.
Config:
pipeline:
export:
obsidian_enabled: true # default — set false to skip vault export
obsidian_vault_dir: obsidian_vault # subfolder name inside output/
Or use --obsidian-vault on the command line for a one-off run without editing config.
Neo4j Load CSV Export
Optional bundle for one-step import into Neo4j 5+. Off by default. When enabled, every run writes the bundle to output/neo4j_csv/ alongside the other outputs.
Bundle contents (see neo4j_csv/ above for the full layout):
- One
nodes_<Label>.csvper concept type with plain headers (id,name,name_confidence,...) - One
relationships_<TYPE>.csvper property (rel-type names sanitized to upper snake_case) import_browser.cypher— paste-and-run for Neo4j Browser (relativefile:/<name>.csvURIs)import_shell.cypher— forcypher-shell -f(absolutefile:///URIs)README.md— bundle-local quick-reference with paste instructions
The scripts use:
- A uniqueness constraint on
(_MykgNode {id})— created on first run,IF NOT EXISTSthereafter MERGEfor every node and edge — idempotent, safe to re-runIN TRANSACTIONS OF 1000 ROWS— handles large bundles without OOM- Per-label domain labels (
:Person,:Organization) plus the shared:_MykgNodelabel that carries the constraint
Config:
pipeline:
export:
neo4j_csv_enabled: false # default — set true to enable
neo4j_csv_dir: neo4j_csv # subfolder name inside output/
Or use --neo4j-csv on the command line for a one-off run without editing config.
Walkthrough Report
A human-readable summary is written to mykg_sessions/<name>/walkthrough.md after every run:
# Regenerate the walkthrough for an existing session
mykg walkthrough --session 2026-05-17T18-31-07
Disable with pipeline.report.enabled: false.
Schema Drift & Pruning
The induced schema only ever grows: Pass 1 and --append-with-grow-schema add concepts and properties, and nothing removes them. After you delete documents and run --append --sync, the graph shrinks but schema.json still declares the old vocabulary. That gap is schema drift. Two commands, neither of which calls an LLM or needs an API key, detect and repair it. "Unused" here means a concept with no instances or a property with no edges (not an "orphan" node, which has no edges).
# 1. Reconcile instances with the current documents
mykg extract-graph docs/ --session s1 --append --sync
# 2. Report unused concepts/properties (writes intermediate/schema_drift.json)
mykg detect-schema-drift --session s1
# 3. Preview the removal plan (writes intermediate/schema_prune_plan.json, changes nothing else)
mykg prune-schema --session s1 --preview
# 4. Apply: choose ONE
mykg prune-schema --session s1 # A. remove every prunable entry (recomputed; ignores the saved plan file)
mykg prune-schema --session s1 --plan <plan file> # B. remove only what your edited plan lists
Each unused entry is classified. locked entries (from --base-schema) are never pruned. structural entries (zero own instances but an instantiated descendant, a surviving child, or the domain/range of a surviving property) are kept; there is no re-parenting. Only prunable entries are removed. Pruning never touches nodes or edges: only zero-instance entries are eligible.
Safer alternative: prune a copy of the session. Undoing a prune by hand (below) is fiddly, so if you are unsure, prune a copy and keep the original untouched. A session is fully self-contained, so copying its folder is enough:
cp -r mykg_sessions/s1 mykg_sessions/s1-pruned
mykg prune-schema --session s1-pruned --preview
mykg prune-schema --session s1-pruned
Compare s1 and s1-pruned; keep whichever you prefer and delete the other (rm -r mykg_sessions/s1-pruned). Undoing then needs no steps at all. Use the manual undo below only when you pruned the session in place.
Undoing a prune. flattened_schema.json is not regenerated automatically on undo:
- Copy
intermediate/schema.pre_prune.jsonoverintermediate/schema.json. mykg approve-schema --session s1(regeneratesschema.ttl).- Delete
intermediate/flattened_schema.json. mykg extract-graph <dir> --session s1 --from-step validate_graph.- Re-run
mykg detect-schema-drift --session s1.
Do not use --from-step schema_flatten for this: it deletes the Pass 2 outputs. Undo restores the schema exactly as it was before the most recent prune, so it is only appropriate before later runs have grown or changed the schema (the backup is overwritten by the next prune).
Known limitations. The writes during apply are not one transaction; they are ordered so that re-running the prune is idempotent after a crash, and mykg approve-schema repairs a stale schema.ttl. There is no cross-process lock: both commands refuse to run while a step in pipeline_state.json is running. Only one backup is kept (schema.pre_prune.json is overwritten by the next prune). validate_graph regenerates nodes.jsonl and edges.jsonl from nodes.json and edge_metadata.json.
Continuing the session after a prune
Note: locks come only from a --base-schema given on a fresh run (recorded in intermediate/base_schema_parsed.json). --append-with-grow-schema does not write that file, so a session without it has no locks: the commands print no locked schema recorded; treating all entries as unlocked.
Typical order: sync, then detect, then prune. The walkthrough shows a Schema Health section once schema_drift.json exists (marked stale if the schema or graph changed since), and the MCP server's read-only mykg_schema_drift tool computes the report live from the graph as loaded in memory. Details: docs/architecture.md.
Troubleshooting
Token Budgets
Each profile's llm: and pipeline: blocks carry a chain of token-budget values sized for that model's context window:
llm.context_window— the model's total context limitllm.max_output_tokens— the output cap reserved for each LLM responsepipeline.pass1.batch_token_targetandpipeline.pass2.concat_batch_token_target/batch_token_target— input budget per LLM call, sized to(context_window − max_output_tokens) × 0.95pipeline.chunking.window_tokens/overlap_tokens— chunk size and overlap for splitting large files, sized to roughlybatch_token_target / 4andwindow_tokens × 0.10
The shipped values are tuned per profile (e.g. claude-cli/anthropic-claude assume a 200K context window, openrouter-free/ollama-local assume 64K). If you switch to a different model — especially on ollama-local or openrouter-free — check that model's actual context window and rescale these values, otherwise window_tokens + max_output_tokens may exceed what the model can actually handle, causing truncated or failed responses.
Use the bundled context-calculator tool to recompute the chain for a new model:
# Compute a full token-budget chain from context window + output cap:
context-calculator --context 64000 --max-output 32000
# Or measure your actual corpus and suggest values for the active profile:
context-calculator --from-config --input-dir my_notes/
--from-config reads the active profile from mykg_config.yaml, measures token counts across your input files, and writes suggested values to mykg_config_candidate.yaml for review before copying them into mykg_config.yaml.
Diagnosing Truncated/Rejected LLM Calls
On JSON parse or validation errors in Pass 1/2, check run.log for a mykg.llm.retry warning just above the failure — always logged, regardless of logging.llm_log:
output truncated (finish_reason=...)— hitmax_output_tokens. Lowerpipeline.chunking.window_tokensor raisellm.max_output_tokens, then recompute withcontext-calculator(see Token Budgets).context length exceeded, request rejected: ...— input didn't fit the context window. Lowerbatch_token_target(pipeline.pass1/pipeline.pass2).
Pass 2 Prep Mode (pass2.prep_mode)
Controls how source files are packed into Pass 2 LLM calls (set per profile in mykg_config.yaml):
batch_chunks(default) — packs chunks across files into token-bounded batches, ignoring file boundaries; best throughput and extraction density.concat— merges whole small files into directory-grouped batches (one LLM call per window) for cross-file context.per_file— one file per extraction unit; cleanest provenance (every entity traces to one source file). Fully updates a modified file on--append— its old nodes are cleanly replaced on re-extraction. (batch_chunkswithbatch_per_file: truegives the same guarantee; mixed-batch modes may retain stale copies — see docs/architecture.md.)
See docs/architecture.md for the full comparison.
Hitting API Rate Limits (HTTP 429)
A 429 surfaces in the log as a retry warning like:
[WARNING] mykg.llm.retry — OpenAI 429 rate-limit (attempt 1/5) — retrying in 2.0s
429 is a "Too Many Requests" error. If you see repeated 429 errors during pass1, pass2, or the orphan-connection pass, your account's requests-per-minute limit is lower than the number of concurrent calls mykg is making. Each profile sets these independently under pipeline::
pass1.max_workers— concurrent schema-induction batch callspass2.max_workers— concurrent per-file extraction callsorphan_pass.max_workers— concurrent orphan-connection calls
Lower these (e.g. from 8 down to 2–4) in the active profile to reduce concurrent requests. This is especially likely on openrouter-free (free-tier models have very low per-minute caps), on gemini with a free-tier key (5 requests/minute/model — drop pass1/pass2/orphan_pass to 1–2), and on lower-tier anthropic-claude/openai accounts. llm.retry_429_max / llm.retry_429_base_delay control automatic backoff on a 429, but a persistent 429 is a signal to reduce max_workers, not just retry harder. claude-cli is unaffected — it is serial by design (max_workers: 1); it doesn't hit API rate limits since there's no API call. agent-claude-code is not API rate-limited either (no API key involved), but it is not serial — its default profile sets pass1/pass2/orphan_pass max_workers > 1 (configurable, like any other profile), since the skill dispatches multiple subagents per wave.
Also check your quota/credits. Some providers return 429 when your account has exhausted its token quota or spending balance, not only for request cadence. If lowering max_workers doesn't help and the 429s persist from the very first call, check that your account still has available tokens/credits (e.g. the OpenAI/Anthropic billing dashboard, or your OpenRouter balance). No max_workers value will clear a 429 caused by a depleted balance — top up or switch to a profile with quota (e.g. ollama-local for local inference, or claude-cli which bills via your Claude Pro/Max plan instead of the API).
Non-Reproducible Extraction (Different Graph Each Run)
If the same corpus produces a different schema or a different set of nodes and edges on each run, sampling temperature is the usual cause.
myKG sends temperature: 0.0 on every LLM call so extraction is repeatable —
it is a structured-output task, not a creative one, and provider defaults
(typically ~1.0) work against that. Two cases fall outside it, and in both the
provider's own sampling applies:
- Models that reject an explicit temperature — OpenAI's o-series and gpt-5 reasoning families — have it omitted automatically and use their own fixed sampling.
claude-cliandagenthave no temperature control at all:claude -pexposes no such flag, and the host Claude Code session servicing the agent inbox has no sampling dial. These two profiles accept the setting and ignore it, so extraction on them is not reproducible in the sense above.
If you need reproducibility, switch to a profile whose provider exposes
temperature (anthropic-claude, openai on a non-reasoning model, gemini,
ollama-local, openrouter-free).
Changed in 0.4.5 — earlier versions sent no temperature at all and inherited each provider's default. If you were relying on that, extraction output will shift. To restore the old behaviour, set an explicit empty
temperature:in the active profile'sllm:block; anullvalue means "send nothing".
Roadmap
- in preparation.
Development
Installation
git clone https://github.com/SenolIsci/mykg && cd mykg
uv sync
Testing
# All non-live tests (fast, no API key needed)
uv run pytest -m "not live" -v
# All tests including live API integration tests
# Requires OPENROUTER_API_KEY in environment or .env.mykg (see sample.env.mykg)
uv run pytest -m live -v
# Single file
uv run pytest tests/test_assembler.py -v
# With coverage (HTML report at htmlcov/index.html)
uv run pytest -m "not live"
open htmlcov/index.html
Healthiness check
A dry-run smoke test that answers one question: is everything alive and right? It
probes every configured LLM endpoint, then exercises the major capabilities —
fetch-web, parse-docs (MinerU), extract-graph, query, walkthrough — end to end
through the real CLI, and prints a summary table.
# Just the endpoints — seconds
uv run pytest tests/test_healthiness.py -k endpoints -v --no-cov
# Everything except MinerU — the usual quick check
uv run pytest tests/test_healthiness.py -m "live and not mineru" -v --no-cov
# Full check including PDF/XLSX conversion via MinerU
uv run pytest tests/test_healthiness.py -m live -v --no-cov
# Point the capability stages at a specific provider
MYKG_E2E_PROFILE=gemini uv run pytest tests/test_healthiness.py -m live -v --no-cov
Each endpoint is checked twice: that it is reachable, and that it can return the JSON
contract Pass 2 depends on — a provider can answer a ping and still fail every real
extraction. Providers with no key configured are skipped, never failed, and failures are
classified into an actionable cause (quota or balance exhausted, api key rejected,
model not available, …) rather than a bare traceback.
ENDPOINTS reach contract
openrouter-free ok ok 2 node(s), 1 edge(s)
anthropic-claude ok ok 2 node(s), 1 edge(s)
gemini FAIL — quota or balance exhausted → top up, or switch profile
ollama-local ok ok 2 node(s), 1 edge(s)
Artifacts land in a pytest temp directory — the crawled HTML, the MinerU-converted
Markdown, and a full session under mykg_sessions/ — so a health check never touches
your real sessions. The gitignored _healthiness_run symlink in the repo root points at
the most recent run.
Two things to expect. A failing endpoint is the slow one: the cause is only known
after the adapter exhausts its 429 backoff ladder, which is what lets the report tell
"rate limited, recovered" apart from "quota exhausted" — lower llm.retry_429_max for a
faster verdict. And on free tiers (Gemini allows 5 requests/minute/model, OpenRouter's
free models are similarly capped) the extraction stages log 429 backoff warnings; the
adapters retry and recover, which is documented behaviour, not a failure.
The whole file carries the live marker, so it is excluded from CI and from
pytest -m "not live". Design notes and measured results are in
docs/healthiness-check.md.
Linting and Formatting
uv run ruff check src/ tests/ # lint
uv run ruff check --fix src/ tests/ # auto-fix
uv run ruff format src/ tests/ # format
Token Budget Calculator
When switching to a model with a different context window:
context-calculator --context 128000 --max-output 16384
Outputs a ready-to-paste YAML snippet for the pipeline: block.
Profiling
python -m cProfile -o profile.out -m mykg.cli extract input_files/
uv run snakeviz profile.out
Design
Storage & runtime model — myKG uses JSONL files (nodes.jsonl, edges.jsonl) as its persistent graph storage and builds an in-memory NetworkX DiGraph for all graph operations (traversal, search, path-finding, hub/orphan detection, MCP server queries). There is no external database dependency — the graph loads from flat files at startup and all computation happens in-process. This keeps the tool self-contained and portable: a session folder is everything you need.
For a thorough description of the architecture, algorithm, data models, and design decisions, see docs/architecture.md.
License
MIT — see LICENSE.
