⟠ GEO-Scope
Empirical AI Answer Visibility Measurement Framework
An open-source framework for empirical measurement of AI answer visibility, entity mentions, recommendations, and citations across generative AI systems.
PyPI package · Hugging Face demo
🏆 Landmark Benchmark (Iran & Türkiye) • Introduction • What It Measures • Scientific Foundation • Benchmarks • Reproducibility • Research • Quickstart • MCP
🔬 Empirical Pilot Benchmark: Commercial AI Visibility in Iran & Türkiye (2026.1)
[!IMPORTANT]
📊 Exploratory Regional Commercial AI Visibility Pilot
This pilot benchmark marks an exploratory empirical baseline evaluating commercial AI search visibility, entity recommendations, and web citation patterns across Iran (
fa-IR) and Türkiye (tr-TR). Conducted with zero synthetic fabrication, 100% real-time search-grounded completions, and cryptographic auditability, it establishes an evidence-backed baseline for Generative Engine Optimization (GEO) research in regional markets.
Explore Benchmark Dataset & Specs • Full Results (English) • گزارش نتایج (فارسی) • Sonuç Raporu (Türkçe) • Nəticələr (Azərbaycan) • النتائج الكاملة (العربية)
Key Benchmark Metrics at a Glance
Market Rankings & Share of Voice (SOV)
🇮🇷 Iran (fa-IR) Commercial Market (72 completions)
🇹🇷 Türkiye (tr-TR) Commercial Market (72 completions)
Scientific Integrity Note: Stradiji received 0 brand mentions; 60 false-positive token matches with the Turkish word 'strateji' were filtered by GEO-Scope's entity disambiguation engine.
Search Engine Grounding & Citation Density
- Perplexity Sonar (
sonar-pro): 1,003 citations (20.90 citations/query, 7.94s avg latency) - Google Gemini Grounding (
gemini-2.5-flash): 944 citations (19.67 citations/query, 17.85s avg latency) - ChatGPT Search (
gpt-4o): 344 citations (7.17 citations/query, 10.42s avg latency)
Top Ecosystem Domains Cited
- Türkiye: Heavily structured around agency directories (
agencies.semrush.com#1 with 22 citations,edvido.com#2 with 16 citations,pr.net.tr#3 with 16 citations). - Iran: Driven by specialized consultant blogs (
iranweblife.com#1 with 21 citations,miladjalayi.com#2 with 14 citations) and digital PR platforms (triboon.net#5 with 12 citations).
🔗 Read Full 5-Language Methodology & Benchmark Report
شروع خیلی ساده (برای تیم)
GEO-Scope پاسخ‌های یک یا چند موتور هوش مصنوعی را بررسی می‌کند و پنج چیز را جداگانه گزارش می‌دهد: آیا نام شما آمده، آیا پیشنهاد شده، آیا لینک یا citation داده شده، آیا مطلبی به شما نسبت داده شده و آیا در فهرست رتبه گرفته‌اید.
نکته‌ی مهم درباره‌ی نیازمندی‌ها
خود موتور به‌تنهایی کلید API ندارد و چیزی را جعل نمی‌کند. برای یک تست بدون هزینه و بدون اینترنت از حالت simulation یا replay استفاده کنید. برای live باید خودتان دسترسی و کلید ارائه‌دهنده‌ی موردنظر را تنظیم کنید؛ Hugging Face، ZeroGPU و «LLM رایگان نامحدود» جزو نیازمندی‌های این بسته نیستند.
نصب و اولین تست بدون کلید
pip install geo-scope
geo-scope demo
این دستور یک SIMULATION FIXTURE می‌سازد؛ یعنی برای یادگیری و بررسی فرمت خروجی است و به مدل زنده وصل نمی‌شود. بنابراین عددهای آن را گزارش واقعی بازار یا رتبه‌ی برند ندانید.
چهار واژه‌ای که باید بدانید
simulation: اجرای آزمایشی قطعی و بدون تماس با مدل.live: اجرای واقعی با provider و هزینه/کلید همان provider.replay: تحلیل دوباره‌ی پاسخ‌های ذخیره‌شده، کاملاً آفلاین.citation: لینکی که موتور پاسخ به‌عنوان منبع نشان داده؛ با «ذکر نام» یکی نیست.
اجرای واقعی با مدل Hugging Face (اختیاری)
دموی عمومی Space عمداً بدون کلید و بدون GPU کار می‌کند و خروجی آن fixture قطعی است. برای یک اندازه‌گیری زنده با مدل متن‌باز، کلید حساب خودتان را فقط در محیط محلی بگذارید:
pip install geo-scope
export HF_TOKEN="hf_..."
# در صورت نیاز مدل قابل‌دسترسی حساب/ارائه‌دهنده را انتخاب کنید
export HF_MODEL="Qwen/Qwen2.5-7B-Instruct"
geo-scope measure \
--entities examples/public_demo/brands.json \
--prompts examples/public_demo/prompts/observed.jsonl \
--providers huggingface_inference \
--mode live \
--out-dir output/hf-live
hf، huggingface و huggingface_inference نام‌های قابل‌استفاده‌اند. این
provider پاسخ مدل را می‌سنجد، اما web search یا citation واقعی به آن اضافه
نمی‌کند؛ بنابراین citationهای خروجی را فقط وقتی گزارش کنید که خود مدل واقعاً
لینک داده باشد. کلید را در README، فایل Space، issue، commit یا لاگ قرار
ندهید. اعتبار رایگان Hugging Face محدود و وابسته به سیاست فعلی حساب است؛
«رایگان نامحدود» وعده‌ی فنی قابل‌اتکایی نیست. برای استفاده‌ی بدون هزینه‌ی
ابری، از simulation/replay یا یک مدل محلی با Ollama استفاده کنید.
اگر هدف فقط یادگیری است، از demo شروع کنید. اگر هدف اندازه‌گیری واقعی visibility است، ابتدا یک فایل entity و یک فایل prompt آماده کنید، provider را تنظیم کنید و بعد live را اجرا کنید. بدون پاسخ واقعی provider، GEO-Scope ادعای visibility نمی‌کند. راهنمای فارسی برای مثال‌های تیمی آماده است.
1. Introduction
Generative AI systems and search-grounded answer engines are rapidly becoming the primary discovery layer for users seeking products, vendors, services, and factual insights.
GEO-Scope is an evidence-first, open-source measurement framework designed to empirically quantify and preserve auditable evidence of how generative AI systems surface entities. It records, normalizes, and analyzes observable AI completions under documented, neutral prompt sets without relying on speculative ranking algorithms or ungrounded claims.
Core Observable Outputs Measured:
- Entity Mentions: Observable presence of brands, products, technologies, and public figures in generated text.
- Recommendations: Explicit linguistic endorsements and ordered top-position recommendations.
- Citations: Grounding source URLs and referenced web domains returned by search-augmented models.
- Attribution: Textual credit linking specific facts, statistics, or claims to source entities.
- Provider Differences: Distributional shifts between live search-grounded answer engines and parametric foundation LLMs.
- Multilingual Behavior: Cross-lingual response variations across 26+ evaluated languages.
2. What GEO-Scope Measures
GEO-Scope enforces a strict taxonomic separation between four independent visibility dimensions:
┌─────────────────────────────────────────────────────────────────────────┐
│ AI RESPONSE VISIBILITY MATRIX │
├───────────────────┬─────────────────────────────────────────────────────┤
│ Mention │ Did the entity appear anywhere in the completion? │
│ Recommendation │ Was the entity explicitly endorsed or recommended? │
│ Citation │ Was a source URL or grounding domain link provided? │
│ Attribution │ Was specific data/claim textually credited to it? │
│ Rank │ Extracted ONLY when a valid ordered list exists. │
└───────────────────┴─────────────────────────────────────────────────────┘
- Mention (
mentioned: true/false):- Captures whether the target entity (or associated canonical aliases/founders) appeared in the generated completion.
- Evaluated via Unicode NFKC normalization, Arabic/Persian letter unification, Zero-Width Non-Joiner (ZWNJ) handling, and negative homonym collision filtering.
- Recommendation (
recommended: true/false):- Strict rule:
mentioned != recommended. - Evaluated based on explicit linguistic recommendation markers (e.g., "We recommend...", "Top pick", "گزینه پیشنهادی") or inclusion in an ordered list answering a recommendation query.
- Strict rule:
- Citation (
cited: true/false):- Identifies presence of target entity web domains in grounding references, markdown hyperlinks, or structured provider citation chunks.
- Attribution (
attributed: true/false):- Distinct from citation: detects explicit textual sourcing phrasing (e.g., "According to [Entity]...", "طبق گزارش [موجودیت]") even if an active URL link was omitted by the model.
- Rank (
rank: 1..N | null):- Extracted strictly from numbered lists or ordinal items. If an informational question yields an unranked mention, rank is set to
nullto prevent artificial ranking bias.
- Extracted strictly from numbered lists or ordinal items. If an informational question yields an unranked mention, rank is set to
3. What GEO-Scope Does NOT Measure
To maintain scientific integrity, GEO-Scope clearly outlines its epistemic boundaries:
- ❌ It does NOT reverse-engineer internal ranking algorithms: GEO-Scope observes external API completions; it cannot inspect internal model weights, attention matrices, or proprietary ranking formulas.
- ❌ It does NOT inspect hidden training data: Observed entity knowledge reflects generated outputs, not full visibility into private training corpora.
- ❌ It does NOT claim causal ranking factors: All reported metrics represent descriptive statistical associations under documented prompts, not causal guarantees.
- ❌ It does NOT guarantee SEO or AI visibility improvements: Measurements provide historical observation, not predictive visibility promises.
- ❌ It does NOT treat simulation as live empirical data: Simulation fixtures are strictly quarantined for testing and CI.
4. Architecture
GEO-Scope operates as a modular, six-stage evidence pipeline:
┌─────────────────────────────────────────────────────────────┐
│ Provider Layer │
│ ┌──────────────────────────┐ ┌────────────────────────┐ │
│ │ Search Answer Engines │ │ Parametric Base LLMs │ │
│ │ (Perplexity, Gemini...) │ │ (OpenAI, Claude...) │ │
│ └──────────────────────────┘ └────────────────────────┘ │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Measurement Engine │
│ (Prompt Provenance · Zero Silent Fallback · Runs) │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Raw Response Storage │
│ (Unparsed API Payloads · Latency · Token Usage) │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Observation Parser │
│ (Multi-Lingual Normalizer · Homonyms · Citations) │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Metrics Calculation │
│ (OMR · Rec Share · Citation Rate · Honest Denominators) │
└──────────────────────────────┬──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Reports + Replayable Bundle │
│ (JSONL Bundles · SHA-256 Checksums · Markdown Summaries) │
└─────────────────────────────────────────────────────────────┘
5. Execution Modes
GEO-Scope provides three mutually exclusive execution modes:
demo (Simulation Fixture)
- Purpose: Rapid offline testing, development fixtures, and CI validation.
- Behavior: Uses local mock completions with prefixed IDs (
simulated_*) and a clear simulation banner. - Guarantee: Simulation data is strictly rejected by the release quality gate and can never enter published empirical benchmarks.
measure (Live Empirical Execution)
- Purpose: Real-world observation runs against live generative AI endpoints.
- Behavior: Dispatches neutral prompt bundles to configured API providers with zero silent fallback.
- Preservation: Saves full unmodified payloads to
raw_responses.jsonlwith exact model governance metadata (requested_provider,actual_provider,search_grounded).
replay (Deterministic Offline Replay)
- Purpose: Independent auditability and benchmark verification without API calls or cost.
- Behavior: Reruns the observation parser and metric calculations directly against preserved
raw_responses.jsonl. - Integrity: Verifies that recomputed metrics match published results bit-for-bit.
6. Measurement Contract v1 & Scientific Foundation
GEO-Scope does not claim universal AI visibility truth. It measures empirical observations under declared, reproducible measurement configurations.
[!IMPORTANT] Fundamental Measurement Axiom
"AI visibility is an observation under a declared measurement system, not a universal ground-truth ranking."
- 🏛️ Scientific Measurement Gate:
docs/SCIENTIFIC_MEASUREMENT_GATE.md - 📄 Methodology:
docs/METHODOLOGY.md - 📐 Machine-Readable Schema:
schemas/v0.3/manifest.schema.json - 🧪 Validation Example Fixture:
examples/public_demo/manifest.json - 📊 Documentation Index:
docs/index.md
Core Measurement Principles
- Mention Definition: A response-level binary observation indicating whether the target entity appears at least once in the completion. Multiple mentions in a single answer do not artificially inflate response-level mention counts.
- Citation Separation: Strict 4-way separation between
entity_mentionedin text,target_domain_cited(root domain),target_url_cited(deep link), andthird_party_source_cited(external authority/review links). Mention and citation are never treated as equivalent. - Recommendation Semantics: Evaluated as true only when the model semantically recommends, selects, or endorses the entity. Ambiguous detections are gated and marked
experimental. - Comparability Rules: Machine-readable comparability verification. Two studies are marked
comparable: trueonly when prompt universe, market/language, provider/model family, measurement definitions, and observation windows match. - Raw Evidence Traceability: Every public observation is linked to prompt ID, raw response or cryptographic SHA-256 hash (
response_hash_sha256), extracted entities, citations, and execution configuration hash.
7. Published Benchmark Releases
GEO-Scope maintains immutable, peer-review-ready benchmark releases under benchmark/releases/ (see the documentation index):
Mechanism & Replication Studies
- Query Fan-Out Brand Introduction: Empirical replication test examining whether brands introduced by the engine's internal search/fan-out queries are more likely to appear in the final answer than retrieved-only brands. Across 50 commercial discovery conversations (300 brand observations), engine-introduced brands achieved a 65.7% mention rate vs 13.1% for retrieved-only brands (5.01× ratio;
REPRODUCED_DIRECTIONALLY). Intermediate query fan-out tracking is preserved as an experimental diagnostic layer and explicitly excluded from composite visibility scores to avoid causal overreach.
Every release bundle contains:
manifest.json: Dataset metadata, provider matrix, and schema version (measurement_contract_version: "1.0").prompts.jsonl: Neutral, categorized prompts.raw_responses.jsonl: Verbatim API completion payloads.observations.jsonl: Granular extracted observation records.metrics.json: Aggregated metrics with explicit failure denominators.checksums.sha256: SHA-256 hashes of all artifacts.
8. Reproducibility & Auditability
1. Cryptographic SHA-256 Verification
Verify that dataset files have not been modified or corrupted:
geo-scope benchmark verify --dataset benchmark/releases/global-ai-answers-2026.2
2. Zero-Network Replay Workflow
Replay metrics directly from preserved raw responses without executing live API calls:
geo-scope replay \
--bundle benchmark/releases/global-ai-answers-2026.2 \
--out-dir output/replay_2026_2
3. Golden Parser Evaluation
Evaluate the deterministic multi-lingual parser against human-labeled ground truth:
geo-scope parser evaluate --golden-set benchmark/golden_sets/v1
Golden Set Benchmark Results (v1, 220 examples across 5 languages):
- Mention F1:
99.75%(Precision: 99.51%, Recall: 100.00%) - Recommendation F1:
100.00%(Precision: 100.00%, Recall: 100.00%) - Citation F1:
100.00%(Precision: 100.00%, Recall: 100.00%) - Attribution F1:
91.56%(Precision: 100.00%, Recall: 84.44%) - Wrong Entity (Homonym) F1:
96.97% - Rank Accuracy:
100.00%
9. Research & Documentation
- 🏛️ Scientific Measurement Gate:
docs/SCIENTIFIC_MEASUREMENT_GATE.md - 📄 Methodology:
docs/METHODOLOGY.md - 📊 Documentation Index:
docs/index.md - 🔬 Research Methods & Protocol:
docs/METHODOLOGY.md - 📄 Research Paper:
docs/research/global-ai-answers-2026.2/global-ai-answers-paper.md - 🔒 Public Reality Audit:
docs/REALITY_AUDIT.md - 📊 Methodology Crosswalk (Public Practice Comparison):
docs/METHODOLOGY_CROSSWALK.md - 🔬 Scientific Benchmark Methodology:
docs/benchmark-methodology.md - 🗺️ Cross-Repository Evidence Map:
docs/EVIDENCE_MAP.md - 📐 Mathematical Formulation & MAVI:
docs/MATHEMATICAL_MODEL.md - 🔌 API Integration Guide:
docs/API_INTEGRATION.md - 🛡️ Release Gate Integrity Protocol:
docs/SCIENTIFIC_MEASUREMENT_GATE.md
10. Installation & Usage
Installation
# Clone the repository
git clone https://github.com/tmolavi/geo-scope.git
cd geo-scope
# Install package in editable mode
pip install -e .
PyPI publishing
The stable package is available at PyPI. Future releases are published from GitHub Actions through Trusted Publishing; no PyPI token is stored in the repository.
To publish a new version:
- Update the version in
pyproject.toml. - Commit and push the release changes to
main. - Create a published GitHub Release for the matching version.
The workflow is .github/workflows/publish.yml
and uses the pypi GitHub environment for tmolavi/geo-scope. The PyPI-side
publisher must remain mapped to Owner tmolavi, Repository geo-scope,
Workflow publish.yml, Environment pypi.
Do not put a PyPI API token in GitHub secrets or commit it to this repository.
Quick Commands
# 1. Run local simulation fixture demo
geo-scope demo
# 2. Execute live empirical measurement (requires API credentials)
geo-scope measure \
--entities entities/iran-seo-agencies.json \
--prompts examples/research_run/prompts.jsonl \
--mode live \
--providers perplexity_sonar,gemini_grounding \
--out-dir output/live_run_01
# 3. Deterministic offline replay
geo-scope replay \
--bundle output/live_run_01 \
--out-dir output/replay_run_01
# 4. Launch interactive local research dashboard
geo-scope serve --host 127.0.0.1 --port 8000
11. Model Context Protocol (MCP)
GEO-Scope includes a native MCP Server (stdio), enabling AI coding assistants and agents (Claude Desktop, Cursor, Antigravity) to query visibility benchmarks and inspect entity evidence chains directly:
{
"mcpServers": {
"geo-scope": {
"command": "/absolute/path/to/geo-scope/.venv/bin/geo-scope",
"args": ["mcp"]
}
}
}
See the Client Integrations Guide for full configuration details.
Citation & Authorship
Developed by Taqi Molavi (Senior SEO Strategist & GEO Systems Architect).
Part of the Molavi GEO Pyramid research initiative.
@software{molavi2026geoscope,
author = {Molavi, Taqi},
title = {GEO-Scope: Empirical AI Answer Visibility Measurement Framework},
year = {2026},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/tmolavi/geo-scope}},
note = {Personal Homepage: https://molavi.pro/}
}
License
This project is licensed under the MIT License — Copyright (c) 2026 تقی مولوی (Taqi Molavi).