🇮🇩 indonesia-civic-stack
Python SDK, MCP tools, and REST wrappers for Indonesian public-sector data. Release 1.2.1 includes 14 scraper modules, 45 unified MCP tools, and 11 REST module routers. These counts describe code, not live availability.
Indonesian public-data portals expose different APIs, HTML structures, browser requirements, and access restrictions. This project provides a shared response schema and reusable clients rather than requiring every civic-tech app to build its own integration. Portal changes can still break individual operations; verify the response and its provenance before drawing conclusions.
Module inventory and maturity
Each row has a scraper, normalizer, and module test file. “Tests present” means
mocked/replayed tests exist, not that they establish current portal liveness.
The MCP column counts decorators in the unified civic_stack/server.py.
JDIH, KSEI, and DJPB are available through the SDK and unified MCP server, but have no individual MCP server or REST router. JDIH search checks up to 50 recent listing links; KSEI securities fetch/search only inspect page 1. Neither is a complete registry search. LHKPN's browser search implementation does not prove that reCAPTCHA or the upstream portal works in your deployment.
Current portal liveness is unverified here. Historical March/April 2026
observations in endpoint status and
geo-restrictions are troubleshooting context,
not present-day health evidence. An HTTP 200 homepage, local /health, or passing
mock/VCR test is not a successful live registry lookup.
Install
pip install indonesia-civic-stack # Core HTTP SDK
pip install "indonesia-civic-stack[mcp]" # + unified MCP server
pip install "indonesia-civic-stack[api]" # + FastAPI and uvicorn
pip install "indonesia-civic-stack[browser]" # + Playwright and Camoufox
pip install "indonesia-civic-stack[all]" # mcp + api + browser
Browser-backed operations also need binaries:
playwright install chromium
python -m camoufox fetch # AHU Camoufox path
AHU, OSS NIB, JDIH, LHKPN search, and OJK's full-list helper use browser paths.
BPJPH's current scraper uses HTTP REST. all does not download browser binaries
or include LHKPN's optional PDF packages. Install pdfplumber directly for PDF
text extraction; the optional scanned-document fallback needs anthropic and
ANTHROPIC_API_KEY, sends the document to an external API, and may incur cost.
There are no pdf or playwright extras. SOCKS HTTP transport additionally
requires pip install "httpx[socks]".
Python SDK
import asyncio
from civic_stack.bpom.scraper import search
async def main():
try:
async with asyncio.timeout(120):
records = await search("paracetamol")
except Exception as exc:
print("Could not verify:", type(exc).__name__)
return
if not records:
print("No records returned; acquisition may be unverified")
for record in records:
print(record.status, record.found, record.source_url, record.fetched_at)
if record.status == "ERROR":
print("Could not verify:", record.result)
elif record.found:
print(record.result) # Check identity, period, and domain-specific status
else:
print("No match reported within the query scope; not proof of absence")
asyncio.run(main())
Every scraper has fetch() and search(), with module-specific signatures.
Specialized helpers include BMKG's get_latest_earthquake, KPU's
get_election_results, OJK's check_waspada, KSEI's get_statistics_links,
and DJPB's get_budget_summary. See usage and limitations
for actual return shapes, scope limits, and unified-tool wrapper caveats.
Response interpretation
Core lookups return CivicStackResponse or lists of envelopes:
{
"result": {"product_name": "Example", "registration_status": "ACTIVE"},
"found": true,
"status": "ACTIVE",
"confidence": 1.0,
"source_url": "https://cekbpom.pom.go.id/example",
"fetched_at": "2026-03-14T06:30:00Z",
"last_updated": null,
"module": "bpom",
"raw": null
}
This is illustrative, not a live response. Status values are ACTIVE, EXPIRED,
SUSPENDED, REVOKED, NOT_FOUND, and ERROR. Error details are under
result.detail, not a top-level detail. Some helpers return raw lists/dicts,
PDF extraction data, or URLs instead of envelopes; status shortcut MCP tools
omit source and timestamp. Do not assume every tool has the full schema.
Check status, found, source_url, and fetched_at before reading result.
Retrieval time is not publication time; verify the source's reporting period.
ERROR is not NOT_FOUND. Current paths in several modules also collapse
network/browser failures into NOT_FOUND or []. Consequently, an empty or
not-found result alone cannot prove acquisition succeeded or that the record
does not exist. BPS fetch() can raise for a missing key; other callers can
also encounter unexpected exceptions. Catch exceptions at the caller boundary.
A found registration is not a general safety/legal guarantee. An alert-list match is a warning, not an endorsement. Partial multi-portal results and scraper confidence values must not be presented as exhaustive or authoritative checks.
Unified MCP server
civic-stack-mcp # stdio, 45 tools
python -m civic_stack.server # packaged equivalent
civic-stack mcp # CLI stdio entry point
python -m civic_stack.server --transport http --host 127.0.0.1 --port 8000
# Individual module server (where present):
python -m civic_stack.bpom.server
For local clients, configure civic-stack-mcp as the command. The repository's
.mcp.json uses .venv/bin/python -m civic_stack.server; create that
virtual environment and install [mcp] before discovery. Discovery does not
install dependencies or ensure portals respond. There is no shared hosted server.
Unified MCP tool inventory
These are 45 registered names, not 45 independently validated operations. Some wrappers call generic scraper functions rather than the specialized helper suggested by the tool name; read wrapper caveats before using election-results, campaign-finance, tender, alert-list, or portal-list tools.
Verify registration offline, without calling tools:
python -c 'import asyncio; from civic_stack.server import mcp; print(len(asyncio.run(mcp.list_tools())))'
REST API
civic-stack api --host 127.0.0.1 --port 8000
uvicorn civic_stack.app:app --host 127.0.0.1 --port 8000
uvicorn civic_stack.bpom.app:app --host 127.0.0.1 --port 8001
The unified app mounts 11 routers. JDIH, KSEI, and DJPB have no REST routes.
Open /docs or /openapi.json for the actual route signatures. Examples:
GET /bpom/search?q=paracetamol
GET /bpjph/search?q=kopi
GET /ahu/search?q=Contoh
GET /ojk/check/Bank%20BCA
GET /kpu/search?q=Contoh
GET /lhkpn/search?q=Contoh
GET /bps/search?q=inflasi
GET /bmkg/forecast/Jakarta
GET /bmkg/earthquake/latest
GET /simbg/search?q=Jakarta
/health only describes the local app; it does not probe upstream portals.
Deployment and security
civic_client() reads PROXY_URL, with environment-based PROXY_MODE=connect
or rewrite selection. A *.workers.dev URL auto-selects rewriting. Browser
proxy handling is separate: a Worker rewrite URL is not a standard Chromium
proxy, AHU and OSS-NIB fall back to PROXY_URL only in CONNECT mode, JDIH uses
its explicit proxy argument, and LHKPN search does not forward its proxy_url
argument to its browser. Do not assume setting an environment variable proxies
every operation. See usage.
Whoever runs a proxy controls the portal responses the server receives. The
REST routers and the standalone AHU/OSS-NIB MCP servers therefore accept a
caller-supplied proxy_url only when it exactly matches PROXY_URL or an entry
in CIVIC_ALLOWED_PROXIES (comma-separated full proxy URLs); anything else is
rejected (HTTP 400 or an ERROR envelope). Configure proxies server-side.
Access can vary by deployment, endpoint, and time. Historical regional tests do not establish a current geo-block or WAF cause. Stop after bounded attempts; do not bypass authenticated tiers or repeatedly retry access controls.
The current unified REST app does not implement API-key authentication or
per-client rate-limit middleware. Setting CIVIC_API_KEY or CIVIC_RATE_LIMIT
alone does not enable either. CIVIC_ALLOWED_PROXIES only gates caller-supplied
proxies; it is not an SSRF allowlist for SDK calls. Deploy behind your own
authentication, quotas, and network controls. Bind loopback
for local use. Module acquisition rate limiters are not public-API abuse controls.
Agent Skills installation
SKILL.md is the portable skill entry point. Its metadata version is
"1.2.1"; it uses trigger phrases in description, not unsupported frontmatter
keys. Installation must retain the complete repository, especially
references/usage.md and the other bundled references, package code, manifest,
and licenses. The installed directory must be named indonesia-civic-stack.
For a host that supports a user-selected skills directory:
# Set this to your host's configured directory; destination must not already exist.
git clone https://github.com/suryast/indonesia-civic-stack.git "$SKILLS_DIR/indonesia-civic-stack"
cd "$SKILLS_DIR/indonesia-civic-stack"
pip install -e ".[mcp]" # if running the bundled MCP code
Do not install only SKILL.md, and do not rename the package to a release
worktree's suffix. Host discovery and Python installation are separate steps.
For version-pinned installs, choose the reviewed release revision explicitly;
cloning the default branch is not a version pin.
Development and offline verification
git clone https://github.com/suryast/indonesia-civic-stack.git
cd indonesia-civic-stack
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all,dev]"
pytest --collect-only -q # derive current collected tests; no fixed advertised total
pytest -v # mocked/VCR replay; no live portal calls intended
pytest tests/jdih/ tests/ksei/ tests/djpb/ -v
ruff check civic_stack/ tests/ scripts/ examples/ app.py server.py
ruff format --check civic_stack/ tests/ scripts/ examples/ app.py server.py
mypy civic_stack/
All 14 modules have test files, including JDIH, KSEI, and DJPB. A passing offline suite validates fixtures/mocks, not upstream operation. Do not record live cassettes in CI; review the actual test isolation and recording settings first.
Read AGENTS.md, CLAUDE.md, and CONTRIBUTING.md before development. The contribution guide states the desired full module contract; the implemented surfaces above take precedence over older blanket inventory statements in other documents.
References and related projects
- Usage and implementation limits
- Historical endpoint observations
- Historical regional access observations
- Portal patterns
- Proxy operations
- Example prompts
- indonesia-civic-signal-monitor
- indonesia-gov-apis
License
The root skill/documentation license is MIT; see LICENSE.
Module licensing varies; retain module license files and consult
LICENSES.md (its older modules/ labels refer to the current
civic_stack/ package directories).
