Openingest

OpenIngest

Open-source data ingestion engine. 500+ connectors. Log-based CDC. Schema drift handling. PII masking. Arrow → Parquet → Lakehouse.

Features

• 500+ Connectors — Databases, SaaS APIs, files, message queues (via WASM plugins, Singer, and Airbyte compatibility)
• Log-Based CDC — Real-time change data capture from Postgres, MySQL, MongoDB
• Automatic Schema Drift — Detect and handle column additions, removals, and type changes
• Incremental + Full Refresh — Configurable sync modes per stream
• Checkpointing & Resume — Exactly-once semantics with pluggable state backends
• Inline PII Masking — Hash, nullify, truncate, mask, or tokenize sensitive columns
• Arrow → Parquet → Lakehouse — Apache Arrow for in-memory, Parquet for storage, with Iceberg / Delta Lake / Apache Hudi writers
• Singer & Airbyte Compatible — Run existing Singer taps and Airbyte sources
• WASM Plugin Architecture — Connectors as sandboxed WASM modules (safe, polyglot, hot-reloadable)
• Single Static Binary — Everything in one binary, no JVM or Python runtime required
• MCP Server — Built-in Model Context Protocol server for AI-assisted data pipeline management
• Connector SDK — Write a new connector in under 100 lines of Rust

Quick Start

# Install
curl -fsSL https://openingest.ai/install.sh | sh

# Run with demo
openingest daemon --config demo/pipeline.yaml

# Create a pipeline
openingest pipeline create -f my_pipeline.yaml

# Trigger a sync
openingest sync <pipeline-id>

# List connectors
openingest connector list

Pipeline Spec

pipeline:
  name: sales_cdc
  source:
    type: postgres
    config:
      host: localhost
      database: prod
    sync_mode: log_cdc
  destination:
    type: iceberg
    config:
      warehouse: s3://my-lakehouse/
  transforms:
    - type: pii_mask
      rules:
        - table: public.users
          column: email
          method: sha256
  schedule: "*/5 * * * *"
  schema_drift: auto_add

Architecture

┌──────────────┐   ┌──────────────────┐   ┌────────────────┐
│  WASM Plugin │   │  Singer Adapter  │   │ Airbyte Adapter│
│  (connectors)│   │  (Python taps)   │   │ (Docker specs) │
└──────┬───────┘   └────────┬─────────┘   └───────┬────────┘
       │                    │                      │
       └────────────────────┼──────────────────────┘
                            │
                    ┌───────▼────────┐
                    │  Pipeline      │
                    │  Engine        │
                    └───────┬────────┘
                            │
              ┌─────────────┼─────────────┐
              │             │             │
        ┌─────▼────┐ ┌─────▼────┐ ┌─────▼────┐
        │ Transform │ │ Schema   │ │Checkpoint│
        │ (PII)     │ │ Drift    │ │Manager   │
        └─────┬────┘ └─────┬────┘ └─────┬────┘
              │             │             │
        ┌─────▼─────────────▼─────────────▼────┐
        │        Arrow RecordBatch              │
        └────────────────┬─────────────────────┘
                         │
        ┌────────────────▼──────────────────────┐
        │  Parquet File Writer                   │
        │  ┌────────┐ ┌────────┐ ┌────────┐     │
        │  │Iceberg │ │ Delta  │ │  Hudi  │     │
        │  └────────┘ └────────┘ └────────┘     │
        └────────────────────────────────────────┘

Performance

Benchmarks run on c6g.4xlarge (16 vCPU, 32GB RAM). Full methodology in BENCHMARKS.md.

Connectors

30 Tier-1 (hand-crafted WASM): Postgres, MySQL, MariaDB, SQL Server, Oracle, MongoDB, DynamoDB, Cassandra, S3, GCS, Azure Blob, Kafka, Redpanda, RabbitMQ, HTTP/REST, GraphQL, CSV, JSON, Parquet, Snowflake, BigQuery, Redshift, ClickHouse, Elasticsearch, Redis, GitHub, Stripe, Salesforce, HubSpot, Shopify

200+ Tier-2 (declarative factory): Auto-generated from OpenAPI specs, JDBC drivers, and API templates

200+ Tier-3 (Singer/Airbyte compatibility): Run existing Singer taps and Airbyte sources

License

Apache 2.0