dlake — Commercient Data Lake / Data Hub CLI
dlake is the CLI for Commercient Data Lake / Data Hub — a backend-as-a-service
for Microsoft SQL Server: instant REST and GraphQL APIs, row-level security, and
MCP access for AI agents, over your ERP, CRM, and database data.
A place to aggregate all your company data from multiple systems so that AI agents of any type can see it all — and, if set up, even write data back. Code your own solutions against all your data, safely and securely. Let loose your staff's creativity on your data without risk.
dlake is the official cross-platform command-line client — in the spirit of
the Stripe and HubSpot CLIs. It authenticates with a tenant API key, manages
profiles, and wraps the platform's REST and admin surfaces with a
scripting-friendly UX.
Product overview: datalake-ms-dab.commercient.com/datalake — the product site, with the full feature tour and how to get started.
Binary distribution. This repository publishes the official
dlakebinaries and release notes. The source code is not published here.
What is Commercient Data Lake / Data Hub?
A managed, per-tenant data platform built on Microsoft SQL Server that turns your business data into an API- and AI-ready lake. Every tenant gets its own managed SQL Server database — schema builder, Data API, events, time travel, row-level security, and object-storage attach are all native Microsoft SQL Server capabilities, not a bolt-on store. (PolyBase external-table attach and vector features require SQL Server 2022+ / 2025.)
- Bring data in — through Commercient SYNC, Commercient Data Lake / Data Hub can read and write to and from more than 150 systems (ERPs, CRMs, e-commerce, and more — see www.commercient.com). Built-in connectors cover HubSpot, Stripe, Salesforce, ServiceTitan, Pipedrive, SQL Server, and any ODBC source (via a small on-prem push agent), with incremental sync, change tracking, scheduling, integrity verification, and conflict handling for bi-directional flows. File ingest (CSV / Parquet / XML) creates and evolves tables automatically.
- Shape it — a full schema builder (tables, views, procedures, triggers, indexes, functions), a data browser, and a SQL editor with background CSV/Parquet exports.
- Serve it — a per-tenant Data API (REST + GraphQL) over exactly the
entities you expose, plus MCP connectors for AI agents (a data plane and
a full admin control plane —
dlake admin listprints every tool your key can use), natural-language querying, row-change events (SSE / polling / signed webhooks), time travel, and row-level security. - Govern it — role-based permissions, scoped API keys enforced down to entity and field level in the database (fail-closed), audit logs, and data quality rules with alerting.
- Extend it to object storage — link S3 buckets, browse/upload/download, export tables straight to a bucket, and (on SQL Server 2022+) attach parquet and CSV files as queryable external tables with automatic schema discovery.
dlake is the terminal/CI way to drive all of it. docs/help.md is a
short overview; the full product help is in the console's Help section and from
dlake guide help once you are signed in (it fetches your tenant's live copy).
- Human-readable tables by default;
--jsoneverywhere for scripting. - Exit codes: 0 ok, 1 error, 2 usage, 3 permission/auth denied.
- Self-contained single-file binaries — no .NET runtime install required.
Install
npm (recommended)
npm install -g @commercient/dlake
dlake login --domain mycompany --api-key dlk_...
The package downloads the platform-matched binary on install and exposes it as
the dlake command. That download is the package's install script: if your npm
only runs install scripts for packages you allow, install with
npm install -g --allow-scripts=@commercient/dlake @commercient/dlake (see
Windows PowerShell for the details, which apply on any OS).
Direct download
Grab the binary for your platform from the latest
Release (or from
https://datalake-ms-dab.commercient.com/downloads/dlake/<version>/<rid>/dlake[.exe]),
verify it against SHA256SUMS, and put it on your PATH.
Windows PowerShell
Two one-time steps on a fresh Windows machine, then one habit:
-
Let PowerShell run npm's command wrappers. The default execution policy blocks the
.ps1wrapper npm installs for global commands ("running scripts is disabled on this system"). Allow local and signed scripts for your own account:Set-ExecutionPolicy -Scope CurrentUser RemoteSigned -
Install with the package's download step allowed. The binary is fetched by the package's install script, and newer npm versions skip install scripts for packages you have not allowed, leaving only an npm warning:
npm install -g --allow-scripts=@commercient/dlake @commercient/dlake dlake --versionIf a command answers "dlake binary not found" after an install, the download step did not run: repeat the install line above, or run
npm rebuild @commercient/dlake. -
Quote
@filearguments. A bare leading@is PowerShell's splatting operator, so--columns @columns.jsonfails withSplattingNotPermitted. Write--columns "@columns.json"(see Argument conventions).
Quickstart
# No account yet? Sign up from the terminal — no browser, no key needed.
dlake register start --email [email protected] --company "Acme Inc" --phone +15551234567 \
--consent-crm-backup --consent-erp-backup --consent-phone
# the backup flags confirm you have made YOUR OWN backups of your CRM/ERP data;
# --consent-phone consents to phone contact
dlake register status --watch # email verified -> provisioned -> seeded
Click the single verification link we e-mail you; everything after that is
automatic. When your lake is seeded, register status prints a once-only,
7-day owner-admin API key and saves it into your profile — copy it, it is
never shown again — and the same terminal can immediately drive both the data
plane and the control plane. A second e-mail carries your welcome message and
the tenant owner's temporary password.
# Authenticate once per tenant; profiles switch between tenants.
dlake login --domain mycompany --api-key dlk_...
dlake status # tenant, key, agent + service health
# API keys & projects
dlake keys list
dlake keys create --name ci-reader # unscoped key; prints the raw key ONCE
# Query & export
dlake query "SELECT TOP 10 * FROM account" --json
dlake export account --format parquet --out ./account.parquet
# Object storage (S3 outlet)
dlake s3 connections list
dlake s3 ls sales
dlake s3 put sales ./q1.csv reports/
dlake s3 get sales reports/q1.csv ./local.csv
dlake s3 export sales account --format parquet # server-side table → bucket
# Generic admin control plane (MCP passthrough)
dlake admin list # list every admin tool your key can use
dlake admin list_schemas # call one directly — no `call` sub-verb
dlake admin create_table --help # every tool self-documents its arguments
Run dlake --help or dlake <command> --help for the full surface. Help needs
no profile. For the passthroughs, dlake admin <tool> --help and
dlake tool <tool> --help show the tool's live argument schema when a profile is
given (--profile <name>); without one they print the generic help for the verb.
Scoped keys need a DAB restart
dlake keys create mints an unscoped (full-access) key. A key restricted to
specific entities is minted on the admin plane:
dlake admin create_api_key --keyName pizza-web --expirationDays 180 --scope @scope.json
dlake admin restart_dab --confirm true # <-- REQUIRED, see below
Warning — after creating (or re-scoping) a scoped key, run
dlake admin restart_dab --confirm true. DAB's running configuration only gains the key's per-key role (key_<id>) when it is regenerated; until you restart, the key returns403 AuthorizationCheckFailedon every request even though the key, its scope, and the entities are all correct. Unscoped / full-access keys are unaffected. In the web UI the same save shows a "Restart Needed" notice and waits for you to press Restart DAB — restarts are always user-initiated. The restart must be driven by a full-scope admin key: a scoped key cannot reach the admin plane, so it can never restart DAB for itself. (Revoking a key is the one exception — it regenerates immediately.) Thecreate_api_keyresponse for a scoped key says so too, withrestartNeeded: true.
A scope entry is { "entityName": "<table/view/proc>" } plus one or more action
booleans — canRead, canCreate, canUpdate, canDelete, canExecute — with
an optional schemaName for an external/lake schema (omit it for the working
schema). entity is rejected ("empty entityName"), and so is a bare
read: true ("no actions selected").
[
{ "entityName": "pizza_menu_items", "canRead": true },
{ "entityName": "usp_PlaceOrder", "canExecute": true }
]
Build an app end-to-end
The full happy path, in the only order that works — schema, an atomic write, the API surface, then a least-privilege key for the front end:
dlake login --domain <tenant> --api-key dlk_... # admin, FULL-SCOPE key
dlake tool get_active_schema # which schema am I in?
dlake admin create_table --tableName pizza_menu_items \
--columns @menu_items.json --primaryKey Sku
dlake admin create_procedure --procedureName usp_PlaceOrder \
--parameters @params.json --body @proc.sql
dlake admin set_entity_exposure --entity pizza_menu_items --expose true
dlake admin set_entity_exposure --entity usp_PlaceOrder --expose true
dlake admin restart_dab --confirm true # publishes the entities
dlake admin create_api_key --keyName pizza-web --expirationDays 180 --scope @scope.json
dlake admin restart_dab --confirm true # publishes the key's role
# then, from the app: POST /auth/<tenant>/api/usp_PlaceOrder
Two restarts, not one: the first publishes the entities, the second the new key's role, and the key can only be minted after the entities exist.
--columns is a list of column objects in the table column shape:
A worked menu_items.json for the table above:
[
{ "name": "Sku", "type": "NVARCHAR", "size": 40 },
{ "name": "Name", "type": "NVARCHAR", "size": 200, "nullable": false },
{ "name": "Price", "type": "DECIMAL", "precision": 10, "scale": 2, "nullable": false },
{ "name": "Active", "type": "BIT", "default": "1" },
{ "name": "CreatedAt", "type": "DATETIME2", "default": "SYSUTCDATETIME()", "isExpression": true }
]
Any other property is refused with a hint naming the one meant. This is not the
REST or procedure-parameter shape: a string length is size (not maxLength),
the type is type (not dataType), and NULL-ability is nullable (not
isNullable). A size on a numeric type, or precision/scale on a string
type, is refused the same way. add_column and alter_column take one object in
this shape.
Three things that bite on the way through:
- The data plane sees only exposed entities.
create_table/create_proceduretouch your database, not the Data API —dlake tool create_record --entity pizza_menu_itemsreturnsEntityNotFounduntilset_entity_exposure --expose trueandrestart_dab --confirm truehave both run. Raw-SQL reads (dlake query,dlake tool query) don't need exposure. - The active schema is read-only from the CLI. All DDL tools operate on the
tenant's active schema (usually
DLO). Read it withdlake tool get_active_schema; it is switched only from the web UI's schema dropdown — there is noset_active_schematool and nouseverb. Qualify raw SQL with it (FROM DLO.pizza_menu_items), and prefix object names per app to keep several apps tidy in one schema. - Procedure parameters are a different shape from table columns, and their
names must start with
@. Acreate_procedure --parametersentry is{ name, dataType, maxLength }, and{"name":"CustomerName"}is rejected with "must start with '@'". A procedure-parameter example:{"name":"@CustomerName","dataType":"NVARCHAR","maxLength":100}. Do not reuse it for--columns, which takes the table column shape above.
Two more small ones: ingest_table needs a natural-key PK — it refuses a
table whose primary key is an IDENTITY column ("Row-by-row upsert needs a key
whose values come from the file"), so seed identity-keyed tables with
create_record or give the table a natural key; and avoid reserved T-SQL words
in column names — a column called LineNo fails because LINENO is reserved
(likewise Key, Order, User, Percent).
Atomic multi-row writes
Data API writes are not transactional — a multi-step write (a header, its line items, a status row)
can partially succeed and leave orphans. When you need all-or-nothing, put the work in a stored
procedure that wraps BEGIN TRAN / COMMIT / ROLLBACK, expose it, and call it as one request:
dlake admin create_procedure --procedureName usp_PlaceOrder --parameters @params.json --body @proc.sql
dlake admin set_entity_exposure --entity usp_PlaceOrder --expose true
dlake admin restart_dab --confirm true
It either commits everything or leaves the database untouched, and it returns the created rows.
Procedure entities are exposed on POST only — a write should not be reachable by a "safe" method.
Argument conventions
Tool arguments (dlake admin <tool> / dlake tool <tool>) accept three forms:
- Scalars —
--tableName Invoice,--confirm true. - Arrays and objects — a value starting with
[or{is sent as JSON (--primaryKey '["Id"]'); a plain comma list still works for arrays of scalars (--primaryKey Id). Malformed JSON is reported as a usage error naming the argument. @file— read the value from a file:--columns @columns.json.@@escapes a literal leading@. This is the reliable form on Windows, where quoted JSON is mangled by the shell before the program sees it. In PowerShell, quote it —--columns "@columns.json"— because a bare leading@is the splatting operator (SplattingNotPermitted).
dlake admin create_table --tableName Invoice --columns @columns.json --primaryKey Id
dlake admin create_table --tableName Invoice --columns "@columns.json" --primaryKey Id
dlake admin <tool> --help documents these forms for every tool that takes an
array or object argument.
Documentation
- Product overview — what the platform does, in one page: https://datalake-ms-dab.commercient.com/datalake/.
- API usage guide — endpoints, auth, scopes, rate limits: see the Data Lake \ Data Hub API guide served from your tenant's Help page.
- AI-agent skills — twenty skill summaries, one per Commercient integration:
what it does for the business and how to get its operating guide.
skills/dlakecovers building and operating a tenant;skills/dlake-integration-setupstands up a new integration; one skill each for the sync products — Normal Sync, ODBC Sync, Generic API Sync, CRMPro with its destination skills for HubSpot, Salesforce, Shopify, Zoho CRM, Dynamics CRM, MDC and Magento, and TxDownloaderPro with its destination skills for Salesforce, HubSpot, Dynamics CRM, Zoho CRM and Shopify — plusskills/dlake-syncagentfor the on-premises agent that runs on the customer's own ERP server. Read one withdlake skills show <name>or install them all withdlake skills install, which refreshes any copies already on disk (add--skip-existingto keep local edits); seeskills/README.md. Whitelisted customers fetch the operational skills, with their per-ERP pages, from the admin plane withdlake admin get_skill <name>. - Permissions — object writes and connection management need
data.ingest.manage; reads accept anydata.ingest.*tier; scoped API keys are enforced server-side (fail-closed) down to entity and field level.
Configuration
Verifying downloads
Every release ships a SHA256SUMS file:
sha256sum -c SHA256SUMS --ignore-missing
The npm postinstall verifies automatically, and its integrity chain does not
follow the mirror: pointing DLAKE_DOWNLOAD_BASE at a private mirror moves the
binary only — SHA256SUMS is still fetched from
datalake-ms-dab.commercient.com, so a mirror can only serve bytes the
publisher already vouched for. Air-gapped installs pin DLAKE_SHA256=<digest>
(recommended) or opt into DLAKE_ALLOW_MIRROR_CHECKSUMS=1, which transfers full
trust to the mirror host and prints a warning. An older DLAKE_VERSION than the
installed package is refused unless DLAKE_ALLOW_DOWNGRADE=1.
Questions & support
For any questions, please email [email protected]. That is the fastest way to reach us for help with the CLI, the platform, API keys, connectors, or a Commercient SYNC integration. You can also open a GitHub issue here for CLI bugs and feature requests.
Product overview for Data Lake / Data Hub: https://datalake-ms-dab.commercient.com/datalake/. Learn more about Commercient and the 150+ systems SYNC connects at www.commercient.com.
License
The dlake binaries are proprietary software, free to use with a Commercient
Data Lake / Data Hub subscription — see LICENSE.