Laya

Multilingual, non-autoregressive System 1 decision engine. Typed decisions over 100+ languages in a single forward pass — 33 ms — trained with reinforcement learning against strictly proper scoring rules (RLCD), with a router that picks the right checkpoint per request.

Installation

python -m pip install laya

With uv, run uv add laya in a uv project or uv pip install laya in a virtual environment.

Python 3.10 or newer. Optional extras: laya[serve] (HTTP server), laya[mcp] (MCP server), laya[langchain] (LangChain and LangGraph), laya[llamaindex] (LlamaIndex selectors), laya[crewai] (CrewAI routing), laya[onnx] (ONNX Runtime), laya[fast] (TileLang GPU fast path), laya[structured] (pydantic models in decide), laya[vision] (Pillow and torchvision, for predict(image=...) and predict(video=...)), laya[audio] (soundfile, to decode audio files). Step-by-step setup for each platform, CPU-only or GPU PyTorch builds, and troubleshooting are in Installation details.

For TypeScript / Node.js / browser, see laya-ts/. npm releases (npm install laya-ts) are published from this repository's laya-ts-v* release tags.

Long documents. laya-multilingual reads up to 8,192 tokens with max_len=8192. Measured accuracy and time by document length, reproducible with research/scripts/bench_long_context.py:

Quickstart

One argument puts vegaml's engine under laya's names: a frozen 0.8B reader with a small physics engine on top, so every answer carries a conformal prediction set, an abstain flag, an out-of-distribution flag and a route decided without a threshold you had to guess at. Text, images, speech and 73k-token documents go through the same call. Python 3.10 or newer; details in vega inside laya.

python -m pip install -U laya   # 0.4.23 or newer: vegaml comes with it

import laya

agent = laya.Agent(vegaml=True)   # the 0.8B universal reader (nandakishorm/vega-U), the default

# Switching the model underneath is one string. Every option keeps the same engine, questions and fields:
#   laya.Agent(vegaml="4b")                                # the Qwen3.5-4B reader from the same Hub repo
#   laya.Agent(vegaml="gemma-1b")                          # the gemma-3-1b-it reader, same repo
#   laya.Agent(vegaml="nandakishorm/vega-U")               # any Hub repo id holding a vega-U checkpoint
#   laya.Agent(vegaml="./my-reader/")                      # a local dir from VegaAgent.attach(...), your own LLM ported in
#   laya.Agent(vegaml=True, backbone="Qwen/Qwen3.5-0.8B")  # name the reader weights explicitly, e.g. a mirror
# Without the argument, laya's own encoders, also by Hub name:
#   laya.Agent("convaiinnovations/laya")                   # English, 33 ms on a T4
#   laya.Agent("convaiinnovations/laya-multilingual")      # 100+ languages, up to 8,192 tokens with max_len=8192
#   laya.Agent("convaiinnovations/laya-typed-decisions")   # fine-tuned for the typed-decisions workflows

state = "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."
questions = {
    "department": {"type": "choice", "instructions": "Which department should handle this?",
                   "criteria": {"billing": "invoices, payments, refunds",
                                "technical": "bugs, outages, system errors",
                                "other": "everything else"}},
    "urgency": {"type": "score", "instructions": "How urgent is this?",
                "criteria": ["not urgent", "soon", "blocking"]},
    "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
}

out = agent.predict(state, questions)
ans = out["answers"]["department"]
print(ans["choice"])             # billing
print(ans["probabilities"])      # {"billing": 0.91, "technical": 0.04, "other": 0.05}
print(ans["prediction_set"])     # ["billing"]: the options the answer could still be, at alpha 0.1
print(ans["abstain"])            # False: the engine is not refusing this one
print(ans["unbound"])            # False: the state looks like something it has seen
print(ans["route"])              # "auto"; "review" when the set, abstain or unbound say so
print(out["answers"]["churn_risk"]["noul"])   # probability the answer is yes

The output keeps laya's shape (choice, score, noul, probabilities, answer_confidence) and adds vega's fields. Gate on route, not on a confidence number: it comes from the conformal set and the fitted band, which stay right on answers where the top probability does not.

Model switching. Keep a laya encoder for short English text and let vega take the rest:

from laya.vega import Router

router = Router(fast=laya.Agent(), vega=agent)
out = router.predict(state, questions)
print(out["routing"])            # {"model": "fast", "reason": "short English text, no fitted head: the fast encoder is enough"}
out = router.predict("मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।", questions)
print(out["routing"]["model"])   # vega: non-English goes to the reader

The rule, in order: an image or audio goes to vega; text longer than the fast model's context goes to vega; non-English or non-Latin text goes to vega; a question with a fitted head goes to vega; everything else goes to the encoder. model="vega" or model="fast" forces one.

Images, speech, video, long documents. The questions and the answer shape do not change with the input:

# text: up to 73,728 tokens in one pass; usage["truncated"] says if a state hit the limit
agent.predict_long(seventy_page_contract, questions)
# image: PIL image, path or HxWx3 array, any format Pillow opens; scaled to ~1 megapixel, no file-size cap
#        needs pip install 'laya[vision]' (Pillow and torchvision), as does video
agent.predict(image="invoice.png", questions=questions)
# audio: path or bytes (wav, flac, ogg, mp3; needs pip install 'laya[audio]') or a 1-D waveform with sampling_rate=;
#        any length, resampled to 16 kHz, transcribed in 30 s windows, transcript counts toward the 73,728 tokens
agent.predict(audio=wave, questions=questions, sampling_rate=16000)
# video: path or bytes, any container ffmpeg opens, any length and resolution (needs the ffmpeg binary);
#        speech is transcribed and decided as text, a silent clip is decided on 4 sampled frames as one image
agent.predict(video="call.mp4", questions=questions)

Fit and calibrate on your own labels. Seconds, no gradient step, and the route is decided for you. Labels are a CSV with a state column and one column per question id; examples/data/routing_labels.csv is a 60-row example that routes messages to billing, HR, PR, sales or technical and rates their urgency:

state,team,urgency
"We were charged twice for the March invoice, please refund the duplicate.",billing,2
"I would like to apply for the open backend engineer role. Where do I send my CV?",hr,0
"A journalist from the Economic Times is asking for a comment on today's outage.",pr,2

The whole loop, from import to the answer, is examples/42_fit_and_calibrate_vegaml.py. It holds out 4 rows per team (20), fits on the other 40, and measures the same 20 rows before and after. Condensed:

import csv, laya

questions = {
    "team": {"type": "choice", "instructions": "Which team should handle this message?",
             "criteria": {"billing": "invoices, payments, refunds", "hr": "jobs, payroll, leave, policy",
                          "pr": "press, media, public statements", "sales": "pricing, demos, quotes, trials",
                          "technical": "bugs, outages, API and product questions"}},
    "urgency": {"type": "score", "instructions": "How soon does this need a reply?",
                "criteria": ["can wait", "this week", "today"]},
}
rows = list(csv.DictReader(open("examples/data/routing_labels.csv")))
held, fit_rows = rows[-20:], rows[:-20]            # the script holds out 4 per team; same idea

agent = laya.Agent(vegaml=True)
print(laya.VegaAgent.min_rows(questions))         # 35: the fewest rows fit() accepts for these questions

before = agent.predict(held[0]["state"], questions)["answers"]["team"]       # the cold engine

report = agent.fit([{"state": r["state"], "gold": {"team": r["team"], "urgency": r["urgency"]}}
                    for r in fit_rows], questions=questions)                # or agent.fit("examples/data/routing_labels.csv", questions=questions)
print(report["n_fit"], report["n_calibrate"])     # 30 fit the heads, 10 are held out inside fit() to calibrate on
print(report["serving"])                          # the questions whose head beat the engine on the same rows
print(report["calibration"]["band"])              # the review line per question, or None where it could not hold

after = agent.predict(held[0]["state"], questions)["answers"]["team"]
agent.save_calibration("fit-routing/")            # heads + calibration; agent.load_calibration("fit-routing/") later

What the script printed on a laptop GPU, every number measured in that run:

minimum rows fit() accepts for these questions: 35 (5 per option of the widest question for the heads, plus a quarter held out to calibrate on, never fewer than 10)

== before: the cold engine, 20 held-out rows
   team accuracy      0.600   ECE 0.221   mean confidence 0.629
   urgency accuracy   0.400   ECE 0.412   MAE 0.65 levels
   team set sizes     {1: 13, 2: 7}   abstain 12   route {'auto': 6, 'review': 14}

== fit + calibrate on 40 rows in 9s: 30 fit the heads, 10 held out to calibrate on
   team     head cv 0.767 vs engine 0.633 on the same rows  ->  SERVED
   urgency  head cv 0.400 vs engine 0.400 on the same rows  ->  left to the engine
   team     temperature 0.83  conformal q-hat 0.987  ECE on the calibration rows 0.281 -> 0.241
   urgency  temperature 6.44  conformal q-hat 0.741  ECE on the calibration rows 0.515 -> 0.142
   review line per question: {'team': None, 'urgency': None}
   ['team', 'urgency']: no line holds precision 0.95 with a 95% bound on 10 calibration rows, so the conformal set routes those answers

== after: heads + calibration, 20 held-out rows
   team accuracy      0.900   ECE 0.206   mean confidence 0.749
   urgency accuracy   0.400   ECE 0.059   MAE 0.65 levels
   team set sizes     {2: 2, 3: 2, 4: 6, 5: 10}   abstain 0   route {'review': 20}

== change on held-out rows: team accuracy +0.300, team ECE -0.015, urgency MAE +0.00

== one held-out message: "Why was a late fee added to this month's bill when we paid on the due "  (gold: billing)
   before: {"choice": "billing", "probabilities": {"billing": 0.999995, "hr": 3e-06, "pr": 0.0, "sales": 1e-06, "technical": 1e-06}, "answer_confidence": 0.999995, "prediction_set": ["billing"], "abstain": false, "unbound": false, "readout": "engine", "route": "auto", "route_reason": "conformal singleton, not abstaining, in distribution"}
   after:  {"choice": "billing", "probabilities": {"billing": 0.674496, "hr": 0.186172, "pr": 0.068743, "sales": 0.041369, "technical": 0.029221}, "answer_confidence": 0.674496, "prediction_set": ["billing", "hr", "pr", "sales", "technical"], "abstain": false, "unbound": null, "readout": "ttt", "route": "review", "route_reason": "band could not hold precision 0.95 on 10 calibration rows, so the conformal rule decides: conformal set at alpha 0.1 holds 5 options"}

== ablation: fitting rows per team, same 20 held-out rows
   per team  rows  team acc  team ECE  urg. acc  urg. MAE  route
   cold      0     0.600     0.221     0.400     0.65      {'auto': 6, 'review': 14}
   7         35    0.800     0.111     0.400     0.65      {'auto': 14, 'review': 6}
   8         40    0.900     0.206     0.400     0.65      {'review': 20}

Read it honestly. The head took routing from 0.600 to 0.900 on rows it never saw, and was refused for urgency because it did not beat the engine on the same rows, so that question stayed exactly where it was. The urgency ECE fell from 0.412 to 0.059 from a temperature alone, with the argmax untouched. Routing is where 10 calibration rows are too few: at alpha 0.1 the conformal quantile on 10 rows is the largest score, so the sets are wide, and no review line can certify 0.95 precision with a 95% bound on fewer than about 50 rows; the output says so in route_reason instead of pretending. The floor of 35 rows gets you a served head; a route you can act on wants a few hundred rows, and report["calibration"]["band"] tells you when you are there.

How this differs from fine-tuning: a fine-tune runs a gradient through the model until it agrees with your labels, which changes every answer the model gives, invalidates any calibration fitted before, and leaves you a new checkpoint to validate. fit never touches the reader. It fits a small logistic head per question over the frozen features, a convex problem with one solution, keeps the head only if it beats chance under cross-validation and serves it only if it also beats the zero-shot engine on the same rows, so a fit can lose and say so, as urgency did above. A quarter of the rows is held out inside fit and calibrate refits three numbers per question on them: a temperature, a conformal threshold so the prediction set covers at 90%, and the review line, placed where precision holds at 0.95 with a 95% Wilson bound to spare. Every other question is untouched by construction, nothing is forgotten, and agent.thresholds() prints what decides route right now.

The laya checkpoints, without vegaml

Leave the argument out and laya is the ModernBERT family it always was: ~150M to 421M parameters, 33 ms a decision on a T4, 100+ languages through the Router.

from laya import Router

router = Router()  # downloads a checkpoint on first use; Router(preload=True) loads all three up front

result = router.predict(state, questions)
print(result["answers"]["department"]["choice"])  # billing
print(result["answers"]["churn_risk"]["noul"])    # probability the answer is yes
print(result["routing"]["model"])                 # english

The Router detects the script and language and sends non-English text to laya-multilingual:

for text in ["मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।",
             "La aplicación se cierra cada vez que abro la configuración."]:
    r = router.predict(text, {"department": questions["department"]})
    print(r["routing"]["model"], r["answers"]["department"]["choice"])
# multilingual billing
# multilingual technical

Long documents on the checkpoints: laya-multilingual reads up to 8,192 tokens. It ships with a 1,024-token limit that cuts long documents off, so pass max_len=8192 for them:

result = router.predict(long_document, questions, model="multilingual", max_len=8192)

In the table above, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary (8 to 17 of 20), so check long-document accuracy on your own data. Short inputs give identical answers with max_len=8192, and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU. Name the checkpoint with model="multilingual", since long mostly-English text would otherwise route to the English checkpoint. With vegaml=True none of this applies: the reader takes 73,728 tokens whole.

From the command line, laya "My payment failed twice" --preset triage answers a ready-made question set. More in the full quickstart and the docs.

Fine-tune for better accuracy

The shipped checkpoints work zero-shot, but fine-tuning on decisions from your own domain is where accuracy jumps. On the typed-decisions benchmark (2,000 decisions across four workflows), the fine-tuned laya-typed-decisions checkpoint scores 0.766 accuracy, against 0.362 for the base English checkpoint on the same decisions.

Fine-tuning notebook (Kaggle 2xT4) and Apple Silicon script (MPS / CPU): run the whole loop (build dataset, train with RLCD, calibrate temperatures, evaluate, export). Details in Fine-Tuning.

Documentation

nandhakishorm.github.io/laya: guides for prediction hooks, schema-driven decisions, Docker and LangChain and LangGraph, plus a full API reference.

What's new in 0.4.28

• Load a trained policy straight from its .zip, verified. Agent(vegaml=True).load_calibration takes the folder or the downloaded .zip itself, nothing to unpack, and returns the questions ready for predict:

  agent = laya.Agent(vegaml=True)
  info = agent.load_calibration("acme-policy.zip")
  agent.predict("the message to decide", info["questions"])
  

  Before anything is installed it checks that every file has the SHA-256 its manifest states, that a signed policy's Ed25519 signature verifies against a key laya trusts (Embedding Space's key is pinned in laya.policy.TRUSTED_KEYS), and that the loaded engine is the one the policy was trained on. Any failure raises laya.policy.PolicyError naming the file, the key or the engine. verify="signed" also refuses an unsigned policy; verify=False skips the checks. save_calibration now writes an unsigned manifest of hashes, so laya's own folders are integrity-checked on load, and folders written before 0.4.28 still load, reported as unverified. cryptography is a new dependency.

What's new in 0.4.27

• Faster CPU inference on long inputs. On transformers 5, ModernBERT's local attention layers compute their window in blocks instead of full-sequence attention masked afterwards, and the decision head packs its projections and, in its final layer, scores only the marker and pooled positions. Same weights, same answers: a 186-key real-weights comparison across the three checkpoints, short and long states, batches and predict_long matched 0.4.26 to six decimals. Measured 26 to 55 percent lower latency between 2k and 7k tokens (#1467).
• Named refusals instead of leaked errors. predict_batch and predict_long refuse a float, string or boolean batch_size with a ValueError before any state is encoded, and None, 0 and negatives still mean no cap (#1458); a wide integer in an adapter answer is refused as malformed rather than raising OverflowError (#1466); the Router's fallback for older agents composes both legacy keywords (#1468); the ONNX exporter's default opset follows the installed torch (#1451).
• laya-ts Router parity. leafNonEnglish reads each line on its own, as Python does (#1450, fixes #1449); maxLoaded is read as int() reads it, so an unreadable cap cannot disable eviction (#1457); hooksTimeout bounds the Router's own hooks (#1459); predictBatch gates on minConfidence (#1460) and groups by the resolved checkpoint (#1461).
• TypeScript HTTP SDK: predictBatch sends independent states with one question set to /v1/systemone/batch in one request (#1471).

Where to go next

Installation details

Python 3.10 or newer. The dependencies set that floor: huggingface_hub 1.x, transformers 5.x and torch 2.14 all require 3.10.

Optional PyTorch build selection: If you need a CPU-only or GPU-specific PyTorch build, follow PyTorch's installation guide after creating your virtual environment and before installing Laya. Replace pip or pip3 in the selected command with the environment's Python executable followed by -m pip.

If you already use a virtual environment, install the PyPI release with:

python -m pip install laya

For a new environment, choose the commands for your platform below. Run them from your project directory; the explicit Python paths keep installation and verification in the same environment.

macOS / Linux (with Python 3.10 or newer):

On Debian/Ubuntu, the system Python may require sudo apt install python3-venv before creating a virtual environment. If venv reports that ensurepip is unavailable, install that package and retry.

python3 -m venv .venv
.venv/bin/python -m pip install laya
.venv/bin/python -I -c "import laya; print(laya.__version__)"

Windows PowerShell (this example uses an installed Python 3.11):

py -3.11 -m venv .venv
.\.venv\Scripts\python.exe -m pip install laya
.\.venv\Scripts\python.exe -I -c "import laya; print(laya.__version__)"

Both checks print the installed Laya version without loading a checkpoint. -I excludes the current directory from the import search path, so a local source copy cannot mask a missing installation. Keep using the same virtual environment's Python when running your application.

Intel GPU (XPU)

Install a supported Intel GPU driver first. For an XPU-enabled PyTorch build, install its wheel before Laya; the default PyPI wheel may be CPU-only. PyTorch's validated hardware and OS list is in the Intel GPU guide.

.\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/xpu
.\.venv\Scripts\python.exe -m pip install laya
.\.venv\Scripts\python.exe -c "import torch; print(torch.xpu.is_available())"

For a source checkout, replace pip install laya with pip install -e .. Laya automatically selects an available XPU when no device is specified; you can also request one explicitly with device="xpu" in laya.load() or Router(device="xpu").

Install from GitHub

To use the development version instead of the PyPI release, create the virtual environment above and replace its installation command with the appropriate command below. Git must be installed.

# macOS / Linux
.venv/bin/python -m pip install "git+https://github.com/NandhaKishorM/laya.git"

# Windows PowerShell
.\.venv\Scripts\python.exe -m pip install "git+https://github.com/NandhaKishorM/laya.git"

Run the same version check afterward. The GitHub version follows the repository's default branch and may differ from the published release.

Install with uv

uv creates the virtual environment, downloads a matching Python if none is installed, and installs into it. The commands are the same on macOS, Linux and Windows PowerShell:

uv venv --python 3.12
uv pip install laya

uv pip install targets the .venv in the current directory without activating it, so run the version check for your platform above afterward. Extras and the GitHub version install the same way: uv pip install "laya[serve]", uv pip install "git+https://github.com/NandhaKishorM/laya.git". For a CPU-only or GPU-specific PyTorch build, add --torch-backend=auto to pick the build that matches the machine's GPU driver, or name one such as --torch-backend=cpu; uv's PyTorch guide has the list.

If your application is a uv project, add Laya as a dependency instead:

uv add laya
uv run python -I -c "import laya; print(laya.__version__)"

--torch-backend applies to uv pip only; in a uv project, uv's PyTorch guide shows how to set the PyTorch index in pyproject.toml.

Model setup and troubleshooting

Continue with the Router quickstart to run inference. Loading a Hub checkpoint requires access to Hugging Face on its first download; the quickstart's Router(preload=True) loads all three configured checkpoints at construction.

• ModuleNotFoundError: No module named 'laya': run both installation and your script with the same virtual environment's Python executable shown above. In an editor, select that interpreter as well.
• Missing rl_agent_config.json: this file ships with a Laya checkpoint alongside model.safetensors; it is not a configuration file you need to create in the source repository. For a local model, pass the directory containing those checkpoint files.
• Checkpoint downloads and cache: Router() is lazy: the first prediction downloads only the selected checkpoint, then reuses the Hugging Face cache. Set HF_HUB_CACHE to move that cache. The English and typed-decisions checkpoints have about 421M parameters each; multilingual has about 322M (see the table above). Download sizes also depend on weight precision and files.
• Memory limits: Router() keeps up to two checkpoints resident by default (max_loaded=2), evicting the least recently used when needed. Use Router(max_loaded=1) to reduce resident memory, at the cost of reloads when switching checkpoints. Router(preload=True) loads all three up front and raises the resident limit to fit them; it is not an out-of-memory remedy.
• Windows + Python 3.14: the model-construction crash reported in #123 was fixed in Laya 0.3.7 by #195 and verified on the reporter's Windows 11 setup without a manual patch. Upgrade older Laya installations with your environment's Python executable followed by -m pip install -U laya.

Command line

Installing the package also installs a laya command for quick local testing, no script needed:

laya "I was charged twice, please refund"            # routing decision only; works offline, no download
laya "Refactor this service" --predict               # full answers (downloads the checkpoint on first use)
laya "Mein Konto wurde zweimal belastet" --lang de   # force a language instead of detecting it
laya "My payment failed twice" --lang-guess en        # a soft hint: nudge routing, still fall through to detection
laya "My payment failed twice" --model ml            # pin a checkpoint: names, aliases and casing all resolve as the SDK resolves them
laya "My payment failed twice" --preset triage       # answer a ready-made preset (triage, email, guard, moderation, router)
laya --batch tickets.txt --predict                   # score a file of requests, one per line, in one batch
cat tickets.txt | laya --batch - --predict --json    # stdin; one JSON line of answers per request
laya "Where is my card" --questions intents.json     # answer your own questions, written in a JSON file
laya "Refund my card" --predict --min-confidence 0.9 # mark an answer the model is unsure of
laya --image invoice.png --preset triage              # a picture, a recording (--audio) or a clip (--video) on the vega lane
laya                                                 # interactive mode

--lang is decisive: a real code picks the checkpoint and skips detection. --lang-guess is the soft sibling core checks in between, so a probable-but-uncertain language can nudge routing without forcing it — and a hint that resolves to nothing falls through to the built-in detector, exactly the Router(lang_guess=...) behaviour the library offers. Neither flag changes the routing decision's precedence rules; --lang-guess just adds a hint those rules read before detection.

Routing alone never downloads a checkpoint, so it returns in milliseconds. --predict loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing. --batch (with or without --predict) sends the whole file through Router.predict_batch in one process, so the requests share checkpoint loads and forward passes — measured 2.6x on 20 tickets vs looping predict one by one, with --batch-size N to bound the forward pass, --sort-by-length to group similarly sized requests inside it, and --json for JSONL output. Batch routing (laya --batch FILE, no --predict) likewise answers with route_batch in one pass, still without loading anything.

--sort-by-length reaches the length grouping Agent.predict_batch has done since #294, which until now only the library call in Batch Mode could ask for. A forward pass pads every state in it to the longest one, so a two-line request sharing a pass with a two-paragraph one spends most of its compute on padding; grouping by length first puts comparable sizes together. Measured on 128 real Yelp reviews of 69-2,293 characters, run as laya --batch FILE --predict --model laya --batch-size 8 with and without the flag, four interleaved rounds on an Apple-silicon Mac at the CLI's own device pick: 41.25 s median unsorted against 29.23 s sorted, a paired median of 1.42x (1.39-1.44x across the four pairs), with all 128 decisions identical and still in input order. research/ measures the same knob at 2.15x over 10,000 synthetic multilingual tickets. It needs --batch-size N with 1 < N < the number of requests — a single pass over the whole file has no group to reorder — and the CLI says so on stderr rather than printing a same-speed result and leaving you to notice.

--questions takes the same question dict the SDK takes, as JSON: either the mapping itself, or {"state_key": "body", "questions": {...}} when the question's instructions name a field other than request. It implies --predict, and a question set with many labels usually wants --head-max-len with it: on 58 MASSIVE-INTENT labels written as one choice question, the English checkpoint goes from 24/58 correct at its default 192-token option budget to 34/58 at --head-max-len 384, for about 1.4x the per-request time on CPU. Widening it further costs the accuracy back, because max_len then leaves fewer tokens for the request itself. Honest limits describes the same budget ceiling for a 77-option question.

Device selection is automatic, in this order: CUDA → MPS → XPU → CPU. Mixed precision is used on CUDA; CPU and MPS run fp32. Override with device=, which is accepted by both entry points:

agent = laya.load("convaiinnovations/laya", device="cpu")      # or "cuda", "mps", "xpu"
router = Router(preload=True, device="mps")

Eager CPU fp32 inference combines packed decision-head projections and SDPA with selective outputs in the final head layer: option markers and token zero for the action head. Every unmasked token remains a head attention key/value. On Transformers 5, supported ModernBERT encoders also compute their existing local attention windows in blocks for sequences of at least 512 tokens; global layers retain full attention. No allowed attention connection or input token is removed. Training, autocast, other devices, compilation/export, custom hooks and unsupported encoders retain the corresponding original execution paths. This uses the same weights and calibration, adds no dependency or external service, and keeps request tensors local to each call without a shared request cache. Floating-point results can differ slightly, including decisions at exact ties. The existing CPU/device, routing-latency and long-context benchmarks provide the before/after evidence. The separate CPU regression harness checks unrounded outputs and public answers against the original execution path.

macOS, including Intel Macs

PyTorch stops publishing macOS x86_64 wheels at 2.2.2 (2.3 and later are Apple-silicon only), while transformers 5.x requires torch >= 2.4 and torch 2.2.2 was built against NumPy 1.x. A plain pip install laya on an Intel Mac therefore resolves a stack that cannot run — torch 2.2.2 with transformers 5.x and NumPy 2.x, and torch fails to initialise NumPy 2, which predict() needs for its final .numpy() conversion. Pin the stack:

pip install "numpy<2" "torch==2.2.2" "transformers==4.57.6" laya

transformers 4.57.x is the last 4.x line and the newest one that runs on torch 2.2.2. The transformers-5 checkpoints load on it unchanged: build_model maps their rope_parameters onto the RoPE attributes 4.x reads, which mmBERT needs because both of its bases are 160000 rather than the 4.x defaults. Apple-silicon and CUDA machines need none of this pinning.

Running from this checkout

setup_laya.sh stands the whole thing up in the checkout: an isolated .venv with the pinned stack, the three checkpoints under models/ (gitignored, ~2.3 GB), and a verification run. It is idempotent, so re-running only fills in what is missing. The numbers below are from the machine it was developed on — an Intel Mac Pro, no CUDA, AMD Radeon GPUs reachable through Metal (MPS); the script itself is not Intel-specific. Commands are relative to the repository root.

./setup_laya.sh                                    # venv + pinned deps + checkpoints + verify
.venv/bin/python verify/laya_smoke_test.py --models ./models            # real weights, device auto -> MPS
.venv/bin/python verify/laya_smoke_test.py --models ./models --device cpu
.venv/bin/python verify/numerics_check.py          # RoPE bases, determinism, SDPA vs eager
.venv/bin/python verify/bench_devices.py           # CPU vs MPS, thread scaling
.venv/bin/python verify/edge_sweep.py              # edge cases: option budgets, truncation, router lifecycle
.venv/bin/python verify/checkpoints.py             # weights vs the recorded sha256 hashes
.venv/bin/python verify/soak_check.py              # drift, RSS and concurrency over 200 calls
.venv/bin/python tests/test_local_e2e.py ./models             # the repo's own e2e suite
./examples/run_all.sh                              # all 41 examples, pass/fail

Measured here, one predict() call answering 4 questions about a short email (median of 10, after warm-up):

MPS is 1.6-3x this 56-core CPU and matches the T4 reference quoted below (32.8 ms/question for laya-multilingual). CPU-only work runs fastest around 16 threads (OMP_NUM_THREADS=16); the default 28 already over-subscribes. The first MPS call pays ~13 s of Metal kernel compilation, so warm up before timing. Individual runs move by a few percent with machine load.

To keep the router fully offline, point it at the local checkpoints rather than the hub repos:

from laya import Router

router = Router(models={"english": "models/laya",
                        "multilingual": "models/laya-multilingual",
                        "typed-decisions": "models/laya-typed-decisions"},
                preload=True)

Full notes — version rationale, the two portability fixes this needed, and every verification run — are in LOCAL_SETUP.md.

Try it locally: web GUI + JSON API

examples/server.py is a self-contained FastAPI app for testing Laya without writing any code: a two-pane playground (edit the request as a form or as JSON, run it with Ctrl+Enter, read each answer's full distribution and calibrated confidence, copy it as curl or Python), plus a plain JSON API (/predict, /predict/batch) for scripting against.

pip install "laya[serve]"
python examples/server.py               # http://127.0.0.1:8000

On Windows, start.bat does the same from a clone: it creates .venv with whichever supported Python is installed, installs CPU torch and the serve extra on first run, then starts the server and opens the browser. start.bat 8080 picks a different port.

Open http://127.0.0.1:8000 in a browser for the playground, or hit it directly:

curl -s localhost:8000/predict -H 'content-type: application/json' -d '{
  "state": {"body": "We were billed twice for March. Please refund it today."},
  "questions": {
    "department": {"type": "choice",
                   "instructions": "Which department should handle this?",
                   "criteria": {"billing": "invoices, payments, refunds", "other": "everything else"}},
    "urgency": {"type": "score",
                "instructions": "How urgent is this?",
                "criteria": ["not urgent", "soon", "critical"]}
  }
}' | python -m json.tool

--no-preload loads checkpoints lazily instead of all three up front; --device cuda|cpu|mps pins the device. See python examples/server.py --help for the rest.

A request body may carry model to pin a checkpoint instead of letting the router choose, and it takes exactly the spellings laya --model takes: names, aliases and casing all resolve through laya.router. GET /models lists the checkpoints and the aliases alongside them.

Both /predict and /predict/batch also accept the four per-call controls laya-serve forwards on /v1/systemone: lang_guess (a soft ISO hint that participates in routing, where lang skips detection entirely), max_len and head_max_len (per-call token-budget overrides, capped by LAYA_MAX_TOKEN_BUDGET), and min_confidence (core's abstention threshold; the answer comes back with abstention and abstention_threshold fields naming whether it fired). An absent key means "inherit what the Router was built with", so a caller never has to send them back as null; a body that names hooks, hooks_raise, hooks_timeout, on_predict_start, or on_predict_end is refused with 422 rather than silently dropped, matching what laya-serve does for the same fields. On /predict/batch the controls are read once, from the batch body: min_confidence becomes a call argument to Router.predict_batch, and the other three are copied into every request dict, so all states in one call share one budget. A max_len written inside a state is not a control there -- it stays part of the state, and the body's value is what every state gets. Per-request budgets do exist one level down, in Router.predict_batch, which cannot share a forward pass between requests naming different budgets and so makes one pass per budget; the batch endpoint just gives an HTTP caller no way to name them separately.

/predict/batch accepts two optional body fields that control the shape of the forward passes without changing any answer: batch_size (states per pass; omit it and the whole batch is one pass) and sort_by_length (group similarly sized states so each pass pads to a shorter maximum — it needs a batch_size below the number of states, since one pass has nothing to reorder). Measured on the running server: 64 real reviews of 91–2,293 characters, one choice question, batch_size: 8, three interleaved rounds on MPS — the default one-pass shape 5.35s median against 3.04s sorted, a paired median of 1.77x (min 1.76x), and 64/64 labels identical in input order. batch_size: 8 on its own was 4.95s, so the length grouping is what earns the second factor: 1.66x over the sized batch.

Python HTTP client

Applications that call a separate laya-serve can use the Python HTTP client without installing the inference dependencies. It provides synchronous predict() and health() calls using the standard library. From this checkout, install it with python -m pip install ./sdk/python in the application environment; keep laya[serve] in the server environment. The separate laya-client distribution is not yet published to PyPI.

JavaScript / TypeScript

laya-client supports JavaScript and TypeScript applications over HTTP to a self-hosted laya-serve /v1/systemone server. It includes inferred answer types, all five question presets, ESM/CommonJS exports, and cancellation.

# From this checkout, install and start the Python inference service:
pip install -e '.[serve]'
LAYA_HOST=127.0.0.1 LAYA_MODELS=english laya-serve

# In another terminal:
cd sdk/typescript
npm ci
npm run build
node examples/triage.mjs

The npm package is named laya-client. Use it when a JavaScript or TypeScript application talks over HTTP to self-hosted Python laya-serve. Use laya-ts when inference must run directly inside JavaScript through its local ONNX runtime, without a Python server. See installation, examples, and publishing and the repository analysis for architecture and scope.

Quickstart: Route Mode (Recommended)

To try the Python SDK in a CPU container, see the Docker Compose quickstart. It runs a sample request and keeps downloaded models between runs.

Laya ships three checkpoints. The built-in Router is the recommended entry point: it evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass.

Since 0.4.25, with vegaml installed: the same Router also takes image=, audio= or video= in place of a state and sends them, states past 8,192 tokens, and questions with fitted heads to laya.Agent(vegaml=True); everything else routes exactly as below. Router(vega=False) turns that off. See the vega lane.

Minimal example (30 seconds)

from laya import Router

router = Router()  # downloads only the selected checkpoint on first use
result = router.predict(
    {"body": "We were billed twice. Please refund the duplicate."},
    {"billing": {"type": "noul", "instructions": "Does the user request a refund?"}},
)
print(result["answers"]["billing"]["noul"])  # P(true), a float from 0.0 to 1.0
print(result["routing"]["model"])            # which checkpoint answered, e.g. 'english'

Input is any state (str, dict, or list — text, email, ticket, JSON document) plus a dict of typed questions (choice = pick one label, score = ordinal levels, noul = yes/no statement check). Each result["answers"][qid] is an answer dict: choice is the selected label, score is the expected ordinal level, and noul is P(true). confidence is a separate field in that dict; probabilities is available for choice and score. result["routing"] explains which checkpoint was picked and why. Values depend on the checkpoint and input. For ready-made question sets see laya.presets (triage_questions, moderation_questions, email_questions, ...).

Full walkthrough:

from laya import Router

# Preload checkpoints into memory for instant sub-35ms routing
router = Router(preload=True)

# 1. State in any language or schema
state = {
    "from": "[email protected]",
    "subject": "Duplicate charge on invoice #4411",
    "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."
}

# 2. Define your typed questions
questions = {
    "department": {
        "type": "choice",
        "instructions": "Which department should handle this request?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, system errors",
            "sales": "pricing, new contracts, plan upgrades",
            "other": "everything else"
        }
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this request?",
        "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]
    },
    "churn_risk": {
        "type": "noul",
        "instructions": "Does the user threaten to cancel or leave?"
    },
    "refund_requested": {
        "type": "noul",
        "instructions": "Does the user explicitly request a refund?"
    }
}

# 3. English state -> automatically routed to laya (ModernBERT-large, 39.5 ms)
res_en = router.predict(state, questions)
print("Department :", res_en["answers"]["department"]["choice"])  # -> billing (confidence: 0.94)
print("Routing    :", res_en["routing"]["model"])                 # -> english

# 4. Hindi state -> automatically routed to laya-multilingual (mmBERT-base, 32.8 ms)
res_hi = router.predict({"body": "मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें।"}, questions)
print("Department :", res_hi["answers"]["department"]["choice"])  # -> billing (confidence: 0.86)
print("Routing    :", res_hi["routing"]["model"])                 # -> multilingual

# 5. Explicit override when you want a specific checkpoint
res_td = router.predict(state, questions, model="typed-decisions")

Every result carries full routing metadata explaining why the choice was made:

res_hi["routing"]
# {
#   'model': 'multilingual',
#   'repo': 'convaiinnovations/laya/multilingual',
#   'reason': 'non-Latin script (devanagari, 100% of letters); the English checkpoint cannot read it',
#   'detection': {'script': 'devanagari', 'script_profile': {'devanagari': 1.0},
#                 'language': None, 'is_english': False, 'language_undecided': True,
#                 'diacritic_rate': 0.0, 'non_latin_fraction': 1.0, 'mixed_segment': None},
#   'workflow': None
# }

All five keys are always present. detection is the laya.lang.analyse result the decision was made from, and is None whenever routing decided before detection ran: model=, task=, a detected workflow, a decisive lang= and a resolving lang_guess all take precedence over it and short-circuit it. workflow names the typed-decisions workflow whose question ids matched, and is None when none did.

Inspect a routing decision without running any forward pass:

router.route({"body": "Der Kunde wurde zweimal belastet"}, questions).reason
# "Latin script but language looks like 'de', not English"

Very short Latin-script text often carries nothing that identifies its language ("Quero cancelar", "Esqueci minha senha"). Such text goes to default, which is "multilingual" since 0.4.0. Text the detector can place is unaffected: identified English still goes to the English checkpoint.

router = Router()                                                   # default="multilingual"
router.route({"body": "Esqueci minha senha"}).model                 # -> multilingual
router.route({"body": "Please refund the duplicate charge"}).model  # -> english

If most of your traffic is English, set it back. The English checkpoint scores 0.820 on English against the multilingual checkpoint's 0.710, and the break-even is around 62% English traffic:

router = Router(default="english")

Running one of the shipped servers rather than your own Router, the same setting is LAYA_DEFAULT_MODEL=english — in the environment of laya-serve, the MCP server or the laya-serve container (docs/docker.md). It is the routing fallback only: text the detector can place is routed on what it detects, whatever this is set to.

Heterogeneous routed batches

If a lazy router receives an interleaved workload whose requests route to different checkpoints, calling predict() in a loop can still cause unnecessary checkpoint churn when the required checkpoints exceed the resident cache, for example with max_loaded=1 or when typed-decisions is also used.

Router.predict_batch() routes the full workload first, groups requests by checkpoint, then groups requests with the same question schema within each checkpoint. Each compatible group is dispatched to Agent.predict_batch() so states can share forward passes, and results are restored to the original request order.

requests = [
    {"state": "Please refund invoice 1", "questions": questions},
    {"state": "تم خصم المبلغ مرتين", "questions": questions},
    {"state": "Please refund invoice 2", "questions": questions},
]

results = Router(max_loaded=1).predict_batch(requests)
# results stay in input order while compatible requests are batched by checkpoint

Each item can independently set model, task, lang, lang_guess, or the token budget (max_len, head_max_len). Use route_batch(requests) when you only want the ordered routing decisions without loading any checkpoint. predict_many is an alias for predict_batch.

Requests are validated before model loading. Different requests may use different question schemas; requests sharing a checkpoint, question schema and token budget are passed together to Agent.predict_batch().

• Per-call hooks reach the batch path. Router.predict_batch takes hooks, on_predict_start, on_predict_end, hooks_raise and hooks_timeout, matching predict; Router.route_batch takes hooks, hooks_raise and hooks_timeout, since it fires only on_route and has no predict events for the convenience callables to bind to (#909). The batched dispatch also composes installed hooks properly, so a set_default_hooks default no longer fires on predict and silently vanishes on predict_batch.

Prediction hooks installed on the Router run once per request, as they do for predict(), so a redaction hook rewrites every state before the model sees it. Requests that share a checkpoint run all their start hooks before their shared forward pass; see docs/hooks/lifecycle.md.

You can also bound the Agent-level forward-pass batch size, and forward the length grouping knob to every group (see Batch Mode):

results = router.predict_batch(requests, batch_size=8, sort_by_length=True)

Why Route: The Evidence

On a shared benchmark (17,416 questions, one T4 GPU, identical questions per model — seed 13, 20 options per choice question, the first 300 test rows per language, from research/results/t4_colab_benchmark.json; tests/test_doc_tables.py checks every cell of this table against that file):

The English checkpoint collapses on non-Latin scripts (Khmer scores 0.000 accuracy at 0.952 confidence, the raw-temperature figure; 0.705 as served after the clamp). Because the model stays confident while being wrong, confidence gating cannot save you. Router detects the script in <0.5 ms pure Python before the forward pass.

Production Preload & Memory

A cold checkpoint build costs seconds; language detection costs microseconds. The lazy default keeps two checkpoints resident — english and multilingual, the only two automatic routing chooses between — so a language flip costs detection only once each has been built. max_loaded=1 rebuilds the checkpoint it just evicted on every switch (measured at a 7.4 s median reload on CPU and 10.3 s on T4), and traffic that only ever sees one language never builds the second, so the default costs a single-language deployment nothing.

For a server or production app, preload:

# Every checkpoint resident in memory; language flips cost detection only (<1 ms)
router = Router(preload=True)
router = Router(preload=True, device="cuda")     # or "mps" (Apple/AMD GPU), "cpu" to force

# Or preload only the specific checkpoints you serve:
router.preload(["english", "multilingual"])

# If your app already built an agent, attach it to avoid duplicate VRAM:
router.attach("english", existing_agent)

# Manage resident memory (default keeps two hot: english + multilingual, LRU eviction)
router = Router(max_loaded=3)       # keep all three hot, e.g. with auto_task_detection
router = Router(max_loaded=1)       # memory-constrained host, reloads on every switch
router.unload()                     # free memory

A rebuild still re-reads the checkpoint, but each checkpoint's tokenizer is parsed once per process and reused by every Agent — including one the Router rebuilds after eviction. The multilingual tokenizer.json alone is 34 MB / 256k vocab, several times the cost of applying its weights. Preloading is still the right answer for a server: it removes the rebuild rather than making it cheaper.

Supplying Your Own Language Detection

Routing asks one question: can the English checkpoint read this state? The built-in detector answers it from the script and a function-word heuristic, and is deliberately dependency-free. That heuristic is best-effort on Latin-script languages it holds no word list for, so a short request can carry no usable signal:

from laya.lang import analyse
analyse("Care este ora in Tokyo?")
# {'script': 'latin', 'script_profile': {'latin': 1.0}, 'language': 'en', 'is_english': True,
#  'language_undecided': False, 'diacritic_rate': 0.0, 'non_latin_fraction': 0.0,
#  'mixed_segment': None}                                     -> the English checkpoint

is_english is the key routing reads. language_undecided separates "read as English" from "carried no usable signal, so the default applies", and mixed_segment names the line or field that made a mostly-English state non-English (None when none did).

If you already run a language-identification model, hand routing the answer instead of relying on the heuristic. lang_guess takes a language code or a callable receiving the state, and is checked after an explicit lang= and before detection:

# A code you already know
router.predict(state, questions, lang_guess="ro")

# A callable, e.g. wrapping fastText, CLD3 or a transformer LID
router.predict(state, questions, lang_guess=lambda s: my_lid(s))

# Or install one for every request on a server
router = Router(preload=True, lang_guess=my_lid)

The hint only decides English or not: a code whose primary subtag is en, eng or english routes to the English checkpoint, and every other code that names a language routes to the multilingual one. "en_US" and "en_US.UTF-8" are read as English, so $LANG can be passed straight through. Returning None, or a code that names no language, makes it abstain and the built-in detector decides as before — so a LID model that is unsure does not force a checkpoint. C, POSIX and C.UTF-8 abstain, which matters because C.UTF-8 is the default $LANG in the official Python image: passing it through no longer pins every request to the multilingual checkpoint, which is what it used to do. The ISO 639-2 special codes und, zxx and mul abstain for the same reason. An explicit model=, task= or lang= still wins, and the default path is unchanged.

Self-Hosting: HTTP Server (Jev-compatible)

laya.serve exposes the Router over HTTP on the same POST /v1/systemone wire protocol as TypeSafe's hosted Jev API. Laya's answer payload is already schema-identical to what Jev returns (choice/score/noul answers and a {input_tokens, output_tokens} usage block), so an existing Jev client — e.g. the hs-jev Haskell client — just needs its baseUrl repointed; nothing else changes. A PHP client that targets laya-serve rather than Jev also exists: marcreichel/laya-php.

pip install "laya[serve]"          # adds fastapi + uvicorn + python-multipart
LAYA_DEVICE=cuda LAYA_PRELOAD=1 laya-serve   # binds 0.0.0.0:8000, preloads all 3 checkpoints

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"body": "billed twice, refund please or we cancel"},
  "questions": {"dept": {"type": "choice", "instructions": "which team?",
                "criteria": {"billing": "refunds", "tech": "bugs"}}}
}'

Questions in that request share the one state. Each is answered against the whole serialized state, so an instruction that names a record (rows[i]) still sees every record in it. One answer per record means one state per record, sent as states to POST /v1/systemone/batch. A state that does not fit the token room the questions leave is truncated before inference, and the server logs the dropped state tokens. The full response reports the same cut on usage (truncated, state_tokens_dropped, truncated_questions); LAYA_JEV_STRICT=1 omits those fields, and the log line is the signal that remains.

To evaluate multiple states in a single call, send a states array to POST /v1/systemone/batch (capped at 64 states; a batch larger than LAYA_MAX_BATCH_TOKENS is split across several forward passes rather than refused):

curl -s localhost:8000/v1/systemone/batch -H 'content-type: application/json' -d '{
  "states": [
    {"body": "billed twice, refund please"},
    {"body": "cannot login, getting 500 error"}
  ],
  "questions": {"dept": {"type": "choice", "instructions": "which team?",
                "criteria": {"billing": "refunds", "tech": "bugs"}}}
}'

The endpoint shares forward passes via Router.predict_batch and returns an array of results in matching order along with aggregated total_usage. Each entry of states is its own context, which is the per-record call: a question on /v1/systemone sees the whole shared state, even when its text names rows[i]. When any result dropped state tokens, the server logs one warning for the batch — which results were cut, and how many tokens each lost — rather than one line per answer.

Configuration is by environment variable: LAYA_HOST, LAYA_PORT, LAYA_DEVICE, LAYA_PRELOAD, LAYA_MODELS (comma list to preload), LAYA_THREADS (cap torch intra-op threads for CPU inference — keep at or below physical cores), LAYA_AUTO_TASK, LAYA_DEFAULT_MODEL (the checkpoint a state with no language evidence falls back to, multilingual by default; set it to english when most of your traffic is English), LAYA_MAX_LOADED (checkpoints resident at once, 2 by default; raise it to 3 when LAYA_AUTO_TASK makes a third one reachable on demand, or the server rebuilds one every time routing switches), LAYA_IDLE_UNLOAD_SECONDS (unload idle checkpoints; 0 disables it, 300 frees device memory after five minutes and makes the next request pay a cold load), LAYA_EXTRA_MODELS (a JSON object {"my-checkpoint": "/path/to/it"} registering extra checkpoints — Hub repo ids, local directories or ["repo", "subfolder"] pairs — beside the bundled ones), and LAYA_API_KEY (when set, clients must send Authorization: Bearer <key>). A client's model field is honoured when it names a Laya checkpoint (english/multilingual/typed-decisions, a LAYA_EXTRA_MODELS name, or a published Hub id). jev-1 and convaiinnovations/laya still mean the router auto-selects by script/language. A path or an unpublished Hub id (/path/to/checkpoint, org/repo) is a 422 on both /v1/systemone and /v1/systemone/batch instead of being answered by another checkpoint. The same body can also carry task, lang, lang_guess, min_confidence, max_len and head_max_len — the controls Router.predict takes that a JSON body can state — while the five hook arguments (hooks, on_predict_start, on_predict_end, hooks_raise, hooks_timeout) are refused with 422, because a hook is a callable that has to live where the server runs.

When publishing the server under a reverse-proxy prefix such as /laya, set LAYA_ROOT_PATH=/laya. The proxy should strip that prefix before forwarding; the setting controls FastAPI-generated URLs and leaves the internal routes unchanged.

Four things differ from Jev when you port a client:

• One shared context. One /v1/systemone request is one state, and every question is decided on that whole state. Several records packed into it, with questions aimed at rows[i], still share that context. Per-record answers are states on /v1/systemone/batch, one state per record. A state larger than the token room is truncated, and laya-serve logs the dropped tokens; under LAYA_JEV_STRICT that log is where the cut stays visible.
• Options per question. A question's options share the checkpoint's option budget, head_max_len (192 tokens on laya, 256 on the other two), not Jev's cap of 255 options. In addition, the HTTP server (laya.serve) enforces an amplification guard of at most 100 choice options per question (MAX_CHOICE_OPTIONS = 100, rejected with 413 before inference). Once options overflow the token budget, around 20 options with a short description each, every option is trimmed to fit, so long or similar labels can reach the model reading the same (Where Jev leads). Once they no longer fit the window at all, the library rejects the request with 422. For more candidates, narrow them first with predict_shortlist (Honest limits).
• Score levels. Every level needs a description. A null level is rejected with 422 rather than scored and echoed back in legend.
• confidence on choice and score answers is 1 minus normalised entropy, a measure of how concentrated the distribution is, not Jev's (n·p_max − 1)/(n − 1). A threshold carried over from Jev does not transfer. For one calibrated number on every question type, gate on answer_confidence, the probability of the reported answer. Both values shift with the number of options, so fit and validate any cutoff on held-out data at the option counts your workload uses (#394).

Nix / NixOS

This repo is a flake. On a machine with an NVIDIA GPU:

nix run .#laya-serve          # build (prebuilt CUDA torch, no compile) and serve
nix develop                   # dev shell: torch-bin, transformers, fastapi, pytest

For a NixOS host, import the module and enable the service:

# flake inputs:  laya.url = "github:<you>/laya";  # or path:/… on the same host
{
  imports = [ laya.nixosModules.default ];
  services.laya-serve = {
    enable = true;
    host = "0.0.0.0";           # or bind to the Tailscale/LAN address
    openFirewall = true;
    device = "cuda";
    models = [ "english" "multilingual" "typed-decisions" ];
    # apiKeyFile = config.age.secrets.laya-api-key.path;  # optional bearer auth
  };
}

The module runs a hardened DynamicUser systemd unit with CUDA device access, caches weights under /var/lib/laya-serve, and reads the bearer token (if any) via LoadCredential so it never enters the store.

Single-Model Mode (Direct SDK)

If you only need a single checkpoint for a dedicated pipeline, you can load models directly:

import laya

# 1. Load a specific checkpoint directly from the hub
agent = laya.load("convaiinnovations/laya")                           # English root
agent_ml = laya.load("convaiinnovations/laya", subfolder="multilingual") # 100+ languages
agent_td = laya.load("convaiinnovations/laya", subfolder="typed-decisions")

# 2. Run all questions in ONE single forward pass (~35 ms on GPU)
result = agent.predict(state, questions)
answers = result["answers"]

print("Department :", answers["department"]["choice"])   # -> billing (confidence: 0.94)
print("Urgency    :", answers["urgency"]["score"])        # -> 1.84 / 2.0
print("Churn Risk :", answers["churn_risk"]["noul"])       # -> 0.892 (89.2% probability)

Passing an empty question dictionary to agent.predict(state, {}) or agent.system_one(state, {}) returns the standard response with "answers": {} and "usage": {"input_tokens": 0, "output_tokens": 0}. The state is not tokenized and no model forward pass runs.

Batch Mode: score many states in one forward pass

predict handles one state per call, which leaves most of the GPU's batch dimension idle. When you have a list of items to score against the same questions — a backlog of tickets, a table of rows, a log slice — predict_batch packs them into shared forward passes:

states = [{"body": t} for t in ticket_texts]           # a list of states

results = agent.predict_batch(states, questions)       # one forward pass for the whole list
# results[i] corresponds to states[i], with the same output shape as predict

# Bound peak memory when the list (or the texts) are large — chunk into passes of N:
results = agent.predict_batch(states, questions, batch_size=64)

# Reduce padding when input lengths vary; results still follow the original state order:
results = agent.predict_batch(states, questions, batch_size=64, sort_by_length=True)

sort_by_length=True groups states by their longest encoded question row, after truncation. It looks ahead at most eight batches and reuses the encoded rows for sorting. This uses more temporary CPU memory for tokenized inputs, and takes effect only when 1 < batch_size < len(states). Benchmark it on your workload and backend: uniform lengths offer little benefit, and changed batch shapes can cause small floating-point differences, including near decision thresholds. Hooks still see states and final results in input order. The option is available on Agent and ONNXAgent.

Results are aligned with states by index and identical in shape to predict. Changing batch shapes can introduce floating-point differences on CPU and GPU; check decision thresholds on your workload, particularly with mixed precision. Batching is a GPU throughput win — on an RTX 5060 Ti, per-decision latency drops from ~10 ms one-by-one to ~1 ms batched (measured ~9–10×). On CPU, increasing batch size alone may not speed up inference; length grouping can help by reducing the padded work in a mixed-length workload. See the CPU measurements and reproduction commands.

ONNXAgent.predict_batch(states, questions, batch_size=..., sort_by_length=...) has the same contract, backed by one ONNX Runtime session run per chunk, so an ONNX deployment gets the same batch API, the same result shape, and the same length grouping. Measured on the English checkpoint (flat fp32 export, Apple M1, 80 support tickets alternating short and ~8x-longer documents, batch_size=4, best of 3): wall clock 53.7 s unsorted → 36.3 s sorted (~1.48x), with 0/80 decision changes and max probability drift 0.0 across queue/noul/score.

Long documents: predict_long

predict/system_one truncate a state that exceeds max_len to a single window (the first, or for a conversation list the last), silently dropping the rest. predict_long scans the whole state in overlapping windows, scores them in shared forward passes (via predict_batch), and aggregates per question:

result = agent.predict_long(state, questions)              # windows the state, one result back
result = agent.predict_long(state, questions, window=256)  # smaller window isolates a localized span
result = agent.predict_long(state, questions, hooks=[AuditLog()])   # the scan, instrumented

• noul takes the strongest window (the statement holds if any window supports it).
• choice / score take the most-confident window — averaging over a long, mostly-neutral document lets the neutral majority out-vote the one window that saw the deciding span.
• A state that already fits one window is passed straight to system_one (identical output, plus usage["windows"] = 1). Without an explicit window, that is any state the questions leave room for, since system_one reads it whole. The key is total: 1 single window, N scanned windows, 0 a hook answered the document before the model read any of it.
• Hooks wrap the inference that answers the state, so on a scanned document on_predict_start fires once with ctx.states holding the decoded windows, not the state you passed in (it was tokenized to produce them). A start hook may replace that list: the answers are aggregated over whatever reached inference, and usage["windows"] counts those states. What a rewritten scan costs is the attribution — answer["window"] names a span of your document, so it is only reported when the scan reached inference unchanged. A hook that means to answer the document calls ctx.skip(...) instead: its result comes back with no answer["window"] and usage["windows"] at 0, because no window scored it. End-hook result reordering keeps an unchanged answer's original span; cloned, replaced or edited answers omit attribution. Reliability annotations are retained.

A smaller window isolates a short deciding span better (it becomes a larger fraction of its window); the default (max(64, max_len - head_max_len - 8) — 312 tokens on the English checkpoint at its 512/192 defaults, 760 on the other two at 1,024/256) favors context and throughput. Either way the window is capped at the room the questions leave for the state inside max_len, so a window is never re-truncated on the way to the model and token_start/token_end describe the span it read: many options leave little room (on the English checkpoint, 2 options leave 483 state tokens and 100 leave 100), and a window past that room used to be cut short silently. Output shape matches predict, with usage["windows"] added.

ONNXAgent.predict_long(state, questions, window=..., stride=..., batch_size=...) has the same contract and the same aggregation rules, with the windows scored through ONNXAgent.predict_batch — one ONNX Runtime session run for all of them, or one per chunk when batch_size bounds memory.

The returned probability is the deciding window's, not a calibrated number for the whole document — a noul max drifts up with the window count even with no signal, and choice can land on a confidently-neutral window when nothing is decisive. Each answer carries answer["window"] (the deciding window's index, token_start/token_end, and count) so you can check the span the answer actually came from:

r = agent.predict_long(state, questions)
r["answers"]["refund"]["window"]   # {'index': 13, 'token_start': 4680, 'token_end': 5432, 'count': 14}

The same scan is reachable from the Router, which routes first and then windows the checkpoint it picked — the same model=/task=/lang= hints, hooks and routing key as predict:

result = router.predict_long(state, questions, model="multilingual")

GPU Fast Path (TileLang)

pip install laya[fast] adds an optional forward built from fused TileLang kernels: GEMM + bias/activation epilogues, GEMM + GEGLU, residual + LayerNorm, in-place RoPE, and a sliding-window flash attention that reads the packed QKV buffer directly. Weights stay resident in bf16 and every (batch, length) bucket is captured as a CUDA graph, so a one-question call no longer pays ~200 kernel launches from Python.

agent = laya.load("convaiinnovations/laya", fast=True)   # or: agent.accelerate()
agent.predict(state, questions)                            # same API, same answers

Numerics: on a fixed set of 60 states the fast path stays within 0.046 of an fp32 forward and within 0.076 of the stock bf16 path (max |Δp| ≤ 0.05 vs fp32 on both checkpoints, argmax agreement ≥ 47/48 per question type; every per-option probability is in benchmarks/results/parity_*.json) — see benchmarks/parity_fast.py and BENCHMARKS.md. The fast path runs in the agent's autocast dtype at the time accelerate() is called: bf16 by default, fp16 if agent.dtype is torch.float16, where it stays within 0.009 of fp32 on the same set (BENCHMARKS.md). Falls back to the stock forward on CPU/MPS or when tilelang is not installed; agent.deaccelerate() restores it. Kernels compile once per shape bucket on first use (a few seconds, cached on disk).

Warm-up before serving: agent.warmup()

compile=True now runs agent.warmup() before returning from load. This moves the batch and single-row graph compiles into startup; startup can take tens of seconds. To defer the work:

agent = laya.load("convaiinnovations/laya", compile=True, compile_warmup=False)
seconds = agent.warmup()         # manual entry point, also accepts shapes=[(rows, tokens, markers)]

If automatic warm-up fails, load emits a RuntimeWarning naming the underlying error and returns with the torch.compile wrapper and compile settings intact. Later requests still use the compiled model and propagate compilation failures; there is no automatic eager fallback. Explicit agent.warmup() calls also propagate failures, even after automatic warm-up has failed.

Warm-up uses synthetic inputs without prediction hooks or prediction caches. It covers common batch and single-row shapes, not every possible guard or CUDA graph shape. compile=False and fast=True do not warm automatically. Manual warm-up still works on both paths. On CUDA, compiled inference pads the masked end of each sequence to a multiple of eight tokens. This avoids extra SDPA graph specialisations on PyTorch versions that distinguish lengths modulo eight; reported token usage still counts the original, unpadded sequence. Inductor caches compiled graphs under TORCHINDUCTOR_CACHE_DIR (by default in /tmp); point it at a persistent directory to keep them across restarts. Alternatively, load with compile=True, compile_cache=True to use $XDG_CACHE_HOME/laya/torchinductor (or ~/.cache/laya/torchinductor). This opt-in sets the process-wide environment variable only if absent; an existing value always wins. It does nothing on eager or TileLang loads.

compile_mode="reduce-overhead" is a separate opt-in for CUDA graphs. With compile=True, it can reduce launch overhead on repeated shapes, but each new shape needs recording and GPU memory can grow. Laya copies outputs out of reusable graph buffers and serializes these CUDA forwards. The default mode remains unchanged. See the engineering notes for measurements and limitations.

vega inside laya: Agent(vegaml=True)

One argument swaps the engine under laya's surface for vegaml's: a frozen 0.8B universal reader with a small physics engine on top. Every method keeps its laya name; the answers gain what the engine knows that an encoder does not.

import laya

agent = laya.Agent(vegaml=True)                  # vegaml ships with laya 0.4.23 and newer
out = agent.predict