AssetOpsBench
AI Agents for Industrial Asset Operations & Maintenance
A unified, open framework for building, orchestrating, and evaluating domain-specific AI agents in Industry 4.0.
📄 Paper · 🤗 Dataset · 🎮 Playground · 📢 IBM Blog · 🎥 Video · 📊 Kaggle · 🚀 Colab
[!IMPORTANT] 🎉 AssetOpsBench is officially accepted at KDD 2026 (Datasets & Benchmarks Track), Jeju, South Korea, alongside our hands-on tutorial Building Reliable Industrial Agents with MCP. See Publications for the full list of 2025–2026 work.
At a Glance
Built for: maintenance engineers, reliability specialists, facility planners, and Industry 4.0 researchers. Powered by: LLMs + Time Series Foundation Models, orchestrated over live sensor data and Industry 4.0 records (FMEA, work orders, alerts). Now with: simplified interface and native MCP (Model Context Protocol) support.
Quick Start
# Clone and install
git clone https://github.com/IBM/AssetOpsBench.git
cd AssetOpsBench
pip install -e .
# Try a scenario (to be enabled)
python -m assetopsbench.run --scenario "List all sensors of Chiller 6 in MAIN site"
Or jump in instantly:
- 🚀 Run on Colab — no install required (illustration of LLM Agent)
- 🎮 Try the HF Playground — interactive demo
- 📖 Read INSTRUCTIONS.md — full setup, MCP servers, plan-execute runner
[!NOTE] Active development is on
main. The codebase used for various publication venues continues to be maintained on separate branches, for example, ACL 2026IndustryAssetEQAand prior experimental work is maintained onmain-0.x.
What is AssetOpsBench?
AssetOpsBench is a unified framework for developing, orchestrating, and evaluating domain-specific AI agents in industrial asset operations and maintenance. It provides reproducible scenarios, agent tooling, and evaluation pipelines for multi-step workflows in simulated industrial environments.
Domain-Specific MCP Servers
Five domain servers (IoT, FMSR, TSFM, WO, Vibration) plus one shared utility server, documented in full — arguments, categories, and backing services — in docs/mcp-servers.md.
The full TSFM MCP surface currently contains 41 tools covering model cards, feature cards, recipe execution, evaluation, and result/run lookup. See docs/mcp-servers.md for the complete reference.
The WO MCP surface contains 15 tools (9 read + 6 write) covering the full Maximo-style work-order lifecycle — query, create, approve, assign, close, cancel — plus costs, KPIs, and scheduling, backed by CouchDB with IBM Maximo mxwo field names. Set AOB_READONLY=1 to expose the 9 read tools only. See docs/mcp-servers.md for the complete reference.
Agent Frameworks
- Plan Execute — plan-and-execute sequential workflow to work with any LLM
- Deep Agent — planning, sub-agents, and virtual filesystem for long-horizon tasks
- Claude Agent — ReAct-based orchestrator using Claude with agent-as-tool delegation
- OpenAI Agent — ReAct-based orchestrator using OpenAI models with agent-as-tool delegation
MCP Environment
The src/ directory contains MCP servers and a plan-execute runner built on the Model Context Protocol. See INSTRUCTIONS.md for setup.
Example Scenarios
Some tasks focus on a single domain, others are multi-step end-to-end workflows. Explore all scenarios on Hugging Face.
Leaderboards
- To be revised (WIP with latest models)
- Evaluated with 7 Large Language Models
- Trajectories scored using LLM Judge (Llama-4-Maverick-17B)
- 6-dimensional criteria measuring reasoning, execution, and data handling
Example: MetaAgent leaderboard
Publications
12+ contributions across 7 top venues in 2025–2026 from the team behind AssetOpsBench.
- [D&B] AssetOpsBench: A Benchmark for Industrial Asset Operations Agents · D. Patel, S. Lin, et al. · 📄 Paper
- [Tutorial] Building Reliable Industrial Agents with MCP: A Hands-on AssetOpsBench Tutorial for AI-Driven Operations · D. Patel, C. Shyalika, et al.
- [Industry] IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenance · C. Shyalika, D. Patel, A. Sheth · arXiv:2604.23446
- [Main] Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring · N. Martinez, F. O'Donncha, W. M. Gifford, N. Zhou, D. C. Patel, R. Vaculin
- [Demo] AssetOpsBench-Live: Privacy-Aware Online Evaluation of Multi-Agent Performance in Industrial Operations · D. Patel, N. Zhou, S. Lin, J. T. Rayfield, C. Shyalika, S. R. Yarrabothula · 🎥 Demo
- [Main] SPIRAL: Symbolic LLM Planning via Grounded and Reflective Search · Y. Zhang, G. Ganapavarapu, S. Jayaraman, B. Agrawal, D. Patel, A. Fokoue · 💻 Code
- [Bridge] Knowledge-Guided AI for Industrial Asset Health Monitoring · S. Lin, D. Patel
- [Tutorial] From Inception to Productization: Hands-on Lab for the Lifecycle of Multimodal Agentic AI in Industry 4.0 · C. Shyalika, S. Ahuja, S. Lin, R. Wickramarachchi, D. Patel, A. Sheth · 🌐 Website · 📊 Slides
- [Workshop(AABA4ET)] Agentic Code Generation for Heuristic Rules in Equipment Monitoring · F. Lorenzi, A. Langbridge, F. O'Donncha, J. Rayfield, B. Eck, S. Rosato
- [Deployed] Deployed AI Agents for Industrial Asset Management: CodeReAct Framework for Event Analysis and Work Order Automation · N. Zhou, D. Patel, A. Bhattacharyya
- [Emmerging] Diversity Meets Relevancy: Multi-Agent Knowledge Probing for Industry 4.0 Applications · C. Constantinides, D. Patel, S. Kimbleton, N. Garg, M. Paracha
- [D&B Track] FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes · C. Constantinides, D. Patel, S. Lin, C. Guerrero, S. D. Patil, J. Kalagnanam · 📄 arXiv · 💻 Code
- [Social] Building Reliable Agentic Benchmarks: Insights from AssetOpsBench (invited talk, 2000+ registered) · D. Patel · 📅 Luma
- [Main] ReAct Meets Industrial IoT: Language Agents for Data Access · J. T. Rayfield, S. Lin, N. Zhou, D. C. Patel
- [Main] Generalized Embedding Models for Industry 4.0 Applications · C. Constantinides, S. Lin, D. C. Patel · 📄 arXiv
- [Findings] Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health Monitoring · S. Lin, D. Patel, C. Constantinides · 📄 ACL Anthology · 💻 Code
Tutorials & Technical Material
📘 Hands-on guides from our team:
- ReActXen IoT Agent (EMNLP 2025)
- FailureSensorIQ (NeurIPS 2025)
- AssetOpsBench Lab (AAAI 2026)
- SPIRAL (AAAI 2026)
- AssetOpsBench Technical Material
AI Competitions
AssetOpsBench powers public AI agent competitions that bring together researchers, students, and practitioners worldwide.
🔴 Live — IJCAI 2026
Industrial Automation Challenge: Benchmarking Physics-Grounded LLMs for Task Reasoning
A new challenge co-located with IJCAI 2026 that pushes LLM agents on physics-grounded industrial reasoning.
- 🌐 Challenge site: ai-industrial-challenge-ijcai
- 📋 IJCAI 2026 competitions: 2026.ijcai.org/competitions
✅ Completed — CODS 2025
AssetOpsBench-Live: AI Agentic Challenge
Launched in September 2025 at CODS 2025, the competition evaluated multi-agent systems on live industrial scenarios.
- 🏆 Competition page: codabench.org/competitions/10206
- 👥 365 participants · 500+ agent submissions
Talks & Events
University Projects & Extensions
AssetOpsBench is being extended by university research groups exploring new asset classes, evaluation paradigms, and agentic architectures. To list your project, open a PR.
- Calibrated Coordination Reduces Overconfident Errors in Multi-Agent LLM Systems – Confidence-weighted aggregation and abstention framework for reducing hallucinated confidence events in multi-agent industrial troubleshooting and operational decision-making benchmarks. Chand Sahil Mansuri, Sadamori Kojaku, Binghamton University.
- Internalizing MCP Tool Knowledge in Small LLMs via QLoRA Fine-Tuning — HPML project using AssetOpsBench to fine-tune ~4B models to internalize MCP tool knowledge and reduce prompt schema overhead. Ayal Yakobe, Columbia University · repo
- SPIN — Structural LLM Planning via Iterative Navigation for Industrial Tasks. Yusuke Ozaki, University at Albany · paper · repo
- Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents — Automated scenario generation, transformer asset integration, and scenario quality evaluation. Rohith Kanathur, Sagar Chethan Kumar, Columbia University · repo · paper
- AgentOpsBench — High-throughput battery analytics MCP server with DNN prognostics (RUL prediction) and 3.3× latency optimization. Siddharth Gowda, Rushin Bhatt, Aryaman Agrawal, Winston Li, Columbia University · repo
- Skill-Knowledge-Augmented Agents on AssetOpsBench — Confidence-gated skill execution with scoped knowledge plugins for industrial fault diagnosis. Vera Mazeeva, Sanskruti Shejwal, Shrey Arora, Mana Abbaszadeh, Columbia University · repo
- Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines. Krish Veera, , , , Columbia University · ·
Call for Scenario Contribution
We are expanding AssetOpsBench to cover a broader range of industrial challenges. We invite researchers and practitioners to contribute new scenarios, particularly in:
- Asset Classes: Turbines, HVAC systems, Pumps, Transformers, CNC Machines, Robotics, Engines
- Task Domains: Prognostics and Health Management, Remaining Useful Life (RUL) estimation, Root Cause Analysis (RCA), Diagnostic Analysis, Predictive Maintenance
How to contribute:
- Define your scenario following our Utterance Guideline and Ground Truth Guideline
- Explore the Hugging Face dataset for examples
- For external public sources and starter asset-class mappings, see External Industrial Dataset Guide
- Submit a Pull Request or open an Issue with the tag
new-scenario - Contact us with questions:
- Dhaval Patel — [email protected]
- Nianjun Zhou — [email protected]
Infrastructure Support
Model API access for AssetOpsBench 2.0 is enabled by TokenRouter (PaleBlueDot AI), a unified API platform providing access to leading AI models through a single API endpoint.
Contributors
Thanks to these wonderful people ✨
Star History
If AssetOpsBench is useful to your work, please ⭐ star the repo, 🍴 fork it, and tell us what you're building.
