Potencialize seu agente
Instalação do publicador. Não é um endpoint do PlugMCP.
Host a copy with our keys and logs.
Ainda não há uma versão empacotada no registro. Siga o README do repositório.
URL remota ou um comando na sua máquina? Como as duas instalações diferem
Rodar no PlugMCP
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
AbrirDo publicador
A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes
A production-ready starter kit for deploying generative AI infrastructure on Amazon EKS (Elastic Kubernetes Service). Deploy LLM serving engines (vLLM, SGLang, TGI), AI gateways, vector databases, GPU-accelerated inference platforms, and AI agent frameworks on Kubernetes with a single CLI command. Built for teams running self-hosted LLMs and AI workloads on AWS.
The starter kit includes the configurable components and examples from several categories:
Before you begin, ensure you have the following tools installed:
npm install
./cli configure
# Example:
# ✔ Enter value for REGION: us-west-2
# ✔ Enter value for EKS_CLUSTER_NAME: genai-on-eks
# ? Enter value for DOMAIN:
This will prompt you to enter values for environment variables. Then, it will save the values on .env.local. There are a few important ones, including:
auto (default) or standardThere are two methods to setup your environment:
To quickly set up a demo environment with infrastructure and essential components and examples:
./cli demo-setup
This command will:
Check Demo Walkthrough on how to setup and use the demo
For a more customized setup, you can use the interactive setup command:
./cli interactive-setup
# Example:
# ✔ Select AI Gateway components to install: litellm
# ✔ Select LLM Model components to install: vllm
# ? Select Embedding Model components to install: (Press <space> to select, <a> to toggle all, <i> to invert selection, and <enter> to proceed)
# ❯◉ Text Embedding Inference (TEI)
This command will:
Note. Unlike the quick demo setup, the selected components and examples may not be deployed in the required order. Some components/examples might need to be refreshed by running the CLI install command again.
This starter kit supports deploying NVIDIA Dynamo for optimized LLM inference on Amazon EKS.
./cli nvidia-platform monitoring install # Prometheus + Grafana
./cli nvidia-platform gpu-operator install # NVIDIA GPU Operator
./cli nvidia-platform dynamo-platform install # Dynamo CRDs, Operator, etcd, NATS
./cli nvidia-platform dynamo-vllm install # Deploy a model with vLLM
./cli nvidia-platform benchmark install # AIPerf concurrency sweep
./cli nvidia-platform aiconfigurator install # TP/PP recommendation + SLA deploy
For full details on platform prerequisites, deployment modes (aggregated vs disaggregated), KV cache routing, monitoring dashboards, benchmarking, and AIConfigurator, see the NVIDIA Platform README.
You can install or uninstall individual components/examples using the CLI:
./cli <category> <component/example> install
# Examples:
# ./cli ai-gateway litellm install
# ./cli strands-agents calculator-agent install
./cli <category> <component/example> uninstall
# Examples:
# ./cli ai-gateway litellm uninstall
# ./cli strands-agents calculator-agent uninstall
The CLI provides commands to manage LLM/Embedding models for the hosting components:
Configure which models should be deployed for a specific component:
./cli llm-model <component> configure-models
./cli embedding-model <component> configure-models
# Example:
# ./cli llm-model vllm configure-models
# ./cli embedding-model tei configure-models
Add and/or remove models for a specific component:
./cli llm-model <component> update-models
./cli embedding-model <component> update-models
# Example:
# ./cli llm-model vllm update-models
# ./cli embedding-model tei update-models
Only add missing models for a specific component:
./cli llm-model <component> add-models
./cli embedding-model <component> add-models
# Example:
# ./cli llm-model vllm add-models
# ./cli embedding-model tei add-models
Remove all models for a specific component:
./cli llm-model <component> remove-all-models
./cli embedding-model <component> remove-all-models
# Example:
# ./cli llm-model vllm remove-all-models
# ./cli embedding-model tei remove-all-models
There are two methods to clean up your environment:
This method gives you more control over the cleanup process:
# Examples:
# ./cli strands-agents calculator-agent uninstall
# ./cli ai-gateway litellm uninstall
# ... uninstall other components/examples as needed
./cli cleanup-infra
This method provides a one-command solution to clean up all examples, components and infrastructure:
./cli cleanup-everything
This command will:
.env and config.json will be loaded first. Then, the configs will be merged/overridden with the values from .env.local and config.local.json if exist.
With a domain name already configured with a Route 53 hosted zone, a single shared ALB with HTTPS is used together with a wildcard ACM cert and Route 53 DNS records to expose all public facing services e.g. litellm. and openwebui..
Alternatively, when the DOMAIN field on .env (or .env.local) is empty, multiple ALBs with HTTP will be created for each public facing service. In this case, only one service requiring the Nginx Ingress basic auth (e.g. Milvus and Qdrant) can be exposed.
Run ./cli litellm install again to update the LiteLLM models.
For Bedrock models, the model list hardcoded on config.json.
# Example:
"bedrock": {
"llm": {
"models": [
{ "name": "amazon-nova-premier", "model": "us.amazon.nova-premier-v1:0" },
{ "name": "claude-4-opus", "model": "us.anthropic.claude-opus-4-20250514-v1:0" },
]
}
}
The default instance families are g6e, g6 and g5g and the default purchasing options are spot and on-demand. You can change the values on terraform/variables.tf and then run ./cli terraform apply again.
Note that the model deployment manifests use nodeSelector like eks.amazonaws.com/instance-family: g6e to lock the specific tested instance family which you will need to adjust accordingly.
For self-hosted models, they will be dynamically detected from the running model pods.
The supported models will have the -neuron suffix. To enable the support, on config.json (or config.local.json), change enableNeuron to true and then install the component again (e.g ./cli llm-model vllm install) which will take ~20-30 mins to build the vLLM Neuron container image and push it to ECR.
When a supported LLM model is deployed for the first time, Neorun just-in-time (JIT) compilation will compile the model which will take ~20-30 mins. The compiled model then will be cached on EFS file system for the subsequent deployments.
Llama-3.1-8B-Instruct, DeepSeek-R1-Distill-Llama-8B, Mistral-7B-Instruct-v0.3 models can the INT8 quantization to run on single inf2.xlarge, but inf2.8xlarge is still required to compile.
config.json (or config.local.json), change from "compile": true to "compile": falseBy default, docker buildx is used to build the multi-arch container images. To disable it, modify the docker section on config.json (or config.local.json) to set "useBuildx": false and arch based on your machine OS arch.
By default, the same region as the EKS cluster will be used. To change it, modify the bedrock section on config.json (or config.local.json) to set region to the preferred region.
You can change the values of the REGION, EKS_CLUSTER_NAME, and DOMAIN fields on .env (or .env.local). Then, Terraform workspace and kubectl context will automatically use those values when running the related ./cli commands.
ECR Pull Through Cache caches external container images (from Docker Hub, GitHub Container Registry) in your private ECR registry. This avoids rate limits from public registries and keeps images within your AWS infrastructure.
Default: Disabled (enable_ecr_pull_through_cache = false)
Why disabled by default?
When to enable:
To enable:
Docker Hub and GitHub Container Registry require authentication for ECR pull through cache. You'll need to provide credentials for both registries.
Get Docker Hub credentials:
Get GitHub credentials:
read:packages scope to pull from GitHub Container RegistryConfigure credentials in config.local.json:
// config.local.json
{
"terraform": {
"vars": {
"enable_ecr_pull_through_cache": true,
"dockerhub_username": "your-dockerhub-username",
"dockerhub_access_token": "your-dockerhub-access-token",
"github_username": "your-github-username",
"github_token": "your-github-personal-access-token"
}
}
}
./cli terraform apply
Important: Keep your credentials in config.local.json (which is gitignored) and never commit them to version control. The credentials are stored securely in AWS Secrets Manager with the ecr-pullthroughcache/ prefix.
Supported registries:
vllm/* → Docker Hublmsysorg/* → Docker Hubollama/* → Docker Hubhuggingface/* → GitHub Container RegistryNote: When you terraform destroy, the cache rules are deleted but the cached ECR repositories remain. To fully clean up, manually delete repositories starting with vllm/, lmsysorg/, ollama/, or huggingface/ from ECR.
⚠️ This repository is intended for demonstration and learning purposes only. It is not intended for production use. The code provided here is for educational purposes and should not be used in a live environment without proper testing, validation, and modifications.
Use at your own risk. The authors are not responsible for any issues, damages, or losses that may result from using this code in production.
Check Security Considerations for more information on the security scans.
Contributions welcome! Please read our Contributing Guidelines and Code of Conduct for more information.
This project is licensed under the MIT License - see LICENSE file.
A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama
Visão geral
A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes
Ferramentas
Ainda não há lista de ferramentas. Servidores remotos recebem uma após a primeira verificação de saúde.
Perguntas frequentes
Copie nesta página a configuração stdio para Cursor, Claude Desktop, Claude Code ou VS Code. Ela usa a instalação do publicador, não um proxy do PlugMCP.
Este registro não lista nenhuma variável de ambiente obrigatória.
Sample Genai On Eks Starter Kit roda localmente via stdio.
Última verificação: 2026-10-27. A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes