Model Garden
Kubernetes-backed model deployment, inference, and fine-tuning service.
Updated 2026-11-03
Verified 2026-11-03
Model Garden is a Go service for deploying, managing, and serving open-source models on Kubernetes GPU node pools. It manages model metadata in PostgreSQL and coordinates runtime deployment through the Kubernetes API.
Responsibilities
- Register and manage model records.
- Deploy models to Kubernetes-backed GPU infrastructure, with an optional security scan before deployment.
- Serve inference routes with API-key authentication.
- Manage Docker images used for model serving.
- Track node pool capacity and orphaned deployments.
- Support fine-tuning datasets and fine-tuning job workflows.
Runtime
| Area | Details |
|---|---|
| Repository | modelgarden |
| Language | Go |
| Default port | 8000 |
| Storage | PostgreSQL through GORM; Azure Blob Storage for datasets |
| Infrastructure dependency | Kubernetes API |
| Default serving image | vllm/vllm-openai:latest (vLLM containers run with VLLM_ALLOW_LONG_MAX_MODEL_LEN=1) |
API Areas
| Path | Purpose |
|---|---|
/api/v1/models/ | Model registration and lifecycle: CRUD plus deploy, stop, status, logs, adapters, and scan-report per model. |
/api/v1/docker-images/ | Serving image management. |
/api/v1/api-keys/ | Inference API-key management. |
/api/v1/inference/{model_id}/... | OpenAI-compatible chat/completions, completions, embeddings, and models; other paths are proxied when wildcard proxying is enabled. |
/api/v1/nodepool/status | GPU node pool visibility and scheduling data. |
/api/v1/fine-tuning/jobs | Fine-tuning jobs: create, list, get, update (PATCH), and cancel, retry, deploy, undeploy, artifacts, logs. |
/api/v1/datasets/ | Fine-tuning datasets: list, upload, folder, download/{path}, preview/{path}, and delete. |
/api/v1/orphaned-deployments | Deployment cleanup visibility. |
/health | Health check. |
/ready | Readiness check. |
GET /api/v1/datasets/ accepts prefix, search, page, and page_size, and returns {data, total, page, page_size, total_pages, prefix, search}. Each entry includes owner, created_by, and updated_by.
Authentication
Management endpoints use Cosmos JWTs validated against the User Service JWKS, and privileged operations require administrative access. Inference endpoints accept API keys with the mg-sk- prefix; API keys are rejected on every other route.
Configuration
Important environment variables:
| Variable | Purpose |
|---|---|
PORT | Service port (default 8000). |
DATABASE_URL / DB_HOST, DB_PORT, DB_USER, DBPASSWORD, DB_NAME, DB_SSLMODE | PostgreSQL connection configuration. |
JWKS_URL, USER_SERVICE_URL | JWT validation and User Service calls. |
K8S_NAMESPACE | Kubernetes namespace for model workloads (default modelgarden). |
K8S_IN_CLUSTER | Selects in-cluster Kubernetes config (default true). |
GPU_NODEPOOL_CONFIG | GPU node pool configuration. |
MAX_GPU_MEMORY_UTILIZATION | GPU scheduling guardrail (default 0.95). |
VLLM_DEFAULT_GPU_MEMORY_UTILIZATION | Default vLLM GPU memory utilization (default 0.9). |
MODEL_CACHE_PVC | Model cache persistent volume claim. |
HUGGINGFACE_TOKEN | Hugging Face access token. |
DEFAULT_DOCKER_IMAGE | Default serving image. |
ENABLE_MODEL_SCAN, MODELSCAN_SCANNER_IMAGE | Pre-deployment model scan (enabled by default) and scanner image. |
AZURE_STORAGE_ACCOUNT_NAME, AZURESTORAGEACCOUNTKEY, AZURE_STORAGE_CONTAINER_NAME | Dataset blob storage. |
DATASET_UPLOAD_MAX_SIZE, DATASET_PREVIEW_MAX_SIZE | Dataset upload limit (default 500 MB) and preview limit (default 1 MB). |
AXOLOTL_IMAGE | Fine-tuning image. |
CORS_ALLOWED_ORIGINS | Allowed browser origins. |
ENABLE_WILDCARD_PROXY | Enables wildcard inference proxying (default false). |
Changelog
See Model Garden Changelog for verified documentation and service-surface changes.