Storage Service

The Storage Service (cosmos-storage-v3) provides file and folder management APIs with multi-hyperscaler support. Built on FastAPI with SQLAlchemy and Alembic, it abstracts blob storage across local filesystem, Azure Blob Storage, and Google Cloud Storage behind one API, so callers never branch on cloud provider.

Why It Matters

Storage is the platform's document management system (DMS): agent-generated files, knowledge base source documents, and user uploads all live here, gated by a fine-grained ACL model rather than plain org-level access.

Architecture At A Glance

• Framework: FastAPI + SQLAlchemy + Alembic, repository pattern with auto-discovered routers.
• Database: PostgreSQL (cosmos_storage_dev).
• Models: Item (unified files/directories), File, FileVersion, Directory, ItemACL.
• Multi-hyperscaler: HYPERSCALER selects the backend — local (filesystem, no SSL), azure (Azure Blob, no SSL required), gcp (GCS, SSL with certificates). The abstraction layer makes upload/download/delete behave identically regardless of backend.
• Processes: the webserver, a Celery worker for redaction tasks, and the jobs background-task supervisor. Cloud deployments start each under opentelemetry-instrument, which reports to Azure Monitor through APPLICATIONINSIGHTS_CONNECTION_STRING.

Key Capabilities

• Unified item model — files and directories share one Item table, so browse, search, move, and bulk operations work uniformly across both.
• Large uploads — multipart uploads up to 300 MB (MAX_FILE_SIZE), and resumable upload sessions that send bytes directly to cloud storage for files up to 6 GiB (MAX_ASYNC_UPLOAD_FILE_SIZE). Audio formats include .m4a.
• Upload-time PII redaction — redact text, images, and embedded media metadata during upload (native or Azure redaction), either for a single file or every file in a ZIP, with a review-and-apply step.
• Versioned files — every upload can create a new FileVersion; version history is queryable per file.
• Bulk and batch operations — bulk-fetch by ID or path (skips items the caller can't view), bulk access-level updates for up to 1,000 items, and batch permission checks for up to 100 items in one call.
• Background ZIP/unzip — archiving a directory or extracting a ZIP runs asynchronously (202 Accepted), respects per-item view permissions, and validates MIME type before extraction.
• Fine-grained ACLs — each ItemACL entry binds a user to an item with an allow/deny effect, granular flags (can_view, can_update, can_delete, can_manage_acl), and optional applies_to_children inheritance. At most one ALLOW and one DENY entry exist per user per item; ALLOW is evaluated before DENY (permissive-by-default), and RBAC *_all permissions bypass DENY entirely.
• Cached permission checks — POST /api/v1/items/check-permissions/ lets the frontend decide which action buttons to show per item without N individual calls; results are cached in Redis for 5 minutes.
• Project-attributed activity — file activity events sent to Observer carry the owning project when the operation happens inside a project, so project-scoped Recent Activity includes file changes.

Configuration

Deployments apply migrations with a Kubernetes migrate job (alembic upgrade head) before the webserver, worker, and jobs roll out, so migrations must be backward-compatible. Locally, run alembic upgrade head inside the storage container, or alembic revision --autogenerate -m "description" to create a new migration.

Integration Points

API Overview

All routes are under /api/v1/ except /health/.

→ Full API reference: Storage API