Architecture
Component Overview
┌─────────────────────────────────────────────────────────────────┐
│ LiteLLM Operator │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────┐ │
│ │ Instance │ │ Organization │ │ Team │ │
│ │ Controller │ │ Controller │ │ Controller │ │
│ │ │ │ │ │ │ │
│ │ - Deployment │ │ - POST /org/* │ │ - POST /team │ │
│ │ - ConfigMap │ │ - Members │ │ - Members │ │
│ │ - Service │ │ - Budget mgmt │ │ - Budgets │ │
│ │ - Ingress/Route │ │ │ │ │ │
│ │ - HPA, PDB │ └────────┬─────────┘ └──────┬───────┘ │
│ │ - Migration Job │ │ │ │
│ │ - SSO/SCIM config │ ┌────────┴────────┐ ┌───────┴───────┐ │
│ │ - License Secret │ │ Model │ │ User │ │
│ │ │ │ Controller │ │ Controller │ │
│ └────────┬──────────┘ │ │ │ │ │
│ │ │ - POST /model/* │ │ - POST /user/*│ │
│ │ │ - Health check │ │ - Budget mgmt │ │
│ │ └────────┬────────┘ └───────┬───────┘ │
│ │ │ │ │
│ │ ┌────────┴────────────────────┘ │
│ │ │ │
│ │ ┌────────┴────────┐ │
│ │ │ VirtualKey │ ┌──────────────────┐ │
│ │ │ Controller │ │ ConfigSync │ │
│ │ │ - POST /key/* │ │ Controller │ │
│ │ │ - Secret mgmt │ │ - Drift detect │ │
│ │ └────────┬────────┘ │ - Prune/adopt │ │
│ │ │ │ - Status counts │ │
│ │ │ └────────┬─────────┘ │
│ ┌────────▼─────────────▼────────────────────▼───────────────┐ │
│ │ LiteLLM API Client │ │
│ │ OrganizationService · ModelService · TeamService │ │
│ │ UserService · KeyService · CustomerService · HealthService│ │
│ └────────────────────────┬───────────────────────────────────┘ │
│ │ │
└───────────────────────────┼──────────────────────────────────────┘
│ HTTP (within cluster)
▼
┌─────────────────────┐
│ LiteLLM Proxy │
│ (Deployment) │
│ REST API :4000 │
└─────────┬───────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│PostgreSQL│ │ Redis │ │ LLM │
│ │ │ (cache) │ │Providers │
└──────────┘ └──────────┘ └──────────┘Controllers
Instance Controller
The most complex controller. It manages all Kubernetes infrastructure for a LiteLLM deployment:
- ConfigMap — generates
proxy_server_config.yamlfrom general settings (including IP allowlisting), router settings (including tag-based routing), fallback chains, retry policies, caching, SSO, and callback configuration - Secrets — master key (auto-generated or from existing Secret), salt key, SSO client credentials
- Migration Job — runs database migrations before Deployment rollout
- Deployment — LiteLLM container with env vars, volumes, probes, security context
- Service — ClusterIP or LoadBalancer
- Ingress / Route — optional external access (Ingress for vanilla K8s, Route for OpenShift)
- HPA — horizontal pod autoscaling based on CPU/memory
- PDB — pod disruption budget for availability
- NetworkPolicy — restrict access to the LiteLLM service
- OpenShift Route — optional Route for OpenShift clusters (TLS edge/passthrough/reencrypt)
- Gateway API HTTPRoute — optional HTTPRoute for Gateway API implementations (Istio, Envoy Gateway, Cilium, etc.)
- SCIM Token — auto-generate and store SCIM bearer token
- License Secret detection — discovers
{instance}-licenseorlitellm-licenseSecrets and injectsLITELLM_LICENSEenv var viasecretKeyRef
Secondary Controllers
Most secondary controllers (Organization, Model, Team, User, Customer, VirtualKey) follow the same pattern:
CR created/updated/deleted
│
├── 1. Fetch CR
├── 2. Check deletion → finalizer cleanup → call LiteLLM DELETE API
├── 3. Ensure finalizer present
├── 4. Resolve instanceRef → get API endpoint + master key
├── 5. Reconcile against LiteLLM API (create or update)
└── 6. Update status (synced, IDs, conditions)Change detection uses a spec hash stored in the litellm.palena.ai/sync-hash annotation. On each reconciliation, the current spec hash is compared to the stored hash — if different, an update is sent to the LiteLLM API.
Credential Controller
The Credential controller is different: LiteLLMCredential is a config-level resource, not an API-level one. There is no POST /credential/new equivalent — credentials live in the proxy's credential_list config section. The controller validates the referenced Secret and counts consuming models, but the actual materialization happens in the Instance controller, which watches LiteLLMCredential and rebuilds the ConfigMap + rolls the Deployment whenever a credential changes. API keys are injected as CREDENTIAL_{NAME}_API_KEY env vars via secretKeyRef, and the ConfigMap uses os.environ/... placeholders so plaintext keys never land on disk.
Guardrail Controller
Like the Credential controller, the Guardrail controller manages a config-level resource. LiteLLMGuardrail CRs are rendered into the guardrails section of proxy_server_config.yaml by the Instance controller — there is no API-level create/update. The Guardrail controller validates that the referenced instance and (if declared) API key Secret exist, and reports Ready / SecretNotFound / InstanceNotFound conditions. The Instance controller watches LiteLLMGuardrail CRs and rebuilds the ConfigMap + rolls the Deployment whenever a guardrail is created, updated, or deleted. Provider API keys are injected as GUARDRAIL_{NAME}_API_KEY env vars via secretKeyRef, referenced from the config as os.environ/.... Per-key/per-team guardrail assignment via spec.guardrails []string on LiteLLMVirtualKey / LiteLLMTeam is a LiteLLM Enterprise feature.
ConfigSync Controller
The ConfigSync controller is a separate controller (not part of the Instance controller) that implements bidirectional config sync. It watches LiteLLMInstance resources and, for each instance with spec.configSync.enabled: true, runs a periodic sync cycle on the configured interval (default 30s).
The sync cycle covers all six API-managed resource types: models, teams, users, keys, organizations, and customers. For each type it:
- Lists CRDs referencing this instance and builds a map of API IDs
- Lists all resources from the LiteLLM API
- Classifies each API resource as managed (has a matching CRD) or unmanaged
- Detects drift on managed resources by comparing key spec fields against the API response
- Handles drift per the
conflictResolutionpolicy and unmanaged resources per theunmanagedResourcePolicy - Updates
status.configSynccounts and theConfigSyncedcondition
The ConfigSync controller is complementary to the per-resource controllers: per-resource controllers handle real-time CRD lifecycle (event-driven, immediate), while ConfigSync provides periodic comprehensive drift detection, unmanaged resource handling, and status aggregation.
Reconciliation Model
The operator uses the standard Kubernetes reconciliation pattern:
- Finalizers ensure cleanup: deleting a CRD calls the corresponding LiteLLM API delete endpoint before removing the Kubernetes resource
- Status conditions report health using standard
metav1.Conditiontypes - Requeue strategies:
- Transient errors (network, API 5xx):
RequeueAfter: 30s - Permanent errors (invalid spec, 400): set status condition, don't requeue
- Enterprise license errors (403 + "enterprise"): set
EnterpriseLicenseRequiredcondition, don't requeue - Healthy state:
RequeueAfter: 5mfor periodic re-sync
- Transient errors (network, API 5xx):
Security Model
- Operator runs with a dedicated ServiceAccount and scoped RBAC
- LiteLLM pods run as non-root with a read-only root filesystem
- Secrets (master key, salt key, provider API keys, license keys) are always read from Kubernetes Secrets via
secretKeyRef— never stored as plaintext in CRDs or read into operator memory - NetworkPolicy restricts which namespaces can reach the LiteLLM service (network layer)
- IP allowlisting restricts API access to specific IP addresses or CIDR ranges (application layer, enterprise)
- Generated virtual keys are stored in Secrets with
ownerReferencesfor automatic garbage collection
Upgrade Strategy
When spec.image.tag changes on a LiteLLMInstance:
- A migration Job runs with the new image
- On success, the Deployment is updated with a rolling update
- Health checks validate new pods
- If auto-rollback is enabled and health checks fail, the Deployment reverts
