LiteLLMInstance
The primary CRD. Deploys a LiteLLM proxy with all infrastructure dependencies.
API Version: litellm.palena.ai/v1alpha1Kind: LiteLLMInstanceShort Name: li
Minimal Example
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
name: my-gateway
spec:
masterKey:
autoGenerate: true
database:
external:
connectionSecretRef:
name: litellm-db
key: DATABASE_URLFull Example
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
name: my-gateway
spec:
image:
repository: ghcr.io/berriai/litellm
tag: main-v1.60.0
pullPolicy: IfNotPresent
replicas: 3
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilization: 70
masterKey:
autoGenerate: true
database:
external:
connectionSecretRef:
name: litellm-db
key: DATABASE_URL
connectionPool:
maxConnections: 20
migration:
enabled: true
timeout: "300s"
redis:
enabled: true
host: redis.default.svc
port: 6379
passwordSecretRef:
name: redis-credentials
key: password
saltKey:
autoGenerate: true
service:
type: ClusterIP
port: 4000
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/litellm
labels:
example.com/identity: irsa
deployment:
annotations:
reloader.stakater.com/auto: "true"
podScheduling:
nodeSelector:
kubernetes.io/arch: arm64
tolerations:
- key: kubernetes.io/arch
operator: Equal
value: arm64
effect: NoSchedule
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: litellm
ingress:
enabled: true
host: litellm.example.com
annotations:
cert-manager.io/cluster-issuer: letsencrypt
configSync:
enabled: true
interval: "30s"
unmanagedResourcePolicy: preserve
conflictResolution: crd-wins
auditChanges: true
caching:
enabled: true
type: redis
ttl: 600
namespace: "prod"
mode: default_on
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "2"
memory: 1Gi
podDisruptionBudget:
enabled: true
minAvailable: 1
security:
networkPolicy:
enabled: true
allowedNamespaces:
- default
- production
ipAllowlist:
enabled: true
allowedIPs:
- "10.0.0.0/8"
- "192.168.1.0/24"
useXForwardedFor: true
generalSettings:
masterKeyRequired: true
maxBudget: "10000.00"
budgetDuration: "30d"
globalMaxParallelRequests: 100
budgetReschedulerMinTime: 300
budgetReschedulerMaxTime: 600
routerSettings:
routingStrategy: least-busy
numRetries: 3
enableTagFiltering: true
defaultMaxParallelRequests: 10
retryPolicy:
TimeoutError: 2
RateLimitError: 3
modelGroupRetryPolicy:
gpt-4:
TimeoutError: 1
providerBudgetConfig:
openai:
budgetLimit: "500.00"
timePeriod: "1d"
anthropic:
budgetLimit: "300.00"
timePeriod: "1d"
fallbacks:
defaultFallbacks: ["gpt-4-mini", "claude-3-haiku"]
modelFallbacks:
- model: gpt-4
fallbacks: ["gpt-4-mini", "claude-3-haiku"]
contentPolicyFallbacks:
- model: gpt-4
fallbacks: ["claude-3-sonnet"]
contextWindowFallbacks:
- model: gpt-4
fallbacks: ["gpt-4-32k", "claude-3-sonnet"]
maxFallbacks: 3
passThroughEndpoints:
- path: /bria
target: https://engine.prod.bria-api.com
headers:
content-type: application/json
headerSecrets:
- headerName: Authorization
prefix: "Bearer "
secretRef:
name: bria-credentials
key: api-key
- path: /langfuse
target: https://us.cloud.langfuse.com
auth: true
forwardHeaders: true
includeSubpath: true
methods: ["GET", "POST"]
defaultQueryParams:
version: "2"
jwtAuth:
enabled: true
adminJwtScope: "litellm_proxy_admin"
adminAllowedRoutes:
- openai_routes
- info_routes
teamIdJwtField: "client_id"
teamIdsJwtField: "groups"
orgIdJwtField: "org_id"
userIdJwtField: "sub"
userEmailJwtField: "email"
publicKeyTtl: 600
oauth2Auth:
enabled: true
configMappings:
- name: clientId
jwtField: "client_id"
litellmAttribute: "team_id"
- name: userId
jwtField: "sub"
litellmAttribute: "user_id"
secretManager:
provider: aws_secret_manager
credentialsSecretRef:
name: litellm-aws-credentials
hostedKeys:
- OPENAI_API_KEY
- ANTHROPIC_API_KEY
storeVirtualKeys: true
prefixForStoredVirtualKeys: "litellm/"
accessMode: read_and_write
aws:
region: us-east-1
rbac:
enabled: true
adminOnlyRoutes:
- /model/new
- /model/delete
- /organization/new
allowedRoutes:
- /chat/completions
- /embeddings
- /key/info
- /user/info
defaultTeamDisabled: true
keyGeneration:
teamKeyGeneration:
allowedTeamMemberRoles: ["admin"]
personalKeyGeneration:
allowedUserRoles: ["proxy_admin"]
rolePermissions:
internal_user:
routes:
- /key/generate
- /key/delete
- /key/info
models:
- gpt-4
- claude-3-haiku
logging:
auditLogs:
enabled: true
retentionDays: 90
turnOffMessageLogging: false
redactUserApiKeyInfo: true
spendLogRetention:
maxRetentionPeriod: "90d"
cleanupInterval: "1d"
adminUI:
disabled: false
adminOnly: true
storeModelInDB: true
defaultTeamDisabled: true
apiDocBaseURL: "https://api.example.com"
docsURL: "/docs"
rootRedirectURL: "/ui"
logoURL: "https://example.com/logo.png"
emailLogoURL: "https://example.com/email-logo.png"
emailSupportContact: "support@example.com"
colorThemeConfigMapRef:
name: litellm-colorsSpec Fields
image
| Field | Type | Default | Description |
|---|---|---|---|
repository | string | ghcr.io/berriai/litellm | Container image repository |
tag | string | latest | Image tag |
pullPolicy | string | IfNotPresent | Image pull policy |
pullSecrets | []SecretRef | — | Image pull secrets |
workload
Controls whether the operator provisions the proxy workload. Omit the block (the default) and the operator creates and owns the Deployment, Service, ConfigMap and ServiceAccount as it always has.
| Field | Type | Default | Description |
|---|---|---|---|
managed | bool | true | Whether the operator owns the proxy workload. Set false to attach to a deployment managed elsewhere |
endpoint | string | — | Base URL of the existing proxy, e.g. http://litellm.platform.svc:4000. Only valid when managed is false. Defaults to http(s)://<metadata.name>.<namespace>.svc:<service.port> |
With managed: false the operator creates nothing and adopts nothing — no existing object is mutated, no owner reference added. It resolves an endpoint and a master key so the entity CRDs (LiteLLMTeam, LiteLLMVirtualKey, LiteLLMBudget, LiteLLMModel, ...) work against a proxy owned by a Helm chart, a GitOps pipeline or an internal platform.
spec:
workload:
managed: false
endpoint: http://litellm.platform.svc:4000
masterKey:
secretRef:
name: litellm-master-key
key: LITELLM_MASTER_KEY
database: {}Behaviour differences when unmanaged:
| Managed (default) | Unmanaged | |
|---|---|---|
| Deployment, Service, ConfigMap, ServiceAccount | Created and reconciled | Never touched |
| Ingress, Route, HTTPRoute, HPA, PDB, NetworkPolicy, ServiceMonitor | Created when enabled | Never touched |
Auto-rollback (spec.upgrade.autoRollback) | Active | Skipped |
Database migration Job (spec.database.migration) | Created when enabled | Never — the proxy owns its schema |
status.ready | At least one Deployment replica ready | Admin API answers /health/liveliness at status.endpoint |
status.replicas / readyReplicas | Deployment counts | 0 |
status.version | spec.image.tag (or latest) | Empty, unless the proxy discloses litellm_version on /health/readiness |
PodsHealthy condition | Set | Absent — the operator owns no pods |
DatabaseReady condition reason | MigrationSkipped / MigrationComplete / … | WorkloadUnmanaged |
Ready condition reason | AllResourcesReady / DeploymentNotReady | ProxyReachable / ProxyNotReachable |
WorkloadUnmanaged condition | Absent | Set when the CR also carries spec sections the operator ignores unmanaged (see below) |
| Health probing, config sync, entity CRDs, finalizer cleanup | Active | Active |
masterKey.autoGenerate is rejected here by a CEL rule: the Secret it would create is only ever built by the managed-workload reconcile path, so on an unmanaged instance the reference would dangle. Reference the existing proxy's admin key with masterKey.secretRef instead.
A CR copied from a managed instance often keeps fields like sso, caching, rbac, or secretManager set — those only take effect through the ConfigMap a managed workload builds. The WorkloadUnmanaged condition names whichever of these the CR actually has set, so one that never set them stays quiet.
Database fields describe the proxy's own database and are only consumed when building the workload, so database: {} is the normal unmanaged value.
spec.database.migration is ignored entirely. An externally-managed proxy owns its own schema: LiteLLM migrates on startup, and whatever deployed it has its own migration hook, so a second migrator would race the real one. The migration Job also takes its image from spec.image.tag — meaningless for a proxy the operator did not deploy, and defaulting to latest — which would run prisma migrate deploy at an arbitrary schema version against a database the operator does not own. DatabaseReady reports WorkloadUnmanaged, and says so explicitly if you configured a migration anyway.
replicas
| Field | Type | Default | Description |
|---|---|---|---|
replicas | int32 | 1 | Number of proxy replicas |
autoscaling
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable HPA |
minReplicas | int32 | 1 | Minimum replicas |
maxReplicas | int32 | — | Maximum replicas (required if enabled) |
targetCPUUtilization | *int32 | — | Target CPU percentage |
targetMemoryUtilization | *int32 | — | Target memory percentage |
masterKey
| Field | Type | Default | Description |
|---|---|---|---|
secretRef | *SecretKeyRef | — | Reference to existing master key Secret |
autoGenerate | bool | false | Auto-generate and store in a Secret |
database
| Field | Type | Description |
|---|---|---|
external | *ExternalDBSpec | External PostgreSQL connection |
cloudnativepg | *CloudNativePGSpec | CloudNativePG cluster reference |
managed | *ManagedDBSpec | Operator-managed single-pod PostgreSQL |
connectionPool | *ConnectionPoolSpec | Connection pool settings |
migration | *MigrationSpec | Database migration settings |
database.migration
The migration Job runs prisma migrate deploy (applying LiteLLM's versioned migration files in litellm-proxy-extras/migrations) — the same command LiteLLM's own componentized chart uses. Required for LiteLLM v1.86+, which disables schema updates in app pods (see PR #27557).
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | true | Run the migration Job before the proxy starts |
timeout | string | "300s" | Migration Job timeout |
useDatabaseImage | bool | false | Run LiteLLM's dedicated litellm-migrations migrations image instead of prisma migrate deploy in the gateway image. When true, only the database image runs — the operator does not also invoke prisma inside the gateway image |
databaseImage | *DatabaseImageSpec | — | Override repo/tag/pullPolicy for the database image. Only consulted when useDatabaseImage: true. Repo defaults to ghcr.io/berriai/litellm-migrations; tag defaults to spec.image.tag so versions stay aligned |
When to enable useDatabaseImage:
- You are on LiteLLM v1.86+ and want the maintained recovery logic (P3005 baseline, P3009/P3018 idempotent retries, v2 migration resolver) for free.
- You are upgrading from a pre-v0.12 operator that ran
prisma db pushagainst your DB — the database image's recovery flow heals the resulting_prisma_migrations/ schema drift automatically.
Tag availability caveat: as of June 2026, ghcr.io/berriai/litellm-migrations only publishes release-candidate tags (e.g. v1.87.0-rc.1, v1.88.0-rc.1). If your gateway tag isn't published as a migrations tag yet, either override databaseImage.tag to one that exists or stay on the gateway-image path. The gateway-image path now invokes the same ProxyExtrasDBManager.setup_database(use_migrate=True, use_v2_resolver=True) entry the dedicated image uses and works on every LiteLLM v1.85+ gateway tag (verified end-to-end against v1.85.3, v1.86.1, and v1.87.0).
Example:
spec:
database:
external:
connectionSecretRef:
name: litellm-db
key: DATABASE_URL
migration:
enabled: true
useDatabaseImage: true # run only ghcr.io/berriai/litellm-migrations
# databaseImage: # optional override (e.g. internal mirror)
# repository: registry.example.com/mirror/litellm-migrations
# tag: v1.87.0podScheduling
Node placement applied consistently to the proxy Deployment Pods and the database migration Job Pod. Changing it creates a fresh migration Job because Kubernetes does not allow its Pod template to be modified in place. An omitted podScheduling, an empty podScheduling: {} and empty values are equivalent and leave the Job name unchanged.
| Field | Type | Description |
|---|---|---|
nodeSelector | map[string]string | Node labels required by LiteLLM Pods |
tolerations | []Toleration | Taints LiteLLM Pods may tolerate |
affinity | *Affinity | Node affinity and pod (anti-)affinity, for rules nodeSelector cannot express: set-based matches, soft/preferred placement, and spreading replicas across nodes or zones |
Note that spec.topologySpreadConstraints is configured separately, at the top level of the spec, and applies to the proxy Deployment only — spread constraints are meaningless for the single-Pod migration Job.
Example:
spec:
podScheduling:
nodeSelector:
kubernetes.io/arch: arm64
tolerations:
- key: kubernetes.io/arch
operator: Equal
value: arm64
effect: NoSchedule
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: litellmredis
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable Redis |
connectionSecretRef | *SecretKeyRef | — | Redis connection URL Secret |
host | string | — | Redis host |
port | int | 6379 | Redis port |
passwordSecretRef | *SecretKeyRef | — | Redis password Secret |
configSync
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable bidirectional config sync |
interval | string | 30s | Sync interval |
unmanagedResourcePolicy | string | preserve | Policy for unmanaged resources: preserve, prune, adopt |
conflictResolution | string | crd-wins | Conflict strategy: crd-wins, api-wins, manual |
auditChanges | bool | false | Emit Kubernetes Events for sync changes |
service
| Field | Type | Default | Description |
|---|---|---|---|
type | string | ClusterIP | Service type |
port | int32 | 4000 | Service port |
generalSettings
| Field | Type | Description |
|---|---|---|
masterKeyRequired | *bool | Require master key for all requests |
proxyBatchWriteAt | int | Batch write interval in seconds |
alertTypes | []string | Alert types to generate notifications for |
alerting | []string | Alert delivery channels (e.g. ["slack"]). Required for alerts to actually fire; for Slack also set SLACK_WEBHOOK_URL via extraEnvVars |
alertingThreshold | *int | Seconds a request may hang before a hanging-request alert fires |
alertToWebhookUrl | map[string]string | Override the delivery webhook per alert type |
backgroundHealthChecks | *bool | Enable periodic background health checks so GET /health returns cached results |
healthCheckInterval | *int | Seconds between background health checks (default 300) |
healthCheckDetails | *bool | Whether GET /health exposes endpoint URLs and error details |
customKeyGenerate | string | Dotted path to a custom key-generation handler |
allowUserAuth | *bool | Allow requests without a key |
maxBudget | *string | Global proxy budget in USD |
budgetDuration | string | Global budget reset duration (e.g., 1d, 7d, 30d) |
globalMaxParallelRequests | *int | Maximum parallel requests across the entire proxy |
budgetReschedulerMinTime | *int | Minimum interval (seconds) between budget reset checks |
budgetReschedulerMaxTime | *int | Maximum interval (seconds) between budget reset checks |
extra | map[string]JSON | Arbitrary keys passed straight through to general_settings (see Raw settings passthrough) |
routerSettings
| Field | Type | Default | Description |
|---|---|---|---|
routingStrategy | string | — | simple-shuffle, least-busy, latency-based-routing, usage-based-routing, usage-based-routing-v2, cost-based-routing |
numRetries | *int | — | Number of retries |
timeout | *int | — | Timeout in seconds |
streamTimeout | *int | — | Timeout in seconds for streaming responses (distinct from timeout) |
retryAfter | *int | — | Seconds to wait before a retry |
cooldownTime | *int | — | Cooldown time in seconds |
enablePreCallChecks | *bool | — | Filter deployments by context-window size and region before calling |
modelGroupAlias | map[string]string | — | Map an alias model-group name to a real one |
retryPolicy | map[string]int | — | Per-error-type retry counts (e.g., TimeoutError: 2, RateLimitError: 3) |
modelGroupRetryPolicy | map[string]map[string]int | — | Per-model-group retry overrides (e.g., gpt-4: {TimeoutError: 1}) |
enableTagFiltering | *bool | — | Enable tag-based routing. Requests with matching tags route to tagged model deployments |
tagFilteringMatchAny | *bool | — | If true, match ANY tag (OR logic). If false (default), ALL tags must match (AND logic) |
defaultMaxParallelRequests | *int | — | Default max parallel requests per model deployment |
providerBudgetConfig | map[string]ProviderBudget | — | Per-provider spending limits |
ProviderBudget:
| Field | Type | Description |
|---|---|---|
budgetLimit | string | Budget limit in USD |
timePeriod | string | Time period for the budget (e.g., 1d, 7d, 30d) |
litellmSettings
Settings rendered under litellm_settings in proxy_server_config.yaml.
| Field | Type | Description |
|---|---|---|
jsonLogs | *bool | Emit structured JSON logs instead of plaintext (recommended for log aggregation) |
checkProviderEndpoint | *bool | Query each provider's own /models endpoint when serving GET /v1/models, so a wildcard deployment lists the upstream's real model names instead of the literal pattern |
extra | map[string]JSON | Arbitrary keys passed straight through to litellm_settings (see Raw settings passthrough) |
Wildcard model discovery
checkProviderEndpoint maps to litellm_settings.check_provider_endpoint. Without it, a wildcard deployment such as up/* is listed either as the literal pattern or expanded from LiteLLM's static built-in model list for that provider. With it, the proxy calls the upstream's /models endpoint and lists what is actually there:
spec:
litellmSettings:
checkProviderEndpoint: true# LiteLLMModel
spec:
modelName: "up/*"
litellmParams:
model: "openai/*"
apiBase: https://upstream.example.com/v1
apiKeySecretRef: {name: upstream-key, key: api_key}It applies to models registered through the API by LiteLLMModel as well as to any declared in the config file.
Two caveats:
- It is a global switch. Every wildcard deployment on the proxy is discovered this way, and each discovery reaches out to the upstream (LiteLLM caches results).
- It only works for providers in LiteLLM's
models_by_providermap. Notablylitellm_proxy/*is not one of them — a wildcard backed bylitellm_proxy/*stays unexpanded no matter what this flag is set to. To front another LiteLLM proxy, useopenai/*with the upstream's/v1base URL, as above; a LiteLLM proxy is OpenAI-compatible.
Raw settings passthrough
generalSettings.extra and litellmSettings.extra pass arbitrary keys straight into their config sections, as an escape hatch for LiteLLM settings this CRD does not model yet:
spec:
litellmSettings:
extra:
some_new_upstream_setting: true
nested_value: {a: [1, 2]}Anything the operator derives from the rest of the spec always wins: a colliding key here is ignored, and the instance emits an ExtraSettingsIgnored warning event naming it. Prefer a typed field where one exists — settings placed here are neither validated nor documented by the operator, and a later release may add a typed field that starts taking precedence over your entry.
fallbacks
Configure model fallback chains. Fallback model names must match model_name values from your LiteLLMModel resources.
| Field | Type | Default | Description |
|---|---|---|---|
defaultFallbacks | []string | — | Global fallback models applied on any error |
modelFallbacks | []ModelFallbackEntry | — | Per-model fallback chains for general errors |
contentPolicyFallbacks | []ModelFallbackEntry | — | Fallbacks for content policy violations |
contextWindowFallbacks | []ModelFallbackEntry | — | Fallbacks for context window exceeded errors |
maxFallbacks | *int | 3 | Maximum number of fallback attempts |
ModelFallbackEntry:
| Field | Type | Description |
|---|---|---|
model | string | Primary model name |
fallbacks | []string | Ordered list of fallback model names |
caching
Response caching configuration. See the Caching guide for detailed examples.
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable response caching |
type | string | redis | Cache backend: redis, redis-semantic, s3, gcs, qdrant, local |
ttl | *int | 600 | Cache TTL in seconds |
namespace | string | — | Cache key namespace for isolation |
supportedCallTypes | []string | — | Restrict caching to specific call types |
mode | string | default_on | default_on (cache all) or default_off (require opt-in) |
redis | *CacheRedisSpec | — | Redis-specific config (omit to reuse spec.redis) |
s3 | *CacheS3Spec | — | S3 backend config |
gcs | *CacheGCSSpec | — | GCS backend config |
qdrant | *CacheQdrantSpec | — | Qdrant semantic cache config |
CacheRedisSpec:
| Field | Type | Default | Description |
|---|---|---|---|
host | string | — | Redis host |
port | *int | — | Redis port |
passwordSecretRef | *SecretKeyRef | — | Redis password Secret reference |
ssl | bool | false | Enable SSL/TLS |
CacheS3Spec:
| Field | Type | Description |
|---|---|---|
bucketName | string | S3 bucket name (required) |
region | string | AWS region |
credentialsSecretRef | *SecretKeyRef | Secret with aws_access_key_id and aws_secret_access_key keys |
CacheGCSSpec:
| Field | Type | Description |
|---|---|---|
bucketName | string | GCS bucket name (required) |
credentialsSecretRef | *SecretKeyRef | Secret with GCS service account JSON |
CacheQdrantSpec:
| Field | Type | Description |
|---|---|---|
url | string | Qdrant server URL (required) |
apiKeySecretRef | *SecretKeyRef | Qdrant API key Secret reference |
collectionName | string | Collection name for cached embeddings |
passThroughEndpoints
Configure pass-through endpoints to proxy arbitrary API requests to upstream services through LiteLLM. Useful for provider-specific APIs (image generation, fine-tuning, embeddings) that aren't covered by the standard chat/completion routing.
| Field | Type | Default | Description |
|---|---|---|---|
path | string | — | Route path on the LiteLLM proxy (e.g., /bria) (required) |
target | string | — | Target URL to forward requests to (required) |
auth | *bool | — | Enable LiteLLM authentication for this endpoint (enterprise) |
forwardHeaders | *bool | — | Forward incoming client headers to the target |
includeSubpath | *bool | — | Forward requests to sub-paths (e.g., /path/sub → target/sub) |
methods | []string | — | HTTP methods to allow. If empty, all methods are allowed |
headers | map[string]string | — | Static headers to add to forwarded requests |
headerSecrets | []HeaderSecretRef | — | Headers sourced from Kubernetes Secrets |
defaultQueryParams | map[string]string | — | Default query parameters added to all forwarded requests |
HeaderSecretRef:
| Field | Type | Default | Description |
|---|---|---|---|
headerName | string | — | HTTP header name (e.g., Authorization) (required) |
prefix | string | — | Prefix prepended to the secret value (e.g., "Bearer ") |
secretRef | SecretKeyRef | — | Reference to Secret containing the header value (required) |
Secret-backed headers are injected as environment variables named PASSTHROUGH_{PATH}_{HEADER} (uppercase, special characters replaced with _). In the generated config, they appear as os.environ/PASSTHROUGH_... references that LiteLLM resolves at runtime.
secretManager
External secret manager integration. When configured, LiteLLM fetches API keys from the provider at runtime instead of reading them from Kubernetes Secrets.
| Field | Type | Default | Description |
|---|---|---|---|
provider | string | — | Provider name (required). One of: aws_secret_manager, aws_kms, azure_key_vault, google_secret_manager, google_kms, hashicorp_vault |
credentialsSecretRef | *SecretRef | — | Reference to a Secret containing provider authentication fields. Omit when using workload identity (IRSA, GKE WI) |
hostedKeys | []string | — | Env var names LiteLLM should resolve from the secret manager |
storeVirtualKeys | *bool | — | Store generated virtual keys in the secret manager |
prefixForStoredVirtualKeys | string | — | Prefix for stored virtual keys (e.g., litellm/) |
accessMode | string | read_only | Access mode: read_only, write_only, read_and_write |
primarySecretName | string | — | Single secret containing multiple key-value pairs as JSON |
aws | *AWSSecretManagerConfig | — | AWS-specific settings |
azure | *AzureKeyVaultConfig | — | Azure-specific settings |
vault | *VaultConfig | — | HashiCorp Vault-specific settings |
AWSSecretManagerConfig:
| Field | Type | Description |
|---|---|---|
region | string | AWS region (required) |
roleARN | string | IAM role ARN for role assumption |
sessionName | string | Session name for role assumption |
webIdentityTokenPath | string | Path to web identity token file (for IRSA on EKS) |
stsEndpoint | string | Custom STS endpoint (for VPC environments) |
AzureKeyVaultConfig:
| Field | Type | Description |
|---|---|---|
vaultURI | string | Azure Key Vault URI (required) |
tenantID | string | Azure tenant ID (required) |
VaultConfig:
| Field | Type | Default | Description |
|---|---|---|---|
address | string | — | Vault server address (required) |
namespace | string | — | Vault namespace |
authMethod | string | approle | Auth method: approle, tls, or token |
appRoleMountPath | string | — | AppRole mount path |
mountName | string | — | KV engine mount name |
pathPrefix | string | — | Path prefix for secrets |
refreshInterval | *int | — | Cache refresh interval in seconds |
See the Secret Managers guide for full usage examples per provider.
jwtAuth
JWT-based authentication configuration (enterprise). When enabled, LiteLLM validates JWT tokens from identity providers and maps claims to roles, teams, and organizations. Public keys are fetched automatically from the IdP's JWKS endpoint. See the JWT/OAuth2 Auth guide for detailed examples.
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable JWT-based authentication |
publicKeyUrl | string | — | Required — JWKS endpoint the proxy fetches signing keys from → JWT_PUBLIC_KEY_URL env var (comma-separate multiple). Without it, no token can be validated |
issuer | string | — | Expected iss claim → JWT_ISSUER env var |
audience | string | — | Expected aud claim → JWT_AUDIENCE env var |
adminJwtScope | string | — | JWT scope value that grants proxy admin access |
adminAllowedRoutes | []string | — | Routes accessible to admin JWT holders (e.g., openai_routes, info_routes) |
teamIdJwtField | string | — | JWT field containing the team ID |
teamIdsJwtField | string | — | JWT field containing team IDs (array) |
orgIdJwtField | string | — | JWT field containing the organization ID |
userIdJwtField | string | — | JWT field containing the user ID |
userIdUpsert | *bool | — | Auto-create the LiteLLM user on first sight of an unknown JWT user id (user_id_upsert) |
userEmailJwtField | string | — | JWT field containing the user email |
userRoleJwtField | string | — | JWT field containing the user role (single value) |
userRolesJwtField | string | — | JWT claim containing a list of roles (e.g. roles); drives role-based model access |
userAllowedRoles | []string | — | Role values (from userRolesJwtField) that map to internal_user (e.g. ["basic_user"]) |
enforceRbac | *bool | — | Enable JWT role-based access control (rendered under litellm_jwtauth); checks the caller's role against spec.rbac.rolePermissions before allowing model access |
objectIdJwtField | string | — | JWT claim holding an object id (user or team, inferred from role mapping) |
endUserIdJwtField | string | — | JWT field containing the end-user ID |
publicKeyTtl | *int | — | TTL in seconds for caching the public key |
scopeModelMappings | map[string][]string | — | Scope→models (key: JWT scope, value: allowed models). Rendered into scope_mappings. Prefer scopeMappings for new configs |
scopeMappings | []JWTScopeMapping | — | Structured scope→models (and optionally routes) list → scope_mappings |
rolesJwtField | string | — | JWT claim containing the caller's roles (used by roleMappings) → roles_jwt_field |
roleMappings | []JWTRoleMapping | — | Map a JWT role → LiteLLM internal role ({role, internalRole}) → role_mappings |
jwtLitellmRoleMap | []JWTLiteLLMRoleMapEntry | — | Map a JWT role → LiteLLM user role ({jwtRole, litellmRole}) → jwt_litellm_role_map |
teamAliasJwtField | string | — | Look up a team by its name/alias claim → team_alias_jwt_field |
orgAliasJwtField | string | — | Look up an org by its name/alias claim → org_alias_jwt_field |
teamIdDefault | string | — | Fallback team id when the team claim doesn't resolve → team_id_default |
teamAllowedRoutes | []string | — | Route allowlist for team tokens → team_allowed_routes |
teamIdUpsert | *bool | — | Auto-create a team on unknown JWT team id → team_id_upsert |
teamClaimFallback | *bool | — | Fall back to the user's single DB team when the team claim is unresolved → team_claim_fallback |
userAllowedEmailDomain | string | — | Grant access to any caller whose JWT email is in this domain → user_allowed_email_domain |
enforceTeamBasedModelAccess | *bool | — | Deny models unless the team has access → enforce_team_based_model_access |
enforceScopeBasedAccess | *bool | — | Enforce model access via scopeMappings → enforce_scope_based_access |
syncUserRoleAndTeams | *bool | — | Sync the user's role + team membership from the IdP → sync_user_role_and_teams |
customValidate | string | — | Dotted path to a custom JWT-validation function baked into the proxy image → custom_validate |
virtualKeyClaimField | string | — | JWT claim to look up in virtual-key mappings → virtual_key_claim_field |
virtualKeyMappingCacheTtl | *int | — | Cache TTL (s) for the virtual-key lookup → virtual_key_mapping_cache_ttl |
routingOverrides | []JWTRoutingOverride | — | Route JWT-shaped tokens to OAuth2 by claim ({iss, clientId, scope, aud, path}) → routing_overrides |
oidcUserinfoEnabled | *bool | — | Fetch extra claims from the IdP UserInfo endpoint → oidc_userinfo_enabled |
oidcUserinfoEndpoint | string | — | IdP UserInfo endpoint URL → oidc_userinfo_endpoint |
oidcUserinfoCacheTtl | *int | — | Cache TTL (s) for UserInfo responses → oidc_userinfo_cache_ttl |
oauth2Auth
OAuth2 machine-to-machine authentication configuration (enterprise). Maps JWT fields to LiteLLM attributes for service-to-service authentication. See the JWT/OAuth2 Auth guide for detailed examples.
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable OAuth2 machine-to-machine authentication |
configMappings | []OAuth2Mapping | — | Mappings from JWT fields to LiteLLM attributes |
OAuth2Mapping:
| Field | Type | Description |
|---|---|---|
name | string | Identifier for this mapping (required) |
jwtField | string | JWT field to read from (required) |
litellmAttribute | string | LiteLLM attribute to map to, e.g., team_id, user_id (required) |
security
| Field | Type | Default | Description |
|---|---|---|---|
networkPolicy.enabled | bool | false | Enable NetworkPolicy |
networkPolicy.allowedNamespaces | []string | — | Namespaces allowed ingress access |
runAsNonRoot | *bool | — | Run as non-root user (OpenShift compatible) |
ipAllowlist | *IPAllowlistSpec | — | IP address filtering (enterprise) |
IPAllowlistSpec:
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable IP address filtering |
allowedIPs | []string | — | Allowed IP addresses or CIDR ranges (required, min 1) |
useXForwardedFor | *bool | — | Use X-Forwarded-For header for client IP detection. Enable when behind a load balancer |
maxRequestSizeMB | *int | — | Maximum request body size in MB (enterprise) |
maxResponseSizeMB | *int | — | Maximum response body size in MB (enterprise) |
rbac
Role-based access control configuration. Controls route restrictions, key generation permissions, and per-role access. Some features require a LiteLLM Enterprise license. See the RBAC guide for detailed examples.
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable RBAC enforcement (enforce_rbac: true in config) |
adminOnlyRoutes | []string | — | Routes restricted to proxy_admin only |
allowedRoutes | []string | — | Routes accessible to all authenticated users. If set, all other routes are blocked |
defaultTeamDisabled | *bool | — | Prevent personal key creation (force team-based keys) |
keyGeneration | *KeyGenerationSettings | — | Key generation restrictions (enterprise) |
rolePermissions | map[string]RolePermission | — | Per-role permission definitions (enterprise) |
KeyGenerationSettings:
| Field | Type | Description |
|---|---|---|
teamKeyGeneration | *TeamKeyGenerationSettings | Team key generation restrictions |
personalKeyGeneration | *PersonalKeyGenerationSettings | Personal key generation restrictions |
TeamKeyGenerationSettings:
| Field | Type | Description |
|---|---|---|
allowedTeamMemberRoles | []string | Team member roles allowed to generate team keys. Values: admin, user |
PersonalKeyGenerationSettings:
| Field | Type | Description |
|---|---|---|
allowedUserRoles | []string | User roles allowed to generate personal keys. Values: proxy_admin, proxy_admin_viewer, internal_user, internal_user_viewer |
RolePermission:
| Field | Type | Description |
|---|---|---|
routes | []string | API routes this role can access |
models | []string | Models this role can use |
logging
Instance-level logging configuration. Controls audit logs, message content logging, API key redaction, and spend log retention.
| Field | Type | Default | Description |
|---|---|---|---|
auditLogs | *AuditLogSpec | — | Audit log configuration (enterprise) |
turnOffMessageLogging | *bool | — | Disable logging of request/response message content. Only metadata (tokens, cost, model) is logged |
redactUserApiKeyInfo | *bool | — | Redact user API key information from logs |
spendLogRetention | *SpendLogRetentionSpec | — | Spend log retention and cleanup configuration |
AuditLogSpec:
| Field | Type | Default | Description |
|---|---|---|---|
enabled | bool | false | Enable audit logging (enterprise). Writes store_audit_logs: true to general_settings |
retentionDays | *int | — | Audit log retention period in days. Written to litellm_settings.audit_log_retention_days |
SpendLogRetentionSpec:
| Field | Type | Description |
|---|---|---|
maxRetentionPeriod | string | Maximum retention period (e.g., 90d, 1y). Written to general_settings.maximum_spend_logs_retention_period |
cleanupInterval | string | Cleanup interval (e.g., 1d, 1h). Written to general_settings.maximum_spend_logs_retention_interval |
extraEnvVars / extraEnvFrom
Inject additional environment variables into the LiteLLM container.
| Field | Type | Description |
|---|---|---|
extraEnvVars | []corev1.EnvVar | Extra env vars appended to the container. Entries override operator-set env vars of the same name (e.g. PROXY_BASE_URL) |
extraEnvFrom | []corev1.EnvFromSource | Extra envFrom sources (Secret or ConfigMap) mounted into the container |
TIP
extraEnvVars is an escape hatch for env vars the operator derives automatically. A common case is PROXY_BASE_URL: the operator derives it from spec.ingress.host, falling back to the in-cluster Service DNS when ingress is not used. If you expose the gateway through an OpenShift Route, a Gateway API HTTPRoute, or an external load balancer, set PROXY_BASE_URL here to your public URL so SSO redirects land on the correct host.
spec:
extraEnvVars:
- name: PROXY_BASE_URL
value: https://gateway.example.comadminUI
Admin UI configuration. Controls UI availability, access restrictions, model persistence, and personal key creation policy.
| Field | Type | Default | Description |
|---|---|---|---|
disabled | *bool | — | Disable the Admin UI entirely. When true, the /ui endpoint returns 404. Injected as DISABLE_ADMIN_UI env var |
adminOnly | *bool | — | Restrict UI access to admin users only (proxy_admin and proxy_admin_viewer). Sets ui_access_mode: "admin_only" in general_settings |
storeModelInDB | *bool | — | Store model definitions in the database instead of the config file. Enables adding/editing models from the UI without proxy restart. Written to general_settings.store_model_in_db |
defaultTeamDisabled | *bool | — | Prevent users from creating personal API keys. Keys can only be created under an assigned team. Written to general_settings.default_team_disabled |
apiDocBaseURL | string | — | Custom base URL for the API reference documentation shown in the UI. Useful when the Admin UI is served from a different host. Injected as LITELLM_UI_API_DOC_BASE_URL env var |
docsURL | string | — | Custom path for the docs endpoint (default: /). Injected as DOCS_URL env var |
rootRedirectURL | string | — | URL to redirect to when the root path is accessed and docsURL is changed. Injected as ROOT_REDIRECT_URL env var |
logoURL | string | — | URL to a hosted logo image displayed in the Admin UI. Injected as UI_LOGO_PATH env var |
emailLogoURL | string | — | URL to a logo image included in email notifications (budget alerts, invitations). Injected as EMAIL_LOGO_URL env var |
emailSupportContact | string | — | Support email address displayed in email notifications. Injected as EMAIL_SUPPORT_CONTACT env var |
colorThemeConfigMapRef | *ConfigMapRef | — | Reference to a ConfigMap containing enterprise_colors.json with custom brand colors (Tremor palette). Mounted at /app/enterprise/enterprise_ui/enterprise_colors.json |
TIP
storeModelInDB: true is recommended for production multi-replica deployments where models should persist across restarts and be shared across instances.
defaultTeamDisabled: true pairs well with the operator's LiteLLMTeam CRD — it enforces that all keys are team-scoped, making spend tracking and access control more predictable.
allowedNamespaces
Which namespaces may reference this instance from spec.instanceRef (with instanceRef.namespace set), and with which kinds of resource. Only the instance's own namespace may by default. The field follows Gateway API's allowedRoutes.
| Field | Type | Default | Description |
|---|---|---|---|
from | string | Same | Same (only the instance's namespace), All (every namespace), or Selector (namespaces whose labels match selector) |
selector | *LabelSelector | — | Namespace label selector; required when from is Selector, not allowed otherwise |
kinds | []string | [LiteLLMVirtualKey] | Kinds of resource that may reference the instance from an allowed namespace. Setting it replaces the default, so include LiteLLMVirtualKey if keys should still be admitted |
The instance's own namespace is always allowed, for every kind. A reference from a namespace or kind that is not allowed reports Synced=False with reason RefNotPermitted, and nothing is created in LiteLLM. teamRef, userRef and the other references stay in the referencing resource's own namespace, so a LiteLLMVirtualKey with a teamRef needs a team in its own namespace, which means listing LiteLLMTeam too.
Choose kinds by what a tenant may do to the whole proxy
A LiteLLMVirtualKey only spends: it cannot affect other tenants' objects, which is why it is the one kind admitted by default. The others reach further, so list them only for namespaces you trust accordingly:
| Kind | What listing it grants |
|---|---|
LiteLLMUser | Admin-level. Can create a proxy_admin user with an initial password and log in to the Admin UI. Can also set userId to an existing user, including an admin, adopting it, updating it to the CR's spec and resetting its password |
LiteLLMTeam | Admin-level. Can set teamId to another tenant's team, adopting it and bringing its models, budget and members to the CR's spec |
LiteLLMOrganization, LiteLLMBudget, LiteLLMCustomer | Create and manage objects shared across the whole proxy, not scoped to the tenant |
LiteLLMModel, LiteLLMCredential and LiteLLMGuardrail cannot be listed, and reject instanceRef.namespace. Each changes what the proxy does for every tenant: a model registered under an existing name joins that model's load balancing, so it would receive other tenants' prompts.
Taking access away. Changing allowedNamespaces stops the affected resources from syncing, but does not by itself revoke what they already created. A key minted earlier keeps working until its CR is deleted, which removes it from LiteLLM even though the reference is no longer allowed. Config sync then depends on what changed:
| Change | unmanagedResourcePolicy: prune | Other policies |
|---|---|---|
| A namespace is no longer allowed | What its resources created counts as unmanaged and is deleted | Left in place, counted as unmanaged |
| A kind is no longer listed | Left in place and not counted as unmanaged, so editing kinds can never delete keys | Left in place, not counted as unmanaged |
# In namespace llm: one shared gateway
spec:
allowedNamespaces:
from: Selector
selector:
matchLabels:
litellm.palena.ai/tenant: "true"
# kinds defaults to [LiteLLMVirtualKey]
---
# In namespace my-app (labelled litellm.palena.ai/tenant=true)
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMVirtualKey
metadata:
name: my-app
namespace: my-app
spec:
instanceRef:
name: my-gateway
namespace: llm
keyAlias: my-app
maxBudget: "20"
budgetDuration: 30dThe key's Secret (my-app-key) is created in my-app, beside the workload that uses it. The operator needs get/list/watch on Namespaces to evaluate selector; the bundled RBAC grants it.
Status Fields
| Field | Type | Description |
|---|---|---|
ready | bool | Whether the instance is fully ready. Managed: at least one Deployment replica is serving. Unmanaged: the admin API answers at endpoint |
replicas | int32 | Current replica count (0 when workload.managed is false) |
readyReplicas | int32 | Ready replica count (0 when workload.managed is false) |
endpoint | string | Endpoint URL the operator uses to reach the admin API: spec.workload.endpoint when set, otherwise the derived in-cluster Service address |
version | string | Current LiteLLM version. Managed: spec.image.tag. Unmanaged: the litellm_version the proxy reports on /health/readiness, or empty — LiteLLM includes that field only when its general_settings sets allow_public_health_readiness_details: true, and the endpoint is unauthenticated so the master key does not unlock it |
database | DatabaseStatus | Database connection status |
redis | *RedisStatus | Redis connection status |
configSync | *ConfigSyncStatus | Config sync status and counts |
secretManager | *SecretManagerStatus | Secret manager status (configured, provider) |
license | *LicenseStatus | License activation status (active, secretName) |
sso | *SSOStatus | SSO configuration status |
scim | *SCIMStatus | SCIM configuration status |
unhealthyPods | []UnhealthyPod | Why proxy pods aren't running (crash loops, image pull failures, OOM kills, unschedulable). Populated only while not ready; capped at 3 |
conditions | []Condition | Standard Kubernetes conditions |
unhealthyPods
| Field | Type | Description |
|---|---|---|
name | string | Pod name |
phase | string | Pod phase (Pending, Running, Failed, …) |
reason | string | Machine-readable cause: CrashLoopBackOff, ImagePullBackOff, OOMKilled, CreateContainerConfigError, Unschedulable, … |
message | string | Human-readable detail (truncated to 512 chars). For a crash loop this is the previous container termination — e.g. last exit code 3 (Error) … — not the back-off timing |
restartCount | int32 | Container restart count; a climbing value is the signature of a crash loop |
Diagnosing an unready gateway
Pod faults surface as a PodsHealthy condition, independent of Ready (which keeps meaning "at least one replica is serving"):
kubectl get litellminstance my-gateway -o jsonpath='{.status.unhealthyPods[0].reason}{"\n"}'
# CrashLoopBackOff
kubectl describe litellminstance my-gateway | grep -A3 PodsHealthy
# PodsHealthy False CrashLoopBackOff
# 1 unhealthy pod(s); my-gateway-5fdb-4rrvz: CrashLoopBackOff — last exit code 3 (Error) ...A Warning event is also emitted when the cause changes (not on every reconcile, so a crash loop won't spam the event stream).
Print Columns
kubectl get li
NAME READY ENDPOINT VERSION AGE
my-gateway True http://my-gateway.default.svc:4000 main-v1.60.0 5d