Skip to content

LiteLLMInstance ​

The primary CRD. Deploys a LiteLLM proxy with all infrastructure dependencies.

API Version: litellm.palena.ai/v1alpha1Kind: LiteLLMInstanceShort Name: li

Minimal Example ​

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: my-gateway
spec:
  masterKey:
    autoGenerate: true
  database:
    external:
      connectionSecretRef:
        name: litellm-db
        key: DATABASE_URL

Full Example ​

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: my-gateway
spec:
  image:
    repository: ghcr.io/berriai/litellm
    tag: main-v1.60.0
    pullPolicy: IfNotPresent

  replicas: 3

  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilization: 70

  masterKey:
    autoGenerate: true

  database:
    external:
      connectionSecretRef:
        name: litellm-db
        key: DATABASE_URL
    connectionPool:
      maxConnections: 20
    migration:
      enabled: true
      timeout: "300s"

  redis:
    enabled: true
    host: redis.default.svc
    port: 6379
    passwordSecretRef:
      name: redis-credentials
      key: password

  saltKey:
    autoGenerate: true

  service:
    type: ClusterIP
    port: 4000

  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/litellm
    labels:
      example.com/identity: irsa

  deployment:
    annotations:
      reloader.stakater.com/auto: "true"

  podScheduling:
    nodeSelector:
      kubernetes.io/arch: arm64
    tolerations:
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: litellm

  ingress:
    enabled: true
    host: litellm.example.com
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt

  configSync:
    enabled: true
    interval: "30s"
    unmanagedResourcePolicy: preserve
    conflictResolution: crd-wins
    auditChanges: true

  caching:
    enabled: true
    type: redis
    ttl: 600
    namespace: "prod"
    mode: default_on

  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "2"
      memory: 1Gi

  podDisruptionBudget:
    enabled: true
    minAvailable: 1

  security:
    networkPolicy:
      enabled: true
      allowedNamespaces:
        - default
        - production
    ipAllowlist:
      enabled: true
      allowedIPs:
        - "10.0.0.0/8"
        - "192.168.1.0/24"
      useXForwardedFor: true

  generalSettings:
    masterKeyRequired: true
    maxBudget: "10000.00"
    budgetDuration: "30d"
    globalMaxParallelRequests: 100
    budgetReschedulerMinTime: 300
    budgetReschedulerMaxTime: 600

  routerSettings:
    routingStrategy: least-busy
    numRetries: 3
    enableTagFiltering: true
    defaultMaxParallelRequests: 10
    retryPolicy:
      TimeoutError: 2
      RateLimitError: 3
    modelGroupRetryPolicy:
      gpt-4:
        TimeoutError: 1
    providerBudgetConfig:
      openai:
        budgetLimit: "500.00"
        timePeriod: "1d"
      anthropic:
        budgetLimit: "300.00"
        timePeriod: "1d"

  fallbacks:
    defaultFallbacks: ["gpt-4-mini", "claude-3-haiku"]
    modelFallbacks:
      - model: gpt-4
        fallbacks: ["gpt-4-mini", "claude-3-haiku"]
    contentPolicyFallbacks:
      - model: gpt-4
        fallbacks: ["claude-3-sonnet"]
    contextWindowFallbacks:
      - model: gpt-4
        fallbacks: ["gpt-4-32k", "claude-3-sonnet"]
    maxFallbacks: 3

  passThroughEndpoints:
    - path: /bria
      target: https://engine.prod.bria-api.com
      headers:
        content-type: application/json
      headerSecrets:
        - headerName: Authorization
          prefix: "Bearer "
          secretRef:
            name: bria-credentials
            key: api-key
    - path: /langfuse
      target: https://us.cloud.langfuse.com
      auth: true
      forwardHeaders: true
      includeSubpath: true
      methods: ["GET", "POST"]
      defaultQueryParams:
        version: "2"

  jwtAuth:
    enabled: true
    adminJwtScope: "litellm_proxy_admin"
    adminAllowedRoutes:
      - openai_routes
      - info_routes
    teamIdJwtField: "client_id"
    teamIdsJwtField: "groups"
    orgIdJwtField: "org_id"
    userIdJwtField: "sub"
    userEmailJwtField: "email"
    publicKeyTtl: 600

  oauth2Auth:
    enabled: true
    configMappings:
      - name: clientId
        jwtField: "client_id"
        litellmAttribute: "team_id"
      - name: userId
        jwtField: "sub"
        litellmAttribute: "user_id"

  secretManager:
    provider: aws_secret_manager
    credentialsSecretRef:
      name: litellm-aws-credentials
    hostedKeys:
      - OPENAI_API_KEY
      - ANTHROPIC_API_KEY
    storeVirtualKeys: true
    prefixForStoredVirtualKeys: "litellm/"
    accessMode: read_and_write
    aws:
      region: us-east-1

  rbac:
    enabled: true
    adminOnlyRoutes:
      - /model/new
      - /model/delete
      - /organization/new
    allowedRoutes:
      - /chat/completions
      - /embeddings
      - /key/info
      - /user/info
    defaultTeamDisabled: true
    keyGeneration:
      teamKeyGeneration:
        allowedTeamMemberRoles: ["admin"]
      personalKeyGeneration:
        allowedUserRoles: ["proxy_admin"]
    rolePermissions:
      internal_user:
        routes:
          - /key/generate
          - /key/delete
          - /key/info
        models:
          - gpt-4
          - claude-3-haiku

  logging:
    auditLogs:
      enabled: true
      retentionDays: 90
    turnOffMessageLogging: false
    redactUserApiKeyInfo: true
    spendLogRetention:
      maxRetentionPeriod: "90d"
      cleanupInterval: "1d"

  adminUI:
    disabled: false
    adminOnly: true
    storeModelInDB: true
    defaultTeamDisabled: true
    apiDocBaseURL: "https://api.example.com"
    docsURL: "/docs"
    rootRedirectURL: "/ui"
    logoURL: "https://example.com/logo.png"
    emailLogoURL: "https://example.com/email-logo.png"
    emailSupportContact: "support@example.com"
    colorThemeConfigMapRef:
      name: litellm-colors

Spec Fields ​

image ​

FieldTypeDefaultDescription
repositorystringghcr.io/berriai/litellmContainer image repository
tagstringlatestImage tag
pullPolicystringIfNotPresentImage pull policy
pullSecrets[]SecretRef—Image pull secrets

workload ​

Controls whether the operator provisions the proxy workload. Omit the block (the default) and the operator creates and owns the Deployment, Service, ConfigMap and ServiceAccount as it always has.

FieldTypeDefaultDescription
managedbooltrueWhether the operator owns the proxy workload. Set false to attach to a deployment managed elsewhere
endpointstring—Base URL of the existing proxy, e.g. http://litellm.platform.svc:4000. Only valid when managed is false. Defaults to http(s)://<metadata.name>.<namespace>.svc:<service.port>

With managed: false the operator creates nothing and adopts nothing — no existing object is mutated, no owner reference added. It resolves an endpoint and a master key so the entity CRDs (LiteLLMTeam, LiteLLMVirtualKey, LiteLLMBudget, LiteLLMModel, ...) work against a proxy owned by a Helm chart, a GitOps pipeline or an internal platform.

yaml
spec:
  workload:
    managed: false
    endpoint: http://litellm.platform.svc:4000
  masterKey:
    secretRef:
      name: litellm-master-key
      key: LITELLM_MASTER_KEY
  database: {}

Behaviour differences when unmanaged:

Managed (default)Unmanaged
Deployment, Service, ConfigMap, ServiceAccountCreated and reconciledNever touched
Ingress, Route, HTTPRoute, HPA, PDB, NetworkPolicy, ServiceMonitorCreated when enabledNever touched
Auto-rollback (spec.upgrade.autoRollback)ActiveSkipped
Database migration Job (spec.database.migration)Created when enabledNever — the proxy owns its schema
status.readyAt least one Deployment replica readyAdmin API answers /health/liveliness at status.endpoint
status.replicas / readyReplicasDeployment counts0
status.versionspec.image.tag (or latest)Empty, unless the proxy discloses litellm_version on /health/readiness
PodsHealthy conditionSetAbsent — the operator owns no pods
DatabaseReady condition reasonMigrationSkipped / MigrationComplete / …WorkloadUnmanaged
Ready condition reasonAllResourcesReady / DeploymentNotReadyProxyReachable / ProxyNotReachable
WorkloadUnmanaged conditionAbsentSet when the CR also carries spec sections the operator ignores unmanaged (see below)
Health probing, config sync, entity CRDs, finalizer cleanupActiveActive

masterKey.autoGenerate is rejected here by a CEL rule: the Secret it would create is only ever built by the managed-workload reconcile path, so on an unmanaged instance the reference would dangle. Reference the existing proxy's admin key with masterKey.secretRef instead.

A CR copied from a managed instance often keeps fields like sso, caching, rbac, or secretManager set — those only take effect through the ConfigMap a managed workload builds. The WorkloadUnmanaged condition names whichever of these the CR actually has set, so one that never set them stays quiet.

Database fields describe the proxy's own database and are only consumed when building the workload, so database: {} is the normal unmanaged value.

spec.database.migration is ignored entirely. An externally-managed proxy owns its own schema: LiteLLM migrates on startup, and whatever deployed it has its own migration hook, so a second migrator would race the real one. The migration Job also takes its image from spec.image.tag — meaningless for a proxy the operator did not deploy, and defaulting to latest — which would run prisma migrate deploy at an arbitrary schema version against a database the operator does not own. DatabaseReady reports WorkloadUnmanaged, and says so explicitly if you configured a migration anyway.

replicas ​

FieldTypeDefaultDescription
replicasint321Number of proxy replicas

autoscaling ​

FieldTypeDefaultDescription
enabledboolfalseEnable HPA
minReplicasint321Minimum replicas
maxReplicasint32—Maximum replicas (required if enabled)
targetCPUUtilization*int32—Target CPU percentage
targetMemoryUtilization*int32—Target memory percentage

masterKey ​

FieldTypeDefaultDescription
secretRef*SecretKeyRef—Reference to existing master key Secret
autoGenerateboolfalseAuto-generate and store in a Secret

database ​

FieldTypeDescription
external*ExternalDBSpecExternal PostgreSQL connection
cloudnativepg*CloudNativePGSpecCloudNativePG cluster reference
managed*ManagedDBSpecOperator-managed single-pod PostgreSQL
connectionPool*ConnectionPoolSpecConnection pool settings
migration*MigrationSpecDatabase migration settings

database.migration ​

The migration Job runs prisma migrate deploy (applying LiteLLM's versioned migration files in litellm-proxy-extras/migrations) — the same command LiteLLM's own componentized chart uses. Required for LiteLLM v1.86+, which disables schema updates in app pods (see PR #27557).

FieldTypeDefaultDescription
enabledbooltrueRun the migration Job before the proxy starts
timeoutstring"300s"Migration Job timeout
useDatabaseImageboolfalseRun LiteLLM's dedicated litellm-migrations migrations image instead of prisma migrate deploy in the gateway image. When true, only the database image runs — the operator does not also invoke prisma inside the gateway image
databaseImage*DatabaseImageSpec—Override repo/tag/pullPolicy for the database image. Only consulted when useDatabaseImage: true. Repo defaults to ghcr.io/berriai/litellm-migrations; tag defaults to spec.image.tag so versions stay aligned

When to enable useDatabaseImage:

  • You are on LiteLLM v1.86+ and want the maintained recovery logic (P3005 baseline, P3009/P3018 idempotent retries, v2 migration resolver) for free.
  • You are upgrading from a pre-v0.12 operator that ran prisma db push against your DB — the database image's recovery flow heals the resulting _prisma_migrations / schema drift automatically.

Tag availability caveat: as of June 2026, ghcr.io/berriai/litellm-migrations only publishes release-candidate tags (e.g. v1.87.0-rc.1, v1.88.0-rc.1). If your gateway tag isn't published as a migrations tag yet, either override databaseImage.tag to one that exists or stay on the gateway-image path. The gateway-image path now invokes the same ProxyExtrasDBManager.setup_database(use_migrate=True, use_v2_resolver=True) entry the dedicated image uses and works on every LiteLLM v1.85+ gateway tag (verified end-to-end against v1.85.3, v1.86.1, and v1.87.0).

Example:

yaml
spec:
  database:
    external:
      connectionSecretRef:
        name: litellm-db
        key: DATABASE_URL
    migration:
      enabled: true
      useDatabaseImage: true        # run only ghcr.io/berriai/litellm-migrations
      # databaseImage:               # optional override (e.g. internal mirror)
      #   repository: registry.example.com/mirror/litellm-migrations
      #   tag: v1.87.0

podScheduling ​

Node placement applied consistently to the proxy Deployment Pods and the database migration Job Pod. Changing it creates a fresh migration Job because Kubernetes does not allow its Pod template to be modified in place. An omitted podScheduling, an empty podScheduling: {} and empty values are equivalent and leave the Job name unchanged.

FieldTypeDescription
nodeSelectormap[string]stringNode labels required by LiteLLM Pods
tolerations[]TolerationTaints LiteLLM Pods may tolerate
affinity*AffinityNode affinity and pod (anti-)affinity, for rules nodeSelector cannot express: set-based matches, soft/preferred placement, and spreading replicas across nodes or zones

Note that spec.topologySpreadConstraints is configured separately, at the top level of the spec, and applies to the proxy Deployment only — spread constraints are meaningless for the single-Pod migration Job.

Example:

yaml
spec:
  podScheduling:
    nodeSelector:
      kubernetes.io/arch: arm64
    tolerations:
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: litellm

redis ​

FieldTypeDefaultDescription
enabledboolfalseEnable Redis
connectionSecretRef*SecretKeyRef—Redis connection URL Secret
hoststring—Redis host
portint6379Redis port
passwordSecretRef*SecretKeyRef—Redis password Secret

configSync ​

FieldTypeDefaultDescription
enabledboolfalseEnable bidirectional config sync
intervalstring30sSync interval
unmanagedResourcePolicystringpreservePolicy for unmanaged resources: preserve, prune, adopt
conflictResolutionstringcrd-winsConflict strategy: crd-wins, api-wins, manual
auditChangesboolfalseEmit Kubernetes Events for sync changes

service ​

FieldTypeDefaultDescription
typestringClusterIPService type
portint324000Service port

generalSettings ​

FieldTypeDescription
masterKeyRequired*boolRequire master key for all requests
proxyBatchWriteAtintBatch write interval in seconds
alertTypes[]stringAlert types to generate notifications for
alerting[]stringAlert delivery channels (e.g. ["slack"]). Required for alerts to actually fire; for Slack also set SLACK_WEBHOOK_URL via extraEnvVars
alertingThreshold*intSeconds a request may hang before a hanging-request alert fires
alertToWebhookUrlmap[string]stringOverride the delivery webhook per alert type
backgroundHealthChecks*boolEnable periodic background health checks so GET /health returns cached results
healthCheckInterval*intSeconds between background health checks (default 300)
healthCheckDetails*boolWhether GET /health exposes endpoint URLs and error details
customKeyGeneratestringDotted path to a custom key-generation handler
allowUserAuth*boolAllow requests without a key
maxBudget*stringGlobal proxy budget in USD
budgetDurationstringGlobal budget reset duration (e.g., 1d, 7d, 30d)
globalMaxParallelRequests*intMaximum parallel requests across the entire proxy
budgetReschedulerMinTime*intMinimum interval (seconds) between budget reset checks
budgetReschedulerMaxTime*intMaximum interval (seconds) between budget reset checks
extramap[string]JSONArbitrary keys passed straight through to general_settings (see Raw settings passthrough)

routerSettings ​

FieldTypeDefaultDescription
routingStrategystring—simple-shuffle, least-busy, latency-based-routing, usage-based-routing, usage-based-routing-v2, cost-based-routing
numRetries*int—Number of retries
timeout*int—Timeout in seconds
streamTimeout*int—Timeout in seconds for streaming responses (distinct from timeout)
retryAfter*int—Seconds to wait before a retry
cooldownTime*int—Cooldown time in seconds
enablePreCallChecks*bool—Filter deployments by context-window size and region before calling
modelGroupAliasmap[string]string—Map an alias model-group name to a real one
retryPolicymap[string]int—Per-error-type retry counts (e.g., TimeoutError: 2, RateLimitError: 3)
modelGroupRetryPolicymap[string]map[string]int—Per-model-group retry overrides (e.g., gpt-4: {TimeoutError: 1})
enableTagFiltering*bool—Enable tag-based routing. Requests with matching tags route to tagged model deployments
tagFilteringMatchAny*bool—If true, match ANY tag (OR logic). If false (default), ALL tags must match (AND logic)
defaultMaxParallelRequests*int—Default max parallel requests per model deployment
providerBudgetConfigmap[string]ProviderBudget—Per-provider spending limits

ProviderBudget:

FieldTypeDescription
budgetLimitstringBudget limit in USD
timePeriodstringTime period for the budget (e.g., 1d, 7d, 30d)

litellmSettings ​

Settings rendered under litellm_settings in proxy_server_config.yaml.

FieldTypeDescription
jsonLogs*boolEmit structured JSON logs instead of plaintext (recommended for log aggregation)
checkProviderEndpoint*boolQuery each provider's own /models endpoint when serving GET /v1/models, so a wildcard deployment lists the upstream's real model names instead of the literal pattern
extramap[string]JSONArbitrary keys passed straight through to litellm_settings (see Raw settings passthrough)

Wildcard model discovery ​

checkProviderEndpoint maps to litellm_settings.check_provider_endpoint. Without it, a wildcard deployment such as up/* is listed either as the literal pattern or expanded from LiteLLM's static built-in model list for that provider. With it, the proxy calls the upstream's /models endpoint and lists what is actually there:

yaml
spec:
  litellmSettings:
    checkProviderEndpoint: true
yaml
# LiteLLMModel
spec:
  modelName: "up/*"
  litellmParams:
    model: "openai/*"
    apiBase: https://upstream.example.com/v1
    apiKeySecretRef: {name: upstream-key, key: api_key}

It applies to models registered through the API by LiteLLMModel as well as to any declared in the config file.

Two caveats:

  • It is a global switch. Every wildcard deployment on the proxy is discovered this way, and each discovery reaches out to the upstream (LiteLLM caches results).
  • It only works for providers in LiteLLM's models_by_provider map. Notably litellm_proxy/* is not one of them — a wildcard backed by litellm_proxy/* stays unexpanded no matter what this flag is set to. To front another LiteLLM proxy, use openai/* with the upstream's /v1 base URL, as above; a LiteLLM proxy is OpenAI-compatible.

Raw settings passthrough ​

generalSettings.extra and litellmSettings.extra pass arbitrary keys straight into their config sections, as an escape hatch for LiteLLM settings this CRD does not model yet:

yaml
spec:
  litellmSettings:
    extra:
      some_new_upstream_setting: true
      nested_value: {a: [1, 2]}

Anything the operator derives from the rest of the spec always wins: a colliding key here is ignored, and the instance emits an ExtraSettingsIgnored warning event naming it. Prefer a typed field where one exists — settings placed here are neither validated nor documented by the operator, and a later release may add a typed field that starts taking precedence over your entry.

fallbacks ​

Configure model fallback chains. Fallback model names must match model_name values from your LiteLLMModel resources.

FieldTypeDefaultDescription
defaultFallbacks[]string—Global fallback models applied on any error
modelFallbacks[]ModelFallbackEntry—Per-model fallback chains for general errors
contentPolicyFallbacks[]ModelFallbackEntry—Fallbacks for content policy violations
contextWindowFallbacks[]ModelFallbackEntry—Fallbacks for context window exceeded errors
maxFallbacks*int3Maximum number of fallback attempts

ModelFallbackEntry:

FieldTypeDescription
modelstringPrimary model name
fallbacks[]stringOrdered list of fallback model names

caching ​

Response caching configuration. See the Caching guide for detailed examples.

FieldTypeDefaultDescription
enabledboolfalseEnable response caching
typestringredisCache backend: redis, redis-semantic, s3, gcs, qdrant, local
ttl*int600Cache TTL in seconds
namespacestring—Cache key namespace for isolation
supportedCallTypes[]string—Restrict caching to specific call types
modestringdefault_ondefault_on (cache all) or default_off (require opt-in)
redis*CacheRedisSpec—Redis-specific config (omit to reuse spec.redis)
s3*CacheS3Spec—S3 backend config
gcs*CacheGCSSpec—GCS backend config
qdrant*CacheQdrantSpec—Qdrant semantic cache config

CacheRedisSpec:

FieldTypeDefaultDescription
hoststring—Redis host
port*int—Redis port
passwordSecretRef*SecretKeyRef—Redis password Secret reference
sslboolfalseEnable SSL/TLS

CacheS3Spec:

FieldTypeDescription
bucketNamestringS3 bucket name (required)
regionstringAWS region
credentialsSecretRef*SecretKeyRefSecret with aws_access_key_id and aws_secret_access_key keys

CacheGCSSpec:

FieldTypeDescription
bucketNamestringGCS bucket name (required)
credentialsSecretRef*SecretKeyRefSecret with GCS service account JSON

CacheQdrantSpec:

FieldTypeDescription
urlstringQdrant server URL (required)
apiKeySecretRef*SecretKeyRefQdrant API key Secret reference
collectionNamestringCollection name for cached embeddings

passThroughEndpoints ​

Configure pass-through endpoints to proxy arbitrary API requests to upstream services through LiteLLM. Useful for provider-specific APIs (image generation, fine-tuning, embeddings) that aren't covered by the standard chat/completion routing.

FieldTypeDefaultDescription
pathstring—Route path on the LiteLLM proxy (e.g., /bria) (required)
targetstring—Target URL to forward requests to (required)
auth*bool—Enable LiteLLM authentication for this endpoint (enterprise)
forwardHeaders*bool—Forward incoming client headers to the target
includeSubpath*bool—Forward requests to sub-paths (e.g., /path/sub → target/sub)
methods[]string—HTTP methods to allow. If empty, all methods are allowed
headersmap[string]string—Static headers to add to forwarded requests
headerSecrets[]HeaderSecretRef—Headers sourced from Kubernetes Secrets
defaultQueryParamsmap[string]string—Default query parameters added to all forwarded requests

HeaderSecretRef:

FieldTypeDefaultDescription
headerNamestring—HTTP header name (e.g., Authorization) (required)
prefixstring—Prefix prepended to the secret value (e.g., "Bearer ")
secretRefSecretKeyRef—Reference to Secret containing the header value (required)

Secret-backed headers are injected as environment variables named PASSTHROUGH_{PATH}_{HEADER} (uppercase, special characters replaced with _). In the generated config, they appear as os.environ/PASSTHROUGH_... references that LiteLLM resolves at runtime.

secretManager ​

External secret manager integration. When configured, LiteLLM fetches API keys from the provider at runtime instead of reading them from Kubernetes Secrets.

FieldTypeDefaultDescription
providerstring—Provider name (required). One of: aws_secret_manager, aws_kms, azure_key_vault, google_secret_manager, google_kms, hashicorp_vault
credentialsSecretRef*SecretRef—Reference to a Secret containing provider authentication fields. Omit when using workload identity (IRSA, GKE WI)
hostedKeys[]string—Env var names LiteLLM should resolve from the secret manager
storeVirtualKeys*bool—Store generated virtual keys in the secret manager
prefixForStoredVirtualKeysstring—Prefix for stored virtual keys (e.g., litellm/)
accessModestringread_onlyAccess mode: read_only, write_only, read_and_write
primarySecretNamestring—Single secret containing multiple key-value pairs as JSON
aws*AWSSecretManagerConfig—AWS-specific settings
azure*AzureKeyVaultConfig—Azure-specific settings
vault*VaultConfig—HashiCorp Vault-specific settings

AWSSecretManagerConfig:

FieldTypeDescription
regionstringAWS region (required)
roleARNstringIAM role ARN for role assumption
sessionNamestringSession name for role assumption
webIdentityTokenPathstringPath to web identity token file (for IRSA on EKS)
stsEndpointstringCustom STS endpoint (for VPC environments)

AzureKeyVaultConfig:

FieldTypeDescription
vaultURIstringAzure Key Vault URI (required)
tenantIDstringAzure tenant ID (required)

VaultConfig:

FieldTypeDefaultDescription
addressstring—Vault server address (required)
namespacestring—Vault namespace
authMethodstringapproleAuth method: approle, tls, or token
appRoleMountPathstring—AppRole mount path
mountNamestring—KV engine mount name
pathPrefixstring—Path prefix for secrets
refreshInterval*int—Cache refresh interval in seconds

See the Secret Managers guide for full usage examples per provider.

jwtAuth ​

JWT-based authentication configuration (enterprise). When enabled, LiteLLM validates JWT tokens from identity providers and maps claims to roles, teams, and organizations. Public keys are fetched automatically from the IdP's JWKS endpoint. See the JWT/OAuth2 Auth guide for detailed examples.

FieldTypeDefaultDescription
enabledboolfalseEnable JWT-based authentication
publicKeyUrlstring—Required — JWKS endpoint the proxy fetches signing keys from → JWT_PUBLIC_KEY_URL env var (comma-separate multiple). Without it, no token can be validated
issuerstring—Expected iss claim → JWT_ISSUER env var
audiencestring—Expected aud claim → JWT_AUDIENCE env var
adminJwtScopestring—JWT scope value that grants proxy admin access
adminAllowedRoutes[]string—Routes accessible to admin JWT holders (e.g., openai_routes, info_routes)
teamIdJwtFieldstring—JWT field containing the team ID
teamIdsJwtFieldstring—JWT field containing team IDs (array)
orgIdJwtFieldstring—JWT field containing the organization ID
userIdJwtFieldstring—JWT field containing the user ID
userIdUpsert*bool—Auto-create the LiteLLM user on first sight of an unknown JWT user id (user_id_upsert)
userEmailJwtFieldstring—JWT field containing the user email
userRoleJwtFieldstring—JWT field containing the user role (single value)
userRolesJwtFieldstring—JWT claim containing a list of roles (e.g. roles); drives role-based model access
userAllowedRoles[]string—Role values (from userRolesJwtField) that map to internal_user (e.g. ["basic_user"])
enforceRbac*bool—Enable JWT role-based access control (rendered under litellm_jwtauth); checks the caller's role against spec.rbac.rolePermissions before allowing model access
objectIdJwtFieldstring—JWT claim holding an object id (user or team, inferred from role mapping)
endUserIdJwtFieldstring—JWT field containing the end-user ID
publicKeyTtl*int—TTL in seconds for caching the public key
scopeModelMappingsmap[string][]string—Scope→models (key: JWT scope, value: allowed models). Rendered into scope_mappings. Prefer scopeMappings for new configs
scopeMappings[]JWTScopeMapping—Structured scope→models (and optionally routes) list → scope_mappings
rolesJwtFieldstring—JWT claim containing the caller's roles (used by roleMappings) → roles_jwt_field
roleMappings[]JWTRoleMapping—Map a JWT role → LiteLLM internal role ({role, internalRole}) → role_mappings
jwtLitellmRoleMap[]JWTLiteLLMRoleMapEntry—Map a JWT role → LiteLLM user role ({jwtRole, litellmRole}) → jwt_litellm_role_map
teamAliasJwtFieldstring—Look up a team by its name/alias claim → team_alias_jwt_field
orgAliasJwtFieldstring—Look up an org by its name/alias claim → org_alias_jwt_field
teamIdDefaultstring—Fallback team id when the team claim doesn't resolve → team_id_default
teamAllowedRoutes[]string—Route allowlist for team tokens → team_allowed_routes
teamIdUpsert*bool—Auto-create a team on unknown JWT team id → team_id_upsert
teamClaimFallback*bool—Fall back to the user's single DB team when the team claim is unresolved → team_claim_fallback
userAllowedEmailDomainstring—Grant access to any caller whose JWT email is in this domain → user_allowed_email_domain
enforceTeamBasedModelAccess*bool—Deny models unless the team has access → enforce_team_based_model_access
enforceScopeBasedAccess*bool—Enforce model access via scopeMappings → enforce_scope_based_access
syncUserRoleAndTeams*bool—Sync the user's role + team membership from the IdP → sync_user_role_and_teams
customValidatestring—Dotted path to a custom JWT-validation function baked into the proxy image → custom_validate
virtualKeyClaimFieldstring—JWT claim to look up in virtual-key mappings → virtual_key_claim_field
virtualKeyMappingCacheTtl*int—Cache TTL (s) for the virtual-key lookup → virtual_key_mapping_cache_ttl
routingOverrides[]JWTRoutingOverride—Route JWT-shaped tokens to OAuth2 by claim ({iss, clientId, scope, aud, path}) → routing_overrides
oidcUserinfoEnabled*bool—Fetch extra claims from the IdP UserInfo endpoint → oidc_userinfo_enabled
oidcUserinfoEndpointstring—IdP UserInfo endpoint URL → oidc_userinfo_endpoint
oidcUserinfoCacheTtl*int—Cache TTL (s) for UserInfo responses → oidc_userinfo_cache_ttl

oauth2Auth ​

OAuth2 machine-to-machine authentication configuration (enterprise). Maps JWT fields to LiteLLM attributes for service-to-service authentication. See the JWT/OAuth2 Auth guide for detailed examples.

FieldTypeDefaultDescription
enabledboolfalseEnable OAuth2 machine-to-machine authentication
configMappings[]OAuth2Mapping—Mappings from JWT fields to LiteLLM attributes

OAuth2Mapping:

FieldTypeDescription
namestringIdentifier for this mapping (required)
jwtFieldstringJWT field to read from (required)
litellmAttributestringLiteLLM attribute to map to, e.g., team_id, user_id (required)

security ​

FieldTypeDefaultDescription
networkPolicy.enabledboolfalseEnable NetworkPolicy
networkPolicy.allowedNamespaces[]string—Namespaces allowed ingress access
runAsNonRoot*bool—Run as non-root user (OpenShift compatible)
ipAllowlist*IPAllowlistSpec—IP address filtering (enterprise)

IPAllowlistSpec:

FieldTypeDefaultDescription
enabledboolfalseEnable IP address filtering
allowedIPs[]string—Allowed IP addresses or CIDR ranges (required, min 1)
useXForwardedFor*bool—Use X-Forwarded-For header for client IP detection. Enable when behind a load balancer
maxRequestSizeMB*int—Maximum request body size in MB (enterprise)
maxResponseSizeMB*int—Maximum response body size in MB (enterprise)

rbac ​

Role-based access control configuration. Controls route restrictions, key generation permissions, and per-role access. Some features require a LiteLLM Enterprise license. See the RBAC guide for detailed examples.

FieldTypeDefaultDescription
enabledboolfalseEnable RBAC enforcement (enforce_rbac: true in config)
adminOnlyRoutes[]string—Routes restricted to proxy_admin only
allowedRoutes[]string—Routes accessible to all authenticated users. If set, all other routes are blocked
defaultTeamDisabled*bool—Prevent personal key creation (force team-based keys)
keyGeneration*KeyGenerationSettings—Key generation restrictions (enterprise)
rolePermissionsmap[string]RolePermission—Per-role permission definitions (enterprise)

KeyGenerationSettings:

FieldTypeDescription
teamKeyGeneration*TeamKeyGenerationSettingsTeam key generation restrictions
personalKeyGeneration*PersonalKeyGenerationSettingsPersonal key generation restrictions

TeamKeyGenerationSettings:

FieldTypeDescription
allowedTeamMemberRoles[]stringTeam member roles allowed to generate team keys. Values: admin, user

PersonalKeyGenerationSettings:

FieldTypeDescription
allowedUserRoles[]stringUser roles allowed to generate personal keys. Values: proxy_admin, proxy_admin_viewer, internal_user, internal_user_viewer

RolePermission:

FieldTypeDescription
routes[]stringAPI routes this role can access
models[]stringModels this role can use

logging ​

Instance-level logging configuration. Controls audit logs, message content logging, API key redaction, and spend log retention.

FieldTypeDefaultDescription
auditLogs*AuditLogSpec—Audit log configuration (enterprise)
turnOffMessageLogging*bool—Disable logging of request/response message content. Only metadata (tokens, cost, model) is logged
redactUserApiKeyInfo*bool—Redact user API key information from logs
spendLogRetention*SpendLogRetentionSpec—Spend log retention and cleanup configuration

AuditLogSpec:

FieldTypeDefaultDescription
enabledboolfalseEnable audit logging (enterprise). Writes store_audit_logs: true to general_settings
retentionDays*int—Audit log retention period in days. Written to litellm_settings.audit_log_retention_days

SpendLogRetentionSpec:

FieldTypeDescription
maxRetentionPeriodstringMaximum retention period (e.g., 90d, 1y). Written to general_settings.maximum_spend_logs_retention_period
cleanupIntervalstringCleanup interval (e.g., 1d, 1h). Written to general_settings.maximum_spend_logs_retention_interval

extraEnvVars / extraEnvFrom ​

Inject additional environment variables into the LiteLLM container.

FieldTypeDescription
extraEnvVars[]corev1.EnvVarExtra env vars appended to the container. Entries override operator-set env vars of the same name (e.g. PROXY_BASE_URL)
extraEnvFrom[]corev1.EnvFromSourceExtra envFrom sources (Secret or ConfigMap) mounted into the container

TIP

extraEnvVars is an escape hatch for env vars the operator derives automatically. A common case is PROXY_BASE_URL: the operator derives it from spec.ingress.host, falling back to the in-cluster Service DNS when ingress is not used. If you expose the gateway through an OpenShift Route, a Gateway API HTTPRoute, or an external load balancer, set PROXY_BASE_URL here to your public URL so SSO redirects land on the correct host.

yaml
spec:
  extraEnvVars:
    - name: PROXY_BASE_URL
      value: https://gateway.example.com

adminUI ​

Admin UI configuration. Controls UI availability, access restrictions, model persistence, and personal key creation policy.

FieldTypeDefaultDescription
disabled*bool—Disable the Admin UI entirely. When true, the /ui endpoint returns 404. Injected as DISABLE_ADMIN_UI env var
adminOnly*bool—Restrict UI access to admin users only (proxy_admin and proxy_admin_viewer). Sets ui_access_mode: "admin_only" in general_settings
storeModelInDB*bool—Store model definitions in the database instead of the config file. Enables adding/editing models from the UI without proxy restart. Written to general_settings.store_model_in_db
defaultTeamDisabled*bool—Prevent users from creating personal API keys. Keys can only be created under an assigned team. Written to general_settings.default_team_disabled
apiDocBaseURLstring—Custom base URL for the API reference documentation shown in the UI. Useful when the Admin UI is served from a different host. Injected as LITELLM_UI_API_DOC_BASE_URL env var
docsURLstring—Custom path for the docs endpoint (default: /). Injected as DOCS_URL env var
rootRedirectURLstring—URL to redirect to when the root path is accessed and docsURL is changed. Injected as ROOT_REDIRECT_URL env var
logoURLstring—URL to a hosted logo image displayed in the Admin UI. Injected as UI_LOGO_PATH env var
emailLogoURLstring—URL to a logo image included in email notifications (budget alerts, invitations). Injected as EMAIL_LOGO_URL env var
emailSupportContactstring—Support email address displayed in email notifications. Injected as EMAIL_SUPPORT_CONTACT env var
colorThemeConfigMapRef*ConfigMapRef—Reference to a ConfigMap containing enterprise_colors.json with custom brand colors (Tremor palette). Mounted at /app/enterprise/enterprise_ui/enterprise_colors.json

TIP

storeModelInDB: true is recommended for production multi-replica deployments where models should persist across restarts and be shared across instances.

defaultTeamDisabled: true pairs well with the operator's LiteLLMTeam CRD — it enforces that all keys are team-scoped, making spend tracking and access control more predictable.

allowedNamespaces ​

Which namespaces may reference this instance from spec.instanceRef (with instanceRef.namespace set), and with which kinds of resource. Only the instance's own namespace may by default. The field follows Gateway API's allowedRoutes.

FieldTypeDefaultDescription
fromstringSameSame (only the instance's namespace), All (every namespace), or Selector (namespaces whose labels match selector)
selector*LabelSelector—Namespace label selector; required when from is Selector, not allowed otherwise
kinds[]string[LiteLLMVirtualKey]Kinds of resource that may reference the instance from an allowed namespace. Setting it replaces the default, so include LiteLLMVirtualKey if keys should still be admitted

The instance's own namespace is always allowed, for every kind. A reference from a namespace or kind that is not allowed reports Synced=False with reason RefNotPermitted, and nothing is created in LiteLLM. teamRef, userRef and the other references stay in the referencing resource's own namespace, so a LiteLLMVirtualKey with a teamRef needs a team in its own namespace, which means listing LiteLLMTeam too.

Choose kinds by what a tenant may do to the whole proxy

A LiteLLMVirtualKey only spends: it cannot affect other tenants' objects, which is why it is the one kind admitted by default. The others reach further, so list them only for namespaces you trust accordingly:

KindWhat listing it grants
LiteLLMUserAdmin-level. Can create a proxy_admin user with an initial password and log in to the Admin UI. Can also set userId to an existing user, including an admin, adopting it, updating it to the CR's spec and resetting its password
LiteLLMTeamAdmin-level. Can set teamId to another tenant's team, adopting it and bringing its models, budget and members to the CR's spec
LiteLLMOrganization, LiteLLMBudget, LiteLLMCustomerCreate and manage objects shared across the whole proxy, not scoped to the tenant

LiteLLMModel, LiteLLMCredential and LiteLLMGuardrail cannot be listed, and reject instanceRef.namespace. Each changes what the proxy does for every tenant: a model registered under an existing name joins that model's load balancing, so it would receive other tenants' prompts.

Taking access away. Changing allowedNamespaces stops the affected resources from syncing, but does not by itself revoke what they already created. A key minted earlier keeps working until its CR is deleted, which removes it from LiteLLM even though the reference is no longer allowed. Config sync then depends on what changed:

ChangeunmanagedResourcePolicy: pruneOther policies
A namespace is no longer allowedWhat its resources created counts as unmanaged and is deletedLeft in place, counted as unmanaged
A kind is no longer listedLeft in place and not counted as unmanaged, so editing kinds can never delete keysLeft in place, not counted as unmanaged
yaml
# In namespace llm: one shared gateway
spec:
  allowedNamespaces:
    from: Selector
    selector:
      matchLabels:
        litellm.palena.ai/tenant: "true"
    # kinds defaults to [LiteLLMVirtualKey]
---
# In namespace my-app (labelled litellm.palena.ai/tenant=true)
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMVirtualKey
metadata:
  name: my-app
  namespace: my-app
spec:
  instanceRef:
    name: my-gateway
    namespace: llm
  keyAlias: my-app
  maxBudget: "20"
  budgetDuration: 30d

The key's Secret (my-app-key) is created in my-app, beside the workload that uses it. The operator needs get/list/watch on Namespaces to evaluate selector; the bundled RBAC grants it.

Status Fields ​

FieldTypeDescription
readyboolWhether the instance is fully ready. Managed: at least one Deployment replica is serving. Unmanaged: the admin API answers at endpoint
replicasint32Current replica count (0 when workload.managed is false)
readyReplicasint32Ready replica count (0 when workload.managed is false)
endpointstringEndpoint URL the operator uses to reach the admin API: spec.workload.endpoint when set, otherwise the derived in-cluster Service address
versionstringCurrent LiteLLM version. Managed: spec.image.tag. Unmanaged: the litellm_version the proxy reports on /health/readiness, or empty — LiteLLM includes that field only when its general_settings sets allow_public_health_readiness_details: true, and the endpoint is unauthenticated so the master key does not unlock it
databaseDatabaseStatusDatabase connection status
redis*RedisStatusRedis connection status
configSync*ConfigSyncStatusConfig sync status and counts
secretManager*SecretManagerStatusSecret manager status (configured, provider)
license*LicenseStatusLicense activation status (active, secretName)
sso*SSOStatusSSO configuration status
scim*SCIMStatusSCIM configuration status
unhealthyPods[]UnhealthyPodWhy proxy pods aren't running (crash loops, image pull failures, OOM kills, unschedulable). Populated only while not ready; capped at 3
conditions[]ConditionStandard Kubernetes conditions

unhealthyPods ​

FieldTypeDescription
namestringPod name
phasestringPod phase (Pending, Running, Failed, …)
reasonstringMachine-readable cause: CrashLoopBackOff, ImagePullBackOff, OOMKilled, CreateContainerConfigError, Unschedulable, …
messagestringHuman-readable detail (truncated to 512 chars). For a crash loop this is the previous container termination — e.g. last exit code 3 (Error) … — not the back-off timing
restartCountint32Container restart count; a climbing value is the signature of a crash loop

Diagnosing an unready gateway ​

Pod faults surface as a PodsHealthy condition, independent of Ready (which keeps meaning "at least one replica is serving"):

bash
kubectl get litellminstance my-gateway -o jsonpath='{.status.unhealthyPods[0].reason}{"\n"}'
# CrashLoopBackOff

kubectl describe litellminstance my-gateway | grep -A3 PodsHealthy
# PodsHealthy   False   CrashLoopBackOff
#   1 unhealthy pod(s); my-gateway-5fdb-4rrvz: CrashLoopBackOff — last exit code 3 (Error) ...

A Warning event is also emitted when the cause changes (not on every reconcile, so a crash loop won't spam the event stream).

bash
kubectl get li
NAME         READY   ENDPOINT                              VERSION          AGE
my-gateway   True    http://my-gateway.default.svc:4000    main-v1.60.0     5d

Released under the Apache 2.0 License.