Skip to content

LiteLLMModel ​

Registers an AI model with a LiteLLM proxy instance. The operator syncs the model to the LiteLLM API via POST /model/new and POST /model/update.

API Version: litellm.palena.ai/v1alpha1Kind: LiteLLMModelShort Name: lm

Example ​

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMModel
metadata:
  name: gpt4o
spec:
  instanceRef:
    name: my-gateway
  modelName: gpt-4o
  litellmParams:
    model: openai/gpt-4o
    apiKeySecretRef:
      name: openai-credentials
      key: OPENAI_API_KEY
    rpm: 500
    tpm: 100000
    timeout: 60
  modelInfo:
    maxTokens: 128000
    inputCostPerToken: 0.0000025
    outputCostPerToken: 0.00001
  tags: ["paid"]

Spec Fields ​

FieldTypeRequiredDescription
instanceRefInstanceRefYesReference to the LiteLLMInstance
modelNamestringYesModel name exposed to clients
litellmParamsLiteLLMModelParamsYesProvider-specific parameters
modelInfo*ModelInfoNoOptional model metadata
tags[]stringNoTags for tag-based routing (requires enableTagFiltering on the instance)

instanceRef.namespace is rejected on a model. A model changes what the whole proxy does for every tenant, so it must live in its instance's namespace; spec.allowedNamespaces on the instance does not apply to it.

litellmParams ​

FieldTypeRequiredDescription
modelstringYesProvider/model string (e.g., openai/gpt-4o, anthropic/claude-sonnet-4-20250514)
credentialRef*CredentialRefNoReference to a LiteLLMCredential in the same namespace. Takes precedence over inline apiBase / apiVersion / apiKeySecretRef
apiBasestringNoAPI base URL for the provider. Ignored if credentialRef is set
apiVersionstringNoProvider API version (e.g., 2024-10-21 for Azure OpenAI / Azure AI Foundry). Ignored if credentialRef is set
apiKeySecretRef*SecretKeyRefNoSecret containing the provider API key. Ignored if credentialRef is set
rpm*intNoRequests per minute limit
tpm*intNoTokens per minute limit
timeout*intNoRequest timeout in seconds
streamTimeout*intNoStream timeout in seconds
maxRetries*intNoMax retries for failed requests
weight*intNoWeighted load-balancing weight across deployments in this model group
order*intNoRouting priority within the model group (lower is preferred; higher-order deployments act as fallbacks)
maxInputTokens*intNoContext-window size used for context-window-aware routing / fallbacks
temperature*float64NoDefault temperature applied to requests to this deployment
topP*float64NoDefault top_p applied to requests to this deployment
maxTokens*intNoDefault max_tokens (completion tokens) sent to the provider. Distinct from modelInfo.maxTokens
seed*intNoDefault seed for reproducible outputs (providers that support it)
organizationstringNoProvider organization ID (e.g. an OpenAI organization)
awsRegionNamestringNoAWS region for Bedrock / SageMaker (e.g. us-east-1)
extraHeadersmap[string]stringNoExtra HTTP headers sent on every upstream request to this deployment
dropParams*boolNoSilently drop request params the provider rejects (e.g. temperature on reasoning models) instead of erroring
vertexProjectstringNoGCP project ID for Vertex AI models
vertexLocationstringNoGCP region for Vertex AI models (e.g. us-central1)
vertexCredentialsSecretRef*SecretKeyRefNoSecret key holding the GCP service-account JSON. The operator reads it and sends vertex_credentials, never logging the value. Ignored when credentialRef is set

Using a shared credential ​

Instead of repeating apiKeySecretRef and apiBase in every model, declare a LiteLLMCredential once and reference it by name:

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMModel
metadata:
  name: gpt4o
spec:
  instanceRef:
    name: my-gateway
  modelName: gpt-4o
  litellmParams:
    model: openai/gpt-4o
    credentialRef:
      name: openai-prod   # → LiteLLMCredential in the same namespace

The operator resolves the credential's api_base / api_version / api_key and writes them inline on the model registered with LiteLLM (and also sends litellm_credential_name: openai-prod for Admin UI association). Resolving inline is deliberate: LiteLLM's request-time resolution of a named credential into a DB-stored model is unreliable on a cold start — a model registered with only litellm_credential_name can come up with no api_base, which breaks Azure deployments. Inline values are restart-safe and always win. See LiteLLMCredential for how credentials are declared.

Azure

For Azure OpenAI / Azure AI Foundry, set apiVersion (e.g. 2024-10-21) on the credential — or inline on the model when not using credentialRef. Without it, the deployment falls back to LiteLLM's default API version, which many Azure models reject.

modelInfo ​

FieldTypeDescription
maxTokens*intMaximum tokens supported
inputCostPerToken*float64Input cost per token in USD
outputCostPerToken*float64Output cost per token in USD
modestringModel type so LiteLLM runs the correct health check / routing. Common values: chat, completion, embedding, image_generation, audio_transcription, audio_speech, moderation, rerank, responses, batch, realtime
baseModelstringMaps this deployment to a known base model for accurate cost tracking. Required for Azure deployments where the deployment name differs from the underlying model
tierstringTier for tier-based routing (free / paid)
regionNamestringRegion for region-based routing (e.g. us-east-1)
accessGroups[]stringAccess groups a key/team must hold to route to this model
supportedEnvironments[]stringEnvironments that expose this deployment (e.g. production, staging, development)
useInPassThrough*boolAllow this deployment to be selected by pass-through endpoints
inputCostPerPixel*float64Cost per pixel in USD for image models
inputCostPerSecond*float64Cost per second in USD for audio / realtime models billed by duration
cacheReadInputTokenCost*float64Cost per token in USD for provider prompt-cache reads
cacheCreationInputTokenCost*float64Cost per token in USD for provider prompt-cache writes
healthCheck*ModelHealthCheckPer-model health-check tuning / disabling (see below)

modelInfo.healthCheck ​

Tunes or disables LiteLLM's health checks for this single deployment. All fields are optional; unset fields fall back to LiteLLM's defaults. These map to the health-check keys under model_info in proxy_server_config.yaml.

FieldTypeDescription
disableBackgroundHealthCheck*boolSkip background health checks for this model (proxy runs with background_health_checks enabled). Useful for providers that bill/rate-limit probes, or models that reject the probe request shape
timeoutSeconds*intHealth-check request timeout override (LiteLLM default 60s)
maxTokens*intmax_tokens used for the health-check request
maxTokensReasoning*intHealth-check max_tokens for reasoning models
maxTokensNonReasoning*intHealth-check max_tokens for non-reasoning models
supportsMaxTokens*boolSet to false to leave max_tokens out of the health-check request (see below). Requires LiteLLM v1.86.0 or later
reasoningEffortstringReasoning effort for reasoning-model probes (e.g. none, low, medium, high)
voicestringVoice for text-to-speech model probes (e.g. alloy)
modelstringOverride the model used for the probe (for wildcard routes, e.g. openai/gpt-4o-mini)

Fixing Unsupported parameter: 'max_tokens' in health checks ​

Newer OpenAI models reject max_tokens and require max_completion_tokens. LiteLLM translates between them only for models it recognizes, so a health check can fail with this error when litellmParams.model is a custom Azure deployment name or an OpenAI-compatible alias, even though the model itself is fine:

text
OpenAIException - Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.

Set supportsMaxTokens: false to leave the parameter out of the probe:

yaml
spec:
  modelName: gpt-6-luna
  litellmParams:
    model: azure/my-gpt-6-luna-deployment
  modelInfo:
    healthCheck:
      supportsMaxTokens: false

maxTokens, maxTokensReasoning and maxTokensNonReasoning are ignored when this is false. Setting maxTokens alone does not fix this error, because the request still sends max_tokens.

This only changes the health check. A client that sends max_tokens to the same deployment can still get this error from the provider.

Disabling health checks for a model ​

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMModel
metadata:
  name: expensive-embeddings
spec:
  instanceRef:
    name: my-gateway
  modelName: text-embedding-3-large
  litellmParams:
    model: openai/text-embedding-3-large
    apiKeySecretRef:
      name: openai-credentials
      key: OPENAI_API_KEY
  modelInfo:
    mode: embedding
    healthCheck:
      disableBackgroundHealthCheck: true

Status Fields ​

FieldTypeDescription
syncedboolWhether the model is synced to LiteLLM
litellmModelIdstringLiteLLM-assigned model ID
lastSyncTime*TimeLast successful sync time
healthstringModel health status from LiteLLM
latencyP50Ms*intP50 latency in milliseconds
latencyP95Ms*intP95 latency in milliseconds
requestsLast24h*int64Request count in last 24 hours
conditions[]ConditionStandard conditions
bash
kubectl get lm
NAME            MODEL    SYNCED   HEALTH    AGE
gpt4o           gpt-4o   true     healthy   3d
claude-sonnet   claude   true     healthy   3d

Multiple Providers for the Same Model ​

Register the same model name from multiple providers for automatic fallback:

yaml
---
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMModel
metadata:
  name: gpt4o-openai
spec:
  instanceRef:
    name: my-gateway
  modelName: gpt-4o
  litellmParams:
    model: openai/gpt-4o
    apiKeySecretRef:
      name: openai-credentials
      key: OPENAI_API_KEY
---
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMModel
metadata:
  name: gpt4o-azure
spec:
  instanceRef:
    name: my-gateway
  modelName: gpt-4o
  litellmParams:
    model: azure/gpt-4o
    apiBase: https://my-deployment.openai.azure.com/
    apiKeySecretRef:
      name: azure-credentials
      key: AZURE_API_KEY

LiteLLM's router handles load balancing and fallback between the two.

Released under the Apache 2.0 License.