Skip to content

Observability ​

Prometheus Metrics ​

The operator exposes Prometheus metrics on :8443/metrics:

MetricTypeDescription
litellm_operator_reconcile_totalCounterReconciliation count per controller
litellm_operator_reconcile_errors_totalCounterError count per controller
litellm_operator_reconcile_duration_secondsHistogramReconciliation latency
litellm_operator_config_sync_totalCounterConfig sync operations
litellm_operator_config_sync_drift_detectedCounterDrift detection events
litellm_operator_managed_instancesGaugeNumber of managed instances
litellm_operator_managed_modelsGaugeNumber of managed models
litellm_operator_managed_teamsGaugeNumber of managed teams
litellm_operator_managed_usersGaugeNumber of managed users
litellm_operator_managed_keysGaugeNumber of managed keys

ServiceMonitor ​

If Prometheus Operator is installed, enable automatic scrape target creation:

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: my-gateway
spec:
  observability:
    serviceMonitor:
      enabled: true
      interval: 30s
      labels:
        release: prometheus

PrometheusRule (Alerting) ​

Enable built-in alerting rules with runbooks:

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: my-gateway
spec:
  observability:
    prometheusRule:
      enabled: true
      labels:
        release: prometheus
      # disabledAlerts:
      #   - LiteLLMHighCPUUsage

The operator creates a monitoring.coreos.com/v1 PrometheusRule with these default alerts:

AlertSeverityDescription
LiteLLMInstanceDowncriticalNo available replicas for 5 minutes
LiteLLMInstanceDegradedwarningFewer replicas than desired for 10 minutes
LiteLLMPodRestartingwarningMore than 3 restarts in the last hour
LiteLLMPodNotReadywarningPod not ready for 10 minutes
LiteLLMHighMemoryUsagewarningMemory usage above 90% of limit for 15 minutes
LiteLLMHighCPUUsagewarningCPU usage above 90% of limit for 15 minutes

Each alert includes:

  • severity and namespace/instance labels for routing
  • summary and description annotations for context
  • runbook annotation with kubectl troubleshooting commands

Individual alerts can be disabled via spec.observability.prometheusRule.disabledAlerts.

Grafana Dashboard ​

Auto-provision a Grafana dashboard via the sidecar pattern:

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: my-gateway
spec:
  observability:
    grafanaDashboard:
      enabled: true
      folder: "LiteLLM"
      labels:
        grafana_dashboard: "1"

The operator creates a ConfigMap with the grafana_dashboard: "1" label. If the Grafana sidecar is configured to watch for this label, the dashboard is automatically imported.

Dashboard panels include:

  • Ready / desired replicas (stat)
  • Pod restarts in the last hour (stat)
  • CPU and memory usage over time (timeseries)
  • Network I/O (timeseries)
  • Deployment conditions (table)

Callbacks ​

LiteLLM supports callback integrations for logging and observability. Configure them via the Instance CRD:

yaml
spec:
  callbacks:
    enabled: true
    callbacks:
      - langfuse
    langfuse:
      publicKeySecretRef:
        name: langfuse-credentials
        key: public-key
      secretKeySecretRef:
        name: langfuse-credentials
        key: secret-key
      host: https://langfuse.example.com

The operator translates callback configuration into environment variables (LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST) and adds the callback list to the proxy_server_config.yaml.

Instance-Level Logging Controls ​

Configure global logging behavior on the LiteLLMInstance:

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: my-gateway
spec:
  logging:
    # Audit logs (enterprise) — track admin actions in the database
    auditLogs:
      enabled: true
      retentionDays: 90

    # Don't log request/response message content (only metadata)
    turnOffMessageLogging: true

    # Redact API key info from logs
    redactUserApiKeyInfo: true

    # Spend log cleanup
    spendLogRetention:
      maxRetentionPeriod: "90d"
      cleanupInterval: "1d"

Per-Team Logging (Enterprise) ​

Route logs to team-specific destinations. Each team can have its own Langfuse project, GCS bucket, LangSmith project, or Arize space:

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMTeam
metadata:
  name: engineering
spec:
  instanceRef:
    name: my-gateway
  teamAlias: engineering
  logging:
    callbacks:
      - name: langfuse
        type: success_and_failure
        credentialsSecretRef:
          name: engineering-langfuse
---
apiVersion: v1
kind: Secret
metadata:
  name: engineering-langfuse
stringData:
  langfuse_public_key: "pk-team-engineering"
  langfuse_secret: "sk-team-engineering"
  langfuse_host: "https://cloud.langfuse.com"

Supported providers: langfuse, gcs_bucket, langsmith, arize.

GDPR: Disable Logging for a Team ​

Completely disable all logging for a team to comply with GDPR or other data privacy requirements:

yaml
apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMTeam
metadata:
  name: eu-customers
spec:
  instanceRef:
    name: my-gateway
  teamAlias: eu-customers
  logging:
    disabled: true

When disabled: true, the operator calls the LiteLLM API to remove all callbacks for the team. No request or response data will be logged.

Status Conditions ​

Every CRD reports health via standard metav1.Condition types:

bash
# Check instance conditions
kubectl get li my-gateway -o jsonpath='{.status.conditions}' | jq

# Check if all models are synced
kubectl get lm -o custom-columns=NAME:.metadata.name,SYNCED:.status.synced

Condition types used:

CRDConditionMeaning
LiteLLMInstanceReadyAll managed resources are healthy
LiteLLMInstanceDatabaseReadyDatabase connection and migrations are healthy
LiteLLMInstanceRedisReadyRedis connection is healthy
LiteLLMInstanceConfigSyncedConfig sync loop is running without errors
Secondary CRDsSyncedResource is synced to the LiteLLM API

Kubernetes Events ​

The operator emits Kubernetes Events for important lifecycle actions:

bash
kubectl get events --field-selector involvedObject.kind=LiteLLMInstance

Released under the Apache 2.0 License.