Observability

Health endpoints

Kubernetes probes

The chart ships probes tuned for a Next.js cold start. Defaults for the app:

Realtime uses /health on port 3002 with a 150-second startup budget.

The generous startup budget matters: a cold Next.js start on a large bundle can take minutes, and a tighter liveness probe will restart the pod mid-boot in a loop. If you customize probes, keep the startup budget well above your observed cold-start time.

app:
  startupProbe:
    httpGet:
      path: /
      port: 3000
    periodSeconds: 5
    failureThreshold: 60

Logs

Both services log structured JSON to stdout. Collect them with whatever you already run — Fluent Bit, Vector, Datadog Agent, Loki.

kubectl logs -n simstudio -l app.kubernetes.io/component=app --tail=200 -f
kubectl logs -n simstudio -l app.kubernetes.io/component=realtime --tail=200 -f
kubectl logs -n simstudio deploy/sim-app -c migrations --tail=100

docker compose -f docker-compose.prod.yml logs -f simstudio

Every API request carries a request ID that appears in all log lines for that request — the fastest way to reconstruct a failing call.

Redacting PII from logs

Enable the PII service and log redaction if execution logs may contain sensitive data:

pii:
  enabled: true
app:
  env:
    INTERNAL_API_BASE_URL: "http://sim-app.simstudio.svc.cluster.local:3000"

See Security for the INTERNAL_API_BASE_URL requirement — the path fails closed without a cluster-reachable value.

Telemetry

Whether telemetry runs depends on how you deploy. A Docker Compose or source deployment enables server telemetry unless you turn it off, so a deployment with an egress policy should decide about this deliberately rather than inherit the default.

Telemetry has two halves that are decided at different times.

Neither half gates POST /api/telemetry, the relay that forwards browser events. See the warning below.

When the server SDK runs it exports three OTLP signals to the same base endpoint: traces at /v1/traces, metrics at /v1/metrics, and application log records at /v1/logs. Sim's own telemetry is scoped to feature-usage statistics, error rates, performance metrics, and AI/LLM operation traces — not workflow content or outputs, API keys, or geolocation.

The log stream is different in kind: it carries every line at or above LOG_LEVEL with its structured metadata, error messages and stack traces, which can include user ids, emails and URLs. That matters if you raise LOG_LEVEL as suggested above, and it is a reason to point the exporter at your own collector rather than leaving it at the default.

NEXT_TELEMETRY_DISABLED=1 stops the server's OpenTelemetry SDK and disables browser collection on newly loaded pages. It does not affect Trigger.dev task telemetry, configured separately below. The Settings → General → Privacy → Allow browser telemetry toggle stops optional browser performance and error diagnostics, discards queued events, and remembers the choice. It does not stop server spans.

app:
  envDefaults:
    NEXT_TELEMETRY_DISABLED: null

Tracing

The chart can deploy an OpenTelemetry Collector for you:

telemetry:
  enabled: true

Or point the app at a collector you already run:

The endpoint is resolved by precedence, first match wins: OTEL_EXPORTER_OTLP_ENDPOINT, then TELEMETRY_ENDPOINT, then the endpoint in apps/sim/telemetry.config.ts. This precedence governs the server SDK only — the /api/telemetry relay reads TELEMETRY_ENDPOINT and nothing else. Sampling resolves the same way: TELEMETRY_SAMPLING_RATIO, then OTEL_TRACES_SAMPLER_ARG, defaulting to 1.0. Editing telemetry.config.ts only changes the last fallback, so an environment variable set anywhere in your stack silently wins over it.

If you enable the chart's Jaeger export, point telemetry.jaeger.endpoint at Jaeger's OTLP gRPC port (4317) — the collector exports over OTLP.

Trigger.dev task telemetry

GRAFANA_OTLP_ENDPOINT, GRAFANA_OTLP_HEADERS, and GRAFANA_DEPLOYMENT_ENVIRONMENT are read only by apps/sim/trigger.config.ts. They route Trigger.dev background-task telemetry to Grafana Cloud and have no effect on the app's own OTLP export — for that, use OTEL_EXPORTER_OTLP_ENDPOINT above.

Metrics

Until an application metrics endpoint ships, build alerting from the signals that do exist:

• Kubernetes state — pod restarts, CrashLoopBackOff, OOMKills, replica count vs desired, PVC utilization (kube-state-metrics). For a point-in-time read on resource pressure, kubectl top pods -n simstudio shows current CPU and memory usage — compare it against the limits from kubectl describe pod. It is the first thing to check when a pod is being OOMKilled.
• Ingress/load balancer — request rate, 5xx rate, p99 latency, websocket connection count.
• PostgreSQL — connection count vs max_connections, replication lag, disk usage, long-running queries.
• Redis — memory usage, evictions, connected clients.
• CronJobs — last successful completion per job.

CloudWatch metrics

Sim's hosted-key metrics recorder arms itself whenever AWS_ACCESS_KEY_ID is present, but every call site is additionally gated on the deployment being Sim Cloud. A self-hosted deployment emits no PutMetricData calls, even with AWS credentials configured for S3. If you want defence in depth against that gate ever changing, deny cloudwatch:PutMetricData on the IAM identity behind those credentials.

What to alert on