Observability

Zil agents ship with pre-wired OpenTelemetry conventions for agent-specific spans. This is one of the highest-value features — most agents in the wild have poor observability. Zil standardizes it.

How it works

Zil provides built-in OpenTelemetry tracing that works across all framework backends. When you call zil.create_agent(), it:

1. Reads observability/config.yaml from your project
2. Sets standard OTel environment variables (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_SERVICE_NAME)
3. Injects resource attributes (agent.name, agent.version) so every span is tagged with the agent’s identity
4. Registers the OTLP exporter via the framework backend’s tracing setup

The framework backend then emits spans for agent invocations, LLM calls, and tool executions — following the OpenTelemetry GenAI Semantic Conventions .

Configuration

# observability/config.yaml
observability:
  tracing:
    exporter: otlp
    endpoint: ${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT}
    sample_rate: 1.0
  resource_attributes:
    service.name: my-agent
  span_conventions:
    - agent.session
    - agent.turn
    - agent.reasoning
    - agent.skill.invoke
    - agent.mcp.tool_call
    - agent.guardrail.check
  required_attributes:
    - agent.name
    - agent.version
    - session.id
    - tokens.input
    - tokens.output
    - cost.usd

Configuration fields

tracing

resource_attributes

Custom OTel resource attributes  attached to all spans. Typically used for service.name. Supports ${ENV_VAR} substitution.

span_conventions

The standard span types emitted by Zil agents:

required_attributes

Attributes that must be present on every span:

Token and cost attributes are mandatory because they enable cost governance — tracking spend per agent, per session, per request.

Dev vs Prod

Local development

Quick debugging — print spans to stderr with no infrastructure:

zil serve --trace-console

This uses OpenTelemetry’s ConsoleSpanExporter — every span is printed as it completes.

Full observability with Docker — use zil serve --docker --trace to get traces, metrics, and logs in one command:

zil serve --docker --trace

This starts a Grafana OTEL-LGTM  container alongside your agent, providing:

• Agent server: http://localhost:8000
• Grafana: http://localhost:3000 (login: admin / admin)

To explore traces:

1. Open Grafana → Explore (compass icon)
2. Select Tempo as the data source
3. Click Search → Run query

Both containers stop on Ctrl+C.

Manual OTLP setup — point to any collector:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318
zil serve --trace

Production (Cloud Run)

Deploy with --trace to send spans to Google Cloud Trace:

zil deploy --project my-project --region us-central1 --trace

View traces at: GCP Console → Observability → Trace Explorer

For other backends, set the endpoint in your .env:

# .env
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://your-collector.example.com/v1/traces

The SDK picks this up automatically via zil.create_agent() — no code changes needed between dev and prod.

SDK integration

Telemetry is wired up automatically when you use zil.create_agent():

import zil
 
agent = zil.create_agent(tools=[...])
# Tracing is active if observability/config.yaml exists
# and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set

To disable automatic telemetry setup (e.g., in tests or when managing OTel yourself):

agent = zil.create_agent(tools=[...], enable_telemetry=False)

For programmatic control:

from zil.sdk import setup_telemetry, setup_console_telemetry
 
# Console exporter for local debugging
setup_console_telemetry(agent_name="my-agent", agent_version="1.0.0")
 
# Or OTLP exporter from config
setup_telemetry(obs_config, agent_name="my-agent", agent_version="1.0.0")

Compatible backends

The otlp exporter works with any OTel-compatible backend:

Environment setup

Set the collector endpoint in your .env:

# .env
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces

For Google Cloud Trace, deploy with zil deploy --trace or configure the GCP OTLP endpoint.

CLI flags

zil serve supports tracing flags:

# Export to OTLP collector
zil serve --trace
 
# Local debugging — no infra required
zil serve --trace-console
 
# Docker with full observability stack
zil serve --docker --trace

Validation

zil validate checks that:

• observability/config.yaml exists
• The cost.usd attribute is present (warns if missing)

✓ observability/config.yaml — present
⚠ observability/config.yaml — missing recommended attribute: cost.usd