Observability

Observability is how well you can tell what your application is doing from the outside, without adding debug code everywhere. For LLM applications it matters even more than usual: the model is non-deterministic, so tracking token usage, latency, errors, and tool calls is what lets you run an unpredictable system with confidence.

Rig emits its telemetry through the tracing crate. Consume it with any tracing-subscriber layer, log facade, or OpenTelemetry exporter you like; Rig installs nothing itself.

What Rig emits

Rig’s spans follow the OpenTelemetry GenAI semantic conventions, so GenAI-aware backends such as Langfuse and Arize Phoenix understand them without extra mapping.

Span	When	Key attributes
`invoke_agent`	one agent run (`agent.prompt(..)`), covering every turn	`gen_ai.agent.name`, total `gen_ai.usage.*`
`chat`	one model call (`generate_content` for Gemini)	`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.request.stream`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, cache and reasoning token counts
`execute_tool`	one tool call	`gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.call.outcome`, `gen_ai.tool.error.type`
`embeddings`, `rerank`, `transcription`, `image_generation`, `audio_generation`	one call to that kind of model	provider, model, and usage

Model-call spans nest under the agent span, and tool spans under the turn that requested them. If you call agent.prompt(..) inside a span of your own, Rig uses your span as the agent span instead of opening a new invoke_agent root.

Recording message content

By default, spans carry structure and token usage but not the content of prompts, responses, or tool arguments and results. Content can contain personal data and makes traces much larger, so you opt in per agent or per run:

let agent = AgentBuilder::new(OpenAI::from_env()?.completion(openai::GPT_5_5))
    .preamble("You are a helpful assistant.")
    .record_content_telemetry(true) // every run of this agent
    .build();

// Or for a single run:
let response = agent.prompt("Hello!").record_content_telemetry(true).await?;

With content on, spans also get gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, and the tool call arguments and results.

Basic setup with `tracing-subscriber`

The simplest setup prints spans and events to stdout:

tracing_subscriber::fmt().init();

That needs RUST_LOG set to show anything useful. To get a sensible default without it, add an EnvFilter, here info for everything and debug for Rig:

tracing_subscriber::registry()
    .with(
        tracing_subscriber::EnvFilter::try_from_default_env()
            .unwrap_or_else(|_| "info,rig=debug".into()),
    )
    .with(tracing_subscriber::fmt::layer())
    .init();

To group Rig’s spans under your own unit of work, instrument your function. Rig adopts your span as the agent span, so the model-call and tool spans appear as its children:

use rig::prelude::*;
use rig::providers::openai::{self, OpenAI};
use tracing::{info, instrument};

#[instrument(name = "process_user_query")]
pub async fn process_query(user_input: &str) -> anyhow::Result<String> {
    info!("Processing user query");

    let agent = AgentBuilder::new(OpenAI::from_env()?.completion(openai::GPT_5_5))
        .name("assistant")
        .preamble("You are a helpful assistant.")
        .build();

    // The agent run and its model call emit spans under `process_user_query`.
    let response = agent.prompt(user_input).await?;

    info!("Query processed successfully");
    Ok(response.output())
}

.name(..) sets gen_ai.agent.name, which makes runs easy to find in a tracing backend.

For production log shipping, tracing-subscriber can format events as JSON with fmt::layer().json() (enable its json feature).

Exporting to OpenTelemetry

To send traces to an OpenTelemetry backend, add a tracing-opentelemetry layer with an OTLP exporter. The dependencies:

[dependencies]
rig = "0.44.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-opentelemetry = "0.33"
opentelemetry = "0.32"
opentelemetry_sdk = { version = "0.32", features = ["rt-tokio"] }
opentelemetry-otlp = "0.32"

Build an OTLP span exporter, wrap it in a tracer provider, and stack the OpenTelemetry layer with a filter and a fmt layer so traces go to both stdout and the exporter:

use opentelemetry::trace::TracerProvider;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{Resource, trace::SdkTracerProvider};
use tracing::Level;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Sends to http://localhost:4318 unless OTEL_EXPORTER_OTLP_ENDPOINT says otherwise.
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_http()
        .with_protocol(opentelemetry_otlp::Protocol::HttpBinary)
        .build()?;

    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_resource(Resource::builder().with_service_name("rig-service").build())
        .build();
    let tracer = provider.tracer("rig-service");

    tracing_subscriber::registry()
        .with(
            tracing_subscriber::EnvFilter::builder()
                .with_default_directive(Level::INFO.into())
                .from_env_lossy(),
        )
        .with(tracing_subscriber::fmt::layer())
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .init();

    let answer = process_query("Hello world!").await?;
    println!("{answer}");

    // Flush buffered spans before exiting.
    provider.shutdown()?;
    Ok(())
}

The Rig repository has complete programs: agent_with_tools_otel (multi-turn agent with tools), and openai_agent_completions_api_otel / openai_streaming_with_tools_otel for OpenAI, plus a collector config and Dockerfile.

Sending traces to Langfuse

Langfuse accepts OTLP directly at https://cloud.langfuse.com/api/public/otel. For local development you can point the exporter there with environment variables and skip running a collector:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ${AUTH_STRING}"

AUTH_STRING is public_key:secret_key from your Langfuse project, base64-encoded.

Using a collector

An OpenTelemetry Collector is worth running when you send to several backends, already run OTel infrastructure, or need to process or redact spans. The collector receives OTLP from your app, optionally transforms it, and exports it. This config renames invoke_agent spans to the agent’s name and forwards to Langfuse:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  transform:
    trace_statements:
      - context: span
        statements:
          # Rename "invoke_agent" spans to the agent's name, when it has one.
          - set(name, attributes["gen_ai.agent.name"]) where name == "invoke_agent" and attributes["gen_ai.agent.name"] != nil

exporters:
  debug:
    verbosity: detailed
  otlphttp/langfuse:
    endpoint: "https://cloud.langfuse.com/api/public/otel"
    headers:
      Authorization: "Basic ${AUTH_STRING}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform]
      exporters: [otlphttp/langfuse, debug]

Run it from the Collector Contrib image with the config baked in:

FROM otel/opentelemetry-collector-contrib:0.135.0
COPY ./config.yaml /etc/otelcol-contrib/config.yaml

The rename lives in the collector because tracing fixes span names at compile time; Rig can’t name the span after your agent itself.

Logs only, no spans

Because Rig uses tracing, you can ignore spans and print only events. This layer prints each event’s level and message:

use std::io::Write;

#[derive(Clone)]
struct MessageOnlyLayer;

impl<S> tracing_subscriber::Layer<S> for MessageOnlyLayer
where
    S: tracing::Subscriber + for<'a> tracing_subscriber::registry::LookupSpan<'a>,
{
    fn on_event(&self, event: &tracing::Event<'_>, _ctx: tracing_subscriber::layer::Context<'_, S>) {
        use tracing::field::{Field, Visit};

        struct MessageVisitor(Option<String>);

        impl Visit for MessageVisitor {
            fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
                if field.name() == "message" {
                    self.0 = Some(format!("{value:?}"));
                }
            }
        }

        let mut visitor = MessageVisitor(None);
        event.record(&mut visitor);

        if let Some(msg) = visitor.0 {
            let level = event.metadata().level();
            let _ = writeln!(std::io::stdout(), "{level} {}", msg.trim_matches('"'));
        }
    }
}

fn init_logging() {
    use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::new("info"))
        .with(MessageOnlyLayer)
        .init();
}

Instrumenting your own model code

If you write your own provider or call models outside an agent, rig::telemetry builds spans with the same GenAI attributes Rig uses, so your calls show up alongside Rig’s:

use rig::telemetry::{GenAiOperation, SpanBuilder};

let span = SpanBuilder::new("my-provider", "my-model", GenAiOperation::Chat)
    .streaming(false)
    .build();
let _guard = span.enter();

rig::observe is a lower-level layer of typed facts about each provider request (attempt number, the provider’s verdict, error envelopes, usage), separate from tracing. Pass an AdapterContext to a direct model call and collect the facts in an ObservationLog:

use std::sync::Arc;
use rig::completion::CompletionRequest;
use rig::observe::{AdapterContext, ObservationLog, Subject};
use rig::providers::openai::{self, OpenAI};

let log = Arc::new(ObservationLog::with_capacity(256));
let context = AdapterContext::new(log.clone(), Subject::scoped("checkout"), "summarize-1");

let model = OpenAI::from_env()?.completion(openai::GPT_5_5);
model.call_observed(CompletionRequest::new("Hello!"), context).await?;

for observation in log.trace().observations {
    println!("{:?}", observation.action);
}

Diagnostic text in these facts is scrubbed of credentials before it is stored.

Troubleshooting

Spans appear out of order. When an operation finishes in under about a millisecond (a fast tool, for example), some backends show spans slightly out of order because of timestamp resolution. Real workloads rarely hit this.
No content in spans. Content recording is off by default; see Recording message content.

Next steps

Error handlingTurn the errors you now trace into typed, recoverable failures.

Recording & ReplayRecord a run once and replay it without a provider.

TestingPin down behaviour with tests around your agents and tools.

Previous
Media Next
Testing