Incidentary Docs

OpenTelemetry

Send traces from your existing OpenTelemetry instrumentation to Incidentary with zero SDK installation and zero code changes.

OpenTelemetry Integration

If your services already export OpenTelemetry traces, you can send them to Incidentary without installing anything. Point your OTel Collector (or SDK exporter) at Incidentary's OTLP endpoint. Your existing instrumentation becomes the data source for causal incident artifacts.

No SDK. No code changes. No new dependency in your services.


OTLP key setup

OTLP ingest keys are separate from workspace API keys. Create one in the dashboard:

  1. Go to Settings --> OTLP Keys
  2. Click Create Key
  3. Copy the token — it is shown once

One key per environment or collector is recommended. Keys can be revoked individually without affecting other collectors.

OTLP keys are not workspace API keys. They start with a different prefix and authenticate only the OTLP ingest endpoint. Do not use a workspace API key (sk_...) as an OTLP bearer token — it will be rejected.


Setup with OTel Collector

Add Incidentary as an exporter in your Collector config. This runs alongside your existing exporters — nothing changes about where your traces go today.

exporters:
  otlphttp/incidentary:
    endpoint: https://api.incidentary.com/api/v1/otlp
    headers:
      authorization: "Bearer <your-otlp-key>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/incidentary, otlphttp/your-existing-backend]

The endpoint accepts both protobuf (application/x-protobuf) and JSON (application/json) content types. Protobuf is recommended for lower overhead.

gRPC alternative

If your Collector exports via gRPC instead of HTTP:

exporters:
  otlp/incidentary:
    endpoint: grpc.incidentary.com:443
    headers:
      authorization: "Bearer <your-otlp-key>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/incidentary]

Setup with SDK-native exporters

For teams not running the Collector, configure your SDK's OTLP exporter directly.

Node.js

npm install @opentelemetry/exporter-trace-otlp-http
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const exporter = new OTLPTraceExporter({
  url: 'https://api.incidentary.com/api/v1/otlp/v1/traces',
  headers: {
    authorization: 'Bearer <your-otlp-key>',
  },
});

Python

pip install opentelemetry-exporter-otlp-proto-http
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="https://api.incidentary.com/api/v1/otlp/v1/traces",
    headers={"authorization": "Bearer <your-otlp-key>"},
)

Go

import "go.opentelemetry.io/otel/exporters/otlp/otlptracehttp"

exporter, err := otlptracehttp.New(ctx,
    otlptracehttp.WithEndpointURL("https://api.incidentary.com/api/v1/otlp/v1/traces"),
    otlptracehttp.WithHeaders(map[string]string{
        "authorization": "Bearer <your-otlp-key>",
    }),
)

What appears in Incidentary

OTel spans are mapped to Incidentary's causal event model automatically.

Service identity

The service.name resource attribute becomes the service name in traces and artifacts. If service.name is missing, the entire resource block is dropped — set it.

The deployment.environment resource attribute sets the environment label. If absent, it defaults to unknown.

Span kind mapping

OTel span kindSemantic attributes presentIncidentary event kind
SERVERhttp.*HTTP_IN
SERVERmessaging.*QUEUE_CONSUME
SERVERrpc.*HTTP_IN
SERVER(none)HTTP_IN
CLIENThttp.*HTTP_OUT
CLIENTmessaging.*QUEUE_PUBLISH
CLIENTrpc.* or db.*HTTP_OUT
CLIENT(none)HTTP_OUT
PRODUCER(any)QUEUE_PUBLISH
CONSUMER(any)QUEUE_CONSUME
INTERNALrpc.* or db.*HTTP_OUT
INTERNAL(none)INTERNAL_TASK

Status codes

If the span carries http.response.status_code (or the legacy http.status_code), that value is used directly. For gRPC spans with rpc.grpc.status_code, code 0 maps to 200 and all other codes map to 500. If neither attribute is present, the OTLP span status is used: OK becomes 200, ERROR becomes 500.

Parent-child relationships

OTel trace IDs and parent span IDs are preserved. Parent-child relationships from the original OTel trace appear as causal links in the Incidentary trace viewer.

Quality tiers

Spans with semantic convention attributes (http.*, messaging.*, db.*, rpc.*) are classified as rich and produce higher-fidelity mappings. Spans without these attributes are generic — they still appear in the trace, but with less specific event kinds.


Coexistence with Incidentary SDKs

You can use both OTel and the Incidentary SDK in the same trace. A typical pattern:

  • OTel for services you don't own or can't modify (third-party, legacy, infrastructure)
  • Incidentary SDK for services you own and want full capture from (pre-arm, elevated capture modes, richer event types)

Events from both sources are assembled into the same causal chain using the trace ID. The trace viewer shows them together.


What is not included (yet)

Pre-arm elevated capture: The Incidentary SDK can switch to elevated capture mode when it detects anomaly signals before an alert fires. OTel spans are always ingested in SKELETON mode — timing, status, and causal structure only. If you need pre-arm triggers with elevated capture, use the Incidentary SDK on the services where that matters.

Custom event type hints: If you want an OTel span to map to a specific Incidentary event type (like job_start or webhook_in), set the incidentary.event_type span attribute. This is optional and most teams will not need it.


Troubleshooting

401 Unauthorized: You are using a workspace API key (sk_...) instead of an OTLP key. Create an OTLP key in Settings --> OTLP Keys.

Spans not appearing: Verify that service.name is set as a resource attribute. Spans without service.name are dropped.

429 Too Many Requests: You have exceeded the ingest rate limit for your workspace plan. Reduce batch frequency or contact support to increase your limit.

422 Unprocessable Entity: The payload could not be parsed. Verify your exporter is sending valid OTLP protobuf or JSON. Check that trace IDs are 16 bytes and span IDs are 8 bytes.

Services appear but are not linked: Parent span IDs are not being propagated between services. Ensure your OTel SDK or Collector is configured with a propagator (W3C TraceContext is the default in most SDKs).

On this page