Ghost Services
How Incidentary detects uninstrumented dependencies from a single SDK install, and how to use ghost services to expand coverage.
Ghost Services
When you install the Incidentary SDK on one service, it captures every outbound call that service makes — HTTP requests, queue publishes, and webhook sends. Incidentary identifies the target of each outbound call and checks whether that target has the SDK installed. Targets that do not have the SDK are surfaced as ghost services.
What ghost services tell you
A ghost service is a dependency you know exists because your instrumented service calls it, but you do not have internal data from. Incidentary shows you:
- That the dependency exists: ghost services appear on the service map and in the coverage scorecard
- Caller-side metrics: latency (p50, p95, p99), error rate, and call volume — measured from the calling service's perspective
- Which instrumented service discovered it: useful when multiple services call the same downstream dependency
- When it was first and last seen: derived from the timestamps of outbound CEs
What ghost services do not tell you
Ghost services are observed from the outside. They do not provide:
- Internal timing breakdowns (what happened inside the service)
- Root cause information (why a call to the ghost service failed)
- Internal error context (database errors, downstream call failures within the ghost service)
To get internal data from a ghost service, install the SDK on that service.
How detection works
- The SDK captures an outbound CE (
HTTP_OUT,QUEUE_PUBLISH,WEBHOOK_OUT) with atarget_serviceattribute - Incidentary checks the service registry for that target
- If no instrumented service matches the target name, a ghost service record is created
- Caller-side metrics are aggregated from the outbound CEs
Detection happens incrementally on ingest — not in a batch job. New ghost services appear within seconds of the first outbound call.
External services
Not every outbound call target is a service you own. CDNs, third-party APIs (Stripe, AWS), and shared infrastructure may appear as ghost services. You can mark these as "external" to exclude them from the coverage scorecard. Incidentary auto-detects common patterns (e.g., *.googleapis.com, *.amazonaws.com).
Coverage scorecard
The coverage scorecard shows:
Service Coverage: 3 / 8 instrumented (37%)
[====------]Ghost services are sorted by call volume, so the highest-impact instrumentation targets appear first. Each ghost service has an "Add SDK" action that shows copy-paste setup instructions pre-filled with your workspace API key.
Ghost services in incident traces
When an incident trace passes through a ghost service, the ghost appears as a gap in the causal chain with a dashed border. You can see that the call was made and what the caller observed (status code, latency), but internal data from the ghost service is missing.
Every ghost service in a trace is a natural adoption moment — the engineer debugging the incident wants to see inside that service. The "Instrument this service" action is available directly from the trace viewer.