Skip to main content

OpenTelemetry

OpenTelemetry

Plugin: otel.plugin Module: otel

Maintained by Netdata

Overview​

Receive OpenTelemetry metrics, logs, and traces from an OpenTelemetry Collector over OTLP/gRPC or OTLP/HTTP. The plugin converts supported metrics into Netdata charts, indexes logs for exploration in the Logs tab, and indexes traces for exploration in the Traces tab. It also builds a service map from the received spans (the topology:tracing Function).

The plugin accepts gauges, sums, explicit-bucket histograms, and summaries. It does not currently ingest exponential histograms. Monotonic cumulative and delta sums are rate-normalized; non-monotonic cumulative sums behave as gauges.

For a maintained Collector configuration and end-to-end verification steps, see Ingest OpenTelemetry Metrics, Logs, and Traces. The separate OpenTelemetry Logs integration explains how received logs appear in Netdata, and Trace Storage and Retention explains how received traces are stored and kept.

The plugin starts automatically and listens for OTLP/gRPC on port 4317. It can also listen for OTLP/HTTP on port 4318, but this listener is off by default: set receivers.otlp.protocols.http.enabled to true to turn it on. Point the Collector's otlp_grpc exporter, or its otlp_http exporter when the OTLP/HTTP listener is on, at the matching port. Keep the exporter's default protobuf encoding and gzip compression: the plugin rejects other compression algorithms, and with encoding: json it rejects any request that carries a NaN or infinite metric value.

Each metric name becomes a Netdata chart context prefixed with otel.. YAML mapping rules can match a metric name and instrumentation scope, select a data point attribute for dimension names, and override chart timing. User mapping files take priority over the stock hostmetrics mappings compiled into the plugin. Metrics without a matching mapping still create charts using default settings.

Mapping file optionDescription
instrumentation_scope.nameRegular expression matching the instrumentation scope name
instrumentation_scope.versionRegular expression matching the instrumentation scope version
dimension_attribute_keyData point attribute whose value becomes the dimension name
interval_secsPer-metric chart update interval override
grace_period_secsPer-metric gap-filling grace period override

Invalid user mapping files are logged and skipped without disabling the stock mappings. Charts expire after their configured inactivity period.

This collector is only supported on the following platforms:

  • Linux
  • macOS

This collector only supports collecting metrics from a single instance of this integration.

Default Behavior​

Auto-Detection​

The plugin starts automatically and listens on 127.0.0.1:4317 for OTLP/gRPC connections. When you turn on the OTLP/HTTP listener, it also listens on 127.0.0.1:4318 for OTLP/HTTP connections.

Limits​

Each export request is limited to 4 MiB, both as received and after decompression. The plugin rejects a larger request as a whole. The plugin also rejects as a whole a request with so many records that the plugin cannot store them together: the log records of one service in the request, or all the spans of the request, are more than 32 MiB after the plugin prepares them. Only a request of hundreds of thousands of records that are almost empty is that large. Each export request can create at most 100 new charts by default. This is a per-request throttle, not a total cardinality limit. By default, log timestamps are accepted from up to 24 hours in the past through 10 minutes in the future; records without a timestamp use their arrival time. A span is accepted when it started no more than 24 hours ago and ends no more than 10 minutes in the future. Log and trace retention are each bounded by whichever limit is reached first: retained file count, retained indexed-data size, or age.

Performance Impact​

Resource use depends on the number and cardinality of received metrics, log and span volume, retention, and query activity. Keep unbounded attributes out of metric identities and dimension names.

Setup​

Prerequisites​

A Netdata installation that includes the OpenTelemetry plugin​

Official Linux packages, static builds, containers, and supported macOS packages include the plugin. Linux source installs using netdata-installer.sh require a compatible Rust toolchain and --enable-plugin-otel; on macOS, the installer enables the plugin automatically when it finds a compatible Rust toolchain.

An OTLP data source​

An OpenTelemetry Collector that exports metrics, logs, or traces to the Agent: the otlp_grpc exporter to port 4317, or the otlp_http exporter to port 4318 after you turn on the OTLP/HTTP listener. The default loopback endpoints accept only a Collector on the same host.

Configuration​

Options​

Edit otel.yaml with edit-config. A user file can contain only the fields that differ from the stock configuration. Configuration resolves in this order: stock file, user file, then environment variables. Restart the Netdata Agent after changing otel.yaml or any NETDATA_OTEL_CFG_* environment variable.

The absolute paths shown below are the defaults for standard Linux packages. The installed stock configuration resolves its configuration and log directories for the installation layout, so macOS and custom-prefix installations use different paths. Use edit-config with otel.yaml to inspect the installed values.

Environment overrides use the NETDATA_OTEL_CFG_ prefix. Names are the uppercase option path with dots replaced by underscores. For example, receivers.otlp.protocols.grpc.tls.cert_file becomes NETDATA_OTEL_CFG_RECEIVERS_OTLP_PROTOCOLS_GRPC_TLS_CERT_FILE, and receivers.otlp.protocols.http.enabled becomes NETDATA_OTEL_CFG_RECEIVERS_OTLP_PROTOCOLS_HTTP_ENABLED. For the default rotation and retention entries, omit the default path segment: logs.retention.default.max_age becomes NETDATA_OTEL_CFG_LOGS_RETENTION_MAX_AGE, and traces.retention.default.max_age becomes NETDATA_OTEL_CFG_TRACES_RETENTION_MAX_AGE.

Configuration is strict. Unknown YAML fields, unknown NETDATA_OTEL_CFG_* variables, malformed values, and the removed logs options of the previous otel.yaml layout prevent the plugin from starting. The effective configuration is logged at startup; remote_storage.uri is redacted in that log.

In a YAML file, an option set to null keeps its current value. To turn a listener on or off, set its enabled option. The OTLP/gRPC listener is on and the OTLP/HTTP listener is off by default. At least one of the two listeners must stay enabled.

Older files configure the OTLP/gRPC listener in an endpoint: section: endpoint.path, endpoint.tls_cert_path, endpoint.tls_key_path, and endpoint.tls_ca_cert_path, and the matching NETDATA_OTEL_CFG_ENDPOINT_* variables. These names still work and log a deprecation warning. If a file or the environment sets an option under both names, the plugin logs a warning and uses the receivers name.

Config options
OptionDescriptionDefaultRequired
receivers.otlp.protocols.grpc.enabledTurns the OTLP/gRPC listener on or off.yesno
receivers.otlp.protocols.grpc.endpointAddress (host:port) on which the OTLP/gRPC listener accepts connections.127.0.0.1:4317no
receivers.otlp.protocols.grpc.tls.cert_filePath to the server TLS certificate for the OTLP/gRPC listener. Set it together with receivers.otlp.protocols.grpc.tls.key_file.no
receivers.otlp.protocols.grpc.tls.key_filePath to the server TLS private key for the OTLP/gRPC listener. Set it together with receivers.otlp.protocols.grpc.tls.cert_file.no
receivers.otlp.protocols.grpc.tls.client_ca_filePath to a CA certificate used to verify client certificates on the OTLP/gRPC listener.no
receivers.otlp.protocols.http.enabledTurns the OTLP/HTTP listener on or off. The listener is off by default.nono
receivers.otlp.protocols.http.endpointAddress (host:port) on which the OTLP/HTTP listener accepts connections.127.0.0.1:4318no
receivers.otlp.protocols.http.tls.cert_filePath to the server TLS certificate for the OTLP/HTTP listener. Set it together with receivers.otlp.protocols.http.tls.key_file.no
receivers.otlp.protocols.http.tls.key_filePath to the server TLS private key for the OTLP/HTTP listener. Set it together with receivers.otlp.protocols.http.tls.cert_file.no
receivers.otlp.protocols.http.tls.client_ca_filePath to a CA certificate used to verify client certificates on the OTLP/HTTP listener.no
auth.enabledRequire each sender to set its tenant name in the X-Scope-OrgID header. Applies to logs and traces.nono
metrics.chart_configs_dirDirectory containing user metric mapping YAML files./etc/netdata/otel.d/v1/metricsno
metrics.interval_secsDefault Netdata chart update interval in seconds.10no
metrics.grace_period_secsTime after the last data point before the plugin begins filling chart gaps.60no
metrics.expiry_duration_secsTime without data after which a chart is removed.900no
metrics.max_new_charts_per_requestMaximum number of new charts one OTLP export request can create.100no
base_dirAbsolute root directory for locally stored OpenTelemetry logs and traces./var/log/netdata/otel/v2no
logs.rotation.default.max_file_sizeFile size at which incoming logs start a new file. The plugin reads a value of 0 as "not set". It then logs a warning when it starts and uses the default value. For a tenant, it uses the value of the default entry.25MBno
logs.rotation.default.max_entriesEntry count at which incoming logs start a new file. The plugin reads a value of 0 as "not set". It then logs a warning when it starts and uses the default value. For a tenant, it uses the value of the default entry.50000no
logs.retention.default.max_filesMaximum number of stored log files.100000no
logs.retention.default.max_total_sizeMaximum total size of stored log files.1GBno
logs.retention.default.max_ageMaximum age of stored log files.7 daysno
traces.rotation.default.max_file_sizeFile size at which incoming spans start a new file. The plugin reads a value of 0 as "not set". It then logs a warning when it starts and uses the default value. For a tenant, it uses the value of the default entry.25MBno
traces.rotation.default.max_entriesSpan count at which incoming spans start a new file. The plugin reads a value of 0 as "not set". It then logs a warning when it starts and uses the default value. For a tenant, it uses the value of the default entry.50000no
traces.retention.default.max_filesMaximum number of stored trace files.100000no
traces.retention.default.max_total_sizeMaximum total size of stored trace files.1GBno
traces.retention.default.max_ageMaximum age of stored trace files, measured on the start time of the newest span in each file.7 daysno
remote_storage.enabledUpload retained OpenTelemetry logs and traces to configured remote object storage.nono
remote_storage.uriRemote storage URI. Supported schemes are fs and s3.fs:///var/log/netdata/otel/v2/remoteno
remote_storage.read_cache_max_sizeMaximum size of the local download cache, shared by logs and traces, for data fetched back from remote storage during queries.1GBno
receivers.otlp.protocols.grpc.endpoint​

The default IPv4 loopback address accepts only local senders. To accept remote senders, bind a non-loopback address and protect the listener with TLS or mutual TLS and network access controls.

receivers.otlp.protocols.grpc.tls.client_ca_file​

Setting a CA certificate enables mutual TLS and therefore also requires the server certificate and key. Clients must present a certificate signed by this CA.

receivers.otlp.protocols.http.endpoint​

The default IPv4 loopback address accepts only local senders. To accept remote senders, bind a non-loopback address and protect the listener with TLS or mutual TLS and network access controls.

receivers.otlp.protocols.http.tls.client_ca_file​

Setting a CA certificate enables mutual TLS and therefore also requires the server certificate and key. Clients must present a certificate signed by this CA.

auth.enabled​

Tenants keep the logs and traces of different senders apart, and the logs and traces sections can set rotation and retention limits per tenant. Metrics do not use tenants. When enabled, the plugin rejects logs and traces without this header. When disabled, the plugin ignores the header and keeps all logs and traces in one default tenant. The header does not prove who the sender is: use TLS or mutual TLS and network access controls to limit which senders can connect.

metrics.chart_configs_dir​

User mappings take priority over the stock mappings compiled into the plugin. Files are processed in filename order. An invalid file is logged and skipped.

metrics.interval_secs​

Timing must satisfy 0 < interval <= 3600, interval < grace, and grace <= expiry. Because the stock configuration explicitly sets all three values, set compatible values together when changing the global timing. An invalid resolved timing configuration logs a warning and falls back to the plugin's hardcoded chart defaults.

metrics.expiry_duration_secs​

The plugin also rejects a metric data point when its timestamp is further from the current time than the expiry of its chart, in the past or in the future. The expiry of a chart is this value, unless a per-metric mapping gives the chart a longer grace period.

metrics.max_new_charts_per_request​

This limits a single request's cardinality burst. It does not cap the total number of charts that successive requests can create. The plugin reads a value of 0 as "not set". It then logs a warning when it starts and uses 100.

base_dir​

The retention limits count only the stored log and trace files. Total disk use under this directory can be higher, because it also holds:

  • the files that hold incoming data (limited by the rotation options),
  • catalog files that list the stored files,
  • the download cache for remote storage (the remote-read subdirectory, limited by remote_storage.read_cache_max_size).
logs.retention.default.max_total_size​

This limit counts only the stored log files. Other files under base_dir also use disk space; see base_dir.

traces.retention.default.max_total_size​

This limit counts only the stored trace files. Other files under base_dir also use disk space; see base_dir.

remote_storage.enabled​

The plugin uploads the stored log and trace files, and also keeps them on local disk under the retention limits. When a query needs a file that retention has deleted from local disk, the plugin downloads it from remote storage into a download cache shared by logs and traces. The plugin waits at most 30 seconds plus one second per MiB for each download, and never more than 5 minutes. A file that takes longer, or that the plugin cannot download, is left out of the answer (traces mark the answer as partial). A query that needs more data than the cache can hold fails with a message to narrow its time range.

remote_storage.uri​

Put non-secret backend options in the query string. Never put credentials in this file or URI. For S3, use the standard AWS environment, credentials file, or instance-role mechanisms available to the Netdata service account.

via File​

The configuration file name for this integration is otel.yaml.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config otel.yaml
Examples​
Accept remote senders with mutual TLS​

Bind beyond loopback only when the endpoint is protected. This example requires each sender to present a certificate signed by the configured CA. Each listener has its own TLS settings. If no sender uses one of the listeners, set its enabled option to false instead.

receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
tls:
cert_file: /etc/netdata/ssl/server-cert.pem
key_file: /etc/netdata/ssl/server-key.pem
client_ca_file: /etc/netdata/ssl/client-ca.pem
http:
enabled: true
endpoint: "0.0.0.0:4318"
tls:
cert_file: /etc/netdata/ssl/server-cert.pem
key_file: /etc/netdata/ssl/server-key.pem
client_ca_file: /etc/netdata/ssl/client-ca.pem

Configure log and trace retention​

A user file needs only the fields that change; omitted fields keep their stock values. Logs and traces have separate settings, each tenant entry inherits omitted fields from default, and a per-tenant key must match the sender's X-Scope-OrgID value when auth.enabled is true.

Config
logs:
retention:
default:
max_total_size: "10GB"
max_age: "30 days"
traces:
retention:
default:
max_total_size: "10GB"
max_age: "30 days"

Metric mapping file​

Without a mapping, all data point attributes contribute to chart identity and the dimension is named value. A mapping selects one string attribute as the dimension name and removes it from chart identity, grouping data points that differ only by that attribute into one multi-dimension chart. Place user mapping files in metrics.chart_configs_dir; omit instrumentation_scope when the metric name alone is unambiguous.

Config
metrics:
"system.cpu.time":
- instrumentation_scope:
name: .*hostmetricsreceiver.*cpuscraper$
dimension_attribute_key: state
interval_secs: 5
"system.network.io":
- instrumentation_scope:
name: .*hostmetricsreceiver.*networkscraper$
dimension_attribute_key: direction
"redis.cpu.time":
- dimension_attribute_key: state

Alerts​

There are no alerts configured by default for this integration.

Metrics​

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Metrics are created dynamically from supported OpenTelemetry data. The exact charts depend on the received metric names, attributes, instrumentation scopes, and mapping rules.

The plugin also makes charts about the data it receives, listed below. These charts count the records of an export when the plugin accepts the export, even when it rejects some of its records. When the plugin cannot store the records of an export of logs or spans (a storage error), the records show as failed, and the sender sends them again. An export that fails more than one time counts each time. A request that the plugin rejects as a whole because it is too large is on no dimension: the sender gets the error. An export that a stop of the Agent cuts is also on no dimension. An export that fails for another reason does not show on these charts.

The plugin has two worker processes: the ingestor receives the data and writes it, and the ledger indexes it. The ingestor tells the ledger about each file it creates, syncs, and closes. netdata.otel.wal_events_queued shows how many of these messages wait to be sent. The number is near zero when the ledger keeps up. When it grows, the ledger is slower than the ingestor, and the memory of the ingestor grows with it.

The charts of the index files (netdata.otel.index_files, netdata.otel.index_size) are totals of all tenants, and the plugin updates them every 30 seconds. netdata.otel.tenants shows how many tenants these totals cover: each tenant has its own files and its own retention limits, so the disk that the plugin needs grows with the number of tenants. The plugin keeps a tenant in this number after the data of the tenant expires. The retention limits are per tenant: compare the totals with the limits that you configured.

Three charts exist only when remote_storage.enabled is true. netdata.otel.index_files_waiting_for_remote shows the files that are over a retention limit while remote storage did not confirm their catalog entry yet. Such files stay on the local disk, so the disk can hold more than the limits while remote storage is unreachable. netdata.otel.uploads and netdata.otel.uploads_finished count the uploads of index files and of catalog files together. An upload is in_progress from the moment the plugin starts it until it ends; the plugin runs a few uploads at the same time and the others wait for their turn. An upload that fails shows as failed each time it fails, and as waiting_for_retry until the plugin tries it again.

The plugin also makes charts for each service, from the traces and the logs that it receives: requests, failed requests and request durations from traces, and log records by severity from logs. A service is the service.namespace and service.name resource attributes, kept apart by tenant and deployment environment. The contexts of these charts start with otel.derived_metrics.. These charts have no settings. They count as follows:

  • A request is a span that starts a trace, or a span of kind SERVER or CONSUMER. A request failed when its span status is ERROR. A request counts at the time it ends.
  • Spans sampled with an OpenTelemetry sampling threshold (ot=th in the trace state) count for the number of spans that they represent.
  • A log record counts at its time (its own time, else its observed time, else the time it arrived), in the severity range of its severity number. A record without a severity number counts as unspecified, also when it has a severity text. The sender or the OpenTelemetry Collector must set the severity number.
  • The average request duration is the mean duration of the requests that ended in each second. Over a longer time, the chart shows the mean of these per-second values. This is not the mean duration of all the requests in that time.
  • The charts count only the spans and the log records that the plugin stores. When an SDK or a Collector drops spans by sampling and does not write a sampling threshold, the request rates are lower than the real rates, and the share of failed requests can change.
  • The charts show data with a delay of 10 seconds, so that data that arrives late still counts in the correct second. Data that arrives later than that counts in the oldest second that is still open, and data with a time in the future counts in the current second. The "Adjusted log records" and "Adjusted requests" charts show this data as late and future.

At most 500 services have their own charts, for traces and for logs separately. The records of more services go to one shared overflow chart set. When a service sends no data for 15 minutes, its charts are removed and its place becomes free. The "Services with derived metrics" and "Expired services" charts count the overflow chart set as one service. The "Adjusted log records" and "Adjusted requests" charts show its data as overflowed. Such a record or request can also count as late or future. The "Adjusted requests" chart counts each stored request span as one, also when a sampling threshold makes it count for more on the request charts.

Per OpenTelemetry instance​

The plugin itself, with charts of the metric data points, log records and spans that it receives, of the per-service charts that it makes from logs and traces, of the indexing operations that make stored logs and traces searchable, of the files that it keeps on the local disk, and of its uploads to remote storage.

This scope has no labels.

Metrics:

MetricDescriptionDimensionsUnit
netdata.otel.metric_data_pointsMetric data points receivedaccepted, rejecteddata points/s
netdata.otel.log_recordsLog records receivedaccepted, rejected, failedrecords/s
netdata.otel.spansSpans receivedaccepted, rejected, failedspans/s
netdata.otel.wal_events_queuedWAL file events waiting to be sent to the ledgereventsevents
netdata.otel.derived_servicesServices with derived metricslogs, tracesservices
netdata.otel.derived_expired_servicesExpired serviceslogs, tracesservices/s
netdata.otel.derived_adjusted_log_recordsAdjusted log recordslate, future, overflowedrecords/s
netdata.otel.derived_adjusted_requestsAdjusted requestslate, future, overflowedrequests/s
netdata.otel.indexing_operationsIndexing operationsrunning, waitingoperations
netdata.otel.indexing_operations_oldest_waitLongest wait of a waiting indexing operationwaitseconds
netdata.otel.indexing_operations_finishedFinished indexing operationssucceeded, failedoperations/s
netdata.otel.index_filesLocal index fileslogs, tracesfiles
netdata.otel.index_sizeLocal index sizelogs, tracesbytes
netdata.otel.tenantsTenantslogs, tracestenants
netdata.otel.index_files_waiting_for_remoteIndex files waiting for remote storagelogs, tracesfiles
netdata.otel.uploadsUploads to remote storagein_progress, waiting_for_retryoperations
netdata.otel.uploads_finishedFinished uploads to remote storagesucceeded, failedoperations/s

Per service​

One service, identified by its resource attributes and tenant.

Labels:

LabelDescription
resource.attributes.service.nameThe service.name resource attribute, or unknown_service when it is not set. Not present on the overflow chart set.
resource.attributes.service.namespaceThe service.namespace resource attribute. Not present when it is not set, and not on the overflow chart set.
resource.attributes.deployment.environment.nameThe deployment.environment.name resource attribute, or the older deployment.environment. Not present when neither is set, and not on the overflow chart set.
tenantThe tenant of the sender (default when tenant authentication is disabled). Not present on the overflow chart set.
otel.metric.overflowSet to true only on the chart set that holds the services over the limit of 500 services.

Metrics:

MetricDescriptionDimensionsUnit
otel.derived_metrics.log_recordsLog recordsfatal, error, warn, info, debug, trace, unspecifiedrecords/s
otel.derived_metrics.requestsRequestssuccess, errorrequests/s
otel.derived_metrics.request_duration_avgRequest duration averageavgseconds
otel.derived_metrics.request_duration_maxRequest duration maximummaxseconds
otel.derived_metrics.request_duration_histogramRequest duration histogram0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10, +Infrequests/s

Troubleshooting​

Known Errors​

failed to bind OTLP/gRPC endpoint 127.0.0.1:4317: Address already in use (os error 98)​

When

Logged in the Agent journal at startup by the otel-plugin/ingestor worker, followed by exited with error code 1 and haven't collected any data. Disabling it. from the Agent. A conflict on the OTLP/HTTP listener fails the same strict way: failed to bind OTLP/HTTP endpoint 127.0.0.1:4318: Address already in use (os error 98).

Cause

Another process already listens on the configured address. Typical sources are a standalone OpenTelemetry Collector on the same host — its own OTLP receivers default to the same conventional ports, 4317 for gRPC and 4318 for HTTP — or several Netdata Agents running in containers that share the host network namespace (--network host), where only one of them can own a given port. The plugin binds both listeners before it advertises anything, so on a conflict it exits once and the Agent disables it until the next Agent restart.

Fix

Find the owner with ss -ltnp 'sport = :4317' (or sport = :4318 for the OTLP/HTTP listener). Then stop that listener, or give this Agent its own addresses in otel.yaml and point every OTLP sender at the new ports. You can also turn off a listener that no sender uses:

receivers:
otlp:
protocols:
grpc:
endpoint: 127.0.0.1:14317
http:
endpoint: 127.0.0.1:14318
# or keep the OTLP/HTTP listener off:
# enabled: false

When several Agents share one network namespace and only one needs to receive OTLP, disable the plugin on the others in netdata.conf:

[plugins]
otel = no

Other Problems​

The plugin does not start​

Check the Agent journal for otel-plugin configuration errors. User YAML and NETDATA_OTEL_CFG_* variables are validated strictly, so a typo or a removed logs option from the previous otel.yaml layout stops startup. For a source install, confirm that a compatible Rust toolchain is available; Linux installs using netdata-installer.sh also require --enable-plugin-otel.

The endpoint is reachable but no data appears​

A successful TCP connection proves only that something is listening. Check that the Collector's exporter uses the right port: 4317 for otlp_grpc, 4318 for otlp_http. The otlp_http exporter needs the OTLP/HTTP listener, which is off by default. With the default endpoints, use 127.0.0.1, not localhost, which can resolve to IPv6. Then look for rejected exports in the Collector log, and verify the resulting chart, log entry, or trace.

A metric does not create the expected chart​

Exponential histograms are not currently ingested. The chart netdata.otel.metric_data_points shows the data points that reached a chart on its accepted dimension and the data points that the plugin dropped on its rejected dimension. A data point without a value is on neither, and so is a gauge or sum data point whose value is not a finite number (NaN, infinity). A histogram or summary data point with such a number in its sum, minimum, maximum or a quantile is charted without that number. For other supported metrics, inspect the Agent journal for rejected user mapping files and verify the metric name, instrumentation scope, and dimension_attribute_key. The resulting chart context is otel.<metric-name>.

Large exports are rejected​

The plugin rejects an export request larger than 4 MiB as a whole, and the Collector drops that batch. The plugin also rejects a request of hundreds of thousands of log records or spans that are almost empty (the answer is InvalidArgument or HTTP 400, and its message gives the number of records). In the two cases, make the batches smaller. On the exporter, cap them by size with sending_queue: { batch: { sizer: bytes, max_size: 3000000 } }. Alternatively, lower send_batch_max_size on the batch processor; it counts items (spans, data points, log records), not bytes.

The Collector reports Unavailable or HTTP 503 for logs or traces​

The plugin could not store the data: its disk is full, or a write to its disk failed. It answers UNAVAILABLE (HTTP 503) with WAL write error, WAL sync error or WAL writer creation failed, and the Agent journal has an error from otel-plugin/ingestor. The Collector keeps the batch and sends it again, so no data is lost while its retry settings allow it. The charts netdata.otel.log_records and netdata.otel.spans show these records on their failed dimension. Make room on the disk that holds base_dir, or repair it. One failure can refuse more than one batch of the same tenant, and a refused batch can be on the disk in part or in full. When the Collector sends such a batch again, the plugin stores it again, so some records can appear two times. Delivery is at least once: see OTLP ingestion for all the causes of copies.

A different case has the same look in the Collector's log: the Agent stopped or restarted. When it stops, the plugin first answers the exports that are in progress, for up to half a second. An export that it could not answer in that time usually ends with a closed connection and no answer; in rare cases the answer is UNAVAILABLE with the plugin is stopping. The same happens to an export in progress when the Agent or the host stops suddenly. Exports that come while the plugin is down get a refused connection. This is not a disk problem, and these records are not on the failed dimension. The plugin can have the records of an export that was cut on its disk already. When the Collector sends the batch again after the restart, the plugin stores it again, so these records can appear two times.

Some exported metric data points are missing​

The plugin rejects a metric data point when its timestamp is further from the current time than the expiry of its chart (metrics.expiry_duration_secs, 15 minutes by default), in the past or in the future. The chart netdata.otel.metric_data_points shows these data points on its rejected dimension, and the plugin logs a warning in the Agent journal. Check the clock of the host that produced the metrics. The plugin also rejects a data point when the name of its metric or of its dimension is longer than 255 bytes, and when its chart already has 4,096 dimensions. For a histogram or a summary, the plugin leaves out the buckets or quantiles that do not fit and does not reject the data point. A chart keeps at most 1,024 labels. When the attributes of a metric give more labels, the plugin leaves out the rest.

Some exported logs are missing​

By default, the plugin rejects log records timestamped more than 24 hours in the past or more than 10 minutes in the future. The Collector logs a Partial success response warning for rejected records. Check the clock of the host that produced the logs, how old the backfilled records are, the Collector log, and the Agent journal.

Some exported spans are missing​

By default, the plugin rejects a span that started more than 24 hours ago or ends more than 10 minutes in the future; a span without an end time is judged by its start. The Collector logs a Partial success response warning for rejected spans, and the plugin logs a warning in the Agent journal. Check the clock of the instrumented application's host, how old the backfilled spans are, the Collector log, and the Agent journal.

Logs written by the previous OpenTelemetry plugin are not visible​

A removed logs option from the previous otel.yaml layout stops the current plugin and prints a migration guide. Replace it with a partial configuration based on the current stock file. If the former file set a custom logs.journal_dir, keep it: it is the only former logs option still accepted, and the plugin uses it only to locate the former plugin's read-only journals.


Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.