X.509 certificate
Plugin: go.d.plugin Module: x509check
Overview
This collector tracks how long X.509 (SSL/TLS) certificates remain valid and whether they have been revoked.
It reads the certificate that a website or any other TLS service presents, including mail servers that use STARTTLS, or a certificate stored in a local file. It charts the time left until the certificate expires and, optionally, the expiration of every certificate in the chain and the revocation status of the server certificate, with the outcome of each revocation check. Stock alerts fire before a certificate expires, when it is revoked, and when revocation checks keep failing for a reason that can be fixed: a failed request, an unusable answer, or a missing issuer certificate.
Every collection interval, each job connects to its source and completes a TLS handshake, after a
STARTTLS command for smtp:// sources, to read the certificates the server presents. A file:// source
is read from disk instead. The collector never requests a page or sends mail.
With check_revocation_status enabled, the collector also downloads the CRLs and queries the OCSP
responders listed in the server certificate. It verifies their answers with the certificate of the issuer
that signed the server certificate: one the server sends or that TLS verification of the server certificate
reached, such as a root CA from tls_ca, or else the one downloaded from the issuer URL listed in the server
certificate. These requests go to the HTTP or HTTPS URLs in the certificate, so the Agent host needs
outbound access to them. Each request times out after 10 seconds; timeout does not apply
to them. Each job reuses a CRL's result for a certificate until the next update time the CRL states.
This collector is supported on all platforms.
This collector supports collecting metrics from multiple instances of this integration.
| Permission | Needed when |
|---|---|
The netdata user can read the file | tls_ca, tls_cert or tls_key is set |
Default Behavior
Auto-Detection
This integration doesn't support auto-detection.
Limits
- The handshake verifies the certificate like any TLS client. A certificate that is expired, self-signed,
or issued for another name fails collection: a running job's charts show gaps instead of the time left,
and a job that starts in this state stays stopped unless
autodetection_retryis set. Settls_cafor a private CA, ortls_skip_verifyto read such certificates anyway. - A
file://source provides only the first certificate in the file, socheck_full_chainhas no effect there. - Revocation status is checked for the server certificate only, and an OCSP
unknownanswer is reported as revoked. Only the HTTP and HTTPS URLs a certificate lists are used, not LDAP or other ones, so a certificate that lists no HTTP or HTTPS CRL or OCSP URL has no revocation status to check: the revocation status chart shows a gap and the revocation check status showsno_revocation_info. - For a certificate that lists CRL or OCSP URLs, revocation status comes only from CRLs and OCSP answers
that verify against the issuer of the server certificate and are current by the Agent host's clock,
within 5 minutes. The issuer certificate must be one the server sends, one that TLS verification reached
through
tls_caor the system's trusted CAs, or the one at the issuer URL listed in the server certificate. Without it the revocation status chart shows a gap, for example withtls_skip_verifyfor a private CA whose certificates list no issuer URL, behind a server that does not send the CA certificate, or for afile://source whose certificate lists none. The revocation status chart also shows a gap when the CA publishes outdated CRLs or OCSP answers, and can show one when the Agent host's clock is more than 5 minutes off. - With an
smtp://source,timeoutcovers the whole SMTP exchange. A mail server that delays its greeting, a common anti-spam measure on port 25, needs atimeoutlonger than that delay.
Performance Impact
The default configuration for this integration is not expected to impose a significant performance impact on the system.
Setup
You can configure the x509check collector in two ways:
| Method | Best for | How to |
|---|---|---|
| UI | Fast setup without editing files | Go to Nodes → Configure this node → Collectors → Jobs, search for x509check, then click + to add a job. |
| File | If you prefer configuring via file, or need to automate deployments (e.g., with Ansible) | Edit go.d/x509check.conf and add a job. |
UI configuration requires paid Netdata Cloud plan.
Prerequisites
No action required.
Configuration
Options
The following options can be defined globally: update_every, autodetection_retry.
Config options
| Group | Option | Description | Default | Required |
|---|---|---|---|---|
| Target | source | Where to read the certificate, as a URL: https, tcp, tcp4 or tcp6 for a TLS server, smtp for an SMTP server with STARTTLS, file for a local PEM file. | yes | |
| timeout | Certificate retrieval timeout, in seconds. Covers the connection, the SMTP exchange and the TLS handshake. | 2 | no | |
| TLS | tls_server_name | Host name sent to the server (SNI) and checked against the certificate, instead of the host in source. Leave empty to use the host in source. | no | |
| tls_skip_verify | Skip TLS certificate and hostname verification, so certificates that are self-signed, expired or issued for another name are still read. | no | no | |
| tls_ca | Path to a CA bundle used to verify the server certificate. Leave empty to use the system trust store. | no | ||
| tls_cert | Path to a client certificate file for mutual TLS. Requires a client key. | no | ||
| tls_key | Path to the client key file for mutual TLS. Requires a client certificate. | no | ||
| Checks | check_full_chain | Report the expiration time of every certificate the server sends, such as intermediates, not only the server certificate. | no | no |
| check_revocation_status | Check whether the server certificate has been revoked, using the CRL and OCSP endpoints listed in it. | no | no | |
| Collection | update_every | Data collection interval, in seconds. | 60 | no |
| autodetection_retry | How often to retry the initial connection when the job fails to start, in seconds. Zero disables retries. | 0 | no | |
| Virtual Node | vnode | Associates this job with a Virtual Node. | no |
source
Network sources need a port. The host name is sent for SNI and, unless tls_skip_verify is set, checked
against the certificate. To use another name for both while connecting to the same address, set
tls_server_name.
| Scheme | Use for | Example |
|---|---|---|
https, tcp | Websites and services that start TLS on connect (LDAPS, IMAPS, SMTPS on port 465) | https://example.com:443 |
tcp4, tcp6 | The same, over IPv4 or IPv6 only | tcp6://example.com:443 |
smtp | SMTP servers that upgrade with STARTTLS (ports 25 and 587) | smtp://smtp.example.com:587 |
file | A PEM file on the Agent host; note the three slashes before an absolute path | file:///etc/ssl/certs/example.com.pem |
tls_server_name
The job still connects to the host and port in source, so it can read the certificate issued for one
name from a server reached at another address:
- each site hosted on one IP address and port, with one job per site;
- an origin server behind a load balancer or CDN, reached at its own address;
- each node of a pool, reached at its address and verified against the shared host name.
A host name set here is sent as SNI even when source is an IP address. An IP address set here sends
no SNI and is checked against the IP addresses listed in the certificate. A file:// source does not
accept this option.
check_full_chain
Servers usually leave the root certificate out of the chain they send, so it is charted only when the server includes it.
check_revocation_status
The revocation charts exist only for the server certificate (depth 0). The revocation status chart
appears with the first check that determines a status, and shows a gap when a check cannot. The
revocation check status chart shows the outcome of every check:
| State | Meaning |
|---|---|
success | The revocation status was determined. |
request_failed | A CRL, OCSP or issuer certificate request failed (DNS, connection, timeout or HTTP status); with several OCSP responders or issuer URLs, every request failed. |
invalid_answer | A CRL or OCSP answer could not be used: it is larger than 64 MiB, cannot be parsed, is an OCSP error status such as unauthorized or tryLater, or failed verification (signature, issuer, scope or freshness). |
issuer_not_found | The certificate of the CA that issued the server certificate is unavailable: the source did not send it, TLS verification did not reach it, and the server certificate lists no HTTP or HTTPS issuer URL, or one that does not serve that certificate. |
no_revocation_info | The certificate lists no HTTP or HTTPS CRL or OCSP URL, so there is nothing to check. |
outside_validity | The certificate is expired or not yet valid. |
Only request_failed, invalid_answer and issuer_not_found count toward the stock alert on failing
revocation checks: the others have no fix in the check, and the expiration alerts cover validity.
via UI
Configure the x509check collector from the Netdata web interface:
- Go to Nodes.
- Select the node where you want the x509check data-collection job to run and click the ⚙ (Configure this node). That node will run the data collection.
- The Collectors → Jobs view opens by default.
- In the Search box, type x509check (or scroll the list) to locate the x509check collector.
- Click the + next to the x509check collector to add a new job.
- Fill in the job fields, then click Test to verify the configuration and Submit to save.
- Test validates the provided settings and checks the collector's startup prerequisites. Successful validation does not guarantee that every metric will be available during collection.
- If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.
via File
The configuration file name for this integration is go.d/x509check.conf.
The file format is YAML. Generally, the structure is:
update_every: 1
autodetection_retry: 0
jobs:
- name: some_name1
- name: some_name2
You can edit the configuration file using the edit-config script from the
Netdata config directory.
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/x509check.conf
Examples
Website
The certificate of an HTTPS website.
Config
jobs:
- name: example_com
source: https://example.com:443
Other TLS service
A service that starts TLS on connect, such as LDAPS, IMAPS or SMTPS on port 465.
Config
jobs:
- name: ldap_example_com
source: tcp://ldap.example.com:636
SMTP server with STARTTLS
A mail server that upgrades the connection with STARTTLS.
Config
jobs:
- name: smtp_example_com
source: smtp://smtp.example.com:587
Local certificate file
A PEM certificate file on the Agent host.
Config
jobs:
- name: example_com_file
source: file:///etc/ssl/certs/example.com.pem
Full chain and revocation status
Also track the intermediate certificates the server sends, and check whether the server certificate has been revoked.
Config
jobs:
- name: example_com
source: https://example.com:443
check_full_chain: yes
check_revocation_status: yes
Self-signed certificate
A certificate the Agent host does not trust. To verify it against your own CA instead, set tls_ca.
Config
jobs:
- name: internal_example
source: https://internal.example.com:443
tls_skip_verify: yes
Several sites on one address
A server that hosts several sites on one IP address and port, checked locally: every job connects to the same address and asks for a different site.
Config
jobs:
- name: site_a_local
source: https://127.0.0.1:443
tls_server_name: site-a.example.com
- name: site_b_local
source: https://127.0.0.1:443
tls_server_name: site-b.example.com
Multi-instance
Note: When you define multiple jobs, their names must be unique.
Certificates of several websites.
Config
jobs:
- name: example_com
source: https://example.com:443
- name: example_org
source: https://example.org:443
- name: example_net
source: https://example.net:443
Alerts
The following alerts are available:
| Alert name | On metric | Description |
|---|---|---|
| x509check_days_until_expiration | x509check.time_until_expiration | SSL cert expiring soon (${label:source} cn:${label:common_name}) |
| x509check_revocation_status | x509check.revocation_status | SSL cert revoked (${label:source}) |
| x509check_revocation_check_failures | x509check.revocation_check_status | Percentage of SSL cert revocation checks in the last 30 minutes that failed on a request, an unusable answer or a missing issuer certificate (${label:source}) |
Metrics
Metrics grouped by scope.
The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.
When a certificate cannot be read, its charts show gaps rather than zeros. When its revocation status cannot be determined, the revocation status chart shows a gap and the revocation check status chart shows why.
Per certificate
One certificate presented by the source; only the server certificate unless check_full_chain is enabled.
Labels:
| Label | Description |
|---|---|
| source | The source option of the job. |
| common_name | Common name (CN) of the certificate subject, for example example.com. |
| depth | Position of the certificate in the chain, 0 for the server certificate and higher values for its issuers toward the root. |
Metrics:
| Metric | Description | Dimensions | Unit |
|---|---|---|---|
| x509check.time_until_expiration | Time Until Certificate Expiration | expiry | seconds |
| x509check.revocation_status | Revocation Status | not_revoked, revoked | boolean |
| x509check.revocation_check_status | Revocation Check Status | success, request_failed, invalid_answer, issuer_not_found, no_revocation_info, outside_validity | status |
Troubleshooting
Diagnostics
Debug Mode
Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.
To troubleshoot issues with the x509check collector, run the go.d.plugin with the debug option enabled. The output
should give you clues as to why the collector isn't working.
-
Navigate to the
plugins.ddirectory, usually at/usr/libexec/netdata/plugins.d/. If that's not the case on your system, opennetdata.confand look for thepluginssetting under[directories].cd /usr/libexec/netdata/plugins.d/ -
Switch to the
netdatauser.sudo -u netdata -s -
Run the
go.d.pluginto debug the collector:./go.d.plugin -d -m x509checkTo debug a specific job:
./go.d.plugin -d -m x509check -j jobName
Getting Logs
If you're encountering problems with the x509check collector, follow these steps to retrieve logs and identify potential issues:
- Run the command specific to your system (systemd, non-systemd, or Docker container).
- Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.
System with systemd
Use the following command to view logs generated since the last Netdata service restart:
journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep x509check
System without systemd
Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:
grep x509check /var/log/netdata/collector.log
Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.
Docker Container
If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:
docker logs netdata 2>&1 | grep x509check
Known Errors
config validation: source is not set
Cause
The job has no source option.
Fix
Set source to the URL of the certificate, for example https://example.com:443.
config validation: timeout must be greater than zero
Cause
timeout is zero or negative.
Fix
Set timeout to a positive number of seconds, or remove it to use the default.
certificate provider init: unsupported scheme scheme in source source
Cause
source does not start with a supported scheme. A plain http:// URL, a udp:// URL (TLS does not run over UDP), or a host name and port without a scheme produce this error.
Fix
Use one of the schemes listed for the source option, for example https://example.com:443.
certificate provider init: source parse: parse source: first path segment in URL cannot contain colon
Cause
source is an IP address and port without a scheme.
Fix
Add the scheme, for example tcp://192.0.2.10:443.
certificate provider init: tls_server_name cannot be used with a file:// source
Cause
tls_server_name is set on a job whose source is a local file, which has no server to ask for a certificate.
Fix
Remove tls_server_name from the job, or point source at the server that presents the certificate.
error on dial to host: dial tcp: address host: missing port in address
Cause
A network source has no port; the collector does not assume one from the scheme.
Fix
Add the port, for example https://example.com:443 or smtp://smtp.example.com:587.
error on dial to host:port: dial tcp: lookup host: no such host
When
The resolver address can appear after the host name, as in lookup example.com on 127.0.0.53:53.
Cause
The host name in source does not resolve from the Agent host.
Fix
Check the host name, and the DNS configuration of the Agent host.
error on dial to host:port: dial tcp address: connect: connection refused
Cause
Nothing listens on that port, or a firewall rejects the connection from the Agent host.
Fix
Check the host and port in source, and that the Agent host can reach them.
error on dial to host:port: dial tcp address: i/o timeout
Cause
The connection was not established within timeout, usually because a firewall drops it silently.
Fix
Check that the Agent host can reach the host and port in source. Otherwise raise timeout.
error on SSL handshake with host:port: context deadline exceeded
Cause
The server did not complete the TLS handshake within timeout. The port may expect a STARTTLS protocol first, or the server is slow or filtered.
Fix
Check that the port starts TLS on connect. For SMTP on ports 25 or 587, use an smtp:// source. Otherwise raise timeout.
error on creating SMTP client: read tcp address: i/o timeout
Cause
The server sent no SMTP greeting within timeout. A port that starts TLS on connect, such as SMTPS on port 465, never sends a plain greeting, and some mail servers delay their greeting to slow down spam.
Fix
For a port that starts TLS on connect, use tcp:// instead of smtp://. Otherwise raise timeout.
error on startTLS with host:port: reply
Cause
The SMTP server rejected the STARTTLS command, usually because it does not offer STARTTLS on that port.
Fix
Check that the server offers STARTTLS on that port. For a port that starts TLS on connect, such as 465, use tcp:// instead of smtp://.
tls: first record does not look like a TLS handshake
Cause
The port answers in plain text, for example HTTP on port 80 or SMTP before STARTTLS.
Fix
Point source at the TLS port, or use an smtp:// source for SMTP with STARTTLS.
tls: server selected unsupported protocol version version
Cause
The server offers only TLS 1.0 or 1.1, which the collector does not accept.
Fix
Enable TLS 1.2 or later on the server. Until then, monitor the certificate file with a file:// source on a host that can read it.
tls: failed to verify certificate: x509: certificate signed by unknown authority
Cause
The certificate is self-signed or issued by a CA that the Agent host does not trust.
Fix
Set tls_ca to the CA certificate that issued it, or set tls_skip_verify: yes.
tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time now is after not_after
Cause
The certificate has expired, or the clock of the Agent host is wrong. While verification fails, a running job's charts show gaps instead of the time left.
Fix
Renew the certificate, or correct the clock. To keep charting an expired certificate, set tls_skip_verify: yes.
tls: failed to verify certificate: x509: certificate is valid for names, not host
Cause
The host name in source, or tls_server_name when it is set, is not one of the names in the certificate.
Fix
Use a host name the certificate covers, in source or in tls_server_name, or set tls_skip_verify: yes.
tls: failed to verify certificate: x509: cannot validate certificate for ip because it doesn't contain any IP SANs
Cause
The certificate is checked against an IP address, from source or tls_server_name, and lists only host names.
Fix
Use the host name of the service in source, or keep the address in source and set tls_server_name to that host name.
tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead
Cause
The certificate names its host only in the Common Name and has no Subject Alternative Names.
Fix
Reissue the certificate with Subject Alternative Names, or set tls_skip_verify: yes.
error on reading path: open path: permission denied
Cause
The collector cannot read the certificate file or one of its parent directories.
Fix
Grant the netdata user read access to the file, for example setfacl -m u:netdata:r /etc/ssl/certs/example.com.pem.
error on reading path: open path: no such file or directory
Cause
The file does not exist at that path. A file:// URL with two slashes before an absolute path, such as file://etc/ssl/cert.pem, drops the first directory.
Fix
Use three slashes before an absolute path, for example file:///etc/ssl/certs/example.com.pem.
error on decoding path: PEM decode failed (not a valid PEM file)
Cause
The file is not in PEM format, for example a binary DER certificate.
Fix
Convert it to PEM: openssl x509 -inform der -in cert.der -out cert.pem.
error on parsing certificate path: invalid X.509 certificate
Cause
The first PEM block in the file is not a certificate, for example a private key.
Fix
Point source at a file whose first PEM block is the certificate.
create tls config: could not read certificate path: reason
Cause
The file set in tls_ca or tls_cert does not exist, or the netdata user cannot read it.
Fix
Check the path, and grant the netdata user read access to the file.
create tls config: could not read key path: reason
Cause
The file set in tls_key does not exist, or the netdata user cannot read it.
Fix
Check the path, and grant the netdata user read access to the file.
create tls config: could not parse any PEM certificates path
Cause
The file set in tls_ca contains no PEM certificate.
Fix
Point tls_ca at a PEM file with the CA certificates.
create tls config: could not parse keypair cert:key
Cause
The files set in tls_cert and tls_key are not a matching PEM certificate and private key.
Fix
Point tls_cert and tls_key at the certificate and the private key of the same pair.
revocation status of source could not be determined: issuer certificate not found: reason
When
check_revocation_status is enabled.
Cause
The certificate of the CA that issued the server certificate is unavailable: the server does not send it, TLS verification did not reach it (for example, tls_skip_verify is set or the source is file://), and the server certificate lists no HTTP or HTTPS issuer URL, or the one it lists is unreachable or does not serve that certificate as a single DER or PEM certificate; reason says which. The revocation status chart shows a gap, and the revocation check status shows request_failed when every issuer URL request failed, otherwise issuer_not_found.
Fix
Configure the server to send the CA certificate that issued its certificate, or set tls_ca to that CA certificate and remove tls_skip_verify. Otherwise check that the Agent host can reach the issuer URL listed in the certificate. A file:// source does not use tls_ca: use a certificate that lists its issuer URL, or set check_revocation_status to no for the job.
revocation status of source could not be determined: no usable OCSP answer: reason
When
check_revocation_status is enabled.
Cause
No OCSP responder listed in the certificate returned a usable answer: one about this certificate, current within 5 minutes, and signed by the issuer or by a responder it authorized for OCSP signing; reason gives each responder's problem. The revocation status chart shows a gap, and the revocation check status shows request_failed when every request failed, otherwise invalid_answer.
Fix
Check that the Agent host can reach the OCSP URLs listed in the certificate and that its clock is correct. For an outdated since reason, the responder serves stale answers; report it to the CA.
revocation status of source could not be determined: the certificate lists no HTTP or HTTPS CRL or OCSP URL
When
check_revocation_status is enabled.
Cause
The server certificate lists no OCSP responder and no CRL that the collector uses (it uses only HTTP and HTTPS URLs, not LDAP ones), so its revocation status cannot be checked. The revocation status chart shows a gap, and the revocation check status shows no_revocation_info.
Fix
Set check_revocation_status to no for this job. For a certificate of your own CA, have the CA include an HTTP CRL distribution point or an OCSP URL in the certificates it issues.
revocation status of source could not be determined: reason
When
check_revocation_status is enabled.
Cause
A CRL listed in the certificate could not be downloaded, parsed or verified, and reason names its URL and the problem; or the certificate is expired or not yet valid, and reason gives the time. The revocation status chart shows a gap. The revocation check status shows request_failed for a CRL that could not be downloaded, invalid_answer for one over 64 MiB or that could not be parsed or verified, and outside_validity for the certificate's validity. A CRL must be a complete version 2 CRL (not a delta or indirect CRL) up to 64 MiB, issued and signed by the certificate's issuer, covering the server certificate, and current within 5 minutes.
Fix
Check that the Agent host can reach the CRL URLs listed in the certificate and that its clock is correct, and renew an expired certificate. For an unsupported crl version reason, have the CA issue version 2 CRLs (with OpenSSL, set crlnumber in the CA configuration). For an outdated since reason, have the CA publish a current CRL.
Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.