Log events
The Nexus Telemetry Fleet server and collector write their logs as one JSON
object per line. Every event line has an event field with a fixed
name, and this page lists those names. The names are stable. Log pipelines and alert rules can key on them. A name
is only renamed or removed as a documented change. The names are the
product’s own, and some use older internal words: board for the
dashboard, beat for a summary, verdict for a state. The text on this
page uses the dashboard’s words.
Each event has a level. ERROR is a failure that requires action. WARN
is a fault the product has worked around. Read its action column.
INFO is a lifecycle step, a state change or a heartbeat. DEBUG is
per-reading detail. TRACE is every message sent and received. At the default NF_LOG=info
a healthy fleet server logs its startup, state changes, and an hourly
heartbeat, and little else.
No log ever contains a position, a token, key material, a licence key, or anything that identifies a person at a site. Readings are not logged.
See Private deployment for reading the fleet server’s log, and Files and repositories for where each log is.
Process and configuration
| Event | Level | Fires when | Action |
|---|---|---|---|
process.panic |
ERROR | The process crashed. It then exits | The service manager restarts it. Report the location field, the place in the code, to support. This is always a bug |
log.initialised |
INFO | Logging started | None. The line records the log format, level and file |
log.config_fallback |
WARN | NF_LOG, NF_LOG_FORMAT or NF_HEARTBEAT_SECS was invalid, or the log directory could not be used, and the default applied |
Fix the variable or the directory |
config.fallback |
WARN | An invalid configuration variable, NF_POLL_INTERVAL for example, was ignored |
Fix the variable |
server.started |
INFO | The fleet server started | None. The line records the version, the data directory, the addresses it listens on and the number of collectors |
collector.started |
INFO | The collector started | None. The line records the version, the collector id, the data directory and the fleet server address |
collector.dish_addr |
INFO | At startup. The line records the address the collector uses to read the terminal | None. Read it when a terminal never appears on the dashboard |
server.heartbeat |
INFO | Every hour, or every NF_HEARTBEAT_SECS |
None. Alert when no heartbeat has arrived for twice the interval |
collector.heartbeat |
INFO | Every hour, or every NF_HEARTBEAT_SECS |
None. Alert when no heartbeat has arrived for twice the interval |
collector.start_failed |
ERROR | On Windows, the collector service stopped because the collector could not start. The error field says why |
Read the error. A failed storage conversion names the file and the step, leaves the file unchanged, and is tried again at the next start |
health.degraded_entered |
WARN | The fleet server became degraded: a health check failed | Investigate the check named in the line. /healthz answers 503 until the check passes |
health.degraded_cleared |
INFO | Every health check passes again | None |
server.drain_complete |
INFO | At shutdown, the dashboard stopped accepting connections and closed the open ones. finished counts those that ended on their own and cut those still open at the end of the grace period |
None. A cut above zero at every restart is a client that holds its connection open past the grace period |
storage.legacy_storage_refused |
ERROR | The fleet server found storage written before 0.0.1-beta.18, encrypted in the file, and this release cannot read it. It stops | Start 0.0.1-beta.20 once on the same data directory with its key files in place. It converts the storage, and this release then starts |
storage.legacy_keys_removed |
INFO | At startup, key files kept by a release before 0.0.1-beta.21 were deleted. Nothing reads them | None |
Readings received by the fleet server
| Event | Level | Fires when | Action |
|---|---|---|---|
ingest.bucket_stored |
DEBUG | A 5-minute summary was stored | None |
ingest.bucket_replayed |
DEBUG | A summary arrived that the fleet server already had | None. A collector sends the current summary again each time it connects |
ingest.bucket_rejected_stale |
WARN | A summary arrived for a period the fleet server already holds, with less in it than the stored one | If one collector repeats this more than once a minute, check the host’s clock and local store |
terminal.report_stored |
INFO | A terminal’s own answer from its collector was stored: the latest status, self-test and full snapshot, and a snapshot and descriptor per firmware, for the Diagnostics view’s hardware table, its Raw form and its download | None |
terminal.report_oversize |
WARN | A terminal’s answer was larger than the 4 MB cap and was dropped | Read the collector’s log. An answer this large means the terminal returned far more than it does today |
ingest.batch_oversized |
WARN | A collector sent more in one batch than the fleet server accepts | Investigate the collector’s host |
ingest.schema_version_skew |
WARN | A collector’s readings use a different schema version from the fleet server’s. They are still read | Update the collector, or the fleet server, so that the versions match |
ingest.samples_dropped |
WARN | Individual readings were dropped because their time was in the future, or more than 10 years in the past | Fix the host’s clock |
ingest.bucket_malformed |
WARN | Summaries were dropped because they were not whole 5-minute periods | Investigate the collector’s host |
ingest.bucket_ancient |
WARN | Whole summaries were dropped because their period was more than 10 years in the past | Fix the host’s clock |
ingest.dwell_over_span |
WARN | A summary recorded more time in one state than the period it covered. The excess was dropped | Investigate the collector’s host |
Collector
| Event | Level | Fires when | Action |
|---|---|---|---|
collector.beat_shipped |
DEBUG | A summary was sent | None |
collector.beat_skipped |
WARN | A whole summary was skipped because one part of it, the terminal, the connection or the location, could not be read | Investigate the part named in the line. The collector tries again at the next interval |
collector.beat_send_failed |
INFO | The connection closed while a summary was being sent. It is sent again after reconnecting | None |
collector.heartbeat_send_failed |
INFO | The connection closed while a heartbeat was being sent | None |
collector.domain_poll_failed |
WARN | One part of the reading, the terminal, the connection or the location, could not be read this interval | If sustained, check that part, for example whether the terminal is reachable |
collector.domain_recovered |
INFO | A part that was failing was read again | None |
collector.event_pushed |
INFO | A state change was sent ahead of the next summary | None |
collector.decommissioned |
INFO | The fleet server completed the collector’s decommission | None |
collector.parked |
WARN | The fleet server refused the collector’s certificate 3 times over at least 10 minutes. The collector keeps reading the terminal and storing its readings, and connects once an hour | Enrol the collector again, or remove it. A host clock that is years wrong, or a certificate rotation the fleet server has not finished, reads the same and clears at a later hourly connection |
collector.parked_still |
INFO | Once a day while the collector is refused. since is when the refusals began |
As collector.parked |
collector.parked_cleared |
INFO | The fleet server accepted the collector’s certificate again | None. The collector reconnects as normal |
collector.relocated |
INFO | On the managed service, after a refusal from its fleet server, the platform directory named another fleet server for the collector’s organisation and the collector connected to it | None. The organisation was moved |
collector.directory_unreachable |
INFO | On the managed service, the platform directory did not answer. The collector connects to the last fleet server it knew | None, unless it persists. The collector keeps working with its last answer |
stream.resume_rewound |
INFO | The fleet server holds fewer of this collector’s readings than the collector sent, because the fleet server was restored from an older copy. The collector sends the missing readings again from its own store | None. The fleet server is made whole from the collectors |
stream.resume_rewind_capped |
WARN | The fleet server is missing more than a week of this collector’s readings. The collector sends none of them again | The readings missing since the restored copy are a gap in the history. Everything before the gap is already on the fleet server |
storage.legacy_store_set_aside |
WARN | The collector’s database was written before 0.0.1-beta.18 and this release cannot read it. The file is renamed collector.db.pre-plain and a new store starts, since the fleet server holds everything already sent |
Delete the renamed file once the terminal reports again |
face.refused |
WARN | The collector’s setup page refused a request that did not come from the collector’s own host, or a change posted from another site | Repeated refusals mean a page on the host is trying the setup page. Nothing was changed |
collector.local_status_started |
INFO | The onboard status page started serving on the site network | None. The line records its address |
collector.local_status_bind_failed |
ERROR | The onboard status page setting was invalid, or its port could not be opened. The page is off | Fix NF_LOCAL_STATUS, or free the port, and restart the collector |
collector.local_status_denied |
WARN | A request to the onboard status page from an address outside the site network was refused | If repeated, something outside the site network can reach the collector. Check the routing and the firewall |
collector.local_status_stopped |
INFO | The onboard status page stopped, because it was turned off or its setting changed | None |
collector.local_status_remote |
INFO | The onboard status page setting arrived from the dashboard | None. applied=false means NF_LOCAL_STATUS on the collector overrides the dashboard |
location.override_loaded |
INFO | At startup, a location source set from the dashboard on an earlier run was read from the data directory | None |
location.reader_started |
INFO | An external GPS source started: serial, NMEA over TCP or UDP, or gpsd | None |
location.serial_detected |
INFO | Serial auto-detect found a port with valid GPS sentences | None |
location.serial_open_failed |
WARN | A serial port could not be opened while looking for a GPS receiver. Logged once for each port | For a permission error, add the service user to the group that owns serial ports, dialout or uucp |
location.override_applied |
INFO | A location source set from the dashboard arrived and was applied | None |
location.locked_ignored |
INFO | A location source arrived from the dashboard and was not applied, because NF_LOCATION_LOCK is set |
Remove the lock if the dashboard is to set the location |
location.dish_backoff |
WARN | The terminal has returned no position for several readings. The collector keeps the last known position | If sustained, the terminal’s Starlink subscription may not include GPS. Set a fixed position or an external source |
location.dish_recovered |
INFO | The terminal returned a position again | None |
Connections, enrolment, sign-in and certificates
| Event | Level | Fires when | Action |
|---|---|---|---|
stream.connected |
DEBUG | A collector connected | None. Not logged at the default level |
stream.closed |
DEBUG | A collector’s connection ended | None |
enrol.accepted |
INFO | A collector enrolled, or a collector enrolled earlier connected again | None |
enrol.ip_cap_override |
INFO | At startup, NF_ENROL_MAX_PER_IP set the enrolment limit |
None |
enrol.ip_cap_invalid |
WARN | NF_ENROL_MAX_PER_IP is not a positive whole number. The default applies |
Fix or unset the variable |
enrol.rejected |
WARN | An enrolment was refused: a bad, expired or used token, a suspended organisation, the rate limit, or the licence. The reason field says which |
Check the source address if one address repeats this. A licence_ reason means the licence requires a key, a renewal or a higher collector limit |
enrol.reattach_refused |
WARN | A collector with a known id enrolled against a record that is revoked or decommissioned, with a token other than that record’s Re-admit token. It was refused and nothing changed | A revoked collector is being enrolled again. Check who holds its id file and token |
stream.dropped |
INFO | A collector’s connection ended from the collector’s side. The reason field is closed_without_close_notify for a host or network that went away, connection_reset, timed_out or other |
None. The collector connects again. Many together is a site or a network event, not the fleet server |
stream.duplicate_identity |
WARN | A second connection authenticated as a collector already connected, and replaced the first. previous_peer and peer are the two addresses |
Once from a new address is a reconnection after a network change. Repeated within the hour from two addresses is one certificate in two places. Revoke it from the terminal’s page and enrol the real collector again |
stream.handshake_failed |
WARN | A connection to the collectors’ port failed its TLS handshake. The reason field is certificate for a certificate the fleet server does not trust, client_alert when the client ended the handshake, or protocol for a client that is not a collector |
Find the host at peer. A collector from another fleet, or with an old certificate, enrols again. Anything else is traffic from the internet |
stream.handshake_timeout |
WARN | A connection to the collectors’ port opened and did not complete its TLS handshake | Find the host at peer. It is a scanner or a stalled connection |
stream.ack_unmatched |
WARN | A collector answered a command that the fleet server has no record of on that connection | Once after a fleet server restart is the collector answering a command sent before the restart. Repeated, the collector and the fleet server are on different versions |
stream.org_moved |
INFO | A collector of an organisation marked as moved connected, and was told to ask the platform directory for its new fleet server | None. The collector follows. A collector still connecting here after a day means the directory was not updated |
org.moved |
INFO | nf-server mark-moved marked an organisation as moved to another fleet server. It is suspended here and its data is kept for the copy |
Copy the organisation to the new fleet server and update the directory. Then run nf-server cancel-org here once the move is proven |
custody.sealed_at_birth |
INFO | A new organisation’s root key was encrypted for the provider’s custody and removed from the fleet server | None |
custody.born |
INFO | A fleet server was set up under the platform root before its first start. The root certificate is in place, the first CA certificate is signed, and the root key was never written to its disk | None. Start the fleet server |
custody.deny_list_imported |
INFO | A list of revoked CA certificates, signed on another fleet server under the platform root, was checked and stored. Collectors receive it with their next CA bundle | None |
custody.window_delivered |
INFO | A sealed stock window was delivered to the queue by root-key deliver-window |
None |
provision.rejected |
WARN | A request to the provisioning API was refused, for a bad credential or the rate limit | If repeated, check the provisioning secret, or investigate the source address |
org.destroyed |
INFO | An organisation was erased outright, with its database, keys, certificate authority and users. Only on a test fleet server started with the destroy unlock, which a production fleet server never is | None |
provision.org_created |
INFO | An organisation was created through the provisioning API | None |
board.settling |
INFO | The fleet server started and is allowing collectors time to reconnect. A terminal that has not reconnected is in the planned outage state until the time ends | None. Expected on every restart. Not logged with NF_SETTLE_SECS=0 |
board.settling_cleared |
INFO | The reconnection time ended | None |
stream.handshake_shed |
WARN | The fleet server refused a collector’s connection because its limit of connections in one second was reached. The collector retries | Expected after a restart. If sustained at other times, investigate the source, or raise NF_STREAM_HANDSHAKE_MAX on a larger host |
collector.stopping |
INFO | The collector was told to stop and is telling the fleet server before it exits | None. The terminal’s state is stopped, not an outage |
collector.goodbye_sent |
INFO | The stop notice was sent | None |
collector.stopped_announced |
INFO | The fleet server received a collector’s stop notice | None. An outage alert is held while the stop lasts |
licence.refreshed |
INFO | The daily licence check returned a newer licence and applied it | None |
licence.census_sent |
DEBUG | The hourly census for an organisation, its enrolled and active collector counts, was recorded by Nexus Telemetry | None |
licence.census_unknown |
DEBUG | Nexus Telemetry answered the census with a status other than success. The status field says which |
None. The next hour’s census is sent as usual |
licence.census_unreachable |
DEBUG | The census could not reach Nexus Telemetry | None. The next hour’s census is sent as usual |
auth.login_succeeded |
INFO | A user signed in | None |
auth.login_failed |
WARN | A sign-in failed | Check the source address if one address repeats this. The rate limit applies |
auth.login_ambiguous |
WARN | A username matched accounts in more than one organisation, and the user was asked for their organisation’s short name | None. Frequent for one username means the same name is in use in several organisations |
auth.rate_limited |
WARN | The sign-in rate limit refused an attempt | Check the source address |
auth.password_changed |
INFO | A user set their own password. Their other sessions ended | None |
auth.password_reset |
INFO | A user set a new password through Trouble signing in? with the code sent to their verified email address. Every session of the account ended, and the user was told by email | None, unless the user did not do it. An Owner then resets the password and checks the address |
auth.password_reset_budget |
WARN | An account has received its 10 reset codes for the day, and a further request sent none. The page read as it always does | Someone other than the user is asking in their name, or the user’s email is not arriving. An Owner’s reset is the way in until the next day |
auth.password_reset_refused |
WARN | A code entered on Trouble signing in? matched no valid code for the address, because it was wrong, expired, or there is no such account | Repeated from one address is guessing. 5 wrong entries end a code, and the rate limit applies |
auth.two_factor_challenged |
INFO | A password was accepted for an account with two-factor authentication, and the code is awaited | None |
auth.two_factor_passed |
INFO | The two-factor code, or a recovery code, was accepted | None |
auth.two_factor_failed |
WARN | A two-factor code was refused | None. 5 failures end the sign-in |
auth.two_factor_abandoned |
WARN | A sign-in ended without the two-factor code | None |
auth.two_factor_enrolled |
INFO | A user set up an authenticator app. Their other sessions ended | None |
auth.two_factor_cleared |
INFO | Two-factor authentication was removed from an account, by the user, an administrator or the command line | None |
auth.email_code_sent |
INFO | A verification code was emailed to a user | None |
auth.email_code_echoed |
INFO | A verification code was written to the log instead of sent, because NF_AUTH_EMAIL_ECHO is set or there is no email key |
Set the email key, or unset NF_AUTH_EMAIL_ECHO. A production fleet server must not log codes |
auth.email_send_failed |
WARN | A verification code could not be sent | Check NF_SENDGRID_API_KEY and the sender address |
auth.email_verified |
INFO | A user verified their email address | None |
auth.sign_in_policy_set |
INFO | An organisation’s sign-in policy was saved | None |
demo.refused |
INFO | A visitor to the demo organisation tried a change the demo does not allow | None |
oidc.client_loaded |
INFO | At startup, the Microsoft sign-in registration was read | None |
oidc.client_invalid |
ERROR | The Microsoft sign-in registration is set but its key or certificate could not be read. Microsoft sign-in is off | Fix the files: a PKCS#8 key and a PEM certificate |
oidc.started |
INFO | A user was sent to Microsoft to sign in | None |
oidc.refused |
WARN | A Microsoft sign-in did not complete. The reason field names the check that failed |
If repeated, check the tenant id and the registration’s redirect address |
oidc.no_account |
WARN | A Microsoft identity was verified and matched no account, because no account has a verified email address equal to it | Add the address to the user’s account and have them verify it |
oidc.ambiguous |
WARN | A Microsoft identity matched accounts in more than one organisation, and the user was asked to sign in from their organisation’s own address | None |
oidc.tested |
INFO | An administrator’s test sign-in through Microsoft succeeded, and the registration can be turned on | None |
oidc.connected |
INFO | A Microsoft identity was connected to an account at its first sign-in | None |
oidc.disconnected |
INFO | A user removed the Microsoft identity from their account | None |
oidc.connected_elsewhere |
WARN | A Microsoft identity matched an account by email address, but the account is connected to a different Microsoft identity. It was refused and nothing changed | A guest with a member’s address, or a member with a new identity. The user disconnects Microsoft sign-in on their account page, then signs in with the new identity |
pki.renewal_flagged |
INFO | A device certificate is near expiry and its renewal is queued | None |
pki.renewal_completed |
INFO | A device certificate was renewed | None |
pki.renewal_proved_by_reconnect |
INFO | A collector reconnected on its renewed certificate before its acknowledgement arrived. The renewal is completed on that connection | None |
pki.stream_cert_renewed |
INFO | The fleet server’s own certificate was renewed | None |
pki.intermediate_promoted |
INFO | The next CA certificate took over from the expiring one | None |
pki.promotion_blocked |
WARN | The current CA certificate is near expiry and there is no next one ready | Restore the root key and extend the signing cover. See Best practice |
pki.runway_low |
WARN | The signing cover kept on the fleet server has dropped below its target. The check runs daily, and the target leaves about a year before the cover runs out | Extend the signing cover |
pki.renewal_window_clipped |
WARN | cert_renewal_window_secs was set to half of cert_lifetime_secs or more, and was reduced to half |
Set the renewal window well below the lifetime. The defaults are a lifetime of 365 days and a window of 30 days |
pki.transport_chain_reloaded |
INFO | The CA certificate on disk changed under a ceremony run from the command line. The running fleet server adopted it and re-issued its own certificate under it | None |
pki.transport_chain_reload_refused |
WARN | The CA certificate on disk changed, but its key is missing or the pair does not validate under the root. The fleet server keeps the chain it has | Read the ceremony’s output. The files under ca/transport/ are inconsistent |
pki.awaiting_root |
ERROR | A fleet server with NF_TRANSPORT_ROOT=platform started with no root certificate on disk and did not create one. It stops |
Run nf-server root-key birth --file <root-payload.json> with the platform root, then start the fleet server again |
update.manifest_checked |
INFO | The collector release manifest was read and names a newer version | None |
update.requested |
INFO | A user started an update from the dashboard, for one collector or for all that are behind | None |
update.command_sent |
INFO | The update command was sent to a collector | None |
update.acked |
INFO | The collector acknowledged the update. The detail field says how it will apply. unsupported-platform means it cannot |
For unsupported-platform, update the collector by hand. See Release channels |
update.failed |
WARN | A collector that had accepted an update could not apply it, and reported why. The running version is unchanged | Read error. A version mismatch means the update was started while a release was being promoted, so start it again. A download or signature failure is on the download host or the package |
update.arrived |
INFO | A collector reconnected on the version it was asked to update to | None. The terminal’s update is complete |
collector.update_command |
INFO | The collector received the update command. The outcome field says what it did |
None |
collector.update_applying |
INFO | On macOS, the collector passed the verified package to the installer | None |
collector.update_failed |
WARN | An update failed. The running version is unchanged | Read the error. The update can be started again from the dashboard |
stream.connected and stream.closed are DEBUG because they fire on every
connection and disconnection. At the default level a collector’s connection
and disconnection are recorded in the terminal’s event history instead. To see them in the log, set NF_LOG=debug, or
NF_LOG=nf_server::stream=debug for those two events only.
Alerts, states and maintenance
| Event | Level | Fires when | Action |
|---|---|---|---|
alert.state_changed |
INFO | A terminal’s state change queued an alert | None |
alert.suppressed |
INFO | A maintenance window, a silence or an acknowledgement stopped an alert’s email. The webhook was still sent, with the reason | None |
alert.acknowledged |
INFO | A user acknowledged an alert | None |
alert.silenced |
INFO | A user silenced a terminal’s alerts | None |
alert.delivered |
INFO | An alert was delivered by email or webhook | None |
alert.shaped |
INFO | A noise rule stopped an alert. The rule field says which: the confirmation wait, a recovery for an outage that was never sent, or a terminal changing state repeatedly |
None |
alert.digest_delivered |
INFO | Several alerts due together were sent as one email | None |
alert.capped |
INFO | The organisation reached its hourly limit of alerts. Further alerts were not sent | If frequent, find the terminal that is changing state repeatedly |
alert.reminder |
INFO | A reminder was sent for an alert still open | None |
alert.stable |
INFO | A terminal that was changing state repeatedly has held one state for 30 minutes | None |
alert.delivery_failed |
WARN | An alert could not be delivered. The fleet server retries, or has given up | Check the email key and the webhook address |
verdict.transition |
INFO | A terminal’s state changed, after the change was confirmed | None. Alerts that share a correlation_key are one incident |
maintenance.window_created |
INFO | A user scheduled a maintenance window | None |
maintenance.window_cancelled |
INFO | A user cancelled a maintenance window | None |
fleet.members_updated |
INFO | A user saved a group’s membership. Groups were fleets in an earlier release | None |
weather.fetch |
INFO | A weather forecast was fetched. stored=false means the request failed and the previous forecast was kept |
None |
weather.fetch_failed |
WARN | The weather service refused a request or timed out. Logged with the weather.fetch line for the same request |
Occasional is normal. If persistent, check the fleet server’s internet access and the weather settings |
weather.budget_exhausted |
WARN | The daily limit of weather requests was reached. Forecasts are not refreshed until the next UTC day | Raise NF_WEATHER_DAILY_CAP only with a paid weather key |
board.slow_build |
INFO | Building the dashboard’s view of the fleet took over 250 milliseconds. The fields say which part | Occasional after a restart is normal. If sustained, the host is too small for the fleet. Read db_ms first |
maint.slow_step |
INFO | One step of the fleet server’s maintenance pass took over a second | Occasional after a restart is normal. A step that grows on every pass requires support |
maint.pass_end |
DEBUG | A maintenance pass completed | None. Its absence for many minutes means the pass is stuck. See which maint.slow_step came before |
rollup.truncated |
INFO | The pass that summarises readings reached its 10 second limit and will continue at the next pass | Normal while catching up. If continuous for hours, the host is too small for the fleet |
ping.unavailable |
WARN | The ping command cannot run on the collector’s host, because it is not installed or is not allowed to send ping packets. Every ping target reads as unreachable until fixed | Install the ping tool, or grant it the CAP_NET_RAW capability, or use tcp:// targets in NF_PING_TARGETS |
ping.defaults_icmp |
INFO | At startup, the collector is using its built-in ping targets | On a network that blocks ping, set tcp:// targets in NF_PING_TARGETS |
connection.target_invalid |
WARN | An NF_CONNECTION_TARGETS entry is not host:port and was ignored |
Fix the entry |
connection.targets_override |
INFO | At startup, NF_CONNECTION_TARGETS replaced the built-in targets |
None |
connection.targets_defaulted |
WARN | NF_CONNECTION_TARGETS was set but had no usable entry, and the built-in targets apply |
Fix the list. Every entry must be host:port |
connection.public_ip_unreachable |
WARN | The lookup of the site’s public address failed. The last known result is kept | If the network blocks the built-in lookup services, set NF_CONNECTION_IP_URLS and NF_CONNECTION_ASN_URLS |
connection.uplink_unrecognised |
WARN | A VPN holds the default route, so the collector cannot tell which interface is the internet connection | Set NF_CONNECTION_UPLINK_INTERFACE to the interface |
failover.route_probe_unavailable |
WARN | The collector cannot read the host’s routes, so it cannot detect a change of connection | Install iproute2. Run a Docker collector with --network host |
failover.route_probe_unsupported |
INFO | Reading the host’s routes is not supported on this platform | None. Expected on Windows today |
Live view
| Event | Level | Fires when | Action |
|---|---|---|---|
live.subscribed |
INFO | A user opened a terminal’s Live view | None |
live.unsubscribed |
INFO | A Live view was closed | None |
live.session_ended |
INFO | A Live view was closed because the user’s session ended | None |
live.rejected_over_cap |
WARN | The limit of open Live views was reached, and a new one was refused | If sustained, many users are watching one organisation. A refused view falls back to a refresh every 5 seconds |
live.dropped_lagging |
WARN | A Live view fell behind and caught up from the current state | None |
live.dropped_slow_client |
WARN | A Live view’s browser could not keep up and one update was skipped | None. Readings are not affected |
live.extra_rejected |
WARN | A collector sent a live item, such as a sky map, larger than the fleet server accepts | Investigate the collector’s host |
operator.tool_requested |
INFO | A user pressed Speed test or Reboot | None |
geocode.upstream_failed |
WARN | The place search on the Location setting got no answer from the map service | Check the fleet server’s internet access. Choosing a point on the map still works |
collector.speed_test_started |
INFO | The collector started a speed test | None. The test uses the connection’s full bandwidth for about a minute |
collector.speed_test_finished |
INFO | The speed test ended | None. The result is in the detail field and on the terminal’s page |
collector.reboot_requested |
INFO | The collector received a reboot instruction | The terminal drops and returns within minutes. The collector declines the reboot when the site’s network would go down with the terminal, and the detail field says so |
collector.ping_targets_applied |
INFO | The collector applied the connectivity check targets set on the dashboard | None. persisted=false means the data directory is not writable |
ping.targets_set |
INFO | A user saved the connectivity check targets | None |
collector.history_window |
INFO | The fleet server asked the collector for a stretch of the terminal’s per-second history, and the collector answered. It is asked when a terminal’s page opens, for the hour, 6-hour and day views, and for a span selected on the week or year | None. Once when a page opens, then once a minute while it is open |
collector.terminal_report |
INFO | The collector sent one of the terminal’s own answers as returned: the status and self-test answers when their text changed since the last send, and the full snapshot and descriptor at the collector’s start, on a firmware change and once a day | None |
collector.diagnostics_export_requested |
INFO | A fleet server asked the collector for the terminal’s full snapshot. Nothing in the dashboard sends this today | None |
chart.history_fallback |
INFO | An hour, 6-hour or day chart could not be drawn from the terminal’s own history, because the terminal is offline or did not answer within 10 seconds. It was drawn from the 5-minute summaries, and the page says so | None when the terminal is offline. Repeated for a terminal that is online, check its connection |
collector.snapshot_shipped |
INFO | The collector sent the terminal’s daily sky map | None. If no map arrives for a day, the collector cannot fetch it from the terminal |