Troubleshooting

The common faults in Nexus Telemetry Fleet, listed by what the dashboard or a log shows, with the steps for each. Each section ends with the page that covers the subject. A collector is the software at a site that reports its terminal to the fleet server. See Add a collector.

# A collector that never appears

1. Read the collector’s log. Enrolment failed: token expired or exhausted means the token was already used, or has expired. Enrolment failed: invalid or revoked token means the token was revoked. Enrolment refused by the fleet server's licence means the fleet server has no valid licence key.
2. Open Tokens in the dashboard and read the token’s Status: used or expired confirms the log.
3. Generate a new token with Add telemetry collector and run the new command on the collector’s host.

See Tokens.

# A collector that cannot reach the terminal

The collector runs and the terminal is on the dashboard, but its readings are empty and the collector’s log repeats collector.domain_poll_failed for the terminal.

1. Read collector.dish_addr at the top of the collector’s log. It records the address the collector reads, http://192.168.100.1:9200 by default.
2. From the collector’s host, check that 192.168.100.1 answers. The terminal answers at that address whether or not it is in Bypass mode.
3. Where the host sits behind another router, add a static route on that router for 192.168.100.1/32 through the interface the terminal is connected to. A site network managed by a service provider requires the provider to add it. No port is opened and nothing is exposed to the internet.
4. Where the terminal is at another address, set NF_STARLINK_ADDR. See Collector configuration.

The log records collector.domain_recovered once the terminal is read again.

# Unknown source name

1. Read the log for Unknown NF_SOURCES entry '...' (known: host, starlink, connection, ping).
2. Compare NF_SOURCES with those 4 names. A misspelt name stops the collector at startup.
3. Correct or remove the name and restart the collector. The log then lists the sources it is reading.

See Collector configuration.

# A terminal in Outage

1. Read the cause on the terminal’s card. unreachable means the collector stopped reporting without notice. A collector that reports its own shutdown puts the terminal in No data, not Outage.
2. Check power and network at the site, and whether anyone stopped the collector.
3. Once readings resume, the card shows Stabilising for 5 minutes, then the state returns to Healthy. Nothing is done on the dashboard.

See States.

# A terminal reading stale

The Live view of a terminal’s page shows a freshness word beside the state, which says how recent the last reading is. stale means the collector is connected and nothing has arrived for more than 3 times its own reporting interval. The Stale threshold setting, 1 day by default, is the separate point at which the terminal is marked offline.

1. On the terminal’s Live view, read last seen for when the last reading arrived.
2. Check power and network at the site, and whether anyone stopped the collector.

While the terminal is past the threshold, the Status view reads Outage, and the List view and the terminal’s own page read Offline.

See States.

# A version reading behind

1. On the List view, the Version column marks a collector that is behind its release channel with an amber behind chip.
2. On the terminal’s Admin view, Software offers Update to <version>. Press it, then press Update in the panel.
3. The update can wait. The collector keeps reporting on its current version until it is updated.

See Release channels.

# An update that stays at updating

The List view’s updating chip clears when the collector reconnects on the new version. It also clears when the collector reports that it could not apply the update, or 10 minutes after the update was sent.

1. Read update.failed in the fleet server’s log. Its error field gives the reason. The running version is unchanged.
2. A version mismatch means the release changed after Update was pressed. Press Update again.
3. A download or signature failure is on the download host or the package. Read the collector’s own collector.update_failed line.

See Release channels.

# A licence chip, or an enrolment refused with 402

A fleet server in a private deployment runs on a licence key. As the key nears expiry and after it, the dashboard shows a licence chip in the top bar.

1. Read the chip: Licence: 1 day, Licence expired or Licence: read-only.
2. With the licence read-only, a change on the dashboard is refused with The licence has lapsed and the dashboard is read-only. Apply a renewed licence key on the Licence page to restore changes., and a collector’s enrolment is refused with HTTP status 402.
3. Open Settings, Licence, paste the new key under Licence key, and press Apply key.

See Private deployment.

# A certificate that cannot renew

1. On the Certificates page, find the collector whose Status shows the amber chip offline · cannot self-renew.
2. Compare its Expires date with today.
3. Get the collector back online before that date. The fleet server sends the queued renewal when the collector connects.

See Certificates page.

# A lost token

1. The fleet server does not store a token, only a hash of it, so a token cannot be shown again.
2. Generate another with Add telemetry collector.
3. Revoke the lost token on Tokens, or leave it to expire.

See Add a collector.

# A collector revoked by mistake

1. Check that the collector is still yours and still running. Recovery is done on its host.
2. On the terminal’s Admin view, press Re-admit. A revoked collector returns only with that token. The fleet server refuses every other token for it.
3. On the host, enrol again with --force, then restart the collector’s service.

sudo nf-collector enrol --server <your-address> --token <token> --force
sudo systemctl restart nf-collector

The collector attaches to its existing record, and its terminal returns to the dashboard with its name and history. Without --force the command refuses, because the collector already has an identity. The restart matters because a refused collector is parked and otherwise waits up to an hour to connect.

See Retire and revoke.

# A collector refused and parked

After the fleet server refuses a collector’s certificate 3 times over at least 10 minutes, the collector parks. It keeps reading the terminal and storing its readings, and connects once an hour. Its log reads collector.parked, and nf-collector status exits 3.

1. Enrol the collector again, or remove it. A revoked collector is enrolled again with its Re-admit token, as above.
2. Restart the collector’s service after the new enrolment, so it connects at once rather than at the next hourly try.

A host clock that is years wrong, or a CA rotation the fleet server has not finished, reads the same. It clears at a later hourly try, and the log then reads collector.parked_cleared.

# A terminal at connecting after a reset

nf-collector reset changes files only and does not start or stop the service. Start the service after the new enrolment:

Until the service starts, the Shipping section of nf-collector status reads daemon not running?. See Command line.

# A CA rotation that does not complete

A CA rotation replaces the CA certificate, and each collector moves to the new one when it next connects. The banner counts the collectors that have not moved yet.

1. On the Certificates page, compare the number in the CA rotation in progress banner with the number of collectors that are offline. If they match, the rotation is waiting for those collectors.
2. Wait. Each collector moves to the new CA certificate when it connects, and the number drops.
3. Press Force-complete only for collectors that will never return. It revokes every collector that has not migrated.

See Certificates page.

# Gaps in a terminal’s readings

1. Note where the gap is in the terminal’s charts.
2. A recent gap on a terminal whose collector was offline is filled in when the collector reconnects. It sends every reading it stored while it was cut off, in blocks, until the gap is filled.
3. A gap while the collector was stopped, or its host asleep, is filled from the terminal’s own record of its last 15 minutes when the collector starts again. Anything older than those 15 minutes remains a gap.
4. Raw readings older than the Raw readings retention period, 7 days by default, are deleted. The 5-minute, hourly and daily summaries have their own longer periods and remain.

See Timings.

# A dashboard unreachable from another host

The fleet server answers only on its own host until NF_LISTEN is set to another address. Before opening it:

1. Create the first user, so that the dashboard requires a sign-in.
2. Put TLS or a reverse proxy in front of the dashboard.
3. Set NF_LISTEN and restart the fleet server.

See Private deployment.