Troubleshooting
Linux journal troubleshooting: use systemd logs to trace a failure
The systemd journal is most useful when you approach it with a service name and time window. Dumping the entire journal after an incident often buries the first relevant error under retries, follow-on failures, and unrelated background noise.
Anchor the investigation to time and unit
Record when the symptom began, then query the affected service unit around that window. Include a few minutes before the visible failure so dependency errors, restarts, resource pressure, or configuration reloads are not missed.
Use boot selection when a problem appeared after reboot; logs from the previous boot may contain the shutdown or crash clue.
Read the first failure before the cascade
A database becoming unavailable can generate hundreds of application errors. The earliest dependency failure is often more actionable than the last request that failed.
Note exit status, signal, restart count, and any referenced file, port, user, permission, or environment variable.
Pair journal output with unit configuration
Inspect the effective unit, drop-ins, environment files, dependencies, restart policy, working directory, and service user. A process that works from a shell can fail under systemd because its environment and permissions are intentionally different.
Use systemd-analyze verify or application-specific config tests where available before restarting repeatedly.
Preserve useful logs
Set journal retention appropriate to available disk and incident needs. Forward or back up critical logs when local loss would erase the only evidence after a filesystem or server failure.
- Filter by unit and time.
- Read the earliest dependency error.
- Inspect the effective unit environment.
- Retain enough history for incident review.