Linux log monitoring: the complete guide to files, tools, and severity

The /var/log directory is the source of truth for every event on a Linux system. Whether you are debugging a failed SSH login, a kernel panic, or a database timeout, the answer is almost always sitting in a text file or a binary journal on your disk.

Effective Linux log monitoring is not just about knowing where these files are, but understanding the differences between distributions and the tools required to parse them in real time.

The Linux logging hierarchy

Most Linux logs are stored in /var/log. However, the specific filenames change depending on whether you are running a Debian-based system (like Ubuntu) or an RPM-based system (like RHEL, CentOS, or Fedora).

The following table maps the most critical log files across these two families:

Critical files to watch

• syslog / messages: The catch-all for system activity. If you don’t know where to start, start here.
• auth.log / secure: Tracks every sudo attempt, SSH login, and password failure. This is your primary security audit trail.
• dmesg: Records hardware and driver initialization. Essential for debugging disk failures or network card issues.

Standard log severity levels

Linux logging follows a standardized severity scale defined by the Syslog protocol. Understanding these levels (0 through 7) allows you to filter out noise and focus on what is actually broken.

In most production environments, you should ignore everything above level 4 (Warning) unless you are actively troubleshooting a specific bug.

Essential CLI toolkit

Monitoring logs from the command line is the fastest way to debug a single server. You should move beyond just cat and master the tools designed for live data.

Basic monitoring commands

The most common way to “watch” a log is tail:

# Watch the last 10 lines and follow new updates
sudo tail -f /var/log/syslog

# Watch and filter for a specific keyword
sudo tail -f /var/log/syslog | grep "error"

If you need to search through a large file without loading the whole thing into memory, use less:

sudo less +F /var/log/auth.log

Pressing Ctrl+C in less stops the “follow” mode and lets you search (using /), then pressing Shift+F resumes the live tail.

Advanced filtering

When you need to extract specific data, like the IP addresses of failed logins, awk is your best friend:

sudo grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c

This command finds failed passwords, extracts the 11th column (usually the IP), and counts the occurrences of each.

Mastering journalctl

Modern Linux distributions use systemd, which writes logs to a binary format called the journal. You cannot use cat or tail on these files directly. Instead, you use journalctl.

The advantage of the binary journal is that it is indexed, making searches significantly faster than grepping through text files.

Common journalctl workflows

# See logs for a specific service (unit)
sudo journalctl -u nginx.service

# See logs since a specific time
sudo journalctl --since "1 hour ago"

# Filter by severity (e.g., only Errors and above)
sudo journalctl -p err..emerg

# Follow the journal in real-time
sudo journalctl -f

One of the most useful flags is -xe, which opens the journal at the end of the file and includes explanatory text for common systemd errors.

Managing log growth with logrotate

Logs can grow indefinitely, eventually filling up your disk and crashing your applications. Linux solves this with logrotate. This utility runs as a cron job to compress, rename, and eventually delete old log files.

You can find the configuration in /etc/logrotate.conf and directory-specific configs in /etc/logrotate.d/.

A typical config for Nginx might look like this:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
}

This tells Linux to rotate the logs daily, keep 14 days of history, and compress the old files to save space.

Centralization: Solving the SSH anti-pattern

SSH-ing into every server to run tail -f works when you have one box. When you have five or ten, it becomes an “anti-pattern” that leads to missing critical errors.

Centralized log monitoring involves installing an agent that tails your logs and sends them to a central dashboard. This allows you to:

1. Search across all servers at once.
2. Set up alerts that fire automatically when an “Error” level log appears.
3. Keep logs even if the original server is terminated.

For teams that want this visibility without the overhead of managing an ELK stack or Loki, Simple Observability provides a zero-config approach. A single-command agent automatically tails your system logs and the systemd journal, centralizing them into a searchable UI with built-in alerting.

Conclusion

Linux log monitoring is a fundamental skill for any sysadmin. By understanding the /var/log hierarchy, mastering journalctl, and ensuring logrotate is doing its job, you can maintain a healthy system. When you outgrow the terminal, look toward centralized solutions that let you monitor your entire fleet from a single pane of glass.