Configuring Logging

Understanding System Logging

In the intricate ecosystem of a Linux server, logging serves as the silent sentinel, capturing the events and activities that transpire within its digital confines. These logs are indispensable for administrators, providing insights into system behavior, troubleshooting clues, and security auditing trails. To comprehend the landscape of system logging, we delve into the mechanisms and methodologies employed by various logging solutions.

Logging Mechanisms

1. Systemd-journald: As the progeny of the Systemd revolution, systemd-journald stands as the vanguard of modern logging. Seamlessly integrated with Systemd, it captures a comprehensive tapestry of system events, from kernel messages to service activities. Its command-line interface, epitomized by journalctl, offers a window into the rich tapestry of log data, empowering administrators with unparalleled insights into system dynamics.
2. Direct Write: Despite the ascendancy of systemd-journald, some venerable services opt for a more traditional approach, inscribing their tales directly onto log files. Apache web servers and Samba file servers, among others, adhere to this time-honored practice. However, this method, while functional, lacks the finesse and extensibility of modern logging solutions.
3. rsyslogd: In the pantheon of logging daemons, rsyslogd emerges as a stalwart, embodying the evolution of syslogd. Tasked with managing centralized log files, rsyslogd perpetuates the legacy of syslogd while embracing modernity. It complements systemd-journald, bridging the gap between tradition and innovation, and furnishes administrators with an arsenal of features for log aggregation, filtering, and forwarding.

The Role of systemd-journald and rsyslogd

In the realm of Red Hat Enterprise Linux (RHEL) 9, systemd-journald reigns supreme as the primary custodian of system logs. Its meticulous journaling of system events, coupled with real-time access via journalctl, delivers unparalleled visibility into system activities. Meanwhile, rsyslogd assumes the mantle of augmenting systemd-journald’s capabilities, providing persistence, centralization, and advanced features such as modular logging and remote logging.

Navigating the Logging Landscape

1. journalctl Command: To unearth the intricacies of system events, administrators wield the journalctl command, excavating the systemd journal for granular insights. Armed with filtering capabilities and temporal queries, they traverse the annals of system history, unraveling the mysteries encoded within.
2. systemctl status Command: A cursory glance into service status and recent log entries is facilitated by the systemctl status command. Offering a succinct summary of systemd unit activities, it furnishes administrators with a bird’s-eye view of service health and operational integrity.
3. Monitoring Log Files: While systemd-journald captures the ephemeral essence of system events, rsyslogd immortalizes them in the hallowed halls of /var/log. By vigilantly monitoring these log files, administrators glean insights into system health, performance, and security incidents.

Interpreting Log Data

In the annals of /var/log, a myriad of log files beckon administrators, each bearing witness to a facet of system activity. From the ubiquitous /var/log/messages to the esoteric /var/log/audit/audit.log, these artifacts harbor insights into authentication attempts, kernel messages, and system startups. Armed with the lexicon of log file interpretation, administrators navigate this labyrinth of log data, deciphering timestamps, origin hosts, service names, and message content.

Live Log File Monitoring

When configuring services on Linux, real-time insight into system activities can be invaluable. This is where tools like tail -f <logfile> come into play, providing a live stream of log entries as they’re appended to a file. In practice, you might have one terminal session dedicated to configuring and testing a service while simultaneously monitoring its log file in another session. The tail -f command allows you to keep a continuous eye on the evolving log data, with the trace persisting until you manually halt it by pressing Ctrl-C.

Using logger

In the realm of logging, the logger command emerges as a versatile tool, enabling users to inject messages directly into the system log from the command line or scripts. Its simplicity belies its power—just type logger followed by the desired message, and voila! The message seamlessly integrates into the log stream, offering administrators an expedient means of conveying information or troubleshooting insights. Moreover, logger grants users the ability to specify the priority and facility of log messages, facilitating targeted logging and testing scenarios.

Working with systemd-journald

At the heart of modern logging lies systemd-journald, the custodian of system log messages in a binary journal format. Interacting with this journal is facilitated by the journalctl command, which offers a plethora of options for querying and filtering log data. From viewing recent events to tailing live log entries, journalctl empowers administrators with unprecedented control over log analysis and system monitoring.

Exploring journalctl Options

Delving deeper into journalctl, administrators uncover a wealth of options for fine-tuning log queries and extracting actionable insights. From viewing boot logs to filtering messages by priority or systemd unit, journalctl offers a cornucopia of capabilities for navigating the labyrinth of system events. Armed with -f for live monitoring, -b for boot log retrieval, and -p for priority-based filtering, administrators traverse the annals of system history with precision and finesse.

In the ever-evolving landscape of system logging, tools like tail -f, logger, and journalctl serve as indispensable allies, providing administrators with the visibility and agility needed to navigate the complexities of modern computing environments.

Preserving the Systemd Journal

The systemd journal, by default, resides in the volatile /run/log/journal directory, which means it’s ephemeral and cleared upon system reboot. However, to ensure persistence across restarts, administrators can establish a permanent home for the journal by creating the directory /var/log/journal. Setting the Storage=auto parameter in /etc/systemd/journald.conf ensures that the journal is written to disk if /var/log/journal exists. This parameter offers different options: volatile, persistent, and none, each dictating where the journal is stored and how it behaves.

Even with the journal stored in /var/log/journal, it’s important to note that it undergoes built-in log rotation, typically monthly. Additionally, it’s constrained by a maximum size of 10 percent of the file system it resides on, ceasing growth if less than 15 percent of the file system remains free. When this occurs, older messages are automatically purged to accommodate new ones. Modifying /etc/systemd/journald.conf allows administrators to adjust these settings as needed.

Configuring rsyslogd

To ensure that log information is directed to the desired locations, administrators can configure the rsyslogd service through /etc/rsyslog.conf and optional drop-in files in /etc/rsyslog.d. The configuration involves understanding different sections within rsyslog.conf, including modules, global directives, and rules.

The rules section is pivotal, defining what information is logged and where it’s logged. This is achieved through facilities, priorities, and destinations. Facilities categorize logged information, while priorities indicate the severity of messages. Destinations specify where messages are written, typically files or device files.

For instance, the rsyslog.conf file may contain rules like:

.info;mail.none;authpriv.none;cron.none /var/log/messages

authpriv. /var/log/secure
mail.* -/var/log/maillog

In this example, different facilities and priorities dictate where various types of messages are logged.

While the syslog facilities and priorities are fixed, the daemon facility serves as a catch-all for unspecified daemons, and the local0 through local7 facilities offer flexibility for custom logging. Adjusting logging configurations often involves specifying priorities alongside facilities to route messages to specific files. For instance:

cron.=debug -/var/log/cron.debug

This line buffers writes efficiently for all cron messages with debug priority to the file /var/log/cron.debug.

Exercise- 1: Changing rsyslog.conf Rules

This exercise demonstrates modifying rsyslog.conf to direct Apache service error messages to syslog and create a rule for logging debug messages to a specific file. Steps include installing Apache, configuring it to log to syslog, updating rsyslog.conf, and restarting services to apply changes. Subsequently, debug messages are logged to /var/log/messages-debug.

Through exercises like these, administrators gain hands-on experience configuring logging behavior, essential for managing system events effectively.

Rotating Log Files

To manage syslog messages effectively and prevent them from consuming excessive disk space, log rotation is employed. Log rotation involves closing an old log file and opening a new one when a certain threshold is reached. This process is typically handled by the logrotate utility, which runs periodically.

When log files are rotated, the old log file is often copied to a file named with the rotation date, such as /var/log/messages-20230608 if rotated on June 8, 2023. By default, logrotate keeps four old log files on the system, automatically removing files older than that period.

However, it’s essential to note that rotated log files are not stored indefinitely; they are deleted. Therefore, if retaining log data beyond the default rotation period is necessary, organizations should consider options like backing up log files or configuring a centralized log server where rotated messages are kept for a more extended period.

The default settings for log rotation are specified in /etc/logrotate.conf.

# Log Rotation Settings
# Rotate log files on a weekly basis
weekly

# Retain 4 weeks worth of old log files
rotate 4

# Create new (empty) log files after rotating old ones
create

# Append date as a suffix to rotated file names
dateext

# Uncomment the following line to enable compression of log files
compress

# Include additional log rotation configurations from /etc/logrotate.d directory
include /etc/logrotate.d

This configuration instructs logrotate to rotate files on a weekly basis, retain four old versions of the file, create new log files after rotating old ones, and use the date as a suffix for rotated files.

If specific files require custom settings, administrators can create configuration files for those files in /etc/logrotate.d. Settings in these specific configuration files override the default settings in /etc/logrotate.conf. The /etc/logrotate.d directory often contains various configuration files for managing specific log files on the system.

For more details about logrotate and its parameters, administrators can refer to the man logrotate command, which provides comprehensive documentation. Through proper log rotation configurations, administrators can efficiently manage log files, ensuring system stability and performance.

Exercise- 2: Using Live Log Monitoring and logger

1. Open a root shell.
2. From the root shell, type tail -f /var/log/messages.
3. Open a second terminal window. In this terminal window, type su - student to open a subshell as user student.
4. Type su - to open a root shell, but enter the wrong password.
5. Look at the file /var/log/messages. You see an error message was logged here.
6. From the student shell, type logger hello. You’ll see the message appearing in the /var/log/messages file in real time.
7. In the tail -f terminal, press Ctrl-C to stop tracing the messages file.
8. Type tail -20 /var/log/secure. This shows the last 20 lines in /var/log/secure, which also shows the messages that the su - password errors have generated previously.

Exercise- 3: Discovering journalctl

1. Type journalctl. You’ll see the content of the journal since your server last started, starting at the beginning of the journal. The content is shown in less, so you can use common less commands to walk through the file.
2. Type q to quit the pager. Now type journalctl --no-pager. This shows the contents of the journal without using a pager.
3. Type journalctl -f. This opens the live view mode of journalctl, which allows you to see new messages scrolling by in real time. Press Ctrl-C to interrupt.
4. Type journalctl, press the Spacebar, and then press the Tab key twice. When prompted to view all possibilities, type y and then press the Enter key. This shows specific options that can be used for filtering. Type, for instance, journalctl _UID=1000 to show messages that have been logged for your student user account.
5. Type journalctl -n 20. The -n 20 option displays the last 20 lines of the journal (just like tail -n 20).
6. Type journalctl -p err. This command shows errors only.
7. If you want to view journal messages that have been written in a specific time period, you can use the --since and --until commands. Both options take the time parameter in the format YYYY-MM-DD hh:mm:ss. Also, you can use yesterday, today, and tomorrow as parameters. So, type journalctl --since yesterday to show all messages that have been written since yesterday.
8. journalctl allows you to combine different options, as well. So, if you want to show all messages with a priority error that have been written since yesterday, use journalctl --since yesterday -p err.
9. If you need as much detail as possible, use journalctl -o verbose. This shows different options that are used when writing to the journal. All these options can be used to tell the journalctl command which specific information you are looking for. Type, for instance, journalctl _SYSTEMD_UNIT=sshd.service to show more information about the sshd Systemd unit.
10. Type journalctl --dmesg. This shows kernel-related messages only. Not many people use this command, as the dmesg command gives the exact same result.

Exercise- 4: Making the systemd Journal Persistent

1. Open a root shell and type mkdir /var/log/journal.
2. Before journald can write the journal to this directory, you have to set ownership. Type chown root:systemd-journal /var/log/journal, followed by chmod 2755 /var/log/journal.
3. Use systemctl restart systemd-journal-flush to reload the new systemd-journald parameters.
4. The Systemd journal is now persistent across reboots.

Exercise- 5: Changing rsyslog.conf Rules

1. By default, the Apache service does not log through rsyslog but keeps its own logging. You are going to change that. To start, type dnf install -y httpd to ensure that the Apache service is installed.
2. After installing the Apache service, open its configuration file /etc/httpd/conf/httpd.conf and verify it has the following line:

ErrorLog syslog:local1

3. Type systemctl restart httpd.

4. Create a line in the /etc/rsyslog.conf file that will send all messages that it receives for facility local1 (which is now used by the httpd service) to the file /var/log/httpd-error.log. To do this, include the following line in the #### RULES #### section of the file:

local1.error /var/log/httpd-error.log

5. Tell rsyslogd to reload its configuration, by using systemctl restart rsyslog.
6. All Apache error messages will now be written to the httpd-error.log file.
7. From the Firefox browser, go to http://localhost/index.html. Because no index.html page exists yet, this will be written to the error log.
8. Create a snap-in file that logs debug messages to a specific file as well. To do this, type echo "*.debug /var/log/messages-debug" /etc/rsyslog.d/debug.conf.
9. Again, restart rsyslogd using systemctl restart rsyslog.
10. Use the command tail -f /var/log/messages-debug to open a trace on the newly created file.
11. From a second terminal, type logger -p daemon.debug "Daemon Debug Message". You’ll see the debug message passing by.
12. Press Ctrl-C to close the debug log file.

Revision: Configuring Logging

Here’s a list of commands related to logging on Red Hat-based systems, along with their descriptions:

1. Install rsyslog:
   • Description: Installs the rsyslog logging service.
   • Example: sudo yum install rsyslog
2. Start rsyslog:
   • Description: Starts the rsyslog service.
   • Example: sudo systemctl start rsyslog
3. Enable rsyslog to start on boot:
   • Description: Configures the rsyslog service to start automatically when the system boots up.
   • Example: sudo systemctl enable rsyslog
4. Check rsyslog status:
   • Description: Displays the status of the rsyslog service, indicating whether it’s running or not.
   • Example: sudo systemctl status rsyslog
5. Edit rsyslog configuration:
   • Description: Opens the rsyslog configuration file in a text editor for editing.
   • Example: sudo vi /etc/rsyslog.conf
6. Restart rsyslog after changes:
   • Description: Reloads the rsyslog service to apply any configuration changes made.
   • Example: sudo systemctl restart rsyslog
7. Send a test log message using logger:
   • Description: Manually sends a test message to the system logger.
   • Example: logger "This is a test message"
8. View logs using journalctl:
   • Description: Displays logs from the systemd journal, including kernel and system logs.
   • Example: sudo journalctl -xe
9. View specific log files:
   • Description: Displays the contents of a specific log file.
   • Example: tail -f /var/log/messages
10. View security-related log entries:
    • Description: Displays security-related log entries from /var/log/secure.
    • Example: tail -f /var/log/secure
11. View mail server logs:
    • Description: Displays logs related to the mail server from /var/log/maillog.
    • Example: tail -f /var/log/maillog
12. View boot process logs:
    • Description: Displays logs related to the system boot process from /var/log/boot.log.
    • Example: tail -f /var/log/boot.log
13. View cron job logs:
    • Description: Displays logs related to cron jobs from /var/log/cron.
    • Example: tail -f /var/log/cron
14. Edit logrotate configuration:
    • Description: Opens the logrotate main configuration file in a text editor for editing.
    • Example: sudo vi /etc/logrotate.conf
15. Test logrotate configuration:
    • Description: Simulates the log rotation process based on the logrotate configuration file.
    • Example: sudo logrotate -d /etc/logrotate.conf

These commands and descriptions should help you understand and utilize logging effectively on Red Hat-based systems.