How to use journalctl to read linux system logs

How to use journalctl to read linux system logs

For many years system and kernel logs were handled by a utility called syslogd. Most Linux-based operating systems have migrated to systemd, which comes with a different log daemon, journald. To interact with these logs, you use the journalctl utility.

Give User Permission to Read System Logs

Only users belonging to the “adm” or “systemd-journal” groups can read systemd logs. Distributions such as Ubuntu already add you as a user to the adm group.

Open a terminal emulator and type the following command:

How to use journalctl to read linux system logs

If you see “adm” or “systemd-journal” in the output, you can skip the rest of the steps in this section. Otherwise, add yourself to the “adm” group.

You would have to restart your login session for this change to take effect (log out and log in). If you can’t do that for various reasons, use this command to log in to the new group without restarting the graphical session.

Don’t close the terminal window. You are now part of the adm group – in the terminal session, but not in your graphical session. If you open a new terminal at this time, your user won’t be logged in to the adm group anymore.

Check If Journal Is Persistent

The systemd logs can be persistent or volatile. On Ubuntu and other distros, they are persistent by default. On Debian 9 they are volatile, meaning they’re kept only in memory (not disk) and disappear at shutdown or reboot. Enter the following command.

How to use journalctl to read linux system logs

If there are multiple entries here, there’s nothing more you have to do. It means journals are kept on disk (persistent). If you only get one entry, then the journal is volatile. Change it to persistent.

Select Which Boot Entry Journal to View

Usually, you will want to see the log for the current boot. On rare occasions you will want to see the previous boot, when something went wrong, for example, after a system crash.

To view the log for the current boot:

How to use journalctl to read linux system logs

For the previous boot, use “-1” instead of “0,” and for two boots ago, “-2” and so on.

Navigate and Search Through the System Journal

After you open the log with journalctl, you can navigate through the text with arrow keys and PAGE UP or PAGE DOWN keys. Other useful keys are:

  • > to go to the end of the output.
  • < to go to the beginning of the output.
  • / to search for a string of text. After you press the slash key, enter the string you want to find, followed by Enter . The string is case sensitive, so “network” won’t find “Network” strings. The search begins from your current view position, downwards. To search upwards, use ? .
  • n find the next match in a current search. N finds the previous.
  • q quits the journalctl utility.

How to use journalctl to read linux system logs

Filter Log Entries by Priority

Sometimes you only want to search for errors, ignoring notices and status messages. Each entry in a log has a priority: emergency, alert, critical, error, warning, notice, info. These are listed in order of importance, emergency being reserved for worst case scenarios (system unusable). Info messages are just informational text, reporting status of programs that work normally.

To only display error messages from the current boot, enter:

How to use journalctl to read linux system logs

If you want to see errors from all boots, just remove the “-b” parameter:

These are the codes you can pass to the “-p” parameter:

  • alert
  • crit
  • debug
  • emerg
  • err
  • info
  • notice
  • warning

Filter Log Entries by Path to Process Executable File or Systemd Unit

Some processes are started and managed by so-called systemd units. To see all logs related to the cron service unit, enter:

You can see what units you have available with:

How to use journalctl to read linux system logs

You can navigate the list with the up and down arrow keys. Press q to quit.

If you would rather use the path to the program’s executable file (binary), simply pass its full path as an argument.

Don’t forget, you can also filter by current boot entry to get rid of unnecessary messages.

Conclusion

Journalctl aims to make it easy to find what you’re looking for. If you want to learn about more advanced parameters you can use, consult the journalctl command manual page.

For years, system log and kernel log have been processed by a utility called syslogd. Most Linux-based operating systems have switched to systemd, which comes with a different daemon log called journald. To interact with these logs, use the journalctl utility.

How to use Journalctl on Linux

  1. Authorize users to read the system log
  2. Check the continuity of the journal
  3. Select whether to boot entry journal
  4. Navigate and search through System Journal
  5. Filter log entries by priority
  6. Filter log entries according to the path to handle executable files or Systemd unit

Authorize users to read the system log

Only users of ‘adm’ or ‘systemd-journal’ can read the systemd logs. Distributions like Ubuntu have allowed users to add to the adm group.

Open a terminal emulator and type the following command:

How to use journalctl to read linux system logs

If you see ‘adm’ or ‘systemd-journal’ in the output, you can skip the remaining steps in this section. If not, add yourself to the ‘adm’ group .

You will have to restart the login session in order for these changes to take effect (logging out and logging in). If you can’t do it for a variety of reasons, use this command to log in to the new group without restarting the graphics session:

Do not close the terminal window. You are now in the adm group, at the current terminal session, but not in the graphics session. If you open a new terminal at this time, you won’t be able to log in to the adm group anymore.

Check log continuity

Systemd log records can be recorded continuously or intermittently. On Ubuntu and other distributions, by default, they will continue. On Debian 9, logs are not recorded continuously, meaning they are only stored in memory (not the drive) and disappear when the user shuts down or restarts. Please enter the following command:

How to use journalctl to read linux system logs

If there are multiple entries here, then you don’t have to do anything else. That means the logs are saved on the drive (continuous). If you only get a single entry, the log is not recorded continuously. Please change it to be recorded continuously.

Choose to view the boot entry log

Normally, you will want to see the log for the current boot time. Sometimes you want to see information about the previous boot, such as after a system problem occurs. But this doesn’t happen very often.

To view the log for the current boot time:

How to use journalctl to read linux system logs

For the previous boot, use ‘-1’ instead of ‘0’, or give two previous booting ‘-2’, etc.

Navigate and search through System Journal

After you open the log with journalctl, you can navigate through the text with the arrow keys and the PAGE UP or PAGE DOWN keys . Other useful keys are:

Want to keep an eye on the user actions and tasks happening on your Linux machine? Here’s how to use the journalctl utility.

Log messages are important for auditing and maintaining a healthy Linux system. Every Linux computer stores log messages for different services or jobs. This guide will explore how to read and analyze log messages using journalctl, a command-line tool for reading log messages written by journald.

What Is journald?

Journald is a system logging service that aggregates log messages into a journal. It is a part of the systemd daemon which is responsible for event logging in Linux. The journal is simply a binary file used for storing log messages generated by journald.

Journal log messages are not persistent, because they are stored in RAM, which is a volatile form of storage. By default, journald logs are lost or wiped whenever your PC reboots or loses power. Linux allocates a fixed amount of RAM to journald logs to avoid clogging your system’s memory.

How to Use the journalctl Command

You can use journalctl to query the systemd journal or journald logs. The system indexes all journald logs to improve efficiency when reading log messages from the journal.

Note: This guide uses sudo to run commands using elevated privileges because the journalctl command will not list all log messages when you run it as a regular Linux user.

View All Log Messages

To view all journald logs, simply run the journalctl command without any arguments:

The journalctl command will list all journald logs on your system in chronological order. The command uses less in the background which gives you the same navigation ability as you generally would have with the less command. For example, you can navigate through the logs using the F and B keys on your keyboard.

If you want to change the order in which the system outputs the logs, i.e. show the latest one first, you can use the -r flag with the command. The -r flag stands for Reverse.

View Kernel journald Logs

Kernel logs are very important on Linux because they contain information related to your system from the time it boots up. To view kernel logs only, specify the -k flag with the journalctl command:

The output will also list some kernel information, such as the kernel version and its name.

Filter journald Logs by a Specific Program

You can also view logs related to a specific program or service using journalctl. For example, to view logs associated with the cron service, run the command below:

View Log Messages in Real-Time

Sometimes you might want to view the logs in real-time as they are being logged. For that, issue the following command:

Use the Ctrl + C keyboard shortcut to exit the real-time view.

Get Log Messages by Date

You can use journalctl to filter and analyze the logs using a timestamp. For example, to display the logs from yesterday until now:

You can be more specific by using a detailed “since” and “until” timestamp, as follows:

Journalctl will only display the log messages for the specified period.

View Log Messages by UID or PID

You can also filter journald logs using the user ID (UID) or the process ID (PID). The basic syntax is:

. where 0 is the UID for the root account. You can also replace UID in the aforementioned command with either PID or GID (group ID).

Formatting the journalctl Output

To view journalctl logs using a specific output format, you should use the journalctl -o command followed by your preferred format. For example, to display the logs in a pretty JSON format, run the command below:

Configuring journald on Linux

This guide has shown you how to view and analyze journald log messages on Linux using the journalctl command. The /var/log/journal directory stores all the journald logs. Note that, not all Linux distros have journald enabled by default.

The systemd daemon uses a centralized logging system called a journal, which is managed by the journald daemon. This daemon collects all log entries generated by the Linux kernel or any other systemd unit service regardless of their origin and stores them in a format that is easy to access and manipulate. The journalctl utility is provided for querying and filtering the data held within the journal as written by the journald service,

While diagnosing or troubleshooting an issue with the server or one of our services, the first place to look is the log entries in the journal. Due to its centralized nature, the journal could potentially contain thousands of log entries that may not be relevant to our current problem. Therefore, it’s necessary to filter out what we don’t need to quickly find the relevant information that will help us solve the problem, and that’s what this tutorial is all about.

By reading through this article, you will learn how to do the following:

  • Query the Systemd journal with `journalctl`.
  • Customize the log output format.
  • Filter journal entries by name, time range, log level, or boot session.
  • Clean up old logs manually and automatically.

Prerequisites

Before you proceed with this tutorial, you will need the following:

  • A Linux server that includes a non-root user with sudo access. We tested all the commands presented in this guide on an Ubuntu 20.04 server.
  • A basic understanding of how logging works in Linux.

Step 1 — Permitting a user to view the system logs

By default, a user can only see log entries from systemd services under the user’s control. If you run the journalctl command without any arguments, you may see the following message at the top of the output:

This informs you that the output produced by the utility does not include entries created by system services and those under other users’ control. The way to ensure that you can view all log messages is by adding the user to an existing group such as adm or systemd-journal .

Go ahead and add the current user to the systemd-journal group using the command below:

After logging out and logging back in again, you’ll be able to to see all the available messages, and not just the ones that pertain to the current user.

Step 2 — Querying the journal with Journalctl

In this section, we will query the systemd journal using journalctl , and view the results in various ways. Enter the command below to see all the logs collected by the journald daemon:

When used alone without any options, the journalctl command will output all the journal entries on the system and pipe them through a pager (usually less ). You can also cause journalctl to print its output directly to the standard output instead of using a pager by including the –no-pager flag. This is useful if you want to process the data further with text processing tools like grep , awk , or sed , or redirect the output to a file.

The first line of the output is a header that describes the time range of the displayed logs:

Below the header, you will see log entries generated by various programs in the following format and sorted from oldest to newest:

Each entry starts with a timestamp, the hostname of the machine, the program that generated the log entry, and its process id. The log message itself comes afterward.

If you want to print only the last few log entries, you can use the -n option, which will restrict the printed entries to specified number:

You can also view incoming log messages in real-time through the -f flag, which imitates tail -f :

In later sections, we’ll look at more sophisticated ways of filtering the logs produced by journalctl so that we can find the information we’re looking for quickly. Let’s discuss customizing the output of the journalctl command first though.

Step 3 — Customizing the log output format

When parsing the log entries produced by journalctl , it may be useful to change the format to an easy to parse format like JSON. This is possible by specifying the json format through the -o option:

The logs we discussed in the above section are the “Journal logs”.

The “Journald” is a system program from the systemd tool that collects data from multiple logs in a binary format. It works the same way as syslog but gives a more efficient way to manage logs.

The Linux system has an effective tool termed “journalctl” that helps the user read and communicate with the “journal logs”. This utility is a standard way to display log messages provided by journald and monitor them.

The syntax of the “journalctl” command is:

How to Use “journalctl” Command?

Execute the “journalctl” command without any argument in a command-line to display all records of journal logs from the oldest entries:

How to use journalctl to read linux system logs

The displayed output is listed in less which means you can use directional keys to search and read log entries.

How to Use “journalctl” Command Options?

The “journalctl” command has several options to display journal logs. These are:

Use the “–no-page” to get logs in the terminal instead of to show with “less” features:

It will display the following result:

How to use journalctl to read linux system logs

To display all log fields whether they are unprintable, use the “–all” option in the command-line prompt:

How to use journalctl to read linux system logs

To limit the output lines of journalctl log, run the “-n” flag with the number of lines you want to display:

How to use journalctl to read linux system logs

Run the “-f” flag to display the journal log in real-time:

How to use journalctl to read linux system logs

To display the record of kernel logs, execute the “-k” flag:

How to use journalctl to read linux system logs

To display the list of boot sessions that include the boot time, run the “–list-boot” option of the “journalctl” command:

Whenever you run the “journalctl” command, it will display the journal logs in local time by default, but if you want to specify the logs in UTC, do it through the given command:

How to use journalctl to read linux system logs

The “journalctl” command allows the user to get the record of logs of a certain period.

For example, to get the log record of yesterday, type:

How to use journalctl to read linux system logs

Or if you want to check the record of the last 2 hours, use:

How to use journalctl to read linux system logs

The “-r” or “–reverse” option is used to display the logs with the newest entries first in command-line:

How to use journalctl to read linux system logs

Execute the “-q” or “–quiet” option if you don’t want to display an irrelevant message like warning or info:

How to use journalctl to read linux system logs

To display the help message of the “journalctl” command and all of its options, use the “–help” option:

How to use journalctl to read linux system logs

Check the version of the “journalctl” command using the “–version” option:

Conclusion:

In this brief tutorial, we have seen how the “journalctl” command and its options work. The “journalctl” command is a command-line utility used to display the record of journal logs. The Journal logs are recorded in a binary format that can be accessed anytime. Using the “journalctl” command, one can manage and analyze the record of all logs.

About the author

Wardah Batool

I am a Software Engineer Graduate and Self Motivated Linux writer. I also love to read latest Linux books. Moreover, in my free time, i love to read books on Personal development.

I would like logging.info() to go to journald (systemd).

Up to now I only found python modules which read journald (not what I want) or modules which work like this: journal.send(‘Hello world’)

How to use journalctl to read linux system logs

3 Answers 3

Help us improve our answers.

Are the answers below sorted in a way that puts the best answer at or near the top?

python-systemd has a JournalHandler you can use with the logging framework.

From the documentation:

An alternative to the official package, the systemd package works with python 3.6. Its source is also on github.

The implementation is a mirror of the official lib, with some minor changes:

or for an even shorter method:

How to use journalctl to read linux system logs

This is a solution without third party modules. It works fine for me and the messages show up in journald.

Not the answer you’re looking for? Browse other questions tagged python logging systemd or ask your own question.

Linked

Related

Hot Network Questions

To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

Site design / logo © 2022 Stack Exchange Inc; user contributions licensed under cc by-sa. rev 2022.4.21.42004

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

The logs we discussed in the section above are the “Journal Logs”.

The “journal” is a system program of the systemd tool that collects data from several logs in a binary format. It works the same as syslog but provides a more efficient way to manage logs.

The Linux system has an effective tool called “journalctl” that helps the user to read and communicate with the “journal logs”. This utility is a standard way to view and monitor log messages provided by journald.

The syntax of the “journalctl” command is:

How do I use the journalctl command?

Run the journalctl command with no argument on a command line to view all journal log records for the oldest entries:

The displayed output is listed in less, which means that you can use the directional buttons to search for and read log entries.

How do I use the journalctl command options?

The journalctl command provides several options for viewing journal logs. These are:

Use “–no-page” to call up logs in the terminal instead of displaying them with “fewer” functions:

The following result is displayed:

How to use journalctl to read linux system logs

To see whether all log fields are unprintable, use the “–all” option at the command line prompt:

How to use journalctl to read linux system logs

To limit the lines of output from journalctl log, run the -n flag with the number of lines you want to display:

How to use journalctl to read linux system logs

Run the -f flag to view the journal log in real time:

How to use journalctl to read linux system logs

To display the record of the kernel logs, execute the “-k” flag:

How to use journalctl to read linux system logs

To view the list of boot sessions that contain the boot time, run the –list-boot option of the journalctl command:

When you run the journalctl command, the journal logs are displayed in local time by default, but if you want to specify the logs in UTC, use the command given:

How to use journalctl to read linux system logs

The “journalctl” command allows the user to get the recordings of the logs for a specific period of time.

For example, to get yesterday’s log record, you would type:

How to use journalctl to read linux system logs

Or if you want to check the recording of the last 2 hours use:

How to use journalctl to read linux system logs

The “-r” or “–reverse” option is used to display the logs with the most recent entries first on the command line:

How to use journalctl to read linux system logs

Execute the “-q” or “–quiet” option if you do not want to display irrelevant messages such as warnings or information:

How to use journalctl to read linux system logs

To see the help message for the journalctl command and all of its options, use the –help option:

How to use journalctl to read linux system logs

Check the version of the “journalctl” command with the “–version” option:

Diploma:

In this short tutorial, we saw how the journalctl command and its options work. The journalctl command is a command line utility that is used to view the recording of journal logs. The journal logs are recorded in a binary format that can be accessed at any time. The “journalctl” command can be used to manage and analyze the recording of all protocols.

How to use journalctl to read linux system logs

There are various commands available to Linux users for troubleshooting desktop and server logs. Learn the basics of the journalctl utility of Systemd and its commands that can be used to view and analyze Systemd Logs in this DebugPoint.com guide.

Many say that Systemd is not good, it is heavy on the system and it is a debated topic always. But you can not deny that it provides a well set of utilities to manage, troubleshoot a system. Imagine you end up with a broken system with no GUI. You probably messed up boot and GRUB as well. In those kinds of scenarios or in general – you can boot from a LIVE system, mount your Linux partition and explore the Systemd logs to find out about the problem.

Systemd has three basic components as follows –

  • systemd: System and service manager for Linux operating systems.
  • systemctl: Command to introspect and control the state of the systemd system and service manager.
  • systemd-analyze: Provides system boot-up performance statistics and retrieve other state and tracing information from the system and service manager

Apart from these three, there are additional services that systemd provides such as – journald, logind, networkd, etc. In this guide we will talk about the journald service of systemd.

journalctl command in Linux is used to view systemd, kernel and journal logs. The logs are presented in the following way:

It displays the paginated output, hence it is a bit easy to navigate through a lot of logs. It prints the log in the chronological order with the oldest first.

Working with journalctl command

1. To display all logs

How to use journalctl to read linux system logs

It displays all the logs in the paginated view.

2. To reverse the order or to display the new entries first.

How to use journalctl to read linux system logs

This will display the logs in the chronological order with the newest first.

3. To display only a few log entries

How to use journalctl to read linux system logs

This will display just 2 log entries.

4. To get log entries containing a specific keyword.

How to use journalctl to read linux system logs

This will display all the entries containing the word Centaur in them.

5. To display priority specific log entries.

How to use journalctl to read linux system logs

It displays all log entries with priority as a warning.

6. To print verbose customized output.

How to use journalctl to read linux system logs

This will display the formatted output in verbose mode.

7. To display the boots of the system.

How to use journalctl to read linux system logs

To display all the boots of the system.

8. To display journalctl help.

How to use journalctl to read linux system logs

How to use journalctl to read linux system logs

The above command will display the help section of the journalctl command.

By default, newer systemd based linux systems now uses two logging services for the system logs:

    systemd-journald, which is configured to only keep logs in memory

To view messages in the system journal, a tool called journalctl can be used. If used without any parameters it will show the full contents of the system journal, presented in a pager (by default less is used). The output of journalctl can be modified by using both options and filters. Options can be used to change the number of lines displayed, to turn on follow mode, change the displayed field, specify a time range, etc. Filters can be used to modify for what services and units information is displayed, which executables to display information for, etc.

journalctl examples

journalctl -ef

Jump to the end of the journal (-e, and enable follow mode (-f). This will keep the journal open on screen, displaying new messages as they come in.

journalctl _SYSTEMD_UNIT=sshd.service

This will display all messages generated by the sshd.service systemd unit.

journalctl -u sshd.service

This will display all messages generated by, and about, the sshd.service systemd unit.

journalctl -p emerg..err

Display all messages in the journal with a priority in the range emerg up to and including err.

If a single priority is specified, for example, -p err, all messages up to and including that level are displayed.

journalctl -b -1

Only show messages from the last system boot. This is useful for searching for information about a system crash. This requires a persistent journal to be configured.

journalctl –since “2015-02-02 20:30:00” –until “2015-03-31 12:00:00”

Displays all messages between February 2, half past eight in the evening, and noon on March 31st. This requires a persistent journal to be configured.

journalctl -o verbose

Use verbose output mode (-o verbose). This will show all fields stored in the journal with their field name and contents. All field names can be used as filters on the journalctl command line.

For a complete list of options and filters, refer to the journalctl(1) man page.

Persisting the journal

By default, CentOS/RHEL 7 stores the system journal in /run/log/journal, which is stored on a tmpfs. This implies that on a reboot all stored information will be lost. If the directory /var/log/journal is present the journal will be stored there, thus enabling a persistent journal across reboots.

Enabling a persistent journal can be done by using the following steps:

1. Create the directory /var/log/journal.

2. Set the group ownership of the new directory to systemd-journal, and the permissions to 2755.

3. Inform systemd-journald that the new location should be used by sending a USR1 signal to it. A reboot will also suffice.

Enabling verbose information

Many tools and services can increase the amount of logging they perform, as well as the amount of information they display when run from the command line, by using various configuration options or command-line flags.

Command-line options typically include -v, which can sometimes be specified multiple times, to increase verbosity, or include a –debug option that can be used. Services will typically have configuration options, either in their main configuration file or in /etc/sysconfig/SERVICENAME, that can be used to increase their logging level and/or verbosity as well. Refer to the documentation for these individual services to increase their verbosity and logging levels.

Journalctl is a utility for querying and displaying logs from journald, systemd’s logging service. Since journald stores log data in a binary format instead of a plaintext format, journalctl is the standard way of reading log messages processed by journald.

In the following paragraphs, we’ll show you several ways of using journalctl to retrieve, format, and analyze your logs. These methods can be used on their own or in combination with other commands to refine your search. To get a full listing of journalctl options, visit the journalctl man page.

When run without any parameters, the following command will show all journal entries, which can be fairly long:

The entries will start with a banner similar to this which shows the time span covered by the log.

Journalctl splits the results into pages, similar to the less command in Linux. You can navigate using the arrow keys, the Page Up/Page Down keys, and the space bar. To quit navigation, press the Q key.

Long entries are printed to the width of the screen and truncated off at the end if they don’t fit. The cut-off portion can be viewed using the left and right arrow keys.

Boot Messages

Journald tracks each log to a specific system boot. To limit the logs shown to the current boot, use the -b switch.

You can view messages from an earlier boot by passing in its offset from the current boot. For example, the previous boot has an offset of -1, the boot before that is -2, and so on. Here, we are retrieving messages from the last boot:

To list the boots of the system, use the following command.

It will show a tabular result like this.

The first field is the offset (0 being the latest boot, -1 being the boot before that, and so on), followed by a Boot ID (a long hexadecimal number), followed by the time stamps of the first and the last messages related to that boot.

Time Ranges

To see messages logged within a specific time window, we can use the –since and –until options. The following command shows journal messages logged within the last hour.

To see messages logged in the last two days, the following command can be used.

The command below will show messages between two dates and times. All messages logged on or after the since parameter and logged on or before the until parameter will be shown.

For greater accuracy, format the date and time as “YYYY-MM-DD HH:MM:SS”. You can also use any format that follows the systemd.time specification .

By Unit

To see messages logged by any systemd unit, use the -u switch. The command below will show all messages logged by the Nginx web server. You can use the –since and –until switches here to pinpoint web server errors occurring within a time window.

The -u switch can be used multiple times to specify more than one unit source. For example, if you want to see log entries for both nginx and mysql, the following command can be used.

Follow or Tail

Journalctl can print log messages to the console as they are added, much like the Linux tail command. To do this, add the -f switch,

For example, this command “follows” the mysql service log.

To stop following and return to the prompt, press Ctrl+C.

Like the tail command, the -n switch will print the specified number of most recent journal entries. In the command below, we are printing the last 50 messages logged within the last hour.

The -r parameter shows journal entries in reverse chronological order, so the latest messages are printed first. The command below shows the last 10 messages from the sshd daemon, listed in reverse order.

Output Formats

The -o parameter enables us to format the output of journalctl query. -o (or –output if we are using the long form parameter name) can take a few values.

  • jsonwill show each journal entry in json format in one long line. This is useful when sending logs to a log centralization or analysis service, since it makes them easier to parse.
  • json-pretty will show each log entry in easy-to-read json format.
  • verbose will show very detailed information for each journal record with all fields listed.
  • cat shows messages in very short form, without any date/time or source server names.
  • shortis the default output format. It shows messages in syslog style.
  • short-monotonic is similar to short, but the time stamp second value is shown with precision. This can be useful when you are looking at error messages generated from more than one source, which apparently are throwing error messages at the same time and you want to go to the granular level.

For example, the following command prints logs from the Apache web server in json-pretty format.

Here’s a sample of the output. You can see a number of important fields including the user, group, syslog facility, and even the code location that generated the message (if available).

By Priority

Use the -p switch to filter out messages based on their priority level. To see what priority levels are available, see the Journald Configuration section of Linux Logging with Systemd and the MaxLevelStore parameter. If a single priority level is specified, all messages with that priority level and below are returned. To use a range of priority levels, provide the start and end levels in the form of FROM…TO. As an example, the command below will output all messages with priority between emergency and critical from last boot.

By User

To find all messages related to a particular user, use the UID for that user. In the following example, we are finding the UID of the user mysql.

This returns a line like this.

And then we are querying the journal for all messages logged by that user.

systemd is a popular Linux utility to manage services & processes. However, sometimes you may find that a particular service managed by systemd is not functioning properly. In such cases, you need to use journalctl that can be used used to query journald, the logging facility of systemd. It allows you to query system boot information from journald to investigate when the said service was working well. In this article, we will look at a few handy journalctl commands to help you easily debug systemd issues.

How to Use Journalctl Command in Linux

Here is how to use journalctl command. You can simply enter the command as-is to get latest systemd log information.

Here is a sample log entry.

Each log entry consists of month, day and time, followed by system hostname, service name and PID of service. This is followed by actual journald information.

If there is an error, the keyword ‘error’ will be mentioned towards the end of log entry. Also it will be highlighted in red color, when you run journalctl. Here is an example.

If you want to list only boot related entries, use -b option.

This will show you the boot information about current boot, also numbered as 0. If you want to see the information related to previous boot use -1 after -b above.

For information about boot before that use -2.

This will help you compare information across different boots. For example, you can check the log entries for same service across different boots.

If you want to find out how many boots have happened run the following command.

You will see the list of boots as shown below.

Created with GIMP

View Journald logs by Time

If you want to view log entries by time, say, for past 1 hour, use –since option with the time frame mentioned in double quotes.

You can also mention time frame in YYYY-MM-DD HH:MM:SS format to be more specific.

View Journald logs by service

If you know the name of service you need to get information you can view its log entries with -u option followed by service name. Here is an example to view log entries of OpenNMS service.

You will see output as shown, including its error message, indicating there is some issue with it.

How to use journalctl to read linux system logsCreated with GIMP

You may also format the output of journalctl command with -o option followed by one of the available formats. Here is the command to get output in json format.

Here are the available formats:

  • json – show each journal entry in json format
  • json-pretty – show each log entry in an easier-to-read json format
  • verbose – detailed information for each entry
  • cat – messages in very short form
  • shortis – default, syslog, output format
  • short-monotonic – similar to short, but includes time stamp value

That’s it. In this article, we have covered many important aspects of journalctl command that you can use to debug systemd logs and services.

In this article I will share the steps to enable persistent logging in systemd-journald without rebooting the node or restarting the systemd-journald service. The systemd journal is configured by default to store logs only in a small ring-buffer in /run/log/journal .

The default storage type in journald.conf is “auto”. In auto storage type the journal logs will not be persistent and will not survive reboots.

How to use journalctl to read linux system logs

Different Storage Type supported with journald.conf

  • The storage type value is controlled using /etc/systemd/journald.conf file. Under [Journal] you can modify the storage type.
  • The supported values are “volatile”, “persistent”, “auto” and “none”
  • Default storage type is configured as “auto”
  • If “volatile“, journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed)
  • If “persistent“, data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy, with a fallback to /run/log/journal during early boot stage and if the disk is not writable
  • The “auto” value will configure journald to store journal log data in the /var/log/journal/ directory. However, the directory must already exist and have the proper permissions set. If it does not exist, then journal data is stored in the volatile /run/log/journal/ directory, and the data is erased when the system shuts down.
  • none” turns off all storage, all log data received will be dropped.

Enable persistent logging in systemd-journald

Method 1:
Now if your current storage type is auto in /etc/systemd/journald.conf , in that case the “auto” value will configure journald to store journal log data in the /var/log/journal/ directory. However, the directory must already exist and have the proper permissions set. If it does not exist, then journal data is stored in the volatile /run/log/journal/ directory, and the data is erased when the system shuts down.

Now in this case to enable persistent logging in systemd-journald you just need to create a directory /var/log/journal/ and systemd-journald will start writing journal log files into the disk rather than memory.

With this change the logs will become persistent and will not be erased after reboot

Single step to be followed

No need to restart the systemd-journald service or reboot the node. As soon as this directory is accessible, systemd-journald will start writing logs under this path

Method 2:
There is no problem in using the steps from Method 1 but again to enable persistent logging in systemd-journal properly we must change the “Storage” type to “persistent“. In which case you need not manually create /var/log/journal directory.

Replace storage type with Storage=persistent

Next you can restart systemd-journald service

With storage option “persistent“, data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable.

You can always use flush to move the journal log files from /run/log/journal to /var/log/journal

here this command asks the Journal daemon to flush any log data stored in /run/log/journal into /var/log/journal , if persistent storage is enabled. This call does not return until the operation is complete

Lastly I hope the steps from the article to enable persistent logging in systemd-journald on Linux was helpful. So, let me know your suggestions and feedback using the comment section.

Related Posts

Didn’t find what you were looking for? Perform a quick search across GoLinuxCloud

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

3 thoughts on “How to enable persistent logging in systemd-journald without reboot”

It solves the problem of persistente of journald logs, but it may create a storage problem. How would one rotate the logs beneath this directory? I mean you probably don’t need lifetime logs for this, maybe just keeping from the last reboot.

In the past SystemV used syslog (or rsyslog) to log events to a log file. Many times there is a separate log file for each service.

With the switch to SystemD, Red Hat, Fedora, CentOS, etc. also introduced a new logging facility and tool called the journal. This centralized the collection of logs and allowed administrators one simple robust tool and one location to inspect and manipulate log data. Here we will cover some of the basics of journalctl, the program used to interface with the logs.

To simply view the logs on your system, you can execute the following command:

This will display the logs with the oldest entries first. Although this is simple, it is not very useful since we do not tend to read logs like a book.

By default journalctl displays the logs in a pager. It shows you one page of logs requiring you to hit the space bar to proceed. Also long log lines WILL NOT wrap, they will trail off the right side of the screen. You can use the right arrow to see the rest of the line.

We will talk more about changing the way the logs are displayed in a different article. Let’s move on to some basic log viewing commands.

Diplaying Logs by Date

More than likely you are looking for an event. One way to find something is the logs is to display the logs from a certain time. You can specify a single time and it will display all logs SINCE that time, or you can specify a time window.

To find all logs since December 25th 2015 at 07:00 PM you can run:

journalctl –since “2015-12-25 19:00:00”

To find all logs between December 25th 2015 and January 1st 2016:

journalctl –since “2015-12-25” –until “2016-1-1”

You can also use the more human friendly relative terms. For example, to see all logs since yesterday:

journalctl –since yesterday

You also have the option to mix the absolute and relative terms:

journalctl –since “2015-12-25” –until “2 hours ago”

Displaying Logs by Unit or Service

Another way to find the logs you need would be to filter the results by unit (or service). For example, if you want to see all the logs vsftpd (FTP software) produced you can specify that in the journalctl command like so:

journalctl -u vsftpd.service

You can mix in a time, or a time window to find logs from a specific service during a specific time.

To find all the vsftpd logs from December 5 2015 to January 8 2016 you can runt he following command:

journalctl -u vsftpd.service –since “2015-12-05 16:17:47” –until “2016-1-8 15:03:02”

You can also request logs from two different services at the same time. This comes in handy when trying to get information about how two services are interacting or debugging an issue.

To see all the logs from vsftpd and firewalld you can run this command:

journalctl -u vsftpd.service -u firewalld.service

You can also specify a time in absolute, relative or any combination.

journalctl -u vsftpd.service -u firewalld.service –since “2 days ago”

Displaying Logs by User or Group

Other things you can do is find logs generated by a specific user (UID) or group (GUID).

For example, let’s say I wanted to see all logs from the user “savona”. First I would find their UID like so:

]# id savona
uid=1000(savona) gid=1000(savona) groups=1000(savona)

Now that I know their UID is 1000, I can use the _UID filter in journalctl like so:

journalctl _UID=1000

And of course I can mix this with a time window:

journalctl _UID=1000 –since “2 days ago”

Displaying Logs by Process ID

You can also use _PID to search for process id, or _GUID for group id.

journalctl _PID=1221

Displaying Kernel Logs

Since all the logs are kept in one place, we can use the same tool (journalctl) to view just the kernel logs:

journalctl -k

The above command will show you all the kernel messages from the current boot. You can specify a different boot using the boot selection option like so:

journalctl -k -b 2

The above command will show you all the kernel messages from 2 boots ago.

Displaying Logs Since Last Boot

The boot selection option will work on it’s own as well. If you would like to see all the logs generated since the last boot up, simply give the -b option:

journalctl -b

Displaying Logs by Priority

You can also select to view logs by priority. The journal uses the same syslog message levels:

0: emerg
1: alert
2: critical
3: error
4: warning
5: notice
6: info
7: debug

To see all logs from priority 4 (warning) and higher:

journalctl -p 4

To see all the logs from priority 3 (error) and higher since last boot:

journalctl -p 3 -b

And of course you can use time windows if you like:

journalctl -p 3 –since “2 days ago”

Tailing or Following the Log

In my opinion on for the most useful commands for viewing logs, follow allows you to view the log as they are bring written. You may of used “tail -f” in the past. The journalctl utility has the same function.

journalctl -f

Also, similar to tail, you can view the last 10 entries by using the -n option like so:

journalctl -n

Or you can see the last 50 entries by specifying a number of the “-n” option:

journalctl -n 50

You can also see the last 50 entries, then begin to follow by mixing the commands like so:

journalctl -n 50 -f

Finding Size of Logs / Log Maintenance

To find how much disk space is being used by the journal simply ask:

journalctl –disk-usage

If you are concerned about disk space, you can trim (remove oldest) the logs.

You can do this by specifying the amount of disk space you want to keep or the time you would like to keep.

For example, if you want to delete all logs and keep just 5GB of data:

journalctl –vacuum-size=5GB

If you want to keep only logs from the last year:

journalctl –vacuum-time=1years

Now you should have a decent idea of how to find the logs you are looking for (I just said that in the voice of Obi-Wan). If you have any questions or require further explanation, please feel free to sound off in the comments.

Despite some of the negative publicity, systemd have brought great new functionality to the Linux operating system. Logging is one of them. In the old days before systemd, checking logs and troubleshooting issues was sometimes very unpleasant thing to do because logs for different programs were scattered in different files and you had to check timestamps in each and every one of them. Of course there was syslog file also, but sometimes was not as verbose as needed. The systemd brings us journal daemon called journald which enables centralized management of logs. This logs can be filtered and manipulated by utility called journalctl. In this article we will go through basic usage of utilities.

What is journald

The journald is service that collects and stores logs from many sources, and make indexed structured log files that are easy to interpret. Some of the sources that journald uses

  • Kernel log messages, via kmsg
  • Simple system log messages, via the libc syslog call
  • Structured system log messages via the native Journal API
  • Standard output and standard error of system services

Idea is to have all log messages centralized, no matter from which application they are coming from.

Setting the time

In order for logs to be useful, you must have correct system time. Logs can have time by UTC or by local time. Generally both time-stamps are saved. So lets set up timezone first

This command sets the time to Central European timezone, but if you are on any other spot of the globe, you just need to enter continent and press tab to see cities you might chose. It doesn’t matter if it is not your city, I am not living in Belgrade either, it is just closest big city. Next lets see the status of the timezone

1) Viewing logs

Now that we have correct time, lets continue to viewing logs. The basic command for viewing logs is

It will get you output similar to this:

That is first line, and followed by many lines. We see that logs begin from last system boot. I last time rebooted the computer yesterday. By default, journald doesn’t save logs across reboots, because that would make log files get to big over time. The log seems similar to syslog, but it has many more sources. We said that both UTC and your local timezone timestamps are saved, so you can also look at the same logs with UTC timestamps.

Yes, I know that logs are huge. We said that logs will be great again! But they are too great, so the journald has neat feature of filtering logs.

If you want to see logs from some point in time till now, you need to use command formatted like this:

2) Filtering logs by time

This is still too big, so we can make them smaller if we add one more operand after –since, and that will be –until

In those five minutes, there were only two log massages.

Since and until flags accept the time stamps in format YYYY-MM-DD HH:MM:SS but that is not only way. If for example get a user complain that your server is not working in last 5 minutes, you can examine logs like this:

It also understands words like yesterday or today so you can look for logs since yesterday or today.

3) Setting up logging across reboots

This this all is more useful if you have the logs persistent across reboots. If you want logs to remain after boots, you need to enable this manually. First create directory where it is going to be logged

Then edit the configuration file to log persistently.

After this you can reboot your machine and check if boots span across reboots with following command:

I rebooted the computer two times and both boots are logged. I thought that previous boot would get saved as well but it did not. So only those reboots that are done after setting persistent logs will be logged.

4) Managing logs from different boots

If you now want to see logs from this boot only, you use command

You can use boot number to pick specific boot

Or you could use boot id

The time based filtering will also work

5) Filtering by service and id

Say you want to see logs by one program only. I will look logs of fprintd

You can also filter logs by UID. First you need to get UID of your user

And next you can use that uid to filter logs

You can also filter by GID

Now can filter by any of those GIDs

Filtering logs by path to to executable:

Seeing logs in real time

If you want your logs to be displayed in plan text for easy copy and paste, you can use “no pager” option

6) Displaying kernel messages and deleting old logs

The journalctl can display kernel messages as well. By default, it will display from current boot

If you want kernel messages from previous boot, you use the -b -1 flag.

To check how much disk space your logs use, you can use this command

If your logs have grown in size, you can use vacuum cleaner to wipe the oldest one. Well not really a vacuum cleaner, but command is called vacuum.

This will reduce the log size to 16MB.

Conclusion

We have gone through basic journald commands that make logging great again. This will hopefully help you greatly when you next time have to troubleshoot some sticky problem with your Linux server or desktop computer with systemd. The systemd is brings us great tools and negative publicity that it got early was just backlash against changing old tools that people were used too. I already got used to systemd and I don’t know how I lived without it. This is all for this article, thank you for reading and have a nice day.

How to use journalctl to read linux system logs

Log messages are important for auditing and maintaining a healthy Linux system. Every Linux computer stores log messages for different services or jobs. This guide will explore how to read and analyze log messages using journalctl, a command-line tool for reading log messages written by journald.

What Is journald?

Journald is a system logging service that aggregates log messages into a journal. It is a part of the systemd daemon which is responsible for event logging in Linux. The journal is simply a binary file used for storing log messages generated by journald.

Journal log messages are not persistent, because they are stored in RAM, which is a volatile form of storage. By default, journald logs are lost or wiped whenever your PC reboots or loses power. Linux allocates a fixed amount of RAM to journald logs to avoid clogging your system’s memory.

How to Use the journalctl Command

You can use journalctl to query the systemd journal or journald logs. The system indexes all journald logs to improve efficiency when reading log messages from the journal.

Note: This guide uses sudo to run commands using elevated privileges because the journalctl command will not list all log messages when you run it as a regular Linux user.

View All Log Messages

To view all journald logs, simply run the journalctl command without any arguments:

The journalctl command will list all journald logs on your system in chronological order. The command uses less in the background which gives you the same navigation ability as you generally would have with the less command. For example, you can navigate through the logs using the F and B keys on your keyboard.

If you want to change the order in which the system outputs the logs, i.e. show the latest one first, you can use the -r flag with the command. The -r flag stands for Reverse.

View Kernel journald Logs

Kernel logs are very important on Linux because they contain information related to your system from the time it boots up. To view kernel logs only, specify the -k flag with the journalctl command:

The output will also list some kernel information, such as the kernel version and its name.

Filter journald Logs by a Specific Program

You can also view logs related to a specific program or service using journalctl. For example, to view logs associated with the cron service, run the command below:

View Log Messages in Real-Time

Sometimes you might want to view the logs in real-time as they are being logged. For that, issue the following command:

Use the Ctrl + C keyboard shortcut to exit the real-time view.

Get Log Messages by Date

You can use journalctl to filter and analyze the logs using a timestamp. For example, to display the logs from yesterday until now:

You can be more specific by using a detailed “since” and “until” timestamp, as follows:

Journalctl will only display the log messages for the specified period.

View Log Messages by UID or PID

You can also filter journald logs using the user ID (UID) or the process ID (PID). The basic syntax is:

…where 0 is the UID for the root account. You can also replace UID in the aforementioned command with either PID or GID (group ID).

Formatting the journalctl Output

To view journalctl logs using a specific output format, you should use the journalctl -o command followed by your preferred format. For example, to display the logs in a pretty JSON format, run the command below:

Configuring journald on Linux

This guide has shown you how to view and analyze journald log messages on Linux using the journalctl command. The /var/log/journal directory stores all the journald logs. Note that, not all Linux distros have journald enabled by default.

You can use the /etc/systemd/journald.conf file to configure or make changes to the journald configuration on your PC. Apart from an effective logging service, there are several other tools that are a must if you are serious about the security of your Linux servers.

Don’t want to compromise on the security of your Linux server? Install these six tools to set up an impenetrable network.

About The Author

How to use journalctl to read linux system logs

Mwiza Kumwenda
(30 Articles Published)

Mwiza develops software by profession and writes extensively on Linux and front-end programming. Some of his interests include history, economics, politics & enterprise-architecture.

Subscribe to our newsletter

Join our newsletter for tech tips, reviews, free ebooks, and exclusive deals!

The systemd journal is systemd’s own logging system. It is equivalent to the syslog in the init system. It collects and stored kernel logging data, system log messages, standard output and error for various systemd services.

A Linux machine with systemd writes the logs to /var/log/journal directory. If you remember the Linux directory structure, /var is where the system logs are stored.

You can either manually view the log files using less command or use the journalctl command. To view all the latest logs, use the command with reverse option.

The thing with logging is that over the time, it starts to grow big. And if you check the disk space in Linux, you’ll see that sometimes, it takes several GB of space.

Let me show you how to clean systemd journal logs and free up disk space on your Linux system.

Clearing systemd journal logs

How to use journalctl to read linux system logs

First check the space taken by journal logs with the du command:

You can also use the journalctl command for the same task:

Both of the command should give approximately the same result:

Now that you know how much space the journal logs take, you can decide if you want to clear the logs or not. If you decide to clear the journal logs, let me show you a couple of ways of doing it.

You can of course use the rm command to delete the files in the log folder but I won’t advise that. The journalctl command gives you the proper way of handling old logs.

First thing you should do is to rotate journal files. This will mark the currently active journal logs as archive and create fresh new logs. It’s optional but a good practice to do so.

Now you have three ways to clear old journal logs. You delete logs older than a certain time or you delete older log files so that total log size is limited to the predefined disk space or you limit number of log files. Let’s see how to use all three methods.

1. Clear journal log older than x days

Keep in mind that logs are important for auditing purpose so you should not delete all of them at the same time. Let’s say you want to keep the log history of just two days. To delete all entries older than two days, use this command:

Here’s what the output may look like:

You can also change the provide time frame in hours like 2h, in minutes like 2m, in seconds like 2s. If you want bigger time units, you can 2weeks, 2months as well.

2. Restrict logs to a certain size

Another way is to restrict the log size. With this, it will delete the journal log files until the disk space taken by journal logs falls below the size you specified.

This will reduce the log size to around 100 MB.

You can specify the size in GB with G, MB with M, KB with K etc.

3. Restrict number of log files

The third way is to limit the number of log files. The journalctl usually has log files for the system and for the users. As the logs get old they are archived in various files.

You can limit the number of archive log files. Let’s say you want to have only five log files.

It will remove the older archive log files leaving only the specified number of log files.

Linux Handbook Abhishek Prakash

How to use journalctl to read linux system logs

Automatically clearing old log files [Requires intermediate knowledge of command line]

What you just did will clean the log files for now. In a month the logs will increase again. You can manually clean them with one of the methods described above. But that is a tedious task and you may not remember to do that regularly.

The good thing is that you can configure systemd to automatically handle old log files.

The journalctl has configuration file at /etc/systemd/journald.conf. There are settings which are commented out. The commented lines basically indicate the default value of those settings parameters (even if they are commented out).

You can change some of these default settings to automatically clean the log files.

You would want to change the following settings:

Setting Description
SystemMaxUse Max disk space logs can take
SystemMaxFileSize Max size of an INDIVIDUAL log file
SystemMaxFiles Max number of log files

Please note that you should be careful while editing configuration files. You must be comfortable using a terminal-based text editor like Vim, Emacs or Nano so that you don’t make silly mistakes while editing the conf file.

I advise to make a backup of the configuration file first:

Now, you should uncomment (remove the # at the beginning of the line) the setting you want to use. For example, I want to restrict the maximum disk space taken by the log files to 250 MB.

You’ll have to use Vim or some other terminal based editor in order to edit this configuration file. Here’s what it looks like me after editing the file.

Keep in mind that after editing the configuration file, you should load the changes:

The journald.conf file can be used to tweak the journalctl settings even further. You can even set the levels of log (info, debug, error etc) you want to see. It’s up to you if you want to change those settings as well.

Linux Handbook Abhishek Prakash

How to use journalctl to read linux system logs

I hope you like this tip to clear systemd journal log files. If you have any questions or suggestions, please leave a comment below.

When I run the following command as the root user in the Centos 7 linux terminal, it produces 57 lines of output:

So how do I change the code below to successfully call this from Java without having to leave my password in a file?

Here is my attempt. When I execute the following code, the console only outputs exit: 1 :

EDIT:

When I add the following:

The following ouput gets generated:

Is the problem related to permissions? When I run journalctl –output=json-pretty UNIT=firewalld.service as root from the linux terminal, I get the 57 lines of output. But when I run journalctl –output=json-pretty UNIT=firewalld.service as a normal user, the terminal tells me that no files were found. I do not want to put my root password in java code. Is there some other way to call journalctl from java without having to leave the system root password in a file?

2 Answers 2

You can add your (normal) user to the group systemd-journal using usermod -a -G systemd-journal . Logout and login to it for the change to take effect. This gives your user access to the system journal files without giving it complete root privileges.

The group may be different for different operating system setups. You can simply take a look at which group the journal files belong to, by using ls -l /var/log/journal/ or ls -l /run/systemd/journal/ if /var/log/journal/ does not exist.

I have tried this in Centos LiveCD, and the group the files belonged to was root . So you can add the user to group root , which is not the same as giving it full root permissions.

Log files contain messages about the system, including the kernel, services, and applications running on it. These contain information that helps troubleshoot issues, or simply monitor system functions. Fedora uses the systemd system and service manager. With systemd, messages for most services are now stored in the systemd journal which is a binary file that must be accessed usinng the journalctl command.

System tools that do not use systemd for their logs continue to place them as plain text files in the /var/log/ directory. In Fedora, there are two ways of accessing system logs:

The command line

A GUI applications

Using the command line to view log files

The journalctl command can be to view messages in the system journal on the command line. For plain text log files, generic tools may be used:

cat , more , less , tail , or head .

the grep command to search for specific information.

any text editor of your choosing (nano/pico/vim/emacs)

Please note that you may require sudo access to view these files.

Using journalctl to view system information

To view all collected journal entries, simply use:

To view a logs related to a specific file, you can provide the journalctl command with a filepath. The example shown below shows all logs of the kernel device node /dev/sda :

To view log for the current boot use the -b option :

To view kernel logs for the current boot, you can add the -k option:

Using journalctl to view log information for a specific service

To filter logs to only see ones matching the “foo” systemd service:

Matches can be combined. For example, to view logs for systemd-units that match foo , and the PID number :

If the separator “+” is used, two expressions may be combined in a logical OR. For example, to view all messages from the foo service process with the PID plus all messages from the foo1 service (from any of its processes):

If two matches refer to the same field, all entries matching either expression are shown. For example, this command will show logs matching a systemd-unit foo or a systemd-unit foo1 :

The files for service modification are stored in a directory within /etc/systemd/system , to know more about systemd, please refer to understanding-and-administering-systemd.adoc#Understanding Systemd Services

Using journalctl to view older logs

To view older logs use the –list-boots option :

This will show a tabular list of boot numbers, their IDs, and the timestamps of the first and last message pertaining to the boot:

with this ID you can use journalctl as usual :

To know more about journalctl , read the man page:

Using Gnome Logs to view log files

The GNOME Logs application provides a convenient GUI tool to view the systemd journal. GNOME Logs is not currently installed by default on Fedora systems.

You can install Gnome Logs using the default software installation application on your system. On a Fedora Workstation install running the GNOME desktop:

Press the Super key

In the Search field type Logs and choose the GNOME Logs item from the list of results

Install the application

You can also install GNOME Logs using the command line with dnf :

In GNOME Logs , you can filter for time periods, search within logs, and display categories.

To select a log file type, from the side bar of GNOME Logs, select the type to view.

To select a time period, from the menu bar, click Log , and select a time period.

To search within logs, select a log file from the results pane.

Click the search icon.

Enter one or more search criterion in the search field.

All Fedora Documentation content available under CC BY-SA 4.0 or, when specifically noted, under another accepted free and open content license.

Create professional documents, spreadsheets and presentations on Ubuntu with LibreOffice, the open source office suite that’s compatible with Microsoft Office.

How to use journalctl to read linux system logs

How to use journalctl to read linux system logs

Linux Mint is one of the most popular desktop Linux distributions and used by millions of people.

How to use journalctl to read linux system logs

How to use journalctl to read linux system logs

Arch Linux is a powerful Linux distribution intended to be lightweight and a simple approach for experts for simplicity and elegance.

​Example Journalctl Output

Example of the output is below for journalctl without parameters

Show Current Boot Messages

Show messages from a specific boot. This will add a match for “_BOOT_ID ebd-block ” data-type=”note” style=”font-size:18px;line-height:120%”>

Boot Messages using Ellipsize fields

Using -b and -l, -l meaning lower case “L”. The default is to show full fields, allowing them to wrap or be truncated by the pager, if one is used.

Getting Boot ID to List

List Journalctl Catalogs

Display Journalctl Log Ranges

The below example gets logs since 3 hours ago on the current system

Display Journalctl Log by Time & Date

​Checking logs on 17th and 18th of April 2018 @ 00:00 until 01:00 would show like below example

Journalctl Display Logs By Unit

Show messages for the specified systemd unit UNIT (such as a service unit), or for any of the units matched by PATTERN. If a pattern is specified, a list of unit names found in the journal is compared with the specified pattern and all that match are used.

You could also specify multiple units from a single command line

Showing Live Journalctl Messages

Show only the most recent journal entries, and continuously print new entries as they are appended to the journal.

List Journalctl Logs Newest Entries ​

Reverse output so that the newest entries are displayed first.

Display Logs Today Only

Get logs produced today by using below command

You could use both –since and –until like below pretty much just like -S and -U but it doesn’t work with -S today, only –since today

Journalctl Output Formats

short – is the default and generates an output that is mostly identical to the formatting of classic syslog files, showing one line per journal entry.

short-full – is very similar, but shows timestamps in the format the –since= and –until= options accept. Unlike the timestamp information shown in short output mode this mode includes weekday, year and timezone information in the output, and is locale-independent.

short-iso – is very similar, but shows ISO 8601 wallclock timestamps.

short-precise – is very similar, but shows timestamps with full microsecond precision.

short-monotonic – is very similar, but shows monotonic timestamps instead of wallclock timestamps.

short-unix – is very similar, but shows seconds passed since January 1st 1970 UTC instead of wallclock timestamps (“UNIX time”). The time is shown with microsecond accuracy.

verbose – shows the full-structured entry items with all fields.

export – serializes the journal into a binary (but mostly text-based) stream suitable for backups and network transfer (see Journal
Export Format[1] for more information).

json – formats entries as JSON data structures, one per line (see Journal JSON Format[2] for more information).

json-pretty – formats entries as JSON data structures, but formats them in multiple lines in order to make them more readable by humans.

json-sse – formats entries as JSON data structures, but wraps them in a format suitable for Server-Sent Events[3].

cat – generates a very terse output, only showing the actual message of each journal entry with no metadata, not even a timestamp.

Display High Priority Journalctl Messages

List only 20 lines with priority “critical” on all logs in system

List 20 lines with high priority “critical” on only 1 unit specified below

  • 0: emerg
  • 1: alert
  • 2: crit
  • 3: err
  • 4: warning
  • 5: notice
  • 6: info
  • 7: debu

Display Only Kernel Messages

Show only 10 lines from kernel only. -k Show only kernel messages. This implies -b and adds the match “_TRANSPORT=kernel”.

System logging is one of the objectives on the Red Hat Certified System Administrator (RHCSA) exam and I thought it would be fun to go over a few things that I learned while studying this section.

Most system logs are logged in text files in the /var/log directory. You can use less or tail to view the contents of the logs. In RHEL, the systemd-journald and rsyslog services handle the syslog messages.

systemd-journald

The systemd-journald service is the core of the operating system event logging. It collects event messages from many sources including the kernel, the boot process, and standard error from daemons as they start up. By default, this journal is stored on a file system that does not persist when the system reboots.

Journalctl is the utility for querying and displaying from from journald. Journald stores data in binary format instead of plain-text format. Using journalctl is the way you read the log messages processed by journald. journald is part of the systemd software bundle. Systemd was a project started in 2010 in an effort to surpass the efficiency of the init daemon by improving the framework for dependencies and allowing more concurrent processing during system booting, for example. Systemd is in charge of managing services – starting them and keeping them alive. In May 2011, Fedora became the first major Linux distribution to enable systemd by default. RHEL 7 was the first RHEL unit to support systemd.

rsyslog

The rsyslog service sorts and writes syslog messages to the log files that do not persist when the system reboots. Rsyslog can be used to “capture” systemd-journald logs and store them in log files that are persistent on reboot. It stores these log files in /var/log. The log messages are sorted to specific log files based on the type of program that sent each message (the facility), and the priority of each syslog message. Some common log files and the types of messages stored are:

/var/log/messages Most syslog messages except for messages related to authentication and email processing, schedule job execution, and debugging.

Hi everyone. Today I’m going to tell you about my project idea. Before this post, I published another one.

What’s this journalctl?

Let’s dig into journalctl man page using the below command;

We will see an output like that

Exit fullscreen mode

So, it’s a command to get systemd logs and it uses systemd-journald.service

What is systemd-journal service

systemd-journald is a system service that collects and stores logging data. It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources. These are can be message types;

  • Kernel Messages
  • Simple Log Messages
  • Audit records

There are a lot of messages you can find.

Let’s See Some journalctl Commands

If you’re using journalctl without any parameters it will show full output;

Exit fullscreen mode

The output

Exit fullscreen mode

You can get json output in pretty format

Exit fullscreen mode

The output

Exit fullscreen mode

You can also get specific outputs for instance chrome’s logs;

Exit fullscreen mode

The output

Exit fullscreen mode

As you see, these are the oldest messages. What about the current boot’s log or specific boot’s messages?

To get a list of boots, use this command;

Exit fullscreen mode

The output

Exit fullscreen mode

To see boot 91’s message, use this command;

Exit fullscreen mode

There are many commands you should know.

What Will We Do?

As we see journalctl useful command to understand system or application logs. But it’s also hard to understand. You have to use a terminal, you have to know all commands. (In this idea you have to) but the end-user may don’t want to know all commands.

We can write a parser in our best programming language. It can be a web project or another terminal project or GUI application.

Users can filter logs between two dates

To do this idea, use this command;

Exit fullscreen mode

  • -S: since
  • -U: until

And search about this command

Users can filter logs by specific services

For example, you want to see logs for apache2 use this command;

Exit fullscreen mode

  • -u: unit

Users can filter logs by specific binary

For example, you want to see logs for chrome use this command;

Exit fullscreen mode

  • _COMM: match for the script name is added to the query

Users can see all boots

I’m an end-user who wants to see all boots. But it’s really hard to see for me. Use this command;

Exit fullscreen mode

Users can see logs from different boots

For instance, we want to see the logs for boot 35, we should use this command;

Exit fullscreen mode

Users can filter logs by priority

To filter logs by priority use this command;

Exit fullscreen mode

You can specify the number or level key.

Exit fullscreen mode

  • -p: priority

These are log levels;

  • 0: emerg
  • 1: alert
  • 2: crit
  • 3: err
  • 4: warning
  • 5: notice
  • 6: info”
  • 7: debug”

Technologies

You can use various technologies to achieve this idea. For example, golang really good programming language. I believe you can do that in Python easily. I’ll choose NodeJS to do that.

Actually these are my thoughts. You can extend them. Your project will have better features than my project’s features.

Sorry for the grammar mistakes.

Thanks for reading ^_^ and if there is something wrong, tell me.

Resources

These resources helped a lot while thinking of this idea. I learned many new things. Remember that you can learn new things while thinking about something.

One of the common themes in modern Linux is the adoption of systemd. Like it or hate it, it is here to stay.

The main component of logging in systemd is the Journal, controlled by journald. Linux users are undoubtedly familiar with invoking journalctl to view Journal logs. As Python developers that target Linux environments, it isn’t unusual to use systemd to manage our logged events.

I like this approach almost as much as I like logging to stdout, as it is consistent, expected (on Linux), and there is plenty of tooling around to support scraping the logs from systemd to push to central logging, aggregation, etc. (take a look at fluentd and Fluent Bit!).

Installing system dependencies

The Python package that we will rely on is appropriately named systemd. But before we can pip install systemd , we need to first grab system dependencies (specific to the distro we’re using).

On RPM-based distributions (RHEL, CentOS, Fedora) this should be as simple as installing systemd-devel . On my beloved CentOS machines, that would be:

On Debian-based distributions, as per the documentation, the dependencies are build-essential , libsystemd-journal-dev , libsystemd-daemon-dev , and libsystemd-dev .

Once these dependencies are installed, you can then install our necessary Python package:

Note: I highly recommend you use a virtual environment for your Python development and deployments (unless containerized), to prevent Python package conflicts. More on this below.

Logging to systemd

Once we have the dependencies installed, logging to systemd is fairly straightforward. I think the best way to show this is with a simple snippet:

The comments above should help explain, but from a higher level we just need to get an instance of a logger (I pass in the module name, but this could’ve been anything). The important part here is to wire up systemd’s Journal by adding a handler to our logger that is of type JournaldLogHandler . I personally like to include the level in my logging messages, so I define a custom formatter for my journald handler.

Virtual environment

Because I rely heavily on virtual environments, I need to make sure systemd sources my virtual environment activation prior to running my Python script as a daemon. This allows me to reap the benefits of a virtual environment (dependency isolation), even while running my code as a daemon.

My usual approach to this is to just create an executable wrapper shell script:

I put this shell script in the same directory as my Python module.

Setting up the systemd unit file and service

Before we can test this out, we need to create a unit file so that systemd knows how to setup and handle our daemon.

Keep this unit file in whatever long term directory you like, typically with the Python source for this application. systemd works heavily off of convention regarding what unit files exist where. Without going into too much depth about systemd (there could be books written, and it is out of the scope of this blog post), we need to make sure our unit file is symlink’d and accessible in /etc/systemd/system :

The next step is to reload systemd so that our new unit file is picked up.

Verify that the unit file is registered by attempting to get the status of our service.

You should see that the service is ‘loaded’ but ‘inactive’. Let’s start our new daemon.

Viewing the log entries with journalctl

Now that our systemd daemon is up and running, let’s take a look at the entries that are (hopefully) waiting for us in the journal!

With the “-u” parameter we are able to filter based on a unit (in our case, we only want to see log entries from our daemon). “-b” tells journalctl to only give us entries since the last boot.

Hopefully your output looks similar to mine!

How to use journalctl to read linux system logs

Summary

I hope this post has illustrated how to easily log to systemd’s Journal from Python! If you’re interested in diving a little more into systemd and unit files, I highly recommend you check out this entry in the Arch wiki. Enjoy!

This post describes how to use the Python bindings of systemd in order to follow and parse log entries of a specific service in real time through systemd’s journal. This is an alternative way to tail log files which can be very easily customized.

Following and parsing log files in real time is one of the most common practices when it comes to service monitoring for misuse. While doing maintenance of my systems during the last days, I experimented with journalctl, part of systemd, and realized that, in case of services that send their log entries to the syslog, it might be a much better idea to parse the log entries as they pass through systemd’s journal instead of tailing the actual log file.

Using the tail and journalctl commands

Log files can be followed with the tail command:

The equivalent command using journalctl is:

As you can see, the only real difference when journalctl is used is that, instead of parsing the actual log file, we use one or more special journal fields to isolate the syslog entries of the service we are interested in monitoring.

Advantages of using journalctl

Using journalctl instead of the tail command has some advantages which facilitate the whole process:

  1. You do not have to worry about log file rotation. Log file rotation is standard practice in Linux systems and, when parsing the actual log file directly, you have to take it into consideration, otherwise your monitoring script may end up tailing the rotated log file. This is not an issue when tailing log entries through systemd’s journal, because at this stage the recorded log entries have not been sent to the actual log files yet.
  2. It is easier to filter log entries. Log entries in the systemd journal are groups of environment variables. Each variable resembles a field which can be used for the isolation of the log entries we are interested in. Read more about the special journal fields you can use.

Invoking journalctl directly from the script

One way of following log entries in the systemd journal from within your Python scripts is to invoke journalctl directly using the subprocess module.

While doing research on the web about this, I found a very useful code snippet on a question at StackOverflow which shows how to use a polling object to follow the output of the tail command in a way that does not block the execution of the script. I had never used select.poll() before, but the code snippet was all I needed in order to understand its usefulness and usage. The following is a modified version of that snippet that polls journalctl for new entries:

Using the Python bindings of systemd

Although calling external commands is quite efficient, it is not my method of choice while writing my scripts. I usually run such scripts using an unprivileged user, so, if the script calls external commands, extra sudo configuration is required. Most of the time this is not a problem, but I tend to use Python bindings, if they are available, as I find this method more convenient.

So, in this case, instead of invoking journalctl as an external command, it’s a lot more convenient to use the Python bindings of systemd in order to read the systemd journal entries. If you run your script using an unprivileged user, all you need to do in order to gain read access to the systemd’s journal is to make the user a member of the systemd-journal system group.

The following source code snippet makes use of the systemd.journal module:

This code is an enhanced version of the polling for journal events example of the official documentation of the systemd Python bindings. It uses the same polling method found in the previous code snippet (posted above) in order to follow log entries as they pass through the systemd journal.

Here are a few important notes:

  1. After moving the cursor to the end of the journal with j.seek_tail() , we call j.get_previous() . In my tests, if this method is not called, older journal entries are returned together with the first entry that is returned by the polling object. I’m not really sure why this happens and the documentation did not help much. Any suggestions or ideas are appreciated.
  2. We check the value of j.process() so that only new log entries are retrieved ( j.process() == journal.APPEND ).

Conclusion

Parsing log files is a very common practice in system administration. Polling systemd’s journal for new log entries, either by calling the external journalctl command from within your scripts or by using a systemd.journal.Reader instance, seems to be a more flexible method compared to using the tail command. However, do note that the techniques described in this post apply to systems that use systemd and only to services that record their log messages to the syslog. In case your service logs directly to its own log file, using the tail command is the only available option.

About George Notaras

George Notaras is the editor of the G-Loaded Journal, a technical blog about Free and Open-Source Software. George, among other things, is an enthusiast self-taught GNU/Linux system administrator. He has created this web site to share the IT knowledge and experience he has gained over the years with other people. George primarily uses CentOS and Fedora. He has also developed some open-source software projects in his spare time.