Understanding the Apache access log: how to view, locate, and analyze

Log files are invaluable tools for developers and system administrators when it comes to debugging issues within web applications. They often serve as the primary source of information when troubleshooting website malfunctions.

Among these logs, the Apache HTTP server’s access log stands out as a key resource for debugging applications and gaining insights into visitor activity.

Read on to understand the details of Apache access logs, including what they record, where to find them, and how to interpret the data they contain. Because the real power of log data lies in the insights gained through analysis, this blog will explore the advantages of using a SaaS Log Analytics platform, such as Sumo Logic, to gain actionable insights from access log data.

What are Apache access logs and their role in web server monitoring?

As mentioned above, the Apache access log file is one of several log files an Apache web server produces. This particular log file records data for all requests processed by the Apache server. So, if an individual visits a webpage on your site, the access log file will contain details regarding this event.

This information is valuable in a variety of situations:

• If a common request is failing for each individual trying to get to a particular web page, the link may be pointing to a page that no longer exists.

• If a certain page on the site is taking longer than it should to load, the log entry might highlight SQL queries that need optimization to enhance server performance.

• If one particular page on the site is very popular, aggregating data from access logs could shine a light on commonly requested resources, helping businesses optimize content.

Where can I find Apache access logs?

The location of the Apache access logs depends on the system running the Apache HTTP server. As most Apache server instances run on Linux distributions, let’s focus on locating Apache access logs on a Linux machine.

On the Ubuntu Linux distribution, for example, access log records are written to the following location by default:

/var/log/apache2/access.log

The default location may vary slightly on other Linux distributions, but you won’t have to look very far in most cases. Ultimately, the log location and format (more on this later) of the access logs are defined by a CustomLog directive, which can be viewed and modified within your Apache HTTP server configuration.

Interpreting Apache access logs

Now that you know what Apache access logs are and where they can be found, you’re ready to interpret the entries so your development team and other IT personnel can make the most of them.

To understand log entries, you need to know the Apache log format. The format for access logs is defined in the CustomLog directive. Below are two popular log formats often used with Apache access logs.

Common Log Format

The Common Log Format is a standardized text file format used by various web servers to generate server log files. With an Apache HTTP server, the Common Log Format can be used to produce access logs that are straightforward enough for developers and administrators to read. In addition, as it is a standardized format used by multiple web servers, CLF-formatted log files can be easily used by many log analysis platforms.

A sample access log record written in the Common Log Format will look like this:

127.0.0.1 - Scott [10/Dec/2019:13:55:36 -0700] "GET /server-status HTTP/1.1" 200 2326

The fields represent:

• 127.0.0.1: IP address of the client that made the request

• Hyphen: The identity of the client. This field is often returned as a hyphen, and Apache’s HTTP server documentation recommends that this particular field not be relied upon except in the case of a controlled internal network.

• Scott: User ID of the requestor

• [10/Dec/2019:13:55:36 -0700]: Date and time of the request

• “GET /server-status HTTP/1.1": Request type and resource being requested

• 200: HTTP status code

• 2326: Size of the object returned to the client.

Combined Log Format

Another format often used with Apache access logs is the Combined Log Format. This format is similar to the Common Log Format but contains a few extra fields to provide more information for log analysis and debugging operations. An access log record that is recorded in the Combined Log Format looks like:

127.0.0.1 - Scott [10/Dec/2019:13:55:36 -0700] "GET /server-status HTTP/1.1" 200 2326 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"

As you can see, the first seven fields are identical to those in Common Log Format. The remaining fields represent two additional properties:

• "http://localhost/": This is the HTTP referrer, which shows the source URL.

• "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36": This is the User Agent, which reveals the client’s browser details.

The CustomLog directive

Earlier, I mentioned that Apache access logs are configured via the CustomLog directive within an Apache configuration file. Let’s take a look at a sample access log configuration to show the flexibility provided by the CustomLog directive:

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

CustomLog /var/log/apache2/access.log combined

This setup defines the combined log format using the LogFormat directive, followed by specifying the access log’s location and format through the CustomLog directive. As you can see, modifying the location or format of the access log is a straightforward process. Additionally, the CustomLog directive offers several other capabilities.

Multiple access logs

To configure multiple access logs for your Apache HTTP server, simply add additional CustomLog directives to add an extra, customized access log file:

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{User-agent}i" agent
CustomLog /var/log/apache2/access.log combined
CustomLog /var/log/apache2/agent_access.log agent

Conditional logs

In addition, access logs can be written conditionally. This could be useful for various reasons, such as excluding records associated with particular clients. Typically, this is done by setting environment variables and referencing them via the “env” clause. Visit the official documentation on the CustomLog directive for more information.

Log rotation and piped logs

Since server logs consume lots of storage, managing archived log files is essential. On a relatively busy Apache server, log files, especially access logs can grow quickly, so it’s important to have established processes for regularly moving or deleting old log files. Luckily, an Apache HTTP server can do this through the use of graceful restarts and piped log processes.

A graceful restart allows an Apache server to switch to a new access log file without losing client connections. This restart lets Apache open and write to new log files without client interruption, allowing the compression or deletion of old log files to save space.

Piped log processes, on the other hand, can allow for log rotation to be performed without a server restart. For example, a program called rotatelogs is included with Apache HTTP server. Rather than simply writing to a file, access log entries can be written through a pipe to this particular program. The rotatelogs program includes options to rotate logs conditionally based on time or size.

Analyzing Apache access logs with Sumo Logic

Collecting massive amounts of log data is only useful if the data can be managed and analyzed effectively. When done properly, it produces valuable insights that can be leveraged to improve web server or application performance.

Integrating with Sumo Logic simplifies Apache log file collection and makes generating valuable visualizations effortless when working with Apache access logs.

Getting started is easy. By configuring a Sumo Logic collector and local file source for the Apache access log, you can be up and running in minutes.

Start enhancing your log management and data analytics processes. Try Sumo Logic today with our 30-day free trial.