Log Parsing: How to Parse Server and Application Logs

Logs are the event journals of servers, applications, and services: every request to a site, every error, every user action leaves a line behind. Log parsing is what you do to turn those lines into structured data — for traffic analytics, error hunting, security-incident investigation, and monitoring. Technically, logs are text files, and the general techniques for taking them apart are the same; but they have enough quirks of their own to deserve a dedicated article within our broader overview of document parsing.

How logs differ from ordinary text

The defining trait: within a single source, the line format is strictly stable. A web server writes every line to one template, which lets you parse millions of lines with a single regular expression. But there are complications too. Logs can be enormous — gigabytes a day — so you can't read them into memory whole, only stream them. They get rotated — old files are renamed and compressed to .gz, and a log file parser has to read the compressed archives. And they contain multi-line records — a stack trace, for instance, spans dozens of lines that all belong to a single event.

Standard web-log formats

The most commonly parsed logs are Nginx and Apache access logs in the "combined" format. One line looks like this:

192.168.1.10 - - [01/Jun/2026:13:55:36 -0500] "GET /product/101 HTTP/1.1" 200 4523 "https://example.com/" "Mozilla/5.0 ..."

In order: IP address, date and time, request method and path, response code, response size, referrer, and User-Agent. Because the format is fixed, it's convenient to take apart with a single regular expression using named groups.

Python

The base and most flexible approach is a regular expression plus streaming reads. Named groups make the result readable.

import re
import gzip

LOG_RE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<path>\S+) [^"]*" '
    r'(?P<status>\d{3}) (?P<size>\S+) '
    r'"(?P<referer>[^"]*)" "(?P<agent>[^"]*)"'
)

def open_log(path):
    # transparently read both plain and compressed files
    return gzip.open(path, "rt") if path.endswith(".gz") else open(path, "r")

with open_log("access.log") as f:
    for line in f:
        m = LOG_RE.match(line)
        if not m:
            continue
        row = m.groupdict()
        if row["status"] != "200":
            print(row["status"], row["method"], row["path"], row["ip"])

When you need not just to parse but to analyze — count top pages, the distribution of response codes, traffic by hour — the parsed lines are handy to drop into pandas and work with as a table.

import pandas as pd

records = [m.groupdict() for line in open_log("access.log")
           if (m := LOG_RE.match(line))]
df = pd.DataFrame(records)

# top-10 most requested pages
print(df["path"].value_counts().head(10))

# share of 5xx errors
errors = df[df["status"].str.startswith("5")]
print(len(errors) / len(df))

From there the result is easy to export to CSV or Excel for a report.

Command line: fast parsing without code

For a one-off task it's often quickest to reach for Unix utilities. awk splits a line into fields and counts on the fly.

# top-10 IPs by request count
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# all requests that ended in a 404
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn

# total bytes served (field 10 is the size)
awk '{sum += $10} END {print sum/1024/1024 " MB"}' access.log

The grep + awk + sort + uniq combo answers most ad-hoc questions about logs and requires nothing to be written.

Ready-made analyzers

If the task is general web-traffic analytics rather than extracting specific fields, there's no need to invent a parser. GoAccess reads Nginx/Apache access logs and builds an interactive report in the terminal or the browser in real time. For complex pipelines with heterogeneous sources, teams use a stack built on Logstash with its grok pattern sets — essentially a library of ready-made named regular expressions for popular log formats.

Structured logs (JSON logs)

Modern applications increasingly write logs not as lines but as JSON — one object per line (the JSON Lines format). Such a log is far simpler and more reliable to parse: no brittle regular expressions needed, each line just deserializes.

import json

with open("app.log") as f:
    for line in f:
        event = json.loads(line)
        if event.get("level") == "ERROR":
            print(event["timestamp"], event["message"])

If you can influence how logs are written, switching to JSON Lines radically simplifies parsing them later — worth baking into a project from the start.

Where the difficulties come from

The most common problems are inconsistent time formats (logs from different systems use different time zones and date templates that must be normalized to one form), multi-line errors (a stack trace breaks line-by-line parsing, and records have to be "stitched" back together by the marker that starts a new record), and volume (analyzing a daily log measured in gigabytes calls for streaming processing and filtering on the fly, not loading it all into memory).

If you need to parse logs from several servers regularly, roll them up into unified analytics, or monitor errors and suspicious activity, scraping.pro can set up log collection and parsing to your formats and pipe the metrics into a system of your choice — delivered as an ongoing data-as-a-service feed. The general techniques for handling arbitrary text that this builds on are covered in the article on parsing TXT files.