Log file analysis without enterprise tools

Nginx and Cloudflare inputs

Nginx logs are often easiest when you control the format. Cloudflare exports may use a different schema, so map fields before aggregating. Never identify a bot from user-agent alone for security decisions; combine it with IP/provider verification when the decision matters.

# Requests by user-agent family
awk -F'"' '{print $6}' access.log \
  | awk 'BEGIN{IGNORECASE=1} /Googlebot/{g++} /GPTBot|ClaudeBot|PerplexityBot/{ai++} !/bot|crawler|spider/{human++} END{print "google",g; print "ai",ai; print "human",human}'

# Status distribution for crawler requests
awk -F'"' '/Googlebot|GPTBot|ClaudeBot|PerplexityBot/ { split($1,a," "); print a[9] }' access.log \
  | sort | uniq -c | sort -nr

A reproducible Python parser

Parse the request line and quoted user-agent with a real parser rather than splitting on spaces. The example below is intentionally conservative: malformed lines are counted, not silently dropped.

import re
from collections import Counter

line_re = re.compile(
    r'^(?P<ts>\S+) \S+ "(?P<request>[^" ]+) (?P<status>\d{3}) '
    r'(?P<bytes>\d+)" "(?P<ref>[^" ]*)" "(?P<ua>[^" ]*)"'
)

bots = ('Googlebot', 'Bingbot', 'GPTBot', 'ClaudeBot', 'PerplexityBot')
counts = Counter(); malformed = 0
with open('access.log', encoding='utf-8', errors='replace') as log:
    for line in log:
        match = line_re.match(line)
        if not match:
            malformed += 1; continue
        row = match.groupdict()
        ua = row['ua']
        group = next((bot for bot in bots if bot.lower() in ua.lower()), 'human/other')
        path = row['request'].split('?', 1)[0]
        counts[(group, row['status'], path)] += 1

for key, total in counts.most_common(30):
    print(total, *key)
print('malformed=', malformed)

Turn requests into crawl decisions

Build a URL-level report with hits, unique days, status classes, bytes, and bot groups. Prioritise URLs that receive repeated 4xx/5xx hits, redirected chains, or large HTML responses while important canonicals receive no crawler requests. Compare deployments, not isolated hours.

# Top non-200 bot URLs
awk -F'"' '/Googlebot|GPTBot|ClaudeBot|PerplexityBot/ { split($1,a," "); if (a[9] !~ /^2/) print a[9], $2 }' access.log \
  | sort | uniq -c | sort -nr | head -50

# A simple decision table
status >= 500  -> investigate origin reliability first
status == 404  -> remove links or restore the canonical resource
status 301/302 -> collapse chains and update internal links
bytes unusually high -> inspect HTML, compression, and cache headers

For Cloudflare, export the same fields into CSV and run the Python parser over the mapped columns. The goal is not a dashboard; it is a defensible list of crawl waste and missed important URLs.