Python regular expressions — practice with a server log: filtering errors, counting login attempts and converting it to CSV (and the IP that knocked three times a second)

BySergio Alejandro Medina Ospina
PracticeFrom scratchFP125 min read

The lab technician handed us the log of one morning on the virtual campus with a question: “is anyone trying to get in by guessing passwords?”. A failed login leaves a request to /campus/login in the log with the code 401 (unauthorised), so the question becomes “which IPs have 401s?”. With what we learnt in the theory, it can be answered in a single line:

import re

LOG = """\
10.0.0.7 [12/10/2026 09:15:02] "GET /campus/inicio" 200 5120 "Firefox/131"
10.0.0.23 [12/10/2026 09:15:40] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:15:51] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:16:03] "POST /campus/login" 200 1024 "Chrome/129"
172.16.4.150 [12/10/2026 09:20:17] "GET /fp1/tema8.pdf" 200 14040 "Safari/17"
10.0.0.7 [12/10/2026 09:22:45] "GET /fp1/tema9.pdf" 404 0 "Firefox/131"
192.168.1.66 [12/10/2026 09:31:08] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:09] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:10] "POST /campus/login" 401 312 "curl/8.5"
10.0.0.41 [12/10/2026 10:02:33] "GET /fp1/entregas" 500 87 "Chrome/129"
-- server restart --
10.0.0.41 [12/10/2026 10:05:12] "GET /fp1/entregas" 200 2210 "Chrome/129"
"""

# Which IPs have failed logins (code 401)? One single line:
print(re.findall(r'(\S+) .*" 401 ', LOG))

When you run it:

['10.0.0.23', '10.0.0.23', '192.168.1.66', '192.168.1.66', '192.168.1.66']

The pattern (\S+) .*" 401 reads like this: a piece without spaces at the start (the IP, which is kept in the group), a space, anything, and then a quote, a space, 401 and another space. Since the pattern has one group, findall returns only the IP of each matching line. And since the dot does not match newlines, .* cannot jump from one line to the next: each match stays inside its own line.

The answer had a catch. 10.0.0.23 fails twice in 11 seconds and gets in on the third try: someone who mistyped. 192.168.1.66, on the other hand, fails three times in three consecutive seconds, and its browser is called curl: a program, not a person. But the list does not say that: the rest of the line does. To see it you have to cut each line into its fields, and that is what we are going to do.

In this practice we work with the full log in three programs: filtering the errors while checking that each line has the right format, counting requests per code, per hour and per IP (to find our curl), and converting the log to CSV with re.sub, reordering the date and anonymising the IPs. All three are intermediate level and build on the regular expressions theory; we will also use dictionaries to count, so if you need to, review associative containers.

How to use this article

Each program is complete in a single file. Copy it, save it with the name given and run it from the terminal with python3 name.py. Under each one you have the real output, a drawing or a table with what happens inside, and an explanation piece by piece. Before looking at the output, try to predict which lines of the log will match each pattern: that is exactly what you need to train with regular expressions.

From the theory, make sure you are clear on:

  • The difference between match, search and fullmatch, and that all three return None if they find nothing.
  • How groups are numbered (by their opening parenthesis) and what m.groups() returns.
  • Why .* is greedy and what [^"]* does.
  • That in the replacement of re.sub, \1, \2… are the text of each group.

All three programs use the same log, twelve lines from one morning on the virtual campus. Each line has this shape: IP, [date time], “METHOD path” in quotes, HTTP code, bytes sent and “browser” in quotes. Line 11 is not a request: it is a mark the server left when it restarted.

LOG = """\
10.0.0.7 [12/10/2026 09:15:02] "GET /campus/inicio" 200 5120 "Firefox/131"
10.0.0.23 [12/10/2026 09:15:40] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:15:51] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:16:03] "POST /campus/login" 200 1024 "Chrome/129"
172.16.4.150 [12/10/2026 09:20:17] "GET /fp1/tema8.pdf" 200 14040 "Safari/17"
10.0.0.7 [12/10/2026 09:22:45] "GET /fp1/tema9.pdf" 404 0 "Firefox/131"
192.168.1.66 [12/10/2026 09:31:08] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:09] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:10] "POST /campus/login" 401 312 "curl/8.5"
10.0.0.41 [12/10/2026 10:02:33] "GET /fp1/entregas" 500 87 "Chrome/129"
-- server restart --
10.0.0.41 [12/10/2026 10:05:12] "GET /fp1/entregas" 200 2210 "Chrome/129"
"""

No program uses input() or reads files. The log is written inside the code, between triple quotes, so that your output is identical to the one you see here. Reading it from a real file is exactly what we will learn in the next topic. The backslash after the opening quotes ("""\) stops the text from starting with an empty line.

Program 1 · Filtering the errors in the log

filtrar.py

The idea

We want a table with the requests that returned an error (code 400 or higher) and, on top of that, the list of lines that do not have the format of a request. One pattern solves both: if a line matches it whole, it is a request and its groups give us each field; if it does not match, it is an odd line.

Write the pattern piece by piece, copying a log line and replacing each field with its piece of regex. Whatever is not a field (brackets, quotes, spaces) stays as it is, escaped if needed.

The code

import re

LOG = """\
10.0.0.7 [12/10/2026 09:15:02] "GET /campus/inicio" 200 5120 "Firefox/131"
10.0.0.23 [12/10/2026 09:15:40] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:15:51] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:16:03] "POST /campus/login" 200 1024 "Chrome/129"
172.16.4.150 [12/10/2026 09:20:17] "GET /fp1/tema8.pdf" 200 14040 "Safari/17"
10.0.0.7 [12/10/2026 09:22:45] "GET /fp1/tema9.pdf" 404 0 "Firefox/131"
192.168.1.66 [12/10/2026 09:31:08] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:09] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:10] "POST /campus/login" 401 312 "curl/8.5"
10.0.0.41 [12/10/2026 10:02:33] "GET /fp1/entregas" 500 87 "Chrome/129"
-- server restart --
10.0.0.41 [12/10/2026 10:05:12] "GET /fp1/entregas" 200 2210 "Chrome/129"
"""

# A good line:  IP [date time] "METHOD path" code bytes "browser"
PATTERN = (r'(\d{1,3}(?:\.\d{1,3}){3}) '               # 1: IP
           r'\[(\d\d/\d\d/\d{4}) (\d\d:\d\d:\d\d)\] '  # 2: date, 3: time
           r'"(GET|POST) (\S+)" '                      # 4: method, 5: path
           r'(\d{3}) (\d+) '                           # 6: code, 7: bytes
           r'"([^"]*)"')                               # 8: browser

errors = 0
bad = []
print(f"{'Time':<10}{'IP':<14}{'Code':>6}  Request")
for n, line in enumerate(LOG.splitlines(), start=1):
    m = re.fullmatch(PATTERN, line)         # fullmatch: the WHOLE line has to match
    if m is None:
        bad.append(n)
        continue
    ip, date, time, method, path, code, size, browser = m.groups()
    if int(code) >= 400:                    # 4xx: client failure; 5xx: server failure
        errors += 1
        print(f"{time:<10}{ip:<14}{code:>6}  {method} {path}")

print(f"Errors: {errors}. Lines that are not requests: {bad}")

When you run it:

Time      IP              Code  Request
09:15:40  10.0.0.23        401  POST /campus/login
09:15:51  10.0.0.23        401  POST /campus/login
09:22:45  10.0.0.7         404  GET /fp1/tema9.pdf
09:31:08  192.168.1.66     401  POST /campus/login
09:31:09  192.168.1.66     401  POST /campus/login
09:31:10  192.168.1.66     401  POST /campus/login
10:02:33  10.0.0.41        500  GET /fp1/entregas
Errors: 7. Lines that are not requests: [11]

The pattern: eight groups, one per field

# A good line:  IP [date time] "METHOD path" code bytes "browser"
PATTERN = (r'(\d{1,3}(?:\.\d{1,3}){3}) '               # 1: IP
           r'\[(\d\d/\d\d/\d{4}) (\d\d:\d\d:\d\d)\] '  # 2: date, 3: time
           r'"(GET|POST) (\S+)" '                      # 4: method, 5: path
           r'(\d{3}) (\d+) '                           # 6: code, 7: bytes
           r'"([^"]*)"')                               # 8: browser

The first thing that stands out is that the pattern is split across five lines. Python automatically joins strings that appear next to each other inside parentheses: r'abc' r'def' is the same as r'abcdef'. That way each piece fits on one line and carries a comment with the number of its groups. Each piece, with its r in front.

The 8 groups of a log lineThe line 10.0.0.23 [12/10/2026 09:15:40] "POST /campus/login" 401 312 "Chrome/129" split into pieces: eight of them are groups numbered 1 to 8 (IP, date, time, method, path, code, bytes and browser) and the rest are brackets, quotes and spaces. Below, the table with each piece of the pattern, its variable and its value.The 8 groups of a log line10.0.0.23␣[12/10/2026␣09:15:40]␣"123POST␣/campus/login"␣401␣312␣"Chrome/129"45678no.piece of the patternvariablevalue1(\d{1,3}(?:\.\d{1,3}){3})ip10.0.0.232(\d\d/\d\d/\d{4})date12/10/20263(\d\d:\d\d:\d\d)time09:15:404(GET|POST)methodPOST5(\S+)path/campus/login6(\d{3})code4017(\d+)size3128([^"]*)browserChrome/129
Whatever is not in parentheses (brackets, quotes, spaces) must be there, but it is not kept.

Piece by piece:

  • IP, group 1: (\d{1,3}(?:\.\d{1,3}){3}). 1 to 3 digits, and then three times “dot and 1 to 3 digits”. The repetition goes in a non-capturing group (?:…), because we only want to repeat, not keep; a normal group would shift the numbering of all the others and would only keep its last round anyway.
  • Date and time, groups 2 and 3: \[(\d\d/\d\d/\d{4}) (\d\d:\d\d:\d\d)\]. The brackets in the log are real brackets, so they are escaped (\[, \]); otherwise the pattern would take them as the start of a set.
  • Method and path, groups 4 and 5: "(GET|POST) (\S+)". The alternative goes inside a group so that the | only affects GET and POST. The path is “one or more characters that are not spaces”.
  • Code and bytes, groups 6 and 7: (\d{3}) (\d+). The code always has 3 digits; the bytes, however many (including 0).
  • Browser, group 8: "([^"]*)". Here you can see why the theory insisted on [^"]*: “anything that is not a quote” cannot overshoot the closing quote.

Writing the path as "(GET|POST) (.*)" instead of (\S+). With fullmatch it would work on this log, because the rest of the pattern forces it to step back, but with search on a longer line the .* would take pieces that are not the path. Describe each field by what it can contain.

The loop: fullmatch validates and cuts at the same time

errors = 0
bad = []
print(f"{'Time':<10}{'IP':<14}{'Code':>6}  Request")
for n, line in enumerate(LOG.splitlines(), start=1):
    m = re.fullmatch(PATTERN, line)         # fullmatch: the WHOLE line has to match
    if m is None:
        bad.append(n)
        continue
    ip, date, time, method, path, code, size, browser = m.groups()
    if int(code) >= 400:                    # 4xx: client failure; 5xx: server failure
        errors += 1
        print(f"{time:<10}{ip:<14}{code:>6}  {method} {path}")
From the log to the error table (p1)Flowchart of p1: each of the 12 log lines goes through re.fullmatch. If it gives None, its number is stored in bad (only line 11). If it matches, m.groups() spreads the 8 groups into variables and, if the code is 400 or more, the line is printed: 7 of the 11.From the log to the error table (p1)LOG: 12 lines of textfor n, line in enumerate(LOG.splitlines(), 1)m = re.fullmatch(PATTERN, line)m is NoneMatchbad.append(n)line 11: the restartm.groups() → 8 variablesip, date, time, method…int(code) >= 400 ?yes: a table row7 linesno: next line4 linesErrors: 7. Lines that are not requests: [11]
fullmatch does two jobs at once: it checks that the line has the format and, with the groups, it cuts it up.
  • enumerate(LOG.splitlines(), start=1) goes through the lines numbered from 1, so the number matches the one you would see in an editor.
  • **re.fullmatch** is the key decision. With search, a line with rubbish in front or behind would match all the same; with fullmatch, the whole line has to have the shape of the pattern. Line 11, -- server restart --, does not start with an IP, so m is None.
  • if m is None stores the number of the bad line and **continue** jumps straight to the next one. This protects the line after it: m.groups() on None would give the AttributeError from the theory.
  • m.groups() returns a tuple of 8 strings, which are spread over 8 variables in one go. If the number of variables did not match the number of groups, Python would raise a ValueError when unpacking: a good check that you counted the parentheses right.
  • int(code) >= 400 converts the code before comparing. Groups are strings, and comparing "401" >= 400 would raise a TypeError.

Let’s check the output: of the 12 lines, 11 are requests. Those with code 400 or higher are five 401s, one 404 and one 500: 5 + 1 + 1 = 7 errors, the 7 rows of the table. The other 4 requests are 200.

The variables you won’t use (date, size, browser) can still be collected in the unpacking. It is clearer than reaching for m.group(3) or m.group(5) by number, and the day you need them they already have a name.

Try it yourself

  • Add a line to the log with the PUT method. Does it appear in the table or in the list of bad lines? Change the pattern to accept GET, POST, PUT and DELETE.
  • Swap fullmatch for search and add a line with extra text at the end (for example, # check). What changes in the output? Why is it not detected?

Program 2 · Counting requests and spotting login attempts

contar.py

The idea

Now we want numbers: how many requests there are for each code, how many in each hour (with a bar of # to see it at a glance), how many bytes the server has sent and how many failed logins each IP has. If an IP reaches 3, we flag it as suspicious.

To count, a dictionary with get(key, 0) + 1. To pull several fields out of a line, a pattern with groups only on the fields you care about; the rest can be described with .* if there is no risk of it overshooting.

The code

import re

LOG = """\
10.0.0.7 [12/10/2026 09:15:02] "GET /campus/inicio" 200 5120 "Firefox/131"
10.0.0.23 [12/10/2026 09:15:40] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:15:51] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:16:03] "POST /campus/login" 200 1024 "Chrome/129"
172.16.4.150 [12/10/2026 09:20:17] "GET /fp1/tema8.pdf" 200 14040 "Safari/17"
10.0.0.7 [12/10/2026 09:22:45] "GET /fp1/tema9.pdf" 404 0 "Firefox/131"
192.168.1.66 [12/10/2026 09:31:08] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:09] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:10] "POST /campus/login" 401 312 "curl/8.5"
10.0.0.41 [12/10/2026 10:02:33] "GET /fp1/entregas" 500 87 "Chrome/129"
-- server restart --
10.0.0.41 [12/10/2026 10:05:12] "GET /fp1/entregas" 200 2210 "Chrome/129"
"""

codes = re.findall(r'" (\d{3}) ', LOG)          # all the codes at once
per_code = {}
for c in codes:
    per_code = per_code.get(c, 0) + 1
print("Requests per code:", dict(sorted(per_code.items())))

per_hour = {}
failed_logins = {}
total_bytes = 0
for line in LOG.splitlines():
    m = re.match(r'(\S+) \[\d\d/\d\d/\d{4} (\d\d):.*" (\d{3}) (\d+) ', line)
    if not m:
        continue                                # the restart line is not a request
    ip, time, code, size = m.groups()
    per_hour[time] = per_hour.get(time, 0) + 1
    total_bytes += int(size)
    if code == "401":
        failed_logins[ip] = failed_logins.get(ip, 0) + 1

for time, n in sorted(per_hour.items()):
    print(f"{time}:00-{time}:59  {'#' * n} {n}")
print(f"Bytes served: {total_bytes} ({total_bytes / 1024:.1f} KB)")
for ip, n in failed_logins.items():
    warning = "  <- suspicious" if n >= 3 else ""
    print(f"{ip:<14}{n} failed logins{warning}")

When you run it:

Requests per code: {'200': 4, '401': 5, '404': 1, '500': 1}
09:00-09:59  ######### 9
10:00-10:59  ## 2
Bytes served: 24041 (23.5 KB)
10.0.0.23     2 failed logins
192.168.1.66  3 failed logins  <- suspicious

All the codes at once with findall

codes = re.findall(r'" (\d{3}) ', LOG)          # all the codes at once
per_code = {}
for c in codes:
    per_code = per_code.get(c, 0) + 1
print("Requests per code:", dict(sorted(per_code.items())))
  • re.findall(r'" (\d{3}) ', LOG) is applied to the whole log, not line by line. It looks for “closing quote, space, three digits, space”, which in this format only appears right before the HTTP code. Since there is a group, the list only has the digits: 11 codes, one per request.
  • The context is what makes the pattern reliable. A bare \d{3} would also find pieces of the IP, the date or the bytes; with the quote in front, only the code.
  • dict(sorted(per_code.items())) sorts the dictionary by key so the output always comes out in the same order: 4 of 200, 5 of 401, 1 of 404 and 1 of 500. They add up to 4 + 5 + 1 + 1 = 11.

A pattern with only the groups you need

for line in LOG.splitlines():
    m = re.match(r'(\S+) \[\d\d/\d\d/\d{4} (\d\d):.*" (\d{3}) (\d+) ', line)
    if not m:
        continue                                  # the restart line is not a request
    ip, time, code, size = m.groups()
    per_hour[time] = per_hour.get(time, 0) + 1
    total_bytes += int(size)
    if code == "401":
        failed_logins[ip] = failed_logins.get(ip, 0) + 1

The pattern in this part is shorter than in program 1, because we only need four fields:

PieceGroupWhat it captures
(\S+)1The IP: the start of the line, up to the first space
\[\d\d/\d\d/\d{4} The date, which is checked but not kept
(\d\d):2Only the hour (no minutes or seconds)
.*Anything: minutes, seconds, method and path
" (\d{3}) (\d+) 3 and 4The code and the bytes, after the quote
  • **re.match is used because the IP is at the start** of the line: here it is exactly what we want.
  • The .* is greedy: it swallows the line to the end and steps back until it finds " ddd ddd . In this format there is only one place where that fits, so the result is right. If the browser could contain something like " 200 5 , the greedy one would go to that last occurrence: that is why the hint said to use it only “if there is no risk of it overshooting”.
  • The restart line does not match, m is None and continue skips it.
  • Each counter is a different dictionary: per_hour counts every request in each hour and failed_logins only the ones with code "401". Note that it is compared with the string "401", because code is a group and has not been converted.

The report

for time, n in sorted(per_hour.items()):
    print(f"{time}:00-{time}:59  {'#' * n} {n}")
print(f"Bytes served: {total_bytes} ({total_bytes / 1024:.1f} KB)")
for ip, n in failed_logins.items():
    warning = "  <- suspicious" if n >= 3 else ""
    print(f"{ip:<14}{n} failed logins{warning}")
  • '#' * n repeats the character as many times as there are requests: 9 between 9:00 and 9:59, and 2 between 10:00 and 10:59. A one-line bar chart.
  • The bytes add up to 24041. Check it against the bytes column of the log: 5120 + 312 + 312 + 1024 + 14040 + 0 + 312 + 312 + 312 + 87 + 2210 = 24041. Divided by 1024 they give 23.48, which .1f writes as 23.5.
  • 10.0.0.23 has 2 failures and 192.168.1.66 has 3, the minimum for the warning. The dictionary remembers the order in which each IP was added, which is why 10.0.0.23 comes first.

A real detector would also take time into account: three failures in three seconds are not the same as three failures in a week. The times of our three requests (09:31:08, 09:31:09 and 09:31:10) are already in the log; working out the difference between them is a good “Try it yourself”.

Try it yourself

  • Add a counter of requests per path (/campus/login, /fp1/entregas…). You will have to add a group to the pattern; watch out that the numbers of the groups you already use do not change… or use m.groups() and unpack into one more variable.
  • Make the suspicious IP warning also appear if the IP has any 401 and its browser contains curl.

Program 3 · Converting the log to CSV with re.sub

reformatear.py

The idea

To open the log in a spreadsheet, we want to turn each request into a CSV line (fields separated by ;) in this order: date and time in ISO format (2026-10-12T09:15:02), the IP anonymised (with the last number set to 0, because a full IP is personal data), the method, the path and the code. Lines that are not requests are discarded and counted.

re.sub with a template can do all the work at once: a pattern that captures each piece you need, in the order it appears in the log, and a template that writes them in the order you want. And re.subn also tells you whether there was a replacement.

The code

import re

LOG = """\
10.0.0.7 [12/10/2026 09:15:02] "GET /campus/inicio" 200 5120 "Firefox/131"
10.0.0.23 [12/10/2026 09:15:40] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:15:51] "POST /campus/login" 401 312 "Chrome/129"
10.0.0.23 [12/10/2026 09:16:03] "POST /campus/login" 200 1024 "Chrome/129"
172.16.4.150 [12/10/2026 09:20:17] "GET /fp1/tema8.pdf" 200 14040 "Safari/17"
10.0.0.7 [12/10/2026 09:22:45] "GET /fp1/tema9.pdf" 404 0 "Firefox/131"
192.168.1.66 [12/10/2026 09:31:08] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:09] "POST /campus/login" 401 312 "curl/8.5"
192.168.1.66 [12/10/2026 09:31:10] "POST /campus/login" 401 312 "curl/8.5"
10.0.0.41 [12/10/2026 10:02:33] "GET /fp1/entregas" 500 87 "Chrome/129"
-- server restart --
10.0.0.41 [12/10/2026 10:05:12] "GET /fp1/entregas" 200 2210 "Chrome/129"
"""

PATTERN = (r'(\d+\.\d+\.\d+)\.\d+ '             # 1: IP without its last number
           r'\[(\d\d)/(\d\d)/(\d{4}) (\S+)\] '  # 2: day, 3: month, 4: year, 5: time
           r'"(\w+) (\S+)" (\d{3}) .*')         # 6: method, 7: path, 8: code; .* the rest
TEMPLATE = r"\4-\3-\2T\5;\1.0;\6;\7;\8"         # the template decides the order

print("datetime;ip;method;path;code")
discarded = 0
for line in LOG.splitlines():
    new, n = re.subn(PATTERN, TEMPLATE, line)   # n is 1 if the line matched, 0 if not
    if n == 0:
        discarded += 1
    else:
        print(new)
print(f"# {discarded} line(s) discarded")

When you run it:

datetime;ip;method;path;code
2026-10-12T09:15:02;10.0.0.0;GET;/campus/inicio;200
2026-10-12T09:15:40;10.0.0.0;POST;/campus/login;401
2026-10-12T09:15:51;10.0.0.0;POST;/campus/login;401
2026-10-12T09:16:03;10.0.0.0;POST;/campus/login;200
2026-10-12T09:20:17;172.16.4.0;GET;/fp1/tema8.pdf;200
2026-10-12T09:22:45;10.0.0.0;GET;/fp1/tema9.pdf;404
2026-10-12T09:31:08;192.168.1.0;POST;/campus/login;401
2026-10-12T09:31:09;192.168.1.0;POST;/campus/login;401
2026-10-12T09:31:10;192.168.1.0;POST;/campus/login;401
2026-10-12T10:02:33;10.0.0.0;GET;/fp1/entregas;500
2026-10-12T10:05:12;10.0.0.0;GET;/fp1/entregas;200
# 1 line(s) discarded

The pattern and the template

PATTERN = (r'(\d+\.\d+\.\d+)\.\d+ '             # 1: IP without its last number
           r'\[(\d\d)/(\d\d)/(\d{4}) (\S+)\] '  # 2: day, 3: month, 4: year, 5: time
           r'"(\w+) (\S+)" (\d{3}) .*')         # 6: method, 7: path, 8: code; .* the rest
TEMPLATE = r"\4-\3-\2T\5;\1.0;\6;\7;\8"         # the template decides the order

This pattern captures exactly what is going to come out and nothing else:

  • Group 1, (\d+\.\d+\.\d+), is the first three numbers of the IP with their dots. The fourth number is described outside the group (\.\d+): it has to be there, but it is not kept, and so it disappears from the result.
  • Groups 2, 3 and 4 are the day, the month and the year separately, because they have to be reordered.
  • Group 5 is the time, with (\S+): everything up to the next space, which comes after the closing bracket. The greedy one also swallows the ] and then gives it back, so that the pattern’s \] can match.
  • Groups 6, 7 and 8 are the method, the path and the code. This time the method is described with \w+, which accepts any word.
  • The **final .*** matches the rest of the line (bytes and browser). It is essential: re.sub only replaces what matches, and without it the bytes and the browser would stay stuck behind the CSV.

The template r"\4-\3-\2T\5;\1.0;\6;\7;\8" decides the output order:

The re.sub template reorders the groups (p3)At the top, the 8 groups of log line 2: 10.0.0, 12, 10, 2026, 09:15:40, POST, /campus/login and 401. Below, the template \4-\3-\2T\5;\1.0;\6;\7;\8, with an arrow from each group to the slot where it is copied. Result: 2026-10-12T09:15:40;10.0.0.0;POST;/campus/login;401.The re.sub template reorders the groups (p3)10.0.01210202609:15:40POST/campus/login40112345678\4–\3–\2T\5;\1.0;\6;\7;\8result2026-10-12T09:15:40;10.0.0.0;POST;/campus/login;401
Each \n in the template copies the text of group n; the rest (hyphens, the T, the semicolons) is written as it is. The template decides the order.

Let’s check it with line 2 of the log, the first 401. The groups are 10.0.0, 12, 10, 2026, 09:15:40, POST, /campus/login and 401. The template writes group 4 (2026), a hyphen, group 3 (10), a hyphen, group 2 (12), a T, group 5 (09:15:40), a ;, group 1 (10.0.0) followed by .0;, and then groups 6, 7 and 8 separated by ;. Result: 2026-10-12T09:15:40;10.0.0.0;POST;/campus/login;401, which is the second line of the output.

Writing the template without the r in front: "\4-\3-\2T…". Python turns \4, \3… into control characters before they reach re, and the output fills up with invisible characters without any error. sub templates, just like patterns, always with r.

The loop: subn to know whether it matched

print("datetime;ip;method;path;code")
discarded = 0
for line in LOG.splitlines():
    new, n = re.subn(PATTERN, TEMPLATE, line)   # n is 1 if the line matched, 0 if not
    if n == 0:
        discarded += 1
    else:
        print(new)
print(f"# {discarded} line(s) discarded")
  • re.subn returns the new line and the number of replacements. Since the pattern describes the whole line, it can only match once: n is 1 if the line was a request and 0 if not.
  • When it does not match, sub returns the original line unchanged. That is why printing new is not enough: the restart line would come out as it is, in the middle of the CSV. n == 0 is the clean way to detect it.
  • Look at the header: datetime;ip;method;path;code, with no spaces. Column names like that cause fewer problems when the CSV is imported into other programs.

Once the date is in year-month-day format, the lines can be sorted as text and they end up in chronological order: "2026-10-12" comes before "2026-11-01" both alphabetically and on the calendar. With day/month/year it doesn’t work: "12/10" would go after "01/11". We will take advantage of this in the last exercise.

Try it yourself

  • Add the bytes as the last column of the CSV. You will have to turn part of the final .* into a group; which group number does it get?
  • Change the anonymisation so that it only keeps the first two numbers (10.0.0.23 → 10.0.0.0, 172.16.4.150 → 172.16.0.0).

Common mistakes in this practice

  • Not escaping the brackets of the log. [\d\d/…] is a set, not a bracket: the pattern changes its meaning completely. Literal brackets, \[ and \].
  • **Asking for groups() without checking for None.** Any line that does not match (like the restart one) gives an AttributeError. First if m is None: continue.
  • Counting the groups wrong. If you add a parenthesis, all the numbers after it change. Use (?:…) for parentheses that only group, and comment the number of each group.
  • Comparing groups with numbers. Groups are strings: int(code) >= 400, or code == "401" in quotes.
  • **Using search to validate.** A line with rubbish in front or behind matches all the same. For “the line has this format”, fullmatch.
  • **Forgetting that sub only replaces what matches.** If the pattern does not cover the whole line, whatever is left outside is copied as it is into the result.

Watch it step by step in Python Tutor

Python Tutor runs the code line by line and shows the value of each variable after every step. Choose Python 3 and paste this minimal version of program 3, with a single line:

import re

line = '192.168.1.66 [12/10/2026 09:31:08] "POST /campus/login" 401 312 "curl/8.5"'
pattern = r'(\S+) \[(\d\d)/(\d\d)/(\d{4}) (\S+)\]'
m = re.search(pattern, line)
groups = m.groups()
iso = re.sub(pattern, r"\4-\3-\2T\5 \1", line)
print(groups)
print(iso)

Expected output:

('192.168.1.66', '12', '10', '2026', '09:31:08')
2026-10-12T09:31:08 192.168.1.66 "POST /campus/login" 401 312 "curl/8.5"

What to watch, step by step:

  1. After line 5, m appears as an re.Match object. Python Tutor does not draw its groups inside: to see them you have to ask for them.
  2. On line 6, groups is drawn as a tuple of five strings: the IP, the day, the month, the year and the time. All in quotes, the numbers too.
  3. On line 7, iso is a new string. Compare it with line, which stays the same: re.sub changes nothing, it returns another text.
  4. Look at iso: the start of the line has changed (2026-10-12T09:31:08 192.168.1.66) but the rest, from the quote onwards, is the same. The pattern in this example **has no .* at the end**, so only the part that matched has been replaced, and what came after has been copied as it is. It is the last common mistake in the list, live.
  5. Add .* to the end of the pattern and run it again: now the rest of the line disappears from the result.

Summary

  • One pattern per line format: copy a real line and replace each field with its piece of regex. Brackets and other metacharacters from the log, escaped.
  • **fullmatch validates and cuts at the same time:** if it returns None, the line does not have the format; if not, m.groups() spreads the fields into variables.
  • Groups only where needed: (?:…) to repeat without capturing, and a comment with the number of each group.
  • **findall over the whole text** is the fastest way to pull one field out of every line, as long as the pattern has enough context not to confuse it with another.
  • Groups are strings: convert with int() before comparing or doing maths.
  • **re.sub with a template reorders fields;** re.subn also tells you whether the line matched. The template, with r in front.

Checklist: have you practised it?

  • Have you run the three programs and checked the outputs?
  • Could you explain why program 1 uses fullmatch and program 2 uses match?
  • Can you number the eight groups of program 1’s pattern by hand without looking at the drawing?
  • Do you know what happens to the restart line in each of the three programs?
  • Could you add a column to program 3’s CSV without the group numbers getting out of place?
  • Have you tried the “Try it yourself” sections without looking at the theory?

If you said yes to almost everything, you are ready for the regular expressions exercises, where I no longer give you the code: validating fields with fullmatch, finding DNIs (Spanish ID numbers) written any old way in an incident report, merging two logs with different date formats, and three snippets with very common bugs to hunt down. See you there!

Next step: exercisesPython regular expressions — exercises with validators, DNIs hidden in a report and two logs to merge (and the alert that said “All good”)

Did this article help you? Here is how you can support SergioLearns →

Data Science and Engineering student at ULPGC. I explain what I learn
at university the way I wish it had been explained to me: with its
doubts, its mistakes and the path to the solution.

Sigue aprendiendo

Leave a Reply

Your email address will not be published. Required fields are marked *