Regex Essentials: Overview

• Regular expressions (regex) are a language for defining text search patterns.
• Python’s re module provides functions like search (find anywhere) and match (anchored at start).
• Patterns include literals, metacharacters (. ^ $ * + ? [] \), character classes (\d, \w, \s), and quantifiers (*, +, ?, {n,m}).
• Greedy quantifiers (*, +) match as much as possible; non-greedy (*?, +?) as little as possible.

Introduction to re.search() vs re.match()

• re.search(pattern, text) scans the entire string for the first occurrence.
• re.match(pattern, text) checks only at the beginning of the string.
• re.findall() and re.finditer() let you retrieve every occurrence of a pattern.
• Always use raw strings (r"...") to define regex patterns, avoiding Python string escapes interfering with regex.

import re

line = "WARN: Disk usage at 91%"
pattern = r"WARN"

print(f"search '{pattern}':", bool(re.search(pattern, line)))
print(f"match '{pattern}':", bool(re.match(pattern, line)))

Common Metacharacters

• . matches any character (except newline).
• ^ anchors at start of string.
• $ anchors at end of string.
• [] defines a set or range of characters, e.g. [A-Z].
• \ escapes metacharacters or introduces special sequences.

import re

test = "Error code: E1234. cxge"

print(f"Dot matches any character: {re.findall(r"c..e", test)}")
print(f"Start anchor (finds): {re.findall(r"^Error", test)}")
print(f"Start anchor (does not find): {re.findall(r"^E1234", test)}")
print(f"End anchor: {re.findall(r"cxge$", test)}")
print(f"Character set: {re.findall(r"[E0-9]+", test)}")

Special Sequences

• \d digit (0–9), \D non-digit.
• \w word character (letters, digits, underscore), \W non-word.
• \s whitespace, \S non-whitespace.
• \b word boundary (zero-width match).

import re

text = "The cat scattered 1024 catalogues."

print(f"Digits: {re.findall(r"\d+", text)}")
print(f"Word characters: {re.findall(r"\w+", text)}")
print(f"Whitespace: {re.findall(r"\s+", text)}")
print(f"Word boundary: {re.findall(r"\bcat\b", text)}")

Quantifier Cheat-Sheet

import re

text = "aaaa"

print(re.findall(r"a?", text))
print(re.findall(r"a*", text))
print(re.findall(r"a+", text))
print(re.findall(r"a{2}", text))
print(re.findall(r"a{1,3}", text))

print(f"Non-greedy a*: {re.findall(r"a*?", text)}")
print(f"Non-greedy a+: {re.findall(r"a+?", text)}")
print(f"Non-greedy a{{1,3}}?: {re.findall(r"a{1,3}?", text)}")

Quantifiers & Greedy vs Non-Greedy

• * / + / {n,} are greedy: match as much as possible.
• Append ? (*? / +? / {n,}?) to make them non-greedy: match as little as possible.
• Greedy quantifiers match the longest possible string that satisfies the pattern. Adding a ? after them makes them non-greedy (or lazy), matching the shortest possible string.

import re

html = "<p>One</p><p>Two</p><></>"

print(f"Greedy: {re.findall(r"<.*>", html)}")
print(f"Non-greedy: {re.findall(r"<.*?>", html)}")

Capturing Groups and Back-References

• Regex lets you check for patterns, but often you need to extract pieces of the match (e.g., IP vs port).
• Capturing groups, defined with (), let you isolate and retrieve substrings from a match.
• Named groups improve readability by giving meaningful labels instead of relying on group numbers.
• Non-capturing groups (?:…) let you apply grouping logic without cluttering captures.
• Back-references allow you to match the same text twice (or more) within one pattern.

Capturing Groups

• Parentheses () both group and capture the matched text inside them.
• Groups are numbered by their opening (, starting at 1; group 0 is the entire match.
• Use match.group(n) for a single group or match.groups() to get all captures as a tuple.
• Capturing is essential when you need to feed specific substrings into further processing.

import re

log_entry = "Ts=2023-10-27T12:00:00Z Level=ERROR User=admin Action=login_fail IP=10.0.0.5"

# Our goal:
# 1. Group 1: The log level
# 2. Group 2: The user name
# 2. Group 3: The IP address

pattern = r"Level=(\w+)\s+User=(\w+).*?\s+IP=([\d\.]+)"

match = re.search(pattern, log_entry)

if match:
    print(f"Full match: {match.group(0)}")
    print(f"Level: {match.group(1)}")
    print(f"User: {match.group(2)}")
    print(f"IP: {match.group(3)}")
    print(f"All groups: {match.groups()}")

Named Capturing Groups

• Syntax: (?P<name>pattern) assigns a label to a capturing group.
• Access by name: match.group('name') makes code self-documenting.
• match.groupdict() returns a dict of all named captures.
• You can still use numeric indices if needed, but names help avoid off-by-one errors.

import re

log_entry = "Ts=2023-10-27T12:00:00Z Level=ERROR User=admin Action=login_fail IP=10.0.0.5"

# Add labels to:
# 1. Group 1: The log level
# 2. Group 2: The user name
# 2. Group 3: The IP address

pattern = r"Level=(?P<level>\w+)\s+User=(?P<user>\w+).*?\s+IP=(?P<ip>[\d\.]+)"

match = re.search(pattern, log_entry)

if match:
    print(f"Full match: {match.group(0)}")
    print(f"Level: {match.group("level")}")
    print(f"User: {match.group("user")}")
    print(f"IP: {match.group("ip")}")
    print(f"All groups: {match.groups()}")
    print(f"Group dictionary: {match.groupdict()}")

Non-Capturing Groups

• Use (?:pattern) when you need grouping for quantifiers or alternation without capturing.
• Keeps your capture numbers focused on what you actually want.
• Prevents unwanted None entries in match.groups() when using optional parts.

import re

log_line1 = "report.txt Status: OK"
log_line2 = "report Status: OK"

# Our goal:
# 1. Group 1: The stem of the filename, with .txt being an optional string
# 2. Group 2: The status code

pattern = r"^(.+?)(?:\.txt)?\s+Status:\s+(.+)$"

match_line1 = re.search(pattern, log_line1)
match_line2 = re.search(pattern, log_line2)

if match_line1: print(match_line1.groups())
if match_line2: print(match_line2.groups())

Back-references

• Refer back to a previous capture using \1, \2, … or (?P=name) for named groups.
• Useful for matching repeated words or balanced constructs (e.g., open/close tags).
• Can make patterns more complex but powerful for advanced text validation.

import re

text = "This this is a test test."
pattern_numbers = r"(?i)\b(\w+)\s+\1\b"
pattern_labels = r"(?i)\b(?P<word>\w+)\s+(?P=word)\b"

print(f"Doubled words: {re.findall(pattern_numbers, text)}")
print(f"Doubled words: {re.findall(pattern_labels, text)}")

html = "<p>Paragraph</p> <b>Bold</b>"
pattern_tags = r"<(\w+)>(.*?)</\1>"

print(f"Tags: {re.findall(pattern_tags, html)}")

Search, Split, and Substitute

• re.findall() and re.finditer() let you retrieve every occurrence of a pattern.
• re.split() handles complex delimiters beyond simple string splits.
• re.sub() performs powerful search-and-replace operations, including reuse of captured groups.

Finding All Matches

• re.findall(pattern, string) returns a list of all non-overlapping matches:
  • No groups → list of matched substrings.
  • With groups → list of tuples of captured substrings.
• re.finditer(pattern, string) returns an iterator of match objects, giving access to .group(), positions, named groups, etc., and is more memory-efficient for large inputs.

import re

text = "Errors found: 404, 500, 403, 500. User IDs: user123, admin99."
config = "timeout=60 retries=3 workers=5"

# Find all error codes:
print(f"Numbers found: {re.findall(r"\d+", text)}")

# findall with groups:
print(f"Key-value pairs: {re.findall(r"(\w+)=(\w+)", config)}")

# finditer
for match in re.finditer(r"(\w+)=(\w+)", config):
    print(f"Whole match: {match.group(0)}; key: {match.group(1)}; value: {match.group(2)} - at {match.start()}-{match.end()}")

Splitting Strings

• Use re.split(pattern, string) to break a string on a regex pattern, not just a fixed substring.
• Always use a raw string literal so backslashes reach the regex engine.
• Simple single-character delimiters: use a character class (never captured), e.g. r"\s*[,;]\s*".
• Complex delimiters (alternation or multi-character): group with non-capturing parentheses, e.g. r"\s*(?:foo|bar|baz)\s*", so they aren’t included in the result list.
• Including delimiters: wrap your delimiter in a capturing group, e.g. r"\s*([,;])\s*", to have the separators appear in the split output.
• Summary:
  • No parentheses or a non-capturing group → delimiters are removed.
  • Capturing group → delimiters appear in the split list.

import re

data = "item1 , item2; item3 ,item4 ;item5"

# 1. Split on comma and semi-colon
pattern1 = r"\s*[,;]\s*"
print(f"Character class split: {re.split(pattern1, data)}")

# 2. Capturing the delimiter
pattern2 = r"\s*([,;])\s*"
print(f"Capturing group split: {re.split(pattern2, data)}")

html = """
<p class='hello'>First paragraph.</p>
<b class='world'>Second paragraph.</b>
End.
"""

pattern3 = r"<.*?class='(?:hello|world)'.*?>|</[pb]>"
print(f"HTML non-capturing split: {re.split(pattern3, html)}")

Substituting Text

• re.sub(pattern, replacement, string, count=0) replaces all (or a limited number) of matches.
• count controls how many replacements to make (default 0 = all).
• Back-references (\1, \g<name>) let you reorder or reuse captured text in the replacement.

import re

text = "User IDs: user123, user456, user123457689. Contact admin789 for help."

# Basic substitution
redacted = re.sub(r"user\d+", "[REDACTED_USER]", text)
print(f"Result of redacting: {redacted}")

# Back-reference for reusing information
redacted_partially = re.sub(r"(u)ser\d+(\d{2})", r"\1[REDACTED_USER]\2", text)
print(f"Result of redacting: {redacted_partially}")

# Limited count of substitutions
redacted_only_two = re.sub(r"(u)ser\d+(\d{2})", r"\1[REDACTED_USER]\2", text, count=2)
print(f"Result of redacting: {redacted_only_two}")

# Named groups for substitution
date_text = "Start: 2023-10-27, End: 2024-01-15"
# Current format YYYY-MM-DD
# Target format DD/MM/YYYY

date_pattern_named = r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})"
replacement_format_named = r"\g<day>/\g<month>/\g<year>"
reformatted_date = re.sub(date_pattern_named, replacement_format_named, date_text)

print(f"Result of date transformation: {reformatted_date}")