Supprimer Rendre public Rendre privé Add tags Delete tags
  Ajouter un tag   Annuler
  Supprimer le tag   Annuler
  • • DevOps notes •
  •  
  • Tags
  • Connexion

Filesystem Paths/shaare/jvk1Ew

  • python
  • python

Working with Filesystem Paths in Python

  • Manipulating paths as plain strings is error-prone and OS-specific.
  • pathlib provides an object-oriented, cross-platform way to handle paths.
  • Path objects offer intuitive operators and methods for most filesystem tasks.

Limitations of String Paths and os.path

  • Using os.path.join, os.path.exists, etc., requires multiple function calls.
  • Code readability suffers when paths are manipulated as plain strings.
  • OS differences ("/" vs "\" separators) must be handled explicitly.

Creating and Combining Path Objects

  • Import Path from pathlib.
  • Create Path objects for directories and files.
  • Use the / operator to join path components cleanly.
from pathlib import Path

config_dir = Path(".")
filename = "settings.yaml"

print(config_dir, type(config_dir))

config_path = config_dir / filename
print(config_path.resolve())

Inspecting Path Properties

  • .exists(), .is_file(), .is_dir() check path state.
  • .parent, .name, .stem, .suffix expose components.
  • .resolve() returns the absolute, canonical path.
service_log = Path("/var/log/app/service.log")

print(f"Exists: {service_log.exists()}")
print(f"Is file? {service_log.is_file()}")
print(f"Is directory? {service_log.is_dir()}")
print(f"Parent: {service_log.parent}")
print(f"Name: {service_log.name}")
print(f"Stem: {service_log.stem}")
print(f"Suffix: {service_log.suffix}")
print(f"Resolved absolute path: {service_log.resolve()}")

Listing Directory Contents

  • .iterdir() yields immediate children of a directory.
  • .glob(pattern) finds entries matching a shell-style pattern.
  • Use "**/*.ext" in glob for recursive searches.
course_parent = Path("..")

print("Immediate children:")

for i, child in enumerate(course_parent.iterdir()):
    print(f"  {child.name} - {child.is_dir()}")
    if i >= 4: break

print("Python files recursively:")

for i, child in enumerate(course_parent.glob("**/*.ipynb")):
    print(f"  {child}")
    if i >= 10: break

Reading and Writing Files with Path

  • .write_text() and .read_text() handle simple text I/O.
  • Use p.open(mode="a") for more control (e.g., appending, binary mode).
  • Path methods automatically manage file open/close.
test_file = Path("demo.txt")

test_file.write_text("Hello, from pathlib!", encoding="utf-8")
print(f"Read back: {test_file.read_text(encoding="utf-8")}")

with test_file.open(mode="a", encoding="utf-8") as file:
    file.write("\nAppended line!")

print(f"Read back: {test_file.read_text(encoding="utf-8")}")

test_file.unlink()
2 months ago Permalien
cluster icon
  • Working with YAML files : Working with YAML files YAML (“YAML Ain’t Markup Language”) focuses on human readability. Indentation replaces braces and brackets, comments are allo...
  • Python Functions Are First‑Class Citizens : Python Functions Are First‑Class Citizens In Python, functions behave like any other object (strings, ints, lists). Because they are "first‑clas...
  • Python Modules and the import System : Python Modules and the import System What is a Module? A module in Python corresponds directly to a single file containing Python code. The module's ...
  • Fixtures in Pytest : Fixtures in Pytest As tests grow more complex, repeating setup and cleanup steps makes tests harder to read and maintain. Pytest fixtures allow centr...
  • Working with Environment Variables : Working with Environment Variables Environment variables are dynamic, named values provided by the operating system to running processes, enabling co...

Declarative Logging/shaare/gWRxQw

  • python
  • python

Declarative Logging Configuration

  • Declarative configuration separates setup from code, making it easier to maintain and adjust.
  • Python’s logging.config module supports both INI-style (fileConfig) and dictionary-based (dictConfig) configurations.
  • Configuration objects can be loaded from files (INI, JSON, YAML) or defined in code.
  • Benefits include environment-specific overrides, less boilerplate, and clearer visibility of logger/handler relationships.

INI-Style Configuration with fileConfig

  • Uses an INI-format file to define loggers, handlers, and formatters.
  • Sections: [loggers], [handlers], [formatters], plus one section per named logger/handler/formatter.
  • Good for simple setups and backwards compatibility, but less flexible for dynamic structures.

Dictionary-Based Configuration with dictConfig

  • Configuration defined as a Python dict, offering full programmatic control.
  • Keys: version, disable_existing_loggers, and mappings for formatters, handlers, loggers, and optionally root.
  • Easy to build or modify at runtime, and to serialize/deserialize via JSON/YAML.

Loading Configuration from JSON or YAML

  • Store the same dict-based schema in a JSON/YAML file for external editing.
  • Read and parse the file, then pass the resulting dict to dictConfig.
  • Enables separation of concerns: ops teams can tweak logging without touching code.

Dynamic and Programmatic Adjustments

  • You can modify the config dict at runtime before calling dictConfig.
  • Handlers, formatters, and levels can be added, removed, or tweaked based on environment variables or feature flags.
  • Example: switch file logging on/off depending on a DEBUG flag.
import logging
import logging.config
import json
from typing import Any, Dict

"""
# Uncomment to test INI configuration

# Declarative logging configuration - INI-file
print("Declarative configuration using INI files")
print("---------\n")

config_path = "declarative-config.ini"

logging.config.fileConfig(
    fname=config_path,
)

app_logger = logging.getLogger("app")
app_logger.debug("INI-style fileConfig is working!")
"""

"""
# Uncomment to test Dictionary configuration

# Declarative logging configuration - Dictionary config
print("Declarative configuration using dictionary config")
print("---------\n")

dict_config: Dict[str, Any] = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "simple": {"format": "%(levelname)-8s - %(message)s"}
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": "INFO",
            "formatter": "simple",
            "stream": "ext://sys.stdout",
        }
    },
    "loggers": {
        "config.dict": {
            "level": "DEBUG",
            "handlers": ["console"],
        }
    },
}

logging.config.dictConfig(dict_config)
config_logger = logging.getLogger("config.dict")
config_logger.debug("dictConfig setup successfully")
config_logger.info("Info goes to console")
"""

"""
# Uncomment to test JSON configuration

# Declarative logging configuration - JSON config
print("Declarative configuration using JSON config")
print("---------\n")

config_path = "declarative-config.json"

with open(config_path, "r") as config_file:
    json_config = json.load(config_file)

logging.config.dictConfig(json_config)
config_logger = logging.getLogger("config.json")
config_logger.debug("JSON config setup successfully")
config_logger.info("Info goes to console")
"""

# Dynamically building config
print("Dynamically building config")
print("---------\n")

base_config: Dict[str, Any] = {
    "version": 1,
    "disable_existing_loggers": True,
    "handlers": {},
    "formatters": {},
    "loggers": {},
}

base_config["formatters"]["simple"] = {
    "format": "%(levelname)-8s - %(message)s"
}

base_config["handlers"]["console"] = {
    "class": "logging.StreamHandler",
    "level": "DEBUG",
    "formatter": "simple",
    "stream": "ext://sys.stdout",
}

base_config["loggers"]["config.dynamic"] = {
    "level": "WARNING",
    "handlers": ["console"],
}

def is_debug():
    return True

if is_debug():
    for logger, _config in base_config["loggers"].items():
        base_config["loggers"][logger]["level"] = "DEBUG"

logging.config.dictConfig(base_config)
config_logger = logging.getLogger("config.dynamic")
config_logger.debug("Dynamic config setup successfully")
config_logger.info("Info goes to console")

Example of declarative-config.ini

[loggers]
keys=root,app

[handlers]
keys=consoleHandler,nullHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=INFO
handlers=consoleHandler

[logger_app]
level=DEBUG
handlers=nullHandler
qualname=app

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[handler_nullHandler]
class=NullHandler
level=NOTSET
formatter=
args=()

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)-8s - %(message)s

Example of declarative-config.json

{
  "version": 1,
  "disable_existing_loggers": false,
  "formatters": {
    "simple": { "format": "%(levelname)-8s - %(message)s" },
    "detailed": {
      "format": "%(asctime)s %(name)s [%(levelname)s]: %(message)s",
      "datefmt": "%Y-%m-%d %H:%M:%S"
    }
  },
  "handlers": {
    "console": {
      "class": "logging.StreamHandler",
      "level": "INFO",
      "formatter": "detailed",
      "stream": "ext://sys.stdout"
    }
  },
  "loggers": {
    "config.json": {
      "level": "DEBUG"
    }
  },
  "root": {
    "level": "DEBUG",
    "handlers": ["console"]
  }
}
2 months ago Permalien
cluster icon
  • Regex : Regex Essentials: Overview Regular expressions (regex) are a language for defining text search patterns. Python’s re module provides functions like...
  • Typing classes : Introduction As our Python automation projects grow, defining custom classes helps model complex objects and should be reflected in type hints for cl...
  • Numbers, strings : Numbers (int and float) int: Whole numbers (e.g., 10, 1024). No overflow due to arbitrary precision. float: Numbers with decimals (e.g., 3.14159). Us...
  • Making HTTP Requests : Making HTTP Requests The requests library simplifies HTTP interactions by abstracting raw HTTP details, making it ideal for DevOps automation tasks. ...
  • Filesystem Operations : Filesystem Operations (os & shutil) DevOps scripts often need to create, delete, copy, and move files and directories as part of automation workflows...

Structured Logging/shaare/3E69ww

  • python
  • python

Introduction to Structured Logging

  • Plain-text logs are hard to parse and brittle to format changes.
  • Structured logging records events as key-value data, making machine parsing trivial.
  • JSON is a de-facto standard: human-readable yet easily ingested by ELK, Splunk, DataDog, etc.
  • Python’s python-json-logger integrates JSON output into the standard logging workflow.

Configuring python-json-logger

  • Install via pip install python-json-logger==3.3.0 (for consistency, I'm pinning the version; removing it will install the latest version available).
  • Replace logging.Formatter with pythonjsonlogger.JsonFormatter.
  • Specify a format string listing the LogRecord attributes you want as JSON keys.
  • Attach to any Handler just like a normal Formatter.

Logging with Extra Context

  • Pass a dict to the extra parameter of logger.<level>().
  • Keys in extra become top-level JSON fields.
  • Use for request IDs, user IDs, session tokens, or any domain data.

Logging Exceptions as JSON

  • Use logger.exception(...) inside an except block.
  • The JsonFormatter automatically adds an exc_info key with the traceback.
  • This preserves full error context for downstream analysis.
# Configuring python-json-logger
print("Configuring python-json-logger")
print("---------\n")

import logging
import sys
from pythonjsonlogger.json import JsonFormatter

json_logger = logging.getLogger("demo.json")
json_logger.setLevel(logging.INFO)

handler = logging.StreamHandler(sys.stdout)
json_formatter = JsonFormatter(
    "{asctime}{levelname}{message}",
    style="{",
    json_indent=4,
    rename_fields={"asctime": "timestamp", "levelname": "level"},
)
handler.setFormatter(json_formatter)

json_logger.addHandler(handler)

json_logger.info("Structured logging initialized")

# Logging with extra context
print("Logging with extra context")
print("---------\n")

extra_context = {
    "user_id": "devops1",
    "request_id": "request-12345abc",
    "source_ip": "10.0.0.5",
}

json_logger.warning(
    "Request took longer than 5s to complete",
    extra=extra_context,
)

# Logging exceptions as JSON
print("Logging exceptions as JSON")
print("---------\n")

try:
    result = 1 / 0
except ZeroDivisionError:
    json_logger.exception(
        "Unexpected calculation error",
        extra={"operation": "division"},
    )
2 months ago Permalien
cluster icon
  • Regex : Regex Essentials: Overview Regular expressions (regex) are a language for defining text search patterns. Python’s re module provides functions like...
  • Configuring Pytest : Configuring Pytest As you start using Pytest extensively, typing -v or -m on the command line every time becomes tedious. Centralize your defaults in...
  • Mocking : Mocking Fundamentals Introduction When unit testing DevOps scripts that interact with external systems, tests can become slow, unreliable, difficult ...
  • Making HTTP Requests : Making HTTP Requests The requests library simplifies HTTP interactions by abstracting raw HTTP details, making it ideal for DevOps automation tasks. ...
  • Generators : Generators Writing a class-based iterator requires __iter__() and __next__(), plus manual state management and StopIteration handling. Generator fu...

Logging to Files/shaare/oXLUGw

  • python
  • python

Logging to Files

Basic File Logging with FileHandler

  • Use logging.FileHandler to write log records to a file.
  • mode='a' (append) preserves existing logs; mode='w' (write) overwrites on each run.
  • You can specify encoding (e.g., 'utf-8') and delay=True to open the file only on first write.

Size-Based Rotation with RotatingFileHandler

  • RotatingFileHandler rotates when the file reaches maxBytes.
  • backupCount determines how many old files to keep (.1, .2, …).
  • New rotations rename existing backups, deleting the oldest beyond backupCount.

Time-Based Rotation with TimedRotatingFileHandler

  • TimedRotatingFileHandler rotates based on elapsed time (when, interval).
  • Common when values (case insensitive): 'S', 'M', 'H', 'D', 'midnight', 'W0'-'W6'
    • 'S' – Rotate every N seconds (as given by interval), useful for very short-lived scripts or testing.
    • 'M' – Rotate every N minutes, good for high-volume services where hourly isn’t fine-grained enough.
    • 'H' – Rotate every N hours, often used for long-running daemons that batch logs hourly.
    • 'D' – Rotate every N days, for simple daily log files without tying to midnight.
    • 'midnight' – Rotate once per day exactly at midnight (local time), regardless of interval, ideal for calendar-aligned logs.
    • 'W0'–'W6' – Rotate weekly on a specific weekday, where W0 = Monday through W6 = Sunday. Use interval weeks between rotations.
  • backupCount limits number of rotated files; use .suffix to customize timestamp format.
import logging
import logging.handlers
import os
import time

def cleanup_log_files(base_name: str):
    for file_name in os.listdir("."):
        if file_name.startswith(base_name):
            os.remove(file_name)

# Basic logging with FileHandler
print("Basic logging with FileHandler")
print("-------\n")

basic_logger = logging.getLogger("file.basic")
basic_logger.setLevel(logging.DEBUG)

basic_fh = logging.FileHandler(
    "basicfile.log", delay=True, encoding="utf-8"
)
basic_fh.setLevel(logging.INFO)

basic_logger.addHandler(basic_fh)

basic_logger.info("INFO: will be written to file")

# Size-based log rotation with RotatingFileHandler
print("Size-based log rotation with RotatingFileHandler")
print("-------\n")

rotating_logs_filename = "rotatingfile.log"

cleanup_log_files(rotating_logs_filename)

rotating_logger = logging.getLogger("file.rotating")
rotating_logger.setLevel(logging.DEBUG)

rotating_fh = logging.handlers.RotatingFileHandler(
    rotating_logs_filename,
    maxBytes=500,
    backupCount=2,
    encoding="utf-8",
)
rotating_fh.setFormatter(
    logging.Formatter("%(levelname)-8s %(message)s")
)

rotating_logger.addHandler(rotating_fh)

for i in range(30):
    rotating_logger.info(f"Entry {i}: {'Z' * 50}")
    time.sleep(0.05)

# Time-based log rotation with TimedRotatingFileHandler
print("Time-based log rotation with TimedRotatingFileHandler")
print("-------\n")

timed_rotating_logs_filename = "timedrotatingfile.log"

cleanup_log_files(timed_rotating_logs_filename)

timed_rotating_logger = logging.getLogger("file.timed")
timed_rotating_logger.setLevel(logging.DEBUG)

timed_rotating_fh = logging.handlers.TimedRotatingFileHandler(
    timed_rotating_logs_filename,
    when="s",
    interval=3,
    backupCount=2,
    encoding="utf-8",
)
timed_rotating_fh.setFormatter(
    logging.Formatter("%(levelname)-8s %(message)s")
)

timed_rotating_logger.addHandler(timed_rotating_fh)

for i in range(30):
    timed_rotating_logger.info(f"Entry {i}: {'Z' * 50}")
    time.sleep(0.5)
2 months ago Permalien
cluster icon
  • Generators : Generators Writing a class-based iterator requires __iter__() and __next__(), plus manual state management and StopIteration handling. Generator fu...
  • Generators and Lazy Pipelines : Generators and Lazy Pipelines You can chain generator functions to form multi-stage data pipelines that process items one at a time. No intermediat...
  • Filesystem Paths : Working with Filesystem Paths in Python Manipulating paths as plain strings is error-prone and OS-specific. pathlib provides an object-oriented, cr...
  • Structured Logging : Introduction to Structured Logging Plain-text logs are hard to parse and brittle to format changes. Structured logging records events as key-value da...
  • Declarative Logging : Declarative Logging Configuration Declarative configuration separates setup from code, making it easier to maintain and adjust. Python’s logging.conf...

Log Levels in Practice/shaare/sMZ4dg

  • python
  • python

Log Levels in Practice

  • Python defines five standard levels with increasing severity:
    • DEBUG (10): Detailed diagnostic information.
    • INFO (20): Confirmation that things are working normally.
    • WARNING (30): An indication of potential problems or deprecation.
    • ERROR (40): A failure in a specific operation.
    • CRITICAL (50): A serious error causing program termination.
  • NOTSET (0) causes a logger to inherit its parent’s effective level.
  • Appropriate use of these levels lets you adjust verbosity without changing code.

Two-Stage Filtering: Logger vs Handler

  • Logger Level: First gate: records below logger.level are discarded immediately.
  • Handler Level: Second gate: each handler only emits records at or above its handler.level.
  • This allows, for example, DEBUG messages to be logged to a file but only WARNING and above to the console.

Configuring Logger & Handlers

  • Use logger.setLevel(...) to control which messages the logger accepts.
  • Use handler.setLevel(...) to control which accepted messages each handler emits.
  • Attach multiple handlers for different ouputs (e.g., console vs file) with independent levels.

# Log levels in practice

import logging
import sys

print("Log levels in practice")
print("------\n")

for lvl in (
    logging.DEBUG,
    logging.INFO,
    logging.WARNING,
    logging.ERROR,
    logging.CRITICAL,
):
    print(
        f"{logging.getLevelName(lvl):8} = {lvl}"
    )

# Two-stage filtering

print("\n")
print("Two stage filtering")
print("------\n")

filter_logger = logging.getLogger("demo.filter")
filter_logger.setLevel(logging.INFO)

stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setLevel(logging.ERROR)

filter_logger.addHandler(stream_handler)

filter_logger.info("INFO: will not be shown")
filter_logger.error("ERROR: will be shown")

# Configuring logs and handlers

print("\n")
print("Configuring logs and handlers")
print("------\n")

data_logger = logging.getLogger("demo.data")
data_logger.setLevel(logging.DEBUG)

data_sh = logging.StreamHandler(sys.stdout)
data_sh.setLevel(logging.ERROR)

data_fh = logging.FileHandler("process.log", "w")
data_fh.setLevel(logging.INFO)

data_logger.addHandler(data_sh)
data_logger.addHandler(data_fh)

data_logger.debug("DEBUG: will be dropped")
data_logger.info("INFO: file only")
data_logger.warning("WARNING: file only")
data_logger.error("ERROR: file and console")
data_logger.critical("CRITICAL: file and console")
2 months ago Permalien
cluster icon
  • Generators : Generators Writing a class-based iterator requires __iter__() and __next__(), plus manual state management and StopIteration handling. Generator fu...
  • Typing classes : Introduction As our Python automation projects grow, defining custom classes helps model complex objects and should be reflected in type hints for cl...
  • Handling Subprocess Errors : Handling Subprocess Errors External commands can fail in multiple ways: non-zero exit codes, missing executables, or hanging processes. Using subpr...
  • Generators and Lazy Pipelines : Generators and Lazy Pipelines You can chain generator functions to form multi-stage data pipelines that process items one at a time. No intermediat...
  • Parametrized Tests : Parametrized Tests Introduction Often, we need to test the same logic with different inputs and outputs, such as validating various IP address or hos...

Logging Anatomy/shaare/2wicQA

  • python
  • python

Python Logging Anatomy

  • Python’s logging module has five core components: Loggers, Log Records, Handlers, Formatters and Filters.
  • Loggers are hierarchical objects your code calls to emit messages at various severity levels.
  • Each call to a logger creates a LogRecord capturing metadata: level, message, timestamp, source, thread/process IDs, exception info, etc.
  • Handlers attached to loggers dispatch records to destinations (console, files, network).
  • Formatters define how a LogRecord is rendered into the final string emitted by a handler.
import logging

root_logger = logging.getLogger()
print(f"Root logger: name={root_logger.name}, level={logging.getLevelName(root_logger.level)}")

app_logger = logging.getLogger("app")
print(f"App logger: name={app_logger.name}, level={logging.getLevelName(app_logger.level)}, parent={app_logger.parent.name}")

network_logger = logging.getLogger("app.network")
print(f"Network logger: name={network_logger.name}, level={logging.getLevelName(network_logger.level)}, parent={network_logger.parent.name}")

Log Records

  • Each logging call (logger.info(), logger.error(), etc.) creates a LogRecord object behind the scenes.
  • A LogRecord includes attributes such as name, levelno, levelname, pathname, lineno, funcName, asctime, message, plus any user-supplied extra data.
  • Handlers and formatters use these attributes to filter and render the log entry.
from logging import LogRecord

record = LogRecord(
    name="app.network",
    level=logging.ERROR,
    pathname="/path/to/file.py",
    lineno=43,
    msg="My log message",
    args=(),
    exc_info=None
)

print("LogRecord contents:")

for attr in ("name", "levelname", "pathname", "msg"):
    print(f"    {attr} => {getattr(record, attr)}")

Handlers

  • Handlers determine where log records are sent (console, file, network, etc.).
  • Each handler has its own level: it filters out any record whose level is below its threshold.
  • Common handlers include:
    • StreamHandler (console),
    • FileHandler (single file),
    • RotatingFileHandler,
    • TimedRotatingFileHandler,
    • SysLogHandler,
    • HTTPHandler,
    • NullHandler.
import sys

demo_logger = logging.getLogger("handler_demo")
demo_logger.setLevel(logging.INFO)
demo_logger.handlers.clear()

stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setLevel(logging.DEBUG)
demo_logger.addHandler(stream_handler)

demo_logger.debug("Debug message: will not show")
demo_logger.info("Info message: will show")
demo_logger.warning("Warning message: will show")
demo_logger.error("Error message: will show")

Formatters

  • Formatters specify the layout of the final log message string.
  • You define a format string using %(attribute)s or %(attribute)d placeholders.
  • Common attributes: asctime, levelname, name, message, filename, lineno, funcName, process, thread.
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
)

stream_handler.setFormatter(formatter)

demo_logger.warning("Formatted warning")
2 months ago Permalien
cluster icon
  • Making HTTP Requests : Making HTTP Requests The requests library simplifies HTTP interactions by abstracting raw HTTP details, making it ideal for DevOps automation tasks. ...
  • Working with CSV files : Working with CSV files CSV (Comma Separated Values) is a plain-text tabular format where each line is a row and fields are delimited (commonly by com...
  • Adding Tests to a Multi-File Project : Adding Tests to a Multi-File Project Standard Project Layout with Tests To maintain a clean and organized codebase, it is standard practice to separat...
  • Dictionaries : Dictionaries (dict) Dictionaries are mutable, insertion-ordered collections of key-value pairs. Keys must be unique and immutable; values can be of an...
  • Logging to Files : Logging to Files Basic File Logging with FileHandler Use logging.FileHandler to write log records to a file. mode='a' (append) preserves existing log...

Context managers/shaare/wo-q7g

  • python
  • python

Context Managers

  • When opening files or acquiring locks, resources must be released even if errors occur.
  • Manual try...finally ensures cleanup but adds boilerplate and potential for mistakes.
  • Forgetting to initialize the resource variable or to call cleanup in every exit path leads to leaks, deadlocks, or corrupted data.
  • Cleaner patterns reduce noise and risk in automation scripts.
f = None

try:
    f = open("my_log.txt", "w")
    f.write("First line\n")
    # Simulate an error
    result = 1 / 0
    f.write("Second line\n")
except:
    print("Error has occurred.")
finally:
    if f:
        print("Closing file.")
        f.close()

print(f"File closed: {f.closed}")

The with Statement Simplifies Cleanup

  • The with statement handles setup and teardown automatically for context managers.
  • For file I/O, with open(...) as f: guarantees f.close() on block exit, even if an exception is raised.
  • Syntax is concise and idiomatic, reducing boilerplate and improving readability.

Common Context Manager Examples

  • Files: with open(...) as f: for automatic file closing.
  • Locks: with threading.Lock(): acquires and releases locks safely.
  • Tempfiles/Dirs: with tempfile.TemporaryDirectory() as d: creates and cleans up temporary directories.
  • Context managers from the standard library cover most resource-management needs.
f = None

try:
    with open("my_log.txt", "w") as f:
        f.write("First line\n")
        # Simulate an error
        result = 1 / 0
        f.write("Second line\n")
except:
    print("Error has occurred.")

print(f"File closed: {f.closed}")
import tempfile, os

dir_name = None

with tempfile.TemporaryDirectory() as tempdir:
    print(f"Created temp dir: {tempdir}")

    dir_name = tempdir
    test_file = os.path.join(tempdir, "test.txt")

    with open(test_file, "w") as file:
        file.write("Hello from temp directory.")

    print(f"Files inside temp dir: {os.listdir(tempdir)}")

try:
    contents = os.listdir(dir_name)
    print(f"Contents of {dir_name}: {contents}")
except FileNotFoundError as e:
    print(f"Expected error accessing removed directory: {e}")

Custom Resource Management: Writing Context Managers

  • Whenever you need custom setup/teardown logic, you can write your own Context Manager.
  • A context manager ensures that teardown always runs, even if errors occur in the block.
  • Two approaches: implement __enter__/__exit__ in a class or use the simpler generator-based decorator.
class MyContextManager:
    def __init__(self, timeout):
        self.timeout = timeout

    def __enter__(self):
        print("Setup complete")
        return "a simple value"

    def __exit__(self, exception_type, exception_value, traceback):
        print(f"Teardown")

        # Commenting out since we replaced *args for explicit
        # exception_type, exception_value, traceback parameters

        # for arg in args:
        #     print(arg)

        return False

with MyContextManager(timeout=30) as cm:
    print(cm)
    print("Inside the block")
    raise ValueError("Simulated problem")

The @contextlib.contextmanager Decorator

  • Provided by the contextlib module to turn a generator into a context manager.
  • Decorated function needs exactly one yield.
  • Code before yield runs as __enter__; code after (or in finally) runs as __exit__.
  • Simplifies many common patterns without writing a full class.

Generator Structure for @contextmanager

  • Wrap the yield in try...finally to ensure teardown even on errors.
  • The value yielded is bound to as var in the with statement (if used).
  • You can catch exceptions inside the generator if you want to suppress them.
import os
from contextlib import contextmanager

@contextmanager
def change_directory(destination):
    """
    Temporarily switch into destination. If the directory does not exist,
    it is created just before the switch.

    Args:
        destination (str): Path to the directory that should become the working directory
    """

    origin_dir = os.getcwd()

    try:
        print(f"Changing into {destination}")
        os.makedirs(destination, exist_ok=True)
        os.chdir(destination)
        yield os.getcwd()
    finally:
        print(f"Reverting to original dir: {origin_dir}")
        os.chdir(origin_dir)

print(f"Start: {os.getcwd()}")

with change_directory("temp_dir") as new_dir:
    print(f"Inside: {new_dir}")

print(f"End: {os.getcwd()}")
2 months ago Permalien
cluster icon
  • For & While Loops : For & While Loops Python provides two main ways to repeat actions: for loops (for iterating over known sequences) and while loops (for repeating as lo...
  • Exceptions : Common Built‑in Exceptions Python ships with a rich hierarchy of exception classes; most automation errors fall into a small, predictable subset. A...
  • Variables, comments : Variables: Naming Values Naming Guidelines: Must start with a letter or underscore (_) and can contain letters, numbers, and underscores. Use snake_...
  • Logging to Files : Logging to Files Basic File Logging with FileHandler Use logging.FileHandler to write log records to a file. mode='a' (append) preserves existing log...
  • Parametrized Tests : Parametrized Tests Introduction Often, we need to test the same logic with different inputs and outputs, such as validating various IP address or hos...

Custom Exceptions: Tailoring Error Signals/shaare/hTTdng

  • python
  • python

Custom Exceptions: Tailoring Error Signals

  • Built-in exceptions are great, but often too generic for application-specific failures.
  • A custom exception like ServiceConnectionError immediately conveys context compared to a plain Exception.
  • Defining a base exception class groups related errors; subclasses add specificity for targeted handling.
  • Catching except BaseError: handles all related issues, while except SpecificError: addresses one case precisely.

Simple Custom Exceptions (Inheritance)

  • Create a new exception by subclassing Exception or another exception class.
  • Using pass is enough when no extra logic or attributes are needed.
  • Catch the base class (AutomationError) to handle any related subclass errors in one block.
  • Use subclasses (FileProcessingError, APICallError) when context-specific handling is required.
class AutomationError(Exception):
    """Base for all automation script errors."""
    pass

class FileProcessingError(AutomationError):
    """Error during file processing stage."""
    pass

class APICallError(AutomationError):
    """Error during an external API call."""
    pass

def process_file(filepath):
    raise FileProcessingError(f"Failed to process file at path: {filepath}")

try:
    process_file("nonexistent.csv")
except FileProcessingError as e:
    print(f"File error: {e}")
except AutomationError:
    print("Other automation error occurred.")

Adding Context with __init__

  • Override __init__ in your exception class to capture context (e.g., filename, invalid value).
  • Store custom attributes on self and build a clear message passed to super().__init__().
  • Inherit from a built-in exception (ValueError) when semantics align, allowing broad catches.
  • Attribute access (e.key_name) provides extra debugging info in handlers.
class ConfigValueError(ValueError):
    """Raised when a config value is invalid."""
    def __init__(self, key_name, invalid_value, message="Invalid configuration value."):
        self.key_name = key_name
        self.invalid_value = invalid_value
        full_message = f"{message} for key '{key_name}': received '{invalid_value}'"
        super().__init__(full_message)

try:
    raise ConfigValueError("timeout", -5, message="Timeout cannot be negative")
except ConfigValueError as e:
    print(f"{e}")
    print(f"   -> key: {e.key_name}")
    print(f"   -> value: {e.invalid_value}")

Raising and Catching Enhanced Custom Exceptions

  • Raise custom exceptions by instantiating them with relevant arguments: raise MyError(arg1, arg2).
  • In except blocks, catch specific exceptions and access their attributes for tailored recovery or logging.
  • Fallback except BaseError: catches any related subclass if no more specific handler exists.
class DeploymentError(Exception):
    """Base class for deployment-related errors."""
    pass

class InvalidEnvironmentError(DeploymentError):
    """Raised when environment is invalid."""
    def __init__(self, env_name, allowed_envs):
        self.env_name = env_name
        self.allowed_envs = allowed_envs
        super().__init__(f"Invalid environment '{env_name}'. Allowed values: {allowed_envs}")

class PackageMissingError(DeploymentError):
    """Raised when required packages are missing."""
    def __init__(self, package_name, host):
        self.package_name = package_name
        self.host = host
        super().__init__(f"Package '{package_name}' is missing on host {host}.")

def deploy_app(environment, package):
    allowed_envs = ["staging", "production"]

    if environment not in allowed_envs:
        raise InvalidEnvironmentError(environment, allowed_envs)

    if environment == "production" and package == "critical-lib":
        raise PackageMissingError(package, f"server-{environment}")

    print(f"Deployment to {environment} with package {package} succeeded.")

for env, pkg in [("dev", "tool"), ("production", "critical-lib"), ("staging", "tool")]:
    try:
        deploy_app(env, pkg)
    except DeploymentError as e:
        print(e)
2 months ago Permalien
cluster icon
  • Pytest Markers : Pytest Markers Markers are decorators (@pytest.mark.) applied to tests to attach metadata. Built-in markers like skip, skipif, xfail, and parametrize...
  • Functions, Docstrings : Functions Functions package reusable code into named blocks, improving modularity, readability, and testability. They prevent duplication (DRY) and ma...
  • The Iteration Protocol : The Iteration Protocol We use for item in sequence: all the time. But how does Python get each item? Iterable: An object that can be looped over. It...
  • Variables, comments : Variables: Naming Values Naming Guidelines: Must start with a letter or underscore (_) and can contain letters, numbers, and underscores. Use snake_...
  • Classes and Objects : Classes and Objects Beyond Built-ins: Python lets you define your own data types using class. Class: A blueprint or template for creating objects. De...

Signaling Errors: The raise Statement/shaare/ul6CvA

  • python
  • python

Signaling Errors: The raise Statement

  • Functions sometimes encounter states they cannot handle and must signal failure clearly.
  • Using raise triggers an exception, integrates with try...except, and stops execution immediately.
  • Prefer exceptions over special return values (None, False) to avoid ambiguous error handling.
  • Raising early enforces preconditions and supports the "fail fast" principle.
def process_servers(server_list):
    if not isinstance(server_list, list):
        # return None - BAD Practice, better to raise TypeError Exception
        raise TypeError("Input 'server_list' must be of type list.")

    # GOOD practice - Handle edge cases without raising exception
    if len(server_list) == 0:
        print("There are no servers to process. Exiting...")
        return

    print(f"Processing {len(server_list)} servers.")

# process_servers("abc") # Uncommenting will raise TypeError
process_servers([])
process_servers(["web01", "web02"])

Raising Built-in Exceptions

  • Built-in exception classes (e.g., TypeError, ValueError, FileNotFoundError) convey standard error semantics.
  • Raise TypeError when the argument’s type is wrong; raise ValueError when its value is out of acceptable range.
  • Use exceptions like OSError, ConnectionError, etc., when the built-in meaning matches your context.
  • Always include a clear, informative message describing the failure.
def set_deployment_replicas(count):
    """Example: enforce input type and value boundaries with built-in Exceptions."""
    try:
        parsed_count = int(count)
    except (ValueError, TypeError):
        raise TypeError(f"Replica count must be int or convertible to int, got {type(count).__name__}")

    if parsed_count < 0 or parsed_count > 100:
        raise ValueError(f"Replica count must be between 0 and 100")

    print(f"Replicas set to {parsed_count}")

for val in [5, -2, "three", 150, "5", 5.0]:
    try:
        set_deployment_replicas(val)
    except (TypeError, ValueError) as e:
        print(f"Caught error: {e}.")
2 months ago Permalien
cluster icon
  • Dictionaries : Dictionaries (dict) Dictionaries are mutable, insertion-ordered collections of key-value pairs. Keys must be unique and immutable; values can be of an...
  • Functions, Docstrings : Functions Functions package reusable code into named blocks, improving modularity, readability, and testability. They prevent duplication (DRY) and ma...
  • Parametrized Tests : Parametrized Tests Introduction Often, we need to test the same logic with different inputs and outputs, such as validating various IP address or hos...
  • Variables, comments : Variables: Naming Values Naming Guidelines: Must start with a letter or underscore (_) and can contain letters, numbers, and underscores. Use snake_...
  • Editable Installs with pyproject.toml : Editable Installs with pyproject.toml The Python interpreter doesn't automatically know about our project's structure. The modern and most robust solu...

Exceptions/shaare/_Jauhg

  • python
  • python

Common Built‑in Exceptions

  • Python ships with a rich hierarchy of exception classes; most automation errors fall into a small, predictable subset.
  • All ordinary run‑time exceptions inherit from Exception, but subclasses convey why something failed (e.g., file missing vs. wrong type).
  • Catching overly broad bases like Exception hides root causes and can mask bugs—prefer the narrowest class you can handle.
  • Understanding the inheritance tree lets you decide when a single except can cover many related problems (e.g., OSError).
import inspect, builtins

def show_tree(base, level=0, max_depth=1):
    if level > max_depth:
        return

    for name, obj in vars(builtins).items():
        if inspect.isclass(obj) and issubclass(obj, base) and obj is not base:
            print("\t" * level + f"- {name}")
            show_tree(obj, level + 1, max_depth)

show_tree(Exception, max_depth=1)

OSError Family: Filesystem & Network Issues

  • Signals problems interacting with the operating system: files, permissions, sockets, paths.
  • Subclasses such as FileNotFoundError, PermissionError, IsADirectoryError, ConnectionRefusedError, and TimeoutError offer granularity.
  • Catch individual subclasses when you can recover differently (create a missing file, prompt for sudo, retry a connection).
  • A single except OSError still groups all OS‑level failures when the response is the same (e.g., log and abort).
try:
    with open('nonexistent.txt', 'r') as file:
        content = file.read()
except FileNotFoundError as e:
    print("File not found")
except PermissionError:
    print("Permission denied when accessing resources.")
except OSError as os_err:
    print(f"General OS error: {os_err}")

KeyError: Missing Dictionary Keys

  • Raised when using dict[key] with a key that is absent.
  • Frequent in config loading, JSON parsing, or environment variable maps.
  • Mitigation patterns: dict.get(key, default), membership tests (if key in cfg), or a tailored except KeyError.
  • Treats missing data distinctly from a wrong value (ValueError) or wrong type (TypeError).
config = {"host": "server1", "port": 8080}
config2 = {"host": "server1", "port": 8080, "api-key": "12345"}

api_key = config.get("api-key", "")
print(f"API Key: {api_key}")

def call_endpoint(config, endpoint):
    """Calls the specified endpoint of the configured host.

    Args:
        config (dict(str)): Dict containing host, port, and api-key
        endpoint (str): The endpoint to hit
    """
    if "api-key" in config:
        print(f"Making API call to endpoint {endpoint} with key: {config["api-key"]}")
    else:
        print("No api-key available, not possible to call API.")

def call_endpoint_exception(config, endpoint):
    """Calls the specified endpoint of the configured host.

    Args:
        config (dict(str)): Dict containing host, port, and api-key
        endpoint (str): The endpoint to hit
    """
    try:
        print(f"Making API call to endpoint {endpoint} with key: {config["api-key"]}")
    except KeyError as missing_key:
        print(f"No required key {missing_key} available, not possible to call API.")

call_endpoint(config, "/users")
call_endpoint(config2, "/users")

call_endpoint_exception(config, "/users")
call_endpoint_exception(config2, "/users")

IndexError: Sequence Index Out of Bounds

  • Triggered when list/tuple indices fall outside the valid range: negative beyond the left edge or ≥ len(seq).
  • Common during iterative processing of dynamic lists or user‑provided indexes.
  • Prevent with bounds checks (if i < len(seq)), safe iteration (for item in seq:), or catch and default.
  • Signals "wrong position" rather than "wrong content".
servers = ["web01", "web02"]

i = 2

if i < len(servers):
    print(servers[i])

try:
    print(servers[i])
except IndexError as e:
    print(f"Index error: {e}. List length is {len(servers)}")

ValueError vs. TypeError

  • ValueError: argument type is acceptable but content/value is invalid (e.g., int("abc")).
  • TypeError: operation applied to an object of the wrong type altogether (e.g., len(5) or "a" + 3).
  • Distinguishing them clarifies whether to validate content or convert types.
  • Catch them separately to craft precise user feedback.
try:
    port = int("http") # ValueError, since literal must be a valid base-10 value
except ValueError as e:
    print(f"Bad numeric string: {e}")

try:
    total = "Errors: " + 5
except TypeError as e:
    print(f"Type mismatch: {e}")

AttributeError: Missing Object Member

  • Raised when an attribute or method doesn't exist on the object referenced.
  • Often results from typos, unexpected None, or polymorphic functions returning different types.
  • Defensive techniques: hasattr(obj, "attr"), if obj is not None:, or narrow except AttributeError.
  • Conveys "object of this type doesn’t support that capability".
class Calculator:
    def add(self, a, b):
        return a + b

calc = Calculator()

if hasattr(calc, "subtract"):
    print(calc.subtract(10, 5))
else:
    print("Object has no attribute 'subtract'")

try:
    print(calc.subtract(10, 5))
except AttributeError as e:
    print(f"AttributeError caught: {e}.")

try:
    print(calc.result)
except AttributeError as e:
    print(f"AttributeError caught: {e}.")

ImportError / ModuleNotFoundError

  • Raised when an import statement cannot locate a module/package.
  • ModuleNotFoundError (Python 3.6+) is the specific subclass; catching ImportError also covers it.
  • Causes: misspelling, package not installed, wrong virtual environment, or PYTHONPATH issues.
  • Typical handling logs instructions and aborts early to avoid cascading failures.
try:
    import non_existent_lib
except ModuleNotFoundError as e:
    print(f"Import failed: {e}. Is the library installed and the correct venv active?")
2 months ago Permalien
cluster icon
  • Automated Testing with Pytest : Assertions in Pytest Pytest uses Python’s built-in assert statement to declare expected conditions in tests, making test code concise and readable. W...
  • Typing : Introduction Python is a dynamically typed language, meaning you can assign values to variables without declaring their types, and type checking happ...
  • Numbers, strings : Numbers (int and float) int: Whole numbers (e.g., 10, 1024). No overflow due to arbitrary precision. float: Numbers with decimals (e.g., 3.14159). Us...
  • Filesystem Operations : Filesystem Operations (os & shutil) DevOps scripts often need to create, delete, copy, and move files and directories as part of automation workflows...
  • Python Functions Are First‑Class Citizens : Python Functions Are First‑Class Citizens In Python, functions behave like any other object (strings, ints, lists). Because they are "first‑clas...

Enhancing Functions: Decorators/shaare/OPgX0g

  • python
  • python

Enhancing Functions: Decorators

  • A decorator is a callable that takes another function, adds behaviour before and/or after it runs, and returns a new callable.
  • They solve cross‑cutting concerns such as logging, timing, permission checks, or retries without cluttering core logic.
  • The magic @decorator_name syntax is shorthand for passing the target function to the decorator and re‑binding the original name to the returned wrapper.

Decorator Anatomy (Manual View)

  • Outer decorator function accepts the target function and creates a wrapper inside it.
  • The wrapper usually takes *args, **kwargs so it can handle any signature.
  • Wrapper executes optional "before" code, calls the original, maybe does "after" code, and returns the original’s result.
  • Returning the wrapper from the decorator completes the transformation.

Using decorators:

  • Manually wrapping illustrates what @ syntax really does behind the scenes.
  • This approach is clear but repetitive: @ eliminates the manual reassignment step.
import time

def simple_task(sleep_duration):
    time.sleep(sleep_duration)
    print("Running a simple task...")

def timing_decorator(original_function):
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = original_function(*args, **kwargs)
        duration = time.perf_counter() - start
        print(f"{original_function.__name__} took {duration:.3f}s")

        return result

    return wrapper

simple_task = timing_decorator(simple_task)
simple_task(0.3)

The @ Syntax

  • Placing @decorator_name directly above def my_func(): triggers my_func = decorator_name(my_func) at definition time.
  • After that line is executed, my_func refers to the wrapper returned by the decorator, so callers automatically get enhanced behaviour.
  • This keeps the decoration visible and close to the function definition, improving readability.
@timing_decorator
def another_task():
    print("Running another task...")

another_task()

Configurable Decorators: Decorators with Arguments

  • A basic decorator adds fixed behavior; sometimes you need to configure that behaviour (e.g. how many retries, which log level).
  • You cannot pass options directly to a plain @decorator, because that decorator receives only the target function.
  • Solution: call a factory that takes options and returns a decorator, then apply it with @factory(option=value).
def timing_decorator(original_function):
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = original_function(*args, **kwargs)
        duration = time.perf_counter() - start
        print(f"{original_function.__name__} took {duration:.3f}s")

        return result

    return wrapper

The Decorator Factory Pattern

  • Factory function receives configuration arguments and returns the actual decorator.
  • The actual decorator still takes the target function and builds a wrapper.
  • The wrapper can access both the factory’s configuration (via a closure) and the call‑time *args / **kwargs for the target function.
  • Three nested layers keep concerns separated: configuration ➜ decoration ➜ runtime.

Applying Decorators with Arguments

  • Use @factory(arg1, arg2…) above the function definition.
  • At definition time Python calls the factory, gets back a decorator, and applies that decorator to the function.
  • Callers of the function automatically get the behaviour configured by the factory.

Example: Retry Decorator Factory

  • A practical DevOps scenario: retry a flaky operation a configurable number of times.
  • The factory takes max_attempts; the wrapper loops until success or until attempts are exhausted, re‑raising the last error.
import random

def retry(max_attempts=3):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    print(f"Attempt {attempt}/{max_attempts}")
                    return func(*args, **kwargs)
                except Exception as e:
                    print(f" Error: {e}")
                    if attempt == max_attempts:
                        raise

        return wrapper
    return decorator

@retry(4)
def sometimes_fails():
    if random.random() < 0.7:
        raise RuntimeError("Flaky failure")
    return "Success!"

print(f"Result: {sometimes_fails()}")

Configurable Decorators: Decorators with Arguments

  • A basic decorator adds fixed behavior; sometimes you need to configure that behaviour (e.g. how many retries, which log level).
  • You cannot pass options directly to a plain @decorator, because that decorator receives only the target function.
  • Solution: call a factory that takes options and returns a decorator, then apply it with @factory(option=value).
def timing_decorator(original_function):
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = original_function(*args, **kwargs)
        duration = time.perf_counter() - start
        print(f"{original_function.__name__} took {duration:.3f}s")

        return result

    return wrapper

The Decorator Factory Pattern

  • Factory function receives configuration arguments and returns the actual decorator.
  • The actual decorator still takes the target function and builds a wrapper.
  • The wrapper can access both the factory’s configuration (via a closure) and the call‑time *args / **kwargs for the target function.
  • Three nested layers keep concerns separated: configuration ➜ decoration ➜ runtime.

Applying Decorators with Arguments

  • Use @factory(arg1, arg2…) above the function definition.
  • At definition time Python calls the factory, gets back a decorator, and applies that decorator to the function.
  • Callers of the function automatically get the behaviour configured by the factory.

Example: Retry Decorator Factory

  • A practical DevOps scenario: retry a flaky operation a configurable number of times.
  • The factory takes max_attempts; the wrapper loops until success or until attempts are exhausted, re‑raising the last error.
import random

def retry(max_attempts=3):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    print(f"Attempt {attempt}/{max_attempts}")
                    return func(*args, **kwargs)
                except Exception as e:
                    print(f" Error: {e}")
                    if attempt == max_attempts:
                        raise

        return wrapper
    return decorator

@retry(4)
def sometimes_fails():
    if random.random() < 0.7:
        raise RuntimeError("Flaky failure")
    return "Success!"

print(f"Result: {sometimes_fails()}")

Decorators & Return Values

  • A decorator’s wrapper replaces the original function, so if it forgets to return the original result the caller receives None.
  • Many real‑world functions produce critical data (e.g. status strings, dictionaries, numeric results); the decorator must be transparent about that value.
  • Fixing this means capturing the result of func(*args, **kwargs) inside the wrapper and returning it unchanged.
def log_calls_broken(func):
    def wrapper(*args, **kwargs):
        print(f"LOG: Calling {func.__name__}")
        func(*args, **kwargs)
        print(f"LOG: Finished {func.__name__}")
    return wrapper

@log_calls_broken
def add(x, y):
    return x + y

print(f"Result seen by caller: {add(2, 3)}")

The Wrapper’s Responsibility

  • The wrapper is the public face of the decorated function; it must faithfully:
    • Call the original with all arguments.
    • Capture its return value.
    • Perform any extra behaviour (log, time, validate).
    • Return the captured value so callers remain unaware of the wrapper.
  • Failure to return breaks contracts and causes subtle bugs.

Capturing return values

  • Capturing is a one‑liner: value = func(*args, **kwargs).
  • After post‑call logic, return value preserves behaviour.
  • You can also inspect or transform value before returning if the decorator’s purpose demands it.
def log_calls(func):
    def wrapper(*args, **kwargs):
        print(f"LOG: Calling {func.__name__}")
        value = func(*args, **kwargs)
        print(f"LOG: Finished {func.__name__}")
        return value
    return wrapper

@log_calls
def multiply(a, b):
    return a * b

print(f"Result seen by caller: {multiply(2, 3)}")

Handling Exceptions in Decorators

  • Wrappers often log exceptions for observability but should re‑raise them so callers can still handle or see errors.
  • Use try ... except ... raise around the call; log inside the except, then re‑raise without arguments to preserve traceback.
  • A decorator that swallows exceptions changes program semantics unless that is its explicit purpose (e.g. retry).
def log_and_reraise(func):
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as err:
            print(f"[ERROR] {func.__name__} raised {err.__class__.__name__}")
            raise
    return wrapper

@log_and_reraise
def fail():
    raise ValueError("simulated problem")

fail()

functools.wraps

  • A decorator replaces the original function object with its wrapper, so introspection tools see the wrapper’s metadata instead of the original’s.
  • Attributes such as __name__, __doc__, __module__, and type‑hint annotations are lost or altered.
  • This confuses debuggers, documentation generators, and anyone relying on help(), inspect, or error traces that reference the function name.
  • Python’s functools module supplies @wraps(original_func); apply it inside your decorator to the wrapper.
  • @wraps copies key metadata from the original function onto the wrapper, so the decorated function still looks like the original externally.
def broken_decorator(func):
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@broken_decorator
def add(a, b):
    """Return the sum of two numbers."""
    return a + b

print("Introspection without @wraps:")
print(f"  __name__: {add.__name__}")
print(f"  __doc__: {add.__doc__}")
from functools import wraps

def correct_decorator(func):
    @wraps(func) # Best practice: Always use it!
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@correct_decorator
def multiply(a, b):
    """Return the product of two numbers."""
    return a * b

print("Introspection with @wraps:")
print(f"  __name__: {multiply.__name__}")
print(f"  __doc__: {multiply.__doc__}")

Stacking Decorators: Applying Multiple Layers

  • Python lets you attach more than one decorator to a single function by writing multiple @decorator lines above the def.
  • Each decorator contributes a distinct slice of behaviour (logging, timing, caching, auth checks) keeping the core function clean.

Application vs. Execution Order

  • Decoration happens bottom‑up when the function is defined:
    1. Decorator nearest the def wraps the original first.
    2. Each line above wraps the result of the previous decoration.
  • Execution happens top‑down (outside‑in) when the decorated function is called: the outermost wrapper runs first, then calls the inner wrapper, and so on until the original function runs.

Order Matters

  • Swapping decorator order changes both side‑effects and final result if wrappers transform the return value.

from functools import wraps

def decorator_A(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("A before")
        result = func(*args, **kwargs)
        print("A after")
        return result
    return wrapper

def decorator_B(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("B before")
        result = func(*args, **kwargs)
        print("B after")
        return result
    return wrapper

@decorator_A
@decorator_B
def foo():
    print("  >>> inside function foo")

@decorator_B
@decorator_A
def bar():
    print("  >>> inside function bar")

foo()

print("----")

bar()
3 months ago Permalien
cluster icon
  • Working with CSV files : Working with CSV files CSV (Comma Separated Values) is a plain-text tabular format where each line is a row and fields are delimited (commonly by com...
  • Structured Logging : Introduction to Structured Logging Plain-text logs are hard to parse and brittle to format changes. Structured logging records events as key-value da...
  • Adding Tests to a Multi-File Project : Adding Tests to a Multi-File Project Standard Project Layout with Tests To maintain a clean and organized codebase, it is standard practice to separat...
  • Dictionaries : Dictionaries (dict) Dictionaries are mutable, insertion-ordered collections of key-value pairs. Keys must be unique and immutable; values can be of an...
  • Handling Subprocess Errors : Handling Subprocess Errors External commands can fail in multiple ways: non-zero exit codes, missing executables, or hanging processes. Using subpr...

Python Functions Are First‑Class Citizens/shaare/AumOhg

  • python
  • python

Python Functions Are First‑Class Citizens

  • In Python, functions behave like any other object (strings, ints, lists).

  • Because they are "first‑class", we can:

    • Bind them to new variable names
    • Pass them around as arguments
    • Return them from other functions
    • Stash them in data structures.
  • This flexibility is the foundation for patterns such as callbacks, plugin registries, and decorators.

  • Assigning Functions to Variables

  • A variable can reference the function object itself, not its return value.

  • Any name that points to the function can be used to call it.

  • This is handy for creating aliases or late‑binding a function into another module.

def greet(name):
    print(f"Hello, {name}!")

say_hello = greet
print(say_hello is greet)
say_hello("Alice")

Passing Functions as Arguments

  • Higher‑order functions accept other callables to customize behavior.
  • Classic examples: sorted(key=...), event callbacks, retry helpers.
  • Lets you build flexible pipelines without hard‑coding every step.
def apply_operation(operation, *operands):
    print(f"Applying {operation.__name__} to {operands}")
    return operation(*operands)

def add(*numbers):
    return sum(numbers)

def mul(*numbers):
    result = 1

    for n in numbers:
        result *= n

    return result

print(apply_operation(add, 1, 2))
print(apply_operation(mul, 1, 2, 3, 4))

Returning Functions from Functions

  • A factory function can create and return a new, customized function.
  • The returned function “remembers” variables from the factory’s scope: this is a closure.
  • Great for building tailored validators, loggers, or API clients on the fly.
def create_api_client(auth_token):
    def api_client(endpoint, method):
        return f"Hitting endpoint {endpoint} with method {method} and auth token {auth_token}"

    return api_client

alice_api_client = create_api_client("alice-token")
bob_api_client = create_api_client("bob-token")

print(alice_api_client("/users", "GET"))
print(bob_api_client("/health", "GET"))

Storing Functions in Data Structures

  • Functions can live inside lists, dicts, sets, and other containers.
  • Enables command dispatch tables, plugin registries, and processing pipelines.
def task_A():
    print("Running task A")

def task_B():
    print("Running task B")

def task_C():
    print("Running task C")

pipeline = [task_B, task_A, task_C]

for task in pipeline:
    task()

command_registry = {
    "start": task_A,
    "process": task_B,
    "stop": task_C
}
command_registry["process"](http://)

Why First‑Class Functions Matter for Decorators

  • Decorators are simply functions that take another function, wrap it, and return a new function.
  • That entire mechanism only works because Python lets us treat functions as data.
  • With this groundwork, we’re ready to explore decorator syntax (@decorator) next.
    python
3 months ago Permalien
cluster icon
  • Dictionaries : Dictionaries (dict) Dictionaries are mutable, insertion-ordered collections of key-value pairs. Keys must be unique and immutable; values can be of an...
  • Parametrized Tests : Parametrized Tests Introduction Often, we need to test the same logic with different inputs and outputs, such as validating various IP address or hos...
  • Logging to Files : Logging to Files Basic File Logging with FileHandler Use logging.FileHandler to write log records to a file. mode='a' (append) preserves existing log...
  • Structured Logging : Introduction to Structured Logging Plain-text logs are hard to parse and brittle to format changes. Structured logging records events as key-value da...
  • Temporary Files and Directories : Temporary Files and Directories Automation scripts often need scratch space for intermediate data without cluttering the filesystem or risking name c...

Generators and Lazy Pipelines/shaare/0-n3aw

  • python
  • python

Generators and Lazy Pipelines

  • You can chain generator functions to form multi-stage data pipelines that process items one at a time.
  • No intermediate lists are built, so memory stays low even for very large streams.
  • Each generator only holds its own minimal state and passes items downstream on demand.

Memory Efficiency

  • Lazy iterables maintain only minimal state (like start, stop, step) regardless of total length.
  • Eager collections (lists, tuples) grow in memory usage as you add items.
  • Use sys.getsizeof() to inspect the in-memory size of objects themselves (not their contents).
# 1. DONE Ingest the log lines
# 2. DONE Filter log lines based on either level or message substring
# 3. DONE Extract and return only the message attribute of the logs

import sys
import json

def read_logs(filepath):
    """Reads the contents of a file line by line.

    Args:
        filepath (str): The path where the file is located.

    Returns:
        generator (dict(str)): The json dictionary for the log line.
    """
    with open(filepath, 'r') as file:
        for line in file:
            line = line.strip()
            if not line:
                continue
            yield json.loads(line)

def filter_logs(logs, level=None, message_substring=None):
    """Filters any iterable containing dictionaries by either level or message_substring (or both)

    Args:
        logs (iterable(dict)): Iterable containing the logs to be filtered.
        level (str): The log level to keep. Defaults to None.
        message_substring (str): The pattern to look for in messages. Defaults to None.

    Returns:
        generator (dict(str)): The json dictionary for the filtered log.
    """

    for log in logs:
        if (
            level is not None
            and log.get("level", "").lower() != level.lower()
        ):
            continue

        if (
            message_substring is not None
            and message_substring.lower() not in log.get("message", "").lower()
        ):
            continue

        yield log

def extract_field(logs, field="message"):
    """Extracts a specific field from any iterable containing dictionaries.

    Args:
        logs (iterable(dict)): Iterable containing the logs to be evaluated.
        field (str): The field to return. Defaults to 'message'.

    Returns:
        generator (str): The value of the extracted field.
    """
    for log in logs:
        yield log.get(field, "").strip()

def get_first_n(logs, n=10):
    """Extracts the first n items from the provided iterable.

    Args:
        logs (iterable(T)): Iterable from which items will be extracted.
        n (int): The number of items to extract.

    Returns:
        generator (T): The item from the iterable.
    """
    count = 0

    for log in logs:
        if count >= n:
            break

        yield log
        count += 1

logs_gen = read_logs("large_logs.txt")
filter_gen = filter_logs(logs_gen, message_substring="user")
extract_gen = extract_field(filter_gen, "message")

for log in get_first_n(extract_gen, 4):
    print(log)

print("Generator object sizes (in bytes):",
      sys.getsizeof(logs_gen),
      sys.getsizeof(filter_gen),
      sys.getsizeof(extract_gen)
     )
3 months ago Permalien
cluster icon
  • List : Lists (list) Lists are ordered, mutable sequences defined with square brackets []. You can add, remove, or change items after creation. Characteristic...
  • Enhancing Functions: Decorators : Enhancing Functions: Decorators A decorator is a callable that takes another function, adds behaviour before and/or after it runs, and returns a new ...
  • The Iteration Protocol : The Iteration Protocol We use for item in sequence: all the time. But how does Python get each item? Iterable: An object that can be looped over. It...
  • Classes and Objects : Classes and Objects Beyond Built-ins: Python lets you define your own data types using class. Class: A blueprint or template for creating objects. De...
  • Working with JSON files : Working with JSON files JSON is the standard format for data exchange in web services and cloud APIs. Python’s built-in json module provides function...

Functions: return vs yield/shaare/UDAb-A

  • python
  • python

Functions: return vs yield

  • Regular functions execute immediately, run to completion, and return a single value (or None).
  • Generator functions return an iterator immediately; their body runs incrementally as values are requested.
  • Understanding this distinction is critical for choosing between eager and lazy workflows.

Regular Function (return) Recap

  • Calling a regular function runs its entire body before returning.
  • A single return exits the function and discards all local state.
  • Useful when you need to compute and return a complete result at once.
def get_list_of_servers():
    print("Regular function started.")
    servers = []

    for i in range(3):
        server_name = f"server-{i}"
        print(f"\tAdding {server_name}")
        servers.append(server_name)

    print("Regular function finished.")

    return servers

servers = get_list_of_servers()
print(f"Returned list: {servers}")

Generator Function (yield) Recap

  • Calling a generator function returns a generator object without running its body.
  • Each yield returns one value and pauses, preserving local variables until the next request.
  • Ideal for producing sequences lazily, especially when the full list is large or unbounded.
def yield_servers(count):
    print("Generator function started.")

    for i in range(count):
        server_name = f"server-{i}"
        print(f"\tYielding {server_name}")
        yield server_name

    print("Generator function finished.")

servers_gen = yield_servers(3)

for server in servers_gen:
    print(f"Server received: {server}")
3 months ago Permalien
cluster icon
  • Working with Environment Variables : Working with Environment Variables Environment variables are dynamic, named values provided by the operating system to running processes, enabling co...
  • Parametrized Tests : Parametrized Tests Introduction Often, we need to test the same logic with different inputs and outputs, such as validating various IP address or hos...
  • Lambda Functions : Lambda Functions Python functions defined with def allow multiple statements, clear naming, and support for docstrings, making them ideal for complex...
  • Classes and Objects : Classes and Objects Beyond Built-ins: Python lets you define your own data types using class. Class: A blueprint or template for creating objects. De...
  • Typing classes : Introduction As our Python automation projects grow, defining custom classes helps model complex objects and should be reflected in type hints for cl...

Generators/shaare/fMZUuQ

  • python
  • python

Generators

  • Writing a class-based iterator requires __iter__() and __next__(), plus manual state management and StopIteration handling.
  • Generator functions let you express the same logic in plain Python functions, using yield to produce values one at a time.
  • Any function with yield becomes a generator: calling it returns a generator object (an iterator) without running its body immediately.
def count_up_to(limit):
    """Generates numbers from 1 up to (and including) the limit.

    Args:
        limit (int): The upper limit for counting.

    Returns:
        generator(int): The generator to lazily count up to limit.
    """
    print("Generator function started...")
    n = 1

    while n <= limit:
        print(f"Yielding {n}")
        yield n
        print(f"Resumed after yielding {n}.")
        n += 1

    print("Generator function finished.")

count_gen = count_up_to(3)
print(f"Returned object: {count_gen} of type {type(count_gen)}")

print("First call to next outside of for loop.")
next(count_gen)

print("Remaining output from for loop.")
for number in count_gen:
    print(number)

Generator Functions & the yield Keyword

  • A function becomes a generator by including yield; no other boilerplate is needed.
  • Calling a generator function returns an object that implements __iter__() and __next__().
  • The code inside runs only when iteration begins (e.g., in a for loop or via next()).
def filter_evens(data):
    """Yield only the even items from the input sequence.

    Args:
        data (iterable(int or float)): The data to iterate through and filter.

    Returns:
        generator(int or float): A generator object that yields the even items.
    """
    print("filter_evens: starting")

    for item in data:
        if item % 2 == 0:
            print(f"filter_evens: yielding {item}")
            yield item

    print("filter_evens: finished")

evens_from_range = filter_evens(range(6))

print(f"Generator object created: {evens_from_range}")

for num in evens_from_range:
    print(f"Received even: {num}")

evens_from_list = filter_evens([0, 1, 2, 3, 4, 5])

print(f"Generator object created: {evens_from_list}")

for num in evens_from_list:
    print(f"Received even: {num}")

How yield Works: Pause and Resume

  • On each next() (or loop iteration), execution runs until it hits yield, returns the value, then pauses with all local state intact.
  • The next next() call resumes immediately after the yield, preserving variables and the instruction pointer.
  • When the function ends (no more yield), a StopIteration is raised automatically.
def demo_three_yields():
    """Demonstrate how having multiple yield statements work."""
    print("Generator started")
    yield 1
    print("Generator resumed after yielding 1.")
    yield 2
    print("Generator resumed after yielding 2.")
    yield 3
    print("Generator finished.")

demo_gen = demo_three_yields()

print(next(demo_gen))
print(next(demo_gen))
print(next(demo_gen))
# print(next(demo_gen)) # Uncommenting will raise a StopIteration Exception because there are no more yields

Generator State

  • Generators keep their local variables alive between yields, making explicit state objects unnecessary.
  • This persistent state allows infinite or long-running sequences without full data storage.
count_gen = count_up_to(5)

print("First call to next outside of for loop.")
print(next(count_gen))

print("Second call to next outside of for loop - now the value yielded is 2.")
print(next(count_gen))

print("Remaining output from for loop - prints from 3 onwards.")
for number in count_gen:
    print(number)
count_gen = count_up_to(5)

# Since generators have state, using the same generator object in nested loops can lead to issues.
# The inner for loop will complete the iteration, and the outer for loop will have a sinle pass.
for num in count_gen:
    for num2 in count_gen:
        print(f" - {num}:{num2}")

# The solution to this is to use distinct generator objects.
for num in count_up_to(5):
    for num2 in count_up_to(5):
        print(f" - {num}:{num2}")

Exhaustion

  • Once a generator’s code path completes (falls off the end or hits return), further next() calls immediately raise StopIteration.
  • A for loop over an exhausted generator does nothing on subsequent passes—you must call the function again for a fresh iterator.
count_gen = count_up_to(2)

print(next(count_gen))
print(next(count_gen))

try:
    print(next(count_gen)) # Will raise StopIteration exception
except StopIteration:
    print("Generator finished")

# Nothing will happen because the generator is already exhausted
for number in count_gen:
    print(number)
3 months ago Permalien
cluster icon
  • Exceptions : Common Built‑in Exceptions Python ships with a rich hierarchy of exception classes; most automation errors fall into a small, predictable subset. A...
  • Declarative Logging : Declarative Logging Configuration Declarative configuration separates setup from code, making it easier to maintain and adjust. Python’s logging.conf...
  • Adding Type Hints to Decorators and Generators : Adding Type Hints to Decorators and Generators Decorators and generators are advanced constructs that require specialized type hints to make their tr...
  • List : Lists (list) Lists are ordered, mutable sequences defined with square brackets []. You can add, remove, or change items after creation. Characteristic...
  • Making HTTP Requests : Making HTTP Requests The requests library simplifies HTTP interactions by abstracting raw HTTP details, making it ideal for DevOps automation tasks. ...

The Iteration Protocol/shaare/sBPSEQ

  • python
  • python

The Iteration Protocol

We use for item in sequence: all the time. But how does Python get each item?

  • Iterable: An object that can be looped over. It's anything you can put on the right side of the in keyword in a for loop. Examples include lists, tuples, strings, dictionaries, sets, files, and range objects.

    • An object is considered iterable if it implements the __iter__() special method.
    • The __iter__() method returns an iterator.
  • Iterator: An object that produces the next value in a sequence when asked. It "remembers" its position in the sequence.

    • An object is an iterator if it implements the __next__() special method. When there are no more items, __next__() raises the StopIteration exception.
    • Iterators normally also implement the __iter__() method, which makes them iterables too.

Example: iterable returning an iterator

class CountTo:
    def __init__(self, max_value):
        self.max = max_value

    def __iter__(self):
        # Each new for-loop call
        # gets its own iterator
        return CountToIter(self.max)

class CountToIter:
    def __init__(self, max_value):
        self.max = max_value
        self.curr = 1

    def __iter__(self):
        # Iterators are iterable
        return self

    def __next__(self):
        if self.curr <= self.max:
            val = self.curr
            self.curr += 1
            return val
        else:
            raise StopIteration

Supports nested loops: Decoupling iterables from iterators allow us to instantiate a single iterable and use it in nested loops without consuming the values from a single iterator.

my_foods = ["apple", "banana", "cherry"]

for food in my_foods:
    for food2 in my_foods:
        if food == food2:
            print(f"Skipping duplicate food: {food}")
            continue
        print(f"Cooking {food} with {food2}")

class CountTo:
    def __init__(self, max_value):
        self.max = max_value

    def __iter__(self):
        return CountToIterator(self.max)

class CountToIterator:
    def __init__(self, max_value):
        self.max = max_value
        self.current = 1

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= self.max:
            val = self.current
            self.current += 1
            return val
        else:
            raise StopIteration

counter = CountTo(5)

for count in counter:
    for count2 in counter:
        print(f"Count: {count} and {count2}")
3 months ago Permalien
cluster icon
  • *args and **kwargs : Flexible Functions: *args and **kwargs We can use the syntax *args and **kwargs to accept a variable number of both positional and keyword arguments....
  • For & While Loops : For & While Loops Python provides two main ways to repeat actions: for loops (for iterating over known sequences) and while loops (for repeating as lo...
  • Custom Exceptions: Tailoring Error Signals : Custom Exceptions: Tailoring Error Signals Built-in exceptions are great, but often too generic for application-specific failures. A custom excepti...
  • Declarative Logging : Declarative Logging Configuration Declarative configuration separates setup from code, making it easier to maintain and adjust. Python’s logging.conf...
  • Context managers : Context Managers When opening files or acquiring locks, resources must be released even if errors occur. Manual try...finally ensures cleanup but a...

Lambda Functions/shaare/PlNZLA

  • python
  • python

Lambda Functions

  • Python functions defined with def allow multiple statements, clear naming, and support for docstrings, making them ideal for complex or reusable logic.
  • In many cases, you need a simple, single-expression function to pass directly to another function without the ceremony of a full definition.
  • The lambda keyword lets you create small, anonymous functions inline, avoiding the verbosity of a def for trivial operations.
  • These one-line functions are particularly handy when you want to supply custom behavior to built-in higher-order functions without polluting your namespace with one-off function names.

Syntax of a lambda Function

  • A lambda function follows the exact pattern: lambda <arguments>: <expression>, where the expression result is implicitly returned.
  • The lambda keyword introduces the function, <arguments> lists its parameters, and a colon separates them from the single expression body.
  • You cannot include multiple statements, loops, or traditional if/else blocks: only a single expression or a ternary expression.
  • Compared to a def function, a lambda is nameless and concise, making it ideal for inline usage where defining a named function would be overkill.
square = lambda x: x * x
print(square(5))

print((lambda a, b: a + b)(3, 4))

Custom Sorting with sorted()

  • The built-in sorted() function accepts an optional key parameter, which should be a function that returns a comparison key for each element.
  • Using a lambda for the key argument lets you define the sorting logic inline without a separate function definition.
  • This approach keeps your code concise and focused, especially when the key logic is a simple attribute extraction or computation.
  • When you need more complex sorting logic, you can still fall back to a named def function for clarity.
services = [("web-app", 3), ("database", 1), ("cache", 5), ("api-gateway", 2)]

print(f"Default sort: {sorted(services)}")

def get_replica_count(svc_tuple):
    return svc_tuple[1]

print(f"Sorting by replica count - standard function: {sorted(services, key=get_replica_count)}")
print(f"Sorting by replica count - lambda function: {sorted(services, key=lambda svc: svc[1])}")

Transforming Data with map()

  • The map(function, iterable) built-in applies a given function to each item in an iterable, producing an iterator of results.
  • Using a lambda with map lets you specify simple transformations inline without an extra function definition.
  • Although list comprehensions are often preferred for readability, map with a lambda can be concise when you already need an iterator or want to emphasize the function-application nature.
  • Remember that map returns a lazy iterator; convert it to a list if you need to access all results at once.
my_numbers = [1, 2, 3, 4]
print(list(map(lambda num: num * 2, my_numbers)))

ports = [80, 443, 8080, 22]
port_descriptions = list(map(lambda port: f"Port {port} is open", ports))

print(port_descriptions)

Filtering Data with filter()

  • The filter(function, iterable) built-in yields only those items for which the function returns True.
  • A lambda in filter is perfect for inline tests, such as checking attributes or simple conditions.
  • As with map, filter returns an iterator, so wrap it in list() to evaluate immediately if needed.
  • While list comprehensions can express filtering more idiomatically in modern Python, filter remains a clear demonstration of higher-order function usage.
ports = [80, 443, 8080, 22, 5432]

privileged_ports = list(filter(lambda port: port < 1024, ports))
print(privileged_ports)

privileged_comprehension = [port for port in ports if port < 1024]
print(privileged_comprehension)
3 months ago Permalien
cluster icon
  • Filesystem Operations : Filesystem Operations (os & shutil) DevOps scripts often need to create, delete, copy, and move files and directories as part of automation workflows...
  • Exceptions : Common Built‑in Exceptions Python ships with a rich hierarchy of exception classes; most automation errors fall into a small, predictable subset. A...
  • Typing : Introduction Python is a dynamically typed language, meaning you can assign values to variables without declaring their types, and type checking happ...
  • Automated Testing with Pytest : Assertions in Pytest Pytest uses Python’s built-in assert statement to declare expected conditions in tests, making test code concise and readable. W...
  • Working with YAML files : Working with YAML files YAML (“YAML Ain’t Markup Language”) focuses on human readability. Indentation replaces braces and brackets, comments are allo...

*args and **kwargs/shaare/qwGUkw

  • python
  • python

Flexible Functions: *args and **kwargs

  • We can use the syntax *args and **kwargs to accept a variable number of both positional and keyword arguments.
def example_function(*args, **kwargs):
    print(f"Positional args: {args}")
    print(f"Keyword args: {kwargs}")

example_function(1, 2, 3, a="Value", b=True)

*args in Definition: Collecting Positionals

  • Uses *args to gather extra positional parameters into a tuple
  • Allows functions to accept any number of positional inputs
  • Common in utilities like custom logging or aggregation functions
def apply_operator(operator, *operands):
    """Applies operator to a variable number of operands. Supports 'add' and 'mul'.

    Args:
        operator (str): The operator to apply. Must be either 'add' or 'mul'.
        *operands (int or float): Zero or more numbers to be combined.

    Returns:
        int or float: The result of applying the operator on the operands.

    Raises:
        ValueError: Raised when operator is not 'add' nor 'mul'.
    """

    if operator == 'add':
        result = sum(operands)
    elif operator == 'mul':
        result = 1
        for n in operands:
            result *= n
    else:
        raise ValueError(f"Unknown operator {operator}. Supported values are 'add' and 'mul'")

    return result

print(apply_operator('add', 1, 2, 3, 4))
print(apply_operator('add', 1, 2, 3, 4, 5, 6, 7))
print(apply_operator('add', 1, 2))

print(apply_operator('mul', 1, 2, 3, 4))
print(apply_operator('mul', 1, 2, 3, 4, 5, 6, 7))
print(apply_operator('mul', 1, 2))

# print(apply_operator('div', 1, 2)) # Uncommenting raises ValueError since div is not supported

**kwargs in Definition: Collecting Keywords

  • Uses **kwargs to gather extra named parameters into a dictionary
  • Ideal for optional configuration flags or settings
  • Enables functions to accept flexible keyword arguments without predefining them
def set_options(**settings):
    print(f"Received dictionary: {settings}")
    for key, value in settings.items():
        print(f"\t{key} = {value}")

set_options(timeout=30, user="admin", retries=5)

Order in Definition Matters

  • Standard positional parameters must come first, some might also have a default value
  • Followed by *args to catch extra positionals
  • Then keyword-only parameters, some might also have a default value
  • Finally **kwargs to catch extra keyword arguments
def process_request(url, method="GET", *headers, timeout, **params):
    print(f"url={url}, method={method}, timeout={timeout}")
    print(f"headers={headers}")
    print(f"params={params}")

process_request("https://www.example.com", timeout=30)
process_request("https://www.example.com", "PUT", timeout=30)
# Equivalent to call above
process_request("https://www.example.com", timeout=30, method="PUT")

process_request(
    "https://www.example.com",
    "PUT",
    "Auth: xyz",
    "Content-Type: application/json",
    timeout=30
)

process_request(
    "https://www.example.com",
    "PUT",
    "Auth: xyz",
    "Content-Type: application/json",
    timeout=30,
    retries=5,
    log_level="DEBUG"
)

* in Call: Unpacking Positional Arguments

  • Uses *sequence to expand a list or tuple into positional arguments
  • Sequence length must match the function’s positional parameters
  • Useful for dynamic argument lists built at runtime
def connect(host, port, timeout):
    print(f"Connecting to {host}:{port} with timeout {timeout}s.")

params = ["db.internal", 5432, 10]
params_with_extra_values = ["db.internal", 5432, 10, "a", True]
connect(*params)
connect(*params_with_extra_values[:3])

** in Call: Unpacking Keyword Arguments

  • Uses **dict to expand key-value pairs into keyword arguments
  • Dictionary keys must match the function’s parameter names
  • Common in configuration-driven function calls
def configure_service(name, version, replicas=1):
    print(f"Setting up {name} v{version} with {replicas} replicas...")

config = {"name": "auth-service", "version": "2.1.0", "replicas": 3}
configure_service(**config)
3 months ago Permalien
cluster icon
  • Handling Authentication : Handling Authentication APIs often require authentication to control access, rate limits, and auditing. Without authentication, requests to protected...
  • Custom Exceptions: Tailoring Error Signals : Custom Exceptions: Tailoring Error Signals Built-in exceptions are great, but often too generic for application-specific failures. A custom excepti...
  • Running Python modules : Running Scripts: python -m vs. python file.py The Core Difference: What is "Entry Point Zero"? The key to understanding the difference lies in the fir...
  • Structured Logging : Introduction to Structured Logging Plain-text logs are hard to parse and brittle to format changes. Structured logging records events as key-value da...
  • Automated Testing with Pytest : Assertions in Pytest Pytest uses Python’s built-in assert statement to declare expected conditions in tests, making test code concise and readable. W...

Classes and Objects/shaare/Qakaww

  • python
  • python

Classes and Objects

  • Beyond Built-ins: Python lets you define your own data types using class.
  • Class: A blueprint or template for creating objects. Defines attributes (data) and methods (behavior). Convention: PascalCase names (MyClass).
  • Object (Instance): A specific item created from a class blueprint. Each object has its own set of attribute values but shares the methods defined by the class. obj1 = MyClass(), obj2 = MyClass(). obj1 and obj2 are distinct objects.

Defining a Class & __init__ (The Constructor)

  • __init__(self, ...): Special method for initialization. self is always the first parameter and represents the instance itself. Other parameters receive arguments passed during object creation.
  • Instance Attributes (self.x = ...): Data attached to this specific object. Created inside methods (usually __init__) using self.attribute_name = value.
class ServiceMonitor:
    """Provides service checks for a single service"""
    def __init__(self, service_name, port):
        """Initializes the monitor for a specific service.

        Args:
            service_name (str): the name of the service.
            port (int): the port to use for checks.
        """
        print(f"Initializing monitor for service {service_name} on port {port}.")
        self.service = service_name
        self.port = port
        self.is_alive = False

Creating Instances (Objects)

  • Mechanism: Call the class name as if it were a function, passing any arguments required by __init__ (after self).
  • Python automatically creates the object and passes it as self to __init__.
nginx_monitor = ServiceMonitor("nginx", 80)
print(isinstance(nginx_monitor, ServiceMonitor))

redis_monitor = ServiceMonitor(service_name="redis", port=6379)
print(isinstance(redis_monitor, ServiceMonitor))

print(nginx_monitor.service)
print(redis_monitor.service)

Instance Methods: Object Behavior

  • Definition: Functions defined inside a class definition.
  • First Parameter: Always self (by strong convention), allowing the method to access and modify the instance's attributes (self.attribute_name).
  • Calling: Use dot notation on an instance: instance.method_name(arguments). Python automatically passes the instance (instance) as the self argument.
class ServiceMonitor:
    """Provides service checks for a single service"""
    def __init__(self, service_name, port):
        """Initializes the monitor for a specific service.

        Args:
            service_name (str): the name of the service.
            port (int): the port to use for checks.
        """
        print(f"Initializing monitor for service {service_name} on port {port}.")
        self.service = service_name
        self.port = port
        self.is_alive = False

    def check(self):
        """Simulates checking the service status"""
        print(f"METHOD: Checking {self.service} on port {self.port}...")
        self.is_alive = True
        print(f"METHOD: Status for service {self.service}: {"Alive" if self.is_alive else "Down"}")
        return self.is_alive

nginx_monitor = ServiceMonitor("nginx", 80)
status = nginx_monitor.check()
print(f"Received status: {status}")

Basic Inheritance: Reusing and Extending

  • Concept: Create a new class (Child/Subclass) that inherits properties (attributes and methods) from an existing class (Parent/Superclass). Promotes code reuse (DRY).
  • Syntax: class ChildClassName(ParentClassName):
  • Inherited Members: The Child automatically gets all methods and attributes defined in the Parent.
  • Specializing: The Child can:
    • Add new attributes and methods.
    • Override parent methods by defining a method with the same name.
  • super(): Inside the Child's methods, use super().method_name(...) to explicitly call the Parent's version of a method (very common in __init__).
class HttpServiceMonitor(ServiceMonitor):
    """Extends ServiceMonitor to add an HTTP endpoint check."""
    def __init__(self, service_name, port, url):
        super().__init__(service_name, port)
        self.url = url

    def ping(self):
        """Ping url provided when creating instance."""
        print(f"METHOD: Pinging url {self.url}")

    def check(self):
        alive = super().check()
        print(f"METHOD: Performing HTTP check on {self.url}")

http_monitor = HttpServiceMonitor("web", 8080, "http://localhost")
nginx_monitor = ServiceMonitor("nginx", 80)

http_monitor.ping()
http_monitor.check()
# nginx_monitor.ping() # Uncommenting will raise AttributeError since ping() is a method only of the subclass
nginx_monitor.check()
3 months ago Permalien
cluster icon
  • Lambda Functions : Lambda Functions Python functions defined with def allow multiple statements, clear naming, and support for docstrings, making them ideal for complex...
  • Typing : Introduction Python is a dynamically typed language, meaning you can assign values to variables without declaring their types, and type checking happ...
  • Python package and subpackage : Introduction to Packages (__init__.py) What is a Package? A Python package provides a way to structure a project's module namespace by using directori...
  • Structured Logging : Introduction to Structured Logging Plain-text logs are hard to parse and brittle to format changes. Structured logging records events as key-value da...
  • Python Modules and the import System : Python Modules and the import System What is a Module? A module in Python corresponds directly to a single file containing Python code. The module's ...

Range, zip/shaare/YcrfAg

  • python
  • python

Efficient Looping: range

  • Creating large lists for loops is memory-intensive (e.g., list(range(1_000_000))).
  • range() stores only start, stop, and step values, not all numbers.
  • Numbers are generated one at a time during iteration, reducing memory usage.
  • Ideal for loops needing a fixed number of iterations without large allocations.
import sys

number_count = 10_000_000

numbers_list = list(range(number_count))
numbers_range = range(number_count)

list_mb = sys.getsizeof(numbers_list) / (1024**2)
range_mb = sys.getsizeof(numbers_range) / (1024**2)

print(f"List size: {list_mb:.2f}")
print(f"Range size: {range_mb:.6f}")
print(f"List uses {(list_mb / range_mb):.2f} more memory!")

Using range()

  • range(stop): iterate from 0 up to (but not including) stop.
  • range(start, stop): iterate from start up to stop.
  • range(start, stop, step): iterate with a custom step increment.
for i in range(5):
    print(f"Retry #{i}")

for year in range(2020, 2024):
    print(f"Processing logs for {year}")

for server_id in range(10, 30, 5):
    print(f"Checking server {server_id}")

Getting Index + Value: enumerate()

  • Use enumerate(iterable, start=0) to get (index, item) tuples.
  • The start parameter sets the initial index value.
servers = ["web01", "web02", "web03"]

for idx, server in enumerate(servers, 1):
    print(f"#{idx}: Processing server {server}")

Parallel Iteration: zip()

  • Use zip(*iterables) to pair items from multiple iterables.
  • Iteration stops when the shortest iterable is exhausted.
hosts = ["hostA", "hostB", "hostC"]
ips = ["10.0.0.1", "10.0.0.2"]
azs = ["us-east-1a", "us-east-1b"]

for host, ip, az in zip(hosts, ips, azs):
    print(f"Host: {host}, IP: {ip}, AZ: {az}")
3 months ago Permalien
cluster icon
  • Read/Write Text Files : Read/Write Text Files Use open() to read/write text files with proper modes and encoding. Specify encoding='utf-8' for portability. Leverage with...
  • Mocking : Mocking Fundamentals Introduction When unit testing DevOps scripts that interact with external systems, tests can become slow, unreliable, difficult ...
  • Configuring Pytest : Configuring Pytest As you start using Pytest extensively, typing -v or -m on the command line every time becomes tedious. Centralize your defaults in...
  • Automated Testing with Pytest : Assertions in Pytest Pytest uses Python’s built-in assert statement to declare expected conditions in tests, making test code concise and readable. W...
  • Functions, Docstrings : Functions Functions package reusable code into named blocks, improving modularity, readability, and testability. They prevent duplication (DRY) and ma...


(110)
3 / 6
Liens par page
  • 20
  • 50
  • 100
Filtrer par liens sans tag
Replier Replier tout Déplier Déplier tout Êtes-vous sûr de vouloir supprimer ce lien ? Êtes-vous sûr de vouloir supprimer ce tag ? Le gestionnaire de marque-pages personnel, minimaliste, et sans base de données par la communauté Shaarli