Learning Generics by extending the prometheus python client

Imagine you’re writing a library to extend the Prometheus Python client and need to add some dynamic labels specific to the environment. Ideally, you would add most of your labels at the collector end to avoid writing extensions, but there are situations where injecting dynamic labels is unavoidable. For these cases, you might want to extend the client in a structured and type-safe manner.

What Are Generics? And Why Do We Use Them in Python?

Generics are a feature commonly associated with statically typed languages. They allow you to create code that works with a variety of types while maintaining type safety. By declaring “placeholder types,” generics help us write higher-level patterns that simplify declarations and reduce redundancy.

While Python doesn’t enforce type checking at runtime (as long as the code is syntactically valid, it will run), tools like type hints and third-party linters like mypy bring the safety of statically typed languages to Python. These tools enable developers to:

Catch type errors early.
Improve code readability.
Document function and class behavior clearly.

Generics in Python

Generics in Python are brought to life using type hints from the typing module. In essence, generics in Python are a programming style adapted from statically typed languages to achieve similar benefits in Python’s dynamic environment.

Extending the Prometheus Python Client with Generics

Below is an example of extending the Prometheus client to support dynamic labels using generics. This approach makes it easier to manage labels dynamically while preserving type safety.

Example Code with Comments

from prometheus_client import Counter as _PromCounter
from prometheus_client import Histogram as _PromHistogram
from typing import TypeVar, Generic, Iterable, Dict, cast

# Create a new generic type, bound to two specific Prometheus metric types
_MetricsTypeT = TypeVar('_MetricsTypeT', _PromCounter, _PromHistogram)

# Base class of type generic; the child class will pass in the type
class _MetricsBase(Generic[_MetricsTypeT]):
    def __init__(self, label_names: Iterable[str]):
        # Default labels dynamically fetched (e.g., environment-specific)
        self.default_labels: Dict[str] = get_default_labels()
        self.all_label_names: list = list(label_names) + list(self.default_labels.keys())
        self._parent_metric: _MetricsTypeT = None  # Parent metric instance

    # Provides dynamic label functionality
    def labels(self, *labelargs, **labelkwargs) -> _MetricsTypeT:
        if labelargs:
            # Append default labels to positional arguments
            labelargs += tuple(self.default_labels.values())
            return cast(_MetricsTypeT, self._parent_metric.labels(*labelargs))
        # Merge default labels into keyword arguments
        labelkwargs.update(self.default_labels)
        return cast(_MetricsTypeT, self._parent_metric.labels(**labelkwargs))


# Child class passing a specific type (_PromCounter) to the base class via generics
class Counter(_MetricsBase[_PromCounter]):
    def __init__(self, name: str, documentation: str, labelnames: Iterable[str] = ()):
        super().__init__(label_names=labelnames)
        # Create the actual Prometheus Counter metric
        self._parent_metric = _PromCounter(
            name=name, documentation=documentation, labelnames=self.all_label_names
        )

Key Points:

Generics in Base Class:
- _MetricsBase accepts a generic type (_MetricsTypeT), which can be either a Prometheus Counter or Histogram.
- This ensures type safety across methods like labels().
Dynamic Label Handling:
- default_labels are fetched dynamically, e.g., from the environment or configuration.
- These labels are automatically added to the user-provided labels when creating or using metrics.
Child Classes:
- Child classes (Counter, Histogram) specify the type of Prometheus metric they represent.
- They inherit the generic behavior while remaining strongly typed.

The Final Word

In my project, I used generics to type my base class _MetricsBase. This class accepts a generic type that is passed in by its child classes (Counter, Histogram). The labels() method dynamically appends environment-specific labels while ensuring type safety. By leveraging mypy or similar tools, I achieved many of the benefits of statically typed languages in Python.

Complete Code

You can find the complete code for this example here.