Course contentsShow
← Cloud and Infrastructure
Lesson 2037 of 2,98943. Observability - LoggingFree lesson

Promtail Log Collection and Pipelines

Configure Promtail to scrape logs, extract labels, and transform log lines before shipping to Loki.

Promtail Log Collection and Pipelines

You'll learn how Promtail scrapes logs from your systems, extracts meaningful labels, and transforms log content before sending it to Loki.

What is Promtail?

Promtail is Loki's log collection agent—think of it as a specialized delivery truck that picks up logs from various sources, labels each package with routing information, and prepares them for efficient storage in Loki. Unlike heavy processors like Logstash, Promtail is lightweight and follows the "labels, not indexing" philosophy.

How Promtail Works

Promtail operates through scraping configurations that define where to find logs (files, systemd journals, syslog streams) and pipeline stages that process each log line before shipping.

Scraping Targets

Promtail discovers log sources through:

  • Static configs: Hardcoded file paths (e.g., /var/log/app/*.log)
  • Service discovery: Kubernetes pods, Docker containers, automatically finding new sources
  • Push receivers: Accept logs pushed via HTTP

Pipeline Stages

Pipelines are sequential processing steps that extract labels and transform logs. Common stages include:

Regex extracts values from log lines into temporary key-value pairs.
Labels promotes extracted values to Loki labels (remember: keep cardinality low!).
Timestamp parses log timestamps instead of using collection time.
Output modifies the final log line content.
Drop filters out unwanted logs before sending.

Think of the pipeline as an assembly line: each log enters, gets stamped with labels, potentially modified or filtered, then exits ready for Loki.

Why Pipelines Matter

Extracting labels at collection time means Loki doesn't need to parse logs during queries—making searches blazingly fast. However, you must balance label extraction with cardinality concerns (remember from earlier Loki lessons: too many unique label combinations hurt performance).

Key Takeaway: Promtail pipelines transform raw logs into labeled, queryable streams by scraping sources, extracting labels through regex/parsing stages, and optionally filtering or modifying content before shipping to Loki.