In this lesson, you will learn how to measure the execution duration of scheduled tasks to preemptively identify resource contention before it impacts system stability.
When multiple automation tasks share a system, the primary danger is "latency creep." If a task that typically takes five seconds starts taking forty seconds, it will eventually overlap with subsequent scheduled jobs, leading to CPU contention, lock exhaustion, or cascading service failures. Monitoring latency is the process of tracking the "start-to-finish" time of these tasks to detect this drift early.
Capturing Latency with systemd-analyze and Journald
While you could manually wrap scripts in date commands, the most efficient way to track latency is by leveraging systemd metadata. Since we are already managing our automations as services, we can utilize systemd-analyze to inspect unit performance.
To track execution time, you can implement a simple "wrapper" strategy within your service unit. By logging the start and end timestamps, you can calculate the duration. A more automated approach involves using the ExecStart command to capture the duration directly into the system journal:
# Example: Wrapping a script to log its duration
ExecStart=/bin/bash -c 'start=$(date +%s); /usr/local/bin/my-task.sh; end=$(date +%s); echo "Task took $((end-start)) seconds" | systemd-cat -t task-monitor'
Predicting Resource Contention
Once you have these duration metrics flowing into journald, you can identify patterns. If your monitoring indicates that a task's duration is increasing linearly, it often suggests a backlog of data or a leaking resource (like an unindexed database table).
By analyzing these logs, you can build a simple threshold monitor. If the execution time exceeds 80% of the interval between scheduled runs, your system should trigger an alert. This allows you to identify contention before the task actually overlaps with the next cycle, giving you the chance to optimize the script or increase the OnCalendar interval before a system bottleneck occurs.
Key Takeaway: By measuring and logging task execution duration, you transform silent background processes into predictable, observable metrics that allow you to anticipate and prevent resource contention.