In this lesson, you will learn how to design a rigorous performance benchmark to evaluate throughput and latency for heavy telemetry workloads in Time-Series Databases (TSDBs).
Defining the Workload Profile
Performance in a TSDB is rarely a single number; it is a tension between ingestion throughput (how many data points per second can we write?) and query latency (how fast can we retrieve them under load?). To benchmark effectively, you must simulate a realistic "telemetry storm."
A robust benchmark mimics the nature of your time-series data:
- Write Workload: Use a tool to generate batches of points with varying tags. Focus on "churn"—how the system handles new metric series (high cardinality) versus updates to existing ones.
- Query Workload: Design a mix of read patterns, including point lookups, range scans (e.g., pulling a week of data), and heavy aggregations (e.g., downsampling over a large time window).
Measuring Throughput vs. Latency
The most critical metric is the latency tail (P95/P99) under sustained throughput. It is easy to sustain high ingestion rates until you hit a bottleneck, at which point latency spikes exponentially.
You can observe these limits using a simple script to measure write-acknowledgment times:
-- Conceptual measure of ingestion latency
-- T1: Start batch write, T2: Receive success
SELECT (t2 - t1) AS write_latency_ms
FROM telemetry_benchmarks
WHERE batch_id = 'test_batch_001';
When benchmarking, keep these variables constant to isolate performance:
- Cardinality: Ensure the number of unique tags remains consistent across runs.
- Concurrency: Use a multi-threaded writer to simulate multiple edge devices pushing data simultaneously.
- Storage State: Measure performance on a "cold" database vs. one already filled with historical data to account for disk compaction and index growth.
As you scale, monitor the Resource Saturation. A TSDB that maintains low latency at 100k points/second but spikes to 500ms at 150k points/second is hitting a physical limit—likely related to indexing metadata or write-ahead log (WAL) contention. By slowly increasing the ingestion rate until the P99 latency breaches your SLO, you identify the "breaking point" of your specific cluster configuration.
Key Takeaway: Effective TSDB benchmarking requires measuring the P99 latency during concurrent ingestion and read operations to identify the exact point where resource saturation creates a performance cliff.