In this lesson, you will learn how to determine whether your high-frequency time-series system should utilize agent-based polling or centralized data streaming.
As we move into specialized data storage, the architecture of how metrics arrive at your database is a foundational design choice. We generally categorize ingestion into two patterns: Pull (Polling) and Push (Streaming).
The Pull Model (Agent-Based Polling)
In a pull-based architecture, your database or an intermediate collector actively initiates a request to the source (the target) to fetch metrics at defined intervals. This is common in environments where you have a central registry of services.
- Pros: The central system controls the load; if the database is struggling, it can simply reduce the poll frequency. It simplifies service discovery, as targets don't need to know where the database is located—they just need to expose an endpoint.
- Cons: You may miss data between intervals if the source is ephemeral or restarts frequently. It also requires the targets to be reachable via a network route from the collector.
The Push Model (Centralized Data Streaming)
In a push-based architecture, sources emit data points to a collector or a message broker (like Kafka or a direct ingestion API) as soon as they are generated.
- Pros: Ideal for short-lived jobs or high-frequency events where you cannot risk losing data between polling cycles. It pushes the responsibility of transmission to the source.
- Cons: It can overwhelm your ingestion layer if all clients "burst" data simultaneously. It also requires clients to have the configuration and network access to reach the central sink.
Decision Framework
To choose between them, consider the nature of your data source:
-- Conceptual: Polling vs. Push metadata
-- Pulling requires a registry of targets:
SELECT target_url, scrape_interval
FROM service_discovery
WHERE status = 'active';
-- Pushing relies on an append-only write stream:
INSERT INTO metrics_buffer (source_id, metric_name, value, ts)
VALUES ('sensor_01', 'temp', 22.5, NOW());
If your infrastructure involves high-frequency data where every micro-event matters (such as real-time anomaly detection), Push is superior. If your environment is stable and you need to strictly control resource utilization across a vast fleet of services, Pull remains the industry standard for monitoring.
Key Takeaway: Choose Push ingestion for event-driven accuracy and ephemeral sources, but use Pull ingestion for robust resource management and simplified service orchestration.