In this lesson, you will learn how to map your Node.js application logs to the Elastic Common Schema (ECS) to ensure your data is indexed efficiently and remains searchable across your entire infrastructure.
When shipping logs to Elasticsearch, sending raw text or inconsistently nested JSON creates "mapping explosions," where the database struggles to index fields because their data types vary. To solve this, we adhere to the Elastic Common Schema (ECS), which enforces standardized field names for common concepts like timestamps, service names, and error details.
Instead of writing custom, ad-hoc JSON objects, align your log payloads with ECS conventions. For instance, always use event.action for specific operations and service.name for identification. This consistency allows Elasticsearch to pre-allocate memory and create optimized indices, resulting in significantly faster query speeds when you eventually need to debug a production issue.
Consider this structured payload format:
const logEntry = {
"@timestamp": new Date().toISOString(), // Standard ISO format
"log.level": "error",
"message": "Failed to process payment",
"service.name": "payment-api",
"event.action": "process_transaction",
"error.code": "ERR_GATEWAY_TIMEOUT",
"trace.id": "a1-b2-c3-d4", // Correlation ID
"user.id": "user_987"
};
logger.info(JSON.stringify(logEntry));
By mapping your logs this way, you avoid the heavy processing overhead required for Elasticsearch to guess the schema of incoming documents. If you log a user.id as a string in one service and an integer in another, your aggregation queries will fail or return incomplete results. Adhering to a rigid, schema-first approach ensures that when you filter by log.level: "error" or group by service.name, the search engine operates at peak performance.
Consistent naming conventions essentially act as a contract between your Node.js runtime and your observability stack. When the index structure is predictable, your queries become faster, your dashboards become more reliable, and your incident response time drops.
Key Takeaway: Adhering to the Elastic Common Schema (ECS) by standardizing field names and types allows Elasticsearch to index your logs efficiently, preventing mapping conflicts and ensuring lightning-fast search performance.