In this lesson, you will learn how to feed real-time script output directly into a remote ingestion endpoint to centralize your logs.
The Concept of Streaming
When you run a script, your logs are typically trapped inside a local file or standard output (stdout). If that server crashes or the disk fills up, that data vanishes. Log streaming solves this by taking your process’s output and "piping" it through a network connection to a remote server (an ingestion endpoint) as soon as the data is generated.
Think of it like a faucet: instead of filling a local bucket (a file), you attach a hose (a pipe) that sends the water directly to a reservoir across town.
Piping to a Remote Endpoint
In Linux, you can leverage the nc (Netcat) utility to act as this "hose." By combining standard input redirection with a network socket, your script can push logs the moment they appear.
Imagine you have a script monitor.sh that generates logs. Instead of just running it, you can stream the output like this:
./monitor.sh | nc ingestion-server.internal 5000
In this command:
./monitor.shperforms its work and prints logs tostdout.- The pipe (
|) captures that stream of data. nc(Netcat) takes that stream and forwards it to the remote server atingestion-server.internalon port5000.
Why this matters
Because you are using standard pipes, you don't need to change your script’s internal logic. Whether you are using echo inside a loop or running a complex diagnostic tool, the system treats the network socket exactly like a file descriptor.
This approach is highly memory-efficient because the logs are never written to your local disk. They are processed as a continuous stream of data, allowing your centralized logging system to aggregate logs from dozens of servers simultaneously, providing you with a single "pane of glass" for your entire infrastructure.
Key Takeaway: You can move logs off-machine in real-time by piping script output directly into a network socket, preventing data loss and enabling centralized monitoring.