Streaming to Cloud Object Storage

In this lesson, you will learn how to architect efficient uploads to cloud providers using multi-part streaming to maximize your bandwidth utilization.

The Problem with Monolithic Uploads

When dealing with large files—such as multi-gigabyte backups or high-resolution video—uploading the entire object in one request is fragile. If the connection drops at 99%, you lose everything. Furthermore, a single sequential upload rarely saturates your available network bandwidth. To solve this, we use multi-part uploads, which split a stream into smaller, independent chunks that can be sent in parallel.

Architecting Parallel Streams

Instead of piping a single ReadStream directly to an HTTP request, we orchestrate the upload by chunking the data source. We create a custom Transform stream that buffers data into fixed-size segments (e.g., 5MB or 10MB blocks). Each block is then pushed into a pool of concurrent upload promises.

By utilizing Backpressure, we ensure that we only read from the disk as fast as the cloud provider can ingest the chunks:

// Conceptual Multi-part logic
const uploadChunk = async (chunk, partNumber) => {
  // Use our stream-based HTTP client to send the segment
  return cloudProvider.uploadPart({ Body: chunk, PartNumber: partNumber });
};

// Using backpressure to regulate stream flow
sourceStream.on('data', (chunk) => {
  if (uploadPool.length >= MAX_CONCURRENCY) {
    sourceStream.pause(); // Stop reading until a slot opens
  }
  processChunk(chunk);
});

Why Multi-part Streaming Wins

1. Fault Tolerance: If one part fails, we only retry that specific 5MB segment rather than the entire file.
2. Bandwidth Utilization: By having multiple concurrent HTTP requests (or streams) active, we keep the network pipe full, even if the latency to the cloud data center is high.
3. Memory Efficiency: We never load the entire file into memory; we only keep a few "in-flight" chunks in the event loop’s buffer at any given time.

When building these pipelines, always ensure that your error handling layer acts as a circuit breaker; if the cloud provider begins returning 503 errors, your orchestrator should pause the ReadStream to prevent data pile-up and memory exhaustion.

Key Takeaway: To maximize throughput, decompose large data streams into smaller, concurrent segments and use explicit backpressure to synchronize the read speed with the aggregate ingestion speed of your cloud provider.