Dynamic Task Joining

In this lesson, you will learn how to orchestrate and collect results from a collection of tasks whose number is determined only at runtime.

The Problem of Indefinite Concurrency

When we use tokio::join!, we are restricted to a fixed number of futures known at compile time. However, real-world services often need to spawn a variable number of tasks—such as processing a batch of user requests or crawling a dynamic list of URLs—and wait for every single one of them to finish before proceeding.

Managing Lifecycles with JoinSet

The idiomatic way to handle dynamic task joining is through a JoinSet. Think of a JoinSet as a managed container for tasks. When you spawn a task into a JoinSet, the runtime takes ownership of it, and the JoinSet tracks its progress. Unlike raw handles, which would require you to store them in a Vec<JoinHandle<T>> and iterate over them manually, the JoinSet acts as a stream of results.

Here is how you manage a dynamic workload:

use tokio::task::JoinSet;

async fn process_items(items: Vec<i32>) -> Vec<i32> {
    let mut set = JoinSet::new();

    for item in items {
        // Spawn tasks dynamically based on the input length
        set.spawn(async move {
            item * 2
        });
    }

    let mut results = Vec::new();
    // Dynamically collect results as they complete
    while let Some(res) = set.join_next().await {
        if let Ok(value) = res {
            results.push(value);
        }
    }
    results
}

Why This Matters

Beyond simple collection, JoinSet excels in resource management. If your task execution is canceled or the JoinSet is dropped, all pending tasks are automatically aborted. This prevents "zombie tasks" from running in the background when the parent operation is no longer interested in the outcome. By using this pattern, you move away from manual handle tracking and toward structured concurrency, ensuring that your task lifecycles are explicitly tied to the scope of the orchestrator.

Key Takeaway: Use JoinSet to treat a dynamic collection of tasks as a unified unit, enabling safe aggregation of results and automatic cleanup of pending work.