In this lesson, you will learn how to implement batch database commits to minimize lock contention and maximize throughput in high-frequency background workers.
The Problem: Write Amplification
When a background worker processes a high-velocity stream of events—such as IoT telemetry or clickstream data—performing a COMMIT for every individual record creates a massive performance bottleneck. Each commit forces the database engine to flush logs to disk and manage transaction locks. In a high-concurrency environment, these frequent round-trips lead to "lock contention," where worker threads spend more time waiting for the transaction log to clear than actually persisting your data.
The Strategy: Batching and Flushing
To optimize write efficiency, we decouple the incoming event stream from the physical write operation using an in-memory buffer. Instead of saving immediately, we collect events into a collection and only issue a single transaction once a specific threshold—either a count or a time duration—is met.
Consider this implementation pattern for a background processor:
public async Task ProcessEventsAsync(ChannelReader<Event> reader, CancellationToken ct)
{
var batch = new List<Event>(100);
await foreach (var item in reader.ReadAllAsync(ct))
{
batch.Add(item);
if (batch.Count >= 100) // Batch threshold
{
await PersistBatchAsync(batch);
batch.Clear();
}
}
}
private async Task PersistBatchAsync(List<Event> events)
{
using var transaction = await _dbContext.Database.BeginTransactionAsync();
await _dbContext.Events.AddRangeAsync(events);
await _dbContext.SaveChangesAsync();
await transaction.CommitAsync();
}
Balancing Throughput and Latency
When implementing this, you must choose your batch size carefully. A larger batch reduces the number of round-trips (improving write throughput), but increases the window of time that records remain in memory. If your worker crashes, any events currently sitting in the List are lost unless you have a robust Fault-Tolerant Message Replay mechanism in place to recover the uncommitted data from your message broker.
By grouping writes into discrete transactions, you reduce the overhead of log synchronization and significantly lower the pressure on database locks, allowing your background services to scale alongside your event stream.
Key Takeaway: Batching database commits transforms high-frequency, individual write operations into unified transactions, drastically reducing lock contention and increasing overall write throughput.