Database-like Indexing for Flat Files

Build simple search indexes on local binary files to enable fast data lookups without a database engine.

In this lesson, you will learn how to create a high-speed "lookup table" that allows you to jump directly to specific records in a large binary file without scanning the entire document.

The Problem: Sequential Scanning

Imagine you have a 2GB binary file containing thousands of user profiles. If you need to find "User ID: 500," reading the file byte-by-byte from the beginning is incredibly slow. This is effectively an $O(n)$ operation—the larger the file, the longer the wait.

The Solution: Building an Index

An index is essentially a separate, small file that acts as a "map." It maps a searchable key (like a User ID) to a specific byte offset in your large data file. By keeping this map in memory (or reading it quickly), you can use fs.read() with an explicit position argument to jump straight to the data.

How it works

Think of your index as an array of pairs: [Key, Offset].

// A simple in-memory index structure
// Maps UserID (key) to the byte position in our data file
const index = new Map();

// When writing a record to the file:
const offset = await getEndOfFilePosition();
await fs.promises.appendFile('data.bin', recordBuffer);
index.set(userId, offset);

// To retrieve a record later:
const position = index.get(targetUserId);
const buffer = Buffer.alloc(recordSize);
await fs.read(fileHandle, buffer, 0, recordSize, position);

By storing the offset (the integer representing how many bytes from the start of the file the record begins), you skip the "scan" entirely. When you perform the read, the operating system jumps directly to that memory address in the file, making retrieval nearly instantaneous, regardless of how large the total dataset is.

Managing the Index

For massive files, you can store this index in a secondary "index file." You would store the Key (e.g., a 4-byte integer) followed by the Offset (e.g., an 8-byte integer) sequentially. You can then use binary search on this index file to find the offset for any key, keeping memory usage constant even if the main data file is gigabytes in size.

Key Takeaway: By decoupling your data from its location and mapping IDs to specific byte offsets, you transform slow sequential file scanning into near-instantaneous direct access.