Testing Bulk Data Import Logic

In this lesson, you will learn how to verify that your Node.js bulk import systems handle partial failures gracefully without corrupting your database state.

The Challenge of Bulk Imports

When importing thousands of records, it is rarely an "all or nothing" operation. Whether due to malformed data in a CSV or a transient database connection timeout, you must decide if your logic should halt immediately or attempt to commit valid records while flagging the invalid ones. Testing this requires simulating partial commit behavior to ensure your system behaves predictably under duress.

Verifying Error Handling

To test this, you need to mock your database repository or persistence layer to inject failure at specific indices. You are looking to confirm that:

1. Valid items are saved.
2. Invalid items trigger the appropriate error logs or "dead letter" storage.
3. The transaction context (if using one) is handled correctly.

Consider a scenario where you import an array of objects. Your test should verify the state of the database after an error occurs mid-stream:

it('should commit valid records and store errors for invalid ones', async () => {
  const bulkData = [{ id: 1, val: 'ok' }, { id: 2, val: 'bad' }, { id: 3, val: 'ok' }];
  
  // Mock the DB to fail only on id: 2
  db.save.mockImplementation(async (item) => {
    if (item.id === 2) throw new Error('Validation failed');
    return { success: true };
  });

  const results = await importService.bulkProcess(bulkData);

  expect(results.processedCount).toBe(2);
  expect(results.errors).toHaveLength(1);
  expect(results.errors[0].id).toBe(2);
});

Ensuring Partial Commit Integrity

If your logic uses a transactional approach (e.g., BEGIN and COMMIT), ensure that a failure in the middle of a batch doesn't accidentally roll back the entire session if the design intent is "partial success." Conversely, if you are performing row-by-row inserts, verify that the operation does not crash the entire worker process when an exception is thrown. Use assertions to check the total count in the database to ensure you haven't introduced duplicate entries upon retrying the import.

Key Takeaway: Always test your import logic by simulating mid-stream failures to confirm that valid data persists while erroneous data is captured, preventing both data loss and silent processing failures.