Course contentsShow
← System Design
Lesson 1859 of 1,91944. Real-World System: Web CrawlerFree lesson

Handling DNS Failures and Timeouts

Implement retries with exponential backoff and fallback DNS servers to handle resolution failures gracefully.

Handling DNS Failures and Timeouts

What you'll learn: How to gracefully handle DNS resolution failures and timeouts in a web crawler using retry strategies and fallback mechanisms.

When your crawler resolves thousands of domain names per second, DNS failures are inevitable—servers go down, networks glitch, or queries simply time out. Without proper handling, these failures would halt your crawl, wasting resources and leaving URLs unprocessed.

Why DNS Failures Matter

DNS resolution is a critical dependency. If a single DNS server fails or responds slowly, it creates a bottleneck. Given your crawler's scale (potentially millions of URLs), even a 1% failure rate means thousands of unresolved domains.

Retry with Exponential Backoff

When a DNS query fails, don't immediately give up. Implement exponential backoff: retry after 1 second, then 2, then 4, doubling the wait time each attempt. This pattern gives transient issues time to resolve without hammering a struggling server.

Think of it like knocking on a door—if no one answers immediately, you wait a bit longer before knocking again, not pounding continuously.

# Conceptual approach
max_retries = 3
base_delay = 1  # second

for attempt in range(max_retries):
    result = dns_resolve(domain)
    if result.success:
        return result.ip
    wait_time = base_delay * (2 ** attempt)
    sleep(wait_time)

Fallback DNS Servers

Configure multiple DNS servers (primary, secondary, tertiary). If your primary resolver fails or times out, immediately switch to the next one. Use public DNS services like Google (8.8.8.8) or Cloudflare (1.1.1.1) as fallbacks.

Set aggressive timeouts (2-5 seconds) per server to fail fast and move to the next option rather than waiting indefinitely.

Key Takeaway: Use exponential backoff retries for transient DNS failures and maintain multiple fallback DNS servers to ensure your crawler remains resilient against resolution infrastructure problems.