Crawling and Application Discovery
What You'll Learn
You'll learn how DAST tools automatically map out a web application's structure to identify all the places attackers might strike.
The Exploration Challenge
Before a DAST scanner can test for vulnerabilities, it needs to know what exists. Imagine trying to secure a building without knowing how many doors, windows, and entry points it has. Application discovery through crawling solves this problem by systematically exploring your web application.
How Crawling Works
A DAST crawler acts like an automated user browsing your site:
- Starts at entry points – Usually your homepage or login page
- Follows links – Clicks every
<a>tag, button, and navigation element it finds - Submits forms – Identifies input fields (search boxes, login forms, comment fields)
- Maps API endpoints – Discovers AJAX calls and REST endpoints through JavaScript analysis
- Records the attack surface – Builds a complete map of URLs, parameters, and inputs
The crawler builds what's called the attack surface: every place where user input enters your application and every endpoint that responds to requests.
Discovery Techniques
Modern crawlers use multiple strategies:
- Traditional spidering: Following HTML links recursively
- JavaScript rendering: Executing JavaScript to find dynamic content (Single Page Applications)
- Form field detection: Identifying every input, textarea, and file upload
- Parameter discovery: Finding URL parameters and hidden form fields
- API documentation parsing: Reading OpenAPI/Swagger specs if available
Why This Matters
Without thorough discovery, DAST tools miss vulnerabilities. A crawler that can't find your API endpoints won't test them. A crawler that ignores JavaScript-rendered content in your React app leaves blind spots an attacker could exploit.
Key Takeaway: DAST crawling automatically discovers and maps your application's attack surface—all endpoints, forms, and inputs—so security testing covers everywhere attackers might probe.