Data Connectors and Loading Documents
What You'll Learn
You'll learn how to use LlamaIndex's built-in loaders to pull data from various sources—files, APIs, databases, and the web—into your RAG pipeline.
The Core Idea
LlamaIndex provides data connectors (also called "readers" or "loaders") that handle the heavy lifting of ingesting documents from different sources. Instead of writing custom code for every file format or data source, you use pre-built loaders that know how to fetch and parse the content.
Think of data connectors as specialized delivery trucks: one truck knows how to pick up parcels from local warehouses (files), another from shipping ports (APIs), another from storage facilities (databases), and yet another from distribution centers (websites). Each truck handles the logistics so you just receive the packages (documents) ready to use.
Common Built-in Loaders
LlamaIndex offers dozens of connectors, including:
- SimpleDirectoryReader: Load all supported files from a folder
- PDFReader: Extract text from PDFs
- DatabaseReader: Query SQL databases
- NotionReader: Pull content from Notion pages
- GoogleDocsReader: Import Google Docs
- WebPageReader: Scrape web pages
- SlackReader: Extract Slack conversations
These loaders return Document objects (which you learned about in the previous lesson) that contain the text and metadata from your sources.
Why This Matters
Data connectors abstract away the complexity of different formats and protocols. You don't need to write PDF parsers, handle database connections, or manage API authentication separately for each source. The connectors handle extraction, and you focus on what to do with the data.
Key Takeaway: LlamaIndex's data connectors provide a unified interface to ingest documents from diverse sources—files, databases, APIs, and web—turning heterogeneous data into standardized Document objects ready for indexing and retrieval.