Documenting Data Sources
You'll learn how to record the essential origin details of your data so you and others can trust, reproduce, and audit your analyses.
What Does It Mean to Document Data Sources?
Documenting data sources means creating a clear, written record of where your data came from, when you obtained it, how you accessed it, and what permissions apply. Think of it as writing a "birth certificate" for your dataset—without it, you have no proof of legitimacy or way to trace back if something goes wrong.
Why These Details Matter
When you pull data from an API, download a CSV from a partner organization, or query a production database, that moment in time matters. The data might change tomorrow. The API endpoint might move. The file might be updated. Without documentation, you can't reproduce your work or explain discrepancies that emerge later.
What to Record
For every data source, capture:
- Origin: Database name, URL, file path, API endpoint, or vendor name
- Extraction timestamp: Exact date and time you pulled the data
- Access method: How you retrieved it (SQL query, API call, manual download, automated script)
- Permissions and licensing: Who authorized access? Any usage restrictions? Legal terms?
For example: "Customer transactions extracted from prod_db.orders on 2024-01-15 at 14:30 UTC via SQL query saved in queries/extract_orders.sql. Access granted by Data Governance team ticket #4521. Data classified as PII—internal use only."
The Practical Impact
Good documentation enables you to answer critical questions: Can we refresh this data? Who do we contact if access breaks? Can we share this externally? It's the foundation for reproducible, trustworthy data science.
Key Takeaway: Always document where data came from, when it was extracted, how it was accessed, and what permissions govern its use—this ensures reproducibility and accountability.