Dataset Discovery
When to Use
- User asks "find me data about X" or "where can I get data on Y"
- User wants to analyze a relationship between variables
- User needs specific study designs (longitudinal, cross-sectional, experimental)
- User asks about specific surveys or cohorts
Step 1: Understand What the Research Question Requires
Before searching, determine the minimum data requirements:
Study design needed:
- "Does X predict CHANGES in Y over time?" → longitudinal (same people measured repeatedly). Cross-sectional data CANNOT answer this — don't settle for it.
- "Is X associated with Y?" → cross-sectional is sufficient (one-time measurement)
- "Does intervention X cause outcome Y?" → experimental (clinical trial with controls)
- "What genes/proteins are involved in X?" → omics (sequencing, expression, proteomics)
Variables needed:
- List the specific exposure, outcome, and confounder variables
- For each variable, note the measurement type (continuous, categorical, biomarker vs self-report)
- Identify minimum confounders needed (age, sex are almost always required; domain-specific confounders depend on the question)
Population needed:
- Age range, geography, clinical status, sample size requirements
- Power analysis: to detect a small effect (r=0.1), you need ~800 subjects at 80% power
Step 2: Search Strategy
Search from broadest to most specific. Use find_tools to discover available dataset search tools — don't rely on memorized tool names.
Layer 1 — Cross-repository search (cast wide net): Search tools that index datasets across thousands of repositories. These find datasets you didn't know existed.
- Search by: research topic keywords, variable names, population descriptors
- Look for: DOI-registered datasets, repository listings, government data portals
Layer 2 — Domain-specific repositories: Search repositories specialized for your data type.
- Health surveys: CDC, NHANES (search by variable name, not topic keywords)
- Genomics: SRA, ENA, ArrayExpress, GEO
- Proteomics: PRIDE, MassIVE
- Metabolomics: MetaboLights, Metabolomics Workbench
- Clinical: ClinicalTrials.gov (for trial data with results)
Layer 3 — Literature-based discovery: Many datasets aren't in any repository — they're described in paper methods sections.
- Search PubMed/EuropePMC for papers that analyzed the relationship you're interested in
- Read their methods: "We used data from [DATASET NAME]" tells you exactly what exists
- Check supplementary materials for deposited data (GEO/SRA accession numbers)
- This is often the MOST effective strategy for finding niche datasets
Step 3: Evaluate Dataset Fitness
For each candidate dataset, assess these dimensions:
Variables:
- Does it contain your SPECIFIC exposure and outcome variables?
- Are they measured the way you need? (biomarker vs self-report, continuous vs categorical)
- Are key confounders available? (missing confounders = biased analysis)
Design match:
- If you need longitudinal: does it follow the SAME individuals over time? How many waves? What's the follow-up interval?
- Beware: "repeated cross-sections" (different people each wave) are NOT longitudinal
- If you need experimental: is there a proper control group? Randomization?
Sample:
- Is the sample large enough for your analysis? (logistic regression needs ~10 events per predictor)
- Does the population match? (age range, geography, clinical characteristics)
- Are there subgroups you need? (stratified by sex, race, disease status)
Access:
- Publicly downloadable (best) vs registration required (days) vs collaboration agreement (months) vs restricted (may be impossible)
- Data format: CSV/TSV (easy), XPT/SAS (need conversion), proprietary database (may need special software)
Quality:
- Is it from a well-known study with published methods? (NHANES, HRS, UK Biobank = high quality)
- Has it been used in peer-reviewed publications? (indicates data is usable)
- What's the response rate / missingness pattern?
Step 4: Download and Analyze
Don't stop at finding datasets — download and analyze them. Write and run Python code via Bash. Never describe what you "would do" — execute it.
Data Loading Cookbook
Choose the loader that matches your data source. When unsure of the format, download a small sample first and inspect.
import requests, io, pandas as pd
# --- Tabular files (most common) ---
df = pd.read_csv("data.csv") # CSV / TSV (use sep="\t" for TSV)
df = pd.read_excel("data.xlsx") # Excel
df = pd.read_stata("data.dta") # Stata
df = pd.read_sas("data.xpt", format="xport") # SAS transport (XPT)
df = pd.read_sas("data.sas7bdat", format="sas7bdat") # SAS native
df = pd.read_parquet("data.parquet") # Parquet
df = pd.read_json("data.json") # JSON (records or columnar)
df = pd.read_fwf("data.dat") # Fixed-width (some legacy surveys)
# --- Download from URL first, then parse ---
resp = requests.get(url, timeout=120)
content = resp.content
# Detect format from URL or content header
if url.endswith(".XPT") or url.endswith(".xpt"):
df = pd.read_sas(io.BytesIO(content), format="xport")
elif url.endswith(".csv") or url.endswith(".csv.gz"):
df = pd.read_csv(io.BytesIO(content))
elif url.endswith(".tsv") or url.endswith(".tsv.gz"):
df = pd.read_csv(io.BytesIO(content), sep="\t")
elif url.endswith(".json"):
df = pd.read_json(io.BytesIO(content))
else:
# Try CSV first, then inspect
df = pd.read_csv(io.BytesIO(content))
# --- REST API pagination (common for GDC, ClinicalTrials.gov, etc.) ---
import json
all_records = []
offset = 0
while True:
resp = requests.get(f"{api_url}?offset={offset}&limit=100", timeout=30)
batch = resp.json().get("data", [])
if not batch:
break
all_records.extend(batch)
offset += len(batch)
df = pd.DataFrame(all_records)Merge, Clean, Analyze
# Merge multiple files on participant/sample ID
merged = df1.merge(df2, on="id_col", how="inner")
# Filter population
subset = merged[(merged["age"] >= 60) & (merged["age"] <= 80)].copy()
# Handle missing values
missing_pct = subset.isnull().mean() * 100
print("Missing % per variable:\n", missing_pct[missing_pct > 0].sort_values(ascending=False))
subset = subset.dropna(subset=["exposure_var", "outcome_var"])
# Quick regression
import statsmodels.formula.api as smf
model = smf.ols("outcome ~ exposure + age + sex", data=subset).fit()
print(model.summary())
# Visualization
import matplotlib.pyplot as plt
plt.scatter(subset["exposure"], subset["outcome"], alpha=0.3)
plt.xlabel("Exposure"); plt.ylabel("Outcome")
plt.savefig("/tmp/scatter.png", dpi=150, bbox_inches="tight")Always run the code and report actual numbers (β, p-value, CI, N).
Step 5: Report Honestly
Structure the report as:
- Best available dataset — name, what it contains, access method, key limitation
- Analysis results — actual statistics (β, p-value, CI, N) from running the code
- Alternative datasets — ranked by fitness, with tradeoffs
- What CANNOT be answered — if no dataset matches the study design needed, say so clearly
- Recommended next steps — apply for access to longitudinal data, replicate in other cohorts
Critical honesty rules:
- Never claim a dataset answers a temporal question if it's cross-sectional
- Distinguish "data exists but needs registration" from "data doesn't exist"
- Report actual computed statistics, not hypothetical analyses
- State the strongest analysis possible with available data, even if it's weaker than what was asked
LOOK UP, DON'T GUESS
Never assume a dataset exists — search for it. Never assume access is public — check. Never assume variables are measured the way you need — verify the codebook.