How to Scrape ClinicalTrials.gov Trial Data in 2026

ClinicalTrials.gov is the world’s largest public registry of clinical studies — more than 585,000 trials — and unlike most data sources covered here, it wants to be queried. The US National Library of Medicine runs an official, well-documented REST API with no login, no API key and no blocking. So why scrape it at all? Because the API is paginated, the study records are deeply nested, and turning “every Phase 3 oncology trial sponsored by a top-20 pharma, recruiting, in the last two years” into a clean tabular dataset is real work. This guide covers what’s in a trial record, how the API is structured, and how to build a maintainable feed.

What’s worth extracting

A single study record on ClinicalTrials.gov is enormous — the full protocol can run to hundreds of fields. For most analysis you want a focused, trial-level subset:

• Identity — NCT registration ID, official and brief titles.
• Sponsor — lead sponsor name and sponsor classification (industry, NIH, academic, other).
• Study design — study type (interventional / observational) and phase (1, 2, 3, 4, or N/A).
• Conditions — the diseases or conditions studied (often multiple).
• Interventions — drugs, devices, procedures or behavioral interventions being tested.
• Enrollment — target or actual participant count.
• Eligibility — sex, minimum/maximum age, and healthy-volunteer eligibility.
• Timeline dates — study start, primary completion, overall completion, first-posted and last-update dates.
• Locations — an array of trial sites (facility, city, country) — critical for patient-recruitment use cases.
• Results availability — whether results have been posted.
• Status — recruiting, active not recruiting, completed, terminated, withdrawn, etc.
• Direct study link and a scrape timestamp.

The sponsor classification plus phase plus condition combination is what powers competitive-intelligence work; the locations array is what powers patient recruitment.

How the API is exposed

This is the official ClinicalTrials.gov API v2, served as JSON. The query endpoint is:

GET https://clinicaltrials.gov/api/v2/studies
  ?query.cond=lung+cancer        # condition
  &query.term=immunotherapy      # free-text term
  &query.spons=Pfizer            # lead sponsor
  &filter.overallStatus=RECRUITING
  &pageSize=100
  &pageToken=<token>

A few practical realities of the v2 API:

• It’s token-paginated. Each response carries a nextPageToken; you follow tokens until there isn’t one. Get the loop wrong and you either miss studies or fetch duplicates.
• Records are deeply nested. Sponsor, eligibility, design and locations all live in nested protocol modules. Flattening them into one row per trial is the bulk of the parsing work.
• No auth, no captcha, no blocking. This is a genuinely open government API. The only constraint is being polite — pace requests and back off on the rare 429 so you don’t hammer a public resource.
• It’s huge. A broad query can match tens of thousands of studies. You need a sensible page size and a way to cap or filter so a run completes cleanly.

Because there’s no anti-bot layer, the engineering challenge is entirely in pagination, nested-field flattening and filter mapping — not evasion.

▶ Run the ClinicalTrials.gov Scraper — filter by condition, free-text term, sponsor and status, then pull tens of thousands of flattened trial records per run from the official API. No login, no API key, no blocking.

Build it yourself vs. use a managed scraper

This is one of the more reasonable APIs to query yourself — it’s open and documented. The case for a managed scraper is about the parsing and the plumbing, not access:

• Building from scratch — implement token pagination correctly, map the filter UI to query params, and flatten the nested protocol modules into stable columns. The flattening is the time sink: locations are an array, eligibility is a sub-object, dates are scattered across modules.
• Using a managed actor — set your filters, run it, get flat rows. The pagination loop and the nested-to-tabular flattening are already done and kept in sync with the v2 schema.

For a recurring feed — say, a weekly refresh of every trial in your therapeutic area — the managed path saves you from re-deriving the flattening logic and from re-checking it every time NLM tweaks the schema.

Schema design for downstream use

A flattened, analysis-ready per-trial record:

{
  "nct_id": "NCT05123456",
  "brief_title": "Study of Drug X in Advanced NSCLC",
  "lead_sponsor": "Acme Oncology",
  "sponsor_class": "INDUSTRY",
  "study_type": "INTERVENTIONAL",
  "phase": "PHASE3",
  "overall_status": "RECRUITING",
  "conditions": ["Non-Small Cell Lung Cancer", "NSCLC"],
  "interventions": ["Drug X", "Pembrolizumab"],
  "enrollment": 480,
  "sex": "ALL",
  "min_age_years": 18,
  "max_age_years": null,
  "healthy_volunteers": false,
  "start_date": "2026-01-15",
  "primary_completion_date": "2027-09-30",
  "locations": [
    { "facility": "Memorial Cancer Center", "city": "Houston", "country": "United States" }
  ],
  "has_results": false,
  "url": "https://clinicaltrials.gov/study/NCT05123456",
  "scraped_at": "2026-05-23T08:00:00Z"
}

Schema choices worth making:

• Keep conditions, interventions and locations as arrays. A trial routinely has several of each; flattening them to a single string destroys the analysis.
• Make max_age_years nullable. Many trials set no upper age bound; null is correct.
• Store sponsor_class separately from sponsor name — industry-vs-academic split is a core competitive-intelligence dimension.
• Keep the NCT ID as the join key. Titles change between updates; the NCT ID is permanent.
• Always store scraped_at. Statuses and dates change as trials progress — you need to know when a row was valid.

Typical use cases

• Pharma and biotech competitive intelligence — track competitor trials by phase, sponsor and condition; spot when a rival moves a program into Phase 3.
• CROs and patient recruitment — find recruiting trials by condition and location at scale using the locations array.
• Investors and analysts — monitor the clinical pipeline across the industry to inform diligence and theses.
• Researchers and academics — build datasets from the largest trials registry for meta-analysis and trend work.
• Journalists and policy teams — analyze trial trends, sponsor activity and enrollment patterns.

The common thread: the value is in structured, filterable, refreshable access. The raw API has everything, but it’s nested JSON behind a pagination token — not a dataset you can query in a notebook until someone flattens it.

Cost math

The scraper is pay-per-event with the result event priced at zero — you pay effectively for the run, not per study. A run pulling tens of thousands of flattened trial records is cents of compute. Scheduling a weekly refresh of your therapeutic area lands in the low single digits per month.

Since the underlying API is free, self-hosting has no data cost either — but you carry the pagination, flattening and schema-maintenance work, plus a place to run and schedule it. For a recurring, clean feed, the managed actor removes that for negligible cost.

Common pitfalls

• Pagination bugs. Token pagination is easy to get subtly wrong — dropped tokens skip studies, reused tokens duplicate them. Validate your counts against the API’s total.
• Flattening arrays to strings. Conditions, interventions and locations are inherently multi-valued. Keep them as arrays or you can’t do location- or condition-level analysis.
• Treating status as static. Trials move from recruiting to active to completed. A snapshot is a point in time — store the timestamp and refresh.
• Ignoring sponsor class. “Sponsor name” alone hides the industry-vs-academic split that most analysis depends on.
• Over-broad queries. Matching tens of thousands of studies with no filter wastes time. Use condition, sponsor and status filters to scope runs.
• Confusing primary vs. overall completion dates. They mean different things in trial timelines — keep both, labeled.

Wrapping up

ClinicalTrials.gov is a rare friendly source: an open, documented government API with no anti-bot wall. The work isn’t access — it’s correct token pagination and flattening deeply nested protocol records into stable, array-aware columns, on a schedule. For a one-off pull in your therapeutic area, the API is approachable. For a recurring, analysis-ready feed across phases, sponsors and conditions, a managed actor hands you flat rows and keeps the flattening in sync with the v2 schema.

▶ Open the ClinicalTrials.gov Scraper on Apify — 585K+ studies, filter by condition, sponsor and status, flattened to tabular rows. No API key, no blocking. Start on Apify’s free monthly credit.