How to Find Every Broken Link on a Website in 2026

Broken links are the quietest form of SEO rot. A page that returned 200 last quarter quietly starts returning 404 after a CMS migration, a deleted product, or an expired external resource — and nothing tells you. Visitors hit dead ends, crawl budget gets wasted on error pages, and link equity drains into nowhere. The only way to catch it is to crawl the entire site and verify every link. This guide walks through how a site-wide broken-link crawler actually works in 2026, what it should record per link, and how the per-result economics shake out.

What’s worth recording per link

A broken-link check is only useful if it tells you where the problem is and what kind of problem it is. For every link discovered on the site, you want a row that captures:

• Status code — the exact HTTP response: 200, 301, 404, 500, 503. The code is the whole point.
• Network errors — not every failure is an HTTP code. Timeouts, DNS failures, and unreachable hosts need their own classification.
• Source page — which page the link was found on. Without this, a list of broken URLs is useless; you can’t fix what you can’t locate.
• Anchor text — the clickable text, so you know which link on a busy page to edit.
• Link type — internal vs. external. Internal breaks are yours to fix; external breaks are vendors disappearing under you.
• Timestamp — when the check ran, so scheduled runs become a time series.

That combination — source page plus anchor text plus status — is what turns a raw error list into an actionable fix queue.

How a link-checking crawl actually works

The crawl has two distinct phases that people often conflate. First, discovery: the crawler starts from one or more seed URLs, fetches each page’s HTML, extracts every <a href>, and follows the internal ones to discover more pages. It stays domain-scoped so it doesn’t wander off into the entire internet. Second, verification: every unique link it found — internal and external — gets an HTTP request to confirm it resolves.

The efficiency tricks matter at scale:

• HEAD with GET fallback. A HEAD request asks only for headers, not the body, so it’s cheap. But plenty of servers don’t implement HEAD correctly, so the crawler falls back to GET when HEAD misbehaves.
• Check each URL once. A link to your homepage might appear on all 5,000 pages. You check it once and reuse the cached result everywhere — otherwise you’d issue 5,000 identical requests.
• Skip assets when following. Images, CSS, and JS get checked as links but aren’t crawled into — there’s no point parsing a .png for more links.
• Pure HTTP, no browser. No headless Chrome means thousands of pages per run without the memory and time cost of rendering.

This is a Crawlee-style HTTP crawler, not a browser farm. That’s what makes it fast and cheap enough to run across a large site.

▶ Run the Broken Link Checker — crawls your whole domain, follows internal links, and verifies every internal and external URL. Returns source page, anchor text, status code, and link type per row. Schedulable for ongoing monitoring.

Concurrency and politeness

Speed comes from concurrency, but concurrency is also how you accidentally DDoS your own site (or someone else’s). A few hundred parallel requests against a small WordPress box will trip rate limits or knock it over. The right move is to cap concurrency to a level the target can absorb, and to let the crawler dedupe aggressively so you’re not hammering the same external host repeatedly.

For external links especially, be conservative — a third-party server returning 429 Too Many Requests will look like a “broken” link in your report when it’s really just your crawler being rude. Lower concurrency and result caching solve most of this for you.

Output schema

One row per link keeps downstream filtering simple. A clean record looks like:

{
  "url": "https://example.com/old-product-page",
  "source_page": "https://example.com/blog/2024-roundup",
  "anchor_text": "our 2024 best-seller",
  "link_type": "internal",
  "status_code": 404,
  "error": null,
  "checked_at": "2026-05-20T09:14:00Z"
}

For monitoring you’ll typically run in one of two modes: a full link audit (every link with its status, useful for a one-time inventory) or a broken-only filter (only the rows where status_code >= 400 or error is set, useful for a recurring alert). The second mode keeps your dataset small and your attention focused.

Use cases

What teams actually do with this:

• SEO audits — broken internal links waste crawl budget and bleed link equity. Finding and fixing them is table-stakes technical SEO.
• Migrations and redesigns — run a baseline crawl before launch, run it again after, and diff the two. Any new 404 is a redirect you forgot.
• Docs and knowledge-base maintenance — large documentation sites accumulate dead anchors and moved pages constantly. A scheduled weekly crawl catches them.
• Affiliate and outbound monitoring — affiliate links expire, partners shut down, and your “recommended tools” page slowly fills with dead ends that cost you commission.
• Scheduled monitoring — the real value is recurring. Set it to run nightly and you find breakage within a day instead of a quarter.

Cost math

The actor is priced pay-per-event: a tiny per-run start fee and zero per result. That economics is unusual and worth dwelling on, because broken-link checking is inherently high-volume — a 5,000-page site can easily surface tens of thousands of links once you count every external reference.

A typical mid-size site audit:

• 5,000 pages crawled, ~40,000 unique links verified.
• One run, results free.
• Effectively the cost of a single Actor start plus the compute, which for an HTTP-only crawl is minimal.

Compare that to a SaaS broken-link tool that charges per page or per seat, and the difference at scale is dramatic. The high-volume nature — seed URL in, tens of thousands of verified links out — is exactly where this model wins. You’re not penalized for having a big site.

Common pitfalls

• Soft 404s. Some sites return 200 with a “page not found” body. A status-code check can’t catch these; you’d need content inspection. Know the limitation.
• External rate limits. As noted, 429 and 403 from third parties can masquerade as broken links. Re-check flagged externals before treating them as dead.
• Auth-gated pages. Anything behind a login will return 401/403 to an unauthenticated crawler. Scope your crawl to public pages or expect noise.
• Redirect chains. A 301 → 301 → 200 link technically works but is slow and wasteful. Decide whether you want to flag long chains as issues.
• Infinite URL spaces. Calendars, faceted search, and session IDs can generate near-infinite URLs. Cap crawl scope so you don’t crawl forever.

Crawl-to-discover vs. check-a-list

Two related jobs people mix up:

• Broken link checker (this actor) — you give it a seed URL, it crawls the site to discover links, then verifies them. Use it when you don’t have the list and want site-wide coverage including source page and anchor text.
• Bulk URL checker — you already have a list of URLs (a sitemap export, a backlink file) and just want each one’s status and redirect chain. No crawling.

If you have the URLs, check the list; if you need to find them, crawl.

Is there a broken link checker API?

There’s no standalone REST API for broken-link data — the crawl is the product. But running this actor via Apify’s API gives you the same thing: kick off a crawl programmatically, poll for completion, and pull the results as JSON. That’s an API-shaped broken-link check you can wire into CI or a scheduled job, without hosting a crawler yourself. For a plain list of URLs rather than a crawl, the bulk URL checker is the lighter call.

Wrapping up

If you just need a one-off check before a launch, run it once and fix the red rows. If you maintain a large site that changes constantly, schedule it — broken links are a chronic condition, not a one-time bug, and the only cure is regular crawling. With free per-result pricing, there’s no reason to ration how often you look.

▶ Open the Broken Link Checker on Apify — HTTP-fast, no browser, schedulable. Get a full link audit or a broken-only list. Start with Apify’s free monthly credit.