How to Crawl a Site's Internal Link Graph in 2026

Search engines understand a website partly through its internal link graph: which pages link to which, with what anchor text, and how authority flows between them. Yet most site owners have no map of their own link structure. They can’t answer “which pages point at our pricing page?” or “how many of our outbound links are nofollow?” without guessing. The answer is to crawl the whole site and export every link as a graph edge. This guide covers what an edge should record, how the crawl differs from a broken-link check, and the economics of mapping a large site.

What’s worth recording per edge

A link graph is a list of edges. Each edge connects a source page to a target URL and should carry enough metadata to analyze structure, anchor distribution, and link attributes:

• Source page — where the link lives.
• Target URL and hostname — where it points.
• Anchor text — the clickable text, central to anchor-distribution analysis.
• Link classification — internal, subdomain, or external. Authority flows differently across each.
• rel flags — parsed from the raw rel attribute into nofollow, sponsored, and ugc booleans. These govern whether link equity passes and whether you’re compliant with sponsorship disclosure rules.
• New-tab flag — whether target="_blank" is set.
• Timestamp.

This is deliberately not a status check. The link-graph crawler maps structure; it does not issue HTTP requests against each target to see if it resolves. That’s a different job (and a different tool). Skipping status verification is what lets it map links at very high volume.

How it differs from broken-link checking

The distinction matters because the two tools look superficially similar but optimize for opposite things:

• A broken-link checker discovers links and verifies each one with an HTTP request. It’s bounded by how fast targets respond, and it cares about status codes.
• A link-graph crawler discovers links and records their attributes without verifying them. It only parses HTML, so it’s bounded by parse speed — dramatically faster and higher-volume.

If you want to know what’s broken, use the checker. If you want to know how your site is structured and how anchors and rel flags are distributed, use the graph crawler. They answer different questions.

How the crawl works

Starting from one or more seed URLs, the crawler fetches each page, extracts every <a href>, and follows the internal ones to discover the rest of the domain. For each link it resolves relative URLs to absolute, classifies the target, parses the rel attribute into flags, deduplicates links per page (so a nav menu repeated on every page doesn’t drown the data), and emits one edge row. It’s pure HTTP HTML parsing — no browser, no status checks — which is exactly why it can map tens of thousands of edges quickly.

▶ Run the Website Link Graph Crawler — crawls a whole site and exports every internal and outbound link as a graph edge with anchor text, classification and nofollow/sponsored/ugc flags. No login, no browser.

Output schema

{
  "source_page": "https://example.com/blog/seo-basics",
  "target_url": "https://example.com/pricing",
  "target_host": "example.com",
  "anchor_text": "see our pricing",
  "link_class": "internal",
  "rel_nofollow": false,
  "rel_sponsored": false,
  "rel_ugc": false,
  "new_tab": false,
  "crawled_at": "2026-05-24T08:00:00Z"
}

With edges in this shape you can load them straight into a graph database or a notebook and compute the things that matter: in-degree per page (how many internal links point at it — a rough internal PageRank proxy), anchor-text distribution per target, and the full list of external domains you link out to with their rel attributes.

Use cases

• Internal-linking SEO audits. Find orphan pages (zero internal in-links), over-linked pages, and pages your money pages should link to but don’t.
• Anchor-text analysis. Audit whether your anchor distribution is natural or whether you’re over-optimizing exact-match anchors to one target.
• Outbound and backlink reporting. List every external domain you link to and whether those links are nofollow or sponsored — useful for partner reporting and link cleanup.
• Compliance. Affiliate and sponsored links are supposed to carry rel="sponsored" (or at least nofollow). Crawl to find paid links missing the right attribute before it becomes a problem.
• Site-structure visualization. Feed the edges into a graph viz to literally see your site’s shape and crawl-budget hotspots.

Cost math

Pay-per-event, small per-run start fee, zero per result. Link mapping is the highest-volume of the crawler family — every page has dozens of links, so edges multiply fast. A 3,000-page site can easily produce 60,000–100,000 edge rows.

• 3,000 pages, ~80,000 link edges.
• One run, results free.
• Cost is the Actor start plus HTTP parse compute — and because there are no status checks, the compute is lower than a broken-link run of the same site.

This is the canonical seed-in, explosion-out shape. The free-per-result model means a comprehensive 100K-edge graph costs the same as a tiny one, so you map the whole site instead of a sample.

Common pitfalls

• Nav/footer link noise. Site-wide navigation produces an edge from every page to every nav target. That’s technically correct but dominates the graph. Filter or weight it out before analyzing content links.
• Pagination and faceted URLs. Faceted navigation can generate huge numbers of near-duplicate target URLs. Decide whether to canonicalize them.
• No status info. Remember the graph crawler doesn’t tell you if a target is alive. A broken internal link still shows up as an edge. Pair with the broken-link checker if you need both.
• rel parsing edge cases. Some sites put multiple tokens in rel (nofollow noopener). Make sure your analysis reads the parsed flags, not the raw string.
• Subdomain scope. Decide up front whether subdomains count as internal. Misclassifying them skews the whole graph.

Wrapping up

You can’t optimize an internal-linking structure you can’t see. Crawl the graph once and you get a concrete map: orphans, hubs, anchor distribution, and every outbound link with its rel flags. With free per-result pricing, mapping a 100,000-edge site is no more expensive than mapping a blog, so there’s no reason not to capture the whole structure.

▶ Open the Website Link Graph Crawler on Apify — every internal and outbound link as a graph edge, with anchor text and rel flags. Start with Apify’s free monthly credit.