GitHub Data Without the GraphQL Rate Limit, at Scale

GitHub’s API is excellent — and for bulk repository intelligence it runs out of headroom almost immediately. The REST API caps unauthenticated requests at 60 per hour per IP. Authenticate and you jump to 5,000 requests per hour, which sounds generous until you remember each repo you actually care about needs several calls (repo, languages, contributors, releases, topics). The GraphQL API trades the request count for a points budget — also effectively 5,000 points per hour — where a single nested query over stargazers or contributors can cost dozens of points each. Try to profile ten thousand repositories in a sitting and you’ll spend most of your wall-clock time sleeping on X-RateLimit-Reset. This guide is about getting GitHub data without the GraphQL rate limit being your bottleneck, by complementing the API with GitHub’s public HTML and search surfaces.

Where the official limits actually bite

It’s worth being precise, because the limits are reasonable for app integrations and brutal for bulk intelligence:

• REST, unauthenticated: 60 requests/hour/IP. Enough to look at one repo. Useless for a crawl.
• REST, authenticated (PAT or App): 5,000 requests/hour. But a “full” repo profile is 4–6 calls, so that’s ~1,000 repos/hour, before search (which has its own tight 30 req/min secondary limit).
• GraphQL: ~5,000 points/hour. Nested connections (issues → comments, repo → stargazers → user) multiply cost. A few rich queries can drain the budget.
• Secondary rate limits: GitHub also throttles bursts and concurrent requests independently, returning 403s with Retry-After even when your primary budget isn’t exhausted.

For monitoring a handful of repos, none of this matters. For mapping an ecosystem — every repo with a given topic, or every fork of a popular project — you hit the ceiling fast.

What the public surface gives you

GitHub’s website is server-rendered and rich. For read-only intelligence, the public pages and the search UI expose most of what the API does, on a different (and looser) budget:

• Repo landing page (https://github.com/{owner}/{repo}) — name, description, topics, star/fork/watcher counts, primary language, the language breakdown bar, license, latest release tag, last-commit recency, and the README.
• About sidebar — homepage URL, topics, “Used by” count, contributor count and avatars.
• /{owner}/{repo}/forks and /stargazers — paginated lists for popularity-graph work.
• Search UI (https://github.com/search?q=...&type=repositories) — the same query language as the API (stars:>1000 language:rust topic:cli), paginated, without consuming your REST/GraphQL budget.
• /{owner}/{repo}/releases and /tags — version history and release notes as HTML.

There are also genuinely public JSON niches worth knowing — for example the unauthenticated https://api.github.com/repos/{owner}/{repo} works but counts against the 60/hr limit, and raw README content is fetchable from raw.githubusercontent.com without touching the API at all.

A hybrid strategy that respects everyone

The right architecture isn’t “abandon the API” — it’s to use each surface where it’s cheapest:

1. Discover with search HTML. Find candidate repos via the search UI so discovery doesn’t burn API budget.
2. Profile with public pages. Pull stars, language breakdown, topics, license, and last-activity from the repo landing page.
3. Reserve the authenticated API for the few fields only it returns cleanly (exact commit history, contributor stats) and only for the repos that survive your filter.

This keeps the expensive, rate-limited calls for a small, high-value subset and gets the breadth from HTML.

▶ Try the GitHub Repository Scraper on Apify — bulk repo intelligence from public pages and search, so the GraphQL points budget stays free for what only the API can do. No auth required.

Living with the limits

Even off-API, GitHub will throttle a single IP hammering thousands of pages a minute. Sane defaults:

1. Cap a single IP around 1–2 requests/second for HTML pages; you’ll stay well clear of bot challenges.
2. Honor Retry-After whenever you do touch the API — both primary and secondary limits use it.
3. Cache aggressively. Repo metadata changes slowly. Re-crawling a repo’s stars every hour is wasteful; daily is plenty for most intelligence.
4. Rotate IPs for true scale. Tens of thousands of repos a day needs a pool; a few thousand fits one address comfortably.

A clean output schema

One row per repository:

{
  "full_name": "rust-lang/rust",
  "owner": "rust-lang",
  "name": "rust",
  "description": "Empowering everyone to build reliable and efficient software.",
  "url": "https://github.com/rust-lang/rust",
  "homepage": "https://www.rust-lang.org",
  "primary_language": "Rust",
  "languages": { "Rust": 92.1, "C": 3.4, "Shell": 2.0 },
  "topics": ["compiler", "language", "rust"],
  "stars": 98231,
  "forks": 12740,
  "watchers": 1450,
  "open_issues": 9821,
  "used_by": 412300,
  "contributors_count": 5100,
  "license": "MIT OR Apache-2.0",
  "default_branch": "master",
  "latest_release": "1.89.0",
  "last_commit_at": "2026-06-04T09:12:00Z",
  "created_at": "2010-06-16T20:39:03Z",
  "archived": false,
  "scraped_at": "2026-06-05T12:00:00Z"
}

Use full_name (owner/repo) as the natural key — it’s GitHub’s own canonical identifier and survives renames better than a numeric ID in practice.

Use cases

• Ecosystem mapping — every repo with a topic or language, ranked by stars and recency, to size a community.
• Developer-tool lead gen — find repos using your category of tool (via dependency/topic search) and surface the maintainers.
• OSS due diligence — before adopting a dependency, pull bus-factor signals: contributor count, last-commit recency, archived flag, issue backlog.
• Trend detection — track star velocity across a watchlist to catch projects breaking out before they trend.
• Competitive monitoring — watch a rival’s repos for new releases, language shifts, or sudden fork spikes.

Build it yourself vs. a managed actor

A single-repo scraper is trivial. Scale is where it gets real: search-result pagination, the language-bar percentages living in a title attribute, “Used by” counts hidden behind a link, fork/stargazer lists that paginate forever, and GitHub’s secondary rate limiter quietly 403-ing your bursts. Stitching the HTML breadth to the occasional authenticated API call — and keeping that within the GraphQL points budget — is exactly the orchestration a managed GitHub repository scraper handles for you.

Common pitfalls

• Counting on the 60/hr unauth API. It’s a trap for crawls — one repo and you’re done for the hour. Use HTML for breadth.
• GraphQL point math. Nested connections cost more than you expect. Measure rateLimit { cost remaining } in your query before scaling it.
• Secondary-limit 403s. These hit even with budget left. Reduce concurrency, don’t just retry.
• Stale star counts from caching. If velocity is your signal, don’t over-cache the popularity fields.
• Renames and redirects. Repos move; owner/repo 301-redirects. Follow redirects and update your key.

Wrapping up

GitHub’s API limits — 60 requests/hour unauthenticated, a ~5,000-point GraphQL budget authenticated, plus secondary burst throttling — are fine for an integration and far too tight for bulk repo intelligence. The fix isn’t to abandon the API; it’s to get the breadth from GitHub’s public pages and search, and spend the rate-limited calls only where they’re irreplaceable. That’s how you get GitHub data without the GraphQL rate limit dictating your throughput. A managed actor wires the two surfaces together so you don’t have to.

▶ Open the GitHub Repository Scraper on Apify — repo metadata, language breakdowns, stars, and activity at scale without draining your API budget. Pay per repository.