How to Scrape Hacker News Stories and Comments in 2026

Hacker News is one of the richest signal sources on the open web for tech trends, product launches, and developer sentiment — and unlike almost every other social platform, it doesn’t fight you. There’s no headless browser, no proxy arms race, no rotating challenge page. The catch is more subtle: the official search backend caps any single query, so naive pagination silently stops returning rows long before you’ve got the full dataset. This guide covers what HN actually exposes, the result-cap problem, and how to pull clean story and comment data at scale in 2026.

The Algolia reality

Hacker News doesn’t run its own search infrastructure — it hands that off to Algolia. The public, key-free endpoints are:

https://hn.algolia.com/api/v1/search          # relevance-ranked
https://hn.algolia.com/api/v1/search_by_date  # newest-first

Both accept a query string, a tags filter (story, comment, poll, show_hn, ask_hn, front_page), and numeric filters like points>100 or created_at_i>.... No authentication, no rate-limit headers worth worrying about under polite use. This is genuinely one of the friendliest data sources on the internet.

So why use a managed scraper at all? Two reasons: the result cap, and schema normalization.

The result-cap problem nobody mentions

Algolia paginates with page and hitsPerPage, but it will not return results past roughly 1,000 hits per query (50 pages × 20, or fewer pages at larger page sizes). Ask for “rust” stories and you’ll get the first ~1,000 and then empty pages — even though HN has tens of thousands of matching items across its history.

The fix is date-windowed (time-sliced) pagination. Instead of paging a single query to exhaustion, you split the time range into windows using the created_at_i numeric filter:

?query=rust&tags=story&numericFilters=created_at_i>1704067200,created_at_i<1706745600

Each window returns under the cap; you walk windows from oldest to newest, narrowing any window that itself hits the cap, and deduplicate by objectID across the boundaries. This is exactly the logic that turns “first 1,000 results” into “every matching item HN has ever had.” Building it correctly — adaptive window sizing, boundary dedup, loop detection — is the part that’s annoying to get right by hand.

▶ Run the Hacker News Search Scraper — keyword search across stories, comments, polls and Show/Ask HN with time-sliced pagination that breaks the Algolia cap. Thousands of clean rows per run, no API key.

What’s worth extracting

Every item — story or comment — normalizes to a flat row. The fields that matter for most downstream work:

• Identity — object_id (stable, the only reliable dedup/join key), HN permalink, item type/tags.
• Content — title, body text (for comments and Ask HN), external URL (for link stories).
• Author — the HN username, useful for tracking prolific posters.
• Signals — points/upvotes, comment count.
• Time — creation timestamp (both ISO and epoch).

That’s the whole record. Because HN content is plain text, it’s unusually clean for LLM corpora — no ad markup, no tracking spans, no infinite-scroll cruft to strip.

A clean per-row schema

When this lands in your warehouse, normalize it like so:

{
  "object_id": "39481234",
  "type": "story",
  "title": "Show HN: I built a local-first sync engine",
  "author": "patio11",
  "url": "https://example.com/sync-engine",
  "text": null,
  "points": 412,
  "num_comments": 188,
  "tags": ["story", "show_hn"],
  "created_at": "2026-05-12T08:14:00Z",
  "created_at_i": 1778646840,
  "permalink": "https://news.ycombinator.com/item?id=39481234",
  "scraped_at": "2026-05-20T09:00:00Z"
}

A few schema choices to make early:

• Keep object_id as a string. It’s numeric today but treat it as an opaque key for joins and dedup.
• Store both created_at and created_at_i. The epoch form is what you’ll re-feed into time-window filters on the next incremental run.
• Keep text and url separate. Stories usually have a URL and no text; comments and Ask HN have text and no URL. Don’t coalesce them.
• Persist tags as an array. A single item can be both story and show_hn; flattening loses the launch signal.

Typical use cases

What people actually build on HN data:

• Tech and product trend research — track the rise and fall of languages, frameworks, and AI models by counting stories and summing points over time.
• Show HN launch tracking — a daily scheduled run filtering tags=show_hn gives you every product launch with its traction.
• Ask HN knowledge mining — Ask HN threads are dense, high-quality Q&A; great as a fine-tuning or retrieval corpus.
• Brand and competitor monitoring — search your company or a competitor’s name with a points>50 floor to catch only the discussions that got traction.
• LLM training data — large comment corpora, clean and plain-text, filtered by topic.
• Newsletter curation — pull the week’s top stories by points in your niche tags.

The common thread is that HN’s value is longitudinal. One snapshot is a leaderboard; a scheduled incremental feed is a trend dataset.

Cost math

Pricing is pay-per-event: a tiny per-run start fee plus a few tenths of a cent per result row. Concretely, at $0.003 per row, a 10,000-row historical backfill of a topic runs about $30 one time. After that, a daily incremental run that only fetches the last 24 hours of a niche keyword might return 50–200 rows — pennies per day.

Compared to building it yourself, you’re not avoiding proxy bills (HN doesn’t need them) — you’re avoiding the time-window pagination engine. That’s the actual work: getting adaptive windowing, boundary dedup, and loop detection right so a backfill doesn’t either stall at 1,000 rows or double-count across window seams.

Common pitfalls

• The 1,000-result cap is silent. Empty pages don’t throw an error; they just return zero hits. If your homegrown scraper “finishes” suspiciously fast, this is why.
• search vs search_by_date. Relevance ranking is non-deterministic across runs; for reproducible backfills always slice by date, not relevance.
• Comment text contains HTML entities. HN encodes >, ', and <p> tags in comment bodies. Decode them before feeding to an LLM.
• Points are a snapshot, not final. A story scraped an hour after posting shows different points than the same story a day later. Log scraped_at so you know when a count was valid.
• Dead and flagged items still appear. Algolia indexes them; filter on your side if you only want live content.

Wrapping up

Hacker News is the rare data source where the hard part isn’t anti-bot defense — it’s quietly working around a result cap that pretends everything is fine. If you only need the top few hundred items on a topic, the raw Algolia endpoint is genuinely usable by hand. If you need full historical backfills or a clean scheduled feed, use a scraper that already implements the time-sliced pagination correctly and hands you flat, deduplicated rows.

▶ Open the Hacker News Search Scraper on Apify — stories, comments, polls and front-page items as export-ready JSON or CSV. Schedule it for a continuous HN feed. Pay per row, start on Apify’s free credit.