How to Scrape Dev.to Articles by Tag and Author in 2026

Dev.to (built on the open-source Forem platform) is one of the densest sources of developer-written content on the web — tutorials, deep dives, hot takes and product write-ups, each tagged and engagement-scored. For content curation, DevRel monitoring, trend tracking, or building an LLM corpus of developer writing, it’s a goldmine. And unlike most platforms, Dev.to ships a genuinely usable public API. This guide covers what’s worth pulling, how that API behaves, and where a managed scraper saves you the pagination and rate-limit grind.

What’s worth extracting

Per article, Dev.to exposes a consistent, clean set of metadata:

• Textual metadata — title and description/summary (the article preview text).
• Canonical URL — the stable link to the post.
• Tags — the topic labels (javascript, rust, webdev, ai, etc.) that power discovery.
• Author identity — username, name, profile info.
• Engagement metrics — public reactions count and comment count, the two signals that separate a viral post from a quiet one.
• Estimated reading time — minutes, useful for filtering long-form versus quick reads.
• Cover image URL — the hero image for newsletter/aggregator rendering.
• Publication timestamps — published-at, for chronological and cadence analysis.

For most pipelines you want title, tags, author, reactions, comments and published-at. Reading time and cover image matter when you’re rendering a digest.

How the data is exposed

Dev.to / Forem ships a clean, public, unauthenticated REST API. The article-listing endpoint is:

GET https://dev.to/api/articles
  ?tag=rust            # filter by topic tag
  &username=ben        # or filter by author
  &page=1
  &per_page=100        # up to ~1000 per page

There’s also a ?top=7 flavor for top articles over a window, and /api/articles/latest for the global newest feed. Crucially, this returns flat-ish article objects directly — no HTML parsing, no headless browser, no bot challenge. That’s a meaningful contrast to scraping most social platforms.

The catch is pagination and rate limits: there’s no single “give me everything” call. You walk pages until the API returns an empty array, and you have to pace those page requests or you’ll get throttled.

Rate limits and anti-abuse

Forem applies per-IP rate limiting on the public API. In practice:

• The article-list endpoints tolerate steady polling but will return HTTP 429 if you blast pages in a tight loop.
• The correct pattern is exponential backoff on 429 and a modest delay between page requests — patient pagination, not parallel flooding.
• Because the endpoints are unauthenticated, there’s no token to rotate; your only lever is pacing (and, if you’re very aggressive, IP rotation).

A managed scraper handles exactly this: it paginates fully through a tag or author feed, backs off on 429 automatically, and stops cleanly at the end of the feed — so you get the whole corpus without writing the retry loop yourself.

▶ Run the Dev.to Articles Scraper — pulls articles by tag, by author, or the global latest feed, fully paginated with automatic 429 backoff. Returns title, tags, reactions, comments, reading time and cover image as flat rows. Pay per row.

Build it yourself vs. use a managed scraper

Because the API is clean and unauthenticated, a quick script is genuinely easy here — more so than for most targets. The honest trade-off:

• Building from scratch — an hour for a single-tag dump. The work creeps in when you want many tags and authors in one run, dedup across overlapping feeds, full-corpus pagination with reliable end-detection, 429 handling, and flat output ready for a warehouse or sheet.
• Using a managed actor — list your tags/authors, set a cap (or unlimited), run, get flat JSON/CSV out. Schedule it for a fresh daily or weekly feed without owning the cron.

For a one-off pull of a single tag, just script it. For an ongoing multi-tag/author monitoring feed, the scraper removes the boilerplate.

Schema design for downstream use

A flat per-article row that’s easy to dedup and append:

{
  "id": 1839204,
  "title": "Why I switched from Express to Hono in 2026",
  "description": "A migration story with benchmarks and gotchas...",
  "url": "https://dev.to/jane/why-i-switched-from-express-to-hono-2026",
  "tags": ["javascript", "webdev", "performance"],
  "author_username": "jane",
  "author_name": "Jane Doe",
  "reactions_count": 412,
  "comments_count": 37,
  "reading_time_minutes": 6,
  "cover_image": "https://media.dev.to/cdn-cgi/image/.../cover.png",
  "published_at": "2026-05-22T09:14:00Z",
  "scraped_at": "2026-05-22T12:00:00Z"
}

Decisions worth making early:

• Dedup on id, not URL. The same article surfaces across multiple tag feeds; the numeric id is the clean dedup key.
• Keep tags as an array. Tag-trend analysis needs to explode posts by tag; don’t flatten to a comma string.
• Store both published_at and scraped_at. Engagement counts grow over time — to chart “reactions at 24h” you need to know when you observed the number.
• Treat engagement as a snapshot. Reactions and comments are live and rising; a single pull is a point-in-time reading, not a final tally.

Typical use cases

• Tech content discovery & curation — feed newsletters, podcasts and internal developer digests with the week’s top posts per tag.
• Author network analysis — collect an author’s full corpus with tags and engagement for outreach or profiling.
• Tag trend tracking — plot topic momentum over time; backfill historical tag feeds to see which technologies are rising.
• DevRel & devtool marketing — monitor mentions of your product, competitor coverage and integration tutorials.
• Technical-writing benchmarking — assess topic coverage and engagement before planning a book, course or docs.
• LLM / NLP training data — assemble a labeled corpus of developer writing with topical tags and clean attribution.
• Competitive analysis — track a rival’s posting cadence, engagement deltas and topic mix.

The thread: the value is in breadth (many tags/authors) plus freshness (a feed, not a one-time dump).

Cost math for the managed approach

Dev.to articles are lightweight rows, so cost scales purely with volume. A daily pull of, say, the top posts across 20 tags is a few thousand rows; at a fraction of a cent per row that’s a few dollars a month. A one-time historical backfill of a popular tag’s full feed might be tens of thousands of rows — still modest, and you only pay for it once.

Against a self-hosted poller you skip the pagination loop, the 429 retry logic and the dedup bookkeeping — small individually, annoying in aggregate, and yours to maintain forever.

Common pitfalls

• Engagement counts move. Reactions and comments keep climbing after publication. To compare posts fairly, normalize by age or re-scrape on a fixed cadence.
• Tag-feed overlap. A post tagged ai and python appears in both feeds — always dedup on id when scraping multiple tags.
• description is a preview, not the full body. The list endpoints return the summary; if you need full article text you fetch the per-article endpoint separately.
• Rate-limit on burst. Parallelizing page requests trips 429 fast; pace them.
• Deleted/unpublished posts. Articles can be removed or unpublished; a URL from an old run may 404 later.

Wrapping up

Dev.to is one of the rare platforms where the public API is clean enough that scraping is mostly about pagination etiquette and good output shape rather than fighting anti-bot defenses. If you need a single tag once, script it. If you want an ongoing, deduplicated, multi-tag/author feed for curation, DevRel or a training corpus, run a managed scraper that already handles the paging, backoff and flattening.

▶ Open the Dev.to Articles Scraper on Apify — schedule it for a fresh feed of trending developer content by tag or author. Flat JSON/CSV, pay per row, free monthly credit to start.