Stack Exchange

Search questions on Stack Overflow and other Stack Exchange sites, and fetch up to 100 questions with their bodies by id, as typed JSON.

At a glance

Two GET endpoints under /v1/structured/stackexchange/. Each answers { endpoint, url, credits, duration_ms, data }. Each call is charged for the data it moves; a request we could not read is 502 TARGET_FAILED and costs nothing. Public data under the network's API terms; show attribution to the authors where you publish it, as the content is licensed CC BY-SA.

curl -G "https://api.scrapelyserp.com/v1/structured/stackexchange/search" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY" \
  -d site=stackoverflow \
  --data-urlencode "query=web scraping" \
  -d sort=votes

import os
import requests

HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPELY_API_KEY']}"}

r = requests.get(
    "https://api.scrapelyserp.com/v1/structured/stackexchange/search",
    headers=HEADERS,
    params={"site": "stackoverflow", "query": "web scraping", "sort": "votes"},
    timeout=30,
)
r.raise_for_status()
data = r.json()
for q in data["data"]["results"]:
    print(q["score"], q["title"])

const res = await fetch(`https://api.scrapelyserp.com/v1/structured/stackexchange/search?${new URLSearchParams({ site: 'stackoverflow', query: 'web scraping', sort: 'votes' })}`, {
  headers: { Authorization: `Bearer ${process.env.SCRAPELY_API_KEY}` },
  signal: AbortSignal.timeout(30_000),
});
if (!res.ok) throw new Error((await res.json()).error.code);
const data = await res.json();
for (const q of data.data.results) console.log(q.score, q.title);

How much you can pull

The network limits how much one address may read in a day, so requests are spread across many exits and the limit rarely shows. When an exit is used up, or asks readers to slow down, the call moves to another exit straight away (up to four tries, no waiting), and that exit rests until its allowance returns. Only if every try is throttled does a call fail: it is 502 TARGET_FAILED, uncharged, and can be sent again. A retry adds a second or two to a call.

Searching reaches pages 1 to 25 (up to 2,500 questions at 100 per page). Fetching by id is far cheaper per question: one call takes up to 100 ids.

Search

Each question has id, title, url, score, answers, views, is_answered, accepted_answer_id, tags, author, author_id, author_reputation, created_at and last_activity_at. next_page is the page to send next, or null at the end. A query that matches nothing returns an empty list.

Full reference → GET /v1/structured/stackexchange/search

Questions by id

The same fields as search plus body_html, the question's body. Results come in the order the ids were sent; ids that do not exist are listed in missing.

Full reference → GET /v1/structured/stackexchange/questions