Wikipedia

Search Wikipedia by keyword and read up to 100 pages by title in one call, with opening text, short description and Wikidata id, as typed JSON.

At a glance

Two GET endpoints under /v1/structured/wikipedia/, for the 16 language editions in en, de, fr, es, it, pt, nl, pl, sv, ru, ja, zh, ar, ko, tr, uk (lang, default en; any other value is a free 400). Each answers { endpoint, url, credits, duration_ms, data }. Each call is charged for the data it moves, and a call we could not read is 502 TARGET_FAILED and costs nothing. Rates are estimates until production traffic measures them.

curl -G "https://api.scrapelyserp.com/v1/structured/wikipedia/pages" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY" \
  --data-urlencode "titles=Albert Einstein|Web scraping"

import os
import requests

HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPELY_API_KEY']}"}

r = requests.get(
    "https://api.scrapelyserp.com/v1/structured/wikipedia/pages",
    headers=HEADERS,
    params={"titles": "Albert Einstein|Web scraping"},
    timeout=30,
)
r.raise_for_status()
data = r.json()
for p in data["data"]["pages"]:
    print(p["title"], "-", p["description"])

const res = await fetch(`https://api.scrapelyserp.com/v1/structured/wikipedia/pages?${new URLSearchParams({ titles: 'Albert Einstein|Web scraping' })}`, {
  headers: { Authorization: `Bearer ${process.env.SCRAPELY_API_KEY}` },
  signal: AbortSignal.timeout(30_000),
});
if (!res.ok) throw new Error((await res.json()).error.code);
const data = await res.json();
for (const p of data.data.pages) console.log(p.title, "-", p.description);

Search

Each result has title, page_id, url, snippet (plain text around the match), word_count and last_modified; total counts every match. A query that matches nothing is an empty results with a total of 0. next_cursor is null at the end.

Full reference → GET /v1/structured/wikipedia/search

Pages by title

data.pages are in the order you asked, one for each title: requested is the title as you sent it and title the page's own, because a title is tidied (albert einstein becomes Albert einstein) and redirects are followed (Einstein is the page Albert Einstein, with redirected_from set). Two titles that lead to one page give two entries. Each page has page_id, url, description (Wikipedia's one-line description), extract (the opening section as plain text), wikidata_id, disambiguation (true for a page that lists several meanings, such as Mercury), last_modified and length. A title that is not a page comes back in data.not_found, and a call where none is a page is a free 404 NOT_FOUND.

Full reference → GET /v1/structured/wikipedia/pages