S3 export

Write every watchlist event to your own S3 bucket (or any S3-compatible store: Cloudflare R2, Wasabi, MinIO, Backblaze B2). One destination per account. The secret access key is write-only: it is stored encrypted and never returned.

Embed

<a href="https://scrapelyserp.com/reference/s3-export"><img src="https://scrapelyserp.com/reference/_api/public/orgs/affc5b0f-d3c2-423a-8143-9d10db5736ae/spaces/e4cbe065-c1c2-4572-b134-0292dedd9c39/pages/b673b69c-cdd1-4651-b78d-47ca3ef5470c/social-card.svg?v=3e7ae33eec8b" alt="S3 export" width="600" height="315"></a>

Endpoints

Method	Path	Does
`GET`	`/v1/exports/s3`	Get the S3 export destination
`PUT`	`/v1/exports/s3`	Create or update the S3 export destination
`DELETE`	`/v1/exports/s3`	Remove the S3 export destination
`POST`	`/v1/exports/s3/test`	Test the S3 connection

How it works, when to use what, and what it costs: S3 export. Base URL, authentication and errors: Overview.

GET /v1/exports/s3

Get the S3 export destination.

The destination with its access key id masked (the secret is never returned), the last connection test, and the 20 most recent uploads. 404 when the account has none.

Request

cURL
Python
Node.js

curl "https://api.scrapelyserp.com/v1/exports/s3" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY"

import os
import requests

HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPELY_API_KEY']}"}

r = requests.get(
    "https://api.scrapelyserp.com/v1/exports/s3",
    headers=HEADERS,
    timeout=30,
)
r.raise_for_status()
data = r.json()

const res = await fetch('https://api.scrapelyserp.com/v1/exports/s3', {
  headers: { Authorization: `Bearer ${process.env.SCRAPELY_API_KEY}` },
  signal: AbortSignal.timeout(30_000),
});
if (!res.ok) throw new Error((await res.json()).error.code);
const data = await res.json();

Response

200 OK
404 Not Found

{
  "object": "s3_destination",
  "bucket": "my-scrapes",
  "region": "us-east-1",
  "endpoint": null,
  "path_style": false,
  "prefix": "scrapely/",
  "enabled": true,
  "access_key_id": "••••EYID",
  "last_test": {
    "at": "2026-10-16T08:59:58.000Z",
    "ok": true,
    "error": null
  },
  "created_at": "2026-10-06T08:00:00.000Z",
  "updated_at": "2026-10-16T09:00:00.000Z",
  "recent_exports": [
    {
      "id": "s3x_8f7e6d5c4b3a2910",
      "kind": "watchlist",
      "key": "scrapely/watchlists/wl_3f1c2a9e8d4b4c61/wlrun_5f0c8a1b2d3e4f56/watchlist.product.changed-1.json",
      "status": "ok",
      "status_code": 200,
      "error": null,
      "bytes": 2381,
      "attempt": 1,
      "at": "2026-10-16T09:15:03.002Z"
    }
  ]
}

{
  "error": {
    "code": "NOT_FOUND",
    "message": "This account has no S3 export destination. Create one with PUT /v1/exports/s3.",
    "docs": "/docs/errors#not_found"
  }
}

Responses

Status	Meaning
`200`	The destination (keys masked) and its 20 most recent uploads.
`401`	`UNAUTHORIZED`: missing, unknown or revoked key.
`404`	`NOT_FOUND`.
`429`	`RATE_LIMITED` or `CONCURRENCY_LIMIT`. Honour `Retry-After`.
`500`	`INTERNAL`: our fault, not charged. Quote `X-Request-Id`.

Response fields

Field	Type	Description
`object`	"s3_destination"	—
`bucket`	string	—
`region`	string	—
`endpoint`	string or null	null: AWS (`https://s3.<region>.amazonaws.com`). Else the https origin of an S3-compatible service.
`path_style`	boolean	`https://<endpoint>/<bucket>/<key>` instead of `https://<bucket>.<endpoint>/<key>`.
`prefix`	string	—
`enabled`	boolean	On: every watchlist event is written to `<prefix>watchlists/<watchlist_id>/<run_id>/<event>-<part>.json`.
`access_key_id`	string	Masked: the last four characters only. The secret access key is never returned.
`last_test`	object or null	—
`last_test.at`	string (date-time)	—
`last_test.ok`	boolean	—
`last_test.error`	string or null	—
`created_at`	string (date-time)	—
`updated_at`	string (date-time)	—
`recent_exports`	object[]	—
`recent_exports[].id`	string	—
`recent_exports[].kind`	string	One of: `watchlist`, `test`.
`recent_exports[].key`	string	—
`recent_exports[].status`	string	One of: `ok`, `failed`.
`recent_exports[].status_code`	integer or null	—
`recent_exports[].error`	string or null	—
`recent_exports[].bytes`	integer	—
`recent_exports[].attempt`	integer	—
`recent_exports[].at`	string (date-time)	—

PUT /v1/exports/s3

Create or update the S3 export destination.

Fields left out keep their current value. Runs a connection test first (unless test: false); enabled: true is refused with a 400 when the test fails, and nothing is saved.

Request body

Name	Type	Required	Description
`bucket`	string	—	S3 bucket naming rules: 3–63 lowercase letters, numbers, dots and hyphens.
`region`	string	—	An AWS region (us-east-1, us-east-2, us-west-1, us-west-2, …). With `endpoint`: the region that service signs with, e.g. `auto` for Cloudflare R2.
`access_key_id`	string	—	Send with `secret_access_key` to set or replace the keys; leave both out to keep the stored ones. Write-only: never returned.
`secret_access_key`	string	—	Write-only. Stored encrypted (AES-256-GCM) and never returned. Write-only: never returned.
`prefix`	string	—	Key prefix for everything written. `""` for the bucket root. Default `scrapely/`.
`endpoint`	string or null	—	An S3-compatible service: https only, a public host, port 443 or 1024–65535, no path. null for AWS.
`path_style`	boolean	—	Default: true with `endpoint`, false for AWS.
`enabled`	boolean	—	`true` needs a passing connection test. Default `true`.
`test`	boolean	—	Write `<prefix>scrapely-connection-test.txt` before saving. `false` skips it (allowed with `enabled: true` only when the bucket, region, endpoint, prefix and keys are unchanged since a passing test). Default `true`.

Request

cURL
Python
Node.js

curl -X PUT "https://api.scrapelyserp.com/v1/exports/s3" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "my-scrapes",
    "region": "us-east-1",
    "access_key_id": "EXAMPLEKEYID",
    "secret_access_key": "example-secret-access-key",
    "prefix": "scrapely/"
  }'

import os
import requests

HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPELY_API_KEY']}"}

r = requests.put(
    "https://api.scrapelyserp.com/v1/exports/s3",
    headers=HEADERS,
    json={
        "bucket": "my-scrapes",
        "region": "us-east-1",
        "access_key_id": "EXAMPLEKEYID",
        "secret_access_key": "example-secret-access-key",
        "prefix": "scrapely/",
    },
    timeout=30,
)
r.raise_for_status()
data = r.json()

const res = await fetch('https://api.scrapelyserp.com/v1/exports/s3', {
  method: 'PUT',
  headers: {
    Authorization: `Bearer ${process.env.SCRAPELY_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    bucket: 'my-scrapes',
    region: 'us-east-1',
    access_key_id: 'EXAMPLEKEYID',
    secret_access_key: 'example-secret-access-key',
    prefix: 'scrapely/',
  }),
  signal: AbortSignal.timeout(30_000),
});
if (!res.ok) throw new Error((await res.json()).error.code);
const data = await res.json();

Response

200 OK
201 Created
400 Bad Request

{
  "object": "s3_destination",
  "bucket": "my-scrapes",
  "region": "us-east-1",
  "endpoint": null,
  "path_style": false,
  "prefix": "scrapely/",
  "enabled": true,
  "access_key_id": "••••EYID",
  "last_test": {
    "at": "2026-10-16T09:15:04.120Z",
    "ok": true,
    "error": null
  },
  "created_at": "2026-10-06T08:00:00.000Z",
  "updated_at": "2026-10-16T09:15:04.120Z",
  "test": {
    "ok": true,
    "key": "scrapely/scrapely-connection-test.txt",
    "status_code": 200,
    "code": null,
    "error": null
  }
}

{
  "object": "s3_destination",
  "bucket": "my-scrapes",
  "region": "us-east-1",
  "endpoint": null,
  "path_style": false,
  "prefix": "scrapely/",
  "enabled": true,
  "access_key_id": "••••EYID",
  "last_test": {
    "at": "2026-10-16T09:15:04.120Z",
    "ok": true,
    "error": null
  },
  "created_at": "2026-10-16T09:15:04.120Z",
  "updated_at": "2026-10-16T09:15:04.120Z",
  "test": {
    "ok": true,
    "key": "scrapely/scrapely-connection-test.txt",
    "status_code": 200,
    "code": null,
    "error": null
  }
}

{
  "error": {
    "code": "MISSING_PARAM",
    "message": "Missing required parameters `access_key_id` and `secret_access_key`.",
    "param": "access_key_id",
    "docs": "/docs/errors#missing_param"
  }
}

Responses

Status	Meaning
`200`	Updated.
`201`	Created.
`400`	`MISSING_PARAM` or `INVALID_PARAM`; `param` names the culprit.
`401`	`UNAUTHORIZED`: missing, unknown or revoked key.
`429`	`RATE_LIMITED` or `CONCURRENCY_LIMIT`. Honour `Retry-After`.
`500`	`INTERNAL`: our fault, not charged. Quote `X-Request-Id`.

Response fields

Field	Type	Description
`object`	"s3_destination"	—
`bucket`	string	—
`region`	string	—
`endpoint`	string or null	null: AWS (`https://s3.<region>.amazonaws.com`). Else the https origin of an S3-compatible service.
`path_style`	boolean	`https://<endpoint>/<bucket>/<key>` instead of `https://<bucket>.<endpoint>/<key>`.
`prefix`	string	—
`enabled`	boolean	On: every watchlist event is written to `<prefix>watchlists/<watchlist_id>/<run_id>/<event>-<part>.json`.
`access_key_id`	string	Masked: the last four characters only. The secret access key is never returned.
`last_test`	object or null	—
`last_test.at`	string (date-time)	—
`last_test.ok`	boolean	—
`last_test.error`	string or null	—
`created_at`	string (date-time)	—
`updated_at`	string (date-time)	—
`test`	object or null	—
`test.ok`	boolean	—
`test.key`	string	—
`test.status_code`	integer or null	—
`test.code`	string or null	The storage service's error code: `AccessDenied`, `NoSuchBucket`, `InvalidAccessKeyId`, `SignatureDoesNotMatch`, `PermanentRedirect`, …
`test.error`	string or null	What went wrong, in plain words.

DELETE /v1/exports/s3

Remove the S3 export destination.

Deletes the destination and its stored keys. Nothing more is written to the bucket; objects already there stay.

Request

cURL
Python
Node.js

curl -X DELETE "https://api.scrapelyserp.com/v1/exports/s3" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY"

import os
import requests

HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPELY_API_KEY']}"}

r = requests.delete(
    "https://api.scrapelyserp.com/v1/exports/s3",
    headers=HEADERS,
    timeout=30,
)
r.raise_for_status()
data = r.json()

const res = await fetch('https://api.scrapelyserp.com/v1/exports/s3', {
  method: 'DELETE',
  headers: { Authorization: `Bearer ${process.env.SCRAPELY_API_KEY}` },
  signal: AbortSignal.timeout(30_000),
});
if (!res.ok) throw new Error((await res.json()).error.code);
const data = await res.json();

Response

200 OK
404 Not Found

{
  "object": "s3_destination",
  "deleted": true
}

{
  "error": {
    "code": "NOT_FOUND",
    "message": "This account has no S3 export destination. Create one with PUT /v1/exports/s3.",
    "docs": "/docs/errors#not_found"
  }
}

Responses

Status	Meaning
`200`	Removed.
`401`	`UNAUTHORIZED`: missing, unknown or revoked key.
`404`	`NOT_FOUND`.
`429`	`RATE_LIMITED` or `CONCURRENCY_LIMIT`. Honour `Retry-After`.
`500`	`INTERNAL`: our fault, not charged. Quote `X-Request-Id`.

Response fields

Field	Type	Description
`object`	"s3_destination"	—
`deleted`	true	—

POST /v1/exports/s3/test

Test the S3 connection.

Writes <prefix>scrapely-connection-test.txt now. With no body, tests the saved destination; with destination fields (as PUT), tests those merged over it without saving.

Request body (optional)

Name	Type	Required	Description
`bucket`	string	—	S3 bucket naming rules: 3–63 lowercase letters, numbers, dots and hyphens.
`region`	string	—	An AWS region (us-east-1, us-east-2, us-west-1, us-west-2, …). With `endpoint`: the region that service signs with, e.g. `auto` for Cloudflare R2.
`access_key_id`	string	—	Send with `secret_access_key` to set or replace the keys; leave both out to keep the stored ones. Write-only: never returned.
`secret_access_key`	string	—	Write-only. Stored encrypted (AES-256-GCM) and never returned. Write-only: never returned.
`prefix`	string	—	Key prefix for everything written. `""` for the bucket root. Default `scrapely/`.
`endpoint`	string or null	—	An S3-compatible service: https only, a public host, port 443 or 1024–65535, no path. null for AWS.
`path_style`	boolean	—	Default: true with `endpoint`, false for AWS.
`enabled`	boolean	—	`true` needs a passing connection test. Default `true`.
`test`	boolean	—	Write `<prefix>scrapely-connection-test.txt` before saving. `false` skips it (allowed with `enabled: true` only when the bucket, region, endpoint, prefix and keys are unchanged since a passing test). Default `true`.

Request

cURL
Python
Node.js

curl -X POST "https://api.scrapelyserp.com/v1/exports/s3/test" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY"

import os
import requests

HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPELY_API_KEY']}"}

r = requests.post(
    "https://api.scrapelyserp.com/v1/exports/s3/test",
    headers=HEADERS,
    timeout=30,
)
r.raise_for_status()
data = r.json()

const res = await fetch('https://api.scrapelyserp.com/v1/exports/s3/test', {
  method: 'POST',
  headers: { Authorization: `Bearer ${process.env.SCRAPELY_API_KEY}` },
  signal: AbortSignal.timeout(30_000),
});
if (!res.ok) throw new Error((await res.json()).error.code);
const data = await res.json();

Response

200 OK
404 Not Found

{
  "object": "s3_test",
  "ok": true,
  "key": "scrapely/scrapely-connection-test.txt",
  "status_code": 200,
  "code": null,
  "error": null,
  "duration_ms": 412
}

{
  "error": {
    "code": "NOT_FOUND",
    "message": "This account has no S3 export destination. Create one with PUT /v1/exports/s3.",
    "docs": "/docs/errors#not_found"
  }
}

Responses

Status	Meaning
`200`	What the storage service answered.
`400`	`MISSING_PARAM` or `INVALID_PARAM`; `param` names the culprit.
`401`	`UNAUTHORIZED`: missing, unknown or revoked key.
`404`	`NOT_FOUND`.
`429`	`RATE_LIMITED` or `CONCURRENCY_LIMIT`. Honour `Retry-After`.
`500`	`INTERNAL`: our fault, not charged. Quote `X-Request-Id`.

Response fields

Field	Type	Description
`object`	"s3_test"	—
`duration_ms`	integer	—
`ok`	boolean	—
`key`	string	—
`status_code`	integer or null	—
`code`	string or null	The storage service's error code: `AccessDenied`, `NoSuchBucket`, `InvalidAccessKeyId`, `SignatureDoesNotMatch`, `PermanentRedirect`, …
`error`	string or null	What went wrong, in plain words.

Updated Oct 19, 2026

Was this page helpful?