S3 export

Have every watchlist event written to your own Amazon S3 bucket, or to any S3-compatible store such as Cloudflare R2, Wasabi, MinIO or Backblaze B2.

Set it up

Each account has one export destination. Set it in Dashboard → Settings → S3 export, or with PUT /v1/exports/s3. You need an existing bucket and an access key that may write to it.

1. Create (or pick) a bucket. Objects are written under a key prefix, scrapely/ by default.

2. Create an access key limited to s3:PutObject on that bucket and prefix (an example policy is below).

3. Enter the bucket, region and keys, then Test connection: we write scrapely-connection-test.txt under the prefix and report what the storage service answered.

4. Save. Exports switch on only after a passing test.

Minimal IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-scrapes/scrapely/*"
    }
  ]
}

Your keys stay secret: The secret access key is encrypted (AES-256-GCM) before it is stored and is never shown or returned again, in the dashboard or the API. Only the last four characters of the access key id are displayed. To change keys, send a new pair; editing anything else keeps the stored ones.

What is written, and where

While the destination is on, every watchlist event is written to the bucket as one JSON object, in the background as each run finishes: an upload never delays or fails the run. It happens whether or not you also receive the event by webhook.

Each object also carries x-amz-meta-scrapely-watchlist-id, x-amz-meta-scrapely-run-id and x-amz-meta-scrapely-event, so you can route it without opening it.

Example bucket

scrapely/watchlists/wl_3c4d5e6f7a8b9c0d/wlrun_7a8b9c0d1e2f3a4b/watchlist.product.changed-1.json
scrapely/watchlists/wl_3c4d5e6f7a8b9c0d/wlrun_7a8b9c0d1e2f3a4b/watchlist.product.changed-2.json
scrapely/watchlists/wl_3c4d5e6f7a8b9c0d/wlrun_7a8b9c0d1e2f3a4b/watchlist.run.completed-1.json
scrapely/scrapely-connection-test.txt

S3-compatible stores

Choose Custom endpoint (or send endpoint) to write to any service that speaks the S3 API. Requests are signed with AWS Signature Version 4 using the region you give.

Allowed endpoints: A custom endpoint must be https://, on port 443 or 1024–65535, and resolve to a public address: private, loopback, link-local and cloud metadata addresses are refused when you save it and again on every upload. Redirects are never followed.

API

Amazon S3

curl -X PUT "https://api.scrapelyserp.com/v1/exports/s3" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "my-scrapes",
    "region": "us-east-1",
    "access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "prefix": "scrapely/"
  }'

# → { "object": "s3_destination", "bucket": "my-scrapes", "access_key_id": "••••MPLE",
#     "enabled": true, "test": { "ok": true, "key": "scrapely/scrapely-connection-test.txt" }, … }

Cloudflare R2

curl -X PUT "https://api.scrapelyserp.com/v1/exports/s3" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "scrapes",
    "region": "auto",
    "endpoint": "https://<account_id>.r2.cloudflarestorage.com",
    "access_key_id": "<r2 access key id>",
    "secret_access_key": "<r2 secret access key>"
  }'

Pause

# Keeps the bucket and keys; nothing more is written until you set it back to true.
curl -X PUT "https://api.scrapelyserp.com/v1/exports/s3" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "enabled": false, "test": false }'

Errors and retries

An upload that fails because the storage service is busy or unreachable (a timeout, a 5xx, SlowDown) is retried after 10 s, 1 min, 5 min, 30 min and 2 h. A refusal (AccessDenied, NoSuchBucket, a wrong key) is not retried: fix the destination and later events export again. Every attempt is listed under Recent exports in Settings and in recent_exports from the API (newest 100 kept).