Bulk Catalog Export

The Bulk Catalog Export pattern pulls Snappy’s catalog into your own data store asynchronously. You queue a job, poll until it completes, then download an NDJSON file with one product per line. This is the right pattern when you need a local copy of the catalog - for blending with other data sources, powering a search index, or serving from your own infrastructure.

When to use

Use bulk export when:

You already have local catalog infrastructure (search index, database, product schema)
You want to blend Snappy products with other sources
Your rendering layer cannot tolerate per-request API latency

Use Real-Time Catalog Access instead for simpler integrations that don’t need a local copy.

The three-phase lifecycle

Every export goes through Queue → Poll → Download:

Queue - POST to an export endpoint to start a job. You get back an exportId.
Poll - GET the job status until status is completed (or failed).
Download - fetch the file(s) from the downloadUrls map in the completed job response.

Two export paths

Filtered products export

Export products from the full catalog with optional filters. catalog and format are required; locations defaults to ["US"]. See Create products export for all filters.

POST /v3/products/exports

{
  "catalog": "marketplace",
  "format": "ndjson",
  "locations": ["US", "CA"],
  "brandId": ["6511b55142c420000d083a55"]
}

Collection-scoped export

Export all products in a specific collection. collectionId, catalog, and locations are required. See Create collections export.

POST /v3/collections/exports

{
  "collectionId": "col_abc123",
  "catalog": "marketplace",
  "locations": ["US"]
}

Both endpoints return 200 OK immediately with the job ID:

{
  "exportId": "6650a1b2c3d4e5f6a7b8c9d0"
}

Each create call starts a new export job - repeated calls with the same body are not deduplicated. Store the exportId of the job in progress and wait for it to finish before starting another.

Recommended usage

Run each export at most once a day. Use webhooks for changes between exports - see Keeping your local copy fresh.
Keep each job to 50 locations or fewer. To cover more countries, split them across multiple jobs and run them one after another.

Poll and download

Both export types share the same poll endpoint:

GET /v3/products/exports/{exportId}

The job moves through pending → processing → completed or failed. When completed, the response includes a downloadUrls map of signed URLs (keyed by file identifier or location code). When failed, it includes an errorMessage. See Export API for the shape of each line in the file.

async function waitForExport(exportId, { pollIntervalMs = 5000, timeoutMs = 300000 } = {}) {
  const start = Date.now();
  while (Date.now() - start < timeoutMs) {
    const res = await fetch(`${SNAPPY}/v3/products/exports/${exportId}`, { headers });
    const job = await res.json();
    if (job.status === "completed") return job;
    if (job.status === "failed") throw new Error(`Export failed: ${job.errorMessage}`);
    await new Promise(r => setTimeout(r, pollIntervalMs));
  }
  throw new Error("Export timed out");
}

async function downloadExport(job) {
  const products = [];
  for (const url of Object.values(job.downloadUrls)) {
    const res = await fetch(url); // signed URL, no auth needed
    const text = await res.text();
    for (const line of text.split("\n")) {
      if (line.trim()) products.push(JSON.parse(line)); // one product per line
    }
  }
  return products;
}

// Full flow:
async function runExport(locations = ["US"]) {
  const startRes = await fetch(`${SNAPPY}/v3/products/exports`, {
    method: "POST",
    headers,
    body: JSON.stringify({ catalog: "marketplace", format: "ndjson", locations }),
  });
  const { exportId } = await startRes.json();
  const job = await waitForExport(exportId);
  return downloadExport(job);
}

import json, time, requests

def wait_for_export(export_id, poll_interval=5, timeout=300):
    start = time.time()
    while time.time() - start < timeout:
        res = requests.get(f"{SNAPPY}/v3/products/exports/{export_id}", headers=headers)
        job = res.json()
        if job["status"] == "completed":
            return job
        if job["status"] == "failed":
            raise RuntimeError(f"Export failed: {job.get('errorMessage')}")
        time.sleep(poll_interval)
    raise TimeoutError("Export timed out")

def download_export(job):
    products = []
    for url in job["downloadUrls"].values():
        res = requests.get(url)  # signed URL, no auth needed
        for line in res.text.splitlines():
            if line.strip():
                products.append(json.loads(line))  # one product per line
    return products

def run_export(locations=None):
    start_res = requests.post(
        f"{SNAPPY}/v3/products/exports",
        headers=headers,
        json={"catalog": "marketplace", "format": "ndjson", "locations": locations or ["US"]},
    )
    export_id = start_res.json()["exportId"]
    job = wait_for_export(export_id)
    return download_export(job)

Keeping your local copy fresh

After the initial bulk export, use the stock-availability-updates webhook to receive incremental updates instead of re-exporting the full catalog on a timer.

stock-availability-updates

Subscribe to this event in your Webhook configuration. Each event carries the products whose availability or pricing changed. Update only those records in your local store.

Re-run a full bulk export periodically as a safety net - at most once a day - and rely on webhooks for real-time freshness in between. This covers any webhook delivery failures.

Common pitfalls

Pitfall	Fix
Starting duplicate jobs	Each create call starts a new job; store the in-flight `exportId` and wait for it to finish before starting another
Polling too aggressively	Use 5-second intervals minimum; the export rarely completes in under 30 seconds
Ignoring the download URL expiry	Download the file(s) as soon as the job completes; the export record and its signed URLs expire 48 hours after creation
Parsing the file as a single JSON document	The file is NDJSON - parse it one line (one product) at a time
Not subscribing to stock webhooks	Your local copy will drift; pair bulk export with `stock-availability-updates`
Exporting more than once a day	Run each export at most daily; use webhooks for changes in between
Requesting too many locations in one job	Keep each job to 50 locations or fewer; split larger sets across sequential jobs

Export API - endpoint reference and NDJSON file format
Real-Time Catalog Access - simpler alternative without local infrastructure
Swag Products Access - swag-specific catalog access

Introduction

Patterns

Recipes

Migration

When to use

The three-phase lifecycle

Two export paths

Filtered products export

Collection-scoped export

Recommended usage

Poll and download

Keeping your local copy fresh

Common pitfalls

​When to use

​The three-phase lifecycle

​Two export paths

​Filtered products export

​Collection-scoped export

​Recommended usage

​Poll and download

​Keeping your local copy fresh

​Common pitfalls

​Related

When to use

The three-phase lifecycle

Two export paths

Filtered products export

Collection-scoped export

Recommended usage

Poll and download

Keeping your local copy fresh

Common pitfalls

Related