Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
Scrape up to 100 independently configured URLs through Scrapfly’s streaming batch endpoint.
scrapfly.scrapeUrls sends one atomic batch while preserving a correlation ID for every result.
Scrapfly offers the Batch API on paid plans. Free-plan requests fail before any URL is scraped.
import { automation, onInvocation } from "automate.ax"
import { scrapfly } from "automate.ax/scrapfly"
export default automation("Scrape product pages", () => {
onInvocation()
scrapfly.scrapeUrls({
requests: [
{
correlationId: "starter",
url: "https://example.com/starter",
format: "markdown",
},
{ correlationId: "pro", url: "https://example.com/pro", renderJs: true },
],
})
})
| Input | Type | Required | Default | Description |
|---|---|---|---|---|
requests | ScrapflyScrapeInput[] | Yes | — | One to 100 inputs accepted by scrapfly.scrapeUrl. |
requests[].correlationId | string | No | request-1, request-2, … | Stable ID used to match streamed parts. Values must be unique. |
{ results }. Each entry contains correlationId and either the same result shape as scrapfly.scrapeUrl or { error: { name, message } }. Scrapfly can complete parts out of order; use correlationId, not array position, to join them.