Start Scrapfly crawl

scrapfly.startCrawl submits a recursive crawl, an inline URL list, or a remote URL list. It does not poll for completion.

import { automation, onInvocation } from "automate.ax"
import { scrapfly } from "automate.ax/scrapfly"

export default automation("Start docs crawl", () => {
  onInvocation()
  scrapfly.startCrawl({
    url: "https://example.com/docs",
    contentFormats: ["markdown", "page_metadata"],
    maxDepth: 3,
    pageLimit: 500,
  })
})

Supply exactly one of url, urlList (up to an API-supported uploaded list), or remoteUrlList. Crawl controls include maxDepth, pageLimit, maxDuration, maxApiCredit, maxConcurrency, delay, path filters, domain/subdomain rules, sitemap and robots behavior, rendering, cache, ASP, proxy, headers, extraction rules, output formats, and dashboard webhook name/events. excludePaths and includeOnlyPaths are mutually exclusive. Returns { crawlerId, status }. Status is PENDING, RUNNING, DONE, or CANCELLED. Use scrapfly.getCrawlStatus and the content actions to inspect subsequent work.

Built-in triggers

Built-in actions

Runtime utilities

Integrations