Crawl website with Firecrawl

crawlWebsiteWithFirecrawl starts a Firecrawl crawl and continues in a correlated child context after Firecrawl reports completion through a callback. It does not hold an action invocation open while the crawl runs. Use it for related pages on one site; use scrapeUrlWithFirecrawl for a single page. Inputs accept literal values or compatible signals. An optional second argument { account } selects the Firecrawl binding.

Example

automations/crawl-help-center.automation.ts

import { automation, onSchedule } from "automate.ax"
import { crawlWebsiteWithFirecrawl } from "automate.ax/firecrawl"

export default automation("Crawl help center", () => {
  onSchedule({ schedule: "0 3 * * 0" })

  crawlWebsiteWithFirecrawl({
    url: "https://help.example.com",
    includePaths: ["articles/.*"],
    excludePaths: ["articles/archive/.*"],
    limit: 100,
    maxDiscoveryDepth: 3,
    scrapeOptions: { formats: ["markdown", "links"] },
  })
})

Inputs

Input	Type	Required	Default	Description
`url`	`string`	Yes	—	Website root URL.
`limit`	`number`	No	`100`	Maximum pages to crawl. Start with a small limit to control duration and credits.
`includePaths` / `excludePaths`	`string[]`	No	None	Regular expressions matched against URL paths.
`regexOnFullURL`	`boolean`	No	`false`	Match path expressions against complete URLs, including query parameters.
`maxDiscoveryDepth`	`number`	No	Provider default	Maximum link-discovery depth; the root and sitemap pages are depth 0.
`sitemap`	`'skip' \| 'include' \| 'only'`	No	`include`	Sitemap discovery mode.
`ignoreQueryParameters`	`boolean`	No	`false`	Avoid scraping the same path with different query parameters.
`ignoreRobotsTxt`	`boolean`	No	`false`	Crawl pages even when robots.txt disallows them.
`deduplicateSimilarURLs`	`boolean`	No	Provider default	Deduplicate URLs Firecrawl considers materially similar.
`crawlEntireDomain`	`boolean`	No	`false`	Follow sibling and parent paths, not only descendants of the starting path.
`allowSubdomains` / `allowExternalLinks`	`boolean`	No	`false`	Expand the crawl beyond the starting hostname.
`prompt`	`string`	No	None	Natural-language instructions from which Firecrawl derives crawl options. Explicit inputs take precedence.
`delay`	`number`	No	None	Seconds between scrapes. A delay reduces crawl concurrency.
`maxConcurrency`	`number`	No	Team limit	Maximum simultaneous page scrapes.
`robotsUserAgent`	`string`	No	Firecrawl default	User-Agent used to evaluate robots.txt.
`scrapeOptions`	`object`	No	Markdown	Per-page formats, browser actions, content filters, cache, proxy, redaction, and target settings accepted by `scrapeUrlWithFirecrawl`.
`zeroDataRetention`	`boolean`	No	`false`	Request enterprise zero-data-retention mode when enabled for the Firecrawl team.

Output

Returns Firecrawl’s crawl job as { id, status, total, completed, creditsUsed?, expiresAt?, next?, data }. expiresAt is a Date; data contains Firecrawl documents in the same shape returned by scrapeUrlWithFirecrawl. Automate.ax supplies a callback URL to Firecrawl, correlates the terminal callback with the crawl job, and continues from the action that started it. A failed crawl fails the continued action path. Use startWebsiteCrawlWithFirecrawl instead when you want the job ID immediately or need to provide your own webhook.

Get started

Guides

Concepts

Management API

Reference

Crawl website with Firecrawl

Example

Inputs

Output

​Example

​Inputs

​Output

Example

Inputs

Output