Crawl website with Firecrawl

firecrawl.crawlWebsite starts a Firecrawl crawl and continues in a correlated child context after Firecrawl reports completion through a callback. It doesn’t hold an action invocation open while the crawl runs. Use it for related pages on one site. Use firecrawl.scrapeUrl for a single page. Inputs accept literal values or compatible signals. An optional second argument { account } selects the Firecrawl binding.

Example

automations/crawl-help-center.automation.ts

import { automation, onSchedule, withPrerequisites } from "automate.ax"
import { firecrawl } from "automate.ax/firecrawl"

export default automation("Crawl help center", () => {
  const scheduled = onSchedule({ schedule: "0 3 * * 0" })

  withPrerequisites(scheduled, () =>
    firecrawl.crawlWebsite({
      url: "https://help.example.com",
      includePaths: ["articles/.*"],
      excludePaths: ["articles/archive/.*"],
      limit: 100,
      maxDiscoveryDepth: 3,
      scrapeOptions: { formats: ["markdown", "links"] },
    }),
  )
})

firecrawl.crawlWebsite introduces a protected callback trigger and therefore another possible root. Scope literal crawl inputs to the trigger that should start them, as shown above. Passing a signal in the crawl input also anchors the starter action to that signal’s context boundary.

Inputs

Input	Type	Required	Default	Description
`url`	`string`	Yes	—	Website root URL.
`limit`	`number`	No	`100`	Maximum pages to crawl. Start with a small limit to control duration and credits.
`includePaths` / `excludePaths`	`string[]`	No	None	Regular expressions matched against URL paths.
`regexOnFullURL`	`boolean`	No	`false`	Match path expressions against complete URLs, including query parameters.
`maxDiscoveryDepth`	`number`	No	Provider default	Maximum link-discovery depth; the root and sitemap pages are depth 0.
`sitemap`	`'skip' \| 'include' \| 'only'`	No	`include`	Sitemap discovery mode.
`ignoreQueryParameters`	`boolean`	No	`false`	Avoid scraping the same path with different query parameters.
`ignoreRobotsTxt`	`boolean`	No	`false`	Crawl pages even when robots.txt disallows them.
`deduplicateSimilarURLs`	`boolean`	No	Provider default	Deduplicate URLs Firecrawl considers materially similar.
`crawlEntireDomain`	`boolean`	No	`false`	Follow sibling and parent paths, not only descendants of the starting path.
`allowSubdomains` / `allowExternalLinks`	`boolean`	No	`false`	Expand the crawl beyond the starting hostname.
`prompt`	`string`	No	None	Natural-language instructions from which Firecrawl derives crawl options. Explicit inputs take precedence.
`delay`	`number`	No	None	Seconds between scrapes. A delay reduces crawl concurrency.
`maxConcurrency`	`number`	No	Team limit	Maximum simultaneous page scrapes.
`robotsUserAgent`	`string`	No	Firecrawl default	User-Agent used to evaluate robots.txt.
`scrapeOptions`	`object`	No	Markdown	Per-page formats, browser actions, content filters, cache, proxy, redaction, and target settings accepted by `firecrawl.scrapeUrl`.
`zeroDataRetention`	`boolean`	No	`false`	Request enterprise zero-data-retention mode when enabled for the Firecrawl team.

Output

Returns Firecrawl’s crawl job as { id, status, total, completed, creditsUsed?, expiresAt?, next?, data }. expiresAt is a Date. data contains Firecrawl documents in the same shape returned by firecrawl.scrapeUrl. Automate.ax supplies Firecrawl with a signed callback URL. The terminal delivery starts an independent root context, and correlation joins its crawl ID with the action that started the job. The callback capability authorizes access to the exact endpoint. It doesn’t create causal inheritance. A failed crawl fails the correlated action path. Use firecrawl.startWebsiteCrawl instead when you want the job ID immediately or need to provide your own webhook.

Built-in triggers

Built-in actions

Runtime utilities

Integrations

Crawl website with Firecrawl

Example

Inputs

Output

​Example

​Inputs

​Output

Example

Inputs

Output