> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automate.ax/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Automate.ax automations are TypeScript programs.
> Use Bun for installation and command examples.
> Use Automate.ax for the product name and automate.ax for the package, CLI, and domain.
> Treat documented public APIs as current; do not invent transitional or deprecated names.

# Crawl website with Firecrawl

> Crawl a site and continue after Firecrawl reports completion.

`crawlWebsiteWithFirecrawl` starts a Firecrawl crawl and continues in a correlated child context after Firecrawl reports completion through a callback. It does not hold an action invocation open while the crawl runs. Use it for related pages on one site; use `scrapeUrlWithFirecrawl` for a single page.

Inputs accept literal values or compatible signals. An optional second argument `{ account }` selects the Firecrawl binding.

## Example

```ts automations/crawl-help-center.automation.ts theme={null}
import { automation, onSchedule } from "automate.ax"
import { crawlWebsiteWithFirecrawl } from "automate.ax/firecrawl"

export default automation("Crawl help center", () => {
  onSchedule({ schedule: "0 3 * * 0" })

  crawlWebsiteWithFirecrawl({
    url: "https://help.example.com",
    includePaths: ["articles/.*"],
    excludePaths: ["articles/archive/.*"],
    limit: 100,
    maxDiscoveryDepth: 3,
    scrapeOptions: { formats: ["markdown", "links"] },
  })
})
```

## Inputs

| Input                                    | Type                            | Required | Default           | Description                                                                                                                            |
| ---------------------------------------- | ------------------------------- | -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                                    | `string`                        | Yes      | —                 | Website root URL.                                                                                                                      |
| `limit`                                  | `number`                        | No       | `100`             | Maximum pages to crawl. Start with a small limit to control duration and credits.                                                      |
| `includePaths` / `excludePaths`          | `string[]`                      | No       | None              | Regular expressions matched against URL paths.                                                                                         |
| `regexOnFullURL`                         | `boolean`                       | No       | `false`           | Match path expressions against complete URLs, including query parameters.                                                              |
| `maxDiscoveryDepth`                      | `number`                        | No       | Provider default  | Maximum link-discovery depth; the root and sitemap pages are depth 0.                                                                  |
| `sitemap`                                | `'skip' \| 'include' \| 'only'` | No       | `include`         | Sitemap discovery mode.                                                                                                                |
| `ignoreQueryParameters`                  | `boolean`                       | No       | `false`           | Avoid scraping the same path with different query parameters.                                                                          |
| `ignoreRobotsTxt`                        | `boolean`                       | No       | `false`           | Crawl pages even when robots.txt disallows them.                                                                                       |
| `deduplicateSimilarURLs`                 | `boolean`                       | No       | Provider default  | Deduplicate URLs Firecrawl considers materially similar.                                                                               |
| `crawlEntireDomain`                      | `boolean`                       | No       | `false`           | Follow sibling and parent paths, not only descendants of the starting path.                                                            |
| `allowSubdomains` / `allowExternalLinks` | `boolean`                       | No       | `false`           | Expand the crawl beyond the starting hostname.                                                                                         |
| `prompt`                                 | `string`                        | No       | None              | Natural-language instructions from which Firecrawl derives crawl options. Explicit inputs take precedence.                             |
| `delay`                                  | `number`                        | No       | None              | Seconds between scrapes. A delay reduces crawl concurrency.                                                                            |
| `maxConcurrency`                         | `number`                        | No       | Team limit        | Maximum simultaneous page scrapes.                                                                                                     |
| `robotsUserAgent`                        | `string`                        | No       | Firecrawl default | User-Agent used to evaluate robots.txt.                                                                                                |
| `scrapeOptions`                          | `object`                        | No       | Markdown          | Per-page formats, browser actions, content filters, cache, proxy, redaction, and target settings accepted by `scrapeUrlWithFirecrawl`. |
| `zeroDataRetention`                      | `boolean`                       | No       | `false`           | Request enterprise zero-data-retention mode when enabled for the Firecrawl team.                                                       |

## Output

Returns Firecrawl's crawl job as `{ id, status, total, completed, creditsUsed?, expiresAt?, next?, data }`. `expiresAt` is a `Date`; `data` contains Firecrawl documents in the same shape returned by `scrapeUrlWithFirecrawl`.

Automate.ax supplies a callback URL to Firecrawl, correlates the terminal callback with the crawl job, and continues from the action that started it. A failed crawl fails the continued action path. Use [`startWebsiteCrawlWithFirecrawl`](/reference/integrations/firecrawl/actions/start-website-crawl) instead when you want the job ID immediately or need to provide your own webhook.