> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automate.ax/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Automate.ax automations are TypeScript programs.
> Use Bun for installation and command examples.
> Use Automate.ax for the product name and automate.ax for the package, CLI, and domain.
> Treat documented public APIs as current; do not invent transitional or deprecated names.

# Scrape a web page

> Render a URL or HTML document and return its content as Markdown.

`scrapeWeb` opens one page in a managed browser and returns its content as Markdown. Automate.ax supplies the browser account, runs the scrape in the background, and counts Cloudflare's reported browser time against the organization's monthly [web scraping allowance](/concepts/platform-resources#count-web-scraping-usage).

Use it when a normal HTTP request does not include content rendered by JavaScript, or when the page needs cookies, HTTP authentication, custom headers, injected scripts or styles, resource filtering, viewport settings, or an explicit wait.

## Scrape a page

```ts automations/scrape-product-page.automation.ts theme={null}
import {
  automation,
  onDashboardRun,
  scrapeWeb,
  sendEmail,
  withPrerequisites,
} from "automate.ax"

export default automation("Scrape product page", () => {
  const run = onDashboardRun({ title: "Scrape product page" })

  const page = withPrerequisites(run, () =>
    scrapeWeb({
      url: "https://example.com/products/launch",
      waitForSelector: { selector: "main", visible: true },
      rejectResourceTypes: ["image", "media", "font"],
    }),
  )

  sendEmail({
    markdown: page,
    subject: "Latest product page",
  })
})
```

The scrape starts after the dashboard run and the email waits for its Markdown result. A terminal browser failure fails the result signal and prevents dependent actions from running.

## Inputs

Provide exactly one page source:

| Input  | Type     | Description                                              |
| ------ | -------- | -------------------------------------------------------- |
| `url`  | `string` | Public or authenticated page URL to open.                |
| `html` | `string` | Inline HTML document to render instead of opening a URL. |

The complete request may be up to 5 MiB after encoding. This limit mainly affects large inline HTML documents.

The action accepts these Browser Run controls. Every input accepts a compatible signal.

| Input                  | Type                                                                        | Description                                                                                                                                                            |
| ---------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `setExtraHttpHeaders`  | `Record<string, string>`                                                    | Arbitrary page request headers. Header names and values pass through unchanged.                                                                                        |
| `cookies`              | `Cookie[]`                                                                  | Browser cookies, including domain or URL, path, expiry, security, same-site, priority, partition, source scheme, and source port controls.                             |
| `authenticate`         | `{ username: string; password: string }`                                    | HTTP authentication credentials.                                                                                                                                       |
| `userAgent`            | `string`                                                                    | Browser user agent.                                                                                                                                                    |
| `addScriptTag`         | `{ content?, id?, type?, url? }[]`                                          | Scripts to inject before extraction.                                                                                                                                   |
| `addStyleTag`          | `{ content?, url? }[]`                                                      | Styles to inject before extraction.                                                                                                                                    |
| `setJavaScriptEnabled` | `boolean`                                                                   | Enable or disable page JavaScript.                                                                                                                                     |
| `viewport`             | `{ width; height; deviceScaleFactor?; hasTouch?; isLandscape?; isMobile? }` | Browser viewport and device emulation.                                                                                                                                 |
| `emulateMediaType`     | `string`                                                                    | CSS media type such as `screen` or `print`.                                                                                                                            |
| `gotoOptions`          | `{ referer?; referrerPolicy?; timeout?; waitUntil? }`                       | Navigation timeout, referrer, policy, and lifecycle wait. `waitUntil` accepts one value or an array of `load`, `domcontentloaded`, `networkidle0`, and `networkidle2`. |
| `waitForSelector`      | `{ selector; timeout?; visible?: true; hidden?: true }`                     | Wait for an element state before extraction. Choose at most one of `visible` or `hidden`.                                                                              |
| `waitForTimeout`       | `number`                                                                    | Additional wait in milliseconds, up to 120 seconds.                                                                                                                    |
| `actionTimeout`        | `number`                                                                    | Markdown extraction timeout after the page loads, up to 120 seconds.                                                                                                   |
| `allowRequestPattern`  | `string[]`                                                                  | Allow requests matching these URL patterns.                                                                                                                            |
| `rejectRequestPattern` | `string[]`                                                                  | Reject requests matching these URL patterns.                                                                                                                           |
| `allowResourceTypes`   | `ResourceType[]`                                                            | Allow only selected browser resource types.                                                                                                                            |
| `rejectResourceTypes`  | `ResourceType[]`                                                            | Reject selected browser resource types.                                                                                                                                |
| `bestAttempt`          | `boolean`                                                                   | Return the best available result when optional browser steps fail.                                                                                                     |

`ResourceType` supports `document`, `stylesheet`, `image`, `media`, `font`, `script`, `texttrack`, `xhr`, `fetch`, `prefetch`, `eventsource`, `websocket`, `manifest`, `signedexchange`, `ping`, `cspviolationreport`, `preflight`, and `other`.

See Cloudflare's [Browser Rendering Markdown API](https://developers.cloudflare.com/api/resources/browser_rendering/subresources/markdown/methods/create/) for provider behavior. Automate.ax validates the public camelCase fields and sends Cloudflare's exact wire names.

## Output

`scrapeWeb` returns a `Signal<string>` containing the rendered page as Markdown.

Browser work runs outside the action sandbox. The starting action queues durable work, and a correlated completion emits the public result signal. Cloudflare's `X-Browser-Ms-Used` value is recorded for every provider response that includes it. Sub-millisecond values round up to a whole millisecond.

Automate.ax disables Cloudflare's shared URL cache for this platform action so authenticated requests remain isolated between organizations.

## Use your own Cloudflare account

Use `cloudflare.extractMarkdown` from `automate.ax/cloudflare` when you want to supply a connected Cloudflare account. It also exposes Cloudflare's `cacheTtl` control, charges that Cloudflare account directly, and does not consume the Automate.ax web scraping allowance.