Scrape a web page

scrapeWeb opens one page in a managed browser and returns its content as Markdown. Automate.ax supplies the browser account, runs the scrape in the background, and counts Cloudflare’s reported browser time against the organization’s monthly web scraping allowance. Use it when a normal HTTP request does not include content rendered by JavaScript, or when the page needs cookies, HTTP authentication, custom headers, injected scripts or styles, resource filtering, viewport settings, or an explicit wait.

Scrape a page

automations/scrape-product-page.automation.ts

import {
  automation,
  onDashboardRun,
  scrapeWeb,
  sendEmail,
  withPrerequisites,
} from "automate.ax"

export default automation("Scrape product page", () => {
  const run = onDashboardRun({ title: "Scrape product page" })

  const page = withPrerequisites(run, () =>
    scrapeWeb({
      url: "https://example.com/products/launch",
      waitForSelector: { selector: "main", visible: true },
      rejectResourceTypes: ["image", "media", "font"],
    }),
  )

  sendEmail({
    markdown: page,
    subject: "Latest product page",
  })
})

The scrape starts after the dashboard run and the email waits for its Markdown result. A terminal browser failure fails the result signal and prevents dependent actions from running.

Inputs

Provide exactly one page source:

Input	Type	Description
`url`	`string`	Public or authenticated page URL to open.
`html`	`string`	Inline HTML document to render instead of opening a URL.

The complete request may be up to 5 MiB after encoding. This limit mainly affects large inline HTML documents. The action accepts these Browser Run controls. Every input accepts a compatible signal.

Input	Type	Description
`setExtraHttpHeaders`	`Record<string, string>`	Arbitrary page request headers. Header names and values pass through unchanged.
`cookies`	`Cookie[]`	Browser cookies, including domain or URL, path, expiry, security, same-site, priority, partition, source scheme, and source port controls.
`authenticate`	`{ username: string; password: string }`	HTTP authentication credentials.
`userAgent`	`string`	Browser user agent.
`addScriptTag`	`{ content?, id?, type?, url? }[]`	Scripts to inject before extraction.
`addStyleTag`	`{ content?, url? }[]`	Styles to inject before extraction.
`setJavaScriptEnabled`	`boolean`	Enable or disable page JavaScript.
`viewport`	`{ width; height; deviceScaleFactor?; hasTouch?; isLandscape?; isMobile? }`	Browser viewport and device emulation.
`emulateMediaType`	`string`	CSS media type such as `screen` or `print`.
`gotoOptions`	`{ referer?; referrerPolicy?; timeout?; waitUntil? }`	Navigation timeout, referrer, policy, and lifecycle wait. `waitUntil` accepts one value or an array of `load`, `domcontentloaded`, `networkidle0`, and `networkidle2`.
`waitForSelector`	`{ selector; timeout?; visible?: true; hidden?: true }`	Wait for an element state before extraction. Choose at most one of `visible` or `hidden`.
`waitForTimeout`	`number`	Additional wait in milliseconds, up to 120 seconds.
`actionTimeout`	`number`	Markdown extraction timeout after the page loads, up to 120 seconds.
`allowRequestPattern`	`string[]`	Allow requests matching these URL patterns.
`rejectRequestPattern`	`string[]`	Reject requests matching these URL patterns.
`allowResourceTypes`	`ResourceType[]`	Allow only selected browser resource types.
`rejectResourceTypes`	`ResourceType[]`	Reject selected browser resource types.
`bestAttempt`	`boolean`	Return the best available result when optional browser steps fail.

ResourceType supports document, stylesheet, image, media, font, script, texttrack, xhr, fetch, prefetch, eventsource, websocket, manifest, signedexchange, ping, cspviolationreport, preflight, and other. See Cloudflare’s Browser Rendering Markdown API for provider behavior. Automate.ax validates the public camelCase fields and sends Cloudflare’s exact wire names.

Output

scrapeWeb returns a Signal<string> containing the rendered page as Markdown. Browser work runs outside the action sandbox. The starting action queues durable work, and a correlated completion emits the public result signal. Cloudflare’s X-Browser-Ms-Used value is recorded for every provider response that includes it. Sub-millisecond values round up to a whole millisecond. Automate.ax disables Cloudflare’s shared URL cache for this platform action so authenticated requests remain isolated between organizations.

Use your own Cloudflare account

Use cloudflare.extractMarkdown from automate.ax/cloudflare when you want to supply a connected Cloudflare account. It also exposes Cloudflare’s cacheTtl control, charges that Cloudflare account directly, and does not consume the Automate.ax web scraping allowance.

Built-in triggers

Built-in actions

Runtime utilities

Integrations

Scrape a page

Inputs

Output

Use your own Cloudflare account

​Scrape a page

​Inputs

​Output

​Use your own Cloudflare account

Scrape a page

Inputs

Output

Use your own Cloudflare account