View all results

Work with browser-driven pages

Some pages load content through JavaScript — single-page applications, lazy-loaded lists, content behind client-side rendering. For these, Ferret can drive a real browser through the Chrome DevTools Protocol (CDP).

This guide covers when to use a browser, how to set one up, and how to wait for dynamic content.

When to use a browser

Use static extraction (WEB::HTML::OPEN without a driver) when the data is in the initial HTML response. Use the cdp driver when the page:

  • renders content with JavaScript
  • loads data asynchronously after the initial page load
  • requires user interaction (clicking, scrolling, filling forms) before content appears
  • depends on browser APIs (cookies, local storage, service workers)

If you are unsure, try static extraction first. It is faster and simpler. Switch to the browser driver only if the data is missing from the static HTML.

Set up a browser

Ferret connects to a Chrome or Chromium instance over CDP.

Option 1: Docker (recommended for reproducible environments)

terminal 
docker run -d -p 9222:9222 montferret/chromium

Option 2: Managed browser (the CLI starts and stops a browser for you)

terminal 
ferret browser start

Option 3: Connect to an existing browser launched with remote debugging enabled.

See CLI Browser for full details on browser management.

Open a page with a browser

Pass { driver: "cdp" } to WEB::HTML::OPEN:

example.fql Ferret v2
query.fql
FQL
LET page = WEB::HTML::OPEN("https://mockery.ferretlang.org", { driver: "cdp" }) RETURN page.title

Once opened, querying and element access work the same as with static pages. The difference is that the page value reflects the live browser DOM, including content added by JavaScript.

Wait for content

JavaScript-rendered pages may not have all content immediately after the page loads. Use WAITFOR to pause until the data appears.

Wait for an element

example.fql Ferret v2
query.fql
FQL
LET page = WEB::HTML::OPEN("https://mockery.ferretlang.org/scenarios/delayed-rendering/", { driver: "cdp" }) WAITFOR EXISTS QUERY ONE '[data-testid="delayed-long"]' IN page USING css TIMEOUT 5s RETURN QUERY '[data-testid="delayed-long"]' IN page USING css

WAITFOR EXISTS re-checks the expression on a polling interval until it is non-empty according to EXISTS semantics or the timeout is reached.

Wait for a value

Use WAITFOR VALUE when you need the result of the check itself:

example.fql Ferret v2
query.fql
FQL
LET page = WEB::HTML::OPEN("https://mockery.ferretlang.org/scenarios/dynamic-products/delayed/", { driver: "cdp" }) LET element = WAITFOR VALUE QUERY ONE '[data-testid="dynamic-product-card"]' IN page USING css TIMEOUT 10s EVERY 200ms RETURN element?.textContent

Wait for network idle

After a navigation or interaction, you may want to wait until the page finishes loading resources:

example.fql Ferret v2
query.fql
FQL
LET page = WEB::HTML::OPEN("https://mockery.ferretlang.org", { driver: "cdp" }) WAITFOR EVENT "network.idle" IN page TIMEOUT 10s RETURN QUERY "article" IN page USING css

Tune the polling

WAITFOR supports several clauses to control timing:

example.fql
read-only
WAITFOR EXISTS QUERY ONE ".result" IN page USING css TIMEOUT 10s EVERY 100ms, 2s BACKOFF EXPONENTIAL JITTER 0.1
  • EVERY — how often to re-check (with optional cap)
  • BACKOFF — how the interval grows (LINEAR, EXPONENTIAL, or NONE)
  • JITTER — randomize the interval to avoid synchronized retries
  • TIMEOUT — maximum wait time

Handle timeouts

When WAITFOR times out, it returns false (or NONE for WAITFOR VALUE). Add ON TIMEOUT RETURN for a custom fallback:

example.fql Ferret v2
query.fql
FQL
LET page = WEB::HTML::OPEN("https://mockery.ferretlang.org", { driver: "cdp" }) LET items = WAITFOR VALUE QUERY ".product" IN page USING css WHEN LENGTH(.) > 0 TIMEOUT 5s ON TIMEOUT RETURN [] RETURN items

Extract from a browser-backed page

Once the content is loaded, extraction works the same as with static pages:

example.fql Ferret v2
query.fql
FQL
LET page = WEB::HTML::OPEN("https://mockery.ferretlang.org/scenarios/dynamic-products/basic/", { driver: "cdp" }) LET grid = WAITFOR VALUE QUERY ONE "#dynamic-products-grid" IN page USING css WAITFOR EXISTS QUERY '[data-testid="dynamic-product-card"]' IN page USING css FOR product IN grid.children LET title = product[~? css`.product-title`] RETURN { title: title?.textContent, price: product[~? css`.product-price`], inStock: product[~? css`.product-in-stock`]?.attributes["data-stock-count"] != "0" }

Navigate within a page

Use NAVIGATE to go to a different URL within the same browser session:

example.fql Ferret v2
query.fql
FQL
LET page = WEB::HTML::OPEN("https://mockery.ferretlang.org", { driver: "cdp" }) LET titleBefore = page.title LET link = QUERY ONE ":nth(1, nav li)" IN page USING css WAITFOR EVENT "navigation" IN page TRIGGER ( DISPATCH "click" IN link ) TIMEOUT 10s LET titleAfter = page.title RETURN { titleBefore, titleAfter }

NAVIGATE_BACK(page) and NAVIGATE_FORWARD(page) move through the browser history.

Next steps

Guide Interact with pages Click elements, fill forms, and trigger browser events. Guide Pagination patterns Collect data across multiple pages using common pagination patterns. Learn Waitfor Expressions Suspend evaluation until a condition becomes true or an event arrives. Browser runtime Browser Start and stop managed browser instances for browser-backed scripts.