Test extraction scripts

As extraction scripts grow, testing helps catch regressions when pages change or scripts are refactored. Lab is Ferret’s test runner — it executes FQL test files and reports pass/fail results.

This guide walks through the testing workflow. For the complete reference, see the Lab documentation.

Install Lab

$go install github.com/nicktomlin/ferret-lab@latest

Or pull the Docker image:

$docker pull montferret/lab

See Lab Installation for all options.

Write a test file

A test file is a regular .fql script that uses assertion functions from the TESTING namespace. Name the file with a .test.fql extension:

tests/
  headings.test.fql

Run the tests

$lab --dir ./tests

Lab discovers all .test.fql files in the directory and runs them. A test passes when it completes without error; it fails when an assertion fails or a runtime error is raised.

Write expected-failure tests

Name a file with .fail.fql to indicate that it should fail:

This test passes only if the script produces an error.

Write YAML test suites

For testing many queries with structured assertions, use .yaml test files:

# tests/api.test.yaml
tests:
  - name: "posts endpoint returns data"
    query: |
      LET response = IO::NET::HTTP::GET("https://jsonplaceholder.typicode.com/posts")
      LET posts = JSON_PARSE(TO_STRING(response))
      RETURN LENGTH(posts)
    assert:
      gt: 0

  - name: "single post has title"
    query: |
      LET response = IO::NET::HTTP::GET("https://jsonplaceholder.typicode.com/posts/1")
      LET post = JSON_PARSE(TO_STRING(response))
      RETURN post.title
    assert:
      not_empty: true

Use fixtures for reproducible tests

Testing against live websites is fragile — the page may change at any time. Lab can serve static HTML fixtures locally so tests run against stable content.

Create a fixtures directory:

tests/
  fixtures/
    products.html
  products.test.fql

Start the fixture server and run tests:

$lab --dir ./tests --static ./tests/fixtures --static-port 8080

Reference the local server in your test:

See Static File Server for details.

Run tests in CI

Lab can run in a CI pipeline. A common setup:

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run FQL tests
        run: |
          lab --dir ./tests --static ./tests/fixtures --static-port 8080

For browser-backed tests, add a Chromium container:

services:
  chromium:
    image: montferret/chromium
    ports:
      - 9222:9222

See CI for the full reference.

Control test execution

Lab supports several options for test execution:

$lab --dir ./tests --concurrency 4 --timeout 30s --retries 2

• --concurrency — run tests in parallel
• --timeout — maximum time per test
• --retries — retry failed tests

See Runners for all options.

Next steps