View all results

Embed Ferret in a Go application

Embedding Ferret lets you run FQL queries inside your Go application — useful for user-defined extraction rules, dynamic data pipelines, or sandboxed scripting. This guide walks through building a Go service that accepts queries over HTTP.

For the full native API reference, see Go embedding.

FQL source is code. Do not insert URLs, selectors, HTML, credentials, request bodies, or other runtime values with fmt.Sprintf or string concatenation. Pass them as parameters instead. See Construct FQL safely.

Set up the project

Create a new Go module:

terminal
mkdir ferret-service && cd ferret-servicego mod init ferret-service

Follow the Go installation instructions to use the consumable-output API in this guide. Then run a one-shot query to verify the setup:

example.go
read-only
package main import ( "context" "errors" "fmt" "log" "github.com/MontFerret/ferret/v2" ) func main() { if err := run(); err != nil { log.Fatal(err) } } func run() (err error) { engine, err := ferret.New() if err != nil { return err } defer func() { err = errors.Join(err, engine.Close()) }() output, err := engine.Run( context.Background(), ferret.NewAnonymousSource(`return { message: "Ferret is running" }`), ) if err != nil { return err } defer func() { err = errors.Join(err, output.Close()) }() content, err := output.Collect(context.Background()) if err != nil { return err } fmt.Println(string(content.Data)) return nil }

Choose a standard library profile

The full standard library includes file system and network access. For a service that runs user-submitted queries, restrict it:

example.go
read-only
import "github.com/MontFerret/ferret/v2/pkg/stdlib"
Profile What it does When to use
stdlib.Full() All function groups Trusted scripts, internal pipelines
stdlib.Safe() Full minus IO (file system + network) User-submitted queries
stdlib.Only(groups...) Only the listed groups Minimal sandboxes
example.go
read-only
engine, err := ferret.New( ferret.WithStdlib(stdlib.Safe()), )

See Runtime Configuration for the full list of groups.

Configure the HTML module

This service needs the HTML module to fetch and query pages. Install it from the contrib repository:

terminal
go get github.com/MontFerret/contrib/modules/web/html@v1.0.0-rc.29

Configure the in-process memory driver for static HTML, then pass the module to ferret.WithModules:

example.go
read-only
import ( "github.com/MontFerret/contrib/modules/web/html" "github.com/MontFerret/contrib/modules/web/html/drivers/memory" ) htmlmod := html.New( html.WithDefaultDriver(memory.New()), ) engine, err := ferret.New( ferret.WithStdlib(stdlib.Safe()), ferret.WithModules(htmlmod), )

The memory driver fetches and parses static HTML without a browser. Ferret modules are installed, configured, and registered explicitly by the host application. See Modules for the complete registration and lifecycle model.

Add browser support

For JavaScript-rendered pages, add the cdp driver alongside the memory driver:

example.go
read-only
import ( "github.com/MontFerret/contrib/modules/web/html" "github.com/MontFerret/contrib/modules/web/html/drivers/cdp" "github.com/MontFerret/contrib/modules/web/html/drivers/memory" ) htmlmod := html.New( html.WithDefaultDriver(memory.New()), html.WithDrivers( cdp.New(), ), ) engine, err := ferret.New( ferret.WithStdlib(stdlib.Safe()), ferret.WithModules(htmlmod), )

Scripts select the driver at query time with { driver: "cdp" }:

example.fql
read-only
let page = web::html::open("https://mockery.ferretlang.org", { driver: "cdp" }) return page.title

Without the driver option, the default memory driver is used.

Format source without executing it

Use pkg/formatter to format FQL independently of an engine. Formatting checks syntax without resolving host functions or executing the program, so scripts can reference functions unavailable to the formatting application.

example.go
read-only
import ( "bytes" "github.com/MontFerret/ferret/v2/pkg/formatter" "github.com/MontFerret/ferret/v2/pkg/source" ) func formatSource(text string) (string, error) { f, err := formatter.New() if err != nil { return "", err } var out bytes.Buffer if err := f.Format(&out, source.New("query.fql", text)); err != nil { return "", err } return out.String(), nil }

formatter.WithSingleQuote(true) prefers single quotes for ordinary string literals; the default prefers double quotes. Quote conversion preserves the exact string value. If a conversion cannot safely preserve that value, the formatter keeps the original spelling. For example, the default formatter retains 'a"b'. Literal bodies, including backslashes and embedded line breaks, are preserved, and template chunks keep their original spelling.

Format writes directly to its writer. Any failure can leave partial output; use the output only after a nil error. The example buffers at the caller so it can discard incomplete output. A formatter can be reused after a failure and shared across goroutines with separate writers.

Native diagnostic spans are zero-based, half-open UTF-8 byte offsets into the original source. Native positions use one-based lines and byte columns. Convert these coordinates to an editor’s coordinate system in the integration layer.

Recovered panics return a structured diagnostic with kind diagnostics.UnexpectedError, available through errors.As. Ordinary writer errors retain their identity for errors.Is. Internal diagnostics can contain panic and stack details; service adapters should classify them and return a generic internal-error response instead of exposing those details.

Compile and reuse a plan

When the same query runs multiple times — with different parameters, in different goroutines, or on a schedule — compile it once and reuse the plan:

example.go
read-only
plan, err := engine.Compile(ctx, ferret.NewSource( "extract-title.fql", `let page = web::html::open(@url) return page.title`, )) if err != nil { return err } defer plan.Close() urls := []string{ "https://mockery.ferretlang.org", "https://mockery.ferretlang.org/scenarios/ecommerce/", } // This snippet also imports "errors". for _, url := range urls { session, err := plan.NewSession(ctx, ferret.WithSessionParam("url", url), ) if err != nil { return err } output, err := session.Run(ctx) if err != nil { return errors.Join(err, session.Close()) } content, collectErr := output.Collect(ctx) closeErr := errors.Join(output.Close(), session.Close()) if err := errors.Join(collectErr, closeErr); err != nil { return err } fmt.Println(string(content.Data)) }

The plan manages an internal pool of virtual machines. Sessions borrow a VM from the pool and return it on close, so reusing a plan avoids recompilation and reduces allocation overhead.

Build an HTTP handler

Collect before writing response headers or bytes so terminal failures can still produce an error status. For forwarding to a writer without HTTP status handling, see Output.

The following handler accepts a JSON body with query and optional params, compiles and runs the query, and returns the result:

example.go
read-only
package main import ( "context" "errors" "encoding/json" "log" "net/http" "time" "github.com/MontFerret/ferret/v2" "github.com/MontFerret/ferret/v2/pkg/stdlib" "github.com/MontFerret/contrib/modules/web/html" "github.com/MontFerret/contrib/modules/web/html/drivers/memory" ) type QueryRequest struct { Query string `json:"query"` Params map[string]any `json:"params,omitempty"` } func main() { if err := run(); err != nil { log.Fatal(err) } } func run() (err error) { htmlmod := html.New( html.WithDefaultDriver(memory.New()), ) engine, err := ferret.New( ferret.WithStdlib(stdlib.Safe()), ferret.WithModules(htmlmod), ) if err != nil { return err } defer func() { err = errors.Join(err, engine.Close()) }() http.HandleFunc("/api/query", func(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { http.Error(w, "method not allowed", http.StatusMethodNotAllowed) return } var req QueryRequest if err := json.NewDecoder(r.Body).Decode(&req); err != nil { http.Error(w, "invalid request body", http.StatusBadRequest) return } ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second) defer cancel() plan, err := engine.Compile(ctx, ferret.NewSource("api-query.fql", req.Query), ) if err != nil { http.Error(w, err.Error(), http.StatusUnprocessableEntity) return } defer plan.Close() session, err := plan.NewSession(ctx, ferret.WithSessionParams(req.Params), ) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } defer session.Close() output, err := session.Run(ctx) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } defer output.Close() content, err := output.Collect(ctx) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if content == nil { w.WriteHeader(http.StatusNoContent) return } w.Header().Set("Content-Type", content.Metadata.ContentType) if _, err := w.Write(content.Data); err != nil { log.Printf("write response: %v", err) } }) log.Println("listening on :8080") return http.ListenAndServe(":8080", nil) }

Test it:

terminal
curl -s -X POST http://localhost:8080/api/query \  -H "Content-Type: application/json" \  -d '{"query": "return upper(@name)", "params": {"name": "ferret"}}'
For production use, consider caching compiled plans by query hash to avoid recompilation on repeated queries.

Handle errors

Compilation and runtime errors are separate concerns:

example.go
read-only
plan, err := engine.Compile(ctx, ferret.NewSource("user-script.fql", query), ) if err != nil { // Compilation error — syntax or semantic issue. // The error includes source location (line and column). // Safe to return to the user. http.Error(w, err.Error(), http.StatusUnprocessableEntity) return }

Handle preparation errors from session.Run, then runtime and encoding errors from consumption:

example.go
read-only
output, err := session.Run(ctx) if err != nil { // Preparation failed, possibly before VM entry due to cancellation. http.Error(w, err.Error(), http.StatusInternalServerError) return } defer output.Close() content, err := output.Collect(ctx) if err != nil { // Execution, encoding, hooks, cancellation, or cleanup failed. http.Error(w, err.Error(), http.StatusInternalServerError) return } _ = content // write only after the terminal error check

Runtime error identities remain available through diagnostic wrapping. Use the public runtime sentinels with errors.Is when the host needs to distinguish zero-divisor failures:

example.go
read-only
switch { case errors.Is(err, runtime.ErrDivisionByZero): // Int, Float, or Duration division by zero case errors.Is(err, runtime.ErrModuloByZero): // Int or Float modulo by zero case errors.Is(err, runtime.ErrInvalidOperation): // another unsupported runtime operation }

runtime.ErrDivisionByZero and runtime.ErrModuloByZero are the canonical identities. The existing vm.ErrDivisionByZero and vm.ErrModuloByZero names remain aliases for source compatibility, but embedding code should prefer the runtime package.

Always set a timeout on the context to prevent runaway queries:

example.go
read-only
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second) defer cancel()

The VM checks cancellation at structural execution boundaries. Native work between those boundaries completes atomically. Any blocking or remote host operation receives the execution context and is responsible for observing cancellation while it retains control. A returned context.Canceled or context.DeadlineExceeded, including a wrapped error, bypasses FQL recovery. Cancellation before VM admission can fail Session.Run; after admission it is reported by output consumption, with recorded failures preserved.

Control concurrency

Three options control how the engine manages concurrent execution:

example.go
read-only
engine, err := ferret.New( ferret.WithStdlib(stdlib.Safe()), ferret.WithModules(htmlmod), ferret.WithMaxActiveSessions(50), ferret.WithMaxVMsPerPlan(8), ferret.WithMaxIdleVMsPerPlan(4), )
Option Effect
WithMaxActiveSessions Engine-wide cap on concurrent sessions. NewSession blocks when full.
WithMaxVMsPerPlan Per-plan cap on total VMs (idle + active). Bounds resource use for hot queries.
WithMaxIdleVMsPerPlan Per-plan idle VM cache size. Trade memory for faster session creation.

See Runtime Configuration for details on how these interact.

Shut down cleanly

Stop accepting new requests and drain active handlers before closing the engine. Each handler owns its output, session, and plan. The complete example gives each query a deadline and waits for HTTP shutdown before releasing the engine.

Complete example

A full runnable service combining everything above:

example.go
read-only
package main import ( "context" "encoding/json" "errors" "log" "net/http" "os" "os/signal" "syscall" "time" "github.com/MontFerret/ferret/v2" "github.com/MontFerret/ferret/v2/pkg/stdlib" "github.com/MontFerret/contrib/modules/web/html" "github.com/MontFerret/contrib/modules/web/html/drivers/cdp" "github.com/MontFerret/contrib/modules/web/html/drivers/memory" ) type QueryRequest struct { Query string `json:"query"` Params map[string]any `json:"params,omitempty"` } func main() { if err := serve(); err != nil { log.Fatal(err) } } func serve() (err error) { htmlmod := html.New( html.WithDefaultDriver(memory.New()), html.WithDrivers( cdp.New(), ), ) engine, err := ferret.New( ferret.WithStdlib(stdlib.Safe()), ferret.WithModules(htmlmod), ferret.WithMaxActiveSessions(50), ferret.WithMaxVMsPerPlan(8), ferret.WithMaxIdleVMsPerPlan(4), ) if err != nil { return err } defer func() { err = errors.Join(err, engine.Close()) }() mux := http.NewServeMux() mux.HandleFunc("/api/query", func(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { http.Error(w, "method not allowed", http.StatusMethodNotAllowed) return } var req QueryRequest if err := json.NewDecoder(r.Body).Decode(&req); err != nil { http.Error(w, "invalid request body", http.StatusBadRequest) return } ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second) defer cancel() plan, err := engine.Compile(ctx, ferret.NewSource("api-query.fql", req.Query), ) if err != nil { http.Error(w, err.Error(), http.StatusUnprocessableEntity) return } defer plan.Close() session, err := plan.NewSession(ctx, ferret.WithSessionParams(req.Params), ) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } defer session.Close() output, err := session.Run(ctx) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } defer output.Close() content, err := output.Collect(ctx) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if content == nil { w.WriteHeader(http.StatusNoContent) return } w.Header().Set("Content-Type", content.Metadata.ContentType) if _, err := w.Write(content.Data); err != nil { log.Printf("write response: %v", err) } }) srv := &http.Server{ Addr: ":8080", Handler: mux, } shutdown, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() served := make(chan error, 1) go func() { served <- srv.ListenAndServe() }() select { case <-shutdown.Done(): case err = <-served: if errors.Is(err, http.ErrServerClosed) { err = nil } } return errors.Join(err, srv.Shutdown(context.Background())) }

Next steps