mentat_duckdb — mentat as a DuckDB loadable extension

A DuckDB loadable extension that stores mentat’s datoms in DuckDB and runs Datalog over them, mirroring the SQLite CLI and pg_mentat. Each mentat store is a schema in the database the extension is loaded into (mentat for the 'default' store, mentat_<name>_<hash> for any other name), holding ordinary DuckDB tables: datoms, timelined_transactions (+ the transactions view), idents, schema, known_parts, meta. The data persists and checkpoints with the DuckDB database, and plain SQL can read it.

The engine is mentat’s own (transactor, algebrizer, projector, pull), reached through a storage seam; store/ (mentat_duckdb_store) holds the DuckDB schema and the DuckDB SQL. Design: docs/duckdb-native-storage-plan.md.

Install from the registry: INSTALL mentat FROM community; LOAD mentat;.

Pinned versions (IMPORTANT — version-lock caveat)

Component Version
duckdb / libduckdb-sys / duckdb-loadable-macros ~1.10506.0 (= DuckDB v1.5.6)
TARGET_DUCKDB_VERSION v1.5.6
USE_UNSTABLE_C_API 1 (required by duckdb-rs today)

Because duckdb-rs currently requires the unstable C API (USE_UNSTABLE_C_API=1), the produced .duckdb_extension loads only into DuckDB v1.5.6. Forward compatibility is not guaranteed. Bumping DuckDB means bumping the crate pin and TARGET_DUCKDB_VERSION together, then rebuilding (plan §9 risk 1).

This crate is extension-only. The loadable-extension feature replaces the DuckDB C API functions; opening a normal DuckDB Connection from this crate panics with “API not initialized”. The extension’s one connection is cloned from the entrypoint’s (the database handle DuckDB passes the entrypoint is valid only during that call), and every mentat call runs on it, one at a time.

Workspace wiring

crates/duckdb is a workspace member but not a default-member (exactly like crates/pg/pg_mentat), so a plain cargo build / cargo test at the repo root never pulls the DuckDB toolchain. Build it explicitly with -p mentat_duckdb or via the Makefile.

Build

Rust 1.90 (see rust-toolchain.toml). The extension-ci-tools submodule provides the metadata-footer + test harness.

cd crates/duckdb
# one-time: clone the ci-tools helper (a git submodule upstream)
git submodule update --init          # or: git clone https://github.com/duckdb/extension-ci-tools

make configure    # creates configure/venv, pip-installs the sqllogictest runner,
                  # writes configure/platform.txt + configure/extension_version.txt
make debug        # cargo build (cdylib) -> append metadata footer
                  #   -> build/debug/mentat.duckdb_extension
# or: make release

Notes: - The crate is a workspace member, so the Makefile exports CARGO_TARGET_DIR=crates/duckdb/target so the ci-tools rust.Makefile finds the artifact where it expects (./target/debug/, or ./target/<triple>/ for the macOS cross builds). - The cdylib is libmentat_duckdb.{so,dylib} / mentat_duckdb.dll (crate name), not libmentat.*; the Makefile sets RUST_LIBNAME per platform. The footer output is still mentat.duckdb_extension. - EXTENSION_VERSION comes from the workspace version in the root Cargo.toml (so it works without .git); the registry’s CI sets its own.

Compile-check only (no loadable footer)

cargo build -p mentat_duckdb   # from repo root; produces a bare .so, NOT loadable

Load & use

DuckDB refuses unsigned extensions unless started with -unsigned (or allow_unsigned_extensions=true). Use a DuckDB v1.5.6 CLI (the pinned target). The standalone CLI:

curl -sfL -o duckdb_cli.zip \
  https://github.com/duckdb/duckdb/releases/download/v1.5.6/duckdb_cli-linux-amd64.zip
unzip duckdb_cli.zip
./duckdb --version   # v1.5.6 (Variegata)

(The make configure venv installs the latest duckdb PyPI wheel available for the host Python; on Python 3.9 that caps at 1.4.5, which can NOT load a v1.5.6 extension — hence the standalone v1.5.6 CLI for load/smoke tests. On Python 3.10+ the venv wheel is 1.5.6 and make test_debug works.)

-- duckdb -unsigned my.duckdb
LOAD './build/debug/mentat.duckdb_extension';

SELECT edn_t('default',
  '[{:db/ident :person/name :db/valueType :db.type/string :db/cardinality :db.cardinality/one}]');
SELECT edn_t('default', '[{:person/name "Alice"} {:person/name "Bob"}]');
SELECT * FROM edn_q('default', '[:find ?e ?name :where [?e :person/name ?name]]', '{}');
-- The datoms are rows in the `mentat` schema of my.duckdb.
SELECT count(*) FROM mentat.datoms;

Function surface

DuckDB call Kind Returns engine call
edn_t(store VARCHAR, edn VARCHAR) scalar, volatile VARCHAR (JSON tx-report) DuckStore::transact
edn_q(store VARCHAR, query VARCHAR, options VARCHAR) table fn rows, all VARCHAR columns DuckStore::q_json
edn_pull(store VARCHAR, pattern VARCHAR, entity BIGINT) scalar, volatile VARCHAR (JSON map) (pull ?e pattern) via DuckStore::q
edn_eval(store VARCHAR, script VARCHAR) scalar, volatile VARCHAR (EDN) mentat_duckdb_store::script (feature script)

Any NULL argument to a scalar yields NULL. store names a mentat store in the current DuckDB database; it is created (schema, tables, bootstrap transaction) on first use. 'default' (or '') is the schema mentat. Any other name maps to mentat_<name>_<hash>: letters, digits and _ are kept, the rest become _, and a hash of the full name keeps distinct names apart, so a path-like name such as '/tmp/demo.mentat' (the pre-1.11 file argument) is just a name and writes no file.

Transactions and connections

Each edn_t runs in its own DuckDB transaction and commits it, so it is atomic (a failed :db.fn/cas leaves nothing behind) but is not part of a caller’s BEGIN … ROLLBACK. Another DuckDB connection or session sees the change once it commits. Each store’s schema is cached per thread and revalidated by one primary-key read of a generation that every schema-changing transaction replaces, so a schema change made elsewhere is picked up on the next call.

Storage layout

The tables mirror the embedded SQLite store’s, column for column, so the engine’s SQL resolves unchanged. A value v is a UNION(i BIGINT, d DOUBLE, s VARCHAR, b BLOB) with SQLite’s storage-class mapping (refs, booleans, longs, instants -> i; doubles -> d; strings, keywords -> s; uuids, bytes -> b), plus plain copies v_i, v_d, v_s that the DuckDB query dialect uses for filters, joins and numeric comparisons (DuckDB scans and joins a UNION much more slowly than plain columns). DuckDB can’t index a UNION and has no partial indexes, so value lookups scan the attribute’s rows.

edn_q options (JSON, same shape pg_mentat accepts)

'', NULL, {} or JSON null = no options. Otherwise a JSON object with:

Key Meaning
"inputs": [v1, v2, ...] Positional, one element per :in binding form ($ source vars are not counted). The count must match.
"asOf": T Query the database as of tx T (inclusive).
"since": T Only datoms transacted after tx T (pair with a history pattern [?e ?a ?v ?tx ?added]).

asOf and since are mutually exclusive; either may be combined with inputs. Unknown keys and malformed JSON are errors.

Binding forms: scalar ?x -> a JSON value; collection [?x ...] -> a JSON array; tuple [?a ?b] -> a JSON array (one row); relation [[?a ?b]] -> an array of arrays. _ placeholders in a tuple/relation consume a value that is discarded. A query may bind either any number of scalars or exactly one collection/tuple/relation (mentat’s QueryInputs has no public way to combine them yet).

JSON -> value (mirrors pg_mentat’s bind_input_value):

JSON mentat value
integer Long, or Ref if the variable is used as an entity/tx, or as the value of a :db.type/ref attribute, in the :where patterns (incl. or/not)
float Double
true / false Boolean
string starting with : Keyword (":person/name")
any other string String

Output rendering (edn_q)

All columns are VARCHAR. A string value is returned raw (Alice, not "Alice"), so it joins against native DuckDB VARCHAR columns. Keywords keep their colon (:person/name); refs and longs are decimal; instants RFC 3339; uuids hyphenated; booleans true/false. Nested values (pull maps, tuples) render as EDN, where strings are quoted.

edn_pull

pattern is a Datomic pull pattern (EDN vector): [*], [:person/name :person/age], [:person/_friend], [:db/id :person/name]. It runs mentat’s own pull ((pull ?e pattern)). The JSON matches pg_mentat’s edn_pull: keys are attribute idents with the colon (":person/name"), ":db/id" is always the entity id, cardinality-many values are arrays, refs are {":db/id": n}, keywords ":ns/name", instants epoch microseconds, bytes hex. Nested map specs ({:person/friend [:person/name]}) are not supported by mentat’s pull grammar yet.

edn_eval (feature script, default ON)

Runs a mino script with the mentat.store/* prims and returns the last value as EDN text. A no-arg (mentat.store/open) opens the store argument; (mentat.store/open "other") opens another store in the same database. (mentat.store/q ...) on an as-of / since db value runs temporal Datalog. mentat.store/with (speculative transact) is not supported on DuckDB stores and returns an error. The interpreter is sandboxed() (no slurp, spit, or other host filesystem prims) and bounded per call to 10M eval steps, 64 MiB heap, and depth 1000. Each call gets a fresh interpreter (no state carries between calls). mino is pure Rust, so the feature adds no toolchain; build with --no-default-features to leave edn_eval out.

SELECT edn_eval('default', '
  (def c (mentat.store/open))
  (mentat.store/transact c [{:person/name "Carol"}])
  (mentat.store/q (mentat.store/db c) (quote [:find ?n :where [_ :person/name ?n]]))');

Running as a server (Quack)

DuckDB v1.5.6 ships the core quack extension: a DuckDB process serves SQL over HTTP to other DuckDB clients. With mentat loaded in that server, every client gets edn_t/edn_q/edn_pull/edn_eval without loading (or even having) the mentat extension, and all of them share one long-lived process and its DuckDB database, which holds the stores.

Start and stop

export MENTAT_QUACK_TOKEN=$(openssl rand -hex 24)   # clients need this
DUCKDB=/path/to/duckdb-v1.5.6 crates/duckdb/server/serve.sh
# mentat quack server: quack:127.0.0.1:9494 pid 4242 log /run/user/1000/mentat-quack-9494.log
crates/duckdb/server/stop.sh

serve.sh binds 127.0.0.1:9494 (MENTAT_QUACK_HOST, MENTAT_QUACK_PORT), writes a pidfile (MENTAT_QUACK_PIDFILE), runs the server in its own session so it outlives the shell, and keeps the CLI’s stdin open (the DuckDB CLI serves only while stdin is open). The token reaches the server through getenv(), so it is not in any process’s argv. MENTAT_QUACK_FOREGROUND=1 runs it in the foreground for a supervisor; server/mentat-quack.service is an example systemd unit. Everything is done by the DuckDB statements the script writes:

LOAD quack; LOAD '/path/to/mentat.duckdb_extension';
SELECT * FROM quack_serve('quack:127.0.0.1:9494',
  token => getenv('MENTAT_QUACK_TOKEN'), disable_ssl => true);

Connect

quack_query(uri, sql, token =>, disable_ssl =>) runs sql on the server and returns its rows. The client needs only LOAD quack, not mentat:

LOAD quack;
SELECT * FROM quack_query('quack:127.0.0.1:9494',
  $$SELECT edn_t('people', '[{:person/name "Alice"}]')$$,
  token => getenv('MENTAT_QUACK_TOKEN'), disable_ssl => true);
SELECT * FROM quack_query('quack:127.0.0.1:9494',
  $$SELECT * FROM edn_q('people', '[:find ?e ?n :where [?e :person/name ?n]]', '{}')$$,
  token => getenv('MENTAT_QUACK_TOKEN'), disable_ssl => true);

The result is an ordinary table, so it joins against the client’s local tables. Python works the same way (duckdb.connect().execute("LOAD quack"), then the same quack_query). ATTACH also works against a live server and exposes its tables (ATTACH 'quack:127.0.0.1:9494' AS r (TOKEN '…', DISABLE_SSL true); SELECT * FROM r.t; CREATE TABLE r.t2 AS …), but NOT its functions: SELECT * FROM r.edn_q(…) fails with “Table Function with name edn_q does not exist”. Call mentat through quack_query. A wrong token fails with Invalid Input Error: Authentication failed (no token: Could not find a Quack authentication token), for both quack_query and ATTACH.

Security

  • The token is the only gate, and it grants everything. A client with the token runs arbitrary SQL in the server process: any mentat store, and DuckDB’s own file functions (read_text('/etc/…'), COPY … TO) as the server’s OS user. Run the server as a dedicated user confined to the store directory (the systemd unit does). DuckDB’s in-process lockdowns do not help here: with enable_external_access = false, lock_configuration = true or autoload_known_extensions = false set, every Quack request fails with HTTP 500 at v1.5.6. The server insists on a token of at least 4 characters; use 32+ random ones.
  • Bind address. Quack accepts only localhost unless allow_other_hostname is set; serve.sh sets it automatically when MENTAT_QUACK_HOST is not loopback, and warns.
  • TLS. The server speaks plain HTTP (serve.sh passes disable_ssl => true; v1.5.6 reports an http:// listen URL either way). Clients use http:// for localhost/127.0.0.1 and https:// for any other host unless given disable_ssl => true (disable_ssl => false forces https:// even on localhost). So for anything beyond one host, terminate TLS in a reverse proxy in front of the server and connect to the proxy without disable_ssl. Plain HTTP across a network sends the token in clear text.
  • Unsigned extension. The server starts DuckDB with -unsigned because mentat is not a signed community extension yet. That flag lets the server load any unsigned extension file, including one a token holder asks it to LOAD; treat that as part of “the token grants everything”.
  • Listen backlog is 5 (ss -ltn), hard-coded in Quack, and somaxconn can’t raise it. From about 32 concurrent clients, or a single query returning 100k+ rows (the client fetches large results over 30+ parallel connections), the queue overflows and clients retry after 1, 2 then 4 s: 1-5 s stalls. With the backlog raised to 4096 (an LD_PRELOAD shim around listen(), measured in benchmarks/results/duckdb-quack-*), throughput at 32 clients went up 22% and worst-case latency fell from 3-4.6 s to under 0.5 s. The real fix is upstream.

Caveats

  • Per-call cost. Each quack_query opens three new TCP connections (no keep-alive), about 2.2 ms per call on localhost, plus ~35-110 ns per result row. In-process DuckDB is always faster; the server’s value is that clients don’t need mentat loaded and can join results with their own tables.
  • ATTACH 'quack:...' exposes the server’s tables, but not its table functions: r.edn_q(...) fails with “Table Function with name edn_q does not exist”. Use quack_query.
  • Calls are serialized. The extension runs every mentat call on one connection, so concurrent clients' mentat calls take turns (their other SQL doesn’t).

Quack is pre-2.0 at DuckDB v1.5.6 (quack build c154811): its wire protocol and function signatures may change in any DuckDB release, and the mentat extension is version-locked to v1.5.6 anyway (see “Pinned versions”). Server and clients must run the same DuckDB version. The stores live in the server’s DuckDB database, so start the server on a database file (not :memory:) to keep them.

Tests

  • test/smoke.sh — standalone DuckDB v1.5.6 CLI; asserts on every output line, that the datoms are DuckDB rows readable without the extension, that no file is written, and on the error paths. DUCKDB=/path/to/duckdb bash test/smoke.sh.
  • test/sql/mentat.test — SQLLogicTest (the registry’s runner). Needs a Python 3.10+ venv with duckdb==1.5.6 and duckdb-sqllogictest-python, e.g. python -m duckdb_sqllogictest --test-dir test/sql --external-extension build/debug/mentat.duckdb_extension.
  • store/tests-harness — the storage backend outside the extension, on a bundled DuckDB (a separate workspace): a differential test running the same transactions and 60 queries (now, as-of and since every transaction) on the SQLite engine and on DuckDB, the shared scripting model suite, persistence and WAL-replay tests. cargo test --manifest-path crates/duckdb/store/tests-harness/Cargo.toml.

Known gaps

  • :db/fulltext values are stored as plain strings, not tokenized; (fulltext ...) queries aren’t supported.
  • Stores live in the database the extension was loaded into, not in another ATTACHed database.
  • Mentat calls run one at a time (one connection).
  • Stores written by mentat 1.10.x (SQLite files) are not migrated.