Automatic document chunking

pgContext can turn an ordinary PostgreSQL document row into a source-linked, query-visible chunk generation. The source row remains authoritative. Chunk rows, contextual prefixes, deterministic fixture embeddings, jobs, and staging records are derived state and can be rebuilt.

The first versioned parser contract covers plain UTF-8 text, Markdown, and HTML. It uses the versioned unicode_words_v1 tokenizer and deterministic token-safe overlap. PDF, Office, OCR, image/layout parsing, network fetches, and model-driven splitting are not part of this contract.

This capability is Experimental. Its complete PG17 and PG18 lifecycle suites pass, but both frozen one-million-row lanes miss the predeclared 1,000-chunks-per-second publication floor: PG17 measures 10.837 chunks per second and PG18 measures 33.325. Stable promotion therefore remains a measured no-go; the threshold was not relaxed after either result.

Setup

The source table must have a primary key named id, a non-null text column, and a non-null positive bigint source-version column. First create a collection and a user-owned projection table, then register an immutable profile and source:

SELECT pgcontext.create_collection('documents', 'app.documents');
SELECT pgcontext.create_document_chunk_projection('app.document_chunks');

SELECT pgcontext.register_chunking_profile(
    'default',
    'markdown_v1',
    384,  -- target tokens
    512,  -- hard tokens per chunk
    32,   -- useful non-final minimum
    64,   -- adjacent overlap
    8388608,
    true  -- store a separate heading/context prefix
);

SELECT pgcontext.register_document_source(
    'documents', 'body', 'body', 'source_version',
    'app.document_chunks', 'default'
);

The projection belongs to the caller, not the extension. Do not alter its column contract while it is registered. pgContext locks and verifies its OID and required column types before reading or publishing. Registration requires the caller to hold SELECT, INSERT, UPDATE, and DELETE on the projection.

Install the optional transactional outbox trigger when inserts, updates, and deletes should enqueue or invalidate work automatically:

SELECT pgcontext.install_document_chunk_trigger('documents', 'body');

An application may instead enqueue bounded source-key batches explicitly:

SELECT pgcontext.enqueue_document_chunking(
    'documents', 'body', ARRAY['doc-1', 'doc-2']
);

Shadow, promote, rollback, and drain

Each source binds a profile alias, not a cached profile value. An alias has one current profile, one optional non-serving shadow, and at most eight retained predecessors. Pointer changes are constant-size catalog updates; they never rewrite every source, job, or projection row.

SELECT pgcontext.prepare_chunking_profile_alias('default', 'next-profile');
SELECT pgcontext.enqueue_document_chunking_profile(
    'documents', 'body', 'next-profile', ARRAY['doc-1', 'doc-2']
);
SELECT pgcontext.promote_chunking_profile_alias('default', 'next-profile');

Before promotion, the shadow is never query-visible. After promotion, current ready chunks win per source key; retained predecessors fill current-version coverage gaps in most-recent-promotion order. Preparing another shadow does not expose it or replace those fallbacks. Rollback selects the newest retained predecessor. Drain only after its remaining coverage and rollback value are no longer needed:

SELECT pgcontext.rollback_chunking_profile_alias('default');
SELECT pgcontext.drain_chunking_profile_alias('default', 'old-profile');

Preparing a different shadow replaces the old shadow pointer. Existing old shadow work is fail-closed and becomes superseded lazily when claim or another fenced lifecycle boundary encounters it.

Worker lifecycle

Workers claim at most 256 jobs for 1–60,000 milliseconds. A claim returns the job ID, a monotonically increasing fencing token, and a versioned chunk_worker_request_v1 JSON envelope containing only text currently visible to the invoker under PostgreSQL ACL and RLS.

SELECT *
FROM pgcontext.claim_document_chunk_jobs(32, 30000, 'chunk-worker-a');

Send each request as one newline-delimited frame to pgcontext-chunk-worker. The persistent process accepts chunk_worker_request_v1, emits chunk_worker_response_v1, and exits cleanly after a shutdown line. It reads no files and performs no network access.

Stage and publish a complete response with the same fenced token:

SELECT pgcontext.stage_document_chunks(:job_id, :lease_token, :response_jsonb);
SELECT pgcontext.publish_document_chunk_generation(:job_id, :lease_token);

Staging reparses and rechunks the current authoritative row. Every identity, span, token count, link, hash, and output field must equal that canonical pass. Publication rereads the row again, rechecks source and projection OIDs, attnums, types, collation, registration revision, source version/hash, lease, ACL, and RLS, inserts the complete projection, records one fake-embedding job per occurrence, and flips the current alias in the same transaction. Partial or stale work is never returned. A previous ready generation remains visible only while its source version and SHA-256 still match an invoker-visible authoritative row. A source edit, delete, SELECT revocation, or RLS change hides stale chunks immediately, before any replacement is ready.

Use these lifecycle controls for long-running workers:

  • heartbeat_document_chunk_job(job_id, lease_token, lease_millis) renews a current lease without weakening its fencing token.
  • checkpoint_document_chunk_job(job_id, lease_token, status, processed_units, total_units) records monotonic parsing, chunking, and embedding progress.
  • fail_document_chunk_job(job_id, lease_token, error_code) terminates a leased attempt with a bounded content-free worker failure code.
  • cancel_document_chunk_job(job_id) cancels queued work immediately. Active work enters cancel_requested; the current worker acknowledges it at a heartbeat, or bounded expiry cleanup terminalizes it and removes staging.
  • retry_document_chunk_job(job_id) requeues a bounded failed or cancelled attempt; a job has at most three claims.
  • document_chunking_progress(collection, source_name) returns content-free job, progress, staging-byte, and current-document counts.

fake_process_document_chunk_job is an internal deterministic certification helper. Production applications should run the external worker protocol.

Reading, rollback, and deletion

Only current complete aliases are returned:

SELECT source_key, occurrence_id, original_text, start_byte, end_byte,
       structure_path, fake_embedding
FROM pgcontext.current_document_chunks(
    'documents', 'body', ARRAY['doc-1']
)
ORDER BY source_key, ordinal;

original_text and its byte/character span are the citation authority. retrieval_text and the optional context_prefix are derived retrieval input; never substitute them for the citation. Document instructions and markup are data, not trusted prompts.

An owner can make a previously published generation current again only when its source key, version, digest, and registration revision still match the current invoker-visible source row:

SELECT pgcontext.rollback_document_chunk_generation(
    'documents', 'body', 'doc-1', :generation_id
);

Deleting a source row through the installed trigger removes every current alias for that source key and retires the published generation. Explicit invalidation is also available through invalidate_document_chunks(collection, source_name, source_keys).

If a ready or retired user-owned projection is modified after publication and fails its stored digest, the collection/source owner can rebuild it:

SELECT pgcontext.rebuild_document_chunk_job(:job_id);

The function first rechecks current source ACL/RLS and identity, removes the generation’s current alias, clears its projection and fake-embedding rows, and requeues the same immutable source version. A worker must claim, stage, and publish it again before it becomes current. Other job states are not eligible.

Frozen limits

  • source keys per enqueue/read: 256; each key: 1–1,024 bytes
  • source document: at most 8 MiB and 2,000,000 tokens
  • adjacent chunk overlap: at most 64 tokens and strictly below the target
  • chunks per generation: at most 16,384
  • parser structure depth: 32
  • parser structure-path segment: at most 512 UTF-8 bytes
  • retained predecessor profiles per alias: 8
  • staged decoded projection: 32 MiB and 1,000,000 JSONB iterator tokens
  • encoded worker frame: 40 MiB
  • contextual prefix: 96 tokens
  • lease: at most 60 seconds; at most three claims
  • complete parser/worker operation: at most 120 seconds, with cooperative interrupt checks during tokenization and output construction

The 120-second value is an outer parser/worker safety ceiling. A detached worker request also freezes the initial lease deadline, which is at most 60 seconds and is therefore the tighter deadline for that request. A database heartbeat protects later database mutations but does not rewrite an already issued worker envelope; controllers must issue work that can finish inside the claim lease.

The worker admits a frame before allocating it. PostgreSQL admits raw JSONB bytes and iterator nodes before constructing a Rust JSON value. Errors and default diagnostics contain typed reasons and identities, never document or chunk text.

Backup and recovery

Profiles and source registrations participate in extension configuration dump. Source and projection tables follow ordinary PostgreSQL backup/WAL rules. Queued jobs, leases, staging rows, and current aliases are derived operational state and are intentionally excluded from logical extension configuration dump; enqueue or rebuild them after a logical restore. Physical recovery and transaction rollback preserve atomic alias publication through PostgreSQL WAL.

Raw staged response JSON is never exposed by membership-filtered views. The visible staging surface contains only content-free counts, byte totals, and timestamps; source keys are omitted from the public job view.

Current scope and composition

The frozen 28-column projection includes nullable page_number and region fields; the certified plain-text, Markdown, and HTML parsers emit them as null. PDF/layout adapters and source MIME, title, URI, language, tenant, and arbitrary metadata bindings remain outside this Experimental parser contract.

Profile aliases support bounded shadow publication, constant-size promotion, ordered retained fallback, rollback, and explicit drain. Published projection rows preserve exact citation spans and can be registered with the existing P11 multi-model and P12 detached-rerank APIs. P13 does not automatically provision those query registrations or choose application-specific embedding/rerank models; that composition remains explicit so PostgreSQL authority and profile selection stay visible to the operator.