Rebuildable Storage Artifacts

pgContext treats segment artifacts as rebuildable cache data. Losing a segment file must not lose primary PostgreSQL table data. This is an intentional operational difference: backups must preserve source tables and pgContext catalog metadata, while acceleration files may be validated and copied for speed or rebuilt after restore.

The current context-storage segment format has a fixed 40-byte header followed by payload bytes. Header fields are little-endian and include:

  • magic bytes identifying pgContext segment files;
  • a format version, currently 1;
  • an endian marker;
  • a segment kind;
  • reserved bytes, currently zero;
  • payload byte length;
  • a deterministic checksum.

Format version 1 is the current segment format. pgContext writes only version 1 artifacts, and the readable-version window is fixed at minimum 1, maximum 1. The loader rejects unknown versions instead of attempting best-effort reads. Future incompatible formats must use a new version number, keep old loader behavior pinned by tests, widen the readable-version window only when backwards loading is supported, and document whether operators should rebuild artifacts from source tables or run an explicit import/export migration.

The checksum covers the header with the checksum field zeroed plus the payload. It is used to detect accidental artifact corruption, not to authenticate data.

Experimental SQL helpers expose the same byte-level contract for tests, diagnostics, and artifact-publication plumbing:

  • pgcontext.encode_artifact_segment(kind text, payload bytea) writes a versioned segment header and checksum around a rebuildable payload. The currently exposed kind is hnsw_graph.
  • pgcontext.validate_artifact_segment(segment bytea) validates segment bytes through the mmap-safe loader and reports kind, payload length, and checksum. Malformed segment bytes raise SQLSTATE XX001; unsupported segment kinds raise SQLSTATE 22023.
  • pgcontext.publish_artifact_segment(build_job_id bigint, segment bytea) validates bytes for a completed visible segment or mmap build job and records collection-owned manifest metadata: build job, artifact target, segment kind, format version, payload length, checksum, and lifecycle state.
  • pgcontext.publish_artifact_segment_file(build_job_id bigint, segment bytea) performs the same validation, writes the bytes through the context-storage atomic writer under a generated PostgreSQL data-directory-relative pgcontext_artifacts/... path, reloads the file through the validator, and records the generated relative path with file_materialized lifecycle state.
  • pgcontext.artifact_segments(collection text) lists those visible manifests, including generated relative paths for file-materialized artifacts.
  • pgcontext.artifact_segment_memory(collection text) reports deterministic mmap budget diagnostics for each visible manifest: payload bytes, fixed header bytes, total mapped bytes, lifecycle state, and whether a file has been materialized.
  • pgcontext.artifact_segment_diagnostics(collection text) reloads file-materialized manifests through the segment loader, rejects catalog paths that escape pgcontext_artifacts/..., and reports constrained text statuses: ready, metadata_only, artifact_missing, checksum_mismatch, artifact_corrupt, metadata_mismatch, or path_rejected. It also reports deterministic repair_advice and cleanup_eligible columns. Ready artifacts need no action; metadata-only, retired, or pathless manifests are not cleanup candidates; rejected paths must be fixed or removed from the catalog before cleanup; and missing, corrupt, checksum-drifted, or metadata-drifted artifacts should be retired or rebuilt after investigation. cleanup_eligible is true only for root-confined materialized artifact paths that retire_artifact_segment may clean up.
  • pgcontext.retire_artifact_segment(artifact_id bigint) marks the manifest retired, then attempts to remove the generated materialized artifact file when it still exists. It tolerates already-missing files, rejects catalog paths that escape pgcontext_artifacts/..., and does not rebuild or replace artifacts.
  • pgcontext.cleanup_artifact_segments(collection text, dry_run boolean) scans visible file-materialized manifests for loader-visible cleanup candidates: missing files, checksum drift, corrupt files, or catalog/file metadata drift. It also scans the generated per-collection artifact directory for regular .pgctxseg files that are not referenced by any visible manifest, which can happen if a backend stops after the atomic file write but before catalog publication. With dry_run = true, it reports the manifests it would retire and orphan files it would remove without mutating catalog rows or files. With dry_run = false, it retires those manifests and removes generated files when they still exist, and removes orphan .pgctxseg files. Ready artifacts, metadata-only artifacts, symlinks, directories, and non-segment files are skipped; escaped catalog paths raise SQLSTATE 22023 instead of being cleaned up.

The metadata-only publisher does not store payload bytes or write files. The file-materialization publisher never accepts an operator-supplied path and does not derive storage paths from artifact names or targets; it uses generated identifier-based paths under the PostgreSQL data directory. The memory diagnostic is catalog-derived and does not open or trust artifact files; the file diagnostic opens only generated, root-confined artifact paths before classifying corruption or catalog drift. The retire and cleanup helpers attempt cleanup only for generated, root-confined artifact paths. These helpers do not make mmap/vector serving stable. pgcontext.artifact_segment_serving_readiness is the experimental read-only gate every mmap serving path must pass: it requires mmap artifacts, file-materialized lifecycle, confined paths, reload and checksum/catalog agreement, and mapped bytes within the caller supplied budget before reporting serving_ready = true. pgcontext.artifact_segment_mmap_payload uses that same gate for one visible artifact and copies validated payload bytes at the SQL compatibility boundary. Internally, pgcontext.search_mmap_hnsw_artifact holds a reader pin and a real read-only OS mapping for the query, validates borrowed bytes, traverses the persisted HNSW links, merges source points added after the generation high-water mark, and rechecks/scores candidates against the authoritative source table. It remains an experimental serving surface and fails closed for not-ready or corrupt artifacts.

The loader rejects malformed artifacts before exposing payload bytes. The same validation path also supports memory-mapped readers by returning a borrowed payload view after checking header fields, payload bounds, section alignment, and checksum integrity without copying payload data. Covered bad paths include truncated headers, unknown versions, wrong endian markers, oversized payload lengths, truncated payloads, misaligned mmap sections, overflowing section ranges, and checksum mismatches. Atomic write/reload, import/export, and loader fuzzing are covered by tests.

HNSW graph artifacts use a portable payload format inside the outer segment: payload magic/version, record count, dimensions, and little-endian records with contiguous node ids, point ids, dense vectors, and base-layer neighbor ids. pgcontext.validate_hnsw_graph_artifact(segment) validates that inner payload before any mmap serving path may consume it, rejecting truncated records, non-contiguous node ids, out-of-range neighbors, and invalid vectors as corrupt artifact input. Source-built artifacts contain real HNSW base links; old or manually constructed edgeless artifacts retain an exact compatibility fallback. Final SQL scores always come from current source-table vectors, not artifact scores.

Segment writers use a same-directory temporary file, sync the temporary file, rename it over the target path, sync the parent directory, and then reload the target through the validator. Readers also reject encoded files larger than the maximum header-plus-payload size before loading bytes into memory.

Import and export operations in context-storage are rebuildable-artifact copy primitives for internal and release-gate use. They are not exposed as stable SQL operator APIs in this release; the SQL validators and manifest APIs above are experimental metadata primitives. Until artifact file publication and serving APIs are explicitly stabilized, operators should use normal PostgreSQL backup/restore for authoritative data and rebuild acceleration artifacts from catalog metadata and source tables.

Snapshot, Export, And Import Procedure

Use PostgreSQL backup/restore as the authoritative recovery path:

  1. Take a normal PostgreSQL base backup, logical dump, or managed-service snapshot that includes user source tables and the pgcontext catalog schema.
  2. Treat pgContext index and segment artifacts as optional cache files. If an artifact copy is useful, copy it only after the writer has completed its atomic rename and the validator accepts the target file.
  3. Restore PostgreSQL data first. Confirm that collections, points, vectors, filters, aliases, model versions, and migration records are present in the catalog before rebuilding derived artifacts.
  4. Rebuild or revalidate pgContext acceleration artifacts from source tables. Do not trust a copied artifact unless its header version, bounds, alignment, and checksum pass the loader.
  5. If a copied artifact fails validation, delete the artifact copy and rebuild it from PostgreSQL source tables. Do not repair binary artifacts by hand.

The operator contract is therefore to snapshot PostgreSQL, optionally copy validated rebuildable artifacts for speed, and prefer rebuild over binary import whenever there is a version, checksum, or ownership mismatch.

The segment loader is covered by a cargo-fuzz target named segment_loader. New binary-loader fixes should add a focused unit test and, when practical, a small corpus seed under fuzz/corpus/segment_loader/.