Repository Guidelines

What This Is

Kazakh stemmer for PostgreSQL full-text search. BFS suffix-stripping over ordered morphological layers (noun: DERIV→PLUR→POSS→CASE→PRED, verb: VVOICE→VNEG→VTENSE→VPERSON) with vowel harmony enforcement, penalty-based candidate scoring, optional lexicon verification, morphophonological stem repair, and an idempotent fixed-point pass (with lexicon) that conflates verbal nouns, denominal verbs, lexicalized participles (-ған/-ген/-қан/-кен), and converbs (-п, gated on the verb-tagged lexicon sibling kaz_stems.dict.verbs) onto their lexicon root. Glide-only words (туризм) are harmony-neutral. Token coverage (measured by eval/measure_stem_coverage.py over 45.7k corpus tokens): 76.5% analyzed, 87.5% recognized (stemmed or dictionary lemma).

No prior Kazakh stemmer exists for PostgreSQL or Elasticsearch. This is the first.

Architecture

Cargo workspace with a shared core library and multiple consumers:

  • core/ — kazsearch-core: pure Rust stemmer (BFS engine, suffix rules, vowel harmony, penalty scoring, lexicon, stem repair). No Postgres dependencies.
  • pg_ext/ — pg_kazsearch: pgrx-based PostgreSQL extension. Thin wrapper that calls kazsearch_core::stem().
  • cli/ — kazsearch-cli: CLI tool (kazsearch stem, analyze, bench, lexicon validate).
  • elastic/ — kazsearch-elastic: Elasticsearch plugin (placeholder).
  • legacy/pg_kazsearch_c/ — archived original C implementation (reference only, not built).

Key core modules:

  • core/src/explore.rs — BFS engine, visit set, penalty scorer, stem repair. The heart.
  • core/src/text.rs — UTF-8 iteration, vowel classification, harmony checks.
  • core/src/rules.rs — Suffix tables for noun and verb layers.
  • core/src/lexicon.rs — Lexicon loader.
  • core/src/lib.rs — stem() entry point, winner selection, sound change undo.

Supporting: scripts/ (lexicon builder), eval/ (scraper, corpus loader, evaluator, CMA-ES optimizer), docker/ (dev container).

Commands

All via just.

Command What it does
just up / just down Start/stop Postgres container
just build Build lexicon + compile Rust extension + install
just reload Build + DROP/CREATE EXTENSION
just cli Build CLI tool
just test-core Run core library unit tests
just test-ext Smoke-test stemmer output via SQL
just psql Interactive psql
just pipeline Full eval: scrape → load → gen queries → evaluate
just optimize CMA-ES penalty weight optimization
just apply-weights Push optimized weights to running DB

Style

Rust: Standard rustfmt. Public API in core/src/lib.rs. Modules mirror the C design: explore, text, rules, lexicon.

Python: snake_case, standalone argparse CLIs.

Commits: Conventional Commits (feat:, fix:, refactor:). One logical change per commit. just build && just test-ext must pass.

Critical Context

  • Kazakh is agglutinative — words stack 5-6 suffixes. Greedy stripping fails; BFS is necessary.
  • Vowel harmony (back/front) is mandatory for suffix validation. Glides (у, и, ю) are transparent.
  • Penalty constants in candidate_penalty (core/src/explore.rs) are empirically tuned via CMA-ES against a real corpus. Changing one can break others.
  • Stem repair reverses morphophonological changes: consonant mutation (б→п, ғ→қ, г→к), vowel elision, and lexicon-based vowel restore.
  • The lexicon safety valve prevents overstemming: if the input word is already in the dictionary and the candidate looks suspicious, return input unchanged.
  • Layer guards in core/src/explore.rs encode real morphotactic constraints — they are not optional and each one prevents a class of mis-stems.

Landing the Plane (Session Completion)

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until git push succeeds.

MANDATORY WORKFLOW:

  1. Note remaining work - Summarize anything that needs follow-up in the handoff
  2. Run quality gates (if code changed) - Tests, linters, builds
  3. PUSH TO REMOTE - This is MANDATORY: bash git pull --rebase git push git status # MUST show "up to date with origin"
  4. Clean up - Clear stashes, prune remote branches
  5. Verify - All changes committed AND pushed
  6. Hand off - Provide context for next session

CRITICAL RULES: - Work is NOT complete until git push succeeds - NEVER stop before pushing - that leaves work stranded locally - NEVER say “ready to push when you are” - YOU must push - If push fails, resolve and retry until it succeeds