Contents

• JSONCDC
  • Copyright and License
  • Status
  • Usage

JSONCDC

JSONCDC provides change data capture for Postgres, translating the Postgres write ahead log to JSON.

It is written in Rust and, being short, is a good skeleton project for other would be plugin authors who’d like to use Rust to write Postgres extensions.

Our library Requires rust stable 1.1 or greater.

Copyright and License

Copyright © 2016 Alex Newman, Jason Dusek Copyright © 2018 Instructure, Inc.

JSONCDC is available under multiple licenses:

• the same license as Postgres itself (licenses/postgres),

• the Apache 2.0 license (licenses/apache).

Status

JSONCDC is presently installable with pgxn, from the testing channel: pgxn install jsoncdc --testing.

Usage

A basic demo:

SELECT * FROM pg_create_logical_replication_slot('jsoncdc', 'jsoncdc');
--- Wait for some transactions, and then:
SELECT * FROM pg_logical_slot_get_changes('jsoncdc', NULL, NULL);

The output format of jsoncdc is very regular, consisting of begin, table, insert, update, delete and message clauses as JSON objects, one per line:

{ "begin": <xid> }
{ "schema": <column names and type>, "table": <name of table> }
...inserts, updates and deletes for this table...
{ "schema": <column names and type>, "table": <name of next table> }
...inserts, updates and deletes for next table...
{ "prefix": <prefix>, "message": <message>, "transactional": <true|false> }
...messages may be mixed in at any point; they don't belong to a table...
{ "commit": <xid>, "t": <timestamp with timezone> }

With pg_recvlogical and a little shell, you can leverage this very regular formatting to get each transaction batched into a separate file:

pg_recvlogical -S jsoncdc -d postgres:/// --start -f - |
while read -r line
do
  case "$line" in
    '{ "begin": '*)                # Close and reopen FD 9 for each new XID
      fields=( $line )
      xid="${fields[2]}"
      exec 9>&-
      exec 9> "txn-${xid}.json" ;;
  esac
  printf '%s\n' "$line" >&9       # Use printf because echo is non-portable
done