oai_fdw 1.15

2026-10-01

  • Enhancements

    Regression tests without network access: make installcheck now only runs the tests that need nothing but a PostgreSQL server. Tests against live repositories and through the Squid proxies are opt-in, with INCLUDE_EXTERNAL_TESTS=1, INCLUDE_LOCAL_TESTS=1 or INCLUDE_ALL_TESTS=1.

    Reproducible builds: the build date shown by oai_fdw_settings() is taken from SOURCE_DATE_EPOCH, if set.

    HTTP 429 flow control: 429 Too Many Requests is handled like 503, honouring Retry-After. Without Retry-After, the wait before each retry doubles, starting at 5 seconds, and so does the wait after network errors, which used to be retried immediately. Other client errors (4xx), also from a proxy, are no longer retried. libcurl’s reason for a failed attempt is written to the server log.

    Validation of from and until: the table options must be a valid YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ value.

  • Bug fixes

    Fixed backend crashes: a NULL element in a setspec array filter, a prefix operator in WHERE, support functions called on a server of another FDW, and ListSets/ListMetadataFormats responses with missing elements crashed the backend.

    Fixed leaks on query cancel: cancelling a query during a request no longer leaks the curl handle, its socket and the response buffer.

    Fixed leaks on errors while parsing: an error while reading a response, e.g. a value that cannot be converted to the database encoding, no longer leaks the response document in OAI_Identify(), OAI_ListSets() and OAI_ListMetadataFormats(), nor libxml2’s copies of the values in queries.

    Support functions require USAGE: OAI_Identify(), OAI_ListSets() and OAI_ListMetadataFormats() now check the USAGE privilege on the foreign server.

    ListSets follows resumptionToken: OAI_ListSets() and IMPORT FOREIGN SCHEMA oai_sets no longer stop after the first page of sets.

    from/until granularity: both are sent in a granularity the repository supports, as announced by Identify, so repositories with day granularity no longer answer badArgument. GetRecord, which takes neither, needs no Identify request.

    Character encoding: values are converted between UTF-8 and the database encoding, in both directions.

    Responses are checked: a response that is not an OAI-PMH document (e.g. an HTML maintenance page) raises an error instead of returning no rows, and OAI errors are raised for all requests. libxml2 parser errors are no longer written to the server’s stderr.

    Pushdown: date constants, infinite and BC timestamps, and constants of other types (e.g. name) are no longer pushed down with a wrong value.

    Pushdown of values known at execution time: conditions comparing with a parameter (e.g. $1 in a generic plan of a prepared statement or PL/pgSQL) or an expression such as now() - interval '1 day' were not pushed down, so they harvested the whole repository. They are now evaluated when the scan starts, and EXPLAIN lists them as Runtime arguments.

    varchar columns: varchar(n) limits are applied, and array operators work on varchar[] setspec columns.

    <metadata> content: the root element is returned, even if a comment precedes it.

    IMPORT FOREIGN SCHEMA: the user mapping is used, and LIMIT TO only imports sets that exist. Sets whose setSpec is longer than 63 characters get unique table names instead of making the import fail.

    User mapping of views: a foreign table queried through a view now uses the user mapping of the view owner, as permissions are checked as that role.

    Redirects: a redirect that is not followed now raises an error naming its target and suggesting request_redirect, instead of reporting an invalid response. A followed redirect no longer warns about the content-type of the redirect response. Neither does a Content-Type header without a space after the colon.

    OAI_HarvestTable: the last time window is no longer skipped, quoted table names work, a current schema whose name needs quoting works, the default end_date is the current time in UTC instead of the session’s local time (which left out the latest records in sessions west of UTC), and same-named tables in other schemas are no longer mixed up. A page_size that is not positive, or a NULL argument, is rejected instead of reporting a completed harvest of nothing. Its “target table created” and “harvester complete” messages are now NOTICEs instead of INFOs, so that they can be silenced with client_min_messages; the per-page messages of exec_verbose remain INFOs.

    Server options: request_redirect, request_max_redirect and the URL scheme are validated, and connect_retry '0' disables retries. Server options are read by one function for queries, the support functions and IMPORT FOREIGN SCHEMA, so they are interpreted the same way everywhere.

    Linking: the library now links against libxml2.

    Build with older libcurl: the build no longer fails with libcurl older than 7.66, e.g. 7.61.1 on RHEL / Rocky Linux 8. oai_fdw_settings() then does not report nghttp2.

    Stalled transfers: a transfer that receives less than one byte per second for 300 seconds is now aborted and retried, instead of hanging forever with the default request_timeout of 0. Timeouts are reported with libcurl’s reason.

    No signals from libcurl: libcurl builds without an asynchronous resolver used SIGALRM for DNS timeouts, which PostgreSQL uses itself (e.g. for statement_timeout). CURLOPT_NOSIGNAL is now set.

oai_fdw 1.14

2026-09-09

  • Enhancements

    Improved EXPLAIN diagnostics: EXPLAIN output now include oai_fdw-specific details for each Foreign Scan node, showing which parameters are pushed down to the remote OAI Server.

    Enhanced version information: The oai_fdw_version() function now returns a comprehensive version string that includes PostgreSQL version, compiler information, and all dependency versions (libxml, librdf, libcurl) in a single formatted output. A new oai_fdw_settings() function provides extended dependency information including optional components like SSL, zlib, libSSH, and nghttp2. The oai_fdw_settings view parses this extended information into a table format for convenient programmatic access to individual component versions.

    Add ‘request_timeout’ to FOREIGN SERVERS: This option sets the maximum time in seconds allowed for a complete HTTP request (connect + transfer). 0 disables the limit (default). Unlike connect_timeout, this applies to the entire duration of the request, including data transfer.

  • Bug fixes

    Fixed invalid libcurl lifecycle: curl_global_init()/curl_global_cleanup() were being called on every OAI request instead of once per backend process. This could interfere with other libcurl users loaded in the same backend (e.g. other FDWs). Global initialization now happens once in _PG_init(); cleanup is left to the OS at process exit.

    Fixed catalog lookup overhead: Foreign table column metadata (name, OAI node mapping, PostgreSQL type, type modifier, and attribute number) is now loaded once during session initialization and cached in the scan state, then passed to the executor via the serialized plan. Previously this information was looked up from the system catalogs (GetForeignColumnOptions) for every column of every row during CreateOAITuple(), causing significant syscache overhead on large result sets.

    Add missing user mapping in helper functions: GetIdentity, ListSets, and ListMetadataFormats were being executed without loading the USER MAPPINGS, which could fail requests if the server needed any sort of authentication. This has now been solved.

    Encoded credentials are no longer written to server logs via libcurl verbose output: When client_min_messages is set to DEBUG3, libcurl’s verbose output is now routed through PostgreSQL’s elog(DEBUG3) via a custom CURLOPT_DEBUGFUNCTION callback (CurlDebugCallback) instead of being written directly to stderr. Crucially, any outgoing or incoming HTTP header whose name matches Authorization: or Proxy-Authorization: has its value replaced with [REDACTED] before being logged, so Bearer tokens, Basic auth credentials (base64-encoded), and proxy passwords never appear in plaintext in the PostgreSQL server log.

    HTTP error response bodies are now truncated in log and error messages: Error bodies included in errdetail() and server-log elog() calls were previously unbounded. A misconfigured proxy returning a large HTML error page would be written verbatim into the PostgreSQL server log. Error bodies are now truncated to 512 bytes (OAI_FDW_MAX_ERROR_BODY) before being included in any message.

    SQL identifiers and literals are now properly quoted in IMPORT FOREIGN SCHEMA: The CREATE FOREIGN TABLE statements generated by IMPORT FOREIGN SCHEMA interpolated the server name, the metadataPrefix and the setSpec directly into the command text without quoting. Since the metadataPrefix and setSpec values are taken from the remote OAI repository’s ListMetadataFormats and ListSets responses, a malicious or compromised repository could inject arbitrary SQL that would then be executed with the privileges of the user running the import. Server names and generated table names are now passed through quote_identifier(), and option values through quote_literal_cstr().

    metadataPrefix is now URL-encoded in GetRecord requests: Unlike the ListRecords and ListIdentifiers code paths, the GetRecord request builder appended the metadataPrefix option to the request URL without passing it through curl_easy_escape(). A value containing & or # could therefore inject additional parameters into the OAI request.

    OAI header values are no longer returned XML-escaped: The identifier, setSpec and datestamp values of a record header were extracted with xmlNodeDump(), which serialises a node back to XML instead of decoding it. Any header value containing a character that has to be escaped in XML was therefore returned to SQL in its escaped form - an identifier such as oai:example.org/a&b came back as oai:example.org/a&amp;b. Besides being wrong on its own, this silently broke pushdown: a query filtering on the true identifier pushed a correct GetRecord request, but the escaped value returned by the scan then failed the local re-check, so a record that exists yielded no rows at all. These values are now read with xmlNodeGetContent(), which resolves XML escapes. The content node is unaffected, as it legitimately holds serialised XML.

    OAI-PMH flow control (HTTP 503 / Retry-After) is now honoured: Section 3.4 of the OAI-PMH specification lets a repository answer 503 Service Unavailable with a Retry-After header to ask a harvester to slow down. Such a response was treated as an ordinary failure and retried immediately, up to connect_retry times with no delay whatsoever, which hammers a repository that has just reported itself overloaded and then aborts the harvest. A 503 is now recognised, and the delay requested through Retry-After is waited out before the next attempt. Both forms allowed by RFC 9110 are accepted (delta-seconds and HTTP-date), a repository that sends no usable Retry-After falls back to 5 seconds, and delays are capped at 300 seconds so that a repository cannot pin a backend down indefinitely. The wait is implemented with WaitLatch() rather than pg_usleep(), so queries remain cancellable while it is in progress.

    Harvests no longer accumulate every page in memory: The scan context holding the records of a page was only reset when the scan was restarted or finished, so LoadOAIRecords() dropped its reference to the previous page without releasing it. Memory use therefore grew with the size of the whole harvest rather than with the size of a single page. The context is now reset before each new page is requested, with the resumption token carried across the reset in a context of its own.

    A page that is empty but carries a resumption token no longer truncates the harvest: OAIFdwIterateForeignScan() loaded at most one page per call, so an empty page arriving with a resumption token ended the scan silently and every record behind that token was lost. Pages are now requested until a record is produced or the repository stops handing out tokens.

    XML response documents are no longer leaked when a request fails: LoadOAIRecords() freed the parsed response only after walking it, so any error raised in between - including every OAI error condition reported by RaiseOAIException(), such as badArgument or cannotDisseminateFormat - skipped the call. As libxml2 allocates these documents outside PostgreSQL’s memory contexts, they were not reclaimed at the end of the query and stayed lost for the life of the backend. The document is now released through PG_TRY()/PG_CATCH() and the pointer cleared, which also removes a stale pointer that could be freed a second time.

    libcurl callbacks no longer raise PostgreSQL errors: WriteMemoryCallback() and HeaderCallbackFunction() could ereport(ERROR) - and used palloc(), which throws on out-of-memory - while running inside curl_easy_perform(). Either would longjmp straight out of libcurl, leaving its handle and connection state inconsistent and skipping the cleanup that follows the transfer. The callbacks now allocate with malloc(), record the condition and return a short count, so the transfer fails cleanly and the caller reports it.

    OAI requests are sent as GET instead of POST: The request arguments were passed through CURLOPT_POSTFIELDS, which made every request a POST even though the log line claimed otherwise. On a 301 or 302 libcurl turns a POST into a GET and drops the body while doing so, so a redirected request reached the repository with no verb, no metadataPrefix and no other argument - relevant for any SERVER created with request_redirect. The arguments are now part of the request URL and survive redirects.

oai_fdw 1.13

2026-02-20

  • Bug Fixes

    Moved proxy authentication options (proxy_user and proxy_password) from SERVER to USER MAPPING. These credentials are now correctly specified in CREATE USER MAPPING instead of CREATE SERVER.

oai_fdw 1.12

2026-01-09

  • Bug Fixes
    • Added URL encoding for resumption tokens in ListRecords and ListIdentifiers requests using curl_easy_escape to properly handle special characters like ‘&’ and ‘=’.
    • Ensured xmldoc is only freed if successfully initialized to prevent crashes from freeing uninitialized memory.
    • Set User-Agent and Accept headers in requests for better server compatibility and to mimic standard HTTP client behavior.

oai_fdw 1.11

2025-09-26

  • New Features

    PostgreSQL 18 support.

  • Bug fixes

    A safeguard has been introduced to handle cases where xmlDocGetRootElement fails to parse the root node of an XML document. Instead of proceeding with an empty set, an error message is now displayed to inform the user of the issue.

oai_fdw 1.10

2024-10-14

  • New Features

    PostgreSQL 17 support.

oai_fdw 1.9

2024-03-21

  • New Features *

    • This feature defines a mapping of a PostgreSQL user to an user in the target triplestore - user and password, so that the user can be authenticated.

  • Bug Fixes *

    • This fix a fixed string length limit in the support functions (512 bytes). I mistakenly assumed that it was defined like that in the OAI-PMH Standard.

oai_fdw 1.8

2023-11-15

  • Performance improvements
    • This intoduces a new pagination logic to go through the result sets from OAI requests. It no longer iterates over the xml document to retrieve the OAI records for building the tuples. Instead, it now parses all records from a result set into a list right after loading the xml response, so that they can accessed later on for building the tuples.

oai_fdw 1.7

2023-09-22

  • New Features *
    • Adds PostgreSQL 16 support

oai_fdw 1.6

2023-04-11

  • New Features *

    • Adds request_redirect and request_max_redirect options (CREATE SERVER). This allows users to enable or disable URL redirects issued by the server and also how many times a redirection may occur in a single http request.

  • Bug Fixes *

    • This bug fix implements a http response validation to avoid a server crash when a support function receives an invalid OAI document from the repository.

oai_fdw 1.5.1

2023-04-11

  • Bug Fixes *
    • This bug fix implements a http response validation to avoid a server crash when a support function receives an invalid OAI document from the repository.

oai_fdw 1.5.0

2022-11-15

  • Enhancements *

    • add number of inserted and updated records in the harvester log messages

  • Bug Fixes *

    • clean up parser in support function calls (libxml2)
    • fix wrong initial page size in the logs for debug1 sessions
    • add missing memory contexts for oai loader and parser.

oai_fdw 1.4.0

2022-10-27

  • New Features *

    • add update path script for oai_fdw upgrades: 1.1 -> 1.4, 1.2 -> 1.4 and 1.3 -> 1.4

  • Bug Fixes *

    • fix memory leak in xmldoc for requests with resumption tokens

oai_fdw 1.3.0

2022-10-21

  • New Features *
    • Connection retry option for HTTP requests (CREATE SERVER).

oai_fdw 1.2.0

2022-10-19

  • New Features *

    • OAI_HarvestTable: new procedure to harvest data from OAI foreign tables and store them into a local table.
    • Connection retry option for HTTP requests (CREATE SERVER).

  • Bug Fixes *

    • Fix memory leaks in the libxml2 parser for ListRecords and ListIdentifiers requests
    • Fix missing timeout parameter for non-proxy OAI requests

  • Enhancements *

    • Add regression tests for PostgreSQL 15.

oai_fdw 1.1.0

2022-10-01

  • New Features *

    • Proxy support for HTTP requests (CREATE SERVER).
    • Connection timeout option for HTTP requests (CREATE SERVER).

  • Bug Fixes *

    • cURL exception handling: Adds missing cURL error message and response code in case of failed http requests. In previous versions these values were never fetched.

  • Enhancements *

    • URL validator no longer checks if the URL is reachable. It now only checks the URL’s syntax.
    • URls with protocols other than HTTP and HTTPS are now rejected before execution, as other protocols are not supported in the OAI-PMH Standard.

oai_fdw 1.0.0

2022-09-05

  • Features *

    This release fully enables usage of the following OAI-PMH 2.0 Requests via SQL queries (see README):

    • ListRecords
    • ListIdentifiers
    • GetRecord
    • Identify
    • ListMetadataFormats
    • ListSets