To run tests in parallel use:
will run 4 configs in parallel
make check-arbitrary-configs parallel=4 ```
To run tests sequentially use:
make check-arbitrary-configs parallel=1
To run only some configs:
Config names should be comma separated
make check-arbitrary-configs CONFIGS=CitusSingleNodeClusterConfig,CitusSmallSharedPoolSizeConfig ```
To run only some test files with some config:
make check-arbitrary-base CONFIGS=CitusSingleNodeClusterConfig EXTRA_TESTS=dropped_columns_1
To get a deterministic run, you can give the random's seed:
make check-arbitrary-configs parallel=4 seed=12312
seed will be in the output of the run.
In our regular regression tests, we can see all the details about either planning or execution but this means we need to run the same query under different configs/cluster setups again and again, which is not really maintanable.
When we don't care about the internals of how planning/execution is done but the correctness, especially with different configs this infrastructure can be used.
check-arbitrary-configs target, the following happens:
- a bunch of configs are loaded, which are defined in
config.py. These configs have different settings such as different shard count, different citus settings, postgres settings, worker amount, or different metadata.
- For each config, a separate data directory is created for tests in
tmp_citus_testwith the config's name.
- For each config,
create_scheduleis run on the coordinator to setup the necessary tables.
- For each config,
sql_scheduleis run on the coordinator if it is a non-mx cluster. And if it is mx, it is either run on the coordinator or a random worker.
- Tests results are checked if they match with the expected.
When tests results don't match, you can see the regression diffs in a config's datadir, such as
We also have a PostgresConfig which runs all the test suite with Postgres. By default configs use regular user, but we have a config to run as a superuser as well.
So the infrastructure tests:
- Postgres vs Citus
- Mx vs Non-Mx
- Superuser vs regular user
- Arbitrary Citus configs
Adding a new test
When you want to add a new test, you can add the create statements to
create_schedule and add the sql queries to
If you are adding Citus UDFs that should be a NO-OP for Postgres, make sure to override the UDFs in
If the test needs to be skipped in some configs, you can do that by adding the test names in the
skip_tests array for
each config. The test files associated with the skipped test will be set to empty so the test will pass without the actual test
Adding a new config
You can add your new config to
config.py. Make sure to extend either
On the CI, upon a failure, all logfiles will be uploaded as artifacts, so you can check the artifacts tab. All the regressions will be shown as part of the job on CI.
In your local, you can check the regression diffs in config's datadirs as in