Skip to content

Testing

Tests are a significant part of the VersionPress project, we care about writing and maintaining them. They live in plugins/versionpress/tests and there are several types of them, from unit to full end2end tests. They all run in a dockerized test environment.

Note: the ./frontend app has its own tests, this section is about core VersionPress tests (PHP code) only.

Dockerized testing environment

Similarly to the development environment, tests utilize Docker & Docker Compose as well. The main benefit is that you don't need to set up things like Selenium or Java locally.

Most tasks are scripted, for example, you just run npm run tests:unit but you can also drop to the raw Docker Compose mode and do things like docker-compose -f docker-compose-tests.yml run --rm tests ....

Note the different docker-compose file

Tests use a separate docker-compose file, add -f docker-compose-tests.yml when running docker-compose.

In that case, one thing to understand is that there are two services in docker-compose-tests.yml to choose from:

  • tests – just a test runner.
  • tests-with-wordpress – starts a WordPress stack.

All scripts also come with a ...:debug version, for example, tests:unit:debug. See Starting a debugging session from command line.

Running tests from command line

  1. Make sure you have run npm install as described above and have Docker running.
  2. Run npm run tests:unit or npm run tests:full.

Unit tests use a simpler tests service and are fast to execute.

The full tests include end2end tests and are relatively slow to run, however, if they pass, there's a good chance that VersionPress works correctly.

Customizing what tests run

tests:custom and tests:custom-with-wordpress scripts allow you to run custom tests easily. Here are some examples:

❕ Notice how PHPUnit arguments come after --.

# Pick a test suite from the default phpunit.xml
npm run tests:custom -- -c phpunit.xml --testsuite Unit

# Create your own phpunit.*.xml (gitignored)
npm run tests:custom -- -c phpunit.custom.xml

# Run specific test class
npm run tests:custom-with-wordpress -- -c phpunit.xml --filter RevertTest

If you want to go entirely custom, use raw docker-compose:

# PhpStorm-like invocation (copy/pasted from its console):
docker-compose -f docker-compose-tests.yml run --rm tests \
  ../vendor/bin/phpunit \
  --bootstrap /opt/versionpress/tests/phpunit-bootstrap.php \
  --no-configuration /opt/versionpress/tests/Unit

Test output

Npm scripts are configured to log in a TestDox format to container's /var/opt/versionpress/logs which is mapped to your local folder ./dev-env/test-logs.

To log in another supported format, run tests manually like this:

docker-compose -f docker-compose-tests.yml run --rm tests \
  ../vendor/bin/phpunit \
  -c phpunit.xml --log-junit /var/opt/versionpress/logs/vp-tests.log

Clean up tests

Tests persist data (MySQL database files, WordPress files etc.) to Docker volumes which are re-attached between test runs. This speeds things up but for tests that are not perfectly isolated, e.g., end2end tests, it can also cause weird test failures. For cases like that, run

npm run tests:cleanup

to ensure that all containers are stopped and volumes deleted. The next test run will be entirely fresh.

Tips for tests

  • If you're trying to narrow down a bug, it's useful to run a smaller test suite via one of the options above and add stopOnFailure="true" to the XML file or --stop-on-failure on the command line.
  • Unit tests can also easily be run using a local php interpreter, basically just run them in PhpStorm.

Running and debugging tests from PhpStorm

PhpStorm makes it easy to select specific tests and to debug them. Also, if you stop debugging, you will see messages gathered so far. There is a one-time setup to go through.

💡 If this doesn't work for you, e.g., due to some Docker Compose bug in PhpStorm, you can start debugging from command line.

First, if you're using Docker for Mac or Docker for Windows, expose a daemon in Docker settings:

image

In PhpStorm, create a new Docker environment in Settings > Build, Execution, Deployment > Docker:

image

In the Docker panel, you should now be able to connect:

image

Next, define a remote interpreter. Make sure you have the PHP Docker plugin enabled and go to Settings > Languages & Frameworks > PHP. Add a new interpreter there:

image

Note that the docker-compose-tests.yml is at the repo root, not inside ./plugins/versionpress:

⚠️ Note: the screenshot uses the old path docker-compose.yml; it should be docker-compose-tests.yml.

image

If this doesn't go smoothly, try unchecking the Include parent environment variables checkbox in the Environment variables field:

image

Select this CLI interpreter as the main one for the project and make sure the path mappings are correct:

image

The final step is to set up a test framework in PHP > Test Frameworks. Add a new PHPUnit by Remote Interpreter:

image

Don't forget to set the Default bootstrap file to /opt/versionpress/tests/phpunit-bootstrap.php.

Now you're ready to run the tests. For example, to run all unit tests, right-click the Unit folder and select Run:

image

Debugging also works, just select Debug instead of Run:

image

This works equally well other types of tests as well, for example, Selenium tests:

image

Starting debugging session from command line

This method is more universal and works for PhpStorm, VSCode and other IDEs. You generally do this:

  1. Set a breakpoint.
  2. Start listening in your IDE.
  3. Launch a debug-enabled script like npm run tests:unit:debug (see package.json).

PhpStorm example

First, make sure you have the 'VersionPress-tests' server defined in Settings > Languages & Frameworks > PHP > Servers. If not, run npm run init-phpstorm.

Then, set a breakpoint in some test and start listening for debug connections in the toolbar.

Run npm run tests:unit:debug in the console, skip the first break at the wp binary and see your breakpoint hit:

image

See this JetBrains help page for more.

VSCode example

In VSCode, install PHP Debug extension and create a launch.json config containing this:

{
  "name": "PHP: Listen for Xdebug",
  "type": "php",
  "request": "launch",
  "port": 9000,
  "pathMappings": {
    "/opt/versionpress": "${workspaceRoot}/plugins/versionpress",
  }
}

Then, start a debugging session in VSCode and set a breakpoint. Run the tests:unit:debug script and see the breakpoint hit:

image

Unit tests

Unit tests are best suited for small pieces of algorithmic functionality. For example, IniSerializer is covered by unit tests extensively.

You can either run unit tests in a dockerized environment as described above or set up a local CLI interpret which makes the execution faster and more convenient.

End2end tests

End2end tests exercise a WordPress site and check that VersionPress creates the right Git commits, that the database is in correct state, etc. These tests are quite heavy and slow to run but if they pass, there's a good chance that VersionPress works correctly. (Before the project had these, long and painful manual testing period was necessary before each release.)

End2end tests use the concept of workers: each test itself is implemented once but how e.g. a post is created or a user deleted is up to a specific worker. There are currently two types of workers:

  1. WP-CLI workers – run WP-CLI commands against the test site.
  2. Selenium workers – simulate real user by clicking in a browser.

In the future, we might add REST API workers; you get the idea.

Currently, the default worker is WP-CLI and the only way to switch workers is to update tests/test-config.yml, the end2end-test-type key. We'll make it more flexible in the future.

Troubleshooting failed tests

The Docker containers are stopped when tests finish running but the data is kept in Docker volumes. You can start the WordPress site again via:

docker-compose -f docker-compose-tests.yml up -d wordpress-for-tests

You can now inspect it:

  • The site is running at http://wordpress-for-tests/wptest – check test-config.yml for the login info. (You'll also need to update your hosts file so that wordpress-for-tests resolves to 127.0.0.1.)
  • Connect to the database via mysql -u root -p or Adminer which you can access by running docker-compose run -d --service-ports adminer and visiting http://localhost:8099. The database name is mysql-for-wordpress.
  • To inspect the site files or the logs, you have two options:
    1. Run docker-compose -f docker-compose-tests.yml run --rm tests sh and use commands like ls -ls /var/www/html/wptest or cd /var/www/html/wptest && git log to explore the files. Type exit when finished.
    2. Run npm run tests:copy-files-to-host to copy files to your local filesystem. This will create two folders, dev-env/wp-for-tests and dev-env/test-logs, where you can conveniently use your local tools (editors, Git GUI clients, etc.). Note that this can be quite resource-intensive, for example, on Docker for Mac, this will overwhelm the system for several minutes.

When you're done, clean up everything by running:

npm run tests:cleanup

This will stop & remove containers, delete volumes and remove temporaray files under dev-env.

Other tests

The project has these other types of tests (folders in the ./plugins/versionpress/tests folder and also test suite names in phpunit.xml so that you can run them using --testsuite <SuiteName>):

  • GitRepositoryTests – test Git repository manipulation in GitRepository.
  • SynchronizerTests – these are quite slow and test that given some INI files on disk, the database is in a correct state after synchronization runs.
  • StorageTests – test that entities are stored correctly as INI files.
  • LoadTests – they are run together with other tests but with very few iterations; manually update their source files and execute them separately to properly exercise them.
  • Selenium – a bit like end2end tests but for rarer cases, like VersionPress not being activated yet.
  • Workflow – exercise cloning and merging between environments.