Search

Parallel Execution

When execution time of your tests is longer than a coffee break, it is a good reason to think about making your tests faster. If you have already tried to run them on SSD drive, and the execution time still upsets you, it might be a good idea to run your tests in parallel. However, PHP runs in a single-process and you can’t parallelize tests natively similarly to how this works in Java or in NodeJS.

Depending on the project size and requirements you can choose how the best to implement parallel testing for your case. In this guide we will overview possible options.

Sharding

Minimal setup can be implemented by executing several independent CI jobs and running. Sharding in Codeception allows to combine stages 1 and 2 so tests could be split by groups on the fly. In this case a pipeline could be simplified to one stage with several jobs.

Each job should have Codeception running a subset of tests set by --shard option:

First job

php vendor/bin/codecept run --shard 1/3

Second job

php vendor/bin/codecept run --shard 2/3

Third job

php vendor/bin/codecept run --shard 3/3

For each job you specify on how many groups tests should be split and the group that should be executed on this agent. I.e. --shard option takes 2 params: --shard {currentGroup}/{numberOfGroups}. So to split tests on 5 machines you need to create 5 jobs with Codeception running these shards: 1/5, 2/5, 3/5, 4/5, 5/5.

Splitting test by shards is done automatically with zero-config. However, in this case you receive as many reports as jobs you have. To aggregate jobs store HTML, XML, and CodeCoverage results as artifacts and add an extra job in the end to merge them. Merging can be done with Robo-paracept toolset described below.

To get an aggregated report without an extra stage and without managing artifacts use Testomat.io. This is a SaaS platform that can receive test results from different parallel run and show them in the one interface.

By running tests with Testomat.io reporter attached results will be sent to a centralized server. By default each execution will create its own report. To store results from different shards in one report set the Run title for them. You can use a common environment variable, like number of a build, to create the unique title which will be the same for all jobs. If build id is stored as $BUILDID variable, execution script for shard #3 can be following:

TESTOMATIO={apiKey} TESTOMATIO_TITLE="Build $BUILDID" ./vendor/bin/codecept run --shard 3/4

Building Pipeline

While sharding provides a simplified setup for testing the complete pipeline schema may look like this.

• On the first stage, tests should be split into groups. The group file should be committed into the repository or passed to next stage as an artifact.
• On the second stage tests are executed. XML, HTML, and CodeCoverage reports must be stored as artifacts.
• On the third stage the results from previous jobs must be collected or aggregated.

To get more control on how the jobs are split excuted and results aggregated you can use a task runner.

Codeception provides a toolset for Robo task runner called robo-paracept for splitting tests into groups and merging resulting JUnit XML reports.

To sum up, we need to install:

• Robo, a task runner.
• robo-paracept - Codeception tasks for parallel execution.

Using Robo and Robo-paracept

Execute this command in an empty folder to install Robo and Robo-paracept :

composer require codeception/robo-paracept --dev

Setting up Robo

Initializes basic RoboFile in the root of your project

php vendor/bin/robo init

Open RoboFile.php to edit it

<?php

class RoboFile extends \Robo\Tasks
{
    // define public methods as commands
}

Each public method in robofile can be executed as a command from console. Let’s define commands for 3 stages and include autoload.

<?php
require_once 'vendor/autoload.php';

class Robofile extends \Robo\Tasks
{
    use Codeception\Task\Merger\ReportMerger;
    use Codeception\Task\Splitter\TestsSplitterTrait;

    public function parallelSplitTests()
    {

    }

    public function parallelRun()
    {

    }

    public function parallelMergeResults()
    {

    }
}

When running robo you should see all 3 that these methods availble as CLI commands:

parallel:split-tests
parallel:run
parallel:merge-results

Step 1: Split Tests

Codeception can organize tests into groups. Starting from 2.0 it can load information about a group from a files. Sample text file with a list of file names can be treated as a dynamically configured group. Take a look into sample group file:

tests/Functional/LoginCept.php
tests/Functional/AdminCest.php:createUser
tests/Functional/AdminCest.php:deleteUser

Tasks from \Codeception\Task\SplitTestsByGroups will generate non-intersecting group files. You can either split your tests by files or by single tests:

public function parallelSplitTests()
{
    // Split your tests by files
    $this->taskSplitTestFilesByGroups(5)
        ->projectRoot('.')
        ->testsFrom('tests/Acceptance')
        ->groupsTo('tests/Support/Data/paracept_')
        ->run();

    /*
    // Split your tests by single tests (alternatively)
    $this->taskSplitTestsByGroups(5)
        ->projectRoot('.')
        ->testsFrom('tests/Acceptance')
        ->groupsTo('tests/Support/Data/paracept_')
        ->run();
    */
}

But what if one group of your tests runs for 5 mins and other for 20mins. In this case, you can balance execution time by using SplitTestsByTime task. It will generate balanced groups taking the execution speed into account.

More splitting strategies are implemented within Robo-paracept package.

Let’s prepare group files:

php vendor/bin/robo parallel:split-tests

The output should be similar to this:

 [Codeception\Task\SplitTestFilesByGroupsTask] Processing 33 files
 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/Support/Data/paracept_1
 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/Support/Data/paracept_2
 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/Support/Data/paracept_3
 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/Support/Data/paracept_4
 [Codeception\Task\SplitTestFilesByGroupsTask] Writing tests/Support/Data/paracept_5

Now we have group files. We should update codeception.yml to load generated group files. In our case we have groups: paracept_1, paracept_2, paracept_3, paracept_4, paracept_5.

groups:
    paracept_*: tests/Support/Data/paracept_*

Let’s try to execute tests from the second group:

php vendor/bin/codecept run Acceptance -g paracept_2

Step 2: Running Tests

At this point you should decide if tests are executed on the same job or use multiple jobs for them. We recommend using multiple jobs, as in this case the burden of parallelization goes to CI server. This makes a lot of sense as a single machine has limited resources. If you split tests into CI jobs, you are limited only to the number of agents (build servers) that the CI can provide. For cloud-based services like GitHub Actions, GitLab CI, CircleCI, etc, this number is unlimited.

Please refer to documentation of your CI platform to learn how to set up multiple jobs runnninng in parallel. Then proceed to merging of results.

In some situations you may want to keep tests running on the same machine and scale it up with more resourses. This makes sense if you have heavy application setup for each test run and setting it up for each machine can waste a lot of resources.

To execute tests in multiple processes Robo has ParallelExec task to spawn background processes.

public function parallelRun()
{
    $parallel = $this->taskParallelExec();
    for ($i = 1; $i <= 5; $i++) {
        $parallel->process(
            $this->taskCodecept() // use built-in Codecept task
            ->suite('Acceptance') // run acceptance tests
            ->group("paracept_$i") // for all paracept_* groups
            ->xml("tests/_log/result_$i.xml") // save XML results
        );
    }
    return $parallel->run();
}

After the parallelRun method is defined you can execute tests with

php vendor/bin/robo parallel:run

Step 3: Merge Results

In case of parallelExec task we recommend to save results as JUnit XML, which can be merged and plugged into Continuous Integration server.

public function parallelMergeResults()
{
    $merge = $this->taskMergeXmlReports();
    for ($i=1; $i<=5; $i++) {
        $merge->from("tests/_output/result_paracept_$i.xml");
    }
    $merge->into("tests/_output/result_paracept.xml")->run();
}

Now, we can execute :

php vendor/bin/robo parallel:merge-results

result_paracept.xml file will be generated. It can be processed and analyzed.

If you prefer HTML reports, they can be generated in the same way:

public function parallelMergeResults()
{
    $merge = $this->taskMergeHtmlReports();
    for ($i=1; $i<=5; $i++) {
        $merge->from("tests/_output/result_$i.html");
    }
    $merge->into("tests/_output/result.html")->run();
}