Parallelism in PHPUnit
2022 Update: This contents of article has been superseded by Laravel’s in-built features and may no longer seem particularly revolutionary. I’m going to leave this article here has a timestamp for when I had figured things out ahead of the curve in 2019. There’s also mentions of Justly below, which is no longer operating and is not associated with the current contents of the previously linked domain. If you’re still interested, please enjoy the read below.
Summary
While this particular use case is about PHP development with Laravel & PHPUnit, you maybe able to apply some of the techniques to your preferred language workflows. We also are running xdebug (unless otherwise mentioned) which can sometimes double the time required to run a simple test.
TLDR:
1. Use parallelism to run your test framework tests in individual threads. This does require more CPU power.
2. Use in combination with an in-memory SQLite database to keep your testing dependencies simple, and keep each test on their own database, so there’s no concurrency issues.
3. Bootstrap once per run of your tests (and only if your migrations/seeds have changed) and then seed any test specific data in the test itself, that can be rolled back if need be.
This probably seems simple, but let’s talk about why this article is long.
Context
The first thing I should mention here is what I started trying to solve, that lead me to write about the techniques I use in my workflow.
Building a web app can turn into a monolith of logic before you know it. Even if you take the high road to make your code as testable as can be by abstracting your domain logic from your controllers, and create services within your code base to encompass the real world processes you’re mimicking, before you know it, this application has a lot of responsibility.
The alternative is creating micro service projects, which can create and sustain a lot of (exponential) overhead, such as setting up pipelines, deployment processes, dependency management, team specialisation and cost. Managing this kind of overhead contributes to the make or break of a small development team on a burn to break even, and so having a single project can soothe these pains.
Considering this, while I believe and practice test driven development when I can, I strongly believe that the strongest development discipline is domain driven design – the two are not mutually exclusive, but when you’re short on time and money, having well modelled code is, in my opinion, more important when you cannot afford all of the testing an idealist dreams of. At least well modelled code can retrofit these tests with ease and if it’s a monolithic source code, then you’ll have a lot of tests to run eventually.
So this leads me to where I am now. In a software-lead business, you need to identify what your key interactions are in your customers experience with your product. For me at Justly, (no longer operating and not associated with the website at the previous domain) the most important is the complex, yet feature-demanding documents we automatically generate for our customers. I previously had thought it was making sure there was no errors around payment and so I had focussed on TDD in this area first.
When I realised this was not the area that needed the most quality control, I began to write tests to ensure our generated documents were exactly as they needed to be. This leads me to having 172 tests with 512 assertions on the English Language output generated for the two precedent-based documents we provide.
To put this in perspective, I have aimed for 100% coverage on these specific logic adapters, as we as a business have to strive to test all major output decisions. When it comes to legal documents, not a single sentence can be incorrect. This means not only are there a lot of tests, but long tests too.
My journey with the testing framework
When I first set out to bootstrap our testing, I knew I had to be as free from dependency as possible when using a database for testing so that I can get it into a continuous integration (CI) process. There was already a language dependency on PHP and I wanted to avoid as many more as possible. SQLite demonstrated that I could achieve what I wanted so long as the testing machines could have access to SQLite via a package on the machine or CI environment.
That’s it. I set our tests run on SQLite. A migration and a base seed for each test, and then an additional data seed if required for the test specifically.
Problem 1:
I have 172 tests for a single feature, that time spent bootstrapping adds up fast. I was seeing around 8 minutes for a run of the feature test. Remember, I’m using xdebug, so this could probably be around 4 minutes without it.
Trying to improve this, I thought if I could migrate and seed the database once before running any test, then simply copy the database, or even better, implement Laravel's DatabaseTransactions trait so that before doing test specific seeds, I’d get huge gains in test time. I was right.
I’d found an improvement, reducing tests with xdebug from 8 minutes to 2.5 minutes for a single feature. I’d been pretty happy for a while, even if this was chewing through CI minutes with each build. The CI server doesn’t run xdebug, so I’m seeing just more than 1 minute for this process.
So here’s my testing bootstrap file, adding this as the bootstrap in phpunit.xml was the way to go.
File: bootstrap/testing.php
<?php
/* Force a clear of clear the configuration cache. */
if (file_exists(sprintf('%s/cache/config.php', __DIR__)) === true) {
unlink(sprintf('%s/cache/config.php', __DIR__));
}
require_once sprintf('%s/autoload.php', __DIR__);
require_once sprintf('%s/app.php', __DIR__);
/**
* The database file used by normal phpunit.
*
* @var string $database
*/
$database = sprintf('%s/../../tmp/tests/database.sqlite', __DIR__);
/* Remove copy of the database so it's fresh when testing. */
if (file_exists($database) === true) {
unlink($database);
}
/* Now create the file, ready for population. */
touch($database);
/* Now migrate and seed the database */
exec('php artisan migrate --seed --database=testing', $databaseLog);
//print_r($databaseLog);
Problem 2: This is just one critical feature we test, and we have many to go.
Our team also has a need to gauge our testing quality off some sort of metric, and one of the metrics we can use to see what’s being tested, is using coverage testing. This requires xdebug to be enabled to complete. This single feature took around 22 minutes to run a coverage test for. Not economical on its own, nor is it for an entire product of features
So I set out to find a better way to run tests. I stumbled across an article that recommended a Github package: Paratest which allows you to run PHPunit tests on individual CPU threads. Personally, this was like Christmas to me.
My host machine has at least 8 idle threads at any given time, so that’s a lot of unused power I could be testing with. I installed the package with composer, then simply changed the testing binary, and set a number of threads to utilise. I also added the --runner WrapperRunner option as per the documentations recommendation.
Before:
./vendor/bin/phpunit --group ClauseAdapterTests
After:
./vendor/bin/paratest --runner WrapperRunner -p 8 --group ClauseAdapterTests
Note:
Both of these commands use the same phpunit.xml file at this point in the journey.
Problem 3:
SQLite couldn’t handle concurrent connections and I was resetting our database with each test. So I now needed a database for each connection. While I did try to create a file-based database for each test, it wasn’t really viable to change our database for each test.
I read up about in-memory databases with SQLite and found that it’s essentially a database unique to a connection. “Perfect!” I thought to myself thinking I would be almost done. I’ll setup a migration and base seed for each test and we’re away.
Set your config('database.connections.testing.database') value to :memory: to utilise this feature.
Problem 4: After all of that, our tests came in at the same time, because of the migrations and initial seeds, DUCK! I said really loudly.
I needed to bootstrap the database with the and seeded data before running the test logic. Fortunately by doing this already with the SQLite file database, I was essentially setup to go. I just had to get the data into memory as I ran each test. The solution I had found was using the SQLite backup API, however this has no PHP library that supports it. I tried a shell script from the PHP bootstrap thread and managed to get the database from a backup and back into memory.
Problem 5: The memory database only exists for the single connection, then it’s gone again, so, every time I did this, I would lose my data.
So my requirement was that I needed to load the data to memory in each test, without any migration and seed overhead, after the connection had been opened.
I decided that after the initial database had been created, I would dump the schema as table creation and insert statements using SQLite’s command for doing so. This gave me something that I could run in each test that literally couldn’t be faster unless the OS was doing it (but that would have it’s own connection)
File: bootstrap/testing.php
<?php
/* Clear the configuration cache. */
if (file_exists(sprintf('%s/cache/config.php', __DIR__)) === true) {
unlink(sprintf('%s/cache/config.php', __DIR__));
}
require_once sprintf('%s/autoload.php', __DIR__);
require_once sprintf('%s/app.php', __DIR__);
/**
* The database file used by normal phpunit and the export for memory loading
* used by paratest.
*
* @var string $database
* @var string $export
*/
$database = sprintf('%s/../../tmp/tests/database.sqlite', __DIR__);
$export = sprintf('%s/../../tmp/tests/export.sql', __DIR__);
/* Remove copy of the database and export, so it's fresh when testing. */
if (file_exists($database) === true) {
unlink($database);
}
if (file_exists($export) === true) {
unlink($export);
}
/* Now create the files, ready for population. */
touch($database);
touch($export);
/* Now migrate and seed the database */
exec('php artisan migrate --seed --database=testing', $databaseLog);
//print_r($databaseLog);
/* Take a sql dump of the database for importing when using SQLite :memory: */
exec(sprintf('sqlite3 %s .dump > %s', $database, $export), $exportLog);
//print_r($exportLog);
Problem 6:
The dump is created as a transaction and I can’t do a nested transaction in the setUp() method of the test as before parent::setUp() is run, I don’t have a database connection where as after, I’m in the middle of an open transaction.
My options are that I can either remove the transaction logic from sequel dump (not really a clean technique) or I can find a place to hook the import before the test begins it’s in it’s own transactions.
I found that the transaction trait I implemented earlier is initialized by the parent::setUp() which contained a call to a method named setUpTraits() which basically starts any of the Laravel traits you might use in testing.
So if I do something like this on my class that extends the base Laravel test:
use Illuminate\Support\Facades\DB;
...
/**
* Boot the testing helper traits.
*
* @return array
*/
protected function setUpTraits()
{
DB::unprepared(file_get_contents(storage_path('tests/export.sql')));
parent::setUpTraits();
}
Now I’m able to run our feature tests in parallel, in memory, with little per-test bootstrapping. Progress report: I’m feeling pretty good. Tired. But good.
Problem 7: A sneaky surprise attack! My test databases are empty! Turns out Paratest ran my bootstrap once when starting, and once again for each test process. This meant my migration, seeds and sequel export was being re-made each time (Causing a blank export through race conditions)
So I moved the bootstrap to be once before, and outside, of Paratest:
php ./bootstrap/testing.php && ./vendor/bin/paratest -c phpunit.paratest.xml --runner WrapperRunner -p 6 --group ClauseAdapterTests
Where phpunit.paratest.xml did not do any bootstrapping. This means that through an environment variable in phpunit.paratest.xml I can also specify to use an in memory database for Paratest specifically, and a regular file-based database for PHPunit tests, persevering my original testing setup for CI.
Technically, between tests, I don’t even need to bootstrap unless there’s been a migration or global seed change. So here’s the results of my journey.
All running 172 tests, 512 assertions
PHPUnit, SQLite database with transactions, xdebug disabled, no coverage: 4 Minutes
PHPUnit, SQLite database with transactions, xdebug enabled, no coverage: 8 Minutes
PHPUnit, SQLite database with transactions, xdebug enabled, with coverage: 22 Minutes
All using 8 processor threads:
Paratest, Memory database with transactions, xdebug disabled, no coverage: 6.74 Seconds
Paratest, Memory database with transactions, xdebug enabled, no coverage: 25.63 Seconds
Paratest, Memory database with transactions, xdebug enabled, with coverage: 2 Minutes
Remember each of these tests is running a SQLite import, and it’s own seeder if required. (99% of these tests have a dedicated seed)
I’m pretty happy with the result. This is the first time I’ve written anything about programming in the last 5 years and this was a success I thought worth documenting. If you have any feedback or suggestions on how to improve this process, tweet me, I’d love to hear it.