Notes from Conferences: Run your tests in hundreds of different environments fast. I mean really fast
Talk: https://www.youtube.com/watch?v=cLptIbomkoM
Problem
- large amount of tests: ~450 tests in test suite
- large amount of environments to run test suite in: > 400
- 7 Python versions
- 20 Python web frameworks
- 2-9 versions of each web framework
- total duration: 38 min 51 sec
The aim is to reduce the total duration of tests running time.
Testing stack:
- Pytest
- Flake8, Black & MyPy
- Tox
- Make
- GitHub Actions
Refactoring Ideas
Running tests in paralel
Tox parallel auto
- first idea was to use
tox --parallel auto - default:
- with
tox --parallel auto:
- improved speed from 40 to 25 minutes, but that was not good enough
Splitting it up
- design:
- custom writted script to parse the tox file and create a yaml file for each configuration
- reality:
- there were only 15 concurrent runners, as there was limited amount of runners on the company level, so when someone else was running tests, there would be less runners available
A bit of both worlds - parallelize CPU + parallelize runners
- design:
- saved 20-30 minutes
Cleaning up YAML file
- removed node.js as it’s not used anymore
- removed Redis and Postgres services, as Redis was not used anymore and Postgres was used only for testing
- result: saved 2 minutes
GitHub Actions Cache
- cache is used for caching dependencies from
pypi - the idea was to cache the whole directory, so the dependencies are not installed over and over again
- it works in a way that if any dependency is change, the cache is invalidated, but in the reality the cache didn’t refresh and they were running the tests against the old dependencies
- they decided not to implement it, as it did not work correctly
RAM drives
- the basic idea was - if you can’t use the cache, use RAM drives
- directories in your file system, which is not sitting on the disk but in the memory
- memory is 50x faster than disk, therefore it might be the better to have virtual environments directly in the memory
- saved 2 minutes
Local PyPI Server
- the virtual environments are in the memory now, but all dependencies still have to be downloaded from the PyPI server, before they’re installed, so the idea is to create a local PyPI server
- it was not implemented
Total Results
Ideas Considered
Tests Running Time
- from ~40 minutes to <7 minutes