Testing

Server

The whole server can be built and run as a docker image, easing deploying and testing the server architecture. In this section, we'll describe the way the server employs testing strategies and outline why we chose particular ways of testing.

Server tests: an overview

The need for testing

As we were developing TeamCode, we noticed an increasing need to test features before they were merged into our develop branch. Having to push a quick fix commit after the merge because we forgot something small definitely occurred a few too many times.

Testing frameworks: self-written, pytest

To improve the situation, we started writing some Python scripts that connected to the server over the websocket connection and verified that some common messages returned the values we expected.

As time went on, we also wrote tests using the pytest framework, which greatly simplified boilerplate testing code and could also generate neat reports.

Code coverage

We used the coverage.py framework to measure code coverage, which was integrated into pytest using the pytest-cov plugin. At the end of the project, we ended up covering about 85 percent of our codebase. This testing could definitely be improved, as not all edge cases are considered even when all of the code is run, but we're pretty happy with the coverage overall.

Linting and type-based verification

Aside from these formal tests, our Python code uses Python 3 type annotations, and we run the mypy type checker for type-based code checking. Additionally, we use the flake8 Python style checker to test for PEP8 compliance.

Running automatic tests

To run all of these tests, the Docker image (as built using the instructions in the installation guide) can be used. The tests require some sample files in the file root, however, so some prep needs to be done. The process is shown below.

cd /path/to/cte/
mkdir -p src/server/file_root/tmp
touch src/server/file_root/{kaas_is_dood,test}.txt
docker build -t psedit . && docker run --mount src=$PWD/src/server/file_root,dst=/file_root,type=bind -it psedit test

At the end of the tests, the coverage checker should comment on the amount of coverage the tests have. Additionally, the output should contain a pytest report, showing the results of running the pytest tests.