godotengine/godot-docs
1
0
Fork 0
mirror of https://github.com/godotengine/godot-docs.git synced 2026-01-04 14:11:02 +03:00
Code Issues Packages Projects Releases Wiki Activity

Document unit testing in the engine and modules (#4017)

Browse Source
This commit is contained in:
Andrii Doroshenko
2020-09-27 16:06:37 +03:00
committed by GitHub
parent 23a8dcaf78
commit 8c5a168966
3 changed files with 357 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
development/cpp/custom_modules_in_cpp.rst
View File

@@ -594,6 +594,72 @@ you might encounter an error similar to the following:
ERROR: Can't write doc file: docs/doc/classes/@GDScript.xml
At: editor/doc/doc_data.cpp:956
.. _doc_custom_module_unit_tests:
Writing custom unit tests
-------------------------
It's possible to write self-contained unit tests as part of a C++ module. If you
are not familiar with the unit testing process in Godot yet, please refer to
:ref:`doc_unit_testing`.
The procedure is the following:
1. Create a new directory named ``tests/`` under your module's root:
.. code-block:: console
cd modules/summator
mkdir tests
cd tests
2. Create a new test suite: ``test_summator.h``. The header must be prefixed
with ``test_`` so that the build system can collect it and include it as part
of the ``tests/test_main.cpp`` where the tests are run.
3. Write some test cases. Here's an example:
.. code-block:: cpp
// test_summator.h
#ifndef TEST_SUMMATOR_H
#define TEST_SUMMATOR_H
#include "tests/test_macros.h"
#include "modules/summator/summator.h"
namespace TestSummator {
TEST_CASE("[Modules][Summator] Adding numbers") {
Ref<Summator> s = memnew(Summator);
CHECK(s->get_total() == 0);
s->add(10);
CHECK(s->get_total() == 10);
s->add(20);
CHECK(s->get_total() == 30);
s->add(30);
CHECK(s->get_total() == 60);
s->reset();
CHECK(s->get_total() == 0);
}
} // namespace TestSummator
#endif // TEST_SUMMATOR_H
4. Compile the engine with ``scons tests=yes``, and run the tests with the
following command:
.. code-block:: console
./bin/<godot_binary> --test --source-file="*test_summator*" --success
You should see the passing assertions now.
.. _doc_custom_module_icons:

1
development/cpp/index.rst
View File

@@ -12,6 +12,7 @@ Engine development
variant_class
object_class
inheritance_class_tree
unit_testing
custom_modules_in_cpp
binding_to_external_libraries
custom_resource_format_loaders

290
development/cpp/unit_testing.rst Normal file
View File

@@ -0,0 +1,290 @@
.. _doc_unit_testing:
Unit testing
============
Godot Engine allows to write unit tests directly in C++. The engine integrates
the `doctest <https://github.com/onqtam/doctest>`_ unit testing framework which
gives ability to write test suites and test cases next to production code, but
since the tests in Godot go through a different ``main`` entry point, the tests
reside in a dedicated ``tests/`` directory instead, which is located at the root
of the engine source code.
Platform and target support
---------------------------
C++ unit tests can be run on Linux, macOS, and Windows operating systems.
Tests can only be run with editor ``tools`` enabled, which means that export
templates cannot be tested currently.
Running tests
-------------
Before tests can be actually run, the engine must be compiled with the ``tests``
build option enabled (and any other build option you typically use), as the
tests are not compiled as part of the engine by default:
.. code-block:: shell
scons tests=yes
Once the build is done, run the tests with a ``--test`` command-line option:
.. code-block:: shell
./bin/<godot_binary> --test
The test run can be configured with the various doctest-specific command-line
options. To retrieve the full list of supported options, run the ``--test``
command with the ``--help`` option:
.. code-block:: shell
./bin/<godot_binary> --test --help
Any other options and arguments after the ``--test`` command are treated as
arguments for doctest.
.. note::
Tests are compiled automatically if you use the ``dev=yes`` SCons option.
``dev=yes`` is recommended if you plan on contributing to the engine
development as it will automatically treat compilation warnings as errors.
The continuous integration system will fail if any compilation warnings are
detected, so you should strive to fix all warnings before opening a pull
request.
Filtering tests
~~~~~~~~~~~~~~~
By default, all tests are run if you don't supply any extra arguments after the
``--test`` command. But if you're writing new tests or would like to see the
successful assertions output coming from those tests for debugging purposes, you
can run the tests of interest with the various filtering options provided by
doctest.
The wildcard syntax ``*`` is supported for matching any number of characters in
test suites, test cases, and source file names:
+--------------------+---------------+------------------------+
| **Filter options** | **Shorthand** | **Examples** |
+--------------------+---------------+------------------------+
| ``--test-suite`` | ``-ts`` | ``-ts="*[GDScript]*"`` |
+--------------------+---------------+------------------------+
| ``--test-case`` | ``-tc`` | ``-tc="*[String]*"`` |
+--------------------+---------------+------------------------+
| ``--source-file`` | ``-sf`` | ``-sf="*test_color*"`` |
+--------------------+---------------+------------------------+
For instance, to run only the ``String`` unit tests, run:
.. code-block:: shell
./bin/<godot_binary> --test --test-case="*[String]*"
Successful assertions output can be enabled with the ``--success`` (``-s``)
option, and can be combined with any combination of filtering options above,
for instance:
.. code-block:: shell
./bin/<godot_binary> --test --source-file="*test_color*" --success
Specific tests can be skipped with corresponding ``-exclude`` options. As of
now, some tests include random stress tests which take a while to execute. In
order to skip those kind of tests, run the following command:
.. code-block:: shell
./bin/<godot_binary> --test --test-case-exclude="*[Stress]*"
Writing tests
-------------
Test suites represent C++ header files which must be included as part of the
main test entry point in ``tests/test_main.cpp``. Most test suites are located
directly under ``tests/`` directory.
All header files are prefixed with ``test_``, and this is a naming convention
which the Godot build system relies on to detect tests throughout the engine.
Here's a minimal working test suite with a single test case written:
.. code-block:: cpp
#ifndef TEST_STRING_H
#define TEST_STRING_H
#include "tests/test_macros.h"
namespace TestString {
TEST_CASE("[String] Hello World!") {
String hello = "Hello World!";
CHECK(hello == "Hello World!");
}
} // namespace TestString
#endif // TEST_STRING_H
The ``tests/test_macros.h`` header encapsulates everything which is needed for
writing C++ unit tests in Godot. It includes doctest assertion and logging
macros such as ``CHECK`` as seen above, and of course the definitions for
writing test cases themselves.
.. seealso::
`tests/test_macros.h <https://github.com/godotengine/godot/blob/master/tests/test_macros.h>`_
source code for currently implemented macros and aliases for them.
Test cases are created using ``TEST_CASE`` function-like macro. Each test case
must have a brief description written in parentheses, optionally including
custom tags which allow to filter the tests at run-time, such as ``[String]``,
``[Stress]`` etc.
Test cases are written in a dedicated namespace. This is not required, but
allows to prevent naming collisions for when other static helper functions are
written to accommodate the repeating testing procedures such as populating
common test data for each test, or writing parameterized tests.
Godot supports writing tests per C++ module. For instructions on how to write
module tests, refer to :ref:`doc_custom_module_unit_tests`.
Assertions
~~~~~~~~~~
A list of all commonly used assertions used throughout the Godot tests, sorted
by severity.
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
| **Assertion** | **Description** |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
| ``REQUIRE`` | Test if condition holds true. Fails the entire test immediately if the condition does not hold true. |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
| ``REQUIRE_FALSE`` | Test if condition does not hold true. Fails the entire test immediately if the condition holds true. |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
| ``CHECK`` | Test if condition holds true. Marks the test run as failing, but allow to run other assertions. |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
| ``CHECK_FALSE`` | Test if condition does not hold true. Marks the test run as failing, but allow to run other assertions. |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
| ``WARN`` | Test if condition holds true. Does not fail the test under any circumstance, but logs a warning if something does not hold true. |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
| ``WARN_FALSE`` | Test if condition does not hold true. Does not fail the test under any circumstance, but logs a warning if something holds true. |
+-------------------+----------------------------------------------------------------------------------------------------------------------------------+
All of the above assertions have corresponding ``*_MESSAGE`` macros, which allow
to print optional message with rationale of what should happen.
Prefer to use ``CHECK`` for self-explanatory assertions and ``CHECK_MESSAGE``
for more complex ones if you think that it deserves a better explanation.
.. seealso::
`doctest: Assertion macros <https://github.com/onqtam/doctest/blob/master/doc/markdown/assertions.md>`_.
Logging
~~~~~~~
The test output is handled by doctest itself, and does not rely on Godot
printing or logging functionality at all, so it's recommended to use dedicated
macros which allow to log test output in a format written by doctest.
+----------------+-----------------------------------------------------------------------------------------------------------+
| **Macro** | **Description** |
+----------------+-----------------------------------------------------------------------------------------------------------+
| ``MESSAGE`` | Prints a message. |
+----------------+-----------------------------------------------------------------------------------------------------------+
| ``FAIL_CHECK`` | Marks the test as failing, but continue the execution. Can be wrapped in conditionals for complex checks. |
+----------------+-----------------------------------------------------------------------------------------------------------+
| ``FAIL`` | Fails the test immediately. Can be wrapped in conditionals for complex checks. |
+----------------+-----------------------------------------------------------------------------------------------------------+
Different reporters can be chosen at run-time. For instance, here's how the
output can be redirected to a XML file:
.. code-block:: shell
./bin/<godot_binary> --test --source-file="*test_validate*" --success --reporters=xml --out=doctest.txt
.. seealso::
`doctest: Logging macros <https://github.com/onqtam/doctest/blob/master/doc/markdown/logging.md>`_.
Testing failure paths
~~~~~~~~~~~~~~~~~~~~~
Sometimes, it's not always feasible to test for an *expected* result. With the
Godot development philosophy of that the engine should not crash and should
gracefully recover whenever a non-fatal error occurs, it's important to check
that those failure paths are indeed safe to execute without crashing the engine.
*Unexpected* behavior can be tested in the same way as anything else. The only
problem this creates is that the error printing shall unnecessarily pollute the
test output with errors coming from the engine itself (even if the end result is
successful).
To alleviate this problem, use ``ERR_PRINT_OFF`` and ``ERR_PRINT_ON`` macros
directly within test cases to temporarily disable the error output coming from
the engine, for instance:
.. code-block:: cpp
TEST_CASE("[Color] Constructor methods") {
ERR_PRINT_OFF;
Color html_invalid = Color::html("invalid");
ERR_PRINT_ON; // Don't forget to re-enable!
CHECK_MESSAGE(html_invalid.is_equal_approx(Color()),
"Invalid HTML notation should result in a Color with the default values.");
}
Test tools
----------
Test tools are advanced methods which allow you to run arbitrary procedures to
facilitate the process of manual testing and debugging the engine internals.
These tools can be run by supplying the name of a tool after the ``--test``
command-line option. For instance, the GDScript module implements and registers
several tools to help the debugging of the tokenizer, parser, and compiler:
.. code-block:: shell
./bin/<godot_binary> --test gdscript-tokenizer test.gd
./bin/<godot_binary> --test gdscript-parser test.gd
./bin/<godot_binary> --test gdscript-compiler test.gd
If any such tool is detected, then the rest of the unit tests are skipped.
Test tools can be registered anywhere throughout the engine as the registering
mechanism closely resembles of what doctest provides while registering test
cases using dynamic initialization technique, but usually these can be
registered at corresponding ``register_types.cpp`` sources (per module or core).
Here's an example of how GDScript registers test tools in
``modules/gdscript/register_types.cpp``:
.. code-block:: cpp
#ifdef TESTS_ENABLED
void test_tokenizer() {
TestGDScript::test(TestGDScript::TestType::TEST_TOKENIZER);
}
void test_parser() {
TestGDScript::test(TestGDScript::TestType::TEST_PARSER);
}
void test_compiler() {
TestGDScript::test(TestGDScript::TestType::TEST_COMPILER);
}
REGISTER_TEST_COMMAND("gdscript-tokenizer", &test_tokenizer);
REGISTER_TEST_COMMAND("gdscript-parser", &test_parser);
REGISTER_TEST_COMMAND("gdscript-compiler", &test_compiler);
#endif
The custom command-line parsing can be performed by a test tool itself with the
help of OS :ref:`get_cmdline_args<class_OS_method_get_cmdline_args>` method.