mirror of
https://github.com/gryf/coach.git
synced 2026-01-07 14:24:16 +01:00
72a1d9d426
* reordering of the episode reset operation and allowing to store episodes only when they are terminated * reordering of the episode reset operation and allowing to store episodes only when they are terminated * revert tensorflow-gpu to 1.9.0 + bug fix in should_train() * tests readme file and refactoring of policy optimization agent train function * Update README.md * Update README.md * additional policy optimization train function simplifications * Updated the traces after the reordering of the environment reset * docker and jenkins files * updated the traces to the ones from within the docker container * updated traces and added control suite to the docker * updated jenkins file with the intel proxy + updated doom basic a3c test params * updated line breaks in jenkins file * added a missing line break in jenkins file * refining trace tests ignored presets + adding a configurable beta entropy value * switch the order of trace and golden tests in jenkins + fix golden tests processes not killed issue * updated benchmarks for dueling ddqn breakout and pong * allowing dynamic updates to the loss weights + bug fix in episode.update_returns * remove docker and jenkins file
61 lines
2.5 KiB
Markdown
61 lines
2.5 KiB
Markdown
# Coach - Tests
|
|
|
|
Coach is a complex framework consisting of various features and running schemes.
|
|
On top of that, reinforcement learning adds stochasticity in many places along the experiments, which makes getting the
|
|
same results run-after-run is almost impossible.
|
|
To address those issues, and ensure that Coach keeps working as expected, we separated our testing mechanism into
|
|
several parts, each testing the framework in different areas and strictness.
|
|
|
|
* **Docker** -
|
|
|
|
The docker image we supply checks Coach in terms of installation process, and verifies that all the components
|
|
are installed correctly. To build the Docke, use the command:
|
|
|
|
```
|
|
docker build . -t coach
|
|
docker run -it coach /bin/bash
|
|
```
|
|
|
|
|
|
* **Unit tests** -
|
|
|
|
The unit tests test sub components of Coach with different parameters and verifies that they work as expected.
|
|
There are currently tens of tests and we keep adding new ones. We use pytest in order to run the tests, using
|
|
the following command:
|
|
|
|
```
|
|
python3 -m pytest rl_coach/tests -m unit_test
|
|
```
|
|
|
|
* **Integration tests** -
|
|
|
|
The integration tests make sure that all the presets are runnable. It's a static tests that does not check the
|
|
performance at all. It only checks that the preset can start running with no import error or other bugs.
|
|
To run the integration tests, use the following command:
|
|
|
|
```
|
|
python3 -m pytest rl_coach/tests -m integration_test
|
|
```
|
|
|
|
* **Golden tests** -
|
|
|
|
The golden tests run a subset of the presets available in Coach, and verify that they pass a known score after
|
|
a known amount of steps. The threshold for the tests are defined as part of each preset. The presets which are
|
|
tested are presets that can be run in a short amount of time, and the requirements for passing are quite weak.
|
|
The golden tests can be run using the following command:
|
|
|
|
```
|
|
python3 rl_coach/tests/golden_tests.py
|
|
```
|
|
|
|
* **Trace tests** -
|
|
|
|
The trace tests run all the presets available in Coach, and compare their csv output to traces we extracted after
|
|
verifying each preset works correctly. The requirements for passing these tests are quite strict - all the values
|
|
in the csv file should match the golden csv file exactly. The trace tests can be run in parallel to shorten the
|
|
testing time. To run the tests in parallel use the following command:
|
|
|
|
```
|
|
python3 rl_coach/tests/trace_tests.py -prl
|
|
```
|