SAC algorithm (#282)

* SAC algorithm

* SAC - updates to agent (learn_from_batch), sac_head and sac_q_head to fix problem in gradient calculation. Now SAC agents is able to train.
gym_environment - fixing an error in access to gym.spaces

* Soft Actor Critic - code cleanup

* code cleanup

* V-head initialization fix

* SAC benchmarks

* SAC Documentation

* typo fix

* documentation fixes

* documentation and version update

* README typo

benchmarks/README.md

@@ -37,6 +37,7 @@ The environments that were used for testing include:
 |**[ACER](acer)**                | ![#2E8B57](https://placehold.it/15/2E8B57/000000?text=+) |Atari           | |
 |**[Clipped PPO](clipped_ppo)**  | ![#2E8B57](https://placehold.it/15/2E8B57/000000?text=+) |Mujoco          | |
 |**[DDPG](ddpg)**                | ![#2E8B57](https://placehold.it/15/2E8B57/000000?text=+) |Mujoco          | |
+|**[SAC](sac)**                  | ![#2E8B57](https://placehold.it/15/2E8B57/000000?text=+) |Mujoco          | |
 |**[NEC](nec)**                  | ![#2E8B57](https://placehold.it/15/2E8B57/000000?text=+) |Atari           | |
 |**[HER](ddpg_her)**                  | ![#2E8B57](https://placehold.it/15/2E8B57/000000?text=+) |Fetch           | |
 |**[DFP](dfp)**                  | ![#ceffad](https://placehold.it/15/ceffad/000000?text=+) |Doom            | Doom Battle was not verified |

benchmarks/clipped_ppo/README.md

@@ -1,6 +1,6 @@
 # Clipped PPO
 
-Each experiment uses 3 seeds and is trained for 10k environment steps.
+Each experiment uses 3 seeds and is trained for 10M environment steps.
 The parameters used for Clipped PPO are the same parameters as described in the [original paper](https://arxiv.org/abs/1707.06347).
 
 ### Inverted Pendulum Clipped PPO - single worker

benchmarks/ddpg/README.md

@@ -1,6 +1,6 @@
 # DDPG
 
-Each experiment uses 3 seeds and is trained for 2k environment steps.
+Each experiment uses 3 seeds and is trained for 2M environment steps.
 The parameters used for DDPG are the same parameters as described in the [original paper](https://arxiv.org/abs/1509.02971).
 
 ### Inverted Pendulum DDPG - single worker

benchmarks/sac/README.md

@@ -0,0 +1,48 @@
+# Soft Actor Critic
+
+Each experiment uses 3 seeds and is trained for 3M environment steps.
+The parameters used for SAC are the same parameters as described in the [original paper](https://arxiv.org/abs/1801.01290).
+
+### Inverted Pendulum SAC - single worker
+
+```bash
+coach -p Mujoco_SAC -lvl inverted_pendulum
+```
+
+<img src="inverted_pendulum_sac.png" alt="Inverted Pendulum SAC" width="800"/>
+
+
+### Hopper Clipped SAC - single worker
+
+```bash
+coach -p Mujoco_SAC -lvl hopper
+```
+
+<img src="hopper_sac.png" alt="Hopper SAC" width="800"/>
+
+
+### Half Cheetah Clipped SAC - single worker
+
+```bash
+coach -p Mujoco_SAC -lvl half_cheetah
+```
+
+<img src="half_cheetah_sac.png" alt="Half Cheetah SAC" width="800"/>
+
+
+### Walker 2D Clipped SAC - single worker
+
+```bash
+coach -p Mujoco_SAC -lvl walker2d
+```
+
+<img src="walker2d_sac.png" alt="Walker 2D SAC" width="800"/>
+
+
+### Humanoid Clipped SAC - single worker
+
+```bash
+coach -p Mujoco_SAC -lvl humanoid
+```
+
+<img src="humanoid_sac.png" alt="Humanoid SAC" width="800"/>