SAC algorithm (#282)

* SAC algorithm

* SAC - updates to agent (learn_from_batch), sac_head and sac_q_head to fix problem in gradient calculation. Now SAC agents is able to train.
gym_environment - fixing an error in access to gym.spaces

* Soft Actor Critic - code cleanup

* code cleanup

* V-head initialization fix

* SAC benchmarks

* SAC Documentation

* typo fix

* documentation fixes

* documentation and version update

* README typo

docs_raw/source/components/agents/index.rst

@@ -21,6 +21,7 @@ A detailed description of those algorithms can be found by navigating to each of
    imitation/cil
    policy_optimization/cppo
    policy_optimization/ddpg
+   policy_optimization/sac
    other/dfp
    value_optimization/double_dqn
    value_optimization/dqn

docs_raw/source/components/agents/policy_optimization/acer.rst

@@ -38,6 +38,7 @@ Each update perform the following procedure:
     .. math::   \text{where} \quad \bar{\rho}_{t} = \min{\left\{c,\rho_t\right\}},\quad \rho_t=\frac{\pi (a_t \mid s_t)}{\mu (a_t \mid s_t)}
 
 3. **Accumulate gradients:**
+
     :math:`\bullet` **Policy gradients (with bias correction):**
 
         .. math::  \hat{g}_t^{policy} & = & \bar{\rho}_{t} \nabla \log \pi (a_t \mid s_t) [Q^{ret}(s_t,a_t) - V(s_t)] \\

docs_raw/source/components/agents/policy_optimization/sac.rst

@@ -0,0 +1,49 @@
+Soft Actor-Critic
+============
+
+**Actions space:** Continuous
+
+**References:** `Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor <https://arxiv.org/abs/1801.01290>`_
+
+Network Structure
+-----------------
+
+.. image:: /_static/img/design_imgs/sac.png
+   :align: center
+
+Algorithm Description
+---------------------
+
+Choosing an action - Continuous actions
++++++++++++++++++++++++++++++++++++++
+
+The policy network is used in order to predict mean and log std for each action. While training, a sample is taken
+from a Gaussian distribution with these mean and std values. When testing, the agent can choose deterministically
+by picking the mean value or sample from a gaussian distribution like in training.
+
+Training the network
+++++++++++++++++++++
+Start by sampling a batch :math:`B` of transitions from the experience replay.
+
+* To train the **Q network**, use the following targets:
+
+  .. math:: y_t^Q=r(s_t,a_t)+\gamma \cdot V(s_{t+1})
+
+  The state value used in the above target is acquired by running the target state value network.
+
+* To train the **State Value network**, use the following targets:
+
+  .. math:: y_t^V = \min_{i=1,2}Q_i(s_t,\tilde{a}) - log\pi (\tilde{a} \vert s),\,\,\,\, \tilde{a} \sim \pi(\cdot \vert s_t)
+
+  The state value network is trained using a sample-based approximation of the connection between and state value and state
+  action values, The actions used for constructing the target are **not** sampled from the replay buffer, but rather sampled
+  from the current policy.
+
+* To train the **actor network**, use the following equation:
+
+  .. math:: \nabla_{\theta} J \approx \nabla_{\theta} \frac{1}{\vert B \vert} \sum_{s_t\in B} \left( Q \left(s_t, \tilde{a}_\theta(s_t)\right) - log\pi_{\theta}(\tilde{a}_{\theta}(s_t)\vert s_t) \right),\,\,\,\, \tilde{a} \sim \pi(\cdot \vert s_t)
+
+After every training step, do a soft update of the V target network's weights from the online networks.
+
+
+.. autoclass:: rl_coach.agents.soft_actor_critic_agent.SoftActorCriticAlgorithmParameters

docs_raw/source/diagrams.xml

@@ -1 +1 @@
-<mxfile modified="2019-02-10T18:35:59.935Z" host="www.draw.io" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36" version="10.2.1" etag="oVRlGCHRoZX2a2BLwc5B" type="device"><diagram id="33c2a640-8c1e-935c-0e0a-86b5dd5c932c" name="Page-1">7V1dd6M40v41uUwOQnxe5qPTPXu6ezPT7767e0lsYrODwYtx0plfvxJGGFQyYFvCkFZ2T48tA8Y8VaWqR1WlK3y/+vk5C9bLb+k8jK9MY/7zCj9cmSZCnkP+Q0fedyOu6e0GFlk0Lw/aD/yI/grLQaMc3UbzcNM4ME/TOI/WzcFZmiThLG+MBVmWvjUPe0nj5reug0UIBn7MghiO/jOa58vdqGcb+/EvYbRYsm9GRvnJczD7c5Gl26T8visTvxR/u49XAbtWefxmGczTt9oQ/nSF77M0zXevVj/vw5g+W/bYduc9Hvi0uu8sTPJeJ2C8O+U1iLchu+fizvJ39jSK3xPSM9AVvntbRnn4Yx3M6KdvBH8ytsxXcfnxIg429Okb5PUsXUWz8vUmz9I/w/s0TrPiqtiZeeHzS/UJe86YjLxEcVw7ch6E3suMjqdJXgqLaZTva8cZxR8ZD+JokZCxOHzJ6dtsVp7lkHfwEZVP7TXM8vBnbah8ZJ/DdBXm2Ts5hH1qeSV+70xu/VLA3/biYjGhWNZEBTvl8w1KEV1UV9/DRF6USIlRczyFoM2DzbI6rxXBTw79Xx8ESx0gX5QF8yjco5akSSgDWFcOsASfBq6m4RgQV0uAq8Nswzm4+obGNZeEpNvUUGqjAZJYhCSyZCCJAJIBmaeGsay++2C4rmT8kNmCnwzAfOvG5iBz/BsP7/9sAGClMXUApeBndmtimMxvqa9B3s0oNBSQOl7V3E6fUBw8h/Fd5R3UHuJd8T8xWj0fP3nq2fu/6DeRR1i+/ffBeddwH2/v2kDLg2wR5k1BDucNjwnCWEPFFoDCxrIwDvLotelniZAqv+EpjcjdVVJS2V4mI5XLyK6xSbfZLCxPq3s+4Eo2L2/Y4K61exDgWgT14L122JoesDnmprHRem+27555ArLbTwC3xJ1AXux+5V5bKqx7KZDbQ4H0VNZzKuOcEtx3KjN5gT7FFCKVzubw4LUa0oZgSQ8bzKbOVeFkDUbHEcUMvowZTaOoBEXk90TR4meqk5TR+eVhlBTqYX4GFGgjxiKjKsO/hLPjdhOSgSTM39Lsz4Lvor8wC+cRjRsMEj1EaUJezCMCQPS83b0dazwxYOCAnSaQotlRWZwA2bN/FDjmS/pv+HNNfnJQIrdO42j2zqDdBKt1TA96ydJVdUYJOIFQI92NtDsg0hZAepaFQU5BC5K6gkbJC8U3ff5PuNPchIK5U+8or+DP02wvKNXJu4ODpAgv5vOIDgfx/qrBc7rNwWlaNoBsWHhA2YCTcqXdAY15CaqFjlcwb5PoJc1W8Tuv/9URmwIw856OxWmyKH4J+SHkE3J/1DqEL8E2bkwNpYwUt7HRUlED9xIGA4ZNe4NB8SZSsYmYPGyiAuKdEJBnnzMTMS9eMDszr45Jwjc6wSSvVL7CzTpNqIEZK+QjoBeNJtnjWwOKAvNIOthg8lNyjkJsPHv21GoPuRxipMOMPKCQjN/RBxPNgvi2/GAVzef0a4RCsBcToy/R0pcNKY4rf6FgcfJchTYFvp6YCJEAoqaEG7PdSBhhbDdFAjkc1r0JYYO7EB+vS2KD+Rvu4naRB2hqTmzPY2pRjwyAX0KuzY8p18jn5IdnEy8k17ahWK5hxJZtizh8GRTBWDFG/bAoKbwvh/jsRFaS5826EpnJz86tyiR7dsb8orvhgukZCTk1GfOz3QvvJfHBt2uNtgSXGvVA21SFtgvQni3TdNNCi5wPabmm91Gda55DMQXhcuUVSQcULlClz5swe4Vh7TR185JxkidA0lMEJLvGRPzJn1FecyfJu3+X9yDZ09y5a1f1CKrufLI83JF4n3zQYbn2ad4n7xU6vIhJ8j75G7ZRu/fJ3xd3/NneZ6WyWgsOaQGagBpwsZPFs4GnqoGL1agBH1RdXA0gacYtV3yIuX3YKMv2ucxFR0CDMoMtfXrXdBGDYVSmitdk79TEWMzVtvhnZ8U2ROtI+wFJnI41L21AehgQTlRMaD5ES2FSzAdkaZqpELuFTbbkGSXVB7NtVvzm/YevQRYFz7Fe9zxi3RMZBgRb2cKnCVmaYD6vICUGIosSOj1QhX4LsuKjtPq8vkauIe7Luu6rUesQi5ZFpUDcIz38w/oD9XCGWbbGkpI7ah8BGafHM9aNU/tzrabPMNDaKTK8jjoXv/2EswMc/EHifEnCP1pBd08WdO5KQy2fdoo2v/xy/AmgOUDnbzclKw+MJck0S51vkA/JZ9U2M+bSbb4u8mLZ6WP1Fi7o+dlDZrxhGLf9imZxZHSmjbhZuykgsmykcza5eQZjgGGAydkUnkDgbEmUTMOUXD7w4C2M5w1pYQQJADWGIFxflexBuI426bxIAGFJH4tdZS4LkzR7dBx7xDMKtgNgR6roZwzDzSZ/1Mirr9FHq3CVll+l1bknjyBofqOMR8A9CpR/BY/BG5XHYPEyYZ6ahjpYIAVC/ouHOUw9aqI9L7T7Uc8/EuYfUeW2K1AJGdOPdbF2eaOpv1fURkFUsCnuoyClmhfGp1yUsPcjNiBEWGfNWn0dKhy9DGXji/X2smCEWBQBfBBjfMk8U2RZXbCqSju1enRGGZH/qC4vxTZH5UEC78oyhCJybpoKsvzW6w7JQVmC2u80oVp1YHoh31l0d7iiHYTzGS3zZ0y3nlP6zyliCRhiTmFKJ2oBEUdFOX/6AmBvtH7QBMXRgFvu5QCH9ONA0I0mEJDUiIvP3EeOgHYSduKS0RbPhrRTHAZZstfO0iBL9w0/eDFZxa5XsApWB5CpiCZ2IOWyXc93aYOls1QL4LTJ7W1y3cv1ZHYmlWpzbolY+cDG4sVbflMOTk8v56/k87IxpKfOdmOYhkxdplixhGfUMSYxSo7jIcuxDezxDoWHpEScfIOY/eYQqlPBPKt9BcN2zj3B6ugXDqN4yf3CHUjKxWlKF/FfUkrKJtvVVb1ZR7HET2OoPFoJmt1Nk7obdB0FyIAjYOBVlYE4ovDJiSlm8+i1gaXz3y3dDemOQnpdonNLw2rqP1efkldlVsfuKps17cq5G4tTgkI5Tm6s/lFtuPheNqrF6Vhx4ptHIE/UGFuROLnQ3dfbrbTCZfou2P7icq69C1178ZKMRrANQcu8HIIqF8Z/8f0+bMHErG7rKhcujWtbeqwm2t7lNBF60iF9XZBfH8KvGZbHFIDbmbugapHbhYvc5TzJOsfrtsKngcyF9lhQEq+sc6HrAswoscJ+YAlG0+DWn1oN2T1D5NYpIi7lwIUc0dEcVpDljBUrb7AYe4zozyy/UMCbkcHyEHq7/wnz/L0EPNjmKRlKs3yZLuiuB19p/F8X2AOUWauknEl+sam4Tn65fYt6erNavVvKwzAHSs542Ep1OS7MRxkJ/2hZnMcGKqD670ro8bONq6hjGrzpDg7Q4fnPo09gFWaHfz3v+TZPOJtl9KbVvOgSdL8rUrdxLUZZNicm1XbcR6sbfyVHUeMCINmsSqG37hx9Au7QTnBLWDKl70FnNVqts1R3dL06xUXlt74wOuk6VVGI90FKtVRaUZHfaI7LivIC5Zxa/+3ynf/BlZS5LF1GEfzGY08wuxrM8LdkSm4w44tcFlUrZVVz/IuulX30zDjBTvdD0g1+j94cE0pdPYad3yevem2W/IjkVYOvYzMwrIa3RLsgWKxHyllYQrKXPLE1LSBYBUmwEFSljc/XGoNS7luhtbY1EM3HUnTyYvs6OzMvfH7po5PzIPReZjKUD0laeCEYcaAhQSdkLMo6QDxDcxJqsM4nDl/DWKn2yZ0dx6F7oPOEYMVTne71CGfU6J5nPmOn13w4t0Nvbp2ue3JQ4vIqkSCj3xMk9LPWHudtoSnK/lblBxPVpU7+R3KDx6HofG8OxLiEAZLEyBSvFb0PSgiiNKSiQ5dWK/rUFB3UDQ2r6Bfzpqel6BhseetClNQp+pA54FrR1Sg6RkDRvQEVvUfzeR02d4fNpqjkWlnYTNxAAJuOm2XEzaaoxlpV3FxxLXqa7YKJs5FYQG8om2aRDpw/gKbzIRnGA/rTyNSKflLgPKyiD7lSrBV9oMB5WEXXDFkvlEDgjM0BA2ekGbLpKzoInDEeMHBGF2PIfMfFQS9FDxFRdffCM7rN5+MjS7DEjFgesXxdH5IkC5PXKEuTldZ4FRqPXAxlCU4biE0l8pVes2Wn4Gbzk71pQdTEbJmMDoUIabZMiv7ZfKqe6Qi0TxlbZmq2rCdMvGMkMJLKfGtTs2UfQNP5XE5sDuhbM73Wit6OEl9rMbCia7Zs+oru8HsBDavomi3rhRJfH0YUHaKkTtE1WzZ9RXcdoOjugIp+scB5WmyZYwCGAwszElSxZSYMlTVbNk2NdxBgXrEgh1QdW4ZhvA5oF+M1Ct80nv1sA+cEWIL+sMh3VaEJ43pQKKnR7M9lYwS47GHxhL63JkVlkKJI4JurI0WxyDnXIVQ3KYqcAUMoLFot1iHUxDSdp9uQO2AIhS+2X9q0FB2QosMquijQ1Yo+LUUHpOiwiq6r6XuhBEhR5ECUlCm6JVqM1oo+LUUHpChi/cGHUHRLtFI+iKJXT7R/G6m+zQL36UOOkrCZ37TaNWDQjIXRFr/X2GmYmQCzT3UWcxD4ht/AVjGqQA9d1parQYUIbLmcckoLLmXfLkR46tZfB1KxAdXsCpJ8XcGqhRxLCkmQKFlvaZdJAkUuoLIm60BV8iPFmvLine truncated
+<mxfile modified="2019-04-24T09:04:49.730Z" host="www.draw.io" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" etag="Oj1WgTFAGXXvHH9O1PX0" version="10.6.3" type="device"><diagram id="33c2a640-8c1e-935c-0e0a-86b5dd5c932c" name="Page-1">7V1bd6M6sv41eUwWQlwfc+l073O6e7J3nzln5uksbBOH2Rg8GCed/etHwgiDSgZsSxjSyszqbcvmYr6qUtWnqtIVvl/9/JwF65dv6SKMr0xj8fMKP1yZJkK2Rf5DR953Iy4qB5ZZtCi/tB/4Ef0VloNGObqNFuGm8cU8TeM8WjcH52mShPO8MRZkWfrW/NpzGjevug6WIRj4MQ9iOPp/0SJ/2Y16trEf/xJGyxd2ZWSUn8yC+Z/LLN0m5fWuTPxc/O0+XgXsXOX3Ny/BIn2rDeFPV/g+S9N892r18z6M6bNlj2133OOBT6v7zsIk73UAxrtDXoN4G7J7Lu4sf2dPo/g9IT0CXeG7t5coD3+sgzn99I3gT8Ze8lVcfryMgw19+gZ5PU9X0bx8vcmz9M/wPo3TrDgrduZeOHuuPmHPGZOR5yiOa99cBKH3PKfjaZKXwmIa5fva94zij4wHcbRMyFgcPuf0bTYvj3LIO/iIyqf2GmZ5+LM2VD6yz2G6CvPsnXyFfWp5JX7vTG59bzfwthcXiwnFS01UsFM+36AU0WV19j1M5EWJlBg1x1MI2iLYvFTHtSL4yaH/64NgqQPkQlmwiMI9akmahDKAdeUAS/Bp4GoajgFxtQS4Osw2nIOrb2hcc0lIuk0NpTYaIIlFSCJLBpIIIBmQeWoYy+q7D4brSsYPmS34yQDMt25sDjLHv/Hw/s8GAFYaUwdQCn5mtyaGyeKW+hrk3ZxCQwGp41XN7fQJxcEsjO8q76D2EO+K/4nR6vn4yVPP3v9Br0QeYfn2nwfnXcN9vL1rAy0PsmWYNwU5XDQ8JghjDRVbAAoby8I4yKPXpp8lQqq8wlMakburpKSyvUxGKpeRnWOTbrN5WB5W93zAmWxe3rDBnWv3IMC5COrBe+1ra/qFzTE3jY3We7N998wDkN1+ALgl7gDyYvcr99pSYd1LgdweCqSnsp5TGeeU4L5TmckL9CmmEKl0NocHr9WQNgRLethgNnWuCidrMDqOKGbwZcxoGkUlKCK/J4oWP1OdpIzOLw+jpFAP8zOgQBsxFhlVGf4lnB23m5AMJGH+lmZ/FnwX/YVZuIho3GCQ6CFKE/JiEREAotl293as8cSAgQN2mkCKZkdlcQJkz/5e4Ji/0H/Dn2vyk4MSuXUaR/N3Bu0mWK1j+qXnLF1VR5SAEwg10t1IuwMibQGk51kY5BS0IKkraJQ8U3zT2b/CneYmFMydekd5BX+eZntBqQ7efTlIivBisYjocBDvzxrM0m0ODtOyAWTDwgPKBpyUK+0OaMxLUC10vIJ5m0TPabaK33n9r76xKQAz7+lYnCbL4peQH0I+IfdHrUP4HGzjxtRQykhxGxstFTVwL2EwYNi0NxgUbyIVm4jJwyYqIN4JAXn2OTMRi+IFszOL6jtJ+EYnmOSVyle4WacJNTBjhXwE9KLRJHt8a0BRYB5JBxtMfkrOUYiNZ8+eWu0hl0OMdJiTBxSS8Tv6YKJ5EN+WH6yixYJeRigEezEx+hItfdmQ4nvlLxQsTp6r0KbA1xMTIRJA1JRwY7YbCSOM7aZIIIfDujchbHAn4uN1SWwwf8Nd3C7yAE3Nie15TC3qkQHwS8i1+THlGvmc/PBs4oXk2jYUyzWM2LJtEYe/BEUwVoxRPyxKCu/LIT47kZVktllXIjP52blVmWTPzphfdDdcMD0jIacmY362e+H9Qnzw7VqjLcGlRj3QNlWh7QK05y9pummhRc6HtFzT+6jONc+hmIJwufKKpAMKF6jS2SbMXmFYO03dvGSc5AmQ9BQByc4xEX/yZ5TX3Eny7p/lPUj2NHfu2lU9gqo7nywPdyTeJx90WK59mvfJe4UOL2KSvE/+hm3U7n3y98V9/2zvs1JZrQWHtABNQA242Mni2cBT1cDFatSAD6ourgaQNOOWKz7E3D5slGX7XOaiI6BBmcGWPr1ruojBMCpTxWuyd2piLOZqW/yzs2IbonWk/YAkTsealzYgPQwIJyomNB+ipTAp5gOyNM1UiN3CJlvyjJLqg/k2K37z/sPXIIuCWazXPY9Y90SGAcFWtvBpQpYmWCwqSImByKKETg9Uod+CrPgorT6vr5FriPuyrvtq1DrEomVRKRD3SA//sP5APZxhlq2xpOSO2kdAxunxjHXj1P5cq+kzDLR2igyvo87Fbz/g7AAHf5A4X5Lwj1bQ3ZMFnTvTUMunnaLNL78cfwBoDtD5203JygNjSTLNUucb5EPyWbXNjLl0m6+LvFh2+Fi9hQt6fvaQGW8Yxm2/olkcGZ1pI27WbgqILBvpnE1unsEYYBhgcjaFJxA4WxIl0zAllw88eAvjeUNaGEECQI0hCNdXJXsQrqNNuigSQFjSx3JXmcvCJM0eHcce8YyC7QDYkSr6GcNws8kfNfLqa/TRKlyl5aW0OvfkEQTNb5TxCLhHgfKv4DF4o/IYLF4mzFPTUAcLpEDIf/Ewh6lHTbQXhXY/6vlHwvwjqtx2BSohY/qxLtYubzT194raKIgKNsV9FKRU88L4lIsS9n7EBoQI66xZq69DhaOXoWx8sd5eFowQiyKAD2KML5lniiyrC1ZVaadWj84oI/If1eWl2OaoPEjgXVmGUETOTVNBlt963iE5KEtQ+50mVKsOTC/kmkV3hyvaQTif0zJ/xnTrOaX/nCKWgCHmFKZ0ohYQcVSU86fPAPZG6wdNUBwNuOVeDnBIPw4E3WgCAUmNuPjMfeQIaCdhJy4ZbfFsSDvFYZAle+0sDbJ03/CDF5NV7HoFq2B1AJmKaGIHUi7b9WKXNlg6S7UATpvc3ibXvVxPZmdSqTbnloiVD2wsXrzlN+Xg9PRy/kw+LxtDeupsN4ZpyNRlihVLeEYdYxKj5DgeshzbwB7vUHhISsTJN4jZbw6hOhXMs9pXMGzn3AOsjn7hMIqX3C/cgaRcnKZ0Ef85paRssl1d1Zt1FEv8NIbKo5Wg2d00qbtB11GADDgCBl5VGYgjCp+cmGK2iF4bWDr/3tLdkO4opNclOrc0rKb+c/UpeVVmdezOslnTrpy7sTglKJTj5MbqH9WGi+uyUS1Ox4oT3zwCeaLG2IrEyYXuvt5upRUu03fB9heXc+1d6NqLl2Q0gm0IWublEFS5MP6L7/dhCyZmdVtXuXBpXNvSYzXR9i6nidCTDunrgvz6EH7NsDymANzO3AVVi9wuXOQu50nWOV63FT4NZC60x4KSeGWdC10XYEaJFfYDSzCaBrf+1GrI7hkit04RcSkHLuSIjuawgixnrFh5g8XYY0R/ZnlBAW9GBsuv0Nv9V5jn7yXgwTZPyVCa5S/pku568JXG/3WBPUCZtUrKmeQXm4rr5Jfbt6inN6vVu6U8DHOg5IyHrVSX48J8lJHwj5bFeWygAqr/roQeP9u4ijqmwZvu4AAdnv88+gBWYXb41/Oeb/OAs1lGb1rNiy5B97sidRvXYpRlc2JSbcd9tLrxZ3IUNS4Aks2qFHrrztEH4A7tBLeEJVP6HnRWo9U6S3VH16tTXFR+6wujk65TFYV4H6RUS6UVFfmN5risKC9Qzqn13y7f+R+cSZnL0mUUwW889gCzq8EMf0um5AYzvshlUbVSVjXHv+ha2UfPjBPsdD8k3eD36M0xodTVY9j5ffKq12bJj0heNfg6NgPDanhLtAuCxXqknIUlJHvJE1vTAoJVkARLQVXa+HytMSjlvhVaa1sD0XwsRScvtq+zM/fC2XMfnVwEofc8l6F8SNLCC8GIAw0JOiFjUdYB4hmak1CDdT5x+BrGSrVP7uw4Dt0DnScEK57qdK9HOKNG9zxzhp1e8+HCDr2FdbruyUGJy6tEgox+T5DQz1p7nLeFpij7W5UfTFSXOvkfyQ0eh6LzvTkQ4xIGSBIjU7xW9D4oIYjSkIoOXVqt6FNTdFA3NKyiX8ybnpaiY7DlrQtRUqfoQ+aAa0VXo+gYAUX3BlT0Hs3nddjcHTabopJrZWEzcQMBbDpulhE3m6Iaa1Vxc8W16Gm2CybORmIBvaFsmkU6cP4Ams6HZBgP6E8jUyv6SYHzsIo+5EqxVvSBAudhFV0zZL1QAoEzNgcMnJFmyKav6CBwxnjAwBldjCHzHRcHvRQ9RETV3QvP6Dafj48swRIzYnnE8nV9SJIsTF6jLE1WWuNVaDxyMZQlOG0gNpXIV3rNlp2Cm81P9qYFUROzZTI6FCKk2TIp+mfzqXqmI9A+ZWyZqdmynjDxjpHASCrzrU3Nln0ATedzObE5oG/N9ForejtKfK3FwIqu2bLpK7rD7wU0rKJrtqwXSnx9GFF0iJI6Rdds2fQV3XWAorsDKvrFAudpsWWOARgOLMxIUMWWmTBU1mzZNDXeQYB5xYIcUnVsGYbxOqBdjNcofNN49rMNnBNgCfrDIt9VhSaM60GhpEazP5eNEeCyh8UT+t6aFJVBiiKBb66OFMUi51yHUN2kKHIGDKGwaLVYh1AT03SebkPugCEUvth+adNSdECKDqvookBXK/q0FB2QosMquq6m74USIEWRA1FSpuiWaDFaK/q0FB2Qooj1Bx9C0S3RSvkgil490f5tpPo2C9ynDzlKwmZ+02rXgEEzFkZb/F5jp2FmAsw+1VnMQeAbfgNbxagCPXRZW64GFSKw5XLKKS24lH27FOGpW38dSMUGVLMrSPJ1BasWciwpJEGiZL2lXSYJFLmAypqsA1XJjxLine truncated

docs_raw/source/selecting_an_algorithm.rst

@@ -198,6 +198,14 @@ The algorithms are ordered by their release date in descending order.
                improve stability it also employs bias correction and trust region optimization techniques.
             </span>
          </div>
+         <div class="algorithm continuous off-policy" data-year="201808">
+            <span class="badge">
+               <a href="components/agents/policy_optimization/sac.html">SAC</a>
+               <br>
+               Soft Actor-Critic is an algorithm which optimizes a stochastic policy in an off-policy way.
+               One of the key features of SAC is that it solves a maximum entropy reinforcement learning problem.
+            </span>
+         </div>
          <div class="algorithm continuous off-policy" data-year="201509">
             <span class="badge">
                <a href="components/agents/policy_optimization/ddpg.html">DDPG</a>