TD3 (#338)

docs_raw/source/algorithms.xml

@@ -0,0 +1 @@
+<mxfile modified="2019-06-13T11:04:47.252Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" etag="OGV5teY4xcR0Xj7nvQBA" version="10.7.7" type="device"><diagram id="Fja6IZyvrddIfr-74nt7" name="Page-1">7V1bk5s2FP41O9M+JIMQ18e9NduZbCdp0jR5ysgg26SAXMC73v76CnMVEphdkLGzbB5sDhJgne/7dCQdkQt4HezeRWizvicu9i9Uxd1dwJsLVQVAt+hHannKLIZuZ4ZV5Ll5ocrwyfsP50Ylt249F8dMwYQQP/E2rNEhYYidhLGhKCKPbLEl8dm7btAKc4ZPDvJ569+em6wzq6Urlf0Oe6t1cWeg5GcCVBTODfEaueSxZoK3F/A6IiTJvgW7a+ynjVe0S1bvt5az5YNFOEz6VPh+fW+gL8GXH+p7HP5156+j3dc3mp4/XPJU/GLs0gbID0mUrMmKhMi/raxXEdmGLk4vq9Cjqsx7QjbUCKjxB06Sp9ybaJsQalongZ+fxTsv+ZpWf6vnR99qZ252+ZX3B0/FQZhET7VK6eG3+rmq2v6oqBcnEfmn9F16j+wXpz+ztSVzU0y2kYM7mi8ttYckilY46SgItdLjlCqYBJg+I60YYR8l3gP7JCjH7Kosl1e9jCL0VCuwIV6YxLUrf0gNtEBOP6jl2MvJpxsNiDTLm1ZXefole4LiqPZTKtMeds+AYP6jH5C/zZuBg2QFuNR/j2svwZ82aO+ZRyo7LLiWnu9fE59E+7pwqaf/SiDUzhj7v7QGCZOaPftLaxSMTXGEIifHs8qhSi1R9YCjBO+6ccWjoKigs+4ChXQ8Vsqj5i5R1jXVgYXoiJBT894LnKNwzpj14Rn60FceBqrDIB+rHAHJIsbRA701CYdxkWEQT0DHwoulkE0NErsIW0uHYSEYiXRU9N7qDO0MnnW2zpNOlcY5wDX6zLn+nIM9OTe0Rx7kYyjo9AyfPu7Vgn5ZJfsmyQxp78Sgwfh3S4oTb+K9Py9pAaBvdtXJ4iq/h5ttWv02WGDXxVFxVfrU2YXZm1Fz7QHkUd9SF3Df9x6mvo4tV2Oob4zEfMVqMB9AnvpAhzz3LVnc146Fi3vPdX38SBtrRkQNETaDBxoBC/Cg8njQZOFBHzk4PgMfADYILrnG+EAQBJfkHd0JhgRSQhEpb24+vKMnL52EtrOq3GHkDmEndULScD/j8ZCEuOHa3IR8bxXSQ4c6jfYZ8Cp1qecg/zI/EezlowVvbCzCQG4EgABDacg2FNBUEyBEXsSm/ewR24jxmdkzPrOmjM/MVsr3ZbgtInirbiyiUgNwGGMxvZtF/9gG9EZkm9AQL87uGm6D78hJB24xX/6AltQLyutjXNNeKPwgQNDHLJfYcCQN+lRWQVRL0NGropkWXZaEzBOxQ0TF6ikq9pSiYo0fRwhV5jMK754VN7w+/utqM4aA8C2vAYZo3gfKkgB7loABEmD3lAAwaWBhcxrwlfe673ubuC26rvkOxZtszXPp7VIMXLE8HZ2IYzDPZId30OSHd1C4xCGLdXBm3QDWaT1ZZ05JOmBO6WNQ83Dl70M+ZjxcOXwKHwPjHKIr0DlNcziUskSh1GXbWOpPFK56DtM47L2aeRlz4nkZYxjvwSvXdtB3VJXNSh87u6UctOdw06wD2S2ws7yk7BZ+0JfNz1BbgNFrXF/XtIkX2IsLzylHKrDYcFyzBSlHotUWaEjzjtraj/deXVFEfblDQgclrRMjLx2EncGYy26k9QHeyaZglVsaAeHYOX9nsKxpaKwPBLGRcGkZyPOCiGg/uRcaCbGmIMMSiDIsAZCW8WHNMeqQbK++6V5Z8DdZvld7wtcJLSjOC4Rt05S6LVgcMARhkaz1QW0eyg6Sib7zlNCYVCba8/9mmThBmYAaO7SdXCaKCHKWiZfJhN5TJrRJp7qhKC10lomTlYlmNGEIZsCOKxPDtpi8epnouyCmK2JcHEkm2lfEZpk4QZloRhOTy4Q2RxPH2R0+dP/ni9bPnr07XIFd5eWsn2nGDMH5BQWTvqBAH7a7YxoIqqeDQa1vZqY+iQyqjSVQ07QPyGBneUkYPJtVgiNAaShCDkLAMhurPRl081qNoGqMTo7PCp4kUBcG5WXlkN9uBOZwvmux0WSFQrC7/Lij/mF7jKbpyMzT6ciKPJgT7chgMatTdkwHginV7CovqSMT5V7JkLrxZiAGPVeXaM4SCUEzOXB6kTybASevdSewu1vvm49hqFJCOQ2wotY3lHu22AJ2MsTSDoitAbrKSxJbCa8ZEqrcvRceELXXvuHUMliZsy3BdtNipuEo20314YMOAERg+DjOZYqrxNsSOPWes2amnyhIMREu4vQj+0UH8diZ/vvTpeFr7MyZJdh3KdztLG1vjioJfoEX/jIBBGkrKB8FFVRxhV9nfLKRoAobkaAIoeZREdqeZVG68fMNlWfly0X6EM1OrfH6pkWrL/n9gCM0KLAbgXUxyVTfjCdI+i/W/MZvzh6v0RK+le7YDWexEaQhyBHXRB21tA0xZ7P+cJIjkhzPR9rEKNh12Nhh1Xx35EgDkuZuyOJF9G0DEq3xXM8tf+TdlvSwelV+Vrz6Dwfg7f8=</diagram></mxfile>

docs_raw/source/components/agents/index.rst

@@ -21,6 +21,7 @@ A detailed description of those algorithms can be found by navigating to each of
    imitation/cil
    policy_optimization/cppo
    policy_optimization/ddpg
+   policy_optimization/td3
    policy_optimization/sac
    other/dfp
    value_optimization/double_dqn

docs_raw/source/components/agents/policy_optimization/td3.rst

@@ -0,0 +1,55 @@
+Twin Delayed Deep Deterministic Policy Gradient
+==================================
+
+**Actions space:** Continuous
+
+**References:** `Addressing Function Approximation Error in Actor-Critic Methods <https://arxiv.org/pdf/1802.09477>`_
+
+Network Structure
+-----------------
+
+.. image:: /_static/img/design_imgs/td3.png
+   :align: center
+
+Algorithm Description
+---------------------
+Choosing an action
+++++++++++++++++++
+
+Pass the current states through the actor network, and get an action mean vector :math:`\mu`.
+While in training phase, use a continuous exploration policy, such as a small zero-meaned gaussian noise,
+to add exploration noise to the action. When testing, use the mean vector :math:`\mu` as-is.
+
+Training the network
+++++++++++++++++++++
+
+Start by sampling a batch of transitions from the experience replay.
+
+* To train the two **critic networks**, use the following targets:
+
+  :math:`y_t=r(s_t,a_t )+\gamma \cdot \min_{i=1,2} Q_{i}(s_{t+1},\mu(s_{t+1} )+[\mathcal{N}(0,\,\sigma^{2})]^{MAX\_NOISE}_{MIN\_NOISE})`
+
+  First run the actor target network, using the next states as the inputs, and get :math:`\mu (s_{t+1} )`. Then, add a
+  clipped gaussian noise to these actions, and clip the resulting actions to the actions space.
+  Next, run the critic target networks using the next states and :math:`\mu (s_{t+1} )+[\mathcal{N}(0,\,\sigma^{2})]^{MAX\_NOISE}_{MIN\_NOISE}`,
+  and use the minimum between the two critic networks predictions in order to calculate :math:`y_t` according to the
+  equation above. To train the networks, use the current states and actions as the inputs, and :math:`y_t`
+  as the targets.
+
+* To train the **actor network**, use the following equation:
+
+  :math:`\nabla_{\theta^\mu } J \approx E_{s_t \tilde{} \rho^\beta } [\nabla_a Q_{1}(s,a)|_{s=s_t,a=\mu (s_t ) } \cdot \nabla_{\theta^\mu} \mu(s)|_{s=s_t} ]`
+
+  Use the actor's online network to get the action mean values using the current states as the inputs.
+  Then, use the first critic's online network in order to get the gradients of the critic output with respect to the
+  action mean values :math:`\nabla _a Q_{1}(s,a)|_{s=s_t,a=\mu(s_t ) }`.
+  Using the chain rule, calculate the gradients of the actor's output, with respect to the actor weights,
+  given :math:`\nabla_a Q(s,a)`. Finally, apply those gradients to the actor network.
+
+  The actor's training is done at a slower frequency than the critic's training, in order to allow the critic to better fit the
+  current policy, before exercising the critic in order to train the actor.
+  Following the same, delayed, actor's training cadence, do a soft update of the critic and actor target networks' weights
+  from the online networks.
+
+
+.. autoclass:: rl_coach.agents.td3_agent.TD3AlgorithmParameters

docs_raw/source/diagrams.xml

@@ -1 +1 @@
-<mxfile modified="2019-04-24T09:04:49.730Z" host="www.draw.io" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" etag="Oj1WgTFAGXXvHH9O1PX0" version="10.6.3" type="device"><diagram id="33c2a640-8c1e-935c-0e0a-86b5dd5c932c" name="Page-1">7V1bd6M6sv41eUwWQlwfc+l073O6e7J3nzln5uksbBOH2Rg8GCed/etHwgiDSgZsSxjSyszqbcvmYr6qUtWnqtIVvl/9/JwF65dv6SKMr0xj8fMKP1yZJkK2Rf5DR953Iy4qB5ZZtCi/tB/4Ef0VloNGObqNFuGm8cU8TeM8WjcH52mShPO8MRZkWfrW/NpzGjevug6WIRj4MQ9iOPp/0SJ/2Y16trEf/xJGyxd2ZWSUn8yC+Z/LLN0m5fWuTPxc/O0+XgXsXOX3Ny/BIn2rDeFPV/g+S9N892r18z6M6bNlj2133OOBT6v7zsIk73UAxrtDXoN4G7J7Lu4sf2dPo/g9IT0CXeG7t5coD3+sgzn99I3gT8Ze8lVcfryMgw19+gZ5PU9X0bx8vcmz9M/wPo3TrDgrduZeOHuuPmHPGZOR5yiOa99cBKH3PKfjaZKXwmIa5fva94zij4wHcbRMyFgcPuf0bTYvj3LIO/iIyqf2GmZ5+LM2VD6yz2G6CvPsnXyFfWp5JX7vTG59bzfwthcXiwnFS01UsFM+36AU0WV19j1M5EWJlBg1x1MI2iLYvFTHtSL4yaH/64NgqQPkQlmwiMI9akmahDKAdeUAS/Bp4GoajgFxtQS4Osw2nIOrb2hcc0lIuk0NpTYaIIlFSCJLBpIIIBmQeWoYy+q7D4brSsYPmS34yQDMt25sDjLHv/Hw/s8GAFYaUwdQCn5mtyaGyeKW+hrk3ZxCQwGp41XN7fQJxcEsjO8q76D2EO+K/4nR6vn4yVPP3v9Br0QeYfn2nwfnXcN9vL1rAy0PsmWYNwU5XDQ8JghjDRVbAAoby8I4yKPXpp8lQqq8wlMakburpKSyvUxGKpeRnWOTbrN5WB5W93zAmWxe3rDBnWv3IMC5COrBe+1ra/qFzTE3jY3We7N998wDkN1+ALgl7gDyYvcr99pSYd1LgdweCqSnsp5TGeeU4L5TmckL9CmmEKl0NocHr9WQNgRLethgNnWuCidrMDqOKGbwZcxoGkUlKCK/J4oWP1OdpIzOLw+jpFAP8zOgQBsxFhlVGf4lnB23m5AMJGH+lmZ/FnwX/YVZuIho3GCQ6CFKE/JiEREAotl293as8cSAgQN2mkCKZkdlcQJkz/5e4Ji/0H/Dn2vyk4MSuXUaR/N3Bu0mWK1j+qXnLF1VR5SAEwg10t1IuwMibQGk51kY5BS0IKkraJQ8U3zT2b/CneYmFMydekd5BX+eZntBqQ7efTlIivBisYjocBDvzxrM0m0ODtOyAWTDwgPKBpyUK+0OaMxLUC10vIJ5m0TPabaK33n9r76xKQAz7+lYnCbL4peQH0I+IfdHrUP4HGzjxtRQykhxGxstFTVwL2EwYNi0NxgUbyIVm4jJwyYqIN4JAXn2OTMRi+IFszOL6jtJ+EYnmOSVyle4WacJNTBjhXwE9KLRJHt8a0BRYB5JBxtMfkrOUYiNZ8+eWu0hl0OMdJiTBxSS8Tv6YKJ5EN+WH6yixYJeRigEezEx+hItfdmQ4nvlLxQsTp6r0KbA1xMTIRJA1JRwY7YbCSOM7aZIIIfDujchbHAn4uN1SWwwf8Nd3C7yAE3Nie15TC3qkQHwS8i1+THlGvmc/PBs4oXk2jYUyzWM2LJtEYe/BEUwVoxRPyxKCu/LIT47kZVktllXIjP52blVmWTPzphfdDdcMD0jIacmY362e+H9Qnzw7VqjLcGlRj3QNlWh7QK05y9pummhRc6HtFzT+6jONc+hmIJwufKKpAMKF6jS2SbMXmFYO03dvGSc5AmQ9BQByc4xEX/yZ5TX3Eny7p/lPUj2NHfu2lU9gqo7nywPdyTeJx90WK59mvfJe4UOL2KSvE/+hm3U7n3y98V9/2zvs1JZrQWHtABNQA242Mni2cBT1cDFatSAD6ourgaQNOOWKz7E3D5slGX7XOaiI6BBmcGWPr1ruojBMCpTxWuyd2piLOZqW/yzs2IbonWk/YAkTsealzYgPQwIJyomNB+ipTAp5gOyNM1UiN3CJlvyjJLqg/k2K37z/sPXIIuCWazXPY9Y90SGAcFWtvBpQpYmWCwqSImByKKETg9Uod+CrPgorT6vr5FriPuyrvtq1DrEomVRKRD3SA//sP5APZxhlq2xpOSO2kdAxunxjHXj1P5cq+kzDLR2igyvo87Fbz/g7AAHf5A4X5Lwj1bQ3ZMFnTvTUMunnaLNL78cfwBoDtD5203JygNjSTLNUucb5EPyWbXNjLl0m6+LvFh2+Fi9hQt6fvaQGW8Yxm2/olkcGZ1pI27WbgqILBvpnE1unsEYYBhgcjaFJxA4WxIl0zAllw88eAvjeUNaGEECQI0hCNdXJXsQrqNNuigSQFjSx3JXmcvCJM0eHcce8YyC7QDYkSr6GcNws8kfNfLqa/TRKlyl5aW0OvfkEQTNb5TxCLhHgfKv4DF4o/IYLF4mzFPTUAcLpEDIf/Ewh6lHTbQXhXY/6vlHwvwjqtx2BSohY/qxLtYubzT194raKIgKNsV9FKRU88L4lIsS9n7EBoQI66xZq69DhaOXoWx8sd5eFowQiyKAD2KML5lniiyrC1ZVaadWj84oI/If1eWl2OaoPEjgXVmGUETOTVNBlt963iE5KEtQ+50mVKsOTC/kmkV3hyvaQTif0zJ/xnTrOaX/nCKWgCHmFKZ0ohYQcVSU86fPAPZG6wdNUBwNuOVeDnBIPw4E3WgCAUmNuPjMfeQIaCdhJy4ZbfFsSDvFYZAle+0sDbJ03/CDF5NV7HoFq2B1AJmKaGIHUi7b9WKXNlg6S7UATpvc3ibXvVxPZmdSqTbnloiVD2wsXrzlN+Xg9PRy/kw+LxtDeupsN4ZpyNRlihVLeEYdYxKj5DgeshzbwB7vUHhISsTJN4jZbw6hOhXMs9pXMGzn3AOsjn7hMIqX3C/cgaRcnKZ0Ef85paRssl1d1Zt1FEv8NIbKo5Wg2d00qbtB11GADDgCBl5VGYgjCp+cmGK2iF4bWDr/3tLdkO4opNclOrc0rKb+c/UpeVVmdezOslnTrpy7sTglKJTj5MbqH9WGi+uyUS1Ox4oT3zwCeaLG2IrEyYXuvt5upRUu03fB9heXc+1d6NqLl2Q0gm0IWublEFS5MP6L7/dhCyZmdVtXuXBpXNvSYzXR9i6nidCTDunrgvz6EH7NsDymANzO3AVVi9wuXOQu50nWOV63FT4NZC60x4KSeGWdC10XYEaJFfYDSzCaBrf+1GrI7hkit04RcSkHLuSIjuawgixnrFh5g8XYY0R/ZnlBAW9GBsuv0Nv9V5jn7yXgwTZPyVCa5S/pku568JXG/3WBPUCZtUrKmeQXm4rr5Jfbt6inN6vVu6U8DHOg5IyHrVSX48J8lJHwj5bFeWygAqr/roQeP9u4ijqmwZvu4AAdnv88+gBWYXb41/Oeb/OAs1lGb1rNiy5B97sidRvXYpRlc2JSbcd9tLrxZ3IUNS4Aks2qFHrrztEH4A7tBLeEJVP6HnRWo9U6S3VH16tTXFR+6wujk65TFYV4H6RUS6UVFfmN5risKC9Qzqn13y7f+R+cSZnL0mUUwW889gCzq8EMf0um5AYzvshlUbVSVjXHv+ha2UfPjBPsdD8k3eD36M0xodTVY9j5ffKq12bJj0heNfg6NgPDanhLtAuCxXqknIUlJHvJE1vTAoJVkARLQVXa+HytMSjlvhVaa1sD0XwsRScvtq+zM/fC2XMfnVwEofc8l6F8SNLCC8GIAw0JOiFjUdYB4hmak1CDdT5x+BrGSrVP7uw4Dt0DnScEK57qdK9HOKNG9zxzhp1e8+HCDr2FdbruyUGJy6tEgox+T5DQz1p7nLeFpij7W5UfTFSXOvkfyQ0eh6LzvTkQ4xIGSBIjU7xW9D4oIYjSkIoOXVqt6FNTdFA3NKyiX8ybnpaiY7DlrQtRUqfoQ+aAa0VXo+gYAUX3BlT0Hs3nddjcHTabopJrZWEzcQMBbDpulhE3m6Iaa1Vxc8W16Gm2CybORmIBvaFsmkU6cP4Ams6HZBgP6E8jUyv6SYHzsIo+5EqxVvSBAudhFV0zZL1QAoEzNgcMnJFmyKav6CBwxnjAwBldjCHzHRcHvRQ9RETV3QvP6Dafj48swRIzYnnE8nV9SJIsTF6jLE1WWuNVaDxyMZQlOG0gNpXIV3rNlp2Cm81P9qYFUROzZTI6FCKk2TIp+mfzqXqmI9A+ZWyZqdmynjDxjpHASCrzrU3Nln0ATedzObE5oG/N9ForejtKfK3FwIqu2bLpK7rD7wU0rKJrtqwXSnx9GFF0iJI6Rdds2fQV3XWAorsDKvrFAudpsWWOARgOLMxIUMWWmTBU1mzZNDXeQYB5xYIcUnVsGYbxOqBdjNcofNN49rMNnBNgCfrDIt9VhSaM60GhpEazP5eNEeCyh8UT+t6aFJVBiiKBb66OFMUi51yHUN2kKHIGDKGwaLVYh1AT03SebkPugCEUvth+adNSdECKDqvookBXK/q0FB2QosMquq6m74USIEWRA1FSpuiWaDFaK/q0FB2Qooj1Bx9C0S3RSvkgil490f5tpPo2C9ynDzlKwmZ+02rXgEEzFkZb/F5jp2FmAsw+1VnMQeAbfgNbxagCPXRZW64GFSKw5XLKKS24lH27FOGpW38dSMUGVLMrSPJ1BasWciwpJEGiZL2lXSYJFLmAypqsA1XJjxLine truncated
+<mxfile modified="2019-06-12T14:38:13.932Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" etag="KqQkpaiTzICBBUdtGW-V" version="10.7.7" type="device"><diagram id="33c2a640-8c1e-935c-0e0a-86b5dd5c932c" name="Page-1">7V1bd6M6sv41eUwWQlwfc+l07zPdPdm7Z87MnJeziE0cZmPwYJxL//qRMMKgkgHbEoa0MrN627K5mK+qVPWpqnSBb5dvn7Ng9fwtnYfxhWnM3y7w3YVpIux75D905H074tj+dmCRRfPyS7uBH9HPsBw0ytFNNA/XjS/maRrn0ao5OEuTJJzljbEgy9LX5tee0rh51VWwCMHAj1kQw9F/RPP8eTvq2cZu/EsYLZ7ZlZFRfvIYzP5cZOkmKa93YeKn4m/78TJg5yq/v34O5ulrbQh/usC3WZrm21fLt9swps+WPbbtcfd7Pq3uOwuTvNcBGG8PeQniTcjuubiz/J09jeL3hPQIdIFvXp+jPPyxCmb001eCPxl7zpdx+fEiDtb06Rvk9SxdRrPy9TrP0j/D2zROs+Ks2Jl54eNT9Ql7zpiMPEVxXPvmPAi9pxkdT5O8FBbTKN/XvmcUf2Q8iKNFQsbi8Cmnb7NZeZRD3sFHVD61lzDLw7faUPnIPofpMsyzd/IV9qnllfi9M7llEv+6ExeLCcVzTVSwUz7foBTRRXX2HUzkRYmUGDXHUwjaPFg/V8e1IvjJof/rg2CpA+RCWTCPwh1qSZqEMoB15QBL8GngahqOAXG1BLg6zDacgqtvaFxzSUi6TQ2lNhogiUVIIksGkgggGZB5ahjL6rt3hutKxg+ZLfjJAMy3rmwOMse/8vDuzwYAVhpTB1AKfma3JobJ/Jr6GuTdjEJDAanjVc3t9AnFwWMY31TeQe0h3hT/E6PV8/GTp569/5NeiTzC8u2/9s67hnt/fdMGWh5kizBvCnI4b3hMEMYaKrYAFDaWhXGQRy9NP0uEVHmFhzQid1dJSWV7mYxULiM7xzrdZLOwPKzu+YAz2by8YYM71/ZBgHMR1IP32tdW9AvrQ24aG633ZvvuiQcgu/0AcEvcAeTF9lfutKXCupcCuT0USE9lPacyzinBfacykxfoY0whUulsDg9eqyFtCJb0sMFs6lwVTtZgdBxRzODLmNE0ikpQRH5PFC1+pjpKGZ1fHkZJoR7mZ0CBNmIsMqoy/Es4O27WIRlIwvw1zf4s+C76C7NwHtG4wSDRQ5Qm5MU8IgBEj5vt27HGEwMGDthpAimaHZXFCZA9+3uBY/5M/w3fVuQnByVyqzSOZu8M2nWwXMX0S09ZuqyOKAEnEGqku5F2B0TaAkjPsjDIKWhBUlfQKHmi+KaP/w63mptQMLfqHeUV/Hma7QSlOnj75SApwov5PKLDQbw7a/CYbnJwmJYNIBsWHlA24KRcaXdAY16CaqHjFcybJHpKs2X8zut/9Y11AZh5S8fiNFkUv4T8EPIJuT9qHcKnYBM3poZSRorbWGupqIF7DoMBw6adwaB4E6lYR0we1lEB8VYIyLPPmYmYFy+YnZlX30nCVzrBJC9UvsL1Kk2ogRkr5COgF40m2eNbA4oC80g62GDyU3KOQmw8e/bUag+5HGKkw4w8oJCM39AHE82C+Lr8YBnN5/QyQiHYiYnRl2jpy4YU3yt/oWBx8lSFNgW+npgIkQCipoQbs91IGGFsN0UCORzWvQlhgzsRH69LYoP5G+7idpEHaGpObE9jalGPDIBfQq7NjynXyOfkh2cTzyTXtqFYrmHElm2KOPw5KIKxYoz6YVFSeF8O8dmJrCSP61UlMpOfnVuVSfbsjPlFd8MF0zMScmoy5me7F97PxAffrDTaElxq1ANtUxXaLkB79pym6xZa5HRIyzW9j+pc8xyKKQiXK69IOqBwgSp9XIfZCwxrp6mb54yTPAGSniIg2Tkm4k++RXnNnSTv/lXeg2RPc+uuXdQjqLrzyfJwR+J98kGH5drHeZ+8V+jwIibJ++Rv2Ebt3id/X9z3T/Y+K5XVWrBPC9AE1ICLnSyeDTxWDVysRg34oOrsagBJM2654kPM7cNGWbbPZS46AhqUGWzp07umixgMozJVvCZ7xybGYq62xT85K7YhWgfaD0jidKx5aQPSw4BwomJC8yFaCpNiPiBL00yF2C5ssiXPKKk+mG2y4jfvPnwJsih4jPW65wHrnsgwINjKFj5NyNIE83kFKTEQWZTQ6YEq9GuQFR+l1ef1NXINcV/WdVeNWodYtCwqBeIe6eEf1h+ohzPMsjWWlNxR+wjIOD6esa6c2p9rNX2GgdZOkeF11Ln47QecHODgDxLnSxL+0Qq6e7Sgc2caavm0U7T55ZfDDwDNATp/uylZeWAsSaZZ6nyDfEg+q7aZMZdu8lWRF8sOH6u3cEbPzx4y4w3DuO1XNIsjozNtxM3aTQGRZSOdk8nNExgDDANMzqbwBAJnS6JkGqbk/IEHb2E8b0gLI0gAqDEE4eqiZA/CVbRO50UCCEv6WGwrc1mYpNmjw9gjnlGwHQA7UkU/YxhuNvmjRl59jT5ahsu0vJRW5548gqD5jTIeAfcoUP4VPAZvVB6DxcuEeWwa6mCBFAj5zx7mMPWoifa80O57Pf9ImH9ElduuQCVkTD/W2drljab+XlEbBVHBpriPgpRqXhifclHCzo9YgxBhlTVr9XWocPAylI3P1tvLghFiUQTwQYzxOfNMkWV1waoq7dTq0RllRP6jurwU2xyVBwm8K8sQisipaSrI8lvPOyQHZQlqv9OEatWe6YVcs+jucEE7COczWubPmG49p/SfU8QSMMScwpRO1AIijopy/vQJwN5o/aAJioMBt9zzAQ7px4GgG00gIKkRF5+5jxwB7STsxCWjLZ4Naac4DLJkp52lQZbuG37wYrKKXa9gFawOIFMRTexAymWzmm/TBktnqRbAaZPb2+S65+vJ7Ewq1ebUErHygY3Fi7f8phwcn17On8nnZWNIT53txjANmTpPsWIJz6hjTGKUHMdDlmMb2OMdCg9JiTj5BjG7zSFUp4J5VvsKhu2ceoDV0S8cRvGS+4U7kJSL05Qu4j+llJRNNsuLerOOYomfxlB5tBQ0u5smdTfoOgqQAUfAwKsqA3FE4ZMTU8zm0UsDS+c/G7ob0g2F9LJE55qG1dR/rj4lr8qsju1Z1ivalXM7FqcEhXKc3Fj9o9pwcV02qsXpUHHim0cgT9QYW5E4udDd19uttMJl+i7Y/uJ8rr0LXXvxkoxGsA1ByzwfgioXxn/x/T5swcSsbusqFy6Na1t6qCba3vk0EXrSIX1dkF8fwq8ZlscUgNuZu6BqkduFi9zlPMk6x+u2wseBzIX2WFASr6xzoesCzCixwn5gCUbT4NafWg3ZHUPk1ikiLuXAhRzRwRxWkOWMFStvsBi7j+jPLC8o4M3IYPkVerv/DvP8vQQ82OQpGUqz/Dld0F0PvtL4vy6weyizVkk5kfxiU3Gd/HL7FvX0ZrV6t5SHYQ6UnPGwlepyXJiPMhL+0bI4jw1UQPXfldDjZxtXUcc0eNMdHKDD858HH8AqzPb/et7zbR5wMsvoTat50TnoflekbuNajLJsTkyq7bgPVjf+TI6ixgVAslmVQm/dOfgA3KGd4JawZErfg85qtFxlqe7oenGMi8pvfWF00nWqohDvg5RqqbSiIr/RHJcV5QXKObb+2+U7/4MzKXNZuowi+I2HHmB2NZjhb8mU3GDGF7ksqlbKqub4Z10r++iZcYKd7oekG/wevTkmlLp6CDu/S1712iz5AcmrBl/HZmBYDW+JdkGwWI+Uk7CEZC95YitaQLAMkmAhqEobn681BqXctUJrbWsgmo+l6OTZ9nV2Zl74+NRHJ+dB6D3NZCgfkrTwQjDiQEOCTshYlHWAeIbmKNRgnU8cvoSxUu2TOzuOQ/dA5wnBiqc63esRzqjRPc98xE6v+XBuh97cOl735KDE5VUiQUa/J0joZ609TttCU5T9rcoPJqpLnfyP5AaPQ9H53hyIcQkDJImRKV4reh+UEERpSEWHLq1W9KkpOqgbGlbRz+ZNT0vRMdjy1oUoqVP0IXPAtaKrUXSMgKJ7Ayp6j+bzOmzuDptNUcm1srCZuIEANh03y4ibTVGNtaq4ueJa9DTbBRNnI7GA3lA2zSIdOH8ATedDMowH9KeRqRX9qMB5WEUfcqVYK/pAgfOwiq4Zsl4ogcAZmwMGzkgzZNNXdBA4Yzxg4IzOxpD5jouDXooeIqLq7plndJvPx0eWYIkZsTxi+bo+JEkWJi9RliZLrfEqNB65GMoSnDYQm0rkK71my47BzeYne9OCqInZMhkdChHSbJkU/bP5VD3TEWifMrbM1GxZT5h4x0hgJJX51qZmyz6ApvO5nNgc0Ldmeq0VvR0lvtZiYEXXbNn0Fd3h9wIaVtE1W9YLJb4+jCg6REmdomu2bPqK7jpA0d0BFf1sgfO02DLHAAwHFmYkqGLLTBgqa7ZsmhrvIMC8YkEOqTq2DMN4HdAuxksUvmo8+9kGzgmwBP1hke+qQhPG9aBQUqPZn8vGCHDZw+IJfW9NisogRZHAN1dHimKRc65DqG5SFDkDhlBYtFqsQ6iJaTpPtyF3wBAKn22/tGkpOiBFh1V0UaCrFX1aig5I0WEVXVfT90IJkKLIgSgpU3RLtBitFX1aig5IUcT6gw+h6JZopXwQRa+eaP82Un2bBe7ShxwlYTO/abVrwKAZC6Mtfq+x4zAzAWaf6izmIPANv4GtYlSBHrqsLVeDChHYcjnllBZcyr5eiPDUrb/2pGIDqtkVJPm6glULOZYUkiBRstrQLpLine truncated

docs_raw/source/selecting_an_algorithm.rst

@@ -214,6 +214,16 @@ The algorithms are ordered by their release date in descending order.
                and therefore it is able to use a replay buffer in order to improve sample efficiency.
             </span>
          </div>
+         <div class="algorithm continuous off-policy" data-year="201509">
+            <span class="badge">
+               <a href="components/agents/policy_optimization/td3.html">TD3</a>
+               <br>
+               Very similar to DDPG, i.e. an actor-critic for continuous action spaces, that uses a replay buffer in
+               order to improve sample efficiency. TD3 uses two critic networks in order to mitigate the overestimation
+               in the Q state-action value prediction, slows down the actor updates in order to increase stability and
+               adds noise to actions while training the critic in order to smooth out the critic's predictions.
+            </span>
+         </div>
          <div class="algorithm continuous discrete on-policy" data-year="201706">
             <span class="badge">
                <a href="components/agents/policy_optimization/ppo.html">PPO</a>