6d40ad1650
* updating the documentation website * adding the built docs * update of api docstrings across coach and tutorials 0-2 * added some missing api documentation * New Sphinx based documentation
2.2 KiB
Actions space: Continuous
References: Continuous control with deep reinforcement learning
Network Structure
Algorithm Description
Choosing an action
Pass the current states through the actor network, and get an action mean vector μ. While in training phase, use a continuous exploration policy, such as the Ornstein-Uhlenbeck process, to add exploration noise to the action. When testing, use the mean vector μ as-is.
Training the network
Start by sampling a batch of transitions from the experience replay.
To train the critic network, use the following targets:
yt = r(st, at) + γ⋅Q(st + 1, μ(st + 1))
First run the actor target network, using the next states as the inputs, and get μ(st + 1). Next, run the critic target network using the next states and μ(st + 1), and use the output to calculate yt according to the equation above. To train the network, use the current states and actions as the inputs, and yt as the targets.
To train the actor network, use the following equation:
∇θμJ ≈ Estρβ[∇aQ(s, a)|s = st, a = μ(st)⋅∇θμμ(s)|s = st]
Use the actor's online network to get the action mean values using the current states as the inputs. Then, use the critic online network in order to get the gradients of the critic output with respect to the action mean values ∇aQ(s, a)|s = st, a = μ(st). Using the chain rule, calculate the gradients of the actor's output, with respect to the actor weights, given ∇aQ(s, a). Finally, apply those gradients to the actor network.
After every training step, do a soft update of the critic and actor target networks' weights from the online networks.