coach/docs_raw/source/components/agents/policy_optimization/acer.rst at c1d1fae3426d537b8444531987a87377a135bbc9

gryf/coach

mirror of https://github.com/gryf/coach.git synced 2025-12-17 19:20:19 +01:00

Files

guyk1971 74db141d5e SAC algorithm (#282 )

* SAC algorithm

* SAC - updates to agent (learn_from_batch), sac_head and sac_q_head to fix problem in gradient calculation. Now SAC agents is able to train.
gym_environment - fixing an error in access to gym.spaces

* Soft Actor Critic - code cleanup

* code cleanup

* V-head initialization fix

* SAC benchmarks

* SAC Documentation

* typo fix

* documentation fixes

* documentation and version update

* README typo

2019-05-01 18:37:49 +03:00

2.5 KiB

Raw Blame History

Actions space: Discrete

References: Sample Efficient Actor-Critic with Experience Replay

Network Structure

Algorithm Description

Choosing an action - Discrete actions

The policy network is used in order to predict action probabilites. While training, a sample is taken from a categorical distribution assigned with these probabilities. When testing, the action with the highest probability is used.

Training the network

Each iteration perform one on-policy update with a batch of the last T_max transitions, and n (replay ratio) off-policy updates from batches of T_max transitions sampled from the replay buffer.

Each update perform the following procedure:

Calculate state values:

V(s_t) = 𝔼_a ∼ π[Q(s_t, a)]
Calculate Q retrace:

Q^ret(s_t, a_t) = r_t + γ^‒ρ_t + 1[Q^ret(s_t + 1, a_t + 1) − Q(s_t + 1, a_t + 1)] + γV(s_t + 1)

where ^‒ρ_t = min{c, ρ_t}, ρ_t = (π(a_t ∣ s_t))/(μ(a_t ∣ s_t))
Accumulate gradients:

• Policy gradients (with bias correction):

ĝ^policy_t = ^‒ρ_t∇logπ(a_t ∣ s_t)[Q^ret(s_t, a_t) − V(s_t)] + 𝔼_a ∼ π⎛⎝⎡⎣(ρ_t(a) − c)/(ρ_t(a))⎤⎦∇logπ(a ∣ s_t)[Q(s_t, a) − V(s_t)]⎞⎠

• Q-Head gradients (MSE):

ĝ^Q_t = (Q^ret(s_t, a_t) − Q(s_t, a_t))∇Q(s_t, a_t)
(Optional) Trust region update: change the policy loss gradient w.r.t network output:

ĝ^{trust − region}_t = ĝ^policy_t − max⎧⎩0, (k^Tĝ^policy_t − δ)/(‖k‖²₂)⎫⎭k

where k = ∇D_KL[π_avg ∥ π]

The average policy network is an exponential moving average of the parameters of the network (θ_avg = αθ_avg + (1 − α)θ). The goal of the trust region update is to the difference between the updated policy and the average policy to ensure stability.

System Message: ERROR/3 (<string>, line 61)

Unknown directive type "autoclass".

.. autoclass:: rl_coach.agents.acer_agent.ACERAlgorithmParameters