coach/docs_raw/source/components/agents/policy_optimization/acer.rst at 2b5d1dabe6390d66893fdb58af1744cdded0d365

gryf/coach

mirror of https://github.com/gryf/coach.git synced 2025-12-18 03:30:19 +01:00

Files

shadiendrawis 2b5d1dabe6 ACER algorithm (#184 )

* initial ACER commit

* Code cleanup + several fixes

* Q-retrace bug fix + small clean-ups

* added documentation for acer

* ACER benchmarks

* update benchmarks table

* Add nightly running of golden and trace tests. (#202)

Resolves #200

* comment out nightly trace tests until values reset.

* remove redundant observe ignore (#168)

* ensure nightly test env containers exist. (#205)

Also bump integration test timeout

* wxPython removal (#207)

Replacing wxPython with Python's Tkinter.
Also removing the option to choose multiple files as it is unused and causes errors, and fixing the load file/directory spinner.

* Create CONTRIBUTING.md (#210)

* Create CONTRIBUTING.md.  Resolves #188

* run nightly golden tests sequentially. (#217)

Should reduce resource requirements and potential CPU contention but increases
overall execution time.

* tests: added new setup configuration + test args (#211)

- added utils for future tests and conftest
- added test args

* new docs build

* golden test update

2019-02-20 23:52:34 +02:00

2.5 KiB

Raw Blame History

Actions space: Discrete

References: Sample Efficient Actor-Critic with Experience Replay

Network Structure

Algorithm Description

Choosing an action - Discrete actions

The policy network is used in order to predict action probabilites. While training, a sample is taken from a categorical distribution assigned with these probabilities. When testing, the action with the highest probability is used.

Training the network

Each iteration perform one on-policy update with a batch of the last T_max transitions, and n (replay ratio) off-policy updates from batches of T_max transitions sampled from the replay buffer.

Each update perform the following procedure:

Calculate state values:

V(s_t) = 𝔼_a ∼ π[Q(s_t, a)]
Calculate Q retrace:

Q^ret(s_t, a_t) = r_t + γ^‒ρ_t + 1[Q^ret(s_t + 1, a_t + 1) − Q(s_t + 1, a_t + 1)] + γV(s_t + 1)

where ^‒ρ_t = min{c, ρ_t}, ρ_t = (π(a_t ∣ s_t))/(μ(a_t ∣ s_t))
Accumulate gradients:

• Policy gradients (with bias correction):

ĝ^policy_t = ^‒ρ_t∇logπ(a_t ∣ s_t)[Q^ret(s_t, a_t) − V(s_t)] + 𝔼_a ∼ π⎛⎝⎡⎣(ρ_t(a) − c)/(ρ_t(a))⎤⎦∇logπ(a ∣ s_t)[Q(s_t, a) − V(s_t)]⎞⎠

• Q-Head gradients (MSE):

ĝ^Q_t = (Q^ret(s_t, a_t) − Q(s_t, a_t))∇Q(s_t, a_t)
(Optional) Trust region update: change the policy loss gradient w.r.t network output:

ĝ^{trust − region}_t = ĝ^policy_t − max⎧⎩0, (k^Tĝ^policy_t − δ)/(‖k‖²₂)⎫⎭k

where k = ∇D_KL[π_avg ∥ π]

The average policy network is an exponential moving average of the parameters of the network (θ_avg = αθ_avg + (1 − α)θ). The goal of the trust region update is to the difference between the updated policy and the average policy to ensure stability.

System Message: ERROR/3 (<string>, line 60)

Unknown directive type "autoclass".

.. autoclass:: rl_coach.agents.acer_agent.ACERAlgorithmParameters