moving the docs to github

docs_raw/docs/algorithms/imitation/bc.md

@@ -0,0 +1,25 @@
+# Behavioral Cloning
+
+**Actions space:** Discrete|Continuous
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\dqn.png">
+
+</p>
+
+
+
+## Algorithm Description
+
+### Training the network
+
+The replay buffer contains the expert demonstrations for the task.
+These demonstrations are given as state, action tuples, and with no reward.
+The training goal is to reduce the difference between the actions predicted by the network and the actions taken by the expert for each state.
+
+1. Sample a batch of transitions from the replay buffer.
+2. Use the current states as input to the network, and the expert actions as the targets of the network.
+3. The loss function for the network is MSE, and therefore we use the Q head to minimize this loss.

docs_raw/docs/algorithms/other/dfp.md

@@ -0,0 +1,25 @@
+# Direct Future Prediction
+
+**Actions space:** Discrete
+
+**References:** [Learning to Act by Predicting the Future](https://arxiv.org/abs/1611.01779)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="../../design_imgs/dfp.png" width=600>
+
+</p>
+
+## Algorithm Description
+### Choosing an action
+
+1. The current states (observations and measurements) and the corresponding goal vector are passed as an input to the network. The output of the network is the predicted future measurements for time-steps $t+1,t+2,t+4,t+8,t+16$ and $t+32$ for each possible action. 
+2. For each action, the measurements of each predicted time-step are multiplied by the goal vector, and the result is a single vector of future values for each action. 
+3. Then, a weighted sum of the future values of each action is calculated, and the result is a single value for each action. 
+4. The action values are passed to the exploration policy to decide on the action to use.
+
+### Training the network
+
+Given a batch of transitions, run them through the network to get the current predictions of the future measurements per action, and set them as the initial targets for training the network. For each transition $(s_t,a_t,r_t,s_{t+1} )$ in the batch, the target of the network for the action that was taken, is the actual measurements that were seen in time-steps $t+1,t+2,t+4,t+8,t+16$ and $t+32$. For the actions that were not taken, the targets are the current values.

docs_raw/docs/algorithms/policy_optimization/ac.md

@@ -0,0 +1,27 @@
+# Actor-Critic
+
+**Actions space:** Discrete|Continuous
+
+**References:** [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783)
+
+## Network Structure 
+<p style="text-align: center;">
+<img src="..\..\design_imgs\ac.png" width=500>
+</p>
+## Algorithm Description
+
+### Choosing an action - Discrete actions
+
+The policy network is used in order to predict action probabilites. While training, a sample is taken from a categorical distribution assigned with these probabilities. When testing, the action with the highest probability is used.
+
+### Training the network
+A batch of $ T_{max} $ transitions is used, and the advantages are calculated upon it.
+
+Advantages can be calculated by either of the following methods (configured by the selected preset) -
+
+1. **A_VALUE** - Estimating advantage directly:$$ A(s_t, a_t) = \underbrace{\sum_{i=t}^{i=t + k - 1} \gamma^{i-t}r_i +\gamma^{k} V(s_{t+k})}_{Q(s_t, a_t)} - V(s_t) $$where $k$ is $T_{max} - State\_Index$ for each state in the batch.
+2. **GAE** - By following the [Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438) paper. 
+
+The advantages are then used in order to accumulate gradients according to 
+$$ L = -\mathop{\mathbb{E}} [log (\pi) \cdot A] $$
+

docs_raw/docs/algorithms/policy_optimization/cppo.md

@@ -0,0 +1,28 @@
+# Clipped Proximal Policy Optimization
+
+**Actions space:** Discrete|Continuous
+
+**References:** [Proximal Policy Optimization Algorithms](https://arxiv.org/pdf/1707.06347.pdf)
+
+## Network Structure
+
+<p style="text-align: center;">
+<img src="..\..\design_imgs\ppo.png">
+</p>
+
+
+## Algorithm Description
+### Choosing an action - Continuous action
+Same as in PPO. 
+### Training the network
+Very similar to PPO, with several small (but very simplifying) changes:
+
+1. Train both the value and policy networks, simultaneously, by defining a single loss function, which is the sum of each of the networks loss functions. Then, back propagate gradients only once from this unified loss function.
+
+2. The unified network's optimizer is set to Adam (instead of L-BFGS for the value network as in PPO). 
+
+3. Value targets are now also calculated based on the GAE advantages. In this method, the $ V $ values are predicted from the critic network, and then added to the GAE based advantages, in order to get a $ Q $ value for each action. Now, since our critic network is predicting a $ V $ value for each state, setting the $ Q $ calculated action-values as a target, will on average serve as a $ V $ state-value target.  
+
+4. Instead of adapting the penalizing KL divergence coefficient used in PPO, the likelihood ratio $r_t(\theta) =\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}}(a|s)}$ is clipped, to achieve a similar effect. This is done by defining the policy's loss function to be the minimum between the standard surrogate loss and an epsilon clipped surrogate loss:
+
+$$L^{CLIP}(\theta)=E_{t}[min(r_t(\theta)\cdot \hat{A}_t, clip(r_t(\theta), 1-\epsilon, 1+\epsilon) \cdot \hat{A}_t)]  $$

docs_raw/docs/algorithms/policy_optimization/ddpg.md

@@ -0,0 +1,32 @@
+# Deep Deterministic Policy Gradient
+
+**Actions space:** Continuous
+
+**References:** [Continuous control with deep reinforcement learning](https://arxiv.org/abs/1509.02971)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\ddpg.png">
+
+</p>
+
+## Algorithm Description
+### Choosing an action
+Pass the current states through the actor network, and get an action mean vector $ \mu $. While in training phase, use a continuous exploration policy, such as the Ornstein-Uhlenbeck process, to add exploration noise to the action. When testing, use the mean vector $\mu$ as-is.
+### Training the network
+Start by sampling a batch of transitions from the experience replay.
+
+* To train the **critic network**, use the following targets:
+
+  $$ y_t=r(s_t,a_t )+\gamma \cdot Q(s_{t+1},\mu(s_{t+1} )) $$
+  First run the actor target network, using the next states as the inputs, and get $ \mu (s_{t+1} ) $. Next, run the critic target network using the next states and $ \mu (s_{t+1} ) $, and use the output to calculate $ y_t $ according to the equation above. To train the network, use the current states and actions as the inputs, and $y_t$ as the targets.
+
+* To train the **actor network**, use the following equation:
+
+  $$ \nabla_{\theta^\mu } J \approx E_{s_t \tilde{} \rho^\beta } [\nabla_a Q(s,a)|_{s=s_t,a=\mu (s_t ) } \cdot \nabla_{\theta^\mu} \mu(s)|_{s=s_t} ] $$
+  Use the actor's online network to get the action mean values using the current states as the inputs. Then, use the critic online network in order to get the gradients of the critic output with respect to the action mean values $ \nabla _a Q(s,a)|_{s=s_t,a=\mu(s_t ) } $. Using the chain rule, calculate the gradients of the actor's output, with respect to the actor weights, given $ \nabla_a Q(s,a) $. Finally, apply those gradients to the actor network.
+
+After every training step, do a soft update of the critic and actor target networks' weights from the online networks.
+

docs_raw/docs/algorithms/policy_optimization/pg.md

@@ -0,0 +1,27 @@
+# Policy Gradient
+
+**Actions space:** Discrete|Continuous
+
+**References:** [Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning](http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\pg.png">
+
+</p>
+
+## Algorithm Description
+### Choosing an action - Discrete actions
+Run the current states through the network and get a policy distribution over the actions. While training, sample from the policy distribution. When testing, take the action with the highest probability. 
+
+### Training the network
+The policy head loss is defined as $ L=-log (\pi) \cdot  PolicyGradientRescaler $. The $PolicyGradientRescaler$ is used in order to reduce the policy gradient variance, which might be very noisy. This is done in order to reduce the variance of the updates, since noisy gradient updates might destabilize the policy's convergence. The rescaler is a configurable parameter and there are few options to choose from:	
+* **Total Episode Return** - The sum of all the discounted rewards during the episode.
+* **Future Return** - Return from each transition until the end of the episode.
+* **Future Return Normalized by Episode** - Future returns across the episode normalized by the episode's mean and standard deviation.
+* **Future Return Normalized by Timestep** - Future returns normalized using running means and standard deviations, which are calculated seperately for each timestep, across different episodes. 
+
+Gradients are accumulated over a number of full played episodes. The gradients accumulation over several episodes serves the same purpose - reducing the update variance. After accumulating gradients for several episodes, the gradients are then applied to the network. 
+

docs_raw/docs/algorithms/policy_optimization/ppo.md

@@ -0,0 +1,24 @@
+# Proximal Policy Optimization
+
+**Actions space:** Discrete|Continuous
+
+**References:** [Proximal Policy Optimization Algorithms](https://arxiv.org/pdf/1707.06347.pdf)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\ppo.png">
+
+</p>
+
+
+## Algorithm Description
+### Choosing an action - Continuous actions
+Run the observation through the policy network, and get the mean and standard deviation vectors for this observation. While in training phase, sample from a multi-dimensional Gaussian distribution with these mean and standard deviation values. When testing, just take the mean values predicted by the network. 
+### Training the network
+1. Collect a big chunk of experience (in the order of thousands of transitions, sampled from multiple episodes).
+2. Calculate the advantages for each transition, using the *Generalized Advantage Estimation* method (Schulman '2015).  
+3. Run a single training iteration of the value network using an L-BFGS optimizer. Unlike first order optimizers, the L-BFGS optimizer runs on the entire dataset at once, without batching. It continues running until some low loss threshold is reached. To prevent overfitting to the current dataset, the value targets are updated in a soft manner, using an Exponentially Weighted Moving Average, based on the total discounted returns of each state in each episode.
+4. Run several training iterations of the policy network. This is done by using the previously calculated advantages as targets. The loss function penalizes policies that deviate too far from the old policy (the policy that was used *before* starting to run the current set of training iterations) using a regularization term. 
+5. After training is done, the last sampled KL divergence value will be compared with the *target KL divergence* value, in order to adapt the penalty coefficient used in the policy loss. If the KL divergence went too high, increase the penalty, if it went too low, reduce it. Otherwise, leave it unchanged. 

docs_raw/docs/algorithms/value_optimization/bs_dqn.md

@@ -0,0 +1,30 @@
+# Bootstrapped DQN
+
+**Actions space:** Discrete
+
+**References:** [Deep Exploration via Bootstrapped DQN](https://arxiv.org/abs/1602.04621)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\bs_dqn.png">
+
+</p>
+
+## Algorithm Description
+### Choosing an action
+The current states are used as the input to the network. The network contains several $Q$ heads, which  are used for returning different estimations of the action $ Q $ values. For each episode, the bootstrapped exploration policy selects a single head to play with during the episode. According to the selected head, only the relevant output $ Q $ values are used. Using those $ Q $ values, the exploration policy then selects the action for acting.
+
+### Storing the transitions
+For each transition, a Binomial mask is generated according to a predefined probability, and the number of output heads. The mask is a binary vector where each element holds a 0 for heads that shouldn't train on the specific transition, and 1 for heads that should use the transition for training. The mask is stored as part of the transition info in the replay buffer. 
+
+### Training the network
+First, sample a batch of transitions from the replay buffer. Run the current states through the network and get the current $ Q $ value predictions for all the heads and all the actions. For each transition in the batch, and for each output head, if the transition mask is 1 - change the targets of the played action to $y_t$, according to the standard DQN update rule:
+
+$$ y_t=r(s_t,a_t )+\gamma\cdot max_a Q(s_{t+1},a) $$
+
+Otherwise, leave it intact so that the transition does not affect the learning of this head. Then, train the online network according to the calculated targets.
+
+As in DQN, once in every few thousand steps, copy the weights from the online network to the target network.
+

docs_raw/docs/algorithms/value_optimization/categorical_dqn.md

@@ -0,0 +1,33 @@
+# Categorical DQN
+
+**Actions space:** Discrete
+
+**References:** [A Distributional Perspective on Reinforcement Learning](https://arxiv.org/abs/1707.06887)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\distributional_dqn.png">
+
+</p>
+
+
+
+## Algorithm Description
+
+### Training the network
+
+1. Sample a batch of transitions from the replay buffer. 
+2. The Bellman update is projected to the set of atoms representing the $ Q $ values distribution, such that the $i-th$ component of the projected update is calculated as follows:
+   $$ (\Phi \hat{T} Z_{\theta}(s_t,a_t))_i=\sum_{j=0}^{N-1}\Big[1-\frac{|[\hat{T}_{z_{j}}]^{V_{MAX}}_{V_{MIN}}-z_i|}{\Delta z}\Big]^1_0 \ p_j(s_{t+1}, \pi(s_{t+1})) $$
+   where:
+   	*  $[ \cdot ] $ bounds its argument in the range [a, b]
+   	*  $\hat{T}_{z_{j}}$ is the Bellman update for atom $z_j$: &nbsp; &nbsp;   $\hat{T}_{z_{j}} := r+\gamma z_j$
+
+
+3. Network is trained with the cross entropy loss between the resulting probability distribution and the target probability distribution.   Only the target of the actions that were actually taken is updated. 
+4. Once in every few thousand steps, weights are copied from the online network to the target network.
+
+
+

docs_raw/docs/algorithms/value_optimization/distributional_dqn.md

@@ -0,0 +1,33 @@
+# Distributional DQN
+
+**Actions space:** Discrete
+
+**References:** [A Distributional Perspective on Reinforcement Learning](https://arxiv.org/abs/1707.06887)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\distributional_dqn.png">
+
+</p>
+
+
+
+## Algorithmic Description
+
+### Training the network
+
+1. Sample a batch of transitions from the replay buffer. 
+2. The Bellman update is projected to the set of atoms representing the $ Q $ values distribution, such that the $i-th$ component of the projected update is calculated as follows:
+   $$ (\Phi \hat{T} Z_{\theta}(s_t,a_t))_i=\sum_{j=0}^{N-1}\Big[1-\frac{|[\hat{T}_{z_{j}}]^{V_{MAX}}_{V_{MIN}}-z_i|}{\Delta z}\Big]^1_0 \ p_j(s_{t+1}, \pi(s_{t+1})) $$
+   where:
+   	*  $[ \cdot ] $ bounds its argument in the range [a, b]
+   	*  $\hat{T}_{z_{j}}$ is the Bellman update for atom $z_j$: &nbsp; &nbsp;   $\hat{T}_{z_{j}} := r+\gamma z_j$
+
+
+3. Network is trained with the cross entropy loss between the resulting probability distribution and the target probability distribution.   Only the target of the actions that were actually taken is updated. 
+4. Once in every few thousand steps, weights are copied from the online network to the target network.
+
+
+

docs_raw/docs/algorithms/value_optimization/double_dqn.md

@@ -0,0 +1,28 @@
+# Double DQN
+
+**Actions space:** Discrete
+
+**References:** [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461.pdf)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\dqn.png">
+
+</p>
+
+
+
+## Algorithm Description
+
+### Training the network
+1. Sample a batch of transitions from the replay buffer. 
+2. Using the next states from the sampled batch, run the online network in order to find the $Q$ maximizing action $argmax_a Q(s_{t+1},a)$. For these actions, use the corresponding next states and run the target network to calculate $Q(s_{t+1},argmax_a Q(s_{t+1},a))$.
+3. In order to zero out the updates for the actions that were not played (resulting from zeroing the MSE loss), use the current states from the sampled batch, and run the online network to get the current Q values predictions. Set those values as the targets for the actions that were not actually played. 
+4. For each action that was played, use the following equation for calculating the targets of the network:
+   $$ y_t=r(s_t,a_t )+\gamma \cdot Q(s_{t+1},argmax_a Q(s_{t+1},a)) $$
+
+
+5. Finally, train the online network using the current states as inputs, and with the aforementioned targets. 
+6. Once in every few thousand steps, copy the weights from the online network to the target network.

docs_raw/docs/algorithms/value_optimization/dqn.md

@@ -0,0 +1,28 @@
+# Deep Q Networks
+
+**Actions space:** Discrete
+
+**References:** [Playing Atari with Deep Reinforcement Learning](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\dqn.png">
+
+</p>
+
+
+
+## Algorithm Description
+
+### Training the network
+
+1. Sample a batch of transitions from the replay buffer. 
+2. Using the next states from the sampled batch, run the target network to calculate the $ Q $ values for each of the actions $ Q(s_{t+1},a) $, and keep only the maximum value for each state. 
+3. In order to zero out the updates for the actions that were not played (resulting from zeroing the MSE loss), use the current states from the sampled batch, and run the online network to get the current Q values predictions. Set those values as the targets for the actions that were not actually played. 
+4. For each action that was played, use the following equation for calculating the targets of the network:​                                                         $$ y_t=r(s_t,a_t)+γ\cdot max_a {Q(s_{t+1},a)} $$ 
+
+
+5. Finally, train the online network using the current states as inputs, and with the aforementioned targets. 
+6. Once in every few thousand steps, copy the weights from the online network to the target network.

docs_raw/docs/algorithms/value_optimization/dueling_dqn.md

@@ -0,0 +1,21 @@
+# Dueling DQN
+
+**Actions space:** Discrete
+
+**References:** [Dueling Network Architectures for Deep Reinforcement Learning](https://arxiv.org/abs/1511.06581)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\dueling_dqn.png">
+
+</p>
+
+## General Description
+Dueling DQN presents a change in the network structure comparing to DQN.
+
+Dueling DQN uses a specialized _Dueling Q Head_ in order to separate $ Q $ to an $ A $ (advantage) stream and a $ V $ stream. Adding this type of structure to the network head allows the network to better differentiate actions from one another, and significantly improves the learning.
+
+In many states, the values of the different actions are very similar, and it is less important which action to take.
+This is especially important in environments where there are many actions to choose from. In DQN, on each training iteration, for each of the states in the batch, we update the $Q$ values only for the specific actions taken in those states. This results in slower learning as we do not learn the $Q$ values for actions that were not taken yet. On dueling architecture, on the other hand, learning is faster - as we start learning the state-value even if only a single action has been taken at this state.

docs_raw/docs/algorithms/value_optimization/mmc.md

@@ -0,0 +1,32 @@
+# Mixed Monte Carlo
+
+**Actions space:** Discrete
+
+**References:** [Count-Based Exploration with Neural Density Models](https://arxiv.org/abs/1703.01310)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="../../design_imgs/dqn.png">
+
+</p>
+
+## Algorithm Description
+### Training the network
+In MMC, targets are calculated as a mixture between Double DQN targets and full Monte Carlo samples (total discounted returns).
+
+The DDQN targets are calculated in the same manner as in the DDQN agent:
+
+$$ y_t^{DDQN}=r(s_t,a_t )+\gamma Q(s_{t+1},argmax_a Q(s_{t+1},a)) $$
+
+The Monte Carlo targets are calculated by summing up the discounted rewards across the entire episode:
+
+$$ y_t^{MC}=\sum_{j=0}^T\gamma^j r(s_{t+j},a_{t+j} ) $$
+
+A mixing ratio $\alpha$ is then used to get the final targets:
+
+$$ y_t=(1-\alpha)\cdot y_t^{DDQN}+\alpha \cdot y_t^{MC} $$ 
+
+Finally, the online network is trained using the current states as inputs, and the calculated targets.
+Once in every few thousand steps, copy the weights from the online network to the target network.

docs_raw/docs/algorithms/value_optimization/n_step.md

@@ -0,0 +1,30 @@
+# N-Step Q Learning
+
+**Actions space:** Discrete
+
+**References:** [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\dqn.png">
+
+</p>
+
+
+
+## Algorithm Description
+
+### Training the network
+
+The $N$-step Q learning algorithm works in similar manner to DQN except for the following changes:
+
+1. No replay buffer is used. Instead of sampling random batches of transitions, the network is trained every $N$ steps using the latest $N$ steps played by the agent.
+
+2. In order to stabilize the learning, multiple workers work together to update the network. This creates the same effect as uncorrelating the samples used for training.
+
+3. Instead of using single-step Q targets for the network, the rewards from $N$ consequent steps are accumulated to form the $N$-step Q targets, according to the following equation: 
+$$R(s_t, a_t) = \sum_{i=t}^{i=t + k - 1} \gamma^{i-t}r_i +\gamma^{k} V(s_{t+k})$$
+where $k$ is $T_{max} - State\_Index$ for each state in the batch
+

docs_raw/docs/algorithms/value_optimization/naf.md

@@ -0,0 +1,22 @@
+# Normalized Advantage Functions
+
+**Actions space:** Continuous
+
+**References:** [Continuous Deep Q-Learning with Model-based Acceleration](https://arxiv.org/abs/1603.00748.pdf)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\naf.png" width=600>
+
+</p>
+
+## Algorithm Description
+### Choosing an action
+The current state is used as an input to the network. The action mean $ \mu(s_t ) $ is extracted from the output head. It is then passed to the exploration policy which adds noise in order to encourage exploration.
+###Training the network
+The network is trained by using the following targets:
+$$ y_t=r(s_t,a_t )+\gamma\cdot V(s_{t+1}) $$
+Use the next states as the inputs to the target network and extract the $ V $ value, from within the head, to get $ V(s_{t+1} ) $. Then, update the online network using the current states and actions as inputs, and $ y_t $ as the targets.
+After every training step, use a soft update in order to copy the weights from the online network to the target network.

docs_raw/docs/algorithms/value_optimization/nec.md

@@ -0,0 +1,28 @@
+# Neural Episodic Control
+
+**Actions space:** Discrete
+
+**References:** [Neural Episodic Control](https://arxiv.org/abs/1703.01988)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="..\..\design_imgs\nec.png" width=500>
+
+</p>
+
+## Algorithm Description
+### Choosing an action
+1. Use the current state as an input to the online network and extract the state embedding, which is the intermediate output from the middleware. 
+2. For each possible action $a_i$, run the DND head using the state embedding and the selected action $a_i$ as inputs. The DND is queried and returns the $ P $ nearest neighbor keys and values. The keys and values are used to calculate and return the action $ Q $ value from the network. 
+3. Pass all the $ Q $ values to the exploration policy and choose an action accordingly. 
+4. Store the state embeddings and actions taken during the current episode in a small buffer $B$, in order to accumulate transitions until it is possible to calculate the total discounted returns over the entire episode.
+
+### Finalizing an episode
+For each step in the episode, the state embeddings and the taken actions are stored in the buffer $B$. When the episode is finished, the replay buffer calculates the $ N $-step total return of each transition in the buffer, bootstrapped using the maximum $Q$ value of the $N$-th transition. Those values are inserted along with the total return into the DND, and the buffer $B$ is reset.
+### Training the network
+Train the network only when the DND has enough entries for querying.
+
+To train the network, the current states are used as the inputs and the $N$-step returns are used as the targets. The $N$-step return used takes into account $ N $ consecutive steps, and bootstraps the last value from the network if necessary:
+$$ y_t=\sum_{j=0}^{N-1}\gamma^j r(s_{t+j},a_{t+j} ) +\gamma^N   max_a Q(s_{t+N},a) $$

docs_raw/docs/algorithms/value_optimization/pal.md

@@ -0,0 +1,32 @@
+# Persistent Advantage Learning
+
+**Actions space:** Discrete
+
+**References:** [Increasing the Action Gap: New Operators for Reinforcement Learning](https://arxiv.org/abs/1512.04860)
+
+## Network Structure
+
+<p style="text-align: center;">
+
+<img src="../../design_imgs/dqn.png">
+
+</p> 
+
+## Algorithm Description
+### Training the network
+1. Sample a batch of transitions from the replay buffer. 
+
+2. Start by calculating the initial target values in the same manner as they are calculated in DDQN
+   $$ y_t^{DDQN}=r(s_t,a_t )+\gamma Q(s_{t+1},argmax_a Q(s_{t+1},a)) $$
+3. The action gap $ V(s_t )-Q(s_t,a_t) $ should then be subtracted from each of the calculated targets. To calculate the action gap, run the target network using the current states and get the $ Q $ values for all the actions. Then estimate $ V $ as the maximum predicted $ Q $ value for the current state:
+   $$ V(s_t )=max_a Q(s_t,a) $$
+4. For _advantage learning (AL)_, reduce the action gap weighted by a predefined parameter $ \alpha $ from the targets $ y_t^{DDQN} $: 
+   $$ y_t=y_t^{DDQN}-\alpha \cdot (V(s_t )-Q(s_t,a_t )) $$
+5. For _persistent advantage learning (PAL)_, the target network is also used in order to calculate the action gap for the next state:
+   $$ V(s_{t+1} )-Q(s_{t+1},a_{t+1}) $$
+   where $ a_{t+1} $ is chosen by running the next states through the online network and choosing the action that has the highest predicted $ Q $ value. Finally, the targets will be defined as -
+   $$ y_t=y_t^{DDQN}-\alpha \cdot min(V(s_t )-Q(s_t,a_t ),V(s_{t+1} )-Q(s_{t+1},a_{t+1} )) $$
+6. Train the online network using the current states as inputs, and with the aforementioned targets.
+
+7. Once in every few thousand steps, copy the weights from the online network to the target network.
+