# Exploration Policies

Exploration policies are a component that allow the agent to tradeoff exploration and exploitation according to a predefined policy. This is one of the most important aspects of reinforcement learning agents, and can require some tuning to get it right. Coach supports several pre-defined exploration policies, and it can be easily extended with custom policies. Note that not all exploration policies are expected to work for both discrete and continuous action spaces.

Exploration Policy | Discrete Action Space | Box Action Space
--- | --- | ---
AdditiveNoise | X | V
Boltzmann | V | X
Bootstrapped | V | X
Categorical | V | X
ContinuousEntropy | X | V
EGreedy | V | V
Greedy | V | V
OUProcess | X | V
ParameterNoise | V | V
TruncatedNormal | X | V
UCB | V | X

## ExplorationPolicy

class `rl_coach.exploration_policies.``ExplorationPolicy`(action_space: rl_coach.spaces.ActionSpace)

  An exploration policy takes the predicted actions or action values from the agent, and selects the action to actually apply to the environment using some predefined algorithm.

  Parameters: | action_space – the action space used by the environment

  `change_phase`(phase)

    Change between running phases of the algorithm :param phase: Either Heatup or Train :return: none

  `get_action`(action_values: List[Union[int, float, numpy.ndarray, List]]) → Union[int, float, numpy.ndarray, List]

    Given a list of values corresponding to each action, choose one actions according to the exploration policy :param action_values: A list of action values :return: The chosen action

  `requires_action_values`() → bool

    Allows exploration policies to define if they require the action values for the current step. This can save up a lot of computation. For example in e-greedy, if the random value generated is smaller than epsilon, the action is completely random, and the action values don’t need to be calculated :return: True if the action values are required. False otherwise

  `reset`()

    Used for resetting the exploration policy parameters when needed :return: None

## AdditiveNoise

class `rl_coach.exploration_policies.``AdditiveNoise`(action_space: rl_coach.spaces.ActionSpace, noise_percentage_schedule: rl_coach.schedules.Schedule, evaluation_noise_percentage: float)

  AdditiveNoise is an exploration policy intended for continuous action spaces. It takes the action from the agent and adds a Gaussian distributed noise to it. The amount of noise added to the action follows the noise amount that can be given in two different ways: 1. Specified by the user as a noise schedule which is taken in percentiles out of the action space size 2. Specified by the agents action. In case the agents action is a list with 2 values, the 1st one is assumed to be the mean of the action, and 2nd is assumed to be its standard deviation.

  Parameters: | action_space – the action space used by the environment noise_percentage_schedule – the schedule for the noise variance percentage relative to the absolute range of the action space evaluation_noise_percentage – the noise variance percentage that will be used during evaluation phases

## Boltzmann

class `rl_coach.exploration_policies.``Boltzmann`(action_space: rl_coach.spaces.ActionSpace, temperature_schedule: rl_coach.schedules.Schedule)

  The Boltzmann exploration policy is intended for discrete action spaces. It assumes that each of the possible actions has some value assigned to it (such as the Q value), and uses a softmax function to convert these values into a distribution over the actions. It then samples the action for playing out of the calculated distribution. An additional temperature schedule can be given by the user, and will control the steepness of the softmax function.

  Parameters: | action_space – the action space used by the environment temperature_schedule – the schedule for the temperature parameter of the softmax

## Bootstrapped

class `rl_coach.exploration_policies.``Bootstrapped`(action_space: rl_coach.spaces.ActionSpace, epsilon_schedule: rl_coach.schedules.Schedule, evaluation_epsilon: float, architecture_num_q_heads: int, continuous_exploration_policy_parameters: rl_coach.exploration_policies.exploration_policy.ExplorationParameters = <rl_coach.exploration_policies.additive_noise.AdditiveNoiseParameters object>)

  Bootstrapped exploration policy is currently only used for discrete action spaces along with the Bootstrapped DQN agent. It assumes that there is an ensemble of network heads, where each one predicts the values for all the possible actions. For each episode, a single head is selected to lead the agent, according to its value predictions. In evaluation, the action is selected using a majority vote over all the heads predictions.

  Note

  This exploration policy will only work for Discrete action spaces with Bootstrapped DQN style agents, since it requires the agent to have a network with multiple heads.

  Parameters: | action_space – the action space used by the environment epsilon_schedule – a schedule for the epsilon values evaluation_epsilon – the epsilon value to use for evaluation phases continuous_exploration_policy_parameters – the parameters of the continuous exploration policy to use if the e-greedy is used for a continuous policy architecture_num_q_heads – the number of q heads to select from

## Categorical

class `rl_coach.exploration_policies.``Categorical`(action_space: rl_coach.spaces.ActionSpace)

  Categorical exploration policy is intended for discrete action spaces. It expects the action values to represent a probability distribution over the action, from which a single action will be sampled. In evaluation, the action that has the highest probability will be selected. This is particularly useful for actor-critic schemes, where the actors output is a probability distribution over the actions.

  Parameters: | action_space – the action space used by the environment

## ContinuousEntropy

class `rl_coach.exploration_policies.``ContinuousEntropy`(action_space: rl_coach.spaces.ActionSpace, noise_percentage_schedule: rl_coach.schedules.Schedule, evaluation_noise_percentage: float)

  Continuous entropy is an exploration policy that is actually implemented as part of the network. The exploration policy class is only a placeholder for choosing this policy. The exploration policy is implemented by adding a regularization factor to the network loss, which regularizes the entropy of the action. This exploration policy is only intended for continuous action spaces, and assumes that the entire calculation is implemented as part of the head.

  Warning

  This exploration policy expects the agent or the network to implement the exploration functionality. There are only a few heads that actually are relevant and implement the entropy regularization factor.

  Parameters: | action_space – the action space used by the environment noise_percentage_schedule – the schedule for the noise variance percentage relative to the absolute range of the action space evaluation_noise_percentage – the noise variance percentage that will be used during evaluation phases

## EGreedy

class `rl_coach.exploration_policies.``EGreedy`(action_space: rl_coach.spaces.ActionSpace, epsilon_schedule: rl_coach.schedules.Schedule, evaluation_epsilon: float, continuous_exploration_policy_parameters: rl_coach.exploration_policies.exploration_policy.ExplorationParameters = <rl_coach.exploration_policies.additive_noise.AdditiveNoiseParameters object>)

  e-greedy is an exploration policy that is intended for both discrete and continuous action spaces.

  For discrete action spaces, it assumes that each action is assigned a value, and it selects the action with the highest value with probability 1 - epsilon. Otherwise, it selects a action sampled uniformly out of all the possible actions. The epsilon value is given by the user and can be given as a schedule. In evaluation, a different epsilon value can be specified.

  For continuous action spaces, it assumes that the mean action is given by the agent. With probability epsilon, it samples a random action out of the action space bounds. Otherwise, it selects the action according to a given continuous exploration policy, which is set to AdditiveNoise by default. In evaluation, the action is always selected according to the given continuous exploration policy (where its phase is set to evaluation as well).

  Parameters: | action_space – the action space used by the environment epsilon_schedule – a schedule for the epsilon values evaluation_epsilon – the epsilon value to use for evaluation phases continuous_exploration_policy_parameters – the parameters of the continuous exploration policy to use if the e-greedy is used for a continuous policy

## Greedy

class `rl_coach.exploration_policies.``Greedy`(action_space: rl_coach.spaces.ActionSpace)

  The Greedy exploration policy is intended for both discrete and continuous action spaces. For discrete action spaces, it always selects the action with the maximum value, as given by the agent. For continuous action spaces, it always return the exact action, as it was given by the agent.

  Parameters: | action_space – the action space used by the environment

## OUProcess

class `rl_coach.exploration_policies.``OUProcess`(action_space: rl_coach.spaces.ActionSpace, mu: float = 0, theta: float = 0.15, sigma: float = 0.2, dt: float = 0.01)

  OUProcess exploration policy is intended for continuous action spaces, and selects the action according to an Ornstein-Uhlenbeck process. The Ornstein-Uhlenbeck process implements the action as a Gaussian process, where the samples are correlated between consequent time steps.

  Parameters: | action_space – the action space used by the environment

## ParameterNoise

class `rl_coach.exploration_policies.``ParameterNoise`(network_params: Dict[str, rl_coach.base_parameters.NetworkParameters], action_space: rl_coach.spaces.ActionSpace)

  The ParameterNoise exploration policy is intended for both discrete and continuous action spaces. It applies the exploration policy by replacing all the dense network layers with noisy layers. The noisy layers have both weight means and weight standard deviations, and for each forward pass of the network the weights are sampled from a normal distribution that follows the learned weights mean and standard deviation values.

  Warning: currently supported only by DQN variants

  Parameters: | action_space – the action space used by the environment

## TruncatedNormal

class `rl_coach.exploration_policies.``TruncatedNormal`(action_space: rl_coach.spaces.ActionSpace, noise_percentage_schedule: rl_coach.schedules.Schedule, evaluation_noise_percentage: float, clip_low: float, clip_high: float)

  The TruncatedNormal exploration policy is intended for continuous action spaces. It samples the action from a normal distribution, where the mean action is given by the agent, and the standard deviation can be given in t wo different ways: 1. Specified by the user as a noise schedule which is taken in percentiles out of the action space size 2. Specified by the agents action. In case the agents action is a list with 2 values, the 1st one is assumed to be the mean of the action, and 2nd is assumed to be its standard deviation. When the sampled action is outside of the action bounds given by the user, it is sampled again and again, until it is within the bounds.

  Parameters: | action_space – the action space used by the environment noise_percentage_schedule – the schedule for the noise variance percentage relative to the absolute range of the action space evaluation_noise_percentage – the noise variance percentage that will be used during evaluation phases

## UCB

class `rl_coach.exploration_policies.``UCB`(action_space: rl_coach.spaces.ActionSpace, epsilon_schedule: rl_coach.schedules.Schedule, evaluation_epsilon: float, architecture_num_q_heads: int, lamb: int, continuous_exploration_policy_parameters: rl_coach.exploration_policies.exploration_policy.ExplorationParameters = <rl_coach.exploration_policies.additive_noise.AdditiveNoiseParameters object>)

  UCB exploration policy is following the upper confidence bound heuristic to sample actions in discrete action spaces. It assumes that there are multiple network heads that are predicting action values, and that the standard deviation between the heads predictions represents the uncertainty of the agent in each of the actions. It then updates the action value estimates to by mean(actions)+lambda*stdev(actions), where lambda is given by the user. This exploration policy aims to take advantage of the uncertainty of the agent in its predictions, and select the action according to the tradeoff between how uncertain the agent is, and how large it predicts the outcome from those actions to be.

  Parameters: | action_space – the action space used by the environment epsilon_schedule – a schedule for the epsilon values evaluation_epsilon – the epsilon value to use for evaluation phases architecture_num_q_heads – the number of q heads to select from lamb – lambda coefficient for taking the standard deviation into account continuous_exploration_policy_parameters – the parameters of the continuous exploration policy to use if the e-greedy is used for a continuous policy