# Memories — Reinforcement Learning Coach 0.12.0 documentation

# Memories

## Episodic Memories

### EpisodicExperienceReplay

class `rl_coach.memories.episodic.``EpisodicExperienceReplay`(max_size: Tuple[rl_coach.memories.memory.MemoryGranularity, int] = (<MemoryGranularity.Transitions: 0>, 1000000), n_step=-1, train_to_eval_ratio: int = 1)

  A replay buffer that stores episodes of transitions. The additional structure allows performing various calculations of total return and other values that depend on the sequential behavior of the transitions in the episode.

  Parameters

    max_size – the maximum number of transitions or episodes to hold in the memory

### EpisodicHindsightExperienceReplay

class `rl_coach.memories.episodic.``EpisodicHindsightExperienceReplay`(max_size: Tuple[rl_coach.memories.memory.MemoryGranularity, int], hindsight_transitions_per_regular_transition: int, hindsight_goal_selection_method: rl_coach.memories.episodic.episodic_hindsight_experience_replay.HindsightGoalSelectionMethod, goals_space: rl_coach.spaces.GoalsSpace)

  Implements Hindsight Experience Replay as described in the following paper: https://arxiv.org/pdf/1707.01495.pdf

  Parameters

    - max_size – The maximum size of the memory. should be defined in a granularity of Transitions

    - hindsight_transitions_per_regular_transition – The number of hindsight artificial transitions to generate for each actual transition

    - hindsight_goal_selection_method – The method that will be used for generating the goals for the hindsight transitions. Should be one of HindsightGoalSelectionMethod

    - goals_space – A GoalsSpace which defines the base properties of the goals space

### EpisodicHRLHindsightExperienceReplay

class `rl_coach.memories.episodic.``EpisodicHRLHindsightExperienceReplay`(max_size: Tuple[rl_coach.memories.memory.MemoryGranularity, int], hindsight_transitions_per_regular_transition: int, hindsight_goal_selection_method: rl_coach.memories.episodic.episodic_hindsight_experience_replay.HindsightGoalSelectionMethod, goals_space: rl_coach.spaces.GoalsSpace)

  Implements HRL Hindsight Experience Replay as described in the following paper: https://arxiv.org/abs/1805.08180

  This is the memory you should use if you want a shared hindsight experience replay buffer between multiple workers

  Parameters

    - max_size – The maximum size of the memory. should be defined in a granularity of Transitions

    - hindsight_transitions_per_regular_transition – The number of hindsight artificial transitions to generate for each actual transition

    - hindsight_goal_selection_method – The method that will be used for generating the goals for the hindsight transitions. Should be one of HindsightGoalSelectionMethod

    - goals_space – A GoalsSpace which defines the properties of the goals

    - do_action_hindsight – Replace the action (sub-goal) given to a lower layer, with the actual achieved goal

### SingleEpisodeBuffer

class `rl_coach.memories.episodic.``SingleEpisodeBuffer`

## Non-Episodic Memories

### BalancedExperienceReplay

class `rl_coach.memories.non_episodic.``BalancedExperienceReplay`(max_size: Tuple[rl_coach.memories.memory.MemoryGranularity, int], allow_duplicates_in_batch_sampling: bool = True, num_classes: int = 0, state_key_with_the_class_index: Any = 'class')
  Parameters

    - max_size – the maximum number of transitions or episodes to hold in the memory

    - allow_duplicates_in_batch_sampling – allow having the same transition multiple times in a batch

    - num_classes – the number of classes in the replayed data

    - state_key_with_the_class_index – the class index is assumed to be a value in the state dictionary. this parameter determines the key to retrieve the class index value

### QDND

class `rl_coach.memories.non_episodic.``QDND`(dict_size, key_width, num_actions, new_value_shift_coefficient=0.1, key_error_threshold=0.01, learning_rate=0.01, num_neighbors=50, return_additional_data=False, override_existing_keys=False, rebuild_on_every_update=False)

### ExperienceReplay

class `rl_coach.memories.non_episodic.``ExperienceReplay`(max_size: Tuple[rl_coach.memories.memory.MemoryGranularity, int], allow_duplicates_in_batch_sampling: bool = True)

  A regular replay buffer which stores transition without any additional structure

  Parameters

    - max_size – the maximum number of transitions or episodes to hold in the memory

    - allow_duplicates_in_batch_sampling – allow having the same transition multiple times in a batch

### PrioritizedExperienceReplay

class `rl_coach.memories.non_episodic.``PrioritizedExperienceReplay`(max_size: Tuple[rl_coach.memories.memory.MemoryGranularity, int], alpha: float = 0.6, beta: rl_coach.schedules.Schedule = <rl_coach.schedules.ConstantSchedule object>, epsilon: float = 1e-06, allow_duplicates_in_batch_sampling: bool = True)

  This is the proportional sampling variant of the prioritized experience replay as described in https://arxiv.org/pdf/1511.05952.pdf.

  Parameters

    - max_size – the maximum number of transitions or episodes to hold in the memory

    - alpha – the alpha prioritization coefficient

    - beta – the beta parameter used for importance sampling

    - epsilon – a small value added to the priority of each transition

    - allow_duplicates_in_batch_sampling – allow having the same transition multiple times in a batch

### TransitionCollection

class `rl_coach.memories.non_episodic.``TransitionCollection`

  Simple python implementation of transitions collection non-episodic memories are constructed on top of.