mirror of
https://github.com/gryf/coach.git
synced 2025-12-17 11:10:20 +01:00
0896f43097
* Add Robosuite parameters for all env types + initialize env flow * Init flow done * Rest of Environment API complete for RobosuiteEnvironment * RobosuiteEnvironment changes * Observation stacking filter * Add proper frame_skip in addition to control_freq * Hardcode Coach rendering to 'frontview' camera * Robosuite_Lift_DDPG preset + Robosuite env updates * Move observation stacking filter from env to preset * Pre-process observation - concatenate depth map (if exists) to image and object state (if exists) to robot state * Preset parameters based on Surreal DDPG parameters, taken from: https://github.com/SurrealAI/surreal/blob/master/surreal/main/ddpg_configs.py * RobosuiteEnvironment fixes - working now with PyGame rendering * Preset minor modifications * ObservationStackingFilter - option to concat non-vector observations * Consider frame skip when setting horizon in robosuite env * Robosuite lift preset - update heatup length and training interval * Robosuite env - change control_freq to 10 to match Surreal usage * Robosuite clipped PPO preset * Distribute multiple workers (-n #) over multiple GPUs * Clipped PPO memory optimization from @shadiendrawis * Fixes to evaluation only workers * RoboSuite_ClippedPPO: Update training interval * Undo last commit (update training interval) * Fix "doube-negative" if conditions * multi-agent single-trainer clipped ppo training with cartpole * cleanups (not done yet) + ~tuned hyper-params for mast * Switch to Robosuite v1 APIs * Change presets to IK controller * more cleanups + enabling evaluation worker + better logging * RoboSuite_Lift_ClippedPPO updates * Fix major bug in obs normalization filter setup * Reduce coupling between Robosuite API and Coach environment * Now only non task-specific parameters are explicitly defined in Coach * Removed a bunch of enums of Robosuite elements, using simple strings instead * With this change new environments/robots/controllers in Robosuite can be used immediately in Coach * MAST: better logging of actor-trainer interaction + bug fixes + performance improvements. Still missing: fixed pubsub for obs normalization running stats + logging for trainer signals * lstm support for ppo * setting JOINT VELOCITY action space by default + fix for EveryNEpisodes video dump filter + new TaskIDDumpFilter + allowing or between video dump filters * Separate Robosuite clipped PPO preset for the non-MAST case * Add flatten layer to architectures and use it in Robosuite presets This is required for embedders that mix conv and dense TODO: Add MXNet implementation * publishing running_stats together with the published policy + hyper-param for when to publish a policy + cleanups * bug-fix for memory leak in MAST * Bugfix: Return value in TF BatchnormActivationDropout.to_tf_instance * Explicit activations in embedder scheme so there's no ReLU after flatten * Add clipped PPO heads with configurable dense layers at the beginning * This is a workaround needed to mimic Surreal-PPO, where the CNN and LSTM are shared between actor and critic but the FC layers are not shared * Added a "SchemeBuilder" class, currently only used for the new heads but we can change Middleware and Embedder implementations to use it as well * Video dump setting fix in basic preset * logging screen output to file * coach to start the redis-server for a MAST run * trainer drops off-policy data + old policy in ClippedPPO updates only after policy was published + logging free memory stats + actors check for a new policy only at the beginning of a new episode + fixed a bug where the trainer was logging "Training Reward = 0", causing dashboard to incorrectly display the signal * Add missing set_internal_state function in TFSharedRunningStats * Robosuite preset - use SingleLevelSelect instead of hard-coded level * policy ID published directly on Redis * Small fix when writing to log file * Major bugfix in Robosuite presets - pass dense sizes to heads * RoboSuite_Lift_ClippedPPO hyper-params update * add horizon and value bootstrap to GAE calculation, fix A3C with LSTM * adam hyper-params from mujoco * updated MAST preset with IK_POSE_POS controller * configurable initialization for policy stdev + custom extra noise per actor + logging of policy stdev to dashboard * values loss weighting of 0.5 * minor fixes + presets * bug-fix for MAST where the old policy in the trainer had kept updating every training iter while it should only update after every policy publish * bug-fix: reset_internal_state was not called by the trainer * bug-fixes in the lstm flow + some hyper-param adjustments for CartPole_ClippedPPO_LSTM -> training and sometimes reaches 200 * adding back the horizon hyper-param - a messy commit * another bug-fix missing from prev commit * set control_freq=2 to match action_scale 0.125 * ClippedPPO with MAST cleanups and some preps for TD3 with MAST * TD3 presets. RoboSuite_Lift_TD3 seems to work well with multi-process runs (-n 8) * setting termination on collision to be on by default * bug-fix following prev-prev commit * initial cube exploration environment with TD3 commit * bug fix + minor refactoring * several parameter changes and RND debugging * Robosuite Gym wrapper + Rename TD3_Random* -> Random* * algorithm update * Add RoboSuite v1 env + presets (to eventually replace non-v1 ones) * Remove grasping presets, keep only V1 exp. presets (w/o V1 tag) * Keep just robosuite V1 env as the 'robosuite_environment' module * Exclude Robosuite and MAST presets from integration tests * Exclude LSTM and MAST presets from golden tests * Fix mistakenly removed import * Revert debug changes in ReaderWriterLock * Try another way to exclude LSTM/MAST golden tests * Remove debug prints * Remove PreDense heads, unused in the end * Missed removing an instance of PreDense head * Remove MAST, not required for this PR * Undo unused concat option in ObservationStackingFilter * Remove LSTM updates, not required in this PR * Update README.md * code changes for the exploration flow to work with robosuite master branch * code cleanup + documentation * jupyter tutorial for the goal-based exploration + scatter plot * typo fix * Update README.md * seprate parameter for the obs-goal observation + small fixes * code clarity fixes * adjustment in tutorial 5 * Update tutorial * Update tutorial Co-authored-by: Guy Jacob <guy.jacob@intel.com> Co-authored-by: Gal Leibovich <gal.leibovich@intel.com> Co-authored-by: shadi.endrawis <sendrawi@aipg-ra-skx-03.ra.intel.com>
112 lines
5.0 KiB
Python
112 lines
5.0 KiB
Python
from rl_coach.agents.td3_exp_agent import TD3GoalBasedAgentParameters
|
|
from rl_coach.architectures.embedder_parameters import InputEmbedderParameters
|
|
from rl_coach.architectures.layers import Dense, Conv2d, BatchnormActivationDropout, Flatten
|
|
from rl_coach.base_parameters import EmbedderScheme
|
|
from rl_coach.core_types import TrainingSteps, EnvironmentEpisodes, EnvironmentSteps
|
|
from rl_coach.environments.robosuite_environment import RobosuiteGoalBasedExpEnvironmentParameters, \
|
|
OptionalObservations
|
|
from rl_coach.filters.filter import NoInputFilter, NoOutputFilter
|
|
from rl_coach.graph_managers.basic_rl_graph_manager import BasicRLGraphManager
|
|
from rl_coach.graph_managers.graph_manager import ScheduleParameters
|
|
from rl_coach.architectures.head_parameters import RNDHeadParameters
|
|
from rl_coach.schedules import LinearSchedule
|
|
|
|
####################
|
|
# Graph Scheduling #
|
|
####################
|
|
|
|
schedule_params = ScheduleParameters()
|
|
schedule_params.improve_steps = TrainingSteps(300000)
|
|
schedule_params.steps_between_evaluation_periods = TrainingSteps(300000)
|
|
schedule_params.evaluation_steps = EnvironmentEpisodes(0)
|
|
schedule_params.heatup_steps = EnvironmentSteps(1000)
|
|
|
|
#########
|
|
# Agent #
|
|
#########
|
|
|
|
agent_params = TD3GoalBasedAgentParameters()
|
|
agent_params.algorithm.use_non_zero_discount_for_terminal_states = False
|
|
agent_params.algorithm.identity_goal_sample_rate = 0.04
|
|
agent_params.exploration.noise_schedule = LinearSchedule(1.5, 0.5, 300000)
|
|
|
|
agent_params.algorithm.rnd_sample_size = 2000
|
|
agent_params.algorithm.rnd_batch_size = 500
|
|
agent_params.algorithm.rnd_optimization_epochs = 4
|
|
agent_params.algorithm.td3_training_ratio = 1.0
|
|
agent_params.algorithm.identity_goal_sample_rate = 0.0
|
|
agent_params.algorithm.env_obs_key = 'camera'
|
|
agent_params.algorithm.agent_obs_key = 'obs-goal'
|
|
agent_params.algorithm.replay_buffer_save_steps = 25000
|
|
agent_params.algorithm.replay_buffer_save_path = './tutorials'
|
|
|
|
agent_params.input_filter = NoInputFilter()
|
|
agent_params.output_filter = NoOutputFilter()
|
|
|
|
# Camera observation pre-processing network scheme
|
|
camera_obs_scheme = [
|
|
Conv2d(32, 8, 4),
|
|
BatchnormActivationDropout(activation_function='relu'),
|
|
Conv2d(64, 4, 2),
|
|
BatchnormActivationDropout(activation_function='relu'),
|
|
Conv2d(64, 3, 1),
|
|
BatchnormActivationDropout(activation_function='relu'),
|
|
Flatten(),
|
|
Dense(256),
|
|
BatchnormActivationDropout(activation_function='relu')
|
|
]
|
|
|
|
# Actor
|
|
actor_network = agent_params.network_wrappers['actor']
|
|
actor_network.input_embedders_parameters = {
|
|
'measurements': InputEmbedderParameters(scheme=EmbedderScheme.Empty),
|
|
agent_params.algorithm.agent_obs_key: InputEmbedderParameters(scheme=camera_obs_scheme, activation_function='none')
|
|
}
|
|
|
|
actor_network.middleware_parameters.scheme = [Dense(300), Dense(200)]
|
|
actor_network.learning_rate = 1e-4
|
|
|
|
# Critic
|
|
critic_network = agent_params.network_wrappers['critic']
|
|
critic_network.input_embedders_parameters = {
|
|
'action': InputEmbedderParameters(scheme=EmbedderScheme.Empty),
|
|
'measurements': InputEmbedderParameters(scheme=EmbedderScheme.Empty),
|
|
agent_params.algorithm.agent_obs_key: InputEmbedderParameters(scheme=camera_obs_scheme, activation_function='none')
|
|
}
|
|
|
|
critic_network.middleware_parameters.scheme = [Dense(400), Dense(300)]
|
|
critic_network.learning_rate = 1e-4
|
|
|
|
# RND
|
|
agent_params.network_wrappers['predictor'].input_embedders_parameters = \
|
|
{agent_params.algorithm.env_obs_key: InputEmbedderParameters(scheme=EmbedderScheme.Empty,
|
|
input_rescaling={'image': 1.0},
|
|
flatten=False)}
|
|
agent_params.network_wrappers['constant'].input_embedders_parameters = \
|
|
{agent_params.algorithm.env_obs_key: InputEmbedderParameters(scheme=EmbedderScheme.Empty,
|
|
input_rescaling={'image': 1.0},
|
|
flatten=False)}
|
|
agent_params.network_wrappers['predictor'].heads_parameters = [RNDHeadParameters(is_predictor=True)]
|
|
|
|
###############
|
|
# Environment #
|
|
###############
|
|
env_params = RobosuiteGoalBasedExpEnvironmentParameters(level='CubeExp')
|
|
env_params.robot = 'Panda'
|
|
env_params.custom_controller_config_fpath = './rl_coach/environments/robosuite/osc_pose.json'
|
|
env_params.base_parameters.optional_observations = OptionalObservations.CAMERA
|
|
env_params.base_parameters.render_camera = 'frontview'
|
|
env_params.base_parameters.camera_names = 'agentview'
|
|
env_params.base_parameters.camera_depths = False
|
|
env_params.base_parameters.horizon = 200
|
|
env_params.base_parameters.ignore_done = False
|
|
env_params.base_parameters.use_object_obs = True
|
|
env_params.frame_skip = 1
|
|
env_params.base_parameters.control_freq = 2
|
|
env_params.base_parameters.camera_heights = 84
|
|
env_params.base_parameters.camera_widths = 84
|
|
env_params.extra_parameters = {'hard_reset': False}
|
|
|
|
|
|
graph_manager = BasicRLGraphManager(agent_params=agent_params, env_params=env_params, schedule_params=schedule_params)
|