mirror of
https://github.com/gryf/coach.git
synced 2025-12-18 19:50:17 +01:00
Robosuite exploration (#478)
* Add Robosuite parameters for all env types + initialize env flow * Init flow done * Rest of Environment API complete for RobosuiteEnvironment * RobosuiteEnvironment changes * Observation stacking filter * Add proper frame_skip in addition to control_freq * Hardcode Coach rendering to 'frontview' camera * Robosuite_Lift_DDPG preset + Robosuite env updates * Move observation stacking filter from env to preset * Pre-process observation - concatenate depth map (if exists) to image and object state (if exists) to robot state * Preset parameters based on Surreal DDPG parameters, taken from: https://github.com/SurrealAI/surreal/blob/master/surreal/main/ddpg_configs.py * RobosuiteEnvironment fixes - working now with PyGame rendering * Preset minor modifications * ObservationStackingFilter - option to concat non-vector observations * Consider frame skip when setting horizon in robosuite env * Robosuite lift preset - update heatup length and training interval * Robosuite env - change control_freq to 10 to match Surreal usage * Robosuite clipped PPO preset * Distribute multiple workers (-n #) over multiple GPUs * Clipped PPO memory optimization from @shadiendrawis * Fixes to evaluation only workers * RoboSuite_ClippedPPO: Update training interval * Undo last commit (update training interval) * Fix "doube-negative" if conditions * multi-agent single-trainer clipped ppo training with cartpole * cleanups (not done yet) + ~tuned hyper-params for mast * Switch to Robosuite v1 APIs * Change presets to IK controller * more cleanups + enabling evaluation worker + better logging * RoboSuite_Lift_ClippedPPO updates * Fix major bug in obs normalization filter setup * Reduce coupling between Robosuite API and Coach environment * Now only non task-specific parameters are explicitly defined in Coach * Removed a bunch of enums of Robosuite elements, using simple strings instead * With this change new environments/robots/controllers in Robosuite can be used immediately in Coach * MAST: better logging of actor-trainer interaction + bug fixes + performance improvements. Still missing: fixed pubsub for obs normalization running stats + logging for trainer signals * lstm support for ppo * setting JOINT VELOCITY action space by default + fix for EveryNEpisodes video dump filter + new TaskIDDumpFilter + allowing or between video dump filters * Separate Robosuite clipped PPO preset for the non-MAST case * Add flatten layer to architectures and use it in Robosuite presets This is required for embedders that mix conv and dense TODO: Add MXNet implementation * publishing running_stats together with the published policy + hyper-param for when to publish a policy + cleanups * bug-fix for memory leak in MAST * Bugfix: Return value in TF BatchnormActivationDropout.to_tf_instance * Explicit activations in embedder scheme so there's no ReLU after flatten * Add clipped PPO heads with configurable dense layers at the beginning * This is a workaround needed to mimic Surreal-PPO, where the CNN and LSTM are shared between actor and critic but the FC layers are not shared * Added a "SchemeBuilder" class, currently only used for the new heads but we can change Middleware and Embedder implementations to use it as well * Video dump setting fix in basic preset * logging screen output to file * coach to start the redis-server for a MAST run * trainer drops off-policy data + old policy in ClippedPPO updates only after policy was published + logging free memory stats + actors check for a new policy only at the beginning of a new episode + fixed a bug where the trainer was logging "Training Reward = 0", causing dashboard to incorrectly display the signal * Add missing set_internal_state function in TFSharedRunningStats * Robosuite preset - use SingleLevelSelect instead of hard-coded level * policy ID published directly on Redis * Small fix when writing to log file * Major bugfix in Robosuite presets - pass dense sizes to heads * RoboSuite_Lift_ClippedPPO hyper-params update * add horizon and value bootstrap to GAE calculation, fix A3C with LSTM * adam hyper-params from mujoco * updated MAST preset with IK_POSE_POS controller * configurable initialization for policy stdev + custom extra noise per actor + logging of policy stdev to dashboard * values loss weighting of 0.5 * minor fixes + presets * bug-fix for MAST where the old policy in the trainer had kept updating every training iter while it should only update after every policy publish * bug-fix: reset_internal_state was not called by the trainer * bug-fixes in the lstm flow + some hyper-param adjustments for CartPole_ClippedPPO_LSTM -> training and sometimes reaches 200 * adding back the horizon hyper-param - a messy commit * another bug-fix missing from prev commit * set control_freq=2 to match action_scale 0.125 * ClippedPPO with MAST cleanups and some preps for TD3 with MAST * TD3 presets. RoboSuite_Lift_TD3 seems to work well with multi-process runs (-n 8) * setting termination on collision to be on by default * bug-fix following prev-prev commit * initial cube exploration environment with TD3 commit * bug fix + minor refactoring * several parameter changes and RND debugging * Robosuite Gym wrapper + Rename TD3_Random* -> Random* * algorithm update * Add RoboSuite v1 env + presets (to eventually replace non-v1 ones) * Remove grasping presets, keep only V1 exp. presets (w/o V1 tag) * Keep just robosuite V1 env as the 'robosuite_environment' module * Exclude Robosuite and MAST presets from integration tests * Exclude LSTM and MAST presets from golden tests * Fix mistakenly removed import * Revert debug changes in ReaderWriterLock * Try another way to exclude LSTM/MAST golden tests * Remove debug prints * Remove PreDense heads, unused in the end * Missed removing an instance of PreDense head * Remove MAST, not required for this PR * Undo unused concat option in ObservationStackingFilter * Remove LSTM updates, not required in this PR * Update README.md * code changes for the exploration flow to work with robosuite master branch * code cleanup + documentation * jupyter tutorial for the goal-based exploration + scatter plot * typo fix * Update README.md * seprate parameter for the obs-goal observation + small fixes * code clarity fixes * adjustment in tutorial 5 * Update tutorial * Update tutorial Co-authored-by: Guy Jacob <guy.jacob@intel.com> Co-authored-by: Gal Leibovich <gal.leibovich@intel.com> Co-authored-by: shadi.endrawis <sendrawi@aipg-ra-skx-03.ra.intel.com>
This commit is contained in:
shadiendrawis
@@ -0,0 +1,54 @@
|
||||
#
|
||||
# Copyright (c) 2019 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import tensorflow as tf
|
||||
import numpy as np
|
||||
|
||||
from rl_coach.architectures.tensorflow_components.layers import Conv2d, BatchnormActivationDropout
|
||||
from rl_coach.architectures.tensorflow_components.heads.head import Head, Orthogonal
|
||||
from rl_coach.base_parameters import AgentParameters
|
||||
from rl_coach.core_types import Embedding
|
||||
from rl_coach.spaces import SpacesDefinition
|
||||
|
||||
|
||||
class RNDHead(Head):
|
||||
def __init__(self, agent_parameters: AgentParameters, spaces: SpacesDefinition, network_name: str,
|
||||
head_idx: int = 0, is_local: bool = True, is_predictor: bool = False):
|
||||
super().__init__(agent_parameters, spaces, network_name, head_idx, is_local)
|
||||
self.name = 'rnd_head'
|
||||
self.return_type = Embedding
|
||||
self.is_predictor = is_predictor
|
||||
self.activation_function = tf.nn.leaky_relu
|
||||
|
||||
self.loss_type = tf.losses.mean_squared_error
|
||||
|
||||
def _build_module(self, input_layer):
|
||||
weight_init = Orthogonal(gain=np.sqrt(2))
|
||||
input_layer = Conv2d(num_filters=32, kernel_size=8, strides=4)(input_layer, kernel_initializer=weight_init)
|
||||
input_layer = BatchnormActivationDropout(activation_function=self.activation_function)(input_layer)[-1]
|
||||
input_layer = Conv2d(num_filters=64, kernel_size=4, strides=2)(input_layer, kernel_initializer=weight_init)
|
||||
input_layer = BatchnormActivationDropout(activation_function=self.activation_function)(input_layer)[-1]
|
||||
input_layer = Conv2d(num_filters=64, kernel_size=3, strides=1)(input_layer, kernel_initializer=weight_init)
|
||||
input_layer = BatchnormActivationDropout(activation_function=self.activation_function)(input_layer)[-1]
|
||||
input_layer = tf.contrib.layers.flatten(input_layer)
|
||||
|
||||
if self.is_predictor:
|
||||
input_layer = self.dense_layer(512)(input_layer, kernel_initializer=weight_init)
|
||||
input_layer = BatchnormActivationDropout(activation_function=tf.nn.relu)(input_layer)[-1]
|
||||
input_layer = self.dense_layer(512)(input_layer, kernel_initializer=weight_init)
|
||||
input_layer = BatchnormActivationDropout(activation_function=tf.nn.relu)(input_layer)[-1]
|
||||
|
||||
self.output = self.dense_layer(512)(input_layer, name='output', kernel_initializer=weight_init)
|
||||
@@ -19,6 +19,7 @@ from .cil_head import RegressionHead
|
||||
from .td3_v_head import TD3VHead
|
||||
from .ddpg_v_head import DDPGVHead
|
||||
from .wolpertinger_actor_head import WolpertingerActorHead
|
||||
from .RND_head import RNDHead
|
||||
|
||||
__all__ = [
|
||||
'CategoricalQHead',
|
||||
@@ -41,5 +42,6 @@ __all__ = [
|
||||
'RegressionHead',
|
||||
'TD3VHead',
|
||||
'DDPGVHead',
|
||||
'WolpertingerActorHead'
|
||||
'WolpertingerActorHead',
|
||||
'RNDHead'
|
||||
]
|
||||
|
||||
@@ -23,6 +23,7 @@ from rl_coach.spaces import SpacesDefinition
|
||||
from rl_coach.utils import force_list
|
||||
from rl_coach.architectures.tensorflow_components.utils import squeeze_tensor
|
||||
|
||||
|
||||
# Used to initialize weights for policy and value output layers
|
||||
def normalized_columns_initializer(std=1.0):
|
||||
def _initializer(shape, dtype=None, partition_info=None):
|
||||
@@ -32,6 +33,29 @@ def normalized_columns_initializer(std=1.0):
|
||||
return _initializer
|
||||
|
||||
|
||||
# Used to initialize RND network parameters
|
||||
class Orthogonal(tf.initializers.orthogonal):
|
||||
def __init__(self, gain=1.0):
|
||||
super().__init__(gain=gain)
|
||||
|
||||
def __call__(self, shape, dtype=None, partition_info=None):
|
||||
shape = tuple(shape)
|
||||
if len(shape) == 2:
|
||||
flat_shape = shape
|
||||
elif len(shape) == 4: # assumes NHWC
|
||||
flat_shape = (np.prod(shape[:-1]), shape[-1])
|
||||
else:
|
||||
raise NotImplementedError
|
||||
a = np.random.normal(0.0, 1.0, flat_shape)
|
||||
u, _, v = np.linalg.svd(a, full_matrices=False)
|
||||
q = u if u.shape == flat_shape else v # pick the one with the correct shape
|
||||
q = q.reshape(shape)
|
||||
return (self.gain * q[:shape[0], :shape[1]]).astype(np.float32)
|
||||
|
||||
def get_config(self):
|
||||
return {"gain": self.gain}
|
||||
|
||||
|
||||
class Head(object):
|
||||
"""
|
||||
A head is the final part of the network. It takes the embedding from the middleware embedder and passes it through
|
||||
|
||||
@@ -109,7 +109,7 @@ class Conv2d(layers.Conv2d):
|
||||
def __init__(self, num_filters: int, kernel_size: int, strides: int):
|
||||
super(Conv2d, self).__init__(num_filters=num_filters, kernel_size=kernel_size, strides=strides)
|
||||
|
||||
def __call__(self, input_layer, name: str=None, is_training=None):
|
||||
def __call__(self, input_layer, name: str=None, is_training=None, kernel_initializer=None):
|
||||
"""
|
||||
returns a tensorflow conv2d layer
|
||||
:param input_layer: previous layer
|
||||
@@ -117,7 +117,8 @@ class Conv2d(layers.Conv2d):
|
||||
:return: conv2d layer
|
||||
"""
|
||||
return tf.layers.conv2d(input_layer, filters=self.num_filters, kernel_size=self.kernel_size,
|
||||
strides=self.strides, data_format='channels_last', name=name)
|
||||
strides=self.strides, data_format='channels_last', name=name,
|
||||
kernel_initializer=kernel_initializer)
|
||||
|
||||
@staticmethod
|
||||
@reg_to_tf_instance(layers.Conv2d)
|
||||
@@ -153,7 +154,7 @@ class BatchnormActivationDropout(layers.BatchnormActivationDropout):
|
||||
@staticmethod
|
||||
@reg_to_tf_instance(layers.BatchnormActivationDropout)
|
||||
def to_tf_instance(base: layers.BatchnormActivationDropout):
|
||||
return BatchnormActivationDropout, BatchnormActivationDropout(
|
||||
return BatchnormActivationDropout(
|
||||
batchnorm=base.batchnorm,
|
||||
activation_function=base.activation_function,
|
||||
dropout_rate=base.dropout_rate)
|
||||
|
||||
Reference in New Issue
Block a user