mani_skill.envs.tasks.mobile_manipulation

Submodules

• mani_skill.envs.tasks.mobile_manipulation.open_cabinet_drawer
• mani_skill.envs.tasks.mobile_manipulation.robocasa

Classes

OpenCabinetDoorEnv	Task Description:
OpenCabinetDrawerEnv	Task Description:
RoboCasaKitchenEnv	Superclass for ManiSkill environments.

Package Contents

class mani_skill.envs.tasks.mobile_manipulation.OpenCabinetDoorEnv(*args, robot_uids='fetch', robot_init_qpos_noise=0.02, reconfiguration_freq=None, num_envs=1, **kwargs)

Bases: OpenCabinetDrawerEnv

Task Description: Use the Fetch mobile manipulation robot to move towards a target cabinet and open the target drawer out.

Randomizations: - Robot is randomly initialized 1.6 to 1.8 meters away from the cabinet and positioned to face it - Robot’s base orientation is randomized by -9 to 9 degrees - The cabinet selected to manipulate is randomly sampled from all PartnetMobility cabinets that have drawers - The drawer to open is randomly sampled from all drawers available to open

Success Conditions: - The drawer is open at least 90% of the way, and the angular/linear velocities of the drawer link are small

Goal Specification: - 3D goal position centered at the center of mass of the handle mesh on the drawer to open (also visualized in human renders with a sphere).

TRAIN_JSON

handle_types = ['revolute', 'revolute_unwrapped']

class mani_skill.envs.tasks.mobile_manipulation.OpenCabinetDrawerEnv(*args, robot_uids='fetch', robot_init_qpos_noise=0.02, reconfiguration_freq=None, num_envs=1, **kwargs)

Bases: mani_skill.envs.sapien_env.BaseEnv

Task Description: Use the Fetch mobile manipulation robot to move towards a target cabinet and open the target drawer out.

Randomizations: - Robot is randomly initialized 1.6 to 1.8 meters away from the cabinet and positioned to face it - Robot’s base orientation is randomized by -9 to 9 degrees - The cabinet selected to manipulate is randomly sampled from all PartnetMobility cabinets that have drawers - The drawer to open is randomly sampled from all drawers available to open

Success Conditions: - The drawer is open at least 90% of the way, and the angular/linear velocities of the drawer link are small

Goal Specification: - 3D goal position centered at the center of mass of the handle mesh on the drawer to open (also visualized in human renders with a sphere).

_after_control_step()

Code that runs after each action has been taken. On GPU simulation this is called right before observations are fetched from the GPU buffers.

_after_reconfigure(options)

Add code here that should run immediately after self._reconfigure is called. The torch RNG context is still active so RNG is still seeded here by self._episode_seed. This is useful if you need to run something that only happens after reconfiguration but need the GPU initialized so that you can check e.g. collisons, poses etc.

_get_obs_extra(info)

Get task-relevant extra observations. Usually defined on a task by task basis

Parameters:

info (dict) –

_initialize_episode(env_idx, options)

Initialize the episode, e.g., poses of actors and articulations, as well as task relevant data like randomizing goal positions

Parameters:

• env_idx (torch.Tensor) –

• options (dict) –

_load_agent(options)

loads the agent/controllable articulations into the environment. The default function provides a convenient way to setup the agent/robot by a robot_uid (stored in self.robot_uids) without requiring the user to have to write the robot building and controller code themselves. For more advanced use-cases you can override this function to have more control over the agent/robot building process.

Parameters:

• options (dict) – The options for the environment.

• initial_agent_poses (Optional[Union[sapien.Pose, Pose]]) – The initial poses of the agent/robot. Providing these poses and ensuring they are picked such that they do not collide with objects if spawned there is highly recommended to ensure more stable simulation (the agent pose can be changed later during episode initialization).

• build_separate (bool) – Whether to build the agent/robot separately. If True, the agent/robot will be built separately for each parallel environment and then merged together to be accessible under one view/object. This is useful for randomizing physical and visual properties of the agent/robot which is only permitted for articulations built separately in each environment.

_load_cabinets(joint_types)

Parameters:

joint_types (list[str]) –

_load_scene(options)

Loads all objects like actors and articulations into the scene. Called by self._reconfigure. Given options argument is the same options dictionary passed to the self.reset function

Parameters:

options (dict) –

compute_dense_reward(obs, action, info)

Compute the dense reward.

Parameters:

• obs (Any) – The observation data. By default the observation data will be in its most raw form, a dictionary (no flattening, wrappers etc.)

• action (torch.Tensor) – The most recent action.

• info (dict) – The info dictionary.

compute_normalized_dense_reward(obs, action, info)

Compute the normalized dense reward.

Parameters:

• obs (Any) – The observation data. By default the observation data will be in its most raw form, a dictionary (no flattening, wrappers etc.)

• action (torch.Tensor) – The most recent action.

• info (dict) – The info dictionary.

evaluate()

Evaluate whether the environment is currently in a success state by returning a dictionary with a “success” key or a failure state via a “fail” key

This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards.

By default if not overriden this function returns an empty dictionary

handle_link_positions(env_idx=None)

Parameters:

env_idx (Optional[torch.Tensor]) –

SUPPORTED_ROBOTS = ['fetch']

Override this to enforce which robots or tuples of robots together are supported in the task. During env creation, setting robot_uids auto loads all desired robots into the scene, but not all tasks are designed to support some robot setups

TRAIN_JSON

property _default_human_render_camera_configs

Add default cameras for rendering when using render_mode=’rgb_array’. These can be overriden by the user at env creation time

property _default_sensor_configs

Add default (non-agent) sensors to the environment by returning sensor configurations. These can be overriden by the user at env creation time

property _default_sim_config

_sample_video_link = 'https://github.com/haosulab/ManiSkill/raw/main/figures/environment_demos/OpenCabinetDrawer-v1_rt.mp4'

a link to a sample video of the task. This is mostly used for automatic documentation generation

agent: mani_skill.agents.robots.Fetch

all_model_ids

handle_types = ['prismatic']

min_open_frac = 0.75

robot_init_qpos_noise = 0.02

class mani_skill.envs.tasks.mobile_manipulation.RoboCasaKitchenEnv(*args, robot_uids='fetch', env_configuration='default', controller_configs=None, gripper_types='default', base_types='default', initialization_noise='default', use_camera_obs=True, use_object_obs=True, reward_scale=1.0, reward_shaping=False, placement_initializer=None, has_renderer=False, has_offscreen_renderer=True, render_camera='robot0_agentview_center', render_collision_mesh=False, render_visual_mesh=True, render_gpu_device_id=-1, control_freq=20, horizon=1000, ignore_done=False, camera_names='agentview', camera_heights=256, camera_widths=256, camera_depths=False, renderer='mujoco', renderer_config=None, init_robot_base_pos=None, seed=None, layout_and_style_ids=None, layout_ids=None, style_ids=None, scene_split=None, generative_textures=None, obj_registries=('objaverse',), obj_instance_split=None, use_distractors=False, translucent_robot=False, randomize_cameras=False, fixtures_only=False, **kwargs)

Bases: mani_skill.envs.sapien_env.BaseEnv

Superclass for ManiSkill environments.

Parameters:

• num_envs – number of parallel environments to run. By default this is 1, which means a CPU simulation is used. If greater than 1, then we initialize the GPU simulation setup. Note that not all environments are faster when simulated on the GPU due to limitations of GPU simulations. For example, environments with many moving objects are better simulated by parallelizing across CPUs.

• obs_mode – observation mode to be used. Must be one of (“state”, “state_dict”, “none”, “sensor_data”, “rgb”, “depth”, “segmentation”, “rgbd”, “rgb+depth”, “rgb+depth+segmentation”, “rgb+segmentation”, “depth+segmentation”, “pointcloud”) The obs_mode is mostly for convenience to automatically optimize/setup all sensors/cameras for the given observation mode to render the correct data and try to ignore unnecessary rendering. For the most advanced use cases (e.g. you have 1 RGB only camera and 1 depth only camera)

• reward_mode – reward mode to use. Must be one of (“normalized_dense”, “dense”, “sparse”, “none”). With “none” the reward returned is always 0

• control_mode – control mode of the agent. “*” represents all registered controllers, and the action space will be a dict.

• render_mode – render mode registered in @SUPPORTED_RENDER_MODES.

• shader_dir (Optional[str]) –

  shader directory. Defaults to None. Setting this will override the shader used for all cameras in the environment. This is legacy behavior kept for backwards compatibility. The proper way to change the shaders used for cameras is to either change the environment code or pass in sensor_configs/human_render_camera_configs with the desired shaders.

  Previously the options are “default”, “rt”, “rt-fast”. “rt” means ray-tracing which results in more photorealistic renders but is slow, “rt-fast” is a lower quality but faster version of “rt”.

• enable_shadow (bool) – whether to enable shadow for lights. Defaults to False.

• sensor_configs (dict) – configurations of sensors to override any environment defaults. If the key is one of sensor names (e.g. a camera), the config value will be applied to the corresponding sensor. Otherwise, the value will be applied to all sensors (but overridden by sensor-specific values). For possible configurations see the documentation see the sensors documentation.

• human_render_camera_configs (dict) – configurations of human rendering cameras to override any environment defaults. Similar usage as @sensor_configs.

• viewer_camera_configs (dict) – configurations of the viewer camera in the GUI to override any environment defaults. Similar usage as @sensor_configs.

• robot_uids (Union[str, BaseAgent, list[Union[str, BaseAgent]]]) – list of robots to instantiate and control in the environment.

• sim_config (Union[SimConfig, dict]) – Configurations for simulation if used that override the environment defaults. If given a dictionary, it can just override specific attributes e.g. sim_config=dict(scene_config=dict(solver_iterations=25)). If passing in a SimConfig object, while typed, will override every attribute including the task defaults. Some environments define their own recommended default sim configurations via the self._default_sim_config attribute that generally should not be completely overriden.

• reconfiguration_freq (int) – How frequently to call reconfigure when environment is reset via self.reset(…) Generally for most users who are not building tasks this does not need to be changed. The default is 0, which means the environment reconfigures upon creation, and never again.

• sim_backend (str) – By default this is “auto”. If sim_backend is “auto”, then if num_envs == 1, we use the PhysX CPU sim backend, otherwise we use the PhysX GPU sim backend and automatically pick a GPU to use. Can also be “physx_cpu” or “physx_cuda” to force usage of a particular sim backend. To select a particular GPU to run the simulation on, you can pass “physx_cuda:n” where n is the ID of the GPU, similar to the way PyTorch selects GPUs. Note that if this is “physx_cpu”, num_envs can only be equal to 1.

• render_backend (str) –

  By default this is “gpu”. If render_backend is “gpu” or it’s alias “sapien_cuda”, then we auto select a GPU to render with. It can be “sapien_cuda:n” where n is the ID of the GPU to render with. If this is “cpu” or “sapien_cpu”, then we try to render on the CPU. If this is “none” or None, then we disable rendering.

  Note that some environments may require rendering functionalities to work. Moreover it is sometimes difficult to determine before running an environment if your machine can render or not. If you encounter some issue with rendering you can first try to double check your NVIDIA drivers / Vulkan drivers are setup correctly. If you don’t need to do rendering you can simply disable it by setting render_backend to “none” or None.

• parallel_in_single_scene (bool) – By default this is False. If True, rendered images and the GUI will show all objects in one view. This is only really useful for generating cool videos showing all environments at once but it is not recommended otherwise as it slows down simulation and rendering.

• enhanced_determinism (bool) – By default this is False and env resets will reset the episode RNG only when a seed / seed list is given. If True, the environment will reset the episode RNG upon each reset regardless of whether a seed is provided. Generally enhanced_determinisim is not needed and users are recommended to pass seeds into the env reset function instead.

_get_obs_extra(info)

Get task-relevant extra observations. Usually defined on a task by task basis

Parameters:

info (dict) –

_initialize_episode(env_idx, options)

Initialize the episode, e.g., poses of actors and articulations, as well as task relevant data like randomizing goal positions

Parameters:

• env_idx (torch.Tensor) –

• options (dict) –

_load_agent(options)

loads the agent/controllable articulations into the environment. The default function provides a convenient way to setup the agent/robot by a robot_uid (stored in self.robot_uids) without requiring the user to have to write the robot building and controller code themselves. For more advanced use-cases you can override this function to have more control over the agent/robot building process.

Parameters:

• options (dict) – The options for the environment.

• initial_agent_poses (Optional[Union[sapien.Pose, Pose]]) – The initial poses of the agent/robot. Providing these poses and ensuring they are picked such that they do not collide with objects if spawned there is highly recommended to ensure more stable simulation (the agent pose can be changed later during episode initialization).

• build_separate (bool) – Whether to build the agent/robot separately. If True, the agent/robot will be built separately for each parallel environment and then merged together to be accessible under one view/object. This is useful for randomizing physical and visual properties of the agent/robot which is only permitted for articulations built separately in each environment.

_load_scene(options)

Loads all objects like actors and articulations into the scene. Called by self._reconfigure. Given options argument is the same options dictionary passed to the self.reset function

Parameters:

options (dict) –

_setup_kitchen_references()

setup fixtures (and their references). this function is called within load_model function for kitchens

evaluate()

Evaluate whether the environment is currently in a success state by returning a dictionary with a “success” key or a failure state via a “fail” key

This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards.

By default if not overriden this function returns an empty dictionary

register_fixture_ref(ref_name, fn_kwargs)

Registers a fixture reference for later use. Initializes the fixture if it has not been initialized yet.

Parameters:

• ref_name (str) – name of the reference

• fn_kwargs (dict) – keyword arguments to pass to get_fixture

Returns:

fixture object

Return type:

Fixture

sample_object(groups, exclude_groups=None, graspable=None, microwavable=None, washable=None, cookable=None, freezable=None, split=None, obj_registries=None, max_size=(None, None, None), object_scale=None, rng=None)

Sample a kitchen object from the specified groups and within max_size bounds.

Parameters:

• groups (list or str) – groups to sample from or the exact xml path of the object to spawn

• exclude_groups (str or list) – groups to exclude

• graspable (bool) – whether the sampled object must be graspable

• washable (bool) – whether the sampled object must be washable

• microwavable (bool) – whether the sampled object must be microwavable

• cookable (bool) – whether whether the sampled object must be cookable

• freezable (bool) – whether whether the sampled object must be freezable

• split (str) – split to sample from. Split “A” specifies all but the last 3 object instances (or the first half - whichever is larger), “B” specifies the rest, and None specifies all.

• obj_registries (tuple) – registries to sample from

• max_size (tuple) – max size of the object. If the sampled object is not within bounds of max size, function will resample

• object_scale (float) – scale of the object. If set will multiply the scale of the sampled object by this value

Returns:

kwargs to apply to the MJCF model for the sampled object

dict: info about the sampled object - the path of the mjcf, groups which the object’s category belongs to, the category of the object the sampling split the object came from, and the groups the object was sampled from

Return type:

dict

EXCLUDE_LAYOUTS = []

SUPPORTED_REWARD_MODES = ['none']

Initialized a Base Kitchen environment.

Parameters:

• robots – Specification for specific robot arm(s) to be instantiated within this env (e.g: “Sawyer” would generate one arm; [“Panda”, “Panda”, “Sawyer”] would generate three robot arms)

• env_configuration (str) – Specifies how to position the robot(s) within the environment. Default is “default”, which should be interpreted accordingly by any subclasses.

• controller_configs (str or list of dict) – If set, contains relevant controller parameters for creating a custom controller. Else, uses the default controller for this specific task. Should either be single dict if same controller is to be used for all robots or else it should be a list of the same length as “robots” param

• base_types (None or str or list of str) – type of base, used to instantiate base models from base factory. Default is “default”, which is the default base associated with the robot(s) the ‘robots’ specification. None results in no base, and any other (valid) model overrides the default base. Should either be single str if same base type is to be used for all robots or else it should be a list of the same length as “robots” param

• gripper_types (None or str or list of str) – type of gripper, used to instantiate gripper models from gripper factory. Default is “default”, which is the default grippers(s) associated with the robot(s) the ‘robots’ specification. None removes the gripper, and any other (valid) model overrides the default gripper. Should either be single str if same gripper type is to be used for all robots or else it should be a list of the same length as “robots” param

• initialization_noise (dict or list of dict) –

  dict containing the initialization noise parameters. The expected keys and corresponding value types are specified below:

  ’magnitude’:

  The scale factor of uni-variate random noise applied to each of a robot’s given initial joint positions. Setting this value to None or 0.0 results in no noise being applied. If “gaussian” type of noise is applied then this magnitude scales the standard deviation applied, If “uniform” type of noise is applied then this magnitude sets the bounds of the sampling range

  ’type’:

  Type of noise to apply. Can either specify “gaussian” or “uniform”

  Should either be single dict if same noise value is to be used for all robots or else it should be a list of the same length as “robots” param

  Note:

  Specifying “default” will automatically use the default noise settings. Specifying None will automatically create the required dict with “magnitude” set to 0.0.

• use_camera_obs (bool) – if True, every observation includes rendered image(s)

• placement_initializer (ObjectPositionSampler) – if provided, will be used to place objects on every reset, else a UniformRandomSampler is used by default.

• has_renderer (bool) – If true, render the simulation state in a viewer instead of headless mode.

• has_offscreen_renderer (bool) – True if using off-screen rendering

• render_camera (str) – Name of camera to render if has_renderer is True. Setting this value to ‘None’ will result in the default angle being applied, which is useful as it can be dragged / panned by the user using the mouse

• render_collision_mesh (bool) – True if rendering collision meshes in camera. False otherwise.

• render_visual_mesh (bool) – True if rendering visual meshes in camera. False otherwise.

• render_gpu_device_id (int) – corresponds to the GPU device id to use for offscreen rendering. Defaults to -1, in which case the device will be inferred from environment variables (GPUS or CUDA_VISIBLE_DEVICES).

• control_freq (float) – how many control signals to receive in every second. This sets the abase of simulation time that passes between every action input.

• horizon (int) – Every episode lasts for exactly @horizon timesteps.

• ignore_done (bool) – True if never terminating the environment (ignore @horizon).

• camera_names (str or list of str) –

  name of camera to be rendered. Should either be single str if same name is to be used for all cameras’ rendering or else it should be a list of cameras to render.

  Note:

  At least one camera must be specified if @use_camera_obs is True.

  Note:

  To render all robots’ cameras of a certain type (e.g.: “robotview” or “eye_in_hand”), use the convention “all-{name}” (e.g.: “all-robotview”) to automatically render all camera images from each robot’s camera list).

• camera_heights (int or list of int) – height of camera frame. Should either be single int if same height is to be used for all cameras’ frames or else it should be a list of the same length as “camera names” param.

• camera_widths (int or list of int) – width of camera frame. Should either be single int if same width is to be used for all cameras’ frames or else it should be a list of the same length as “camera names” param.

• camera_depths (bool or list of bool) – True if rendering RGB-D, and RGB otherwise. Should either be single bool if same depth setting is to be used for all cameras or else it should be a list of the same length as “camera names” param.

• renderer (str) – Specifies which renderer to use.

• renderer_config (dict) – dictionary for the renderer configurations

• init_robot_base_pos (str) – name of the fixture to place the near. If None, will randomly select a fixture.

• seed (int) – environment seed. Default is None, where environment is unseeded, ie. random

• layout_and_style_ids (list of list of int) – list of layout and style ids to use for the kitchen.

• layout_ids ((list of) LayoutType or int) – layout id(s) to use for the kitchen. -1 and None specify all layouts -2 specifies layouts not involving islands/wall stacks, -3 specifies layouts involving islands/wall stacks, -4 specifies layouts with dining areas.

• style_ids ((list of) StyleType or int) – style id(s) to use for the kitchen. -1 and None specify all styles.

• generative_textures (str) – if set to “100p”, will use AI generated textures

• obj_registries (tuple of str) – tuple containing the object registries to use for sampling objects. can contain “objaverse” and/or “aigen” to sample objects from objaverse, AI generated, or both.

• obj_instance_split (str) – string for specifying a custom set of object instances to use. “A” specifies all but the last 3 object instances (or the first half - whichever is larger), “B” specifies the rest, and None specifies all.

• use_distractors (bool) – if True, will add distractor objects to the scene

• translucent_robot (bool) – if True, will make the robot appear translucent during rendering

• randomize_cameras (bool) – if True, will add gaussian noise to the position and rotation of the wrist and agentview cameras

SUPPORTED_ROBOTS = ['fetch', 'none']

Override this to enforce which robots or tuples of robots together are supported in the task. During env creation, setting robot_uids auto loads all desired robots into the scene, but not all tasks are designed to support some robot setups

property _default_human_render_camera_configs

Add default cameras for rendering when using render_mode=’rgb_array’. These can be overriden by the user at env creation time

property _default_sensor_configs

Add default (non-agent) sensors to the environment by returning sensor configurations. These can be overriden by the user at env creation time

property _default_sim_config

property _default_viewer_camera_config

_ep_meta

fixtures_only = False

generative_textures = None

init_robot_base_pos = None

layout_and_style_ids

obj_instance_split = None

obj_registries = ('objaverse',)

placement_initializer = None

randomize_cameras = False

reward_scale = 1.0

reward_shaping = False

translucent_robot = False

use_distractors = False

use_object_obs = True