mani_skill.envs.tasks.digital_twins

Submodules

• mani_skill.envs.tasks.digital_twins.base_env
• mani_skill.envs.tasks.digital_twins.bridge_dataset_eval
• mani_skill.envs.tasks.digital_twins.so100_arm

Classes

PutCarrotOnPlateInScene	Base Digital Twin environment for digital twins of the BridgeData v2
PutEggplantInBasketScene	Base Digital Twin environment for digital twins of the BridgeData v2
PutSpoonOnTableClothInScene	Base Digital Twin environment for digital twins of the BridgeData v2
SO100GraspCubeEnv	Task Description:
StackGreenCubeOnYellowCubeBakedTexInScene	Base Digital Twin environment for digital twins of the BridgeData v2

Package Contents

class mani_skill.envs.tasks.digital_twins.PutCarrotOnPlateInScene(**kwargs)

Bases: mani_skill.envs.tasks.digital_twins.bridge_dataset_eval.base_env.BaseBridgeEnv

Base Digital Twin environment for digital twins of the BridgeData v2

evaluate()

Evaluate whether the environment is currently in a success state by returning a dictionary with a “success” key or a failure state via a “fail” key

This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards.

By default if not overriden this function returns an empty dictionary

get_language_instruction(**kwargs)

objects_excluded_from_greenscreening = ['bridge_carrot_generated_modified', 'bridge_plate_objaverse_larger']

object ids that should not be greenscreened

scene_setting = 'flat_table'

class mani_skill.envs.tasks.digital_twins.PutEggplantInBasketScene(**kwargs)

Bases: mani_skill.envs.tasks.digital_twins.bridge_dataset_eval.base_env.BaseBridgeEnv

Base Digital Twin environment for digital twins of the BridgeData v2

_load_lighting(options)

Loads lighting into the scene. Called by self._reconfigure. If not overriden will set some simple default lighting

evaluate(*args, **kwargs)

Evaluate whether the environment is currently in a success state by returning a dictionary with a “success” key or a failure state via a “fail” key

This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards.

By default if not overriden this function returns an empty dictionary

get_language_instruction(**kwargs)

objects_excluded_from_greenscreening = ['eggplant']

object ids that should not be greenscreened

scene_setting = 'sink'

class mani_skill.envs.tasks.digital_twins.PutSpoonOnTableClothInScene(**kwargs)

Bases: mani_skill.envs.tasks.digital_twins.bridge_dataset_eval.base_env.BaseBridgeEnv

Base Digital Twin environment for digital twins of the BridgeData v2

evaluate()

Evaluate whether the environment is currently in a success state by returning a dictionary with a “success” key or a failure state via a “fail” key

This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards.

By default if not overriden this function returns an empty dictionary

get_language_instruction(**kwargs)

objects_excluded_from_greenscreening = ['table_cloth_generated_shorter', 'bridge_spoon_generated_modified']

object ids that should not be greenscreened

class mani_skill.envs.tasks.digital_twins.SO100GraspCubeEnv(*args, robot_uids='so100', control_mode='pd_joint_target_delta_pos', greenscreen_overlay_path=None, domain_randomization_config=SO100GraspCubeDomainRandomizationConfig(), domain_randomization=True, base_camera_settings=dict(fov=52 * np.pi / 180, pos=[0.5, 0.3, 0.35], target=[0.3, 0.0, 0.1]), spawn_box_pos=[0.3, 0.05], spawn_box_half_size=0.2 / 2, **kwargs)

Bases: mani_skill.envs.tasks.digital_twins.base_env.BaseDigitalTwinEnv

Task Description: A simple task where the objective is to grasp a cube with the SO100 arm and bring it up to a target rest pose.

Randomizations: - the cube’s xy position is randomized on top of a table in a region of size [0.2, 0.2] x [-0.2, -0.2]. It is placed flat on the table - the cube’s z-axis rotation is randomized to a random angle

Success Conditions: - the cube is lifted, grasped, and the robot returns to a rest pose above the surface of the table

Parameters:

domain_randomization_config (Union[SO100GraspCubeDomainRandomizationConfig, dict]) –

_before_control_step()

Code that runs before each action has been taken via env.step(action). On GPU simulation this is called before observations are fetched from the GPU buffers.

_get_obs_agent()

Get observations about the agent’s state. By default it is proprioceptive observations which include qpos and qvel. Controller state is also included although most default controllers do not have any state.

_get_obs_extra(info)

Get task-relevant extra observations. Usually defined on a task by task basis

Parameters:

info (dict) –

_initialize_episode(env_idx, options)

Initialize the episode, e.g., poses of actors and articulations, as well as task relevant data like randomizing goal positions

Parameters:

• env_idx (torch.Tensor) –

• options (dict) –

_load_agent(options)

loads the agent/controllable articulations into the environment. The default function provides a convenient way to setup the agent/robot by a robot_uid (stored in self.robot_uids) without requiring the user to have to write the robot building and controller code themselves. For more advanced use-cases you can override this function to have more control over the agent/robot building process.

Parameters:

• options (dict) – The options for the environment.

• initial_agent_poses (Optional[Union[sapien.Pose, Pose]]) – The initial poses of the agent/robot. Providing these poses and ensuring they are picked such that they do not collide with objects if spawned there is highly recommended to ensure more stable simulation (the agent pose can be changed later during episode initialization).

• build_separate (bool) – Whether to build the agent/robot separately. If True, the agent/robot will be built separately for each parallel environment and then merged together to be accessible under one view/object. This is useful for randomizing physical and visual properties of the agent/robot which is only permitted for articulations built separately in each environment.

_load_lighting(options)

Loads lighting into the scene. Called by self._reconfigure. If not overriden will set some simple default lighting

Parameters:

options (dict) –

_load_scene(options)

Load assets for a digital twin scene in

Parameters:

options (dict) –

compute_dense_reward(obs, action, info)

Compute the dense reward.

Parameters:

• obs (Any) – The observation data. By default the observation data will be in its most raw form, a dictionary (no flattening, wrappers etc.)

• action (torch.Tensor) – The most recent action.

• info (dict) – The info dictionary.

compute_normalized_dense_reward(obs, action, info)

Compute the normalized dense reward.

Parameters:

• obs (Any) – The observation data. By default the observation data will be in its most raw form, a dictionary (no flattening, wrappers etc.)

• action (torch.Tensor) – The most recent action.

• info (dict) – The info dictionary.

evaluate()

Evaluate whether the environment is currently in a success state by returning a dictionary with a “success” key or a failure state via a “fail” key

This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards.

By default if not overriden this function returns an empty dictionary

sample_camera_poses(n)

Parameters:

n (int) –

SUPPORTED_OBS_MODES = ['none', 'state', 'state_dict', 'rgb+segmentation']

The string observation modes the environment supports. Note that “none” and “any_texture” are special keys. none indicates no observation data is generated. “any_texture” indicates that any combination of image textures generated by cameras are supported e.g. rgb+depth, normal+segmentation, albedo+rgb+depth etc. For a full list of supported textures see

SUPPORTED_ROBOTS = ['so100']

Override this to enforce which robots or tuples of robots together are supported in the task. During env creation, setting robot_uids auto loads all desired robots into the scene, but not all tasks are designed to support some robot setups

property _default_human_render_camera_configs

Add default cameras for rendering when using render_mode=’rgb_array’. These can be overriden by the user at env creation time

property _default_sensor_configs

Add default (non-agent) sensors to the environment by returning sensor configurations. These can be overriden by the user at env creation time

property _default_sim_config

agent: mani_skill.agents.robots.so100.so_100.SO100

base_camera_settings

what the camera fov, position and target are when domain randomization is off. DR is centered around these settings

domain_randomization = True

whether randomization is turned on or off.

domain_randomization_config

domain randomization config

spawn_box_half_size = 0.1

spawn_box_pos = [0.3, 0.05]

class mani_skill.envs.tasks.digital_twins.StackGreenCubeOnYellowCubeBakedTexInScene(**kwargs)

Bases: mani_skill.envs.tasks.digital_twins.bridge_dataset_eval.base_env.BaseBridgeEnv

Base Digital Twin environment for digital twins of the BridgeData v2

evaluate()

Evaluate whether the environment is currently in a success state by returning a dictionary with a “success” key or a failure state via a “fail” key

This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards.

By default if not overriden this function returns an empty dictionary

get_language_instruction(**kwargs)

MODEL_JSON = 'info_bridge_custom_baked_tex_v0.json'

objects_excluded_from_greenscreening = ['baked_green_cube_3cm', 'baked_yellow_cube_3cm']

object ids that should not be greenscreened