mani_skill.envs.tasks.digital_twins.base_env

Classes

BaseDigitalTwinEnv

Base Environment class for easily setting up evaluation digital twins for real2sim and sim2real

Module Contents

class mani_skill.envs.tasks.digital_twins.base_env.BaseDigitalTwinEnv(**kwargs)

Bases: mani_skill.envs.sapien_env.BaseEnv

Base Environment class for easily setting up evaluation digital twins for real2sim and sim2real

This is based on the [SIMPLER](https://simpler-env.github.io/) and currently has the following tricks for making accurate simulated environments of real world datasets

Greenscreening: Add a greenscreened real image to the background to make the images more realistic and closer to the distribution of real world data. To use the functionality in your own custom task you can do the following:

class MyTask(BaseDigitalTwinEnv):
    def __init__(self, **kwargs):
        self.rgb_overlay_paths = {"camera_name": "path/to/greenscreen/image.png"}
        super().__init__(**kwargs)
    def _load_scene(self, options: dict):
        # load your objects as usual e.g. a cube at self.cube

        # exclude the robot and cube from the greenscreen process
        self.remove_object_from_greenscreen(self.robot)
        self.remove_object_from_greenscreen(self.cube)

Use self.remove_object_from_greenscreen(object: Actor | Link | Articulation) to exclude those objects from the greenscreen process.

_after_reconfigure(options)

Add code here that should run immediately after self._reconfigure is called. The torch RNG context is still active so RNG is still seeded here by self._episode_seed. This is useful if you need to run something that only happens after reconfiguration but need the GPU initialized so that you can check e.g. collisons, poses etc.

Parameters:

options (dict) –

_get_obs_sensor_data(apply_texture_transforms=True)

Get data from all registered sensors. Auto hides any objects that are designated to be hidden

Parameters:

apply_texture_transforms (bool) – Whether to apply texture transforms to the simulated sensor data to map to standard texture formats. Default is True.

Returns:

A dictionary containing the sensor data mapping sensor name to its respective dictionary of data. The dictionary maps texture names to the data. For example the return could look like

{
    "sensor_1": {
        "rgb": torch.Tensor,
        "depth": torch.Tensor
    },
    "sensor_2": {
        "rgb": torch.Tensor,
        "depth": torch.Tensor
    }
}

Return type:

dict

_green_sceen_rgb(rgb, segmentation, overlay_img)

returns green screened RGB data given a batch of RGB and segmentation images and one overlay image

_load_scene(options)

Load assets for a digital twin scene in

Parameters:

options (dict) –

remove_object_from_greenscreen(object)

remove an actor/articulation/link from the greenscreen process

Parameters:

object (Union[mani_skill.utils.structs.articulation.Articulation, mani_skill.utils.structs.actor.Actor, mani_skill.utils.structs.link.Link]) –

property _default_human_render_camera_configs

Add default cameras for rendering when using render_mode=’rgb_array’. These can be overriden by the user at env creation time

property _default_sim_config

_objects_to_remove_from_greenscreen: list[mani_skill.utils.structs.actor.Actor | mani_skill.utils.structs.link.Link] = []

list of articulations/actors/links that should be removed from the greenscreen process

_rgb_overlay_images: dict[str, torch.Tensor]

dict mapping camera name to the image torch tensor

_segmentation_ids_to_keep: torch.Tensor = None

torch tensor of segmentation ids that reference the objects that should not be greenscreened

rgb_overlay_mode: str = 'background'

which RGB overlay mode to use during the greenscreen process. The default is ‘background’ which enables greenscreening like normal. The other option is ‘debug’ mode which will make the opacity of the original render and greenscreen overlay both 50%. The third option is “none” which will not perform any greenscreening.

rgb_overlay_paths: dict[str, str] = None

dict mapping camera name to the file path of the greenscreening image