mani_skill.envs.tasks.digital_twins.base_env
============================================

.. py:module:: mani_skill.envs.tasks.digital_twins.base_env


Classes
-------

.. autoapisummary::

   mani_skill.envs.tasks.digital_twins.base_env.BaseDigitalTwinEnv


Module Contents
---------------

.. py:class:: BaseDigitalTwinEnv(**kwargs)

   Bases: :py:obj:`mani_skill.envs.sapien_env.BaseEnv`


   Base Environment class for easily setting up evaluation digital twins for real2sim and sim2real

   This is based on the [SIMPLER](https://simpler-env.github.io/) and currently has the following tricks for
   making accurate simulated environments of real world datasets

   Greenscreening: Add a greenscreened real image to the background to make the images more realistic and closer to the distribution
   of real world data. To use the functionality in your own custom task you can do the following:

   .. code-block:: python

       class MyTask(BaseDigitalTwinEnv):
           def __init__(self, **kwargs):
               self.rgb_overlay_paths = {"camera_name": "path/to/greenscreen/image.png"}
               super().__init__(**kwargs)
           def _load_scene(self, options: dict):
               # load your objects as usual e.g. a cube at self.cube

               # exclude the robot and cube from the greenscreen process
               self.remove_object_from_greenscreen(self.robot)
               self.remove_object_from_greenscreen(self.cube)


   Use `self.remove_object_from_greenscreen(object: Actor | Link | Articulation)` to exclude those objects from the greenscreen process.


   .. py:method:: _after_reconfigure(options)

      Add code here that should run immediately after self._reconfigure is called. The torch RNG context is still active so RNG is still
      seeded here by self._episode_seed. This is useful if you need to run something that only happens after reconfiguration but need the
      GPU initialized so that you can check e.g. collisons, poses etc.



   .. py:method:: _get_obs_sensor_data(apply_texture_transforms = True)

      Get data from all registered sensors. Auto hides any objects that are designated to be hidden

      :param apply_texture_transforms: Whether to apply texture transforms to the simulated sensor data to map to standard texture formats. Default is True.
      :type apply_texture_transforms: bool

      :returns: A dictionary containing the sensor data mapping sensor name to its respective dictionary of data. The dictionary maps texture names to the data. For example the return could look like

                .. code-block:: python

                    {
                        "sensor_1": {
                            "rgb": torch.Tensor,
                            "depth": torch.Tensor
                        },
                        "sensor_2": {
                            "rgb": torch.Tensor,
                            "depth": torch.Tensor
                        }
                    }
      :rtype: dict



   .. py:method:: _green_sceen_rgb(rgb, segmentation, overlay_img)

      returns green screened RGB data given a batch of RGB and segmentation images and one overlay image



   .. py:method:: _load_scene(options)

      Load assets for a digital twin scene in



   .. py:method:: remove_object_from_greenscreen(object)

      remove an actor/articulation/link from the greenscreen process



   .. py:property:: _default_human_render_camera_configs

      Add default cameras for rendering when using render_mode='rgb_array'. These can be overriden by the user at env creation time


   .. py:property:: _default_sim_config


   .. py:attribute:: _objects_to_remove_from_greenscreen
      :type:  list[Union[mani_skill.utils.structs.actor.Actor, mani_skill.utils.structs.link.Link]]
      :value: []


      list of articulations/actors/links that should be removed from the greenscreen process


   .. py:attribute:: _rgb_overlay_images
      :type:  dict[str, torch.Tensor]

      dict mapping camera name to the image torch tensor


   .. py:attribute:: _segmentation_ids_to_keep
      :type:  torch.Tensor
      :value: None


      torch tensor of segmentation ids that reference the objects that should not be greenscreened


   .. py:attribute:: rgb_overlay_mode
      :type:  str
      :value: 'background'


      which RGB overlay mode to use during the greenscreen process. The default is 'background' which enables greenscreening like normal. The other option is 'debug' mode which
      will make the opacity of the original render and greenscreen overlay both 50%. The third option is "none" which will not perform any greenscreening.


   .. py:attribute:: rgb_overlay_paths
      :type:  dict[str, str]
      :value: None


      dict mapping camera name to the file path of the greenscreening image