mani_skill.envs.tasks.mobile_manipulation ========================================= .. py:module:: mani_skill.envs.tasks.mobile_manipulation Submodules ---------- .. toctree:: :maxdepth: 1 /api/mani_skill/envs/tasks/mobile_manipulation/open_cabinet_drawer/index /api/mani_skill/envs/tasks/mobile_manipulation/robocasa/index Classes ------- .. autoapisummary:: mani_skill.envs.tasks.mobile_manipulation.OpenCabinetDoorEnv mani_skill.envs.tasks.mobile_manipulation.OpenCabinetDrawerEnv mani_skill.envs.tasks.mobile_manipulation.RoboCasaKitchenEnv Package Contents ---------------- .. py:class:: OpenCabinetDoorEnv(*args, robot_uids='fetch', robot_init_qpos_noise=0.02, reconfiguration_freq=None, num_envs=1, **kwargs) Bases: :py:obj:`OpenCabinetDrawerEnv` **Task Description:** Use the Fetch mobile manipulation robot to move towards a target cabinet and open the target drawer out. **Randomizations:** - Robot is randomly initialized 1.6 to 1.8 meters away from the cabinet and positioned to face it - Robot's base orientation is randomized by -9 to 9 degrees - The cabinet selected to manipulate is randomly sampled from all PartnetMobility cabinets that have drawers - The drawer to open is randomly sampled from all drawers available to open **Success Conditions:** - The drawer is open at least 90% of the way, and the angular/linear velocities of the drawer link are small **Goal Specification:** - 3D goal position centered at the center of mass of the handle mesh on the drawer to open (also visualized in human renders with a sphere). .. py:attribute:: TRAIN_JSON .. py:attribute:: handle_types :value: ['revolute', 'revolute_unwrapped'] .. py:class:: OpenCabinetDrawerEnv(*args, robot_uids='fetch', robot_init_qpos_noise=0.02, reconfiguration_freq=None, num_envs=1, **kwargs) Bases: :py:obj:`mani_skill.envs.sapien_env.BaseEnv` **Task Description:** Use the Fetch mobile manipulation robot to move towards a target cabinet and open the target drawer out. **Randomizations:** - Robot is randomly initialized 1.6 to 1.8 meters away from the cabinet and positioned to face it - Robot's base orientation is randomized by -9 to 9 degrees - The cabinet selected to manipulate is randomly sampled from all PartnetMobility cabinets that have drawers - The drawer to open is randomly sampled from all drawers available to open **Success Conditions:** - The drawer is open at least 90% of the way, and the angular/linear velocities of the drawer link are small **Goal Specification:** - 3D goal position centered at the center of mass of the handle mesh on the drawer to open (also visualized in human renders with a sphere). .. py:method:: _after_control_step() Code that runs after each action has been taken. On GPU simulation this is called right before observations are fetched from the GPU buffers. .. py:method:: _after_reconfigure(options) Add code here that should run immediately after self._reconfigure is called. The torch RNG context is still active so RNG is still seeded here by self._episode_seed. This is useful if you need to run something that only happens after reconfiguration but need the GPU initialized so that you can check e.g. collisons, poses etc. .. py:method:: _get_obs_extra(info) Get task-relevant extra observations. Usually defined on a task by task basis .. py:method:: _initialize_episode(env_idx, options) Initialize the episode, e.g., poses of actors and articulations, as well as task relevant data like randomizing goal positions .. py:method:: _load_agent(options) loads the agent/controllable articulations into the environment. The default function provides a convenient way to setup the agent/robot by a robot_uid (stored in self.robot_uids) without requiring the user to have to write the robot building and controller code themselves. For more advanced use-cases you can override this function to have more control over the agent/robot building process. :param options: The options for the environment. :type options: dict :param initial_agent_poses: The initial poses of the agent/robot. Providing these poses and ensuring they are picked such that they do not collide with objects if spawned there is highly recommended to ensure more stable simulation (the agent pose can be changed later during episode initialization). :type initial_agent_poses: Optional[Union[sapien.Pose, Pose]] :param build_separate: Whether to build the agent/robot separately. If True, the agent/robot will be built separately for each parallel environment and then merged together to be accessible under one view/object. This is useful for randomizing physical and visual properties of the agent/robot which is only permitted for articulations built separately in each environment. :type build_separate: bool .. py:method:: _load_cabinets(joint_types) .. py:method:: _load_scene(options) Loads all objects like actors and articulations into the scene. Called by `self._reconfigure`. Given options argument is the same options dictionary passed to the self.reset function .. py:method:: compute_dense_reward(obs, action, info) Compute the dense reward. :param obs: The observation data. By default the observation data will be in its most raw form, a dictionary (no flattening, wrappers etc.) :type obs: Any :param action: The most recent action. :type action: torch.Tensor :param info: The info dictionary. :type info: dict .. py:method:: compute_normalized_dense_reward(obs, action, info) Compute the normalized dense reward. :param obs: The observation data. By default the observation data will be in its most raw form, a dictionary (no flattening, wrappers etc.) :type obs: Any :param action: The most recent action. :type action: torch.Tensor :param info: The info dictionary. :type info: dict .. py:method:: evaluate() Evaluate whether the environment is currently in a success state by returning a dictionary with a "success" key or a failure state via a "fail" key This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards. By default if not overriden this function returns an empty dictionary .. py:method:: handle_link_positions(env_idx = None) .. py:attribute:: SUPPORTED_ROBOTS :value: ['fetch'] Override this to enforce which robots or tuples of robots together are supported in the task. During env creation, setting robot_uids auto loads all desired robots into the scene, but not all tasks are designed to support some robot setups .. py:attribute:: TRAIN_JSON .. py:property:: _default_human_render_camera_configs Add default cameras for rendering when using render_mode='rgb_array'. These can be overriden by the user at env creation time .. py:property:: _default_sensor_configs Add default (non-agent) sensors to the environment by returning sensor configurations. These can be overriden by the user at env creation time .. py:property:: _default_sim_config .. py:attribute:: _sample_video_link :value: 'https://github.com/haosulab/ManiSkill/raw/main/figures/environment_demos/OpenCabinetDrawer-v1_rt.mp4' a link to a sample video of the task. This is mostly used for automatic documentation generation .. py:attribute:: agent :type: Union[mani_skill.agents.robots.Fetch] .. py:attribute:: all_model_ids .. py:attribute:: handle_types :value: ['prismatic'] .. py:attribute:: min_open_frac :value: 0.75 .. py:attribute:: robot_init_qpos_noise :value: 0.02 .. py:class:: RoboCasaKitchenEnv(*args, robot_uids='fetch', env_configuration='default', controller_configs=None, gripper_types='default', base_types='default', initialization_noise='default', use_camera_obs=True, use_object_obs=True, reward_scale=1.0, reward_shaping=False, placement_initializer=None, has_renderer=False, has_offscreen_renderer=True, render_camera='robot0_agentview_center', render_collision_mesh=False, render_visual_mesh=True, render_gpu_device_id=-1, control_freq=20, horizon=1000, ignore_done=False, camera_names='agentview', camera_heights=256, camera_widths=256, camera_depths=False, renderer='mujoco', renderer_config=None, init_robot_base_pos=None, seed=None, layout_and_style_ids=None, layout_ids=None, style_ids=None, scene_split=None, generative_textures=None, obj_registries=('objaverse', ), obj_instance_split=None, use_distractors=False, translucent_robot=False, randomize_cameras=False, fixtures_only=False, **kwargs) Bases: :py:obj:`mani_skill.envs.sapien_env.BaseEnv` Superclass for ManiSkill environments. :param num_envs: number of parallel environments to run. By default this is 1, which means a CPU simulation is used. If greater than 1, then we initialize the GPU simulation setup. Note that not all environments are faster when simulated on the GPU due to limitations of GPU simulations. For example, environments with many moving objects are better simulated by parallelizing across CPUs. :param obs_mode: observation mode to be used. Must be one of ("state", "state_dict", "none", "sensor_data", "rgb", "depth", "segmentation", "rgbd", "rgb+depth", "rgb+depth+segmentation", "rgb+segmentation", "depth+segmentation", "pointcloud") The obs_mode is mostly for convenience to automatically optimize/setup all sensors/cameras for the given observation mode to render the correct data and try to ignore unnecessary rendering. For the most advanced use cases (e.g. you have 1 RGB only camera and 1 depth only camera) :param reward_mode: reward mode to use. Must be one of ("normalized_dense", "dense", "sparse", "none"). With "none" the reward returned is always 0 :param control_mode: control mode of the agent. "*" represents all registered controllers, and the action space will be a dict. :param render_mode: render mode registered in @SUPPORTED_RENDER_MODES. :param shader_dir: shader directory. Defaults to None. Setting this will override the shader used for all cameras in the environment. This is legacy behavior kept for backwards compatibility. The proper way to change the shaders used for cameras is to either change the environment code or pass in sensor_configs/human_render_camera_configs with the desired shaders. Previously the options are "default", "rt", "rt-fast". "rt" means ray-tracing which results in more photorealistic renders but is slow, "rt-fast" is a lower quality but faster version of "rt". :type shader_dir: Optional[str] :param enable_shadow: whether to enable shadow for lights. Defaults to False. :type enable_shadow: bool :param sensor_configs: configurations of sensors to override any environment defaults. If the key is one of sensor names (e.g. a camera), the config value will be applied to the corresponding sensor. Otherwise, the value will be applied to all sensors (but overridden by sensor-specific values). For possible configurations see the documentation see :doc:`the sensors documentation `. :type sensor_configs: dict :param human_render_camera_configs: configurations of human rendering cameras to override any environment defaults. Similar usage as @sensor_configs. :type human_render_camera_configs: dict :param viewer_camera_configs: configurations of the viewer camera in the GUI to override any environment defaults. Similar usage as @sensor_configs. :type viewer_camera_configs: dict :param robot_uids: list of robots to instantiate and control in the environment. :type robot_uids: Union[str, BaseAgent, list[Union[str, BaseAgent]]] :param sim_config: Configurations for simulation if used that override the environment defaults. If given a dictionary, it can just override specific attributes e.g. ``sim_config=dict(scene_config=dict(solver_iterations=25))``. If passing in a SimConfig object, while typed, will override every attribute including the task defaults. Some environments define their own recommended default sim configurations via the ``self._default_sim_config`` attribute that generally should not be completely overriden. :type sim_config: Union[SimConfig, dict] :param reconfiguration_freq: How frequently to call reconfigure when environment is reset via `self.reset(...)` Generally for most users who are not building tasks this does not need to be changed. The default is 0, which means the environment reconfigures upon creation, and never again. :type reconfiguration_freq: int :param sim_backend: By default this is "auto". If sim_backend is "auto", then if ``num_envs == 1``, we use the PhysX CPU sim backend, otherwise we use the PhysX GPU sim backend and automatically pick a GPU to use. Can also be "physx_cpu" or "physx_cuda" to force usage of a particular sim backend. To select a particular GPU to run the simulation on, you can pass "physx_cuda:n" where n is the ID of the GPU, similar to the way PyTorch selects GPUs. Note that if this is "physx_cpu", num_envs can only be equal to 1. :type sim_backend: str :param render_backend: By default this is "gpu". If render_backend is "gpu" or it's alias "sapien_cuda", then we auto select a GPU to render with. It can be "sapien_cuda:n" where n is the ID of the GPU to render with. If this is "cpu" or "sapien_cpu", then we try to render on the CPU. If this is "none" or None, then we disable rendering. Note that some environments may require rendering functionalities to work. Moreover it is sometimes difficult to determine before running an environment if your machine can render or not. If you encounter some issue with rendering you can first try to double check your NVIDIA drivers / Vulkan drivers are setup correctly. If you don't need to do rendering you can simply disable it by setting render_backend to "none" or None. :type render_backend: str :param parallel_in_single_scene: By default this is False. If True, rendered images and the GUI will show all objects in one view. This is only really useful for generating cool videos showing all environments at once but it is not recommended otherwise as it slows down simulation and rendering. :type parallel_in_single_scene: bool :param enhanced_determinism: By default this is False and env resets will reset the episode RNG only when a seed / seed list is given. If True, the environment will reset the episode RNG upon each reset regardless of whether a seed is provided. Generally enhanced_determinisim is not needed and users are recommended to pass seeds into the env reset function instead. :type enhanced_determinism: bool .. py:method:: _get_obs_extra(info) Get task-relevant extra observations. Usually defined on a task by task basis .. py:method:: _initialize_episode(env_idx, options) Initialize the episode, e.g., poses of actors and articulations, as well as task relevant data like randomizing goal positions .. py:method:: _load_agent(options) loads the agent/controllable articulations into the environment. The default function provides a convenient way to setup the agent/robot by a robot_uid (stored in self.robot_uids) without requiring the user to have to write the robot building and controller code themselves. For more advanced use-cases you can override this function to have more control over the agent/robot building process. :param options: The options for the environment. :type options: dict :param initial_agent_poses: The initial poses of the agent/robot. Providing these poses and ensuring they are picked such that they do not collide with objects if spawned there is highly recommended to ensure more stable simulation (the agent pose can be changed later during episode initialization). :type initial_agent_poses: Optional[Union[sapien.Pose, Pose]] :param build_separate: Whether to build the agent/robot separately. If True, the agent/robot will be built separately for each parallel environment and then merged together to be accessible under one view/object. This is useful for randomizing physical and visual properties of the agent/robot which is only permitted for articulations built separately in each environment. :type build_separate: bool .. py:method:: _load_scene(options) Loads all objects like actors and articulations into the scene. Called by `self._reconfigure`. Given options argument is the same options dictionary passed to the self.reset function .. py:method:: _setup_kitchen_references() setup fixtures (and their references). this function is called within load_model function for kitchens .. py:method:: evaluate() Evaluate whether the environment is currently in a success state by returning a dictionary with a "success" key or a failure state via a "fail" key This function may also return additional data that has been computed (e.g. is the robot grasping some object) that may be reused when generating observations and rewards. By default if not overriden this function returns an empty dictionary .. py:method:: register_fixture_ref(ref_name, fn_kwargs) Registers a fixture reference for later use. Initializes the fixture if it has not been initialized yet. :param ref_name: name of the reference :type ref_name: str :param fn_kwargs: keyword arguments to pass to get_fixture :type fn_kwargs: dict :returns: fixture object :rtype: Fixture .. py:method:: sample_object(groups, exclude_groups=None, graspable=None, microwavable=None, washable=None, cookable=None, freezable=None, split=None, obj_registries=None, max_size=(None, None, None), object_scale=None, rng=None) Sample a kitchen object from the specified groups and within max_size bounds. :param groups: groups to sample from or the exact xml path of the object to spawn :type groups: list or str :param exclude_groups: groups to exclude :type exclude_groups: str or list :param graspable: whether the sampled object must be graspable :type graspable: bool :param washable: whether the sampled object must be washable :type washable: bool :param microwavable: whether the sampled object must be microwavable :type microwavable: bool :param cookable: whether whether the sampled object must be cookable :type cookable: bool :param freezable: whether whether the sampled object must be freezable :type freezable: bool :param split: split to sample from. Split "A" specifies all but the last 3 object instances (or the first half - whichever is larger), "B" specifies the rest, and None specifies all. :type split: str :param obj_registries: registries to sample from :type obj_registries: tuple :param max_size: max size of the object. If the sampled object is not within bounds of max size, function will resample :type max_size: tuple :param object_scale: scale of the object. If set will multiply the scale of the sampled object by this value :type object_scale: float :returns: kwargs to apply to the MJCF model for the sampled object dict: info about the sampled object - the path of the mjcf, groups which the object's category belongs to, the category of the object the sampling split the object came from, and the groups the object was sampled from :rtype: dict .. py:attribute:: EXCLUDE_LAYOUTS :value: [] .. py:attribute:: SUPPORTED_REWARD_MODES :value: ['none'] Initialized a Base Kitchen environment. :param robots: Specification for specific robot arm(s) to be instantiated within this env (e.g: "Sawyer" would generate one arm; ["Panda", "Panda", "Sawyer"] would generate three robot arms) :param env_configuration: Specifies how to position the robot(s) within the environment. Default is "default", which should be interpreted accordingly by any subclasses. :type env_configuration: str :param controller_configs: If set, contains relevant controller parameters for creating a custom controller. Else, uses the default controller for this specific task. Should either be single dict if same controller is to be used for all robots or else it should be a list of the same length as "robots" param :type controller_configs: str or list of dict :param base_types: type of base, used to instantiate base models from base factory. Default is "default", which is the default base associated with the robot(s) the 'robots' specification. None results in no base, and any other (valid) model overrides the default base. Should either be single str if same base type is to be used for all robots or else it should be a list of the same length as "robots" param :type base_types: None or str or list of str :param gripper_types: type of gripper, used to instantiate gripper models from gripper factory. Default is "default", which is the default grippers(s) associated with the robot(s) the 'robots' specification. None removes the gripper, and any other (valid) model overrides the default gripper. Should either be single str if same gripper type is to be used for all robots or else it should be a list of the same length as "robots" param :type gripper_types: None or str or list of str :param initialization_noise: dict containing the initialization noise parameters. The expected keys and corresponding value types are specified below: :`'magnitude'`: The scale factor of uni-variate random noise applied to each of a robot's given initial joint positions. Setting this value to `None` or 0.0 results in no noise being applied. If "gaussian" type of noise is applied then this magnitude scales the standard deviation applied, If "uniform" type of noise is applied then this magnitude sets the bounds of the sampling range :`'type'`: Type of noise to apply. Can either specify "gaussian" or "uniform" Should either be single dict if same noise value is to be used for all robots or else it should be a list of the same length as "robots" param :Note: Specifying "default" will automatically use the default noise settings. Specifying None will automatically create the required dict with "magnitude" set to 0.0. :type initialization_noise: dict or list of dict :param use_camera_obs: if True, every observation includes rendered image(s) :type use_camera_obs: bool :param placement_initializer: if provided, will be used to place objects on every reset, else a UniformRandomSampler is used by default. :type placement_initializer: ObjectPositionSampler :param has_renderer: If true, render the simulation state in a viewer instead of headless mode. :type has_renderer: bool :param has_offscreen_renderer: True if using off-screen rendering :type has_offscreen_renderer: bool :param render_camera: Name of camera to render if `has_renderer` is True. Setting this value to 'None' will result in the default angle being applied, which is useful as it can be dragged / panned by the user using the mouse :type render_camera: str :param render_collision_mesh: True if rendering collision meshes in camera. False otherwise. :type render_collision_mesh: bool :param render_visual_mesh: True if rendering visual meshes in camera. False otherwise. :type render_visual_mesh: bool :param render_gpu_device_id: corresponds to the GPU device id to use for offscreen rendering. Defaults to -1, in which case the device will be inferred from environment variables (GPUS or CUDA_VISIBLE_DEVICES). :type render_gpu_device_id: int :param control_freq: how many control signals to receive in every second. This sets the abase of simulation time that passes between every action input. :type control_freq: float :param horizon: Every episode lasts for exactly @horizon timesteps. :type horizon: int :param ignore_done: True if never terminating the environment (ignore @horizon). :type ignore_done: bool :param camera_names: name of camera to be rendered. Should either be single str if same name is to be used for all cameras' rendering or else it should be a list of cameras to render. :Note: At least one camera must be specified if @use_camera_obs is True. :Note: To render all robots' cameras of a certain type (e.g.: "robotview" or "eye_in_hand"), use the convention "all-{name}" (e.g.: "all-robotview") to automatically render all camera images from each robot's camera list). :type camera_names: str or list of str :param camera_heights: height of camera frame. Should either be single int if same height is to be used for all cameras' frames or else it should be a list of the same length as "camera names" param. :type camera_heights: int or list of int :param camera_widths: width of camera frame. Should either be single int if same width is to be used for all cameras' frames or else it should be a list of the same length as "camera names" param. :type camera_widths: int or list of int :param camera_depths: True if rendering RGB-D, and RGB otherwise. Should either be single bool if same depth setting is to be used for all cameras or else it should be a list of the same length as "camera names" param. :type camera_depths: bool or list of bool :param renderer: Specifies which renderer to use. :type renderer: str :param renderer_config: dictionary for the renderer configurations :type renderer_config: dict :param init_robot_base_pos: name of the fixture to place the near. If None, will randomly select a fixture. :type init_robot_base_pos: str :param seed: environment seed. Default is None, where environment is unseeded, ie. random :type seed: int :param layout_and_style_ids: list of layout and style ids to use for the kitchen. :type layout_and_style_ids: list of list of int :param layout_ids: layout id(s) to use for the kitchen. -1 and None specify all layouts -2 specifies layouts not involving islands/wall stacks, -3 specifies layouts involving islands/wall stacks, -4 specifies layouts with dining areas. :type layout_ids: (list of) LayoutType or int :param style_ids: style id(s) to use for the kitchen. -1 and None specify all styles. :type style_ids: (list of) StyleType or int :param generative_textures: if set to "100p", will use AI generated textures :type generative_textures: str :param obj_registries: tuple containing the object registries to use for sampling objects. can contain "objaverse" and/or "aigen" to sample objects from objaverse, AI generated, or both. :type obj_registries: tuple of str :param obj_instance_split: string for specifying a custom set of object instances to use. "A" specifies all but the last 3 object instances (or the first half - whichever is larger), "B" specifies the rest, and None specifies all. :type obj_instance_split: str :param use_distractors: if True, will add distractor objects to the scene :type use_distractors: bool :param translucent_robot: if True, will make the robot appear translucent during rendering :type translucent_robot: bool :param randomize_cameras: if True, will add gaussian noise to the position and rotation of the wrist and agentview cameras :type randomize_cameras: bool .. py:attribute:: SUPPORTED_ROBOTS :value: ['fetch', 'none'] Override this to enforce which robots or tuples of robots together are supported in the task. During env creation, setting robot_uids auto loads all desired robots into the scene, but not all tasks are designed to support some robot setups .. py:property:: _default_human_render_camera_configs Add default cameras for rendering when using render_mode='rgb_array'. These can be overriden by the user at env creation time .. py:property:: _default_sensor_configs Add default (non-agent) sensors to the environment by returning sensor configurations. These can be overriden by the user at env creation time .. py:property:: _default_sim_config .. py:property:: _default_viewer_camera_config .. py:attribute:: _ep_meta .. py:attribute:: fixtures_only :value: False .. py:attribute:: generative_textures :value: None .. py:attribute:: init_robot_base_pos :value: None .. py:attribute:: layout_and_style_ids .. py:attribute:: obj_instance_split :value: None .. py:attribute:: obj_registries :value: ('objaverse',) .. py:attribute:: placement_initializer :value: None .. py:attribute:: randomize_cameras :value: False .. py:attribute:: reward_scale :value: 1.0 .. py:attribute:: reward_shaping :value: False .. py:attribute:: translucent_robot :value: False .. py:attribute:: use_distractors :value: False .. py:attribute:: use_object_obs :value: True