mani_skill.agents.base_real_agent#

Classes#

BaseRealAgent

Base agent class for representing real robots, sensors, and controlling them in a real environment. This generally should be used with the mani_skill.envs.sim2real_env.Sim2RealEnv class for deploying policies learned in simulation

Module Contents#

class mani_skill.agents.base_real_agent.BaseRealAgent(sensor_configs=dict())[source]#

Base agent class for representing real robots, sensors, and controlling them in a real environment. This generally should be used with the mani_skill.envs.sim2real_env.Sim2RealEnv class for deploying policies learned in simulation to the real world.

Parameters:: sensor_configs (dict[str, BaseSensorConfig]) – the sensor configs to create the agent with.

__getattr__(name)[source]#: Delegate attribute access to self._sim_agent if the attribute doesn’t exist in self. This allows accessing sim_agent properties and methods directly from the real agent. Some simulation agent include convenience functions to access e.g. end-effector poses or various properties of the robot.

abstract capture_sensor_data(sensor_names=None)[source]#

Capture the sensor data asynchronously from the agent based on the given sensor names. If sensor_names is None then all sensor data should be captured. This should not return anything and should be async if possible.

Parameters:: sensor_names (Optional[list[str]]) –

get_proprioception()[source]#

Get the proprioceptive state of the agent, default is the qpos and qvel of the robot and any controller state.

Note that if qpos or qvel functions are not implemented they will return None.

abstract get_qpos()[source]#: Get the current joint positions in radians of the agent as a torch tensor. Data should have a batch dimension, the shape should be (1, N) for N joint positions.

abstract get_qvel()[source]#: Get the current joint velocities in radians/s of the agent as a torch tensor. Data should have a batch dimension, the shape should be (1, N) for N joint velocities.

abstract get_sensor_data(sensor_names=None)[source]#

Get the desired sensor observations from the agent based on the given sensor names. If sensor_names is None then all sensor data should be returned. The expected format for cameras is in line with the simulation’s format for cameras.

{
    "sensor_name": {
        "rgb": torch.uint8 (1, H, W, 3), # red green blue image colors
        "depth": torch.int16 (1, H, W, 1), # depth in millimeters
    }
}

whether rgb or depth is included depends on the real camera and can be omitted if not supported or not used. Note that a batch dimension is expected in the data.

For more details see https://maniskill.readthedocs.io/en/latest/user_guide/concepts/sensors.html in order to ensure the real data aligns with simulation formats.

Parameters:: sensor_names (Optional[list[str]]) –

get_sensor_params(sensor_names=None)[source]#

Get the parameters of the desired sensors based on the given sensor names. If sensor_names is None then all sensor parameters should be returned. The expected format for cameras is in line with the simulation’s format is:

{
    "sensor_name": {
        "cam2world_gl": [4, 4], # transformation from the camera frame to the world frame (OpenGL/Blender convention)
        "extrinsic_cv": [4, 4], # camera extrinsic (OpenCV convention)
            "intrinsic_cv": [3, 3], # camera intrinsic (OpenCV convention)
    }
}

If these numbers are not needed/unavailable it is okay to leave the fields blank. Some observation processing modes may need these fields however such as point clouds in the world frame.

Parameters:: sensor_names (list[str]) –

abstract reset(qpos)[source]#

Reset the agent to a given qpos. For real robots this function should move the robot at a safe and controlled speed to the given qpos and aim to reach it accurately. :param qpos: the qpos in radians to reset the agent to.

Parameters:: qpos (mani_skill.utils.structs.types.Array) –

abstract set_target_qpos(qpos)[source]#

Set the target joint positions of the agent. :param qpos: the joint positions in radians to set the agent to.

Parameters:: qpos (mani_skill.utils.structs.types.Array) –

abstract set_target_qvel(qvel)[source]#

Set the target joint velocities of the agent. :param qvel: the joint velocities in radians/s to set the agent to.

Parameters:: qvel (mani_skill.utils.structs.types.Array) –

abstract start()[source]#

Start the agent, which include turning on the motors/robot, setting up cameras/sensors etc.

For sensors you have access to self.sensor_configs which is the requested sensor setup. For e.g. cameras these sensor configs will define the camera resolution.

For sim2real/real2sim alignment when defining real environment interfaces we instantiate the real agent with the simulation environment’s sensor configs.

abstract stop()[source]#: Stop the agent, which include turning off the motors/robot, stopping cameras/sensors etc.

_sim_agent: mani_skill.agents.base_agent.BaseAgent = None[source]#

property controller[source]#

property qpos[source]#: Get the current joint positions of the agent.

property qvel[source]#: Get the current joint velocities of the agent.

robot[source]#: a reference to a fake robot/articulation used for accessing qpos/qvel values.

sensor_configs[source]#

mani_skill.agents.base_real_agent#

Classes#

Module Contents#

This Page