mani_skill.agents.base_real_agent#

class mani_skill.agents.base_real_agent.BaseRealAgent(sensor_configs: Dict[str, BaseSensorConfig] = {})[source]#

Bases: object

Base agent class for representing real robots, sensors, and controlling them in a real environment. This generally should be used with the mani_skill.envs.sim2real_env.Sim2RealEnv class for deploying policies learned in simulation to the real world.

Parameters:: sensor_configs (Dict[str, BaseSensorConfig]) – the sensor configs to create the agent with.

capture_sensor_data(sensor_names: List[str] | None = None)[source]#: Capture the sensor data asynchronously from the agent based on the given sensor names. If sensor_names is None then all sensor data should be captured. This should not return anything and should be async if possible.

property controller#

get_proprioception()[source]#

Get the proprioceptive state of the agent, default is the qpos and qvel of the robot and any controller state.

Note that if qpos or qvel functions are not implemented they will return None.

get_qpos()[source]#: Get the current joint positions in radians of the agent as a torch tensor. Data should have a batch dimension, the shape should be (1, N) for N joint positions.

get_qvel()[source]#: Get the current joint velocities in radians/s of the agent as a torch tensor. Data should have a batch dimension, the shape should be (1, N) for N joint velocities.

get_sensor_data(sensor_names: List[str] | None = None)[source]#

Get the desired sensor observations from the agent based on the given sensor names. If sensor_names is None then all sensor data should be returned. The expected format for cameras is in line with the simulation’s format for cameras.

{
    "sensor_name": {
        "rgb": torch.uint8 (1, H, W, 3), # red green blue image colors
        "depth": torch.int16 (1, H, W, 1), # depth in millimeters
    }
}

whether rgb or depth is included depends on the real camera and can be omitted if not supported or not used. Note that a batch dimension is expected in the data.

For more details see https://maniskill.readthedocs.io/en/latest/user_guide/concepts/sensors.html in order to ensure the real data aligns with simulation formats.

get_sensor_params(sensor_names: List[str] | None = None)[source]#

Get the parameters of the desired sensors based on the given sensor names. If sensor_names is None then all sensor parameters should be returned. The expected format for cameras is in line with the simulation’s format is:

{
    "sensor_name": {
        "cam2world_gl": [4, 4], # transformation from the camera frame to the world frame (OpenGL/Blender convention)
        "extrinsic_cv": [4, 4], # camera extrinsic (OpenCV convention)
            "intrinsic_cv": [3, 3], # camera intrinsic (OpenCV convention)
    }
}

If these numbers are not needed/unavailable it is okay to leave the fields blank. Some observation processing modes may need these fields however such as point clouds in the world frame.

property qpos#: Get the current joint positions of the agent.

property qvel#: Get the current joint velocities of the agent.

reset(qpos: Tensor | ndarray | Sequence)[source]#: Reset the agent to a given qpos. For real robots this function should move the robot at a safe and controlled speed to the given qpos and aim to reach it accurately. :param qpos: the qpos in radians to reset the agent to.

robot#: a reference to a fake robot/articulation used for accessing qpos/qvel values.

set_target_qpos(qpos: Tensor | ndarray | Sequence)[source]#: Set the target joint positions of the agent. :param qpos: the joint positions in radians to set the agent to.

set_target_qvel(qvel: Tensor | ndarray | Sequence)[source]#: Set the target joint velocities of the agent. :param qvel: the joint velocities in radians/s to set the agent to.

start()[source]#

Start the agent, which include turning on the motors/robot, setting up cameras/sensors etc.

For sensors you have access to self.sensor_configs which is the requested sensor setup. For e.g. cameras these sensor configs will define the camera resolution.

For sim2real/real2sim alignment when defining real environment interfaces we instantiate the real agent with the simulation environment’s sensor configs.

stop()[source]#: Stop the agent, which include turning off the motors/robot, stopping cameras/sensors etc.

mani_skill.agents.base_real_agent#

This Page