robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
This release of robosuite refactors our backend to leverage DeepMind's new mujoco bindings. Below, we discuss the key details of this refactoring:
Now, installation has become much simpler, with mujoco being directly installed on Linux or Mac via pip install mujoco. Importing mujoco is now done via import mujoco instead of import mujoco_py
The new DeepMind mujoco bindings do not ship with an onscreen renderer. As a result, we've implented an OpenCV renderer, which provides most of the core functionality from the original mujoco renderer, but has a few limitations (most significantly, no glfw keyboard callbacks and no ability to move the free camera).
The following briefly describes other changes that improve on the pre-existing structure. This is not an exhaustive list, but a highlighted list of changes.
Standardize end-effector frame inference (#25). Now, all end-effector frames are correctly inferred from raw robot XMLs and take into account arbitrary relative orientations between robot arm link frames and gripper link frames.
Improved robot textures (#27). With added support from DeepMind's mujoco bindings for obj texture files, all robots are now natively rendered with more accurate texture maps.
Revamped macros (#30). Macros now references a single macro file that can be arbitrarily specified by the user.
Improved method for specifying GPU ID (#29). The new logic is as follows:
render_device_gpu_id=-1, MUJOCO_EGL_DEVICE_ID and CUDA_VISIBLE_DEVICES are not set, we either choose the first available device (usually 0) if macros.MUJOCO_GPU_RENDERING is True, otherwise use CPU;CUDA_VISIBLE_DEVICES or MUJOCO_EGL_DEVICE_ID are set, we make sure that they dominate over programmatically defined GPU device id.CUDA_VISIBLE_DEVICES and MUJOCO_EGL_DEVICE_ID are both set, then we use MUJOCO_EGL_DEVICE_ID and make sure it is defined in CUDA_VISIBLE_DEVICES
robosuite docs updated
Add new papers
We would like to introduce the newest members of our robosuite core team, all of whom have contributed significantly to this release! @awesome-aj0123 @snasiriany @zhuyifengzju
This release of robosuite brings powerful rendering functionalities including new renderers and multiple vision modalities, in addition to some general-purpose camera utilties. Below, we discuss the key details of these new features:
In addition to the native Mujoco renderer, we present two new renderers, NVISII and iGibson, and introduce a standardized rendering interface API to enable easy swapping of renderers.
NVISII is a high-fidelity ray-tracing renderer originally developed by NVIDIA, and adapted for plug-and-play usage in robosuite. It is primarily used for training perception models and visualizing results in high quality. It can run at up to ~0.5 fps using a GTX 1080Ti GPU. Note that NVISII must be installed (pip install nvisii) in order to use this renderer.
iGibson is a much faster renderer that additionally supports physics-based rendering (PBR) and direct rendering to pytorch tensors. While not as high-fidelity as NVISII, it is incredibly fast and can run at up to ~1500 fps using a GTX 1080Ti GPU. Note that iGibson must be installed (pip install igibson) in order to use this renderer.
With the addition of these new renderers, we also introduce a standardized renderer for easy usage and customization of the various renderers. During each environment step, the renderer updates its internal state by calling update() and renders by calling render(...). The resulting visual observations can be polled by calling get_pixel_obs() or by calling other methods specific to individual renderers. We provide a demo script for testing each new renderer, and our docs also provide additional information on specific renderer details and installation procedures.
In addition to new renderers, we also provide broad support for multiple vision modalities across all (Mujoco, NVISII, iGibson) renderers:
[0, 255]. This is set during environment construction with the use_camera_obs argument.[0, 1]. This is set during environment construction with the camera_depths argument.camera_segmentations argument.In addition to the above modalities, the following modalities are supported by a subset of renderers:
Specific modalities can be set during environment and renderer construction. We provide a demo script for testing the different modalities supported by NVISII and a demo script for testing the different modalities supported by iGibson.
We provide a set of general-purpose camera utilities that intended to enable easy manipulation of environment cameras. Of note, we include transform utilities for mapping between pixel, camera, and world frames, and include a CameraMover class for dynamically moving a camera during simulation, which can be used for many purposes such as the DemoPlaybackCameraMover subclass that enables smooth visualization during demonstration playback.
The following briefly describes other changes that improve on the pre-existing structure. This is not an exhaustive list, but a highlighted list of changes.
Standardize EEF frames (#204). Now, all grippers have identical conventions for plug-and-play usage across types.
Add OSC_POSITION control option for spacemouse (#209).
Improve model class hierarchy for robots. Now, robots own a subset of models (gripper(s), mount(s), etc.), allowing easy external access to the robot's internal model hierarchy.
robosuite docs updated
Add new papers
Fix default OSC orientation control (valid default rotation matrix) (#232)
Fix Jaco self-collisions (#235)
Fix joint velocity controller clipping and tune default kp (#236)
A big thank you to the following community members for spearheading the renderer PRs for this release! @awesome-aj0123 @divyanshj16 @fxia22
This release of robosuite tackles a major challenge of using simulators: real-world transfer! (Sim2Real)
We present two features to significantly improve the sim2real transferrability -- realistic sensor modeling (observables) and control over physics modeling parameters (dynamics randomization).
This standardizes and modularizes how observations are computed and gathered within a given env. Now, all observations received from the env.step() call can be modified programmatically in either a deterministic or stochastic way. Sensor realism has been increased with added support for individual sensor sampling rates, corruption, delay, and filtering. The OTS behavior (obs dict structure, default no corruption / delay / filtering) has been preserved for backwards compatibility.
Each Observable owns its own _sensor, _corrupter, _delayer, and _filter functions, which are used to process new data computed during its update() call which is called after every simulation timestep, NOT policy step! (Note, however that a new value is only computed once per sampling period, NOT at every update() call). Its functionality is described briefly described below:
_sensor: Arbitrary function that takes in an observation cache and computes the "raw" (potentially ground truth) value for the given observable. It can potentially leverage pre-computed values from the observation cache to compute its output. The @sensor decorator is provided to denote this type of function, and guarantees a modality for this sensor as well._corrupter: Arbitrary function that takes in the output from _sensor and outputs the corrupted data._delayer: Arbitrary function that takes no arguments and outputs a float time value (in seconds), denoting how much delay should be applied to the next sampling cycle_filter: Arbitrary function that takes in the output of _corrupter and outputs the filtered data.All of the above can either be (re-)defined at initialization or during runtime. Utility functions have been provided in the base.py environment module to easily interact with all observables owned by the environment.
Some standard corrupter and delayer function generators are provided ([deterministic / uniform / gaussian] [corruption / delay]), including dummy no-ops for standard functions. All of this can be found in observables.py, and has been heavily documented.
An example script demo'ing the new functionality can be found in demo_sensor_corruption.py.
Physical parameters governing the underlying physics model can now be modified in real-time via the DynamicsModder class in mjmod.py. This modder allows mid-sim randomization of the following supported properties, sorted by element group (for more information, please see Mujoco XML Reference)
density: Density of the medium (i.e.: air)viscosity: Viscosity of the medium (i.e.: air)position: (x, y, z) Position of the body relative to its parent bodyquaternion: (qw, qx, qy, qz) Quaternion of the body relative to its parent bodyinertia: (ixx, iyy, izz) diagonal components of the inertia matrix associated with this bodymass: mass of the bodyfriction: (sliding, torsional, rolling) friction values for this geomsolref: (timeconst, dampratio) contact solver values for this geomsolimp: (dmin, dmax, width, midpoint, power) contact solver impedance values for this geomstiffness: Stiffness for this jointfrictionloss: Friction loss associated with this jointdamping: Damping value for this jointarmature: Gear inertia for this jointThe new DynamicsModder follows the same basic API as the other Modder classes, and allows per-parameter and per-group randomization enabling. Apart from randomization, this modder can also be instantiated to selectively modify values at runtime. Detailed information can be found on our docs page along with an informative example script.
The following briefly describes other changes that improve on the pre-existing structure. This is not an exhaustive list, but a highlighted list of changes.
robosuite docs have been completely overhauled! Hopefully no more broken links or outdated APIs (#159)
Tuned parallel jaw grippers (PandaGripper and RethinkGripper) for improved grasp stability and significantly reduced sliding
Tuned default gains for teleoperation when using keyboard device
Added small frictionloss, damping, and armature values to all robot joints by default to improve stability and reduce no-op drift over time
Simulation now uses realistic values for air density and viscosity
tune_camera.py is more flexible: can now take string names as inputs and also modify any camera active in the simulation (e.g.: eye_in_hand!)
Add macros for automatically concatenating image frames from different cameras
Add new papers
Improve documentation semantics
Fix interpolator dimension bug (#181)
Fixed xml object naming bug (#150)
Fixed deterministic action playback (however, a caveat, and a test script to test this) (#178)
Fix contact geom bug with Jaco, and a test script to test contact geoms for all robots (#180)
Fix OSC position default orientation and when to update orientation goals
Fix OSC orientation to be consistent with global coordinate axes frame of reference
Fix spacemouse import bug (#186)
A big thank you to the following community members for contributing PRs for this release! @hermanjakobsen @zhuyifengzju
While most surface-level functionality hasn't changed, the underlying infrastructure has been heavily reworked to reduce redundancy, improve standardization and ease-of-usage, and future-proof against expected expansions. Specifically, the following standards were pursued:
MujocoModel)self.object and self.object_name, just have self.object, since it already includes its own name reference in self.object.name.This is not an exhaustive list, but includes the key features / changes in this PR most relevant to the common user that should greatly streamline environment prototyping and debugging.
Now, all (robot, gripper, object) models inherit from the MujocoModel class, which defines many useful properties and methods, including references to the model joints, contact geoms, important sites, etc. This allows much more standardized usage of these models when designing environments.
We do not expect robosuite to remain solely manipulation-based. Therefore, all environment properties and methods common to manipulation-based domains were ported to ManipulationEnv, allowing future robot task domains to be added with little reworking. Similarly, common properties / methods common to Single or TwoArm environments were ported to SingleArmEnv and TwoArmEnv, respectively. This both (a) removes much redundant code between top-level env classes, and (b) frees users to focus exclusively on the environment prototyping unique to their use case without having to duplicate much boilerplate code. So, for example, Lift now has a class hierarchy of MujocoEnv --> RobotEnv --> ManipulationEnv --> SingleArmEnv --> Lift. Note that similar changes were made to the Robot and RobotModel base classes.
All object classes now are derived from MujocoObject, which itself is a subclass of MujocoModel. This standardizes the interface across all object source modalities (Generated vs. XML based), and provides the user with an expected set of properties that can be leveraged when prototyping custom environments. Additionally, complex, procedural object generation has been added with the CompositeObject class, of now which the HammerObject and PotWithHandles object are now subclasses of (as examples of how to design custom composite objects).
CompositeObject and CompositeBodyObject classes have now been added. A CompositeObject is composed of multiple geoms, and a CompositeBodyObject is composed of multiple objects (bodies). Together, this allows for complex, procedural generation of arbitrary object shapes with potentially dynamic joint interactions. The HammerObject and PotWithHandlesObject are examples of the CompositeObject class, and HingedBoxObject is an example of the CompositeBodyObject class.
All collision geoms now belong to group 0, while visual geoms belong to group 1. This means that methods can automatically check for the geom type by polling it's group attribute from its element or during sim. Moreover, all collision geoms are assigned solid rgba colors based on their semantic role (e.g.: robot vs. gripper vs. arena vs. objects). If rendering onscreen, you can easily toggle visualizing the visual and collision geoms by pressing 1 or 0, respectively. This can be useful for debugging environments and making sure collision bodies are formed / interacting as expected.
Because of this improved structure, many methods can now take advantage of this standardization. Some especially relevant methods are discussed briefly below:
env.get_contacts(model) (any env): This method will return the set of all geoms currently in contact with the inputted model. This is useful for debugging environments, or checking to see if certain conditions are met when designing rewards / interactions.
env._check_grasp(gripper, object_geoms) (only manipulation envs): This method will return True if the inputted gripper is grasping the object specified by object_geoms, which could be a MujocoModel or simply a list of geoms that define the object. This makes it very easy to design environments that depend on certain grasping requirements being met.
env._gripper_to_target(gripper, target, ...) and env._visualize_gripper_to_target(gripper, target, ...) (only manipulation envs): Methods to help streamline getting relevant distance info between a gripper gripper and target. Target can be a MujocoModel or any specific element (body, geom, site) name. The former calculates the distance, while the latter will set the gripper eef site sphere's color to be proportional to the distance to target. Both are useful for environment prototyping and debugging.
model.set_sites_visibility(sim, visible) (any MujocoModel): This method will set all the sites belonging to model in the current sim to either be visible or not depending on the visible arg. This is useful for quick debugging or teleoperation, to aid the user in visualizing specific points of reference in sim.
The following briefly describes other changes that improve on the pre-existing structure. Again, this is not an exhaustive list, but a highlighted list of changes.
MountModel class added; pedestals used by robots are now assigned to this class and added to a RobotModel in a similar fashion to how the GripperModel is added. This allows abstraction of the robot model from its base mount model.
Abstracted site visualizations to a wrapper (VisualizationWrapper). This wrapper provides fine-grained control over sites being visualized within the environment: can specify whether to visualize site groups belonging to the wrapped env. This is controlled via keywords provided by a given environment. For example, for ManipulationEnv classes, this includes gripper, robot, and env keys, each of which control its associated site visualization.
Added openGL and openCV image convention option as a macro
Added macros.py in robosuite.utils and single file to store all macros for our repo. This includes numba macros and now includes instance randomization and image convention macros. Users can modify these macros mid-script by importing the macros module and modifying the module-level vars directly.
Placement samplers were no longer belong to Task class, but are separate. This is more intuitive, and allows for more modularity when designing future Task subclasses. Moreover, the placement sampler classes were refactored for more intuitive usage.
Refactor all top-level environments in a standardized fashion
Add functionality to modify cameras from Arena class; tuned cameras for Door, TwoArmHandover/PegInHole tasks
Renamed / modified a bunch of stuff so it's more semantically accurate / intuitive
Tuned Wipe environment with alternate compressed object observation space (this is enabled by default) and default environment parameters, such as table height / size and wipe marker sampling locations.
Update GymWrapper class to be more robust to general usage -- now, automatically flattens image observations so that it is Gym-compatible and also extends from the Gym Env class directly.
Add GPU device arg in environments for setups with multiple GPUs
Add new papers (#118)
Improve documentation
Fixed grasping bug where a grasp is incorrectly inferred if a robot's two fingers are touching an object. This resulted in incorrect rewards being received which could negatively impact reward-based training. Grasps are now inferred correctly so robot cannot "cheat" a grasping-based reward.
Fixed singular value problem with OSC controller (#136). Control loop computations now utilizes numpy.pinv instead of our implementation of it.
Fix absolute control and control limits setting for OSC controller.
Fix model XML saving method. We now use env.sim.model.get_model() instead of env.model.get_model() so that we don't save a stale version of the current simulation snapshot.
Fix wiping gripper mass (too high before, leading to bad force-torque sensor readings)
Fix agentview cameras for Door, TwoArmHandover, and TwoArmPegInHole environments (#123)
Fix OSC controller bug that doesn't automatically re-update initial goal orientation upon reset
The first major version of robosuite. For more information, please check out https://robosuite.ai
v0.2.0 release with MuJoCo 1.5 support
Initial release of Surreal Robotics Suite