Abstract
Predictive models can be particularly helpful for robots to effectively manipulate terrains in construction sites and extraterrestrial surfaces. However, terrain state representations become extremely high-dimensional especially to capture fine-resolution details and when depth is unknown or unbounded. This paper introduces a learning-based approach for terrain dynamics modeling and manipulation, leveraging the Graph-based Neural Dynamics (GBND) framework to represent terrain deformation as motion of a graph of particles. Based on the principle that the moving portion of a terrain is usually localized, our approach builds a large terrain graph (potentially millions of particles) but only identifies a very small active subgraph (hundreds of particles) for predicting the outcomes of robot-terrain interaction. To minimize the size of the active subgraph we introduce a learning-based approach that identifies a small region of interest (RoI) based on the robot's control inputs and the current scene. We also introduce a novel domain boundary feature encoding that allows GBNDs to perform accurate dynamics prediction in the RoI interior while avoiding particle penetration through RoI boundaries. Our proposed method is both orders of magnitude faster than naïve GBND and it achieves better overall prediction accuracy. We further evaluated our framework on excavation and shaping tasks on terrain with different granularity.
Method Overview
Previous graph-based neural dynamics (GBND) models often applied to tasks with hard boundaries, e.g., table top. However, no such rigid boundary exists in terrains. Instead, a not strictly-enforced soft boundary presents, where the trrain outside the boundary is not affected by robot-terrain interaction due to energy dissipation. Our method learns to predict the energy dissipation boundary and rollout particles within the boundary to simulate terrain dynamics under robot-terrain interaction on a per-step basis.
Efficiency and Effectiveness
High quality energy dissipation boundary prediction brings extrodinary prediction accuracy because GBND tends to make mistakes by predicting small motions of non-moving particles, i.e., jittering. Having particles outside the boundary not feed into GBND provides a strong inductive bias for our model, which is crucial, and not presented in previous GBND models, nor simple methods such as naïvely select a non-adaptive boundary. Moreover, tight boundary prediction also significantly reduces the number of particles to be rolled out, which leads to a much faster prediction speed, essentially benefits downstream sampling based planning algorithms, e.g., MPPI.
Our method (left) only predicts the dynamics of the highlighted particles in the predicted RoI. Accuracy maintained.
With around 1 hour of automatic real world interaction, our framework can easily finetune the model to adapt to different terrains, e.g., pebbles, sand, etc. We evaluate the real world dynamics prediction both quantitatively and qualitatively as shown below. Dynamics rollouts are performed in an open-loop manner.
Pebbles
Prediction (left) and Ground Truth (right)
Sand
Prediction (left) and Ground Truth (right)
Planning
Planning on learned GBND models is straightforward. We can adapt weighted sampling algorithms, e.g., MPPI, to sample control sequences and rollout in parallel. The best control sequence is selected based on certain metric, e.g., difference between goal heightmap and current scene heightmap.
Pebbles, Hole
Pebbles, Moat
Sand, Hole
Sand, Moat
Better dynamics model, better planning
Pebbles, Hole
Pebbles, Moat
Sand, Hole
Sand, Moat