University of Toronto,Aeva Inc,Aeva,Aeva, Inc

Frequency-Modulated Continuous-Wave (FMCW) lidar is a recently emerging technology that additionally enables per-return instantaneous relative radial velocity measurements via the Doppler effect. In this letter, we present the first continuous-time lidar-only odometry algorithm using these Doppler velocity measurements from an FMCW lidar to aid odometry in geometrically degenerate environments. We apply an existing continuous-time framework that efficiently estimates the vehicle trajectory using Gaussian process regression to compensate for motion distortion due to the scanning-while-moving nature of any mechanically actuated lidar (FMCW and non-FMCW). We evaluate our proposed algorithm on several real-world datasets, including publicly available ones and datasets we collected. Our algorithm outperforms the only existing method that also uses Doppler velocity measurements, and we study difficult conditions where including this extra information greatly improves performance. We additionally demonstrate state-of-the-art performance of lidar-only odometry with and without using Doppler velocity measurements in nominal conditions. Code for this project can be found at: https://github.com/utiasASRL/steam_icp.

Data,,, CSIRO,University of Sydney, NVIDIA,ETH Zurich

Quantification of uncertainty in point cloud matching is critical in many tasks such as pose estimation, sensor fusion, and grasping. Iterative closest point (ICP) is a commonly used pose estimation algorithm which provides a point estimate of the transformation between two point clouds. There are many sources of uncertainty in this process that may arise due to sensor noise, ambiguous environment, initial condition, and occlusion. However, for safety critical problems such as autonomous driving, a point estimate of the pose transformation is not sufficient as it does not provide information about the multiple solutions. Current probabilistic ICP methods usually do not capture all sources of uncertainty and may provide unreliable transformation estimates which can have a detrimental effect in state estimation or decision making tasks that use this information. In this work we propose a new algorithm to align two point clouds that can precisely estimate the uncertainty of ICP's transformation parameters. We develop a Stein variational inference framework with gradient based optimization of ICP's cost function. The method provides a non-parametric estimate of the transformation, can model complex multi-modal distributions, and can be effectively parallelized on a GPU. Experiments using 3D kinect data as well as sparse indoor/outdoor LiDAR data show that our method is capable of efficiently producing accurate pose uncertainty estimates.

Apple Inc.,Universidad de Zaragoza,I,A. Universidad de Zaragoza

Deformable Monocular SLAM algorithms recover the localization of a camera in an unknown deformable environment. Current approaches use a template-based deformable tracking to recover the camera pose and the deformation of the map. These template-based methods use an underlying global deformation model. In this paper, we introduce a novel deformable camera tracking method with a local deformation model for each point. Each map point is defined as a single textured surfel that moves independently of the other map points. Thanks to a direct photometric error cost function, we can track the position and orientation of the surfel without an explicit global deformation model. In our experiments, we validate the proposed system and observe that our local deformation model estimates more accurately the targeted deformations of the map in both laboratory-controlled experiments and in-body scenarios undergoing quasi-isometric deformations, with changing topology or discontinuities.

National University of Defense Technology,Peking University

High-dimensional nonlinear state estimation is at the heart of inertial-aided navigation systems (INS). Traditional methods usually rely on good initialization and find difficulty in handling large inter-frame transformations due to fast camera motion. We opt to tackle these challenges by solving the depth inertial odometry (DIO) problem with random optimization. To address the exponentially increased amount of candidate states sampled for the high-dimensional state space, we propose a highly efficient variant of random optimization based on the idea of active subspace. Our method identifies the active dimensions which contribute the most significantly to the decrease of the cost function in each iteration, and samples candidate states only within the corresponding subspace. This allows us to efficiently explore the 18D state space of DIO and achieve good optimality by sampling and evaluating only thousands of candidate states. Experiments show that our method attains highly robust and accurate DIO under fast camera motions and low light conditions, without needing a slow-motion warm-up for initialization.

Massachusetts Institute of Technology,MIT

We describe a general approach for maximum a posteriori (MAP) inference in a class of discrete-continuous factor graphs commonly encountered in robotics applications. While there are openly available tools providing flexible and easy-to-use interfaces for specifying and solving inference problems formulated in terms of either discrete or continuous graphical models, at present, no similarly general tools exist enabling the same functionality for hybrid discrete-continuous problems. We aim to address this problem. In particular, we provide a library, DC-SAM, extending existing tools for inference problems defined in terms of factor graphs to the setting of discrete-continuous models. A key contribution of our work is a novel solver for efficiently recovering approximate solutions to discrete-continuous inference problems. The key insight to our approach is that while joint inference over continuous and discrete state spaces is often hard, many commonly encountered discrete-continuous problems can naturally be split into a “discrete part” and a “continuous part” that can individually be solved easily. Leveraging this structure, we optimize discrete and continuous variables in an alternating fashion. In consequence, our proposed work enables straightforward representation of and approximate inference in discrete-continuous graphical models. We also provide a method to approximate the uncertainty in estimates of both discrete and continuous variables.

idriverplus.com,Beijing Idriverplus Technology Co. Ltd.,Tsinghua University,,,,,,,,,,,,,,,,,,,,Beijing Idriverplus Technology Co., Ltd.

Iterated Extended Kalman Filter is a promising and widely-used estimator for real-time localization applications. It iterates the observation equation to find a better linearization point and, simultaneously, only maintains the state estimation in a single time to save the computation resources. Inspired by the recent development of the iterative closest point algorithm, this paper investigates an acceleration approach to the iterations in iterative error state Kalman filters (IESKFs). We show that the IESKF can be seen as a fixed point problem, and the Anderson acceleration (AA) can be elegantly applied to the iterations of IESKF since the error state naturally lies in the tangent space and does not require additional transforms. However, the tangent space of the current estimation may change during the iterations, so we should switch the tangent space to the starting point to perform Anderson acceleration. We propose the AA-IEKF and apply it to the lidar-inertial odometry (LIO) systems to estimate the ego-motion of a lidar. The experiments show that the Anderson acceleration can efficiently reduce the number of iterations in ESKF and achieve a lower computational cost.

Nagoya University Graduate School,National Institute of Advanced Industrial Science and Technology,Nat. Inst. of Advanced Industrial Science and Technology,National Institute of Advanced Industrial Science and Technology (AIST),National Instisute of Advanced Industrial Science and Technology

This paper presents a LiDAR odometry estimation framework called Generalized LOAM. Our proposed method is generalized in that it can seamlessly fuse various local geometric shapes around points to improve the position estimation accuracy compared to the conventional LiDAR odometry and mapping (LOAM) method. To utilize continuous geometric features for LiDAR odometry estimation, we incorporate tiny neural networks into a generalized iterative closest point (GICP) algorithm. These neural networks improve the data association metric and the matching cost function using local geometric features. Experiments with the KITTI benchmark demonstrate that our proposed method reduces relative trajectory errors compared to the GICP and LOAM methods.

Shenyang Institute of Automation Chinese Academy of Sciences,National University of Defense Technology,Shenyang Institute of Automation, Chinese Academy of Sciences,Shenyang Institute of Automation，Chinese Academy of Scien

Loop closing is a fundamental part of simultaneous localization and mapping (SLAM) for autonomous mobile systems. In the field of visual SLAM, bag of words (BoW) has achieved great success in loop closure. The BoW features for loop searching can also be used in the subsequent 6-DoF loop correction. However, for 3D LiDAR SLAM, the state-of-the-art methods may fail to effectively recognize the loop in real time, and usually cannot correct the full 6-DoF loop pose. To address this limitation, we present a novel Bag of Words for real-time loop closing in 3D LiDAR SLAM, called BoW3D. Our method not only efficiently recognizes the revisited loop places, but also corrects the full 6-DoF loop pose in real time. BoW3D builds the bag of words based on the 3D LiDAR feature LinK3D, which is efficient, pose-invariant and can be used for accurate point-to-point matching. We furthermore embed our proposed method into 3D LiDAR odometry system to evaluate loop closing performance. We test our method on public dataset, and compare it against other state-of-the-art algorithms. Our BoW3D shows better performance in terms of F1 max and extended precision scores in most scenarios with superior real-time performance. It is noticeable that BoW3D takes an average of 50 ms to recognize and correct the loops on KITTI 00 (includes 4K+ 64-ray LiDAR scans), when executed on a notebook with an Intel Core i7 @2.2 GHz processor. We release the implementation of our method here: https://github.com/YungeCui/

In this letter, we propose a novel method to merge a Gaussian mixture on matrix Lie groups and present its application for a simultaneous localization and mapping problem with symmetric objects. The key idea is to predetermine the weighted mean called a midway point and merge Gaussian mixture components at the associated tangent space. Through this rule, the covariance matrix captures the original density more accurately, and the need for the back-projection is spared when compared to the conventional merge. We highlight the midway-merge by numerically evaluating dissimilarity metrics of density functions before and after the merge on the rotational group. Furthermore, we experimentally discover that the rotational error of symmetric objects follows heavy-tailed behavior and formulate the Gaussian sum filter to model it by a Gaussian mixture noise. The effectiveness of our approach is validated through virtual and real-world datasets.

Although research studies in pneumatic soft robots develop rapidly, most pneumatic actuators are still controlled by rigid valves and conventional electronics. The existence of these rigid, electronic components sacrifices the compliance and adaptability of soft robots. Current electronics-free valve designs based on soft materials are facing challenges in behaviour consistency, design flexibility, and fabrication complexity. Taking advantages of soft material 3D printing, this paper presents a new design of a bi-stable pneumatic valve, which utilises two soft, pneumatically-driven, and symmetrically-oriented conical shells with structural bistability to stabilise and regulate the airflow. The critical pressure required to operate the valve can be adjusted by changing the design features of the soft bi-stable structure. Multi-material printing simplifies the valve fabrication, enhances the flexibility in design feature optimisations, and improves the system repeatability. In this work, both a theoretical model and physical experiments are introduced to examine the relationships between the critical operating pressure and the key design features. Results with valve characteristic tuning via material stiffness changing show better effectiveness compared to the change of geometry design features (demonstrated largest tunable critical pressure range from 15.3 to 65.2 kPa and fastest response time

Purdue University,Gensler Baltimore,Purdue

Programmable surfaces (PSs) consist of a 2D array of actuators that can deform in the third dimension, providing the ability to create continuous 3D profiles. Discrete PSs can be realized using an array of independent solid linear actuators. Continuous PSs consist of actuators that are mechanically coupled, providing deformation states that are more similar to real surfaces with reduced complexity of the control electronics. However, continuous PSs have been limited in size by the lack of the control systems required to take into account the complex internal coupling between actuators in the array. In this work, we computationally explore the deformation of a fully continuous PS with 81 independent actuation pixels based on ionic bending actuator. We establish a control strategy using machine learning (ML) regression models. Both forward and inverse control are achieved based on the training datasets which are derived from the finite element analysis (FEA) data of our PS. The prediction of surface deformation achieved by forward control with accuracy under 1% is 15000 times faster than FEM. And the real-time inverse control of continuous PSs that is to reproduce any arbitrary pre-defined surfaces, which possess high practical value for tactile display or human-machine interactive devices, is first proposed in the letter.

In this letter, we propose a physics-based framework to exploit magnets in robotic manipulation. More specifically, we suggest equipping soft and underactuated hands with magnetic elements, which can generate a magnetic actuation able to synergistically interact with tendon-driven and pneumatic actuations, engendering a complementarity that enriches the capabilities of the actuation system. Magnetic elements can act as additional Degrees of Actuation (DoAs), robustifying the motion control of the device and augmenting the hand manipulation capabilities. We investigate the interaction of a soft hand with itself for enriching possible hand shaping, and the interaction of the hand with the environment for enriching possible grasping capabilities. Physics laws and notions reported in the manuscript can be used as a guidance for DoAs augmentation and can provide tools for the design of novel soft hands.

Origami processes can generate both rigid and compliant structures from the same homogeneous sheet material. We advance the origami robotics literature by showing that it is possible to construct an arbitrary rigid kinematic chain with prescribed joint compliance from a single tubular sheet. Our "Kinegami" algorithm converts a Denavit-Hartenberg specification into a single-sheet crease pattern for an equivalent serial robot mechanism by composing origami modules from a catalogue. The algorithm arises from the key observation that tubular origami linkage design reduces to a Dubins path planning problem. The automatically generated structural connections and movable joints that realize the specified design can also be endowed with independent user-specified compliance. We apply the Kinegami algorithm to a number of common robot mechanisms and hand-fold their algorithmically generated single-sheet crease patterns into functioning kinematic chains. We believe this is the first completely automated end-to-end system for converting an abstract manipulator specification into a physically realizable origami design that requires no additional human input.

Southwest Research Institute; Arizona State University,Arizona State University

A gait entrainment study was conducted using a lightweight soft robotic hip exosuit (SR-HExo) that can apply perturbations at the hip joint during treadmill walking. Periodic perturbations were applied by flat fabric Pneumatic Artificial Muscle actuators starting at a subject’s preferred gait frequency and increasing up to 15% higher in 3% increments. Anterior hip flexion perturbations and posterior hip extension perturbations were tested in two separate experiments. All 11 healthy participants showed successful entrainment in all 12 experimental conditions (i.e., from preferred gait frequency to 15% higher in both flexion and extension perturbation directions). This study confirmed that there exists a single stable point attractor during gait entrainment to unilateral, unidirectional hip perturbations, which is consistent with previous ankle studies. Phase-locking was consistently observed around toe-off phase of the gait cycle (GC). Group averaged results showed gait synchronization with extension perturbations occurred earlier in the gait cycle (around 50% GC where the hip angle reaches maximum extension) than with flexion perturbations (just after 60% GC where the transition from maximum hip extension towards hip flexion occurs). Other gait entrainment characteristics (success rate of entrainment, basin of entrainment, and transient response) observed in this study posits the potential of the SR-HExo for entrainment-based gait training in rehabilitation contexts.

UNIVERSIDAD DE SEVILLA,Universidad de Sevilla,University of Seville

This paper presents the first design of a soft, 3D-printed in flexible filament, lightweight UAV, capable of performing full-body perching using soft tendons, specifically landing and stabilizing on pipelines and irregular surfaces without the need for an auxiliary system. The flexibility of the UAV can be controlled during the additive manufacturing process by adjusting the infill rate distribution. However, the increase in flexibility implies difficulties in controlling the UAV, as well as structural, aerodynamic, and aeroelastic effects. This article provides insight into the dynamics of the system and validates the flyability of the vehicle for densities as low as 6%. Within this range, quasi-static arm deformations can be considered, thus the autopilot is fed back through a static arm deflection model. At lower densities, strong non-linear elastic dynamics appear, which translates to complex modeling, and it is suggested to switch to data-based approaches.

Shandong University,Volvo Construction Equipment Technology (China) Co., Ltd,shandong university,Harbin Institute of Technology

Most current in-pipe robots are usually designed for pipes of a specific size. In this paper, we propose a novel inchworm-like in-pipe robot based on the concept of tensegrity for moving in pipes with varying diameters. Firstly, a tensegrity-based robotic module capable of two kinds of shape change is designed. One kind is extension in the axial direction accompanied by contrac-tion in the radial direction, which is the basis for the wave-like crawling movement of the in-pipe robot. The other kind is ex-pansion in the radial direction while keeping changeless in the axial direction, enabling the module adaptable to pipes with different diameters. Then, the geometrical equilibrium configu-ration of the tensegrity module is determined, followed by kinematic analysis using force density method. By cascading three modules, the in-pipe crawling robot is developed. Finally, a series of experiments are performed to test the shape change-ability and friction force of the tensegrity module, and the mo-bility, load capacity, and adaptability of the in-pipe robot. The results validate that the robot can crawl in horizontal pipes, vertical pipes, and elbow pipes under the control of a simple actuation sequence. Furthermore, the robot has the abilities to adapt to pipes with different diameters varying from 100 mm to 180 mm. It is suggested that the usage of tensegrity structures brings about higher adaptability, flexibility, and mobility to the in-pipe crawling robot.

Mobile robots that can adapt to an extensive range of terrains play essential roles in many applications. Millipedes are one of the most terrain-adaptive creatures in nature due to their multi-legged locomotion and flexible body. Inspired by natural millipedes, we report an untethered robotic millipede with a 6-segments soft-rigid hybrid body that can actively bend and 24 legs driven by low-pressure microfluidic actuators. The 24 microfluidic actuators are driven by two independent low-pressure sources from miniature pumps, which allows the untethered locomotion of the robotic millipede in small size (length, 23 cm; width, 5 cm; height, 4 cm) and lightweight (150 g). Using a pre-defined gait for the multi-legs, the robotic millipede can locomote with a maximum speed of 30.96 cm/min (1.35 body length per minute) and a minimum turning radius of 15 cm (0.65 body length). Experiments also demonstrated that the robot was able to locomote effectively in various uneven terrains. Utilizing its passive or active mode of its flexible body, the robot could also achieve adaptive moves. The robotic millipede has the potential to perform a variety of environment exploration tasks by remotely controlling and transmitting real time images wirelessly.

Vrije Universiteit Brussels,Vrije Universiteit Brussel,Vrije Universiteit Brussel (VUB)

Soft robotics modeling is a research topic that is evolving fast. Many techniques are present in literature but most of them require analytical models with a lot of equations that are time-consuming, hard to resolve, and not so easy to handle. For this reason, the help of a soft mechanics simulator is essential in this field. In fact, this paper presents a tutorial on how to build a soft-robot model using an open-source Finite Element Analysis (FEA) simulator, called SOFA. This software is able to generate a simulation scene from a code written in Python or XML, so it can be used by people that with different fields of competence like mechanical knowledge, knowledge of material properties and programming skills. As a case study, a Python simulation of a cable-driven soft actuator that makes contact with a rigid object is considered. The basic working principles of SOFA required to make a scene are explained step by step. In particular, it shows how to simulate the mechanics and animate the bending behavior of the actuator. Furthermore, it will be shown also how to retrieve and save data from simulation, demonstrating that SOFA can easily adapt to a multi-disciplinary subject as the research in soft-robotics, but also be useful for teaching simulation and programming language principles to engineering students.

Tomoya Takahashi, Masahiro Watanabe, Kazuki Abe, Kenjiro Tadakuma, Naoto Saiki, Masashi Konyo, Satoshi Tadokoro

Inflatable structures used in soft robotics applications have unique characteristics. In particular, the tip-extension structure, which extends the structure from its tip, can grow without creating friction with the environment. However, these inflatable structures need high pressure to maintain their stiffness under various conditions. Excessive inner pressure limits their application in that it prevents the structure from maintaining its curved shape and from complying with specifications. This study aimed to simultaneously lower the pressure and increase the rigidity of the structure. Our work resulted in the proposal of a mechanism that combines a skeleton structure consisting of multi-joint links with functions to increase the rigidity. Insertion of this mechanism into an inflatable structure obviates the need for high inner pressure, yet enables the structure to bend and maintain the intended shape. We devised a design based on rigid articulated links and combined it with a membrane structure that utilizes the advantages of the tip-extension structure. The experimental results show that the payload of the structure designed to operate at low pressure increases compared to that of the membrane-only structure. The findings of this research can be applied to long robots that can be extended into open space without drooping and to mechanisms that enable structures to wrap around the human body.

Ben Gurion University of the Negev,BGU,Ben Gurion University

This paper analyzes the crawling locomotion of a wave-like robot in curved tubes. We use an energy-based approach to determine the optimal crawling orientation of the robot that minimizes the surface energy while advancing. The results showed that the robot rotated its body along the roll direction so that the wave motion would be in the same plane as the curvature plane of the tube. The incorporation of a passive bending joint along the plane of the wave motion decreased the surface energy and enhanced the robot’s ability to advance in even tighter curves. Given these findings we designed and manufactured two new robots with either one or two passive bending joints. We molded custom flexible surfaces and tubes and experimentally tested our robots in them. These validating experiments indicated that the bending joints substantially improved the robots’ ability to traverse curved tubes (see video).

Toyohashi University of Technology,Kyushu University,Okayama University

This paper proposes a palm-sized omnidirectional mobile robot with two torus wheels. A single torus wheel is made of an elastic elongated coil spring in which the two ends of the coil connected each other and is driven by a piezoelectric actuator (stator) that can generate 2-degrees-of-freedom (axial and angular) motions. The stator converts its thrust force and torque into longitudinal and meridian motions of the torus wheel, respectively, making the torus work as an omnidirectional wheel on a plane. In this paper, we build a control system of a piezo-driven 2-degrees-of-freedom torus wheel and evaluate its performance measures, such as the transient characteristics, the orientation accuracy and the payload capacity. An omnidirectional robot with the two torus wheels is constructed, and the feedback control for a desired planar motion is demonstrated. The design inspired by a ring torus represents the possibility toward the creation of an unprecedentedly simple, light, and compact 2-wheel omnidirectional robot.

Massachusetts Institute of Technology,MIT

Modular robotic units that can change their size at will presents an exciting pathway for modular robotics. However, current attempts have been relatively limited, requiring tethers, complex fabrication or slow cycle times. In this work, we present AuxBots: an auxetic-based approach to create high force, fast cycle time self-contained modules. By driving the auxetic shell's inherent mathematical expansion with a motor and leadscrew, these robots are capable of expanding their volume by 274% in 0.7 seconds with a maximum strength to weight ratio of 76x. These force and expansion properties enable us to use these modules in conjunction with flexible wire constraints to get shape changing behavior and independent locomotion. We demonstrate the power of this modular system by using a limited number of AuxBots to mimic the flipper-style locomotion of mudskippers and sea turtles. These structures are entirely untethered and can still move forward even as some AuxBots stall out, achieving the key modular robotics goal of versatility and robustness.

Beihang university,Beihang University

How to design an anthropomorphic hand imitating specific human hand grasps with as few actuators as possible is still a challenge. This paper presents a method for obtaining a simplified configuration of anthropomorphic hand imitating specific human hand grasps based on the motion analyses of the human hand. A participation matrix which characterizes a human hand grasp on joint motion level is constructed according to the motion participation of each finger joint. By adding all participation matrices of expected human hand grasps together a total participation matrix can be derived, and through mathematical processing a simplified anthropomorphic hand configuration can be obtained. Following the proposed method, a simplified anthropomorphic hand configuration that imitates six basic human hand grasps was obtained. A series of grasp experiments with the anthropomorphic hand prototype were conducted to validate the grasping capability as well as the proposed simplified configuration design method. This method can help to obtain a reasonably simplified configuration of an anthropomorphic hand when expected human hand grasps are definite.

University of Alicante,ETH Zurich Robotic Systems Laboratory,Robotic System Lab, ETH,ETH Zurich

The process of robot design is a complex task and the majority of design decisions are still based on human intuition or tedious manual tuning. A more informed way of facing this task is computational design methods where design parameters are concurrently optimized with corresponding controllers. Existing approaches, however, are strongly influenced by predefined control rules or motion templates and cannot provide end-to-end solutions. In this paper, we present a design optimization framework using model-free meta reinforcement learning, and its application to the optimizing kinematics and actuator parameters of quadrupedal robots. We use meta reinforcement learning to train a locomotion policy that can quickly adapt to different designs. This policy is used to evaluate each design instance during the design optimization. We demonstrate that the policy can control robots of different designs to track random velocity commands over various rough terrains. With controlled experiments, we show that the meta policy achieves close-to-optimal performance for each design instance after adaptation. Lastly, we compare our results against a model-based baseline and show that our approach allows higher performance while not being constrained by predefined motions or gait patterns.

Hwi-Su Kim, Jongwoo Park, Myeongsu Bae, Dongil Park, Chanhun Park, Hyunmin Do, Taeyong Choi, Doo-hyeong Kim, Jinho Kyung

Korea Institute of Machinery & Materials,Korea Institue of Machinery & Materials,Dyence tech,Korea Institute of Machinery and Materials (KIMM),KIMM,Korea Institute of Machinery and Materials,Korea Institute of Machinery & Materials (KIMM)

In recent years, human-robot cooperation has enhanced productivity and achieved high payload, speed, and accuracy. Integrating typical industrial robots in human-robot cooperation is challenging because their arms may cause serious injuries to humans during a collision due to malfunction or errors due to robot operators. Therefore, counterbalance robot arms that are capable of counterbalancing the gravitational torques due to the robot mass have been developed to decrease the required capacity of the motors and speeds of these robots. In this research, we propose an advanced counterbalance mechanism using gear units and springs to improve the durability and reliability compared to the previously proposed wire-based counterbalance mechanism, which is difficult to apply to a commercialized product because it can easily be broken or stretched when an excessive force is applied for a long period. Moreover, our proposed method was extended to a multi-DOF system using a parallelogram mechanism based on a timing belt and pulleys to achieve multi-DOF robotic arms. A 2-DOF counterbalanced arm was designed to verify the effectiveness of the proposed mechanism. The simulations and experimental results showed that the proposed mechanism effectively reduced the gravitational torques of each joint of the multi-DOF arm.

Permanent-Magnetically Amplified Robotic Gripper with Less Clamping Width Influence on Compensation Realized by a Stepless Width Adjustment Mechanism

Machines such as robotic grippers use powerful actuators or gearboxes to exert large loads at the expense of energy consumption, volume, and mass. We propose a stepless force amplification mechanism that assists clamping by a pair of permanent magnets, in which the external control force required to adjust their distance, and thus the output force, is suppressed by compensation springs. For further sophistication, we invented a new width adjuster using a lever. By separating the actuation of fingers and compensated magnets temporarily, the adjuster eliminated the nonlinear influence of the object width on the clamping force. The prototype gripper for proof of concept revealed that the adjuster successfully linearized the width-force characteristic with an inclination of 0.15 N/mm, which is sufficiently insignificant compared to the major output force of approximately 50 N. The force amplification effect coexisted with this phenomenon, such that the clamping force was amplified to 137.5% while maintaining the energy consumption of a DC motor, and the force-energy efficiency was multiplied by 1.39. Thus, able to be driven by a weaker, smaller, and lighter actuator, the gripper contributes to extension of the operation time of robots with limited power supply.

This paper proposes the concept design of a novel bio-inspired dual-axis compliant micromanipulator with millimeter working strokes dedicated to fiber alignment. It subtly mimics the gripping and rubbing function of the human hand consisting of the forefinger, purlicue, and thumb. Compared with traditional dual-axis grippers, its advantages lie in millimeter-level stroke, bi-directional rotation, less slippage, and comprehensive force sensing. To achieve dexterous and reliable manipulation, a two-degree-of-freedom (2-DOF) flexible decoupling mechanism and a displacement reversing mechanism based on the leaf-shaped flexible hinge are introduced. A prototype driven by two voice coil motors is fabricated for experimental testing. Three high-precision strain gauges with temperature compensation are glued on the sensitive region to measure the gripping force and rubbing force. Experimental results show that the gripping and rubbing strokes of the manipulator are up to 2.3 mm and 2.1 mm, respectively. For a custom-made fiber flag with a diameter of 200 um, the rotation stroke of more than 1000 degrees has been achieved, which cannot be realized by previous work with the same level of compact mechanism design.

Flapping wing robots show promise as platforms for safe and efficient flight in near-human operations, thanks to their ability to agile maneuver or perch at a low Reynolds number. The growing trend in the automatization of these robots has to go hand in hand with an increase in the payload capacity. This work provides a new passive morphing wing prototype to increase the payload of this type of UAV. The prototype is based on a biased elastic joint and the holistic research also includes the modelling, simulation and optimization scheme, thus allowing to adapt the prototype for any flapping wing robot. This model has been validated through flight experiments on the available platform, and it has also been demonstrated that the morphing prototype can increase the lift of the robot under study by up to 16% in real flight and 10% of estimated consumption reduction.

Florida State University,Tencent America

We propose a variant of alternating direction method of multiplier (ADMM) to solve constrained trajectory optimization problems. Our ADMM framework breaks a joint optimization into small sub-problems, leading to a low iteration cost and decentralized parameter updates. Starting from a collision-free initial trajectory, our method inherits the theoretical properties of primal interior point method (P-IPM), i.e., guaranteed collision avoidance and homotopy preservation throughout optimization, while being orders of magnitude faster. We have analyzed the convergence and evaluated our method for time-optimal multi-UAV trajectory optimizations and simultaneous goal-reaching of multiple robot arms, where we take into consider kinematics-, dynamics-limits, and homotopy-preserving collision constraints. Our method highlights an order of magnitude's speedup, while generating trajectories of comparable qualities as state-of-the-art P-IPM solver.

Frontier-based exploration is widely adopted for exploring an unknown region. The conventional frontier-based exploration for a mobile robot may collide with three-dimensional (3D) obstacles or can suffer from a slower exploration time because the robot may move to another place before completely exploring the current area. To solve this problem, in this paper, we propose a new exploration algorithm by considering a path traveled by a mobile robot and segmenting a two-dimensional (2D) map. The segmented 2D map is generated in real-time by using the position of the robot and the location of the detected frontiers. To apply our algorithm to the actual experiment, we develop an object detection-based exploration algorithm that can remarkably reduce the probability of collision with 3D obstacles. To verify the effectiveness of our proposed algorithm, we perform simulations (Gazebo) and experiments (in the real world) to compare the conventional approach and our algorithm in a cluttered environment. The simulation and experiment results show that our algorithm can satisfactorily shorten the exploration path and time.

University of California, San Diego,Los Alamos National Laboratory,UC San Diego

In this paper, we propose a Wasserstein metric-based random path planning algorithm. Wasserstein Safe RRT (W-Safe RRT) provides finite-sample probabilistic guarantees on the safety of a returned path in an uncertain obstacle environment. Vehicle and obstacle states are modeled as distributions based upon state and model observations. We define limits on distributional sampling error so the Wasserstein distance between a vehicle state distribution and obstacle distributions can be bounded. This enables the algorithm to return safe paths with a confidence bound through combining finite sampling error bounds with calculations of the Wasserstein distance between discrete distributions. W-Safe RRT is compared against a baseline minimum encompassing ball algorithm, which ensures balls that minimally encompass discrete state and obstacle distributions do not overlap. The improved performance is verified in a 3D environment using single, multi, and rotating non-convex obstacle cases, with and without forced obstacle error in adversarial directions, showing that W-Safe RRT can handle poorly modeled complex environments.

Shenzhen Techonology University,Shenzhen University,Oakland University

Obstacle avoidance for robotic manipulators can be challenging when they operate in unstructured environments. This problem is probed with the sim-to-real (sim2real) deep reinforcement learning, such that a moving policy of the robotic arm is learnt in a simulator and then adapted to the real world. However, the problem of sim2real adaptation is notoriously difficult. To this end, this work proposes (1) a unified representation of obstacles and targets to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model combining the unified representation with the deep reinforcement learning control module that can be trained by interacting with the environment. Such a representation is agnostic to the shape and appearance of the underlying objects, which simplifies and unifies the scene representation in both simulated and real worlds. We implement this idea with a vision-based actor-critic framework by devising a bounding box predictor module. The predictor estimates the 3D bounding boxes of obstacles and targets from the RGB-D input. The features extracted by the predictor are fed into the policy network, and all the modules are jointly trained. Our experiments in simulated environment and the real-world show that the end-to-end model of the unified representation achieves better sim2real adaption and scene generalization than state-of-the-art techniques.

University of Maryland, College Park,University of Maryland

Bidirectional path and motion planning approaches decrease planning time, on average, compared to their unidirectional counterparts. In single-query feasible motion planning, using bidirectional search to find a continuous motion plan requires an edge connection between the forward and the reverse search tree. Such a tree–tree connection requires solving a two-point boundary value problem (BVP). However, obtaining a closed-form two-point BVP solution can be difficult or impossible for many systems. While numerical methods can provide a reasonable solution in many cases, they are often computationally expensive or numerically unstable for the purposes of single-query sampling-based motion planning. To overcome this challenge, we present a novel bidirectional search strategy that does not require solving the two-point BVP. Instead of connecting the forward and reverse trees directly, the reverse tree’s cost information is used as a guiding heuristic for forward search. This enables the forward search to quickly grow down the reverse tree—converging to a fully feasible solution without the solution to a two-point BVP. In this article, we propose two algorithms that use this strategy for single-query feasible motion planning for various dynamical systems, performing experiments in both simulation and hardware testbeds. We find that these algorithms perform better than or comparable to existing state-of-the-art methods with respect to quickly finding an initial feasible solution.

University of Stuttgart,TU Berlin,Bilkent University

Robotic construction assembly planning aims to find feasible assembly sequences as well as the corresponding robot-paths and can be seen as a special case of task and motion planning (TAMP). As construction assembly can well be parallelized, it is desirable to plan for multiple robots acting concurrently. Solving TAMP instances with many robots and over a long time-horizon is challenging due to coordination constraints, and the difficulty of choosing the right task assignment. We present a planning system which enables parallelization of complex task and motion planning problems by iteratively solving smaller subproblems. Combining optimization methods to jointly solve for manipulation constraints with a sampling-based bi-directional space-time path planner enables us to plan cooperative multi-robot manipulation with unknown arrival-times. Thus, our solver allows for completing subproblems and tasks with differing timescales and synchronizes them effectively. We demonstrate the approach on multiple construction case-studies to show the robustness over long planning horizons and scalability to many objects and agents. % of our algorithm. Finally, we also demonstrate the

Northwestern Polytechnical University,Zhejiang Lab

By limiting the planning domain to “L2 Informed Set”, some sampling-based motion planner (SBMP) (e.g. Informed RRT*, BIT*) can solve the geometric motion planning problems efficiently. However, the construction of informed set (IS) will be very challenging, when further differential constraints are considered. For the time-optimal kinodynamic motion planning problem, this paper defines a modified time informed set (MTIS) to limit the planning domain. Due to drawing inspiration from Hamilton-Jacobi-Bellman (HJB) reachability analysis, MTIS, compared with the original TIS, can not only help save the running time of SBMP, but also extend the applicable scope from linear systems to polynomial nonlinear systems with control constrains. On this basis, a spatio-temporal sampling strategy adapted to MTIS is proposed. Firstly, MTIS is used to estimate the optimal cost and the valid tree structure is reused, so that we do not need to provide a solution trajectory in advance. Secondly, this strategy is generic, allowing it to be combined with common SBMP (such as SST, etc.) to accelerate convergence and reduce memory complexity. Several simulation experiments also demonstrate the effectiveness of proposed method.

The Hongkokng University of Science and Technology,the Hong Kong University of Science and Technology,Hong Kong University of Science and Technology,KTH Royal Institute of Technology,HKUST,The Hong Kong University of Technology

Safely interacting with other traffic participants is one of the core requirements for autonomous driving, especially in intersections and occlusions. Most existing approaches are designed for particular scenarios and require significant human labor in parameter tuning to be applied to different situations. To solve this problem, we first propose a learning-based Interaction Point Model (IPM), which describes the interaction between agents with the protection time and interaction priority in a unified manner. We further integrate the proposed IPM into a novel planning framework, demonstrating its effectiveness and robustness through comprehensive simulations in highly dynamic environments.

We propose and experimentally demonstrate a reactive planning system for bipedal robots on unexplored, challenging terrain. The system includes: a multi-layer local map for assessing traversability; an anytime omnidirectional Control Lyapunov Function (CLF) for use with a Rapidly Exploring Random Tree Star (RRT*) that generates a vector field for specifying motion between nodes; a sub-goal finder when the final goal is outside of the current map; and a finite-state machine to handle high-level mission decisions. The system also includes a reactive thread that copes with robot deviations via a vector field, defined by a closed-loop feedback policy. The vector field provides real-time control commands to the robot's gait controller as a function of instantaneous robot pose. The system is evaluated on various challenging outdoor terrains and cluttered indoor scenes in both simulation and experiment on Cassie Blue, a bipedal robot with 20 degrees of freedom. All implementations are coded in C++ with the Robot Operating System (ROS) and are available at https://github.com/UMich-BipedLab/CLF_reactive_planning_system.

Robots often need to accomplish complex tasks in unknown environments, which is a challenging problem, involving autonomous exploration for acquiring necessary scene knowledge and task planning. In traditional approaches, the agent first explores the environment to instantiate a complete planning domain and then invokes a symbolic planner to plan and perform high-level actions. However, task execution is inefficient since the two processes involve many repetitive states and actions. Hence, this paper proposes a framework to co-optimize robot exploration and task planning in unknown environments. To afford robot exploration and symbolic planning not being independent and separated, we design a unified structure named subtask, which is exploited to decompose the robot exploration and planning phases. To select the appropriate subtask each time, we develop a value function and a value-based scheduler to co-optimize exploration and task processing. Our framework is evaluated in a photo-realistic simulator with three complex household tasks, increasing task efficiency by 25%-29%.

Nara Institute of Science and Technology

This paper explores a Deep Reinforcement Learning (DRL) approach for designing image-based control for edge robots to be implemented on Field Programmable Gate Arrays (FPGAs). Although FPGAs are more power-efficient than CPUs and GPUs, a typical (DRL) method cannot be applied since they are composed of many Logic Blocks (LBs) for high-speed logical operations but low-speed real-number operations. To cope with this problem, we propose a novel DRL algorithm called Binarized P-Network (BPN), which learns image-input control policies using Binarized Convolutional Neural Networks (BCNNs). To alleviate the instability of reinforcement learning caused by a BCNN with low function approximation accuracy, our BPN adopts a robust value update scheme called Conservative Value Iteration, which is tolerant of function approximation errors. We confirmed the BPN's effectiveness through applications to a visual tracking task in simulation and real-robot experiments with FPGA.

Deep reinforcement learning has enabled robots to learn motor skills from environmental interactions with minimal to no prior knowledge. However, existing reinforcement learning algorithms assume an episodic setting, in which the agent resets to a fixed initial state distribution at the end of each episode, to successfully train the agents from repeated trials. Such reset mechanism, while trivial for simulated tasks, can be challenging to provide for real-world robotics tasks. Resets in robotic systems often require extensive human supervision and task-specific workarounds, which contradicts the goal of autonomous robot learning. In this paper, we propose an extension to conventional reinforcement learning towards greater autonomy by introducing an additional agent that learns to reset in a self-supervised manner. The reset agent preemptively triggers a reset to prevent manual resets and implicitly imposes a curriculum for the forward agent. We apply our method to learn from scratch on a suite of simulated and real-world continuous control tasks and demonstrate that the reset agent successfully learns to reduce manual resets whilst also allowing the forward policy to improve gradually over time.

University of Illinois at Urbana-Champaign,University of Illinois Urbana-Champaign,University of Illinois

A reinforcement learning (RL) control policy could fail in a new/perturbed environment that is different from the training environment, due to the presence of dynamics variations. For controlling systems with continuous state and action spaces, we propose an add-on approach to robustifying a pre-trained RL policy by augmenting it with an L1 adaptive controller (L1AC). Leveraging the capability of an L1AC for fast estimation and active compensation of dynamic variations, the proposed approach can improve the robustness of an RL policy that is trained either in a simulator or in the real world without consideration of a broad class of dynamics variations. Numerical and real-world experiments empirically demonstrate the efficacy of the proposed approach in robustifying RL policies trained using both model-free and model-based methods.

Queensland University of Technology

Many hierarchical reinforcement learning algorithms utilise a series of independent skills as a basis to solve tasks at a higher level of reasoning. These algorithms don't consider the value of using skills that are cooperative instead of independent. This paper proposes the Cooperative Consecutive Policies (CCP) method of enabling consecutive agents to cooperatively solve long time horizon multi-stage tasks. This method is achieved by modifying the policy of each agent to maximise both the current and next agent's critic. Cooperatively maximising critics allows each agent to take actions that are beneficial for its task as well as subsequent tasks. Using this method in a multi-room maze domain and a peg in hole manipulation domain, the cooperative policies were able to outperform a set of naive policies, a single agent trained across the entire domain, as well as another sequential HRL algorithm.

University of Southern California,UNIVERSITY OF SOUTHERN CALIFORNIA

In the paradigm of robot learning-from-demonstrations (LfD), understanding and evaluating the demonstrated behaviors plays a critical role in extracting control policies for robots. Without this knowledge, a robot may infer incorrect reward functions that lead to undesirable or unsafe control policies. Prior work has used temporal logic specifications, manually ranked by human experts based on their importance, to learn reward functions from imperfect/suboptimal demonstrations. To overcome reliance on expert rankings, we propose a novel algorithm that learns from demonstrations, a partial ordering of provided specifications in the form of a performance graph. Through various experiments, including simulation of industrial mobile robots, we show that extracting reward functions with the learned graph results in robot policies similar to those generated with the manually specified orderings. We also show in a user study that the learned orderings match the orderings or rankings by participants for demonstrations in a simulated driving domain. These results show that we can accurately evaluate demonstrations with respect to provided task specifications from a small set of imperfect data with minimal expert input.

CSE, UMN,CSE, University of Minnesota,University of Minnesota

Tumbling robots are simple platforms that are able to traverse large obstacles relative to their size, at the cost of being difficult to control. Existing control methods apply only a subset of possible robot motions and make the assumption of flat terrain. Reinforcement learning allows for the development of sophisticated control schemes that can adapt to diverse environments. By utilizing domain randomization while training in simulation, a robust control policy can be learned which transfers well to the real world. In this paper, we implement autonomous setpoint navigation on a tumbling robot prototype and evaluate it on flat and uneven terrain. The flexibility of our system demonstrates the viability of nontraditional robots for navigational tasks.

Recent successes aside, reinforcement learning still faces significant challenges in its application to the real-world robotics domain. Guiding the learning process with additional knowledge offers a potential solution, thus leveraging the strengths of data- and knowledge-driven approaches. However, this field of research encompasses several disciplines and hence would benefit from a structured overview. In this paper, we propose the concept of guided reinforcement learning that provides a systematic approach towards accelerating the training process and improving the performance for real-world robotic settings. We introduce a classification that structures guided reinforcement learning approaches and shows how different sources of knowledge can be integrated into the learning pipeline in a practical way. Based upon this, we describe available approaches in this field and evaluate their specific impact in terms of efficiency, effectiveness, and sim-to-real transfer within the robotics domain.

Fudan University,FuDan University,Zhejiang University

Reinforcement learning needs to learn policies through trial and error. The unstable policies in the early stage of training make it expensive (and time-consuming) to train directly in the real environment, which may cause disastrous consequences. The popular solution is to use the simulator to train the policy and deploy it in a real environment. However, the modeling error and external disturbance between the simulation and the real environment may fail the physical deployment, resulting in the sim2real transfer problem. In this letter, we propose a novel robust adversarial reinforcement learning framework, which uses the ensemble training of multi-adversarial agents that can adaptively adjust adversaries’ strength to enhance RL policy’s robustness. More specifically, we take the accumulative reward as feedback and construct a PID controller to adjust the adversary’s output magnitude to perform the adversarial training well. Experiments in the simulated and the real environment show that our algorithm improves the generalization ability of the policy for the modeling error and the uncertain disturbance simultaneously, outperforming the next best prior methods across all domains. The algorithm was further proven to be effective in a sim2real transfer task through the load experiment of a real racing drone, and the tracking performance is better than the PID-based flight controller.

Applying reinforcement learning to autonomous driving entails particular challenges, primarily due to dynamically changing traffic flows. To address such challenges, it is necessary to quickly determine response strategies to the changing intentions of surrounding vehicles. This paper proposes a new policy optimization method for safe driving using graph-based interaction-aware constraints. In this framework, the motion prediction and control modules are trained simultaneously while sharing a latent representation that contains a social context. To reflect social interactions, we illustrate the movements of agents in graph form and filter the features with the graph convolution networks. This helps preserve the spatiotemporal locality of adjacent nodes. Furthermore, we create feedback loops to combine these two modules effectively. As a result, this approach encourages the learned controller to be safe from dynamic risks and renders the motion prediction robust to abnormal movements. In the experiment, we set up a navigation scenario comprising various situations with CARLA, an urban driving simulator. The experiments show state-of-the-art performance in navigation strategy and motion prediction compared to the baselines. The code is available online.

University of Freiburg,Albert-Ludwigs-Universität Freiburg,University of Technology Nuremberg

Accurate value estimates are important for off-policy reinforcement learning. Algorithms based on temporal difference learning typically are prone to an over- or underestimation bias building up over time. In this paper, we propose a general method called Adaptively Calibrated Critics (ACC) that uses the most recent high variance but unbiased on-policy rollouts to alleviate the bias of the low variance temporal difference targets. We apply ACC to Truncated Quantile Critics [1], which is an algorithm for continuous control that allows regulation of the bias with a hyperparameter tuned per environment. The resulting algorithm adaptively adjusts the parameter during training rendering hyperparameter search unnecessary and sets a new state of the art on the OpenAI gym continuous control benchmark among all algorithms that do not tune hyperparameters for each environment. ACC further achieves improved results on different tasks from the Meta-World robot benchmark. Additionally, we demonstrate the generality of ACC by applying it to TD3 [2] and showing an improved performance also in this setting.

University of Shanghai for Science and Technology,Shanghai Jiao Tong University,ShangHai JiaoTong university

Legged robots have great potential to be extraterrestrial exploration rovers of extraordinary versatility. Minimizing power consumption is of vital importance in the scenarios of extraterrestrial explorations. The actuation pattern, which refers to the combination of necessary actuators that output torque, has a significant influence on the power consumption of legged robots. This article seeks to investigate the effect of actuation patterns on the power consumption of legged robots that perform motion in a quasi-static manner. The power consumption model of legged robots considering actuation patterns is deduced. Based on that, the effect of the actuation pattern on mechanical power and heat power, which are the main power-loss terms, is investigated. The lowest power consumption under various conditions achieved by different actuation patterns is investigated. Simulation results show that the power consumption can be reduced by choosing the actuation pattern properly. Furthermore, the principles of selecting the optimal actuation pattern from the perspective of power consumption are summarized, which are expected to facilitate the minimal power consumption motion planning of legged robots.

Agency for Defense Development,KAIST

All vessels operating in a marine environment are required to comply with the international regulations for preventing collisions at sea (COLREGs), which provide the guidelines and evasive procedures required to resolve potential conflicts between vessels. However, not all vessels strictly abide by COLREGs, often leading to dangerous situations. This paper presents a novel approach for robust collision avoidance in encounter situations involving COLREG-violating vessels. A probabilistic velocity obstacle algorithm based on intent inference is designed and implemented with consideration of the tradeoff between the adherence to traffic rules and the proactive evasive actions for safety. One-to-one and multi-ship encounter situations in the presence of rule-violating vessels are examined through Monte-Carlo simulations, and the results are discussed to demonstrate the feasibility and performance of the proposed approach.

Shanghai jiao tong University,Shanghai Jiao Tong University,Shanghai Jiaotong University

The distinct design concepts of the vehicles operating in air and water is one of the tremendous challenges that constrain the development of the hybrid aerial underwater vehicle (HAUV). This incompatibility consequently results in the enlarging volume and weight of the existing prototypes, as well as the unmatched maneuvering characteristics in both domains. This letter presented a novel miniaturized and lightweight HAUV, "Nezha-mini", which weighs 953g and is only A4-scaled. Besides, the low cost and high modularity allow the convenient repair and remanufacturing. Nezha-mini reconciles the complete multi-domain maneuverability within 50m aerially and 6m underwater whilst sufficing for the rapid and stable cross-domain locomotion, which benefits from the selection and unique layout of the propulsion system, as well as our proposed multi-modal control strategy and the cross-domain triggering mechanism. The results of the field experiments are in good agreement with the dynamics simulation, demonstrating the performance of multi-domain locomotion in real environments. The preliminary exploration in this letter provides a referential solution for the miniaturization of the highly maneuverable HAUVs for practical applications and creates a feasible platform for the future clustering and networking of HAUVs.

Northwestern Polytechnical University

Most of the existing underwater legged robots are capable of moving on small-angled slopes, but few of them can climb the large-angled slope or transition from one plane to another, such as transition from horizontal plane to vertical plane. In this paper, we propose a motion planning method of a hybrid underwater hexapod robot (HUHR) driven by six C-shape legs and eight thrusters. By analyzing the relationship between rotation and displacement of the hip joint, we establish a single-leg kinematic model. By analyzing the force at the touchpoint, we propose a locomotion mechanism to ensure no slip of the C-shape leg. Based on the central pattern generator (CPG) and tripod gait, we design an aperiodic mapping between the oscillator outputs and the desired rotation angles of hip joints. Overall, a gait planning and control method for our robot is proposed to realize continuous legged locomotion from one plane to another, including directional climbing and transition between them. Furthermore, the effectiveness of the proposed method has been verified on HUHR.

Accurate, self-consistent bathymetric maps are needed to monitor changes in subsea environments and infrastructure. These maps are increasingly collected by underwater vehicles, and mapping requires an accurate vehicle navigation solution. Commercial off-the-shelf (COTS) navigation solutions for underwater vehicles often rely on external acoustic sensors for localization, however survey-grade acoustic sensors are expensive to deploy and limit the range of the vehicle. Techniques from the field of simultaneous localization and mapping, particularly loop closures, can improve the quality of the navigation solution over dead-reckoning, but are difficult to integrate into COTS navigation systems. This work presents a method to improve the self-consistency of bathymetric maps by smoothly integrating loop-closure measurements into the state estimate produced by a commercial subsea navigation system. Integration is done using a white-noise-on-acceleration motion prior, without access to raw sensor measurements or proprietary models. Improvements in map self-consistency are shown for both simulated and experimental datasets, including a 3D scan of an underwater shipwreck in Wiarton, Ontario, Canada.

Zhejiang University,Zhejiang Univ.,China

Underwater positioning is critical to autonomous underwater vehicles (AUVs) for navigation and geo-referencing. The rapid attenuation of the electromagnetic wave in the underwater environment prevents the use of traditional positioning methods such as the Global Positioning System, whereupon acoustic methods like ultra-short baseline (USBL) positioning systems play an important role in AUV navigation. However, the high cost and complexity of classical USBL systems have stifled the democratization of these technologies, which leads to a new method called passive inverted ultra-short baseline (piUSBL) positioning. In a typical piUSBL system, a single beacon is placed at a reference point, periodically broadcasting a positioning signal. A passive USBL receiver, time-synchronized to the beacon, is mounted on an AUV to get one-way travel-time (OWTT) slant range and azimuth estimates. The passive nature of the receiver means the system is inexpensive, low-power, and lightweight. Particularly, the omnidirectional broadcasted signals offer a feasible solution for concurrent multi-AUV navigation. This letter demonstrates a full-stack design and development of a piUSBL positioning system, and presents evaluations of the accuracy and reliability of the system through a series of experiments. More significantly, a successful sea trial of a disc-shaped AUV outfitted with our piUSBL was conducted in the South China Sea.

UC Berkeley Mechanical Engineering,University of California Berkeley,Apptronik,University of California at Berkeley,UC Berkeley

Reduced traction limits the ability of mobile robotic systems to resist or apply large external loads, such as tugging a massive payload. One simple and versatile solution is to wrap a tether around naturally occurring objects to leverage the capstan effect and create exponentially-amplified holding forces. Experiments show that an idealized capstan model explains force amplification experienced on common irregular outdoor objects – trees, rocks, posts. Robust to variable environmental conditions, this exponential amplification method can harness single or multiple capstan objects, either in series or in parallel with a team of robots. This adaptability allows for a range of potential configurations especially useful for when objects cannot be fully encircled or gripped. This versatility is demonstrated with teleoperated mobile platforms to (1) control the lowering and arrest of a payload, (2) to achieve planar control of a payload, and (3) to act as an anchor point for a more massive platform to winch towards. We show the simple addition of a tether, wrapped around shallow stones in sand, amplifies holding force of a low-traction platform by up to 774x.

Korea Advanced Institute of Science and Technology

Inflatable structures have attracted considerable research attention in many fields owing to their numerous advantages, such as being light and able to engage in interactions safely. However, in most cases, the inflatable structure can only have one stable configuration, which is undesirable for robotic arms. This study proposes a novel inflatable structure that can be easily reconfigured into multiple stable configurations, even with single-body inflation. In the proposed mechanism, the structure length can be freely adjusted, and its respective joints can be set in the desired directions to facilitate the reconfiguration of its pose. An additional advantage of the proposed mechanism is that it can withstand external forces as well as its own weight. This study analyzes and experimentally validates the shape locking and load-carrying properties of the proposed mechanism. Further, the fabrication process and design guidelines for the proposed mechanism are presented. Through a suitable demonstration, the proposed mechanism is shown to exhibit multiple stable configurations and lock its poses.

University of Science and Technology of China,City University of Hong Kong

Soft continuum robots have shown tremendous potential for medical and industrial applications owing to their flexibility and continuous deformability. However, their telescopic and bending capabilities and variable stiffness are still limited. This study proposes a novel origami-inspired soft continuum robot to possess large telescopic and bending capabilities while improving stiffness based on the principle of antagonistic actuation. The soft robot consists of dual origami structures. The inner forms an air chamber actuated by pneumatics, and the outer is controlled by nine tendon-driven actuators. The proposed design uses the advantages of a hybrid actuation to achieve motion and stiffness control. The performance of the soft robot is studied experimentally based on single and three robot modules. Results show that the robot has an excellent stretch ratio and a maximum bending angle of 180°. The robot can also increase stiffness to resist the bending deformation induced by self-weight and loads.

Univ. Lille, Inria, CNRS, Centrale Lille, UMR ,,,, CRIStAL,University of Lille - INRIA,Inria - Lille Nord Europe,INRIA

The Finite Element Method (FEM) is a powerful modeling tool for predicting the behavior of soft robots. However, its use for control can be difficult for non-specialists of numerical computation: it requires an optimization of the computation to make it real-time. In this paper, we propose a learning-based approach to obtain a compact but sufficiently rich mechanical representation. Our choice is based on non- linear compliance data in the actuator/effector space provided by a condensation of the FEM model. We demonstrate that this compact model can be learned with a reasonable amount of data and, at the same time, be very efficient in terms of modeling, since we can deduce the direct and inverse kinematics of the robot. We also show how to couple some models learned individually in particular on an example of a gripper composed of two soft fingers. Other results are shown by comparing the inverse model derived from the full FEM model and the one from the compact learned version. This work opens new perspectives, namely for the embedded control of soft robots, but also for their design. These perspectives are also discussed in the paper.

The University of Tokyo,Tokyo Denki University,Riverfield Inc.,the University of Tokyo

One of the recent developments in physical reservoir computing, which uses the complex dynamics of a physical system as a computational resource, is the use of a pneumatic pipeline system as a computational resource. This uses the dynamics of air for computation, and because it is lightweight and power-saving, it is used for gait-assist control using a soft exoskeleton with pneumatic rubber artificial muscles. In this study, we verified that by feeding back the estimated information to a pneumatic pipeline system, the pneumatic physical reservoir computing can generate periodic pressure changes as a stable limit cycle, such as those seen in walking. A pneumatic reservoir with feedback loops was modeled to generate limit cycles in the simulation, and it was confirmed that the system could generate limit cycles with high accuracy even from initial positions far from the target limit cycle. This system is expected to be applied to assist walking movements with a soft exoskeleton with a lightweight computational device.

Carnegie Mellon University,New York University

Pneumatic soft robots present many advantages in manipulation tasks. Notably, their inherent compliance makes them safe and reliable in unstructured and fragile environments. However, full-body shape sensing for pneumatic soft robots is challenging because of their high degrees of freedom and complex deformation behaviors. Vision-based proprioception sensing methods relying on embedded cameras and deep learning provide a good solution to proprioception sensing by extracting the full-body shape information from the high-dimensional sensing data. But the current training data collection process makes it difficult for many applications. To address this challenge, we propose and demonstrate a robust sim-to-real pipeline that allows the collection of the soft robot's shape information in high-fidelity point cloud representation. The model trained on simulated data was evaluated with real internal camera images. The results show that the model performed with averaged Chamfer distance of $8.85$ mm and tip position error of $10.12$ mm even with external perturbation for a pneumatic soft robot with a length of $100.0$ mm. We also demonstrated the sim-to-real pipeline’s potential for exploring different configurations of visual patterns to improve vision-based reconstruction results. The code and dataset are available at https://github.com/DeepSoRo/DeepSoRoSim2Real.

Shageenderan Sapai, Junn Yong Loo, Ze Yang Ding, Chee Pin Tan, Raphael Phan, Vishnu Monn Baskaran, Surya G. Nurzaman

Monash University,Monash Malaysia,Monash University Malaysia

Recently, data-driven models such as deep neural networks have shown to be promising tools for modelling and state inference in soft robots. However, voluminous amounts of data are necessary for deep models to perform effectively, which requires exhaustive and quality data collection, particularly of state labels. Consequently, obtaining labelled state data for soft robotic systems is challenged for various reasons, including difficulty in the sensorization of soft robots and the inconvenience of collecting data in unstructured environments. To address this challenge, in this paper, we propose a semi-supervised sequential variational Bayes (DSVB) framework for transfer learning and state inference in soft robots with missing state labels on certain robot configurations. Considering that soft robots may exhibit distinct dynamics under different robot configurations, a feature space transfer strategy is also incorporated to promote the adaptation of latent features across multiple configurations. Unlike existing transfer learning approaches, our proposed DSVB employs a recurrent neural network to model the nonlinear dynamics and temporal coherence in soft robot data. The proposed framework is validated on multiple setup configurations of a pneumatic-based soft robot finger. Experimental results on four transfer scenarios demonstrate that DSVB performs effective transfer learning and accurate state inference amidst missing state labels.

University of California San Diego,UCSD,University of California, San Diego

State estimation from measured data is crucial for robotic applications as autonomous systems rely on sensors to capture the motion and localize in the 3D world. Among sensors that are designed for measuring a robot's pose, or for soft robots, their shape, vision sensors are favorable because they are information-rich, easy to set up, and cost-effective. With recent advancements in computer vision, deep learning-based methods no longer require markers for identifying feature points on the robot. However, learning-based methods are data-hungry and hence not suitable for soft and prototyping robots, as building such bench-marking datasets is usually infeasible. In this work, we achieve image-based robot pose estimation and shape reconstruction from camera images. Our method requires no precise robot meshes, but rather utilizes a differentiable renderer and primitive shapes. It hence can be applied to robots for which CAD models might not be available or are crude. Our parameter estimation pipeline is fully differentiable. The robot shape and pose are estimated iteratively by back-propagating the image loss to update the parameters. We demonstrate that our method of using geometrical shape primitives can achieve high accuracy in shape reconstruction for a soft continuum robot and pose estimation for a robot manipulator.

Enrico Franco, Ayhan Aktas, Shen Treratanakulchai, Arnau Garriga-casanovas, Abdulhamit Donder, Ferdinando Rodriguez Y Baena

Imperial College London,Imperial College,Imperial College, London, UK

In this article we investigate the discrete-time model based control of a planar soft continuum manipulator with proprioceptive sensing provided by fiber Bragg gratings. A control algorithm is designed with a discrete-time energy shaping approach which is extended to account for control-related lag of digital nature. A discrete-time nonlinear observer is employed to estimate the uncertain bending stiffness of the manipulator and to compensate constant matched disturbances. Simulations and experiments demonstrate the effectiveness of the controller compared to a continuous time implementation.

Soft optical sensing strategies are rapidly developing for soft robotic systems as a means to increase the controllability of soft compliant robots. In this paper, we present a roughness tuning strategy for the fabrication of soft optical sensors to achieve the dual functionality of shape sensing combined with contact recognition within a single multi-modal sensor. The molds used to fabricate the soft sensors are roughened via laser micromachining to achieve asymmetrical sensor responses when bent in opposite directions. We demonstrate the integration of these sensors into a fully soft robotic platform consisting of a multi-directional bending module with integrated 3D shape sensing and a gripper with tip position monitoring along with contact force recognition. We show the accuracy of our sensing strategy in validation experiments and a pick-andplace task is performed to demonstrate the robot’s functionality.

James J. Davies, Mai Thanh Thai, Trung Thien Hoang, Nguyen Chi Cong, Phuoc Thien Phan, Kefan Zhu, Dang Bao Nhi Tran, Van Ho, Hung La, Q P Ha, Nigel Lovell, Thanh Nho Do

University of New South Wales,UNSW Sydney,RMIT,Japan Advanced Institute of Science and Technology,University of Nevada at Reno,University of Technology Sydney

Following biology’s lead, soft robotics has emerged as a perfect candidate for actuation within complex environments. While soft actuation has been developed intensively over the last few decades, soft sensing has so far slowed to catch up. A largely unresearched area is the change of the soft material properties through prestress to achieve a degree of mechanical sensitivity tunability within soft sensors. Here, a new 3D force sensor which employs novel hydraulic filament artificial muscles capable of sensitivity tunability is introduced. Using a neural network (NN) model, the new soft 3D sensor can precisely detect external forces based on the change of the hydraulic pressures with error of ~1.0, ~1.3, and ~0.94 % in the x, y, and z-axis directions, respectively. The sensor is also able to sense large force ranges, comparable to other similar sensors available in the literature. The sensor is then integrated into a soft robotic surgical arm for monitoring the tool-tissue interaction during the ablation process.

Tsinghua University,Tsinghua Shenzhen International Graduate School,Tencent,Ryerson University

Vibration perception is essential for robotic sensing and dynamic control. Nevertheless, due to the rigorous demand for sensor conformability and stretchability, enabling soft robots with proprioceptive vibration sensing remains challenging. This paper proposes a new liquid metal-based stretchable e-skin via a kirigami-inspired design to enable soft robot proprioceptive vibration sensing. The e-skin is fabricated into 0.1mm ultrathin thickness, ensuring its negligible influence on the overall stiffness of the soft robot. Moreover, the working mechanism of the e-skin is based on the ubiquitous triboelectrification effect, which transduces mechanical stimuli without external power supply. To demonstrate the practicability of the e-skin, we built a soft gripper consisting of three soft robotic fingers with proprioceptive vibration sensing. Our experiment shows that the gripper can accurately distinguish the grain category (six grains with the same mass, 99.9% accuracy) and the packaging quality (100% accuracy) by simply shaking the gripped bottle. In summary, a soft robotic proprioceptive vibration sensing solution is proposed; it helps soft robots to have a more comprehensive awareness of their self-state and may inspire further research on soft robotics.

University of Cambridge,University College London

As soft robotic systems become increasingly complex, there is a need to develop sensory systems which can provide rich state information to the robot for feedback control. Multi-axis force sensing and control is one of the less explored problems in this domain. There are numerous challenges in the development of a multi-axis soft sensor: from the design and fabrication to the data processing and modelling. This work presents the design and development of a novel multi-axis soft sensor using a gelatin-based ionic hydrogel and 3D printing technology. A learning-based modelling approach coupled with sensor redundancy is developed to model the environmentally dependent soft sensors. Numerous real-time experiments are conducted to test the performance of the sensor and its applicability in closed-loop control tasks. Our results indicate that the soft sensor can predict force values and orientation angle within 4% and 7% of their total range, respectively.

Harvard University,Harvard University Science and Engineering Building

Soft, wearable systems hold promise for a wide variety of new or enhanced applications in the realm of human-computer interaction, physiological monitoring, wearable robotics, and a host of other human-centric devices. Soft sensor systems have been developed concurrently in order to allow these wearable systems to respond intelligently with their surroundings. A recently reported sensing mechanism based on the strain-mediated contact in anisotropically resistive structures (SCARS) is an attractive solution due to its high sensing resolution, low-profile nature, and high mechanical resilience. Furthermore, the resistance-based output provides a simple electronic readout, facilitating its use in a wide variety of applications. However, previous iterations of the sensing mechanism have exhibited stress relaxation and hysteretic behaviors that limit the scope of its use. Here, we report an iteration of the SCARS mechanism that uses silicone-based materials with low mechanical loss in order to improve the sensor signal stability and bandwidth. A new fabrication approach is developed which permits the incorporation of a liquid elastomer adhesive layer while also preserving the SCARS sensing functionality. The silicone-based SCARS sensors exhibited fast stress relaxation response (< 1 s) and reduced cyclic drift properties by more than half that of previously reported designs. A physiological monitoring demonstration is presented, validating that the new sensor design is mechanically resilient to such applications and has potential for use in real-world wearable use cases.

Carnegie Mellon University,Northwestern University,The University of Texas at Austin

Whisker-based tactile sensors have the potential to perform fast and accurate 3D mappings of the environment, complementing vision-based methods under conditions of glare, reflection, proximity, and occlusion. However, current algorithms for mapping with whiskers make assumptions about the conditions of contact, and these assumptions are not always valid and can cause significant sensing errors. Here we introduce a new whisker sensing system with a tapered, flexible whisker. The system provides inputs to two separate algorithms for estimating radial contact distance on a whisker. Using a Gradient-Moment (GM) algorithm, we correctly detect contact distance in most cases (within 4% of the whisker length). We introduce the Z-Dissimilarity score as a new metric that quantifies uncertainty in the radial contact distance estimate using both the GM algorithm and a Moment-Force (MF) algorithm that exploits the tapered whisker design. Combining the two algorithms ultimately results in contact distance estimates more robust than either algorithm alone.

Queen Mary University of London

Distributed sensor arrays capable of detecting multiple spatially distributed stimuli are considered an important element in the realisation of exteroceptive and proprioceptive soft robots. This paper expands upon the previously presented idea of decoupling the measurements of pressure and location of a local indentation from global deformation, using the overall stretch experienced by a soft capacitive e-skin. We employed machine learning methods to decouple and predict these highly coupled deformation stimuli, collecting data from a soft sensor e-skin which was then fed to a machine learning system comprising of linear regressor, gaussian process regressor, SVM and random forest classifier for stretch, force, detection and localisation respectively. We also studied how the localisation and forces are affected when two forces are applied simultaneously. Soft sensor arrays aided by appropriately chosen machine learning techniques can pave the way to e-skins capable of deciphering multi-modal stimuli in soft robots.

This paper presents the novel use of air gaps in flexible optical light pipes to create coded segments for use in bend localization. The OptiGap sensor system allows for the creation of extrinsic intensity modulated bend sensors that function as flexible absolute linear encoders. Coded segment patterns are identified by a Gaussian naive Bayes classifier running on an STM32 microcontroller. Fitting of the classifier is aided by a custom software suite that simplifies data collection and processing from the sensor. The sensor model is analyzed and verified through simulation and experiments, highlighting key properties and parameters that aid in the design of OptiGap sensors using different light pipe materials and for various applications. This system allows for realtime and accurate bend localization in many robotics and automation applications, in wet and dry conditions.

University of Bristol,Kyoto University,Department of engineering, University of Bristol, UK

Soft devices employ variable stiffness to ensure safety and improve the robustness in the interaction between robots and objects. Using soft materials is one of the most popular approaches to design a variable-stiffness device, while the use of silicone sponge remains less explored in this field. Here we present a novel silicone-sponge-based variable-stiffness device (SVD). The SVD is easy-to-make and low-cost, and fabricated by an air-tight bellow enclosing a silicone sponge core. This allows easy access to the hyper-elastic response of the porous sponge whilst stiffness tuning of the device via pneumatic pressure difference. A detailed mathematical model of the SVD is proposed, by which the stiffness can be precisely controlled by the pressure difference applied. The stiffness of SVD can be tuned in the range of [1.55, 22.82]×10^3 N/m, up to 14.7 times increase. The high stiffness is easily triggered by a low pressure difference (ΔP < 12 kPa). The SVD is a versatile and compact module, with small axial size (10 mm height) and light weight (14.3 g), making it highly suitable for integration in a wide range of robotics and industrial applications. This, in addition to its easy-to-fabricate and low-cost features, may appeal to the robotics community at large. We further detail its working principle, fabrication processes, mathematical model and automated control methods to show its versatility.

University of Pennsylvania,Worcester Polytechnic Institute

We propose a novel design for a lightweight and compact tunable stiffness actuator capable of stiffness changes up to 20x. The design is based on the concept of a coiled spring, where changes in the number of layers in the spring change the bulk stiffness in a near linear fashion. We present an elastica nested rings model for the deformation of the proposed actuator and empirically verify that the designed stiffness-changing spring abides by this model. Using the resulting model, we design a physical prototype of the tunable-stiffness coiled-spring actuator and discuss the effect of design choices on the resulting achievable stiffness range and resolution. In the future, this actuator design could be useful in a wide variety of soft robotics applications, where fast, controllable, and local stiffness change is required over a large range of stiffnesses.

Electrostatic actuators provide a promising approach to creating soft robotic sheets, due to their flexible form factor, modular integration, and fast response speed. However, their control requires kilo-Volt signals and understanding of complex dynamics resulting from force interactions by on-board and environmental effects. In this work, we demonstrate an untethered planar five-actuator piezoelectric robot powered by batteries and on-board high-voltage circuitry, and controlled through a wireless link. The scalable fabrication approach is based on bonding different functional layers on top of each other (steel foil substrate, actuators, flexible electronics). The robot exhibits a range of controllable motions, including bidirectional crawling (up to ~0.6 cm/s), turning, and in-place rotation (at ~1 degree/s). High-speed videos and control experiments show that the richness of the motion results from the interaction of an asymmetric mass distribution in the robot and the associated dependence of the dynamics on the driving frequency of the piezoelectrics. The robot's speed can reach 6 cm/s with specific payload distribution.

National University of Singapore,Ritsumeikan University

Soft robots have long been attractive to robotic engineers due to their remarkable dexterity; however, reports that standardize soft actuators into modularized off-shelf devices akin to rigid robots are still rare, and the mechanical efficiency of existing designs is still limited. This work identifies origami folding to enable the design of LEGO-like modularized soft actuators with high mechanical efficiency in terms of payload capability and workspace. Herein, three modularized origami actuators that can generate translational, bending, and twisting motion are designed, prototyped, and tested. The translational actuator can contract to 40% of its original length, and the twisting and bending actuators can exert 31° and 52° angular motions, respectively. The translational actuator can exert a blocked force of about 821 times self-weight. The motion of origami soft actuators is accurately modeled using rigid body kinematics, and complex systems built by them are captured by homogeneous transformation. Finally, the modularized design and efficient kinematic model are verified on a manipulator and a reconfigurable letter. Benefiting from the unprecedented modularity and mechanical efficiency, these LEGO-like origami actuators are promising for practical applications like food handling and healthcare.

University College London,Kings College London

Soft robotic structures may play a major role in the 4th industrial revolution. Researchers have successfully demonstrated the advantages of soft robotics over traditional robots made of rigid links and joints in many application areas. Variable stiffness links (VSL) and joints (VSJ) have been investigated to achieve on-demand forces and, at the same time, be inherently safe in interactions with humans. However, a thorough characterisation of soft and rigid robotic components is still required. This paper investigates the influence of antagonistically actuated, stiffness-controllable joint-link units (JLUs) on the performance of collaborative robots (i.e. stiffness, load capacity, repetitive precision) and characterizes the difference compared with rigid units. A JLU is made of a combination of a VSL, a VSJ, and their rigid counterparts. Experimental results show that the VSL has minor differences in terms of stiffness (0.62 ~ 0.95), output force (0.93 ~ 0.94), and repetitive precision compared with the rigid link. For the VSJ, our results show a significant gap compared with the servo motor with regards to maximum stiffness (0.14 ~ 0.21) and repetitive position precision (0.07 ~ 0.25). However, similar performance on repetitive force precision and better performance on the maximum output force (1.54 ~ 1.55 times) are demonstrated.

Queen Mary University of London

One of the biggest problems with soft robots is precisely the fact that they are soft. Indeed the softer they are, the less force they can exert on the environment. Researchers have proposed a number of stiffening methods, but all of them have drawbacks, such as locking the shape of the device in a way that precludes further adjustments. In this paper we propose a stiffening method inspired by the internal structure of the mammalian penis. The soft actuation chamber is divided into small compartments that trap the actuation fluid, leading to locally amplified pressure increase under certain conditions. At the same time, the proposed solution does not affect the actuation mechanism, allowing the actuator to be adjusted in one direction just as if it was in non-stiffened mode, while offering a stiff response in the opposite direction. Our prototype achieves an increase in stiffening of approximately a factor of two. The paper describes the concept, the mathematical justification of the working principle, the prototype design, its implementation and our experimental results.

This paper discusses the effect of axial backbone compression on tendon-driven continuum robots. A new mechanics model for compensating for this effect that does not require tendon tension sensing or knowledge of manipulator material properties/stiffnesses is introduced and analyzed. In addition, we provide an analytical expression for the minimum preload on the tendons to achieve a given bend, a quantity determined empirically thus far. Our model is computationally efficient and achieves real time control on low cost hardware. The analysis is supported by experimental results demonstrating significant improvement over kinematics in open loop control of a tendon-driven continuum hose robot.

South China University of Technology,Apple Inc,Clemson University,Clemson Univeristy

This article proposes a novel semi-supervised learning framework FourStr} (Four-Stream formed by two two-stream models) that focuses on the improvement of fusion and labeling efficiency for 3D multi-sensor detector. FourStr adopts a multi-sensor single-stage detector named adaptive fusion network (AFNet) as the backbone and trains it through the semi-supervision learning (SSL) strategy Stereo Fusion. Note that multi-sensor AFNet and SSL Stereo Fusion can benefit each other. On the one hand, the Four-stream composed of two AFNets naturally provides rich inputs and large models for SSL Stereo Fusion. While other SSL works have to use massive augmentation to obtain rich inputs, and deepen and widen the network for large models. On the other hand, by the novel three fusion stages and Loss Pruning, Stereo Fusion improves the fusion and labeling efficiency for AFNet. Finally, extensive experiments demonstrate that FourStr performs excellently on outdoor dataset (KITTI and Waymo Open Dataset) and indoor dataset (SUN RGB-D), especially for the small contour objects. And compared to the fully-supervised methods, FourStr achieves similar accuracy with only 2% labeled data on KITTI (or with 50% labeled data on SUN RGB-D).

Technische Universität Berlin,TU Berlin

We present a robust method to visually segment scenes into objects based on motion and appearance. Both these cues provide complementary information that we fuse using two interconnected recursive estimators: One estimates object segmentation from motion as a probabilistic clustering of tracked 3D points, and the other estimates object segmentation from appearance as a probabilistic image segmentation. The interconnected estimators provide a probabilistic and consistent object segmentation in real time, which makes them well suited for many downstream robotic tasks. We evaluate our method on one such task, kinematic structure estimation, on a dataset of interactions with articulated objects and show that our fusion improves object segmentation by 70% and in turn estimated kinematic joints by 26% over a purely motion-based approach. Furthermore, we show the necessity of probabilistic modeling for downstream robotic tasks, achieving 339% of the performance of a recent multimodal but deterministic RNN for object segmentation on the estimation of kinematic structure.

Northwestern Polytechnical University

Accurate and timely onboard perception is a prerequisite for mobile robots to operate in highly dynamic scenarios. The bio-inspired event camera can capture more motion details than a traditional camera by triggering each pixel asynchronously and therefore is more suitable in such scenarios. Among various perception tasks based on the event camera, ego-motion removal is one fundamental procedure to reduce perception ambiguities. Recent ego-motion removal methods are mainly based on optimization processes and may be computationally expensive for robot applications. In this paper, we consider the challenging perception task of detecting fast-moving objects from an aggressively operated platform equipped with an event camera, achieving computational cost reduction by directly employing IMU motion measurement. First, we design a nonlinear warping function to capture rotation information from an IMU and to compensate for the camera motion during an asynchronous events stream. The proposed nonlinear warping accuracy by 10%-15%. Afterward, we segmented the moving parts on the warped image through dynamic threshold segmentation and optical flow calculation, and clustering. Finally, we validate the proposed detection pipeline on public datasets and real-world data streams containing challenging light conditions and fast-moving objects.

TU Berlin,Technische Universitaet Berlin,Technische Universität Berlin

Robots need to understand articulated objects, such as drawers. The state of articulated structures is commonly estimated using vision, but visual perception is limited when objects are occluded, have few salient features, or are not in the camera's field of view. Audio sensing does not face these challenges, since sound propagates in a fundamentally different way than light. Therefore we propose to fuse vision and audio sensing to overcome the challenges faced by vision alone. We estimate motion in several drawers and show that an audio-visual approach estimates drawer motion more reliably than only vision -- even in settings where the purely visual approach completely breaks down. Additionally, we perform an in-depth analysis of the regularities that govern how motion in drawers shapes their sound.

Developing embodied agents in simulation has been a key research topic in recent years. Exciting new tasks, algorithms, and benchmarks have been developed in various simulators. However, most of them assume deaf agents in silent environments, while we humans perceive the world with multiple senses. We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear. Sonicverse models realistic continuous audio rendering in 3D environments in real-time. Together with a new audio-visual VR interface that allows humans to interact with agents with audio, Sonicverse enables a series of embodied AI tasks that need audio-visual perception. For semantic audio-visual navigation in particular, we also propose a new multi-task learning model that achieves state-of-the-art performance. In addition, we demonstrate Sonicverse's realism via sim-to-real transfer, which has not been achieved by other simulators: an agent trained in Sonicverse can successfully perform audio-visual navigation in real-world environments. Sonicverse is available at: https://github.com/StanfordVL/Sonicverse.

Inria Grenoble,Inria Grenoble Rhône-Alpes,Hacettepe University,INRIA,Univ Lyon

Semantic grids can be useful representations of the scene around an autonomous system. By having information about the layout of the space around itself, a robot can leverage this type of representation for crucial tasks such as navigation or tracking. By fusing information from multiple sensors, robustness can be increased and the computational load for the task can be lowered, achieving real time performance. Our multi-scale LiDAR-Aided Perspective Transform network uses information available in point clouds to guide the projection of image features to a top-view representation, resulting in a relative improvement in the state of the art for semantic grid generation forhuman (+8.67%) and movable object (+49.07%) classes in the nuScenes dataset, as well as achieving results close to the state of the art for the vehicle, drivable area and walkway classes, while performing inference at 25 FPS.

In this paper, we address the problem of estimating the in-hand 6D pose of an object in contact with multiple vision-based tactile sensors. We reason on the possible spatial configurations of the sensors along the object surface. Specifically, we filter contact hypotheses using geometric reasoning and a Convolutional Neural Network (CNN), trained on simulated object-agnostic images, to promote those that better comply with the actual tactile images from the sensors. We use the selected sensors configurations to optimize over the space of 6D poses using a Gradient Descent-based approach. We finally rank the obtained poses by penalizing those that are in collision with the sensors. We carry out experiments in simulation using the DIGIT vision-based sensor with several objects, from the standard YCB model set. The results demonstrate that our approach estimates object poses that are compatible with actual object-sensor contacts in 87.5% of cases while reaching an average positional error in the order of 2 centimeters. Our analysis also includes qualitative results of experiments with a real DIGIT sensor.

LiDAR-Camera online calibration is of great significance for building a stable autonomous driving perception system. For online calibration, a key challenge lies in constructing a unified and robust representation between multimodal sensor data. Most methods extract features manually or implicitly with an end-to-end deep learning method. The former suffers poor robustness, while the latter has poor interpretability. In this paper, we propose CalibDepth, which uses depth maps as the unified representation for image and LiDAR point cloud. CalibDepth introduces a sub-network for monocular depth estimation to assist online calibration tasks. To further improve the performance, we regard online calibration as a sequence prediction problem, and introduce global and local losses to optimize the calibration results. CalibDepth shows excellent performance in different experimental setups.

INRIA Rennes - Bretagne Atlantique,Centre Inria de l'Université de Rennes,Inria center at University of Rennes

In this paper we propose a visual servoing approach that controls the deformation of a suspended tether cable subject to gravity from visual data provided by a RGB-D camera. The cable shape is modelled with a parabolic curve together with the orientation of the plane containing the tether. The visual features considered are the parabolic coefficients and the yaw angle of that plane. We derive the analytical expression of the interaction matrix that relates the variation of the visual features to the velocities of the cable extremities. Singularities are demonstrated to occur if and only if the cable is taut horizontally or vertically. An image processing algorithm is also developed to extract in real-time the current features fitting the parabola to the cable from the observed point cloud. Simulations and experimental results demonstrate the efficiency of our visual servoing approach to deform the tether cable toward a desired shape configuration.

Université de Rennes ,, IRISA,Université of Rennes ,-- IRISA/Inria rba,Univ Rennes, Inria, CNRS, IRISA

We propose a new visual servoing method that controls a robot's motion in a latent space. We aim to extract the best properties of two previously proposed servoing methods: we seek to obtain the accuracy of photometric methods such as Direct Visual Servoing (DVS), as well as the behavior and convergence of pose-based visual servoing (PBVS). Photometric methods suffer from limited convergence area due to a highly non-linear cost function, while PBVS requires estimating the pose of the camera which may introduce some noise and incurs a loss of accuracy. Our approach relies on shaping (with metric learning) a latent space, in which the representations of camera poses and the embeddings of their respective images are tied together. By leveraging the multimodal aspect of this shared space, our control law minimizes the difference between latent image representations thanks to information obtained from a set of pose embeddings. Experiments in simulation and on a robot validate the strength of our approach, showing that the sought out benefits are effectively found.

Centre for Transformative Garment Production,Tohoku University,The University of Hong Kong

This paper proposes CNN-based visual servoing for simultaneous positioning and flattening of a soft fabric part placed on a table by a dual manipulator system. We propose a network for multimodal data processing of grayscale images captured by a camera and force/torque applied to force sensors. The training dataset is collected by moving the real manipulators, which enables the network to map the captured images and force/torque to the manipulator’s motion in Cartesian space. We apply structured lighting to emphasize the features of the surface of the fabric part since the surface shape of the non-textured fabric part is difficult to recognize by a single grayscale image. Through experiments, we show that the fabric part with unseen wrinkles can be positioned and flattened by the proposed visual servoing scheme.

IDSIA USI-SUPSI,IRISA CNRS UMR,,,,,University of Trento

Nowadays ubiquitous robots must be adaptive and easy to use. To this end, dynamical system-based imitation learning plays an important role. In fact, it allows to realize stable and complex robotic tasks without explicitly coding them, thus facilitating the robot use. However, the adaptation capabilities of dynamical systems have not been fully exploited due to the lack of closed-loop implementations making use of visual feedback. In this regard, the integration of visual information allows higher flexibility to cope with environmental changes. This work presents a dynamical system-based imitation learning for visual servoing, based on the large projection task priority formulation. The proposed scheme enables complex and stable visual tasks, as demonstrated by a simulation analysis and experiments with a robotic manipulator.

Nantes Université, LS,N, IRT Jules Verne,IRT Jules Verne,CNRS/LS,N,Inria center at University of Rennes

Cable-Driven Parallel Robots (CDPRs) are well-adapted to large workspaces since they replace rigid links by cables. However, they lack in positioning accuracy and new control methods are necessary to achieve profile-following tasks. This paper presents a control scheme designed for these tasks, relying on a combination of accurate boarded distance sensors and of a less accurate remote camera. The profile-following task is divided into two subtasks that are partially conflicting: maintaining a parallel orientation and a constant distance with the surface to follow, and following a trajectory between two points on the surface. The data fusion to solve the redundancy is based on the Gradient Projection Method. This control scheme is validated experimentally on a CDPR prototype and shown to provide the expected behaviour.

This paper presents a spectral domain registration-based visual servoing scheme that works on 3D point clouds. Specifically, we propose a 3D model/point cloud alignment method, which works by finding a global transformation between reference and target point clouds using spectral analysis. A 3D Fast Fourier Transformation (FFT) in R3 is used for the translation estimation, and the real spherical harmonics in SO(3) are used for the rotations estimation. Such an approach allows us to derive a decoupled 6 degrees of freedom (DoF) controller, where we use gradient ascent optimisation to minimise translation and rotational costs. We then show how this methodology can be used to regulate a robot arm to perform a positioning task. In contrast to the existing state-of-the-art depth-based visual servoing methods that either require dense depth maps or dense point clouds, our method works well with partial point clouds and can effectively handle larger transformations between the reference and the target positions. Furthermore, the use of spectral data (instead of spatial data) for transformation estimation makes our method robust to sensor-induced noise and partial occlusions. We validate our approach by performing experiments using point clouds acquired by a robot-mounted depth camera. Obtained results demonstrate the effectiveness of our visual servoing approach.

Università degli Studi di Napoli Federico II,Università di Napoli Federico II

This paper presents a novel autonomous endoscope control method for the dVRK’s Endoscopic Camera Manipulator (ECM), which allows the camera to track the surgical instruments on the Patient Side Manipulator (PSM). An Image-based Visual Servoing (IBVS) is enforced by the addition of a visibility constraint that ensures the identified surgical tool remains in the camera’s Field Of View (FOV) for the continued availability of image feedback and a joint limits avoidance constraint that prevents the ECM from exceeding its joint limits. The work relies on an optimization approach, with constraints performed using the Control Barrier Functions concept (CBFs). The goal is to minimize the surgeon’s cognitive and physical workload by removing the time-consuming job of camera reorientation, offering an enforced method compared to the traditional IBVS endoscopic camera controller.

Safe motion control in unknown environments is one of the challenging tasks in robotics, such as autonomous navigation. Control Barrier Function (CBF), as a strong mathematical tool, has been widely used in many safety-critical systems to satisfy safety requirements. However, there are only a handful of recent studies on safety controllers with perception inputs. Common assumptions in most of the works are that the CBF is already known and obstacles have predefined shapes. In this work, we introduce a novel Vision-based Control Barrier Function (V-CBF), which enables generalization to new environments and obstacles of arbitrary shapes. We then derive CBF safety conditions over RGB-D space and relate those to actual robot control inputs. To train the CBF function, we introduce a method to generate ground truth with desired properties complying with CBF and a method to generate part of the CBF as an image-to-image translation problem. We finally demonstrate the efficacy of V-CBF on the safe control of an autonomous car in CARLA simulator.

Jilin University,Changchun University of Science and Technology

In this paper, we propose a multi-object tracking framework for videos captured by UAVs, considering motion imperfection in the following two aspects: 1) motion blurring of objects due to high-speed motion of the UAV and the objects, deteriorating the performance of the detector; 2) motion coupling of the global movement of the UAV camera with the object motion, resulting in the objects trajectory in adjacent frames more difficult to predict. For motion blurring, this paper proposes a hybrid deblurring module that deals with the blurred frames while retaining the clear frames, trading off between video tracking performance and spatio-temporal consistency. For motion coupling, we proposed a motion compensation module to align adjacent frames by feature matching, and the corrected target position is obtained in the next frame to alleviate the interference of camera movement with tracking. We evaluate the proposed methods on VisDrone dataset and validate that our framework achieves new state-of-the-art performance on UAV-based MOT systems.

The University of Hong Kong,The Chinese University of Hong Kong,Beihang University,City University, HONG KONG,University of Hong Kong

Motion deblurring is a critical ill-posed problem that is important in many vision-based robotics applications. The recently proposed event-based double integral (EDI) provides a theoretical framework for solving the deblurring problem with the event camera and generating clear images at high frame-rate. However, the original EDI is mainly designed for offline computation and does not support real-time requirement in many robotics applications. In this paper, we propose the fast EDI, an efficient implementation of EDI that can achieve real-time online computation on single-core CPU devices, which is common for physical robotic platforms used in practice. In experiments, our method can handle event rates at as high as 13 million event per second in a wide variety of challenging lighting conditions. We demonstrate the benefit on multiple downstream real-time applications, including localization, visual tag detection, and feature matching.

University of Technology Sydney,Imperial College London

This work addresses the issue of motion compensation and pattern tracking in event camera data. An event camera generates asynchronous streams of events triggered independently by each of the pixels upon changes in the observed intensity. Providing great advantages in low-light and rapid-motion scenarios, such unconventional data present significant research challenges as traditional vision algorithms are not directly applicable to this sensing modality. The proposed method decomposes the tracking problem into a local SE(2) motion-compensation step followed by a homography registration of small motion-compensated event batches. The first component relies on Gaussian Process (GP) theory to model the continuous occupancy field of the events in the image plane and embed the camera trajectory in the covariance kernel function. In doing so, estimating the trajectory is done similarly to GP hyperparameter learning by maximising the log marginal likelihood of the data. The continuous occupancy fields are turned into distance fields and used as templates for homography-based registration. By benchmarking the proposed method against other state-of-the-art techniques, we show that our open-source implementation performs high-accuracy motion compensation and produces high-quality tracks in real-world scenarios.

Current robotic hand manipulation narrowly operates with objects in predictable positions in limited environments. Thus, when the location of the target object deviates severely from the expected location, a robot sometimes responds in an unexpected way, especially when it operates with a human. For safe robot operation, we propose the EXit-aware Object Tracker (EXOT) on a robot hand camera that recognizes an object's absence during manipulation. The robot decides whether to proceed by examining the tracker's bounding box output containing the target object. We adopt an out-of-distribution classifier for more accurate object recognition since trackers can mistrack a background as a target object. To the best of our knowledge, our method is the first approach of applying an out-of-distribution classification technique to a tracker output. We evaluate our method on the first-person video benchmark dataset, TREK-150, and on the custom dataset, RMOT-223, that we collect from the UR5e robot. Then we test our tracker on the UR5e robot in real-time with a conveyor-belt sushi task, to examine the tracker's ability to track target dishes and to determine the exit status. Our tracker shows 38% higher exit-aware performance than a baseline method. The dataset and the code will be released at https://github.com/hskAlena/EXOT.

Rutgers University,Columbia University

We present Mono-STAR, the first real-time RGB-D 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods.