| A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Session | Paper Title | Author List | Keywords | Abstract | |||||||||||||||||||||
2 | Award Finalists 1 | Neural MP: A Neural Motion Planner | Dalal, Murtaza;Yang, Jiahui;Mendonca, Russell;Khaky, Youssef;Salakhutdinov, Ruslan;Pathak, Deepak | Keywords: Big Data in Robotics and Automation, Data Sets for Robot Learning, Machine Learning for Robot Control | Abstract: The current paradigm for motion planning generates solutions from scratch for every new problem, which consumes significant amounts of time and computational resources. For complex, cluttered scenes, motion planning approaches can often take minutes to produce a solution, while humans are able to accurately and safely reach any goal in seconds by leveraging their prior experience. We seek to do the same by applying data-driven learning at scale to the problem of motion planning. Our approach builds a large number of complex scenes in simulation, collects expert data from a motion planner, then distills it into a reactive neural policy. We then combine this with lightweight optimization to obtain a safe path for real world deployment. We perform a thorough evaluation of our method on 64 motion planning tasks across four diverse environments with randomized poses, scenes and obstacles, in the real world, demonstrating an improvement of 23%, 17% and 79% motion planning success rate over state of the art sampling, optimization and learning based planning methods. All code, models and datasets will be released on acceptance. Video results available at mihdalal.github.io/neuralmotionplanner. | |||||||||||||||||||||
3 | Award Finalists 1 | Interactive Navigation for Legged Manipulators with Learned Arm-Pushing Controller | Bi, Zhihai;Chen, Kai;Zheng, Chunxin;Li, Yulin;Li, Haoang;Ma, Jun | Keywords: Mobile Manipulation, Autonomous Vehicle Navigation, Collision Avoidance | Abstract: Interactive navigation is crucial in scenarios where proactively interacting with objects can yield shorter paths, thus significantly improving traversal efficiency. While existing methods primarily rely on body-velocity-based pushing for relocation of large obstacles (which could be comparable to the size of a robot), they prove ineffective in narrow or constrained spaces where the robot's dimensions restrict its manipulation capabilities, thus compromising clearance and overall navigation performance. This paper introduces a novel interactive navigation framework for legged manipulators, featuring an active arm-pushing mechanism for effective obstacle clearance. The framework enables the robot to dynamically detect and reposition movable obstacles in space-constrained environments, leading to more feasible and efficient navigation paths. At the core of this framework, we develop a reinforcement learning-based arm-pushing controller with a two-stage reward strategy for large-object manipulation. Specifically, this strategy progressively refines the pushing behavior, first guiding the manipulator to a pre-pushing zone for kinematically feasible contact configuration, and then maintaining end-effector positioning at appropriate contact points for stable object displacement while preventing toppling. Through simulations and ablation studies, we validate the robustness of the arm-pushing controller, showing that the two-stage reward strategy improves policy convergence and long-term performance in large-object manipulation. Real-world experiments further underscore its effectiveness in confined environments with movable obstacles, achieving shorter paths and reduced traversal time. The open-source project can be found at url{https://github.com/Zhihaibi/Interactive-Navigation-for -legged-manipulator.git}. | |||||||||||||||||||||
4 | Award Finalists 1 | Implicit Disparity-Blur Alignment for Fast and Precise Autofocus in Robotic Microsurgical Imaging | Fu, Pan;Li, Zhen;Zhang, Ming-Yang;Zhai, Yu-Peng;Wang, Junzheng;He, Wenhao;Bian, Gui-Bin | Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Visual Servoing | Abstract: Creating an intelligent surgical environment requires not only advanced robotic systems but also optimized microscopic imaging. However, autofocus remains a fundamental challenge, with current methods suffering from slow iterative processes or directional ambiguity, which compromises real-time performance. This paper presents an implicit disparity-blur alignment approach for robotic microsurgical autofocus, integrating stereo geometryÂ’s monotonic depth cues with defocus characteristics for rapid convergence. A novel physics-guided dual-stream network is developed to encode implicit depth representations through hierarchical cross-pathway feature fusion, enabling reliable focus prediction without explicit stereo matching in blur-degraded regions. An ROI-aware attention module is proposed to dynamically optimize focus-critical regions, coupled with learnable physics-guided kernel learning for precise Z-offset estimation. The approach achieves a top directional accuracy of 94.85% and a single-pass focus error of 0.20 mm with an inference time of 53 ms on a surgical dataset, which outperforms state-of-the-art methods in reducing iteration count by 22.8% and inference time by 51.8%. An intelligent robotic microscope prototype is developed, with validation through ex vivo tests demonstrating its ability to enable fast and precise multi-region focusing for microsurgeries. | |||||||||||||||||||||
5 | Award Finalists 1 | Accelerating Layered Manufacturing-Based 3D Printing through Optimized Non-Printing Travel-Path Planning and Infill Strategies | Wang, Liuyin;Hua, Weijian;Jin, Yifei;Shen, Yantao | Keywords: Additive Manufacturing, Intelligent and Flexible Manufacturing | Abstract: This paper presents a rapid 3D printing framework that enhances the efficiency of commercially available layered manufacturing-based 3D printers. Unlike traditional methods that simplify printing regions to a single point or rely on predefined entry and exit points, our approach utilizes an improved Traveling Salesman Problem (TSP) algorithm to autonomously generate an optimized, cyclic printing path, while automatically assigning entry and exit points for each region. This minimizes non-printing paths and improves efficiency. Additionally, we propose a principal axis calculation method for irregular shapes, aligning better with geometric orientation. This optimization enhances infill uniformity and surface smoothness. Simulations and experimental results demonstrate that the proposed framework improves printing efficiency while maintaining print quality, with promising applicability to large-scale and complex 3D printing models. | |||||||||||||||||||||
6 | Award Finalists 1 | Designing a Magnetic Endoscope for in Vivo Contact-Based Tissue Scanning Using Developable Roller | Greenidge, Nikita Jasmine;Marzi, Christian;Calmé, Benjamin;Martin, James William;Scaglioni, Bruno;Mathis-Ullrich, Franziska;Valdastri, Pietro | Keywords: Medical Robots and Systems, Sensor-based Control, Underactuated Robots | Abstract: Magnetic manipulation has been adopted as a method of actuation in both wireless capsule endoscopy and softtethered endoscopy, with the goal of improving gastrointestinal procedures. However, by nature of magnetic manipulation, these endoscopes are typically limited to a maximum of five degrees of freedom (DoF). With the need to introduce additional contactbased sensing modalities for subsurface investigation into these systems as well as to improve overall dexterity, it is both practically and clinically beneficial to recover the lost DoF i.e. the roll around the main axis. This paper presents a method of achieving the magnetic manipulation of an underactuated device by leveraging developable surfaces, specifically, the oloid shape. The design of a clinically relevant magnetic endoscope with all its ancillary elements, as well as contact sensors, is proposed and demonstrated in vivo. The contact sensor data from the in vivo experiments show that for sweeping motions over 100�of roll, contact between the endoscope’s sensor region and the colon wall can be maintained for 74% of the motion. | |||||||||||||||||||||
7 | Award Finalists 1 | PhysGCN-DL: Physics-Informed Graph Convolutional Networks with Diversity-Aware Loss Optimization for Multimodal Pedestrian Trajectory Prediction | Jiang, ZiHan;Liu, Ruonan;Yibo, Zhou;Lu, Haibo;Yang, Boyuan;Lin, Di;Zhang, Weidong | Keywords: Intelligent Transportation Systems, Deep Learning Methods, AI-Based Methods | Abstract: Pedestrian trajectory prediction ensures safe navigation in autonomous driving and intelligent robots. Existing methods, including physical models, machine learning, and deep learning, have shown promising results but still face challenges in handling dynamic environments, social interactions, and high-dimensional data. In this paper, we propose a novel PhysGCN-DL model within the itransformer framework to address these challenges. Our model incorporates physically-inspired dynamic interaction modeling by representing physical interactions between pedestrians as edge weights in graph convolution. This approach captures the heterogeneity of pedestrian movement and improves the interpretability of social interactions. Additionally, we introduce a new loss function that balances diversity and accuracy in trajectory prediction, ensuring robustness in dense and sparse environments. Experimental results demonstrate the superiority of our model in predicting diverse and accurate pedestrian trajectories compared to existing methods. | |||||||||||||||||||||
8 | Award Finalists 1 | Reinforcement Learning Assist-As-Needed Control Promotes Recovery of Walking Speed Following Ankle Weight Perturbations | Li, Andy;Li, Haoran;Teker, Aytac;Hernandez-Rocha, Mariana;Gebre, Biruk;Nolan J., Karen;Pochiraju, Kishore;Zanotto, Damiano | Keywords: Rehabilitation Robotics, Prosthetics and Exoskeletons, Wearable Robotics | Abstract: Self-selected walking speed is a key outcome for exercise-based rehabilitation programs following lower-extremity trauma. This work introduces a novel reinforcement learning-based assist-as-needed (RL-AAN) controller for ankle exoskeletons, aimed at gait speed training. Built on an actor–critic architecture, the RL-AAN controller integrates a control objective that balances the trade-off between expected stride velocity (SV) errors and exoskeleton assistance. This approach allows the exoskeleton to progressively reduce ankle plantar- and dorsiflexion (PDF) assistance as the user’s performance improves, promoting active participation. The desired assistive torque is computed as the product of the actor output and the wearer’s biomechanical ankle PDF moment, estimated by a subject-agnostic model, thereby ensuring personalized and biomechanically relevant assistance. In a proof-of-concept study with healthy individuals walking on a self-paced treadmill with ankle weights, the RL-AAN controller outperformed a conventional Fixed-K controller—achieving greater immediate speed increases during assisted walking (14.2% vs. 10.0% relative to unassisted perturbed walking) and inducing short-term gait speed adaptation post-training, not observed with the conventional controller. These findings highlight the potential of RL-AAN control for subject-tailored gait training, with promising clinical implications for exercise-based rehabilitation in individuals with neurological or musculoskeletal gait impairments. | |||||||||||||||||||||
9 | Award Finalists 1 | Autonomous Hiking Trail Navigation Via Semantic Segmentation and Geometric Analysis | Reed, Camndon;Arend Tatsch, Christopher Alexander;Gross, Jason;Gu, Yu | Keywords: Field Robots, Vision-Based Navigation, Robotics and Automation in Agriculture and Forestry | Abstract: Natural environments pose significant challenges for autonomous robot navigation, particularly due to their unstructured and ever-changing nature. Hiking trails, with their dynamic conditions influenced by weather, vegetation, and human traffic, represent one of these challenges. This work introduces a novel approach to autonomous hiking trail navigation that balances trail adherence with the flexibility to adapt to off-trail routes when necessary. The solution is a Traversability Analysis module that integrates semantic data from camera images with geometric information from LiDAR to create a comprehensive understanding of the surrounding terrain. A planner uses this traversability map to navigate safely, adhering to trails while allowing off-trail movement when necessary to avoid on-trail hazards or for safe off-trail shortcuts. The method is evaluated through simulation to determine the balance between semantic and geometric information in traversability estimation. These simulations tested various weights to assess their impact on navigation performance across different trail scenarios. Weights were then validated through autonomous field tests at the West Virginia University Core Arboretum, demonstrating the method's effectiveness in a real-world environment. | |||||||||||||||||||||
10 | Mobile Manipulation 1 | GeT-USE: Learning Generalized Tool Usage for Bimanual Mobile Manipulation Via Simulated Embodiment Extensions | Wu, Bohan;de La Sayette, Paul;Fei-Fei, Li;MartÃn-MartÃn, Roberto | Keywords: Mobile Manipulation, Bimanual Manipulation, Deep Learning in Grasping and Manipulation | Abstract: The ability to use random objects as tools in a generalizable manner is a missing piece in robots' intelligence today to boost their versatility and problem-solving capabilities. State-of-the-art robotic tool usage methods focused on procedurally generating or crowd-sourcing datasets of tools for a task to learn how to grasp and manipulate them for that task. However, these methods assume that only one object is provided and that it is possible, with the correct grasp, to perform the task; they are not capable of identifying, grasping, and using the best object for a task when many are available, especially when the optimal tool is absent. In this work, we propose GeT-USE, a two-step procedure that learns to perform real-robot generalized tool usage by learning first to extend the robot's embodiment in simulation and then transferring the learned strategies to real-robot visuomotor policies. Our key insight is that by exploring a robot's embodiment extensions (i.e., building new end-effectors) in simulation, the robot can identify the general tool geometries most beneficial for a task. This learned geometric knowledge can then be distilled to perform generalized tool usage tasks by selecting and using the best available real-world object as tool. On a real robot with 22 degrees of freedom (DOFs), GeT-USE outperforms state-of-the-art methods by 30-60% success rates across three vision-based bimanual mobile manipulation tool-usage tasks under unseen real-world objects, environments, and lighting conditions. | |||||||||||||||||||||
11 | Mobile Manipulation 1 | Globally-Guided Geometric Fabrics for Reactive Mobile Manipulation in Dynamic Environments | Merva, Tomas;Bakker, Saray;Spahn, Max;Zhao, Danning;Virgala, Ivan;Alonso-Mora, Javier | Keywords: Mobile Manipulation, Constrained Motion Planning, Collision Avoidance | Abstract: Mobile manipulators operating in dynamic environments shared with humans and robots must adapt in real time to environmental changes to complete their tasks effectively. While global planning methods are effective at considering the full task scope, they lack the computational efficiency required for reactive adaptation. In contrast, local planning approaches can be executed online but are limited by their inability to account for the full taskÂ’s duration. To tackle this, we propose Globally-Guided Geometric Fabrics (G3F), a framework for real-time motion generation along the full task horizon, by interleaving an optimization-based planner with a fast reactive geometric motion planner, called Geometric Fabrics (GF). The approach adapts the path and explores a multitude of acceptable target poses, while accounting for collision avoidance and the robotÂ’s physical constraints. This results in a real-time adaptive framework considering whole-body motions, where a robot operates in close proximity to other robots and humans. We validate our approach through various simulations and real-world experiments on mobile manipulators in multi-agent settings, achieving improved success rates compared to vanilla GF, Prioritized Rollout Fabrics and Model Predictive Control. | |||||||||||||||||||||
12 | Mobile Manipulation 1 | PlaceNet: Obstacle Aware Mobile Manipulator Base Placement through Deep Learning | Navarro, Alex;Pryor, Mitchell | Keywords: Mobile Manipulation, Deep Learning in Grasping and Manipulation, Task and Motion Planning | Abstract: In this work, we present PlaceNet: a deep learning framework for mobile manipulator base placement which provides solutions to the shortcomings common in the state-of-the-art. Our method addresses the lack of obstacle awareness of reachability methods and the limited generalization of learning methods. Using only the raw pointcloud and task pose data as input, PlaceNet learns the concepts of reachability and obstacle occlusions in an environment-independent manner, enabling its use in situations outside its training experiences. Tests comparing PlaceNet to inverse reachability and heuristic methods demonstrated state-of-the-art performance in both the In-Distribution and Out-Of-Distribution test sets, achieving as high as 98% success rate for problems with many solutions, and an 82% success rate overall. PlaceNet can be trained on grounded pointcloud data from any source without the need for dynamic simulation, marking it as an accessible alternative to similar frameworks which require expensive, high-performance GPUs for running simultaneous simulation and training or which depend on labor intensive data collection. PlaceNet is lightweight during deployment and can easily run with low latency on affordable hardware, including laptop GPUs and the NVIDIA Jetson line for embedded deployment. | |||||||||||||||||||||
13 | Mobile Manipulation 1 | Benchmarking Long-Horizon Mobile Manipulation in Multi-Room Dynamic Environments | Zhang, Junbo;Ma, Kaisheng | Keywords: Mobile Manipulation, Deep Learning Methods, Visual Learning | Abstract: Long-horizon reasoning and task execution are crucial for complex mobile manipulation tasks in household environments. Existing benchmarks and methods primarily focus on single-room or single-object mobile manipulation scenarios, limiting the scope of long-horizon planning and scene-level understanding. To address this gap, we introduce a novel benchmark for long-horizon mobile manipulation in multi-room household environments. Our task requires agents to follow a sequence of language instructions, each directing the movement of specific objects across receptacles and rooms. In this task, we investigate the role of long-term memory by constructing a hierarchical scene graph that captures the relationships between objects, furniture, and rooms. This scene graph-based memory is dynamically updated as the agent explores the environment, which effectively aligns the scene information with the targets and environmental context specified in the language instructions. Additionally, we benchmark the proposed task in dynamic environments where objects can be relocated during task execution, simulating real-world scenarios. Our results demonstrate that the scene graph-based memory significantly improves the agent's performance in long-horizon mobile manipulation tasks. Moreover, dynamically updating the state of objects within the scene graph enables the agent to better adapt to dynamic conditions. | |||||||||||||||||||||
14 | Mobile Manipulation 1 | Manipulate-To-Navigate: Reinforcement Learning with Visual Affordances and Manipulability Priors | Zhang, Yuying;Pajarinen, Joni | Keywords: Mobile Manipulation, Legged Robots, Reinforcement Learning | Abstract: Mobile manipulation in dynamic environments is challenging due to movable obstacles blocking the robotÂ’s path. Traditional methods, which treat navigation and manipulation as separate tasks, often fail in such "manipulate-to-navigate" scenarios, as obstacles must be removed before navigation. In these cases, active interaction with the environment is required to clear obstacles while ensuring sufficient space for movement. To address the manipulate-to-navigate problem, we propose a reinforcement learning-based approach for learning manipulation actions that facilitate subsequent navigation. Our method combines manipulability priors to focus the robot on high manipulability body positions with affordance maps for selecting high-quality manipulation actions. By focusing on feasible and meaningful actions, our approach reduces unnecessary exploration and allows the robot to learn manipulation strategies more effectively. We present two new manipulate-to-navigate simulation tasks called Reach and Door with the Boston Dynamics Spot robot. The first task tests whether the robot can select a good hand position in the target area such that the robot base can move effectively forward while keeping the end effector position fixed. The second task requires the robot to move a door aside in order to clear the navigation path. Both of these two tasks need first manipulation and then navigating the base forward. Results show that our method allows a robot to effectively interact with and traverse dynamic environments. Finally, we transfer the learned policy to a real Boston Dynamics Spot robot, which successfully performs the Reach task. | |||||||||||||||||||||
15 | Mobile Manipulation 1 | Dynamic Open-Vocabulary 3D Scene Graphs for Long-Term Language-Guided Mobile Manipulation | Yan, Zhijie;Li, Shufei;Wang, Zuoxu;Wu, Lixiu;Wang, Han;Zhu, Jun;Chen, Lijiang;Liu, Jihong | Keywords: Mobile Manipulation, Long term Interaction, Semantic Scene Understanding | Abstract: Enabling mobile robots to perform long-term tasks in dynamic real-world environments is a formidable challenge, especially when the environment changes frequently due to human-robot interactions or the robot's own actions. Traditional methods typically assume static scenes, which limits their applicability in the continuously changing real world. To overcome these limitations, we present DovSG, a novel mobile manipulation framework that leverages dynamic open-vocabulary 3D scene graphs and a language-guided task planning module for long-term task execution. DovSG takes RGB-D sequences as input and utilizes vision-language models (VLMs) for object detection to obtain high-level object semantic features. Based on the segmented objects, a structured 3D scene graph is generated for low-level spatial relationships. Furthermore, an efficient mechanism for locally updating the scene graph, allows the robot to adjust parts of the graph dynamically during interactions without the need for full scene reconstruction. This mechanism is particularly valuable in dynamic environments, enabling the robot to continually adapt to scene changes and effectively support the execution of long-term tasks. We validated our system in real-world environments with varying degrees of manual modifications, demonstrating its effectiveness and superior performance in long-term tasks. Our project page is available at https://bjhyzj.github.io/dovsg-web. | |||||||||||||||||||||
16 | Mobile Manipulation 1 | CushionCatch: A Compliant Catching Mechanism for Mobile Manipulators Via Combined Optimization and Learning | Chen, Bingjie;Fan, Keyu;Yang, Qi;Cheng, Yi;Liu, Houde;Dong, Kangkang;Xia, Chongkun;Han, Liang;Liang, Bin | Keywords: Mobile Manipulation, Manipulation Planning, Learning from Demonstration | Abstract: Catching flying objects with a cushioning process is a skill commonly performed by humans, yet it remains a significant challenge for robots. In this paper, we present a framework that combines optimization and learning to achieve compliant catching on mobile manipulators (CCMM). First, we propose a high-level capture planner for mobile manipulators (MM) that calculates the optimal capture point and joint configuration. Next, the pre-catching (PRC) planner ensures the robot reaches the target joint configuration as quickly as possible. To learn compliant catching strategies, we propose a network that leverages the strengths of LSTM for capturing temporal dependencies and positional encoding for spatial context (P-LSTM). This network is designed to effectively learn compliant strategies from human demonstrations. Following this, the post-catching (POC) planner tracks the compliant sequence output by the P-LSTM while avoiding potential collisions due to structural differences between humans and robots. We validate the CCMM framework through both simulated and real-world ball-catching scenarios, achieving a success rate of 98.70% in simulation, 92.59% in real-world tests, and a 28.7% reduction in impact torques. The open source code has be released for the reference of the community. Waiting for Solar LLM summarize... | Predictive Reachability for Embodiment Selection in Mobile Manipulation Behaviors | Feng, Xiaoxu;Horii, Takato;Nagai, Takayuki | Keywords: Mobile Manipulation, Reinforcement Learning | Abstract: Mobile manipulators require coordinated control between navigation and manipulation to accomplish tasks. Typically, coordinated mobile manipulation behaviors have base navigation to approach the goal followed by arm manipulation to reach the desired pose. Selecting the embodiment between the base and arm can be determined based on reachability. Previous methods evaluate reachability by computing inverse kinematics and activate arm motions once solutions are identified. In this study, we introduce a new approach called predictive reachability that decides reachability based on predicted arm motions. Our model utilizes a hierarchical policy framework built upon a world model. The world model allows the prediction of future trajectories and the evaluation of reachability. The hierarchical policy selects the embodiment based on the predicted reachability and plans accordingly. Unlike methods that require prior knowledge about robots and environments for inverse kinematics, our method only relies on image-based observations. We evaluate our approach through basic reaching tasks across various environments. The results demonstrate that our method outperforms previous model-based approaches in both sample efficiency and performance, while enabling more reasonable embodiment selection based on predictive reachability. | |||||||||||||||||
17 | In-Hand Manipulation | Learning Dexterous In-Hand Manipulation with Multifingered Hands Via Visuomotor Diffusion | Koczy, Piotr;Welle, Michael C.;Kragic, Danica | Keywords: In-Hand Manipulation, Imitation Learning, Multifingered Hands | Abstract: We present a framework for learning dexterous in-hand manipulation with multifingered hands using visuomotor diffusion policies. Our system enables complex in-hand manipulation tasks, such as unscrewing a bottle lid with one hand, by leveraging a fast and responsive teleoperation setup for the four-fingered Allegro Hand. We collect high-quality expert demonstrations using an augmented reality (AR) interface that tracks hand movements and applies inverse kinematics and motion retargeting for precise control. The AR headset provides real-time visualization, while gesture controls streamline teleoperation. To enhance policy learning, we introduce a novel demonstration outlier removal approach based on HDBSCAN clustering and the Global-Local Outlier Score from Hierarchies (GLOSH) algorithm, effectively filtering out low-quality demonstrations that could degrade performance. We evaluate our approach extensively in real-world settings and provide all experimental videos on the project website: https://dex-manip.github.io/ | |||||||||||||||||||||
18 | In-Hand Manipulation | Wearable Roller Rings to Augment In-Hand Manipulation through Active Surfaces | Webb, Hayden;Chanrungmaneekul, Podshara;Yuan, Shenli;Hang, Kaiyu | Keywords: In-Hand Manipulation, Multifingered Hands | Abstract: In-hand manipulation is a crucial ability for reorienting and repositioning objects within grasps. The main challenges in this are not only the complexity of the computational models, but also the risks of grasp instability caused by active finger motions, such as rolling, sliding, breaking, and remaking contacts. This paper presents the development of the Roller Ring (RR), a modular robotic attachment with active surfaces that is wearable by both robot and human hands to manipulate without lifting a finger. By installing the angled RRs on hands, such that their spatial motions are not colinear, we derive a general differential motion model for manipulating objects. Our motion model shows that complete in-hand manipulation skill sets can be provided by as few as only 2 RRs through non-holonomic object motions, while more RRs can enable enhanced manipulation dexterity with fewer motion constraints. Through extensive experiments, we test the RRs on both a robot hand and a human hand to evaluate their manipulation capabilities. We show that the RRs can be employed to manipulate arbitrary object shapes to provide dexterous in-hand manipulation. | |||||||||||||||||||||
19 | In-Hand Manipulation | Open-Loop Deep Reinforcement Learning Control of Soft Robotic In-Hand Manipulations | Suske, Gabriel;Pilch, Samuel;Beger, Artem;Heidingsfeld, Julia Laura;Sawodny, Oliver | Keywords: In-Hand Manipulation, Modeling, Control, and Learning for Soft Robots, Reinforcement Learning | Abstract: In-hand manipulation tasks using hand-like robotic grippers offer a promising approach to accomplish various tasks in a human-centered environment. Due to the inherent safety of soft robots, in-hand manipulations performed by soft robots provide great opportunities for future human-robot collaboration, which is the scope of this paper. By modeling a new and innovative soft-robotic gripper known as the Anthropomorphic Soft Gripper and synthesis of an open-loop controller with deep reinforcement learning, it is demonstrated how the movement of objects by in-hand manipulations can be accomplished. Moreover, this work explores the application of deep reinforcement learning methods without the employment for domain randomization. As noted by Bhatt et al. the inherent soft properties of soft robotic grippers enable remarkably robust in-hand manipulation in open-loop control, giving the impetus for the approach that is being followed in this work. Motion sequences generated in simulation are successfully transferred to the real anthropomorphic soft gripper and validated in experiments. | |||||||||||||||||||||
20 | In-Hand Manipulation | Vibration-Induced Friction Modulation to Enable Controlled Sliding for In-Hand Manipulation | Mane, Shambhuraj;Jagetia, Anuj;Naukudkar, Samruddhi;Morgan, Andrew;Calli, Berk | Keywords: In-Hand Manipulation, Grippers and Other End-Effectors, Dexterous Manipulation | Abstract: Achieving controlled sliding of objects on finger surfaces is a significant challenge for robots, substantially constraining their ability to perform complex in-hand manipulation tasks. In this work, we investigate the role of surface vibration in modulating the effective friction at the object-finger contact locations to facilitate controlled sliding. We demonstrate that friction at contact points can be reduced by applying targeted vibrations at specific locations on a robotic finger, creating regions that are suitable for sliding. In this way, we create sticking/sliding regions on finger surfaces on demand and can easily switch between sliding and rolling contacts. To investigate this phenomenon, we embedded an array of vibration modules into robotic fingers. We first analyzed the velocity fields created by surface vibrations on a single finger. Then, we developed a method to select the appropriate activation states of the modules that achieve the desired velocity field at a given object location. Utilizing these fingers and the vibration selection method, we formed a two-finger robotic hand and demonstrated controlled sliding and rotation of a held object within the hand. To the best of our knowledge, this is the first work that utilizes vibration-induced friction modulation for in-hand manipulation that can achieve combinations of object sliding and rolling actions. | |||||||||||||||||||||
21 | In-Hand Manipulation | On the Role of Jacobians in Robust Manipulation | Grace, Joshua;Chanrungmaneekul, Podshara;Hang, Kaiyu;Dollar, Aaron | Keywords: In-Hand Manipulation, Dexterous Manipulation | Abstract: Traditional robot control relies on analytical methods that require precise system models, which are hard to apply in real-world settings and limit generalization to arbitrary tasks. However, systems like serial manipulators and passively adaptive hands feature inherently stable regions without control discontinuities like loss of contact or singularities. In these regions, approximate controllers focusing on the correct direction of motion enable successful coarse manipulation. When coupled with a rough estimation of the motion magnitude, precision manipulation is achieved. Leveraging this insight, we introduce a novel inverse Jacobian estimation method that independently estimates the primary motion direction and magnitude of the manipulator's actuators. Our method efficiently estimates the direct mapping from task to actuator space with no need for a priori system knowledge enabling the same framework to control both hands and arms without compromising task performance. We present a novel control method with no a priori knowledge for precision manipulation. Experiments on the Yale Model O hand, Yale Stewart Hand, and a UR5e arm demonstrate that the inverse Jacobians estimated via our approach enable real-time control with submillimeter precision in manipulation tasks. These results highlight that online self-ID data alone is sufficient for precise real-world manipulation. | |||||||||||||||||||||
22 | Robot Safety 1 | Semantically Safe Robot Manipulation: From Semantic Scene Understanding to Motion Safeguards | Brunke, Lukas;Zhang, Yanni;Römer, Ralf;Naimer, Jack;Staykov, Nikola;Zhou, Siqi;Schoellig, Angela P. | Keywords: Robot Safety, AI-Enabled Robotics | Abstract: Ensuring safe interactions in human-centric environments requires robots to understand and adhere to constraints recognized by humans as "common sense" (e.g., "moving a cup of water above a laptop is unsafe as the water may spill" or "rotating a cup of water is unsafe as it can lead to pouring its content"). Recent advances in computer vision and machine learning have enabled robots to acquire a semantic understanding of and reason about their operating environments. While extensive literature on safe robot decision-making exists, semantic understanding is rarely integrated into these formulations. In this work, we propose a semantic safety filter framework to certify robot inputs with respect to semantically defined constraints (e.g., unsafe spatial relationships, behaviors, and poses) and geometrically defined constraints (e.g., environment-collision and self-collision constraints). In our proposed approach, given perception inputs, we build a semantic map of the 3D environment and leverage the contextual reasoning capabilities of large language models to infer semantically unsafe conditions. These semantically unsafe conditions are then mapped to safe actions through a control barrier certification formulation. We demonstrate the proposed semantic safety filter in teleoperated manipulation tasks and with learned diffusion policies applied in a real-world kitchen environment that further showcases its effectiveness in addressing practical semantic safety constraints. Together, these experiments highlight our approach's capability to integrate semantics into safety certification, enabling safe robot operation beyond traditional collision avoidance. | |||||||||||||||||||||
23 | Robot Safety 1 | Analysis and Mitigation of Inconsistencies in Blockchain-Enabled Robot Swarms | Simionato, Giada;Strobel, Volker;Cimino, Mario G. C. A.;Dorigo, Marco | Keywords: Swarm Robotics, Multi-Robot Systems, Robot Safety | Abstract: Recent research has demonstrated that blockchain-enabled robot swarms—where robots coordinate using blockchain technology— can secure robot swarms by neutralizing malicious and malfunctioning robots. This security is achieved through blockchain technology’s consistency properties. However, prior work addressed malfunctions at the information level, that is, it studied how to neutralize robots that stored information in the blockchain that did not correspond to the real-world state (i.e., it studied the oracle problem). In contrast, this study focuses on inconsistencies at the blockchain protocol level. We analyze how network partitions, which may arise from robots’ local-only communication capabilities, malfunctioning hardware, or external attacks, can lead to inconsistent information in a robot swarm. In order to mitigate these disruptions, we propose a decentralized approach to detect partitions and a corresponding response. We study our approach in a swarm robotics simulator, where we demonstrate its effectiveness in reducing blockchain inconsistencies. | |||||||||||||||||||||
24 | Robot Safety 1 | Integrating Opinion Dynamics into Safety Control for Decentralized Airplane Encounter Resolution | Qi, Shuhao;Tang, Zhiqi;Sun, Zhiyong;Haesaert, Sofie | Keywords: Robot Safety, Collision Avoidance | Abstract: As the airspace becomes increasingly congested, decentralized conflict resolution methods for airplane encounters have become essential. While decentralized safety controllers can prevent dangerous midair collisions, they do not always ensure prompt conflict resolution. As a result, airplane progress may be blocked for extended periods in certain situations. To address this blocking phenomenon, this paper proposes integrating bio-inspired nonlinear opinion dynamics into the airplane safety control framework, thereby guaranteeing both safety and blocking-free resolution. In particular, opinion dynamics enable the safety controller to achieve collaborative decision-making for blocking resolution and facilitate rapid, safe coordination without relying on communication or preset rules. Extensive simulation results validate the improved flight efficiency and safety guarantees. This study provides practical insights into the design of autonomous controllers for airplanes. | |||||||||||||||||||||
25 | Robot Safety 1 | Robots That Suggest Safe Alternatives | Jeong, Hyun Joe;Chen, Rosy;Bajcsy, Andrea | Keywords: Robot Safety, Safety in HRI, Reinforcement Learning | Abstract: Goal-conditioned policies, such as those learned via imitation learning, provide an easy way for humans to influence what tasks robots accomplish. However, these robot policies are not guaranteed to execute safely or to succeed when faced with out-of-distribution goal requests. In this work, we enable robots to know when they can confidently execute a user's desired goal, and automatically suggest safe alternatives when they cannot. Our approach is inspired by control-theoretic safety filtering, wherein a safety filter minimally adjusts a robot's candidate action to be safe. Our key idea is to pose alternative suggestion as a safe control problem in goal space, rather than in action space. Offline, we use reachability analysis to compute a goal-parameterized reach-avoid value network which quantifies the safety and liveness of the robotÂ’s pre-trained policy. Online, our robot uses the reach-avoid value network as a safety filter, monitoring the human's given goal and actively suggesting alternatives that are similar but meet the safety specification. We demonstrate our Safe ALTernatives (SALT) framework in simulation experiments with Franka Panda tabletop manipulation. We find that SALT is able to learn to predict successful and failed closed-loop executions, is a less pessimistic monitor than open-loop uncertainty quantification, and proposes alternatives that consistently align with those that people find acceptable. | |||||||||||||||||||||
26 | Robot Safety 1 | Safe, Task-Consistent Manipulation with Operational Space Control Barrier Functions | Morton, Daniel;Pavone, Marco | Keywords: Robot Safety, Manipulation Planning, Optimization and Optimal Control | Abstract: Safe real-time control of robotic manipulators in unstructured environments requires handling numerous safety constraints without compromising task performance. Traditional approaches, such as artificial potential fields (APFs), suffer from local minima, oscillations, and limited scalability, while model predictive control (MPC) can be computationally expensive. Control barrier functions (CBFs) offer a promising alternative due to their high level of robustness and low computational cost, but these safety filters must be carefully designed to avoid significant reductions in the overall performance of the manipulator. In this work, we introduce an Operational Space Control Barrier Function (OSCBF) framework that integrates safety constraints while preserving task-consistent behavior. Our approach scales to hundreds of simultaneous constraints while retaining real-time control rates, ensuring collision avoidance, singularity prevention, and workspace containment even in highly cluttered settings or during dynamic motions. By explicitly accounting for the task hierarchy in the CBF objective, we prevent degraded performance across both joint-space and operational-space tasks, when at the limit of safety. We validate performance in both simulation and hardware, and release our open-source high-performance code and media on our project webpage, https://stanfordasl.github.io/oscbf/ | |||||||||||||||||||||
27 | Robot Safety 1 | Secure Safety Filter: Towards Safe Flight Control under Sensor Attacks | Tan, Xiao;Sundar, Junior;Bruzzone, Renzo;Ong, Pio;T. Lunardi, Willian;Andreoni, Martin;Tabuada, Paulo;Ames, Aaron | Keywords: Robot Safety, Robust/Adaptive Control, Collision Avoidance | Abstract: Modern autopilot systems are prone to sensor attacks that can jeopardize flight safety. To mitigate this risk, we proposed a modular solution: the secure safety filter, which extends the well-established control barrier function (CBF)-based safety filter to account for, and mitigate, sensor attacks. This module consists of a secure state reconstructor (which generates plausible states) and a safety filter (which computes the safe control input that is closest to the nominal one). Differing from existing work focusing on linear, noise-free systems, the proposed secure safety filter handles bounded measurement noise and, by leveraging reduced-order model techniques, is applicable to the nonlinear dynamics of drones. Software-in-the-loop simulations and drone hardware experiments demonstrate the effectiveness of the secure safety filter in rendering the system safe in the presence of sensor attacks. | |||||||||||||||||||||
28 | Robot Safety 1 | SHIELD: Safety on Humanoids Via CBFs in Expectation on Learned Dynamics | Yang, Lizhi;Werner, Blake;Cosner, Ryan;Fridovich-Keil, David;Culbertson, Preston;Ames, Aaron | Keywords: Robot Safety, Machine Learning for Robot Control, Collision Avoidance | Abstract: Robot learning has produced remarkably effective “black-box” controllers for complex tasks such as dynamic locomotion on humanoids. Yet ensuring dynamic safety, i.e., constraint satisfaction, remains challenging for such policies. Reinforcement learning (RL) embeds constraints heuristically through reward engineering, and adding or modifying constraints requires retraining. Model-based approaches, like control barrier functions (CBFs), enable runtime constraint specification with formal guarantees but require accurate dynamics models. This paper presents SHIELD, a layered safety framework that bridges this gap by: (1) training a generative, stochastic dynamics residual model using real-world data from hardware rollouts of the nominal controller, capturing system behavior and uncertainties; and (2) adding a safety layer on top of the nominal (learned locomotion) controller that leverages this model via a stochastic discrete-time CBF formulation enforcing safety constraints in probability. The result is a minimally-invasive safety layer that can be added to the existing autonomy stack to give probabilistic guarantees of safety that balance risk and performance. In hardware experiments on an Unitree G1 humanoid, SHIELD enables safe navigation (obstacle avoidance) through varied indoor and outdoor environments using a nominal (unknown) RL controller and onboard perception. | |||||||||||||||||||||
29 | Robot Safety 1 | Feasibility Analysis of Real-Time Robustness Certification | Seferis, Emmanouil;Kollias, Stefanos | Keywords: Robot Safety, Deep Learning Methods, Vision-Based Navigation | Abstract: The robustness certification of deep neural networks (DNNs) is crucial in many safety-critical domains. Randomized Smoothing (RS) has emerged as the current state-of-the-art method for DNN robustness verification that successfully scales on large DNNs used in practice, has achieved excellent results, and has been extended for a large variety of adversarial perturbation scenarios. However, an important cost in RS is during inference, since it requires passing tens or hundreds of thousands of perturbed samples through the DNN to perform the verification. In this work we aim to address this, and explore what happens as we decrease the number of samples by orders of magnitude, and the effect on the certified radius. Surprisingly, we find that emph{the performance reduction in terms of average certified radius is not too large, even if we decrease the number of samples by two orders of magnitude, or more}. Moreover, we find that the resulting certified radius reduction can be mitigated using off-the-self methods designed to improve RS performance. This can pave the way for dramatically faster robustness certification, unlocking the possibility of performing it in real-time, which we demonstrate. We perform a detailed analysis, both theoretically and experimentally, and show promising results on the standard CIFAR-10 and ImageNet datasets. | |||||||||||||||||||||
30 | Motion Control 1 | Design and Flight Control of a Novel Thrust-Vectored Tricopter Using Twisting and Tilting Rotors | Li, Xinliang;Chen, Zheyu;Wei, Jingbo;Qin, Zijie;Chen, Weijian;Liu, Kun | Keywords: Aerial Systems: Mechanics and Control, Motion Control | Abstract: This paper presents a novel, compact overactuated tricopter featuring a servo-driven twisting and tilting mechanism, preventing the adverse effects of internal force contradiction during flight. Each arm's vectored thrust is provided by a single motor, with the twisting and tilting angles controlled by two vertically mounted servos. These components are collectively mounted within a 3D-printed semi-ring structure, and are rigidly attached to the fuselage via carbon tubes at the twist end. To address the asymmetry inherent in the tricopter configuration, we conducted a qualitative analysis of the disturbances introduced by the actuators. Additionally, we emphasize the need to include gyroscopic torque effects caused by arm rotations. This issue is addressed using a control allocation method, with our proposed improved Force Decomposition (FD)-based iteration offering a low-cost computational solution. The dynamic models of the motion of rotational joints, identified and employed as virtual sensors, contribute to the estimation of the improved control effective matrix. This overactuated tricopter can operate like a conventional rotorcraft, with the added capability of achieving attitude adjustments through manual control inputs. Finally, we demonstrate the tricopterÂ’s advantages by comparing simulations and flight experiments, both with and without the application of the improved method. | |||||||||||||||||||||
31 | Motion Control 1 | Online Anti-Swing Trajectory Refinement for Variable-Length Cable-Suspended Aerial Transportation Robot | Yu, Hai;Yang, Zhichao;He, Wei;Han, Jianda;Fang, Yongchun;Liang, Xiao | Keywords: Aerial Systems: Mechanics and Control, Intelligent Transportation Systems, Underactuated Robots | Abstract: Aerial robots have demonstrated significant potential in suspended cargo transportation, especially in industries such as logistics and food delivery. Due to the underactuated and nonlinear dynamics of the cable-suspended system, directly tracking a given trajectory with a multicopter without modifying its controller often leads to significant payload swing. This compromises the safety and stability of the cargo. To address the aforementioned issue, this paper proposes an online trajectory refinement method for a variable-length cable-suspended aerial transportation robot, independent from the control layer. By incorporating payload swing angle information, the reference trajectory is refined in real-time, effectively suppressing payload oscillations during transportation. Specially, Lyapunov techniques and LaSalle's invariance theorem are employed to rigorously guarantee the feasibility of the designed trajectory refinement scheme. Finally, hardware experiments are conducted to validate the effectiveness and superiority of the proposed method. The results demonstrate that the refined trajectory not only enables precise positioning of the multicopter, but also effectively suppresses payload oscillations during transportation, significantly enhancing the safety and reliability of the aerial cargo delivery. | |||||||||||||||||||||
32 | Motion Control 1 | Differential-Flatness-Based Tracking Control for Tractor-Trailers in Reversing Maneuvers | Yang, Bo;Zhuang, Zhenhao;Yu, Zitian;Wang, Qian;Wei, Junqing;Mo, Yilin;Yang, Wen | Keywords: Autonomous Vehicle Navigation, Wheeled Robots, Motion Control | Abstract: In this paper, we propose a differential-flatness-based control~(DFBC) framework for precise tracking control of tractor-trailers, particularly during reversing maneuvers, which are challenging due to the unstable nonlinear kinematics. The proposed controller leverages the differential flatness property of tractor-trailers, equivalently transforming the nonlinear kinematics into a Brunovsky canonical from, thus allowing the incorporation of a linear feedback mechanism within the space defined by the flat outputs and their derivatives. Compared to the traditional linear quadratic regulator~(LQR) controllers based on Jacobian linearization, the proposed DFBC method achieves higher precision and stability in reversing maneuvers. We also implement the proposed DFBC method on a self-developed 1/10 scale autonomous tractor-trailer to showcase its effectiveness in real-world scenarios. | |||||||||||||||||||||
33 | Motion Control 1 | Steering Elongate Multi-Legged Robots by Modulating Body Undulation Waves | Flores, Esteban;Chong, Baxi;Soto, Daniel;Goldman, Daniel | Keywords: Biologically-Inspired Robots, Redundant Robots, Motion Control | Abstract: Centipedes exhibit great maneuverability in diverse environments due to their many legs and body-driven control. By leveraging similar morphologies and control strategies, their robotic counterparts also demonstrate effective terrestrial locomotion. However, the success of these multi-legged robots is largely limited to forward locomotion; steering is substantially less studied, in part because of the difficulty in coordinating a high degree-of-freedom robot to follow predictable, planar trajectories. To resolve these challenges, we take inspiration from control schemes based on geometric mechanics(GM) in elongate systems' locomotion through highly damped environments. We model the elongate, multi-legged system as a ``terrestrial swimmer" in highly frictional environments and implement steering schemes derived from low-order templates. We identify an effective turning strategy by superimposing two traveling waves of lateral body undulation and further explore variations of the ``turning wave" to enable a spectrum of arc-following steering primitives. We test our hypothesized modulation scheme on a robophysical model and validate steering trajectories against theoretically predicted displacements producing steering radii between 0 and 0.6 body length. We then apply our control framework to Ground Control Robotics' elongate multi-legged robot, Major Tom, using these motion primitives to autonomously navigate around obstacles and corners on indoor and outdoor terrain. Our work creates a systematic framework for controlling these highly mobile devices in the plane using a low-order model based on sequences of body shape changes. | |||||||||||||||||||||
34 | Motion Control 1 | LESO-Based NMPC Tracking Control of Climbing Robot on Large Components with Variable Curvature (I) | Tan, Ke;Gong, Zeyu;Tao, Bo;Zhang, Yuhao;Shi, Ying;Gu, Zhenfeng;Wu, Chong;Ding, Han | Keywords: Climbing Robots, Industrial Robots, Motion Control | Abstract: Wheeled climbing robots have great application prospects in the machining of large components with variable curvature. However, its accurate motion control on variable curvature surfaces faces two fatal challenges. The varied contact states between the robotÂ’s wheels and the variable curvature surfaces make it difficult to establish an accurate kinematics model. Additionally, there exists a different degree of robot slippage when the robot moves in different attitudes due to the dragging effect of gravity. To overcome the above problems, we first present a kinematics modeling method with instantaneous plane constraints on a variable curvature surface. Subsequently, a linear extended state observer (LESO)-based nonlinear model predictive control (NMPC) scheme is designed, in which the NMPC is used to calculate the nominal control inputs and the LESO is used to estimate and compensate for lumped disturbance brought by the robot slippage and surface constraints. Experiments on a real wind turbine blade with variable curvature show that the proposed control scheme can well eliminate the influence of lumped disturbance, and the climbing robots can achieve unbiased (AVG < 0.1 mm) and high-precision (RMSE < 2 mm) trajectory tracking. provides a solution for automatic and high-precision trajectory tracking for climbing robot with localization system in factories, offering the potential for high-quality machining of large components. | |||||||||||||||||||||
35 | Motion Control 1 | Adaptive Observer-Based Sliding Mode Control for Piezoelectric Nanopositioning System (I) | Chen, Liheng;Tan, Yongjie;Fu, Shasha | Keywords: Robust/Adaptive Control, Motion Control | Abstract: This paper proposes a novel adaptive global sliding mode control strategy for a two-degree-of-freedom (2- DOF) piezoelectric nanopositioning system based on the integral extended state observer (IESO) technique. Its uniqueness is that a global robustness property is generated in the whole precision motion control process, which effectively circumvents sensitivity to the perturbations during the reaching phase. First, to generate an accurate disturbance estimation for compensation control, the IESO is constructed by incorporating an integral action into the observer design. Then, a global sliding mode control approach is developed for the nanopositioning system to ensure global robustness against the unknown hysteresis nonlinearity and crossaxis coupling motion. Moreover, an adaptive rule is established for the global sliding mode controller, which does not require a priori knowledge of the estimation error, hysteresis, and crosscoupling nonlinearity in the control design. Experimental studies are conducted to demonstrate the effectiveness and superiority of the proposed motion control scheme over existing ones. | |||||||||||||||||||||
36 | Micro/Nano Robots 1 | An Intelligent Skeleton Based on Liquid Metal for Biohybrid Actuator Powered by Muscle | Lu, Xiaoqi;Zhang, Yuyin;Gan, Yuanjie;Gao, Shen;Wang, Yue;Liu, Na;Yue, Tao | Keywords: Micro/Nano Robots, Soft Sensors and Actuators, Soft Robot Materials and Design | Abstract: Biological machines that use biological cells and soft materials in combination to obtain a sense of the environment driven by bioenergy and generate driving force are called biohybrid actuators. With the development of tissue engineering and organoid technology, researchers have applied biohybrid actuators technology to the research of precision medicine and targeted drug delivery, but the research on feedback and evaluation of biohybrid actuation performance is limited to visual and simulation calculations. Therefore, we hope to develop an intelligent crawling skeleton for sensing function, which can be used to evaluate the actuation ability of muscle actuators, and eventually realize the high-precision control of biohybrid actuators. In this work, an intelligent crawling skeleton based on three-dimensional liquid metal is proposed to detect and feedback the crawling of C2C12 muscle actuators. Three-dimensional muscle tissue was composed of mixing hydrogels and cells, and the functionalization of muscle rings was promoted using static mechanical forces and external electric field stimulation. The composite crawling skeleton is fabricated by inverting mold and soft lithography technology. The skeleton can adapt to large deformations above 90 degrees and is more sensitive to deformations by adjusting materials with different elastic modulus. Inspired by the tendon-bone structure, the intelligent crawling skeleton can obtain the deformation degree of the biohybrid actuator in the crawling process according to the characteristics of the deformation from the muscle tissue, and put forward a good idea for the feedback and closed-loop control of the biohybrid actuators. | |||||||||||||||||||||
37 | Micro/Nano Robots 1 | Design of DNA Origami-Engineered Tetrahedral Nanorobots | Chen, Haowen;Liu, Fengyu;Huang, Qiang;Arai, Tatsuo;Liu, Xiaoming | Keywords: Micro/Nano Robots, Nanomanufacturing, Automation at Micro-Nano Scales | Abstract: DNA nanorobots have emerged as a promising technology in the biomedical field, owing to their nanoscale dimensions, programmable structures, and minimal physiological toxicity. However, existing DNA-based carriers often exhibit limited stability in complex biological environments and lack efficient targeted drug delivery capabilities. Addressing these challenges requires the development of DNA nanorobots with highly controllable and multifunctional designs for precise biomedical applications. This study introduces a conformational transition design of tetrahedral DNA structures, leveraging a multi-resolution molecular dynamics simulation model validated by all-atom molecular modeling. Through this approach, we dynamically analyzed the stability of the nanomechanical structures using simulation-based methods. Utilizing this design strategy, we successfully constructed a deformable tetrahedral DNA nanorobot characterized by high stability and efficient drug delivery capabilities. Furthermore, we demonstrated the precise release of therapeutic agents with low physiological toxicity and high controllability by employing the DNA nanorobot for in vitro targeting of circulating tumor cells (CTCs). These findings highlight the potential of the proposed DNA origami-based nanorobot design method to advance the development of nanorobots in biomedicine and their application in targeted therapeutic interventions. | |||||||||||||||||||||
38 | Micro/Nano Robots 1 | On-Chip Dynamic Mechanical Characterization: From Cells to Nucleus | Ge, Jingjin;Chen, Zhuo;Bai, Chenhao;Liu, Fengyu;Li, Yuke;Kojima, Masaru;Huang, Qiang;Arai, Tatsuo;Liu, Xiaoming | Keywords: Micro/Nano Robots, Medical Robots and Systems | Abstract: 自动表å¾�机械性能 æœ¬ç ”ç©¶æ��出了一ç§�新的肿瘤和æ£å¸¸ç»†èƒž 使用窄微通é�“诱导和测é‡�的方法 细胞å�˜å½¢ã€‚动æ€�力å¦è¡¨å¾?串行连接微通é�“的技术模æ‹?肿瘤细胞在生物å¦ä¸Šçš„å�˜å½¢å’Œè¿�ç§?相关环境，实现明确差异化 肿瘤和æ£å¸¸ç»†èƒžä¹‹é—´é€šè¿‡è¿žç» 压缩。高速æˆ�åƒ�结å�ˆè®¡ç®—机 视觉和图åƒ�处ç�†æŠ€æœ¯æœ‰åŠ©äºŽå¿«é€?以å�Šå¯¹è‚¿ç˜¤ç»†èƒžçš„准确自动分æž�。使用三ç§�肿瘤细胞系和三ç§�æ£å¸¸ç»†èƒžç³»çš„实éª?细胞系验è¯�了该方法。æ¤å¤–ï¼Œè¿™é¡¹ç ”ç©¶ æ�ç¤ºäº†ç»†èƒžæ ¸çš„æœºæ¢°æ€§èƒ½ 确定整体细胞力å¦ï¼Œä½¿ç”?肿瘤细胞和æ£å¸¸ç»†èƒžä¹‹é—´çš„å·®å¼‚å½’å› äº?原å�æ ¸åŠ›å¦çš„å�˜åŒ–。这ç§�方法表æ˜?早期癌症诊æ–çš„å‰�景ã€?Micro/Nano Robots 1 | A Study on the Generation of Single Cell Droplets Via the Combination of Lateral-Field Optoelectronic Tweezers and Electrowetting-On-Dielectric | Zhao, Jiawei;Huang, Shunxiao;Xiong, Hongyi;Gan, Chunyuan;Ye, Jingwen;Niu, Wenyan;Feng, Lin | Keywords: Micro/Nano Robots, Biological Cell Manipulation, Automation at Micro-Nano Scales | Abstract: Microfluidic technology is currently a popular approach in the field of single-cell research, which is used to reveal the heterogeneity among cells. However, most of the existing microfluidic technologies for single-cell research lack the ability to control the microenvironment of single cells after isolating them. In this work, a technology that combines lateral field optoelectronic tweezers (LOET) with electrowetting-on dielectric (EWOD) is used to separate cells into single cells and then encapsulate each single cell within an individual droplet, generating single-cell droplets. More importantly, it also enables the control of the microenvironment of the separated single cells. The driving control of the single- cell droplets is achieved through the EWOD, which has good application prospects in the field of single-cell research. | |||||||||||||||||
39 | Micro/Nano Robots 1 | Achieving Lift-To-Weight Ratio >3.5 in Piezoelectric Direct-Driven Insect-Scale Flapping-Wing MAVs | Lu, Xiang;Chen, Jie;Chen, Yang;Deng, Zixin;Wu, Yulie;Wu, Xuezhong;Xiao, Dingbang | Keywords: Micro/Nano Robots, Biologically-Inspired Robots, Mechanism Design | Abstract: Insect-scale flapping-wing micro aerial vehicles (FWMAVs) employing piezoelectric direct-drive configurations eliminate traditional kinematic chains through direct coupling of the wing and actuator. While this design approach significantly reduces structural complexity and manufacturing costs compared to transmission-dependent systems, it inherently limits wing stroke amplitude and consequent lift generation. This paper presents a novel lift-enhancement strategy for piezoelectric direct-drive FWMAVs, effectively improving payload capacity through optimized aerodynamic performance. The redesigned X-configuration prototype demonstrates outstanding metrics: 68 mm wingspan with 212 mg total mass achieves 7.47 mN maximum lift (exceeding 3.5:1 lift-to-weight ratio) and 1.25 m/s takeoff speed. Experimental validation confirms 39% payload capacity improvement and 34% lift-to-weight ratio enhancement compared to baseline designs. This enhancement establishes our robot as the current state-of-the-art in piezoelectric direct-drive FWMAVs regarding lift-to-weight ratio. | |||||||||||||||||||||
40 | Micro/Nano Robots 1 | Dual-Mode Motion Control of Multi-Stimulus Deformable Miniature Robots with Adaptive Orientation Compensation in Unstructured Environments | Zhong, Shihao;Li, Wenbo;Yang, Haotian;Niu, Zhenyang;Hou, Yaozhen;Huang, Qiang;Wang, Huaping | Keywords: Micro/Nano Robots, Automation at Micro-Nano Scales, Soft Robot Materials and Design | Abstract: Miniature robots hold great promise for performing micromanipulation tasks within hard-to-reach confined spaces. However, effectively maneuvering across complex and unstructured terrain, achieving adaptive morphogenesis, and developing adaptive multimodal locomotion strategies remain challenges for these robotic systems. Here, we develop a multi-stimulus-responsive deformable miniature robot integrated with an adaptive multimodal motion control method. Sodium alginate hydrogel and graphene-coated magnetic elastomer are integrated into the sheet-shaped robot to enable responsiveness to temperature, humidity, and magnetic fields. A kinematic gait model is designed to control oscillatory motion in the semi-contracted state and rotational motion in the fully contracted state of the miniature robot. To automatically mitigate angular deviation between the robot's motion direction and the intended path, an adaptive orientation compensation control algorithm based on Support Vector Regression (SVR) is proposed. Experimental results demonstrate that the proposed robot exhibits capabilities for flexible and accurate navigation within unstructured environments (e.g., rock piles and stomach models), and is further shown to be capable of cargo transport. The proposed adaptive morphogenesis robots, enabled by dual-mode motion control, hold significant potential for targeted delivery and other micromanipulation applications in complex, unstructured, and confined environments. | |||||||||||||||||||||
41 | Micro/Nano Robots 1 | Enhanced Rolling Motion of Magnetic Microparticles by Turning Interface Lubrication | Li, Yuke;Liang, Xiyue;Chen, Zhuo;Liao, Hongzhe;Zhao, Yue;Kojima, Masaru;Huang, Qiang;Arai, Tatsuo;Liu, Xiaoming | Keywords: Micro/Nano Robots, Medical Robots and Systems, Dynamics | Abstract: Micro-nano robots must break the symmetry of the flow field to generate net displacement in the low Reynolds number environment. The spherical micro-robots utilize the frictional forces generated through interaction with the surface. We designed a magnetic microroller robot powered by the rotating AC magnetic field. Here, we employed dual measurements of laser ranging and computer vision to demonstrate that a single 100 μm microroller maintains a lubrication film of 1 to 15 μm with the surface during normal motion. We found that the translational velocity of the microroller is correlated with the lubrication film thickness. Based on the robot's gravity, we controlled an additional downward gradient magnetic field to effectively increase the load of robot and reduce the lubrication film thickness, thereby controllably increasing the translational velocity of the robot. For example, the gradient magnetic field generated by superimposing a 30 mA direct current input can reduce the lubrication film thickness from 8 μm to 4 μm in a 10 Hz rotating magnetic field, and increase the translational velocity from 230 μm/s to 460 μm/s. The enhancement of the robot's motion performance enables it to better control its movement in fluids. Finally, we validated the strategy for controllable acceleration of micro-scale particles rolling on surfaces, applied to control fluid motion in multiple arteries within blood vessels. These results offer deeper insights into the physical motion mechanism of surface robots and hold significant implications for future applications in biomedical engineering. | |||||||||||||||||||||
42 | Micro/Nano Robots 1 | On-Demand Motion Conversion of Magnetic Helical Microrobots Using Chemistry and Microstructural-Modified Surface Wettability Modulation | Hou, Yaozhen;Bai, Shanming;Siyi, Li;Nie, Ruhao;Du, Jiabao;Huang, Qiang;Wang, Huaping | Keywords: Micro/Nano Robots, Automation at Micro-Nano Scales, Robotics and Automation in Life Sciences | Abstract: Magnetic helical microrobots have been widely applicated in environmental remediation, sensing, targeted medical applications, and so on. However, for locomotion and manipulation in unstructured liquid environments, the capabilities of distinguished motions and on-demand parking/starting over a team of microrobot are essential. Here, we propose a method for achieving on-demand motions conversion of helical microrobots by modulating surface wettability through surface chemical modification and surface microstructural modifications. An obvious difference shows that microrobot after chemical modification exhibit hydrophilicity and microrobot after microstructural modification exhibit hydrophobicity, where the latter possess higher moving step-out frequency and maximum forward velocity compared to the microrobot after surface chemical modification. The step-out frequencies and maximum velocities of the three types of microrobots (chemistry-modified, unmodified, and pimples-modified) are 13 Hz, 16 Hz, 22 Hz, and 385 μm/s, 511 μm/s, 649 μm/s. Furthermore, our method has demonstrated that can be employed to achieve effective on-demand targeted motions and modal conversion in liquid environment. We anticipate that the method can be potentially employed to achieve precise targeted drug delivery and surgery in biomedical applications | |||||||||||||||||||||
43 | Motion and Path Planning 1 | Dynamic Risk-Aware MPPI for Mobile Robots in Crowds Via Efficient Monte Carlo Approximations | Trevisan, Elia;Mustafa, Khaled;Notten, Godert Christiaan;Wang, Xinwei;Alonso-Mora, Javier | Keywords: Motion and Path Planning, Collision Avoidance, Robot Safety | Abstract: Deploying mobile robots safely among humans requires the motion planner to account for the uncertainty in the other agents' predicted trajectories. This remains challenging in traditional approaches, especially with arbitrarily shaped predictions and real-time constraints. To address these challenges, we propose a Dynamic Risk-Aware Model Predictive Path Integral control (DRA-MPPI), a motion planner that incorporates uncertain future motions modelled with potentially non-Gaussian stochastic predictions. By leveraging MPPI’s gradient-free nature, we propose a method that efficiently approximates the joint Collision Probability (CP) among multiple dynamic obstacles for several hundred sampled trajectories in real-time via a Monte Carlo (MC) approach. This enables the rejection of samples exceeding a predefined CP threshold or the integration of CP as a weighted objective within the navigation cost function. Consequently, DRA-MPPI mitigates the freezing robot problem while enhancing safety. Real-world and simulated experiments with multiple dynamic obstacles demonstrate DRA-MPPI’s superior performance compared to state-of-the-art approaches, including Scenario-based Model Predictive Control (S-MPC), Frenét planner, and vanilla MPPI. Videos of the experiments can be found at https://autonomousrobots.nl/paper_websites/dra-mppi. | |||||||||||||||||||||
44 | Motion and Path Planning 1 | Towards Map-Agnostic Policies for Adaptive Informative Path Planning | Rückin, Julius;Morilla-Cabello, David;Stachniss, Cyrill;Montijano, Eduardo;Popovic, Marija | Keywords: Motion and Path Planning, Aerial Systems: Perception and Autonomy, Reinforcement Learning | Abstract: Robots are frequently tasked to gather relevant sensor data in unknown terrains. A key challenge for classical path planning algorithms used for autonomous information gathering is adaptively replanning paths online as the terrain is explored given limited onboard compute resources. Recently, learning-based approaches emerged that train planning policies offline and enable computationally efficient online replanning performing policy inference. These approaches are designed and trained for terrain monitoring missions assuming a single specific map representation, which limits their applicability to different terrains. To address this limitation, we propose a novel formulation of the adaptive informative path planning problem unified across different map representations, enabling training and deploying planning policies in a larger variety of monitoring missions. Experimental results validate that our novel formulation easily integrates with classical non-learning-based planning approaches while maintaining their performance. Our trained planning policy performs similarly to state-of-the-art map-specifically trained policies. We validate our learned policy on unseen real-world terrain datasets. | |||||||||||||||||||||
45 | Motion and Path Planning 1 | Vision-Language Guided Adaptive Robot Action Planning: Responding to Intermediate Results and Implicit Human Intentions | Cai, Weihao;Mori, Yoshiki;Shimada, Nobutaka | Keywords: Motion and Path Planning, AI-Based Methods, Human-Robot Collaboration | Abstract: Recent advances in research have demonstrated that Vision-Language Models (VLMs) are a promising technology for robot task planning. This paper presents a novel approach that leverages visual prompts and VLMs to generate feasible robot action sequences for achieving shared tasks through human-robot collaboration while simultaneously estimating human intentions. Our method enhances VLMsÂ’ understanding of the environment by utilizing annotations (bounding boxes and labels) and dynamically infers human intentions based on changing environmental conditions to generate optimal robot action sequences to achieve common goals. Additionally, the system incorporates a mechanism to regenerate new sequences through VLM analysis when action failures or external interference occur. Furthermore, by designing prompts as versatile modules for diverse tasks, our proposed technology offers a new approach to robot action planning that excels in both efficiency and adaptability. | |||||||||||||||||||||
46 | Motion and Path Planning 1 | Causal-Planner: Causal Interaction Disentangling with Episodic Memory Gating for Autonomous Planning | Yuan, Yibo;Fang, Jianwu;Zhou, Yang;Yang, Zhao;Lv, Chen;Xue, Jianru | Keywords: Motion and Path Planning, Autonomous Agents, Imitation Learning | Abstract: Autonomous vehicle trajectory planning faces significant challenges in dynamic traffic environments due to the complex and mixed causal relationships between critical scene elements (e.g., pedestrians, vehicles, road markings) and safe decision-making. To identify the causal factors influencing planning outcomes, we propose Causal-Planner, which disentangles the scene interaction graph into causal and confounding components via attention-based adversarial graph learning. Additionally, we introduce a long-short-term episodic memory gating (LSTEM) module that enhances causal interaction disentangling by adaptively capturing evolving causal relationships in dynamic scenarios through bidirectional gated memory fusion. Extensive experiments on the nuPlan dataset suggest that Causal-Planner achieves competitive performance, performing well in both Test-random and Test-hard scenarios under open-loop and closed-loop evaluations. | |||||||||||||||||||||
47 | Motion and Path Planning 1 | Motion Planning and Control with Unknown Nonlinear Dynamics through Predicted Reachability | Zhang, Zhiquan;Puthumanaillam, Gokul;Vora, Manav;Ornik, Melkior | Keywords: Motion and Path Planning, Autonomous Agents, Planning, Scheduling and Coordination | Abstract: Autonomous motion planning under unknown nonlinear dynamics presents significant challenges. An agent needs to continuously explore the system dynamics to acquire its properties, such as reachability, in order to guide system navigation adaptively. In this paper, we propose a hybrid planning-control framework designed to compute a feasible trajectory toward a target. Our approach involves partitioning the state space and approximating the system by a piecewise affine (PWA) system with constrained control inputs. By abstracting the PWA system into a directed weighted graph, we incrementally update the existence of its edges via affine system identification and reach control theory, introducing a predictive reachability condition by exploiting prior information of the unknown dynamics. Heuristic weights are assigned to edges based on whether their existence is certain or remains indeterminate. Consequently, we propose a framework that adaptively collects and analyzes data during mission execution, continually updates the predictive graph, and synthesizes a controller online based on the graph search outcomes. We demonstrate the efficacy of our approach through simulation scenarios involving a mobile robot operating in unknown terrains, with its unknown dynamics abstracted as a single integrator model. | |||||||||||||||||||||
48 | Motion and Path Planning 1 | BSSM: GPU-Accelerated Point-Cloud Distance Metric for Motion Planning | Gonçalves, Vinicius Mariano;Krishnamurthy, Prashanth;Tzes, Anthony;Khorrami, Farshad | Keywords: Motion and Path Planning, Collision Avoidance, Computational Geometry | Abstract: We propose the BSSM: Point-Cloud based (B)iased (S)igned (S)mooth (M)etric, which is used to compute a distance metric between a manipulator and its environment. Unlike many methods that requires that the environment is modeled using simple geometric primitives such as spheres, boxes, and cylinders, our proposed metric directly utilizes point clouds. The proposed metric has properties of being smooth (infinitely differentiable), signed (yielding non-zero values upon overlap), and biased. The latter is a novel feature that, as demonstrated by our simulation results, offers advantages for motion planning. This metric is suitable for GPU parallelization and simulation studies and a real experiment are offered to investigate its benefits. | |||||||||||||||||||||
49 | Motion and Path Planning 1 | Neural Configuration Distance Function for Continuum Robot Control | Long, Kehan;Parwana, Hardik;Fainekos, Georgios;Hoxha, Bardh;Okamoto, Hideki;Atanasov, Nikolay | Keywords: Motion and Path Planning, Collision Avoidance, Motion Control | Abstract: This paper presents a novel method for modeling the shape of a continuum robot as a Neural Configuration Euclidean Distance Function (N-CEDF). By learning separate distance fields for each link and combining them through the kinematics chain, the learned N-CEDF provides an accurate and computationally efficient representation of the robot's shape. The key advantage of a distance function representation of a continuum robot is that it enables efficient collision checking for motion planning in dynamic and cluttered environments, even with point-cloud observations. We integrate the N-CEDF into a Model Predictive Path Integral (MPPI) controller to generate safe trajectories for multi-segment continuum robots. The proposed approach is validated for continuum robots with various links in several simulated environments with static and dynamic obstacles. | |||||||||||||||||||||
50 | Motion and Path Planning 1 | Reactive Model Predictive Contouring Control for Robot Manipulators | Yoon, Junheon;Baek, Woo-Jeong;Park, Jaeheung | Keywords: Motion and Path Planning, Collision Avoidance, Optimization and Optimal Control | Abstract: This contribution presents a robot path-following framework via Reactive Model Predictive Contouring Control (RMPCC) that successfully avoids obstacles, singularities and self-collisions in dynamic environments at 100 Hz. Many path-following methods rely on the time parametrization, but struggle to handle collision and singularity avoidance while adhering kinematic limits or other constraints. Specifically, the error between the desired path and the actual position can become large when executing evasive maneuvers. Thus, this paper derives a method that parametrizes the reference path by a path parameter and performs the optimization via RMPCC. In particular, Control Barrier Functions (CBFs) are introduced to avoid collisions and singularities in dynamic environments. A Jacobian-based linearization and Gauss-Newton Hessian approximation enable solving the nonlinear RMPCC problem at 100 Hz, outperforming state-of-the-art methods by a factor of 10. Experiments confirm that the framework handles dynamic obstacles in real-world settings with low contouring error and low robot acceleration. | |||||||||||||||||||||
51 | Medical Robots and Systems 1 | Development of a Novel Miniaturized Dexterous Manipulator with Variable Stiffness for NOTES | Cong, Rong;Wu, Xipeng;Qian, Chao;Zhang, Kaijie;Duan, Xing-guang;Li, Changsheng | Keywords: Flexible Robotics, Medical Robots and Systems, Mechanism Design | Abstract: Natural Orifice Transluminal Endoscopic Surgery (NOTES) holds great promise due to its ability to eliminate external incisions, reduce trauma, and accelerate recovery. However, the adoption of NOTES is hindered by the limited capabilities of existing instruments, particularly in achieving the required balance between compact size, dexterity, and load capacity. This paper introduces a novel robotic manipulator designed for NOTES, featuring a 5 mm diameter and 7 degrees of freedom (DoF). The manipulator incorporates an innovative 3-PRS flexible parallel mechanism combined with a continuum parallel structure, achieving enhanced dexterity and variable stiffness functionality within a miniaturized design. A kinematic and variable stiffness analysis is performed, and experimental validation demonstrates its bending performance and stiffness modulation. Additionally, the feasibility and practicality of the robotic system are confirmed through a peg-transfer experiment, proving its potential for real-world surgical applications. This research offers a viable solution for enhancing the performance of NOTES instruments. | |||||||||||||||||||||
52 | Medical Robots and Systems 1 | Design of a Soft Automatic Anchoring System for Enhanced Mobility and Stability in Colonoscopy Robots | Liang, Yiying;Wang, Xuchen;Shu, Jing;Zhang, Huayu;Zhu, Puchen;Xia, Xianfeng;Ma, Xin | Keywords: Medical Robots and Systems, Flexible Robotics | Abstract: Ensuring both mobility and stability during colonoscopy is crucial for enhancing procedural efficiency and reducing the risk of complications. Traditional colonoscopy robots face challenges due to the fixed diameter of the colonoscope and the variability in colon anatomy. To address this, we propose a soft automatic anchoring system (SAAS) to enhance mobility and stability in colonoscopy robots. The SAAS features proximal and distal soft balloon anchors, employing origami principles to achieve a 42.2% higher maximum expansion capability compared to conventional flat surface anchors. With real-time pressure feedback, the SAAS automatically anchors upon contact detection, ensuring precise anchoring without over-inflation while adapting to colon diameter variations and robot posture changes. Experimental results show a tenfold reduction in displacement during the stability test, significantly enhancing the robot's performance under external loads. Performance comparison tests in phantom further demonstrated notable improvements in the efficiency of colonoscopy procedures using the SAAS. This system has the potential to greatly enhance the safety, precision, and overall efficiency of colonoscopy, offering substantial benefits for both medical practitioners and patient outcomes. | |||||||||||||||||||||
53 | Medical Robots and Systems 1 | Head-Mounted Robotic Needle Positioning: Learning from Augmented Reality Demonstration of Neuronavigation and Planning | Fang, Zhiwei;Hung, Hok Man;Gao, Huxin;Ren, Hongliang | Keywords: Medical Robots and Systems, Virtual Reality and Interfaces | Abstract: Robotic needle positioning tasks in neurosurgery often face challenges due to insufficient perception of planar guidance images during surgery. In this work, we propose an Augmented Reality (AR) interface to help perform the robotic needle positioning tasks by learning from demonstration (LfD). Enhanced immersion in the workflow is achieved by displaying surgical scenes and calculated navigation information. The framework utilizes mixed interactive interfaces in virtual and real environments, enhancing demonstration efficiency and quality. A head-mounted display and an optical tracking system are utilized to perform the visualization and needle tracking. Gaussian Mixture Model (GMM) and Gaussian Mixture Re- gression (GMR) are employed to learn a robust and smooth trajectory policy from demonstrations. Experiments on robot reproduction of the needle positioning task achieved a final positioning error of 0.6 mm and an average trajectory error of 1.07 mm. Comparative user studies with haptic device-based teleoperation exhibit a low completion time of 62.76 s and reduced workload of the proposed system. | |||||||||||||||||||||
54 | Medical Robots and Systems 1 | An Inflatable Soft Robotic Manipulator with Decoupled Dual-Wrist Design for Advanced Endoscopy | Lou, Hanqi;Yang, Jianlin;Zhou, Zhangxi;Chen, Junhong;Runciman, Mark;Mylonas, George | Keywords: Medical Robots and Systems, Tendon/Wire Mechanism, Soft Robot Materials and Design | Abstract: Minimally Invasive Surgery has advanced surgical practice, yet early-stage gastrointestinal cancer treatment remains challenging. Endoscopic Submucosal Dissection offers a solution but faces maneuverability constraints in complex anatomical environments. This paper presents a novel inflatable soft robotic manipulator with a biocompatible thin-film shell and tendon-driven antagonistic actuation. The robot remains compact at 6.5 mm of diameter, expanding to 11.7 mm to enhance stiffness for force exertion and precise manipulation. Featuring two decoupled wrists with four degrees of freedom, it enables dexterous motion for advanced endoscopic procedures. The study details design, fabrication, actuation modeling, workspace evaluation, and simulated retraction experiments in a constrained environment. Results demonstrate high repeatability, high master-slave control accuracy, effective workspace utilization, and feasibility for endoluminal applications, enhancing robotic-assisted endoscopic procedures with improved dexterity and adaptability under soft actuation constraints. | |||||||||||||||||||||
55 | Medical Robots and Systems 1 | Augmented Bridge Spinal Fixation: A New Concept for Addressing Pedicle Screw Pullout Via a Steerable Drilling Robot and Flexible Pedicle Screws | Kulkarni, Yash;Sharma, Susheela;Rezayof, Omid;Kapuria, Siddhartha;Amadio, Jordan P.;Khadem, Mohsen;Tilton, Maryam;Alambeigi, Farshid | Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles | Abstract: To address the screw loosening and pullout limitations of rigid pedicle screws in spinal fixation procedures, and to leverage our recently developed Concentric Tube Steerable Drilling Robot (CT-SDR) and Flexible Pedicle Screw (FPS), in this paper, we introduce the concept of Augmented Bridge Spinal Fixation (AB-SF). In this concept, two connecting J- shape tunnels are first drilled through pedicles of vertebra using the CT-SDR. Next, two FPSs are passed through this tunnel and bone cement is then injected through the cannulated region of the FPS to form an augmented bridge between two pedicles and reinforce strength of the fixated spine. To experimentally analyze and study the feasibility of AB-SF technique, we first used our robotic system (i.e., a CT-SDR integrated with a robotic arm) to create two different fixation scenarios in which two J-shape tunnels, forming a bridge, were drilled at different depth of a vertebral phantom. Next, we implanted two FPSs within the drilled tunnels and then successfully simulated the bone cement augmentation process. | |||||||||||||||||||||
56 | Medical Robots and Systems 1 | Uncertainty-Aware Shared Control for Vision-Based Micromanipulation | Tian, Huanyu;Huber, Martin;Zeng, Lingyun;Han, Zhe;Bennett, Wayne;Silvestri, Giuseppe;Chavez-Badiola, Alejandro;Mendizabal-Ruiz, Gerardo;Bergeles, Christos | Keywords: Medical Robots and Systems, Physical Human-Robot Interaction, Computer Vision for Medical Robotics | Abstract: This paper presents an uncertainty-aware shared control and calibration method for micromanipulation using a digital microscope and a tool-mounted, multi-joint robotic arm, integrating real-time human intervention with a visual-motor policy. Our calibration algorithm leverages co-manipulation control to calibrate the hand-eye relationship without requiring knowledge of the kinematics of the microtool mounted on the robot while remaining robust to camera intrinsics errors.Experimental results show that the proposed calibration method achieves a 39.6% improvement in accuracy over established methods. Additionally, our control structure and calibration method reduces the time required to reach single-point targets from 5.74 s (best conventional method) to 1.91 s, and decreases trajectory tracking errors from 392 um to 40 um. These findings establish our method as a robust solution for improving reliability in high-precision biomedical micromanipulation. | |||||||||||||||||||||
57 | Medical Robots and Systems 1 | CapsDT: Diffusion-Transformer for Capsule Robot Manipulation | He, Xiting;Su, Mingwu;Jiang, Xinqi;Bai, Long;Ren, Hongliang | Keywords: Medical Robots and Systems, Machine Learning for Robot Control, Vision-Based Navigation | Abstract: Vision-Language-Action (VLA) models have emerged as a prominent research area, showcasing significant potential across a variety of applications. However, their performance in endoscopy robotics, particularly endoscopy capsule robots that perform actions within the digestive system, remains unexplored. The integration of VLA models into endoscopy robots allows more intuitive and efficient interactions between human operators and medical devices, improving both diagnostic accuracy and treatment outcomes. In this work, we design CapsDT, a Diffusion Transformer model for capsule robot manipulation in the stomach. By processing interleaved visual inputs, and textual instructions, CapsDT can infer corresponding robotic control signals to facilitate endoscopy tasks. In addition, we developed a capsule endoscopy robot system, a capsule robot controlled by a robotic arm-held magnet, addressing different levels of four endoscopy tasks and creating corresponding capsule robot datasets within the stomach simulator. Comprehensive evaluations on various robotic tasks indicate that CapsDT can serve as a robust vision-language generalist, achieving state-of-the-art performance in various levels of endoscopy tasks while achieving a 26.25% success rate in real-world simulation manipulation. | |||||||||||||||||||||
58 | Medical Robots and Systems 1 | Toward Safer GI Endoscopy with a Novel Robot-Assisted Endoscopic System | Feng, Guang;Wang, Sai;Li, Jinhua;Zuo, Siyang | Keywords: Medical Robots and Systems, Modeling, Control, and Learning for Soft Robots, Robot Safety | Abstract: Increased demand for minimally invasive surgery has accelerated the adoption of natural ori�ce transluminal endoscopic surgery. Meanwhile, the increasing complexity of endoscopic procedures has raised the demand for endoscopist competence. In this paper, we present a novel endoscopic system to perform robotic intervention with a flexible endoscope to enhance the safety and efficiency of endoscopy. The system is composed of an endoscopic manipulation module (EMM) for control of endoscopic interventions, a 6 DOFs robotic arm for adjusting the pose of EMM, and a haptic interface for tele-control. The compact system enables endoscopists to remotely and stably operate flexible endoscopes with one hand. Mas-ter-slave control strategy with active constraint is proposed to guide the motion of the flexible endoscope within the safe working space. The proposed system is validated by recruiting subjects to perform target area localization and endoscopic examinations through highly realistic upper digestive tract phantom. The system can reduce the time of target area localization by at least 10% compared to manual control. The mean angle errors can be reduced by 75% with the help of the constraint force. Results demonstrate the potential clinical value of the system in efficient operation and safety control. | |||||||||||||||||||||
59 | Computer Vision Applications | On the Benefits of Visual Stabilization for Frame and Event-Based Perception | Rodriguez-Gomez, Juan Pablo;Martinez-de Dios, J.R.;Ollero, Anibal;Gallego, Guillermo | Keywords: Computer Vision for Automation, Biologically-Inspired Robots | Abstract: Vision-based perception systems are typically exposed to large orientation changes in different robot applications. In such conditions, their performance might be compromised due to the inherit complexity of processing data captured under challenging motion. Integration of mechanical stabilizers to compensate for the camera rotation is not always possible due to the robot payload constraints. This paper presents a processing-based stabilization approach to compensate for the cameraÂ’s rotational motion both on events and on frames(i.e., images). Assuming that the cameraÂ’s attitude is available, we evaluate the benefits of stabilization in two perception applications: feature tracking and estimating the translation component of the cameraÂ’s ego-motion. The validation is performed using synthetic data and sequences from well-known event-based vision datasets. The experiments unveil that stabilization can improve feature tracking and camera ego-motion estimation accuracy in 27.37% and 34.82%, respectively. Concurrently, stabilization can reduce the processing time of computing the cameraÂ’s linear velocity by at least 25%. | |||||||||||||||||||||
60 | Computer Vision Applications | GNN Topology Representation Learning for Deformable Multi-Linear Objects Dual-Arm Robotic Manipulation (I) | Caporali, Alessio;Galassi, Kevin;Zanella, Riccardo;Palli, Gianluca | Keywords: Computer Vision for Automation, Perception for Grasping and Manipulation, Dual Arm Manipulation | Abstract: Deformable Multi-Linear Objects (DMLOs), or Branched Deformable Linear Objects (BDLOs), are flexible objects that possess a linear structure similar to DLOs but also feature branching or bifurcation points where the object's path diverges into multiple sections. The representation of complex DMLOs, such as wiring harnesses, poses significant challenges in various applications, including robotic systems' perception and manipulation planning. This paper proposes an approach to address the robust and efficient estimation of a topological representation for DMLOs leveraging a graph-based description of the scene obtained via graph neural networks. Starting from a binary mask of the scene, graph nodes are sampled along the objects' estimated centerlines. Then, a data-driven pipeline is employed to learn the assignment of graph edges between nodes and to characterize the node's type based on their local topology and orientation. Finally, by utilizing the learned information, a solver combines the predictions and generates a coherent representation of the objects in the scene. The approach is experimentally evaluated using a test set of complex real-world DMLOs. Within an offline evaluation, the proposed approach achieves a Dice score exceeding 90% in predicting graph edges. Similarly, the identification accuracy of branch and intersection points in the graph topology is above 90%. Additionally, the method demonstrates efficient performance, achieving a runtime of over 20 FPS. In an online assessment employing a dual-arm robotic setup, the approach is successfully applied to disentangle three automotive wiring harnesses, demonstrating the effectiveness of the proposed approach in a real-world scenario. | |||||||||||||||||||||
61 | Computer Vision Applications | MambaXCTrack: Mamba-Based Tracker with SSM Cross-Correlation and Motion Prompt for Ultrasound Needle Tracking | Zhang, Yuelin;Lei, Long;Yan, Wanquan;Zhang, Tianyi;Tang, Raymond Shing-Yan;Cheng, Shing Shin | Keywords: Computer Vision for Medical Robotics, Deep Learning Methods, Medical Robots and Systems | Abstract: Ultrasound (US)-guided needle insertion is widely employed in percutaneous interventions. However, providing feedback on the needle tip position via US image presents challenges due to noise, artifacts, and the thin imaging plane of US, which degrades needle features and leads to intermittent tip visibility. In this paper, a Mamba-based US needle tracker MambaXCTrack utilizing structured state space models cross-correlation (SSMX-Corr) and implicit motion prompt is proposed, which is the first application of Mamba in US needle tracking. The SSMX-Corr enhances cross-correlation by long-range modeling and global searching of distant semantic features between template and search maps, benefiting the tracking under noise and artifacts by implicitly learning potential distant semantic cues. By combining with cross-map interleaved scan (CIS), local pixel-wise interaction with positional inductive bias can also be introduced to SSMX-Corr. The implicit low-level motion descriptor is proposed as a non-visual prompt to enhance tracking robustness, addressing the intermittent tip visibility problem. Extensive experiments on a dataset with motorized needle insertion in both phantom and tissue samples demonstrate that the proposed tracker outperforms other state-of-the-art trackers while ablation studies further highlight the effectiveness of each proposed tracking module. | |||||||||||||||||||||
62 | Computer Vision Applications | LiVeDet: Lightweight Density-Guided Adaptive Transformer for Online On-Device Vessel Detection | Zhang, Zijie;Fu, Changhong;Cao, Yongkang;Li, Mengyuan;Zuo, Haobo | Keywords: Computer Vision for Transportation, Deep Learning for Visual Perception, Intelligent Transportation Systems | Abstract: Vision-based online vessel detection boosts the automation of waterways monitoring, transportation management and navigation safety. However, a significant gap exists in on-device deployment between general high-performance PCs/servers and embedded AI processors. Existing state-of-the-art (SOTA) online vessel detectors lack sufficient accuracy and are prone to high latency on the edge AI camera, especially in scenarios with dense vessels and diverse distributions. To solve the above issues, a novel lightweight framework with density-guided adaptive Transformer (LiVeDet) is proposed for the edge AI camera to achieve online on-device vessel detection. Specifically, a new instance-aware representation extractor is designed to suppress cluttered background noise and capture instance-aware content information. Additionally, an innovative vessel distribution estimator is developed to direct superior feature representation learning by focusing on local regions with varying vessel density. Besides, a novel dynamic region embedding is presented to integrate hierarchical features represented by multi-scale vessels. A new benchmark comprising 100 high-definition, high-frame rate video sequences from vessel-intensive scenarios is established to evaluate the efficacy of vessel detectors under challenging conditions prevalent in dynamic waterways. Extensive evaluations on this challenging benchmark demonstrate the robustness and efficiency of LiVeDet, achieving 32.9 FPS on the edge AI camera. Furthermore, real-world applications confirm the practicality of the proposed method. | |||||||||||||||||||||
63 | Computer Vision Applications | EvTTC: An Event Camera Dataset for Time-To-Collision Estimation | Sun, Kaizhen;Li, Jinghang;Dai, Kuan;Liao, Bangyan;Xiong, Wei;Zhou, Yi | Keywords: Computer Vision for Transportation, Collision Avoidance, Data Sets for Robotic Vision | Abstract: Time-to-Collision (TTC) estimation lies in the core of the forward collision warning (FCW) functionality, which is key to all Automatic Emergency Braking (AEB) systems. Although the success of solutions using frame-based cameras (e.g., Mobileye's solutions) has been witnessed in normal situations, some extreme cases, such as the sudden variation in the relative speed of leading vehicles and the sudden appearance of pedestrians, still pose significant risks that cannot be handled. This is due to the inherent imaging principles of frame-based cameras, where the time interval between adjacent exposures introduces considerable system latency to AEB. Event cameras, as a novel bio-inspired sensor, offer ultra-high temporal resolution and can asynchronously report brightness changes at the microsecond level. To explore the potential of event cameras in the above-mentioned challenging cases, we propose EvTTC, which is, to the best of our knowledge, the first multi-sensor dataset focusing on TTC tasks under high-relative-speed scenarios. EvTTC consists of data collected using standard cameras and event cameras, covering various potential collision scenarios in daily driving and involving multiple collision objects. Additionally, LiDAR and GNSS/INS measurements are provided for the calculation of ground-truth TTC. Considering the high cost of testing TTC algorithms on full-scale mobile platforms, we also provide a small-scale TTC testbed for experimental validation and data augmentation. All the data and the design of the testbed are open sourced, and they can serve as a benchmark that will facilitate the development of vision-based TTC techniques. | |||||||||||||||||||||
64 | Computer Vision Applications | WeatherDG: LLM-Assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation | Qian, Chenghao;Guo, Yuhu;Mo, Yuhong;Li, Wenjing | Keywords: Computer Vision for Transportation, Intelligent Transportation Systems, Deep Learning for Visual Perception | Abstract: In this work, we propose a novel approach, namely WeatherDG, that can generate realistic, weather-diverse, and driving-screen images based on the cooperation of two foundation models, i.e., Stable Diffusion (SD) and Large Language Model (LLM). Specifically, we first fine-tune the SD with source data, aligning the content and layout of generated samples with real-world driving scenarios. Then, we propose a procedural prompt generation method based on LLM, which can enrich scenario descriptions and help SD automatically generate more diverse, detailed images. In addition, we introduce a balanced generation strategy, which encourages the SD to generate high-quality objects of tailed classes under various weather conditions, such as riders and motorcycles. This segmentation-model-agnostic method can improve the generalization ability of existing models by additionally adapting them with the generated synthetic data. Experiments on three challenging datasets show that our method can significantly improve the segmentation performance of different state-of-the-art models on target domains. Notably, in the setting of ''Cityscapes to ACDC'', our method improves the baseline HRDA by 13.9% in mIoU. | |||||||||||||||||||||
65 | Computer Vision Applications | Autonomous Hyperspectral Characterisation Station: Robot Aided Measuring of Polymer Degradation (I) | Azizi, Shayan;Asadi, Ehsan;Howard, Shaun;Muir, Benjamin Ward;O'shea, Riley;Bab-Hadiashar, Alireza | Keywords: Computer Vision for Automation, Reactive and Sensor-Based Planning, Task Planning | Abstract: This paper addresses a gap between the capabilities and utilisation of robotics and automation in laboratory settings and builds upon the concept of Self-Driving Labs (SDL). We introduce an innovative approach to the temporal characterisation of materials. The article discusses the challenges posed by manual methods involving established laboratory equipment and presents an automated hyperspectral characterisation station. This station integrates robot-aided hyperspectral imaging (HSI), complex material characterisation modelling, and automated data analysis, offering a non-destructive and comprehensive approach. This work explains how the proposed assembly can automatically measure the half-life of biodegradable polymers with higher throughput and accuracy than manual methods. The investigation explores the effect of pH, number of average molecular weight (Mn), end groups, and blends on the degradation rate of polylactic acid (PLA). The novel contributions of the paper lie in introducing an adaptable classification station for characterisation and presenting an innovative methodology for polymer degradation rate measurements. The proposed system holds promise for expediting the development of high-throughput screening and characterisation methods within advanced material and chemistry laboratories. Note to Practitioners—The characterisation and classification of materials hold significant importance within the realms of materials science, chemistry, manufacturing, and the circular economy. We introduce an innovative approach by employing robotics to automate hyperspectral imaging and processing to accelerate and improve material characterisation within a laboratory environment. This methodology facilitates the time-dependent characterisation of materials. This automated system encompasses sample manipulation and handling for hyperspectral scanning, accompanied by automated image and data processing procedures to minimise manual interventions effectively. The developed system can be seamlessly integrated into lab settings, offering objective, accurate, and automated measurement of biodegradable polymer degradation rates. | |||||||||||||||||||||
66 | Computer Vision Applications | Decompose-Compose Feature Augmentation for Imbalanced Crack Recognition in Industrial Scenarios (I) | Chen, ZhuangZhuang;Xu, Chengqi;Hu, Tao;Wang, Li;Chen, Jie;Li, Jianqiang | Keywords: Computer Vision for Manufacturing, Computer Vision for Transportation | Abstract: Automated crack recognition has achieved remarkable progress in the past decades as a critical task in structure health monitoring, to ensure safety and durability in many industrial scenarios. However, imbalanced crack recognition remains challenging due to the scarcity of crack samples and the consequential limited diversity. To resolve this, Artificial Intelligence Generated Content (AIGC) has been gradually adopted to generate synthetic data and reduce reliance on large amounts of labeled crack samples. This paper assumes that a crack sample in the feature space can be regarded as a combination of crack and background semantics. Then, the decompose-compose feature augmentation framework (DeCo) is proposed to perform crack data synthesis in the feature space by randomly composing crack and background semantic-relevant features. Specifically, the contrastive learning-based decomposing loss is proposed to enforce two encoders to separately learn crack and background semantics from crack samples with a theoretical guarantee. After that, an effective cross-instance feature union strategy is proposed to synthesize diverse crack samples by composing the crack-relevant features from a crack sample and background-relevant features across other training samples. Experimental results show that DeCo performs favorably against state-of-the-art competitors in imbalanced crack recognition tasks. | |||||||||||||||||||||
67 | Computer Vision for Medical Robotics | Unsupervised Liver Deformation Correction Network Using Optimal Transport for Image-Guided Liver Surgery | Liu, Mingyang;Li, Geng;Yu, Hao;Du, Xinzhe;Song, Rui;Li, Yibin;Meng, Max Q.-H.;Min, Zhe | Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems | Abstract: In this paper, we propose a novel unsupervised liver deformation correction method, Learning Coherent Point Drift Network (LCNet), for image-guided liver surgery (IGLS). In order to address the complete to partial registration problem, we first crop the preoperative liver point cloud to obtain the overlapping regions with the intraoperative point cloud. The cropped preoperative point set is then fed into the subsequent registration network for alignment. Secondly, we establish reliable correspondences between two point sets using the optimal transport (OT) module by leveraging both original points and learnt features, which are robust to the rigid transformation. Finally, we compute the displacements by solving the involved matrix equation in the Transformation module, where the point localisation noise is explicitly considered. In addition, we present three variants of the proposed approach, i.e., LCNet, LCNet-ED and LCNet-WD, where LCNet outperforms the other two, which demonstrates the superiority of the Chamfer loss. We have extensively evaluated LCNet on the MedShapeNet dataset consisting of 615 different liver shapes of real patients, and 3Dircadb consisting 20 another liver models of real patients. For example, when the overlap ratio is 25%, the deformation magnitude is 8 mm, the maximum noise magnitude is 2 mm and the rotation angle lies in the range of [-45^{circ},45^{circ}], LCNet achieves a root-mean-square error (RMSE) value being 3.21 mm on MedShapeNet, outperforming Lepard and RoITr, which are 5.41 mm (p<0.001) and 4.90 mm (p<0.001) respectively. Extensive experimental results, under different deformation and noise levels, demonstrate that LCNet exhibits significant improvements over existing state-of-the-art registration methods and holds significant application potential in IGLS. | |||||||||||||||||||||
68 | Computer Vision for Medical Robotics | A Deep Learning-Driven Autonomous System for Retinal Vein Cannulation: Validation Using a Chicken Embryo Model | Wang, Yi;Zhang, Peiyao;Esfandiari, Mojtaba;Gehlbach, Peter;Iordachita, Ioan Iulian | Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems, Vision-Based Navigation | Abstract: Retinal vein cannulation (RVC) is a minimally invasive microsurgical procedure for treating retinal vein occlusion (RVO), a leading cause of vision impairment. However, the small size and fragility of retinal veins, coupled with the need for high-precision, tremor-free needle manipulation, pose significant challenges for surgeons. These limitations underscore the necessity for robotic assistance to enhance surgical accuracy, stability, and reproducibility. This study presents an automated robotic system with a top-down microscope and cross-sectional B-scan optical coherence tomography (OCT) imaging for precise depth sensing. Deep learning-based models enable real-time needle navigation, contact detection, and vein puncture recognition, using a chicken embryo model, a well-established biological surrogate for human retinal veins. The system autonomously detects needle position and puncture events with 85% accuracy. The experiments demonstrate notable reductions in navigation and puncture times compared to manual methods. Our results demonstrate the potential of integrating advanced imaging and deep learning to automate microsurgical tasks, providing a pathway for safer and more reliable RVC procedures with enhanced precision and reproducibility. | |||||||||||||||||||||
69 | Computer Vision for Medical Robotics | Deep Coarse-To-Fine Networks for Robust Segmentation and Pose Estimation of Surgical Suturing Threads | Zhou, Xinyao;Liu, Yuxuan;Zhang, Musen;Li, Jinkai;Guo, Yao;Yang, Guang-Zhong | Keywords: Computer Vision for Medical Robotics, Surgical Robotics: Laparoscopy | Abstract: Autonomous suturing is a critical challenge in robot-assisted surgery, where accurate segmentation and pose estimation of suturing threads are essential prerequisites. However, suturing threads are easily occluded by moving instruments and embedded in deformable tissues which make the task much more challenging. To address this, we propose a coarse-to-fine network for detailed segmentation and pose estimation of suturing threads. The coarse stage aims to capture global thread structure, while the fine stage refines the detailed structure through error residual correction. A spatial context fusion module is incorporated to improve the perception of occluded regions, and weighted balanced cross entropy loss as well as hard sample mining strategy is implemented to enhance small target segmentation performance. To deal with severe occlusions, topological constraints are utilized to effectively identify and reconstruct invisible thread segments. Experiments have been conducted on three datasets collected from different surgical scenes including phantom, endoscopy, and microsurgery. Both quantitative and qualitative results have demonstrated that our proposed framework outperforms baseline methods on segmentation and pose estimation of suturing threads, particularly in detecting occluded threads. Our proposed framework generalizes well across different surgical scenarios, showing its potential for automatic suturing. | |||||||||||||||||||||
70 | Computer Vision for Medical Robotics | Directed Spatial Consistency-Based Partial-To-Partial Point Cloud Registration with Deep Graph Matching | Zhou, Jingwen;Fu, Kexue;Du, Xinzhe;Song, Rui;Li, Yibin;Meng, Max Q.-H.;Min, Zhe | Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems | Abstract: 3D point cloud registration is an essential problem in computer vision, robotics, surgical navigation and augmented reality. Accurate registration of partially overlapped intraoperative point clouds (e.g., femoral reconstruction) remains critical yet challenging in orthopedic navigation due to incomplete overlap and dynamic noise. In this study, we propose a partial-to-partial point cloud registration framework based on directional spatial consistency. First, we extract overlapped areas from partially overlapping point clouds and leverage the point registration graph matching module to calculate the hard point matching matrix. Second, we sample nodes from the source point cloud and generate translation-invariant edge vectors (direction/scale-preserving) via their k-nearest neighbors, guided by predicted point correspondences. This bypasses translation ambiguities by encoding spatial consistency through edges, reducing pose estimation to 3DoF alignment (rotation). The loss explicitly couples point-level matches with edge-level geometric constraints for dual optimization. Building upon this framework, we extract reliable overlapping edge representations and prune their similarity matrix by thresholding low-confidence scores, effectively suppressing spurious matches. The proposed edge-aware matching mechanism further exploits the translation invariance of local structures to refine point correspondences with enhanced accuracy. Finally, we introduce a bidirectional registration mechanism to reinforce optimization stability, achieving state-of-the-art performance across benchmarks. Extensive experiments on ModelNet40, ShapeNet, and MedShapeNet validate our method under diverse scenarios: partial-to-partial, unseen categories, partial-to-full, and cross-dataset generalization, surpassing existing methods in registration accuracy. This code will be publicly released https://github.com/pidan0824/DSCGM. | |||||||||||||||||||||
71 | Computer Vision for Medical Robotics | Revisiting 3D Curve to Surface Registration Using Tangent and Normal Vectors for Computer-Assisted Orthopedic Surgery | Zhang, Zhengyan;Du, Xinzhe;Min, Zhe;Song, Rui;Li, Yibin;Meng, Max Q.-H. | Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems | Abstract: In this paper, we present a novel curve-to-surface registration method, termed Bi-directional Hybrid Mixture Model Registration based on Dual-constrained Tangent and Normal Vectors (BiHMM-DTN). While hybrid registration models incorporating tangent and normal vectors (HMM-TN) demonstrate success, their geometric constraints prove inadequate or inappropriate for sparse intraoperative point sets, frequently yielding suboptimal optimization outcomes. By critically revisiting the geometric constraints of HMM-TN, we propose a dual-constraints-based hybrid mixture model registration framework with enhanced intraoperative point cloud acquisition protocols. To deal with noise and outliers in preoperative and intraoperative point sets—caused by reconstruction inaccuracies and tracking errors respectively—our approach employs a bi-directional registration mechanism for curve-to-surface registration. We provide rigorous proofs validating the geometric completeness of the dual constraints within this mechanism. The BiHMM-DTN framework is formulated as a maximum likelihood estimation (MLE) problem and optimized using an expectation-maximization (EM) algorithm. Futhermore, to enhance convergence stability and accelerate optimization, the rotation matrix is updated iteratively through successive incremental steps. Extensive experiments on human femur and hip models demonstrate that our method outperforms state-of-the-art approaches, including both traditional optimization and deep learning methods, under various noise and outlier conditions. Furthermore, real-world phantom experiments highlight the potential clinical value of our method for surgical navigation applications. The codes and data are available at url{https://github.com/sam-zyzhang/BiHMM-DTN.git}. | |||||||||||||||||||||
72 | Computer Vision for Medical Robotics | Gaussian Splatting with Reflectance Regularization for Endoscopic Scene Reconstruction | Li, Chengkun;Chen, Kai;Qiu, Shi;Chan, Jason Ying-Kuen;Dou, Qi | Keywords: Computer Vision for Medical Robotics | Abstract: Endoscopic reconstruction plays a crucial role in surgical robotics. The dynamic lighting conditions and integrated camera-light source in endoscopic scenes create a distinct reconstruction challenge: shape ambiguity. To mitigate this, we propose a Gaussian Splatting (GS) based framework for endoscopic scene reconstruction, enhanced with reflectance regularization. We embed every 3D Gaussian point with physical reflective attributes and combine this representation with a physically based inverse rendering framework. By jointly training 3DGS for view synthesis with this reflectance regularization, we are able to attain high-quality geometry without changing the volume rendering pipeline. Our experiments demonstrate the superiority in both geometry representation and rendering performance compared to existing GS approaches, making it a practical solution for endoscopic applications. Project is available at: https://med-air.github.io/GSR2. | |||||||||||||||||||||
73 | Reinforcement Learning 1 | Deep Reinforcement Learning-Based Mapless Navigation for Mobile Robot in Unknown Environment with Local Optima | Hu, Yiming;Wang, Shuting;Xie, Yuanlong;Zheng, Shiqi;Shi, Peng;Rudas, Imre J.;Cheng, Xiang | Keywords: Reinforcement Learning, Collision Avoidance | Abstract: Local optima issues challenge mobile robot mapless navigation with the dilemma of avoiding collisions and approaching the target. Planning-based methods rely on environmental models and manual strategies for updating local paths to guide the robot. In contrast, learning-based methods are capable of processing original sensor data to navigate the robot in real time but struggle with local optima problems. To address this, we designed reward rules that punishes the robot for revisiting passed areas that may traps the robot, and reward it for exploring local areas in diverse ways and escaping from local optima areas. Then, we improved the Soft Actor-Critic (SAC) algorithm by making its temperature parameter adaptive to the current training status, and memorize them in experiences for strategy updating, bringing additional exploratory behaviors and necessary stability into the training. Finally, with the assistance of auxiliary networks, the robot learns to handle various navigation tasks with local optima risks. Simulations demonstrate the advantages of our method in terms of both success rate and path efficiency compared to several existing methods. Experiments verified the proposed method in real-world scenarios. | |||||||||||||||||||||
74 | Reinforcement Learning 1 | Point Cloud-Based End-To-End Formation Control Using a Two Stage SAC Algorithm | Li, Mingfei;Liu, Haibin;Xie, Feng;Huang, He | Keywords: Reinforcement Learning, Sensor-based Control, AI-Based Methods | Abstract: This study develops a novel end-to-end formation strategy for leader-follower formation control of mobile robots that uses onboard LiDAR sensors in non-communication environments. The main contributions of this paper are twofold: Firstly, we propose a point cloud-based LiDAR servoing control method (PCLS) aimed at ensuring mobile robots achieve the predefined formation performance without direct communication. Secondly, an innovative two-stage Soft Actor-Critic (TSSAC) algorithm is presented, specifically designed for end-to-end training of PCLS. This algorithm skillfully combines the strengths of a distance-based agent (serving as a ``teacher") and a point cloud-based agent (serving as a ``student"), effectively addressing the issues of slow convergence and insufficient generalization in deep reinforcement learning methods that use high-dimensional features (such as point clouds, images) as inputs. Furthermore, as part of our method, we designed a novel reward function and normalized the point cloud inputs to provide consistent incentives for the agent across diverse formation tasks, thereby facilitating better learning and adaptation to formation tasks in different environments. Finally, through extensive experiments conducted in the Gazebo simulator and real-world environments, we confirmed the effectiveness of the proposed method. Compared to other formation control strategies, our approach relies solely on onboard LiDAR sensors, without the need for additional communication devices, while ensuring excellent transient and steady-state performance. The video of the test in the real-world environment can be found at https://youtu.be/jhggsLUczPA | |||||||||||||||||||||
75 | Reinforcement Learning 1 | Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning | Chen, Xiaocong;Wang, Siyu;Yu, Tong;Yao, Lina | Keywords: Reinforcement Learning | Abstract: Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties associated with various actions and environmental stochasticity. Traditional approaches primarily emphasize mitigating epistemic uncertainty by learning risk-averse policies, often overlooking environmental stochasticity. In this study, we propose an uncertainty-aware distributional offline RL method to simultaneously address both epistemic uncertainty and environmental stochasticity. We propose a model-free offline RL algorithm capable of learning risk-averse policies and characterizing the entire distribution of discounted cumulative rewards, as opposed to merely maximizing the expected value of accumulated discounted returns. Our method is rigorously evaluated through comprehensive experiments in both risk-sensitive and risk-neutral benchmarks, demonstrating its superior performance. | |||||||||||||||||||||
76 | Reinforcement Learning 1 | IHGSL: Interpretable Heuristic Graph Structure Learning for Multi-Robot Autonomous Collaborative Systems | Han, Yue;Li, Hanqi;Liu, Cuiwei;Liang, Chen;Sun, Zhixiao | Keywords: Reinforcement Learning, Cooperating Robots, Representation Learning | Abstract: In multi-robot systems, capturing the complex and dynamic interaction relationships is essential for enhancing autonomous collaboration. However, existing learning-based approaches usually overlook the understanding of these relationships, leading to reliability issues and hindering their application to real-world scenarios. This paper proposes a novel approach called Interpretable Heuristic Graph Structure Learning (IHGSL) to better comprehend the complex collaborative relationships in multi-robot systems. We first construct a predicate space to define diverse predicates that express fundamental relationships. Then we employ the variational information bottleneck technique to acquire a latent representation of the current observation by aligning it with the historical trajectory. On this basis, the predicates that the robot should currently focus on the most are learned, and some interaction relationships are established accordingly. Thereby an interpretable relationship graph is generated heuristically to guide the achievement of multi-robot autonomous collaborative decision-making. Through experimental evaluation, we demonstrate the process of relationship inference, thus validating the interpretability of IHGSL. Compared with existing methods, IHGSL also achieves superior collaboration performance, which highlights the effectiveness of the learned heuristic graph structure. | |||||||||||||||||||||
77 | Reinforcement Learning 1 | Visual Multitask Policy Learning with Asymmetric Critic Guided Distillation | Srinivasan, Krishnan;Xu, Jie;Ang, Henry;Heiden, Eric;Fox, Dieter;Bohg, Jeannette;Garg, Animesh | Keywords: Reinforcement Learning, Imitation Learning, Dexterous Manipulation | Abstract: We present Asymmetric Critic Guided Distillation (ACGD) a framework for learning multi-task dexterous manipulation policies that can manipulate articulated objects using images as input. ACGD is a scalable student-teacher distillation approach that utilizes behavior cloning to distill multiple expert policies into a single vision-based, multi-task student policy for dexterous manipulation. The expert policies are trained with traditional RL techniques with access to privileged state information of both the robot and the manipulated object, while the distilled student policy operates under realistic sensory constraints, specifically using only camera images and robot proprioception. During distillation, we use an expert-critic that provides action labels and value estimates to refine the student's action sampling through a dual IL/RL objective. In the multi-task setting, we achieve this through an aggregate critic for different single-task experts. Our approach exhibits strong performance compared to a number of state-of-the-art imitation learning (IL) and reinforcement learning (RL) baselines. We evaluate across a variety of multi-task dexterous manipulation benchmarks including bimanual manipulation, single-hand object articulation tasks, and a tendon-actuated hand and achieves state-of-the-art performance with 10-15% improvement over baseline algorithms. Visit our website (https://critic-guided-distillation.github.io) for more details. | |||||||||||||||||||||
78 | Reinforcement Learning 1 | GACL: Grounded Adaptive Curriculum Learning with Active Task and Performance Monitoring | Wang, Linji;Xu, Zifan;Stone, Peter;Xiao, Xuesu | Keywords: Reinforcement Learning, Machine Learning for Robot Control, Continual Learning | Abstract: Curriculum learning has emerged as a promising approach for training complex robotics tasks, yet current applications predominantly rely on manually designed curricula, which demand significant engineering effort and can suffer from subjective and suboptimal human design choices. While automated curriculum learning has shown success in simple domains like grid worlds and games where task distributions can be easily specified, robotics tasks present unique challenges: they require handling complex task spaces while maintaining relevance to target domain distributions that are only partially known through limited samples. To this end, we propose Grounded Adaptive Curriculum Learning, a framework specifically designed for robotics curriculum learning with three key innovations: (1) a task representation that consistently handles complex robot task design, (2) an active performance tracking mechanism that allows adaptive curriculum generation appropriate for the robot's current capabilities, and (3) a grounding approach that maintains target domain relevance through alternating sampling between reference and synthetic tasks. We validate GACL on wheeled navigation in constrained environments and quadruped locomotion in challenging 3D confined spaces, achieving 6.8% and 6.1% higher success rates, respectively, than state-of-the-art methods in each domain. | |||||||||||||||||||||
79 | Reinforcement Learning 1 | RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms | El Asri, Zakariae;Laiche, Ibrahim;Rambour, Clément;Sigaud, Olivier;Thome, Nicolas | Keywords: Reinforcement Learning, Machine Learning for Robot Control, Model Learning for Control | Abstract: Learning a controller directly on the robot requires extreme sample efficiency. Model-based reinforcement learning (RL) methods are the most sample efficient, but they often suffer from a too long inference time to meet the robot control frequency requirements. In this paper, we address the sample efficiency and inference time challenges with two contributions. First, we define a general framework to deal with inference delays where the slow inference robot controller provides a sequence of actions to feed the control-hungry robotic platform without execution gaps. Then, we compare several RL algorithms in the light of this framework and propose RT-HCP, an algorithm that offers an excellent trade-off between performance, sample efficiency and inference time. We validate the superiority of RT-HCP with experiments where we learn a controller directly on a simple but high frequency FURUTA pendulum platform. | |||||||||||||||||||||
80 | Reinforcement Learning 1 | DRARL: Disengagement-Reason-Augmented Reinforcement Learning for Efficient Improvement of Autonomous Driving Policy | Zhou, Weitao;Zhang, Bo;Cao, Zhong;Li, Xiang;Cheng, Qian;Liu, Chunyang;Zhang, Ya-Qin;Yang, Diange | Keywords: Robot Safety, Reinforcement Learning, Continual Learning | Abstract: With the increasing presence of automated vehicles on open roads under driver supervision, disengagement cases are becoming more prevalent. While some data-driven planning systems attempt to directly utilize these disengagement cases for policy improvement, the inherent scarcity of disengagement data (often occurring as a single instance) restricts training effectiveness. Furthermore, some disengagement data should be excluded since the disengagement may not always come from the failure of driving policies, e.g. the driver may casually intervene for a while. To this end, this work proposes disengagement-reason-augmented reinforcement learning (DRARL), which enhances driving policy improvement process according to the reason of disengagement cases. Specifically, the reason of disengagement is identified by an out-of-distribution (OOD) state estimation model. When the reason doesnÂ’t exist, the case will be identified as a casual disengagement case, which doesnÂ’t require additional policy adjustment. Otherwise, the policy can be updated under a reason-augmented imagination environment, improving the policy performance of disengagement cases with similar reasons. The method is evaluated using real-world disengagement cases collected by autonomous driving robotaxi. Experimental results demonstrate that the method accurately identifies policy-related disengagement reasons, allowing the agent to handle both original and semantically similar cases through reason-augmented training. Furthermore, the approach prevents the agent from becoming overly conservative after policy adjustments. Overall, this work provides an efficient way to improve driving policy performance with disengagement cases. | |||||||||||||||||||||
81 | RGB-D Perception 1 | Towards Label-Free 3D Visual Grounding with Vision Foundation Models | Wu, Xiaopei;Hou, Yuenan;Lin, Binbin;Zhu, Xinge;Ma, Yuexin;Liu, Haifeng;Cai, Deng;Sun, Xiao | Keywords: RGB-D Perception, Recognition | Abstract: 3D visual grounding is pivotal for enabling intelligent agents to find the target object in the 3D scene given the linguistic descriptions. However, contemporary methods are typically hindered by the scarcity of large-scale 3D datasets with fine-grained annotations and the complexity of modeling spatial relationships in the 3D space. Inspired by the exceptional performance of Vision Foundation Models (VFMs) and Vision Language Models (VLMs), we propose a novel label-free 3D visual grounding method, termed LF-3DVG , that minimizes the heavy reliance on fine-grained annotations and leverages the off-the-shelf vision foundation models for zero-shot 3D visual grounding. Our LF-3DVG is comprised of two main components, i.e., VFM-guided 3D Object Detection and VLM- based 3D Visual Grounding. Specifically, we first utilize SAM3D to generate high-quality instance masks for the objects in the 3D scene. Since SAM3D cannot provide categorical information, we further employ Semantic-SAM to assign class labels for the detected masks. As to the VLM-based 3D Visual Grounding, we first feed multi-view images and textual descriptions to the VLM for 2D visual grounding. To lift the 2D predictions to the 3D space, we design the 2D-3D object association module to effectively match the 2D detection results with the 3D boxes produced by the 3D detector, yielding the ultimate 3D visual grounding results. Through extensive experiments on the ScanRefer and Sr3D/Nr3D benchmarks, we demonstrate that our method consistently outperforms previous approaches. Our algorithm can also boost the performance of 3D visual grounding when given labeled training samples and can be seamlessly integrated into contemporary 3D visual grounding models. | |||||||||||||||||||||
82 | RGB-D Perception 1 | LR^2Depth: Large-Region Aggregation at Low Resolution for Efficient Monocular Depth Estimation | Ning, Chao;Xuan, Weihao;Gan, Wanshui;Yokoya, Naoto | Keywords: RGB-D Perception, Computer Vision for Automation, Deep Learning Methods | Abstract: Monocular depth estimation (MDE) is crucial for various computer vision applications, but existing methods often struggle to balance inference speed and accuracy when processing large-region visual information. This paper introduces LR^2Depth, a novel MDE method that addresses this challenge by utilizing large-kernel convolution on low-resolution feature maps for efficient large-region feature aggregation. Our approach leverages the fact that each pixel on low-resolution feature maps corresponds to a larger region of the original image, allowing for fast and accurate depth predictions at a lower inference cost. Extensive experiments on NYU-Depth-V2, KITTI, and SUN RGB-D datasets demonstrate that LR^2Depth not only achieves state-of-the-art performance but also operates approximately twice as fast as previous MDE methods. Notably, at the time of submission, LR^2Depth secured the top-1 position on the KITTI depth prediction online benchmark. | |||||||||||||||||||||
83 | RGB-D Perception 1 | Adjacent-View Transformers for Supervised Surround-View Depth Estimation | Guo, Xianda;Yuan, Wenjie;Zhang, Yunpeng;Yang, Tian;Zhang, Chenming;Zhu, Zheng;Zou, Qin;Chen, Long | Keywords: RGB-D Perception, Computer Vision for Transportation | Abstract: Depth estimation has been widely studied and serves as the fundamental step of 3D perception for robotics and autonomous driving. Though significant progress has been made in monocular depth estimation in the past decades, these attempts are mainly conducted on the KITTI benchmark with only front-view cameras, which ignores the correlations across surround-view cameras. In this paper, we propose an Adjacent-View Transformer for Supervised Surround-view Depth estimation to jointly predict the depth maps across multiple surrounding cameras. Specifically, we employ a global-to-local feature extraction module that combines CNN with transformer layers for enriched representations. Further, the adjacent-view attention mechanism is proposed to enable the intra-view and inter-view feature propagation. The former is achieved by the self-attention module within each view, while the latter is realized by the adjacent attention module, which computes the attention across multi-cameras to exchange the multi-scale representations across surround-view feature maps. In addition, AVT-SSDepth has strong cross-dataset generaliza- tion. Extensive experiments show that our method achieves superior performance on both DDAD and nuScenes datasets. | |||||||||||||||||||||
84 | RGB-D Perception 1 | OpenFusion++: An Open-Vocabulary Real-Time Scene Understanding System | Xiaofeng, Jin;Frosi, Matteo;Matteucci, Matteo | Keywords: RGB-D Perception, Semantic Scene Understanding, Range Sensing | Abstract: 实时开放è¯�汇场景ç�†è§£å¯¹äºŽè§†è§‰è¯è¨€å¯¼èˆªã€�具体智能和增强现实ç‰åº”用ä¸çš„高æ•?3D 感知至关é‡�è¦�。但是，现有方法å˜åœ¨å®žä¾‹åˆ†å‰²ä¸�精确ã€�é�™æ€�è¯ä¹‰æ›´æ–°å’Œå¤�æ�‚查询处ç�†å�—é™�的问题。为了解决这些问题，我们æ��出äº?OpenFusion++，这是一ç§�基äº?TSDF 的实æ—?3D è¯ä¹‰å‡ 何é‡�建系统。我们的方法通过èž�å�ˆæ�¥è‡ªåŸºç¡€æ¨¡åž‹çš„置信度图æ�¥ä¼˜åŒ– 3D 点云，通过基于实例区域的自适应缓å˜åŠ¨æ€�更新全局è¯ä¹‰æ ‡ç¾ï¼Œå¹¶é‡‡ç”¨å�Œè·¯å¾„ç¼–ç �框架将对象属性与环境上下文集æˆ�以实现精确的查询å“�应。在 ICL ã€?Replica ã€?ScanNet å’?ScanNet++ æ•°æ�®é›†ä¸Šçš„实验表明，OpenFusion++ 在è¯ä¹‰å‡†ç¡®æ€§å’ŒæŸ¥è¯¢å“�应性方é�¢éƒ½æ˜Žæ˜¾ä¼˜äºŽåŸºçº¿ã€?RGB-D Perception 1 | Monocular One-Shot Metric-Depth Alignment for RGB-Based Robot Grasping | Guo, Teng;Huang, Baichuan;Yu, Jingjin | Keywords: RGB-D Perception, Grasping, Computer Vision for Automation | Abstract: Accurate 6D object pose estimation is a prerequisite for successfully completing robotic prehensile and nonprehensile manipulation tasks. At present, 6D pose estimation for robotic manipulation generally relies on depth sensors based on, e.g., structured light, time-of-flight, and stereo-vision, which can be expensive, produce noisy output (as compared with RGB cameras), and fail to handle transparent objects. On the other hand, state-of-the-art monocular depth estimation models (MDEMs) provide only affine-invariant depths up to an unknown scale and shift. Metric MDEMs achieve some successful zero-shot results on public datasets, but fail to generalize. We propose a novel framework, monocular one-shot metric-depth alignment, MOMA, to recover metric depth from a single RGB image, through a one-shot adaptation building on MDEM techniques. MOMA performs scale-rotation-shift alignments during camera calibration, guided by sparse ground-truth depth points, enabling accurate depth estimation without additional data collection or model retraining on the testing setup. MOMA supports fine-tuning the MDEM on transparent objects, demonstrating strong generalization capabilities. Real- world experiments on tabletop 2-finger grasping and suction- based bin-picking applications show MOMA achieves high success rates in diverse tasks, confirming its effectiveness. | |||||||||||||||||
85 | RGB-D Perception 1 | Learning-Based Keypoints Detection with Topological Order on Deformable Linear Objects from Incomplete Point Clouds | Li, Can;Liu, Jingyang;Sun, Lei | Keywords: RGB-D Perception, Object Detection, Segmentation and Categorization | Abstract: Detection of deformable linear objects (DLOs) in three-dimensional space is essential for robotic manipulation of DLOs. However, their complex deformations and high degrees of freedom make perception highly susceptible to occlusions, noise, and data missing. To address these challenges, we propose a deep learning-based method that leverages the topological properties of DLOs to robustly detect keypoints from incomplete point clouds while preserving the topological order of keypoints. Our approach initializes a sequence of keypoints that adheres to the topological structure of DLOs. Then, these ordered keypoints are refined through bidirectional sequence learning. Simulation results demonstrate that our method generates accurate, uniform, and smooth keypoint sequences under varying levels of occlusion. Compared to existing baselines, our approach achieves superior performance. Real-world experiments further validate the generalization capability of our method in unseen and challenging scenarios involving occlusion and self-occlusion while maintaining real-time performance. | |||||||||||||||||||||
86 | RGB-D Perception 1 | Self-Supervised Enhancement for Depth from a Lightweight ToF Sensor with Monocular Images | Ding, Laiyan;Jiang, Hualie;Chen, Jiwei;Huang, Rui | Keywords: RGB-D Perception, Deep Learning Methods, Deep Learning for Visual Perception | Abstract: Depth map enhancement using paired high-resolution RGB images offers a cost-effective solution for improving low-resolution depth data from lightweight ToF sensors. Nevertheless, naively adopting a depth estimation pipeline to fuse the two modalities requires groundtruth depth maps for supervision. To address this, we propose a self-supervised learning framework, SelfToF, which generates detailed and scale-aware depth maps. Starting from an image-based self-supervised depth estimation pipeline, we add low-resolution depth as inputs, design a new depth consistency loss, propose a scale-recovery module, and finally obtain a large performance boost. Furthermore, since the ToF signal sparsity varies in real-world applications, we upgrade SelfToF to SelfToF* with submanifold convolution and guided feature fusion. Consequently, SelfToF* maintain robust performance across varying sparsity levels in ToF data. Overall, our proposed method is both efficient and effective, as verified by extensive experiments on the NYU and ScanNet datasets. The code is available at https://github.com/denyingmxd/selftof. | |||||||||||||||||||||
87 | Deep Learning for Visual Perception 1 | Curriculum-Based Augmented Fourier Domain Adaptation for Robust Medical Image Segmentation (I) | Wang, An;Islam, Mobarakol;Xu, Mengya;Ren, Hongliang | Keywords: Deep Learning Methods, Deep Learning for Visual Perception, AI-Based Methods | Abstract: Accurate and robust medical image segmentation is fundamental and crucial for enhancing the autonomy of computer-aided diagnosis and intervention systems. Medical data collection normally involves different scanners, protocols, and populations, making domain adaptation (DA) a highly demanding research field to alleviate model degradation in the deployment site. To preserve the model performance across multiple testing domains, this work proposes the Curriculum-based Augmented Fourier Domain Adaptation (Curri-AFDA) for robust medical image segmentation. In particular, our curriculum learning strategy is based on the causal relationship of a model under different levels of data shift in the deployment phase, where the higher the shift is, the harder to recognize the variance. Considering this, we progressively introduce more amplitude information from the target domain to the source domain in the frequency space during the curriculum-style training to smoothly schedule the semantic knowledge transfer in an easier-to-harder manner. Besides, we incorporate the training-time chained augmentation mixing to help expand the data distributions while preserving the domain-invariant semantics, which is beneficial for the acquired model to be more robust and generalize better to unseen domains. Extensive experiments on two segmentation tasks of Retina and Nuclei collected from multiple sites and scanners suggest that our proposed method yields superior adaptation and generalization performance. Meanwhile, our approach proves to be more robust under various corruption types and increasing severity levels. In addition, we show that our method is also beneficial in the domain-adaptive classification task with skin lesion datasets. | |||||||||||||||||||||
88 | Deep Learning for Visual Perception 1 | TCNet: A Temporally Consistent Network for Self-Supervised Monocular Depth Estimation | Zhu, Ying;Liu, Hong;Wu, Jianbing;Liu, Mengyuan | Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Visual Learning | Abstract: Despite significant advances in self-supervised monocular depth estimation methods, achieving temporally consistent and accurate depth maps from frame sequences remains a formidable challenge. Existing approaches often estimate depth maps for individual frames in isolation, neglecting the rich geometric and temporal coherence present across frames. Consequently, this oversight leads to temporally inconsistent outputs, resulting in noticeable temporal flickering artifacts. In response, this paper presents TCNet, a Temporal Consistent Network for self-supervised monocular depth estimation. Specifically, we propose an Inter-frame Temporal Fusion (ITF) module to emphasize the influence of preceding images on the depth estimation of the current frame. The Temporal Consistency Loss (TCL) is proposed to leverage the temporal constraints between the depth maps of adjacent frames. Besides, TCNet can also be applied to both single-frame and multi-frame scenarios during inference. Experimental evaluations on the KITTI dataset demonstrate that our method surpasses state-of-the-art depth estimation methods in accuracy and temporal consistency. Our code will be made public. | |||||||||||||||||||||
89 | Deep Learning for Visual Perception 1 | DPSN: Dual Prior Knowledge Induced Tactile Paving and Obstacle Joint Segmentation Network | Song, Youqi;Li, Wenqi;Zhang, Zhao;Wu, Yu;Jin, Zilong;Wang, Changbo;He, Gaoqi | Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Vision-Based Navigation | Abstract: Accurate semantic segmentation of both tactile paving and the obstacle is crucial for the safe mobility of visually impaired individuals. However, existing methods face two major challenges: (i) discontinuous segmentation fragments; (ii) Inaccurate obstacle recognition. To address challenge (i), we propose incorporating appearance priors of complete tactile pavings to prevent the model from directly learning irregular ground truth masks. To tackle challenge (ii), we propose introducing cross-modal semantic priors to complement the semantic information of obstacles. We implemented these strategies in proposed Dual Prior knowledge induced tactile paving and obstacle joint Segmentation Network (DPSN). Based on bilateral network architecture, DPSN merges obstacle category masks into tactile paving categories, constructing a complete tactile paving mask. Utilizing the complete mask, DPSN transfer appearance prior knowledge to detail features from boundary and structural perspectives. Concurrently, DPSN leverages the CLIP Text Encoder to guide visual feature decoding by attention mechanisms, transferring rich cross-modal semantic prior knowledge to the visual feature maps. Furthermore, we propose the TPO-Dataset, the first dataset for joint tactile paving and obstacle segmentation acquired from actual scenes. Experiments demonstrate that DPSN achieves state-of-the-art results on the TPO-Dataset, with relative gains of 27.16% in obstacle IoU and 30.53% in accuracy metrics compared to baseline methods. Notably, DPSN achieves real-time performance at 88.25 FPS on the maximum scale of 2048×512 resolution. | |||||||||||||||||||||
90 | Deep Learning for Visual Perception 1 | DCT-Diffusion: Depth Completion for Transparent Objects with Diffusion Denoising Approach | Zhou, Zhenning;Shen, Weiqing;Sun, Han;Wang, Yizhao;Cao, Qixin | Keywords: Deep Learning for Visual Perception, RGB-D Perception, Deep Learning Methods | Abstract: Transparent objects are common in industrial automation and daily life. However, accurate visual perception of these objects remains challenging due to their reflective and refractive properties. Most previous studies fail to capture contextual information and typically rely on regression-based methods at the decoder stage, suffering from overfitting and unsatisfactory object details. To overcome these limitations, we present a novel depth completion framework for transparent objects with dif-fusion denoising approach (DCT-Diffusion). First, we adopt a transformer-based encoder to globally learn the depth rela-tionships from different parts of the input by modeling long-distance dependencies. Then, we propose to introduce the diffusion model to generate refined depth maps from random depth distribution. Through iterative refinement, our model can progressively enhance depth map details and achieves fi-ne-grained performance. Lastly, a conditioned fusion module is developed, which utilizes encoder features as visual conditions and fuses them with the denoising block at each step using augmented attention. Extensive comparative studies and cross-domain experiments prove that the DCT-Diffusion out-performs previous methods and significantly improves the robustness and generalization ability. Moreover, visualization results further illustrate that our method can generate depth maps with more complete geometry and clearer boundaries, achieving satisfactory results. | |||||||||||||||||||||
91 | Deep Learning for Visual Perception 1 | Stimulating Imagination: Towards General-Purpose "Something Something Placement" | Wu, Jianyang;Gu, Jie;Ma, Xiaokang;Qiu, Fangzhou;Tang, Chu;Chen, Jingmin | Keywords: Deep Learning for Visual Perception, Perception for Grasping and Manipulation | Abstract: General-purpose object placement is a fundamental capability of an intelligent generalist robot: being capable of rearranging objects following precise human instructions even in novel environments. This work is dedicated to achieving general-purpose object placement with "something something" instructions. Specifically, we break the entire process down into three parts, including object localization, goal imagination and robot control, and propose a method named SPORT. SPORT leverages a pre-trained large vision model for broad semantic reasoning about objects, and learns a diffusion-based pose estimator to ensure physically-realistic results in 3D space. Only object types (movable or reference) are communicated between these two parts, which brings two benefits. One is that we can fully leverage the powerful ability of open-set object recognition and localization since no specific fine-tuning is needed for the robotic scenario. Moreover, the diffusion-based estimator only need to "imagine" the object poses after the placement, while no necessity for their semantic information. Thus the training burden is greatly reduced and no massive training is required. The training data for the goal pose estimation is collected in simulation and annotated by using GPT-4. Experimental results demonstrate the effectiveness of our approach. SPORT can not only generate promising 3D goal poses for unseen simulated objects, but also be seamlessly applied to real-world settings. | |||||||||||||||||||||
92 | Deep Learning for Visual Perception 1 | DroneKey: Drone 3D Pose Estimation in Image Sequences Using Gated Key-Representation and Pose-Adaptive Learning | Hwang, Seo-Bin;Cho, Yeong-Jun | Keywords: Deep Learning for Visual Perception, Computer Vision for Automation, Deep Learning Methods | Abstract: Estimating the 3D pose of a drone is important for anti-drone systems, but existing methods struggle with the unique challenges of drone keypoint detection. Drone propellers serve as keypoints but are difficult to detect due to their high visual similarity and diversity of poses. To address these challenges, we propose DroneKey, a framework that combines a 2D keypoint detector and a 3D pose estimator specifically designed for drones. In the keypoint detection stage, we extract two key-representations (intermediate and compact) from each transformer encoder layer and optimally combine them using a gated sum. We also introduce a pose-adaptive Mahalanobis distance in the loss function to ensure stable keypoint predictions across extreme poses. We built new datasets of drone 2D keypoints and 3D pose to train and evaluate our method, which have been publicly released. Experiments show that our method achieves an AP of 99.68% (OKS) in keypoint detection, outperforming existing methods. Ablation studies confirm that the pose-adaptive Mahalanobis loss function improves keypoint prediction stability and accuracy. Additionally, improvements in the encoder design enable real-time processing at 44 FPS. For 3D pose estimation, our method achieved an MAE-angle of 10.62°, an RMSE of 0.221m, and an MAE-absolute of 0.076m, demonstrating high accuracy and reliability. The code and dataset are available at https://github.com/kkanuseobin/DroneKey. | |||||||||||||||||||||
93 | Deep Learning for Visual Perception 1 | L2COcc: Lightweight Camera-Centric Semantic Scene Completion Via Distillation of LiDAR Model | Wang, Ruoyu;Ma, Yukai;YaoYi, Yaoyi;Tao, Sheng;Li, Haoang;Zhu, Zongzhi;Liu, Yong;Zuo, Xingxing | Keywords: Deep Learning for Visual Perception, Computer Vision for Transportation, Computer Vision for Automation | Abstract: Abstract— Semantic Scene Completion (SSC) constitutes a pivotal element in autonomous driving perception systems, tasked with inferring the 3D semantic occupancy of a scene from sensory data. To improve accuracy, prior research has implemented various computationally demanding and memory- intensive 3D operations, imposing significant computational requirements on the platform during training and testing. This paper proposes L2COcc, a lightweight camera-centric SSC framework that also accommodates LiDAR inputs. With our proposed efficient voxel transformer (EVT) and three types of cross-modal knowledge modules (FSD, TPVD, PAD), our method substantially reduce computational burden while main- taining high accuracy. The experimental evaluations demon- strate that our proposed method surpasses the current state-of- the-art vision-based SSC methods regarding accuracy on both the SemanticKITTI and SSCBench-KITTI-360 benchmarks, respectively. Additionally, our method is more lightweight, ex- hibiting a reduction in both memory consumption and inference time by over 25%. Code is available at the project page: https://studyingfufu.github.io/L2COcc/. | |||||||||||||||||||||
94 | Deep Learning for Visual Perception 1 | Self-Distilled Stereo Matching: Real-Time Domain Generalization for Robotic Depth Perception | Zhang, Xuxin;Li, Kunhong;Zhang, Yongjian;Song, Zhuo;Jiang, Runqing;Zhang, Ye;Guo, Yulan | Keywords: Deep Learning for Visual Perception, Machine Learning for Robot Control | Abstract: While human vision inherently achieves robust cross-domain depth estimation through binocular coordination, robotic systems employing stereo matching still confront significant challenges in maintaining robustness across domains when performing real-time environmental depth perception. Furthermore, most stereo matching methods struggle with challenging regions such as object boundaries and non-overlapping areas on the left side of the left image, resulting in disparity maps that are relatively indistinct and lacking fine details. In this paper, we propose Learning More in Challenging Areas (LMC) to alleviate this problem, which enhances the domain generalization of the model through targeted training on challenging regions. LMC is a simple yet effective data-driven training framework primarily based on self-distillation. Specifically, 1) We pre-train models on a high-frequency dataset to improve perception ability on object boundaries; 2) We develop a self-distillation training strategy to benefit learning in non-overlapping areas on the left side of the left image; 3) We design an adaptive difficult area mask to balance the loss weight on other undefined challenging regions. Under our proposed training framework, GwcNet achieves 33% and 23% performance improvements in autonomous driving benchmarks KITTI 2012 and KITTI 2015 respectively, while preserving real-time inference efficiency without computational overhead. | |||||||||||||||||||||
95 | Deep Learning Methods 1 | YO-CSA-T: A Real-Time Badminton Tracking System Utilizing YOLO Based on Contextual and Spatial Attention | Lai, Yuan;Zhiwei, Shi;Zhu, Chengxi | Keywords: Deep Learning Methods, Visual Tracking, AI-Based Methods | Abstract: The 3D trajectory of a shuttlecock required for a badminton rally robot for human-robot competition demands real-time performance with high accuracy. However, the fast flight speed of the shuttlecock, along with various visual effects, and its tendency to blend with environmental elements, such as court lines and lighting, present challenges for rapid and accurate 2D detection. In this paper, we first propose the YO-CSA detection network, which optimizes and reconfigures the YOLOv8s model's backbone, neck, and head by incorporating contextual and spatial attention mechanisms to enhance model's ability in extracting and integrating both global and local features. Next, we integrate three major sub-tasks—detection, prediction, and compensation—into a real-time 3D shuttlecock trajectory detection system. Specifically, our system maps the 2D coordinate sequence extracted by YO-CSA into 3D space using stereo vision, then predicts the future 3D coordinates based on historical information, and re-projects them onto the left and right views to update the position constraints for 2D detection. Additionally, our system includes a compensation module to fill in missing intermediate frames, ensuring a more complete trajectory. We conduct extensive experiments on our own dataset to evaluate both YO-CSA's performance and system effectiveness. Experimental results show that YO-CSA achieves a high accuracy of 90.43% mAP@0.75, surpassing both YOLOv8s and YOLO11s. Our system performs excellently, maintaining a speed of over 130 fps across 12 test sequences. | |||||||||||||||||||||
96 | Deep Learning Methods 1 | RCGNet: RGB-Based Category-Level 6D Object Pose Estimation with Geometric Guidance | Yu, Sheng;Zhai, Di-Hua;Xia, Yuanqing | Keywords: Deep Learning Methods, Computer Vision for Automation, Computer Vision for Manufacturing | Abstract: While most current RGB-D-based category-level object pose estimation methods achieve strong performance, they face significant challenges in scenes lacking depth information. In this paper, we propose a novel category-level object pose estimation approach that relies solely on RGB images. This method enables accurate pose estimation in real-world scenarios without the need for depth data. Specifically, we design a transformer-based neural network for category-level object pose estimation, where the transformer is employed to predict and fuse the geometric features of the target object. To ensure that these predicted geometric features faithfully capture the object's geometry, we introduce a geometric feature-guided algorithm, which enhances the network's ability to effectively represent the object's geometric information. Finally, we utilize the RANSAC-PnP algorithm to compute the object's pose, addressing the challenges associated with variable object scales in pose estimation. Experimental results on benchmark datasets demonstrate that our approach is not only highly efficient but also achieves superior accuracy compared to previous RGB-based methods. These promising results offer a new perspective for advancing category-level object pose estimation using RGB images. | |||||||||||||||||||||
97 | Deep Learning Methods 1 | Application of LLM Guided Reinforcement Learning in Formation Control with Collision Avoidance | Yao, Chenhao;Yuan, Zike;Liu, Xiaoxu;Zhu, Chi | Keywords: Deep Learning Methods, Multi-Robot Systems, Big Data in Robotics and Automation | Abstract: Multi-Agent Systems (MAS) excel at accomplishing complex objectives through the collaborative efforts of individual agents. Among the methodologies employed in MAS, Multi-Agent Reinforcement Learning (MARL) stands out as one of the most efficacious algorithms. However, when confronted with the complex objective of Formation Control with Collision Avoidance (FCCA): designing an effective reward function that facilitates swift convergence of the policy network to an optimal solution. In this paper, we introduce a novel framework that aims to overcome this challenge. By giving large language models (LLMs) on the prioritization of tasks and the observable information available to each agent,our framework generates reward functions that can be dynamically adjusted online based on evaluation outcomes by employing more advanced evaluation metrics rather than the rewards themselves. This mechanism enables the MAS to simultaneously achieve formation control and obstacle avoidance in dynamic environments with enhanced efficiency, requiring fewer iterations to reach superior performance levels. Our empirical studies, conducted in both simulation and real-world settings, validate the practicality and effectiveness of our proposed approach. | |||||||||||||||||||||
98 | Deep Learning Methods 1 | NaviDiffuser: Tackling Multi-Objective Robot Navigation by Weight Range Guided Diffusion Model | Zhang, Xuyang;Feng, Ziyang;Qiu, Quecheng;Peng, Jie;Li, Haoyu;Ji, Jianmin | Keywords: Deep Learning Methods, Collision Avoidance, Learning from Experience | Abstract: The data-driven paradigm has shown great potential in solving many decision-making tasks. In the robot navigation realm, it also sparked a new trend. People believe powerful data-driven methods can learn efficient and general navigation policies from a vast offline dataset. However, robot navigation tasks differ from common planning tasks and present unique challenges. It often involves multi-objective optimization to meet arbitrary and ever-changing human preferences. It should also overcome the short-sighted problem to obtain globally optimal performance. Furthermore, high planning frequency is needed to address real-time demands. These factors obstruct the application of data-driven methods in robot navigation. To address these challenges, we integrate one of the most powerful data-driven methods, the diffusion model, into robot navigation. Our proposed approach, NaviDiffuser, utilizes a novel classification label to guide the diffusion model in capturing the complex connections between navigation and human preferences. Its Transformer network backbone outputs action sequences to alleviate short-sightedness. It also includes special distillation skills to boost the planning speed and quality. We conduct experiments in both simulated and real-world scenarios to evaluate our approach. In these experiments, NaviDiffuser not only demonstrates an extremely high arrival rate but also adjusts its navigation policy to align with different human preferences. | |||||||||||||||||||||
99 | Deep Learning Methods 1 | Annotation-Free Curb Detection Leveraging Altitude Difference Image | Ma, Fulong;Hou, Peng;Liu, Yuxuan;Liu, Yang;Liu, Ming;Ma, Jun | Keywords: Deep Learning Methods, Computer Vision for Transportation, Deep Learning for Visual Perception | Abstract: Road curbs are considered as one of the crucial and ubiquitous traffic features, which are essential for ensuring the safety of autonomous vehicles. Current methods for detecting curbs primarily rely on camera imagery or LiDAR point clouds. Image-based methods are vulnerable to fluctuations in lighting conditions and exhibit poor robustness, while methods based on point clouds circumvent the issues associated with lighting variations. However, it is the typical case that significant processing delays are encountered due to the voluminous amount of 3D points contained in each frame of the point cloud data. Furthermore, the inherently unstructured characteristics of point clouds poses challenges for integrating the latest deep learning advancements into point cloud data applications. To address these issues, this work proposes an annotation-free curb detection method leveraging Altitude Difference Image (ADI) (as shown in Fig. 1), which effectively mitigates the aforementioned challenges. Given that methods based on deep learning generally demand extensive, manually annotated datasets, which are both expensive and labor-intensive to create, we present an Automatic Curb Annotator (ACA) module. This module utilizes a deterministic curb detection algorithm to automatically generate a vast quantity of training data. Consequently, it facilitates the training of the curb detection model without necessitating any manual annotation of data. Finally, by incorporating a post-processing module, we manage to achieve state-of-the-art results on the KITTI 3D curb dataset with considerably reduced processing delays compared to existing methods, which underscores the effectiveness of our approach in curb detection tasks. | |||||||||||||||||||||
100 | Deep Learning Methods 1 | R2LDM: An Efficient 4D Radar Super-Resolution Framework Leveraging Diffusion Model | Zheng, Boyuan;Lu, Shouyi;Huang, Renbo;Huang, Minqing;Lu, Fan;Tian, Wei;Zhuo, Guirong;Xiong, Lu | Keywords: Deep Learning Methods, Intelligent Transportation Systems, Autonomous Vehicle Navigation | Abstract: We introduce R2LDM, an innovative approach for generating dense and accurate 4D radar point clouds, guided by corresponding LiDAR point clouds. Instead of utilizing range images or birdÂ’s eye view (BEV) images, we represent both LiDAR and 4D radar point clouds using voxel features, which more effectively capture 3D shape information. Subsequently, we propose the Latent Voxel Diffusion Model (LVDM), which performs the diffusion process in the latent space. Additionally, a novel Latent Point Cloud Reconstruction (LPCR) module is utilized to reconstruct point clouds from high-dimensional latent voxel features. As a result, R2LDM effectively generates LiDAR-like point clouds from paired raw radar data. We evaluate our approach on two different datasets, and the experimental results demonstrate that our model achieves 6- to 10-fold densification of radar point clouds, outperforming state-of-the-art baselines in 4D radar point cloud super-resolution. Furthermore, the enhanced radar point clouds generated by our method significantly improve downstream tasks, achieving up to 31.7% improvement in point cloud registration recall rate and 24.9% improvement in object detection accuracy. | |||||||||||||||||||||