Predictive Visual Servoing for Autonomous UAV Landing
Camera-guided positioning and controlled descent of a simulated aerial vehicle.
Camera observations during simulated autonomous landing, illustrating target alignment and progressive descent.
Motivation
Vision-based autonomous control relies on continuous interpretation of visual observations and their evolution over time. In dynamic scenarios, such as autonomous landing, predicting future positioning errors and motion trends may provide useful information for closed-loop control.
The motivation behind this project was to explore the capabilities of xLSTM as a temporal prediction model within a visual servoing architecture, investigating how learned temporal dependencies can support camera-guided positioning and motion control.
An autonomous landing scenario was developed in NVIDIA Isaac Sim as an experimental environment. Using a downward-facing camera, the system detects a landing marker, estimates positioning errors, and executes a controlled descent with xLSTM-assisted visual servoing.
Project Architecture
The architecture integrates xLSTM-based temporal prediction into a ROS 2 visual servoing control loop. Visual observations are processed to estimate the position of the landing target and its distance from the camera.
Two predictive functions are investigated:
- future horizontal positioning errors (X/Y), used to assist motion correction, and
- future camera depth (Z), used to support descent-speed adjustment during final approach.
The predicted information complements the feedback controller, which generates guidance commands for the simulated aerial vehicle in NVIDIA Isaac Sim.
Closed-loop architecture integrating xLSTM-based temporal prediction into vision-guided autonomous landing.
Experimental Results
The experimental setup was evaluated through simulated autonomous landing episodes in NVIDIA Isaac Sim. The following trajectories illustrate the positioning and descent behavior recorded during an xLSTM-assisted landing sequence.
The recorded motion demonstrates progressive horizontal alignment with the landing area, followed by a controlled vertical descent. Position values are obtained from the simulator and expressed relative to the final recorded position.
Horizontal Positioning (X/Y)
The horizontal positioning trajectory illustrates the evolution of the vehicle's X and Y coordinates during target alignment. The controller continuously adjusts the simulated motion based on visual observations, with temporal predictions available to assist the correction process.
Recorded horizontal positioning trajectory during simulated autonomous landing.
Vertical Descent (Z)
The vertical trajectory illustrates the progression from the initial positioning phase toward the landing surface. The descent sequence incorporates camera-depth observations and xLSTM-based depth prediction to support vertical motion control during the final approach.
Recorded vertical trajectory showing the autonomous descent sequence.
The presented trajectories originate from a recorded simulation episode. Position data represents simulator ground truth rather than camera-estimated measurements.
xLSTM Temporal Prediction
The xLSTM model predicts the evolution of visual positioning errors based on previous camera observations. The following figure illustrates predicted horizontal positioning errors compared with subsequently observed camera measurements during a simulated landing episode.
Temporal prediction of horizontal visual positioning errors using xLSTM. Predictions are aligned with their corresponding future observation times. Simulation data from NVIDIA Isaac Sim.
Conclusion
This experiment demonstrates the integration of xLSTM-based temporal prediction into a vision-guided autonomous landing architecture using NVIDIA Isaac Sim and ROS 2.
The simulated landing sequences illustrate how visual observations, temporal predictions, and closed-loop control can be combined to support autonomous positioning and controlled descent. The recorded results provide practical insight into the application of predictive models within robotic feedback-control systems.
The project provides a foundation for further investigation into prediction-assisted visual servoing, particularly under conditions involving dynamic disturbances, observation delays, and temporary loss of visual feedback.
References
This experiment builds upon the xLSTM architecture introduced by Beck et al. and developed within the research activities of NXAI.
- Beck, M. et al. (2024). xLSTM: Extended Long Short-Term Memory . NeurIPS 2024.
- xLSTM — Official GitHub Repository
- NXAI GmbH — Research and development of advanced AI architectures.