AI for Industry Challenge: Vision-Based Robotic Insertion

Motivation

This project was developed as part of the AI for Industry Challenge by Intrinsic, focused on autonomous robotic manipulation in a constrained simulation environment. The goal was to build a perception-driven pipeline capable of localizing task-relevant objects, handling deformable cable behavior, and executing cable insertion without relying on ground-truth information.

The objective was to establish a reliable insertion pipeline under changing conditions, including variations in board position, socket location, and connector type within a simulation environment.

Project description

The solution focuses on a fully autonomous insertion pipeline, combining vision-based localization, task-level decision logic, contact-aware execution, and recovery behavior. The system was designed to operate under challenge constraints, including Docker-based deployment, limited runtime assumptions, and variability across evaluation trials.

The challenge toolkit provided a complete simulation environment, including the robot, gripper, wrist-mounted cameras, task objects, and evaluation infrastructure. Participants were responsible for developing the insertion policy and packaging the solution into a deployable container.

The environment supported extensive local testing under varying conditions before final submission. Policies were evaluated through an automated scoring system that measured insertion performance across multiple randomized trials.

System overview

The solution combines motion planning, robot control, computer vision, and AI components into a unified execution pipeline. Separate workflows were implemented for the two connector types defined by the challenge task.

For each task, the correct socket had to be localized before insertion. This was achieved using a combination of classical machine vision techniques and AI-based classification methods. The estimated pose was then transformed into the robot coordinate system and used to generate insertion motions.

During insertion, force sensor feedback was used to detect contact conditions, compensate for alignment errors, and prevent excessive insertion forces.

Ground-truth information was available during development and debugging. However, all evaluation runs were performed without access to ground-truth data, requiring the final solution to rely entirely on perception.

Technologies used

• ROS 2
• OpenCV
• Python
• Docker
• Gazebo simulation
• Force/torque sensing
• AI-based image classification

Key challenges addressed

Task variability made it impractical to solve the challenge using traditional fixed-position robot programming. Variations in board geometry, socket location, and cable behavior introduced uncertainty that required perception-driven decision making and adaptive execution.

In addition, the deformable cable frequently caused occlusions and introduced external forces that affected insertion accuracy. The policy therefore had to remain robust across a wide range of operating conditions.

• Removal of ground-truth dependence
• Vision-based socket and cable localization
• Handling deformable cable interference
• Force/contact-aware execution logic
• Recovery and search strategies under uncertain alignment
• Packaging into a constrained Docker runtime

Outcome

The project achieved stable scoring across evaluation trials and ranked 54th out of 160 participating teams.

Beyond the ranking, the primary outcome was the development of a fully autonomous robotic manipulation pipeline built from scratch, combining perception, control logic, force-aware execution, recovery behavior, and deployment within a constrained competition environment.

What is intentionally not shown

Exact policies, thresholds, search logic, and implementation details are intentionally omitted.