Umair LatifVision Systems Architect
Back to systems

ABB Robotics R&D

Robotic Perception for Mining Automation

Robotics R&D Engineer / Apr 2023 - Oct 2023

Built a robotic perception workflow for mining automation using ROS 2, YOLO object detection, point-cloud position estimation, and Gazebo simulation to translate visual detections into spatial information usable by robotic system logic.

ROS 2YOLO detectionPoint-cloud localizationGazebo simulationRobotic perceptionPython

At a glance

Technical Overview and Key Facts

Role
Robotics R&D Engineer
System type
Robotic perception R&D workflow
Core stack
ROS 2, YOLO, point clouds, 3D localization, Gazebo, Python
Constraints
Detection-to-position reasoning, ROS 2 interfaces, simulation before hardware risk
Delivered
YOLO-to-point-cloud localization workflow, ROS 2 perception components, Gazebo validation

Problem

In mining automation, object detection alone is not enough. A robot must understand where the detected object exists in space, whether the estimate is usable, and how that perception output connects to the rest of the robotic system before physical action is considered.

My Role

Developed the perception workflow between camera input and robot decision logic. Connected YOLO detections to point-cloud reasoning, estimated object position, structured ROS 2 integration, and used Gazebo simulation to evaluate behavior before exposing physical hardware to risk.

Impact

Helped bridge the gap between computer vision output and robotic action planning by converting object detections into spatial information that a robotic system could reason about inside a ROS 2 and Gazebo-based R&D workflow.

System Architecture

  • Perception input layer: camera and depth/point-cloud data provided the visual and spatial input for the robotic perception workflow.
  • Detection layer: YOLO identified objects of interest in the image stream and produced candidate detections.
  • 3D localization layer: point-cloud processing converted detections into spatial position estimates that could support robotic reasoning.
  • ROS 2 integration layer: detection and position outputs were structured into ROS 2 interfaces for downstream robotic system logic.
  • Simulation validation layer: Gazebo simulation provided a controlled environment to review perception behavior before real-world hardware testing.

Tools

ROS 2YOLOPoint clouds3D localizationGazeboPython

Robotics Constraints

  • Perception output needed to support robotic decision-making, not only image-level detection.
  • Object detections had to be converted into 3D position estimates before they could support planning logic.
  • ROS 2 interfaces needed to keep perception outputs clear, structured, and usable by downstream components.
  • Gazebo simulation had to reduce hardware risk before real-world robotic trials.
  • The workflow needed to support R&D iteration without treating simulation results as full deployment validation.

What I Delivered

  • Connected YOLO object detection to point-cloud-based position estimation.
  • Built ROS 2 perception components for robotic system integration.
  • Structured perception outputs so they could be used by downstream robot logic.
  • Used Gazebo simulation to evaluate behavior before physical hardware trials.
  • Produced a workflow that translated image detections into spatial information for robotic reasoning.
  • Moved the perception workflow beyond bounding boxes by making detection output spatially meaningful.

Simulation Validation & Integration

Simulation reduced hardware risk before physical trials.

Validation focused on whether perception results could become spatial information a robotic system could use inside simulated mining-operation workflows. The goal was to test the detection-to-position-estimation chain before exposing real hardware to unnecessary risk.

  • YOLO object-detection behavior.
  • Point-cloud position-estimation behavior.
  • Detection-to-3D localization workflow.
  • ROS 2 message integration.
  • Gazebo simulation review.
  • Interface behavior between perception and robot logic.

Implementation Highlights

The valuable work sat between visual detection and robotic action.

Connected object detection to 3D position estimation instead of stopping at bounding boxes.
Used point-cloud reasoning to make perception output spatially useful.
Integrated perception outputs through ROS 2 for downstream robotic logic.
Used Gazebo simulation to evaluate robotic vision behavior before physical trials.
Worked in an R&D setting where perception needed to support future physical task execution.

Next Step

Need a vision system that survives production?

Email UmairOpen LinkedInView more systems