Real-Time Caterpillar Detection and Tracking in Orchard Using YOLO-NAS Plus SORT

Abstract

Detecting and tracking caterpillars in orchard environments is crucial for advancing precision agriculture but remains challenging due to occlusions, variable lighting, wind interference, and the need for precise small-object detection. This study presents a real-time deep learning approach that integrates the YOLO-NAS object detection model with the SORT tracking algorithm to overcome these challenges. Evaluated in a jujube orchard, the proposed method significantly improved small caterpillar detection and tracking. Using an RGB-D camera operating at 30 frames per second, the system successfully detected caterpillars measuring 2–5 cm at distances of 20–35 cm, corresponding to resolutions of 21 × 6 to 55 × 10 pixels. The integration of YOLO-NAS with SORT enhanced detection performance, achieving a ~9% increase in true positive detections and an ~8% reduction in false positives compared to YOLO-NAS alone. Even for the smallest caterpillars (21 × 6 pixels), the method achieved over 60% true positive detection accuracy without false positives within 1 s inference. With an inference time of just 0.2 milliseconds, SORT enabled real-time tracking and accurately predicted caterpillar positions under wind interference, further improving reliability. Additionally, selective corner tracking was employed to identify the head and tail of caterpillars, paving the way for future laser-based precision-targeting interventions focused on the caterpillar head.