Deep Learning-Based Markerless Hand Tracking for Freely Moving Non-Human Primates in Brain–Machine Interface Applications

Yuhang Liu, Miao Wang, Shuaibiao Hou, Xiao Wang, Bing Shi

The motor cortex of non-human primates plays a key role in brain–machine interface (BMI) research. In addition to recording cortical neural signals, accurately and efficiently capturing the hand movements of experimental animals under unconstrained conditions remains a key challenge. Addressing this challenge can deepen our understanding and application of BMI behavior from both theoretical and practical perspectives. To address this issue, we developed a deep learning framework that combines Yolov5 and RexNet-ECA to reliably detect the hand joint positions of freely moving primates at different distances using a single camera. The model simplifies the setup procedure while maintaining high accuracy, with an average keypoint detection error of less than three pixels. Our method eliminates the need for physical markers, ensuring non-invasive data collection while preserving the natural behavior of the experimental subjects. The proposed system exhibits high accuracy and ease of use compared to existing methods. By quickly and accurately acquiring spatiotemporal behavioral metrics, the method provides valuable insights into the dynamic interplay between neural and motor functions, further advancing BMI research.