A Spatiotemporal Multimodal Transformer for Pre-Harvest Apple and Pear Quality Prediction

Zhitao Fu, Huaren Shen, Yan Shi, Yiheng Zhang, Luyao Xiao, YJ Li, M Dong

This study addresses the problem of pre-harvest fruit commodity grade prediction in intelligent orchards. To overcome the limitations of conventional post-harvest grading systems, including delayed quality evaluation, the limited representation capability of single-modality approaches, and the difficulty of modeling heterogeneous agricultural information collected throughout the growing season, this study proposes STF-Former, a spatiotemporal multimodal Transformer framework for pre-harvest fruit quality prediction. Rather than relying solely on static visual observations, the proposed framework formulates fruit quality prediction as a multimodal spatiotemporal learning problem by jointly exploiting phenotypic evolution, environmental dynamics, and field management information across different growth stages. Through stage-aware temporal modeling and cross-modal representation learning, STF-Former captures the dynamic interactions between fruit developmental processes and environmental water–fertilizer conditions, providing a unified prediction framework for both commodity grades and key quality indicators. This design improves the interpretability of quality formation and establishes a methodological framework for integrating heterogeneous agricultural data in precision orchard management. Experimental results demonstrate that the proposed method achieves significant advantages in both classification and regression tasks. In the commodity grade classification task, STF-Former achieves an Accuracy of 0.887, a Precision of 0.875, a Recall of 0.861, a Macro-F1 score of 0.868, and an AUC of 0.924, substantially outperforming traditional machine-learning methods (Random Forest and XGBoost) as well as mainstream unimodal deep-learning models. In the regression task for key quality indicators, superior performance is consistently achieved across multiple agronomic metrics, where the mean absolute error (MAE) is 3.41 mm for fruit diameter, 11.85 g for single fruit weight, 0.057 for coloration index, 0.81 °Brix for soluble solid content, and 2.68 N for firmness, with an overall R2 reaching 0.846. These results validate the effectiveness and robustness of multimodal spatiotemporal learning for accurate pre-harvest fruit quality prediction. The proposed framework provides a practical technical solution for intelligent orchard management, harvest planning, and data-driven precision agriculture, while offering a scalable paradigm for integrating multimodal sensing and temporal learning in smart agricultural systems.