Walnut Ripeness Detection Based on Coupling Information and Lightweight YOLOv4

495155d1-e80d-4a3b-87e7-f861944615ff20220121063105639naun:naunmdt@crossref.orgMDT DepositInternational Journal of Circuits, Systems and Signal Processing1998-446410.46300/9106http://www.naun.org/cms.action?id=30291720221720221610.46300/9106.2022.16https://npublications.com/journals/circuitssystemssignal/2022.phpWalnut Ripeness Detection Based on Coupling Information and Lightweight YOLOv4KaixuanCuiSchool of Technology, Beijing Forestry University, Beijing, 100083, ChinaShuchaiSuKey Laboratory for Silviculture and Conservation of Ministry of Education, Beijing, 100083, ChinaJiaweiCaiSchool of Technology, Beijing Forestry University, Beijing, 100083, ChinaFengjunChenSchool of Technology, Beijing Forestry University, Beijing, 100083, ChinaTo realize rapid and accurate ripeness detection for walnut on mobile terminals such as mobile phones, we propose a method based on coupling information and lightweight YOLOv4. First, we collected 50 walnuts at each ripeness (Unripe, Mid-ripe, Ripe, Over-ripe) to determine the kernel oil content. Pearson correlation analysis and one-way analysis of variance (ANOVA) prove that the division of walnut ripeness reflects the change in kernel oil content. It is feasible to estimate the kernel oil content by detecting the ripeness of walnut. Next, we achieve ripeness detection based on lightweight YOLOv4. We adopt MobileNetV3 as the backbone feature extractor and adopt depthwise separable convolution to replace the traditional convolution. We design a parallel convolution structure with depthwise convolution stacking (PCSDCS) to reduce parameters and improve feature extraction ability. To enhance the model’s detection ability for walnuts in the growth-intensive areas, we design a Gaussian Soft DIoU non-maximum suppression (GSDIoU-NMS) algorithm. The dataset used for model optimization contains 3600 images, of which 2880 images in the training set, 320 images in the validation set, and 400 images in the test set. We adopt a multi-training strategy based on dynamic learning rate and transfer learning to get training weights. The lightweight YOLOv4 model achieves 94.05%, 90.72%, 88.30%, 76.92 FPS, and 38.14 MB in mean average precision, precision, recall, average detection speed, and weight capacity, respectively. Compared with the Faster R-CNN model, EfficientDet-D1 model, YOLOv3 model, and YOLOv4 model, the lightweight YOLOv4 model improves 8.77%, 4.84%, 5.43%, and 0.06% in mean average precision, 74.60 FPS, 55.60 FPS, 38.83 FPS, and 46.63 FPS in detection speed, respectively. And the lightweight YOLOv4 is 84.4% smaller than the original YOLOv4 model in terms of weight capacity. This paper provides a theoretical reference for the rapid ripeness detection of walnut and exploration for the model’s lightweight.11020221102022239247https://npublications.com/journals/circuitssystemssignal/2022/a582005-029(2022).pdf10.46300/9106.2022.16.29https://npublications.com/journals/circuitssystemssignal/2022/a582005-029(2022).pdf10.5897/ajb11.2852Y. Li, S. Ma, Y. Wang, et al., “The dynamics of fat, protein and sugar metabolism during walnut (Juglans regia L.) fruit development,” AFRICAN JOURNAL OF BIOTECHNOLOGY, vol. 11, no. 5, pp. 1267–1276, Jan. 2012. 10.3390/s20030637N. A. Aliteh, K. Minakata, K. Tashiro, et al., “Fruit Battery Method for Oil Palm Fruit Ripeness Sensor and Comparison with Computer Vision Method,” Sensors, vol. 20, no. 3, pp. 637–650, Jan. 2020. 10.1016/j.scienta.2020.109231R. Sinambela, T. Mandang, I. Subrata, and W. Hermawan, “Application of an inductive sensor system for identifying ripeness and forecasting harvest time of oil palm,” Scientia Horticulturae, vol. 265, pp. 109231, Jan. 2020. 10.3390/s17010052N. Misron, N. A. Aliteh, N. H. Harun, et al., “Relative Estimation of Water Content for Flat-Type Inductive-Based Oil Palm Fruit Maturity Sensor,” Sensors, vol. 17, no. 1, pp. 52–61, Dec. 2016. 10.1016/j.biosystemseng.2020.04.001S. Landahl and L. A. Terry, “Non-destructive discrimination of avocado fruit ripeness using laser Doppler vibrometry,” Biosystems Engineering, vol. 194, pp. 251–260, Apr. 2020. 10.3390/foods10020323N. Arai, M. Miyake, K. Yamamoto, and I. Kajiwara, “Soft mango firmness assessment based on rayleigh waves generated by a laser-induced plasma shock wave technique,” Foods, vol. 10, no. 2, pp. 323–338, Jan. 2021. 10.1364/ao.56.001753E. N. Obledo-Vázquez, and J. Cervantes-Martínez, “Laser-induced fluorescence spectral analysis of papaya fruits at different stages of ripening,” Applied optics, vol. 56, no. 6, pp. 1753–1756, Feb. 2017. 10.1002/fsn3.1788N. Aghilinategh, M. J. Dalvand, and A. Anvar, “Detection of ripeness grades of berries using an electronic nose,” Food Science & Nutrition, vol. 8, no. 9, pp. 4919–4928, Jun. 2020. 10.1016/j.cclet.2020.04.032Q. Zhao, Z. Duan, Z. Yuan, et al., “High performance ethylene sensor based on palladium-loaded tin oxide:Application in fruit quality detection,” Chinese Chemical Letters, vol. 31, no. 8, pp. 2045–2049, May. 2020. 10.3390/s150100899M. Baietto and A. D. Wilson, “Electronic-nose applications for fruit identification, ripeness and quality grading,” Sensors, vol. 15, no. 1, pp. 899–931, Jan. 2015. 10.3390/s18103256L. Y. Chen, C. C. Wu, T. I. Chou, S. W. Chiu, and K. T. Tang, “Development of a Dual MOS electronic nose/camera system for improving fruit ripeness classification,” Sensors, vol. 18, no. 10, pp. 3256–3266, Sep. 2018. 10.1016/j.microc.2021.106048K. R. Borba, F. Oldoni, T. Monaretto, L. A. Colnago, and M. D. Ferreira, “Selection of industrial tomatoes using TD-NMR data and computational classification methods,” Microchemical Journal, vol. 164, no. 4, pp. 106048, Feb. 2021. 10.1016/j.asoc.2015.07.009N. Goel and P. Sehgal, “Fuzzy classification of pre-harvest tomatoes for ripeness estimation – An approach based on automatic rule learning using decision tree,” Applied Soft Computing, vol. 36, pp. 45–56, Jul. 2015. 10.1049/iet-ipr.2018.6524R. Hamza and M. Chtourou, “Design of fuzzy inference system for apple ripeness estimation using gradient method,” IET Image Processing, vol. 14, no. 3, pp. 561–569, Feb. 2020. 10.1016/j.compag.2017.12.029L. F. Santos Pereira, S. Barbon, N. A. Valous, and D. F. Barbin, “Predicting the ripening of papaya fruit with digital imaging and random forests,” Computers and Electronics in Agriculture, vol. 145, pp. 76–82, Dec. 2017. 10.1016/j.scienta.2019.05.065I. H. Kao, Y. W. Hsu, Y. Z. Yang, et al., “Determination of Lycopersicon maturity using convolutional autoencoders,” Scientia Horticulturae, vol. 256, pp. 108538, Jun. 2019. 10.1007/s13369-018-03695-5F. M. A. Mazen and A. A. Nashat, “Ripeness classification of bananas using an artificial neural network,” Arabian Journal for Science and Engineering, vol. 44, no. 8, pp. 6901–6910, Jan. 2019. 10.1016/j.postharvbio.2015.07.006P. Rungpichayapichet, B. Mahayothee, M. Nagle, P. Khuwijitjaru, and J. Müller, “Robust NIRS models for non-destructive prediction of postharvest fruit ripeness and quality in mango,” Postharvest Biology and Technology, vol. 111, pp. 31–40, Jan. 2016. 10.1007/s12161-019-01506-7Y. Y. Pu, D. W. Sun, M. Buccheri, et al., “Ripeness classification of bananito fruit (Musa acuminata, AA): a comparison study of visible spectroscopy and hyperspectral imaging,” Food Analytical Methods, vol. 12, no. 8, pp. 1693–1704, May. 2019. 10.1016/j.jfoodeng.2017.06.031S. Munera, J. M. Amigo, J. Blasco, et al., “Ripeness monitoring of two cultivars of nectarine using VIS-NIR hyperspectral reflectance imaging,” Journal of Food Engineering, vol. 214, no. 8, pp. 29–39, Dec. 2017. 10.1016/j.patcog.2021.108102X. Bai, X. Wang, X. L. Liu, et al., “Explainable deep learning for efficient and robust pattern recognition: A survey of recent developments,” Pattern Recognition, vol. 120, pp. 108102, Dec. 2021. 10.37394/232014.2020.16.23S. Muni Rathnam, G. Siva Koteswara Rao, “A Novel Deep Learning Architecture for Image Hiding,” WSEAS Transactions on Signal Processing, vol. 16, pp. 206-210, Feb. 2020. J. Xue, Y. Wang, A. Qu, et al., “Image segmentation method for Lingwu long jujubes based on improved FCN-8s,” Transactions of the Chinese Society of Agricultural Engineering, vol. 37, no. 5, pp. 191–197, Mar. 2021. 10.1109/access.2020.3038184Y. P. Huang, T. H. Wang, and H. Basanta, “Using Fuzzy Mask R-CNN Model to Automatically Identify Tomato Ripeness,” IEEE Access, vol. 8, pp. 207672–207682, Nov. 2020. 10.1016/j.biosystemseng.2020.12.002S. Parvathi and S. T. Selvi, “Detection of maturity stages of coconuts in complex background using Faster R-CNN model,” Biosystems Engineering, vol. 202, pp. 119–132, Jan. 2021. 10.1155/2020/8859237W. Chen, S. Lu, B. Liu, G. Li, and T. Qian, “Detecting Citrus in Orchard Environment by Using Improved YOLOv4,” Scientific Programming, vol. 2020, pp. 8859237, Nov. 2020. 10.3390/agronomy10071016A. Kuznetsova, T. Maleva, and V. Soloviev, “Using YOLOv3 algorithm with pre-and post-processing for apple detection in fruit-harvesting robot,” Agronomy, vol. 10, no. 7, pp. 1016-1034, Jul. 2020. X. Liu, C. Cheng, J. Li, et al., “Identification Method of Strawberry Based on Convolutional Neural Network,” Transactions of the Chinese Society for Agricultural Machinery, vol. 51, no. 2, pp. 237–244, Feb. 2020. 10.3390/s21041492G. Li, Y. Huang, Z. Chen, et al., “Practices and Applications of Convolutional Neural Network-Based Computer Vision Systems in Animal Farming: A Review,” Sensors (Basel, Switzerland), vol. 21, pp. 1492–1492, Feb. 2021. 10.1109/isspit.2008.4775650Roumen Kountchev, Roumiana Kountcheva, “Image Segmentation based on Adaptive Mode Quantization and 2D Histograms Analysis,” WSEAS Transactions on Signal Processing, vol. 15, pp. 121-128, Mar. 2019. 10.1109/cvpr46437.2021.01283A. Bochkovskiy, C. Y. Wang, and H. Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 2020. H. Gao, Y. L. Tian, F. Y. Xu, and S. Zhong, “Survey of Deep Learning Model Compression and Acceleration,” Journal of Software, vol. 32, no. 1, pp. 68–92, Jun. 2020. A. Howard, M. Sandler, G. Chu, et al., “Searching for mobilenetv3,” In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1314–1324, Nov. 2019. A. G. Howard, M. Zhu, B. Chen, et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” https://arxiv.org/abs/1704. 04861, Apr. 2017. 10.1109/cvpr.2018.00474M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4510–4520, Mar. 2019. 10.1109/cvpr.2017.195F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258, Jul. 2017. 10.1109/iccv.2017.593N. Bodla, B. Singh, R. Chellappa, and L. S. Davis, “Soft-NMS–improving object detection with one line of code,” In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5561–5569, Aug. 2017.