Tongue size and shape classification fusing segmentation features for traditional Chinese medicine diagnosis

The size and shape of the tongue can reflect different pathological changes of the human body in Traditional Chinese Medicine (TCM). Recently, convolutional neural networks (CNNs) have been widely used for the classification of the color, thickness and teeth marks of the tongue. However, only a few works have been devoted to tongue size and shape classification, which is also key evidence for tongue diagnosis. In this work, we proposed an efficient deep network, TSC-WNet, for tongue size and shape classification. The proposed TSC-WNet consists of two subnetworks, i.e. TSC-Net and TSC-UNet. While TSC-Net is a straightforward and effective classification backbone, TSC-UNet is built for tongue segmentation and offers complementary beneficial features to enhance the classification performance of the networks. Our classification backbone requires fewer parameters than classic CNNs like AlexNet, VGG16 and ResNet18, and achieves better classification performance. Employing TSC-Net as the encoder, the TSC-UNet was used to provide the segmentation information for helping better tongue size and shape classification. Two different datasets, i.e. FJTCM/SZU and BioHit, were employed for performance evaluation. The experimental results show that TSC-Net achieves at least 2% higher accuracy and F₁ score than the baseline networks. Ablation studies show that the fusion of TSC-Net and TSC-UNet at both input and feature levels can further improve the accuracy and F₁ score by about 2%. The code is available at: https://github.com/Yating-Huang/TSC-WNet.