Objective:To construct a feature visualization system utilizing deep learning for superficial esophageal squamous cell carcinoma (SESCC) under magnifying endoscopy with narrow band imaging (ME-NBI) to predict the infiltration depth of SESCC.Methods:The feature visualization system consisted of four models: two for segmenting the intrapapillary capillary loops (IPCL) area and avascular area (AVA) in ME-NBI images of SESCC lesions (models 1 and 2, respectively), one for obtaining the principal component of color (PCC) in ME-NBI images of SESCC lesions (model 3), and another for automatically predicting the depth of SESCC infiltration based on the features extracted from the first three models (model 4). A total of 2 341 ME-NBI images of SESCC lesions from April 2016 to October 2021 were used to develop the feature visualization system, which was divided into 3 datasets: dataset 1 (1 077 ME-NBI images) was used to train and test models 1-3, dataset 2 (1 069 ME-NBI images) was expanded by 20 times through feature combination to generate 21 380 feature synthetic images to train and test model 4, and dataset 3 (195 ME-NBI images), containing 146 ME-NBI images with lesion invasion depth from the epithelium to the upper 1/3 of the submucosa (EP-SM1), and 49 ME-NBI images with lesion invasion depth from the middle 1/3 to the lower 1/3 of the submucosa (SM2-SM3), was used to validate the diagnostic performance of the feature visualization system in predicting the invasion depth of SESCC (EP-SM1/SM2-SM3). In order to evaluate the superiority of the feature visualization system, the prediction results of dataset 3 of the traditional deep learning system (trained directly with ME-NBI images), single-item feature models (single-item IPCL feature model, single-item AVA feature model and single-item PCC feature model) were compared with the prediction results of the feature visualization system. In order to evaluate the clinical utility of the feature visualization system, 4 expert physicians (with more than 10 years of endoscopic operation, expert physician group) and 5 senior physicians (with more than 5 years of endoscopic operation, senior physician group) were invited to participate in the human-computer competition to diagnose dataset 3, and the results were compared with the feature visualization system.Results:The accuracy, sensitivity and specificity of the feature visualization system in predicting the invasion depth of SESCC (EP-SM1/SM2-SM3) were 83.08% (162/195), 82.88% (121/146) and 83.67% (41/49), respectively. The above indicators were 60.00% (117/195), 52.05% (76/146) and 83.67% (41/49) for the traditional deep learning system, 74.87% (146/195), 75.34% (110/146) and 73.47% (36/49) for the single IPCL feature model, 58.97% (115/195), 60.27% (88/146) and 55.10% (27/49) for single AVA feature model, 71.28% (139/195), 71.23% (104/146) and 71.43% (35/49) for single PCC feature model, respectively. The results were 66.67%, 78.22% and 32.24% in senior physician group, and 72.31%, 85.96% and 31.63% in expert physician group, respectively. The accuracy of the feature visualization system in predicting the invasion depth of SESCC was significantly higher than that of the other 6 groups ( P<0.05). The sensitivity of feature visualization system was slightly higher than that of senior physician group ( χ2=1.59, P=0.21) and single-item IPCL feature model ( χ2=2.51, P=0.11), slightly lower than that of expert physician group ( χ2=0.89, P=0.35), and significantly higher than that of three other groups ( P<0.05). The specificity of the feature visualization system was similar to the traditional deep learning system ( χ2=0.00, P=1.00), slightly higher than that of single-item IPCL feature model ( χ2=1.52, P=0.22) and single-item PCC feature model (χ2=2.11, P=0.15), and significantly higher than that of the single AVA feature model ( χ2=9.42, P<0.01), senior physician group ( χ2=44.71, P<0.01) and expert physician group ( χ2=43.57, P<0.01). Conclusion:The developed deep learning-based feature visualization system using ME-NBI shows excellent diagnostic performance in predicting the infiltration depth of SESCC (EP-SM1/SM2-SM3), surpassing the accuracy levels of experienced endoscopists with over 10 years of experience.

Objective:To develop an artificial intelligence-based system for measuring the size of gastrointestinal lesions under white light endoscopy in real time.Methods:The system consisted of 3 models. Model 1 was used to identify the biopsy forceps and mark the contour of the forceps in continuous pictures of the video. The results of model 1 were submitted to model 2 and classified into open and closed forceps. And model 3 was used to identify the lesions and mark the boundary of lesions in real time. Then the length of the lesions was compared with the contour of the forceps to calculate the size of lesions. Dataset 1 consisted of 4 835 images collected retrospectively from January 1, 2017 to November 30, 2019 in Renmin Hospital of Wuhan University, which were used for model training and validation. Dataset 2 consisted of images collected prospectively from December 1, 2019 to June 4, 2020 at the Endoscopy Center of Renmin Hospital of Wuhan University, which were used to test the ability of the model to segment the boundary of the biopsy forceps and lesions. Dataset 3 consisted of 302 images of 151 simulated lesions, each of which included one image of a larger tilt angle (45° from the vertical line of the lesion) and one image of a smaller tilt angle (10° from the vertical line of the lesion) to test the ability of the model to measure the lesion size with the biopsy forceps in different states. Dataset 4 was a video test set, which consisted of prospectively collected videos taken from the Endoscopy Center of Renmin Hospital of Wuhan University from August 5, 2019 to September 4, 2020. The accuracy of model 1 in identifying the presence or absence of biopsy forceps, model 2 in classifying the status of biopsy forceps (open or closed) and model 3 in identifying the presence or absence of lesions were observed with the results of endoscopist review or endoscopic surgery pathology as the gold standard. Intersection over union (IoU) was used to evaluate the segmentation effect of biopsy forceps in model 1 and lesion segmentation effect in model 3, and the absolute error and relative error were used to evaluate the ability of the system to measure lesion size.Results:(1)A total of 1 252 images were included in dataset 2, including 821 images of forceps (401 images of open forceps and 420 images of closed forceps), 431 images of non-forceps, 640 images of lesions and 612 images of non-lesions. Model 1 judged 433 images of non-forceps (430 images were accurate) and 819 images of forceps (818 images were accurate), and the accuracy was 99.68% (1 248/1 252). Based on the data of 818 images of forceps to evaluate the accuracy of model 1 on judging the segmentation effect of biopsy forceps lobe, the mean IoU was 0.91 (95% CI: 0.90-0.92). The classification accuracy of model 2 was evaluated by using 818 forceps pictures accurately judged by model 1. Model 2 judged 384 open forceps pictures (382 accurate) and 434 closed forceps pictures (416 accurate), and the classification accuracy of model 2 was 97.56% (798/818). Model 3 judged 654 images containing lesions (626 images were accurate) and 598 images of non-lesions (584 images were accurate), and the accuracy was 96.65% (1 210/1 252). Based on 626 images of lesions accurately judged by model 3, the mean IoU was 0.86 (95% CI: 0.85-0.87). (2) In dataset 3, the mean absolute error of systematic lesion size measurement was 0.17 mm (95% CI: 0.08-0.28 mm) and the mean relative error was 3.77% (95% CI: 0.00%-10.85%) when the tilt angle of biopsy forceps was small. The mean absolute error of systematic lesion size measurement was 0.17 mm (95% CI: 0.09-0.26 mm) and the mean relative error was 4.02% (95% CI: 2.90%-5.14%) when the biopsy forceps was tilted at a large angle. (3) In dataset 4, a total of 780 images of 59 endoscopic examination videos of 59 patients were included. The mean absolute error of systematic lesion size measurement was 0.24 mm (95% CI: 0.00-0.67 mm), and the mean relative error was 9.74% (95% CI: 0.00%-29.83%). Conclusion:The system could measure the size of endoscopic gastrointestinal lesions accurately and may improve the accuracy of endoscopists.