1.Comparison of Automatic Evaluation Methods for Pattern Hallucinations of Large Language Models in Traditional Chinese Medicine Syndrome Differentiation
Qinwei WU ; Yuzhu GAO ; Xingyue GOU ; Junyu YAO ; Chuangan ZHOU ; Zhengchun XUE ; Zhirong XU ; Xinlin CHEN ; Dong CAO
Journal of Traditional Chinese Medicine 2026;67(17):1845-1852
ObjectiveTo compare the performance of different automatic evaluation methods for detecting pattern hallucinations generated by large language models (LLMs) in traditional Chinese medicine (TCM) syndrome differentiation and to identify a strategy with higher overall discriminative performance against expert manual judgment. MethodsA standardized TCM syndrome-differentiation dataset containing 598 cases was constructed from case reports published in Chinese core journals indexed by Peking University Core Journals or Chinese Science Citation Database. Each of the 598 cases was submitted to five Chinese LLMs, with each model generating one syndrome-pattern output per case, yielding 2990 outputs in total. The 598 outputs generated by Qwen3-Next-80B-A3B-Thinking were independently annotated for syndrome-pattern hallucinations by two licensed TCM physicians with intermediate or higher professional titles. Disagreements were adjudicated by a third licensed TCM physician with a senior associate professional title and more than 10 years of clinical experience. The resulting consensus annotations served as the reference labels for training and validating the automated evaluation method. Based on these reference labels, three categories of automated evaluation methods were developed and compared following a progressive strategy from text-level semantic similarity, to syndrome-element structural consistency, and finally multi-feature fusion. Method A employed sentence-embedding models based on semantic similarity between pattern texts. Method B adopted a rule-based threshold method using weighted Jaccard coefficients of disease-location and disease-nature syndrome elements. Method C utilized a machine-learning classifier integrating syndrome-element matching scores, missing and redundant syndrome-element counts, and multiple semantic-similarity features. Five-fold cross-validation was used to evaluate the discriminative performance of each method against manual judgment. The optimal method was then applied to uniformly assess the pattern hallucination rates of five LLMs. ResultsIn method A, text2vec-large-chinese showed best overall performance, with an area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (PR-AUC), F1-score, sensitivity, and specificity of 0.849, 0.884, 0.794, 0.801, and 0.691, respectively. In method B, the corresponding metrics of the rule-based syndrome-element Jaccard threshold method were 0.848, 0.851, 0.817, 0.845, and 0.684, respectively; those of the LLM-based syndrome-element Jaccard method were 0.872, 0.906, 0.795, 0.764, and 0.780, respectively. In method C, the multilayer perceptron (MLP) classifier showed the best overall performance, with AUROC, PR-AUC, F1-score, sensitivity, and specificity of 0.933, 0.951, 0.871, 0.857, and 0.846, respectively. Under the unified evaluation framework combining DeepSeek-R1 syndrome-element extraction and MLP classifier, the estimated pattern hallucination rates of the five LLMs ranged from 56.0% to 77.4%. ConclusionAmong the three automatic evaluation methods, Method C integrating syndrome-element discrepancy features and text semantic features using an MLP classifier showed the highest overall discriminative performance against manual judgment, and is more suitable for evaluating syndrome hallucinations in LLM-based TCM syndrome differentiation tasks.
2.Application of left echography in the diagnosis of false ventricular aneurysm and mural thrombus
Yanling XUE ; Xiaojing MA ; Shurui XIE ; Juan XIA ; Yafeng HE ; Zhengchun YU
Journal of Chinese Physician 2024;26(10):1460-1463
Objective:To evaluate the value of left echography (LVO) in the diagnosis of false ventricular aneurysm complicated with mural thrombus.Methods:The clinical data of 10 patients with suspected pseudoventricular aneurysm examined by thoracic echocardiography (TTE) in Wuhan Asian Heart Hospital from January 2018 to March 2024 were retrospectively analyzed. All patients underwent LVO examination to further diagnose pseudoventricular tumor and whether it was complicated with mural thrombosis. Computed tomography angiography (CTA) or cardiac magnetic resonance (CMR) examination was used as the gold standard to analyze the diagnostic value of LVO in the diagnosis of pseudoventricular tumor.Results:Among the 10 suspected pseudoventricular tumors examined by TTE, LVO detected 6 cases of left ventricular pseudoaneurysm and 1 case of right ventricular pseudoaneurysm; CTA confirmed that 6 cases of left ventricular pseudoaneurysm detected by LVO were correctly diagnosed, 1 case of right ventricular pseudoaneurysm was misdiagnosed, CMR diagnosed right ventricular diverticula, LVO diagnosis accuracy was 6/7, and 4 cases of thrombi were detected. The detection rate was 4/4. The maximum transverse diameter of the tumor body of the communicating mouth/false ventricular aneurysm was 0.46±0.04. 1 patient underwent coronary artery bypass grafting and resection of false ventricular aneurysm. 1 patient underwent coronary artery interventional stent surgery; 4 routine conservative drug treatment, follow-up observation; One case of right ventricular diverticulum did not require special treatment.Conclusions:LVO contrast agent can clearly show the tumor body and location, measure the tumor entrance and size, and show mural thrombus. It is the first choice for the identification of false ventricular tumor. The diverticula was similar to the image of false ventricular aneurysm, and the sensitivity and specificity of right ventricular wall motion were higher in CMR than in LVO.
3.Prognostic significance of preoperative neutrophil-lymphocyte ratio combined with tu-mor-associated neutrophils in patients with gastric cancer
Xiaopei HUANG ; Jiangbo ZHU ; Wenjun CHANG ; Xiaofang ZHANG ; Tianbao ZHANG ; Zhengchun KANG ; Wei XUE
Chinese Journal of Clinical Oncology 2018;45(2):77-82
Objective:To investigate the prognostic significance of preoperative neutrophil-lymphocyte ratio(NLR)combined with tu-mor-associated neutrophils(TANs)in the stroma of gastric cancer tissues of patients.Methods:One hundred and twenty-six gastric cancer tissue samples from patients enrolled in Changhai hospital from June 2006 to May 2011 were divided into four groups accord-ing to NLRs in preoperative peripheral blood combined with high or low infiltration of neutrophils in gastric cancer tissues.The 5-year survival of the four groups was then compared,and their correlations with clinicopathologic features and prognosis were analyzed.Re-sults:High NLRs in peripheral blood combined with low infiltrating TANs in gastric tissues was associated with lower differential grade (P<0.001)and larger tumor size(P=0.026).Of the four groups,patients with high NLR in peripheral blood combined with low infiltrat-ing TANs in gastric cancer tissues demonstrated the lowest survival rates,whereas those with low NLR and high infiltrating TANs had highest survival rates, and this difference was statistically significant (P<0.05). Univariate and multivariate Cox regression analyses showed that high NLR in peripheral blood combined with low infiltrating TANs in gastric cancer tissues(P<0.05)was an independent factor indicating poor prognosis.Conclusions:NLR in preoperative peripheral blood combined with infiltrating TANs in gastric cancer tissues can be used as a prognostic indicator for patients with gastric cancer,especially high NLR in preoperative peripheral blood com-bined with low infiltrating TANs in gastric cancer tissue indicates poor prognosis.
4.Predicting the radiosensitivity with Raf kinase inhibitor protein in nasopharyngeal carcinoma
Siwei LI ; Zhuokai HE ; Xue BAI ; Zhengchun LIU ; Bo ZHAO ; Meilian LIU
Chinese Journal of Radiological Medicine and Protection 2013;33(4):380-383
Objective To investigate the relationship between RKIP expression and the efficiency of radiotherapy in NPC patients and evaluate the possibility of using RKIP as a predictor of radiosensitivity.Methods A total of 180 patients with NPC in Sun Yat-sen University Cancer Center without evidence of distant metastasis at initial diagnosis were enrolled in this study,who had received intensity-modulated radiotherapy alone.Patients were classified into 2 groups according to criteria below:patients with biopsy proven recurrent diseases occurring at nasopharynx and/or neck within 5 years after radiotherapy were classified as the radioresistant group.The pathological type at relapse was the same as the previous one before treatment.Patients with a minimum follow-up of 5 years after radiotherapy without evidence of recurrence at the original site of the tumor were classified as the radiosensitive group.Patients in the 2 groups were matched according to the factors related with radiosensitivity.RKIP was examined by immunohistochemical staining before radiotherapy.The relationship between RKIP expression and the effect of radiotherapy were analyzed.Results The positive rate of the RKIP expression in the radiosensitive group versus the radioresistant group was 80.0% versus 26.7%.The positive rate (x2 =12.498,P <0.01) and the intensity of the RKIP expression (x2 =51.429,P < 0.01) were significantly different between 2 groups with a negative correlation with radio-resistance to NPC (r =-0.344,-0.535,respectively,P < 0.01).Based on the RKIP expression,the radiosensitivity,specificity,accuracy,positive predictive value,negative predictive value,false positive and false negative were predicted as follows:80.0%,73.3%,77.2%,75.0%,78.6%,26.7%,and 20.0%,respectively.Conclusions RKIP protein shows negative correlation with radioresistance to NPC and could serve as a biomarker in preliminarily screening the intrinsic radiosensitivity of NPC.

Result Analysis
Print
Save
E-mail