1.Comparison of Automatic Evaluation Methods for Pattern Hallucinations of Large Language Models in Traditional Chinese Medicine Syndrome Differentiation
Qinwei WU ; Yuzhu GAO ; Xingyue GOU ; Junyu YAO ; Chuangan ZHOU ; Zhengchun XUE ; Zhirong XU ; Xinlin CHEN ; Dong CAO
Journal of Traditional Chinese Medicine 2026;67(17):1845-1852
ObjectiveTo compare the performance of different automatic evaluation methods for detecting pattern hallucinations generated by large language models (LLMs) in traditional Chinese medicine (TCM) syndrome differentiation and to identify a strategy with higher overall discriminative performance against expert manual judgment. MethodsA standardized TCM syndrome-differentiation dataset containing 598 cases was constructed from case reports published in Chinese core journals indexed by Peking University Core Journals or Chinese Science Citation Database. Each of the 598 cases was submitted to five Chinese LLMs, with each model generating one syndrome-pattern output per case, yielding 2990 outputs in total. The 598 outputs generated by Qwen3-Next-80B-A3B-Thinking were independently annotated for syndrome-pattern hallucinations by two licensed TCM physicians with intermediate or higher professional titles. Disagreements were adjudicated by a third licensed TCM physician with a senior associate professional title and more than 10 years of clinical experience. The resulting consensus annotations served as the reference labels for training and validating the automated evaluation method. Based on these reference labels, three categories of automated evaluation methods were developed and compared following a progressive strategy from text-level semantic similarity, to syndrome-element structural consistency, and finally multi-feature fusion. Method A employed sentence-embedding models based on semantic similarity between pattern texts. Method B adopted a rule-based threshold method using weighted Jaccard coefficients of disease-location and disease-nature syndrome elements. Method C utilized a machine-learning classifier integrating syndrome-element matching scores, missing and redundant syndrome-element counts, and multiple semantic-similarity features. Five-fold cross-validation was used to evaluate the discriminative performance of each method against manual judgment. The optimal method was then applied to uniformly assess the pattern hallucination rates of five LLMs. ResultsIn method A, text2vec-large-chinese showed best overall performance, with an area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (PR-AUC), F1-score, sensitivity, and specificity of 0.849, 0.884, 0.794, 0.801, and 0.691, respectively. In method B, the corresponding metrics of the rule-based syndrome-element Jaccard threshold method were 0.848, 0.851, 0.817, 0.845, and 0.684, respectively; those of the LLM-based syndrome-element Jaccard method were 0.872, 0.906, 0.795, 0.764, and 0.780, respectively. In method C, the multilayer perceptron (MLP) classifier showed the best overall performance, with AUROC, PR-AUC, F1-score, sensitivity, and specificity of 0.933, 0.951, 0.871, 0.857, and 0.846, respectively. Under the unified evaluation framework combining DeepSeek-R1 syndrome-element extraction and MLP classifier, the estimated pattern hallucination rates of the five LLMs ranged from 56.0% to 77.4%. ConclusionAmong the three automatic evaluation methods, Method C integrating syndrome-element discrepancy features and text semantic features using an MLP classifier showed the highest overall discriminative performance against manual judgment, and is more suitable for evaluating syndrome hallucinations in LLM-based TCM syndrome differentiation tasks.
2.Predictive value of the initial MELD score and its derivative scores for early survival rate after liver transplantation in patients with liver failure
Man LAI ; Xin WANG ; Qinwei YAO ; Haixia LIU ; Ying XU ; Li HE ; Guangming LI
Organ Transplantation 2022;13(4):489-
Objective To explore the predictive values of the initial model for end-stage liver disease (MELD) score, MELD combined with serum sodium (MELD-Na) score and MELD combined with serum lactic acid (MELD-Lac) score for early survival rate after liver transplantation in patients with liver failure. Methods Clinical data of 135 recipients undergoing liver transplantation for liver failure were retrospectively analyzed. All patients were divided into the early survival group (
3.Serum exosomal miR-335-5p serves as a diagnostic biomarker of triple-negative breast cancer
Lina YAO ; Guozhong CHEN ; Lei ZHANG ; Qinwei ZHANG
Chinese Journal of Endocrine Surgery 2021;15(5):458-462
Objective:To investigate the expression and diagnostic value of microRNA-335-5p (miR-335-5p) and Human Vascular endothelial growth factor receptor 1 (VEGFR-1, FLT1) in serum exosomal bodies of patients with triple negative breast cancer (TNBC) .Methods:Gene Expression Omnibus database (GEO) analysis was performed and differentially expressed miRNA in breast cancer tissues was screened. From Jan. 2016 to Nov. 2017, the peripheral blood samples of 56 TNBC patients in People’s Hospital Affiliated to Ningbo University were collected, and the exosomes were isolated and identified. FLT1 was selected as the target gene of miR-335-5p by using bioinformatics analysis. Expression levels of miR-335-5p and FLT1 in serum exosome were detected by qRT-PCR. The relationship between miR-335-5p and clinicopathological parameters of TNBC patients was analyzed. The diagnostic value of miR-335-5p was evaluated by receiver operating characteristic (ROC) , and the survival prognosis of miR-335-5p and FLT1 was analyzed by Kaplan Meier survival curve.Results:The expression of miR-335-5p in the serum exosome of TNBC patients was lower than that of the control group, while the expression of FLT1 in the serum exosome was higher than that of the control group ( P<0.05) . The expression of miR-335-5p was related to tissue grade ( χ2=22.02, P<0.000 1) , degree of differentiation ( χ2=20.67, P<0.000 1) and lymph node metastasis ( χ2=4.667, P=0.030 8) in patients with TNBC ( P<0.05) . The area under ROC diagnosed by miR-335-5p was 0.809 8 (95% CI: 0.726 3-0.893 2, P<0.000 1) . Kaplan-Meier survival analysis showed that the overall survival time of patients with low expression of miR-335-5p was significantly shorter than that of patients with high expression of miR-335-5p ( P=0.004 5) . The high expression of its target gene FLT1 was associated with low survival rate ( P=0.048 0) . Conclusion:Serum exosomal miR-335-5p can be used as an important index to predict the prognosis of TNBF and is expected to play a role in diagnosis and treatment of TNBC.
4.Valuation Traceability of Reference Extract of Total Lactones from Ginkgo Leaf for Quantitative Analysis
Qinwei HUANG ; Bei SHI ; Ruwei WANG ; Jianbiao YAO ; Ming ZHU ; Cheng ZHENG
Herald of Medicine 2017;36(5):554-557
Objective To prepare the phmaceutical reference materials of total lactones extract from ginkgo leaf for quantitative analysis.Methods A HPLC determination method was developed to investigate the uniformity and stability of the reference extract of ginkgo leaf total lactones using with bilobalide, ginkgolide A, ginkgolide B and ginkgolide C as the indexes.Three laboratories participated in the collaborative calibration test.Results The four components in reference extract had good uniformity and stability with RSD less than 2.0%;the marked values of the four components had been determined through statistical data analysis which provided by assigned traceability values.The values of bilobalide, ginkgolide A, ginkgolide B, and ginkgolide C were 39.54%, 29.03%, 15.96% and 11.69%, respectively.Conclusion The reference extract of ginkgo leaf total lactones can be prepared for quality control in future quantitative analysis.
6.Translation of Female Sexual Function Index (FSFI) into Chinese and its reliability and validity evaluation
Qingqing LOU ; Qinwei ZHANG ; Huilan YAO ; Yuan TIAN ; Jiajia JI ; Chao LIU
Chinese Journal of Practical Nursing 2013;(10):23-26
Objective To translate the Female Sexual Function Index(FSFI) into Chinese and establish its psychometric properties among ordinary Chinese people and diabetes patients.Methods A two-phase study design was applied.The Chinese version of FSFI was established by translation and back translation,then the reliability and validity of the FSFI were evaluated.Results The content validity coefficient of FSFI was 0.953.The test-retest reliability in each dimension had good correlations (r value was 0.817~0.922),with the highest correlation coefficient in vaginal lubrication dimension (0.922) and arousal dimension the minimum (0.817).In all samples (including diabetes and non diabetes) a reliability coefficients of the Cronbach alpha of each dimension was from 0.760 to 0.874.The Cronbach alpha of each dimension for the diabetic group was from 0.783 to 0.882,and from 0.757 to 0.865 in the non-diabetes group.Pearson correlation of each dimension was very good in total samples,the diabetes group,and the non-diabetes group.Conclusions The psychometric properties of the FSFI demonstrated satisfactory validity and reliability.The Chinese version of FSFI is a reliable and valid measure to evaluate the sexual function in Chinese women.
7.New nursing method for urinary fistula complicated after gynecological malignancy operation
Yufeng CHEN ; Qinwei QIAN ; Yingyan YAO
Chinese Journal of Practical Nursing 2011;27(1):35-36
Objective To study the new nursing method of patients complicated with urinary fistula after gynecological malignancy operation. Methods Nursing measures were provided with sufficient nutritional support and prevention of urinary tract infection by perineal nursing, vaginal plugging by OB sliver,and using endovaginal catheter to 23 patients complicated with urinary fistula after gynecological malignancy operation. Results 18 patients recovered by treatment and nursing, 4 mild cases cured after discharge from hospital and 1 patient died of MODS and multiple metastasis. Conclusions Vaginal plugging by OB sliver, perineal nursing and sufficient nutritional support have positive effect for patients complicated with urinary fistula after gynecological malignancy operation.

Result Analysis
Print
Save
E-mail