1.Comparison of Automatic Evaluation Methods for Pattern Hallucinations of Large Language Models in Traditional Chinese Medicine Syndrome Differentiation
Qinwei WU ; Yuzhu GAO ; Xingyue GOU ; Junyu YAO ; Chuangan ZHOU ; Zhengchun XUE ; Zhirong XU ; Xinlin CHEN ; Dong CAO
Journal of Traditional Chinese Medicine 2026;67(17):1845-1852
ObjectiveTo compare the performance of different automatic evaluation methods for detecting pattern hallucinations generated by large language models (LLMs) in traditional Chinese medicine (TCM) syndrome differentiation and to identify a strategy with higher overall discriminative performance against expert manual judgment. MethodsA standardized TCM syndrome-differentiation dataset containing 598 cases was constructed from case reports published in Chinese core journals indexed by Peking University Core Journals or Chinese Science Citation Database. Each of the 598 cases was submitted to five Chinese LLMs, with each model generating one syndrome-pattern output per case, yielding 2990 outputs in total. The 598 outputs generated by Qwen3-Next-80B-A3B-Thinking were independently annotated for syndrome-pattern hallucinations by two licensed TCM physicians with intermediate or higher professional titles. Disagreements were adjudicated by a third licensed TCM physician with a senior associate professional title and more than 10 years of clinical experience. The resulting consensus annotations served as the reference labels for training and validating the automated evaluation method. Based on these reference labels, three categories of automated evaluation methods were developed and compared following a progressive strategy from text-level semantic similarity, to syndrome-element structural consistency, and finally multi-feature fusion. Method A employed sentence-embedding models based on semantic similarity between pattern texts. Method B adopted a rule-based threshold method using weighted Jaccard coefficients of disease-location and disease-nature syndrome elements. Method C utilized a machine-learning classifier integrating syndrome-element matching scores, missing and redundant syndrome-element counts, and multiple semantic-similarity features. Five-fold cross-validation was used to evaluate the discriminative performance of each method against manual judgment. The optimal method was then applied to uniformly assess the pattern hallucination rates of five LLMs. ResultsIn method A, text2vec-large-chinese showed best overall performance, with an area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (PR-AUC), F1-score, sensitivity, and specificity of 0.849, 0.884, 0.794, 0.801, and 0.691, respectively. In method B, the corresponding metrics of the rule-based syndrome-element Jaccard threshold method were 0.848, 0.851, 0.817, 0.845, and 0.684, respectively; those of the LLM-based syndrome-element Jaccard method were 0.872, 0.906, 0.795, 0.764, and 0.780, respectively. In method C, the multilayer perceptron (MLP) classifier showed the best overall performance, with AUROC, PR-AUC, F1-score, sensitivity, and specificity of 0.933, 0.951, 0.871, 0.857, and 0.846, respectively. Under the unified evaluation framework combining DeepSeek-R1 syndrome-element extraction and MLP classifier, the estimated pattern hallucination rates of the five LLMs ranged from 56.0% to 77.4%. ConclusionAmong the three automatic evaluation methods, Method C integrating syndrome-element discrepancy features and text semantic features using an MLP classifier showed the highest overall discriminative performance against manual judgment, and is more suitable for evaluating syndrome hallucinations in LLM-based TCM syndrome differentiation tasks.
2.Experience of Microsurgical Operation in the Treatment of Supratentorial Hypertensive Intracerebral Hemorrhage
Zhengchun SUN ; Xinliang CAO ; Xudong MA
Chinese Journal of Minimally Invasive Surgery 2014;(12):1154-1156
Objective To explore the curative effects of microsurgical clearance for supratentorial hypertensive intracerebral hemorrhage. Methods A retrospective analysis of clinical data of 30 cases of supratentorial hypertensive intracerebral hemorrhage in our hospital from January 2010 to January 2013 was carried out. According to the head CT images, the position of incision was determined. Under orotracheal intubation general anesthesia, a bone flap craniotomy was performed. A cortex fistula 2-3 cm in length was made. Then by using an automatic retraction device, the hematoma was revealed from the shallower to the deeper, and microscopic clearance of hematoma was completed. Results All the 30 cases of operation were successful. Re-examination of head CT scans on the first postoperative day showed hematoma was completely removed in 14 cases, was 90%cleared in 11 cases, and 80%in 5 cases. Four patients died within 4 weeks after operation (13.3%):2 patients died of cerebral hernia caused by brainstem function failure, 1 patient died of lung infection, and 1 patient died of multiple organ failure. The postoperative survival rate was 86.7%.The ADL grade assessment 3 months after operation showed gradeⅠin 7 cases, grade Ⅱin 9 cases, grade Ⅲin 6 cases, grade Ⅳ in 3 cases, and grade Ⅴin 1 case. Conclusion Microscopic hematoma clearance and open haemostatic intervention has advantages of little brain tissue damage, good haemostatic results, and satisfactory efficacy.
3.Platelet-rich plasma trigger point injection treatment for chronic achilles insertion tendonitis
Guoyou ZOU ; Weitao JIA ; Minqian ZHENG ; Xiaozu XU ; Zhengchun CAO ; Jun YIN ; Ya WU
Chinese Journal of General Practitioners 2013;12(8):657-659
Retrospective analysis was performed for the clinical data of 15 chronic insertion achilles tendinitis patients undergoing platelet-rich plasma (PRP) trigger point injection.The scores of Validated Victorian Institute of Sports Assessment-Achilles (VAS-A) and foot function index (FFI) improved greatly versus pre-treatment (all P < 0.05).Tendon insertion structure inflammation decreased significantly on magnetic resonance imaging.At the last follow-up,all patients recovered normal gait and daily activity.The trigger point injection of PRP is efficacious for chronic insertion achilles tendinopathy.

Result Analysis
Print
Save
E-mail