1.2023 Survey on User Experience of Artificial Intelligence Software in Radiology by the Korean Society of Radiology
Eui Jin HWANG ; Ji Eun PARK ; Kyoung Doo SONG ; Dong Hyun YANG ; Kyung Won KIM ; June-Goo LEE ; Jung Hyun YOON ; Kyunghwa HAN ; Dong Hyun KIM ; Hwiyoung KIM ; Chang Min PARK ;
Korean Journal of Radiology 2024;25(7):613-622
Objective:
In Korea, radiology has been positioned towards the early adoption of artificial intelligence-based software as medical devices (AI-SaMDs); however, little is known about the current usage, implementation, and future needs of AI-SaMDs.We surveyed the current trends and expectations for AI-SaMDs among members of the Korean Society of Radiology (KSR).
Materials and Methods:
An anonymous and voluntary online survey was open to all KSR members between April 17 and May 15, 2023. The survey was focused on the experiences of using AI-SaMDs, patterns of usage, levels of satisfaction, and expectations regarding the use of AI-SaMDs, including the roles of the industry, government, and KSR regarding the clinical use of AI-SaMDs.
Results:
Among the 370 respondents (response rate: 7.7% [370/4792]; 340 board-certified radiologists; 210 from academic institutions), 60.3% (223/370) had experience using AI-SaMDs. The two most common use-case of AI-SaMDs among the respondents were lesion detection (82.1%, 183/223), lesion diagnosis/classification (55.2%, 123/223), with the target imaging modalities being plain radiography (62.3%, 139/223), CT (42.6%, 95/223), mammography (29.1%, 65/223), and MRI (28.7%, 64/223). Most users were satisfied with AI-SaMDs (67.6% [115/170, for improvement of patient management] to 85.1% [189/222, for performance]). Regarding the expansion of clinical applications, most respondents expressed a preference for AI-SaMDs to assist in detection/diagnosis (77.0%, 285/370) and to perform automated measurement/quantification (63.5%, 235/370). Most respondents indicated that future development of AI-SaMDs should focus on improving practice efficiency (81.9%, 303/370) and quality (71.4%, 264/370). Overall, 91.9% of the respondents (340/370) agreed that there is a need for education or guidelines driven by the KSR regarding the use of AI-SaMDs.
Conclusion
The penetration rate of AI-SaMDs in clinical practice and the corresponding satisfaction levels were high among members of the KSR. Most AI-SaMDs have been used for lesion detection, diagnosis, and classification. Most respondents requested KSR-driven education or guidelines on the use of AI-SaMDs.
2.Effects of Expert-Determined Reference Standards in Evaluating the Diagnostic Performance of a Deep Learning Model: A Malignant Lung Nodule Detection Task on Chest Radiographs
Jung Eun HUH ; Jong Hyuk LEE ; Eui Jin HWANG ; Chang Min PARK
Korean Journal of Radiology 2023;24(2):155-165
Objective:
Little is known about the effects of using different expert-determined reference standards when evaluating the performance of deep learning-based automatic detection (DLAD) models and their added value to radiologists. We assessed the concordance of expert-determined standards with a clinical gold standard (herein, pathological confirmation) and the effects of different expert-determined reference standards on the estimates of radiologists’ diagnostic performance to detect malignant pulmonary nodules on chest radiographs with and without the assistance of a DLAD model.
Materials and Methods:
This study included chest radiographs from 50 patients with pathologically proven lung cancer and 50 controls. Five expert-determined standards were constructed using the interpretations of 10 experts: individual judgment by the most experienced expert, majority vote, consensus judgments of two and three experts, and a latent class analysis (LCA) model. In separate reader tests, additional 10 radiologists independently interpreted the radiographs and then assisted with the DLAD model. Their diagnostic performance was estimated using the clinical gold standard and various expertdetermined standards as the reference standard, and the results were compared using the t test with Bonferroni correction.
Results:
The LCA model (sensitivity, 72.6%; specificity, 100%) was most similar to the clinical gold standard. When expertdetermined standards were used, the sensitivities of radiologists and DLAD model alone were overestimated, and their specificities were underestimated (all p-values < 0.05). DLAD assistance diminished the overestimation of sensitivity but exaggerated the underestimation of specificity (all p-values < 0.001). The DLAD model improved sensitivity and specificity to a greater extent when using the clinical gold standard than when using the expert-determined standards (all p-values < 0.001), except for sensitivity with the LCA model (p = 0.094).
Conclusion
The LCA model was most similar to the clinical gold standard for malignant pulmonary nodule detection on chest radiographs. Expert-determined standards caused bias in measuring the diagnostic performance of the artificial intelligence model.
3.Conventional Versus Artificial Intelligence-Assisted Interpretation of Chest Radiographs in Patients With Acute Respiratory Symptoms in Emergency Department: A Pragmatic Randomized Clinical Trial
Eui Jin HWANG ; Jin Mo GOO ; Ju Gang NAM ; Chang Min PARK ; Ki Jeong HONG ; Ki Hong KIM
Korean Journal of Radiology 2023;24(3):259-270
Objective:
It is unknown whether artificial intelligence-based computer-aided detection (AI-CAD) can enhance the accuracy of chest radiograph (CR) interpretation in real-world clinical practice. We aimed to compare the accuracy of CR interpretation assisted by AI-CAD to that of conventional interpretation in patients who presented to the emergency department (ED) with acute respiratory symptoms using a pragmatic randomized controlled trial.
Materials and Methods:
Patients who underwent CRs for acute respiratory symptoms at the ED of a tertiary referral institution were randomly assigned to intervention group (with assistance from an AI-CAD for CR interpretation) or control group (without AI assistance). Using a commercial AI-CAD system (Lunit INSIGHT CXR, version 2.0.2.0; Lunit Inc.). Other clinical practices were consistent with standard procedures. Sensitivity and false-positive rates of CR interpretation by duty trainee radiologists for identifying acute thoracic diseases were the primary and secondary outcomes, respectively. The reference standards for acute thoracic disease were established based on a review of the patient’s medical record at least 30 days after the ED visit.
Results:
We randomly assigned 3576 participants to either the intervention group (1761 participants; mean age ± standard deviation, 65 ± 17 years; 978 males; acute thoracic disease in 472 participants) or the control group (1815 participants; 64 ± 17 years; 988 males; acute thoracic disease in 491 participants). The sensitivity (67.2% [317/472] in the intervention group vs. 66.0% [324/491] in the control group; odds ratio, 1.02 [95% confidence interval, 0.70–1.49]; P = 0.917) and false-positive rate (19.3% [249/1289] vs. 18.5% [245/1324]; odds ratio, 1.00 [95% confidence interval, 0.79–1.26]; P = 0.985) of CR interpretation by duty radiologists were not associated with the use of AI-CAD.
Conclusion
AI-CAD did not improve the sensitivity and false-positive rate of CR interpretation for diagnosing acute thoracic disease in patients with acute respiratory symptoms who presented to the ED.
4.Effects of Implementing Artificial Intelligence-Based Computer-Aided Detection for Chest Radiographs in Daily Practice on the Rate of Referral to Chest Computed Tomography in Pulmonology Outpatient Clinic
Wonju HONG ; Eui Jin HWANG ; Chang Min PARK ; Jin Mo GOO
Korean Journal of Radiology 2023;24(9):890-902
Objective:
The clinical impact of artificial intelligence-based computer-aided detection (AI-CAD) beyond diagnostic accuracy remains uncertain. We aimed to investigate the influence of the clinical implementation of AI-CAD for chest radiograph (CR) interpretation in daily practice on the rate of referral for chest computed tomography (CT).
Materials and Methods:
AI-CAD was implemented in clinical practice at the Seoul National University Hospital. CRs obtained from patients who visited the pulmonology outpatient clinics before (January–December 2019) and after (January–December 2020) implementation were included in this study. After implementation, the referring pulmonologist requested CRs with or without AI-CAD analysis. We conducted multivariable logistic regression analyses to evaluate the associations between using AI-CAD and the following study outcomes: the rate of chest CT referral, defined as request and actual acquisition of chest CT within 30 days after CR acquisition, and the CT referral rates separately for subsequent positive and negative CT results.Multivariable analyses included various covariates such as patient age and sex, time of CR acquisition (before versus after AICAD implementation), referring pulmonologist, nature of the CR examination (baseline versus follow-up examination), and radiology reports presence at the time of the pulmonology visit.
Results:
A total of 28546 CRs from 14565 patients (mean age: 67 years; 7130 males) and 25888 CRs from 12929 patients (mean age: 67 years; 6435 males) before and after AI-CAD implementation were included. The use of AI-CAD was independently associated with increased chest CT referrals (odds ratio [OR], 1.33; P = 0.008) and referrals with subsequent negative chest CT results (OR, 1.46; P = 0.005). Meanwhile, referrals with positive chest CT results were not significantly associated with AI-CAD use (OR, 1.08; P = 0.647).
Conclusion
The use of AI-CAD for CR interpretation in pulmonology outpatients was independently associated with an increased frequency of overall referrals for chest CT scans and referrals with subsequent negative results.
5.2020 Clinical Practice Guideline for Percutaneous Transthoracic Needle Biopsy of Pulmonary Lesions: A Consensus Statement and Recommendations of the Korean Society of Thoracic Radiology
Soon Ho YOON ; Sang Min LEE ; Chul Hwan PARK ; Jong Hyuk LEE ; Hyungjin KIM ; Kum Ju CHAE ; Kwang Nam JIN ; Kyung Hee LEE ; Jung Im KIM ; Jung Hee HONG ; Eui Jin HWANG ; Heekyung KIM ; Young Joo SUH ; Samina PARK ; Young Sik PARK ; Dong-Wan KIM ; Miyoung CHOI ; Chang Min PARK
Korean Journal of Radiology 2021;22(2):263-280
Percutaneous transthoracic needle biopsy (PTNB) is one of the essential diagnostic procedures for pulmonary lesions. Its role is increasing in the era of CT screening for lung cancer and precision medicine. The Korean Society of Thoracic Radiology developed the first evidence-based clinical guideline for PTNB in Korea by adapting pre-existing guidelines. The guideline provides 39 recommendations for the following four main domains of 12 key questions: the indications for PTNB, pre-procedural evaluation, procedural technique of PTNB and its accuracy, and management of post-biopsy complications. We hope that these recommendations can improve the diagnostic accuracy and safety of PTNB in clinical practice and promote standardization of the procedure nationwide.
6.Clinical Implementation of Deep Learning in ThoracicRadiology: Potential Applications and Challenges
Eui Jin HWANG ; Chang Min PARK
Korean Journal of Radiology 2020;21(5):511-525
Chest X-ray radiography and computed tomography, the two mainstay modalities in thoracic radiology, are under activeinvestigation with deep learning technology, which has shown promising performance in various tasks, including detection,classification, segmentation, and image synthesis, outperforming conventional methods and suggesting its potential forclinical implementation. However, the implementation of deep learning in daily clinical practice is in its infancy and facingseveral challenges, such as its limited ability to explain the output results, uncertain benefits regarding patient outcomes, andincomplete integration in daily workflow. In this review article, we will introduce the potential clinical applications of deeplearning technology in thoracic radiology and discuss several challenges for its implementation in daily clinical practice.
7.Implementation of a Deep Learning-Based ComputerAided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19
Eui Jin HWANG ; Hyungjin KIM ; Soon Ho YOON ; Jin Mo GOO ; Chang Min PARK
Korean Journal of Radiology 2020;21(10):1150-1160
Objective:
To describe the experience of implementing a deep learning-based computer-aided detection (CAD) system for the interpretation of chest X-ray radiographs (CXR) of suspected coronavirus disease (COVID-19) patients and investigate the diagnostic performance of CXR interpretation with CAD assistance.
Materials and Methods:
In this single-center retrospective study, initial CXR of patients with suspected or confirmed COVID-19 were investigated. A commercialized deep learning-based CAD system that can identify various abnormalities on CXR was implemented for the interpretation of CXR in daily practice. The diagnostic performance of radiologists with CAD assistance were evaluated based on two different reference standards: 1) real-time reverse transcriptase-polymerase chain reaction (rRT-PCR) results for COVID-19 and 2) pulmonary abnormality suggesting pneumonia on chest CT. The turnaround times (TATs) of radiology reports for CXR and rRT-PCR results were also evaluated.
Results:
Among 332 patients (male:female, 173:159; mean age, 57 years) with available rRT-PCR results, 16 patients (4.8%) were diagnosed with COVID-19. Using CXR, radiologists with CAD assistance identified rRT-PCR positive COVID-19 patients with sensitivity and specificity of 68.8% and 66.7%, respectively. Among 119 patients (male:female, 75:44; mean age, 69 years) with available chest CTs, radiologists assisted by CAD reported pneumonia on CXR with a sensitivity of 81.5% and a specificity of 72.3%. The TATs of CXR reports were significantly shorter than those of rRT-PCR results (median 51 vs. 507 minutes; p < 0.001).
Conclusion
Radiologists with CAD assistance could identify patients with rRT-PCR-positive COVID-19 or pneumonia on CXR with a reasonably acceptable performance. In patients suspected with COVID-19, CXR had much faster TATs than rRT-PCRs.
8.Calcaneal Osteomyelitis Presenting as a Paradoxical Reaction during Treatment of Multidrug-Resistant Tuberculosis
Yong Hyun HAN ; Chang Hwa LEE ; Min Joon BAE ; Kihun HWANG
Clinical Pain 2019;18(2):102-106
Tuberculosis in the foot progresses gradually; thus, diagnosis is usually delayed, and early treatment is rarely provided. If osteomyelitis occurs due to delayed diagnosis and treatment, surgical treatment should be considered. We report the case of a 46-year-old man with osteomyelitis of the calcaneus who was diagnosed with multidrug-resistant pulmonary tuberculosis and he was treated with anti-tuberculosis drugs. Bilateral adrenal masses, abscess of both testes and a small wound in the left plantar heel were observed. Both adrenal masses and abscess were regarded as paradoxical reaction of anti-tuberculosis treatment. After 1 month, he developed a pain in the left plantar heel that was compatible with calcaneal osteomyelitis in radiological features. He underwent right orchiectomy for right scrotal abscess aggravation and surgical treatment for left calcaneal osteomyelitis. Mycobacterium tuberculosis was confirmed by polymerase chain reaction. The patient was immobilized by cast for 8 weeks and the heel pain gradually improved.
Abscess
;
Calcaneus
;
Delayed Diagnosis
;
Diagnosis
;
Foot
;
Heel
;
Humans
;
Middle Aged
;
Mycobacterium tuberculosis
;
Orchiectomy
;
Osteomyelitis
;
Polymerase Chain Reaction
;
Testis
;
Tuberculosis
;
Tuberculosis, Multidrug-Resistant
;
Tuberculosis, Pulmonary
;
Wounds and Injuries
9.Effect of Computerized Neuropsychologic Test in Subacute Post-Stroke Patient With Cognitive Impairment.
Chang Hwa LEE ; Won Sik MOON ; Yong Hyun HAN ; Po Sung JUN ; Gi Hun HWANG ; Ho Joong JUNG
Kosin Medical Journal 2018;33(1):51-63
OBJECTIVES: To investigate the effects of Computerized Neuropsychologic Test (CNT) on cognitive function and daily life performance in subacute post-stroke patients with cognitive impairment. METHODS: Korean Mini-Mentals State Examination (K-MMSE), Korean version of Modified Barthel Index (K-MBI) were investigated in 125 subacute post-stroke patients with cognitive impairment. We analyzed K-MMSE and K-MBI which were conducted 63 patients who had received CNT and 62 patient who had not received CNT from baseline to 8 weeks follow-up. In the experimental group, initial K-MMSE and K-MBI were conducted 13.3 ± 6.8 weeks after the onset of stroke and their age was 63.4 ± 13.3. In the control group, initial K-MMSE and K-MBI were conducted 13.2 ± 7.7 weeks after the onset of stroke and their age was 65.1 ± 11.6. RESULTS: The 8 weeks follow-up total K-MMSE score and total K-MBI score of experimental group were significantly higher than control group (P < 0.05). In K-MMSE subsection, orientation, judgement, recall, language & visual reconstruction were significantly higher in experimental group than control group (P < 0.05). In K-MBI subsection, personal hygiene, bathing self, toilet, dressing, ambulation, chair/bed transfer were significantly higher in experimental group than control group (P < 0.05). The change of total K-MMSE score of experimental group was significantly correlated with change of total K-MBI score (P < 0.05), but control group was not (P > 0.05). In K-MMSE subsection, change of orientation, registration, language and visual reconstruction were correlated with total K-MBI s core after CNT. Especially, the experimental group, total K-MBI score of the left hemisphere damage group was significantly higher than the right hemisphere damage group (P < 0.05). CONCLUSIONS: This study shows that CNT is effective on subacute post-stroke patients with cognitive impairment. Improvement of cognitive function can expect a positive outcome on daily life performance, in particular, it can be expected to improve the prognosis of patients with stroke, the left hemisphere lesions.
Bandages
;
Baths
;
Cognition
;
Cognition Disorders*
;
Cognitive Therapy
;
Follow-Up Studies
;
Humans
;
Hygiene
;
Neuropsychological Tests*
;
Prognosis
;
Stroke
;
Walking
10.Open Bronchus Sign on CT: A Risk Factor for Hemoptysis after Percutaneous Transthoracic Biopsy.
Hyungjin KIM ; Chang Min PARK ; Soon Ho YOON ; Eui Jin HWANG ; Jong Hyuk LEE ; Su Yeon AHN ; Jin Mo GOO
Korean Journal of Radiology 2018;19(5):880-887
OBJECTIVE: We hypothesized that open bronchi within target pulmonary lesions are associated with percutaneous transthoracic needle biopsy (PTNB)-related hemoptysis. We sought to analyze and compare patient characteristics and target features as well as biopsy-related factors between patients with and without PTNB-related hemoptysis. MATERIALS AND METHODS: We retrospectively analyzed 1484 patients (870 males and 614 females; median age, 66 years) who had undergone 1569 cone-beam CT (CBCT)-guided PTNBs. Patient characteristics (sex, age, and pathologic diagnosis), nodule features (nodule type, size, location, and presence of an open bronchus in target nodules), and biopsy-related factors (biopsy needle size, pleura-to-target distance, blood test results, open bronchus unavoidability [OBU] index, etc.) were investigated. OBU index, which was assessed using the pre-procedural CBCT, was a subjective scoring system for the probability of needle penetration into the open bronchus. Univariate analysis and subsequent multivariate logistic regression analysis were conducted to reveal the independent risk factors for PTNB-related hemoptysis. For a subgroup of nodules with open bronchi, a trend analysis between the occurrence of hemoptysis and the OBU index was performed. RESULTS: The independent risk factors for hemoptysis were sex (female; odds ratio [OR], 1.918; p < 0.001), nodule size (OR, 0.837; p < 0.001), open bronchus (OR, 2.101; p < 0.001), and pleura-to-target distance (OR, 1.135; p = 0.003). For the target nodules with open bronchi, a significant trend between hemoptysis and OBU index (p < 0.001) was observed. CONCLUSION: An open bronchus in a biopsy target is an independent predictor of hemoptysis, and careful imaging review may potentially reduce PTNB-related hemoptysis.
Biopsy*
;
Biopsy, Needle
;
Bronchi*
;
Cone-Beam Computed Tomography
;
Female
;
Hematologic Tests
;
Hemoptysis*
;
Humans
;
Image-Guided Biopsy
;
Logistic Models
;
Lung Neoplasms
;
Male
;
Needles
;
Odds Ratio
;
Retrospective Studies
;
Risk Factors*

Result Analysis
Print
Save
E-mail