Mapping Drug Terms via Integration of a Retrieval-Augmented Generation Algorithm with a Large Language Model
10.4258/hir.2024.30.4.355
- Author:
Eizen KIMURA
1
;
Yukinobu KAWAKAMI
;
Shingo INOUE
;
Ai OKAJIMA
Author Information
1. Department of Medical Informatics, Medical School of Ehime University, Toon, Ehime, Japan
- Publication Type:Original Article
- From:Healthcare Informatics Research
2024;30(4):355-363
- CountryRepublic of Korea
- Language:English
-
Abstract:
Objectives:This study evaluated the efficacy of integrating a retrieval-augmented generation (RAG) model and a large language model (LLM) to improve the accuracy of drug name mapping across international vocabularies.
Methods:Drug ingredient names were translated into English using the Japanese Accepted Names for Pharmaceuticals. Drug concepts were extracted from the standard vocabulary of OHDSI, and the accuracy of mappings between translated terms and RxNorm was assessed by vector similarity, using the BioBERT-generated embedded vectors as the baseline. Subsequently, we developed LLMs with RAG that distinguished the final candidates from the baseline. We assessed the efficacy of the LLM with RAG in candidate selection by comparing it with conventional methods based on vector similarity.
Results:The evaluation metrics demonstrated the superior performance of the combined LLM + RAG over traditional vector similarity methods. Notably, the hit rates of the Mixtral 8x7b and GPT-3.5 models exceeded 90%, significantly outperforming the baseline rate of 64% across stratified groups of PO drugs, injections, and all interventions. Furthermore, the r-precision metric, which measures the alignment between model judgment and human evaluation, revealed a notable improvement in LLM performance, ranging from 41% to 50% compared to the baseline of 23%.
Conclusions:Integrating an RAG and an LLM outperformed conventional string comparison and embedding vector similarity techniques, offering a more refined approach to global drug information mapping.
