Objective: This study aimed to confirm whether the methods for assessing the reported causal relationship between dietary supplement intake and adverse events are reliable in the clinical setting.
Design: The relationships between supplement intake and adverse events were assessed using two algorithms proposed in our previous report, and causal relationships were evaluated.
Methods: Twelve raters with a high probability of handling adverse event information examined 200 records of dialogues with supplement users.  Each rater independently assessed the causal relationship using the two algorithms.  The relationships between supplement intake and adverse events were assessed for all 200 cases.  Variability in the evaluation among raters was analyzed for each occupation and the whole group of raters.  The distributions of evaluation were analyzed, and inter-rater reliability was evaluated using the intraclass correlation coefficient (ICC) and Fleiss’ kappa coefficient.
Results: All events of 200 cases seemed to be slight and within the range of variation in daily life.  Almost all cases were classified into two categories as “Possible” and “Lack of Information” by each rater.  The ICC values for all raters, pharmacists, dieticians, and health care workers were 0.644, 0.573, 0.678, and 0.694, respectively, and the kappa coefficients using the two algorithms were 0.466, 0.426, 0.468, and 0.519 and 0.481, 0.478, 0.465, and 0.517, respectively.  There were moderate levels of agreement based on the kappa coefficients and ICC values.
Conclusion: The two algorithms proposed in our previous report may be reliable in the clinical setting.  Their reliability could be enhanced by establishing a unified method of accumulation and recording adverse events for supplement intake, which should be evaluated by more raters using more cases of adverse events.