Purpose: The present study aimed to develop 2 deep learning (DL) systems incorporating detection functions for the diagnosis of carotid artery calcifications (CACs) on panoramic radiographs and to compare their diagnostic performances using CAC-based, side-based, and patient-based evaluations. Materials and Methods: Panoramic radiographs from 290 patients with CACs and 290 control patients without CACs were used to develop 2 detection models: one designed to detect individual CACs across the entire radiograph (System 1) and another designed to detect CACs within the limited bilateral cervical areas (System 2). CAC-based performance was evaluated using recall, precision, and F1-score. Side-based and patient-based performances were assessed usingsensitivity, specificity, positive predictive value, negative predictive value, accuracy, and the area under the receiveroperating characteristic curve (AUC). Results: For System 1, CAC-based recall, precision, and F1-score were 0.81, 0.68, and 0.74, respectively. For System 2,the corresponding values were 0.90, 0.67, and 0.77. Side-based sensitivity, specificity, and AUC were 0.87, 0.80, and 0.83 for System 1, and 0.93, 0.84, and 0.89 for System 2. Patient-based sensitivity, specificity, and AUC were 0.93, 0.73,and 0.83 for System 1, and 0.95, 0.70, and 0.83 for System 2. Although a relatively large number of false positives were observed in CAC-based assessments, side-based and patient-based performances showed improvement. Conclusion Side-based and patient-based performances were sufficient when calculated on the basis of CAC-basedevaluations for diagnosing CACs on panoramic radiographs. When conducting studies of this type, performance assessments should include side-based and patient-based evaluations in addition to CAC-based analyses.