Accurate prediction of drug-induced adverse drug reactions (ADRs) is crucial for drug safety evaluation, as it directly impacts public health and safety. While various models have shown promising results in predicting ADRs, their accuracy still needs improvement. Additionally, many existing models often lack interpretability when linking molecular structures to specific ADRs and frequently rely on manually selected molecular fingerprints, which can introduce bias. To address these challenges, we propose ToxBERT, an efficient transformer encoder model that leverages attention and masking mechanisms for simplified molecular input line entry system (SMILES) representations. Our results demonstrate that ToxBERT achieved area under the receiver operating characteristic curve (AUROC) scores of 0.839, 0.759, and 0.664 for predicting drug-induced QT prolongation (DIQT), rhabdomyolysis, and liver injury, respectively, outperforming previous studies. Furthermore, ToxBERT can identify drug substructures that are closely associated with specific ADRs. These findings indicate that ToxBERT is not only a valuable tool for understanding the mechanisms underlying specific drug-induced ADRs but also for mitigating potential ADRs in the drug discovery pipeline.