Objective:To develop a deep learning model for predicting three-dimensional (3D) voxel-wise dose distributions for intensity-modulated radiotherapy (IMRT).Methods:A total of 110 postoperative rectal cancer cases treated by IMRT were considered in the study, of which 90 cases were randomly selected as the training-validating set and the remaining as the testing set. A 3D deep learning model named 3D U-Res-Net was constructed to predict 3D dose distributions. Three types of 3D matrices from CT images, structure sets and beam configurations were fed into the independent input channel, respectively, and the 3D matrix of IMRT dose distributions was taken as the output to train the 3D model. The obtained 3D model was used to predict new 3D dose distributions. The predicted accuracy was evaluated in two aspects: the average dose prediction bias and mean absolute errors (MAEs)of all voxels within the body, the dice similarity coefficients (DSCs), Hausdorff distance(HD 95) and mean surface distance (MSD) of different isodose surfaces were used to address the spatial correspondence between predicted and clinical delivered 3D dose distributions; the dosimetric index (DI) including homogeneity index, conformity index, V50, V45 for PTV and OARs between predicted and clinical truth were statistically analyzed with the paired-samples t test. Results:For the 20 testing cases, the average prediction bias ranged from -2.12% to 2.88%, and the MAEs varied from 2.55% to 5.75%. The DSCs value was above 0.9 for all isodose surfaces, the average MSD ranged from 0.21 cm to 0.45 cm, and the average HD 95 varied from 0.61 cm to 1.54 cm. There was no statistically significant difference for all DIs, except for bladder Dmean. Conclusions:This study developed a deep learning model based on 3D U-Res-Net by considering beam configurations input and achieved an accurate 3D voxel-wise dose prediction for rectal cancer treated by IMRT.