Objective To evaluate the efficiency of intra-tumor and peri-tumor ultrasound radiomics models based on machine learning algorithms for predicting benign and malignant Breast Imaging Reporting and Data System (BI-RADS) 4 breast lesions, and provide insights into early diagnosis of breast cancer. Methods A retrospective analysis was conducted based on the medical records of 450 female patients who underwent breast ultrasound examination in the Affiliated Cancer Hospital of Xinjiang Medical University from June 2020 to April 2022. The patients were divided into the benign (n = 199) and malignant (n = 195) groups according to pathological examination, and randomized into the training (n = 275) and validation (n = 119) sets at a ratio of 7∶3. Radiomics features were extracted and screened. Intra-tumor, peri-tumor, and intra-tumor + peri-tumor ultrasound radiomics models were constructed based on three machine learning algorithms, including logistic regression (LR), support vector machine (SVM), and multi-layer perceptron (MLP). Receiver operating characteristics (ROC) curves, calibration curves, and decision curves were plotted to evaluate the efficacy of the radiomics models for prediction of benign and malignant breast lesions. Results A total of 17 intra-tumor, 16 peri-tumor, and 17 intra-tumor + peri-tumor radiomics features were selected for model construction. Based on LR, MLP, and SVM algorithms, the intra-tumor + peri-tumor radiomics models showed higher predictive efficacy than intra-tumor and peri-tumor radiomics models. The predictive efficacy of intra-tumor, peri-tumor, and intra-tumor + peri-tumor radiomics models were higher based on the SVM algorithm than based on LR and MLP algorithms. For the intra-tumor radiomics model based on the SVM algorithm, the area under the ROC curve (AUC), accuracy, sensitivity, and a specificity were 0.909, 0.851, 0.860, and 0.842, respectively, in the training set and 0.866, 0.832, 0.847, and 0.817, respectively, in the validation set. For the peri-tumor radiomics model based on the SVM algorithm, these values were 0.899, 0.855, 0.882, and 0.827, respectively, in the training set and 0.844, 0.815, 0.847, and 0.783, respectively, in the validation set. For the intra-tumor + peri-tumor radiomics model based on the SVM algorithm, these values were 0.943, 0.876, 0.860, and 0.892, respectively, in the training set and 0.881, 0.849, 0.915, and 0.783, respectively, in the validation set. Conclusion The intra-tumor and peri-tumor ultrasound radiomics models based on machine learning algorithms are highly valuable for prediction of benign and malignant BI-RADS 4 breast lesions. The intra-tumor + peri-tumor ultrasound radiomics model based on the SVM algorithm has the optimal efficacy for prediction of benign and malignant BI-RADS 4 breast lesions.