Background: We developed a machine learning-based predictive model for diagnosing prostate cancer within the gray zone of prostate-specific antigen (PSA) levels, leveraging transrectal prostate ultrasound video clips. Methods: Data were collected for patients with suspected prostate cancer, characterized by intermediate PSA levels between 4 and 10 ng/mL, who visited the Department of Urology, Dongyang People’s Hospital, which is affiliated with Wenzhou Medical University, from 20 August 2021 to 30 September 2023. Among the final selection of 508 patients, a total of 851 features were extracted from the ultrasound video clips, reduced the dimensionality using least absolute shrinkage and selection operator regression, and finally selected 25 features. The selected features were employed to construct radiomics models based on four machine learning algorithms support vector machine (SVM), random forest (RF), adaptive boosting (ADB) and gradient boosting machine (GBM). The performance of the model was comprehensively assessed using receiver operating characteristic (ROC) curve analysis, with diagnostic effectiveness measured through metrics such as the area under the curve (AUC), sensitivity, specificity and overall accuracy. Results: The RF model demonstrated an AUC of 0.89, accuracy of 0.81, sensitivity of 0.81, specificity of 0.79, positive predictive value of 0.91 and F1 score of 0.77. As compared to the RF model, the SVM, ADB and GBM models showed similar values for AUC (range 0.80–0.86), accuracy (range 0.75–0.79), sensitivity (range 0.80–0.81), specificity (range 0.65–0.75), positive predictive value (range 0.83–0.89) and F1 score (range 0.72–0.76). In the validation set, following comprehensive evaluation, the RF model exhibited the best performance among the four models. Conclusions: The four machine learning models each had diagnostic value for detecting prostate cancer in patients within the PSA “gray zone”, with the RF model demonstrating the highest predictive performance.