Diagnosing prostate cancer in the PSA gray zone through machine learning and transrectal ultrasound video

1. Introduction

Prostate cancer (PCa) remains one of the most common malignancies affecting men worldwide, characterized by a high incidence rate and varying degrees of aggressiveness [1]. Clinically significant prostate tumors pose a significant health risk due to their potential for progression and metastasis [2]. The diagnostic approaches for PCa have traditionally relied on a combination of digital rectal examinations (DRE), prostate-specific antigen (PSA) testing [3], and advanced imaging techniques such as transrectal ultrasound (TRUS) and magnetic resonance imaging (MRI) [4].

PSA testing, in particular, is a cornerstone in the early detection of PCa. PSA is a protein produced by both normal and malignant prostate cells, with elevated levels of PSA often serving as a marker for potential malignancy [3, 5]. However, there is a so-called “PSA gray zone”, typically defined as a PSA range between 4 and 10 ng/mL, presents a diagnostic challenge [6]. This range is particularly problematic due to an increased risk of false-positive results, as PSA levels within this gray zone are neither clearly indicative of benign conditions nor definitively diagnostic for PCa. As a result, additional evaluation is required to improve diagnostic accuracy.

To address the diagnostic challenges associated with the PSA gray zone, additional biomarkers and advanced imaging techniques have been explored. For instance, recent research has investigated the potential of prostate health index scores [7], 4Kscore tests, and multiparametric MRI to enhance diagnostic accuracy within this uncertain range [8, 9]. Despite these diagnostic advances, many studies are significantly constrained by small sample sizes, limiting their generalizability. Other studies have relied on static imaging [10], which may not capture the dynamic behavior of tumor. Moreover, some new modalities are cost-prohibitive and not universally available, further restricting their practical application in routine clinical practice [11]. These shortcomings underscore the need for innovative and accessible methods to improve the specificity and sensitivity of PCa detection in the PSA gray zone, providing an impetus for the development of novel, dynamic and cost-effective solutions.

The recent surge in machine learning (ML) techniques offers a promising avenue for addressing the challenges and complexities associated with the PSA gray zone. With the capacity to analyze vast datasets and identify subtle patterns, machine learning can potentially unveil novel imaging biomarkers and improve decision-making processes [12]. The integration of ML with dynamic imaging modalities, such as TRUS videos, could provide a more detailed and nuanced understanding of tumor characteristics, facilitating improved risk stratification for patients with PSA levels in the gray zone.

Our study harnessed the power of machine learning in conjunction with TRUS video data to extract pertinent radiomic features. Through this approach, we developed a model capable of accurately distinguishing between benign conditions and malignancies within the PSA gray zone. The rationale for combining ML with TRUS video data lies in the ability of dynamic imaging to provide a temporal dimension to tissue characterization, which static images cannot capture. By training a model with this enriched data, we can effectively capture the heterogeneity of prostate tissue and identify malignancies with greater precision.

As far as we are aware, this study represents the first attempt to combine machine learning with TRUS video data for risk assessment in the PSA gray zone of prostate tumors, and the largest-scale investigation to date. Our research not only advances the diagnostic capabilities for PCa but also introduces a scalable and cost-effective tool that can be readily implemented in diverse clinical settings. By bridging the gap between advanced ML techniques and practical clinical applications, we believe our model is a significant milestone in the management of PCa, particularly for patients navigating the diagnostic uncertainty of the PSA gray zone.

2. Methods and materials

2.1 General information

A retrospective study was performed on clinical data collected from patients presenting with suspected prostate cancer at the Department of Urology, Dongyang Hospital, affiliated with Wenzhou Medical University, between August 2021 and September 2023. Initially, 1189 cases were reviewed, of which 508 cases were eventually included, comprising 187 cases of PCa and 321 cases of benign prostatic lesions. Participant selection criteria were rigorously operationalized as follow: (1) The preoperative assessment encompassed serum biomarkers associated with prostate-specific antigen (PSA), including total PSA (tPSA), free PSA (fPSA) and the fPSA/tPSA ratio, all measured within the range of 4 to 10 ng/mL; (2) completion of TRUS examination; and (3) patients who underwent transperineal prostate biopsy and received pathological diagnostic results. The exclusion criteria were: (1) previous prostate treatment such as hormonal or radiation therapy, and (2) incomplete clinical data or imaging examinations.

Among the 508 patients, 436 cases were randomly selected for training and validating the models, whereas the remaining 72 were used for assessing the derived models. The study protocol was evaluated and approved by the ethics committee of the hospital and subsequently registered. Fig. 1 illustrates the process of participant inclusion and exclusion.

Fig. 1.Flowchart of the study inclusion and exclusion criteria. PSA: prostate-specific antigen.

2.6 Statistical analysis

Statistical computations were performed utilizing Statistical Package of Social Sciences (SPSS) version 25.0 (IBM Corp., Armonk, NY, USA). Levene’s test was used to assess normality. For normally distributed continuous variables, descriptive analysis is presented as mean ± standard deviation, and inter-group differences were analyzed using independent sample t-tests. Non-normal distributions are described by median and interquartile range (IQR), with statistical contrasts performed through Mann-Whitney U test. Categorical variables were quantified as frequency distributions (expressed in percentages) and subjected to statistical analysis using the χ² test or continuity correction test. Receiver operating characteristic (ROC) curves were utilized to assess the diagnostic performance of the models for prostate cancer (PCa), including metrics such as area under the curve (AUC), sensitivity, specificity and diagnostic accuracy. A statistical significance threshold of p < 0.05 was applied. The overall flowchart for the study is presented in Fig. 2.

Fig. 2.Overall study flowchart, including feature extraction, feature selection, machine learning and model validation. ADB: adaptive boosting; RF: random forest; GBM: gradient boosting machine; ROC: receiver operating characteristic; AUC: area under the curve; SVM: support vector machine; TPR: true positive rate; MSE: mean square error.

3. Results

3.2 Feature selection

A total of 851 distinct features were derived from the ultrasound video clip datasets for each individual case, which were then subjected to feature selection using either t-tests or Mann-Whitney U tests based on the normality evaluation performed with Levene’s test (p > 0.05). In LASSO regression, 10-fold cross-validation was employed to determine the optimal λ coefficient. After LASSO dimensionality reduction, 25 features were retained: 7 original features and 18 wavelet features (Table 2). Additionally, Fig. 3 demonstrates the process of identifying key parameters from the features within the training set and the subsequent development of the linear predictor. Meanwhile, Fig. 4 displays the optimal penalty coefficient (λ) determined during the model optimization process.

Table 2.The subset of radiomics features ultimately selected by the LASSO algorithm.

Feature(s)	Image type	Feature class	Feature name	LASSO coefficients
1	original	shape	Least Axis Length	0.033571
2	original	shape	Maximum 2D Diameter Column	−0.093023
3	original	shape	Maximum 2D Diameter Row	0.005961
4	original	shape	Mesh Volume	0.087038
5	original	shape	Sphericity	−0.060414
6	original	shape	Surface Volume Ratio	0.024544
7	original	GLCM	Difference Average	−0.014693
8	Wavelet-LHL	First-order	Kurtosis	−0.030508
9	Wavelet-LHL	First-order	Median	0.034010
10	Wavelet-LHL	GLCM	ClusterShade	−0.021096
11	Wavelet-LHL	GLCM	Inverse Variance	0.001262
12	Wavelet-LHL	GLCM	Joint Average	0.008146
13	Wavelet-LHL	First-order	Median	−0.026531
14	Wavelet-LHL	First-order	Median	0.053700
15	Wavelet-LHL	First-order	Minimum	−0.046910
16	Wavelet-LHL	GLSZM	Gray Level Variance	0.100897
17	Wavelet-LHL	GLSZM	Small Area High Gray Level Emphasis	0.094075
18	Wavelet-LHL	First-order	Minimum	−0.059024
19	Wavelet-LHL	First-order	Minimum	0.016990
20	Wavelet-LHL	First-order	Total Energy	0.037929
21	wavelet-HHH	GLCM	Imc2	−0.061461
22	Wavelet-HHH	GLSZM	High Gray Level Zone Emphasis	−0.068249
23	Wavelet-HHH	GLSZM	Small Area Low Gray Level Emphasis	−0.039780
24	wavelet-LLL	GLSZM	Id	0.158603
25	Wavelet-LLL	GLSZM	Idn	−0.045269
GLCM: gray-level co-occurrence matrix; GLSZM: gray-level size zone matrix; LASSO: least absolute shrinkage and selection operator. Imc2: incremental matrix completion2; Idn: identifier numeric; Id: identifier.

Fig. 3.Identification of key feature parameters within the training set and development of the linear predictor. (A) Spearman’s correlation coefficients were estimated for the twenty-five selected features. (B) Feature classification weights. GLCM: gray-level co-occurrence matrix; GLSZM: gray-level size zone matrix; Imc²: incremental matrix completion²; Idn: identifier numeric; Id: identifier.

Fig. 4.Determination of the optimal penalty coefficient, λ. (A) Ten-fold cross-validation was employed to select the tuning parameters for the LASSO model. (B) Solution path of LASSO coefficients for the 25 features. MSE: mean square error; coef: coefficients.

Among the salient features delineated in Table 2, the original shape-related features are particularly noteworthy, as they provide critical insights into the morphology and margins of tumors within the US images. These features are instrumental in facilitating accurate diagnostic assessments. Additionally, several wavelet features capture intricate internal variations of the tumors, potentially reflecting subtle pathological changes that may signify malignancy or distinct tissue characteristics, thereby enhancing the interpretative depth of the model.

3.3 Model effectiveness

The RF model exhibited an AUC of 0.89, accuracy of 0.81, sensitivity of 0.81, specificity of 0.79, positive predictive value of 0.91, and F1 score of 0.77. The SVM model demonstrated an AUC of 0.80, accuracy of 0.79, sensitivity of 0.81, specificity of 0.75, positive predictive value of 0.89, and F1 score of 0.76. The ADB model achieved an AUC of 0.85, accuracy of 0.75, sensitivity of 0.80, specificity of 0.65, positive predictive value of 0.83, and F1 score of 0.72. Lastly, the GBM model obtained an AUC of 0.86, accuracy of 0.75, sensitivity of 0.80, specificity of 0.65, positive predictive value of 0.83, and F1 score of 0.72. Upon comprehensive analysis of the evaluation metrics across the four models in the validation set, the performance of the RF model surpassed that of GBM, ADB and SVM, exhibiting superior predictive performance and the best predictive efficacy for PSA gray zone PCa when compared to the other three models. The diagnostic performances of the RF, SVM, ADB and GBM models are summarized in Table 3, whereas the ROC curves are depicted in Fig. 5.

Table 3.Performance outcomes for each machine learning models.

Data set	AUC	Accuracy	Sensitivity	Specificity	Positive predictive value	Negative predictive value	F1-score
SVM
	Validation set	0.85	0.85	0.91	0.74	0.87	0.81	0.83
	Test set	0.80	0.79	0.81	0.75	0.89	0.60	0.76
RF
	Validation set	0.90	0.85	0.89	0.76	0.89	0.76	0.83
	Test set	0.89	0.81	0.81	0.79	0.91	0.60	0.77
ADB
	Validation set	0.86	0.76	0.81	0.63	0.84	0.57	0.71
	Test set	0.85	0.75	0.80	0.65	0.83	0.60	0.72
GBM
	Validation set	0.85	0.80	0.86	0.68	0.84	0.71	0.78
	Test set	0.86	0.75	0.80	0.65	0.83	0.60	0.72
AUC: area under the curve; SVM: support vector machine; RF: random forest; ADB: adaptive boosting; GBM: gradient boosting machine.

Fig. 5.ROC curves of categorical multivariate models in the validation set (A) and test set (B). ROC: receiver operating characteristic; SVM: support vector machine; AUC: area under the curve; RF: random forest; ADB: adaptive boosting; GBM: gradient boosting machine.

4. Discussion

Prostate biopsy is commonly considered the definitive method for diagnosing prostate cancer (PCa). Nevertheless, this invasive procedure may lead to considerable psychological burden in patients with PSA levels in the gray zone. Therefore, it is necessary to combine multiple techniques for diagnosing PCa in patients within the PSA gray zone. PSA has revolutionized the diagnosis of PCa, especially in the detection of early asymptomatic cases. It is noteworthy that when PSA levels are between 4–10 ng/mL, there is a significant overlap in the number of patients with BPH and PCa [15]. In recent years, the application of multiparametric MRI (mpMRI) has further improved the sensitivity and specificity of PCa diagnosis. The Prostate Imaging Reporting and Data System version 2 (PI-RADS v2) is in clinical use, but its predictive value for prostate biopsy results in patients in the PSA gray zone remains limited [16].

TRUS and mpMRI are two widely utilized imaging techniques for the evaluation of prostate diseases. TRUS is extensively used in clinical practice due to its safety, lack of radiation, low cost and simplicity [17, 18]. TRUS can usually clearly visualize the contours of the prostate, as well as indicate the transition zone with isoechoic and hypoechoic areas, with around 70% of prostate cancers are located in the peripheral zone (which generally appear as hypoechoic lesions). However, PCa situated within the transition zone can present detection challenges, as the neoplasm might be concealed by the hypoechoic background tissue. Although grayscale TRUS is helpful in diagnosing BPH, it is unreliable for detecting PCa [19, 20]. The accuracy of TRUS in detecting PCa in the general population is only moderate [21]. In studies concerning the PSA gray zone, the detection rate for PCa is 25% [22]. In this study, out of 508 patients with suspected PCa in the PSA gray zone, 187 (36%) were diagnosed with PCa by biopsy, a slightly higher fraction than in previous research results. This variation could potentially stem from a multitude of factors, including but not limited to ethnic diversity, geographical distinctions and dietary practices.

Conversely, liquid biopsies have gained significant prominence as a valuable tool in cancer research, serving as a complement to conventional tissue biopsies [23]. This non-invasive technique enables real-time assessment of genomic, transcriptomic, and epigenomic alterations in both primary and metastatic tumors. By integrating data from multiple liquid biopsies, it becomes possible to thoroughly evaluate tumor heterogeneity and subclonal evolution, which aids in the development of personalized therapeutic strategies [24]. Various clinical studies to date have demonstrated the utility of liquid biopsy using cell-free DNA and extracellular vesicles (EVs) as tools in distinct clinical stages of PCa management. Specifically, circulating tumor DNA is increasingly used in advanced cases of metastatic castration-resistant PCa, as it targets genetic mutations within the tumor, enabling treatment choices based on these mutations. When a tumor grows, the expression of certain molecules within EVs may increase. This approach is key because, if these specific molecules can be pinpointed early on, even at the early stages of PCa, it could greatly aid in early detection [25]. Although liquid biopsy has demonstrated enormous potential to improve the current practice, additional assays for risk stratification are required to supplement the current model [26]. To accompany the PSA test for PCa screening, high-specificity tests are needed. After PSA screening, a key challenge is determining how to further select the best candidates for needle biopsy to avoid such invasiveness in a non-cancer population. Even though the future of PCa treatment using liquid biopsy holds great promise, addressing the existing challenges is imperative for its successful integration into routine clinical practice, making personalized and precise PCa management a reality for patients.

Prostate enlargement with increasing age leads to a concomitant increase in prostate volume and cell count, which, in turn, results in elevated PSA secretion by epithelial cells. The diagnostic efficacy of total PSA (tPSA) values for PCa is notably lower within the PSA gray zone compared to cases with PSA levels >10 ng/mL [27]. Notably, Liu constructed a PCa prediction model for 197 patients in the PSA gray zone using PSA and its derivatives, with sensitivity and specificity for predicting PCa of 75.4% and 75.8%, respectively [28]. However, the aforementioned study failed to account for imaging-based assessments.

Radiomics in prostate cancer (PCa) represents a rapidly advancing field of study, holding great promise for providing non-invasive and longitudinal biomarkers to support personalized medicine [29]. This innovative approach utilizes machine learning techniques to convert visual image data into detailed quantitative metrics, extracting extensive feature information from medical images, which are then used to build predictive models [30]. Radiomics facilitates robust quantitative analysis of imaging data, enabling non-invasive evaluation of tumor biology, and has been extensively applied in the diagnosis of PCa, assessment of tumor aggressiveness, and support of clinical decision making [31]. Notably, radiomics-based methods for diagnosing PCa malignancy and GS scoring have demonstrated good predictive performance [32].

By high-throughput data extraction and building efficient and stable predictive models (such as our extraction of features from video clips), radiomics can offer auxiliary diagnostic capabilities for clinical applications. Feature selection is crucial in ML research, as the absence of high-throughput feature extraction can result in data redundancy and overfitting [33, 34, 35]. Jin predicted early cervical cancer lymph node metastasis using ML methods, extracting 106 radiomic features from lymph node ultrasound images. By integrating LASSO and ridge regression, key features were selected from high-dimensional data to mitigate the risk of overfitting. Subsequently, six features were identified for classification analysis, demonstrating a strong correlation with tumor texture complexity and heterogeneity [36]. In addition, Wang conducted ML prediction research on PCa in TRUS video clips, achieving SVM model AUCs of 0.78 and 0.75 in the validation and test sets, respectively. The diagnostic performance of the SVM model was higher than diagnostics based on MRI (AUC values of 0.78 vs. 0.65/0.75 and 0.75 vs. 0.65/0.72, respectively), indicating that it achieves greater diagnostic efficacy and consistency [37]. Furthermore, Li determined that an SVM model based on MRI achieved good diagnostic performance for the malignancy of PI-RADS category 3 prostate lesions (training set AUC = 0.93 and validation set AUC = 0.89) and invasion prediction (training set AUC = 0.92 and validation set AUC = 0.85) [38]. Qi established SVM, RF, ABD and GBM machine learning models for PCa prediction using TRUS video clips combined with MRI, with AUCs of 0.87, 0.85, 0.84 and 0.81, respectively, indicating high predictive performance [39]. Additionally, Nketiah et al. [40] combined texture features extracted from T2 weighted image with quantitative apparent diffusion coefficient values and dynamic contrast-enhanced pharmacokinetic parameters for predicting PCa GS grading. Their approach achieved a higher predictive performance (AUC = 0.91) compared to the RF model within the study (AUC = 0.88). This discrepancy could be explained by the limited sample size and the absence of an independent validation cohort in their analysis [40]. While these studies differ in focus, they collectively underscore the significant diagnostic and predictive value of ML models for PCa. They also highlight the versatility of ML applications across various imaging modalities and datasets.

Chen conducted a regression analysis on age, PSA-related indicators, TRUS and mpMRI to construct a binary logistic regression model to predict and evaluate the diagnostic value of PCa in patients in the PSA gray zone. Their results indicated that when selecting a critical value above 0.36, the AUC, sensitivity, and specificity of the PCa prediction model were 0.79, 73.64% and 64.23%, respectively [41]. Our predictive model demonstrated superior diagnostic performance compared to theirs, likely due to the ability of machine learning (ML) models to extract a greater amount of image feature information. As part of this work, a total of 851 feature types were extracted and subsequently filtered using the LASSO algorithm, resulting in the retention of 25 non-zero feature types that showed reliable correlations with PCa. Predictive models were then constructed using SVM, RF, ADB and GBM algorithms. In the validation set, the AUC values achieved were 0.85, 0.90, 0.86 and 0.85, respectively. Corresponding accuracy rates were 0.79, 0.81, 0.75 and 0.75; positive predictive values were 0.89, 0.91, 0.83 and 0.83; and F1 scores were 0.76, 0.77, 0.72 and 0.72, respectively. In contrast, Liu achieved AUC values of the validation set using logistic regression, Gaussian Naive Bayes, light gradient-boosting machine (LGBM) Classifier, and extreme gradient boosting (XGBoost) ML models of 0.918, 0.893, 0.906 and 0.893, respectively, with accuracies of 0.811, 0.792, 0.794 and 0.800, and F1 values of 0.758, 0.765, 0.749 and 0.771 [42]. This outcome indicates that the diagnostic performance of the four machine learning models selected in their study is higher than that of our study, which may be due to differences in samples or ML model selection.

Despite the inclusion of 508 cases in this study, the test set comprised a relatively limited sample size of 72 cases, which may introduce potential bias. Furthermore, while the ratio of sample sizes between the BPH group and the PCa group was maintained at approximately 2:1 to achieve balance, the overall sample size remained moderate. This constraint increases the risk of classifier models encountering issues such as overfitting and reduced generalizability. As a result, expanding the test set sample size is crucial to enhance the reliability and robustness of the findings.

Although the RF model demonstrated good predictive results within this unit, it has not been applied and validated in other clinical settings. Therefore, multicenter studies are needed for further testing and validation. Additionally, prospective studies are required to verify the reliability and accuracy of this model, to improve the predictive efficacy and detection rate of PCa in patients with PSA “gray zone” levels. Such efforts could potentially lower the rate of unwarranted prostate biopsies in this patient population.

This study used prostate biopsy pathology as a reference rather than prostatectomy pathology, which may have introduced bias [43]. For patients diagnosed with PCa through biopsy, some patients may have missed the opportunity for surgical intervention, potentially affecting the outcomes.

5. Conclusions

Machine learning models for PCa diagnosis, utilizing algorithms such as SVM, RF, ADB and GBM, have shown significant diagnostic potential for patients with PSA “gray zone” level. Among these models, the RF algorithm demonstrated the highest predictive performance. These ML models offer promising tools to aid clinical decision-making, supporting physicians in achieving accurate diagnoses and tailoring personalized treatment plans. Additionally, they provide valuable guidance for prostate biopsy decisions and optimizing follow-up strategies for patients within the PSA “gray zone”.

Abbreviations

PCa, Prostate cancer; PSA, prostate specific antigen; AUC, area under the curve; ROC, Receiver operating characteristic; SVM, support vector machine; RF, random forest; ADB, adaptive boosting; GBM, gradient boosting machine; TRUS, Transrectal ultrasound; MRI, Magnetic resonance imaging; ML, Machine learning; ICCs, Intraclass correlation coefficients; LASSO, Least absolute shrinkage and selection operator; SPSS, Statistical Package for the Social Sciences; IQR, Interquartile range; tPSA, Total prostate-specific antigen; BPH, Benign prostatic hyperplasia; HGPIN, High-grade prostatic intraepithelial neoplasia; GS, Gleason score; US, Ultrasound; mpMRI, Multiparametric magnetic resonance imaging; DRE, digital rectal examinations; fPSA, free prostate-specific antigen; GLRLM, gray-level run length matrix; GLSZM, gray-level size zone matrix; NGDM, neighborhood gray-tone difference matrix; SVM, support vector machine; ADB, adaptive boosting; EVs, extracellular vesicles; PI-RADS, Prostate Imaging Reporting and Data System; LGBM, light gradient-boosting machine; XGBoost, extreme gradient boosting; CLCM, collaborative latent class model; IPR, individual prediction reliability.

Availability of data and materials

The data presented in this study are available on reasonable request from the corresponding author.

Author contributions

QW, CYW and MLZ—designed the research study and implemented the code and contributed to writing the manuscript. JY, JXZ, YJ, YHD, XBS, LYJ, KW, ZBH and XYQ—managed data collection and processing; contributed to data analysis; and assisted in writing the manuscript. JCY, ZPW and DX—as corresponding authors; supervised the research; managed the project; reviewed the manuscript; and provided guidance on the research project. All authors contributed to editorial changes in the manuscript and read and approved the final version.

Ethics approval and consent to participate

The Medical Ethics Committee of Dongyang People’s Hospital 2024-YX-030. Comparison of machine learning models based on transrectal ultrasound videos for the diagnosis of prostate cancer in the PSA gray zone. All participants provided their consent to participate in the study.

Acknowledgment

We extend our sincere gratitude to the Department of Urology at Dongyang Hospital, as affiliated with Wenzhou Medical University, for their support and cooperation. We also appreciate the technical support and equipment provided by the Esaote MyLab™ ClassC color Doppler ultrasound diagnostic system. Special thanks to the three experienced ultrasound doctors with over ten years of expertise in transrectal ultrasound diagnosis, who conducted the ultrasound scans and collected the 10-second transverse section video clips of the prostate. Additionally, we acknowledge the contributions of all medical personnel and research team members involved in the collection and processing of patient data.

Funding

This work was supported by the National Key Research and Development Program of China (2022YFF0608403, 2023YFC2414004) and the Pioneer and Leading Goose R&D Program of Zhejiang (2023C04039) and the Key Science and Technology Project of Jinhua, Zhejiang Province (2022-3-016, 2022-3-017 and 2023-3-012).

Conflict of interest

The authors declare no conflict of interest.