Background: High coal dust exposure threatens miners’ occupational health and safety. While dust control is mainly implemented by enterprises, miners’ active participation is crucial for effective collaborative prevention. This study develops a feature reduction, genetic algorithm, and backpropagation (RS-GA-BP) hybrid evaluation model integrating behavioral and psychosocial factors to quantitatively assess collaborative dust prevention performance in coal mines. Methods: Guided by the theory of collaborative dust prevention among coal miners, this study incorporated human factors into the evaluation framework of prevention effectiveness. Using the rough set method, five key influencing factors were identified from twelve candidate variables. A prediction model for collaborative dust prevention effectiveness (RS-GA-BP) was then developed and applied by optimizing the BP neural network with a genetic algorithm. Results: The results indicated that technical context, conformity tendency, group cohesion, group driving force, and group dissipative force were the principal factors influencing collaborative prevention outcomes. Based on 955 survey responses from front-line coal miners, the model was trained and validated. The results showed that the GA-BP model outperformed the traditional BP model in terms of root mean square error, mean absolute error, and mean absolute percentage error, achieving a prediction accuracy of 95.73%. Conclusion The research results indicate that the RS-GA-BP model can effectively evaluate the dust prevention and control effectiveness among coal miners, thereby enriching the methodological framework for assessing dust prevention effectiveness in coal mines.