OBJECTIVE: To establish a deep-learning architecture based on faster region-based convolutional neural networks (Faster R-CNN) algorithm for detection and sorting of red ginseng (Ginseng Radix et Rhizoma Rubra) with internal defects automatically on an online X-ray machine vision system. METHODS: A Faster R-CNN based classifier was trained with around 20 000 samples with mean average precision value (mAP) of 0.95. A traditional image processing method based on feedforward neural network (FNN) obtained a bad performance with the accuracy, recall and specificity of 69.0%, 68.0%, and 70.0%, respectively. Therefore, the Faster R-CNN model was saved to evaluate the model performance on the defective red ginseng online sorting system. RESULTS: An independent set of 2 000 red ginsengs were used to validate the performance of the Faster R-CNN based online sorting system in three parallel tests, achieving accuracy of 95.8%, 95.2% and 96.2%, respectively. CONCLUSION The overall results indicated that the proposed Faster R-CNN based classification model has great potential for non-destructive detection of red ginseng with internal defects.