Timely and rapid diagnosis is crucial for faster and proper malaria treatment planning. Microscopic examination is the gold standard for malaria diagnosis, where hundreds of millions of blood films are examined annually. However, this method’s effectiveness depends on the trained microscopist’s skills. With the increasing interest in applying deep learning in malaria diagnosis, this study aims to determine the most suitable deep-learning object detection architecture and their applicability to detect and distinguish red blood cells as either malaria-infected or non-infected cells. The object detectors Yolov4, Faster R-CNN, and SSD 300 are trained with images infected by all five malaria parasites and from four stages of infection with 80/20 train and test data partition. The performance of object detectors is evaluated, and hyperparameters are optimized to select the best-performing model. The best-performing model was also assessed with an independent dataset to verify the models’ ability to generalize in different domains. The results show that upon training, the Yolov4 model achieves a precision of 83%, recall of 95%, F1-score of 89%, and mean average precision of 93.87% at a threshold of 0.5. Conclusively, Yolov4 can act as an alternative in detecting the infected cells from whole thin blood smear images. Object detectors can complement a deep learning classification model in detecting infected cells since they eliminate the need to train on single-cell images and have been demonstrated to be more feasible for a different target domain.