In this study, an artificial intelligence-based machine vision system was developed for in-line particle size analysis during the pellet layering process. Drug-layered pellets were produced by coating microcrystalline cellulose cores with an ibuprofen-containing layering liquid until the target drug content was achieved. Drug content increases with pellet size; therefore, particle size monitoring can ensure product safety and quality. The direct imaging system, consisting of a rigid endoscope, a light source, and a high-speed camera, provides real-time information about pellet size and layer uniformity, enabling timely intervention in the case of out-of-spec products. A convolutional neural network-based instance segmentation algorithm was employed to detect particles in focus, ensuring that pellet size could be accurately determined despite the dense flow of the particles. After training the model, the performance of the developed system was assessed by analysing the particle size distribution of pellet cores with variable sizes within the 250-850 μm size range. The endoscopic system was tested in-line at a larger scale during the drug layering of inert pellet cores. The particle size data acquired in real time with the endoscopic imaging system corresponded with the reference methods, demonstrating the feasibility of the proposed machine vision-based method as a process analytical technology tool for in-line process monitoring.