Objective To develop a multi-modal deep learning method for automatic classification of immune-mediated glomerular diseases based on images of optical microscopy(OM),immunofluorescence microscopy(IM),and transmission electron microscopy(TEM).Methods We retrospectively collected the pathological images from 273 patients and constructed a multi-modal multi-instance model for classification of 3 immune-mediated glomerular diseases,namely immunoglobulin A nephropathy(IgAN),membranous nephropathy(MN),and lupus nephritis(LN).This model adopts an instance-level multi-instance learning(I-MIL)method to select the TEM images for multi-modal feature fusion with the OM images and IM images of the same patient.By comparing this model with unimodal and bimodal models,we explored different combinations of the 3 modalities and the optimal methods for modal feature fusion.Results The multi-modal multi-instance model combining OM,IM,and TEM images had a disease classification accuracy of(88.34±2.12)%,superior to that of the optimal unimodal model[(87.08±4.25)%]and that of the optimal bimodal model[(87.92±3.06)%].Conclusion This multi-modal multi-instance model based on OM,IM,and TEM images can achieve automatic classification of immune-mediated glomerular diseases with a good classification accuracy.

Objective To develop a multi-modal deep learning method for automatic classification of immune-mediated glomerular diseases based on images of optical microscopy(OM),immunofluorescence microscopy(IM),and transmission electron microscopy(TEM).Methods We retrospectively collected the pathological images from 273 patients and constructed a multi-modal multi-instance model for classification of 3 immune-mediated glomerular diseases,namely immunoglobulin A nephropathy(IgAN),membranous nephropathy(MN),and lupus nephritis(LN).This model adopts an instance-level multi-instance learning(I-MIL)method to select the TEM images for multi-modal feature fusion with the OM images and IM images of the same patient.By comparing this model with unimodal and bimodal models,we explored different combinations of the 3 modalities and the optimal methods for modal feature fusion.Results The multi-modal multi-instance model combining OM,IM,and TEM images had a disease classification accuracy of(88.34±2.12)%,superior to that of the optimal unimodal model[(87.08±4.25)%]and that of the optimal bimodal model[(87.92±3.06)%].Conclusion This multi-modal multi-instance model based on OM,IM,and TEM images can achieve automatic classification of immune-mediated glomerular diseases with a good classification accuracy.