Mixed reality (MR), characterized by the ability to integrate digital data into human real feeling, is a new technique in medical imaging and surgical navigation. MR has tremendous value in surgery, but its application in oromaxillofacial head and neck oncology surgery is not yet reported. This paper reports the application of MR in oromaxillofacial head and neck oncology surgery. The merits, demerits, and present research situations and prospects of MR are further discussed.
Augmented Reality ; Humans ; Surgery, Computer-Assisted

Objective: To examine the clinical feasibility of mixed reality navigation (MRN) technology based on multimodal imaging for the resection of intracranial eloquent lesions. Methods: Fifteen patients with intracranial eloquent lesions admitted to the Department of Neurosurgery, the First Medical Center, People's Liberation Army General Hospital from September 2020 to September 2021 were retrospectively enrolled. There were 7 males and 8 females, aged (50±16) years (range: 16 to 70 years). Postoperative pathological diagnosis included meningioma (n=7), metastatic carcinoma (n=3), cavernous hemangioma, glioma, ependymoma, aneurysmal changes and lymphoma (n=1, respectively). The open-source software was used to perform the three-dimensional visualization of preoperative images, and the self-developed MRN system was used to perform the fusion and interaction of multimodal images, so as to formulate the surgical plan and avoid damaging the eloquent white matter fiber tracts. Traditional navigation, intraoperative ultrasound and fluorescein sodium angiography were used to determine the extent of lesion resection. The intraoperative conditions of MRN-assisted surgery were analyzed, and the setup time and localization error of MRN system were measured. The changes of postoperative neurological function were recorded. Results: MRN based on multimodal imaging was achieved in all patients. The MRN system setup time (M(IQR)) was 36 (12) minutes (range: 20 to 44 minutes), and the localization error was 3.2 (2.0) mm (range: 2.6 to 6.7 mm). The reliability of eloquent white matter fiber tracts localization based on MRN was rated as "excellent" in 11 cases, "medium" in 3 cases, and "poor" in 1 case. There were no perioperative death and no new impairment in motor, language, or visual functions after operation. Transient limb numbness occurred in 1 patient after operation, and recovered to the preoperative state in 2 weeks after operation. Conclusion: The MRN system based on multimodal imaging can improve the surgical accuracy and safety, and reduce the incidence of iatrogenic neurological dysfunction.
Humans ; Augmented Reality ; Reproducibility of Results ; Retrospective Studies ; Multimodal Imaging

Mixed reality is a new three-dimensional presentation technology that combines the virtual digital world with the real world, which has been initially applied in the field of hepatobiliary surgery. Compared with virtual reality, augmented reality and three-dimensional visualization technology, mixed reality technology has unique advantages in preoperative evaluation and formulation of surgical plan, real-time accurate navigation during operation and three-dimensional virtual teaching. And it is a new generation of auxiliary tool for precision hepatobiliary surgery. This paper describes the application and research progress of mixed reality technology in the field of hepatobiliary surgery, and discusses its application potential and current limitations.
Augmented Reality ; Humans ; Imaging, Three-Dimensional ; Surgery, Computer-Assisted ; Technology ; Virtual Reality

Brain-computer interface (BCI) has great potential to replace lost upper limb function. Thus, there has been great interest in the development of BCI-controlled robotic arm. However, few studies have attempted to use noninvasive electroencephalography (EEG)-based BCI to achieve high-level control of a robotic arm. In this paper, a high-level control architecture combining augmented reality (AR) BCI and computer vision was designed to control a robotic arm for performing a pick and place task. A steady-state visual evoked potential (SSVEP)-based BCI paradigm was adopted to realize the BCI system. Microsoft's HoloLens was used to build an AR environment and served as the visual stimulator for eliciting SSVEPs. The proposed AR-BCI was used to select the objects that need to be operated by the robotic arm. The computer vision was responsible for providing the location, color and shape information of the objects. According to the outputs of the AR-BCI and computer vision, the robotic arm could autonomously pick the object and place it to specific location. Online results of 11 healthy subjects showed that the average classification accuracy of the proposed system was 91.41%. These results verified the feasibility of combing AR, BCI and computer vision to control a robotic arm, and are expected to provide new ideas for innovative robotic arm control approaches.
Augmented Reality ; Brain-Computer Interfaces ; Computers ; Electroencephalography ; Evoked Potentials, Visual ; Humans ; Photic Stimulation ; Robotic Surgical Procedures

Computer-assisted technology are gradually integrated into dental education and clinical treatment. As a cutting-edge technology in computer-aided medicine, augmented reality can not only be used as an aid to dental education by presenting three-dimensional scenes for teaching demonstration and experimental skills training, but also can superimpose virtual image information of patients onto real lesion areas for real-time feedback and intraoperative navigation. This review explores the current applications and limitations of augmented reality in dentistry to provide a reference for future research.
Humans ; Augmented Reality ; Oral Medicine ; Surgery, Computer-Assisted/methods* ; Imaging, Three-Dimensional

OBJECTIVES: This study aimed to build a surgical navigation system based on mixed reality (MR) and optical positioning technique and evaluate its clinical applicability in craniomaxillofacial trauma bone reconstruction. Me-thods We first integrated the software and hardware platforms of the MR-based surgical navigation system and explored the system workflow. The systematic error, target registration error, and osteotomy application error of the system were then analyzed via 3D printed skull model experiment. The feasibility of the MR-based surgical navigation system in craniomaxillofacial trauma bone reconstruction was verified via zygomatico-maxillary complex (ZMC) reduction experiment of the skull model and preliminary clinical study. RESULTS: The system error of this MR-based surgical navigation system was 1.23 mm±0.52 mm, the target registration error was 2.83 mm±1.18 mm, and the osteotomy application error was 3.13 mm±1.66 mm. Virtual surgical planning and the reduction of the ZMC model were successfully conducted. In addition, with the guidance of the MR-based navigation system, the frontal bone defect was successfully reconstructed, and the clinical outcome was satisfactory. CONCLUSIONS The MR-based surgical navigation system has its advantages in virtual reality fusion effect and dynamic navigation stability. It provides a new method for doctor-patient communications, education, preoperative planning, and intraoperative navigation in craniomaxillofacial surgery.
Humans ; Surgical Navigation Systems ; Augmented Reality ; Plastic Surgery Procedures ; Skull/surgery*

OBJECTIVE: To explore the application of mixed reality technique for the surgery of oral and maxillofacial tumors. METHODS: In this study, patients with a diagnosis of an oral and maxillofacial tumor who were referred to Department of Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology from December 2018 to January 2020 were selected. The preoperative contrast-enhanced computed tomography data of the patients were imported into StarAtlas Holographic Medical Imaging System (Visual 3D Corp., Beijing, China). Three-dimensional (3D) model of tumor and key structures, such as skeleton and vessels were reconstructed to three-dimensionally present the spatial relationship between them, followed with the key structures delineation and preoperative virtual surgical planning. By using mixed reality technique, the real-time 3D model was displayed stereotactically in the surgical site. While keeping sterile during operation, the surgeon could use simple gestures to adjust the 3D model, and observed the location, range, and size of tumor and the key structures adjacent to the tumor. Mixed reality technique was used to assist the operation: 3D model registration was performed for guidance before tumor excision; intraoperative real-time verification was performed during tumor exposure and after excision of the tumor. The Likert scale was used to evaluate the application of mixed reality technique after the operation. RESULTS: Eight patients underwent mixed reality assisted tumor resection, and all of them successfully completed the operation. The average time of the 3D model registration was 12.0 minutes. In all the cases, the surgeon could intuitively and three-dimensionally observe the 3D model of the tumor and the surrounding anatomical structures, and could adjust the model during the operation. The results of the Likert scale showed that mixed reality technique got high scores in terms of perceptual accuracy, helping to locate the anatomical parts, the role of model guidance during surgery, and the potential for improving surgical safety (4.22, 4.19, 4.16, and 4.28 points respectively). Eight patients healed well without perioperative complications. CONCLUSION By providing real-time stereotactic visualization of anatomy of surgical site and guiding the operation process through 3D model, mixed reality technique could improve the accuracy and safety of the excision of oral and maxillofacial tumors.
Augmented Reality ; China ; Humans ; Imaging, Three-Dimensional ; Neoplasms ; Retrospective Studies ; Surgery, Computer-Assisted

Objective: To investigate the application effect of augmented reality and mixed reality navigation technology in three-dimensional(3D) laparoscopic narrow right hepatectomy(LRH). Methods: A retrospective analysis was performed on the clinical data of 5 patients with hepatic malignancy admitted to the First Department of Hepatobiliary Surgery,Zhujiang Hospital,Southern Medical University from September 2020 to June 2021,all of whom were males,aged from 42 to 74 years.Preoperative evaluation was performed using the self-developed 3D abdominal medical image visualization system; if all the 5 patients were to receive right hemihepatectomy,the remnant liver volume would be insufficient,so LRH were planned.During the operation,the independently developed 3D laparoscopic augmented reality and mixed reality surgical navigation system was used to perform real-time multi-modal image fusion and interaction between the preoperative 3D model and 3D laparoscopic scene.Meanwhile,intraoperative ultrasound assisted indocyanine green fluorescence was used to determine the surgical path.In this way,the LRH under the guidance of augmented reality and mixed reality navigation was completed.The predicted liver resection volume was evaluated before surgery,actual resected liver volume,surgical indicators and postoperative complications were analyzed. Results: All the 5 patients completed LRH under the guidance of augmented reality and mixed reality navigation technology,with no conversion to laparotomy.The median operative time was 300 minutes(range:270 to 360 minutes),no intraoperative blood transfusion was performed,and the median postoperative hospital stay was 8 days(range:7 to 9 days).There were no perioperative deaths,or postoperative complications such as liver failure,bleeding,or biliary fistula. Conclusion: For patients who need to undergo LRH,the use of augmented and mixed reality navigation technology can safely and effectively guide the implementation of surgery,retain more functional liver volume,improve surgical safety,and reduce postoperative complications.
Adult ; Aged ; Augmented Reality ; Hepatectomy/methods* ; Humans ; Imaging, Three-Dimensional ; Laparoscopy/methods* ; Liver Neoplasms/surgery* ; Male ; Middle Aged ; Retrospective Studies ; Technology