Emotion can reflect the psychological and physiological health of human beings, and the main expression of human emotion is voice and facial expression. How to extract and effectively integrate the two modes of emotion information is one of the main challenges faced by emotion recognition. In this paper, a multi-branch bidirectional multi-scale time perception model is proposed, which can detect the forward and reverse speech Mel-frequency spectrum coefficients in the time dimension. At the same time, the model uses causal convolution to obtain temporal correlation information between different scale features, and assigns attention maps to them according to the information, so as to obtain multi-scale fusion of speech emotion features. Secondly, this paper proposes a two-modal feature dynamic fusion algorithm, which combines the advantages of AlexNet and uses overlapping maximum pooling layers to obtain richer fusion features from different modal feature mosaic matrices. Experimental results show that the accuracy of the multi-branch bidirectional multi-scale time sensing dual-modal emotion recognition model proposed in this paper reaches 97.67% and 90.14% respectively on the two public audio and video emotion data sets, which is superior to other common methods, indicating that the proposed emotion recognition model can effectively capture emotion feature information and improve the accuracy of emotion recognition.
Humans ; Emotions ; Algorithms ; Facial Expression ; Time Perception ; Neural Networks, Computer ; Speech

Multi-task learning (MTL) has demonstrated significant advantages in the field of physiological signal measurement. This approach enhances the model's generalization ability by sharing parameters and features between similar tasks, even in data-scarce environments. However, traditional multi-task physiological signal measurement methods face challenges such as feature conflicts between tasks, task imbalance, and excessive model complexity, which limit their application in complex environments. To address these issues, this paper proposes an enhanced multi-scale spatiotemporal network (EMSTN) based on Eulerian video magnification (EVM), super-resolution reconstruction and convolutional multilayer perceptron. First, EVM is introduced in the input stage of the network to amplify subtle color and motion changes in the video, significantly improving the model's ability to capture pulse and respiratory signals. Additionally, a super-resolution reconstruction module is integrated into the network to enhance the image resolution, thereby improving detail capture and increasing the accuracy of facial action unit (AU) tasks. Then, convolutional multilayer perceptron is employed to replace traditional 2D convolutions, improving feature extraction efficiency and flexibility, which significantly boosts the performance of heart rate and respiratory rate measurements. Finally, comprehensive experiments on the Binghamton-Pittsburgh 4D Spontaneous Facial Expression Database (BP4D+) fully validate the effectiveness and superiority of the proposed method in multi-task physiological signal measurement.
Humans ; Neural Networks, Computer ; Signal Processing, Computer-Assisted ; Face/physiology* ; Video Recording ; Facial Expression ; Heart Rate ; Algorithms

OBJECTIVE: To investigate the changes and symmetry of facial soft tissue during posed smile, to analyze the feature of posed smile in different gender, and verify the reproducibility of posed smile. METHODS: Three-dimensional (3D) facial images of 41 adults (16 males and 25 females with an average age of 26.76±2.70 years) which were taken by FaceScan three-dimensional sensor, including one rest position and two posed smile images. Then these images were imported into 3D soft tissue analysis software for model repositioning. 3D morphable model method (3DMM) was carried out to automatic landmarks setting. After that, the measurement of the eyes, cheeks, nose and perioral area were carried out for 3D soft tissue analysis. Finally, the changes and symmetry of the soft tissues between the two expression states and the gender differences during the posed smiles were compared. Meanwhile, the reproducibility of posed smile was statistically tested. RESULTS: Compared with the rest position, except for nasolabial angle (1.45°±7.65°), the measurements of 3D soft tissue in other region were changed in posed smile (P < 0.001). It should be noted that the eye region was also significantly changed (P < 0.001). Furthermore, the prominent feature of posed smile was that the alar base length became longer, the upper and lower vermilions were narrow and thin, and the mentolabial furrows became shallow. Meanwhile the chin extended anteriorly while the mouth retracted; During posed smile, the labial fissure asymmetry [2.78 (1.73, 3.49) mm], mid-infraorbital asymmetry [2.36 (1.22, 3.27) mm] and outercanthal asymmetry [2.31(1.29, 2.80) mm] were most apparent. Compared with the rest position, the asymmetry was not significantly increased except for cheilion and alar curvature points during the posed smile (P>0.05). In the posed smile, the changes of the right palpebral fissure height and the thickness of lower vermilion (|Li-Stoi| _z) of males were greater than those of females (P < 0.05), and asymmetry of exocanthion and cheekbone increased more than that of females (P < 0.05). There was no obvious difference between the two posed smiles. CONCLUSION In this study, during the posed smile the soft tissues of the eyes, cheeks, nose, lips and chin changed in different degrees, and the asymmetry of cheilion and alar curvature point was greater than that of the rest position; In addition, the reproducibility of posed smile was excellent, which can be a reference for clinical aesthetics and functional research of smile.
Humans ; Smiling/physiology* ; Female ; Male ; Adult ; Imaging, Three-Dimensional/methods* ; Face/anatomy & histology* ; Young Adult ; Facial Expression

Reach-to-grasp movements require integrating information on both object location and grip type, but how these elements are planned and to what extent they interact remains unclear. We designed a new experimental paradigm in which monkeys sequentially received reach and grasp cues with delays, requiring them to retain and integrate both cues to grasp the goal object with appropriate hand gestures. Neural activity in the dorsal premotor cortex (PMd) revealed that reach and grasp were similarly represented yet not independent. Upon receiving the second cue, the PMd continued encoding the first, but over half of the neurons displayed incongruent modulations: enhanced, attenuated, or even reversed. Population-level analysis showed significant changes in encoding structure, forming distinct neural patterns. Leveraging canonical correlation analysis, we identified a shared subspace preserving the initial cue's encoding, contributed by both congruent and incongruent neurons. Together, these findings reveal a novel perspective on the interactive planning of reach and grasp within the PMd, providing insights into potential applications for brain-machine interfaces.
Animals ; Motor Cortex/physiology* ; Hand Strength/physiology* ; Macaca mulatta ; Psychomotor Performance/physiology* ; Neurons/physiology* ; Male ; Cues ; Movement/physiology* ; Gestures

Social working memory (SWM)-the ability to maintain and manipulate social information in the brain-plays a crucial role in social interactions. However, research on SWM is still in its infancy and is often treated as a unitary construct. In the present study, we propose that SWM can be conceptualized as having two relatively independent components: "externally oriented SWM" (e-SWM) and "internally oriented SWM" (i-SWM). To test this external-internal hypothesis, participants were tasked with memorizing and ranking either facial expressions (e-SWM) or personality traits (i-SWM) associated with images of faces. We then examined the neural correlates of these two SWM components and their functional roles in empathy. The results showed distinct activations as the e-SWM task activated the postcentral and precentral gyri while the i-SWM task activated the precuneus/posterior cingulate cortex and superior frontal gyrus. Distinct multivariate activation patterns were also found within the dorsal medial prefrontal cortex in the two tasks. Moreover, partial least squares analyses combining brain activation and individual differences in empathy showed that e-SWM and i-SWM brain activities were mainly correlated with affective empathy and cognitive empathy, respectively. These findings implicate distinct brain processes as well as functional roles of the two types of SWM, providing support for the internal-external hypothesis of SWM.
Humans ; Memory, Short-Term/physiology* ; Male ; Female ; Empathy/physiology* ; Young Adult ; Magnetic Resonance Imaging ; Adult ; Brain/diagnostic imaging* ; Brain Mapping ; Facial Expression ; Social Behavior ; Facial Recognition/physiology* ; Social Perception ; Personality/physiology*

OBJECTIVES: This study aimed to analyze the correlations and proportional relationships between maxillary anterior teeth (MAT) and facial measurements in Han Chinese individuals with the most attractive smiles, as evaluated by dental professionals. METHODS: Ten dentists with more than 5 years of clinical experience from different professional directions in a tertiary stomatological hospital were selected to evaluate the smile attractiveness of volunteers by visual analogue scale (VAS). Eighty-eight Han volunteers with the most attractive smile were selected. The perceived width of the MAT, the dimensions (height and width) of the maxillary central incisors (MCI), and the facial dimensions (intercanthal distance, interzygomatic distance, interalar distance, facial height and lower facial height) of the volunteers were measured on the frontal photos of the smile, digital oral model, and 3D face model. Pearson correlation analysis was performed to analyze linear correlations, and regression analysis was carried out to explore the proportional relationships. Reliability analysis using the intraclass correlation coefficient verified the stability of these proportional relationships. In addition, the correlations between MAT perceived width and the proportional relationships of (MCI) height to width ratio, with facial dimensions were explored and their reliability was verified. RESULTS: In Han Chinese individuals with the most attractive smiles, as evaluated by dental professionals, the Pearson correlation coefficients among MAT perceived widths were 0.813, 0.389, and 0.560. A proportional relationship existed between the lateral incisor and central incisor, and the ratio was 0.729. No significant correlations were found between MCI and the inner canthal distance, zygomatic distance, interalar distance, facial height, or the lower one-third facial height except for a negative correlation (r=-0.357) between MCI height and facial height in males and a positive correlation (r=0.249) between MCI width and interalar width when genders were combined. CONCLUSIONS Correlations exist among MAT perceived widths in Han Chinese individuals with the most attractive smiles, as evaluated by dental professionals. Partial correlations are observed between MCI height and width and facial measurements. The perceived width of the lateral incisor can serve as a reference indicator for predicting the perceived width of the central incisor, providing a reference for the aesthetic restoration of MAT in the Han ethnicity population.
Adult ; Female ; Humans ; Male ; China ; Esthetics, Dental ; Face/anatomy & histology* ; Incisor/anatomy & histology* ; Maxilla/anatomy & histology* ; Smiling ; East Asian People

This study aims to address the limitations in gesture recognition caused by the susceptibility of temporal and frequency domain feature extraction from surface electromyography signals, as well as the low recognition rates of conventional classifiers. A novel gesture recognition approach was proposed, which transformed surface electromyography signals into grayscale images and employed convolutional neural networks as classifiers. The method began by segmenting the active portions of the surface electromyography signals using an energy threshold approach. Temporal voltage values were then processed through linear scaling and power transformations to generate grayscale images for convolutional neural network input. Subsequently, a multi-view convolutional neural network model was constructed, utilizing asymmetric convolutional kernels of sizes 1 × n and 3 × n within the same layer to enhance the representation capability of surface electromyography signals. Experimental results showed that the proposed method achieved recognition accuracies of 98.11% for 13 gestures and 98.75% for 12 multi-finger movements, significantly outperforming existing machine learning approaches. The proposed gesture recognition method, based on surface electromyography grayscale images and multi-view convolutional neural networks, demonstrates simplicity and efficiency, substantially improving recognition accuracy and exhibiting strong potential for practical applications.
Electromyography/methods* ; Neural Networks, Computer ; Humans ; Gestures ; Signal Processing, Computer-Assisted ; Machine Learning ; Pattern Recognition, Automated/methods* ; Algorithms ; Convolutional Neural Networks

Research on facial micro-expression analysis has been going on for decades. Micro-expression can reflect the true emotions of individuals, and it has important application value in assisting auxiliary diagnosis and disease monitoring of mental disorders. In recent years, the development of artificial intelligence and big data technology has made the automatic recognition of micro-expressions possible, which will make micro-expression analysis more convenient and more widely used. This paper reviews the development of facial micro-expression analysis and its application in forensic psychiatry, to look into further application prospects and development direction.
Humans ; Forensic Psychiatry ; Artificial Intelligence ; Mental Disorders/diagnosis* ; Facial Expression ; Emotions

OBJECTIVE: To automatically construct lips symmetry reference plane (SRP) based on posed smile, and to evaluate its advantages over conventional digital aesthetic design. METHODS: Eighteen subjects' three-dimensional facial and dentition data were gathered in this study. The lips SRP of experimental groups were used with the standard weighted Procrustes analysis (WPA) algorithm and iterative closest point (ICP), respectively. A reference plane defined by experts based on regional ICP algorithm, served as the truth plane. The angle error values between the lips SRP of WPA algorithm in the experimental groups and the truth plane were evaluated in this study, and the lips SRP of ICP algorithm of the experimental groups was calculated in the same way. The lips SRP based on posed smile as a reference for aesthetic design and evaluate preliminary clinical application. RESULTS: The average angle error between the lips SRP of WPA algorithm and the truth plane was 1.78°±1.24°, which was smaller than that between the lips SRP of ICP and the truth plane 7.41°±4.31°. There were significant differences in the angle errors among the groups (P < 0.05). In the aesthetic design of anterior teeth, automatically constructing the lips SRP of WPA algorithm based on posed smile and the original symmetry plane by re-ference compared with the prosthetic design, the subjects' scores on the lips SRP of WPA algorithm based on posed smile (8.48±0.57) were higher than those on the original symmetry plane (5.20±1.31). CONCLUSION Automatically constructing the lips SRP of WPA algorithm based on posed smile was more accurate than ICP algorithm, which was consistent with the truth plane. Moreover, it can provide an important reference for oral aesthetic diagnosis and aesthetic analysis of the restoration effect. In the aesthetic design of anterior teeth, automatically constructing the lips SRP of WPA algorithm based on posed smile can improve the patients' satisfaction in esthetic rehabilitation.
Esthetics, Dental ; Humans ; Lip ; Smiling ; Tooth ; Workflow

Surface electromyography (sEMG) is a weak signal which is non-stationary and non-periodic. The sEMG classification methods based on time domain and frequency domain features have low recognition rate and poor stability. Based on the modeling and analysis of sEMG energy kernel, this paper proposes a new method to recognize human gestures utilizing convolutional neural network (CNN) and phase portrait of sEMG energy kernel. Firstly, the matrix counting method is used to process the sEMG energy kernel phase portrait into a grayscale image. Secondly, the grayscale image is preprocessed by moving average method. Finally, CNN is used to recognize sEMG of gestures. Experiments on gesture sEMG signal data set show that the effectiveness of the recognition framework and the recognition method of CNN combined with the energy kernel phase portrait have obvious advantages in recognition accuracy and computational efficiency over the area extraction methods. The algorithm in this paper provides a new feasible method for sEMG signal modeling analysis and real-time identification.
Algorithms ; Electromyography ; Gestures ; Humans ; Neural Networks, Computer ; Signal Processing, Computer-Assisted