Objective To investigate the effects of Niuhuang (Bovis Calculus, BC) and Shexiang (Moschus) (BC-Moschus) on human hepatocellular carcinoma (HCC) cells SMMC-7721 and a nude mouse model of subcutaneous xenografts, and to explore its anti-HCC mechanism. Methods The BC-Moschus combination was applied to two liver cancer models in vivo and in vitro. SMMC-7721 was divided into the BC-Moschus group and the control group, and different doses (rude drug dosage 0.625, 1.25, 2.5, and 5 mg/mL) of BC-Moschus extract were used for the intervention. The proliferation ability of HCC cells was detected using the Cell Counting Kit-8 (CCK-8) assay, and the migration ability was detected by a wound healing assay. A subcutaneous xenograft model was prepared using nude mice with human HCC. Specific pathogen-free-grade BALB/c nude mice (5-week-old) were randomly divided into the following groups (n = 6 per group): control (0.9% physiological saline 0.2 mL/d), BC-Moschus [BC 45.5 mg/(kg·d)+ Moschus 13 mg/(kg·d)], and cisplatin (DDP, intraperitoneal injection 5 mg/kg per week) groups. All groups were administered for 14 d. The volume and mass of the subcutaneous xenografts in nude mice were observed. The expression levels of phosphatidylinositol-3 kinase/protein kinase B/mammalian target of rapamycin (PI3K/AKT/mTOR) pathway, apoptosis-associated factor p70 S6 Kinase (S6K), Bax, Bcl-2, caspase-3, and caspase-9 in nude mice subcutaneous xenografts were measured by real-time quantitative PCR (RT-qPCR) and Western blot. Terminal Deoxynucleotidy Transferase-Mediated dUTP Nick-End Labeling (TUNEL) was used for quantitative analysis of apoptotic cells. Results The CCK-8 assay demonstrated that the BC-Moschus combination inhibited HCC cell proliferation in a superior manner to the use of BC and Moschus alone, and the inhibition effect was dose- and time-dependent (P < 0.01). The wound healing assay showed that the BC-Moschus combination inhibited HCC cell migration (P < 0.01). In the subcutaneous xenograft model of nude mice with human HCC, we found that the tumor volume and weight of the BC-Moschus group were lower than those of the control group (P < 0.01). The levels of the PI3K/AKT/mTOR signaling pathway and S6K protein in the BC-Moschus and DDP groups were significantly decreased (P < 0.01). The expression level of the anti-apoptotic gene Bcl-2 was downregulated (P < 0.05), and the expression of the pro-apoptotic gene Bax and apoptosis-related factors caspase-3 and caspase-9 were significantly upregulated (P < 0.01). The TUNEL assays further confirmed that the combination of the BC-Moschuas could promote HCC (P < 0.01). Conclusion The BC-Moschus combination inhibited the proliferation and migration ability of HCC cells SMMC-7721 and effectively inhibited the growth of subcutaneous xenografts in nude mice. The mechanism may be closely related to the downregulation of the PI3K/AKT/mTOR pathway, regulation of apoptosis-related protein caspase-3, caspase-9, Bcl-2, and Bax expression, and promotion of apoptosis.