OBJECTIVE: To characterize the molecular mechanism underlying the antineoplastic activity of Celastrus orbiculatus Thunb. extracts (COE). METHODS: The human hepatocellular carcinoma HepG2 cells with mammalian target of rapamycin (mTOR) knockdown expressed (HepG2/mTOR) were constructed using molecular biological technology. In vitro, the HepG2/mTOR cells were treated with COE at various concentrations (10, 20, 40, 80, 160 and 320 µ g/mL). Cell viability was determined using 3-(4, 5-dimethylthiazol-2-yl)-2, 5-diphenyltetrazolium bromide (MTT) assays. According to the half-maximal inhibitory concentration (IC) value (140 mg/L), the concentrations of COE in the subsequent experiment was set to alleviate cytotoxicity. The HepG2/mTOR cells were divided into 5 groups: negative control (untreated), COE treatment groups (40, 80, 120 mg/L COE) and positive control group (cisplatin, DDP, 2 mg/L), respectively. Wild-type HepG2 cells were used as a blank control. The effects of COE on the cell apoptosis were analyzed by flow cytometry and transmission electronic microscopy (TEM), respectively. The protein expression levels of mTOR signal pathways were determined by Western blotting. In vivo, HepG2/mTOR cells (2 × 10 cell/mice) were subcutaneously injected into the right flank of nude mice. Thirty-six female nude mice were randomly assigned to 6 groups according to body weight (6 mice per group) as follows: solvent vehicle control, Banmao Capsule treated group (BM, 195 mg/kg), Tegafur, Gimeracil and Oteracil Potassium Capsules (10 mg/kg) treated group, and different dosages of COE (10, 20, 40 mg/kg) groups. Tumor growth was monitored and immunohistochemical staining was used to examine the expression of apoptosis-related proteins in tumor tissues. RESULTS: COE inhibited the proliferation significantly in a concentration-dependent manner in HepG2/mTOR cells (P<0.01). COE significantly induced the apoptosis of HepG2/mTOR cells (P<0.01), and the apoptotic bodies can be observed under TEM. COE significantly inhibits the proteins expression of mTOR-related signal pathways. In vivo, COE significantly inhibited tumor growth in nude mice (P<0.01). Moreover, the results showed that COE down-regulated the expression of Bcl-2 and Bcl-xL, and up-regulated the levels of Bax and caspase-3 protein (P<0.01). CONCLUSION COE was a potential chemotherapeutic drug in HCC treatments via targeting mTOR signal pathway.