In the United States, about 40,000 new cases of oral cancer are diagnosed each year and nearly 7,800 patients died from it in 2012. Omega-3 polyunsaturated fatty acids have been found to have anticancer effects in a variety of cancer cell lines and animal models, but their effect in oral cancer remains unclear. This study was designed to examine the effect of docosahexaenoic acid (DHA, a kind of omega-3 fatty acid) on oral cancer cells and the molecular mechanism of its action. We found that exposure of squamous cell carcinoma-4 (SCC-4) and squamous cell carcinoma-9 (SCC-9) human oral cancer cells to DHA induced growth inhibition in a dose- and time-dependent manner. Meanwhile, in addition to the elevated levels of apoptotic markers, such as cleaved PARP, subG1 portion and TUNEL-positive nuclei, DHA led to autophagic vesicle formation and an increase in autophagic flux, indicating the involvement of both apoptosis and autophagy in the inhibitory effects of DHA on oral cancer cells. Further experiments revealed that the apoptosis and autophagy induced by DHA were linked to inhibition of mammalian target of rapamycin (mTOR) signaling by AKT inhibition and AMP-activated protein kinase (AMPK) activation in SCC-9 cells. Together, our results suggest that DHA induces apoptosis- and autophagy-associated cell death through the AMPK/AKT/mTOR signaling pathway in oral cancer cells. Thus, utilization of omega-3 fatty acids may represent a promising therapeutic approach for chemoprevention and treatment of human oral cancer.