Objectives: This study aimed to describe the difficulties faced by physicians in providing palliative care to patients with cancer in facilities without palliative care specialists. Methods: Semi-structured interviews were conducted with 11 physicians involved in cancer treatment who were affiliated with facilities having no palliative care specialists such as Diplomate or Board Certification of the Specialty Board of Palliative Medicine of the Japanese Society for Palliative Medicine. The interview data were analyzed using qualitative content analysis. Results: The participants had “difficulties in palliative care consultation” because they could not consult with specialists or medical staff at their own facility, when immediate response to cancer symptoms was required. This was partly due to “difficulties in regional cooperation”. In addition, the participants had “difficulties in alleviation of symptoms” for highly complex symptoms of patients with cancer. Behind these difficulties, there was “difficulties in foundation of providing palliative care for individual physicians” including limited time available to the participants. Conclusion: These findings show that there is a necessity to establish a continuous external consultation system for specialists to respond to the immediacy of changes in symptoms and highly complex symptoms.