Introduction: Physical inactivity is common among persons with disabilities (PWDs), thereby leading to harmful secondary complications. Therefore, this cross-sectional study aimed to identify the barriers to physical activity and exercise amongst PWDs in Malaysia. Methods: A total of 102 adults (age 15–65 years) with physical disabilities were recruited from a government-funded hospital. The participants completed the ‘Barriers to Physical Exercise and Disability’ questionnaire via interview by telephone calls or a one-to-one approach. Results: Most participants (90.2%) were interested in engaging in exercise programs. However, the mean concern index score was 2.83 ± 1.35. The majority of the respondents identified five major barriers. These barriers include health concerns (36.3%), transport barriers (36.3%), lack of energy and motivation (36.3%, 31.4%), and exercise program costs (23.54%). Conclusion: Many participants were interested in beginning an exercise program despite the barriers. This observation shows that if the barrier is eliminated, then, this community would be able to participate in an exercise program regularly. Results from the study can inspire health care providers to devise strategies for the promotion of physical activity participation and long-term adherence between PWDs.