This study examined whether routine work time was associated with exercise time, moderate-to-vigorous physical activity (MVPA), and step counts (SC) among middle-aged and older adults. A 7-day diary survey was conducted with 158 adults, and 138 participated in the 1-year follow-up survey for measuring routine work time and exercise time. An accelerometer was used to measure MVPA and SC, and a questionnaire assessed perceived barriers and self-efficacy. Daily analyses revealed that while longer routine work time was associated with shorter exercise time after adjusting for perceived exercise barriers and exercise self-efficacy, it was associated with higher amounts of MVPA and SC. Longitudinal analysis showed that increased routine work time was associated with decreased exercise time and increased MVPA and SC. Changes in perceived barriers and self-efficacy did not mediate these associations. Actual lack of time would inhibit exercise behavior independently of perceived barriers and self-efficacy but elevates MVPA and SC.