Abstract This study aimed to examine the associations of sedentary time, and substituting sedentary time with physical activity and sleep, with cardiometabolic health markers while accounting for a full 24 h of movement and non-movement behaviors, cardiorespiratory fitness (CRF), and other potential confounders. The participants were 4585 members of the Northern Finland Birth Cohort 1966, who wore a hip-worn accelerometer at the age of 46 years for 14 consecutive days. Time spent in sedentary behaviors, light-intensity physical activity (LPA), and moderate-to-vigorous-intensity physical activity (MVPA) were determined from the accelerometer and combined with self-reported sleep duration to obtain the 24-h time use. CRF was estimated from the peak heart rate in a submaximal step test. An isotemporal substitution paradigm was used to examine how sedentary time and substituting sedentary time with an equal amount of LPA, MVPA, or sleep were associated with adiposity markers, blood lipid levels, and fasting glucose and insulin. Sedentary time was independently and adversely associated with the markers of cardiometabolic health, even after adjustment for CRF, but not in partition models including LPA, MVPA, sleep, and CRF. Substituting 60, 45, 30, and 15 min/day of sedentary time with LPA or MVPA was associated with 0.2%–13.7% favorable differences in the cardiometabolic health markers after accounting for LPA, MVPA, sleep, CRF, and other confounders. After adjustment for movement and non-movement behaviors within the 24-h cycle, reallocating additional time to both LPA and MVPA was beneficially associated with markers of cardiometabolic health in middle-aged adults regardless of their CRF level.