Degrading face stimuli reduces face discrimination in both young and older adults, but the brain correlates of this decline in performance are not fully understood. We used functional magnetic resonance imaging to examine the effects of degraded face stimuli on face and non-face brain networks and tested whether these changes would predict the linear declines seen in performance. We found decreased activity in the face network (FN) and a decrease in the similarity of functional connectivity (FC) in the FN across conditions as degradation increased but no effect of age. FC in whole-brain networks also changed with increasing degradation, including increasing FC between the visual network and cognitive control networks. Older adults showed reduced modulation of this whole-brain FC pattern. The strongest predictors of within-participant decline in accuracy were changes in whole-brain network FC and FC similarity of the FN. There was no influence of age on these brain-behavior relations. These results suggest that a systems-level approach beyond the FN is required to understand the brain correlates of performance decline when faces are obscured with noise. In addition, the association between brain and behavior changes was maintained into older age, despite the dampened FC response to face degradation seen in older adults.