Objective This study aimed to explore the characteristics and factors related to changes in cognitive function in vulnerable individuals with cognitive impairment during the coronavirus disease 2019 (COVID-19) pandemic. Methods Among patients who visited a local university hospital with subjective cognitive complaints, those who had been tested for cognitive function at least once after the onset of COVID-19 and tested regularly at least three times within the last 5 years were included (1st, the initial screening; 2nd, the test immediately before the COVID-19 pandemic; 3rd, the most recent test after the pandemic). Finally, 108 patients were included in this study. They were divided into groups according to whether the Clinical Dementia Rating (CDR) was maintained/improved and deteriorated. We investigated the characteristics of the changes in cognitive function and related factors during COVID-19. Results When comparing CDR changes before and after COVID-19, there was no significant difference between the two groups (p=0.317). Alternatively, the main effect of the time when the test was conducted was significant (p<0.001). There was also a significant difference in the interaction between the groups and time. When the effect of the interaction was analyzed, the CDR score of the maintained/ improved group significantly decreased before COVID-19 (1st–2nd) (p=0.045). After COVID-19 (2nd–3rd), the CDR score of the deteriorated group was significantly higher than that of the maintained/improved group (p<0.001). Mini-Mental State Examination recall memory and changes in activity during COVID-19 were significantly associated with CDR deterioration. Conclusion Memory dysfunction and decreased activity during the COVID-19 pandemic are strongly related to the deterioration of cognitive impairment.