Atrophy patterns of deep gray matter nuclei in Alzheimer's disease and frontotemporal dementia

Background While cortical atrophy has been widely studied in dementia, emerging evidence highlights the role of subcortical degeneration, particularly in deep gray matter structures such as the thalamus, claustrum, and basal nuclei, in both Alzheimer's disease (AD) and frontotemporal dementia (FTD). However, disease-specific subcortical patterns remain undercharacterized. Objective To quantify deep gray matter atrophy across the AD continuum (mild cognitive impairment and AD) and three FTD subtypes (bvFTD, svFTD, PNFA), and to assess longitudinal atrophy, cognitive associations, and diagnostic classification. Methods We applied a novel segmentation pipeline (sTHOMAS) to T1-weighted MRI data from 380 participants in the ADNI cohort and 274 participants in the FTLDNI cohort, with longitudinal follow-up available for 237 participants. Group differences were assessed using ANCOVA (adjusted for age and sex), followed by post hoc comparisons and effect size estimation (Cohen's d). Neuropsychological associations were examined using partial correlations. A hierarchical Random Forest model was trained to classify diagnostic groups. Results Pronounced atrophy was observed in the mediodorsal, anteroventral, pulvinar thalamic nuclei, and nucleus accumbens, and claustrum in both AD and FTD, but was significantly greater in bvFTD. Longitudinal analysis revealed the fastest progression in bvFTD. Classification achieved 96.8% accuracy (AUC = 0.99) for AD versus FTD and 77.7% accuracy (AUC = 0.83) for PNFA versus svFTD. Subcortical atrophy correlated to executive, language, and semantic deficits. Conclusions Atrophy in the mediodorsal, pulvinar, anteroventral thalamic nuclei, nucleus accumbens, and claustrum distinguishes AD from FTD and differentiates FTD subtypes. These subcortical structures represent promising biomarkers for diagnosis and monitoring of neurodegeneration.