The etiology of late onset Alzheimer’s disease (LOAD) depends on multiple factors, among which the APOE ε4 allele is the most adverse genetic determinant and conscientiousness represents an influential personality trait. A potential association of both factors with brain structure in young adulthood may constitute a constellation that sets the course toward or against the subtle disease progression of LOAD that starts decades before clinical manifestation. Hence, in the present study, we examined the modulating effects of APOE ε4 on the relation between personality dimensions, including conscientiousness, and total grey matter volume (GMV) in young healthy adults using an a priori genotyping design. 105 participants completed an inventory assessing the Five Factor Model of Personality (NEO-FFI) and a structural MRI scan. Total GMV was estimated using both Freesurfer as well as VBM8. Across all participants, total GMV was positively associated with extraversion and negatively related to age. In APOE ε4-carriers—but not in APOE ε4-non-carriers—conscientiousness was negatively associated with total GMV. In line with the hypothesis of antagonistic pleiotropy of the APOE ε4 allele, this result suggests that young APOE ε4-carriers with increased total GMV may particularly benefit from cognitive advantages and thus have a lower need to engage in conscientious behavior. In this subset of young APOE ε4-carriers, the reduction in conscientiousness could then bring along adverse health behavior in the long run, potentiating the risk for LOAD. Hence, young APOE ε4-carriers with increased total GMV may be at a particularly high risk for LOAD.