The factor structure of the Eating Disorder Examination-Questionnaire (EDE-Q) has proven difficult to replicate, including in vegans, whose eating behaviors differ from omnivores in important ways. We sought to assess fit of data from vegans and omnivores with the most recently proposed brief three-factor model of the EDE-Q, which retains only seven of the original 28 EDE-Q items. We examined fit indices of the EDE-Q brief three-factor model in vegans (i.e., individuals refraining from all animal products, n = 318) and omnivores (i.e., individuals not restricting intake of animal products, n = 200) in single-group confirmatory factor analyses (CFA). Configural and metric invariance across the two groups was examined in multi-group CFA. Data from omnivores exhibited good model fit. Fit in vegans was slightly worse, but still adequate and superior to alternative models. Findings from multi-group CFA supported configural, but not metric invariance across the two groups. We document satisfactory fit of data from vegans and omnivores with the EDE-Q brief three-factor model, suggesting that it is better suited for quantifying disordered eating than the original four-factor, full three-factor, or alternative two-, full one-, and brief one-factor versions, including in individuals who abstain from animal products.