The breeding and non-breeding 'castes' of eusocial insects provide a striking example of role-specific selection, where each caste maximises fitness through different morphological, behavioural and physiological trait values. Typically, queens are long-lived egg-layers, while workers are short-lived, largely sterile foragers. Remarkably, the two castes are nevertheless produced by the same genome. The existence of inter-caste genetic correlations is a neglected consequence of this shared genome, potentially hindering the evolution of caste dimorphism: alleles that increase the productivity of queens may decrease the productivity of workers and vice versa, such that each caste is prevented from reaching optimal trait values. A likely consequence of this 'intralocus caste antagonism' should be the maintenance of genetic variation for fitness and maladaptation within castes (termed 'caste load'), analogous to the result of intralocus sexual antagonism. The aim of this review is to create a research framework for understanding caste antagonism, drawing in part upon conceptual similarities with sexual antagonism. By reviewing both the social insect and sexual antagonism literature, we highlight the current empirical evidence for caste antagonism, discuss social systems of interest, how antagonism might be resolved, and challenges for future research. We also introduce the idea that sexual and caste antagonism could interact, creating a three-way antagonism over gene expression. This includes unpacking the implications of haplodiploidy for the outcome of this complex interaction.