Local adaptation in hermaphrodite species can be based on a variety of fitness components, including survival, as well as both female and male sex-functions within individuals. When selection via female and male fitness components varies spatially (e.g. due to environmental heterogeneity), local adaptation will depend, in part, on variation in selection through each fitness component, and the extent to which genetic trade-offs between sex-functions maintain genetic variation necessary for adaptation. Local adaptation will also depend on the hermaphrodite mating system because self-fertilization alters several key factors influencing selection and the maintenance of genetic variance underlying trade-offs between the sex-functions (sexually antagonistic polymorphism). As a first step to guide intuition regarding sex-specific adaptation in hermaphrodites, we develop a simple theoretical model incorporating the essential features of hermaphrodite mating and adaptation in a spatially heterogeneous environment, and explore the interaction between sex-specific selection, self-fertilization and local adaptation. Our results suggest that opportunities for sex-specific local adaptation in hermaphrodites depend strongly on the extent of self-fertilization and inbreeding depression. Using our model as a conceptual framework, we provide a broad overview of the literature on sex-specific selection and local adaptation in hermaphroditic plants and animals, emphasizing promising future directions in light of our theoretical predictions.