Land use intensification is associated with loss of biodiversity and altered ecosystem functioning. Until now most studies on the relationship between biodiversity and ecosystem functioning focused on random loss of species, while loss of rare species that usually are the first to disappear received less attention. Here we test if the effect of rare microbial species loss on plant productivity depends on the origin of the microbial soil community. Soils were sampled from three land use types at two farms. Microbial communities with increasing loss of rare species were created by inoculating sterilized soils with serially diluted soil suspensions. After 8 months of incubation, the effects of the different soil communities on abiotic soil properties, soil processes, microbial community composition, and plant productivity was measured. Dilution treatments resulted in increasing species loss, which was in relation to abundance of bacteria in the original field soil, without affecting most of the other soil parameters and processes. Microbial species loss affected plant biomass positively, negatively or not at all, depending on soil origin, but not on land use history. Even within fields the effects of dilution on plant biomass varied between replicates, suggesting heterogeneity in microbial community composition. The effects of medium and severe species loss on plant biomass were similar, pointing toward a saturating effect of species loss. We conclude that changes in the composition of the soil microbial community, including rare species loss, can affect plant productivity, depending on the composition of the initial microbial community. Future work on the relation between function and species loss effects should address this variation by including multiple sampling origins.