The GTPase Rab11 plays key roles in receptor recycling, oogenesis, autophagosome formation, and ciliogenesis. However, investigating Rab11 regulation has been hindered by limited molecular detail describing activation by cognate guanine nucleotide exchange factors (GEFs). Here, we present the structure of Rab11 bound to the GEF SH3BP5, along with detailed characterization of Rab-GEF specificity. The structure of SH3BP5 shows a coiled-coil architecture that mediates exchange through a unique Rab-GEF interaction. Furthermore, it reveals a rearrangement of the switch I region of Rab11 compared with solved Rab-GEF structures, with a constrained conformation when bound to SH3BP5. Mutation of switch I provides insights into the molecular determinants that allow for Rab11 selectivity over evolutionarily similar Rab GTPases present on Rab11-positive organelles. Moreover, we show that GEF-deficient mutants of SH3BP5 show greatly decreased Rab11 activation in cellular assays of active Rab11. Overall, our results give molecular insight into Rab11 regulation, and how Rab-GEF specificity is achieved.