Endocytic cargo and Rab GTPases are segregated to distinct domains of an endosome. These domains maintain their identity until they undergo fission to traffic cargo. It is not fully understood how segregation of cargo or Rab proteins is maintained along the continuous endosomal membrane or what machinery is required for fission. Endosomes form contact sites with the endoplasmic reticulum (ER) that are maintained during trafficking. Here, we show that stable contacts form between the ER and endosome at constricted sorting domains, and free diffusion of cargo is limited at these positions. We demonstrate that the site of constriction and fission for early and late endosomes is spatially and temporally linked to contact sites with the ER. Lastly, we show that altering ER structure and dynamics reduces the efficiency of endosome fission. Together, these data reveal a surprising role for ER contact in defining the timing and position of endosome fission.