We report a very rare case of granular cell tumor with a center ulcer of the rectum, which was detected in a 47-year-old man during screening colonoscopy. It is difficult to distinguish granular cell tumor from other subepithelial tumors. However, our case had a center ulcer unlike previous cases. We confirmed the diagnosis using histological and immunohistochemical examinations. In addition to our case report, we discuss the morphology, size, layer of invasion, and treatment of rectal granular cell tumors based on a literature review.