 PURPOSE: This randomized phase II trial is studying giving capecitabine together with radiation therapy to see how well it works with or without panitumumab in treating patients with advanced rectal cancer. 