 RATIONALE: Drugs used in chemotherapy, such as capecitabine, work in different ways to stop the growth of tumor cells, either by killing the cells or by stopping them from dividing. Giving radiation therapy that uses a 3-dimensional 3-D image of the tumor to help focus thin beams of radiation directly on the tumor, and giving radiation therapy in higher doses over a shorter period of time, may kill more tumor cells and have fewer side effects. Monoclonal antibodies, such as panitumumab, can block tumor growth in different ways. Some block the ability of tumor cells to grow and spread. Others find tumor cells and help kill them or carry tumor-killing substances to them. It is not yet known whether giving capecitabine together with 3-D conformal radiation therapy is more effective with or without panitumumab in treating patients with advanced rectal cancer. 