 Phase II trial to study the effectiveness of combining gefitinib with fluorouracil, leucovorin, and irinotecan in treating patients who have advanced or recurrent colorectal cancer. Biological therapies such as gefitinib may stop the growth of tumor cells by blocking the enzymes necessary for tumor cell growth. Drugs used in chemotherapy use different ways to stop tumor cells from dividing so they stop growing or die. Combining gefitinib with fluorouracil, leucovorin, and irinotecan may kill more tumor cells 