 This phase I trial studies the side effects and best dose of pomalidomide when given together with dexamethasone in treating patients with primary central nervous system lymphoma that has come back relapsed or does not respond to treatment refractory or intraocular lymphoma that is newly diagnosed, relapsed or refractory. Pomalidomide may stimulate the immune system to kill cancer cells. Drugs used in chemotherapy, such as dexamethasone, work in different ways to stop the growth of cancer cells, either by killing the cells, stopping them from dividing, or by stopping them from spreading. Giving pomalidomide together with dexamethasone may kill more cancer cells. 