Storing An Led's Previous state when power is removed with eeprom now
With the components that you have listed, you will not be able to build a system that will remember the state of a LED after power-off. The EEPROM that you listed can hold a value without any power, but the problem is that it has a (relatively) complex input/output protocol (I2C) compared to the other parts you have on hand.

I think you will find that there are very few methods for storing data that are both persistent while power is removed, and electrically simple enough that you could build it with the parts you have. 



Given your requirements, a toggle switch is still probably the simplest method for storing a state, followed by some arrangement of latching relays.

If you wanted to store the LED state on the EEPROM you purchased, then the easiest way to interface with that device is to use a microcontroller. For example, you could get Texas Instrument's Launchpad board for around $5. But, many microcontrollers have built-in storage that would be more than adequate for storing the state of a LED.

You could "fake" your requirements by making a small, 1-bit memory cell (SR-Latch) and powering just the latch with a coin-cell battery. When you remove power, you would then only remove the power to the LED, not the SR latch, and this could last a long time. You would need to use low-current parts (CMOS), but this would be a very easy method as well.