High capacity  - EEPROM chip
At higher capacities, SPI flash is way cheaper than eeprom, with capacities in the 1-32 megabit range. Main difference is it is block, not byte eraseable and 3.3v only. ALways SPI, not I2C, but I2C gets too slow for large memories anyway.
For gb capacites you're looking at NAND flash, &amp; you need to deal with pretty large erase blocks, ECC, page-at-a-time writes and bad-block mapping, so SD will probably be easier &amp; cheaper as it does ECC internally.