Is it okay to put a solar cell and battery into resin?
You can put these items into resin. Most resins will work, however polyester is more aggressive, and care must be taken. 

For example the battery is wrapped in a plastic shell, if this softens and there's a solder-spike connected to a positive (+) this may pierce the plastic and connect to the battery shell. Often the shell is negative (-) and this will create a short circuit.

Also badly mixed polyester is much more likely to get incredibly hot at the core, potentially severely damaging your battery chemistry.

The best choice is a low aggression resin like epoxy, to be sure. 

Make sure your electrical connections are well made, most cheap epoxies do shrink a little, by far not enough to stress the battery or solar cell beyond their limits - that is, if you do mix them accurately enough. If your connections are half hearted they may experience just enough stress to introduce unwanted extra resistance.
For example battery terminals can be plated for low-corrosion, which is hard to solder to, while initially it may cling, it is very easy to tear off. You're probably best off getting a battery with solder-tabs for battery packs or PCB mounting.

All that said, I have potted many many things containing any combination of electronics, including all of your mentioned parts, into cheap resins, epoxyd and others, and never had major failure.
If you want the "block" to stick neatly inside a box after curing, shrinking is an issue. You may need special potting compound, or, alternatively, get a bulk package of decent glue epoxy.