How do the stored chemicals in batteries drive electronic current from their terminal to outside devices?
A battery, at its heart, is what we call a redox cell: some compounds are stripped of their electrons, and others gain those electrons. The movement of electrons between these two compounds can be harnessed to provide electricity, but that's taking us a step too far. What we want to understand here is why the electrons move. To do this, we'll have to understand the redox cell. As much as I hate to assume things, you're going to have to understand a bit about ions and equilibrium to understand this explanation.

(Note: the explanation I'm about to use is heavily taken from the excellent Chemguide site, run by Jim Clark. I'll include a link at the end.)

Let's say I place a bar of magnesium in a water bath. I'll take a look at the magnesium bar a short while after I place it in the water. If I could visualize atoms and electrons, this is what I'd see:



Some of the atoms of magnesium leave the bar and become ions in the solution, solvated by water (and other things, but that's not crucial to this explanation). Since they become ions, they leave electrons on the metal bar. Because the ions are positively charged, they are attracted to the negative charge on the metal bar, and tend to hang around there.

If I wait a little longer, I'll see more and more ions coming off the bar. As they start to build up in the water around the metal, every once in a while, one of the ions will hit the bar and reclaim its electrons. This is what we'll see then:



If I wait long enough, eventually, the number of atoms coming off the metal will be equal to the number of ions reabsorbing onto the metal. If I take out the "bites" in the metal bar to simplify the situation, this is what I see:



Notice how there's quite a buildup of electrons on the magnesium. This is because magnesium is a very active metal, like lithium. What if we used a much less active metal, one that doesn't have a large propensity to lose atoms into solution? An example of such a metal would be copper. Here's a side-by-side of the two:



Because the magnesium is a lot more active, it tends to lose ions more, and so it has more electron buildup. If we connect the two with a wire, we give the electrons a way to flow from the high concentration to the low concentration. We can then place something along the wire in the middle (say, a lightbulb) and harvest some energy!



This, of course, is a hilariously oversimplified scheme. For one, as the electrons go to the copper, more and more magnesium ions break off to replace them. At some point, there are so many positive ions in the solution that they draw the electrons back toward them, through electrostatic attractions (which is a fancy way of saying that positive and negative charges attract each other). To solve this, we'd put a salt bridge between the solutions that lets ions migrate back and forth. 

There are other problems too: for example, this setup wouldn't work in real life because there are no ions in solution: real cells use salt solutions of the metal e.g. $\ce{MgCl2}$ or $\ce{CuSO4}$. There's also the issue that this battery is liquid and wouldn't do so well if you tipped it over, the fact that the voltage (which is really the difference between the charge on the metal bars) is only about 1.1 volts, yada yada. That's not central to the explanation.

The fact that certain compounds are more likely to gain electrons and some are more likely to lose them is the central reason why things like batteries can exist. If you'd like to read more about it, I pulled this explanation from chemguide.co.uk, run by Jim Clark. I am currently ready to graduate with a B.Sc in chemistry, and his website is still one of the best resources for conceptual learning that I've come across---as a matter of fact, I learned about things on that website (eutetic mixtures and azeotropes) that I still have yet to come across in four years of university-level study. If you'd like to read more into redox reactions, the redox equilibria page is at this link right here.