How do I cut a popsicle stick lengthwise?
Your question is limited by not having an indication of the resources available to you. One of the better tools would be a scroll saw, often called a jig saw. The tabletop versions are well suited to light duty cuts such as popsicle sticks.

Another suitable tool, somewhat unwieldy, is the rotary hobby tool, often called a dremel tool, from the brand name frequently associated with this type of item. There are various bits available for this device, but it would be challenging to handle safely for such small pieces.

Only recently, there appears on the 'net a number of videos showing people using either a rotary tool such as the dremel, or a hand drill and even a right-angle disc grinder. The latter is seriously overpowered for this application, but the key feature is that the cutting "bit" is a circle of paper. 

Multiple circles of the same diameter are cut, glued together using ordinary adhesive and attached to the rotary device. The faster the speed one can manage, the better the results. Even wood as thick as 25 mm (1 inch) is shown being cut with a paper disc!

I used The YouTube, searched for "cutting with paper" and found a number of returns. One of them shows the rotary tool at the very start. I did not view the entire video, having seen others representative of this method.

Cutting with paper

EDIT: An additional method of attacking the sticks would be to use a box knife or similar sharp instrument. Place a solid straight edge on the stick and score repeatedly along the edge. Use as much force as necessary to cut into the surface, but not so much as to crack the stick along grain lines.

If you can ensure exact positioning, consider to score one side then the other before attempting to split the stick in two. It may be necessary to alternate sides when scoring in order to slice completely through the stick.