What does one need to know in order to build a software synthesizer?
Get Real Sound Synthesis for Interactive Applications, and download the Synthesis Toolkit.  That will give you practical introductions and quality source code for most digital synthesis methods in use.  Then pick whichever method you find most interesting, and start modifying the corresponding STK code to do what you want.  Then try building something similar from scratch.

In addition, like hotpaw2 suggested, anything from JOS is also great introductory material.  In particular, Introduction to Digital Filters with Audio Applications and Mathematics of the Discrete Fourier Transform both start from the basics, and don't assume a lot of background.

From a digital filter/effects perspective, another good starting point is to check out Faust.  You don't even have to learn their block-diagram signal processing language, you can just use the examples to dump out VST, LADSPA, etc. template code as quick way of getting something running [which you can then modify].  

That being said, it's even easier to start out with sound rendering, which is basically the same thing, but without worrying about real-time operation or interactivity.  You just save wav files to disk.  It's a great way to experiment with new techniques before going through the trouble of making them run quickly.