How to go about designing a custom airfoil?
Wing sweep, incidence, and airfoil geometry will all have to be adjusted to achieve the best performance at a certain air speed, density, and Reynolds number.

The software that egid mentioned is a good place to start. While studying aerospace in college, XFOIL was used for basic airfoil analysis in our classes.

Another program used was AVL. This analyzes the aircraft as a whole and can model other things such as dynamic stability.

I'm glad that XPlane was also mentioned, even though I wouldn't consider it the most scientific tool for testing wing geometry. It is an interesting way to test configurations since it does use the airplane geometry to simulate the performance characteristics.

In order to consider more variables such as sweep, dihedral, and different airfoils over the wingspan, computational fluid dynamics (CFD) software is used. ANSYS Fluent is one example. There are many other packages out there, and anyone doing serious commercial airfoil design will have proprietary software and methods for this.

Of course, even in the days of supercomputers and CFD, wind tunnel testing is still very important. Scale models can be tested to validate the computer models and demonstrate performance.

As far as requirements, they can get very complicated. For aircraft, of course, the primary goal is to provide the lowest drag at cruise conditions. However, many other factors include:


Preventing flutter
Provide room for sufficient supporting structure
Provide room for fuel 
Methods to manufacture the shape
Good stall characteristics
No bad aerodynamics with fuselage and engines
Enough lift with flaps for low landing speed
Able to accommodate flaps, slats, spoilers, and ailerons
Tip treatment like winglets/raked tips
Wingspan must fit into available space (airport, hangar, aircraft carrier, etc.)


And probably many more. Of course, the air speed and conditions will also affect the fuselage, empennage (tail) and its control surfaces, and engines, so this will also be taken into account.