Who makes the call to start in terrible weather conditions?
Who else might make that call? The airline?


Yes, if the airline thinks the weather is too bad at a particular airport for safe operations, it can just cancel all flights from that airport without asking the pilots.


  The airport?


If the airport closes completely, then it is closed. No flights can depart any more, even if the pilots consider it safe.


  Is it really only up to the pilot to make this decision?


If neither the airline cancelled the flight nor the airport has closed, it is indeed up to the pilots, or more specifically to the PIC (Pilot In Command, the Captain on an airline flight). They are ultimately responsible for the safe operation of the aircraft and have to make the decision:


  4.5.1 The pilot-in-command shall be responsible for the safety of all crew members, passengers and cargo on board when
  the doors are closed. The pilot-in-command shall also be responsible for the operation and safety of the aeroplane from the
  moment the aeroplane is ready to move for the purpose of taking off until the moment it finally comes to rest at the end of the
  flight and the engine(s) used as primary propulsion units are shut down.


(ICAO Annex 6 Part 1 - 4.5 Duties of pilot-in-command)

The SIC (Second In Command) also gets a say in this because of crew resource management, meaning if they think it is unsafe to depart, they can and should speak up.


  Are there official rules about departing in such conditions?


The respective national aviation authority may have rules on when a departure is permitted, but this will depend on the exact location. The pilots will however have received rules and training from their airline, which should indicate when a departure is allowed. The most important aspect to consider are the aircraft limitations. There is no legal crosswind limit for a takeoff, but e.g. the Boeing 737 FCOM lists a demonstrated value:


  The maximum demonstrated takeoff and landing crosswind is 33 knots.


(Boeing 737 NG FCOMv1 1.10.2 Limitations - Operating Limitations)

It would be considered unsafe to depart with a stronger crosswind.

Note that limitations for landing are typically stricter than for departure and the pilots are legally required to check the expected weather at the destination airport before departing.

In Europe, Eurocontrol can also delay a departure due to expected weather en-route or at the destination airport. See What is the difference between en-route delays and airport delays?.