What are theoretically sound programming languages for graph problems?
You might want to look at The Graph Programming Language GP. From the linked page:


  GP (for Graph Programs) is a rule-based, nondeterministic programming language for solving graph problems at a high level of abstraction, freeing programmers from handling low-level data structures. The core of GP consists of four constructs: single-step application of a set of conditional graph-transformation rules, sequential composition, branching and iteration. 


Sandra Steinert devoted her PhD thesis to the topic.

There's also a Hoare logic for reasoning about the correctness of such programs.