How can I formally verify fuzzy commitment scheme based security protocol? I am currently working on designing a security protocol which involves usage of fuzzy commitment schemes, for. eg Reed-Solomon codes which allows us to tolerate a certain level of error. 

I was wondering how one would go about formally verifying a security protocol which would use a scheme such as Reed-Solomon codes.

I searched online, and found out about tools out there such as AVISPA. ProVerify and even Casper/FDR which would help me test security protocols, however they do not seem to work with fuzzy commitment schemes.
This might not be the answer you are looking for, but as you are looking for a formal verification, I would advise you to take a look at Coq. Even though mainly used by Academics, it provides a logical framework and an interface to write formal and interactive proofs.

Based this language there exists some libraries dedicated to cryptographic proof :


CertiCrypt
EasyCrypt (tutorial)


Example of proof : 


Full Proof Cryptography: Verifiable Compilation of Efficient Zero-Knowledge Protocols
Formal certification of ElGamal encryption
Verification of a Cryptographic Primitive: SHA-256


PHD : Formal certification of game-based cryptographic proofs

Other formal tools (interactive provers) can be used such as Isabelle, Agda, Fstar and HOL.
And some more readings :


Computer Verification in Cryptography
Probabilistic Relational Verification for Cryptographic Implementations




Disclaimer: I use Coq on daily basis...