How to add mathematical rigor to systems research?
You sound like me five years ago! Here's what I did to become more theoretical.


Go back to basics.  I went back and relearned Calculus and Linear Algebra.  Watch YouTube videos on different subject matters. Mathematical maturity is the key.  You get this by reading lots of maths (any maths will help).   I also started learning Real Analysis.
Search Google for books/resources on 'how mathematicians think?'  Also learn how to do proofs.  There are plenty of books available.
Start reading theoretical CS papers/books, and learn a math tool; e.g., a branch of math programming such as linear program, non-linear, etc...


Best of luck with it.  Unfortunately, there is no short term solution, but the payoff is well worth the effort and frustration!

Proofing stuff in general is hard.  Otherwise, any man and his dog will be able to do it!  Books on how to construct proofs will help here.  They will get you into the correct mindset and also provide the framework that will allow you to 'hang' your proof.