How are Vivado's projects directories structured?
Don't add the project itself to git; instead add some sort of script to generate the project and required IP cores. There are all sorts of issues you'll run in to if you commit the actual project, the least of which being absolute paths in various places, not to mention a hard version dependence on Vivado. It should be possible to have Vivado export a TCL script to generate the project, so just commit that and use it to regenerate the whole project. Alternatively, it is possible to use some other build framework such as make or cmake to write out TCL scripts and run them in Vivado to do much the same thing. 

Here is how I usually do it with makefiles:

make file: https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/fpga/Makefile

vivado.mk that the above makefile includes: https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/common/vivado.mk