Is there a method to provide physical stimulus to design inside FPGA like we do in VHDL test bench?
There is a Xilinx Core called VIO ( Virtual Input/Output) which uses the JTAG interface to drive and read values. Find more info here:
https://www.xilinx.com/products/intellectual-property/vio.html

I am not sure about an Altera equivalent though, if that's what you are after.

HTH