How can we tell whether a phenomena was a chemical reaction or not?
For some chemical reactions, it would be very difficult to prove without chemical analysis of the materials before and after the reaction. However, with modern analysis techniques, you could show whether a reaction took place. For example, if you tempered steel or converted diamond to graphite, analysis would show that it was a physical change or change in the crystal structure that occurred.  

For a low-tech example, consider the combustion of some condensed-phase hydrocarbon in oxygen. If you ignite the hydrocarbon under oxygen in a container sealed except for an oxygen source and an exit to a condenser, you would see the hydrocarbon vanish and water would appear in the condenser. For this we'll just ignore carbon dioxide production. You then easily confirm that the water is not just another form of the hydrocarbon by measuring it's freezing and boiling temperatures. You could then repeat the experiment without ignition of the hydrocarbon and see that you get nothing. 
This would confirm beyond any reasonable doubt that you've had a chemical reaction that converted the hydrocarbon to water.