How is Bitcoin governed by mathematics? I keep hearing talks from Bitcoin evangelists that claim Bitcoin isn't governed by any central authority, rather it's governed by mathematics.

What exactly do they mean when they say "governed by mathematics"?  

That the network is secure by means of math instead of some central authority (e.g. a bank overseeing everything)? If so, what math?  The math that makes PoW possible?
This question may be opinion based in some ways, but I'm going to attempt to answer it as it is an important concept, and somewhat unlike traditional systems.

Bitcoin is governed by math, in some places. At the end of the day, it's software, and even the math rules can be updated to use different rules, or removed altogether by means of a soft or hard fork (depending on how drastic the change is).

As for the math, there are primarily three sections where it is in play.

The first is mining - the hash function is used to calculate a value, which is compared against the mining target.

The second is signatures - elliptic curve signatures are used to enforce utxo ownership (for standard outputs anyways).

The third is the block reward - The Bitcoin block reward halves every 210,000 blocks, lea

That said, none of these are particularly math-y. As with any computer program, math is used to enforce certain functionality.

It may be more appropriate to say that Bitcoin is governed by code - The code is ultimately what decides things such as block reward halving, signature technicalities, and mining process. As the abundance of altcoins shows, there are many permutations and combinations on all of these parameters that still result in valid and functional networks. Even Bitcoin has gone through a series of network upgrades that have altered some of these rules, such as moving from an infinite supply to a fixed supply.

As chytrik mentioned in the comments, the infinite supply fix was actually a bug fix (the infinite supply was a side effect of C++ behaviour, and would not have occurred for equivalent code in many other languages). 

Bitcoin can indeed be looked at as a model of social governance governing the code, and the code governing the network.