Is there any way that we can perform speech recognition without using Fourier transforms?
All recognition tasks (doesn't even have to be speech recognition) are reductions of a very high-dimensional signal (your speech recording's dimension is the number of audio samples!) to a low-dimensional signal. 

As such, it is generally advisable to transform the input signal through an easy operation to a representation where the dimensionality can be more easily reduced. For speech, the frequency domain is such a representation – physically, only a very limited set of discrete frequencies is sufficient to make up speech. 

Hence, having a frequency-domain transform (typically, in speech recognition) early on in the process makes a lot of sense: after the transform, only very few coefficients contain information, and need to be evaluated further.

Hence, since speech models are based on the physics of speaking, and that is a time-frequency physical problem, it's very likely that an algorithm that actively tries to stay away from frequency domain will not perform good.

Nowadays, however, many classification problems are solved using Deep Neural Networks; these basically are just a network of nonlinear functions with a constant factor and offset for each of them. Training such a Neural Network to do something useful is just finding the right factors and offsets. There's heuristics that allow us to do that automatedly at large scale.

Such networks can of course be trained directly on discrete time-domain speech signals, and might have good recognition rates after sufficient training has happened. 

Notice that training such Neural Networks is not a trivial task in itself; it's usually an intentionally or inherently stochastic procedure, and might or might not lead to a functioning classifier. There's a lot of experience and luck involved building a good Neural Network classifier instead of just "something that does something close to what I want".

However, the size and hence cost, and the effort spent on training, and the chance that learning doesn't lead to a working result, can be influenced by the way the signal is preprocessed. Think of it like this: Imagine you'd have to learn a language by listening to someone through a bad phone connection, compared to having a good phone at each end of the line that does some error correction and sound enhancements. 

A typical preprocessing step here would be some sort of frequency-domain transform. Simply because the result of that more compactly describes the speech content than the time representation, it'll also make the Neural Network training's job of finding important and unimportant parts of the signal easier.

So, to conclude:

Yes, you can avoid the extremely simple Fourier transform (or one of the many other frequency-domain representations, many of which are based on the Fourier transform), it would probably make your life much harder than necessary.