What are the models that have the potential to replace neural networks in the near future?
This is going backwards, but it kind of follows the logic of the arguments.
In terms of efficiency, I can see a few major problems with classical neural networks.
Data collection and preprocessing overhead
Large neural networks require a lot of data to train. The amount can vary depending on the size of the network and the complexity of the task, but as a rule of thumb it is usually proportional to the number of weights. For some supervised learning tasks, there simply isn't enough high-quality labelled data. Collecting large amounts of specialised training data can take months or even years, and labelling can be cumbersome and unreliable. This can be partially mitigated by data augmentation, which means &quot;synthesising&quot; more examples from the ones you already have, but it is not a panacea.
Training time vs. energy tradeoff
The learning rate is usually pretty small, so the training progress is slow. A large model that could take weeks to train on a desktop CPU can be trained in, say, two hours by using a GPU cluster which consumes several kW of power. This is a fundamental tradeoff due to the nature of the training procedure. That said, GPUs are getting increasingly efficient - for example, the new nVidia Volta GPU architecture allows for 15.7 TFLOPs while consuming less than 300 W.
Non-transferrability
Right now, virtually every different problem requires a custom neural network to be designed, trained and deployed. While the solution often works, it is kind of locked into that problem. For example, AlphaGo is brilliant at Go, but it would be hopeless at driving a car or providing music recommendations - it was just not designed for such tasks. This overwhelming redundancy is a major drawback of neural networks in my view, and it is also a major impediment to the progress of neural network research in general. There is a whole research area called transfer learning which deals with finding ways of applying a network trained on one task to a different task. Often this relates to the fact that there might not be enough data to train a network from scratch on the second task, so being able to use a pre-trained model with some extra tuning is very appealing.

The first part of the question is more tricky. Leaving purely statistical models aside, I haven't seen any prominent approaches to machine learning that are radically different from neural networks. However, there are some interesting developments that are worth mentioning because they address some of the above inefficiencies.
Neuromorphic chips
A bit of background first.
Spiking neural networks have enormous potential in terms of computational power. In fact, it has been proven that they are strictly more powerful than classical neural networks with sigmoid activations.
Added to that, spiking neural networks have an intrinsic grasp of time - something that has been a major hurdle for classical networks since their inception. Not only that, but spiking networks are event-driven, which means that neurons operate only if there is an incoming signal. This is in contrast to classical networks, where each neuron is evaluated regardless of its input (again, this is just a consequence of the evaluation procedure usually being implemented as a multiplication of two dense matrices). So spiking networks employ a sparse encoding scheme, which means that only a small fraction of the neurons are active at any given time.
Now, the sparse spike-based encoding and event-driven operation are suitable for hardware-based implementations of spiking networks called neuromorphic chips. For example, IBM's TrueNorth chip can simulate 1 million neurons and 256 million connections while drawing only about 100 mW of power on average. This is orders of magnitude more efficient than the current nVidia GPUs. Neuromorphic chips may be the solution the training time / energy tradeoff I mentioned above.
Also, memristors are a relatively new but very promising development. Basically, a memristor is a fundamental circuit element very similar to a resistor but with variable resistance proportional to the total amount of current that has passed through it over its entire lifetime. Essentially, this means that it maintains a &quot;memory&quot; of the amount of current that has passed through it. One of the exciting potential applications of memristors is modelling synapses in hardware extremely efficiently.
Reinforcement learning and evolution
I think these are worth mentioning because they are promising candidates for addressing the problem of non-transferrability. These are not restricted to neural networks - being reward-driven, RL and evolution are theoretically applicable in a generic setting to any task where it is possible to define a reward or a goal for an agent to attain. This is not necessarily trivial to do, but it is much more generic than the usual error-driven approach, where the learning agent tries to minimise the difference between its output and a ground truth. The main point here is about transfer learning: ideally, applying a trained agent to a different task should be as simple as changing the goal or reward (they are not quite at that level yet, though...).