Are artificial networks based on the perceptron design inherently limiting?
In the perceptron design generally used in Artificial Neural Networks, we know precisely what a single neuron is capable of computing. It can compute a function

$$f(x) = g(w^{\top} x),$$

where $x$ is a vector of inputs (may also be vector of activation levels in previous layer), $w$ is a vector of learned parameters, and $g$ is an activation function. We know that a single node in such an ANN can compute precisely that, and nothing else. This observation could be interpreted as "of course it is limited; it can do precisely his and nothing else".

The universal function approximation theory tells us (very informally here) that if a Neural Network is "big enough", has at least 1 hidden layer, and has non-linear activation functions, it may in theory learn to approximate any function reasonably well. If we add recurrence (i.e. an RNN), we also get, in theory, Turing completeness. Based on this, we could say that they are not particularly limited in theory... but of course there are many complications in practice:


How big is "big enough"?
How do we effectively learn our parameters? (SGD is the most common approach, but can get stuck in local minima; global optimization methods like evolutionary algorithms wouldn't get stuck... but I don't believe that they're famous for being fast either).
etc.


Just the observation that they may not be highly limited in theory of course doesn't mean that there wouldn't be anything else that works better in practice either. I can very well imagine that a more complex model (trying to simulate additional functionality that we also observe in the brain) may be more capable of learning more complex functions more easily.

An important caveat there is that it tends to be the case that more complex function approximators tend to be more difficult to train in practice. We understand very well how to effectively train a linear function approximator. They also typically aren't very data-hungry. The downside is, they can only approximate linear functions.

We also understand quite well how to train, for example, Decision Trees. They're still quite easy models to understand intuitively, they can learn more complicated functions than just linear functions. I'd say we have a worse understanding of how to train them well than linear functions, but still a good understanding.

ANNs as they are used now... it looks like they are more powerful in practice than the two mentioned above, but there's also still more "mystery" surrounding them (in particular the deep variants). We can train them quite well, but we don't understand everything about them as well as we'd like.

Intuitively, I'd expect that trend to continue if we try to imitate the brain more faithfully. I wouldn't be surprised if there exist more powerful things out there, but they'll be more complex to understand, more difficult to train, maybe also more data-hungry (current ANNs already tend to be very data-hungry).