We discuss the future of massively parallel computing from a fundamental architecture standpoint. Our central thesis is that various versions of Moore's Laws will all unavoidably break down over the next two to three decades, due to fundamental limitations imposed by the laws of physics (especially quantum mechanics). Therefore, the end to scaling-up von Neumann-based architectures by adding more cores and other incremental technology advancements is within sight. How are these challenges to be overcome, in order to continue making computation faster, cheaper, more energy efficient and more reliable? One possible answer lies in connectionist massively parallel models, based on (artificial) neural networks and other paradigms from biology, neuroscience and statistical physics. We discuss what would such massively parallel connectionist computers be like if they were to be based on "implementing neural networks in the hardware", and undertake several thought experiments comparing and contrasting the hypothetical connectionist parallel machines with the "classical" multicore supercomputers and large-scale distributed computing systems.