In this paper, we address the capabilities and cost‐effectiveness of the current models of neural networks for carrying out general computation. We show that neural networks are formally capable of performing any conventional computation. However, we show that for hard ’algorithmic’ problems, the speed with which the solution is achieved comes at the cost of an excessive size for the network.