Spiking Neuron Models
Spiking neuron models simulate a neuron as an entity that receives inputs from dendrites, integrates them, and then makes a binary decision whether to fire. Firing results in messages being sent to post synaptic neurons. Spiking neurons typically operate with a simulated time step of around 1ms.
Hardware does not appear to be the bottleneck to human brain level performance. See also HardwareOverhang.
What is missing?
- Getting the wiring right
- We currently have a major gap in our understanding of how the brain works
- We understand the high level: the cerebrum, cerebellum, thalamus, amygdala, etc.
- We understand the low level: neurons, axons, dendrites, neurotransmitters, myelin, etc.
- But we understand very little about the intermediate level: how neurons are wired together to form the components of the brain
- Understanding learning and memory
- We currently don't have a good understanding of how the brain encodes information
Progress is intended to be made in these areas, especially the first area, by the Human Brain Project and the BRAIN Initiative.
Communication not FLOPS
Communication of firing events from a neuron to its post synaptic neurons is the bottleneck for spiking neuron models, not floating point performance (FLOPS). With around 1,000 synapses per neuron, communicating the firing event to post synaptic neurons swamps the time taken in deciding whether to fire:
- In a Von Neumann architecture computer each synaptic communication is going to result in a cache miss, necessitating a main memory access which might take 50ns. This will take perhaps 100 cycles on a current generation server; longer on future generation servers. Communication to all the post-synaptic neurons will thus takes around 100,000 cycles.
- Supposing a mean firing rate of 10Hz, and a 1ms time step, then a neuron fires once in every 100 time steps. Deciding whether to fire only requires 10 or so machine instructions (Izhikevich, 2004; Which Model to Use for Cortical Spiking Neurons?). So the total time spent deciding whether to fire per firing event is 1,000 cycles (assuming one instruction per cycle).
Thus the communication costs exceed the computation costs by a factor of at least 100 today. This points to the importance of non-Von Neumann architectures such as TrueNorth for spiking neuron models.
Note that the concerns raised here are not relevant to deep learning NeuralNetworks because most layers in deep learning networks only have a few inputs per unit.
A word of warning on FLOPS
FLOPS are commonly measured using the Linpack benchmark which doesn't stress the memory hierarchy. Making matters worse Intel's version of Linpack uses vector instruction extensions greatly boosting performance over regular Linpack. Vector extensions are useless for synaptic communication. The impact of vector extensions can be seen by observing an Amazon EC2 c4.xlarge instance obtains 46GFLOPS on the Intel Linpack benchmark, despite having a single core and a clock rate of only 2.9GHz. It is achieving 16 FLOPs per cycle! Focusing on Intel's Linpack measured FLOPS understates the computing needs for a spiking neuron simulation by perhaps a factor of 1,600 today. This number is likely to increase in the future as main memory latencies aren't keeping up with system clock rates, and further vector optimization of Linpack occurs.