Latency and bandwidth

It never ceases to amaze me how people don't distinguish between different but connected concepts. The classic examples are energy and power. Of course, they are related - very simply - but they are not simply interchangeable. One example - "Power efficiency is really important in the design of cellphones" - well, yes, but for most purposes it is energy efficiency that really matters - energy efficiency determines how long a cell phone works on a single charge. So, running at high power for a short time may be a better solution than low power for a long time.

Fortunately the relationship between latency and bandwidth (as in a memory system) isn't as simple as that between energy and power. If it were, designing embedded computing systems wouldn't be such an engineering challenge. (I say "embedded" because this adds interesting economic constraints - I often point out to people that they can buy - retail - a complex embedded multimedia system (e.g. DVD player) for less than the price - wholesale - of a PC processor).

In engineering terms it is reasonably easy to deliver memory bandwidth to a system - albeit at some cost. We can increase the frequency of interfaces, we can increase with width of interfaces, we can manage (buffer) memory accesses so that they use (paged) memories efficientlt and we can pipeline our systems. Unfortunately, only one of these - increasing frequency - improves latency. Even more unfortunately, processors are generally very sensitive to latency.

So what do we do? It seems easy to me - we have three choices; one, stop using processors; two, decrease system latency; three, make processors less sensitive to latency. Choice one would be a step back to the dark ages; choice two is hard but needs to be done; choice three is also hard and also needs doing. Big caches, multithreading, vector processing here we come.