Right, so, artificial networks also need sleep!

Andy Hab
2 min readNov 21, 2022

I should say that it is artificial neural networks that seem to need sleep, or a rest. But isn’t one point of an artificial network that that of it not needing sleep, or rest?

For those of you who don’t know, artificial neural networks are networks built by engineers in the computing space to mimic the brain’s cells and therefore hope to get better computing, or different, computing outcomes. Therefore, it comes as a real surprise that, apparently, rest improves their performance. It comes as a real surprise because these are not biological entities, there are numerous reason we need rest as human beings, as biological beings. One is that there is a slow build-up of toxic material as we function. Another is that there is also constant genetic damage that needs to be repaired.

So why do these networks need rest? First let’s understand what happens. These networks are designed to mimic neuronal functions — so far so good — and they have become really good at some things such as computational speed. But there is something called catastrophic forgetting — no, not like when we stand in the supermarket and can’t for the life remember what we wanted. This is when these networks learn sequentially, one task after another, new information can then overwrite old information and it is gone, “forgotten”.

Golden et al. at the University of California have now reported that when these artificial networks are trained on new tasks but with periods off-line mimicking sleep, they could replay old memories but without using old training data. This is due to the patterns that are replicated during our biological sleep. In our sleep the synapses, connections that is, between neurons are strengthened. You brain essentially replays your day’s input and strengthens memories during sleep (that’s one of many reasons sleep is so important). When this process was replicated it mitigated this catastrophic forgetting.

So, fascinating it is, that we are building artificial neural networks that replicate the brain’s processes for better computational processing power but also fascinating that these artificial networks also improve performance with sleep.

This goes to show that good old biology, and evolution, seems to have got it right. And for us also another reminder of the importance of getting a good night’s sleep.

--

--

Andy Hab

Sharing fascinating, fun, and important knowledge on the brain and human behaviour - most days. And masters track athlete - still going strong!