Meta-analyses of brain training studies are powerful but only for their intended purpose.
The controversy over brain training is well known. The science is young and the promises are over-stated by brain training game developers. While many individual studies have found very promising results, several review studies have found that effects are weak or non-existent. The most recent review study was sensationally reported in the Daily Mail (UK, November 19th) this past week with the negative headline "Brain training is 'a waste of time': Computer games designed for older people are only effective if undertaken with expert supervision". However, as usual, a closer introspection of the original scientific paper reveals a different and more positive story.
The first clue to the over-simplified reporting in the Daily Mail was the use of the phrase "waste of time", which of course does not appear in the original scientific report. It makes one wonder if the desire to report problems with brain training is not unlike the well-noted media tendency to play up global warming denial commentary from minor scientists (and even non-scientists). Those sensational stories put about by the global warming deniers make for riveting headlines, but they hardly represent the bigger picture. Here too, the rush to find fault with brain training may be jumping ahead of the evidence. Let's see what the brain training review paper reported in the Daily Mail actually found.
The paper written by Amit Lampit, Harry Hallock, and Michael Valenzuel of the University of Sydney, and published in PLOS Medicine, reported on a "meta-analysis" of 52 studies of the effects of brain training in older adults. The analysis covered a wide variety of types of brain training, involving 4,885 participants, 2,358 of which were control participants who were used for comparison purposes.
The Good News
Overall the study found that across the 52 studies, the whole range of brain training methods, taken together, significantly increased visuospatial skills, working memory, processing speed, verbal memory and non verbal memory. Some of these effects were modest while others were moderate in size, but these were all statistically significant. The brain training methods combined did not seem to increase attention or executive function (planning and judgment). In other words, brain training significantly increased most of the important measures that psychologists use to assess the effects of these programmes.
They also studied some other factors that may increase or decrease the benefits of brain training. They suggested that the quality of the design of the studies seems to effect what is found, but most importantly, that supervised and specifically prescribed brain training works better than randomly selected brain training undertaken alone. In fact, the study found that when taken without guidance and supervision, brain training on average tends not to effective. That is fine. Every treatment has "boundary conditions", or conditions under which it does not work so well. Many medicines have severe boundary conditions and may not work in combination with other types of medicines, but we do not conclude that the medicines therefore do not work. To do so would be an over-statement.
The Meaning(lessness) of Meta-Studies of Poorly Defined Treatments
The meta-study is a sound approach to answering questions of the kind asked in the University of Sydney study, but unfortunately it rests on the assumption that we have clarity on what counts as "brain training", which we do not. Just about anything can count as brain training if the manufacturer of the system says so. As a result, the studies being generalized about in these kinds of meta-analyses involve training systems that are very different to one another and many of which are bound to not work. The theories and data that support each one are different and are conceivably even contradictory in some cases. So while we can trust the findings of this study, we still cannot conclude anything about individual brain training methods – only about the brain training industry as a whole. This in turn makes no comment on the possibility of creating very exciting forms of brain training in the future based on emerging new approaches.
Imagine a meta-analysis of cancer treatments for a particular form of cancer that found that as a whole they do not work. This is no way proves that no individual method works, or that the effort to develop a cancer treatment is a waste of time. So it needs to be writ large; this finding does not suggest that all individual forms of brain training are ineffective, even when taken alone. The study was not designed to investigate that issue. And the original studies on which it was based all have their own forms of evidence that specific forms of brain training do or do not work to improve one type of intellectual skill or another.
But the Australian study is not the only review in town. Other meta-analyses of brain training have also been published this year. Another such study published by Trinity College Dublin researchers in Ageing Research Reviews not only corroborates the findings of the University of Sydney study, but is a bit more positive in its conclusions. In that study, thirty-one randomized controlled trials were examined, which covered 1806 older brain training participants. Compared to people who received others forms of mental stimulation than brain training, the brain training was shown to improve executive function and composite measures of cognitive function. When brain training was compared to no training at all, it was found to significantly improve three types of memory (face-name recall, immediate recall, word pairs), as well as user's own reports on general mental ability. That study also found that training prescribed for a user and given under supervision works best.
It's still early days
It is still quite early in the evolution of brain training methods. The field is dominated by a few approaches, several of which are only loosely based on neuroscience and others in my own field, which are heavily based on learning theory and no neuroscience at all. There is room for everybody in science and only time will tell which of today's theoretical and empirical innovations will crack the brain training nut. But precisely because it is so early, we should not expect too much from brain training interventions or give up on the dream of identifying core skills that can be easily taught and improvements in which lead to generalized improvements in daily functioning and across a wide range of intellectual tasks.
So let's be careful of sensational newspaper headlines. Meta-analyses of brain training studies are powerful but only for their intended purpose. Their reliability also depends upon scientists having clearly defined what brain training is in the first instance, and we are far from that point. A meta-analysis of brain training merely tells us if the industry is producing tools that work for their stated purpose. They cannot tell us whether it is possible or not to enhance intellectual functioning with game-based interventions. Based on current developments in my own field of Relational Frame Theory, and Relational Skills Training, we believe that the future is bright, and that the headlines will one day tell a different story
Share This Article
