Falta de credibilidad redux

Readers may recall that just last week this blog reported on - and took issue with - a fierce attack by Leontxo García, chess correspondent of El País, on a study carried out by the Education Endowment Foundation on the Chess in Primary Schools programme. Among other unfavourable comments, he said that the Foundation

should never have designed the study with such an absence of rigour - it should have suspended it, or altered it, on seeing how it was turning out and never should have published it, on the grounds of its complete lack of credibility.

As I said at the time, I am far from sure that Sr Garcia is quite the right person to be talking about rigour in academic studies or their interpretation. But that ad hominem aside, would it not be interesting to seek the opinion of the Dr Kevan Collins who is head of the EEF, and is among the individuals to be criticised by García in El País?

Perhaps it would. So I did.

To: Dr Kevan Collins
31 October 2016

Dear Dr Collins

Sorry to bother you. My name's Justin Horton and I'm a chess blogger. I am writing about the recent Education Endowment Foundation study into the Chess in Schools and Commuities project.

I wonder if you were aware that this study (and your comments on it) has recently been attacked, in quite aggressive terms, in the leading Spanish newspaper El País for, among other things, an alleged "lack of rigour" and "lack of credibility". A link (in Spanish) is here. I don't know if you read Spanish, but I would be interested in your comments!

Yours

Justin Horton

As it happens, Dr Collins was kind enough to write back. His reply is published with permission.

From: Dr Kevan Collins
1 November 2016

Dear Justin,

Thank you for the note. Yes, we've seen the piece and appreciate that some colleagues are disappointed with the outcomes. We have an absolute obligation to report the findings of every trial we support. It's worth noting that Chess in Schools and the Community (CSC) a large charity were responsible for the delivery and design of the intervention - we fund promising ideas (to date, over 130 studies involving more than 7000 schools) but don’t have a vested interest in the individual programme. As I understand the criticisms are levelled at the intensity and dosage of the intervention not the evaluation itself. From the independent randomised controlled study [Dr Collins' emphasis - ejh] it appears that offering children in English classrooms chess lessons once a week delivered by a trained chess coach did not lead to improvements in attainment. We are confident that this was a rigorous and reliable study.

Best Wishes

Kevan

So there you have it. Make your own judgement about the study and the rights and wrongs of the controversy: just for now, let me draw your attention to

we have an absolute obligation to report the findings of every trial we support

which is quite an effective rebuttal to García's

nunca debió publicarlo

never should have published it.

As far as I can see it is Leontxo García's move.