Resisting the siren song of storytelling in science 

Ulysses (also known as Odysseus) stands with his hands tied behind his back to his ship's mast, as the Sirens, depicted as large black raptor birds with the heads of women with black hair, fly around him and his crew. His crew row on past the rocky cliffs in the background, their heads wrapped in cloth so they cannot hear the Sirens' song.

Ulysses and the Sirens, John William Waterhouse, 1891 (public domain)

[3 min read]

Storytelling might get you a TED Talk with millions of views or a popular science book on the bestsellers list à la Malcolm Gladwell, but, asks our Research Integrity Manager Matt Hodgkinson, does it make for good research?

Are you sitting comfortably? Then I’ll begin. Once upon a time, a Very Clever and Very Important Scholar, who lived in an Ivory Tower, told his students, “There are two possible articles you can write: (a) the article you planned to write when you designed your study or (b) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct answer is (b).” [1] They cheered and clapped and said he was Very Wise, for this would surely help them on their quest for Impact.

Another Very Witty Scholar, who liked to trick people into eating too much soup by secretly refilling their bowls as they ate, proudly told of The Grad Student Who Never Said “No”, who could get the best results from the most disappointing data [2]. It was just like swapping a boring cow for magic beans.

These Scholars and their friends cloaked themselves in their glamorous, shiny papers and paraded about town. What a sight to behold! But these scholars had not counted on the Trolls, who rudely laughed at them in public, and the Data Thugs, who roughly pulled and tugged at their clothes, paper by paper, until they were quite, quite naked [3].

A black and white woodcut of The Emperor walking in his nightclothes, surrounded by courtiers holding a canopy over him and crowds watching from either side. A little boy is pointing at him.

Illustration of “The Emperor’s New Clothes”, Vilhelm Pedersen, 1849 (public domain)

 

This tale I tell is true, but not all have heeded its warning. In an editorial published last spring and recently trending on social media, David Montagnes and his colleagues urged “writers to consider starting at the end of their study, starting with writing their main conclusions, which provide the basis of the Discussion, and then work backwards” [4].

I’m sure it’s not the intention – and indeed the authors add a cautionary note at the end of their article – but this kind of advice leads to Hypothesising After the Results are Known (HARKing) [5] and other Questionable Research Practices (QRPs) [6] that massage research into a neat narrative.

Their advice, like that of the naked scholars before them, is risky:

  • “Write up only the Results that relate to your conclusions” is meant to make authors focus, but can also mean leaving contrary results on the cutting-room floor;
  • Selecting take-home messages can mean cherry-picking outcomes that suit a predetermined narrative;
  • Suggesting not including extensive comparison to prior studies encourages glossing over inconsistent findings.

The “don’ts” in Table 2 counsel against bad practice, but these warnings are not specified throughout the text. This storytelling approach is the opposite of recent Open Science practices such as Registered Reports [7].

To avoid such pitfalls, start writing with your research questions, a summary of what is known from the existing literature, and then the methods, before moving to your results and discussion. Post hoc theorising must be declared. Contradictions in your results and versus the previous literature must be noted. These processes will keep you honest – and they will keep the Sirens at bay and the Data Thugs away.

p.s. Stuart Ritchie has more thoughts [8].

  1. Bem, D. J. (2004). Writing the empirical journal article. In J. M. Darley, M. P. Zanna, & H. L. Roediger III (Eds.), The compleat academic: A career guide (pp. 185–219). American Psychological Association. https://psychology.yale.edu/sites/default/files/bemempirical.pdf
  2. Wansink, B. (2016). The Grad Student Who Never Said “No”. PhD Advice https://web.archive.org/web/20170312041524/http:/www.brianwansink.com/phd-advice/the-grad-student-who-never-said-no
  3. Smith, G. N. (2020). Torturing Data Can Destroy a Career: The Case of Brian Wansink. Mind Matters News. https://mindmatters.ai/2020/12/torturing-data-can-destroy-a-career-the-case-of-brian-wansink/
  4. Montagnes, D. J. S., Montagnes, E. I. & Yang, Z. (2022). Finding your scientific story by writing backwards. Mar Life Sci Technol 4, 1–9. https://doi.org/10.1007/s42995-021-00120-z 
  5. Kerr, N. L. (1998). HARKing: Hypothesizing After the Results are Known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
  6. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
  7. Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192
  8. Ritchie, S. (2022). Science isn’t storytelling. Science Fictions. https://stuartritchie.substack.com/p/science-isnt-storytelling