Bad science: how to learn from science in the media

Science is all around us – but so is pseudoscience. Few of us read the original research papers behind every ‘science’ story, so how do we know what to believe? And why aren’t all media stories about science reliable? This classroom activity aims to teach students:

• The difference between observational and intervention studies
• Why we need to carefully scrutinise media reports about the health outcomes associated with different aspects of diet
• That the decision to change lifestyle is often dependent upon a range of factors.

Introduction

1. Ask students to imagine that they have decided to emigrate. They are going to live and work in one of the following regions: Sweden / rural Greece / Melbourne, Australia. Show students images of these regions^w1.
2. In small groups, they should discuss all the things that might be different about living in these situations, such as pollution, diet, healthcare, pace of life, prosperity, types of employment, and climate.
3. Explain that after a number of years, the wrinkling of their skin will be measured, quite scientifically. Which of the factors they have identified might make a difference to the amount of wrinkling? Establish that there could be a number of causes.

It is often hard to find out which of many possible causes has produced a particular outcome, such as skin wrinkling, especially if some of the possible causes are interlinked. In a scientific study we try to change only one thing at a time. This may sometimes be difficult, and we need to think of a way to account for that shortcoming, or consider conducting a different kind of study.

Main activity

1. Ask the students to imagine that they are working as scientists and have been asked to conduct research into the use of olive oil in people’s diets as a way of reducing skin wrinkling. The hypothesis is that ‘more olive oil consumed leads to fewer wrinkles’.
2. Ask students to decide how they could set up such an investigation. Get them to think about such factors as:
   
   • The size of the groups
   • How to make it a fair test
   • How to control the variables.
3. What problems are there with conducting such research?

Explain that there are two main types of study that scientists could use to answer this question, an observational study and an intervention study. Observational studies are when scientists find people who have already brought the change they are studying into their lives (e.g. who has been using olive oil in their diet and who hasn’t?).

4. 
   
   

   
   Ask the students to consider the advantages and disadvantages of this kind of study.
   Draw out that observational studies use existing behaviours, so are cheap and easy to do, but may struggle to isolate single variables. People in the study group may well use different amounts of olive oil in their diets, but there are almost certainly going to be lots of other differences as well. As a result, it may be extremely difficult to clearly identify the extent to which the presence of olive oil is the significant factor in preventing the wrinkling of skin.

Explain that intervention studies (‘trials’) are when the scientists control the variables (e.g. you are going to have olive oil in your diet, but he is not).

5. Again, ask the students what they think the advantages and disadvantages of this type of study are.
   Ideas should include that intervention studies involve a much better control of variables and the groups can be balanced to eliminate other variables. However, they are more expensive to run and may be unethical: imagine researching smoking in this way. They may also take longer: if you want to examine the effect of a lifetime of eating olive oil on life expectancy, you would have to start your experiment with children, but wait perhaps up to 70 years until you had your answer.
6. Return to the investigations proposed by the students, and ask them to determine which type of study they used.

Explain that in 2001 a detailed scientific study was conducted into the wrinkling of skin on people who lived in Sweden, Greece and Australia, and that you are going to share the findings. Depending on the age and ability of the students, you could do this in one (or more) of three ways:

• Explain verbally, making key points on the board
• Give students copies of the edited findings (student worksheet 1 which can also be downloaded below^w2)
• Give students copies of the original research paper (br Purba et al., 2001).
   

7. Ask the students to work in groups and explore what the research showed.
   
   • Was this an observational or an intervention study?
   • The study found an association between features of people’s diets and the amount of wrinkling they had. That might be because different diets cause wrinkles to different extents. But what alternative explanations are there? Are there factors which might be independently associated with both diet and wrinkles, such as social class, working outdoors, sunlight exposure, smoking, and so on? (In this situation, scientists would call these alternative explanations ‘confounding variables’.)
   • Did this study prove that changing your diet will help you get fewer wrinkles?
      

8. Ask the students to work in small groups. Each group has to write a short piece of text (no more than 50 words) for a local newspaper using this report as the basis.

   The editor has made it clear that they want something engaging about how readers can enjoy the summer sun without being affected by it. Keeping the editor happy might be difficult because the research has shortcomings and does not give a clear answer. The pieces should be written in large writing on sugar paper and displayed around the room.
    

9. Ask students to assess each other’s work and give each piece two marks (out of five): one mark for ‘how engaging it is’ and the other for ‘how accurate it is’.
10. Show the students the extract from the Daily Mirror (‘Sun protection on a plate’, student worksheet 2, which can also be downloaded below^w2), part of a longer article with a series of tips about improving diet. Ask the students to discuss the extent to which the article’s conclusion is reasonable. Emphasise that we are not saying that olive oil is not good for you, but asking whether this conclusion is entirely justified.

     
    

    
    
     

    
    Some students may think that the news stories make the good advice in a rather dry research paper accessible to a wide range of people. Others may feel that it is not quite as simple as that, and that if you want smoother skin in old age you might have to do a bit more than consume more olive oil. Some may feel that simplifying the story to make it accessible, and leaving out the caveats, has also made it misleading.
     
11. Draw attention to the abstract from the research paper, which said that “This study illustrates that skin wrinkling in a sun-exposed site in older people of various ethnic backgrounds may be influenced by the types of foods consumed.” Remind students of the confounding variables they identified earlier in the lesson and ask them the extent to which either their report or the one from the Daily Mirror recognised these.
     
12. Ask students to work in pairs to list the pros and cons of observational and intervention studies.

Take a show of hands – if students saw the newspaper headline tomorrow, ‘Scientific study shows butter causes skin cancer’, would they stop eating butter immediately?