You can't just go on telly and make up statistics, can you?

From ChanceWiki
Jump to navigation Jump to search

Take with a pinch of sodium chloride

It seems we can't buy anything unless it has the approval of boffins. But what does any of it mean? Margaret McCartney examines the suspect science that we swallow, apply and absorb every day in an online GuardianUnlimited article from The Guardian (Aug 18th 2005).

When you read or hear something like '8 out of 10 people prefered X to Y', what are the details behind this sample survey result. The article give Pantene Pro-V as an example. They have recently been telling us, via shiny spreads in various magazines and TV ads, that its Anti-Breakage Shampoo, will lead to "up to 95% less breakage in just 10 days". It transpires that the sample size is just 48 and the survey was not a blind one.

Further investigations reveal that 10 samples of hair were tested, three times and the results were "significant". Furthermore the associated adds were vetted and approved by the Broadcast Advertising Clearing Centre.

In another example the UK's Advertising Standards Authority (ASA), with a staff of 100, look at all the major newspapers daily, but with an estimated 30m adverts printed every year in the UK, it is impossible for them to look at them all. In a recent case, a slimming pill advertising was withdrawn after making claims that were found to be based on a study on just 44 people. The ASA decided that this was too small a study to be valid.

The ASA director of communications "Talking generally, we may accept a small sample size as reasonable proof, but this would really depend on the statistical significance of whatever tests were done. Conditional claims lead to a host of different claims, especially when 'modal verbs' are used. We might ask them to change 'can' to 'could' if they didn't have 100% proof of the 'can'. But we would also expect them to hold proof relating to the 'could'."

The article lists other examples of dubious statistics such as "93% say their skin felt softer, and 79% say their skin was firmer with each application" (of a skin care prodct) or a more serious example about medicinal benefits such as 'the drug "effectively reduces the risk of a heart attack" by "preventing build up of harmful plaques in your coronary arteries" and "reducing your risk of coronary heart disease"'.

"The key issue is that of evidence. If you don't have evidence to justify claims of benefit, then the whole argument begins to fall apart." says Dr Ike Iheanacho, editor of the Drug and Therapeutics Bulletin, a journal published by Which?.

The article finishes with the warning that marketing and science have got together and bred a weird hybrid form of sales-experiments that have taken over our advertising culture.