Criticisms of Significance Testing

Kirk 1996 (cited here) http://www.ejournal.aiaer.net/vol21109/4.%20Latchanna%20&%20Masoomeh.pdf

• Statistical testing is vital for statistical analysis.
•  But now is the time to incorporate Practical Significance
•  There are 3 points that need to be adequately met to be considered as in good empirical standing.
•         o Statistical Significance:
•                • relies on Sampling Variation
•                • Shows that the difference in the observed difference of means is most likely not due to chance.
•                • BUT DOES NOT how much more effective one treatment is than the other..
•                • This has been forgotten about and simply the importance of gaining a significant result. This has contributed in the ‘File Drawer Effect’
•          o Practical Significance o Reliability Criticisms
•                   May be statistically significant but not necessarily Practically significant
•          o Problem due to the relationship of the degrees of freedom of the group and the Statistical Significance
•                • As in some circumstances a significant result can come from a large population even if it only highlights a SMALL relationship between 2 variables.
•                • In some of these cases the importance of a significant result reported may not hold any practical value as the effect of the relationship of the two variables are too miniscule.
• – Null hypothesis is not correctly understood and therefore inadequately interpreted, Kirk 1996
• – ALPHA LEVEL.
•         o To control the probability of committing a type one and 2 error. Olejnik 1984 (http://0-pao.chadwyck.co.uk.unicat.bangor.ac.uk/articles/displayItem.do?QueryType=articles&QueryIndex=journal&ResultsID=135B2172EF21097761&ItemNumber=5&BackTo=journalid&BackToParam=QueryType=journals|ItemID=h101|issue=53:1%20(1984:Fall)&journalID=h101)
•          o Kirk 1996; Young 1993 (cited here: http://www.ejournal.aiaer.net/vol21109/4.%20Latchanna%20&%20Masoomeh.pdf0 It has been pointed out that simply labeling results as significant or non-significant is rendered inadequate, as the probability is a continuous measure. For example if we labeled those who are 5ft tall as ‘normal height’ and regarded those shorter than this height ‘short’ – this would mean that even if an individual were to be ½ less than 5ft; they would not classify as ‘normal height’.
•                   • A more relevant example practiced in research would be obtaining a p value of 0.49 and regarding this as non-significant whereas getting a p value of 0.51 as significant. However the difference of these two values is a meager 0.02 decimal. Should such a small difference lead to such contrasting classifications? – Which is also a contributor to the ‘File drawer problem’.
•          o MISUSE OF RESULTS.
•                   • Some associate significance testing with reliability or replicability. Vacha Haase & Nilsson 1998 http://www.ejournal.aiaer.net/vol21109/4.%20Latchanna%20&%20Masoomeh.pdf
•                    • This then leads to the assumption that a p value of 0.001 is more important than a p value of 0.05. But p levels do not measure the chance that the result obtained will be reproduced when the experiment is replicated. Cohen 1994 (http://www.ics.uci.edu/~sternh/courses/210/cohen94_pval.pdf)
•                    • The use of the term highly significant or stating that the small p value as evidence of a strong effect displayed is very misleading. As the smaller the p-value does not indicate the strength of the observed relationship or difference. Friedman 1968; Vacha-Haase & Thompson 1998 (cited here: http://www.ejournal.aiaer.net/vol21109/4.%20Latchanna%20&%20Masoomeh.pdf0)
•                    • Linked to this incorrect assumption is also the notion that a non-significant results indicates that there was no effect present at all. When in actual fact it means that there was not enough evidence to reject the null hypothesis (at the set alpha level employed). Therefore a non-significant result does not mean that the null hypothesis was true – Schmidt and Hunter (1995) cited in (http://www.ejournal.aiaer.net/vol21109/4.%20Latchanna%20&%20Masoomeh.pdf0)