[quoting Reeves, 23 Oct 96.a] Reasoning that, because we can do experimental research in physics and make better cars and planes, we can conduct experimental research of the kind you espouse with human learning and as a result invent better education and training seems to be just as fallacious as the "Quantum Leap" inference from Heisenberg's "uncertainty principle" to alternative methodologies in educational research.
This is an empirical question. Science might work for education and it might not. Time will tell. I tend to think it will work, but we need a deeper understanding of the rules of educational discourse in order to create good teaching machines. I don't think logic gives us an answer to this question so the word fallacious is irrelevant.
Come on, Steve. If you're going to be a positivist, at least play the game according to the rules. Do you really think your study and Furnham and Gunter's represent a sufficient body of research to support a principle of instructional design that text is better than audio for learning facts? Wow? If it was this simple, why did Lee Cronbach and Richard Snow (1977) as well as a whole generation of educational researchers give up on the quest for establishing an empirical basis for aptitude-treatment interactions?
Maybe they couldn't design good experiments. Whether my and Furnham and Gunter's experiments establish a principle is a sociological question. If enough people find it convincing and no big anomalies emerge, it will be a principle.
In an earlier contribution to ITForum, I described nine signs of pseudoscience that are often found in your brand of empirical research studies. These are:
1. Specification error--Vague definitions of the primary independent variables (e.g., text versus audio).
What's vague about text/audio? My guess is that everyone reading this knows what those words mean.
2. Lack of linkage to robust theory--Little more than nominal attention to the underlying learning and instructional theories that are relevant to the investigation.
Huh? I directly addressed the most famous hypothesis in educational technology. That hypothesis stood essentially unchallenged for 12 years. Ain't that robust enough for you?
3. Inadequate literature review--Cursory literature review focused on the results of closely related studies with little or no consideration of alternative findings.
Clark did the literature review. It took him a full year to do it. His review is the alternative and I addressed it directly. My findings were paired with Furnham and Gunter's to make a narrow but salient point.
4. Inadequate treatment implementation--Infrequent (usually single) treatment implementation often averaging less than 30 minutes.
Clark's hypothesis does not say that media has no influence on learning PROVIDED the learning is more than 30 minutes. If I can establish that media have an influence even on short-term learning, that is enough to negate the hypothesis. This was not a single treatment because it replicated FURNHAM AND GUNTER who performed 14 comparisons. This is a hell of lot more replication than you usually see.
5. Measurement flaws--Precise measurement of easy-to-measure variables (e.g., time); insufficient effort to establish the reliability and validity of measures of other variables.
Reliability is a factor IF you get no significant difference, because the no significant difference may be caused by surplus error variance going into the denominator. As you know I did not get no significant difference, so if my measuring device was a little unreliable that would indicate that the t-value was in fact GREATER than what I measured. Maybe asking people to write down everything they can remember is an invalid measure of how much people can remember. Maybe they really could remember more but didn't write it down. How come the text group was less lazy? Note that Furnham and Gunter used other measuring devices and got similar results even with cued recall.
6. Inconsequential outcome measures--A lack of intentionality in the learning context, usually represented by outcome measures that have little or no relevance for the subjects in the study.
So what if the students didn't care? What explains the difference between text and audio? Everybody knows intentionality is not required for learning.
7. Inadequate sample sizes--Small samples of convenience, e.g., the ubiquitous undergraduate teacher education or psychology majors.
Hmm. Furnham and Gunter: 128 school children, 117 military students, 101 adults...
8. Inappropriate statistical analysis--Use of obscure statistical procedures in an effort to tease statistically significant findings out of the data.
The t-test?
9. Meaningless discussion of results--Rambling, often incoherent, rationales for failing to find statistically significant findings.
My results were significant at the .0002 level.