A deeper analysis of those studies published in ETR&D and JCBI which are empirical in intent and quantitative in method yields a dismal picture of the quality of contemporary research in our field. In an earlier article published in the now defunct JCBI (Reeves, 1993), I presented an analysis of five studies published in refereed journals from the literature on learner control. I characterized the research reported in these articles as pseudoscience. Figure 3 summarizes the characteristics of pseudoscience in the field of instructional technology.
Specification error--Vague definitions of the primary independent variables (e.g., learner control versus program control).
Lack of linkage to robust theory--Little more than nominal attention to the underlying learning and instructional theories that are relevant to the investigation.
Inadequate literature review--Cursory literature review focused on the results of closely related studies with little or no consideration of alternative findings.
Inadequate treatment implementation--Infrequent (usually single) treatment implementation often averaging less than 30 minutes.
Measurement flaws--Precise measurement of easy-to-measure variables (e.g., time); insufficient effort to establish the reliability and validity of measures of other variables.
Inconsequential outcome measures--A lack of intentionality in the learning context, usually represented by outcome measures that have little or no relevance for the subjects in the study.
Inadequate sample sizes--Small samples of convenience, e.g., the ubiquitous undergraduate teacher education or psychology majors.
Inappropriate statistical analysis--Use of obscure statistical procedures in an effort to tease statistically significant findings out of the data.
Meaningless discussion of results--Rambling, often incoherent, rationales for failing to find statistically significant findings.
My analysis of recent volumes of ETR&D and JCBI indicates that pseudoscience continues to dominate research in the field of instructional technology. A conservative review of the thirty-nine "empirical-quantitative" studies reported in ETR&D indicates that twenty-eight of them (72%) can be identified as examples of pseudoscience in that they possess two or more of the characteristics in Figure 3. In JCBI, thirty-four (61%) of the fifty-six "empirical-quantitative" studies published during this period suffer two or more signs of pseudoscience. This analysis is evidence of a research malaise of epidemic proportions.
The question inevitably arises with respect to why so many pseudoscience studies get published. At least part of the answer rests in the incestuous nature of the relationships among the people conducting these studies and the people charged with peer review of these submissions. The editorial and review boards of these journals include many of the same people whose research studies exemplify pseudoscience. Not only does the insular nature of the review process assure these researchers of a venue for their pseudoscience reports, but it also at least partially explains the under representation of alternative approaches of inquiry.
For another perspective on the steady flow of pseudoscience in instructional technology, it is necessary to look at its source. Most of it emanates from colleges and schools of education that have graduate programs in instructional technology. The faculty in these programs are subject to the same "publish or perish" pressure as their colleagues in arts and sciences. They quickly learn that it is the number of refereed publications they can amass, not the relevance or value of their research, that really matters when they come up for tenure and promotion.
Needless to say, this problem is hardly limited to instructional technology programs. Colleges and schools of education reward pseudoscience in every discipline from early childhood education though vocational education. A new report issued by the Holmes Group called Tomorrow's schools of education, calls for tenure and promotion guidelines to be revamped so that professors are rewarded less for research and publication and more for work in the public schools (Nicklin, 1995). If such a radical shift in the reward structure was accomplished, would we continue to conduct pseudoscience when we could be rewarded for making a difference in the schools where the needs are so great? I doubt it.
Unfortunately, the likelihood of changing the reward structure within universities seems remote. However, as instructional technologists, we do not have to wait for such a change to occur. Another way of increasing the relevance of IT research would be to call a moratorium on our efforts to find out "how" instructional technology affects learning through pseudoscience research. Instead, we should turn our attention to making education work better through instructional technology. It would seem that we stand a better chance of having a positive influence on education if we engage in interpretivist, postmodern, and/or developmental research situated in schools with relevant problems.
Can reports of alternative forms of research be published? Of course! After all, the same people who conduct IT. research are the gatekeepers who determine what is accepted for publication in our most important journals. This is what peer review is! We are all in this together, and if we want to fundamentally change the nature of our research, we can. We can conduct socially responsible research, and at the same time, still meet the practical requirements of the larger academic "game" by providing our scholars with outlets in refereed publications, albeit ones that have been radically improved in terms of goals, methods, and relevancy.
The relevance of pseudoscience research studies is a moot point. Even if the researchers themselves ascribe to the highest ideals of scientific inquiry, research so flawed has little relevance for anyone other than the people who conduct and publish it. But it is not enough to criticize research in instructional technology as characterized by pseudoscience and social irrelevance. Alternatives to the old ways must be found. Some may demur, believing that instructional technologists are incapable of conducting valid, socially relevant research, and that they should stick to instructional design and evaluation, leaving educational research to cognitive psychologists or practitioners better equipped to conduct it. I disagree. I think we can conduct meaningful research provided we acknowledge the sterility of our existing research base and build anew from a foundation of sound learning theory and rededicated concern for the social impact of our research.
What would be the nature of a new socially relevant research agenda? A change in direction toward developmental research is exemplified by the dissertation study conducted by Idit Harel at M.I.T. (1991). Harel's (1991) Instructional Software Design Project (ISDP) represents a unique effort to use programming as a cognitive tool within a software design context. Harel's ISDP combines Papert's "constructionist" theory (1993) with Perkins "knowledge as design" pedagogy (1986). In her dissertation research, seventeen fourth grade students used Logo for a semester to create software products that were intended to teach fractions to third grade students. Her study combined quantitative and qualitative research methods within a quasi-experimental research design to investigate the effects of this "learners as designers" approach.
Harel reports that the fourth grade students spent an average of seventy hours working on their software design projects. The actual nature of the software the students designed was open, but they were two requirements for students in the program: (1) writing in a "Designer's Notebook" every day, and (2) attending periodic "Focus Sessions" about software design, Logo programming, and fractions. A teacher and the researcher were available at all times to help the students with their design efforts. Although each of the students produced a separate software product, collaboration among the students was encouraged.
Harel compared the differences in Logo skills and fractions knowledge between the seventeen students in the ISDP and thirty-four other students in two classes who were studying Logo and fractions via "a traditional teaching method" (p. 263). No significant differences were found in pretests among the three classes. Harel reports that "In general, the 17 children of the experimental class did better than the other 34 children on all posttests (Fractions and Logo)" (p. 272). Although not all differences were statistically significant, the general trend was quite positive in terms of specific learning outcomes as measured by multiple measures including paper-and-pencil tests, computer exercises, video-taped observations, and interviews.
The major part of Harel's (1991) study is a detailed description of the activities and metacognition of one student, "Debbie," over the four month period of the project. Harel's wrote that her detailed analysis of Debbie's work as well as her observations of other students indicated that "Throughout ISDP, the students were constantly involved in metacognitive acts: learning by explaining, creating, and discussing knowledge representations, finding design strategies, and reflecting on all of the above" (p. 359). In addition to positive cognitive effects in terms of metacognition, Harel concluded that the ISDP students acquired enhanced cognitive flexibility, better control over their problem-solving, and greater confidence in their thinking abilities. She notes however that the study did not include any direct measures of thinking skills, but her own interpretations of the students' metacognition and problem-solving processes based upon observations and analysis of documentation such as their Designer's Notebooks.
What a contrast exists between Harel's (1991) dissertation and the morass of pseudoscience endemic in our field! In Harel's research, pedagogical models grounded in robust learning theories were identified, and subsequently, powerful technologies were used to implement these models. In the latter, the power of various forms of technology to instruct is assumed, and reductionist experiments are conducted to detect its effects. In Harel's study, children with authentic needs experienced a powerful learning opportunity over a period of months. In most pseudoscience studies, undergraduates earn "extra credit" for less than an hour of their time spent using some form of mediated "treatment" that has little or no relevance for them.
In a landmark paper about educational research, Salomon (1991) describes the contrast between analytic and systemic approaches to research. Salomon claims that this contrast transcends the "basic versus applied" or "quantitative versus qualitative" arguments that so often dominate debates about the relevancy of educational research. Salomon concludes that the analytic and systemic approaches are complementary, arguing that "the analytic approach capitalizes on precision while the systemic approach capitalizes on authenticity" (p. 16).
While I agree with Salomon in theory, the dominance of pseudoscience in IT invalidates this complementarity in practice. The ugly truth is that many of us who engage in analytic research approaches consistently violate many of the basic premises of this paradigm, especially with respect to the testing of meaningful hypotheses derived from strong theory (Reeves, 1993). Although we may eventually be able to conduct valid, socially responsible analytic studies in this field, that time has not yet arrived. We need a valid body of systemic (interpretivist, postmodern, and developmental) research before we begin to have theoretical foundations strong enough to pursue an analytical agenda.
Is IT research socially responsible? At the present time, much of it, especially the pseudoscience research so prevalent in our field, is not. Are we asking the wrong questions? For the most part, yes. Can we change this sad state of affairs? Of course, if we have the will! Salomon (1991) points the way. A major benefit of systemic research in education is that it yields new questions and nurtures the development of new theory. A moratorium on analytic studies in our field could give us the theoretical foundations for a socially relevant analytic research agenda early in the 21st Century. There are hopeful signs as indicated by the studies reported by Harel (1991), Jonassen (in press), and Lehrer (1993) and the methodological prescriptions of Neuman (1989), Newman (1990), and Salomon (1991).
I believe that the research malaise in IT stems in part from the "mindlessness" that is endemic in so much of our professional and personal lives as we near the 21st Century. Although some would attempt to redirect or revive the pseudoscience approach to IT. research (cf., Ross & Morrison, 1989), it is clear that much of our research continues to suffer from the same mindless misconceptions and irrelevance identified by previous critics (Lumsdaine, 1963; Mielke, 1968; Schramm, 1977; Clark, 1983; Salomon, 1991). The social psychologist, Ellen Langer, documents the terrible costs of mindless behavior in education, health care, and business in her 1989 book, Mindfulness. She writes:
"When we are behaving mindlessly, that is to say, relying on categories drawn in the past, endpoints to development seem fixed. We are then like projectiles moving along a predetermined course. When we are mindful, we see all sorts of choices and generate new endpoints. Mindful involvement in each episode of development makes us freer to map our own course." (pp. 96-97)
The demise of JCBI and the prevalence of pseudoscience in our field are signals that we need to become more mindful about our research. If we continue as before, mindlessly conducting pseudoscience, the obsolescence of our field per se is a likely outcome. Already, the most innovative learning and performance environments are not coming out of Departments of Instructional Technology (cf., Cognition and Technology Group at Vanderbilt, 1992; Gery, 1995). On the other hand, as Langer (1989) emphasizes, mindfulness opens up all kinds of possibilities. Let us seize this opportunity to stop being pawns in "someone else's costly construction of reality" (p. 28) and realize that we, and we alone, can assure the validity and social relevance of research in instructional technology.
1. I am grateful to Marcy Driscoll, Don Ely, Kent Gustafson, Mike Hannafin, John Hedberg, and Walter Dick for their generous guidance in the development of this revised classification scheme. Of course, I take full responsibility for the flaws that will no doubt be revealed in its organization.
2. The full text of the original Dean Lecture paper may be obtained from the InTRO World Wide Web page (http://129.8.48.23/InTRO/InTRO.html) organized by John Farquhar, Steve Harmon, Marshall Jones, and Dan Surry.
3. ETR&D is a product of the integration of two journals previously published by the Association for Educational Communications and Technology (Educational Communications and Technology Journal and Journal of Instructional Development). ETR&D is divided into two sections, a research section and a development section. This analysis only considered the research section of ETR&D.
Carroll, J. B. (1973). Basic and applied research in education: Definitions, distinctions, and implications. In H. S. Broudy, R. H. Ennis, & L. I. Krimerman (Eds.), Philosophy of educational research (pp. 108-121). New York: John Wiley & Sons.
Casti, J. L. (1994). Complexification: Explaining a paradoxical world through the science of surprise. New York: Harper Collins.
Casti, J. L. (1989). Paradigms lost: Images of man in the mirror of science. New York: William Morrow.
Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21-29.
Clark, R. E. (1983). Reconsidering research on learning with media. Review of Educational Research, 53(4), 445-459.
Cognition and Technology Group at Vanderbilt (1992). The Jasper experiment: An exploration of issues in learning and instructional design. Educational Technology Research and Development, 40(1), 65-80.
Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116-126.
Driscoll, M. P. (1995). Paradigms for research in instructional systems. In G. L. Anglin (Ed.), Instructional technology: Past, present, and future (pp. 322-329). Englewood, CO: Libraries Unlimited.
Eisner, E. W. (1991). The enlightened eye: Qualitative inquiry and the enhancement of educational practice. New York: Macmillan.
Farley, F. H. (1982). The future of educational research. Educational Researcher, 11(8), 11-19.
Gery, G. (1995). Preface to special issue on electronic performance support systems. Performance Support Quarterly, 8(1), 3-6.
Harel, I. (Ed.). (1991). Children designers: Interdisciplinary constructions for learning and knowing mathematics in a computer-rich school. Norwood, NJ: Ablex Publishing.
Hlynka, D., & Belland, J. C. (Eds.). (1991). Paradigms regained: The uses of illuminative, semiotic and post-modern criticism as modes of inquiry in educational technology: A book of readings. Englewood Cliffs, NJ: Educational Technology Publications.
Jonassen, D. H. (in press). Mindtools for schools. New York: Macmillan.
Kozma, R. B. (1994). Will media influence learning? Reframing the debate. Educational Technology Research and Development, 42(2), 7-19.
Langer, E. J. (1989). Mindfulness. Reading, MA: Addison-Wesley.
Lehrer, R. (1993). Authors of knowledge: Patterns of hypermedia design. In S. P. Lajoie & S. J. Derry (Eds.), Computers as cognitive tools (pp. 197-227). Hillsdale, NJ: Lawrence Erlbaum.
Lumsdaine, A. A. (1963). Instruments and media of instruction. In N. Gage (Ed.), Handbook of research on teaching. Chicago: Rand McNally.
Mielke, K. W. (1968). Questioning the questions of ETV research. Educational Broadcasting, 2, 6-15.
Neuman, D. (1989). Naturalistic inquiry and computer-based instruction: Rationale, procedures, and potential. Educational Technology Research and Development, 37(3), 39-51.
Newman, D. (1990). Opportunities for research on the organizational impact of school computers. Educational Researcher, 19(3), 8-13.
Nicklin, J. L. (1995, February 3). Education-school group issues scathing, self-critical report. The Chronicle of Higher Education, p. A17.
Perkins, D. N. (1986). Knowledge as design. Hillsdale, NJ: Lawrence Erlbaum.
Reeves, T. C. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer-Based Instruction, 20(2), 39-46.
Robinson, R. S. (1995). Qualitative research P A case for case studies. In G. L. Anglin (Ed.), Instructional technology: Past, present, and future (pp. 330-339). Englewood, CO: Libraries Unlimited.
Ross, S. M., & Morrison, G. R. (1989). In search of a happy medium in instructional technology research: Issues concerning external validity, media replications, and learner control. Educational Technology Research and Development, 37(1), 19-33.
Salomon, G. (1991). Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20(6), 10-18.
Schramm, W. (1977). Big media, little media. Beverly Hills, CA: Sage Publications.
Seabrook, J. (1994, June 6). My first flame. The New Yorker, p. 70-79.
Yeaman, A. R. J. (1994). Deconstructing modern educational technology. Educational Technology, 34(2), 15-24.
The buttons that appear below will be found at the bottom of each page of the discussion. The first button will take you back to the previous page (in this case, to the beginning of paper #5). The middle button will take you to the ITForum home page. The last button takes you forward into the discussion as it progressed on-line.