VOLUME 13, NUMBER 2, 1999
Editorial
Monitoring and Evaluation
Michael G. Moore
Articles
(for abstracts, click here)
Lead Article:
Comparing Distance Learning and Classroom Learning: Conceptual Considerations
Patricia L. Smith and Connie L. Dillon
Review of Lead Article:
Toward a Systems Theory of Distance Education
Farhad Saba
Authors' Response:
Toward a Systems Theory of Distance Education: A Reaction
Patricia L. Smith and Connie L. Dillon
A New Method for Analyzing Patterns of
Interaction
Daniel C. A. Hillman
Avoiding the Industrialization of Research
Universities: Big and Little Distance Education
D. Randy Garrison and Terry D. Anderson
Grass Roots
Looking at Distance Learning through Both Ends of the Camera
Joan M. Whitworth
Implementing an Internet Tutorial for
Web-Based Courses
Sherri Smith and Andrea Benscoter
Interview
Speaking Personally with Neil Butcher
Modupe Irele
Book Review
Objectives, Competences and Learning Outcomes
by Reginald F. Melton
Darcy W. Hardy
EDITORIAL
Monitoring and Evaluation
Michael G. Moore
In distance education, because the learner is separated from the instructor, and the instructor is separated from the administering agency, success of the whole enterprise is dependent on an effective monitoring and evaluation system. It is only by knowing how to use evaluation materials and procedures provided by the teaching institution that the instructor can know if the distant learners are experiencing difficulties. The right kind of evaluation data makes it possible to determine precisely what kind of help a particular individual needs. Similarly, a good monitoring system tells administrators what problems instructors and students are experiencing and indicates if delays or breakdowns occur in the communication systems—while there is still enough time to take remedial action.
Effective monitoring requires a network of indicators that collect data on learner and instructor performance. This must be done frequently and routinely, and the data has to be relayed with similar frequency to a control center where it can be evaluated. Evaluation in this context is the process of analyzing the feedback data gathered by the monitoring system, reviewing it, and making decisions about how well the distance education system and its various components are operating, as learners, instructors, designers, administrators, and communication resources work together to accomplish short-term and long-term goals. For an educational system, the most important of these goals are learning outcomes, although other goals are legitimate and may be monitored and evaluated, for example, maintaining cost-effectiveness or rectifying demographic imbalances in the student population.
One of the few generalizations that can be made about any distance education program—whatever the communications media used and the content level-is that a good monitoring and evaluation system is likely to lead to a successful program, and a poor system is almost certain to lead to failure. What, then, are some of the features of a good system? There are, I believe, three key features.
The first is the preliminary specification of good learning objectives. From the beginning of the course design process until the final summative evaluation of the project, no matter how large or small the course or how long or brief its duration, the central questions remain the same, namely: did each student produce evidence of having learned what was required as specified in the learning objectives, and if not, why not? All evaluation must address this question, and whether or not evaluators can show that the project was effective will ultimately depend on how well the objectives of the project have been stated, at all levels.
When advising course teams, I always urge—and try to lead—an exhaustive effort to articulate, in as much detail as team members' patience will allow, what they believe their students should learn and how that learning will be demonstrated as a result of their study in every module (typically the most gross division of a course), unit, lesson, and part of each lesson. This is not, as some will suggest, behaviorism run rampant. Nor does it limit distance teaching to merely low-level, easily measured cognitive objectives at the expense of learner creativity, learner involvement, or learner self-direction. Neither does it deny the development of problem-solving skills or knowledge and sensitivities in the affective domain. All these and similar high-level learning-provided it can be defined by subject specialists-can be articulated to the students and their instructors in terms of what the learners will be able to do and present as evidence of their accomplishment by the end of the module, unit, lesson, or part of the lesson. If, as is sometimes suggested, the behavior of a successful student in a given subject truly cannot be described, then indeed it would be difficult to specify a learning objective. But then it is equally impossible to construct a teaching program when that which one is trying to teach is not known! Fortunately, there are very few such cases; more often than not, the inability to define learning in terms of student behavior is a result of lack of knowledge by the instructor(s) about the procedure. When instructors are taught to achieve such specificity regarding their goals, most appreciate the better quality that such clear vision brings to their teaching.
The second key to successful monitoring and evaluation is the construction and, later, the handling of the products submitted by students or trainees as evidence of learning—commonly known as assignments. It is the assignments that provide the indicators referred to earlier. They are the source of feedback signals that should alert authorities throughout the system whenever a problem arises.
In the vast majority of courses, the assignment is a written document handed to an instructor in person at a study site or sent by mail, either electronic or hard copy. It may be an essay, a mathematical calculation, the report of observations of natural phenomena, an experiment, or a social event; it could be a multiple choice test, analysis of a case study, or solution to a problem; it may be a work of art, a poem, or a piece of music. Use of tape recordings, audio or video, allows the student to report on an even wider range of learned accomplishments than text alone permits. All that is needed to design interesting and suitable assignments, besides a crystal clear awareness of the learning the student is expected to demonstrate, is a creative interest in the task and an appreciation of the instructional value that truly interesting and challenging assignments add to the course.
A related awareness is that of time limitations—the absence of which explains many unsuccessful assignments. Every lesson of every course has to be completed within a defined period of student time, and that budget must include the time needed to complete the assignment. If course designers ask for more in an assignment than can be accomplished within the time budget, obviously there will be a greater or lesser degree of failure, through no fault of the student or, perhaps, the instruction. When failure occurs, evaluators need to consider several remedies that will be discussed later, but it is worth saying here that one remedy is to consider whether the assignment itself is unachievable in the allotted time.
Designing assignments to test achievement of well-articulated learning objectives as indicators of progress or problems is critical to successful monitoring and evaluation, but the value of the assignments is only as good as the system set up to handle them. Thus, the third key to good monitoring and evaluation is a good data gathering and reporting system.
Many years of research provide some significant knowledge about assignments and assignment handling. We know that distant learners are more likely to continue and complete a course if they have frequent assignments. We also know that there is a close relationship between students' propensity to continue or drop a course and the length of delay between assignment submission and return. We know that early success in assignment completion is especially important and that the capacity to tolerate frustration with assignments grows with experience in distance learning. From such research and experience we know that in a typical course it may be desirable to require submission of assignments as frequently as once a week. When this is the case, the instructor has two responsibilities: one is to respond weekly to the student, which has implications for workload that most institutions new to distance education in these Web-frenetic days have yet to appreciate; the other is to make weekly reports of the results of the assignments to the agency's administration. After the instructor evaluates the assignment, whether weekly or less frequently, s/he must have procedures and documents (and be paid for the time!) to record certain data, including the date of receipt for assignments and scores or grades given.
In a major distance education system there is likely to be a regional as well as a central administration, so that reports must be provided for evaluation at both the regional and the central levels. The region reviews reports from instructors and submits composite reports or reports of exceptional instances to the center. In a dual-mode institution, reports of student progress may be presented to both the academic and the distance teaching department. Whatever the particular administrative structure, however, all share the need for reports to be reviewed by senior staff in the system who are able to recognize symptoms of system failure. At higher levels (i.e., beyond the instructor), monitoring is a default system; regional and central administrators do not normally review satisfactory assignments or look in-depth at instructors or study sites where students show evidence of satisfactorily meeting learning objectives. Like a pilot in the cockpit who looks for red lights, not green, administrators are primarily interested not in the indicators that show where the system is working (i.e., the students are learning), but in the warning signals that indicate that some part of the system is inoperative or operating below expectations.
More specifically, if a student fails to complete an assignment while other students of the same instructor complete it, the instructor is alerted to identify and rectify the problem that particular student is experiencing. However, if all or many students of the same instructor have difficulty with an assignment, and students of other instructors do not, evaluators must ascertain what circumstances cause difficulty for that particular group of students: perhaps the instructor is misinterpreting evaluation criteria; perhaps the group of students did not receive a package of learning materials; or maybe an incorrect interpretation was given at a study site tutorial meeting. At an even more general level, if all the students in a region fail to complete the assignment, and those in other regions do it successfully, it suggests a regional breakdown: perhaps Internet connections failed; maybe the region in question did not receive a television broadcast that reached other regions; perhaps assignment packages arrived late and assignments were rushed; or maybe that particular region missed a briefing and training session. Finally, evaluators have to deal with situations in which large numbers of students across the whole system perform badly on an assignment. Among possible causes is that the teaching material was inappropriate, the objective was unattainable, or the assignment itself was an ineffective measure of the objective.
The monitoring and evaluation sub-system plays a critical part in the success of any quality distance education project. To do so, however, there must be clearly specified learning objectives and instructional materials and procedures developed to help students and trainees achieve those objectives. Assignments must be designed to test exactly—no more, no less—what is expected from the learning program. There must exist a network of people who know their roles in the monitoring system. And failure must be identified quickly and efficiently.
Comparing Distance Learning and Classroom
Learning: Conceptual Considerations
Patricia L. Smith and Connie L. Dillon
Comparison studies have been widely criticized as offering little conceptually to the field of distance learning. However, these studies can serve an important role in advancing our understanding of the phenomenon of distance education. The problem with comparison studies lies not in the "comparison" but with the media/method confound. This article proposes a schema system based on media attribute theory that can be used to classify both media and delivery systems based on research related to learning and motivation. It is important that comparative studies explain more than just which technologies were used; they must also explain why and how the media and delivery systems were used to support learning and motivation.
A New Method for Analyzing Patterns
of Interaction
Daniel C. A. Hillman
This case study examined a new method for analyzing patterns of interaction in face-to-face and asynchronous computer-mediated classrooms. Using software and a coding system developed by the author, the text of all spoken and written discourse was analyzed from four face-to-face (FTF) courses and two courses taught via computer-mediated communication (CMC). The results indicated that the interaction patterns in the computer-mediated courses resembled discussion, whereas the patterns in the FTF courses resembled recitation. In addition, problems with comparing synchronous FTF courses and asynchronous CMC are examined.
Avoiding the Industrialization of
Research Universities: Big and Little Distance Education
D. Randy Garrison and Terry D. Anderson
The pressure to increase access to higher education while reducing costs raises serious questions with regard to the purpose and goals of the traditional research-intensive university. Moreover, there is considerable rhetoric about reinventing universities based on adoption of communications and learning technologies without clearly defining or articulating educational goals. In contrast to the big industrial model of distance education, an approach to distance education is described that is consistent with the traditional goals and values of creating knowledge through a critical community of learners. This approach, or model, is labeled "little distance education" and its characteristics are defined. Meeting the needs of a new market for continuing professional education available at a distance is also discussed.
Looking at Distance Learning through
Both Ends of the Camera
Joan M. Whitworth
This case study follows an instructor and her students (thirty-three elementary teachers) as they experience for the first time a graduate-level science course delivered via compressed video technology. The study focuses on teacher learning and coping strategies. Data were gathered from videotaped records of classes, informal interviews with students and site facilitators, open-ended student surveys, and instructor and student journals. Findings are reported in the discussion section of the study.
Implementing an Internet Tutorial
for Web-Based Courses
Sherri Smith and Andrea Benscoter
The Division of Criminal Justice at the Florida Gulf Coast University (FGCU) offers an entire criminal justice degree program via distance learning, principally through computer-based Internet courses. During the Spring 1998 semester, an Internet tutorial was provided during the first week of classes to students in all Internet courses in the program. Its purpose was to familiarize students with basic computer skills and Internet usage. This article focuses on the use of the Internet tutorial in Web-based courses, including an explanation of the tutorial, a discussion of its benefits for students and faculty, and implications for future computer-based Internet courses.
