What is Distance Learning

Qualitative Measures

Comparison/control group designs are used to determine the specific effects of the intervention outcomes. These approaches are limited in terms of the breadth information they provide, and they are difficult to implement outside the laboratory environment. Even though experimental design and control groups traditionally have been advocated in evaluation studies, qualitative methods have been given increasing attention in recent years. Qualitative designs seek to describe and explain the program within the larger context of the educational setting.

Rather than entering the study with a pre-specified classification system for measuring program outcomes, the evaluator tries to understand the program and its outcomes from a more qualitative or participant perspective. The emphasis is on detailed description and in-depth understanding as it emerges from direct contact and experience with the program and its participants. Using more ethnographic methods of gathering data, qualitative techniques rely on observations, interviews, case studies, and other means of fieldwork.

There are a number of reasons to use qualitative design:

1. The program emphasizes individual outcomes.
2. There is an interest in the dynamics of program processes and program implementation.
3. Program staff wants detailed descriptive information to assist in program improvement.
4. Unobtrusive observation is needed.
5. Unanticipated outcomes or unexpected side effects are a concern.
6. There is a need to add depth, detail and meaning to empirical findings.

The greatest possible dangers to qualitative evaluation are an inexperienced evaluator and the loss of objectivity. If an evaluator is a participant in the project, or has a stake in Its outcomes, there is a threat to objective observation. This can be offset by using a recognized expert in the subject, who is also independent of the project's success or failure, perform the evaluation.

Strengths of Qualitative Measures

1. These models and their variations employ the quality of divergent responses to performance assessment of higher order and critical thinking skills.
2. By being relatively untied to pre-established objectives, criteria, and outcomes measures, these methods lend themselves well to the detection and interpretation of anticipated factors and results that may shed new light on program strengths and weaknesses.
3. Provide good documentation on lessons learned for others to use who are contemplating developing or installing similar programs.

Weaknesses of Qualitative Measures

1. Results are not easily compared or aggregated with other studies.
2. There is no standard means to control for evaluator bias or lack of suitability for the evaluation task.
3. Replicating the evaluation at other sites is highly problematic, given the reliance on evaluator subjectivity at the expense of standard fixed evaluation criteria.
4. Lack of systematic random sampling and statistically analyzable data makes generalization difficult.
5. Validity is greatly dependent on evaluator expertise and independence.
6. Measuring quality can be labor intensive and therefore expensive.

For the purposes of evaluating educational technology projects, combination of both approaches is desirable.

Summary

Evaluation is the act of making judgments about a program’s worth. Evaluation is different from research although both may use the same methods. Learning outcomes and learner progress are very important to all useful evaluation strategies.

Research design isolates the variables being studied. This is best accomplished in a laboratory where the researcher can exercise control over the conditions of the experiment.

In conducting field research, the control of possible confounding variables occurs by establishing identical conditions between two groups, and then randomly assigning students to participate in either the experimental group or the control group. The only difference in the group’s experience is the educational technology application being studied. Extraordinary efforts are made to prevent any other differences from occurring that might contaminate the results. The measurements of the variables being studied are precise and specific. Mathematical procedures are applied to test the results statistically to determine if the differences noted between the two groups could have occurred by chance alone. The process is designed to enable the researchers to accept or reject their hypothesis concerning the effects of educational technology on the learning outcomes.

The certainty of the relationship between the cause and effect of results is usually expressed as a percentage of probability. This is often described as the significance level. For example, "The results were determined to be significant at the .05 level." This means the likelihood that the differences between the two groups occurred by chance alone is only five percent.

Evaluation, on the other hand, is very practical. Its purpose is to help people make decisions about a distance learning intervention. These decisions may involve whether to continue or end the program, change it to improve its application or whether to expand the technology to other classrooms or other disciplines. The use of the evaluation results are not theoretical, they are practical and specific to a particular program.

Because of this, the evaluation approach is much different from research. While research tries to reduce the number of variables being studied, evaluation examines as many factors as possible. The idea is to describe as much as possible all the things that could have affected the program.

Many people have the mistaken impression that all evaluation is done after a project is completed. Actually, formative evaluation will improve a project during development and implementation phases. Formative evaluation provides feedback during the program development and implementation. The progress the learner is making can be monitored to see what works and what doesn't, allowing the administrators to fine-tune the project with midcourse corrections. Formative evaluation involves data collection, analysis and documentation from initiation of a project to its completion.

Summative evaluation determines the overall effectiveness of a project. The data from the formative evaluation is helpful in analyzing the final results and making recommendations. The evaluation approach must be designed to fit the technology project.

The literature clearly suggests that distance education is in an expansion phase with new institutions joining the ranks of those who are currently offering telecourses and other distance learning programs through a variety of media. The role of the Internet and world wide web and hybrid models being used in the California Innovation Programs are not yet covered substantively in the research literature. However, statewide descriptive data are positive.

Outcome Measurement

The term "performance measurement" refers to the regular, ongoing, measurement and reporting on important performance aspects of programs. The primary focus is to track the outcomes (or results) of programs. For example, feedback on problems found during a formative evaluation could lead to program efficiencies by learning from the experiences that others had early in a project. It can also identify success stories worthy of local or national coverage.

The outcome information derived from a performance measurement process has several more specific uses including the following:

1. To help identify where problems exist and where action is needed to improve program outcomes.
2. To help focus programs on the mission of achieving results.
3. To help motivate employees to continually seek to improve services to their customers.
4. To assist in budget development and justification.
5. To help track whether actions taken in the past have led to improved outcomes.
6. To better communicate with elected officials and the public.

Limitations of Outcome Measurement Information

The major limitations of outcome measurements are the following:

1. Outcome data obtained will usually not tell the impact of the program on the measured outcomes. Data on outcomes do not tell why the results are as they are. Usually other factors outside the control of the program (and probably the organization as a whole) contribute to the results. This particularly applies to indicators that attempt to measure desired end outcomes. This means that worse-than expected, or better-than-expected performance should not be the occasion for automatic blame-setting or, alternatively, praising the program. Additional examination is needed to access causes for shortfalls, or better than expected outcomes.
2. Managers of publicly-supported programs need to know whether they are winning or losing. The score does not indicate why the score is as it is, but provides vital information for program decision making. Projects should provide explanatory information about unexpected or unusual outcomes, along with outcome data.
3. The state-of-the-art of outcome measurement is limited. Perfect measurement and complete coverage of all relevant program outcomes should not be expected. The objective of practical outcome measurement is to provide the information on program quality and outcomes, not perfect information.
4. Outcome measurement requires time and effort to develop the process, to collect the information each year, to tabulate and analyze it, and to report it. The key question is whether over the long haul the information will provide the necessary information.

To determine the extent to which the program itself has affected the outcomes, sometimes called program "impact," more in-depth analysis is needed. Ad hoc, special program evaluations can be undertaken to estimate program impacts and help determine why programs fall short of, or exceed, performance expectations. Formal program evaluations are done infrequently on most programs, especially small programs, and, thus, do not provide the regular feedback on program progress needed to help managers manage.

As outcome measurement data becomes available, the information should be highly useful to those undergoing program evaluation studies. Regularly collected outcome data should also help the Department and its program offices determine their future evaluation needs, i.e., identifying areas that the outcome measurement data indicate need attention.

Categories of Outcome Information

It is useful to distinguish between various categories of outcome information. Outcome indicators can usually be classified as one of the following:

Inputs. These indicate the amount of resources, such as the as the amount of funds and number of employees involved in a particular distance learning program.

Outputs. These indicate the products and services produced by a program. Outputs are important for measuring internal work performance, but do not indicate the extent to which progress has occurred toward achieving a program’s purpose. For example, an output indicator might be "the number of math instructional modules produced." Outputs will generally measure the activities of the program and individual projects, rather than the activities of students or teachers who participate.

Outcomes. These provide information on events, occurrences, or conditions that indicate progress toward achievement of the mission and objectives of the program. It is usually useful to distinguish between intermediate outcomes and end outcomes.

Intermediate Outcomes are outcomes that are expected to lead to the ends desired, but are not themselves "ends." Intermediate outcomes generally indicate the extent of progress toward an ultimate, end result (such as higher student achievement). The distinction between outputs and intermediate outputs is not always clear. For example, the indicator, "Number of courses by type, provided to participants by educational institutions." can be classified as an intermediate outcome because it measures to what extent schools are actually participating in the program, rather than measuring, more passively the availability of the course.

End Outcomes are the desired results of a program. For example, a key end outcome "the percentage of students whose test scores improved significantly in courses in which distance learning technologies had been introduced and were a significant part of the instruction."

People will likely disagree in some cases as to whether an indicator is an end or intermediate outcome. Context is important, and since program missions may change over time, the classification of particular indicators may also change. When in doubt, it is often helpful to refer back to the mission /objective statement to make such determinations.

Outcome Indicators

A standard set of outcome indicators for distance learning programs includes:

1. the overall project mission statement;
2. the general objectives that relate to the mission;
3. more specific, but still general, outcomes sought that relate to each objective;
4. the specific performance indicator (s) for which data need to be collected to track progress on the indicator; and
5. the likely data source(s) of each indicator.

Data Sources and Recommended Data Collection Procedures

The outcome indicators that are finally selected must face the test feasibility and practicality. That is, the distance learning project should be able to obtain reasonably accurate data on each indicator, in a reasonably timely manner, and for an affordable cost in staff time and dollars. This section addresses the following data collection procedures suggested for each outcome indicator.

It is important to determine who will be responsible for the collection of which items of information. In addition, it must be clear to where and by whom they will be processed.

This discussion focuses only on collecting data on outcomes. Program impacts that indicate the extent to which the project/program affected the observed outcomes are not discussed. All outcome indicators are affected by outside factors - factors not fully controllable by the program or its individual projects. For obtaining information on project and program impacts, in-depth ad hoc evaluations are needed. To undertake such in-depth evaluations effectively, comparison groups will usually be needed.

A Distance Learning Program Evaluation Model

A simple distance learning program self evaluation contains these features. It assumes that the target audience is the program administrator, superintendent, board, and state agency.

• Purpose -- the program goal and specific objectives
• Target -- description of the targeted user
• Intervention -- a description of the distance learning intervention(s) and activities to support the intervention(s)
• Participants -- descriptive statistics on the learners -- see the TOPSpro summary data used for the California report (Innovation Program Reports)
• What did participants learn -- curricula content and learning gain data drawn from standardized testing. CASAS is used in California to measure reading and listening skills gains.
• How did participants apply their learning -- learning mastery data drawn from authentic or alternative assessments
• Participant satisfaction -- learner evaluations and comments on the intervention
• Staff self evaluation -- instructor evaluations and comments on the intervention
• Summary and Recommendations -- an examination of the intervention's strengths, weaknesses, learner participation and outcomes, and meeting the distance learning objectives

When possible, a third party evaluator should design and conduct the program evaluation. Emphasizing the practical aspects is important, especially when incorporating user evaluation comments and making program recommendations.

Adult Basic Instruction Delivered Through the Internet

Features of Internet Delivered Instruction

Distance learning takes place when a teacher and the students are separated by physical distance, and media and / or technology (i.e., voice, video, data, and/or print) are used to bridge the instructional gap. Online education refers to any form of learning/ teaching that takes place via a computer network. Physically, the computer network could be a local area network (LAN), an intranet within a particular organization, a wide area network (WAN), or it could be the global Internet and World Wide Web. Whatever the connection, the teacher and students share a common link through communication lines. This is often referred to as elearning.

Elearning has two adult basic education audiences – professional development and anytime student instruction. The professional development provides tutoring and continuing education services to the adult education field. Online learning should increase substantially over the next few years for "just in time" types of information related to grants and new initiatives and on going professional development. ABE, ESL, GED, high school subjects and career education modules and courses are being delivered over the Internet. The course length instructional resources are slowly becoming available. However, few California organizations and staff have initiated online learning.

The most common function used in online education is electronic mail (email) that allows students and teachers to send messages and file attachments to each other. In addition, web conferencing capabilities let participants conduct multi–person discussions either in real–time (often called "chats") or on a delayed basis (asynchronous). There are also groupware" programs designed to facilitate the work of groups. This technology may be used to communicate, cooperate, coordinate, solve problems, compete, or negotiate. Online education also involves access to databases in the form of text files or multimedia web pages, as well as the exchange of information (e.g., assignments, course materials) via file transfers.

Elearning is richest when students enroll in a course at about the same time. This managed enrollment enables the teacher to use the associated communications tools to permit student to student communication as well as two way teacher – student communications.

Delivery Models of Internet Instruction

The key to effective distance education is focusing on the needs of the learners, the requirements of the instructional content, and the constraints faced by the teacher, before selecting a delivery system. Typically, this systematic approach will result in a mix of media, each serving a specific purpose. For example:

• A strong print component can provide much of the basic instructional content in the form of a course text, as well as assigned readings, the syllabus, and day–to–day schedule.
• Interactive audio or video conferencing can provide real time face–to–face (or voice–to–voice) interaction. This is also an excellent and cost–effective way to incorporate guest speakers and content experts. The real time delivery also can be saved and streamed on demand for the asynchronous learner.
• Web conferencing, chat, and electronic mail can be used to send messages, assignment feedback, and other targeted communication to one or more class members. It can also be used to increase interaction among students.
• Pre–recorded video tapes can be used to present class lectures and visually oriented content.
• Simulations and interactive group learning games will be added to the mix as bandwidth increases.

Using a student centered approach, the educator's task is to carefully select among the technological options. The goal is to build a mix of instructional media, meeting the needs of the learner in a manner that is instructionally effective and economically prudent.

Pros and Cons of Internet Instruction

Learning and teaching online is much different than a traditional classroom experience even when used as part of a conventional class. Since most communication takes place via written messages (or files), writing skill and the ability to put thoughts into words are paramount. People who have poor writing skills may be at a disadvantage in an online environment. On the other hand, having to write everything gives people a chance to think about their responses, especially in an asynchronous setting, where the student does not need to respond immediately. One benefit of any class involving Internet instruction is plenty of practice writing, often resulting in improved communication skills. For many learners, this outcome is just as important as the subject matter being learned.

Asynchronous Internet instruction also changes the social dynamics of education — putting everyone (students and teachers) on equal footing. Under usual circumstances, everyone can post messages, so each online participant has the same opportunity to contribute ideas or comments. Consider the situation of the WWW; a web page or site created by a high school student has exactly the same accessibility as one created by a college professor. Similarly, anyone on the Internet can send a message to anyone else, regardless of who they are.