An Evaluation Plan for

Facilitate.com, a Collaborative Groupware

Prepared by Ms. Judy Milton
The University of Georgia

For

Ms. Julia Szerdy
Mr. Michael McCall
Facilitate.com

September 17, 1998

Introduction

This evaluation plan describes the procedures that will be used to evaluate Facilitate.com, a web-based collaborative groupware currently being used at The University of Georgia for distance learning classes. This plan includes a brief description of the background of this project, the purpose of the evaluation, the questions and decisions to be addressed, methods and instrumentation, the sample, limitations, logistics, and a timeline.

This plan has been prepared by Ms. Judy Milton, a Graduate Research Assistant in the Department of Adult Education, under the guidance of Dr. Thomas C. Reeves, Professor in the Department of Instructional Technology at the University of Georgia. It has been prepared for Mr. Michael McCall and Ms. Julia Szerdy, the software developers, and in collaboration with Dr. Karen Watkins at The University of Georgia and Dr. Mike Davis at the University of Manchester, England, faculty for the distance learning course.

Background

Facilitate.com was developed as a network-based groupware, primarily for use in business organizations. In 1994 the software was transferred to a web-based delivery format, providing the means for increased use in asynchronous or different time / different place situations. In the past four years, it has been used increasingly for asynchronous work and distance learning by several universities, including The University of Georgia. The most recent version of it, version 5.0 being released in September 1998, will be used in the class in which this evaluation will be conducted.

Reflective Technologies in Work-Based Learning is a graduate level course jointly taught by the Department of Adult Education at The University of Georgia in Athens, Georgia and the Centre for Adult and Higher Education at the University of Manchester, Manchester, England. The course is primarily experiential in nature, focusing on the dynamics of group interaction and interpersonal communication skills through experience in T-groups and action science case studies. The course begins with a week of face-to-face classroom experience followed by six weeks of small group case work on the web using Facilitate.com.

This is the second time that this course is being taught using Facilitate.com software. Informal feedback from the participants and incidental comments in the on-line conversations during the first course in the summer of 1997 lead to the enhancement and addition of several features in the Facilitate.com software.

Purpose

The intent of this evaluation plan is to provide systematic and reliable data to use for decisions related to the continued development and future use of the software. It addresses issues of usability of the software by students and faculty, the acceptability of its use in distance courses, and the feasibility of how it may be used within an educational environment.

This evaluation is both formative in its intent to improve and enhance the product and it use, and summative in its intent to estimate the effectiveness and value of the most recent version of the product, Version 5.0.

Goals

The goals of this evaluation are to:

Questions

This evaluation plan is designed to answer these questions:

Audiences

The developers of Facilitate.com software, Michael McCall and Julia Szerdy, are the primary audience for this evaluation.

Additional audiences for the findings of this evaluation include:

Decisions

The information provided by the findings of this evaluation may be used to make decisions about the development and use of the software. The primary decisions will be made by the software developers, but these decisions, as well as the findings of the evaluation itself, may influence the decisions made by other participants in the process.

Decisions about changes and improvements in the software will be made by the developers of the software, and may be based on several criteria:

Other participants in the evaluation may also make decisions based on evaluation findings and how that information is used. The basis for these decisions may include:

Methods and Instrumentation

Data for this evaluation will be collected using questionnaires, focus groups, and student journals. The text of the on-line small group case will serve as supporting evidence for the data. The following instruments may be found in the Appendix to this plan:

Additional documentation such e-mail communications and a log of technical problems will be maintained to supplement the formal collection of information.

Sample

The sample for this evaluation will be approximately 15-20 students enrolled in EADU 6000/MD501, Reflective Technologies in the Work-based Learning, a graduate course taught jointly by The University of Georgia, Athens, Georgia and the University of Manchester, Manchester, England. The participants in the evaluation will reflect the diverse composition of the class in gender, ethnic background, and computer skill level. The students participants in the evaluation will be selected after the final arrangements for the class are determined (approximately October 1). The sample will also include Dr. Karen Watkins and Dr. Mike Davis, who are co-teaching the course and will be facilitating the on-line group work.

Limitations

This evaluation plan has two primary limitations that may limit the usefulness of the results: the type of participants in the course and the content of the course.

The participants in this evaluation study are all graduate students, with a range of technology skills. This raises two possible implications for generalizability of the findings. First, as graduate students, their expectations for both content and delivery of a course of this type may be very different from those of undergraduate students. Second, different levels of technology skills may raise different sets of needs and expectations of the technology itself. These different expectations may create a dilemma in making decisions related to the software and its use.

The course content is very experiential in nature. The learning occurs in the group interactions on the casework, rather than through the traditional model of teacher-delivered instruction. The issues about the technology that arise from the group interactions and the implications for the software development may be completely different issues and implications from those that might be found in other courses. Issues about user interface and acceptability of use in distance courses, therefore, may not be generalizable to different contexts.

Logistics

This evaluation is being designed and implemented by Ms. Judy Milton, under the direction of Dr. Thomas C. Reeves, Professor in the Department of Instructional Technology at The University of Georgia. Ms. Milton is responsible for the collection and analysis of the data and the preparation of the final report.

Participation in the evaluation will be explained when the students meet in Manchester the first week of October. Those students asked to keep journals will begin when the groups begin the on-line casework in October and continue through November. Questionnaires will be conducted on-line the third week of November. A focus group will be conducted during that same week.

Time Line

This evaluation is planned for September through December 1998. Version 5.0 of Facilitate.com becomes available in September. The graduate class participating in the evaluation is scheduled to meet during October and November. Collection of evaluation data is planned for October through December. The final report will be submitted by December 10, 1998.

Questionnaire for the Evaluation of Software

Please complete this evaluation of the web-conferencing software, Facilitate.com, that was used in this course by responding to the following statements. Choose the one response from the popup menu that best reflects your opinion. In addition, please provide any comments or suggestions regarding each item in the text area below each statement. Your individual responses to this evaluation will be anonymous. Your name will not be added to any of the comments, and the responses will not be linked to any one person.
  1. Web conferencing was an effective tool for conducting small group work in a distance learning environment. Why or why not?
  2. My small group was able to work effectively over the web. Why or why not?
  3. I found it beneficial to work asynchronously on the cases. Why or why not?
  4. The software supported my learning in the distance format. Why or why not?
  5. I thought the software was easy to use. Why or why not?
  6. The software did not allow me to have the type of interactions that I expect in a distance environment. Why?
  7. I thought the software got in the way of what we were supposed to be doing. Why or why not?
  8. The one thing I liked most about the software was __________________ .
  9. The one thing that I hated about the software was ___________________ .
  10. Some things about the software that I would change are _____________ .

Focus Group Protocol

  1. Let’s talk for a minute about your first impressions of using the web-conferencing software? How difficult or how easy was it for you to use the software?
  2. What were your reactions when you first began using it for the action science cases?
  3. Did the technology ever get in the way of the casework? How?
  4. Based on your experience, what do you see as the positive reasons for conducting the group work this way?
  5. What do you see as the negatives?
  6. Do you think the software had any effect on the way your group functioned? If so, how?
  7. What would you want to change or do differently with the software?
  8. What are some other ways that this software might be used in the university?
  9. If the software developers were here now, what would you want to tell them?

Student Journals

Three to four students in the class will be asked to keep informal notes during the on-line work to note:

These notes will be collected and analyzed for themes and specific suggestions after the small group case work ends in mid-November.