Chapter 10: Quality Assurance of Online Courses
University of Oklahoma, USA
Copyright © 2002, Idea Group Publishing.
THE CONCERN ABOUT QUALITY ASSURANCE
The Potential to Compromise Quality
Today, many university programs are integrating online classes into their curriculums. According to Stephen Ehrmann, reasons for online course offerings typically fall into two categories.
One type wanted to use distance learning technology to increase enrollments, often by reaching out to certain types of people who would not otherwise get an education. Some reviewers charge that these proposals were cheating students of most of the support needed for excellence: laboratories, rich libraries, interactive seminars, and informal interaction on campus.
The other type of proposal used computer technology to change what students learned or how they learned. Some reviewers accused such proposals of being tiny bastions of expensive exclusivity, hoarding rich resources for the lucky or the strong, excluding the vast majority of learners who were most in need of excellent teaching.
In other words, most technology proposals were designed either to enlarge the number of learners or to improve what some learners could learn, but not both. (Ehrmann, 1999)
Both objectives, improving access and improving quality, offer incentive for creation of online classes and degree programs. The benefits of using the economical Internet for distance learning are just beginning to be explored and documented. It is almost assured that, as more classes are offered online and become interchangeable at different universities, the proliferation of use by students will increase because of the “any- where, anytime nature” of the Internet.
Based upon the author’s experience, the convenience of Internet delivery and anytime online class availability are often the primary reasons students enroll in online courses. The quality of the online educational experience is often a secondary consideration to the student. Many times these students are willing to miss the in-class experience and interaction in order to receive credit for online coverage of certain required subject matter. Because of this attitude, there is also the potential for students and potential employers to perceive an online class or degree program as an “alternative” or the “next best thing” to attending traditional lecture classes. This attitude needs to be minimized if students are to utilize online and traditional classes equivalently in their degrees. The same level of quality for the class experience and content should be expected if online and traditional lecture classes are to be considered equivalent. Making an online class as engaging as a lecture class using the Internet is a worthy goal. “Jumping in with both feet is not for timid souls. Internet offerings require large amounts of time in the preparation of course materials. Everything must be viewed in a global sense for an entire semester at the offset.” (Kubala, 1998).
Assuring that the quality of the online class experience is equivalent to the traditional lecture class experience, using a medium that embodies working independently at one’s own pace, and communicating anonymously is a challenge. In fact, at this current stage of development, it is unproven as to whether this objective can be achieved. Class administrators and students must ask themselves, “Is there a trade-off of class quality for the convenience of the delivery method?” Motivations for offering and taking online classes should be fully explored. Class quality should not be compromised for the sake of posting a class online for business reasons, because others are doing it or for the convenience using the Internet brings. Motivations for offering online classes should be based in both improving access and quality. Achieving this level of quality will greatly influence online class acceptability across curriculums and universities, hopefully resulting in better use of classes and Instructor expertise.
At this stage of online education development, there is great potential for the traditional university experience to be reshaped using the Internet.
Strategies for quality assurance of online offerings must be defined and implemented if this is to happen. Optimizing use of the Internet to remove the physical limitations of the traditional university is just beginning. The quality of online classes will improve as Internet technology advances. As with any emerging endeavor, programs that implement quality assurance plans for online class offerings will probably have greater success in this effort.
Equal Quality Expectation for Traditional and Online Construction Classes
At the April 1999 National Construction Industry Education Forum (NCIEF) in Las Vegas, Nevada, discussion focused on quality assurance of online construction courses. During this discussion between construction academicians and construction industry representatives, it was agreed that online construction classes should meet the same quality expectations as traditional lecture classes. It was further recognized that these expectations should be evaluated the same way for both types of classes. It was hoped that all classes in a construction program, regardless of the delivery method, would be held to the same level of quality expectation. It was further hoped that by maintaining the same level of quality expectation issues of curriculum suitability, faculty receiving workload credit and acceptance of class credits at other universities would be minimized. This would also assure potential employers in the construction industry that the current education experience would not be compromised.
Comparisons of online classes to traditional lecture classes are just being documented to support this NCIEF discussion. Schulman and Sims (1999) found that “the learning of the online students is equal to the learning of in- class students,” comparing pre- and post-tests of knowledge for both groups of participants. This quantitative comparison addresses one of the components of quality assessment—comparison of outcomes or grades. Their evidence shows that the online learning environment can be as effective as the traditional lecture environment. Regarding student performance, Dominguez and Ridley (1999) found that “distance education supporters recognize the value and necessity of employing the same yardstick to establish the legitimacy of their programs. Institutions that offer distance education programs to adult learners in any form —from correspondence to web-based courses— have been honor bound to establish that such courses provide student learning and content equivalent to that found with campus-based instruction.”
Evaluating students’ qualitative perceptions of the learning experience is the other component that must be evaluated for quality assurance of a class. Further discussion about the cns4913online Construction Equipment and Methods classes shows that the online learning environment can be as effective as the traditional lecture environment. Recognizing that the traditional lecture and online delivery methods can and should be evaluated the same is a necessary step toward establishing quality expectations for online construction classes. For online classes to be equivalent to traditional lecture classes, there must be a quality equivalency of student performance and the class experience as well.
Online Class Quality Assessment Using In-Place Criteria
A convenient, efficient and equivalent means of gathering online class quality assessment is to use criteria already in place for traditional lecture classes. Most universities have a quality assurance strategy in place for current course offerings. Along with outcomes assessment, the online students’ perception of the class experience is measured and used for quality assurance of the class. This qualitative perception typically influences the students’ level of interaction, attitude and ultimate feeling about the class experience. These considerations subsequently affect the students’ outcome assessments.
Part of class quality assurance for University of Oklahoma College of Architecture classes is a subjective student survey taken at the end of each class offering. Student reaction to the instructor, class content and content delivery are rated. The results of the survey are to be used by the instructor to refine and improve the class. Student perceptions of the class are also used as criteria for the instructor’s yearly faculty evaluation.
Part of the previously mentioned NCIEF quality assurance discussion focused on the cns4913online Construction Equipment and Methods classes offered Spring 1998 and Spring 1999 by the University of Oklahoma (OU) Construction Science Division (CNS). The classes were the first complete semester length, online construction classes to be offered to Associated Schools of Construction member programs. Twenty-five students from nine geographically separate accredited university construction programs successfully completed the online classes. As part of the lecture and online class evaluations, the instructor used the OU College of Architecture (CoA) Non-Studio Course Evaluation to gather student ratings concerning the course instructor, course content and student performance. As with traditional classes, the online students’ perception of the class experience was measured and used for assessment and improvement of the class. Success of the online classes was evaluated by comparing traditional lecture and online class assessments using the Non-Studio Course Evaluation instrument, class exercise, and exam grades.
THE CONSTRUCTION EQUIPMENT AND METHODS CLASSES
The objective of the CNS 4913, Construction Equipment and Methods class was to provide an overview of different types of construction equipment and methods. Students investigated different types of equipment, performance criteria, the selection process, the economics of ownership, and inclusion of equipment costs in the project estimate. American Council of Construction Education accredited (ACCE) construction programs are encouraged to include this content in their curriculums, though not necessarily in a dedicated class. This class is not part of the OU CNS “core” of classes, but all CNS students are required to successfully complete it. Assignments and exams were medium level difficulty. Correct answers to exercises were determined using algorithmic formula, visual association or procedural understanding. Learning the formula or process was typically the objective of the exercises and the correct answer was a product of this understanding. In other problem-solving exercises, assessment was based on the approach to the problem and the steps of the solution, not just the correct answer. The typical lecture class atmosphere was open, using student inquiries to drive the lecture when appropriate. Students typically enjoyed the class because of the visual-based learning, the overall level of class difficulty, and the classroom atmosphere.
The 1998 Class
During the Spring 1998 semester, a web-based online Construction Equipment and Methods class was taught in conjunction with a 26 student lecture class meeting twice a week at OU. The online class was administered from a custom-built website designed, implemented and maintained to replace the traditional lecture delivery. The online teaching model required the online class student to check the website regularly, as the traditional teaching model required the traditional class student to attend lectures regularly. The website was formatted like a “book” of organized information to be used as a class information resource. Students were required to use Windows 95 and Office 97 for assignments. Email, the telephone, the Chat feature and limited desktop video conferencing were used for communication between students and the instructor.
The interactive online course was started and completed by 11 students from five accredited construction programs across the United States. Each student enrolled in an elective or directed reading class with an assigned faculty sponsor at their university. Invited faculty sponsors were initially responsible for finding students in their programs interested in taking the class. This method of recruitment was used due to the new and unproven delivery method. Student reasons for taking the online class included interest in taking an online course for the first time, using the class to conveniently help fulfill their degree requirements or the content was not offered in their program. The cns4913online class was the first comprehensive, semester- length construction course to be offered by an Associated Schools of Construction (ASC) program to other ASC programs using the Internet. Participating construction programs included Oklahoma State, Texas A&M, Auburn, East Carolina and Cincinnati Universities.
Students in the 1998 OU CNS 4913 lecture class did not have access to the class website. Course content was delivered in lecture format requiring note taking, but allowing for questions and observations. The delivery sequence was the same for both classes. Students in both classes were given the same homework assignments and exams at approximately the same times during the semester. Class assessment for both groups was mostly closed- book and required a faculty proctor for all examinations. Online students were given the same amount of time to complete their exams as the lecture class students. Sponsor professors served in this capacity for their respective students. The primary purpose of the parallel classes was for direct performance and administration comparison between the two delivery strategies.
An interesting observation was made as the semester progressed. Lectures typically contained and followed information in the website. Traditional students took notes during these lectures to be used for homework and exams. The online students did not attend lectures, but actually obtained a better hardcopy set of notes than the students attending the traditional lectures by printing the website pages for reference. This was not what the author envisioned at the beginning of the class. The trade-off, obviously, is that the online students missed the lecture class interaction reinforcing which information was the most important. This class interaction was also a more convenient format in which to answer questions.
The 1999 Class
Based upon this experience, in addition to the results of the CoA Course Evaluations and subjective assessment gathered from students at the end of the class, the website and class were upgraded. Prior to the start of the Spring semester, the class was advertised to all ASC faculty using an email listserv provided by the ASC. A promotional website was created and posted so that interested students could preview the website operation, class structure, content and how the class worked. During Spring 1999, 14 students from Oklahoma State, Texas A&M, Texas Tech, Auburn, University of Wisconsin-Stout and Cal Poly-San Luis Obispo participated in the online class along with 26 OU students in the lecture group. During these classes all students used the website for information, communication and to obtain homework exercises. The instructor decided to fully optimize the opportunity students had to work independently by making examinations open book with no outside faculty intervention required for online students or lecture students. Exams were similar to those given in the 1998 classes, but used more problem-solving and discussion questions. Students confirmed an ethics statement on each exam, stating that they worked individually.
Because it was the second time the online class was offered, the author took special note of the reasons that students requested enrollment. Students’ reasons for taking the class were very similar to the 1998 class. Most students viewed the class as a convenient means to fill a curriculum gap with minimal effort required of on-site faculty. For many the class was considered acceptable as an elective, a replacement for a missed class or as makeup work. Several students were aware of the class because of affiliation with an online class member or a faculty sponsor from the 1998 class. Several students took the class because the content was not addressed in their curriculums and they wanted to go into equipment intensive construction.
QUALITY COMPARISON OF THE LECTURE AND ONLINE CLASSES
Paralleling the CNS 4913 class deliveries provided a unique opportunity for comparison of students’ quality perceptions of traditional lecture and online strategies. The author believed that if the online students’ quality perceptions and the outcome assessment grades were similar to those of the lecture class, then it could be assumed that the online class was as effective as the lecture class and the quality of the classes was equivalent.
With the help of Ken Williamson, Ph.D., Texas A&M University, the 1998 lecture and web student groups were compared to determine whether the participants were similar in background, knowledge and attitude. The Test of Logical Thinking (TOLT) was administered between groups to compare basic reasoning ability. The test was given once as a pretest. The Test of Construction Attitude (TOCA) was used to investigate empowerment, attitude and motivation toward professional construction management. It was also used to evaluate student attitudes and motivation toward learning and instruction within and between groups. The TOCA was administered pre- and post-course.
The OU CoA Non-Studio Course Evaluation was used for the 1998 and 1999 classes to gather students’ perceptions of instructor performance, course content and self-evaluation at the end of each semester. Additional information was also collected from the online classes using a participant survey at the conclusion of the course offerings, instructor observations and online participant communications.
Results of the TOLT instrument indicated that the difference between the two groups was not significant. Results of the TOCA indicated that the online students considered themselves to be more professional than the lecture class. These instruments were not used in the Spring 1999 class.
Results of the OU CoA Non-Studio Course Evaluation for the CNS 4913 lecture and online classes are shown below. The questions are listed in order as they appear on the evaluation form. The mean evaluation ratings for the class for each question are noted in the responses columns on the right. Columns list the mean evaluations by delivery method (lect = lecture or ol = online) and the semester of the class (98 or 99). It should be noted that responses in the web classes have greater percentage influence on the overall average ratings for the questions than in the larger lecture classes
1. Adequate preparation for class by Instructor
2. Expressing clearly and concisely by Instructor
3. Clarifying material in reading assignments by Instructor
4. Availability and helpfulness outside of class by instructor
5. Well thought out assignments
6. Reasonable coverage of materials by exams and quizzes
7. Adequate measuring of knowledge by exams and quizzes
8. Knowledge about the prerequisite subjects
9. Time required for homework assignments
10. Credit given for homework assignments
11. Number of exams
12. Rating Instructor compared to other faculty members
13. Rating the course with the course content only
14. Rating the course with the gaining of knowledge
15. Rating oneself as a student in the course
The course evaluation instrument is divided into four categories. Questions 1-7 rate qualities of the instructor and administration of the class. A rating scale of 1 = definitely yes, and 5 = definitely no was used. Questions 8- 10 rate prerequisite knowledge required for the class and class assignments. A rating scale of 1 = far too much, and 5 = far too little was used. Question 11 rates the number of exams. A rating scale of 1 = far too many, and 5 = far too few was used. The last category, questions 12-15 compare the Instructor to other Instructors and the students’ perception of themselves. A rating scale of 1 = excellent, and 5 = poor was used.
Upon initial review, it appears that several of the evaluation questions are not applicable because of the online delivery of the class. For instance, questions dealing with instructor communication (Questions 2 and 4) and comparison of the instructor to other faculty members (Question 12) do not seem applicable because of the anonymity of the class delivery. These questions are still applicable used in the context of online class delivery. Effective communication and availability of the instructor are still an integral part of a successful class experience regardless of the delivery method. “No reply” to an email or missing a Chat session parallels missing an “office hour.” Effective communication demands fulfilling defined obligations to students, such as office hours or online Chat sessions. The instructor’s performance can be compared to other construction faculty at the participants’ programs. Though the online relationship is mostly anonymous and using email, the instructor’s personality and attitudes are still very evident in correspondence and grading. The instructor’s effort to be personable in email and telephone conversations and to share observations and interests can greatly influence students’ perception of the instructor, just as in face-to-face communication.
Eleven assignments, three exams and a comprehensive final exam were completed during both class offerings. There was no notable difference in any homework or exam grades. The average final grades for the 1998 lecture and online classes were 86.38 and 81.64 respectively. The average final grades for the 1999 lecture and online classes were 85.58 and 87.28 respectively.
Observations Based on Comparison
The actual numerical ratings for each question are not the focus of this comparison. It is very important, however, that ratings for both methods of class delivery were consistent for both semesters the classes were offered. There are no trends or evidence of different quality perception between the classes based upon answers in the survey. The final grades for online and lecture participants were not significantly different for either course offering. To the author, these comparisons demonstrate that the teaching techniques and styles are suitable for both delivery methods. Based upon this comparison of the Construction Equipment and Methods classes, the author also concludes that students can evaluate online and lecture classes using the same quality criteria. It should be noted, however, that meeting and exceeding the quality expectation of any class, regardless of the delivery method, is strongly influenced by the motivation of the instructor and students.
The web classes ratings of Question 2 (Expressing clearly and concisely by instructor) and Question 4 (Availability and helpfulness outside of class by instructor) are very close for both semesters. These ratings are consistent with the lecture classes’ ratings. It should be noted that of the 26 online students completing the Non-Studio Class Evaluation during the two class offerings, three participants marked “not applicable” on the assessment form for these questions.
Though still in the excellent (rating = 1) to good (rating = 2) range like the Spring 98 classes, the rating of Question 12 (Rating instructor compared to other faculty members) for Spring 99 was 1.24 for the lecture class and 1.91 for the online class. A possible reason for this difference is that the online students that semester did not interact with the instructor enough to form a more favorable opinion.
Interaction using email, the telephone or Chat demands greater communication efficiency than oral discussion in a classroom setting. This is perhaps the greatest limitation of the online delivery method. Based upon the subjective survey at the end of the classes, almost all online students felt this was the greatest weakness of the class. Several suggested that mandatory times for interaction be included in the class format. Contradictory to this suggestion, it should be noted that when the instructor was available in Chat at announced specified times, participation by online students was limited. This perhaps can be attributed to the great effort expended by the instructor to communicate in a timely, understandable and comprehensive manner concerning content questions and feedback about students’ performance. Based upon the subjective survey at the end of the classes, the availability of the class on the Internet all the time was the greatest strength. Many students also commented about the effectiveness of the class website.
Performance levels of several individuals in both groups decreased at the end of the semester. This is a trend typically occurring in most classes. It was typical for students to collaborate on homework with other class members at their universities. Rarely did they collaborate with someone from another university, unless required for a specific exercise.
DIRECTIONS AND FUTURE CHALLENGES
Meeting the Quality Challenges
To meet the challenges of assuring online and traditional class quality equivalence, several items must be considered when developing and administering online classes. Based upon the author’s experience, the following items should be part of the quality assurance strategy for online class development and administration. These items should be considered regardless of the delivery method, but deserve special consideration due to the nature of online classes.
Match Content to Required Assessment
Testing required for evaluation of a student’s understanding and use of presented information is the primary content suitability consideration for online delivery. Rigor of exercises and tests must be equally demanding for online and lecture classes to be considered equivalent.
If group or essay-based exercises and exams are used, the class is a strong candidate for online delivery. Authentic performance exercises assess the student’s ability to apply knowledge to solve real-life problems. Exercises can require planning and application of knowledge in new and different ways. The abundant and convenient resources available online can add great depth to these types of problem-solving exercises. Classes incorporating a large visual component, such as the construction equipment class, are very good examples of this suitability. Manufacturers’ websites providing information about equipment, specifications and services are ideal for many technology subjects. One of the greatest advantages of the Internet as a medium for delivering a class is the ability to greatly enhance and increase the information that can be included in class content. With minimal instructor effort, exploration of information contained in linked websites was incorporated into equipment selection and management learning exercises. Developing a custom website is an excellent opportunity for the instructor to offer self-collected resources and to express observations and suggestions about specific topics not covered in traditional class materials. Much of the content of the cns4913online website addressed topics that the instructor felt were inadequately addressed in the textbook and other course resources.
Classes requiring outside supervision for exercises and exams are not the best candidates for online delivery. The assessment method requires greater effort to assure the demonstrated learning. If this cannot be done efficiently and economically, than perhaps the content is not as suitable for online delivery at this time. “Developing effective and reliable assessment methods for online class participants perhaps will demand the greatest effort for innovation and departure from traditional practices.” (Ryan, 1999).
“Open book” testing is the easiest assessment method for online classes, because it requires the least amount of faculty intervention. However the rigor of “open book” assessment is suspect when using automatically graded true/false, yes/no, numerical, fill-in-the-blank, specific phrase or multiple-choice answers. Many times the right answer is easily found in the content if the student takes the time to search. Unless time limits are imposed for completion of questions or activities, developed competency is not truly demonstrated. It should be noted, however, that this form of combining assessment and content review is an effective means of teaching online. Questions can be used to prompt the student to review certain content or complete an activity. Exercises or assessment can be crafted so that questions are sequenced and must be answered correctly before proceeding to new information. This is an effective means of focusing students’ efforts on required knowledge and regulating their progression through the class.
As assessment becomes more interactive and automated greater responsibility will be placed on the student to work independently and follow specified guidelines. Efficient verification of students’ identities is a limitation for online delivery today. Eventually, technology will overcome this issue, but for now it strongly influences the assessment technique that is used. Computerized adaptive testing has great potential for minimizing this concern. Using this method, the difficulty of the next question is determined or adjusted based on the previous responses. Selection of questions is database driven and the assessment fashions itself individually for each student as answers are provided. This is an excellent format for totally automated, individualized online assessment.
Determine Required Collaboration and Communication
Two of the primary benefits of using the Internet are the opportunity to work independently at one’s own pace and to communicate anonymously and economically from a remote site. The instructor must decide how strongly these benefits are to be embraced in the online administration strategy. Required collaboration demands much greater communication effort and efficiency. Success of exercises requiring collaboration will greatly depend on students’ characteristics, backgrounds and motivations. Group interaction is an effective teaching technique, but must be used appropriately. Communicating face-to-face is much easier than current email, Chat or desktop-to-desktop audio/video communication. It should be noted that communication using the Internet will continue to become more efficient as real-time audio and video are more effectively combined and delivered. This future interface will minimize many of the current communication limitations.
Communications required for an instructor-paced online class are similar to communications required for traditional lecture classes. The instructor sequences content coverage, exercise durations and assessment. Communication by announcement on the website or email from the instructor is typically required for class activities to be initiated and completed. Classes incorporating extensive group collaboration, group interaction and a feedback loop require greater communication effort by the instructor and students. The instructor should anticipate spending extra time communicating with students and motivating these students to communicate with each other. As observed by the instructor, during collaborative online class activities, effective and timely communication between group members has the potential to be as challenging as the actual completion of the exercise objectives.
An automated or self-paced class format requires the least instructor interaction for class administration. This format truly embraces the benefits of the Internet. However, creating and implementing an automated online format requires much greater initial effort than an instructor-paced class. The entire content and delivery strategy must be determined before implementing the class. Automatic mechanisms must be built into the class to pace student content review and assessment. Much greater effort must be expended to make content presentation organized, comprehensive and interesting. Inter- active features, video and animation can be used to support learning and make the experience more enjoyable for the student.
Optimize Internet Capabilities
Internet presentation and communication technologies are becoming increasingly “user friendly,” incorporating more efficient and interactive features. Having an understanding of both content and these technological capabilities will greatly enrich the style and delivery format that can be incorporated into an online class. Online classes fully optimizing these capabilities to deliver content, exercises and assessment have the potential to be better than traditional lecture classes. Meeting this challenge demands the greatest instructor organization and coordination effort.
Internet capabilities are often incorporated into online classes by someone other than the instructor of the class or by placing packaged information into courseware shells to create class websites. The drawback to these methods of online class creation is that the Instructor may not understand the capabilities of Internet for making content delivery interactive and more engrossing to the student. The use of posted PowerPoint slides or videos of recorded lectures is a rather non-creative approach to content delivery using the Internet. This is basically using the Internet for traditional teaching. If information delivery without regard for the quality of the experience is the objective, then this format is suitable. Ultimately these classes will be perceived as “correspondence courses,” using the Internet for information transfer and communication.
The Online Class Quality Challenge
One of the primary reasons that teaching takes place in a classroom is that it is the easiest medium for content delivery and learning assessment. It is the path of least resistance for teaching and learning. The lack of face-to-face interaction makes the online experience much more demanding. Today, the online experience requires more motivation for successful teaching and learning than the traditional method. The instructor’s motivation and enthusiasm to create and administer an effective class and the students’ willingness to accept a greater responsibility for communicating are two of the primary influences on class success. A necessary step for any class quality assurance is determining how quality is to be evaluated by students. Comparison of assessment results and class evaluations for the Construction Equipment and Methods classes supports the NCIEF recognition that online construction courses can and should be evaluated the same as traditional lecture classes. Quality expectations for online and lecture classes should be the same for the classes to be considered equal. Otherwise, online classes will continue to be considered an “alternative” or the “next best thing” to attending traditional lecture classes. As part of a quality assurance plan, online instructors should seek to expand teaching methods and approaches using the Internet. There is great potential for using automated online content delivery for self-paced interactive learning exercises taking advantage of animation, video and audio. Content can be crafted to take advantage of the growing pool of resources available on the Internet.
How these capabilities are embraced should be part of an online class quality assurance strategy. Offering resource rich online classes using the Internet is a cost efficient opportunity for industry and academia to partner. There is great potential for exceptional classes to be industry-sponsored productions, combining emerging capabilities of the Internet, the best Instructors and the latest industry driven content.
Dominguez, P. S. and Ridley, D. (1999). Reassessing the Assessment of Distance Education Courses. Available on the World Wide Web at: http://www.thejournal.com/magazine/vault/A2223.cfm.
Ehrmann, S. C. (1999). Access and/or Quality? Redefining Choices in the Third Revolution. Available on the World Wide Web at: http://www.tltgroup.org/resources/or%20quality.htm.
Kubala, T. (1998). Addressing Student Needs: Teaching on the Internet. Available on the World Wide Web at: http://www.thejournal.com/magazine/vault/A2026.cfm.
The Quality Assurance Agency for Higher Education. (2001). Distance Learning Guidelines. Available on the World Wide Web at: http://www.qaa.ac.uk/public/dlg/guidelin.htm.
Ryan, R. (1999). Best practice suggestions for custom building a technology class website and administering the class. Journal of Construction
Education. Available on the World Wide Web at: http://www.ascweb.org/ jce/, 4(1), 4-17.
Ryan, R. C. (2000). Student assessment comparison of lecture and online construction equipment and methods classes. T. H. E. Journal, 27(6), 78-83.
Schulman, A. H. and Sims, R. L. (1999). Learning in an online format versus an in-class format: An experimental study. T. H. E. Journal, 26(11), 54-56.
Suen, H. K. and Parkes, J. (2001). Challenges and Opportunities in Distance Education Evaluation. Available on the World Wide Web at: http://www.music.ecu.edu/DistEd/EVALUATION.html.