Skip to main content

Full text of "ERIC EJ969449: Principled Assessment Strategy Design for Online Courses and Programs"

See other formats

Principled Assessment Strategy Design for Online Courses 
and Programs 

Janet McCracken, Sunah Cho, Afsaneh Sharif, Brian Wilson and Jeff Miller 
The University of British Columbia, Vancouver, Canada 

Abstract: As the demand for online learning environments grow in higher education, so does the need 
for systematic application of learning and educational theory to the design, development and delivery 
of assessment strategies within these environments. However, there is little guidance in the form of 
principled design frameworks that can assist the design practitioner in the development of online 
assessment strategies. From four cases, we have identified six design principles that represent the 
collective experience of our team of design practitioners in creating assessment strategies for online 
teaching and learning environments; (a) technology affordances, (b) alignment of objectives with 
assessment, (c) discipline-specific practices and approaches, (d) meaningful and timely feedback, (e) 
authenticity and transferability and (f) transparency of assessment criteria. We present in-situ 
qualitative case studies that articulate how these principles have informed our design practice in 
online assessment strategy development. 

Keywords: online assessment, distance learning, design principles, assessment design, case studies 

1. Introduction 

Designing an assessment strategy requires an understanding of the differences in the ways that 
teachers and learners think about the sequence of events that occur in an online course. To the 
teacher, assessment is at the end of the teaching-learning sequence of events, but to the student it is 
at the beginning (Biggs, 2003). We believe that it is good design practice to articulate assessments as 
early as possible in the design process so that the development of objectives, learning activities and 
media resources are more clearly aligned with the outcomes that instructors are striving to achieve. 
We also believe that the articulation of assessment strategies in a clear and definitive manner serves 
to align the teaching activities of the teacher and the learning activities of the learner towards the 
same goal. 

The creation and delivery support of online assessments present particular challenges and 
opportunities as will be addressed within each of the cases. In addition to general best practices 
concerned with creating representative and valid assessments and rubrics, online assessment design 
requires knowledge of the technical environments and the features they each offer to the designer 
and instructor, as well as the ability to create a reasonably sustainable assessment that the instructor, 
who may have limited technical expertise can manage. For example, managing grading for online 
group assessments requires the development of specialized rubrics and reporting that most 
instructors may find daunting without the help of a designer who understands the technical side of the 

While there are a number frameworks that address online assessment of online discussions 
(Anderson. T., 2004; Levine, S. 2002), it is hard to find more generic frameworks that can guide online 
assessment strategy development. Therefore, we have identified the following design principles that 
represent the collective experience of our team of design practitioners; a) technology affordances, b) 
alignment of objectives with assessment, c) discipline-specific practices and approaches, d) 
meaningful and timely feedback, e) authenticity and transferability and f) transparency of assessment 
criteria. We briefly discuss each of these principles here followed by the presentation of four cases in 
assessment strategy design. 

ISSN 1479-4403 107 ©Academic Publishing International Ltd 

Reference this paper as: McCracken J, Cho, S, Sharif, A, Wilson, B, and Miller, J. “Principled Assessment 
Strategy Design for Online Courses and Programs” The Electronic Journal of e-Learning Volume 10 Issue 1 
2012, (ppl 07-119), available online at 

Electronic Journal of e-Learning Volume 10 Issue 1 2012 

Technology affordances : This principle is concerned with mapping technology to the kinds of 
interactions that lead to learning. In assessment strategy design, this mapping requires selecting tools 
and technologies that can support both formative and summative strategies. For example, if students 
are asked to engage in asynchronous discussions, we need to decide; a) is this discussion going to 
be assessed, b) how will this discussion be assessed, c) what form will the assessment take and d) 
how will the results of the assessment be communicated to the student. Technologies can support 
these different kinds of assessment requirements in different ways. Ultimately the creation of the 
assessment strategy must incorporate the affordance of the technologies that are selected to support 
the teacher and the student. In each of our cases, there is a rationale for the selection tools, 
technologies and approaches that are best suited to the particular learning goals and contexts for 
each case. 

Alignment of objectives, instructional methods and assessments : Biggs (2003) contends that we need 
to assess actual learning outcomes to see how well they match intended learning outcomes. 
However, there is more to this principle than to merely check off which assessments match which 
objective. The method of assessing the student based on the way in which the student is taught is 
also part of the alignment principle, alluded to above in the discussion of technology affordances. For 
example, if a series of case studies are used to teach students how to carry out an analysis, the 
assessment form should be aligned to this method. So we go beyond the matching of objectives to 
the alignment of intended goals, instructional methods and assessment methods. In our cases, the 
intended learning goals are pursued and integrated throughout timely and sufficient formative and 
summative assessments. 

Discipline-specific practices and approaches: As designers who work collaboratively with faculty 
members in the design and development of assessments, we are keenly aware of the particular 
strategies that disciplines and fields use to assess student learning. While most of the research 
literature on distance and online learning has been concerned with discovering general principles and 
practices that may be applied across disciplines and contexts, we draw from work that focuses on 
discipline specific issues in teaching and learning to inform our design strategies. We concur with 
Donald (2002) who suggests that in order to improve teaching and learning we must acquire a deep 
understanding of the contexts and constraints which students encounter in different disciplines. 
Research on student learning in the disciplines has provided highly compelling evidence that 
understanding of the experience of the learner in relation to the content they are attempting to learn 
provides potential for designing powerfully transformative learning activities (Ramsden, 1992; Gibbs, 
1992; Biggs, 1999; Bowden & Marton, 1998). Therefore, this design principle promotes the idea that 
designers must be concerned with creating assessments that are aligned to the particular nature of 
the difficulties that students encounter in their understanding of the principles and concepts 
associated with disciplinary practices and approaches. 

Meaningful and timely feedback: Our assessment strategy design must be concerned with how data 
on student understanding is to be collected, analysed and reported in ways that may be quite difficult 
to accomplish in face-to-face environments. For example, online tests and quizzes can provide 
immediate feedback to students - particularly important for formative assessment. However, the most 
effective type of feedback for improving learning is specific to the individual student, and there is no 
getting around the fact that this type of specific, timely and meaningful feedback is labour-intensive. 
However, technology can help to reduce some of the time involved in creating and communicating 
feedback. For example, audio recording can accompany written comments. Laurillard (2002) 
proposes that not only is feedback essential to the student to help them make sense of the material 
they are attempting to learn, but that the instructor must deeply engage in understanding the nature of 
the difficulties that students are revealing in their assessments in order to make adjustments to the 
way that the material is presented Technology can support multiple modes of communication 
including any combination of student-student, student-faculty, faculty-student, faculty-faculty, student- 
others, others-students, etc, providing more options for ways in which feedback may be accomplished 
(Gayton &McEwan, 2007). 

Authenticity and transferability : Much of the literature about authentic assessment focuses on 
assessing students in real-world contexts (Brown et al., 1989). According to Herrington, Oliver and 
Reeves (2003), there are a number of technology affordances that can be employed to support 
authenticity. For example, an online course of study can incorporate a metaphor based on a realistic 
and authentic context to preserve the complexity of the real-life setting. Students accessing the site 


©Academic Publishing International Ltd 

Janet McCracken et at. 

can move freely around the resources as required, rather than in a linear manner through modules or 
sections of text in a set sequence. Problems presented to students can use the full capacities of the 
technology to present situations and scenarios in video clips, text links and images to give meaning 
and purpose to the students’ endeavors, and to provide motivation to complete the task. Technology 
can provide collaboration tools that may mimic more real-world interactions that occur in workplaces. 
In a similar way, the ability for students to transfer their knowledge from formal educational settings to 
either professional work settings or other formal settings is a complex issue. Therefore, any intended 
assessment should promote application of the knowledge into real world issues. 

Transparency of assessment criteria : Literature on assessment argues strongly that providing 
students with a well articulated assessment strategy as well as the criteria by which they will be 
assessed helps to orient them to the amount of time and effort that will be required of them to 
complete the requirements of the course (Brown & Glasner, 1999: Gibbs & Rowntree, 1999; Thorpe, 
2000). The use of online forms for sharing grading rubrics for assignments of all types is highly valued 
by students. The transparent assessment criteria and procedures will help students understand the 
expectations of the instructor and the course from the beginning. Also they will help students check in 
with their learning progress during the course. 

The process of teaching a course and the process of designing a course in online environments both 
represent a complex planning enterprise consisting of decisions framed within a set of constraints and 
opportunities. While we consider teaching as a specialized form of design (Wiggins & McTighe, 2005), 
the design professional who plans for a future implemented course works with simultaneous 
abstracted attention to multiple constructs; the structure of the content, the needs of the instructional 
team to create conceptually sound and interesting materials that will convey the stated goals of the 
course, and a keen awareness of the ways in which students will interpret and understand the 
material and forms of interaction created within the course. Bowden and Marton (1998) refer to this as 
having the ability to be focally aware of certain things in the foreground at the same time as being 
aware of more peripheral things, with the focal awareness changing over time depending on time and 
circumstances. This is a very good image of how we work in our design practice and while we limit the 
scope of our paper to a discussion of our approach to assessment strategies within this complex 
environment, it is within this larger context that we practice as designers. 

We present four cases from our design practice, and end with a reflection of how our assessment 
strategy design is informed by identifying principles and by the experience of instructors and students, 
and design practitioners in situ. 

2. Case presentations 

We present four cases as exemplars of design practice in online assessment strategy development, 
each of which has been developed by an Instructional Designer in cooperation with a faculty member 
at the University of British Columbia as part of a program or course design initiative. Each of these 
cases represents the outcome of collaborative work between instructional designers and course 
authors over several years through a process of continuous refinement based on the experience of 
the online instructors and students. In all cases, the course authors and instructors were highly 
engaged in the development of the strategies and instruments represented here. 

Case 1: Professionalism and Peer Assessment 

The UBC Dental Hygiene Undergraduate Degree Program places high value on students developing 
the skills necessary to be practicing professionals during their course of study, using peer assessment 
as one means of assessing whether or not the core competencies of professionalism, critical thinking, 
collaboration and communication have been achieved. These competencies are among those that 
have been identified as essential abilities for health care professionals for the 21 s1 century and apply 
to all courses within the program (Dental Hygiene Educators Canada, 2008; Verma S., Paterson M., 
Medves J., 2006; World Health Organization, 2007). 

Rationale. A primary consideration in choosing to implement a peer assessment model is to support 
the learner in the development of their role as a health care professional and, in particular, in 
developing decision-making abilities which include taking responsibility for one’s own learning, making 
constructive contributions to the learning of others, and reflecting on self- and peer-performance. 
Another key objective is to support student participation in the decision-making process, which 

www.eiel orq 


ISSN 1479-4403 

Electronic Journal of e-Learning Volume 10 Issue 1 2012 

includes focusing on initiating decisions and providing meaningful feedback to colleagues about their 
professional decisions. In this way, peer assessment forms a foundational role in the overall 
assessment strategy of the program and provides a vital means for students to demonstrate their 
acquisition of the skills commensurate with a professional dental hygienist. 

The need for peer assessment arose in part due to the demands and constraints of group and 
collaborative work required in the Dental Hygiene courses. As such, assessing a student’s ability not 
only to learn the course content but also the ability to function within a professional setting became a 
key consideration. Peer assessment was designed to help students refine their conflict resolution 
skills and their ability to develop a respectful team environment. In order to do this we designed a 
rubric for assessing these skills according to clear and explicit criteria readily available to the students. 
Such transparency in assessment criteria not only makes it easier for students to understand what 
constitutes good work and what is necessary to achieve program benchmarks, but also helps 
students to undertake a process of reflective practice, including the unanticipated and often unknown 
path that learning often takes. This in turn helps close the gap between current and desired 
competencies, especially in a professional setting. (McNamara & Brown, 2009; Boud & Falchikov 

Assessment Strategies. In all 3 and 6 credit courses, students are expected to both self and peer 
assess, in addition to being assessed for professionalism by the instructor. Each term, students are 
organized into teams of between 4 and 6 members and are responsible for providing peer 
assessment for each team member. Peer assessment constitutes 20% of the final grade, half of which 
is determined by the instructor. Students are required to achieve a peer assessment minimum of 60% 
to be considered successful. 

Students are assessed on a 5 point scale in the areas of: participating in group discussions, helping to 
keep the group focused on tasks, contribution of useful ideas; quality of work, quantity of work, and 
general conduct. The same rubric is used for all three assessments: self, peer and instructor 
( Students are provided with 
extensive descriptors of the criteria needed for assessment of the criteria needed for assessment at 
the start of term and these descriptors are woven throughout the course narrative during group and 
individual activities. 

The professionalism grade not only assesses the learner’s contributions to the class discussions, but 
also applies to the group work. Providing thoughtful feedback to their colleagues and reflecting on 
their own professional competence enables learners to develop their ability to take responsibility for 
their learning, in addition to contributing to the learning of others. However, as the peer assessment is 
a summative, high stakes form of assessment, it is critical that students feel that the grades and 
comments assigned are an honest reflection of the work performed. Therefore, peer assessments 
remain anonymous to the students, but not the instructors, who are able to access the grades 
assigned and any comments made by the students. 

The Dental Hygiene Program is delivered primarily through WebCT Vista, so it is important that we 
create a smooth, reliable and low-maintenance integration of the peer assessment framework within 
the LMS that takes into account user support requirements, institutional resources and course 
sustainability. In consultation with Faculty, it was decided that combining the native grading form tool 
in WebCT Vista with the discussion board would provide the most cost-effective option while offering 
the necessary functionality. This approach provides a low-threshold solution that does not require 
extensive faculty or student training prior to implementation and does not require the integration or 
development of an external application within the WebCT Vista environment. It also provides the 
students with easy, integrated access to the grading rubric throughout the course for reference and 
the only setup required on the part of the instructor is the creation of a new discussion thread that is 
dedicated to self and peer assessment for each student in a group. The result, from a design 
perspective, is a clear and effective workflow that is easily implemented and maintained. 

Lessons Learned. One of the drawbacks of this implementation is that the statistical treatment of peer 
review data is limited. Attaching a grading form to a discussion post in WebCT Vista does not allow for 
easy statistical review by the instructor, which means instructors must manually calculate, or employ 
other tools, should they wish to analyze the results further or compare the results to previous course 
offerings. This can be both time consuming and frustrating for the instructor. In addition, although an 


©Academic Publishing International Ltd 

Janet McCracken et at. 

aggregate of responses is readily provided, individual student responses must be viewed by clicking 
an ‘Expand/Collapse’ button beside each student’s name. While not overly onerous, this small task 
can make the process seem tedious when working with a large cohort. Figure 1 shows a completed 
peer assessment. 

Objective/Criteria Performance Indicators 

Extensive Substantive 

Indicators of Indicators of 

Criteria 5/5 Criteria 4/5 

Participated in 67% 
group discussions ( 4 ) 

Helped keep 
group focused on 

useful ideas 

Quality of work 
Quantity of work 
General conduct 




( 2 ) 






( 2 ) 

50% I 


67% I 


117% I 
( 1 ) 

17% I 
( 1 ) 

83% 17% 

|(5) (1) 


(Click the reviewer's name to view the individual evaluations.) 

© Brockton Oval 

Solid Indicators of 
Criteria 3/5 


( 0 ) 


( 0 ) 


( 0 ) 


( 0 ) 


( 0 ) 


( 0 ) 

Criteria not met 
2/5 or 1/5 
depending on 
degree to which 



( 0 ) 


( 0 ) 


( 0 ) 

0 % 

( 0 ) 


( 0 ) 


( 0 ) 

Figure 1: Completed peer assessment 

Students initially require detailed instructions on the use of grading forms, without which they may 
provide their peer assessment by simply replying to the initial message from the instructor, thereby 
making their comments public and personally identifiable. However, once students are made aware of 
how grading forms work, they find them easy to use. After the first implementation it was noted that 
further faculty training would be required for them to be responsible for the management of the 
framework, including setup each term. In response to this, a screencast was developed and made 
available to all instructors. Once such training resources were made available, instructors have found 
implementation reasonably easy and effective. 

Case 2: Virtual Patients: Assessment and Treatment Planning for Advanced Periodontal 

Educators and designers in health professions regularly employ virtual patients in different forms to 
respond to students needs for real patient encounters and opportunities to practice in safe, accessible 
and responsive environments (Ellaway, Ft., Poulton, T., Fors, U., McGee, JB. & Albright, S., 2008). 
The Assessment and Treatment Planning for Advanced Periodontal Diseases course has been using 
virtual patient cases as a form of assessment for the last few years. It is a fourth year course with 
approximately 20 students in each term that is primarily for dental hygiene degree students and builds 
upon basic knowledge acquired from diploma dental hygiene education and dental hygiene practice. 
Using virtual patient cases, students learn to assess each patient’s specific needs, develop a 
periodontal diagnosis, develop a comprehensive dental hygiene care plan based on current and 
reliable evidence, and evaluate the effectiveness of their dental hygiene care with follow-up strategies 
in a team environment. 

Rationale. Assessment is one of the key components of the educational experience in the dental 
curriculum as the results of assessment can be used to evaluate students’ readiness to enter their 
fields as entry-level practitioners. Assessment in the dental curriculum requires higher order thinking 
skills (i.e. analysis, evaluation, etc.) as well as effective communication skills, procedural skills, 
problem-solving and critical thinking skills. To facilitate transfer of learning from formal education to 

www.eiel orq 


ISSN 1479-4403 

Electronic Journal of e-Learning Volume 10 Issue 1 2012 

practitioner contexts in health professions, it is imperative that assessment is designed to mimic 
authentic environments (Huwendiek, S., Reichert, F. Bosse, HM., et al, 2009). 

Assessment Strategies. The course development team incorporated a number of instructional 
strategies (Huwendiek, S., Reichert, F. Bosse, HM., et al, 2009) when designing the virtual patients 
cases. Cases are designed to predict the learner’s performance outside of the instructional 
environment and in a “real” situation; cases unfold in a series of stages over 2 or 3 weeks to provide a 
record/baseline of performance over time, and multiple types of assessments within a case are 
provided to address learning style differences and enhance motivation. Learners are encouraged to 
work interactively and collaboratively with peers in a small team and later with the whole class. The 
small team format helps learners to master the skills necessary to assess, diagnose, plan, implement, 
and evaluate (ADPIE) (2004) a case study. Students’ professionalism is also assessed through the 
content of discussion, participation and contribution to the groups and by self and peer assessment 
forms (see previous case study in this paper for a discussion of the peer assessment strategy). 

The course makes ample use of grading rubrics; these rubrics match expectations for particular 
aspects of the assignment to grade ranges. Comments on submitted work will reflect performance 
according the expectation laid out in rubrics. The instructor will provide feedback in timely manner. For 
virtual patient cases, detailed rubrics were developed for grading case reports and professionalism. 
Figure 2, shows the grading rubric for case reports. 

Grade Range 


Extensive Indicators of 


90% plus 

» Identified the overt problem(s) as well as potential problems not readily apparent in 
the case 

■ Provides creative, innovative, novel recommendations/solutions to the problem(s) 

■ Provides highly individualized solutions to the problem(s) 

■ search of the evidence is broad in scope and depth 

■ Provides a critical analysis ot evidence with the best information synthesized into the 

• Recommendations/solutions are supported with extensive, relevant, and credible 

Substantive Indicators of 


80 to 89% 

■ Identified the main problem and all secondary problems 

■ Recommendations/solutions are comprehensive and case-specific 
• Indications that search for evidence was comprehensive 

■ Evidence was analyzed and the best information was incorporated into the report 

■ Recommendations are supported with comprehensive, relevant, and credible 


Indicators of Criteria 

70 - 79% 

■ Identifies the main problem and a few secondary problems 

« Recommendations/solutions are individualized; within the realm of solutions that a 
dental hygienist would be expected to think of for the problem 

■ Search for evidence was relevant and focused 

• Evidence was analyzed, summarized and the appropriate information integrated into 
the report 

■ Recommendations are supported with relevant, diverse, and credible resources 

Criteria Met 


• Identifies the main problem 

■ Provided basic recommendations/solutions that are applicable to every case, but are 
not individualized for this particular case 

■ Indications that a basic search for evidence was conducted 

■ Evidence was summarized and integrated into the report 

■ Recommendations are generally supported with evidence 

Criteria not met 


• Did not identify the problem(s) 

■ Recommendations are vague, confusing and/or irrelevant to the case 

■ Search for evidence appears cursory or superficial 

• Evidence was not analyzed to determine its strengths and weaknesses 

■ Little evidence that recommendations grounded in credible resources 

Figure 2: Virtual patient case rubric 

Initially, learners work through a sample case together to become familiar with the case study 
approach and what is expected of them (there is no grade attached to the sample case). The 
instructor works closely with learners through sample case using different technologies such as 
Wimba and takes an active role communicating with learners using Mail, Chat and Discussion forums 
in WebCT Vista. The instructor also provides clear guidelines and tips about expectations concerning 
participation, interaction, and the assignment and provides feedback and support in a timely manner 
to ensure learning and understanding. 


©Academic Publishing International Ltd 

Janet McCracken et at. 

As shown in Figure 3, each case is divided into different rounds, and each round is divided into 
Disclosure and Tasks sections. In the Round 1 Disclosure, the students receive information about 
patient’s medical history, dental history (including Odontogram and radiographs, see Figures 3 and 4), 
oral health care routine and oral self-image. In Round 1 Tasks, students are asked to work with their 
team members to answer different questions, to apply the skills obtained from the previous blocks and 
identify their client’s risk factors for periodontal disease within their teams. Then they move on to the 
analysis of their client’s clinical and radiographic findings with their team and continue their teamwork 
to arrive at an evidence-based dental hygiene diagnosis and dental hygiene care for their patient. At 
the conclusion of the case, students are asked to develop a further plan of action for their client’s 
continuing care. The same format is used for Round 2. The subsequent second case study is more 
complicated than the first to provide learners with additional challenges. 

• 'XXX • — — * 

SAssessment and Treatment Planning f 

If-acuity oTDcntrttfy 

• - ■ 

Advanced Periodontal Diseases 

Cases > Sample Case—Linda > Round 1 

Round 1 Disclosure % 


Round 1 Tasks 

Round I Sample Responses % 

Figure 3: Sample virtual patient case 

Figure 4: Radiograph for sample virtual patient 

Lessons Learned. While designing a virtual patient case can be complicated and costly, designers 
with limited budget and resources can still develop good cases using simple text and images. What is 
important is to make the cases as interactive as possible, and to create relevant questions and 
explanations tailored to the clinical reasoning process. The instructor uses Virtual Patient in the 
course to enhance the learning experience and to aid in the transition from simulation to practice. As 
the cases are “real” patient cases, they give learners opportunities to see exactly how the new 
clinician will encounter the condition in practice. In response to the question “if you had more budget 
and time, how you would have developed the cases?” the instructor replied, “More interactive, for 
example, we would have a virtual patient ‘in the chair’ and would have a ‘real’ scenario and see how 
the students ask questions to achieve an assessment. Instead of the information given to them, they 
would have to ask questions to achieve more information. This approach encourages higher level 
thinking processes in contrast to only identifying a ‘diagnosis’. It is more valuable for a student to be 

www.eiel orq 


ISSN 1479-4403 

Electronic Journal of e-Learning Volume 10 Issue 1 2012 

encouraged to solve problems with a systematic methodology that can be applied in any situation 
than to support only the development of a single diagnosis. Students find the cases challenging and 
thought provoking, as they have never seen such advanced conditions, thereby providing an 
opportunity to apply the knowledge they have learned in various courses and focus that knowledge on 
a relevant case. 








C r j 





P '+?> 

Figure 5: Odontogram for sample virtual patient 

Case 3: Improving Problem-solving skills through extended Biology case studies 

Problem-solving and critical thinking are important skills that help to solve relevant real-world practical 
problems and are crucial for students who pursue careers in health sciences (e.g., biochemistry, 
microbiology, physiology, genetics and molecular biology). Biology 200, Introduction to Cell Biology, is 
an online foundational course in undergraduate life science education consisting of reflective 
assignments, self-tests, problem workshops and extended case studies, all of which contribute to 
helping students to develop these skills. 

Each course module is organized around an extended case study. The three case studies count for 
20% of the final grade. The key objectives for the case studies lie in the experience with problem 
solving in life science, experience with putting ideas about biological issues into writing and 
experience with taking collaborative approaches to learning and problem solving across all the course 

Rationale. The rationale for developing the extended Biology case studies is to help student to go 
beyond simply memorizing biology facts by taking constructivism and active learning perspectives. 
They were designed and developed to offer examples of real-world problems and challenge students 
to apply their learning. It is expected that students will begin to understand the connections between 
elements of basic cell biology and real health problems faced by populations around the world. In this 
sense, the case study plays an important role in the overall assessment in the course and provides 
the students with an opportunity to demonstrate their knowledge and apply the knowledge to resolve 
real problems, which are aligned with the course content and its objectives. 

The extended case studies use a “cycle” model as a process guide for student work. The cycle model 
consists of five phases as shown in Figure 6. 

The cycle starts with the initial challenge. In the next two phases students generate ideas as an 
individual or a group, and research additional perspectives for more information and revise ideas if 
necessary. In the fourth phase, students evaluate their findings and exchange feedback from peers. 
As a final phase, students go public with their solutions to the challenge in a measured and thoughtful 
way. This process model teaches students not only how to engage in a particular case, but teaches 
them a general problem-solving framework that will extend well beyond the course to their careers in 


©Academic Publishing International Ltd 

Janet McCracken et at. 

health sciences. For the set-up and submission of the case studies, WebCT Vista Discussion Forum, 
Assessment Tool, Turnitin, and Assignment Drop Box were chosen. As the course is delivered 
through WebCT Vista, the reliable and smooth integration of the case studies within the LMS was a 
primary consideration. 

Apply & Research 

Evaluate & Revise 

Figure 6: BIOL 200 case study cycle 

Assessment strategies: Students are engaged in three different assessments: self, peer and instructor 

assessment. The following are instructions to students. 

* Case Study 1 is worth 5% of your final grade. You will complete a report on your diagnosis of 
malaria in blood samples from a Kenyan Village. 

- Case Study 2 is worth 5% of your final grade. Your team will write a report explaining the 
genetic bases of the different forms of hearing loss that you have investigated in a Nepalese 
community. In addition, your fellow team-members will also be asked to rate your participation in 
the team-based work through a process of peer review. You may gain an additional 1-point from a 
positive peer review. 

■ Case Study 3 is worth 10% of your final grade. Your team will prepare a report for the Global 
Health Organization. You may gain an additional 1 -point from peer review. (BIOL 200, Jan. 2008) 

In addition to the WebCT Vista Assignment Drop Box, Turnitin is employed for the improvement of 

student writing for the final reports of the case studies; Students are expected to submit their reports 

www.eiel orq 


ISSN 1479-4403 

Electronic Journal of e-Learning Volume 10 Issue 1 2012 

through Turnitin before they submit their final reports to the Assignment Dropbox. The main objective 
of using Turnitin in this course is to help students avoid committing plagiarism. The instructor allows 
students to rewrite their reports if any plagiarism is found and submit their updated report to the 
Assignment Drop Box. In this course, Turnitin is used as an educational guide. 

Lessons Learned : One of the drawbacks of the case studies is in students’ unequal participation in 
the group work. Even though each member’s contribution to their case studies is expected, some 
students demonstrated perfunctory and procrastinated participation in their group work. The problems 
stem from various reasons such as the heavy workloads for the course activities, language problems 
as a non-native speaker, being disoriented in problem-solving skills, etc. However, most of the student 
feedback on the case studies is generally positive, and focuses on how their research skills have 
improved, how group work has helped them to share and articulate information with peers, and how 
from the instructor’s regular feedback and stimulated interest by the topics the case studies has 
engaged them more in the discussions. The following comments are cited from evaluations of the 
course taught in September 2010. 

Case study 2 was a lot more interesting and researching the topic of interest helped tie in 

the fundamentals of cell biology. (Student A) 

Unexpectedly, this Distance Education course turned out to be as good or better than the 

regular classroom course! Because of the relatively small group size, we had an 

opportunity for more discussion, group work and regular feedback from Dr. Jane Doe. 

Excellent set of activities and practice problems, and an interesting approach to learning! 

(Student B) 

The success of the course can therefore be attributed to a combination of 3 main factors; 1) frequent 
and meaningful feedback from the instructor to the discussion forum and to student assignments, 2) 
tight alignment of objectives with assessment methodology and strategies and 3) the provision of 
authentic and transferrable practices for students pursuing life sciences. And while our assessment 
design strategy is clearly important, we affirm that no matter how well we design activities or 
assessments, the success of a course cannot be achieved unless the course instructor is actively 
engaged in implementing the strategy. 

Case 4: The Design Wiki; assessing collaborative writing in the context of a multi-year 
community knowledge-building project 

ETEC510: Design of Technology Supported Learning Environments is a graduate level course within 
UBC’s online Masters of Educational Technology (MET). ETEC510 is a core course and each year 
approximately 100 students take the course. The students in the course come from several different 
professional contexts: 75% are in-service teachers in the primary or secondary level; and 25% are 
higher instructors or educators in government and corporate training contexts. A key learning 
outcome of the course is for students to “[djevelop skills in the design of educational media, and the 
integration of design thinking with scholarship in education.” (ETEC510, 2011) This case will focus on 
an individual or paired activity called the ETEC510 Design Wiki, an activity worth 20% of student final 
mark, where students collaborate on an extensive community knowledge base that is public to the 

Rationale : The rationale for the Design Wiki assignment activity is to provide a space where all 
ETEC510 students will engage actively over the duration of one semester, with the twin goals of 
learning and communication in a publication medium that affords editing, linking, and dialogue as 
persistent and critical elements in the production of useful knowledge (Scardamalia & Bereiter, 1994). 
Alongside the course materials created by the course authors (Mary Bryson and Jeff Miller), the 
Design Wiki materials are a knowledge base that is many times larger than the original course, and 
students express a strong sense of ownership and responsibility for the material that they have 
authored, edited and curated year to year. By running this activity outside of the UBC’s Learning 
Management System, WebCT Vista, we have been able to preserve student work beyond the 
duration of a course as well as provide public access to this growing educational knowledge base 
developed by MET students, (ETEC510 Design Wiki site, 

For the Design Wiki assignment, we use Mediawiki because it is open source software. For many of 
the educators in the MET, exposure to open-source options is quite useful, particularly if they work in 
cash-strapped school districts. It is worth noting that students have regularly gone on to transfer this 


©Academic Publishing International Ltd 

Janet McCracken et at. 

collaborative knowledge-building approach into their own classroom practice, often within days of 
starting the activity. 

Assessment strategies : The assessment criteria for the Design wiki were designed to encourage 
students to engage in collaborative knowledge building with their peers. Students are asked to 
consider the contribution they are making to the Design Wiki as well as the quality of their writing. For 
new entries, students are assessed on three main components; is it a meaningful contribution to the 
Design Wiki, is it educationally significant, and does it provide a unique critical perspective. Students 
are also assessed on the nature of the resources they contribute; are they diverse, useful and 
connected to the topic. Finally, students are assessed on the quality of their writing in terms of the 
Wikipedia format we ask them to use, as well as for clarity and for proper documentation. For 
revisions of existing entries, students are assessed in a similar fashion on the focus and resources of 
the entry, but they are assessed on a rationale they write to justify their revisions. Many of these 
rationales are now attached to secondary pages (discussion pages) that are created alongside the 
main page for an article in MediaWiki. These sub-textual pages make it is possible for students and 
other readers to see both the top-level article and the messy process of knowledge creation that goes 
on beneath the surface within this collaborative writing space. 

One of the most important tools within the wiki for instructors is the history page that tracks the 
revision history of an article. By looking at and comparing versions, it is easy for the instructor to 
identify student edits as well as to assess the amount of time students have put into the authoring or 
editing. The history page, along with the discussion page for each article provides context relating to 
the writing process. These pages also provide students with a means to contribute their own 
responses to the work developed by their peers or to reflect upon their own work. It has been 
interesting to see that students end up receiving feedback from their peers, even though we don’t 
require that students provide such feedback to one another. 

Lessons Learned : The growth of the Design wiki has created challenges relating to the structure of 
the activity and our assessment approach. Starting from a state of tabula rasa in the first year, the 
Design wiki has grown enormously and in 2011 has over 230 topics, some of which have been edited 
and revised multiple times. Close to 500 students who have taken the course in the last 5 years and 
made contributions to the Design Wiki. The activity has been revised and expanded several times 
moving from a focus on the creation of new entries, to revision of existing entries, to revision of the 
overall organizational structure of the Design Wiki. Students engage with the Design Wiki as authors 
and editors, and students now often curate areas of the Design wiki so as to build coherence, look for 
overlap between entries and to look for opportunities to improve the quality of this peer-authored, 
collaborative knowledge base. 

The Design wiki has also prompted students and instructors to discuss the nature of writing in a 
shared space where it is possible for one student to make edits and revisions to another student’s 
work. And while writing in shared spaces can create tensions between students, particularly if they 
feel that their work has been overwritten or changed, the negotiation of what ends up on the main 
page of the entry, and the process by which revisions come about, leads to critically engaged and 
engaging conversations both within the Design Wiki and in the discussions that are posted to WebCT 
Vista by the students. Perhaps most importantly, the experience of negotiating writing in this 
community site has an impact on how these students, who are themselves teachers, design spaces 
and activities to support learning with their own students 

3. Discussion and conclusion 

We have seen in the cases presented that clearly articulated assessment strategies are vital to the 
effective design of online courses and programs. The peer assessment case demonstrates a 
program-level solution to the need to provide a tool that assesses professional skills in group-level 
and individual-level performance within an online context. The virtual patient case illustrates how low- 
tech resources used within a phased set of “rounds” help students to engage in authentic practice. 
The design of extended case studies in the problem-solving case in Biology shows how an iterative 
process model for problem solving supported by discussion and feedback contributes to an effective 
online learning experience for students. And finally, the Design Wiki case illustrates how an open- 
source environment designed for the purpose of encouraging collaborative learning within a well- 
specified assessment strategy leads to a persistent shared workspace that has a life beyond a single 

www.eiel orq 


ISSN 1479-4403 

Electronic Journal of e-Learning Volume 10 Issue 1 2012 

While some argue (Norton, 2004) that making assessment criteria more explicit in higher education 
may have a deleterious effect on students' learning, our design practice demonstrates that 
establishing meaningful learning activities aligned to course objectives as a purposeful and iterative 
collaborative process of working with instructors from initial design through to delivery and evaluation 
is an effective design practice. 

We have seen how discipline-specific practices and teaching approaches must be well understood in 
order to develop a rationale for an assessment strategy in two of our cases that are designed around 
systematic diagnostic problem solving and use both generic and discipline-specific models in their 
assessment strategies. 

All of our cases are concerned with how best to match the affordance of technologies and tools that 
can be used and customized to meet the purpose for online assessment. The design wiki tools in 
particular provide tracking and revision histories that afford visibility of individual contributions to the 
collaborative learning process. 

Our current work continues to further articulate the design principles and the common elements within 
and between the cases in order to develop a framework that designers can use to guide the design of 
online assessment strategies. We are also interested in developing a more comprehensive view of the 
online learning environments through focusing on the experiences of the students and the instructors 
as a way of improving our practice and in considering new models of assessment. The process of 
articulating and sharing our design practice has had a positive impact on our teams’ awareness of the 
critical role of online assessment design in our work. 


American Dental Education Association. Exhibit 7: competencies for entry into the profession of dental hygiene. J 
Dent Educ 2004; 68(7):745-9 

Anderson, T. (2004). Towards a Theory of Online Learning. In T. Anderson & F. Elloumni, (eds.) Theory and 
Practice of Online Learning, (pp 33-60). Athabasca: Athabasca University Press. 

Biggs, J. (2003) Aligning teaching and assessing to course objectives, paper presented at the International 
Conference on Teaching and Learning in Higher Education: New trends and Innovations, University of 
Averio, 13-17 April 2003. Available online at: 
Biggs, J. B. (1999). Teaching for quality learning. Buckingham SRHE: Open University Press. 

Brown, J.S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. 
EducationalResearcher, 18(1), 32-42. 

Brown, S. & Glasner, A. (1999). Assessment Matters in Higher Education, Buckingham: Open University Press 
Bowden, J., & Marton, F. (1998). The University of Learning: Beyond Quality and Competence in Higher 
Education. London, Kogan Page. 

Boud, D. & Falchikov, N. (2007) Aligning assessment with long-term learning, Assessment and Evaluation in 
Higher Education, Vol 31, Number 4, August 2006, pp 399- 413 
BIOL 200: Introduction to Cell Biology. Course description, 

Canadian Association of Public Health Dentistry (CAPHD). Discipline Competencies for Dental Public Health. 

Edmonton, AB: CAPHD, 2008, Available from: 

Dental Hygiene Educators Canada (DHEC), Canadian Dental Hygienists Association (CDHA), National Dental 
Hygiene Certification Board (NDHCB), Federation of Dental Hygiene Regulatory Authorities (FDHRA). 
National Dental Hygiene Competencies for Entry-to-Practice. Ottawa, ON: Authors, 2008. 

DHYG Professionalism Rubric. 
Donald, J. (2002). Learning to Think: Disciplinary Perspectives. San Francisco, Jossey-Bass 
Ellaway, R., Poulton, T., Fors, U., McGee, JB, & Albright, S. (2008). Building a virtual patient commons. Informa 
healthcare. 30 (2), 170-174 (doi: 10.1080/01421590701874074) 

ETEC510. Course description, 
ETEC510 Design Wiki site, 

Gibbs, G. (1992) Assessing More Students. Oxford: Oxford Centre for Staff Development. Oxford: Oxford Centre 
for Staff Development. 

Gibbs, G. & Rowntree, D. (1999) Teaching in Higher Education; theory and evidence. Introduction, Milton 
Keynes, Open University 

Gayton, J. & McEwan, B. (2007) Effective Online Instructional and Assessment Strategies, The American Journal 
of Distance Education, 21(3), 117-132, Lawrence Erlbaum Associates, Inc. 

Herrington, J., Oliver, R. and Reeves, T. (2003). Patterns of engagement in authentic online learning 
environments, Australian Journal of Educational Technology, 2003, 19(1), 59-71. 

Huwendiek, S., Reichert, F. Bosse, HM., de Leng, BA., van der Vleuten, CP., Hagg, M. Hoffmann, GF., Tonshoff, 
B (2009). Design principles for virtual patients: a focus group study among students. Med Educ. 43 (6), 580- 
8 . 


©Academic Publishing International Ltd 

Janet McCracken et at. 

Kramer, G.A. et al. (2009). Dental Student Assessment Toolbox. Journal of Dental Education. 73(1) 12-35. 
Retrieved from http://www.jdentaled.Org/cgi/content/full/73/1/12 

Lauriilard, D. (2002) Rethinking University Teaching: A Conversational Framework for the Effective Use of 
Learning Technologies. 2nd edition. London: RoutledgeFalmer. 

Levine, S., (2002). Replacement myth. Retrieved May 14, 2004, from 

McNamara, J. & Brown, C. (2009) Assessment of online discussion in work-integrated learning, Campus-Wide 
Information Systems, Vol. 26 Iss: 5, pp.413 - 423 

Norton, L. (2004) Using assessment criteria as learning criteria: a case study in psychology, Assessment & 
Evaluation in Higher Education; Dec2004, Vol. 29 Issue 6, p687-702 

Public Health Agency of Canada (PHAC). Core Competencies for Public Health in Canada: Release 1.0. Ottawa: 
PHAC, 2007. Available on line at and 

Ramsden, P. (1992). Learning to teach in higher education. London: Routledge. 

Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of 
the Learning Sciences, 3(3), 265-283. 

Thorpe, M. (2000) Encouraging students to reflect as part of the assignment process; student responses and 
tutor feedback. Active Learning in Higher Education, vol 1, no. 1, pp. 79-92 

Verma S, Paterson M, Medves J. (2006) Core competencies for health care professionals: what medicine, 
nursing, occupational therapy and physiotherapy share. J Allied Health 2006 Fall; 35(2): 109-15. 

Wiggins, G. & McTighe, J. (2005, 2nd Edition) Understanding. Association for Supervision and Curriculum 

World Health Organization. (2007) A safer future: global public health security in the 21 st century. Geneva, CH; 

www.eiel orq 


ISSN 1479-4403