Usability Evaluation of the Course Management Features of Sakai

Usability Evaluation of the Course Management Features of Sakai

Speaker(s): Jonathan Howarth, Aaron Zeckoski

Date: Thursday 2:05 pm - 2:45 pm
Room: INTL 3

Session Abstract

We present the results of a usability evaluation of the course management features of Sakai. The evaluation consists of lab sessions with five representative users and a walkthrough by a practicing usability professional. The usability problems and suggested solutions have been reviewed with three Sakai developers.

Presentation Materials

The presentation consists of the following materials:

1. PowerPoint deck
2. Videos accompanying the PowerPoint deck - If you extract the videos in this zip file and place them in the same directory as the PowerPoint deck, you should be able to click on the links in the PowerPoint slides to play the videos.

We have generated the following two reports of the usability evaluation:

1. Summary Report - This is a summary of the full report that contains brief descriptions of the evaluation setup and the major groups of usability problems.
2. Full Report - This is the full report that contains detailed information on the setup of the evaluation, so that it can be repeated for successive versions of Scholar. In addition, this version contains detailed information on the usability problems experienced by participants in the study.

To Address Questions Asked After The Presentation

After I gave the presentation, an audience member asked the following two questions:

1. How was I able to make claims about the usability of Scholar (Sakai) when the evaluation only included 5 participants?
2. What are we supposed to do with this evaluation?

These are reasonable questions, and I would like to answer them here for the benefit of individuals who were not able to attend the presentation.

There are two types of usability evaluations: formative and summative. This evaluation was a formative evaluation. Formative evaluations are performed during development to help improve the usability (specifically, the interaction design) of a system. Formative evaluations generally involve 3 to 8 participants and are intended to help usability engineers find usability problems. Summative evaluations are performed after development to compare systems. For example, an institution considering adopting Sakai or purchasing Blackboard would use the results of a comparative summative study as a data point in making a decision. Summative studies are more rigorous and require a larger number of resources. For more information on the difference between the two types of evaluation, please read the introduction of the following paper:

Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 373-410 (available at http://www.leaonline.com/doi/pdf/10.1207/S15327590IJHC1501_13)

We intend the formative evaluation that we performed to serve as a checkpoint evaluation. We provide the tasks and measures in the full report, so that the individual institutions in the Sakai community can repeat the evaluation in an iterative fashion as they update their local instances of Sakai to get information on usability problems experienced by their own users. We included counts of usability problems, numbers of assists, and time on task values as additional data points. We do not make any claims based on statistical analyses of these data; we simply refer to them for trends. Other institutions can use these values as a rough measuring stick for their initial evaluations. Thereafter, they can compare data between evaluations to get an idea of if they are improving.

Podcasts

• Session leaders are encouraged to post their podcasts on the main Atlanta Podcasts page (a central repository of podcasts) and may also choose to link to them from their session page. See the main Atlanta conference wiki page for more details.

Additional Information

• Session leaders are also encouraged to appoint a session convener, a podcast recorder and/or a note-taker and post the minutes of their session on a Page (see Add Page link near top-right.)

• Participants and Session Leaders are encouraged to post Comments (see Comment form below) or create additional Pages as needed to facilitate collaboration (see Add Page link near top-right.)

• • Child Pages for this session (Added Pages will automatically appear in this list)