2005-10-30 Ben at MIT heuristic test
The MIT usability team did a heuristic review of the Sakai Gradebook on 10/25/2005, based on observing one user test (we were using this initial test to list obvious problems that should be addressed before doing more testing). During the test we encountered several usability issues with the default Sakai style sheet.
One of the big issues that arose in the test was confusion around what was a link. The recommendation was to have the CSS adjusted to make the links underlined. This includes the table headers, which no one realized were clickable.
A related issue was that the 'tabs' in the OOTB template don't look like tabs. Our user interpreted them as breadcrumbs and had a lot of trouble getting from 'My Workplace' to his class. He couldn't tell if he was in his class or not. The presence of a big tab looking graphic in the upper right hand corner of the screen added to the confusion.
I've attached an image that outlines the CSS-related issues we saw in this test.
2005-11-12 Michelle at Michigan style sheets
More Style Sheet User Testing
At Michigan, we did user testing with 8 subjects comparing our old CTools style sheet/skin to a new proposed CTools style sheet/skin. With either the current or new CTools skin, users were asked to correctly identify their location, available "clickable" items, and the function of some user interface controls. Their answers yield an overall "score" for each design. The average score for correctly answering the questions using the current CTools interface was 78.5%. The average score for correctly answering the questions using the new CTools interface was 72.2%.
Users rated the designs equally as to how confusing or clear they are (3.5 out of 5, 1 being 'Confusing' and 5 being 'Clear'). However, users rated the new design more highly when asked about how appealing the designs are (4.25 out of 5 for the new design; 3.5 out of 5 for the current design; 5 being 'Appealing').
Users were most challenged with the same areas of both designs:
- identifying what site they are in
- identifying some clickable icons are clickable
- identifying the function of the "reset" icon
- to a lesser degree, predicting what would happen when clicking the left-hand menu entry for the tool that they are currently in
These areas can be made less challenging by:
- increasing the size and contrast of items that identify the current site
- increasing the size and perceived "clickability" of icons by making them look more 3-dimensional
- adding a text label to the "reset" icon, or choosing another icon design
- or preferred, removing the "reset" icon and making the left-hand menu entry for the tool you are currently in clickable to "reset" the tool
The full results report, with the screens tested, the evaluation protocol, and the data spreadsheet are posted here for reference for others who want to run similar testing with their users.
| Name | Size | Creator (Last Modifier) | Creation Date | Last Mod Date | Comment | ||
|---|---|---|---|---|---|---|---|
 |
 ResultsWithScreens.pdf |
1.71 MB | Aaron Zeckoski | Nov 21, 2005 | Nov 21, 2005 | UM Style Sheet/Skin User Testing Results | |
|
 Evaluation Data.xls |
63 kB | Aaron Zeckoski | Nov 21, 2005 | Nov 21, 2005 | UM Style Sheet/Skin User Test Data Spreadsheet | |
|
 Evaluation Protocol.doc |
47 kB | Aaron Zeckoski | Nov 21, 2005 | Nov 21, 2005 | UM Style Sheet/Skin User Test Evaluation Protocol |
Comments (3)
Nov 16, 2005
Gonzalo Silverio says:
Some history this skin was produced for Vivie by a contractor. For that reason I...Some history - this skin was produced for Vivie by a contractor. For that reason I have been loath to change it, don't rightly feel it is something that I could do without consultation.
I'd be willing to make changes to portal and tool if Ben's findings constitute a reason to do so and this DG concurs. Do let me know.
Nov 17, 2005
Michelle Bejian Lotia says:
I think these findings are pretty consistent with the major findings from the us...I think these findings are pretty consistent with the major findings from the user testing on local skin changes I did for UM. One suprise - users had little problem identifying what feature they were in on the left menu when the current feature was underlined and the others were not. (This is in a local design Gonzalo did for our implementation.) That was quite inconsistent with what we're told about link underlinining (the current feature should NOT be underlined if it's not clickable) but I think it worked for users because it made that feature pop out more.
Nov 17, 2005
Aaron Zeckoski says:
This pretty much lines up with the dicussion that we had a couple of weeks ago o...This pretty much lines up with the dicussion that we had a couple of weeks ago on the sakai-dg-ui list. Links underlined, better consistency, things of that nature. I think an updated stylesheet would be nice to have.