Since the beginning of January, Mairin Duffy and I have been working on a usability test plan to test the installer program for Fedora, Anaconda. The usability testing will be taking place at DevConf in the Czech Republic in February. I have finished the test plan and this blog post is about the process of how it was created and contains all of the final documentation.
1) User Survey
We created a user survey the user will take before the usability test is run. This will ensure that we have the correct users that match our user profiles to test the interface with our specific use cases. The type of users we would like to test are:
- Novice users who may be new to linux who just want to install a desktop system
- Experienced linux users who may dabble in system administration, are very technical, and like trying out different technologies
- Professional system administrators who work with advanced storage devices and configurations, typically on the job, not for fun
Once these three user profiles were identified, I wrote the first draft of the survey. Once I was done with the first draft, it was sent to the anaconda-devel-list for feedback. After getting a lot of valuable feedback, Mairin and I went through every suggestion we were given and discussed whether we should make changes or not and how we should change it. I went on to make the second draft of the survey, sent it off for feedback, and the only feedback we received was in regards to typographical errors and changes. I made these small changes and the final draft was done. The first draft of the survey, suggestions, and final draft can be found on the Fedora wiki page here.
2) Test Scripts
After Mairin discussed our potential usability use cases with the Anaconda development team supervisor, David Cantrell, they decided on the following 5 use cases:
- Non-English/US Install: 4 of the novice users will simply go through the installation process without any special requirements or customizations.
- Dual Boot Install: 1 of the experienced/professional users will install Fedora 18 alongside a Windows install.
- Customized LVM Install: 1 of the experienced/professional users will install Fedora 18 with a specific LVM disk layout we provide for them.
- Customized RAID Install: 1 of the experienced/professional users will install Fedora 18 with a specific RAID layout we provide for them.
- Preserve /home directory: 1 of the experienced/professional users will install Fedora 18 on top of Fedora 17, which was already installed on the Virtual Machine, without deleting or overwriting the Fedora 17 /home directory.
I created a separate test script for each of these 5 use cases. The ‘Introduction’ and ‘Conclusion’ of each test script is the same, but the ‘Testing’ section of the script is different. The rough drafts and final test scripts can be found on the Fedora wiki page here.
3) Administrator Worksheet
I created an administrator worksheet that the administrators of the usability testing can have with them during the tests to take notes. This worksheet is supposed to aid them in taking notes by giving them some structure and giving them specific things to note throughout the installation process. The worksheet can be found on the Fedora wiki page here.
These are the main three components of the usability testing for Anaconda. Since this is done I am now reading several reviews on the Fedora 18 installer and am categorizing and prioritizing the problems people had so we can address the most critical ones in Fedora 19. I will also be going through some usability bugs and coming up with potential solutions to these. Things are moving forward quickly and it feels good! 🙂