We are planning on doing some usability testing of the mobile prototype, below we have outlined the test plan. Since we haven't tested this prototype much, we are focusing our initial test plan primarily on finding out how they experience the navigation - not necessarily how they experience the admin theme.
Name of the project: Toolbar testing
Type of study: Moderated
Focus of the study: To get answers to the following high-level questions:
How do users use the toolbar on their mobile devices and on their desktops?
Are the users able to easily navigate to the tasks that they want to perform?
Do users understand the interaction pattern of the toolbar?
How does the experience differ when on mobile and when on desktop?
Note: Since this study focuses on mobile and desktop, to avoid order effect - half of the participants will see the mobile version and then the desktop. The order of tasks will be reversed for the other half.
Number of participants: 5 to 6 existing (Intermediate) Drupal users
Participant profile:
Existing Drupal users who have own a smart phone
Users who have a need to access/update their website on the phone
Haven't seen the new mobile toolbar design
Time frame of every session: 35 minutes
Compensation offered: $20 for external (provided by Acquia)
Pre-Session Questions
- Can you briefly describe your role with regards to your site? And how long have you been using Drupal for?
- Can you tell us about which toolbar that you predominantly use?
- How has been the experience with that toolbar? What are the things that you like the most and the least about that toolbar?
- What are your needs when it comes to accessing/updating your website via your phone?
- Besides calling/ texting, what are the common tasks that you perform on your phone?
Moderator to participant: You are about to see is a prototype. I will give you a scenario and a few tasks to perform on mobile and on desktop. But interact with the prototype as you are doing it on your site. I am particularly interested in getting your feedback about the toolbar (interaction pattern of the toolbar and anything else that you deem important).
Scenario
P1,P3,P5 - Performs this task on a mobile device
P2,P4,P6 - Performs this task on a desktop
Based on a friend’s request you decided to give a new toolbar a try for your website. This is your first time interacting with it. Please take a few moments to explore or do what you would normally do when you are trying something new for the very first time.
Question: So, what are your initial impressions?
Order of tasks for P1, P3, P5
Task 1 (on mobile)
You recently tried a new recipe and want to write that on your website. How would you do that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 2 (on mobile)
One of your menu items says "Recipe", you would like to change that to "Recipes". Where would you go to change that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Moderator to participant: Now that you have tried the mobile version, I would like to perform a few tasks on the desktop.
Task 3 (on desktop)
One of your friends, Jeff Smith wants to help with your website. You want to give him the necessary privileges to edit things that he has written. How would you proceed to do this? Note: He is a registered user on the site
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 4 (on desktop)
Something went wrong and you want to take your site offline, what would you do?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Order of tasks for P2,4,6
Task 1 (on desktop)
You recently tried a new recipe and want to write that on your website. How would you do that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 2 (on desktop)
One of your menu items says "Recipe", you would like to change that to "Recipes". Where would you go to change that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Moderator to participant: Now that you have tried the desktop version, I would like to perform a few tasks on mobile.
Task 3 (on mobile)
One of your friends, Jeff Smith wants to help with your website. You want to give him the necessary privileges to edit things that he has written. How would you proceed to do this? Note: He is a registered user on the site
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 4 (on mobile)
Something went wrong and you want to take your site offline, what would you do?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Post Session
- Now that you have had some time to explore this toolbar, what do you make of it?
- What did you think about the interaction pattern of the toolbar?
- Was your experience same or different on mobile vs. desktop? If it was different, how was it different?
- What was there something that was missing or confusing while using on desktop/ mobile?
- If implemented, how will this UI affect the way you work?
- What are your thoughts about having a toolbar like this, for your desktop experience?
- How does this compare to the toolbar that you are currently using?
- How would you feel if the mobile toolbar was different than the desktop in terms of design?
Participant Briefing:
Welcome, my name is .....
Outline of the session: Session will be a couple questions, a series of tasks, then a couple post-session questions.
Please think aloud
Describe your steps… what you are looking for… etc
Ask for clarification if needed, I will be be neutral throughout the test.
Please provide candid opinions about what you see. My role is of a neutral observer. So anything good or bad you say, it is not going to affect my feelings.
Keep in mind we are testing the prototype, not you As I mentioned that this is a prototype, so remember that it is not fully functional. Some of the things are working and some are not. But assume as if everything is working and go from there. Try to complete the tasks as if you were doing this for real. Spend as little or as much time as you normally would doing these task. It is ok if you cannot complete each task, and we may not get to every task
We will be recording the session, so that might share anything relevant with our team.We just record your audio and the screen, so use mouse when describing an area of screen.
Do I have your permission to record this session?
We may post the highlights from this conversation online. Your full name will not be associated with the data.
This should take about 35 minutes. The time is now ___ so we should be done around ___.
Any questions before we begin?
Feedback
As always before doing the testing we post these on the groups.drupal.org/usability for further feedback. We will revise the draft on the received feedback and start testing.
Anyone who wishes to help out testing is welcome to, although we probably do not have the funds to go beyond our 6 participant count.