Evaluative Research | Usability Test
Formative Usability Test - A Prototype for Volunteer Onboarding System
Problem: Provide feedback for the design and flow of the onboarding process for volunteers in a Figma prototype
Summary: Qualitative usability test provided initial feedback on design and flow of the onboarding process for designers, which helped to improve the initial onboarding flow and design prior to handing off to development.
Team: Led research along with 3 other researchers on the team, worked collaboratively with designers, developers, and product manager.
Tools: Figma, Zoom
Background: Pre-launch of a new volunteer onboarding system that is designed for a civic tech organization. Our goal was to provide feedback on the flow and design of the Figma prototype for one of the positions before generalizing it to other positions.
Stakeholders: Civic tech organization leadership
Research Goal: Evaluate the efficiency and clarity of the navigation flow for first-time users.
Research Plan & Methodology
Participants:
Recruited 6 participants from the organization who had never seen the initial design of the onboarding process.
Tasks Tested:
Task 1: Select one practice area from the page.
Task 2: Select the rating for all the skills on the page.
Task 3: Go to the next page and select the dates and times of your availability.
Task 4: Modify search to see the updated results.
Task 5: Recognize and make sense of the error message.
Baseline Metrics Collected:
Task success rate, Time on task (estimate).
Testing Setup:
Prototype Type: Mid-fidelity prototype.
Tools Used: Figma, Zoom
Test Format: Moderated, Remote
Example Vignettes
The following are some examples of vignettes we used in this study. The vignettes are either outcome oriented or socially oriented scenarios of use cases of analytics in the teaching and learning context. Users are asked to rate these vignettes on expected usefulness, privacy concerns, and adoption likelihood.
Poll: While prepping for tomorrow’s CHEM 101 class, Taylor pulls up a Canvas visualization of last year’s responses to a clicker/Tophat poll on the same topic. He sees that 350 of his 395 students chose the same wrong answer in the poll, and decides to add a second clarifying example to the lecture materials. (STEM, large, in-class activity, class prep)
Social: During his 30-student mechanical engineering design class, Bayley monitors team dynamics using a social behavior analytics dashboard (e.g., classification of facial expressions, gestures, body positions). When he sees Team 3 slowing down, he heads over for a check in. (STEM, small, social interactions, class management)
Prereqs: Alex is developing a third-year course in machine learning for the new data sciences major. He reviews assignment and test grades for current majors during their first two years, and he can see that about 30% perform poorly on programming intensive assignments. He expands his early review and extra resources relating to core programming skills. (STEM, small, grades, new course development)
Collaboration: Priya is about to have a coaching session with one of her industrial engineering teams. To prepare, she reviews team communication analytics that show how frequently members communicate and update shared documents, as well as which members are texting and emailing others. This gives her insight into the team dynamics. (STEM, small, social interactions, mentoring)
Key Findings & User Pain Points
Usability Issues Identified
The following summarized the major usability challenges participants encountered.
Recommendations & Design Iterations
Actionable Recommendation:
Modified the design of the error message, focused the highlight on error itself instead of the broader area.
Separated skill pages into several sections to help users feel the sense of progression.
Improved the design of the "Edit skills and availability" page to improve user success rate.
Added search bar on job listing page to help users to explore interested positions.
Outcomes:
We provided feedback to the design team and improved the design accordingly.
Impact & Reflection
Lessons Learned:
Collecting baseline metrics is important to keep track of improvements overtime.
Prioritize improvement based on urgency and level of importance, improvements can be made overtime, it does not have to be made all at once.
It is important to recruit a diverse group of participants to ensure that they are representative of your user group.
Next Steps & Future Considerations
Conduct a subsequent card sort study to better categorize skills lists for all positions to improve skills matrix usability.
Follow the same improved design for other positions' onboarding process.
Collect a user confidence rating score as additional metrics in future usability test.