Evaluating the usability of GitHub's new Pull Request experience through a comprehensive remote study
Brief
Evaluate the effectiveness and user-friendliness of GitHub's new Pull Request experience, focusing on recently introduced UI enhancements.
Result
Assessed the learning curve and success of experienced GitHub users in completing code review tasks efficiently using new UI. Shared recommendations with GitHub.
This is a research project where I wore the following hats:
Usability Testing | Remote A/B Testing, Pre-Test, Post-Test, and System Usability Score (SUS) Questionnaires
Data Analysis | Excel
Visualization | Figma
Timeline | March '24 - June '24
Team | 2 moderators, 1 product designer, 1 PM
GitHub's Pull Request feature is fundamental to how teams review and improve code on GitHub.
To start a new feature or propose a change to existing code developers use a pull request which acts a base for a team to coordinate details and refine changes.
As GitHub introduces new UI elements and interaction patterns, it is essential to evaluate the usability and effectiveness of these changes.
Changes to an expected experience may inadvertently create usability hurdles, leading to confusion, reduced productivity, and user frustration. And assessing the impact of these changes will ensure a smooth transition to the updated experience.
This study will examine how effectively existing users can complete essential tasks within the updated UI.
By conducting a comprehensive usability study, GitHub aims to gain insights into the strengths and weaknesses of the new Pull Request interface.
Additionally, the study will explore user attitudes towards the new interface, unveiling potential pain points, usability bottlenecks, and areas where the design may fall short in meeting user expectations or aligning with established mental models.
By identifying these usability challenges and gathering user feedback, GitHub can make informed decisions to refine the Pull Request experience, ensuring a seamless and productive workflow for developers.
The testing will use a remote A/B testing approach. Participants will perform predefined tasks using both, current and new UIs under the observation of a moderator and an analyst.
After completing the tasks in each scenario, participants will provide open-ended feedback, insights, and any challenges encountered to inform the evaluation process.
A participants will complete the SUS questionnaire. This will help gather quantitative data on participants' perceptions of usability, effectiveness, and learnability of both UIs.
We recorded each session to capture participants' live interactions with the interface, to aid analysis and gain valuable insights. Post-test, semi-structured interviews enhanced our understanding of user experience.
The measures used collected both quantitative and qualitative data.
While all participants finished the tasks in both the old and new UI, a few attempts were made for tasks in the new UI.
The System Usability Scale (SUS) scores showed clear differences between the existing and new UIs, with the existing UI scoring higher at 77 compared to the new UI's 67.
50% users faced difficulties locating the comment widget in the new UI.
The position of the comment widget moved from the extreme left (existing UI) to the extreme right (new UI) causing pause and confusion in users.
Additionally, a lack of contrast between the avatar and the background when in light mode raised visibility issues for the comment widget.
Enhance the visibility of the comment widget by adjusting contrast of avatar colors
Improve discoverability of the comment widget by positioning the option more prominently
To maintain user familiarity retain the comment widget on the left side as in the existing UI
67% users faced difficulties resolving a comment in the new UI.
The resolve icon is gray in color and seems "unintuitive" and "disabled" to users. The tooltip on hover that says 'Resolve conversation' helps.
Enhance visibility by using a distinct color (like green) for the resolve icon to indicate its active state
Maintaining text in the button can assist users in understanding the functionality of each interface element
.png)
