Evaluating the usability of GitHub's new Pull Request experience through a comprehensive remote study
In this comprehensive usability study, I collaborated with GitHub to evaluate the effectiveness and user-friendliness of their new Pull Request experience, focusing on recently introduced UI enhancements. The study aimed to assess long-time GitHub users' learning curve and comfort in navigating the platform, complete code review tasks efficiently, and gather valuable feedback to improve the overall developer experience.
This is a research project where I wore the following hats:
Usability Testing | Remote A/B Testing, Pre-Test Questionnaire, Post-Test Questionnaire, System Usability Score (SUS) Questionnaire
Data Analysis | Excel
Visualization | Figma
Duration | 10 weeks
Team | Collaboration with 4 classmates from Human-Centered Design & Engineering, UW
GitHub's Pull Request feature is fundamental to how teams review and improve code on GitHub.
Start a new feature or propose a change to existing code with a pull request—a base for your team to coordinate details and refine your changes.
As GitHub introduces updates and enhancements to this feature, it is essential to evaluate the usability and effectiveness of these changes to ensure a seamless and productive user experience.
Most crucially, the introduction of new UI elements and interaction patterns may inadvertently create usability hurdles, leading to confusion, reduced productivity, and user frustration.
Assessing the impact of these changes on existing users is crucial to identify areas for improvement and ensure a smooth transition to the updated experience.
This study will examine how effectively existing users can complete essential tasks within the updated UI.
By conducting a comprehensive usability study, GitHub aims to gain insights into the strengths and weaknesses of the new Pull Request interface.
Additionally, the study will explore user attitudes towards the new interface, unveiling potential pain points, usability bottlenecks, and areas where the design may fall short in meeting user expectations or aligning with established mental models.
By identifying these usability challenges and gathering user feedback, GitHub can make informed decisions to refine the Pull Request experience, ensuring a seamless and productive workflow for developers.
The evaluation will primarily utilize a remote A/B testing approach. Participants will perform predefined tasks using both within the GitHub Pull Request Experience.
After completing the tasks assigned during each scenario, participants will provide open-ended feedback, insights, and any challenges encountered to inform the evaluation process.
After completing tasks within each scenario, participants will complete the SUS questionnaire. This will help gather quantitative data on participants' perceptions of usability, effectiveness, and learnability of both the UIs.
We recorded each session to capture participants' live interactions with the interface, to aid analysis and gain valuable insights. Additionally, interviews and questionnaires following scenarios enhanced our understanding of the user experience.
While all participants finished the tasks in both the old and new UI, a few attempts were made for tasks in the new UI.
The System Usability Scale (SUS) scores showed clear differences between the existing and new UIs, with the existing UI scoring higher at 77 compared to the new UI's 67.
50% users faced difficulties locating the comment widget in the new UI.
The position of the comment widget moved from the extreme left (existing UI) to the extreme right (new UI) causing pause and confusion in users.
Additionally, a lack of contrast between the avatar and the background when in light mode raised visibility issues for the comment widget.
67% users faced difficulties resolving a comment in the new UI.
The resolve icon is gray in color and seems unintuitive and "disabled" to users until they hover over it and see the tooltip that says 'Resolve conversation'.
