Assignment Creation on MathWorks | UX Case Study |
Assignment Creation on MathWorks | UX Case Study |
Aim: To design a GUI for assignment creation on MATLAB.
Process:
Research (Semi Structured Interviews, Contextual Inquiry, Affinity Mapping, Task Analysis, Comparative Analysis)
User Needs Analysis (Personas, User Journey Maps, Design Implications)
Design (Prototyping, User Testing)
Team: Collaborated with Arpit Mathur and Joe Geiger from Mathworks, and Charlotte Koh, Zezhi Guo, and Yuqi Hu from Georgia Tech.
Tools: Qualtrics, Miro, Figma, Maze
Background
Problem Statement
Research Questions
MATLAB is a programming platform for engineers and scientists. It is the industry standard for engineering education.
There is currently no existing solution to streamline the process of assignment creation.
Lab instructors need a way to create MATLAB assignments efficiently and easily because their current methods of creating MATLAB assignments is time-consuming, inefficient, and lacks effective mechanisms for ensuring academic integrity.
What are the characteristics of lab instructors and their classes?
How does MATLAB fit into the overall lesson plan?
What is the user experience of professors creating and grading MATLAB assignments?
How are MATLAB assignments created and graded?
What are their pain points and challenges? What works?
What solutions exist already?
What is the reason professors use MATLAB over other softwares?
Process
1) Context and User Needs Research
Semi Structured Interviews
Contextual Inquiry
Affinity Mapping
Task Analysis
Comparative Analysis
2) Outlining User Needs
Personas
User Journey Maps
Design Implications
3) Prototype Design and Evaluation
Ideation
Sketching
Lo-fi prototyping
Hi-fi prototyping
User testing and feedback
Research
Semi-Structured Interviews and Contextual Inquiry
Method Justification
By enabling us to gain insights into the pain-points, successes, and challenges of creating and grading MATLAB assignments, interviews would give us a more detailed understanding of the variety of use cases, which was necessary for us to better understand both our users and our problem space.
Information Goals
Insights into the process of creating and grading MATLAB assignments
Types of assignments and use cases that involve MATLAB
Existing solutions and tools
Role
Brainstorming questions, conducting 2 interviews, note-taking.
Procedure
We formulated a set of research questions, and broke these down into sub research questions. From these, we derived questions for the interview and contextual inquiry. We ensured that we had a rationale for each question so as to optimize the questions asked.
In order to get a holistic understanding of the problem area, we had to get insights from all our stakeholders. We recruited participants through purposive sampling by emailing professors teaching aerospace, electrical, and mechanical engineering at Georgia Tech. We conducted 5 interviews with professors and 4 interviews with their TAs.
A contextual inquiry was embedded into each interview. We asked professors to walk us through their assignment creation process. We noted down the materials they referenced, the tools they used, the components they included, and the steps they followed.
Data
We organised our data from interview notes into an affinity map on Miro. The high-level themes we identified are as follows:
Why MATLAB?
Limitations of MATLAB
Characteristics of Assignments
Assignment Creation Process
Assignment Grading Process
Difficulties when Grading
Comparative Analysis
Information Goals
To identify any gaps in the market and to evaluate existing systems to identify what works and what doesn’t, in order to better design a solution for our user group.
Limitations
There is no primary competitor in the market as there is no existing solution for our specific problem area. Thus, we had to analyse our secondary competitors which provide solutions for a more broad problem scope.
Details
We did a comparative analysis of 4 secondary competitors - MATLAB Grader, Coassemble, Google Colab, zyBooks. For each competitor, we assessed the following:
Strategic Level
Strengths
Weaknesses
Scope level
Target Audience
Use Cases
Structure Level
UI
Collaboration Features
Pricing
Findings - Task Analysis
Information Goals
To gain a low-level understanding of the exact steps involved during assignment creation and grading so as to identify pain-points, inefficiencies, and patterns, with the aim to determine how we can optimize and streamline the process.
Role
Brainstorming and implementation.
Details
We used our findings from the contextual inquiries and interviews to outline the steps.
Assignment Creation
Our research highlighted the 3 main steps in the assignment creation process - Brainstorming, Organizing, and Implementing.
Brainstorming usually involves looking through the course syllabus, determining learning objectives for a specific assignment, and deciding how these can be best tested.
Organizing involves looking at multiple sources such as past assignments, textbooks, and online to collate material for the assignment.
Implementing involves the use of various tools such as MS word, powerpoint, MATLAB, and LaTeX for the creation of the assignment.
Findings - Themes Revealed
Tediousness & Inefficiency
Our findings highlight the inefficiencies involved in the assignment creation process. There is constant cycling between multiple platforms, and repeated referral to various sources of information. Instructors have expressed concerns about the time-consuming nature of the assignment creation process, as well as the technical challenges associated with it. Addressing these issues through an efficient assignment creation solution can save instructors a significant amount of time in the long run.
Preparation
There is a significant portion of the assignment creation process devoted to preparation for the assignment. This includes evaluating the course syllabus and learning objectives, reusing/recycling past assignments, and coming up with new and innovative ways of testing the student’s knowledge. Professors have expressed how this is often the most difficult part of the assignment creation pipeline, and hence there is need for intervention.
Variation in Evaluation Methods
Our findings highlighted the various types of assignments. Professors included a variety of components in their assignments, including MATLAB scripts, text, images, graphs, and CSV data files. While some professors only evaluated the output of the students code, other professors required students to write. Some professors ran the students code, while other professors just went over it. Our solution would hence need to accommodate various methods of assessment.
Focus on Assignment Creation
While our findings did highlight difficulties in the grading process, our comparative analysis led us to discover that there already exist solutions to solve the inefficiencies in the process, such as autograding softwares like gradescope. Thus, we decided to focus on steamlining the assignment creation process.
User Needs
Personas
We use our findings to construct two user personas.
The Innovative Instructor - Lisa
Background: Lisa is 42 years old, and is an experienced instructor in the field of dynamics and control. She values creativity and innovation in her teaching methods.
Approach to Assignment Creation: Lisa leverages various sources to create assignments, including textbooks, online materials, fellow colleagues and her own experience. She uses LaTeX and MATLAB scripts to design assignments, incorporating various components like problem statements, starter code, CSV data, and images. She often uses old assignments as a starting point and seeks inspiration from them.
Key Goals: Lisa aims to create assignments that assess student’s conceptual understanding and critical thinking. She refrains from using AI technology to generate assignments and believes in personally refining questions to avoid cheating.
Assignment Format: She provides her students with starter MATLAB script for the problem, and expects students to edit and submit the completed MATLAB code with a PDF report explaining the process.
Assessment Method: She emphasizes the flow and logic of students’ code and grades them based on rubrics. She also runs the code to check the output.
Pain points:
It is time-consuming and difficult to create new problems that cater to a wide range of academic abilities.
It is challenging to strike the right balance between creating assignments that are descriptive enough to guide students and creative enough to engage them.
She use a variety of tools and software to create assignments (e.g., MATLAB, LaTeX, Word). She also uses components such as problem statements, starter code, CSV data, and images. Coordinating and managing these elements can be complex.
User story: As an innovative instructor, I want a way to efficiently manage and integrate different sources and tools to simplify the process of generating problems for assignments, so that I can focus on enhancing the learning experience.
The Resourceful Educator - Mike
Background: Mike is 57 years old, and he is a mechanical engineering professor who is known for his resourceful and traditional approach to assignment creation.
Approach to Assignment Creation: Mike uses Word or Powerpoint to craft assignments. He recycles old assignments, but he carefully refines and updates them.
Key Goals: Mike’s primary goal is to have efficiency in the assignment creation process and for students to obtain a concrete output for their code.
Assignment Format: He provides his students with a problem on a PDF and expects them to submit MATLAB code for the same along with a PDF of the final output of the code.
Assessment Method: He evaluates the output of the student’s code.
Pain points:
It is time-consuming to download and run all the individual codes, so he is only able to evaluate the outputs.
The students often share their outputs with each other. Because he does not run the students code, he is unable to determine whether the student’s code produces the output they submit in the PDF.
There is a lack of automation in the assignment creation process. He often ends up doing repetitive manual work when generating assignments, which can be inefficient.
User story: As a resourceful educator, I want a way to automate assignment creation to allow for more efficient creation of assignments, while also preventing plagiarism from previous assignments, so that I can create assignments more efficiently.
Design Implications
Based on our personas and journey maps, we come up with the following design recommendations to be followed when designing the solution:
Design
Ideation and Sketches
For our ideation process, each team member came up with 5 ideas using the ‘Crazy 8s’ ideation process. Each idea was created keeping in mind our devised design implications. We presented them to each other, combined ideas, and eventually came up with 2 top ideas to pursue for our sketching phase.
Our 2 ideas were:
1.Assignment Marketplace
2.Modular Assignment Editor
Idea 1 - Assignment Marketplace
Design Idea
Assignment management system where professors are able to easily store, categorize, and modify their past assignments.
Assign categories to their assignments such as field, difficulty level, learning outcome, student group characteristics, keywords to organize them better.
Allow tracking of changes and easy modification of past assignment problems.
Allow users to drop their past assignments and the system will automatically scan and make them digital for easy modification.
Create a marketplace where instructors can purchase or exchange OTHER PROFESSORS assignment problems, including partial or full assignments. An online library that encompasses all MATLAB assignment problems created by professors from all over the world.
Use an in-app currency to motivate professors to trade or contribute assignments.
Design Implications Addressed
Import and modify existing assignments.
Collaborate and brainstorming with colleagues.
Catering to different academic abilities.
Idea 2 - Modular Assignment Creation
Design Idea
Modular assignment creation platform where you can drag and drop various elements like code blocks, text, images, tables, CSV files, etc, so that professors don’t need to resort to multiple tools.
Allow users to vary parameters quickly and test their solution code to check its output, logic and flow.
Design Implications Addressed
Import and modify existing assignments.
Lo-Fidelity Wireframes
As one of the biggest pain points for professors is organizing their assignments and brainstorming questions, we could create a platform by which professors exchange and brainstorm assignments through a marketplace, while also providing them with a repository to collate all of their individual materials. The design of an assignment marketplace is substantiated by our insights, as it aligns with the professors needs for an organizational and collaborative platform.
Thus, for our final idea, we created lo-fidelity wireframes to show the flow of uploading an assignment to the repository, and purchasing an assignment from the marketplace.
User Testing for Lo-Fi Wireframes
Goal
Identify critical issues to address, to guide development of high-fidelity prototype
Methodology
2 Professors
Moderated session
Task based scenarios
Upload an assignment
Purchase an assignment
Role: Moderator
Findings
Hi-Fidelity Prototype
Based on the design system of Mathworks, we created our hi-fidelity wireframes. In the following video demo, the flows of a) uploading an assignment, b) publishing it to marketplace, and c) purchasing an assignment from the marketplace have been demonstrated. We addressed our design implications from our user testing in the following ways:
Persisted with the points system
Enabled full but restricted (PNG) view of assignment
Created onboarding flow
Included more obvious indications of point system (eg. ‘Notice’ message next to assignments for purchase)
Incorporated rewards for filling in assignment details to prevent junk uploads
User Testing for Hi-Fidelity Wireframes
Goal
To evaluate usability of the platform
Methodology
Cognitive walkthrough - 2 experts
Heuristic Evaluation - 1 expert
Task based scenario - 2 users
Role: Moderator for Cognitive Walkthroughs and task scenarios, Note taker
Success Metrics
Task Performance Metrics
Task Success
Efficiency (average number of steps to complete)
Self-Reported Metrics
Rating scale of 1-5 (5 is most difficult)
Findings
Success Evaluation
4/4 successful task completions.
Difficulty rating <2.
Task 2 took an average of 1.13 steps more than the minimum number of steps.
Conclusion
User quotes like “I would totally use this.”, “This seems pretty straightforward.”, “This solves a lot of problems!” showed us that our prototype and idea have been successfully executed to solve the problem statement! We created this word cloud from words that came up in our user testing sessions:
Future Work
Research and test alternative version of point system that uses real money
Research and test alternative terminologies to improve clarity and explainability
Implement visual design fixes pointed out by expert testers
Passed on to Mathworks UX Research team for further work
Lessons Learnt
Importance of continuous iteration, learning about end user
Innovative thinking