Exploratory Research
Scenario-Based Exploration of Learning Analytics Applications (2020)
Overview: This project focuses on understanding the perceptions of data privacy in learning analytics applications among two key stakeholders in higher education: students and instructors.
My Role: I led the end-to-end research process.
Stakeholders: Administrators, Instructors, Students
Duration: 2 months
Background
Higher education systems collect vast amounts of student data daily, yet many students and instructors are unaware of this or lack a clear understanding of how the data is used in learning analytics technologies. While much research has focused on the technical methods underpinning learning analytics, such as machine learning and AI, less attention has been given to the human stakeholders involved. My study aims to explore the ethical considerations and perspectives of these stakeholders regarding the use of student data.
How do students and instructors think of student data privacy in the current institutional data practices?
Research Process
Literature Review to examine questions that has been asked to the users, and what additional information that we can learn from them regarding student data privacy.
User Interviews to compare and contrast two groups of users who are on opposite side of the power dynamic and their perspectives on data privacy.
Scenarios to probe data privacy related questions in a specific use case where users are given more context.
Sample Scenarios
Student Scenario: College algebra class is offered online which has over 300 students enrolled in it. This class also have an AI powered virtual assistant, who can answer questions from students. In addition, the class also use predictive model to provide real-time feedback to students on his/her performance in the course. Alerts are sent to students when they are falling behind and connect students with advising and tutoring services. A summary of questions from students is shared with instructors regularly. In addition, the virtual assistant can also help students to identify areas of weakness in the knowledge points and provide adaptive learning support for students.
Instructor Scenario: An instructor is teaching a large general education class online. This class normally has 300 students per session. In addition to traditional tools (e.g. LMS, videos) you use to teach, this course has a virtual assistant that is powered by AI. Students can reach the virtual assistant anywhere any time. The course can provide real time feedback to students. Students are informed about their progress in class. The virtual assistant not only can answer students’ questions, but also provide the instructor with a summary of questions that students ask. In addition, alerts would be sent to the instructor and the student if they are falling behind in class.
I created three sets of scenarios that captures existing and future use cases (e.g. a virtual assistant in class, aggregated report from Learning Management System, etc.) where learning analytics is implemented in certain aspects of the tool in teaching and learning. The goal is probe students and instructors perspectives on a issue in a similar context. The following example scenario describes a virtual assistant used in a large class. I also made sure both students and instructors are asked comparable questions in the semi-structured interviews.
Voices from Students and Instructors
Instructors and students shared some ideas in common, but also had surprisingly different ideas about data privacy.
“I don’t like the idea of reducing teaching to something that you can train a robot to do? Yeah. I mean, if that’s the case, why do we need universities?” - Edward, Chemistry Instructor, 20 years teaching experience
"...he uses Grammarly to like help, check, and do his word processing and sentence checking and things like that. And as a result, he is terrible at grammar... we can rely on technology so much that it inhibits our ability to learn."- Alison, English Instructor, 7 years teaching experience
“As a student, I feel like we kind of sign our lives away to like Canvas and the university, because we’re giving them all our data, all of our web browser history and stuff like that. I think we probably, if we want that control, we can probably fight for it a little bit, but I have a feeling that we don’t get as much control as we like to think we have.” - Emily, Sophomore
“<MyUni> has a lot of information on me, my computer has a ton of information on me. And anyone that can access that information, can use that data in positive or negative ways. And I should definitely be aware of that.” - Rakesh, Sophomore
Insights
Illuminated by literature on student data privacy, design of learning technology, and learning analytics, I derived five design implications from the study.
These results were presented at the university wide Learning Analytics Committee as a study to inform future use and development of learning analytics technology at the institution. Subsequently, the committee decided to add a student representative position on the Learning Analytics Committee to ensure their voice are included in the institutional decision-making process.
Reflections
User research is not just about (directly) designing products. Even though research ultimately is about contributing to design solutions, not all research has to be directly related to design to contribute to design. Many analytics products already been implemented at the time of research, but the research still revealed important information about strategies moving forward. And it does not involve designing any products.
User research can be quick and easy, with limited resources. I was a one person team and I conducted this project under limited time and budget resource, it was a small study, but it revealed valuable insights.
Select a diverse sample is critical to represent diverse perspectives. In this case, participants in a dicipline that focuses on data and technology could think very different compared to someone who is in a humanities dicipline.