Generative Research | Exploratory Research
Scenario-Based Exploration of Learning Analytics Applications (2020)
Problem: Investigate the ethical considerations and perspectives of students and instructors regarding the use of student data in learning analytics applications.
Summary: Led a two-month exploratory research project to understand student and instructor perceptions of data privacy in learning analytics within higher education, utilizing literature reviews, user interviews, and scenario-based probes.
Team and Tools: Conducted independently; employed qualitative research methods and scenario-based interviews.
Background: Higher education institutions collect extensive student data daily. However, both students and instructors often lack awareness or understanding of how this data is utilized in learning analytics technologies.
Stakeholders: University administrators, instructors, students, and educational technologists
How do students and instructors think of student data privacy in the current institutional data practices?
Objectives: Explore how students and instructors perceive data privacy in institutional data practices and learning analytics applications.
Methodology:
Conducted a literature review to identify existing research and gaps.
Performed user interviews with both students and instructors to gather qualitative insights.
Developed scenario-based probes to elicit detailed responses on specific use cases involving learning analytics.
Participants: Included a diverse group of university students and instructors across various disciplines.
Research Approach
Research Process
Literature Review to examine questions that has been asked to the users, and what additional information that we can learn from them regarding student data privacy.
User Interviews to compare and contrast two groups of users who are on opposite side of the power dynamic and their perspectives on data privacy.
Scenarios to probe data privacy related questions in a specific use case where users are given more context.
Challenges: Ensured scenarios were realistic and relatable to both students and instructors to effectively probe their perceptions.
Sample Scenarios
Student Scenario: College algebra class is offered online which has over 300 students enrolled in it. This class also have an AI powered virtual assistant, who can answer questions from students. In addition, the class also use predictive model to provide real-time feedback to students on his/her performance in the course. Alerts are sent to students when they are falling behind and connect students with advising and tutoring services. A summary of questions from students is shared with instructors regularly. In addition, the virtual assistant can also help students to identify areas of weakness in the knowledge points and provide adaptive learning support for students.
Instructor Scenario: An instructor is teaching a large general education class online. This class normally has 300 students per session. In addition to traditional tools (e.g. LMS, videos) you use to teach, this course has a virtual assistant that is powered by AI. Students can reach the virtual assistant anywhere any time. The course can provide real time feedback to students. Students are informed about their progress in class. The virtual assistant not only can answer students’ questions, but also provide the instructor with a summary of questions that students ask. In addition, alerts would be sent to the instructor and the student if they are falling behind in class.
I created three sets of scenarios that captures existing and future use cases (e.g. a virtual assistant in class, aggregated report from Learning Management System, etc.) where learning analytics is implemented in certain aspects of the tool in teaching and learning. The goal is probe students and instructors perspectives on a issue in a similar context. The following example scenario describes a virtual assistant used in a large class. I also made sure both students and instructors are asked comparable questions in the semi-structured interviews.
Voices from Students and Instructors
Instructors and students shared some ideas in common, but also had surprisingly different ideas about data privacy.
“I don’t like the idea of reducing teaching to something that you can train a robot to do? Yeah. I mean, if that’s the case, why do we need universities?” - Edward, Chemistry Instructor, 20 years teaching experience
"...he uses Grammarly to like help, check, and do his word processing and sentence checking and things like that. And as a result, he is terrible at grammar... we can rely on technology so much that it inhibits our ability to learn."- Alison, English Instructor, 7 years teaching experience
“As a student, I feel like we kind of sign our lives away to like Canvas and the university, because we’re giving them all our data, all of our web browser history and stuff like that. I think we probably, if we want that control, we can probably fight for it a little bit, but I have a feeling that we don’t get as much control as we like to think we have.” - Emily, Sophomore
“<MyUni> has a lot of information on me, my computer has a ton of information on me. And anyone that can access that information, can use that data in positive or negative ways. And I should definitely be aware of that.” - Rakesh, Sophomore
User Feedback
Findings
Key Insights:
Both students and instructors expressed concerns about data privacy but differed in specific apprehensions.
Instructors worried about the potential reduction of teaching roles due to AI integration.
Students felt a lack of control over their data, perceiving that they "sign their lives away" to institutional platforms.
Themes: Concerns about AI replacing human roles, lack of transparency in data usage, and the need for greater control over personal data.
Recommendations:
Increase transparency in how student data is collected and utilized within learning analytics tools.
Develop policies that clearly define the role of AI in educational settings to alleviate fears of job displacement among instructors.
Implement features that allow students to have more control over their data, enhancing trust in institutional data practices.
Outcome: Provided actionable insights to university administrators and educational technologists to inform the ethical implementation of learning analytics applications.
Impact
Reflections
User research is not just about (directly) designing products. Even though research ultimately is about contributing to design solutions, not all research has to be directly related to design to contribute to design. Many analytics products already been implemented at the time of research, but the research still revealed important information about strategies moving forward. And it does not involve designing any products.
User research can be quick and easy, with limited resources. I was a one person team and I conducted this project under limited time and budget resource, it was a small study, but it revealed valuable insights.
Select a diverse sample is critical to represent diverse perspectives. In this case, participants in a dicipline that focuses on data and technology could think very different compared to someone who is in a humanities dicipline.