Deep Dive
Privacy Concerns for Learning Analytics Among Instructors (2021)
Overview: A UX research project that explores the relationship between data privacy concerns and learning analytics product adoption among university instructors.
My Role: I led all phases of research process, including survey design, data collection, data analysis, and disseminate research findings.
Duration: 4 months
Tools: Qualtrics, SPSS
Background
Learning analytics applications are increasingly prevalent in higher education, offering instructors a variety of tools to support different aspects of teaching and learning. However, adopting these tools is not always straightforward for instructors, as numerous factors influence their decisions. My project specifically explored how concerns around data privacy impact instructors’ intentions to adopt learning analytics technology.
What is the relationship between instructors' perceptions of data privacy and their adoption of the learning analytics technology?
Research Process
Secondary Research
I examined existing research and discovered that a few studies have identified challenges for instructors' adoption of learning analytics applications for teaching and learning, but no study has explored instructors perceived usefulness, data privacy concerns, and their impact on adoption.
Methodology
I selected a survey methodology to capture a broad range of instructors' perspectives on data privacy and learning analytics. To make the survey engaging and contextually relevant, I incorporated vignettes—brief, hypothetical scenarios in teaching—which are an effective and efficient way to elicit user perspectives in realistic situations. Surveys provided a fast and cost-effective method for examining relationships between concepts, enabling timely insights without the need for complex experimental setups. The chosen analysis approach further facilitated a rapid exploration of these relationships, making it ideal for this study.
Literature Review to identify issues has been examined about privacy and adoption of learning analytics applications among instructors.
Survey to discover a large group of users thoughts on privacy and adoption in a short period of time.
Vignettes to evoke reactions from users in a series of different situations.
Example Vignettes
The following are some examples of vignettes we used in this study. The vignettes are either outcome oriented or socially oriented scenarios of use cases of analytics in the teaching and learning context. Users are asked to rate these vignettes on expected usefulness, privacy concerns, and adoption likelihood.
Poll: While prepping for tomorrow’s CHEM 101 class, Taylor pulls up a Canvas visualization of last year’s responses to a clicker/Tophat poll on the same topic. He sees that 350 of his 395 students chose the same wrong answer in the poll, and decides to add a second clarifying example to the lecture materials. (STEM, large, in-class activity, class prep)
Social: During his 30-student mechanical engineering design class, Bayley monitors team dynamics using a social behavior analytics dashboard (e.g., classification of facial expressions, gestures, body positions). When he sees Team 3 slowing down, he heads over for a check in. (STEM, small, social interactions, class management)
Prereqs: Alex is developing a third-year course in machine learning for the new data sciences major. He reviews assignment and test grades for current majors during their first two years, and he can see that about 30% perform poorly on programming intensive assignments. He expands his early review and extra resources relating to core programming skills. (STEM, small, grades, new course development)
Collaboration: Priya is about to have a coaching session with one of her industrial engineering teams. To prepare, she reviews team communication analytics that show how frequently members communicate and update shared documents, as well as which members are texting and emailing others. This gives her insight into the team dynamics. (STEM, small, social interactions, mentoring)
Findings
The following findings and design implications were uncovered:
There is a negative correlation between privacy and percieved usefulness among instructors, in another word, the more privacy concerns they have over a specific use case, the less likely they would adopt such an application for teaching.
Instructors had a higher levels privacy concern over situations to do with social interactions of students compared to situations that related to learning outcomes (e.g. grades).
Further, we speculated that the rated likelihood of adoption might be mediated by the tradeoff between usefulness and privacy.
Reflection
The more you research, the more you uncover. Depends on the levels of detail that you want to get into, you can discover different things with a different method, a different sample, or at a different time.
Users are always changing, so should design. Same study can be done at different times and get different results. Because users change as maturity of a certain technology grows, and users gaining more experience with it.
Never make assumptions before talking to users. Even if you think you know a lot about a topic, your users may still surprise you in some ways.