Generative Research | Exploratory Research
IncluSight: Visualizing Inclusive Teaching Elements for Instructors' Personalized Feedback
Problem: Instructors face challenges in identifying and implementing inclusive practices within their syllabi, lacking accessible and actionable feedback tools.
Impact: Developed a scalable solution utilizing data visualization and Large Language Models (LLMs) to provide actionable feedback on course syllabi, empowering instructors to enhance inclusive teaching practices.
Team: Collaborated with engineer, designer, product manager, and stakeholders (Instructors, educational researchers, and curriculum developers aiming to enhance inclusivity in teaching);
Tools: Python Plotly Dash, ChatGPT 3.5 Turbo API, JavaScript (Svelte), and D3.js.
My Role: I am the sole researcher led all phases of research, design, prototype development, and prototype evaluation.
Research Methodology: Conducted secondary research, field studies, stakeholder interviews, and prototyping; employed LLMs for data analysis, utilized visualization techniques.
How can the visualization of inclusive teaching practices in course syllabi support instructors in reflecting on (and improving) their inclusive teaching approaches?
Research Process
Field Study
Concept Development
Validation Research
Literature Review Through secondary research, I discovered that there are countless ways to make a course more inclusive. However, many instructors find it overwhelming to learn about all these approaches or fully understand the rationale behind them. Some hesitate to implement certain practices, unsure of their effectiveness or worried about unintentionally offending students. This highlights the importance of relieving instructors from the burden of navigating and verifying these methods, allowing them to focus on fostering a more inclusive learning environment.
Concept Development Through the early stage research, I identified that data visualization could be an effective feedback mechanism for instructors because it provides personalized insights, highlights key areas—both strengths and opportunities for improvement—and serves as a motivating tool for enhancing their work. To support the analysis, I utilized Large Language Models (LLMs) and prompt engineering to examine course syllabi, generating data for visualization. Preliminary mockups were developed and validated with stakeholders to ensure they aligned with user needs and expectations.
Validation Research I developed data visualizations based on the syllabi of 10 participating instructors who teach at least one undergraduate course and used these visualizations as design probes in in-depth interviews. The goal was to explore whether visualizations effectively deliver feedback, assess the value of personalized feedback, and uncover additional design possibilities.
Field Study I observed two groups of instructors working together to make their curricula more inclusive. I attended their group meetings, and conducted document analysis of the shared files in their online work space. I also conducted stakeholder interviews with the lead instructors/senior administrators of the two working groups.
Design Iteration Based on the feedback I gathered from instructors, I implemented the feedback elements into the current design and developed a low-fi prototype for the next steps.
Literature Review
Design Iteration
Design Rationale
Navigating Challenges in the Inclusive Teaching Problem Space
The inclusive teaching problem space presents several key challenges. It is a complex and continually evolving domain that intersects with politically sensitive issues, making it difficult to pinpoint clear, universally accepted actions. Additionally, motivating instructors to overhaul their courses to align with inclusive teaching practices adds another layer of complexity. Encouraging meaningful change requires not only clear guidance but also addressing concerns, resistance, and the diverse needs of both instructors and students.
Design Principle
Based on these challenges we identified, I derived three design principles to approach this problem:
Accessible: Ensure that the design is accessible to all instructors, regardless of their technological proficiency or familiarity with inclusive teaching practices. This includes creating user-friendly interfaces, providing diverse resources, and offering multiple ways to engage with the content to accommodate different learning and teaching styles.
Actionable: Design solutions should provide clear, specific, and practical steps that instructors can easily implement in their courses. The focus should be on translating complex concepts into tangible actions, making it straightforward for instructors to integrate inclusive practices without feeling overwhelmed.
Assisted: The design should offer continuous guidance and support, helping instructors navigate the complexities of inclusive teaching. This could involve providing just-in-time feedback, offering examples of best practices, and creating a supportive community or network where instructors can share experiences and learn from one another.
Design Decisions
Design Decision I: We selected course syllabi as the focus of our analysis due to their critical role as boundary objects within university organizations. Syllabi are required for every course, making them a standardized and essential document. This choice also supports future scalability, as the consistent structure across syllabi facilitates broader application in future iterations.
Stakeholder Input
Design Decision II: We decided to utilize Large Language Models (LLM) and prompt engineering to analyze course syllabi using the Inclusive Teaching Inventory developed by one of our stakeholders. I collaborated with the stakeholder to adapt the inventory to better fit our specific needs. LLM was chosen as a cost-effective solution with the potential for easy scalability in future applications, aligning with our goals for broader impact and efficiency.
Design Decision III: We selected data visualization as the format to present our analysis results to users. This decision was driven by the need to simplify complex information in a visually engaging way, allowing users to quickly identify trends and focus on key areas of interest. Additionally, data visualizations are accessible to individuals from diverse backgrounds, making them an inclusive and effective tool for communication.
We engaged stakeholders in discussions around the design concepts and presented the data visualization mockups for their feedback. The stakeholders believe that it is comprehensible, and provided valuable insights on areas for improvement, which informed the refinement of the design. Additionally, I collaborated with engineers to ensure that the proposed solution was both feasible and aligned with technical constraints. To enhance the user experience, I also consulted with designers, who offered suggestions on how to scaffold the visualizations to improve user comprehension and ease of use. This collaborative approach ensured that the final design was both effective and practical from a business and technical standpoint.
Data Visualization Design Decisions
Concept Development
I evaluated three visualization methods—Tree Map, Dendrogram, and Sunburst Chart—for representing hierarchical syllabus data generated by LLM analysis. After consulting with data visualization experts, the Sunburst Chart was selected for its adaptability and effectiveness in visualizing hierarchical data while providing a clear and intuitive user experience for the initial concept test.
Figure 1. Hierarchical Data Visualizations a) Tree map b) Dendrogram c) Sunburst
Scaffolding Information for Comprehension
I utilized the ChatGPT 3.5 Turbo API with prompt engineering to conduct a comprehensive analysis of syllabi using the Inclusive Teaching Inventory. I then leveraged Python Plotly Dash to create a dynamic dashboard that visualizes the data generated by ChatGPT. This approach not only demonstrated the potential of integrating AI-driven analysis with user-friendly visual representations but also provided a scalable solution for future iterations.
To ensure users can efficiently comprehend the information, I implemented a scaffolding approach to the data visualization design to help reduce users' cognitive load. We followed the information seeking mantra, "Overview first, zoom and filter, then details-on-demand" to design the process of revealing the information.
Step1 Present a high-level bar chart to present a bar chart initial overview of the syllabi measured against the Inclusive Teaching Inventory. There are eight categories derived from the inventory. The bar chart format was chosen because it is universally accessible, even to instructors from diverse academic disciplines, providing an easy entry point into the data.
Figure 2. a) Bar Chart demonstrates the overview of categories of inclusive teaching practices in the syllabus
Figure 2. b) Category Specific Sunburst Chart to allow users to dive deeper into categories of interest
Figure 2. c) Full Sunburst Chart to give user a at glance view of inclusive teaching practices represented in the syllabus.
Design Tools
Step 2 To enhance comprehensibility and guide users through the data, I developed individual sunburst charts for Category-Specific Exploration. This approach served two key purposes: it enabled users to delve deeper into specific areas of interest, encouraging targeted exploration, and it gradually introduced users to the sunburst chart format, building familiarity before presenting the comprehensive view.
Step 3 Finally, I present a full sunburst chart that integrate all data into one comprehensive visualization. Building on the exploration of individual categories, this comprehensive view allowed users to see the entire dataset at a glance, providing a clear and complete picture.
By following a scaffolding process, users were gradually guided from focused exploration to a holistic understanding, making the data more digestible and supporting a more intuitive experience. This approach empowered users to make informed decisions based on the insights provided.
Key Insights: Instructors require clear, actionable feedback on syllabi; data visualizations can effectively convey complex information; LLMs offer scalable analysis solutions.
Themes: Need for accessible tools, importance of actionable recommendations, and value of visual feedback in promoting inclusive teaching.
The following presents user perceptions on using data visualization as a way to get feedback from the syllabi, and their insights on how it could better support them:
Findings and Insights
I'm in love with this. This is great! This is very helpful because you discover in a visual and quick fashion how you can improve.… But I go beyond this because the syllabus somehow connects you to the way that you practice your teaching. -P1 (Engineering Professor)
I think it could be more effective to include some examples on specifically how to implement it in syllabus.... -P2 (Psychology Professor)
I think it would be useful to look across courses. I’m running different classes in a given semester; I’d be curious to compare from one to the other. I think it’d be interesting to compare multiple sections... -P3 (Information Science Professor)
Improve Interactivity: Initially, I created static visualization artifacts and collected user feedback to assess the concept's viability. The next step involves developing an interactive web application using D3.js to enhances user interactivity, allowing for a more engaging and dynamic experience.
Integrate Examples: User feedback has highlighted the need for more concrete examples of inclusive teaching practices. In response, I intend to incorporate specific examples into the tool to provide clear, actionable guidance for instructors looking to implement these strategies in their classrooms.
Design Iteration
This exploratory research, part of my dissertation project, is conducted in collaboration with an enterprise-level learning analytics product team at a large university. It's the early phase of exploring how visualization can be used to provide actionable feedback for inclusive teaching practices. By leveraging large language models (LLMs) and interactive data visualizations, we developed a practical, scalable solution that empowers instructors to reflect, learn, and implement meaningful improvements in their teaching.
This approach offers multiple advantages: it is cost-effective, highly adaptable, and scalable, addressing the complex and evolving challenges of a wicked problem (inclusive teaching). Recognizing the potential impact of this work, the product team plans to adopt elements of this research into their existing learning analytics tool.
Contribution
Approach to Problem Discovery: In tackling problem identification, I opted for field studies, document analysis, and one-on-one stakeholder interviews. While this approach was practical, I recognize the value in exploring alternative methods, such as contextual inquiries, diary study, and surveys, to capture more nuanced insights. It's crucial to remain open to diverse methodologies to address varying project needs effectively.
Pushing Boundaries: As design researchers, it's essential to push technological boundaries. In hindsight, I could have explored AI more aggressively in this project, even if the results were uncertain. Such experimentation offers valuable learning opportunities, regardless of the outcome.
Continuous Learning: The evolving nature of technology underscores the importance of continuous learning. Drawing from diverse disciplines enriches creativity and problem-solving. My educational background has been instrumental in connecting the dots and driving innovation in this project.