What feedback mechanisms would better support and encourage instructors to reflect on inclusive teaching practices in the classroom?

What strategies can be best support and motivate instructors to implement inclusive teaching practices in their classroom?

The Problem

The current approach to engaging instructors in inclusive teaching practices is largely "top-down," offering resources that provide high-level guidelines for implementing inclusive teaching in classrooms. However, these guidelines are often abstract and theoretical, lacking personalized, actionable insights for instructors. This leaves the responsibility on instructors to interpret and apply these guidelines on their own, which can be overwhelming and ineffective.

The Solution

IncluSight is an interactive visualization solution designed to provide instructors with individualized feedback on their inclusive teaching practices through the analysis of course syllabi. As a universally required document for university courses, syllabi serve as a consistent and accessible resource for evaluation. Using large language models (LLMs) and prompt engineering, I analyzed syllabi against criteria established by prior research. The results of this analysis were then used to develop a visualization tool that delivers actionable, personalized feedback to instructors. This approach is both tangible and scalable, with the potential for seamless integration into the learning analytics systems used by institutions.

Research Process

Secondary Research

Field Study

Stakeholder Collaboration

Ideation and Prototyping

User Feedback

Secondary Research I identified the need to streamline and support practical, impactful adoption of inclusive teaching practices in higher education. I also found that current approach to inclusive teaching is ineffective in supporting instructors.

Field Study I conducted a field study with two instructor groups revamping their curriculum for inclusivity, revealing challenges in sharing and evaluating inclusive practices effectively, underscoring the need for streamlined solutions.

Stakeholders Collaboration I collaborated with stakeholders through interviews to uncover key insights, identifying course syllabi as a critical yet underutilized resource for inclusive pedagogy. Together, we defined design requirements to create a practical, goal-aligned solution.

Ideation and Prototyping I leveraged Large Language Models (LLMs) and Prompt Engineering to analyze course syllabi and generate scalable, cost-effective data visualizations for instructor feedback. Preliminary mockups were created and validated with stakeholders to ensure alignment with their needs.

User Feedback and Concept Validation I created a low-fidelity prototype using Python Dash to test data visualizations and conducted in-depth instructor interviews to assess its usefulness, refine the design, and validate the value of visualizing inclusive teaching practices.

Data Visualization Design Decisions

Early Stage Implementation

We evaluated three visualization methods—Tree Map, Dendrogram, and Sunburst Chart—for representing hierarchical syllabus data generated by LLM analysis. After consulting with data visualization experts, the Sunburst Chart was selected for its adaptability and effectiveness in visualizing hierarchical data while providing a clear and intuitive user experience for the initial concept test.

Figure 1. Hierarchical Data Visualizations a) Tree map b) Dendrogram c) Sunburst

Scaffolding Information for Comprehension

I utilized the ChatGPT 3.5 Turbo API with prompt engineering to conduct a comprehensive analysis of syllabi using the Inclusive Teaching Inventory. I then leveraged Python Plotly Dash to create a dynamic dashboard that visualizes the data generated by ChatGPT. This approach not only demonstrated the potential of integrating AI-driven analysis with user-friendly visual representations but also provided a scalable solution for future iterations.

To ensure users can efficiently comprehend the information, I implemented a scaffolding approach to the data visualization design to help reduce users' cognitive load.

Step1 Present a high-level bar chart to present a bar chart initial overview of the syllabi measured against the Inclusive Teaching Inventory. There are eight categories derived from the inventory. The bar chart format was chosen because it is universally accessible, even to instructors from diverse academic disciplines, providing an easy entry point into the data.

Figure 2. a) Bar Chart demonstrates the overview of categories of inclusive teaching practices in the syllabus

Figure 2. b) Category Specific Sunburst Chart to allow users to dive deeper into categories of interest

Figure 2. c) Full Sunburst Chart to give user a at glance view of inclusive teaching practices represented in the syllabus.

Design Tools

Step 2 To enhance comprehensibility and guide users through the data, I developed individual sunburst charts for Category-Specific Exploration. This approach served two key purposes: it enabled users to delve deeper into specific areas of interest, encouraging targeted exploration, and it gradually introduced users to the sunburst chart format, building familiarity before presenting the comprehensive view.

Step 3 Finally, I present a full sunburst chart that integrate all data into one comprehensive visualization. Building on the exploration of individual categories, this comprehensive view allowed users to see the entire dataset at a glance, providing a clear and complete picture.

By following a scaffolding process, users were gradually guided from focused exploration to a holistic understanding, making the data more digestible and supporting a more intuitive experience. This approach empowered users to make informed decisions based on the insights provided.

We conducted interviews with instructors who are interested in making their courses more inclusive. The following presents user perceptions on IncluSight, our data visualization tool for inclusive elements in syllabi for instructors' feedback:

Initial User Feedback

I'm in love with this. This is great! This is very helpful because you discover in a visual and quick fashion how you can improve. Because at the end, it's about making the syllabus better… But I go beyond this because the syllabus somehow connects you to the way that you practice your teaching. -Santiago (Engineering Professor)

I think it could be very useful....I think it gives kind of an organized way to think about...what we're including in our syllabi and why... if that's something I really want to focus on cultivating in my classroom, then here are some very tangible ways I can try to do that through my syllabi....I'm thinking particularly... engineering faculty that we work with, they like to understand the why, but they always want the applied, like, what can I actually do, you know? And so I think that this kind of gives a nice framework for like, this is what you could actually do to try to accomplish these things in. -Sarah (Engineering Professor)

It simply dissects a very significant idea. "Reduce stress" is an overwhelmingly ambiguous idea, but it breaks it down into individual components. So don't go into the syllabus with the goal of reducing stress but figuring out a way to create advice for success (helps to reduce stress) for students in my syllabus. That sounds much more tangible. And there are ways I can already think of to do that. And the consequence of that would be to reduce stress, which is my goal overall to begin with. So it's extremely helpful in that regard. -Logan (Psychology Professor)

This exploratory research, part of my dissertation project, is conducted in collaboration with an enterprise-level learning analytics product team at a large university. It's the early phase of exploring how visualization can be used to provide actionable feedback for inclusive teaching practices. By leveraging large language models (LLMs) and interactive data visualizations, we developed a practical, scalable solution that empowers instructors to reflect, learn, and implement meaningful improvements in their teaching.

This approach offers multiple advantages: it is cost-effective, highly adaptable, and scalable, addressing the complex and evolving challenges of inclusive education. Recognizing the potential impact of this work, the product team plans to adopt elements of this research into their existing learning analytics tool.

Impact

• Refine Visualization Choices: In the initial phase, I developed three distinct visualizations and consulted with experts in the field before settling on the final choice. This decision was primarily driven by the desire to evaluate the effectiveness of visual feedback across a diverse range of instructors and academic disciplines. Moving forward, I plan to explore additional visualization options and refine the color schemes based on user testing feedback.

• Improve Interactivity: Initially, I created static visualization artifacts and collected user feedback to assess the concept's viability. The next step involves developing an interactive web application that enhances user interactivity, allowing for a more engaging and dynamic experience.

• Integrate Examples: User feedback has highlighted the need for more concrete examples of inclusive teaching practices. In response, I intend to incorporate specific examples into the tool to provide clear, actionable guidance for instructors looking to implement these strategies in their classrooms.

Next Steps

• Approach to Problem Discovery: In tackling problem identification, I opted for field studies, document analysis, and one-on-one stakeholder interviews. While this approach was practical, I recognize the value in exploring alternative methods, such as contextual inquiries, diary study, and surveys, to capture more nuanced insights. It's crucial to remain open to diverse methodologies to address varying project needs effectively.

• Pushing Boundaries: As design researchers, it's essential to push technological boundaries. In hindsight, I could have explored AI more aggressively in this project, even if the results were uncertain. Such experimentation offers valuable learning opportunities, regardless of the outcome.

• Continuous Learning: The evolving nature of technology underscores the importance of continuous learning. Drawing from diverse disciplines enriches creativity and problem-solving. My educational background has been instrumental in connecting the dots and driving innovation in this project.

Reflections