What feedback mechanisms would better support and encourage instructors to reflect on inclusive teaching practices in the classroom?

What strategies can be best support and motivate instructors to implement inclusive teaching practices in their classroom?

Research Process

Secondary Research

Field Study

Stakeholder Collaboration

Ideation and Prototyping

User Feedback

Secondary Research I began by conducting secondary research to examine existing methods for helping instructors integrate inclusive teaching practices in higher education. The literature review highlighted a wide range of pedagogical approaches proven effective in fostering inclusive learning environments. However, the sheer volume of these practices presents a challenge for instructors, many of whom find it difficult to navigate and implement them effectively. Even instructors with the best intentions often feel overwhelmed, fearing that their efforts may not fully address students' diverse needs or could inadvertently alienate some students. This presents an opportunity to streamline and support the adoption of inclusive practices in a way that is both practical and impactful.

Field Study I conducted a field study with two groups of instructors focused on revamping their curriculum to enhance inclusivity. This initiative, while crucial, posed significant challenges, yet instructors from both programs actively contributed to the process. They identified and implemented inclusive practices within their courses and made necessary revisions based on these practices. However, they encountered difficulties in effectively sharing these practices with others in a simple, accessible manner. Additionally, assessing the impact and effectiveness of these practices proved to be a complex and time-consuming task, highlighting a need for streamlined solutions that facilitate both sharing and evaluation within the educational community.

Stakeholders Collaboration I collaborated closely with stakeholders to gain deeper insights into their ongoing efforts. Through targeted interviews, I identified the key aspects of their work. One stakeholder, with the support of her research team, had compiled a valuable resource list focused on course syllabi—an often overlooked yet crucial document in inclusive pedagogy. This finding highlighted a significant opportunity to leverage syllabi more effectively. Building on this discovery, I worked with stakeholders to define design requirements that would address their needs, ensuring that the solution was both practical and aligned with their goals for creating inclusive educational environments.

Ideation and Prototyping In collaboration with my research team, we explored various design approaches and technologies to address the challenge. We ultimately decided to leverage Large Language Models (LLM) for data analysis on course syllabi and generate visualizations to facilitate feedback for individual instructors. This approach not only offered a cost-effective solution but also provided scalability for future applications. Data visualizations were chosen for their ability to deliver quick, direct feedback and effectively guide users' attention. I then developed preliminary mockups to validate these concepts with stakeholders, ensuring alignment with their needs and expectations.

User Feedback and Concept Validation To validate the initial concepts, I developed a low-fidelity prototype using Python Dash to showcase the data visualizations. This prototype was not the final product but served as a testing ground for the effectiveness of visualizing inclusive teaching practices. I conducted in-depth interviews with instructors across various disciplines to gather insights on the prototype’s usefulness, their perceptions of using LLM within the application, and areas for improvement. This feedback was crucial in refining the approach and determining the value of data visualizations in supporting inclusive teaching practices.

Design Rationale

Navigating Challenges in the Inclusive Teaching Problem Space

The inclusive teaching problem space presents several key challenges. It is a complex and continually evolving domain that intersects with politically sensitive issues, making it difficult to pinpoint clear, universally accepted actions. Additionally, motivating instructors to overhaul their courses to align with inclusive teaching practices adds another layer of complexity. Encouraging meaningful change requires not only clear guidance but also addressing concerns, resistance, and the diverse needs of both instructors and students.

Design Principle

Based on these challenges we identified, I derived three design principles to approach this problem:

Accessible: Ensure that the design is accessible to all instructors, regardless of their technological proficiency or familiarity with inclusive teaching practices. This includes creating user-friendly interfaces, providing diverse resources, and offering multiple ways to engage with the content to accommodate different learning and teaching styles.

Actionable: Design solutions should provide clear, specific, and practical steps that instructors can easily implement in their courses. The focus should be on translating complex concepts into tangible actions, making it straightforward for instructors to integrate inclusive practices without feeling overwhelmed.

Assisted: The design should offer continuous guidance and support, helping instructors navigate the complexities of inclusive teaching. This could involve providing just-in-time feedback, offering examples of best practices, and creating a supportive community or network where instructors can share experiences and learn from one another.

Design Decisions

Design Decision I: We selected course syllabi as the focus of our analysis due to their critical role as boundary objects within university organizations. Syllabi are required for every course, making them a standardized and essential document. This choice also supports future scalability, as the consistent structure across syllabi facilitates broader application in future iterations.

Stakeholder Input

Design Decision II: We decided to utilize Large Language Models (LLM) and prompt engineering to analyze course syllabi using the Inclusive Teaching Inventory developed by one of our stakeholders. I collaborated with the stakeholder to adapt the inventory to better fit our specific needs. LLM was chosen as a cost-effective solution with the potential for easy scalability in future applications, aligning with our goals for broader impact and efficiency.

Design Decision III: We selected data visualization as the format to present our analysis results to users. This decision was driven by the need to simplify complex information in a visually engaging way, allowing users to quickly identify trends and focus on key areas of interest. Additionally, data visualizations are accessible to individuals from diverse backgrounds, making them an inclusive and effective tool for communication.

We engaged stakeholders in discussions around the design concepts and presented the data visualization mockups for their feedback. The stakeholders believe that it is comprehensible, and provided valuable insights on areas for improvement, which informed the refinement of the design. Additionally, I collaborated with engineers to ensure that the proposed solution was both feasible and aligned with technical constraints. To enhance the user experience, I also consulted with designers, who offered suggestions on how to scaffold the visualizations to improve user comprehension and ease of use. This collaborative approach ensured that the final design was both effective and practical from a business and technical standpoint.

Data Visualization Design Decisions

Implementation

The data generated by the LLM, which analyzed the syllabi using the Inclusive Teaching Inventory, resulted in a hierarchical dataset. After evaluating several visualization methods, we identified three potential options for presenting the data: a) Tree Map, b) Dendrogram, and c) Sunburst Chart (See Figure 1). While a Tree Map is suitable for continuous data, its use with our dataset posed challenges, such as the varying shapes of squares due to text display, which could lead to misinterpretations of size. Although a Dendrogram could work, it complicates the application of tooltips needed for providing additional information. Ultimately, we selected the Sunburst Chart as our preferred visualization method. This decision was based on its adaptability and effectiveness in representing hierarchical data, making it the most suitable choice for our needs.

Figure 1. Hierarchical Data Visualizations a) Tree map b) Dendrogram c) Sunburst

Scaffolding the Data Visualizations

For the final prototype, I utilized the ChatGPT 3.5 Turbo API with prompt engineering to conduct a comprehensive analysis of syllabi using the Inclusive Teaching Inventory. I then leveraged Python Plotly Dash to create a dynamic dashboard that visualizes the data generated by ChatGPT. Below are examples of these visualizations (Figure 2). This approach not only demonstrated the potential of integrating AI-driven analysis with user-friendly visual representations but also provided a scalable solution for future iterations.

Given that the Inclusive Teaching Inventory contains 88 items, visualizing this large dataset can be overwhelming. To ensure users can efficiently comprehend the information, I implemented a scaffolding approach to the data visualization design.

• Initial Overview: I began with a high-level bar chart to present the overall categories from the inventory. The bar chart format was chosen because it is universally accessible, even to instructors from diverse academic disciplines, providing an easy entry point into the data.

• Category-specific Exploration: Next, I developed individual sunburst charts for each of the 8 categories. This approach served two purposes:

  • It allowed users to dive deeper into specific categories of interest, promoting targeted exploration.

  • It gradually familiarized users with the sunburst chart format before introducing the comprehensive view.

• Comprehensive Visualization: Finally, I presented the full sunburst chart, integrating all the data into a single visualization. This step allowed users to see the complete picture after having explored the individual components in detail and get a at glance view of the data.

  This scaffolding process not only makes the data more digestible but also supports a more intuitive user experience, facilitating effective decision-making based on the insights provided.

Prototype Dashboard

Figure 2. a) Bar Chart demonstrates the overview of categories of inclusive teaching practices in the syllabus

Figure 2. b) Category Specific Sunburst Chart to allow users to dive deeper into categories of interest

Figure 2. c) Full Sunburst Chart to give user a at glance view of inclusive teaching practices represented in the syllabus.

We conducted interviews with instructors who are interested in making their courses more inclusive. The following presents user perceptions on IncluSight, our data visualization tool for inclusive elements in syllabi for instructors' feedback:

User Feedback

I'm in love with this. This is great! This is very helpful because you discover in a visual and quick fashion how you can improve. Because at the end, it's about making the syllabus better… But I go beyond this because the syllabus somehow connects you to the way that you practice your teaching. -Santiago (Engineering Professor)

I think it could be very useful....I think it gives kind of an organized way to think about...what we're including in our syllabi and why... if that's something I really want to focus on cultivating in my classroom, then here are some very tangible ways I can try to do that through my syllabi....I'm thinking particularly... engineering faculty that we work with, they like to understand the why, but they always want the applied, like, what can I actually do, you know? And so I think that this kind of gives a nice framework for like, this is what you could actually do to try to accomplish these things in. -Sarah (Engineering Professor)

It simply dissects a very significant idea. "Reduce stress" is an overwhelmingly ambiguous idea, but it breaks it down into individual components. So don't go into the syllabus with the goal of reducing stress but figuring out a way to create advice for success (helps to reduce stress) for students in my syllabus. That sounds much more tangible. And there are ways I can already think of to do that. And the consequence of that would be to reduce stress, which is my goal overall to begin with. So it's extremely helpful in that regard. -Logan (Psychology Professor)

This research introduced a practical and scalable solution to the complex, evolving challenge of inclusive teaching. By leveraging large language models (LLMs) and data visualizations, we provided instructors with actionable feedback that enables them to reflect, learn, and implement improvements in their teaching practices. This approach is not only cost-effective but also offers significant potential for future scalability, making it a valuable tool for enhancing inclusivity in education.

This approach is versatile and can be applied to other complex and evolving problem spaces, such as data privacy and cybersecurity awareness. By using data visualizations and large language models (LLMs), the methodology offers a practical, scalable solution that allows stakeholders to gain actionable insights and adapt to changing challenges in these critical areas.

Impact

This project is ongoing, and I am preparing for further studies to continue refining IncluSight.

• Refine Visualization Choices: In the initial phase, I developed three distinct visualizations and consulted with experts in the field before settling on the final choice. This decision was primarily driven by the desire to evaluate the effectiveness of visual feedback across a diverse range of instructors and academic disciplines. Moving forward, I plan to explore additional visualization options and refine the color schemes based on user testing feedback.

• Improve Interactivity: Initially, I created static visualization artifacts and collected user feedback to assess the concept's viability. The next step involves developing an interactive web application that enhances user interactivity, allowing for a more engaging and dynamic experience.

• Integrate Examples: User feedback has highlighted the need for more concrete examples of inclusive teaching practices. In response, I intend to incorporate specific examples into the tool to provide clear, actionable guidance for instructors looking to implement these strategies in their classrooms.

Next Steps

• Approach to Problem Discovery: In tackling problem identification, I opted for field studies, document analysis, and one-on-one stakeholder interviews. While this approach was practical, I recognize the value in exploring alternative methods, such as contextual inquiries, diary study, and surveys, to capture more nuanced insights. It's crucial to remain open to diverse methodologies to address varying project needs effectively.

• Pushing Boundaries: As design researchers, it's essential to push technological boundaries. In hindsight, I could have explored AI more aggressively in this project, even if the results were uncertain. Such experimentation offers valuable learning opportunities, regardless of the outcome.

• Continuous Learning: The evolving nature of technology underscores the importance of continuous learning. Drawing from diverse disciplines enriches creativity and problem-solving. My educational background has been instrumental in connecting the dots and driving innovation in this project.

Reflections