The TLSS Evaluation Framework
Please click on the tabs below to learn more.
In addition to implementing a systematic data collection, analysis and reporting process, a standing goal of the Teaching and Learning Support Service (TLSS)’ evaluative framework is to examine and enhance varied facets of its work. Based on what these data indicate regarding trends in campus teaching and learning, opportunities for future innovation, visioning and strategic planning may emerge.
|To see detail, place the cursor of your mouse over the plus symbols:|
Germain-Rutherford & Groen (2018)
All while integrating existing data tracked by individual sectors within the service (predominantly data related to participation or use of service); the framework has provided direction for the collection of data from multiple sources to be used in complementarity to report on a series of indicators. Using an adaptation of the Theory of Change (Hart et al., 2009) and Guskey’s (2002) Model of Teacher Development, the figure below outlines different levels of indicators in three interdependent groupings.
The first grouping, in blue, represents an increasing level of sophistication in the examination of impact. Beginning with transformation of use and satisfaction with specific services, followed by transformation of thinking and conceptualisation (in terms of knowledge, skills, attitudes and values) of those using TLSS services and resources, and thirdly transformation of practice as exemplified by changes in instructional approaches and engagement in the examination of their effectiveness.
The second grouping, in orange, represents the subsequent impact of the multiple individual levels of transformation (in blue). For instance, how a particular instructor’s transformation in thinking and practice might have an impact on the learning of their students or the thinking and practice of some of their colleagues.
The third grouping, in red, indicates transformation in the culture of teaching and learning at the meso-level (department/faculty) and macro-level (institutional). This transformation may be influenced by the direct input and advocacy of the TLSS or indirectly through the collective impact of an increasing number of individuals who have transformed their thinking and practices (individual level transformation in blue).
The Framework Development Process
A participatory approach was adopted from the work of Cousins and Earl (1995) to ensure meaningful engagement across service-level stakeholders. This process allowed for a more comprehensive mapping of the services being tracked along with the development of ownership in the process by members of the TLSS team. To guide stakeholder engagement, a reflective worksheet was created (click here to view a copy). The structure of questions borrowed from both the Theory of Change (Hart et al., 2009) and Guskey’s Education Impact Framework (Guskey, 2002).
Development work drew on the contributions of evaluation scholars such as: Amundsen and Wilson (2012), Chalmers (2015), Guskey (2002), Hines (2017), Kirkpatrick (1994), Kreber and Brook (2001).
Types of Data Sources
A series of data collection sources, instruments, and processes were put in place to collect evidence associated with each indicator. These included the following during the first data collection phase in winter/spring 2018. For multiple data sources, various informants (including instructors, students, administrators, and TLSS team members) provided feedback in terms of expected, perceived, and experienced impact in an effort to aid in data triangulation.
Next to many of the data sources below are examples of data collection instruments that we have used. Please feel free to use and adapt these and to let us know how it goes. We would be happy to send these to you directly and collaborate on projects regarding their use and enhancement (Contact: Jovan Groen at jgroen@uOttawa.ca).
- Descriptive statistics regarding use of services
- General consultation statistics and tracking of insights (Consultation Tracking Strategy)
- Online project and resource development exit interviews (Sample Questions)
- Targeted professor/student surveys (regarding a particular pedagogical practice, use of a specific instructional tool or a specific learning space)
- Targeted Course Initiative Survey Templates
- Learning Management System Survey Template
- Lecture Capture Survey Template
- Videoconferencing Survey Template
- Active Learning Classroom Survey Template
- New professor orientation program follow-up survey (Sample Survey)
- TA Orientation program follow-up survey (Sample Survey)
- TLSS grant recipient follow-up survey (Sample Survey)
- Online course build student surveys
- Class observations (Sample Observation Form 1 and Form 2)
- Anecdotal feedback
Initial Framework Development Timeline (2017/2018)
Data are revealing evidence of impact in both expected and unexpected ways. For instance, expected in that there is a diversity of instructional approaches at work across the levels and disciplinary boundaries of the institution. Somewhat unexpected in that participation overall has been quite high and that the experiences and insights shared by professors, students and staff alike have been exceptionally rich.
These findings are not only permitting us to carefully weave a coherent narrative of impact, but, as initially suggested, to test the processes used to most efficiently collect the data in a way that solicits the greatest participation and the most meaningful and constructive feedback. See the TLSS Annual Report here to view some of the data at work!
The TLSS welcomed 184 professors for
369 individual consultations, representing 383 hours
Emerging Themes Regarding Transformation of Teaching Practices
Across disciplines, professors report that practices with most impact on student learning are:
- case studies;
- student presentations;
- reflective in-class student activities
Influence of Instructional spaces on teaching and learning
Via systematic instructor feedback and class observations, the instructional approaches used by professors and the nature of student engagement is heavily influenced by the space that they are teaching and learning in. Requests for flexible spaces which are designed with active learning in mind continue to increase.
Professor Perceptions as Compared to Student Experiences
Across over a dozen contexts, instructional activities said to be used in-class, match what the students in the same classes are saying that they experience. However, discrepancies of 5-15% emerge between professor and student views of the amount of in-class time spent lecturing.
The Voice of the Professors
The initial grant given to us by the TLSS gave us the opportunity to rethink how we deliver content and engage students in classes (large or small). Is has opened new teaching and grant opportunities and allowed us to develop new ideas on how to engage our students or deliver content that could benefit our students. It stimulated us and gave us the opportunity to think differently about our classrooms, our students, and reflect on our teaching skills and experience in ways we would not have otherwise.
Just to thank you for the meeting, the interest you’ve shown and the good atmosphere. After this meeting, I have a better view of what is doable, and it’s more than I thought. So it’s encouraging for the next step and the next phases of the project.
Tech support was fast and efficient. The proximity and immediate availability, during class setup and while in session, provided a safety net that allowed instructors to feel safe trying new things and "taking risks" with technology.
The Voice of the Students
I had a blended learning international development course that I thoroughly enjoyed. The flexibility of not always having to go to campus was great. I liked being able to complete the course material at home (i.e., watch videos) because often in class it can be hard to focus. I liked being able to do the online component of the course at my convenience.
Technology, in general, is a barrier. For example, I would not be able to complete most of my work without Internet or computer access. Often my professors are not tech savvy, and many features go unused or the site is not updated regularly.
Echo360 is honestly the most valuable resource available to me as a student. I am a student that attends almost 100% of lectures but I STILL go back and re-listen to all lectures to ensure I fully grasp concepts, and I can go back and clarify anything I have not been able to understand in class.
Systematic Data Collection
The development of an evaluation framework is the first part of an effort by the TLSS to systematically evidence the nature and extent of the impact that its services and initiatives have on instructional practices, on student learning, and on the culture of advancement and innovation in teaching and learning. It is intended that this practice become a habit as it increasingly becomes embedded in the educational and media development work of the service.
The Evaluation Process
During this first data analysis and interpretation process, methods to streamline the format used to collect and store data, along with methods to more accurately and efficiently sort and analyse data, are being noted for further development. This first iteration is already proving to be helpful in terms of determining what criteria will we used to prioritise the indicators that we chose to track and the data sources that we use to evidence them.