What do you do if new technology isn't improving student learning outcomes?

Sometimes we race to declaring technology should immediately improve student outcomes. But technology is often one piece of the puzzle among a variety of other factors. I do think we should look at student outcomes but we should define and track other KPIs too. These should be directly tied to the technology's purpose and should be more immediate in time frame. Has the technology saved staff time? Has the technology improved student attendance? Has the technology improved communication with families? Has college and career exploration activity participation increased? Choose the indicator that makes sense given the tech being implemented. Student outcomes may improve as well. But that takes some time.

If new technology isn't improving student learning outcomes, I would first assess the reasons behind the lack of improvement. This might involve gathering feedback from students and teachers, analyzing data on technology usage, and identifying potential barriers or limitations. Then, I'd collaborate with educators and instructional designers to adjust the technology implementation, customize learning experiences, and provide targeted support or training where needed. Flexibility, iteration, and a focus on student needs would guide my approach to ensuring technology enhances learning effectively.

First, reassess its implementation and impact. Start by reviewing the original goals and objectives set for the technology to determine if they are being met. Consider gathering feedback from students, teachers, and other stakeholders to understand their experiences and challenges with the technology. Evaluate the effectiveness of the technology in enhancing student engagement, understanding, and achievement. Identify any barriers or challenges that may be hindering the technology's effectiveness, such as lack of training, inadequate resources, or compatibility issues. Based on your assessment, make adjustments to the implementation plan, provide additional training and support.

Technology must fit within the ecosystem in which educators work, so it’s important to start by assessing its alignment with both the learning process and the broader technical infrastructure. Has the technology been deployed to add value to the educational experience? Is it positioned to become part of educators’ routines? At the same time, does it integrate and work well with other tools that teachers, staff, and administrators must use? Or is it creating extra work? Such sources of friction must be thoroughly analyzed, as they significantly impact implementation and its potential benefits.

While feedback is very important for increasing quality, we should be critical to the feedback received. It can be either ingratiating, or written by people who are infuriated for reasons outside of the course. Often, those who could provide the most valuable feedback decide not to provide any: for instance those who are too disappointed about the time already wasted in the course that they do not want to spend aditional time for feedbacks; or those who are content but do not want to express their opinion (time pressure at the end of a course etc).

When evaluating the effectiveness of a system, data doesn't lie. It works well in tandem with feedback from teachers and students but allows for the conversations to be more targeted since data identifies areas of strength and areas of growth. For example, an analysis of how students performed on unit assessments before and after new technology was introduced can highlight whether or not there was improvement. This can be the entry point for honest and collaborative dialogue with teachers during department meetings or during one-on-one meetings to determine the impact the new program is having and whether shifts need to be made.

Often, the use of technology is easiest where it provides little value added (e.g. presenting pure knowledge; replicating worn-out standard games etc.). Solutions that provide sustainable practical learning experiences which impact the way we work in daily life are very time consuming to prepare and extremely costly to build.

Re-evaluate Implementation Focus on How Students Engage with Technology Find balance: Make sure students have opportunities for hands-on activities, critical thinking, and social interaction outside the digital world.

When new technology is introduced, it can be challenging to learn how to use it effectively for all parties involved. One instance that comes to mind is during the pandemic when teachers were required to incorporate Canvas LMS into their online or blended learning environment. Unfortunately, some teachers found it difficult to transfer this knowledge effectively to their students, which caused confusion. If the platform had been introduced with clear goals and evaluation metrics, the rollout could have gone more smoothly. Therefore, it's essential to establish clear and measurable goals when introducing new technology to avoid confusion.