What the Pandemic Did to Student Brains | A Conversation with Nancy Weinstein

If you asked a teacher what changed most in their students since the pandemic, well, almost every single one would say attention. But what if they're wrong? What if that misdiagnosis is actually making things harder for children, not easier? Hey, I'm Shane Leaning.

Welcome to Education Leaders, the chat-topping international podcast for leaders in schools around the world. I'm an author and organizational coach, and in this show, I bring you practical ideas and honest conversations to help you lead with confidence and clarity. This episode is supported by CC and the International Curriculum Association. Stay tuned to learn more.

Now, my guest today is Nancy Weinstein, Chief Innovation Officer, Otis, which is a school-based data analytics platform. Nancy co-founded Mindprint, which is a cognitive assessment company that has been tracking the cognitive skills across tens of thousands of students all the way back since 2015. And yes, that meant right through and beyond the pandemic. Now, she's a publisher author, she's a National Science Foundation funded researcher, and someone who has turned some genuinely surprising data into practical tools that teachers can use.

Intrigued? Let's jump in. So as part of the Mindprint solution, we actually look at cognitive variability across students. So not limited to special education or gifted, but all students, because all students have variability, all adults have variability, and understanding that variability unlocks full potential.

And so that's sort of the ethos and the way we operate with Mindprint. So at the core, we use a cognitive assessment that was developed at University of Pennsylvania's Brain Behavior Lab, an Ivy League university here in the US, normed and validated through our National Institutes of Health. And so we've been collecting data since 2015 within Mindprint. Penn Medicine's been collecting data from this assessment since, I think, 2010 in children.

And we have data from children starting at age eight, all the way up until adulthood. And so we've been collecting data all the way through the pandemic. And when I say data, we collect what we call cognitive skills. So accuracy and speed scores on complex reasoning, verbal, visual abstract, visual spatial reasoning.

Long-term memory, both verbal and visual. Executive functions, that's a big one. Attention, working memory, cognitive flexibility, and then processing speed. So we collect that data on all students in about an hour.

We can do it very efficiently and very accurately. And so we had been collecting data, like I said, from 2015. And then we, like all ed tech companies, were doing our best to keep operational during the pandemic. Our assessment is entirely online.

And it's self-administered. So we were able to test students during the pandemic while they were in school. And then even in the return to school, some students were home, some students were remote. But through that period, we continued to test students and obviously still do today.

But when we were reading all the research coming out of the pandemic about achievement gaps and learning gaps and kids were coming back to school, behaving differently, acting differently, we said, let's look at this almost from an epidemiological perspective. We've got all this data on, and when we look back, it was about 35,000 students, age 8 to 21, and said, how have they actually changed during the pandemic? Have those cognitive skills actually changed over time and worked with a researcher to understand the data? Incredible.

I mean, this is big. And I assume there's not many studies that have looked in this in this way around the world really. It's quite unique. So I'm curious, what changed in that time?

And I'd love to also just ask, why is that relevant now? I think some leaders might be going in and getting pandemic. Why are we still talking about that? So I'd love to get your thoughts on that.

Well, with the pandemic, I think if you asked a teacher who's been in the classroom for 15 years, they would say kids are different now than they were five, 10 years ago, and that they can see a change and they feel the need to adapt their teaching. And I think that's the real why, which is why do we look at cognitive skills at all, and why are we looking at them more deeply since the pandemic? And the real reason is, is if students are achieving, if we deliver a lesson and every kid understands it, and then they go and they take the assessment at the end of their chapter or the end of the school year and they're all doing great, we should keep going and doing what we're doing. But if we're seeing that students don't seem to be understanding as much, learning as much, retaining as much, being able to do problem solving, self-control, all those factors, it's one thing to observe it because we hear teachers still observing it, right?

Underlying why is teachers are still saying it's different and I'm having a hard time with classroom management. And when we look at PISA scores, we see that they're down in a lot of countries. So that's the why. And then, well, we can look and just say, well, kids are different.

Like, I don't know what to do with kids are different. If I were a classroom teacher, like they're different, okay, now what? What does that even mean, right? Like, does it mean the same thing to you as it does to me, to any teacher?

What does it mean in third grade versus ninth grade? But if we can actually look at these cognitive skills, which we basically explain and talk to teachers about as being, they're the root cause. And you can address, if you know the challenges, memory and students don't remember the content, okay, we know a lot. Learning science knows a lot about memory.

They know a lot about which strategies work in memory. And if we know that's the root cause, we can give students strategies for memory. But if we just look at their tests and say, oh, they knew it in January and then they didn't do it well on the end of your summative assessment, do we say it was memory? Or do we say, maybe they never really knew it.

Maybe they never really did understand it. And that's why they didn't do well. Or maybe they can't sustain focus and they can't sit through a full length long exam anymore. Which is it?

Well, we need to know which it is so we can address it. So what was the things you specifically found then? Like what's actually changed over that time and what should school leaders listening around the world be thinking about right now? I'm gonna answer that question for you, but only if you promise to ask me what didn't change.

Is that a promise? Okay, I'm writing that down, Nancy. I'm gonna ask you. Next question, I promise.

Because I think we do. We talk to teachers all over the world and we always start with, cure those skills that we measure. We measure students' ability to understand new information, their ability to remember it, their executive functions, and then their cognitive flexibility. And we say, what do you think changed?

Because they're the ones observing the behavior and they're making decisions now based on the behavior that they're seeing about what they think changed. And it's always fascinating to see what they think changed versus what actually changed. Oh, so tell me about that. What is this thing that you find between this perception of what people think and what you actually found?

This is great. So what teachers always talk about is attention. Students can't focus the same way that they used to. So if I used to give a 45-minute lesson, now, after 20 minutes, my kids are done.

Their minds are elsewhere and I can't get them to focus. And these are teachers, again, who have been teaching long before the pandemic. So what we found, they all go to attention. And then they'll talk about some other skills.

They'll talk about behavior and flexible thinking. That's sort of like a number two. And then there's other sort of variations of it. So what we found, and we found this consistently across the board.

And what we also found is we can go deeper into the data. But we found the same trends consistently across age groups and socioeconomic status. Okay. But we saw that the changes that we saw, and we'll talk about it, were bigger in younger students and lower-income students.

So keep that in your head. So the big changes we saw, the biggest change which most people don't guess is verbal memory. So students' ability to remember the information that we're teaching them in the classroom. So think about a teacher teaches the lesson on Monday, maybe assigns some homework, maybe doesn't, feels like kids got it, they understood it.

Come back to class on Tuesday, and they continue the lesson, and the kids are just looking at the teacher, like, I have no idea what you're talking about. And so the teacher's wondering, do they never really understand it? Do they not listen to me? In essence, students don't remember as much.

The average student will not remember as much, they'll probably remember about half as much now as they would before the pandemic. So think about a list of 10 vocabulary words. You'd give kids 10 vocabulary words in third grade, and ask them to remember them, go through them in class, come back the next day. Let's say five years ago, they would remember all 10.

Now the average student would remember five. Half. It is literally dropped that significantly. So what does that mean from a teacher's perspective?

Learning is cumulative, particularly in a subject like math, or children learning how to read, it is cumulative. And so if teachers don't realize that students aren't remembering at the same rate as they used to, one, that's really frustrating for a teacher because they think that the children are disengaged, they don't care, they're not putting in the effort to do their homework maybe, you could think a lot of things. One of the things you might think is they never understood it in the first place. So I thought they understood it, but they really didn't understand it.

So then that means they go back and reteach. So now that we're reteaching when we shouldn't be reteaching, and there's this mismatch when really what the teacher needs to know is we need to build in a little bit more repetition. So, well, nobody likes to memorize and nobody likes those stacks of flashcards, right? Not good memories, but they are really effective for a reason, and it's what students need for the content that's most important.

So that's always a big eye-opener for teachers because it's something that is in some way so small, like that idea of five minutes of repetition a day, but it would solve so many challenges that we're now seeing in the classroom. That's so interesting because I don't know what it's like over there, Nancy, but in the UK, for example, there's been a big movement of kind of a lot of science and learning principles over the last five or 10 years. And as part of that, we know a lot about the importance of retrieval practice and to be able to go back to these. And what you've just described to me has now just made this even more important that we are leaning into that because not only is this affecting everyone, but you said also young learners or lower socioeconomic backgrounds are affected even more.

Did I hear that right? You did, exactly. So their memory is not nearly as strong. They were impacted more.

And so when you're younger, everything is new and requires more repetition. And so you can imagine in classrooms, it's transformative if teachers understand for most students, this is what's interfering, not their ability to understand, not their attention, and it changes. It changes what you do and like you said, so retrieval practice is so important for learning and creating a strong base of knowledge for students. And yet it's the thing that with technology, we're like, oh, they don't need to do that anymore.

This is so true. I see a LinkedIn post on that every day. And there's no replacement, right? You can't make connections, right?

Making connections is another big strategy. You can't make connections if there's nothing to connect. Are you now doubly worried like with these latest movements and bringing in AI and things, like the world we're moving into technologically, is there a risk of compounding these challenges that we've only just started to discover? I think so.

I think the science will show us that. I don't know if you saw the recent study out of MIT, which just solidifies what we would think. So there is a recent study about letting students use AI in their writing from their initial first draft versus letting them use the AI after they had already written a draft and then use the AI to help them edit. And when both the quality of the essay and then asked to talk about what they learned in that process, there was dramatic difference between the students that started it on their own and then you use the AI to edit versus the students who used AI from the beginning.

There's no surprise in a way. Learning scientists could predict it. We still need to measure it because sometimes the results will surprise us, but it is consistent with the research. And the great news is the data will lead us to the right solution.

We know what to do. We can look at the data on where students are and what they need, tie it to the right learning strategies and really take students in the right direction and accelerate their learning. The challenge is we don't have enough of bringing that research to practice in the classroom. Well, this makes me super grateful that there's people like you in the world doing this kind of work and talking about this and measuring this so that we can figure out what to do.

I think this is all more important because some of these areas may be not intuitive to us. Like you said, a lot of teachers would talk about attention and baby behavior being a challenge, but bringing to light that the verbal memory is something that we should be addressing. It's not necessarily intuitive for leaders in schools to be dealing with, so I think it's really important that we're getting that message. Tell me if this sounds familiar.

Your board asks for a school-wide report on academic achievement and you've got 24 hours to deliver it, so you start digging. But elementary data, well, that is on one system. Middle school is on another. High school data is somewhere else.

And then there's exam boards on top of that. Well, by the time you've tracked down the 10 to 15 places your data lives, there is no time left for actual analysis. But the thing is, that status quo is not normal. It is just what we've gotten used to.

And the good news is CC changes that. CC brings every part of your school into one simple platform so you can spend less time compiling reports and more time leading. If that sounds familiar, you can check out CC.org or S-I-S-I.org.

This episode is supported by the International Curriculum Association. The ICA have been around for 30 years now, championing quality, unlocking potential and improving learning in international schools. And what I really love is that right at their core is the model for improving learning. This is a model focused on the learning experience and they have got tons of great curriculum materials, PD resources and even an accreditation pathway for schools just like yours.

If you're interested, head to internationalcurriculum.com. I wonder then, Nancy, you've just mentioned the memory thing. I know there were also some other points and I'd love to kind of key in on one particularly, which is you found big drops in something called flexible thinking.

I'd love to get your take on that. What does that look like in a classroom and what have you found? So flexible thinking is this idea of being able to take feedback and adapt and adjust your approach. And isn't that what we're really asking students to do every day in the classroom, right?

If they're not making mistakes, taking feedback and learning from it, then what is school all about if not that? And so what the data suggests is that students' ability to take feedback and adapt has dropped significantly and that makes sense. And so flexible thinking impacts some of our brightest learners. So students who have great critical thinking skills might still have weaker flexible thinking.

Okay. So this idea that if you write a paper and the teacher gives you feedback and says, I want you to rewrite it and think about this instead, how does the student respond to that? Do they say, oh, that's really interesting and I'd love to try and incorporate that perspective or the teacher doesn't like me. And so that's why they gave me negative feedback.

Or when we're doing math problems and teachers ask students to try it a different way and they just get stuck because they see it only one way. And we know flexible thinking drives all of creative problem solving. And what do we need students to do now more than ever in this age of technology and AI is to be creative problem solvers. And yet this skill has really declined significantly in students, at least during the pandemic, and we'll continue to monitor and see if it goes back up.

What I saw in the United States, in the schools that I worked with, but also in my own house is it made sense to me why our students might have weaker flexible thinking. Because as parents, as teachers, we were so concerned about their mental health during the pandemic, being on the computer. We were afraid that they were so fragile that we were afraid to give them the corrective feedback. There was a lot of great inflation here during the pandemic.

They didn't have the natural cadence of adapting to different teachers in different classrooms. Think about walking in the hallways in any given day and taking that feedback and you're speaking too loudly or you can't behave that way. They lost all of that, particularly our youngest learners. And we were so afraid of upsetting them that I think even as parents, a lot of us didn't give our children as much constructive feedback, if you will, as we might otherwise have.

Perhaps teachers didn't give as much constructive feedback. It's okay, they got the core concept. We're not gonna try and go any further. We'll leave them alone.

It was sort of a general feeling and I think that students really didn't develop those skills of taking feedback and saying, what do I do with this feedback? How do I process it? How do I react to it? And what do I do differently?

It's making me think of a word from medicine, one of my favorite terms that I came across a few years ago. The idea of iatrogenic effects, which is, you know, in terms in medicine of, you know, you put in something in place to help someone and then there's an unintended consequence around the side. So we put in place kind of these support mechanisms to help young people with their mental health and help them feel okay and comfortable. And yet there's this iatrogenic effect that's happened in that now that has set them up to struggle with their flexible thinking and to be able to take feedback and adapt.

This is fascinating. It is fascinating. The thing about flexible thinking and memory, so on the positive note, flexible thinking is probably the most malleable of all the skills that we assess. So if we know that our students have weaker flexible thinking, and that's the challenge, not that they're disengaged or they're being stubborn and they don't wanna listen, but we view it as, okay, it's cognitive flexibility.

It's where they are now, and there are strategies to help support and develop that skill. Now we're giving teachers a solution and we're getting away from the problem that kids are different. So we can validate teachers. Yes, they are different.

Here are at least two very specific ways that we've seen across the board that kids are the most different. And here are the strategies that you can now bring into your classroom to address those challenges. So if we can get the word out, I think we can really help teachers and solve what feels like a near universal problem. So that to me is super exciting, and you can be super optimistic about that.

I think that is exciting. And what I really like is that the way you've described it, it's a slightly different framing. You can see this all as kind of these behavioral things with children, and then you feel like it's difficult to do, but actually there are pedagogical strategies. There are strategies in pedagogy that we can use science and learning strategies to help.

So it's not just this, oh, no, kids are different. Therefore, everyone's just having a challenging time. There are really good things, just like when we mentioned their retrieval practice, and I'm sure lots of other strategies that teachers could find. I wanna come on to one other thing, Nancy, but I wonder just, are there any pointers where if a leader's watching and they're like, yeah, I wanna give my teachers some of these strategies.

I want to kind of enable them. Do you have any advice on where they might look to kind of pick some of that stuff up? So there are great resources. John Hattie has his visible learning, and there are 500 evidence-based strategies.

And so we always say, what works in education is known. So there's John Hattie, there's Bob Marzano. We at Mindprint have basically curated the strategies in our own version of them and made them free to everyone if people wanted to go on our website. But there are several other folks who have sort of curated a list of evidence-based strategies.

So to me, the question is not what the strategies are. Daniel Willingham, if anybody's read his work, amazing, under the cognitive scientists. It's about prioritization for teachers, because I think that's where the breakdown is, which is, we'll go back to John Hattie, 500 strategies. Jeez, where do I start?

And so the fallback is always to go to, well, when I try this, this is what works, so this is what I'm gonna do. And it's too much. But if we can pinpoint its memory or its flexible thinking, then we can narrow down the strategies to those that are the most important. What I love about that is that you've made this finding which I think is pretty huge and everyone should know about it.

But you're not just kind of saying, so here's some brand new strategies that we've created to deal with this. Actually, there are established science and learning, good quality. A lot of people have been doing this thinking and we just need to prioritize into a few of those areas that are already known. So we're not creating something brand new to deal with the fallout of the pandemic.

We're able to lean into strategies that we know are good strategies anyway, which I find incredibly hopeful. I wonder, Nancy, before we finish today, there was one other finding that you found which I just thought was really interesting because in the study, you actually tested teachers too, right? We did, we did. So leaders listening are gonna be like, oh really, they're gonna be fascinated about this.

What did you find and what does it mean for how leaders should be supporting their staff as well? What we found is that teachers are in many ways no different from their students. So we saw that with the adults, their memory also declined, but realistically, they're teachers. They know their content, they know their subject matter, they don't need to memorize a whole lot.

So even though their memory may be declined, they can use their sticky notes when they need to and they don't need to worry about it, right? They're not in school where they need to learn a lot of new things, but their flexible thinking also declined significantly. And so think about the school leader telling teachers, we have to try this new program or we need to do this differently and it's hard for them to take the feedback just as much as their students. So think about classrooms where you have students not responding the way you would expect and then the teachers also having a hard time of what do I do when a student doesn't respond?

Sociologically, it's fascinating, but it does explain a lot of the reasons teachers maybe were leaving the classroom because they were really frustrated and they didn't know how to respond to students behaving differently because they were struggling with their own flexible thinking. To be clear at Mindprint, we only work with teachers, so we only have teacher data. I imagine this is not limited to just teachers and it could explain a lot in the world order. But I do think it's fascinating and we often when we go in and work with schools, the heads of school are almost like it's the teacher data and us working with the teachers and sharing the data with the teachers and making the teachers self-aware that is almost more important to them than the student data.

I think that's absolutely fascinating and I've had quite a lot of just anecdotal conversations with leaders over the last few years who have said things like, I've just noticed a difference in staff or struggling with things that maybe used to be automatic in their practice or just many staff sometimes forgetting meetings and things like that and just seems like what's happening here and potentially the staff are just going through the same thing that their students have gone through, which is totally understandable and I'd recommend anyone listening go and have a look at this report and some of this research because if anything, it potentially is gonna make you a much more empathetic leader to kind of understand what's happened. Exactly, great. I mean, if we can understand why people are behaving in an unexpected way, it really does help in terms of how we can support them and feel about it and not take it personally and really address the root cause. So this conversation I found super interesting, super challenging as well, right?

The thing that keeps coming back to me is how often we observe a problem in schools and then think we know what it is, but actually the data is telling a different story. So a couple of things I'd encourage you to think about. First, if your teachers are saying students aren't retaining things in the way they used to, well, maybe trust that and point them towards some retrieval practice, maybe space repetition. It's not new.

It's just probably now more important than ever. And second, that flexible thinking finding, wow, that's really struck me. Students who struggle to take feedback, well, they're not actually just being stubborn. There might actually be a cognitive reason for that and knowing that would change totally how we respond.

And finally, don't forget that teachers may be going through the same thing too, right? A little more empathy, a little less frustration might go a long way right now. You can find Nancy and some of her brilliant work using the links in the show notes. Education Leaders is hosted by me, Shane Leaning, thanks to the show editor, Pete McGill, production assistant by Skyler O'Sturman and the original music by Guillermo Silva.

And thank you as always so, so much for tuning in today. If we don't speak before, I'll see you here next week. If you wanna learn more about CC or the International Curriculum Association, check out the links in the show notes.