Rethink Imaging

EP 27 • September 11, 2025

Peer Learning: From Toxic Scoring to Transformative Education

Featured Guest

Dr. David Larson, MD, MBA

Professor of Radiology; Founding Director, AI Development and Evaluation (AIDE) Lab • Stanford University School of Medicine

Listen Now41 min

Also on:

Let's Talk

Beyond dose

Ask us what your imaging data can actually show you.

Start a Conversation

Share This Episode

Subscribe to Rethink Imaging

Most radiology peer review programs score a sample of cases on a one-to-four scale and log the misses. Dr. David Larson, professor of radiology at Stanford University School of Medicine and founding director of its AI Development and Evaluation Lab, spent years watching that model fail. Conferences turned into arguments about the score instead of the case. Radiologists wrote down case numbers to figure out who made the mistake, or whether they did. In his second visit to Rethink Imaging, he tells host Chris St. John how his team at Cincinnati Children’s threw out scoring altogether and rebuilt case review around feedback, learning, and improvement.

The conversation covers what replaces the score: three pillars of peer learning, ground rules that keep meetings free of blame, and a case mix that includes great catches alongside misses. Larson explains how the model separates system problems from individual error, how three misplaced PICC lines at Cincinnati Children’s became a department-wide reporting campaign, and why participation and psychological safety say more about a program than any number. He also gets into the business case, what AI developers should take from peer review’s failures, and how the approach works in community hospitals as well as academic centers.

Host

Chris St. John

Host, Rethink Imaging • Imalogix

Featured Guest

Dr. David Larson, MD, MBA

Professor of Radiology; Founding Director, AI Development and Evaluation (AIDE) Lab • Stanford University School of Medicine

Watch the Episode

Key Takeaways

Traditional peer review scores a random sample of cases on a one-to-four scale and logs misses for possible remediation. Larson found the conferences spent more time debating the score than the case, with radiologists writing down case numbers to work out who made the miss.
At Cincinnati Children’s, his team tossed scoring out and rebuilt review on three pillars: feedback, learning, and improvement. Feedback about a miss is delivered privately and constructively, and there is only one right way to receive it: say thank you.
Peer learning meetings open with ground rules (no fault-finding, no blame), then go straight into cases, including repetitive perception cases like a “missed fracture clinic” and great calls. Trainees get to see that attendings are not perfect without anyone being singled out.
The model separates system problems from individual error: unoptimized protocols, scheduling gaps that put a radiologist out of their depth, and IT dead ends get fixed as systems issues. Three misplaced PICC lines, one fatal cardiac tamponade among them, became a reporting campaign led by Dr. Neil Johnson that made PICC line reporting far more careful.
Larson’s advice on the business case: if the goal of peer review is revenue, the goal is wrong. Measure participation and cultural change instead of scores, and anyone building AI feedback tools should first study why score-based systems did harm.

Full Transcript

[00:00:01] David Larson: I went down the path of kind of a traditional score based peer review, and we brought it to the conference. And it turns out we’re talking in this conference more about the score than we were about the case. And it was a very adversarial type of conference. And someone said, that should be a four. No one should ever miss this. And then I would see people like, we write down the case number to see, number one, who is the idiot who missed that case? Or number two, was I the idiot who missed that case? So it in concept, like, in theory, it looks like it makes sense. In reality, it doesn’t. So at Cincinnati Children’s, we completely redid the way we thought about it. We tossed scoring out altogether.

[00:00:40] Chris St John: Welcome to Frame by Frame Rethink Imaging, a podcast by Imologix. Here, we explore the intricate world of medical imaging, aiming to dissect the field and inspire both professionals and curious minds alike. I’m your host, Chris St. John. Today on Frame by Frame Rethink Imaging, we are so excited to welcome back to the podcast doctor David Larson, the director of AI development and the evaluation lab at Stanford University School of Medicine, where he is monitoring the performance, ensuring the reliability, and the effectiveness at AI. He is living at the intersection of AI and quality. So over his eleven years at Stanford, he’s led education, clinical operations, and performance improvement. He really brings a systems thinking approach towards advancing healthcare delivery. Doctor Larson is also a seasoned pediatric radiologist, having served at Cincinnati Children’s Hospital and as well at the Utah Radiology Associates. His work in dose optimization, patient safety, and technology has placed him at the forefront of radiology and medical imaging, earning recognition for transforming both education and patient care. Today, we are gonna focus on one of his most influential contributions, shaping peer learning as a transformative alternative to traditional peer review. Welcome back to the show, doctor Larson.

[00:02:07] David Larson: Thanks so much, Chris. Great to be back.

[00:02:09] Chris St John: Should we get I I feel like we just need to, like, set the stage a little bit. Right? So in I believe it in at RSNA in 2017, you had a hot topic discussion where you described peer learning as a fundamentally different paradigm from peer review, anchored in feedback, anchored in learning and improvement rather than just numeric scores. Could you unpack that a little bit for us and lay a little bit of groundwork and just kinda get into what makes this framework more effective than the classic score based peer review model?

[00:02:43] David Larson: Sure. Absolutely. Well and and thank you again. It really is a pleasure to be back with you and talking about a topic that I I love and have has been dear to me for a long time. Well, maybe we’ll start with the beginning of what score based peer review is, you know, the concept behind that. It’s admirable in its ambition, right, in its in its effort to try to improve quality or at least ensure quality. Right? So the concept is for score based peer review that, you know, you have a random sample of cases of your peers that you you take a look at and you assess on, you know, a score of one to four, one to three. And, you know, for those that are, you know, of misses, essentially, then, you know, just keep track of those. And then it’s not exactly clear what happens after that. But, you know, presumably, if someone is a a liar, then they are either you know, there’s remediation or they would be removed from practice. Right? That’s that’s kind of the concept behind it. And it’s it’s a very kinda quality assurance kind of mindset. Right? And it it actually has it it echoes kind of what in many fields, actually, when they first focus on quality, they focus on, you know, minimizing the errors, right, And looking at kind of the scoring, like, you would monitor a machine in the same in a similar way, right, to see how many errors it’s producing. If it’s producing too many errors, then, you know, then there’s a problem there. The problem is that’s not how humans work, especially, you know, uh, professionals, right, doing work that is very difficult or at least historically has been very difficult to quantify in that way, but also are people who con are dynamic, right, who constantly learn and who interface with each other and have relationships with each other. And so those same people who are scoring or judging, those who are in their work are also the same people that they’re, you know, they’re having a conference with at noon and, you know, they they’re their professional colleagues. Right? Right. Well, that can’t be discounted. So when I I’ll tell you a story of the background. When I went to Cincinnati Children’s, I was one year out of fellowship and my the chair at the time, Lane Donnelly, he gave me two assignments. One was to implement a standardized structured reporting program, which nobody ever implemented before, and then the second was peer review. Those are, like, the two things that no radiologist wants you to touch. Like, it was, like, the worst thing that you, you know, junior attending can come in to do. Let me tell you how what your report should look like, and we’re gonna standardize them all. And, also, let me tell you about all your mistakes. And so I learned a lot, especially in fact, I think in many ways, being junior was probably an advantage because I couldn’t just come in and and I didn’t want to, anyway, come in, you know, hit people over the head. It it was not what we wanna do. So I went down the path of kind of a traditional score based peer review, and it brought it to the conference. And it turns out we’re talking in this conference more about the score than we were about the case. And it was a very adversarial type of of conference, and someone said, that should be a four. No one should ever miss this. And someone said, I think that’s a two, if not a one. Like, that is completely understandable. I’m not even sure if it’s a real finding. And they would go back and forth and, you know and then I would see people, like, we write down the the case number to see, you know, number one, you know, who is the idiot who missed that case? Or number two, was I the idiot who missed that case? Right? So it in concept, like, in theory Yeah. Like, it sounds like it makes sense. In reality, it doesn’t. So at at Cincinnati Children’s, we completely redid the way we thought about it. We tossed scoring out altogether. We had a a section that we would review the cases together. We would inform everybody of misses. We added great call cases. And we made it a much more dynamic and learning environment. And that was the that’s really the the concept behind it. That so that moves us from a QA mindset we talked about at the beginning where the, you know, modern quality doesn’t doesn’t think about things in terms of QA. They don’t think about finding the bad apples anymore. You know, a sophisticated quality program, you know, is all about in kind of embedding continuous improvement in the process in the system. And professionals, like, instinctively, we we know this. Right? And we’re actually really good at this if we’re given the right environment to be collegial and supportive and constructive, even difficult conversations, even with missus. And so that’s what we really tried to do is frame that in a way that could bring feedback to individuals. We could learn together. And if we can change the the system to improve the system, then we do that to minimize the chance of making those errors in the future.

[00:07:21] Chris St John: It makes a lot of sense to me. The numerical scoring is it’s, like, naturally intimidating to some degree too. But hearing particularly, like, how people were arguing with each other over it rather than, like, getting back to what can we actually do to improve this, it it it feels so inherently human. And so to tie it back a little bit, I believe we were talking about, like, those three pillars. Right? Feedback, learning, and improvement?

[00:07:50] David Larson: Yeah. So the three pillars, as you say, feedback, learning, and improvement. So feedback, you know, when you make a mistake, if somebody, your colleague or anybody in the system, so to speak, comes across that mistake, you should learn about it. I should know about that. Right? That should be given to me. It should be done in a constructive way, confidential, private way, and, you know, supportive way. Right? And so and and then when I’m receiving that feedback, there’s also there’s a choreography. Right? I need to know how to receive it gracefully. Right? And there’s only one answer when somebody tells me about my miss, and that is thank you. Right? I may actually we can have a discussion. I may even look at it, you know, more carefully and say, I’m not sure that’s a miss, but it’s about the discussion keeping it constructive. So that’s the individual feedback part. Right? The learning part then comes as we, as a group of individuals, come together. Right? And when we look at a case that somebody missed, we don’t care who missed it. It doesn’t that in fact, we generally we try we encourage people just take the names out of it if you can de identify it because, you know, the the principle is there but for the grace of God go I. Right? I could easily have been the person that made this mistake. I know whether I was or not because I would never go into a conference without having already been given the feedback, but it could have been me. And now let’s learn about what we can do to not, you know, make that mistake or strategies that people use to to improve, you know, in what they learned, what the experience has taught them. And then improvement is actually something that is really important, and it’s been a really gratifying part of this where sometimes the human is the last person who catches all the misses, and sometimes they don’t catch it, you know, and then you you look at it and say, well, it was human error. But in reality, there were a lot of things that could have happened to set that up so they wouldn’t have been, uh, it wouldn’t have been an error in the first place, so it wouldn’t have have been missed. So, for example, if a technologist has a a concern, like, they can raise that. If they don’t have anywhere to put that in the IT system, that’s an IT problem. Right? If a protocol is not done so that the image quality is optimized and I miss a finding that is more subtle than it needed to be because the protocol wasn’t optimized, we need to go fix that protocol. Right? If somebody wasn’t available because of scheduling, because we didn’t have the right expertise available and I was now reading out of my depth, you know, that’s a systems problem. Right? So the whole thing is all about learning. So if you get out of the mindset of, you know, let’s go and look at that bad apple, right, and say how many misses did that person make and and change the whole frame to say, let’s not worry about that. Let’s worry about how we can as get together as individuals, as a group, and as a system, then you start to improve over time. It’s better patient care and quite frankly, it’s a heck of a lot more enjoyable place to work.

[00:10:34] Chris St John: Yeah. I mean, of of course. I think you have said, right, that traditional scoring can drive radiologists, quote, inward against each other and against practice leaders. I mean, after after seeing different departments kind of transition from SCORE to these peer learning discussions, have you observed these sort of, you know, like, either a positive, surprising cultural change within these institutions?

[00:10:58] David Larson: Oh, yeah. Absolutely. In many ways, it’s actually easier in most ways, it’s easier to go from score based peer review to peer learning than from no peer review to peer learning. Because when you go to from no peer review to nothing to peer learning, it’s like, really? We have to talk about our misses? But when you go to, you know, to not to score based peer review, people are already they’re already trying. These are people who, like, take what seems to be logical and apply a framework, and you’re almost always well meaning individuals. Everybody usually is well meaning. That’s why it’s like this Greek tragedy. Right? That, like, all these well meaning individuals that end up, you know, because of their their good intentions, end up actually causing harm because there’s fundamental dynamics underneath it that there’s not that’s not accounted for by the way that they approach it. Right? So when they now move from that score based peer review into peer learning, it unleashes what they’ve been wanting to do really in the first place, and that is to to truly learn, uh, review cases. And it changes when a miss you know, when you put people on the defensive, you’re you are adding insult to injury, right, inherently. And so it’s in it’s the design of the system as insult to injury. When instead, everybody talks about their misses and everybody just assumes there are going to be missus and we’re gonna learn from them, it’s a time that people kinda rally around each other. At least it feels like it. Right? It feels understandable. It feels like we’re all human and we have grace. And the administrators may even have grace and recognize that, you know, that this is a human endeavor, and that’s a good thing. And almost always, you know, for the most part, that is what that’s the magic of it. That’s why it works so well is because you have people, you know, brilliant people who are incredibly dedicated. But every once in a while, when it doesn’t go, you know, you miss it, uh, when there’s a missed case, well, then let’s learn from it. And if you use that humanity, you use that group dynamic in a constructive way rather than to find, you know, to kind of ferret out the the bad apple, so to speak. It changes the tenor. Right? And I witnessed it at Cincinnati Children’s, and I’ve witnessed it at many places, and we hear the same thing when people say they they switched over. It takes more work to run a peer learning program, but the the culture change is is impactful.

[00:13:17] Chris St John: Yeah. So you wrote a paper transition from peer review to peer learning in which you detailed, like, some early hurdles such as legal discoverability fields, data overload. Which of those challenges do you think are the toughest for, you know, new adopters who would like to take this on in the future? Uh, and which do you feel like can be solved by, like, tweaks to workflow or technology or AI?

[00:13:43] David Larson: Yeah. Great question. So I I think there’s kind of two steps. One is getting over the hurdle of and it’s usually local administrators allowing you to move away from a score based peer review program into a peer learning program because it’s a different paradigm. And for administrators who are used to looking at numbers, like, that feels really scary and counterintuitive. Um, and it feels like they’re also often kinda used to people trying to game the system. It feels like people are trying to game the system. But almost always when, you know, they look into it, they’re well meaning and have an open mind and have, you know, reasonable trust in their radiologists, then what they find is, number one, lots of other people have done this. And number two, like, they’re not we’re we’re not trying to pull something over. We’re not trying to be lazy. We’re not trying to get out of something. We’re trying to free ourselves from the shackles of this, quite frankly, toxic and and poisonous construct. It not it wasn’t intended to be that way, but it’s just the outcome of it. Right? So it’s having that that’s the first hurdle. And once you get over that, then it it does take more work. And that’s usually the hardest thing is it requires somebody to actually go through the cases that were missed, requires to look at them, you know, assess them, have with some consult with people. It requires providing feedback in a confidential and supportive way. That actually takes time to do that. Right? It requires preparing the cases for a conference and then having that discussion, everybody coming together and meet together and and review. So it’s really about the time. And I would say we could use additional tools to help with a lot of, kind of, the details of gathering that information and keeping it confidential. Right, that most tools are not able to do that. And when they do, quite frankly, a lot of the IT developers, they don’t understand, like, some of those nuances. They don’t understand, like, how critical you it is to never ever accidentally reveal the the identity of the person who made the miss in a conference. Right? So you have to build a tool that will facilitate that. So, yeah, those are the those are the main things, but they’re totally doable. And, you know, if you streamline it, it actually doesn’t take a whole lot more time. And it is it tends to be, like, some of the most beneficial learning that practices do. You can get c we have practically, we’ve got CME credit for it. Practices where people come in on their days off to go to this this meeting because it’s their most valuable meeting of, you know, the whole practice or the whole in in their in their department. Do you have any advice for for folks who would be looking to streamline like that? There are some IT tools that can help. For one, I’d say, mainly, if you’re looking to streamline or if you’re looking to do anything with your peer learning program, there are lots of other people who have figured this out. We have a committee at the ACR. It was founded by, uh, Jen Broder, the peer learning committee. Jen Broder started it at, uh, she’s she’s the Lahey Clinic, and now Rich Sharp, who’s done just a tremendous amount of work first when he was in Kaiser in Denver and now leads this nationally as he’s transitioned over to Mayo Scottsdale. And there’s a whole community around this. Lots of people who have gone through the journey of having those conversations with their administrators of, you know, setting up something that is feasible, you know, of going working through all those, you know, all kinds of challenges. And and the community is, like, unbelievably willing to help. Like, these are the people who have, like, made this their, you know, crusade. Right? This has been their passion for so many years. And to see it come to light, there’s nothing that we love and they love more than somebody calling up to say, hey. I’m trying to set this up. Do you have any advice? Like, absolutely. We’ll get you started.

[00:17:23] Chris St John: I love that. And so, I mean and so what would you say for like, how would you adapt a peer learning design for, like, a smaller, more rural community hospital with, like, limited subspecialty coverage versus, you know, a larger academic center like where you are at Stanford?

[00:17:40] David Larson: Yeah. Great. So first of all, good question, and we get this a lot. And I would say, you know, that’s the beauty of peer learning is it it it’s very flexible. The concept is very flexible, and it should, almost by definition, be adapted to your local environment. That’s what peer learning is all about. It’s you learning along with your peers. Right? It’s reinforcing that relationship. It’s improving your performance in the construct that you have. So, for example, at an academic department like radiology, one challenge we have is we don’t have one peer learning program. We have 11 peer learning programs, basically, one in every division. Right? Because it doesn’t make any sense for, you know, me as a pediatric radiologist to go to the breast imaging, you know, peer review conference or, you know, for musculoskeletal to come to IR. Right? So at the smaller sites, I’d say, or any site, tailor it to what makes sense for you. So we had this really wonderful presentation that was a a practice from, I believe, kind of the Northern Virginia and West Virginia area, very small practice. And they would get together once a month, and they would just keep track of their missed cases. And they would just come and, like, have a list, and they go through and pull it up on their packs. And they’d, you know, have maybe a handful of people, half a dozen people, and just go through the missed cases. And, like, great. Yeah. That’s it. Right? Because that’s plenty to get you talking about, like, here are things that we missed here, you know. And then then they would also provide feedback, you know, they let people know, you know, when you’re in a small group, you can’t de identify it anyway. So you don’t necessarily need a great big, you know, de identification system and and data management system. You just let people know. It’s a culture thing. Right? You let people know about issues that come up, you have a discussion, and then you just come, you know, with an open mind and you share and you learn and you continue to try to get better. That’s that’s what it’s all about. We’ve had we’ve also had environments where you’ve got, like, a handful of one or or maybe two subspecialists within a larger practice that they’re kind of the only subspecialist. Right? And so they want to have a peer learning experience in their subspecialty, and they’ve gotten together, like, in a regional area and get they they come and they share their misses as well. So it all works. Like, it’s mainly about, you know, who are who are your peers, who you’re learning from, get together with that group, and learn from your misses.

[00:19:58] Chris St John: Yeah. And and and so this follow-up question may be, like, a bit that you might have already covered this, so forgive me. But so how would you structure these meetings? You know? Or, like, how would you how would you explain to someone I I feel like you just covered it in some way, but Sure. To dig our teeth in a little bit more.

[00:20:15] David Larson: Yeah. So I’ve seen it done, you know, a handful of different ways. But for the most part, they’re they’re pretty much the same. The first part, at least at first, when you start the meetings, usually, you kinda have to have a preamble to say, okay, this is what, you know, peer learning is about. Here are the ground rules. You know, we’re not gonna find fault. We’re not gonna blame. We’re not gonna try to identify. We’re not gonna, you know, talk about how bad this you know, like, that’s we’re not that’s what we’re not gonna do. What we are gonna do is focus on learning and we’re not, you know, we’re gonna focus on how we can each of us individually can improve. Right? So that that baseline and then you just dive into cases. So here’s the first case. And, you know, often what we find is, like, there’s the perception cases and then there’s the, you know, diagnosis cases. And perception cases is kinda like we’d call it our missed fracture clinic. You know, it’s just, here’s a fracture. Oh, whoops. Yeah. I can see how you can move that. Okay. Here’s another fracture. Oh, shoot. Okay. I can see how you can miss that. Yeah. And and it’s actually really helpful. It’s kinda like I likened it to you know, I don’t know if you’ve ever seen those bumper stickers that says start seeing motorcycles. Right? Have you ever seen those? The concept is, if you just think process this and you see enough of these, right, just that repetition of I know it’s a lot of the same type of missed cases, but it reinforces that now when I go to the reading room and I see that, you know, distal tibia fracture that is, like, that’s oblique, that’s just right overlying the fibula because you can easily get fooled and think it’s the overlying fibula on that lateral view, but it’s actually distal tibia fracture. Like, okay, now I’ve looked more carefully for that. Right? So that’s the first part. And then the second part is the the interpretation cases, and those are usually longer discussions. And, you know, people weigh in and they give, you know, you this is where you start to see a lot of the expertise come out where people, you know, people have great who have more experience will kind of identify, you know, learning aspects, like how, you know, they can what they do to try to or how they would approach it, right, and what they would do to try to have to make sure they come to the right conclusion. Or it it sometimes it’s like those cases are also the ones that really, uh, elicit the kind of process and systems improvement part. It’s like, well, you know, the like, oh, I well, a person who read this, it turns out was, you know, this there was out of their subspecialty and, you know, they didn’t have extra coverage. Great. That’s a good conversation starter for subspecialty or for the, uh, systems issues. So it it turns out it it ends up being a bit organic. It’s structured in that it’s all case based. That’s the key. It’s gotta be case it’s about actual cases. And the the the fun thing to watch is the the great calls. Right? Where you put in a case, you don’t tell them it was a miss or a great call. You just put it in, and someone says, woah, this is a tough case. I can totally see how they missed it. And they say, well, actually they didn’t miss it. Right? Somebody caught it. It’s like, woah, that’s fantastic. Because the material is the same. Right? Yeah. It’s whether whether you actually caught it or not, if you could have missed it, then it’s a it’s a great peer learning case. Right? Because, you know, the only difference between it is the person happened to catch it at that time. Right. Are there

[00:23:20] Chris St John: any cases or specifics that stand out in your mind where you were like, wow, holy crap.

[00:23:27] David Larson: Well, I have to say there maybe I shouldn’t, but

[00:23:30] Chris St John: You you don’t no no pressure, please.

[00:23:32] David Larson: No. There’s one radiologist at Cincinnati Children’s when we started it, Chris Anton, that would constantly have these amazing calls. Like, he does a lot of pediatric musculoskeletal imaging. And everybody, you know, knows that he’s brilliant, and he he he’s just he’s just very down to earth person, but, you know, he can man, he’s got a great eye. And we would just have these impossible cases, and I’m trying to think. There are a lot of, like, those subtle musculoskeletal fractures, like, the like, it’s the tiniest of a hint that you really have to squint and make it big. And you just you just see them. And it’s it’s just so enriching. Right? Because, you know, if you think about, like, all the learning activities you do in radiology, there’s nothing more, you know, impactful than something that it could be you. Right? It’s like you it it this is my practice. This is a case that just I just happen to be not the one that clicked on it, but someone else clicked on it. And if I had clicked on it, I would have gone down the tubes. You got my attention. I made those off in the lecture, you know, at the, uh, CME course. But this one, that’s, like, gonna be my next case that I’m gonna, like, fail if I don’t, you know, pay attention. Like, and it’s in a constructive way because it’s it’s free. You know. I’m not gonna miss it. I can’t hurt anybody because it’s already done, but now we can we can learn from it. So it tends to have, like, it was, you know, we had it ended up being with one of the most in many places, it often is one of the most popular conferences. The trainees actually really appreciate it too. It’s a really great exercise for trainees because number one, they get to see that their attendings aren’t perfect without, you know, having people having to be individually humiliated. Right? And number two, they also get to see, holy cow, some of our attendings are amazing, and I need to up my game to on both accounts, I need to up my game.

[00:25:18] Chris St John: Yeah. Yeah. Yeah. Yeah. And so, like, we’ve talked about how the goal is not to score error, but rather to learn from error. But are there are there metrics that we feel or that you feel are important, whether or not it’s like a KPI or an indicator? Is there anything else that you’re trying to track to process that these sessions are driving change? Or is it is it truly just, like, rooted in conversation?

[00:25:46] David Larson: Well, it’s a great question. And and one we get a lot. We really emphasize that the main measure is participation. You know, what you’re trying to do is foster an active, healthy participating, you know, participatory dialogue, right, where you have people who are active in submitting cases. They’re not a there there’s there’s psychological safety. So that’s one one aspect of one measure of psychological safety is when people are submitting cases. In fact, I would say the sign of the the best sign of a mature peer learning program where there’s true psychological safety is when they submit their own misses. Right? Then they’ve they clearly now are they’re comfortable that they’re they’re not it’s not gonna be used against them. Right? And then, otherwise, it’s also you can look at the improvements that come out of of the program. But in terms of, you know, calculating, you know, some KPI about how some performance KPI, it really not right now. Yeah. Not at this point. Now I will say well, actually, I’m gonna say that with a caveat two caveats. One, there’s what you often see with a peer learning program when it becomes more mature is you you sometimes see a campaign. That’s what we some some people would call it, right, where there’s a theme. These themes start arising. You start to see misses in the same the same types of misses. So one that was really impactful for me, this was incredibly impactful, was at Cincinnati Children’s Hospital, we had a kid several cases, actually, three cases where the the PICC line the PICC is a peripheral indwelling central catheter. So it’s it’s pretty common that people get this catheter for, you know, for chemotherapy infusion or something that lasts for a long time. And so the the catheter tip, it it you wanna put it right above the heart. It goes centrally, but you don’t want it in the heart. Right? And when you do, it can actually it can actually perforate the heart, the muscle wall, and then it causes a hole, and then blood leaks out, and then, you know, it gets restricted by the pericardium, which is a sac around the heart, and then you you can have real problems. So there’s actually a patient who had died from a cardiac tamponade, and then two patients that experienced cardiac tamponade from myocardial perforation from a PICC line, which these seem to be incredibly innocuous, but actually it can be really dangerous. So we presented all those cases together. It it well, here’s the other problem. Because they seem to be so innocuous, people often don’t take them seriously. It’s like it’s where the pick is noted. Like, well, no. Actually, it needs to be in that specific location, and that location needs to be defined. So Neil Johnson was a pediatric radiologist, interventionalist at, uh, Cincinnati Children’s led a huge campaign to really refine the exactly how those things should be managed and how should they should be reported. They were presented a a cine image of a heart beating with the that that PICC line hammering into the muscle. Right? And it’s like, oh, I see it. And then we can presented because at first, you know, they say or presenting PICC lines, there was rules arise. Let me tell you a little bit more. We present that, and then we present three cases, one of whom died, and this was this was earlier. And then we present, like, how well we’re reporting those, and their answer is we’re not reporting them well. And you come out of that and say, wow, you know, that was talk about a change agent, you know, after that, then we started monitoring how well those were being reported and they’re much more carefully reported. I mean, that was certainly in me, you know, was made me think about it very seriously. Right? So when you have those campaigns, then, you know, they can actually drive improvement, dedicated improvement over time, you start to see that.

[00:29:29] Chris St John: Yeah. Let’s talk about I hate getting into financials. Right? But, like, let’s talk about needing to make a business case for the c suite. You know, hospital CFOs, I feel like, are naturally going to ask for, you know, some sort of ROI or some sort of proof in the pudding when confronted with a quality initiative. How how would you frame a business case for peer learning if if leadership is so focused on, you know, the dollar value?

[00:29:57] David Larson: I personally struggle with this one because it’s hard not to, quite frankly, to get a little emotional about this because Sure. I think if your if your purpose is to make more money from, you know, from peer, you know, peer review, it’s I think you’re you’re approaching it wrong. Personally, I think you’re in the wrong business.

[00:30:13] Chris St John: Yeah. No.

[00:30:14] David Larson: I I think that, you know, you we we ethically, that is actually what we the core of what we do, which is, you know, continuous improvement of excellence in the care of our and it does have, you know, increase it’s it’s continuous improvement. So I would say to any CEO, if you have any care at all about continuous improvement, well, there you go. What often happens is you find that workflows were not managed well. That gives you an opportunity to change your workflow. Right? Or manage you find out that other there are other problems that you are putting patients at risk or even talking about the legal liability. I mean, there’s, like, you know, the all the cases that you can never quantify that you didn’t hurt people, you know, you didn’t get sued because you’re keeping people at the top of their game. It’s also the culture. You know? It creates a better culture to recruit and retain people. It decreases burnout and people checking out. So I’d say I guess I would ask maybe you can’t really ask this, but I’d be tempted to ask, what what is the dollar denomination that you put on your culture as a CEO? Yeah. And if your answer is, well, we don’t care about it, so it’s all dollars, and it’s like, well, why are you in health care? I know that’s a bit, you may be a bit, um, cynical, but whatever dollar value you put on culture, use that number and apply that to peer learning.

[00:31:34] Chris St John: I think that’s a great answer.

[00:31:36] David Larson: That’s a great Well, I hope so, but I I guess I’m a little cynical.

[00:31:40] Chris St John: It doesn’t I mean, it doesn’t feel cynical to me. Right?

[00:31:43] David Larson: It should be a great answer. That’s why we’re here, you know. And it’s like three beauty of of pure learning is actually when people just let go of their there’s so many apprehensions. Oh, we’re not gonna have score. Like, we don’t have the scoring anymore, and it’s gonna take time. And now people, you know, may, you know, slack off. It’s like the sky is gonna fall. And I’ve never the sky never falls. Right? The score based peer review programs never did any good in the first place, and they did harm. Right? So, actually, the benefit is going to be you’re going to replace the thing that was doing harm and already taking time and making people hate you because you’re pushing this. And that’s the irony. The harder you push, the more they hate you, even if you’re doing it for good intentions. Right? Just because it’s constructed to be poisonous, not intentionally, but that’s just that’s just the outcome of it. It changes that into a constructive and a collaborative relationship. So would you prefer to have a more collaborative relationship with your radiologist? Put a dollar value on that as well.

[00:32:41] Chris St John: Yeah. In fact, I think, you know, you in your AJR article on peer learning, I you you were highlighting the importance of sharing insights as well with referring clinicians too.

[00:32:51] David Larson: Mhmm. Mhmm.

[00:32:52] Chris St John: Can you talk a little bit about the peer learning discussions you’ve had and, like, whether or not is that improving interdisciplinary trust? Or, like, how are you capturing that impact?

[00:33:02] David Larson: Yeah. Yeah. Oh, it’s great. Great question. So really where it starts, I mean, the the peer learning conference, you generally don’t have, you know, people outside the group come to peer learning program because it’s all about your peers. But you often will have representatives. You know, you hopefully, you have radiologists who have relationships with each of your major referring clinician groups. Right? And so that’s where, you know, for example, actually, I’ll I’ll go back to what Neil did, Sysony Children’s. He was working carefully, you know, with the surgeons and with the the hospital leadership and, uh, nursing administration, for example, in managing PICC line. Right? So all that cross disciplinary effort, and he was going to meetings and they were coming up with, you know, design of how things should be managed and and education components and feedback. Like, all those things are happening across the whole health system. Well, now you when you’ve got a peer learning program, you’ve always got a conduit to say, we will come and take the feedback that you just gave us or take this initiative that you’re working on to improve care, and we’ll put it into our peer learning program. So you’ve automatically now you you have opened the door to invite, you know, feedback on difficult discussions that otherwise are really hard to have spontaneously are ad hoc because no one likes to come into the meeting and to say, oh, we’ve got, you know, the orthopedists are now complaining again. Right? That’s that ends up being kind of the way it feels. And it’s like, no, no, no, we’ve changed we’ve already changed the mindset. When you walk into this room and you’re in this meeting, the mindset’s already different. The mindset is we’re here to learn what we can do better. Oh, the orthopedist have something that they think we could do better? Great. Let’s talk about it. Right? Oh, yeah. The oncologist would like these measurements because when we don’t put these measurements, then it causes problems and you’ve illustrated that, you know, what what was the outcome of not having that? That makes a lot of sense. Okay. We can work on that. Right? So that’s where you end up seeing it ends up feeling organizationally like the Department of Radiology is receptive because it’s designed now to be receptive because you have a conduit for of receptivity. Right? Whereas before, maybe we can try to be receptive, but we don’t have, like, a construct to put it in.

[00:35:16] Chris St John: Yeah. Doctor Larson, with health system consolidation, remote workflows, and AI reshaping radiology, the the peer learning landscape is evolving super quick. Yep. Looking to the future, what is your vision for peer learning two point o? Do you think we could, you know, start to see a national learning framework? Where do you see things headed?

[00:35:37] David Larson: Great. Oh, that’s a great question. So I’m really excited. I’m bullish on this. Yeah. Um, if we do it right, I it’s actually probably gonna you’re gonna see it go both ways, I think. And it’s gonna be sad to see the places that that don’t and have to learn the hard way, but I think we’re gonna there are gonna be some places who get this right. So the first question is like a national peer peer learning program. A lot of people have offered that and thought about that. I you know, and I thought about that at first, but pretty quickly started to realize, you know, peer learning is mainly about your relationships. Right? It’s about the content. That’s part of it. But it’s also that, you know, again, that kind of fear that, like, this could be me, and it’s also about my peer and my colleague, and it’s somebody I know who trusts and has given me feedback on a case that that I very well may be seeing in the future. Right? So it becomes very personal when it’s a local program. And I think for the most part, these pro these programs will stay local. I think a national peer learning program only will emerge to the extent that it is reinforcing a community because really that’s what peer learning is about. It’s about community. Right? So that’s my opinion on that. I think what what we’re going to see is the integration of AI and peer learning. And I think there’s a huge opportunity for that if programs that number one, programs that have a great peer learning program in place, again, already have that conduit. And now when you can add, you know, quantitative assessment, you can add a lot more cases that are being brought to your attention, the learning can go way up. And the second thing is that I I think that in what’s going to happen again with enlightened practices going forward with AI is that it will provide the opportunity for cases to be routed to the experts more quickly Mhmm. To get second opinions, other people weigh in, you know, to have greater sophistication and greater transparency about the clinical content of what’s coming to us, about prevalence, about feedback, about, you know, if we have, you know, our positivity rate of our our polynomials cases is like zero point five percent. Let’s have a talk with the emergency physicians because, you know, we’re over ordering. Right? I think we’re gonna see a lot more a lot more information that’s coming out, and we can use that if we decide to to become better colleagues Yeah. To operate more as a system and as more as teams. Right? If we can learn to move beyond being isolated independent operators who can never be told anything, you know, and never work with anybody else to actually working together as a team, uh, I think that’s those are going to be the practices that thrive, and they’re gonna be practices that you people are gonna wanna work in if they do it well.

[00:38:15] Chris St John: Yeah. I feel like we’ve just, like, barely touched on the AI stuff this episode, which, honestly, talking about radiology and AI, I don’t mind taking a small little break from it. But since you brought it up, like, as AI tools are getting more and more prevalent, they’re increasingly flying outliers and errors. How how do you see the pure learning complementing or contrasting with, like, algorithm driven feedback?

[00:38:41] David Larson: I would say whoever is developing these anything that’s providing feedback Mhmm. You know, whether you are a radiologist or you’re, you know, a data scientist or an informaticist Yeah. Go and read the literature. Understand what a score based peer review system is and what it does, what, you know, feedback that’s not done in a really thoughtful and careful way, what that tends to do, and compare that to, you know, what you can do if you’re really thoughtful and careful and if you’re trying to reinforce the culture. Those, you know, as you build these systems, you will have an impact on the culture whether you know it or not or whether you care or not. And that culture, you know, it has antibodies as it should. It’s self protecting. Right? And so if you try to implement feedback and it’s done even, you know, even in a way that’s just slightly insensitive, Like, it’s just not really careful about, you know, protecting confidentiality, or it’s just there’s a risk that potentially, you know, it could be done in a negative way. Like, you’re gonna have, you know, significant negative impact on what you’re trying to do. So I would say, just be careful, you know, fools rush in or angels fear to tread. You know, you are working in dangerous territory. It’s human psychology and sociology that is really unwieldy. And and it’s it’s, uh, it’s understandable, mainly, if you just be a human and think about it as a human and stop thinking about radiologists as widgets and machines. If you do that, actually, you’d probably be in a lot better place. But not everyone will do that. They’ll implement these things. In some ways, they’ll do it wrong. And so then I encourage the rest of the community, radiology community, call that out, you know, to not accept this type of sloppy and, you know, unthoughtful not, whatever that word is, you know, not, you know, conceived or or or not deeply thought through design. Expect have high expectations of the people who you’re designing your systems and don’t accept, you know, sloppiness. It may the the technology may be wizardry, but it may look like it’s magic. But if it has a negative impact, it doesn’t matter. It’s not thoughtful.

[00:40:53] Chris St John: Yeah. Some very wise words there. Okay. That’s it for today. Thank you so much for joining us here on Frame by Frame. Once again, doctor Larson is the director of AI development and the evaluation lab at Stanford University School of Medicine. Doctor Larson, it has been my pleasure to have you back, and I can’t wait to talk to you again.

[00:41:15] David Larson: Awesome. Thanks so much, Chris. It’s been a real pleasure for me too.

[00:41:18] Chris St John: Thank you so much. Frame by Frame Rethink Imaging is brought to you by Imalogix. Here, you’ll find engaging interviews with thought leaders, experts, and patients sharing stories that showcase the transformative power of medical imaging. To discover how Imologix is rethinking imaging in health care, visit imologix.com. Be sure to subscribe to Frame by Frame Rethink Imaging on Apple Podcasts, Spotify, or wherever you listen. And from all of us here at Emalogix, thanks for tuning in.

Questions from This Episode

How is peer learning different from traditional peer review?
Traditional peer review scores a random sample of cases on a numerical scale and tracks misses, with remediation or removal from practice as the implied consequence. Peer learning drops the scores entirely and treats misses as material everyone learns from. It rests on three pillars: feedback, learning, and improvement.
Why do score-based peer review systems fail?
Larson saw his own score-based conferences turn into debates about the score rather than the case, with attendees writing down case numbers to identify who made the miss. That adversarial design puts people on the defensive, which he describes as adding insult to injury. Defensiveness shuts down exactly the learning the process was supposed to produce.
Does peer learning only work at large academic centers?
No. Larson argues the model adapts from small rural practices to large academic centers, with adjustments for practice size and subspecialty coverage. He also makes the case that local, relationship-based programs outperform national frameworks because the relationships between colleagues are what drive the improvement.