BrainGPT: Advancing neuroscientific research with AI

[Nicky Cartridge] I’m your host, Nicky Cartridge, and I’m delighted to welcome our guest, Professor Bradley Love, to talk all about BrainGPT, a large language model tool to assist with neuroscientific research.

Brad, welcome to Visionary Voices.

[Bradley Love] Great. Yeah. Thanks so much, Nikki, for having me here. I’m looking forward to it.

[Nicky Cartridge] We are delighted to have you. AI is such a hot topic at the moment. I’m really excited to hear about BrainGPT and how it might impact research in the future. But I wondered if we could start off by talking a little bit about your own journey into AI and deep learning research.

[Bradley Love] Yeah. Sure. So, historically, I’m not much of a tools person. I’ve always been more on the the the theory side, but I got this sense, coming from doing research in neuroscience, in particular in cognitive neuroscience and computational neuroscience, that science wasn’t really working in a sense that I’m sure many of your listeners have this feeling of being overwhelmed by the scientific literature.

And they should feel that way because the scientific literature is growing exponentially.

I just had the sense it wasn’t working, that scientists weren’t being able to integrate all te the relevant findings. And it just wasn’t really a human readable literature anymore. And important insights were being lost. And science wasn’t progressing as efficiently as it could. And so, I mean, it seems really obvious, like you mentioned, giving the success of these large language models and synthesizing huge texts to start thinking about how can we use them to do science better.

[Nicky Cartridge] Amazing. So, I mean, you’ve obviously been working in this area for such a long time.

How does it feel that large language models are becoming general use amongst the general public and, so widely used now.

[Bradley Love] Yeah. I mean, I think it’s even surprised some people that are experts working in that area just as the scale of these models as they’re trained on larger and larger training sets, larger and larger collections of text, and they have more and more parameters. I mean, it used to be if a model had a hundred million parameters, that was a large language model. Now there’s models with trillions of parameters.

And larger doesn’t necessarily mean better in most things, but it seems like in this arena, the scaling laws still haven’t tapped out. So you’re still seeing, some improvements. So it’s exciting. You know, it just, it just seems even as the underlying technologies aren’t radically changing, it feels like there’s new capabilities and potential applications, including in science and health that are, coming online.

[Nicky Cartridge] Yeah. It’s really exciting.

I remember attending an ophthalmology conference. Oh my gosh. It must have been about seven or eight years ago at least now. And machine learning and retinal imaging was a really big topic. And I remember at the time thinking, like, this is a bit mind blowing. But to see it kind of just be implemented in all different all different industries, but just thinking about medicine especially, it’s exciting. There’s always something new happening.

[Bradley Love] Yeah. No. I agree. I may also feel like, a bit privileged to have been around longer than some of my younger colleagues because this pace of progress isn’t something that’s always happened, you know? So I was, you know, doing models of cognition, decision making, and brain function all the way back in like the late nineties in graduate school. And I would not have predicted that we’d have vision models that work today and speech perception models, large language models that could do like limited forms of kind of common sense reasoning.

And or like this project we did trying to predict neuroscience results from training on scientific texts. Like, yeah, so it’s just, I guess I’m amazed because I just remember being younger and thinking this wasn’t going to be possible. And usually it’s the other way. Usually we think there’s going to be all these amazing things in the future.

We’re going to have flying cars and fusion power and live forever, whatever, you know, it is and be on Mars. And, it just never happens as soon as we think it is. And I think this is like a case where if you went back twenty five years, you wouldn’t think then that it would be like it is now. So I find it exciting and invigorating.

[Nicky Cartridge] Yeah. That’s really exciting. So I suppose I’m just gonna jump in with the big question. So what is BraiGPT? And could you tell us a bit about how it works?

[Bradley Love] Yeah. Sure. Sure. It’s actually not that fancy or complex. So we could say a little bit how these large language models work very, very generally.

So, they’re really trained in – it’s called a self supervised way. So they don’t really have a teacher. They supervise themselves.

And the most common way they’re trained is just to predict the next word. So or the next it could be anything. The next token, they call it. But, say if it’s a journal article, it could literally just be, like, the amygdala has the function.

Just trying to predict every word from the context, the previous words. And by doing that, it turns out just by trying to predict the next thing that’s coming, the model builds some kind of internal understanding of the scientific literature. Because it turns out it’s just easier to predict the next thing’s coming if you kind of understand, you know, what the text is about. So it’s kind of a neat case because it’s something you using a very simple training process that’s able to kind of extract the general patterns that are, like, underlying, you know, these billions of lines of texts and the scientific literature.

And people might say, well, that’s not really understanding. And you could have reasonable debates about that. But I mean, it’s sort of there’s a lot of things that seem like relatively simple processes. Natural selection is pretty simple, but look at all the unexpected things it leads to.

And so I kind of see this relatively simple training to predict the next word could lead to distilling the basic patterns that underlie science. So anyway, we in the BrainGPT project, we use both off the shelf models that large corporations and groups have trained and that also augmented them in ways. And so the important thing that we did is that we didn’t use commercial offerings that are a bit closed off. So many of your listeners will know about OpenAI’s chat GPT type models or maybe anthropics cloud models.

And those are amazing models. And I use them myself.

But, you don’t have access to the underlying weights like in the neural network that the model has been trained on. So when I said predict the next words, there’s literally billions of little weights or connections that are being adjusted to do that prediction. And that’s sort of the guts of these models.

And, there’s a lot of models out there, like, Meta has a model, named Llama that’s freely available and really good. And, what’s good about that from a scientific perspective is I’ll tell you explain shortly how we use these models, but you could also have many listeners worry about reproducibility, how rigorous is the work. And so everything we’ve done is kind of open and things that, you know, people could build off of or use such as, you know, practitioners or other scientists. So anyway, what we did with these models is we used them to calculate how likely a pattern of text would be.

So like if you did this method, how likely is this result? And basically, does it go with the pattern of the larger scientific literature? And so we assessed how well these large language models could differentiate between patterns of results that actually occurred in an experiment, in a neuroscience experiment, versus ones that didn’t. So it could be as simple as implicating, say, like, the interior versus posterior hippocampus or something.

So pretty, pretty subtle differences.

And we compared the models’ abilities to neuroscience professors’ abilities to do the same thing. And I guess what surprised us was that these models were in the mid eighties of accuracy, like eighty five percent, the better ones.

Whereas the humans, the professors, experts, they were above chance, but they’re more like at sixty three percent. And so it’s just kind of it’s not saying that human experts aren’t truly experts and there’s more aspects of expertise than prediction, but it’s showing that the scientific literature is immense.

And I don’t think people could draw on all the nuances of it, whereas the models can, you know, maybe a scientist reads a certain author’s work or they’re not reading in other areas, whereas these models are pulling across everything. And I think it shows they’re distilling this pattern. It’s also showing that the scientific literature is somewhat redundant because these models are just basically preferring things that follow the pattern, which means there was something in their training set that conformed to this new finding. And so it’s maybe a little disconcerting for scientists to think that most of what’s published, you know, maybe it didn’t even have to be because there’s all kinds of precursors that would strongly indicate, you know, what the scientists observed. You know, it just had to be that way based on, you know, thousands of things that came before. But I don’t think that people have that ability to see these subtle patterns.

[Nicky Cartridge] Could you tell us a little bit about, I guess, for our audience, they’d be interested to think how they might be able to apply this to their future research. Could you give us a little bit about the applications and how it could be used?

[Bradley Love] Yeah.. So I think this is gonna open up all kinds of applications. Because one thing that was really important, that I didn’t touch on, is that the predictions from the models were well calibrated. Just meaning the more there’s you could get a measure of confidence from the model and its decision just like you can from a person.

And it turns out the models are calibrated. So the more confident they were in their decision, the more likely it was to be correct.

So this is good because you need to be able to work with something, team with it, trust it. But it also is gonna open up the door to all kinds of tools, like, you know, so you could maybe use systems like this to assess how replicable, finding is. So, you know, it’s very costly and time consuming to redo a whole study. But you could say, okay, given these methods and these possible patterns of results, which one is the most likely?

And, get a sense of the likelihood of replication. You could also figure out, in the future, make tools with this to figure out how to tune your experiment design to make it the most powerful, the most efficient, the most likely to observe your results. So I kind of see these models that pretty soon they’ll be, you know, serving as assistants to generate new knowledge by making predictions, running experiments. So right now, there’s a lot of commercial packages available to summarize, provide instant reviews of things.

But I guess I’ve been more excited about what we’re working on and what’s coming, which is more to predict, anticipate what’s going to happen, and help direct scientists in how they design their experiments. So everything we’ve done is freely available to share. But we need to package it up, frankly, in a way that non experts could easily use. And so that’s something we’re working on now.

But I mean, if it’s not us, somebody else is going to be offering this very soon. So I think, something for listeners to get ready for in the near future, like not years from now, but months from now. Oh, I should I forgot to mention too that we actually did train these models additionally on neuroscience findings, and they could get better yet. So you could take these models off the shelf that are generic.

It turns out they’ve been trained on a lot of scientific literature, including bio archive, Wikipedia. You know? So are the journal articles too. So they have a lot of knowledge, but, you know, it might be nice to augment them with more.

And so we train them on twenty additional years of neuroscience literature from popular, you know, respect respected journals. And the models got a little bit better, but actually only a few percent better. And they’re already better than people. Like, they’ve already had so much knowledge in them.

But, you know, we just thought, well, we should just give them yet more and yet more let them do, better yet. So that just even opens up other possibilities. If there’s some fields that are not, like, maybe as visible in the training regimen of these models, it is possible to augment the models to, you know, give them that knowledge. If someone’s in a fairly specialist field, that might not have been sufficiently covered in the training of the model.

And if your listeners are interested in the work we’ve done in this project that I’ve been discussing, there’s going to be a paper coming out in Nature Human Behavior this Wednesday, 27th November, in which we do this demonstration of these large language models using open weights. So these are models that are accessible to anybody. You could inspect, and you could replicate what we’ve done. We’ve shown that these models are much more effective than human experts in forecasting neuroscience results.

And again, the difference between the right and the wrong answer and our benchmark we made to measure the ability of models and human experts to forecast neuroscience results are fairly subtle and cover all areas of neuroscience. So everything is very high level from behavior, cognitive type phenomena to the down to the cellular molecular, you know, DNA pretty much level. So that’s the one nice thing about working in neuroscience is that it’s a pretty wide ranging field. But I wanna highlight everything we report in that paper, while our benchmark is in neuroscience, our experts are neuroscience.

There’s nothing particular to neuroscience in how we train the model, the methods we use. So, you know, if someone were in chemistry and physics or any knowledge intensive field, this could be rolled out there, just as well, which I find exciting because I think it’s just gonna make it more likely this kind of work’s gonna take off. But yeah, so in that paper, we showed not just that, these models surpass human experts, but they do so kind of in a sensible way. Like I mentioned that their confidence is calibrated.

So I think this is gonna be really important because in other work we showed that you could team a human and one of these large language models together to form a team that’s more effective in either one alone. And that might seem counterintuitive because it’s like, well, if these models are better, why don’t we use them and just forget about people? But it turns out the models and people make very different kinds of mistakes.

And so, people’s confidence, maybe surprisingly, is calibrated as well. Like, when they’re more confident, they’re more likely to be correct. So there’s methods we come up with where you could combine the two weighted by their confidence and their decision, and it does better yet. So I think there’s still a future for Wow. In the overall system.

Trying to think what else we went through in the paper. But yeah, I think those are the highlights. It’s a pretty short paper. So, yeah, I mean, if people are interested, please take a look at, Nature Human Behavior on the twenty seventh of November.

[Nicky Cartridge] Yeah. Amazing. I will include a link below our video as well so people can find it if they want to learn more. It’s amazing, and it just feels like the, like, the opportune opportunities with it are kind of endless, really.

But I suppose just to put you on the spot. Sure. What are your predictions over, say, the next five years? Where do you think this will lead, and where do you think we’ll be?

[Bradley Love] Oh, yeah. I mean, I mean, I think it’s funny because it just seems foolhardy to try to make a prediction in five years in the future. And that that wasn’t, again, always true. I’m not used to things progressing this quickly.

And it seems like the pace of progress is picking up. And you might hear, you know, people maybe less optimistic talking about like another AI winter, which has happened multiple times in the last fifty years where, like, it looked like something was promising and then it just fell apart. I guess the difference this time is that, like, the models are already better at people at some things, you know, it’s like, if they’re falling apart, we’re falling apart too. So I think even of these models because this is one prediction.

I think the models are gonna get better and more capable across the board, And some problems will always remain like, so you, but be reduced. Like, so your listeners might have come across that these models, you know, get accused of hallucinating, which is basically just making up facts. And that is completely true that they do this and it’s because they just tell general patterns.

And so sometimes the general pattern doesn’t contain something that actually happened in reality. Like, so a lot of times you have to get a hallucination, like, a paper title for that was for a paper that was never written, but it was like something the author would have written, you know? It’s like so you get it follows the pattern.

Yeah. So I think that’s gonna get diminished, those kinds of problems, but they’ll still be lurking.

The way we got around it was by focusing on the future and prediction because there really isn’t hallucination because it’s all about conforming to the pattern. And I think that’s why the models work so well in the case we use them for. So I think you’ll see more things on on that end whereas most of the efforts now are in kind of looking back to summarize facts and retrieve things which is good but I think in a weird way what we’re doing plays more of the strengths of the model the models are going to progress and their shortcomings are going to diminish. But even if they don’t, I just think the capabilities of the models are such that we’re gonna see really exciting tools and products, for scientists and medical practitioners that are gonna become available because like there’s already, the models are already good enough to do interesting things.

And I think it’s like a lot of things like, you know, when the internet came on, it wasn’t like we don’t, we didn’t use it like we use it now. It took over a decade before it really infiltrated our lives and changed how we do things. And so I think a lot of technology would like that. So I think the basic capabilities are there now.

People are already using them and they’re gonna get better and better. But I just think even if things didn’t get better, it’s like we’re already basically a revolution’s already locked in in how we do science and just basically anything knowledge intensive. And I mean, that’s why I’ve changed my emphasis. Like I said, I’m historically not a tools person.

I’m like into the individual scientists being creative and connecting the dots. And this isn’t really the world I necessarily expected or wanted even. But it’s just, I think it’s just reality. This is what’s happening.

And, we’re gonna. I don’t think humans are gonna be doing these kinds of things alone in the future in five years, which is your time horizon. I don’t think there’s any way. So might as well start working on it now and trying to make it better and, be a part of it because I just think this is going to be all our reality in five years. Yeah.

[Nicky Cartridge] Yeah. Absolutely. It certainly feels that way. Thank you, Brad, for that excellent overview, and I’m so excited to see what happens in the future with BRAIN GPT and beyond. So that just leads me to thank our audience for listening today. We welcome any questions you have about today’s episode. You can find us on LinkedIn by searching InTouch Medical Media, and don’t forget to subscribe to our podcast by visiting Podbean and searching visionary voices insights for health care professionals.

Thank you, Brad, and thank you everyone. Goodbye for now.