You can’t make an LLM only reference the data it’s summarising. Everything an LLM outputs is a collage of text and patterns from its original training data, and it’s choosing whatever piece of that data seems most likely given the existing text in its context window. If there’s not a huge corpus of training data, it won’t have a model of English and won’t know how to summarise text, and even restricting the training data to medical notes will stop mean it’s potentially going to hallucinate something from someone else’s medical notes that’s commonly associated with things in the current patient’s notes, or it’s going to potentially leave out something from the current patient’s notes that’s very rare or totally absent from its training data.
Well, I can’t claim to be an expert on the subject, at any rate, but there are plenty of models which are local-only and are required to directly reference the information they interpret. I’d assume a HIPAA-compliant model would need to be more like an airgapped NotebookLM than ChatGPT.
But I would also assume the risk of hallucinations or misinterpretations is the reason why a clinician would still need to review the AI summary to add/correct details before signing off on everything, so there’s probably still some risk. Whether that risk of error is greater or less than an overworked resident writing their notes a couple days after finishing a 12-hour shift is another question, though.
If you end up integrating LLMs in a way where it could impact patient care that’s actually pretty dangerous considering their training data includes plenty of fictional and pseudo scientific sources. That said it might be okay for medical research applications where accuracy isn’t as critical.
For what it’s worth, I don’t mean to say that this is something that hospitals and health networks should be doing per se, but that they are doing right now. I’m sure it has benefits for them, as another user somewhere further in this post described, otherwise I don’t think all these doctors would be so eager to use it.
I work for a non-profit which connects immigrants and refugees to various services, among them being healthcare. I don’t know all of the processes they use when it comes to LLM-assisted documentation, but I’d like to think they have some protocols in place to help preserve accuracy. If they don’t, that’s why this is on our radar, but so is malpractice in general (which is thankfully rare here, but it does happen).
You can’t make an LLM only reference the data it’s summarising. Everything an LLM outputs is a collage of text and patterns from its original training data, and it’s choosing whatever piece of that data seems most likely given the existing text in its context window. If there’s not a huge corpus of training data, it won’t have a model of English and won’t know how to summarise text, and even restricting the training data to medical notes will stop mean it’s potentially going to hallucinate something from someone else’s medical notes that’s commonly associated with things in the current patient’s notes, or it’s going to potentially leave out something from the current patient’s notes that’s very rare or totally absent from its training data.
Well, I can’t claim to be an expert on the subject, at any rate, but there are plenty of models which are local-only and are required to directly reference the information they interpret. I’d assume a HIPAA-compliant model would need to be more like an airgapped NotebookLM than ChatGPT.
But I would also assume the risk of hallucinations or misinterpretations is the reason why a clinician would still need to review the AI summary to add/correct details before signing off on everything, so there’s probably still some risk. Whether that risk of error is greater or less than an overworked resident writing their notes a couple days after finishing a 12-hour shift is another question, though.
If you end up integrating LLMs in a way where it could impact patient care that’s actually pretty dangerous considering their training data includes plenty of fictional and pseudo scientific sources. That said it might be okay for medical research applications where accuracy isn’t as critical.
For what it’s worth, I don’t mean to say that this is something that hospitals and health networks should be doing per se, but that they are doing right now. I’m sure it has benefits for them, as another user somewhere further in this post described, otherwise I don’t think all these doctors would be so eager to use it.
I work for a non-profit which connects immigrants and refugees to various services, among them being healthcare. I don’t know all of the processes they use when it comes to LLM-assisted documentation, but I’d like to think they have some protocols in place to help preserve accuracy. If they don’t, that’s why this is on our radar, but so is malpractice in general (which is thankfully rare here, but it does happen).