AI scribes in practice: common errors to consider

During the webinar, we asked attendees to share examples of when an AI scribe has produced errors or inconsistencies. Many said errors were “frequent” and inconsistencies related to the names of people, places, and medications. Some found that important details were noted incorrectly or omitted. Others found that the AI scribe “hallucinated” and notes contained information that was not discussed in the consultation. 

Below are some errors our attendees have experienced: 

• “I asked if there was chest pain. The answer was no but the AI scribe wrote yes.” 
• “Mine is good at eliminating small talk but sometimes fails to pay attention to social history.” 
• “Wrong side recorded by scribe for the breast cancer.” 
• “Olmesartan instead of irbesartan.” 
• “I have seen a key error (missing the word ‘not’ as in has epilepsy vs does not have epilepsy) in a letter that noted at the bottom ‘dictated not sighted’, which was of concern.” 
• “I have noticed that it cannot always tell the difference between voices and so at times the tool has thought the patient is going on holiday or moving house when in fact that is me sharing information as to why I won’t be at work next week, for example.” 
• “Mishearing and ‘hallucinations’ occur and need careful checking. (But so did my dictated reports in the past.)” 
• “During clinical teaching sessions, I have observed that the AI stated the age of the patient was 21 years old instead of 12 years. Fortunately, the Registrar noticed it and corrected the note prior to finalising it.” 
• “Family history noted as ‘Father happy with progress’.” 
• “[An AI tool] captured ‘Sweatshop’ instead of ‘Abattoir’ for a workplace this week.” 
• “Opioids to a paediatrics case.” 
• “Transcribing oral contraceptive pill for a male patient.” 
• “Worryingly, one AI interaction made up a whole neurological exam when I didn’t perform one. Probably based on a statistical prediction given one to two neuro findings I described.” 

Many attendees noted that they are used to checking and correcting their AI-generated notes before processing the record, as they cannot – and should not – trust the accuracy of a scribe. 

“The key thing to remember with an AI scribe is that it is a tool in the arsenal, and a very good assistant, but not a replacement for our clinical reasoning and thus will have limitations.”

 – Dr Vicky Phan, Medical & Risk Adviser, Avant 

AI tools can contain diagnostic errors and biases, so you may consider how you can adapt your consultation style to capture physical examinations and articulate clinical reasoning. 

Tips for improving the output of AI scribes: 

• Recognise and understand the limitations of the tool you are using. It may not decide what is important or pick up on nuanced conversation, for example. 
• Verbalise your findings out loud. 
• Explain to a patient that you are going to dictate using medical jargon and will explain to them afterwards. 
• Speak clearly, explain your clinical reasoning and repeat information when necessary. 
• Capture both verbal and non-verbal cues (for example, a patient may look pale and unwell or have a limp, and these important visual findings and relevant ‘negatives’ should be recorded when completing the note). 

“Always check and correct each and every note before you finalise it. And do it within the tool if it is not integrated into your software. This will also help the AI tool correct spelling errors and adapt to your style of notetaking.” 

– Dr Vicky Phan, Medical & Risk Adviser, Avant 

Webinar attendees shared the ways they are addressing challenges with AI scribes: 

• “I have used medical students and RMOs as ‘real intelligence’ scribes and they have made mistakes like this quite often. They need to be checked, and AI is no different.” 
• “I give patients a copy of my letters and then, at the next consultation, ask if there are any concerns or errors.” 
• “Verbalise your exam. Keep your thinking visible. Synthesise your findings and verbalise these.” 
• “Whenever I make a mistake, I contact the patient, apologise and write or talk to the patient.” 

There are also advantages of using a scribe tool. For example, the detail may exceed what you write manually. One member commented that the scribe sometimes captures information from the patient that they had missed writing down. 

“Don’t let AI scribes dumb you down. Make sure you still know how to do a proper clinical record yourself – be alert every single time.”

 – Dr Patrick Clancy, Senior Medical Adviser, Avant 

AI scribes can reduce the administrative burden and improve efficiency, but they should not be relied upon to produce accurate notes. It is your responsibility to file correct medical records, regardless of whether you use an AI scribe or not. We advise always checking and correcting notes before finalising them and ensuring they are an accurate representation of your consultation. 

"There are many benefits of AI tools in reducing administrative and cognitive load. But I would encourage you to use the extra free time in your diary to have lunch or go home on time to your family, rather than seeing more patients.” 

– Dr Vicky Phan, Medical & Risk Adviser, Avant 

Avant members can watch our ‘AI scribes: cascading consequences’ webinar on-demand. We also offer a Measuring Outcomes activity for this webinar for 0.5 CPD hours. 

Avant factsheet: An introduction to artificial intelligence  

Avant factsheet: Artificial intelligence for medical documentation 

Avant factsheet: AI scribes – a checklist of things to consider 

For medico-legal advice, please contact us here, or call 1800 128 268, 24/7 in emergencies.