AI scribes - a checklist of things to consider

Use this checklist to consider your need for an AI scribe, how you will use it and the risks it may present. If you are unsure about the answer to any of the questions, contact the AI scribe provider(s) and ask them.

If you are not comfortable with the answers provided, you may wish to explore other options. You can also contact our Medico-legal Advisory Service or Risk Advisory Service for advice.

Avant does not endorse or recommend any particular AI scribe.

Generative AI large language models such as ChatGPT were developed for general use rather than for medical purposes whereas AI medical scribes have been developed for use in a clinical context.  Before you start using an AI scribe, consider these questions:

• Was the AI designed for a medical purpose?
  This is the most important question to ask. We recommend you do not use AI in a clinical context if it was not developed for a medical purpose or for use in healthcare.
• Can you test the AI scribe’s outputs to ensure that they are accurate, fit for purpose and make clinical sense?
  AI scribes are not subject to regulatory oversight so you need to satisfy yourself that the AI scribe you are considering is helpful, appropriate and safe to use in the clinical context in which you work

Obligations under privacy legislation will apply depending on how the AI manages identifiable patient data. You should be able to find the answers to the following questions in the AI provider’s privacy notice and privacy information on their website. If you can’t find the answers, ask.

Compliance with Australian privacy law

Look for a statement on the AI provider’s website or marketing material about compliance with Australian privacy law. Also check if the AI provider has a privacy notice or privacy information you can provide to patients.

Some of the questions you should consider when determining if there is compliance with privacy laws are:

• Is any data collected or retained by the AI provider?          
• Is data identifiable or de-identified/redacted?
• If data is retained:
  • is it encrypted?
  • can you access it?
  • how long is it held?
  • is it stored on servers within Australia or overseas?
    If data is sent and stored overseas, you may have additional privacy obligations (see the OAIC’s APP 8 Cross-border disclosure of personal information).
• Are there any proposed secondary uses of the data?
  Find out if the AI provider will use the data for any other purposes, such as for research, education or machine learning. Can any third parties access the data for these or other purposes? Specific patient consent may be required for the AI provider to use data for other purposes. 
• What cyber security measures are in place?
• If there is a system failure, can records or data be recovered if necessary?
• What will the AI provider do if there is a data breach or cyber security incident involving the AI scribe? Will they notify you and give you information on how you can respond, including how to disengage from the system?
  Given the prevalence of cyber security breaches, check what safeguards the AI provider has in place.  

For doctors and patients to trust AI, they need to know that it’s being used and how. If you decide to use an AI scribe, make sure you and your staff have a sound understanding about how it works, so you can explain it to your patients.

Consider these questions:

• Does the AI provider have information you can give to patients about how the AI scribe works?
• Does the AI scribe incorporate a prompt to ensure consent is obtained at each consultation?
• If asked by a patient, or a regulator, can you and your staff explain why you are using the AI scribe in your practice?

AI service agreements can outline a range of terms and conditions, features, and levels of service, and costs may vary. Make sure you are comfortable with the terms of any agreement before you sign up. Check what you will get, for what price and the time period.

Consider these questions:

• Are there any disclaimers, warranties and/or indemnity clauses shifting responsibility to the user of the tool and absolving the developer of responsibility?  
  The law does not require an indemnity clause to be included in a software services contract. Agreeing to a broad indemnity clause may have implications for your Avant insurance policy. Your Avant Practitioner Indemnity Insurance Policy will not cover liability you take on under a contract if you take on liability beyond the general law. If the provider’s terms and conditions include an indemnity clause, consider seeking advice before agreeing to it.
• Is the service agreement adaptable if there is a change in the technology (such as if another product is developed that is more appropriate)?
• How will the AI be updated and are updates automatic and included in the terms and cost of the agreement?
  Make sure the AI scribe will be suitable for the long term and find out how the AI provider plans to maintain it.
• How much notice must you give to terminate the agreement and are there any penalties?
• How do you report glitches found when using the AI scribe and how will these be managed by the AI provider? 

If you decide to start using an AI scribe, make sure you and your colleagues are trained in its use. The AI provider may offer practice sessions as part of available training.

It’s important the AI scribe assists you and operates as intended.  If it doesn’t, have a plan for how you will manage a glitch, system failure or how to stop using the scribe altogether. Consider these questions in your practice:

• When will you check the clinical notes, referrals and other outputs from the AI scribe?
  Anything produced by the AI scribe should be considered a draft. Always review and check any output for accuracy during, or as soon as possible after, the consultation, before including it in the medical record, sending it to a third party or using it for Medicare billing purposes. See our factsheet Artificial intelligence for medical documentation
• What checks are in place to ensure that the AI scribe is operating as intended?
  Plan to regularly review the tool and have a way to report and manage unintended outcomes or other performance issues. AI based on machine learning improves over time by learning from new data but it also has the potential to change in a way that was unintended or that degrades performance.
• What will you do if there is an unintended outcome or a system failure?
  Make sure you have a plan for how to discontinue use of the tool including a timeframe for doing this. Your business continuity plan should describe how you would work without the AI scribe.  This may also apply if a patient does not consent to use of the AI scribe.
• Can you access historical data and information about how the AI worked at a particular point in time?
  You may need to access this information if there is a system failure, or if there is a later claim or complaint about your use of AI.

Finally, make sure you:

• update your privacy policy and information to include information about the use of the AI scribe in your practice
• update your cyber security policy to include breaches or suspected breaches involving the AI scribe
• appoint a person in your practice to be responsible for monitoring use of the AI scribe and ensuring all updates have been actioned.  

Avant factsheet - An introduction to artificial intelligence

Avant factsheet - Artificial Intelligence for medical documentation

For medico-legal advice, please contact us on nca@avant.org.au or call 1800 128 268, 24/7 in emergencies.