An evidence problem for family justice: social work use of transcription AI, harmful hallucinations and the deletion of original audio

Social worker use of transcription AI may have led to some efficiencies for social workers, but it is impacting the integrity of written documents, with significant risks downstream for family justice.

Recent Ada Lovelace Institute research has identified harmful hallucinations from these technologies in real-world social work cases, including one AI-generated summary which stated a person had expressed suicidal ideation when they had not.

If a family court were to receive such a hallucination in social work evidence, they would be hamstrung to determine it to be as such, since these tools delete the original audio in 30 days as standard, thereby losing an authoritative piece of evidence by which the court could determine what is authentic and what is hallucinated.

JUSTICE is advocating for a systems-approach to arriving at agreed standards for AI tools which will impact the fairness of court proceedings, rather than these standards being decided unilaterally by the commercial vendors. This includes data and audio preservation and auditability standards. This work has started in criminal justice by police and the CPS: their joint operating procedure highlights the importance of retaining original audio when AI is used for transcription in a criminal context.

JUSTICE recommends the right standards for family proceedings – which take into account the need to preserve audio for the purpose of authenticating AI-generated transcripts, but also consider data limitation and practical storage capacity constraints – should be determined through system-wide discussions, including the perspectives of the judiciary, social workers, people affected by care processes, and regulators who establish standards for social work.

Background

There has been rapid expansion of the use of transcription AI across public services in the UK in the last year or so, including by social workers: in early 2025, one AI transcription tool – Magic Notes from private company, Beam – was already in active use by 85 local authorities for social care.

Transcription AI is distinct from more traditional methods of automated transcription due to its incorporation of generative AI in the form of large language models (LLMs). LLMs help fill in gaps and infer content when audio quality is poor or contains long silences. However, by doing so, modern AI transcription tools have integrated a hallucination risk as a design choice to improve performance: there is an increased likelihood of plausible text output because of the LLM, but also an increased likelihood of fabrications.

The risk of AI hallucination

The rapid adoption of these technologies in social work has not been matched with greatly increased research and evaluation. To help plug the gap, the Ada Lovelace Institute conducted qualitative research, published in February 2026, based on conversations with frontline social work practitioners using AI to transcribe and summarise their interactions.

The research found the tools can deliver efficiencies, however it also identified a risk of harmful hallucinations and misrepresentations of people’s experiences being entered into statutory care records. For example, one social worker noted, after an interaction with a client, that a Magic Notes summary incorrectly “indicated that there was suicidal ideation”, but “at no point did the client actually, you know, talk about suicidal ideation or planning, or anything”.

The research also highlighted inconsistency in safeguards, challenges to existing accountability models, and an absence of impact evaluation and monitoring beyond efficiency, for example there is very little evidence on how AI transcription tools impact the experiences and outcomes of people who draw on care.

JUSTICE’s concerns and recommendations

Along with recommendations for better research, JUSTICE also agrees with the need for clarity and guidance on using these tools in statutory processes and formal proceedings.

In producing such guidance, a systems-approach is needed, which considers the downstream impact of AI-transcribed records coming into evidence in statutory proceedings.

It is clear that no AI transcription tool will be 100% accurate, and there are good arguments that this should not be a bar to use, since of course human notetakers are not failsafe.

JUSTICE therefore suggests the question is: if harmful mistakes do make it into care records or care assessments which are then submitted into court proceedings, what safeguards are needed to ensure the court can do its job and ensure fair and safe proceedings?

The court process should enable fair challenge to the local authority evidence, including the accuracy of any transcription in evidence.

Applying the best evidence principle, and some common sense, JUSTICE observes the courts will be best placed to resolve any factual dispute between parties over the accuracy of any AI transcribed documentation by hearing the original audio recordings. These will simply elucidate if the transcribed notes are accurate or not. E.g. in the example in the research report above, if suicidal ideation was mentioned or if it was hallucinated by the AI tool.  

The court’s ability to access best evidence in exercising its fact-finding role however will be hampered if that original audio does not exist. Surprisingly, JUSTICE can confirm that Magic Notes is designed to delete all original audio within 30 days.

This is potentially a fatal flaw when it comes to safeguards since it undermines the court’s function – to allow for fair challenge and the ability for inaccurate evidence to be rectified where it matters most, when what is said may have legal consequences. For example, if suicidal ideation is relied upon as evidence of threshold being met in a care case, the absence of an audio recording undermines the court's ability to do its job in the event of a dispute, ie the parent saying "I didn't say that”.

There is also a risk of the opposite happening, namely that it becomes beneficial for litigants to make spurious allegations of inaccuracy of care records, in the knowledge that no authoritative audio recording exists (known as the liar’s dividend).  

Notably, not retaining audio is at odds with what has been agreed as best practice in the criminal justice context. The CPS and National Police Chiefs’ Council have agreed a Joint Operating Procedure (JOP), which highlights the importance of retaining source information, and inputs (i.e. audio recordings).

Next steps

JUSTICE voiced the above concerns in a workshop hosted by the Ada Lovelace Institute in June. Some participants agreed, whilst others raised various problems with retaining audio recordings for AI transcriptions, including data protection concerns and concerns about capacity and security of data storage solutions. JUSTICE does not consider any of these problems nullify our concerns, nor its identified solution, however they must of course be taken into account.  

It is unrealistic to expect any design change in these tools to be instigated by the vendors. Beam makes Magic Notes for the market, and any change in design will be a response to the demands of their customers. Those customers – social workers – need an agreed set of standards for the use of AI transcription in social work which reflects system-wide perspectives of what safeguards are needed, including safeguards for the courts to do their job.

JUSTICE suggests this work is urgently required. System-wide discussions should take place to identify these standards, and JUSTICE of course recommends that audio retention is considered as a critical piece of the puzzle. Such discussions should incorporate the perspectives of the judiciary, people affected by care processes, and regulators who establish standards for social work.