Ministry of Social Development commissioned the University of Auckland to consider how predictive modelling could be used to target early intervention to reduce the risk of child abuse and neglect, and improve outcomes for children and young people.
The University of Auckland’s research unit developed a predictive risk model for children in a cohort who had contact with the benefit system before age two. These children accounted for 83% of all children in for whom findings of substantiated maltreatment were recorded by age 5.
This research indicates that predicting risk modelling had a fair, approaching good, power in predicting which of the young children having contact with the benefit system would be the subject of substantiated maltreatment by age five. This is similar to the predictive strength of mammograms for detecting breast cancer in the general population.This report provides further detail on the predictive risk modelling system outlined in the White Paper for Vulnerable Children. The research indicates that bringing together administrative data can significantly improve the identification of at-risk children by linking Ministry of Social Development (MSD) benefit, care and protection and youth justice data.
Purpose
Correctly assessing the likelihood that a child will have a maltreatment finding at some future time enables scarce child protection and early intervention preventive resources to be strategically targeted. It means that a suite of appropriate programmes of varying intensity can be offered to children and caregivers at all levels of risk.
Key Results
- Predictive Risk Modelling (PRM) is an automated algorithm which harvests data from a variety of sources. In this analysis, we use PRM to generate a risk score for the probability of a maltreatment finding for each child at the start of any main welfare benefit spell involving the child.
- A maltreatment finding is defined as a substantiated finding of emotional, physical or sexual abuse or neglect by age 5.
- We consider a "core algorithm", applying it to children under age 2 and predicting the risk of a maltreatment finding. Although 5.4% of all New Zealand children have a finding of maltreatment by age 5, the rate is substantially higher for children seen on a main welfare benefit (13%) than for children never seen on a benefit by age 2 (1.4%).
- Of all children having a finding of maltreatment by age 5, 83% are seen on a benefit before age 2, translating into a very high "capture" rate.
- The national prevalence rate of maltreatment finding for under 5 year olds in New Zealand is more than 20 times the risk of breast cancer in women aged 50 to 60 for whom routine screening is offered.
- Performance of Predictive Risk Models is usually summarised by the percentage area under the Receiver Operator Characteristic (ROC) curve. A model with 100% area under the ROC curve is said to have perfect fit. The core algorithm applied to children under age 2 has fair, approaching good, strength in predicting maltreatment by age 5 with an area under the ROC curve of 76%. This is similar to the predictive strength of mammograms for detecting breast cancer in the general population.
- The most at risk children identified by the model represent 37% of all children in New Zealand who will have a maltreatment finding by age 5. This group comprises 5% of children overall.
- The highest risk group of children identified are almost 30 times more likely to have a maltreatment finding than the lowest risk group identified.
- If children with the 20% most risky benefit spells are offered a programme which can reduce maltreatment by 10%, we would need 27 families to take up the programme in order to avoid 1 child having a maltreatment finding. This is called the Numbers Needed to Treat (NNT).
- If the 20% most risky benefit spells are offered a programme such as the Nurse Family Partnership which has been shown in overseas studies to reduce maltreatment rates by 46%, we estimate that it would cost $48,000 per maltreatment finding avoided.
- If programme participation begins as soon as children start a spell in the top 20% most risky spells, only a small fraction will have a maltreatment finding within the first year. Therefore, the PRM identifies risk early enough for prevention services to be effective.
- A full ethical evaluation of PRM is necessary before implementation. Additionally, an ethical framework should be developed to guide agencies in their responses to the use of automated child risk scores. Preliminary ethical analysis suggests that mandatory policies for high risk families need to be treated extremely cautiously; we anticipate far fewer ethical concerns if scores are used to engage high risk families in voluntary services.
- Operationalising the PRM requires strong engagement with providers and frontline staff. Therefore, a careful, deliberate and phased implementation (following the ethical evaluation) is necessary.