Purpose
Wave 4 Report:
Pae Whakatupuranga I Functional Family Therapy - Cross Generations (FFT-CG) is a pilot programme aimed at breaking the intergenerational cycle of justice involvement for rangatahi/young people and improving wellbeing for them and their whānau/aiga/families.
This happens through the facilitation of positive change in family systems. This programme is an adaptation of the original Functional Family Therapy (FFT) model (referred to as FFT Standard), which is designed and owned by US based FFT-LLC, but has been adapted in order to be culturally appropriate for the Aotearoa context.
This evaluation report is the final evaluation of the pilot programme after the two initial formative evaluations and the first impact evaluation. It constitutes wave four and the second impact evaluation.
The primary evaluation question for this impact evaluation is: To what extent has the programme achieved its wellbeing and pilot infrastructure outcomes, by mid-2022?
Six sub-questions consider specific dimensions of progress towards the outcomes:
- How well did the programme achieve outcomes for Māori and Pacific Peoples, including the outcomes they wanted to achieve for their whānau/family through FFT-CG?
- What helped or hindered the programme from achieving its outcomes?
- What are the unintended consequences of the programme?
- What has been learned that can improve the effectiveness of the programme in the future?
- How have COVID-19 Alert Levels affected the delivery and uptake of the programme, and the programme’s intended outcomes?
- What lessons have been learned about making the programme sustainable?
Methodology
Wave 4 Report:
This evaluation employed a mixed-method approach of both qualitative and quantitative analyses. The qualitative analysis draws on 20 interviews in total, ten of which were with youth, whānau, aiga and families who have used Pae Whakatupuranga I FFT-CG services, and two were with therapists in focus groups. The remaining interviews consisted of: the Kia Puāwai Kaumātua; Kia Puāwai management (including the Operations Manager, the Practice Lead and the Intake Specialist); the Programme Manager (Oranga Tamariki); and referrers (Police and Corrections).
The interviews were conducted between 4 April and 8 October 2022. The ten interviews with client families were conducted with three Māori whānau, three Pasefika aiga, and four Pākehā families remotely via zoom or telephone. They represent the three largest client bases of Pae Whakatupuranga | FFT-CG. Remote interviewing was used as a health safety measure because of the ongoing impacts of the COVID-19 pandemic.
Interviewers and interviewees were matched culturally for the interviews with whānau, aiga and families (Māori, Pasefika, Pākehā). Interviews with management, therapists, the kaumātua and stakeholders were also conducted by Zoom or phone. The quantitative analysis interrogated the data stored in the: FFT-LLC database (CSS); Kia Puāwai database (HCC); Outcome Questionnaire website; and in a separate spreadsheet with data from the Cultural Satisfaction form. Tabular analysis was employed to analyse the extent of engagement and equity of outcomes across ethnicity, gender and referral sources.
The outcome data are measured by responses to OQ (Outcomes Questionnaire), the YOQ (Youth Outcomes questionnaire) and the YOQ-SR (Youth Self Report) before and after treatment, and responses to COM-A (Client Outcome Measure (Adolescent)), COM-P (Client Outcome Measure (parent)), and Cultural Satisfaction form at discharge. We use tables and graphs instead of t-tests to evaluate the differences in measured outcomes before and after treatment when sample size is small.
Key Results
Wave 4 Report:
Snapshot of progress as at 30 June 2022:
The numbers:
119 families have started Pae Whakatupuranga I FFT-CG:
22 are active
42 completed the programme
55 exited early.
Working well:
- Improved family communication and dynamics enabling better conversations between young people and their whānau/aiga/families.
- The Pae Whakatupuranga interweaving of the Māori and Pasefika frameworks with FFT international is now well established.
- The cultural approach to therapy is greatly appreciated by all clients, including Pākehā.
- The ongoing cultural training and supervision has developed confidence and skills among the therapists.
- Despite difficulties, the programme managed the COVID-19 pandemic isolation period well.
Challenges
- Need to increase frequency and regularity of Pasefika cultural supervision for therapists to better address higher drop out and lower completion rates.
- Need to acquire evidential data on very high numbers of ‘never began’ referrals and on the reduction in referrals during the pilot and increase since.
MINISTRY UPDATE: As at March 2023 60 people have completed the programme and 61 have exited early.
A summary of findings in relation to the six key evaluation questions is set out below. How well did the programme achieve outcomes for Māori and Pacific Peoples Over the course of the pilot, 97 cases were closed (i.e., not including the currently active cases because their outcomes are not yet determined). Māori and Pasefika had lower completion rates than Pākehā, being 40% and 31% respectively, while 64% of Pākehā completed by comparison.
The therapeutic course has five stages of treatment: Engagement; Motivation; Relational assessment; Behaviour change; and Generalisation. If we add those who reached the final generalisation stages of the course to those who fully completed, then the Māori rate increases to 50%, the Pasefika to 38%, and Pākehā to 71%.
It should be borne in mind though, that the client base is drawing upon households experiencing serious social, legal and economic difficulties for whom the programme was designed. Furthermore, the Māori and Pasefika referrals are more than two and a half times more likely to be referred for delinquent behaviour than the Pākehā referrals.
Māori and Pasefika whānau and aiga who were interviewed consistently spoke of how the therapy had improved their communication and family dynamics. They referred to finding new ways of talking with each other about their feelings and resolving conflicts. They also spoke of learning ways to discuss hard issues in a safe manner so that the matters did not escalate. The quantitative data showed that many clients began treatment with high levels of distress which were subsequently reduced, particularly for those under 18 years.
The distinctive cultural achievement of the entire programme is demonstrated in the cultural approach to therapy. The Pae Whakatupuranga interweaving of the Māori and Pasefika cultural frameworks with FFT enables the therapists to fully engage in the cultural worldviews, while also drawing on the developed wisdom and experience of the international FFT organisation. Māori and Pasefika clients registered high cultural satisfaction scores and said they felt comfortable and that their cultural values were respected because the therapists understood their ways of doing things.
There is a need to develop more frequent and regular Pasefika cultural training and supervision. As with the Māori cultural supervision, it is appreciated and very helpful for therapists. The proportionally higher drop out and lower completion rates for Pasefika clients indicate the need to increase cultural competence and confidence among the
therapists.
Overall, the psychological, social and behavioural outcome measures showed that for 10 to 17 year-olds, 76 percent of parents and caregivers, and 74 percent of the young people reported improvement scores as a result of treatment. For 18 to 25 year-olds, 67 percent of parents and caregivers reported improvements, but only 37 percent of the young people did. On an outcome scale filled in just before discharge, the results also noted improvements. On a scale of 0 to 5 where 0 registers things are worse, 1 registers no change, and 5 registers most things changed successfully, the family dynamic and behavioural change scores of parents and caregivers and the young people ranged between 3.67 and 4.00 (three indicated things were somewhat better, and four that things were a lot better).