Transition Support Service evaluation findings 2022

Methodology

The evaluation was phased to align with the three-year roll-out of TSS and to contribute to the ‘learn and grow’ approach:

• The first phase was a formative evaluation to provide early insights into specific aspects of TSS.
• The second phase was a process focused evaluation to provide information about what was working well and where there were challenges to inform service delivery responses.
• The third and final phase continues the process evaluation and following full implementation, includes a focus on outcomes. A secondary purpose is to support the Oranga Tamariki Action Plan (OTAP) and organisational change.

Evaluation activities in each phase of the evaluation are summarised in Figure 1.

Evaluation questions (Appendix 1) guide the evaluation. The evaluation questions are reviewed annually by the Oranga Tamariki Evidence Centre, Malatest International and an Evaluation Advisory group. The questions guide the different phases of the evaluation – not all questions are applicable to each evaluation activity.

This report summarises findings and informs the process phase of the evaluation (Figure 1). Information was sourced from:

• Oranga Tamariki TSS administrative data
• Just Sayin’ 22 – the third Just Sayin’ survey completed by 408 young people (www.justsayin.nz) and previous Just Sayin’ surveys (Just Sayin’ 20 augmented by an additional survey of 16 year olds, and Just Sayin’ 21
• regional case studies with a focus on Oranga Tamariki transition support and regional TSS partners
• high level findings from interviews with 26 young people from the draft Year 2 Ngā Haerenga report – included to augment the voices of young people.

2.1. The Just Sayin’ survey has been a part of the TSS evaluation for three years

Part of the TSS evaluation is an annual survey of young people eligible for support by transition workers – the Just Sayin’ surveys. The Just Sayin’ surveys have been distributed in 2020 (an initial survey of 17-19 year olds complemented by a later survey of 16 year olds), 2021 and 2022 to seek feedback from young people aged 16 to 21 who are eligible for support from a transition worker. More details about the surveys are provided in Appendix 2. An overview report from the 2021 Just Sayin’ survey can be found here:

www.orangatamariki.govt.nz/about-us/research/ourresearch/transitions-service-synthesis-report/ 

For a full list of responses for each question see the Just Sayin’ 22 technical report and the Just Sayin’22 Fact Sheets. The Fact Sheets will be published on the Oranga Tamariki website as they become available. This report is based on the responses of 408 young people between the ages of 16 and 218 who were eligible for support from a transition worker. This figure represents 18% of all eligible young people, 24% of young people with contact details supplied by Oranga Tamariki and 57% of young people with contact details we confirmed to be valid (Figure 2). 

The demographic profile of survey respondents is summarised in Appendix 2. As in previous years, females and young people who identified as non-Māori and nonPacific were slightly over-represented amongst the survey respondents.

The progressive roll-out of the TSS meant young people aged over the survey years resulting in differences in age groups. In 2020, most of the potentially eligible cohort were still in care. In later years, more had left care and were supported by a TSS partner.

Young people who completed Just Sayin’ 2022 came from across Aotearoa New Zealand, with a similar spread to years prior. The areas with the smallest representation were Gisborne, Marlborough and West Coast (Figure 3).

2.2. The regional case studies provide the perspectives of Oranga Tamariki staff and TSS partners

The regional case studies complement the Just Sayin’ surveys by providing in-depth qualitative perspectives from Oranga Tamariki social workers and the TSS partners.

Regional case study details and aims are summarised in Figure 4 below. The regional case studies focus on Oranga Tamariki staff and TSS partners to balance the 2021 evaluation programme which focused on young people’s voices in different settings, including youth justice.

The regions selected for case studies included regions with high proportions of rangatahi Māori and Pacific young people and to provide a contrast between urban and rural settings (see Appendix 3). Information from the case studies provides insights into the selected regions and TSS partners but may not be generalisable to other regions.

2.3. Strengths and limitations of the information informing this report

The main strength of the report is the synthesis of findings from different sources. Our approach was based on gathering and interweaving different stakeholders’ perspectives from in-depth interviews with a regional selection of TSS partners and Oranga Tamariki staff, and survey responses from young people. In-depth interviews with young people to hear about their experiences was the focus of the evaluation in 2021. Together these perspectives informed the development of a collective fabric of experiences, processes, and findings.

Our ability to conduct evaluation and research in a culturally safe and competent way led by our Kaupapa Māori and Pacific evaluation teams has enhanced theoutcomes of our work, ensuring that participants are able to contribute their thoughts and experiences to those who understand their contexts.

The main limitation of the Just Sayin’ surveys is our inability to contact young people for whom Oranga Tamariki do not have contact details. Therefore, the findings in this report reflect the only views of the young people who responded to the survey.

Young people for whom Oranga Tamariki do not have contact details may hold different views and be living in different circumstances. Although we worked with TSS partners to include young people they supported, many of the young people without contact details may also not be engaged with a TSS partner. A small number of young people declined the survey because they had completed it in previous years and had seen no change in the TSS.