Purpose
The evaluation aimed to understand:
- Sector understanding and readiness to use the new provisions
- Sector experience of the new provisions
- The level of use and early influence of the new provisions.
Methodology
The data collection methods for the evaluation included:
- First online survey
- In-depth Interviews
- Second online survey
Online surveys
Survey one was disseminated in February and March 2020. The survey was designed by Oranga Tamariki and implemented by Synergia. The survey was designed to identify the use and understanding of the new provisions. It asked respondents about their awareness of the provisions, involvement with information sharing, processes for consulting tamariki, and factors that supported or challenged engagement with and use of the new provisions. It also asked for respondents’ ideas for improving people’s understanding and use of the new provisions. A total of 143 people responded to the survey.
Sample: The contacts for the survey respondents were provided to Synergia by Oranga Tamariki. Respondents were contacts that Oranga Tamariki had previously engaged with on the new provisions, including during the development of the information sharing guidance (available on the Oranga Tamariki website). This purposeful sample was designed to ensure that people invited to take part in the survey would have some knowledge of the new provisions due to their engagement with Oranga Tamariki. The survey also had an open link so that people could easily and readily share it with their colleagues and wider networks.
The survey was available to all Oranga Tamariki staff, and sent to other government agencies, and NGOs working in the sector. The survey was disseminated via SurveyMonkey and was sent out via email and the Oranga Tamariki staff intranet.
Survey two was disseminated in February and March 2021. The online survey related to the awareness, understanding, use and influence on practice of the new provisions. A total of 514 people responded to the survey.
Synergia and Oranga Tamariki developed the second survey, which included some of the same questions from the first survey for comparison. Synergia disseminated the second survey via SurveyMonkey.
Sample: Oranga Tamariki identified a variety of channels for communicating about the survey with the wider sector. Like the first survey, the second survey also had an open link so that people could easily and readily share it with their colleagues and wider networks. The survey link was sent to contacts from the government information sharing
inter-agency working group for inclusion in their agency’s communications as appropriate. This included:
- Ministry of Health
- Ministry of Education
- ACC
- Ministry of Justice
- Ministry of Social Development
- Police.
The link was also included in Oranga Tamariki Update for Partners newsletters and sent to Social Service Providers Aotearoa (SSPA), Aotearoa New Zealand Association of Social Workers (ANZASW), Women’s Refuge and Social Workers in Schools for them to distribute to their networks and include in their communications (e.g., newsletters and emails).
A survey link was also directly emailed to those that were sent the first survey and a link to the survey was included on the Oranga Tamariki information sharing web page. The survey was distributed to Oranga Tamariki staff through internal communication channels, including Pānui, a fortnightly newsletter for all Oranga Tamariki staff and the weekly operations email for frontline staff. A link to the survey was also published on Te Pae.
Interviews
Semi-structured interviews with the sector were conducted from August to December 2020. Synergia and the Oranga Tamariki information sharing project team worked collaboratively to create the interview schedule. The interview questions were designed to help Oranga Tamariki gain a more in-depth insight into how the sector had been using the new provisions, the influence of the new provisions on practice and ideas for improvements. The interview schedule was used to guide the interviews.
Interviewees were purposively sampled to ensure that they had used the new provisions. Oranga Tamariki supported Synergia by connecting us to relevant organisations and people to interview, many of whom were involved in the early consultation process to inform the guidance for the legislation.
In total, we contacted over 62 organisations. From this, 23 organisations responded as having used the new provisions, and 32 people from these organisations took part in an interview. Of the 23 organisations formally interviewed, nine were Māori organisations.
A general inductive approach for analysing qualitative evaluation data was used (Thomas, 2006). The general inductive approach provides an easily used and systematic set of procedures for analysing qualitative data that can produce reliable and valid findings.
Formative feedback
Rapid insights reports were provided to the Oranga Tamariki project team at the end of each data collection phase. The survey insights report contained a descriptive analysis of the data alongside data summary charts. The interview insights report used a general inductive approach for analysing qualitative evaluation data to present findings from the interviews. These reports provide more detail into specific examples from the findings.
Limitations and challenges
Key limitations and challenges to the evaluation design and implementation were:
- There were a small number of responses to survey one.
- There were lower numbers of responses to the surveys from some areas of the sector, for example health and justice.
- Although 360-degree case studies were planned, due to COVID-19, the evaluation was unable to hear from tamariki and whānau on their views and experiences of the new information sharing provisions.
Key Results
Key insights from the evaluation include:
- The new provisions are still being embedded into practice, and this takes time. It requires a significant change in mindset and practice. This shift involves moving from viewing information sharing as a privacy breach to seeing information sharing as a way to enable organisations to better support tamariki and whānau.
- There were varying levels of awareness of the new provisions, from people who were aware of the new provisions and using them confidently to people who were not yet aware of them.
- The multiple legislative changes that came into effect at the same time as the information sharing provisions in the Act led to overload and some confusion which challenged implementing the new provisions. This included Care Standards regulations, other significant changes to the Oranga Tamariki Act, including the addition of section 7AA and changes to the Family Violence Act 2018 which also included new provisions relating to sharing information. Later the Privacy Act 2020 also came into force.
- Relationships and trust were important foundations for information sharing practices and could be both a barrier and enabler to information sharing.
- Some sectors were more engaged than others in information sharing prior to the introduction of the new provisions and have used the new provisions more than others. For example, people in the family violence sector were already sharing information more than people in the health sector. They have therefore adapted the provisions more quickly as there is less practice change required.
- Many people wanted more training opportunities, sector specific guidance and case examples to increase their confidence in using the new provisions. Training was requested mostly around understanding the terms “good faith” and what is and isn’t included in the term “wellbeing”.
- There was also a need to continue to raise awareness of the new provisions across the whole sector, including Oranga Tamariki.
- The provisions were enabling people to share information with organisations that they had not had previous relationships with.
- When people understand and confidently use the provisions, they can positively influence information sharing practices by supporting best practice for information sharing.