Partners for Change Outcomes Management System

Methodology

Evaluation design

The evaluation used a mixed methods sequential explanatory design. As understanding the experiences of trial participants was always going to be central to the evaluation, qualitative data was the priority; the quantitative and qualitative analysis and findings were integrated during the interpretation stage.

Data collection and analysis methods

Literature scan

A literature scan, including books, journal articles, websites and grey literature, was undertaken. In addition to On Becoming a Better Therapist: Evidence-based Practice One Client at a Time, by Barry Duncan (2014), and the material on the www.betteroutcomesnow.com website, other key texts included:

• PCOMS Document (SAMHSA, 2017);
• Report (Partnering for Outcomes Foundation Aotearoa, n.d.);
• Report (Connect+Co, 2016) (unpublished version of above with appendices); and
• Various book chapters and articles (for example, Anker, Duncan, & Sparks, 2009; Bohanske & Franczak, 2010; Bringhurst et al., 2006; Cooper, 2013; Duncan et al., 2003, 2006; Duncan & Sparks, 2016; Lambert & Shimokawa, 2011; Low, 2012; Manthei, 2015; Miller & Duncan, 2004; Miller et al., 2003; Reece, Norsworthy, & Rowlands, 2009; She et al., 2018).

Based on recommendations by Superu the scan also reviewed evidence based and systematic review websites relevant to statutory social work. These included:

• Campbell Collaboration;
• Cochrane Library;
• California Evidence Based Clearing House for Child Welfare;
• Crime Solutions;
• Office of Juvenile Justice and Delinquency Prevention;
• Blueprints; and
• Investing in Children Programme materials.

Documentation

The evaluation also reviewed the following Oranga Tamariki documents:

• Oranga Tamariki PCOMS Pilot Plan July 2017 (11 August 2017, v.1_2_1);
• Oranga Tamariki PCOMS Logic Model (10 October 2017, v.3);
• Oranga Tamariki PCOMS 2-pager guidance (7 February 2018);
• Oranga Tamariki PCOMS PowerPoint presentation for Managers and Practice Leaders (undated);
• Oranga Tamariki PCOMS Training day PowerPoint presentation (19 Feb 2018, modified
  version);
• Oranga Tamariki PCOMS 23 May 2018 workshop presentation slides from Barry Duncan; and
• Oranga Tamariki Core Practice Standards (undated).

Survey

The PCOMS online survey of Oranga Tamariki staff attending PCOMS training (n = 52), was designed by the Oranga Tamariki Evidence Centre, and administered before the commencement of the evaluation. The survey was run twice, (November 2017, and April/June 2018) and achieved a response rate of 42% (noting that some of those who had attended the training were no longer using PCOMS and so on that basis may have chosen not to respond). Material was provided for the evaluation in the form of Survey Monkey output presentational data, as well as a consolidated Excel spreadsheet. Analysis was limited to presentational graphics.

Administrative data

For Oranga Tamariki PCOMS administrative data, access was provided to the PCOMS BON website. BON does not allow for raw data to be downloaded and externally analysed, and so the use of descriptive statistics was not practical. Outcomes Rating Scale (ORS) summary report screens in relation to each client, social worker, team, or Oranga Tamariki overall, were available.

Interviews

Face-to-face semi-structured interviews were held with 15 Oranga Tamariki social workers, supervisors and managers from across all four sites. In association with Oranga Tamariki, individuals were selected on the basis that they were best placed to help the evaluators address the key evaluation questions. Therefore, those who had made little use of PCOMS (or had stopped using it altogether) were under-represented.

The Oranga Tamariki Evidence Centre undertook an ethics review of the developed Participant Information Sheet and consent form. The consent form ensured that it was clear that the participation of social workers in the evaluation was entirely voluntary, and that what they said in their interviews would be treated confidentially, and any information that might individually identify them would not be included in the report without their agreement. They were also told that it could be possible for an informed person to deduce who had said a particular thing.

Oranga Tamariki also approved the developed interview guide. When approached, all 15 prospective participants agreed to take part and all made themselves available for interview on their respective agreed date and time. All interviewees signed a consent form.

Evaluation limitations

Some of the limitations identified in relation to this study are listed as follows:

1. This was a small-scale evaluation, which limits somewhat the ability to generalise.
2. The questionnaire includes some responses from Tauranga social workers; The Tauranga site subsequently ceased their involvement with the PCOMS trial. This reduced the overall sample size.
3. Several of the social workers appear to have had little experience of actually using PCOMS at the time that they were surveyed, and this may have impacted upon their responses.
4. While a large proportion of the social workers who were involved in the PCOMS trial were interviewed as part of the evaluation, as with all qualitative interviews the experiences and views of those interviewed are not necessarily representative of all others.