Purpose
The purpose of the IWS evaluation is to determine the effectiveness of IWS in supporting clients to maintain employment over the longer-term. This report presents the qualitative component of the evaluation. It complements the quantitative evaluation completed by Insights MSD.
Methodology
The qualitative component aimed to better understand client experiences of In-Work Support and the In-Work Payment and how best to support clients to remain in work by exploring the following four domains of the intervention logic for the service:
- Clients make an informed decision about opting in/out of IWS (uptake).
- The support that clients receive is useful and timely (service provision).
- The IWP incentivises HEEC clients to opt into IWS and remain in work for the first 12 months (incentive).
- IWS and IWP act as a springboard for HEEC, HCID and JS clients to remain off a benefit over the longer-term post-IWS (impact).
For the fourth of these domains, the qualitative component explored perceptions of impact (ie, perceptions of the difference that the service made to participants). It was not intended that this study would estimate the degree to which it was the service itself, rather than other factors, that accounted for the changes people experienced. That was the task of the quantitative evaluation.
This report describes findings from in-depth interviews with 68 IWS clients and six CSRs. We used the data collected through interviews to construct composite case stories that represent a range of IWS client profiles which were then validated in focus groups with CSRs.
Participants were selected from the population of clients who were receiving or had received the IWS service. All HEEC clients selected had received the associated IWP. We added a booster sample of HCID/JS clients as the number of clients in these groups in the initial sample was small.
In common with many other studies with this population group, the proportion of clients approached who agreed to participate was low. HEEC clients had the highest response rate (51%, n = 27), followed by HCID clients (36%, n = 26). JS clients had the lowest response rate (11%, n = 2).
Given the very low numbers of JS clients interviewed, we do not separately report results for this client group, but do report results for the combined HCID/JS group. The low response rates mean a particularly cautious approach needs to be taken to generalising what participants told us in their interviews as indicative of the experiences of IWS/IWP clients overall. This is especially the case for HCID and JS clients.
Client interviews
We selected clients from the population of clients who had been offered, or were receiving, or had received, the IWS service between February 2015 and August 2017. The selection of HEEC clients was limited to clients who were eligible to receive the IWP.
We added a booster sample of HCID/JS clients as the number of clients in these groups in the initial sample was small.
HEEC, HCID and Jobseeker clients were intended to be grouped into a further three categories:
- Off-benefit at the 12 month completion of IWS
- Returned to benefit during IWS
- Opted out of IWS.
We agreed target interview numbers with MSD for each group across the nine client categories. The sample supplied by MSD in November 2017 identified HEEC clients by category as: HEEC offbenefit, HEEC on-benefit, and HEEC opt-out. However, HCID and JS clients were not able to be categorised as this information was extracted from a different database which did not contain information about who was offered and who declined the service and some of these clients may have self-referred to the service. We were able to use the supplied IWS start and end dates to assign
tentative categories to some of the clients but were unable to identify opt-out HCID and JS clients until after speaking to them.
This constraint meant we were unable to speak to any HCID and JS opt-out clients. These clients may have self-selected out of interviews as several HCID and JS clients who were contacted declined to be interviewed, some because they had no memory of the IWS and others for unspecified reasons. We also spoke to more HCID clients and fewer JS clients than expected, due to clients not responding or declining to be interviewed, and the greater number
of HCID clients on the contact list. Table 1 shows the completed interviews at the close of the interview phase compared to the initial interview goals.
We conducted interviews using semi-structured interview guides which were based on the evaluation questions. Interview guides are appended.
Clients were selected from a contact list provided by MSD and sent an introductory text explaining the evaluation, notifying them they would receive a phone call to offer them an interview, and offering a $40 koha in the form of a supermarket voucher to thank them for their participation. Clients were then called later the same day or the following day and asked if they would be willing to participate in an interview. Three attempts were made to contact clients before they were marked as having made no response.
In total, we attempted to contact 173 clients. Interviews were successfully completed with 68 (39%) clients, 35 (20%) declined to be interviewed, and 80 (46%) did not respond (Table 2). Given the very low numbers of JS clients interviewed, we do not report results for this client group separately, but do report results for the combined HCID/JS group. The low response rates mean a cautious approach needs to be taken to generalising what participants told
us in their interviews as reflecting the experiences of IWS/IWP clients overall. This is especially the case for HCID and JS clients.
Key Results
Do clients make an informed decision about opting in/out of IWS? (uptake)
Successes
The CSRs are a committed and cohesive team. All have Work and Income call centre backgrounds and enjoyed the opportunity to provide IWS. CSRs had a comprehensive script to use when they invited people into IWS.
Most clients gained a clear understanding of the service and were able to make an informed decision about opting in or out. Receipt of the IWP positively influenced client understanding of the IWS service and the choice of opting in. The HEECs are more likely to be satisfied with the amount and content of IWS information received than the HCID and JS clients.
For the most part, interviewed clients who declined the service did so because they were managing well on their own, and did not feel like they needed any support.
Challenges
Most clients who are unclear about what the service could offer them were HCID/JS clients who did not receive the IWP. These clients are less enthusiastic about the service and have fewer expectations of the service as a whole.
Some clients opted out of IWS due to their negative perceptions of Work and Income. CSRs recognised that some people think they are a hoax caller, and they directed clients to the IWS 0800 number and website in those instances. A small number of clients experienced language barriers and required a translator.
Most people who opted out of IWS have poor memories of what happened when the service was offered.
7.3. Is the support that clients receive useful and timely? (service provision)
Successes
Overall, clients were positive about their experiences of IWS. They found the CSRs to be friendly and responsive. CSRs had more flexibility in their work to spend time with clients and to listen to them. The IWS service has changed some clients’ negative perceptions of Work and Income.
Contact methods were client-centred. When clients were unavailable during work hours, the CSRs contacted them by text, enabling clients to call back at a convenient time. Some clients required intensive support and they contacted IWS frequently, but most were happy to have the service in the background. Knowing that ‘someone is there’ is reassuring to clients.
IWS helped clients navigate difficult situations such as workplace conflicts. CSRs armed the clients with information and skills to resolve issues, where in the past the client may have left their job and returned to benefit.
The majority of clients were contacted within two to three weeks of having their benefit cancelled, although some were contacted a month or six weeks into their new job.
Clients and CSRs agree that the 12 month duration of the service is about right, but overall, clients said that six to 18 months would benefit them. This is due to the variability in client needs. Some were very confident in their employment after six months and others required on-going support after the year is up.
Challenges
CSRs would have been aided in their work if they were able to provide food grants and emergency assistance to clients in crisis.
CSRs provide information and connect clients with other services, but both CSRs and clients feel that the service would deliver better support by providing tangible assistance.
CSRs do not provide individualised case management, and clients were contacted by different members of the team. Most clients were happy with this but a small number would have preferred to speak to the same CSR each time.
It was suggested by some clients that information about IWS before starting their new job would have been helpful.
7.4. To what extent does IWP incentivise HEEC clients to opt into IWS and remain in work for the first 12 months? (incentive)
Successes
The IWP made a positive difference in the lives of many clients. It allowed them to buy work tools or clothing, to put deposits on cars or fix their existing cars, pay debt or backlogged bills, and treat their families.
Findings suggest that the combination of the IWS and the IWP incentivised clients to stay in work longer than they otherwise would have. The IWP provided a safety net for clients. Spreading the IWPs over 12 months, with the last instalment being one of the largest, motivated many clients to stay in work.
HEEC clients are more engaged with the service and more positive about the service than those who do not receive the IWP, namely the HCID and JS clients.
Challenges
The ability of casual, temporary and seasonal workers to stay in work is influenced to a greater degree by the availability of local employment opportunities than by the IWP. However, IWS and the IWP helped clients to negotiate the jobless period ahead.
7.5. To what extent do IWS and IWP act as a springboard for HEEC, HCID and JS clients to remain off benefit over the longer-term post-IWS? (perceptions of impact).
Successes
IWS helped clients to develop a routine and adjust to working life. CSRs send regular emails containing tips and tricks that give clients extra knowledge to help them succeed in their jobs.
Most clients want to stay in their job long-term, however this often depends on the work environment, pay rates and co-workers.
Clients and CSRs interviewed agree that IWS and IWP smooth the transition into longer-term employment, but this also has a wider effect in the lives of clients. With the support of IWS and the financial safety net of IWP, many clients build confidence in their work and also in other aspects of their lives, leading to greater independence and a better quality of life.
Challenges
For some clients, location, employment opportunities and health issues can be mitigated but not solved by the IWS or IWP, and they returned to a benefit.