The Health and Disability Commission (HDC), in conjunction with CBG Health Research Limited (CBG), developed and implemented an electronic, real-time system to capture feedback from people interacting with mental health and addiction services (the RTF system). The intention of the RTF was to develop a system that will ensure that the voices of consumers, family/whānau are heard and contribute to quality improvement. This report summarises the information gathered as part of the evaluation of the RTF, which used evaluation questions in the RTF survey, an online survey of service provider staff, site visits, and interviews.
Key Results
Service provider participation in the pilot: A RTF system has been developed and is being used to collect feedback from consumers, families/whānau at each of the seven pilot sites. As at 28 September 2014, 1,721 consumers, family/whānau had completed the feedback questions. A major achievement of the pilot is that the seven service providers who took part in the pilot are positive about the potential value of a RTF system in improving the quality of the services they provide to consumers, family/whānau. Following the end of the pilot period, all seven sites will continue using the RTF system to collect feedback and are committed to developing ways to use feedback to improve their services.
Setting up the system: Feedback is collected through an online survey with consumers, families/ whānau providing their feedback by completing questions on a portable tablet. Wifi connectivity is required to upload the data in ‘real-time’ to the server. Pilot sites considered that they had had the support they required from HDC and CBG. Linking to wifi and local IT issues have been a challenge for the pilot and highlight the need for local IT teams to be involved at the start in setting up the RTF system in new sites. At some sites when something went wrong the tablet sat unused until someone with expertise in setting it up became available.
Developing the questions: Service providers and consumers, family/whānau were consulted about the question content and format and a first version of the questions developed and piloted. Two subsequent versions of the questions were developed in response to feedback on earlier versions. The third and current version was rolled out just prior to the evaluation and included simplified questions, translation into the main languages used at the service provider sites, and optional site specific questions. Consumers like the simplicity of the ‘smiley-face’ response options and the short length of the survey. Although welcomed, the language of the translation (too formal and at a higher literacy level) was not quite right for some service users. Other specific concerns raised by service provider staff at the time of the evaluation may reflect a lack of familiarity with the latest version of the survey. Some sites also requested further customisation of the questions to their site such as not using the word consumer.
Collecting feedback: The sites had different approaches to collecting feedback. Active approaches through a person explaining the RTF system and inviting feedback were more effective than passive approaches. Active approaches included reception staff offering the tablets to consumers, consumer advocates spending time at the site and inviting feedback, and clinical staff inviting feedback during individual and group session and in consumers’ homes.
Passive approaches included displaying posters and leaving the tablets on pedestals. Some sites asked for information resources to be developed by HDC (simplified pamphlet, diagrammatic resource) that could be used to explain the RTF system.
Tablets were provided to sites for the pilot and a limited number meant that there were not enough tablets to cover all locations within the pilot sites. Pilot sites have indicated a willingness to purchase more tablets for future use.
There was inconsistency in the extent administrative and reception staff offered the tablet to consumers, family/whānau depending on which staff are on duty and their workloads. Reception and administrative staff were not confident to ask for feedback from people they felt were upset and/or angry.onsumers, family/whānau provided feedback before appointments and there Many cwas uncertainty about how often to invite feedback from the same people.
Consumer, family/whānau reactions: Most consumers, family/whānau enjoy using the tablets, though some have difficulty with the technology or language and require explanation from staff. Consumers, family/whānau value the opportunity to record their feedback but want to know how it is used.
Displaying the feedback: In planning the pilot, HDC and the advisory group expected that by the end of the pilot period sites would be displaying feedback results to staff and consumers, family/whānau, and service providers would be starting to use the feedback to contribute to quality improvement.
A major learning from the pilot was that it took longer than they expected to develop feedback questions that were relevant to providers and consumers, family/whānau, to work with service providers to put RTF systems in place and have staff familiar with using them, and for service providers to start to use the feedback. However, the time it takes to embed new systems and make changes is reported in evaluation of other similar projects.
At the time of the evaluation, although feedback was available online sites had generally not been providing feedback to consumers and many staff had not seen feedback. Consumers are enthusiastic about access to results and staff like the idea of providing access, for example through a screen in the clinic waiting area or a poster.
Some of the reasons why the results were not being displayed and used are likely to be addressed thorough the new version of the survey. That is concern that results were not an accurate reflection of their service (too positive) and that the results do not change.
Using the feedback to make changes: At the time of the evaluation, almost all staff reported that no changes had been made as a result of the feedback although some sites had plans to make changes.
The enthusiasm of the pilot sites about the value of receiving feedback and the potential use of the feedback to make changes suggest that the feedback will be used to improve services. However, data need to be more consistently collected and sites need support to know how to use the feedback data as part of a change process. While the core questions are high-level and results unlikely to change rapidly, there is the potential to use the open-ended and site specific optional 5 questions to track changes in initiatives set up by the sites. Use of the optional questions in this way would confirm the value of the feedback being ‘real-time’. Changes to the analytics displayed to include trend data and making raw data available to the sites will also help progress towards sites’ use of the feedback data.
Wider roll-out: The Mental Health Commissioner, the advisory group and the pilot site managers support the expansion of the real-time feedback system with 49% of surveyed staff strongly agreeing and 29% agreeing that the feedback system should be expanded to other practices.
HDC and the advisory group have committed to a second phase of the pilot and are working with the Health Quality and Safety Commission and the Ministry of Health to explore ways to support service providers to use feedback in quality improvement.