Introduction

About the evaluation context

Family Safety Victoria (FSV) engaged PricewaterhouseCoopers Consulting (Australia) Pty Ltd (PwC) in partnership with PwC’s Indigenous Consulting (PIC) and Australian Catholic University (ACU) to conduct an evaluation of the first four operational areas of The Orange Door in September 2018. The Orange Door (previously known as Support and Safety Hubs) is a new way for women, children and young people experiencing family violence, and families who need assistance with the care and wellbeing of children to access the services they need to be safe and supported.

This evaluation focuses on the establishment, operations and initial service offering of The Orange Door in Barwon, Bayside Peninsula (BPA), Mallee and North East Melbourne (NEMA) areas that commenced operations between May and July 2018. The evaluation seeks to identify key lessons and opportunities for improvements to establishment activities to support the implementation of The Orange Door in 2019 and beyond, and to inform enhancements across existing and new areas. As such, the evaluation is intended to identify opportunities for improvement at the area (micro-level), across area (organisational-level) and statewide (system-wide) levels to enhance operations and service delivery in the future.

Throughout this report we have discussed the context in which our findings need to be considered. This is critically important when considering the opportunities for improvement identified in the final chapter of this report. It was not expected that The Orange Door would be operating seamlessly in an integrated manner at the time in which the 2018 evaluation was undertaken. The Orange Door represents a new way of working for FSV and the partner agencies. FSV is a new agency, integrated practice with victim survivors, perpetrators and child and family services is a new model, and the model required new facilities, new processes and new staff. The scale of reform and change that The Orange Door has faced in the early establishment phase cannot be underestimated. There has been enormous effort on the part of FSV, all agencies and their staff involved to establish The Orange Door and this was evident to our evaluation team during fieldwork and stakeholder interviews. The findings are not intended as a critique of these efforts, rather as a method to identify opportunities for improvements to the establishment, operations and initial service offering in current areas and in future areas where The Orange Door will be rolled out.

About The Orange Door

The Orange Door is a new model of service entry and coordination for family violence and family support services in Victoria (previously known as Support and Safety Hubs) which responds to key recommendations of the Royal Commission into Family Violence and delivers on the vision described in Roadmap for reform: strong families, safe children. Recommendations were based on findings that people often don’t know where to go for help, can be bounced around the service system and may experience fragmented or uncoordinated responses and that efforts to hold perpetrators to account and in view were insufficient.

The Orange Door is intended to be safe and welcoming and provide quick and simple access to support. The Orange Door statewide concept (Support and Safety Hubs Statewide Concept, July 2017) was informed by a series of design discussions with people who have experienced the service systems. It was also informed by consultations with the community sector and agencies, and across government that focused on ways to provide easy and seamless access to the full range of supports and services that individuals and families need.[1] The service model for the first rollout was intended as a foundational model whereby certain elements, roles and functions may evolve over time given the phased approach to implementation. However key functionality of the model was outlined in FSV’s Support and Safety Service Model (April 2018) and comprises following key features:

• Access: ensuring a safe and convenient entry to The Orange Door, including telephone and online access, a community-based hub, outreach practitioners and referrals from other professionals.
• Screening, identification and triage: identifying risks and needs of all people in a family, in order to determine whether The Orange Door is the appropriate response and then to prioritise The Orange Door response.
• Assessment and planning: whereby practitioners work with people to develop a clear understanding of risks, strengths and issues, identify support needs and identify the type and intensity of service responses required.
• Connecting people to the right services: involving matching services to the needs of a person and family based on the needs and risks identified through assessment and planning, and then connecting the person/family to the services or providing this support directly.

The Orange Door will be established across the 17 Department of Health and Human Services areas. The first four areas examined as part of this evaluation were:

• The Orange Door in Bayside Peninsular area (BPA)
• The Orange Door in Barwon
• The Orange Door in North East Melbourne area (NEMA)
• The Orange Door in Mallee.

Figure 1: Locations of the first four areas of The Orange Door and the primary physical sites

Mallee

Physical location: Mildura

Commenced: 31 May 2018

FTE: 33.4 as of October 2018

Partner organisations: 4

Bayside Peninsula

Physical location: Frankston

Commenced: 14 May 2018

FTE: 80.3 as of October 2018

Partner organisations: 10

North East Melbourne

Physical location: Heidleberg

Commenced: 10 July 2018

FTE: 64.6 as of October 2018

Partner organisations: 7

Barwon

Physical location: Geelong

Commenced: 31 May 2018

FTE: 43.9 as of October 2018

Partner organisations: 4

NB: The number of partner organisations includes community service organisations and Aboriginal services (these numbers do not include staff from FSV, DHHS or Vic Police).

Source: Figures derived from FSV commencement reports and List of community service organisations in launch areas. The Orange Door in Inner Gippsland has commenced but is out of scope for this evaluation. FTE refers to funded positions for 2018-2019.

Evaluation approach and framework

A developmental approach to evaluation

We took a developmental approach to the evaluation to gain insights about the evolution of The Orange Door. Our approach used traditional methods of data collection to enable early insights to rapidly inform changes to current operations of The Orange Door and for the establishment of The Orange Door in further areas. Point-in-time opportunities for improvement also provide FSV and partner agencies of The Orange Door the opportunity to revisit how they are operationalising the service model.

Developmental evaluation is an evaluation approach that can assist evaluators to understand new innovations, initiatives and programs that are being implemented in complex, changing or uncertain environments. The approach facilitates real-time testing of hypotheses, rapid iterations and changes to program design, with feedback on the initiative allowing for rapid and evolving improvements to be made (even while the evaluation is still occurring).

Given the recent establishment of The Orange Door and the evolving nature of service operations in the context of ongoing establishment processes, it is not possible to conduct a traditional ‘summative’ evaluation that makes conclusions about system outcomes and answers the question ‘does the model work’? As such, a developmental evaluation approach allows the rigour of traditional methods of evaluation (such as data analysis, interviews, etc) to be applied while accommodating the dynamic environment in which The Orange Door is evolving in terms of operations. Our developmental evaluation approach enables early insights to rapidly inform changes to the way The Orange Door is operating and to establishment processes. It also focusses on systems, practices and processes that are being trialled or in the early stages, so we expected to identify a range of issues that are not working well, or are not fully developed yet.

This evaluation, by the nature of its focus is a process evaluation. We are focused on examining a range of service infrastructure, systems, processes and procedures during their early implementation to assess if these will contribute to the success of The Orange Door. It is not an outcome evaluation, focused on the impact of The Orange Door on clients. This will in time be an important focus for FSV and partner agencies to evaluate, but it is not possible to conduct a summative outcome evaluation at this time. As such, much of this report focuses on processes and systems – which may seem somewhat removed from the intent of the reforms to keep victims safe and build stronger families. However, processes and procedures in The Orange Door areas are the steps and tools needed in order to achieve the outcomes. At this early stage – it is important that these systems and processes are working effectively to meet the needs of The Orange Door clients.

FSV has specifically commissioned a developmental evaluation at the earliest possible stage of The Orange Door roll out in order to learn about what is working and what needs to be altered, rapidly, for The Orange Door to be successful. This needs to be acknowledged in the context of our evaluation findings that do identify some aspects that need to be strengthened or altered as the service model evolves. This evaluation is to our knowledge, only the second developmental evaluation commissioned by the Victorian Department of Health and Human Services. Given that FSV is itself a new agency, the commissioning of an evaluation of this type demonstrates a commitment by FSV to focus efforts on quality improvement. We believe that this type of perspective and approach will assist FSV well over the coming years as the agency guides the significant reforms it has responsibility for.

Evaluation purpose and framework

The purpose of the evaluation is to:

• Identify key lessons and opportunities for improvements to establishment activities and processes to support implementation of the second roll out of The Orange Door in 2019 and beyond.
• Identify key lessons and opportunities for improvements to the initial service offering in the first four areas, to inform enhanced operations across existing and new areas.

Early findings were provided to FSV throughout the course of the evaluation to provide opportunities for changes to be applied to current operations, as well as establishment processes for the next rollout of The Orange Door.

The scope of the evaluation was defined by five core evaluation questions developed with FSV, as presented below:

1. Establishment: What changes can be made to establishment approaches to improve future implementation and operations of Orange Door Sites?
2. Alignment within intention: To what extent is the Orange Door operating as intended by the initial service offering?
3. Client experience: To what extent is the Orange Door initial service offering contributing to improving client experience and client and system outcomes?
4. To what extent does the Orange Door workforce have the resources, capacity and specialist expertise to undertake the full range of functions articulated in the initial service offering?
5. Local integration and coordination: To what extent is the Orange Door supporting service integration and coordination at the local level?

To ensure the evaluation findings are practical and useful, PwC revised an initial evaluation framework in collaboration with FSV to guide all evaluation activities and outline the evaluation questions, indicators, measures and methods of data collection for the 2018 evaluation. The evaluation framework was based on the initial framework provided by FSV, The Orange Door: Initial Evaluation Framework, which was intended to inform the design of all evaluations of The Orange Door.

This report seeks to answer the core evaluation questions and associated lines of inquiry. The complexity of the evaluation has meant that many themes and findings cut across and are related to multiple evaluation questions and lines of inquiry. In addition, there is some commonality between indicators. As such, this report has made references between content across sections to avoid repetition.

Evaluation methodology

A mixed methods approach was taken to inform The Orange Door 2018 Evaluation, and comprised of a range of data collection methods. Figure 2 summarises the methods used to inform this report. A detailed methodology is provided in Appendix B.

Figure 2: Evaluation methods

In line with our professional standards for evaluation, we have applied the following principals to our analysis and reporting of evaluation evidence:

• Our evaluation analysis triangulated data to develop the common themes and findings, as well as to validate findings that may have arisen from other data sources.
• We analysed data from interviews with multiple individuals and workforce roles in The Orange Door before determining the common theme or finding.
• We analysed data across The Orange Door areas to validate evaluation themes and findings, and noted where differences were observed.
• Throughout the report we have identified where we had conflicting data, or where we were not confident in the strength of evidence. We have at times, also identified the type of additional data that would be required to develop definitive and robust conclusions.
• At times, we have provided quotes from practitioners or clients to illustrate a key theme or finding. However, findings are generated from multiple evaluation sources rather than the viewpoint of an individual (no matter how senior the stakeholder).
• At times, we have deliberately not identified the role of an interviewee where doing so is likely to identify the individual.
• We have provided quantitative data in certain parts of this report to illustrate our findings against a particular line of inquiry. However, to ensure ‘readability’ of this report by a lay audience, we have not presented the findings of all quantitative analysis conducted for this evaluation.

Report structure

The report is structured to answer each of the evaluation questions methodically.

• Section 2 provides the overarching context for the evaluation
• Section 3 describes and assesses the establishment processes and governance arrangements for The Orange Door (Evaluation Q1).
• Section 4 explores the fidelity of The Orange Door service model at the four areas, and analyses the extent to which it is delivering on its intended objectives (Evaluation Q2)
• Section 5 examines client experience and outcomes (Evaluation Q3)
• Section 6 considers the workforce (Evaluation Q4)
• Section 7 examines service integration and coordination within The Orange Door (Evaluation Q5)
• Section 8 provides conclusions and identifies opportunities to inform improvements to current operations – at both the area and state level – as well as changes to implementation of The Orange Door in the next areas
• Appendices containing supporting materials and further detail regarding key aspects of the evaluation.

[1] FSV, ‘Support and Safety Hubs: Service model’, April 2018.