The scale of change that The Orange Door has faced in the early establishment phase cannot be underestimated. The context of the family violence reforms and the early stages of the roll out of The Orange Door are critical when considering the findings of this evaluation. It was not expected that The Orange Door would be operating seamlessly in an integrated manner at the time in which the 2018 evaluation was undertaken. The Orange Door represents a new way of working for FSV and the partner agencies and FSV is a new agency itself. Integrated practice with victim survivors, perpetrators and child and family services is a new model, based in new facilities, with new processes, and new staff.
There has been enormous effort on the part of FSV, all agencies and their staff involved to establish The Orange Door and this was evident to our evaluation team during field work and stakeholder interviews. The findings recognise this effort, and the opportunities for improvement identified in the report are intended to further support and develop the effective delivery of integrated services and integrated practice in future areas.
It is the nature of a developmental evaluation to allow for rapid change to implementation in response to emerging evaluation findings. FSV have already made changes, or are addressing some issues identified in our findings and subsequent recommendations. Examples of this are facilities upgrades, a new ‘go live’ criteria before services commence, the development of a workforce strategy, and further CRM iterations, all undertaken during the period that the evaluation was conducted.
In line with a developmental approach to evaluation of a program in the very early stages of implementation, we frame our ‘recommendations’ in terms of opportunities we have identified. Making ‘recommendations’ implies that FSV are unaware of issues or do not already have a staged plan to address these. It also assumes that The Orange Door will not have evolved between the time of our evaluation field work and the provision of this evaluation report. As such, we frame our advice in relation to the opportunities we see are available at this point in time to strengthen The Orange Door model and initial service offering, and further develop establishment processes for subsequent areas as The Orange Door is rolled out across Victoria.
Overarching themes of our findings and opportunities
The many findings of the evaluation can be grouped into a number of overarching themes that relate to more than one evaluation question or line of inquiry, or relate to only one The Orange Door area.
Overarching themes of the evaluation findings are:
- The commitment by FSV, partner agencies and the workforce to the concept of integrated practice and The Orange Door model has been sufficient to overcome most of the start-up challenges.
- Foundational key concepts in The Orange Door model are clearly evident and are defined in a range of practice guidance and foundational documents. These need refining to operationalise them in practice.
- Consistency of practices and processes within and between The Orange Door areas now needs focus. Clearer operational guidance is needed around the processes and tools that can be standardised across The Orange Door areas as opposed to those that allow for local variation. This may become clearer as more areas are established.
- The volume of change in practice posed by The Orange Door should not be underestimated and needs to be factored into planning and development in the first 12 months in any new area where The Orange Door is established.
- The integration of perpetrator services in The Orange Door requires focused effort to include these elements of the service system in a more effective way.
- The Orange Door is still perceived as primarily a family violence-focused initiative by child wellbeing practitioners.
- CIP has been positive for information sharing to inform risk assessment and practice.
- Outcomes and the experience of clients was unable to be assessed with confidence as part of this evaluation. Further exploration of how to validate findings through client feedback should be a focus of future evaluation effort.
- Data quality issues in this early stage of implementation have had an impact on service delivery and performance management within each The Orange Door area.
Assessment of opportunities
The developmental nature of the evaluation, the scale of the evaluation and the multiple lines of inquiry have understandably generated a significant number of findings and opportunities. It is not possible for FSV and/or partner agencies to implement all of these, and as such, we have used a matrix to assess the value of each opportunity and our view on the effort of implementation. In this assessment we have defined:
Value: The contribution made towards The Orange Door service model and associated reforms.
Implementation: The likely degree of implementation complexity.
Figure 13 shows this matrix and situates findings within different quadrants using a traffic light system.
Figure 13: Matrix to assess opportunities for improvement to The Orange Door
The relationship of the opportunities to the evaluation questions
The opportunities are presented in order to address each of the evaluation questions:
cross-referencing an opportunity for improvement against key findings from the evaluation
identifying our assessment of the value and implementation effort of each opportunity using a traffic light system
identifying if the opportunity relates to specific operational areas, future areas or the broader FSV program of work.
The role of FSV as a ‘system steward’ during early implementation of The Orange Door
At the foundation of The Orange Door concept is a need for collective leadership through the organisations in The Orange Door partnership – where each organisation brings their specific skills and services to the collaborative effort of The Orange Door intake and assessment system. It is to be expected that as the model evolves and service processes and procedures become embedded, that each HLG will move from their early focus on operational issues to a more strategic focus on localising the service to meet the needs of their clients. This is aligned with the concept model of The Orange Door – and in time, this should be achievable.
However, at this early stage of implementation, uncertainty over how to operationalise the model and the pressures of establishment indicate that FSV needs to play a larger role as ‘system steward’ than originally anticipated. The nature and outcomes of The Orange Door will be influenced by many different stakeholders (including partner agencies) working together. In this context, FSV will need to oversee the ways in which The Orange Door is being implemented and attempt to steer the system more directly towards certain outcomes, to reinforce positive attributes of the model, and to identify and correct any process and operational issues that detract from positive client outcomes. This goes to the issue of which operational issues of The Orange Door needs statewide standardisation and what can be localised for each area. In the early days of The Orange Door, it is advised that FSV adopt a more overt system steward role to lead The Orange Door more directly towards the outcomes we all hope The Orange Door can achieve for Victorians. With this in mind, we have identified a range of opportunities for improvements to be made, and have deliberately indicated where we believe FSV should take a leadership role in progressing these. This does not mean that FSV alone are responsible, or even that they can alone make these improvements. Many will require collaborative efforts by the partner agencies but at this stage, FSV have a role to lead the partnerships in addressing these improvement areas.
Download conclusion and opportunity summary:
Updated