Context and background

The Orange Door is a new way for women, children and young people experiencing family violence, and families who need assistance with the care and wellbeing of children to access the services they need to be safe and supported. It is important to acknowledge that The Orange Door is one of many critical reforms currently underway in Victoria and that this is a time of immense change in the family violence and family and child wellbeing service systems. Other critical reforms that are currently being implemented in Victoria include the introduction of the Family Violence Information Sharing; the Child Information Sharing Scheme; Roadmap for Reform: strong families, safe children; Free from violence – Victoria’s prevention strategy; initiatives as part of Building from Strength: 10-year industry plan for family violence prevention and response; rollout of the Specialist Family Violence Courts model; and Family Violence Risk Assessment and Risk Management Framework (MARAM).

The scale of change associated with the reforms and the dynamic nature of reform implementation are important in understanding the establishment and initial operations of The Orange Door in the first four areas. The Orange Door plays a pivotal role in shifting from the existing models that have been identified by the Royal Commission into Family Violence as not meeting the needs of families and victims of family violence, towards linked up, integrated responses to family violence and children and family wellbeing concerns. To this end, the importance of The Orange Door to the overall reforms cannot be underestimated – nor can the significant change management for services, their workforces and the broader service system – required for Victoria to reach this vision.

The magnitude of Family Safety Victoria’s (FSV) task to progress a new service delivery model and establish four new integrated services, secure accommodation and contract partner agencies in 9 to 12 months, while establishing its own agency at the same time, was an enormous undertaking. We note that the Royal Commission recommended introduction of the Support and Safety Hubs by 1 July 2018 (recommendation 37), which placed pressure on FSV to establish The Orange Door areas quickly after its own establishment in July 2017.

Evaluation approach and framework

Family Safety Victoria (FSV) FSV engaged PricewaterhouseCoopers Consulting (Australia) Pty Ltd (PwC) in partnership with PwC’s Indigenous Consulting (PIC) and Australian Catholic University (ACU) to conduct an evaluation of the first four operational areas of The Orange Door in September 2018. This evaluation focuses on the establishment, operations and initial service offering of The Orange Door in Barwon, Bayside Peninsula (BPA), Mallee and North East Melbourne (NEMA) areas that commenced operations between May and July 2018. The evaluation seeks to identify key lessons and opportunities for improvements to establishment activities to support the implementation of The Orange Door in 2019 and beyond, and to inform enhancements across existing and new areas.

This evaluation is occurring at the very start of a significant change process. As such, we would anticipate the foundational building blocks underpinning change to be evolving as Victoria continues to implement the major family violence and child and family service reforms.

We were engaged to undertake a developmental approach to the evaluation to gain insights about the evolution of The Orange Door. Developmental evaluation is an evaluation approach that can support understanding of new innovations, initiatives and programs that are being implemented in complex, changing or uncertain environments. The approach facilitates real-time testing of hypotheses, rapid iterations and changes to program design, with feedback on the initiative allowing for rapid and evolving improvements to be made (even while the evaluation is still occurring). The developmental approach for this evaluation enabled early insights to inform changes to current operations rapidly and to identify improvement for future areas where The Orange Door will be rolled out.

Given the recent establishment of The Orange Door and the evolving nature of service operations while establishment processes are still ongoing, it is not possible to conduct a traditional ‘summative’ evaluation that makes conclusions about system outcomes and can answer the question ‘does the model work’? As such, a developmental evaluation approach allows the rigour of traditional methods of evaluation (such as secondary data analysis, interviews, etc) to be applied while accommodating the dynamic environment in which The Orange Door areas are evolving in terms of operations.

This evaluation, by the nature of its focus is a process evaluation. We are focused on examining a range of service infrastructure, systems, processes and procedures during their early implementation to assess if these will contribute to the success of The Orange Door. It is not an outcome evaluation, focused on the impact of The Orange Door on clients. will in time be an important focus for FSV and partner agencies to evaluate, but it is not possible to conduct a summative outcome evaluation at this time. As such, much of this report focuses on processes and systems – which may seem somewhat removed from the intent of the reforms to keep victims safe and build stronger families. However, processes and procedures in The Orange Door are the steps and tools needed in order to achieve these outcomes. At this early stage – it is important that these systems and processes are working effectively to meet the needs of clients of The Orange Door.

A developmental evaluation is by its nature, focusing attention on systems, practices and processes that are being trialled or in the early stages of implementation. We would expect to identify a range of things that are not working well, or are not fully developed yet. This evaluation, is to our knowledge, only the second developmental evaluation commissioned by the Victorian Department of Health and Human Services (DHHS). Given that FSV is itself a new agency, the commissioning of an evaluation of this type demonstrates a commitment by FSV to focus efforts on quality improvement, and when needed, to reflect on how the agency’s performance contributes to the overall success of the reforms. We believe this type of perspective, approach and willingness to learn, will serve FSV well over the coming years to guide the significant reforms the agency has responsibility for.

Underlying the work of FSV and our evaluation is a commitment to see the service system evolve to respond effectively and efficiently to families who need support, to hold perpetrators to account and to help keep women and children safe from family violence. To this end, this initial evaluation of The Orange Door needs to focus sharply on the structures and systems that are essential to be in place to facilitate long term success of The Orange Door. Key design decisions of the service model were out of scope for The Orange Door evaluation including whether the partnership model was the right approach for service delivery. The scope of the evaluation was defined by five core evaluation questions developed in collaboration with FSV:

1. Establishment: What changes can be made to establishment approaches to improve future implementation and operations of Orange Door Sites?
2. Alignment with intention: To what extent is the Orange Door operating as intended by the initial service offering?
3. Client experience: To what extent is the Orange Door initial service offering contributing to improving client experience and client and system outcomes?
4. Workforce: To what extent does the Orange Door workforce have the resources, capacity and specialist expertise to undertake the full range of functions articulated in the initial service offering?
5. Local integration and coordination: To what extent is the Orange Door supporting service integration and coordination at the local level?

Evaluation methodology

A mixed methods approach was taken to inform this evaluation, which comprised of a range of data collection methods summarised in this figure.

Overarching evaluation findings

Overarching themes of our evaluation findings are:

1. Given the significant time pressures and establishment context of FSV itself, establishing The Orange Door in the first four areas has been an enormous achievement. The areas for improvement are what would be expected in this stage of implementation, recognising the constraints that FSV was operating under.
2. The commitment by FSV, partner agencies and the workforce to The Orange Door model and the concept of integrated practice has been sufficient to overcome most start-up challenges. This commitment to the vision and concept is a strong foundation for the roll out of The Orange Door across Victoria.
3. Foundational key concepts as part of The Orange Door model have been developed and exist in a range of practice guidance and foundational documents. Next steps will need to focus on refining these further and ensuring that they are fully operationalised in practice at The Orange Door.
4. Clearer operational guidance is needed around the processes and tools that can be standardised across The Orange Door areas as opposed to those that allow for local variation. Consistency of practices and processes within and between the areas needs to be a focus now for the areas that have been established. This may become clearer as more areas are established.
5. The volume of change in practice posed by The Orange Door cannot be underestimated and the sequencing of implementation and culture change needs future consideration in the first 12 months before commencement of any new area.
6. The integration of perpetrator services in The Orange Door requires focused effort to include these services in a more effective way.
7. The Orange Door is perceived as a family violence-focused initiative by child wellbeing practitioners, and focused effort is needed to integrate child wellbeing and family violence services.
8. Even in these early stages of operations, the Central Information Point (CIP) has been very positive for information sharing to inform risk assessment and practice at The Orange Door.
9. Outcomes and the experience of clients was unable to be assessed with confidence in this evaluation. Further exploration of how to validate findings through client feedback is needed. We understand that involving clients in program and service planning forms part of an existing program of work by FSV. A specific exploration of how clients with intersectional characteristics are experiencing The Orange Door, including how they are referred and how The Orange Door and targeted services work together to provide a response would be useful.
10. Data quality issues in this early stage of implementation have had an impact on service delivery and performance management within each of The Orange Door areas. Even during this evaluation, data improvements were evidenced through planned upgrades to the information technology system however a focus is needed on continued improvement in order to effectively monitor how The Orange Door is functioning to meet the needs of Victorian families.
11. Suggested areas for future investigation include: adequacy of risk assessment and risk management for clients over time; service system interface including services referring into The Orange Door and services receiving referrals from The Orange Door; the service system’s perceived scope of The Orange Door; client experience including clients who disengage from service; and clients experiences of The Orange Door – including the intersectional experiences of clients from diverse cohorts.

Detailed evaluation findings and opportunities

It is important to realise that where evaluation findings indicate that a process or system is not working as described or intended in the foundational documents for The Orange Door, that these be viewed at this early stage as lessons in the evaluation of the reforms, rather than as ‘failures’ of The Orange Door. This is entirely consistent with usual findings from developmental approaches to evaluation of early initiatives, pilots or demonstration trials. Hence, it is not productive at this point of implementation to ‘benchmark’ The Orange Door against outcomes or achievements that would be expected only when the model is fully evolved and operational across all 17 areas.

Q1. What changes can be made to establishment approaches to improve future implementation and operations of The Orange Door sites?

A number of changes to establishment processes were identified by the evaluation, including refinements to existing and planned operations and guidance. We have identified key findings related to key lines of inquiry for our evaluation.

Condensed timeframes

Extensive foundational work was required to be completed within a compressed timeframe to operationalise The Orange Door. This evaluation found that the timeframe negatively impacted on a range of establishment and operational activities. Evidence of the impact of this compressed timeframe was still being seen at The Orange Door areas at the time of this evaluation, despite each area managing to provide the full suite of foundational services.

Change management processes

Significant change management to support the implementation of The Orange Door service model is needed by both FSV and partner organisations. Initial focus has, as would be expected, been on establishment and operational activities in a compressed timeframe and now there is a need to shift the focus to change management processes that can strategically move the workforce and sector towards the vision outlined in the statewide concept for The Orange Door.

Workforce mobilisation

The Orange Door areas are not currently at full contingency of staffing. There have been a range of challenges in recruitment of appropriately skilled staff and in retaining those staff which are being addressed by FSV and partner agencies through the development of a workforce strategy for The Orange Door. There is a significant opportunity to increase clarification and standardisation of roles and responsibilities across partner agencies.

Initial induction training was provided and the content of this induction training has already evolved in response to feedback about it needing to be tailored to specific workforce needs. Feedback from practitioners at the time of field work was that additional content on the practice model, including how perpetrator services are integrated within The Orange Door, was needed.

Foundational documents

The foundational documents[1] that provide overarching direction for Orange Door are not well understood or used by practitioners at this point in time. Further operational ‘translation’ of these documents is needed to assist the workforce to understand how to operationalise the model. There is an opportunity for FSV to lead a process to co-develop (with practitioners), practical operational procedures at Orange Door that define which functions and processes across areas require state-wide standardised processes, and which can be adapted in response to local needs. For functions that require state-wide consistency, a program of work is needed to define process and procedures and how integration is to be operationalised.

Sequencing before service commencement

Opportunities for improving sequencing before service commencement include: revising the ‘go live’ criteria including levels of staffing contingency, changing the sequencing of implementation of key roles and capabilities, providing additional time for The Orange Door workforce to embed processes and systems prior to service delivery commencing and considering the different staffing/workforce resource needs for the first six months of operation compared with ongoing needs. A range of physical infrastructure issues impacted operations and colocation of the workforce in some areas at service commencement. These have been, or continue to be addressed by FSV.

Collaboration and governance

Collaboration is central for internal information sharing to support service delivery. Despite early stage challenges with establishing these connections, information sharing across partner organisations represented in The Orange Door is already happening within each area primarily through the client record management system (CRM), CIP, colocation and multidisciplinary working arrangements and through access to employing agencies’ client record systems.

Pre-existing working relationships have accelerated collective leadership in governance arrangements. The Partnership Agreements formalise the governance for each area and include an agreed vision and direction for The Orange Door. However, the realisation of the vision and intent of the Partnership Agreements is complicated when there are numerous employing agencies involved in the partnership which may have different interests in and expectations of The Orange Door. Over time it would be expected that the partnership approach would mature and there are opportunities for FSV to accelerate this process. Refinements to partnership and service agreements may be needed after the initial implementation phase, noting that there is a 12 month review built into the agreement which is currently being progressed by FSV.

The matrix management model has been challenging to implement in the context of an innovative service model and new partnership structure very early in the implementation. As roles and responsibilities become clearer, and accountability and decision-making are defined and operationalised in each of The Orange Door areas, the matrix management model may become more effective for the workforce.

Q2. To what extent is The Orange Door operating as intended by the initial service offering?

This evaluation found that The Orange Door offers all functions required to operate and provide service to clients as outlined in the service model. As expected at this very early stage of implementation, some of these functions require refinement and/or standardisation to work effectively and efficiently for all clients. However, early feedback from some clients indicates that, as intended, some clients are experiencing ‘seamless’ services. Feedback from a number of clients also indicates that there is room to improve the knowledge and understanding of service system referrals by The Orange Door practitioners.

Feedback loop – or ‘review and monitoring function’

The Orange Door has a key role in helping to ensure that services are effective and that positive outcomes for clients are maximised.

The ‘review and monitoring’ function articulated in the service model is not yet fully operational. The service model indicates that The Orange Door will, in time, monitor case plans and client outcomes through strong feedback loops with core services and improved data and information systems to collect information about the status of referrals and interventions provided to clients. This review and monitoring function will move The Orange Door beyond being merely a ‘front-door’ for a siloed service system and will help to drive integration and more effective service delivery. While this review and monitoring function is not currently established, this was not intended to be operational in the first four areas in the establishment phase and can be expected to become fully operational over time as Orange Door rolls out across the state.

Demand for The Orange Door

A combination of high demand, staffing vacancies and a lack of clarity by The Orange Door workforce about how to operationalise appropriate screening, identification and triage processes has resulted in significant wait times for some clients in the early stages of implementation. As the model is new (and never implemented anywhere in the world to our knowledge), demand could not have been accurately predicted prior to commencement of service. This will need to be monitored to determine if this is a point in time or systemic issue.

FSV will be able to update demand projections for the next areas of The Orange Door based on learnings from this evaluation. Demand projections would also need to consider caseload and staffing noting the specialty mix across the three client streams needed and the different time/volume of resources needed of cases from different client streams (as found in this evaluation).

There are also a range of opportunities for FSV to improve the monitoring of workflow and demand. For example, FSV may consider investigating the extent to which child wellbeing cases take longer to progress through The Orange Door than family violence-related cases, or explore how data reporting from CRM can show ‘tracking’ of clients’ pathways through The Orange Door from initial contact through to case closure. This could provide insight into potential causes of demand pressures within the overall system.

Connection with the broader service system

The capacity of the system to provide tailored service responses is currently limited by The Orange Door workers’ knowledge of services in the area. It will be enhanced once the ‘review and monitoring’ function is implemented and the Hubs Leadership Group (HLG) can better understand the capacity of the service system to receive referrals from The Orange Door or action allocation of services and respond accordingly. Data limitations mean that we are unable to make firm evaluative judgements about whether the service responses are tailored to need and risk at this point in time.

Screening, triage and assessment and ‘case management’

In these early stages of the roll out, The Orange Door workers would benefit from clearer guidance on screening, triage and assessment using examples from practice. Examples could include detail of where the role of The Orange Door in screening, intake and assessment ends, and where case management begins. This needs to be considered in light of the findings that many of the (limited sample of) clients interviewed expressed a desire for greater support from The Orange Door, even after services had been allocated to them, or after they had been referred to other services. There are opportunities for FSV to prioritise its existing work plan and provide guidance to The Orange Door workforce about the interventions and active engagement and risk monitoring that is appropriate for the different situations experienced by clients (notwithstanding that each case and client is unique).

Risk assessments are undertaken

The evaluation found that risk and needs assessments appear to be undertaken (albeit sometimes inconsistently within each area of The Orange Door and/or between practitioners). Consistency in recording risk assessments requires monitoring by FSV to ensure that the quality and consistency of data recording continues to grow as service delivery and the model evolves. Validation through client feedback is required to determine if risk and needs assessments lead to responses that are tailored to the needs of the client.

The foundational work carried out by FSV and partner agencies means that practitioners have access to a range of tools to assist them to assess risk. The introduction of the MARAM is expected to improve the consistency of risk assessment and risk management at The Orange Door. The child wellbeing tools are not yet standardised across areas (nor was there a standardised tool prior to the establishment of The Orange Door).

A commitment to client-centric practice

note, the evaluation found that practitioners have a strong commitment to a client-centric approach to practice and the concept of integrated practice. Even in this early stage, practitioners report practice aspects that enable the voices of Aboriginal clients for example, to play a strong role in service choice and in determining the support provided. However, we also found that client preferences and choice appear more likely to be taken into account in relation to referral rather than assessment and planning at this early stage, and client preference and choice is more likely to be offered to adult clients of The Orange Door rather than child clients.

Q3. To what extent is The Orange Door initial service offering contributing to improving client experience and client and system outcomes?

A number of lines of inquiry were investigated to determine how The Orange Door was contributing to client and system outcomes.

Accessibility and awareness of The Orange Door

The evaluation found that the physical premises of The Orange Door have been established in locations that are visible and accessible for most clients and in close proximity to community service providers and public transport. Data limitations prevent us making evaluative judgements on whether The Orange Door is accessible and responsive to the needs of clients with diverse needs; and further investigation of this is required.

The physical space at The Orange Door appears to be broadly functional and fit for purpose and FSV has undertaken significant work on the facilities. However, some physical site specific safety concerns (primarily for the workforce that relate to security issues) remain and have been communicated to FSV during the course of this evaluation. Our understanding is that FSV continues to undertake physical safety assessments (for clients and for the workforce) at each site independently. There are safety benefits and disadvantages of each site that need to be balanced in considering how to improve safety for clients and the workforce, or in selecting a location. For example, a main street increases access for clients but also visibility of the workforce and clients.

The physical locations and branding of The Orange Door support community awareness of the role of The Orange Door. However, there is only emerging understanding of the role as an intake and assessment service (notably by those organisations who had previously referred to family violence or Child FIRST services). There is an opportunity to monitor community awareness of the role of The Orange Door over the first 12 months of operations of the first four areas to determine what, if any, clarification or awareness raising within the community might be needed.

The early implementation of The Orange Door has meant that publicly identifiable access points have not been operationalised at the time of this evaluation and will be established in line with the planned roll out.

The Orange Door within the broader service system and network

The evaluation found that in this early stage, the role of the Service System Navigator has not been sufficiently defined and understood by The Orange Door workforce to enable development of service system pathways and connections. The focus of the role will likely shift from operations to the service system once operational issues are resolved.

While information sharing within The Orange Door is occurring, there appears to be limited information shared with clients of The Orange Door at this point in time – and practitioners report being unclear about what information they are allowed to share with clients under the new legislation. This is likely to be due to practitioners being risk averse about inadvertently sharing information obtained through the family violence and child information sharing legislation.

Evolving understanding of ‘keeping perpetrators in view’

There is an evolving understanding about what it means at an operational level for The Orange Door to ‘keep perpetrators in view’. Work with perpetrators by The Orange Door is limited as it is dependent on their voluntary engagement with the service. Many potential clients do not respond to attempts by The Orange Door to contact them. This was anticipated, is common in the existing system, and is reflected in the guidance documents for The Orange Door. FSV and partner agencies could leverage the skills and experiences of Aboriginal practitioners and services who have a history of working in an integrated way to improve the capacity of the rest of the workforce to work in an integrated way to keep perpetrators in view.

Information transfer as a key to collaboration and client experience

While information sharing within The Orange Door is occurring, there appears to be limited information shared with clients of The Orange Door at this point in time – and practitioners report that they are unclear about what information can be shared with clients under the new legislation. This is likely to be due to practitioners being risk averse about inadvertently sharing information obtained through the family violence and child information sharing legislation.

CRM data on connection to services is limited and appears to be inconsistently recorded at this early stage. As such, the evaluation was unable to determine if service responses for clients are tailored based on shared information.

CIP has been effective for improving risk assessments and outcomes for clients, particularly those who were previously unknown to The Orange Door. However, there is a need for clearer operational clarity for the workforce as to when and how often CIP requests can be made.

The Orange Door workforce is collaborating on assessment and planning supported by physical colocation and personal working relationships. The collaboration is inconsistent across the areas and not yet embedded within practice, especially when working with perpetrators. Collaboration on assessment and planning is most evidently displayed by Aboriginal practitioners and aligns with their community-led approach to service provision.

Q4. To what extent does The Orange Door workforce have the resources, capacity and specialist expertise to undertake the full range of functions articulated in the initial service offering?

A number of lines of inquiry were investigated to explore the workforce’s capacity to undertake their role in The Orange Door.

A commitment to the concept of The Orange Door

The evaluation found that there is a commitment from The Orange Door workforce to develop capability in other specialisations and work in an integrated manner that improves outcomes for clients. A culture of collaboration is evolving at The Orange Door in line with the vision described in FSV’s Interim Integrated Practice Framework and tends to happen organically between individuals, and generally between certain disciplines.

A lack of understanding about how to operationalise integrated practice

Within the positive culture of collaboration however, at this point in time, there is a lack of clarity and understanding from practitioners about how to operationalise integrated practice. Despite the written guidance and documents produced about The Orange Door – concepts, framework, operational guidance, etc prior to commencement, the evaluation found that the format and volume of these documents deterred practitioners from engaging with and understanding key information to inform how they integrate services. There is a key opportunity for FSV to lead a process to co-design with practitioners across the disciplines and/or relevant state-wide stakeholders to co-produce state-wide guidance on how to operationalise integrated practice at The Orange Door. There is also the opportunity for FSV to focus on integrated practice skill development in the workforce strategy currently in development.

The ability of the current workforce at The Orange Door to work in an integrated manner is further currently impacted by:

• Staff shortages and high demand.
• Limited structural and formal opportunities for practitioners to learn about other disciplines and practice approaches. Time dedicated for reflective practice for teams and further professional development and training would support integrated practice.
• Limited understanding of The Orange Door workforce about the role of the Integrated Practice Lead and the Service System Navigator.
• Limited understanding about the role of the Advanced Family Violence Practitioner role. Although this role was established to provide guidance to all practitioners working in family violence, there is a perception that the Advanced Family Violence Practice Lead role pertains only to practice with victim survivors. As perpetrator service responses are an emerging field of practice, increased leadership in this practice area may be required to build the capacity of the family violence and child wellbeing workers to identify and respond to risk posed by perpetrators.

There is a key opportunity for FSV to provide further operational clarity around core roles. This might be through working with partner agencies to agree on standardised position descriptions for all roles (including existing areas) for all partner agencies to use as the basis for recruitment.

Q5. To what extent is The Orange Door supporting service integration and coordination at the local level?

A number of lines of inquiry were investigated to explore how The Orange Door was supporting service integration and coordination at a local level. The focus of the evaluation was on internal information transfer and sharing within The Orange Door (and did not investigate the sharing of information to external agencies beyond data on referrals in and out of The Orange Door).

Information sharing as a key mechanism for coordination

Information sharing is contributing to coordinating responses to family violence and child wellbeing within The Orange Door through key mechanisms including the CRM, use of partner agencies client databases and CIP. This evaluation did not investigate information sharing between The Orange Door and external practitioners (beyond processes for referral in and out of The Orange Door). The CRM, CIP and partner agencies client databases are core tools for coordinating and integrating services and improvements are likely to continue as planned upgrades to the CRM are made in response to feedback from practitioners. The intended extent of information sharing within The Orange Door would be strengthened, particularly in relation to duplication of effort, access to information about who clients can connect with, and a feedback loop (a review and monitoring function) with the range of services in the area and The Orange Door (to communicate the outcomes of referrals to services). Information sharing is a key foundation for collaboration, service integration and coordination of support.

Connecting clients to other services can be delayed at times. The reasons for delay may include: difficulties in obtaining information required for assessment; lack of knowledge and clarity among The Orange Door practitioners about the services available; and the lack of an established ‘review and monitoring’ function at this point in time.

Establishing processes and procedures for interfaces with external agencies

At this early stage of implementation of The Orange Door, at an operational level, the responsibility for establishing interfaces (including information sharing) with external agencies appears to be pooled between the Service System Navigator, the Hub Leadership Group (HLG), the Hub Manager and FSV, meaning no one person or group has overall accountability. This may be appropriate in the longer term, but in initial implementation stages, it would beneficial for one role/group to assume leadership.

Possible differences in service needs for child wellbeing cases

The extent and nature of assessment required in child wellbeing cases generally means that the time between initial intake, assessment and then referral to services may be longer for people experiencing child wellbeing issues than for other clients of The Orange Door. This requires further investigation to determine the cause and length of time.

A need for focus on the impact of The Orange Door on the broader service system

This evaluation did not focus on the capacity of the service system within which The Orange Door is established. This is important contextual information for determining the extent to which The Orange Door is supporting service integration and coordination at the local level. There are opportunities for FSV to monitor emerging ‘gaps’ in service capacity in the service sector to which The Orange Door refers. There is also the opportunity for The Orange Door to include a focus on aligning the new family violence risk identification and risk management framework and resources (the MARAM) with The Orange Door and local services in order to improve integration in risk assessment and risk management across the service system, and we understand this this is part of FSV’s current work plan.

Opportunities for improvement and the role of FSV as a system steward

At the foundation of The Orange Door concept is a need for collective leadership through the organisations in The Orange Door partnership – where each organisation brings their specific skills and services to the collaborative effort of The Orange Door. It is to be expected, that as the model evolves and service processes and procedures become embedded, that each HLG will move from their early focus on operational issues to a more strategic focus on how to localise the service to meet the needs of their clients. This is aligned with the concept of The Orange Door – and in time, this should be achievable.

However, at this early stage of implementation, uncertainty over how to operationalise the model combined with the pressures of establishment, means that FSV may need to play a larger role as ‘system steward’ than originally anticipated. The system and outcomes of The Orange Door will be influenced by many different stakeholders (including partner agencies) working together. In this context, FSV will need to oversee the ways in which The Orange Door is being implemented and attempt to steer the system more directly towards certain outcomes, to reinforce positive attributes of the model and to identify and correct against any process and operational issues that detract from positive client outcomes. This goes to the issue of which operational processes of The Orange Door needs statewide standardisation and which can be localised for each area.

In the early days of The Orange Door, it is advised that FSV adopt a more overt system steward role to lead The Orange Door more directly towards the outcomes we all hope The Orange Door can achieve for Victorians. With this in mind, we have identified a range of opportunities for improvements to be made, and have deliberately indicated where we believe FSV should take a leadership role in progressing these. These cover a range of different levers including management, funding, directing operations, workforce development support etc. This does not mean that FSV alone is responsible, or even that they can alone, make these improvements. Many will require collaborative efforts by the partner agencies but at this stage, FSV has a role to lead the partnerships in addressing these improvement areas. It will be through the continued collective investment of the many partner organisations who form The Orange Door, and through FSV leadership, that the vision of The Orange Door can be achieved

[1] Including the Support and Safety Hubs: Statewide concept (July 2017), Support and Safety Hubs: Service model (April 2018), Support and Safety Hubs: Service specifications (April 2018), Support and Safety Hubs: Interim operational guidelines (April 2018), Support and Safety Hubs: Interim integrated practice framework (April 2018) Support and Safety Hubs perpetrator practice guidance (April 2018)

In line with this, we deliberately use gendered language to reflect that the majority of family violence is perpetrated by men towards women.

Aboriginal

Throughout this document, we use the term Aboriginal to refer to both Aboriginal and Torres Strait Islander people. Use of the terms Aboriginal, Aboriginal and Torres Strait Islander and Indigenous are retained in the names of programs and initiatives and unless reported otherwise are inclusive of all Aboriginal and Torres Strait Islander peoples.

ACU

Australian Catholic University.

AFM

Affected Family Member identified in an L17 referral.

Allocation

The process for allocating core services to clients.

Assign or assignment

The internal designation of a practitioner to a client in The Orange Door during intake. Note that practitioners call this ‘allocation’, however, to distinguish this from the formal definition, we have termed it assignment.

Barwon

The Orange Door in Barwon, with the main physical site situated in Geelong.

BPA

The Orange Door in Bayside Peninsula area (BPA), with the physical site situated in Frankston.

Case (in The Orange Door)

Once a person has been screened (see below) and identified as requiring a service response, they will have an individual case relating to them.

CIP

The Central Information Point.

CRM

Client Relationship Management system used by The Orange Door.

CSO

Client Support Officers, who greet walk-in clients to The Orange Door and in some areas, answer calls to The Orange Door phone number.

DHHS

The Department of Health and Human Services.

Direct contact

Refers to clients of The Orange Door directly referring themselves by calling, emailing or physically presenting at The Orange Door.

Diverse

Refers to the variety of different needs clients may have based on the different factors that make up an individual, including age, gender, culture, religion, social status and sexual orientation.

DPC

The Department of Premier and Cabinet

Employing agencies

Family violence, child wellbeing and Aboriginal services (partner services) that make up the partnership and employ practitioners and practice leads in The Orange Door. Employing agencies deliver ‘core services’.

FTE

Full-time equivalent.

HLG

Hub Leadership Group

L17s

Victoria Police officers who attend family violence incidents make referrals (L17s) to specialist family violence services, family services and child protection as necessary

Mallee

The Orange Door in Mallee, with the main physical site situated in Mildura.

NEMA

The Orange Door in North Eastern Melbourne area (NEMA), with the main physical site situated in Heidelberg.

NDIS

National Disability Insurance Scheme.

OLG

Operational Leadership Group.

PIC

PwC’s Indigenous Consulting.

RAMPs

Risk Assessment and Management Panels.

RCFV

Royal Commission into Family Violence (2016).

Referral

The term used for when clients of The Orange Door are connected with non-core services.

Respondent

Alleged perpetrator of family violence as identified in a L17 referral.

Section 38 consultation

Community-based Child and Family Services, which includes all of the partner organisations in The Orange Door, are able to consult with Child Protection at any time under Section 38 of the Children, Youth and Families Act (2005).

Screening

A screening record is created to capture a referral or direct contact into The Orange Door. Where there is more than one person identified in the referral, a screening record will relate to all people included in the referral.

SPaRO

Strategic Planning and Reporting Officer is employed by FSV and based at The Orange Door.

The Orange Door areas

The first four areas where The Orange Door was established in Barwon, BPA, Mallee, NEMA.

Third-party referrals

Referrals made to The Orange Door typically by phone or email by a party other than the client, Child Protection or police (through L17 reports), for example, by a teacher/nurse.

TRAM

Tools for Risk Assessment and Management.

Victim Survivor

More commonly referring to a person with lived experience of family violence. In this report, the term ‘victim survivor’ is used as aligned with the Statewide Concept.

About the evaluation context

Family Safety Victoria (FSV) engaged PricewaterhouseCoopers Consulting (Australia) Pty Ltd (PwC) in partnership with PwC’s Indigenous Consulting (PIC) and Australian Catholic University (ACU) to conduct an evaluation of the first four operational areas of The Orange Door in September 2018. The Orange Door (previously known as Support and Safety Hubs) is a new way for women, children and young people experiencing family violence, and families who need assistance with the care and wellbeing of children to access the services they need to be safe and supported.

This evaluation focuses on the establishment, operations and initial service offering of The Orange Door in Barwon, Bayside Peninsula (BPA), Mallee and North East Melbourne (NEMA) areas that commenced operations between May and July 2018. The evaluation seeks to identify key lessons and opportunities for improvements to establishment activities to support the implementation of The Orange Door in 2019 and beyond, and to inform enhancements across existing and new areas. As such, the evaluation is intended to identify opportunities for improvement at the area (micro-level), across area (organisational-level) and statewide (system-wide) levels to enhance operations and service delivery in the future.

Throughout this report we have discussed the context in which our findings need to be considered. This is critically important when considering the opportunities for improvement identified in the final chapter of this report. It was not expected that The Orange Door would be operating seamlessly in an integrated manner at the time in which the 2018 evaluation was undertaken. The Orange Door represents a new way of working for FSV and the partner agencies. FSV is a new agency, integrated practice with victim survivors, perpetrators and child and family services is a new model, and the model required new facilities, new processes and new staff. The scale of reform and change that The Orange Door has faced in the early establishment phase cannot be underestimated. There has been enormous effort on the part of FSV, all agencies and their staff involved to establish The Orange Door and this was evident to our evaluation team during fieldwork and stakeholder interviews. The findings are not intended as a critique of these efforts, rather as a method to identify opportunities for improvements to the establishment, operations and initial service offering in current areas and in future areas where The Orange Door will be rolled out.

About The Orange Door

The Orange Door is a new model of service entry and coordination for family violence and family support services in Victoria (previously known as Support and Safety Hubs) which responds to key recommendations of the Royal Commission into Family Violence and delivers on the vision described in Roadmap for reform: strong families, safe children. Recommendations were based on findings that people often don’t know where to go for help, can be bounced around the service system and may experience fragmented or uncoordinated responses and that efforts to hold perpetrators to account and in view were insufficient.

The Orange Door is intended to be safe and welcoming and provide quick and simple access to support. The Orange Door statewide concept (Support and Safety Hubs Statewide Concept, July 2017) was informed by a series of design discussions with people who have experienced the service systems. It was also informed by consultations with the community sector and agencies, and across government that focused on ways to provide easy and seamless access to the full range of supports and services that individuals and families need.[1] The service model for the first rollout was intended as a foundational model whereby certain elements, roles and functions may evolve over time given the phased approach to implementation. However key functionality of the model was outlined in FSV’s Support and Safety Service Model (April 2018) and comprises following key features:

• Access: ensuring a safe and convenient entry to The Orange Door, including telephone and online access, a community-based hub, outreach practitioners and referrals from other professionals.
• Screening, identification and triage: identifying risks and needs of all people in a family, in order to determine whether The Orange Door is the appropriate response and then to prioritise The Orange Door response.
• Assessment and planning: whereby practitioners work with people to develop a clear understanding of risks, strengths and issues, identify support needs and identify the type and intensity of service responses required.
• Connecting people to the right services: involving matching services to the needs of a person and family based on the needs and risks identified through assessment and planning, and then connecting the person/family to the services or providing this support directly.

The Orange Door will be established across the 17 Department of Health and Human Services areas. The first four areas examined as part of this evaluation were:

• The Orange Door in Bayside Peninsular area (BPA)
• The Orange Door in Barwon
• The Orange Door in North East Melbourne area (NEMA)
• The Orange Door in Mallee.

Figure 1: Locations of the first four areas of The Orange Door and the primary physical sites

Mallee

Physical location: Mildura

Commenced: 31 May 2018

FTE: 33.4 as of October 2018

Partner organisations: 4

Bayside Peninsula

Physical location: Frankston

Commenced: 14 May 2018

FTE: 80.3 as of October 2018

Partner organisations: 10

North East Melbourne

Physical location: Heidleberg

Commenced: 10 July 2018

FTE: 64.6 as of October 2018

Partner organisations: 7

Barwon

Physical location: Geelong

Commenced: 31 May 2018

FTE: 43.9 as of October 2018

Partner organisations: 4

NB: The number of partner organisations includes community service organisations and Aboriginal services (these numbers do not include staff from FSV, DHHS or Vic Police).

Source: Figures derived from FSV commencement reports and List of community service organisations in launch areas. The Orange Door in Inner Gippsland has commenced but is out of scope for this evaluation. FTE refers to funded positions for 2018-2019.

Evaluation approach and framework

A developmental approach to evaluation

We took a developmental approach to the evaluation to gain insights about the evolution of The Orange Door. Our approach used traditional methods of data collection to enable early insights to rapidly inform changes to current operations of The Orange Door and for the establishment of The Orange Door in further areas. Point-in-time opportunities for improvement also provide FSV and partner agencies of The Orange Door the opportunity to revisit how they are operationalising the service model.

Developmental evaluation is an evaluation approach that can assist evaluators to understand new innovations, initiatives and programs that are being implemented in complex, changing or uncertain environments. The approach facilitates real-time testing of hypotheses, rapid iterations and changes to program design, with feedback on the initiative allowing for rapid and evolving improvements to be made (even while the evaluation is still occurring).

Given the recent establishment of The Orange Door and the evolving nature of service operations in the context of ongoing establishment processes, it is not possible to conduct a traditional ‘summative’ evaluation that makes conclusions about system outcomes and answers the question ‘does the model work’? As such, a developmental evaluation approach allows the rigour of traditional methods of evaluation (such as data analysis, interviews, etc) to be applied while accommodating the dynamic environment in which The Orange Door is evolving in terms of operations. Our developmental evaluation approach enables early insights to rapidly inform changes to the way The Orange Door is operating and to establishment processes. It also focusses on systems, practices and processes that are being trialled or in the early stages, so we expected to identify a range of issues that are not working well, or are not fully developed yet.

This evaluation, by the nature of its focus is a process evaluation. We are focused on examining a range of service infrastructure, systems, processes and procedures during their early implementation to assess if these will contribute to the success of The Orange Door. It is not an outcome evaluation, focused on the impact of The Orange Door on clients. This will in time be an important focus for FSV and partner agencies to evaluate, but it is not possible to conduct a summative outcome evaluation at this time. As such, much of this report focuses on processes and systems – which may seem somewhat removed from the intent of the reforms to keep victims safe and build stronger families. However, processes and procedures in The Orange Door areas are the steps and tools needed in order to achieve the outcomes. At this early stage – it is important that these systems and processes are working effectively to meet the needs of The Orange Door clients.

FSV has specifically commissioned a developmental evaluation at the earliest possible stage of The Orange Door roll out in order to learn about what is working and what needs to be altered, rapidly, for The Orange Door to be successful. This needs to be acknowledged in the context of our evaluation findings that do identify some aspects that need to be strengthened or altered as the service model evolves. This evaluation is to our knowledge, only the second developmental evaluation commissioned by the Victorian Department of Health and Human Services. Given that FSV is itself a new agency, the commissioning of an evaluation of this type demonstrates a commitment by FSV to focus efforts on quality improvement. We believe that this type of perspective and approach will assist FSV well over the coming years as the agency guides the significant reforms it has responsibility for.

Evaluation purpose and framework

The purpose of the evaluation is to:

• Identify key lessons and opportunities for improvements to establishment activities and processes to support implementation of the second roll out of The Orange Door in 2019 and beyond.
• Identify key lessons and opportunities for improvements to the initial service offering in the first four areas, to inform enhanced operations across existing and new areas.

Early findings were provided to FSV throughout the course of the evaluation to provide opportunities for changes to be applied to current operations, as well as establishment processes for the next rollout of The Orange Door.

The scope of the evaluation was defined by five core evaluation questions developed with FSV, as presented below:

1. Establishment: What changes can be made to establishment approaches to improve future implementation and operations of Orange Door Sites?
2. Alignment within intention: To what extent is the Orange Door operating as intended by the initial service offering?
3. Client experience: To what extent is the Orange Door initial service offering contributing to improving client experience and client and system outcomes?
4. To what extent does the Orange Door workforce have the resources, capacity and specialist expertise to undertake the full range of functions articulated in the initial service offering?
5. Local integration and coordination: To what extent is the Orange Door supporting service integration and coordination at the local level?

To ensure the evaluation findings are practical and useful, PwC revised an initial evaluation framework in collaboration with FSV to guide all evaluation activities and outline the evaluation questions, indicators, measures and methods of data collection for the 2018 evaluation. The evaluation framework was based on the initial framework provided by FSV, The Orange Door: Initial Evaluation Framework, which was intended to inform the design of all evaluations of The Orange Door.

This report seeks to answer the core evaluation questions and associated lines of inquiry. The complexity of the evaluation has meant that many themes and findings cut across and are related to multiple evaluation questions and lines of inquiry. In addition, there is some commonality between indicators. As such, this report has made references between content across sections to avoid repetition.

Evaluation methodology

A mixed methods approach was taken to inform The Orange Door 2018 Evaluation, and comprised of a range of data collection methods. Figure 2 summarises the methods used to inform this report. A detailed methodology is provided in Appendix B.

Figure 2: Evaluation methods

In line with our professional standards for evaluation, we have applied the following principals to our analysis and reporting of evaluation evidence:

• Our evaluation analysis triangulated data to develop the common themes and findings, as well as to validate findings that may have arisen from other data sources.
• We analysed data from interviews with multiple individuals and workforce roles in The Orange Door before determining the common theme or finding.
• We analysed data across The Orange Door areas to validate evaluation themes and findings, and noted where differences were observed.
• Throughout the report we have identified where we had conflicting data, or where we were not confident in the strength of evidence. We have at times, also identified the type of additional data that would be required to develop definitive and robust conclusions.
• At times, we have provided quotes from practitioners or clients to illustrate a key theme or finding. However, findings are generated from multiple evaluation sources rather than the viewpoint of an individual (no matter how senior the stakeholder).
• At times, we have deliberately not identified the role of an interviewee where doing so is likely to identify the individual.
• We have provided quantitative data in certain parts of this report to illustrate our findings against a particular line of inquiry. However, to ensure ‘readability’ of this report by a lay audience, we have not presented the findings of all quantitative analysis conducted for this evaluation.

Report structure

The report is structured to answer each of the evaluation questions methodically.

• Section 2 provides the overarching context for the evaluation
• Section 3 describes and assesses the establishment processes and governance arrangements for The Orange Door (Evaluation Q1).
• Section 4 explores the fidelity of The Orange Door service model at the four areas, and analyses the extent to which it is delivering on its intended objectives (Evaluation Q2)
• Section 5 examines client experience and outcomes (Evaluation Q3)
• Section 6 considers the workforce (Evaluation Q4)
• Section 7 examines service integration and coordination within The Orange Door (Evaluation Q5)
• Section 8 provides conclusions and identifies opportunities to inform improvements to current operations – at both the area and state level – as well as changes to implementation of The Orange Door in the next areas
• Appendices containing supporting materials and further detail regarding key aspects of the evaluation.

[1] FSV, ‘Support and Safety Hubs: Service model’, April 2018.

Context for the evaluation

It is important to situate the work of the first (2018) evaluation of The Orange Door within the broader context of the family violence reforms and the transformation of the social services sector in Victoria. The evaluation has taken into account this broader context when considering findings and opportunities for improvement. For example, we have been mindful that it is very early in the implementation of The Orange Door and therefore some issues are to be expected during early implementation and planning. Our findings and opportunities have been developed to be forward looking with the intention of improving the future rollout of The Orange Door.

In this context, it is important to realise that where evaluation findings indicate that a process or system is not working as in line with the intent described in the foundational documents for The Orange Door – that these be viewed as lessons in the evaluation of the reforms, rather than as ‘failures’ of The Orange Door. It is not productive at this point to ‘benchmark’ The Orange Door against outcomes or achievements that would be expected only when the model is fully evolved and operational across all 17 areas.

Developmental evaluations of initiatives in the first twelve months of any innovative service model (such as The Orange Door), are expected to result in inherent challenges and transition friction in relation to:

• conceptual issues
• governance and oversight processes (including monitoring and reporting)
• interface with existing policy and existing legislation
• structural and systems approaches to service delivery
• infrastructure, IT and other resources
• workforce issues
• administrative needs that are specific to start up phases
• interaction with the broader service system within which the service is being developed

As such, it is unsurprising that this developmental evaluation has identified elements that can be strengthened as The Orange Door is rolled out across Victoria.

Purpose of the family and child and family violence reforms and The Orange Door

The Royal Commission recognised that “the way that various parts of the family violence system work together make it harder for women and children to access support, and for the service system to collaborate to provide a broad range of services”[1]. The Royal Commission recommended the establishment of Support and Safety Hubs to “provide a single, area-based entry point into local specialist family violence services, perpetrator programs and Integrated Family Services and link people to other support services” (Recommendation 37).

Our evaluation is grounded in the original purpose of family violence reforms and The Orange Door. The four domains that the Victorian Government’s family violence reforms are working to achieve, outlined in the Family Violence Outcomes Framework are:

• Family violence and gender inequality are not tolerated.
• Victim survivors, vulnerable children and families, are safe and supported to recover and thrive.
• Perpetrators are held to account, engaged and connected.
• Preventing and responding to family violence is systemic and enduring.

The reforms are underpinned by a recognition of the gaps in the current system, particularly the challenges in accessing services and the experience of services by Aboriginal people and people from diverse communities such as multicultural communities, LGBTIQ people and older people. The reforms are underpinned by a commitment to Aboriginal self-determination as outlined in the Aboriginal 10-Year Family Violence Agreement, Dhelk Dja: Safe Our Way – Strong Culture, Strong Peoples, Strong Families. They are also underpinned by an understanding of diversity and intersectionality, recognising that many forms of social characteristics intersect. The Everybody Matters: Inclusion and Equity Statement outlines principles to guide the reform approach and work to ensure the inclusiveness of services including that services are underpinned by human rights, and approaches that are strengths-based, trauma-informed, culturally safe and person centred.

The Orange Door is a critical part of the Roadmap to Reforms: strong families, safe children, which aims to strengthen and reform the family and child wellbeing sector in Victoria. The Roadmap articulates a key role for The Orange Door in terms of triage and risk assessment, early intervention and wrap around services for families and children where there are concerns about child wellbeing. It anticipates the need to leverage from existing Child FIRST alliances and systems and integrate them into the new The Orange Door model of service delivery [2].

It is important to acknowledge that The Orange Door is one of many critical reforms currently underway in Victoria and that this is a time of immense change in the family violence service system. Other critical reforms that are currently being implemented in Victoria include: the introduction of the Family Violence Information Sharing Scheme; the Child Information Sharing Scheme; the Free from violence prevention strategy; initiatives as part of Building from Strength: 10-year industry plan for family violence prevention and response; rollout of the Specialist Family Violence Courts model; The Family Violence Risk Assessment and Risk Management Framework (MARAM); and the Family Violence Outcomes Framework.

The scale of change of the family violence and family and child wellbeing reforms and the dynamic nature of reform implementation will impact The Orange Door workforce and practice at this early implementation point during the evaluation.

The Victorian Government’s Support and Safety Hubs Statewide Concept (July 2017) describes the vision for The Orange Door, which is that it “will be accessible, safe and welcoming to people, providing quick and simple access to the support and safety they need. The Hubs will also focus on perpetrators of family violence, to keep them in view and to connect them to services that assist in holding them accountable for their actions and changing their behaviour.”[3] Thus The Orange Door plays a pivotal role in shifting service provision from the existing models that have been identified as failing to meet the needs of victims of family violence, to shifting towards linked up, integrated responses to family violence risk and for children and families in need of support. To this end, the importance of The Orange Door to the overall reforms cannot be underestimated – nor can the significant change management required for Victoria to reach this vision. This evaluation is occurring at the very start of this significant change process, and as such, we would anticipate seeing the foundational building blocks to be evolving in order to move Victoria towards this vision.

Scale and magnitude of family violence reforms

The family violence reforms are system-wide and transformative, spanning from primary prevention activities to early intervention, response and recovery. They impact across many departments, agencies and the community sector, with interdependencies and interactions between reforms. As a result of these extensive reforms, the workforce requires significant transformation and expansion, to work in a different way alongside different services.

For example, the establishment of The Orange Door has impacted the movement of the local community sector in those areas where it is operational as practitioners move between case management services and The Orange Door. Similarly, we recognise that the first four areas of The Orange Door were established at the same time as the establishment and expansion of the Central Information Point (CIP) and the redevelopment of the Family Violence Risk Assessment and Risk Management Framework. Both of these initiatives impact on the work and practice of The Orange Door and the implementation of the Family Violence Information Sharing Scheme and the Child Information Sharing Scheme.

We also acknowledge that the detail of the reforms is still being defined, implemented and refined. Where possible we have made reference to work currently underway to address issues identified.

Timing of the family violence reforms and the evaluation’s field work

The evaluation team is cognisant of the timing of the evaluation within the broader rollout of The Orange Door. FSV was first established in July 2017, the service model signed off in December 2017, and the first four areas commenced service delivery in May 2018. The timing of the implementation to design and set up a new service with multiple delivery partners was extremely tight. This included all of the logistics to identify appropriate properties, lease them, establish a new CRM, implement the new family violence risk assessment management framework and associated tools, establish new governance, and recruit to The Orange Door.

We also acknowledge that the evaluation team conducted site visits and collected data from October – December 2018 and the pace of the reforms and implementation means that some of the issues raised during the evaluation have been addressed already. For example, we understand that there have been steps to introduce reflective practice across disciplines in one of The Orange Door areas since our evaluation fieldwork. This was an opportunity identified to FSV early in the evaluation which has been subsequently actioned. We also understand that FSV is progressing workforce training, a workforce strategy, and a performance framework, as well as addressing physical infrastructure issues. It is important to acknowledge in the context of our evaluation findings and identified opportunities for improvement that FSV have an agreed ongoing project plan for the further development of The Orange Door.

Maturity of the social services system

In addition to the family violence system reforms, the Victorian social services system has experienced reforms to the child and family service sector through the Roadmap to Reform: Strong families, safe children initiatives, and is at the same time continuing to implement the National Disability Insurance Scheme (NDIS).

This reform environment has meant major change for the social services sector workforce. There has been huge investment in family violence services including The Orange Door as well as investment in child protection practitioners and family services in Victoria. The supply of social workers is limited and while there has been some expansion of training, there are still constraints on the supply of experienced practitioners. Many of the experienced practitioners have only worked with perpetrator services or family violence services and there are strong cultural issues to overcome in asking these services to work alongside each other. See section 6.3.1 for further discussion of the challenges between practice perspectives of the different workforces who have come together in The Orange Door.

'If you look at a service, they are down on staff in The Orange Door but also in their own agency. They are struggling to find people with the skills. So many jobs that became available at the one time without an adequate workforce management process.' – Stakeholder interview.

While there are challenges to increasing workforce supply, practice is also changing. The redeveloped risk assessment and risk management framework (MARAM) was released in 2018, with resources and tools and training under development. Our understanding is that new risk assessment framework for statutory child protection and the child and family practice framework (the Best interests Case Practice Model) are also currently being redeveloped and will be aligned to the MARAM practice approaches.

Family Safety Victoria is currently progressing the development of a workforce strategy which will assist in strengthening the workforce planning for The Orange Door. This strategy builds on the 10 Year Industry Plan for the family violence sector[4] and the Community Services Industry Plan[5] both of which address issues of workforce sustainability and strength. However, this workforce strategy can only influence and direct workforce initiatives within the family violence and child and family services sector – it cannot expect to influence the broader workforce issues within the Victorian social service sector that are currently under pressure from major national reforms in disability and aged care.

In particular, it is important to recognise the infancy of perpetrator services that currently are primarily men’s behaviour change programs. The research and development of the evidence-base on what works in perpetrator practice is at a much earlier stage of maturity and development than research and practice with family violence victim services and children and families. This is a reality of the current state of practice that The Orange Door service model has begun with – it should then be expected that a significant focus will need to be made to strengthen this type of practice and to integrate it well within multidisciplinary practice within The Orange Door.

Local service system context

The Orange Door has been established within the context of local services and local relationships and networks that already exist and are varied from area to area. The number and nature of services within the local service system and the nature and complexity of existing relationships impact the establishment and effective operation of The Orange Door in an area. For example, the strength of the existing service in Barwon working with perpetrators prior to The Orange Door being established has been one of the reasons for the successful integration and operation of that service within The Orange Door. Our identified opportunities recognise that the local service system impacts on FSV and partner agency’s capacity to deliver intake and assessment services as well as provide an effective interface to the broader service system. Furthermore, for many diverse communities, targeted services will exist and be accessible in some areas of The Orange Door and not in others. Service offerings will differ markedly across the state for many client cohorts and this will therefore impact on the role of The Orange Door in each area.

The strength and nature of the existing service system in each of The Orange Door areas needs to be considered in evaluating a new service such as The Orange Door. Initial implementation and early service delivery of The Orange Door can predictably be influenced by the strengths and weaknesses in the existing social services system and may not necessarily reflect strengths and weaknesses of the new service or model itself.

[3] Royal Commission into Family Violence, Report and Recommendations Volume II, 2016 p.264

[4] Victorian Department of Health, ‘Roadmap to Reform: strong families, safer children, the first steps. April 2016.

[5] FSV, ‘Support and Safety Hubs: Statewide model’, April 2018.

[6] FSV, ‘Building from Strength: 10-Year Industry Plan for Family Violence Prevention and Response, 2017

[7] VCOSS ‘Community Services Industry Plan’, August 2018

To what extent have establishment planning and operationalisation activities enabled implementation of The Orange Door foundational service model?

In evaluating the contribution that establishment have made to early implementation of The Orange Door model, we have examined a number of factors including: establishment processes, activities and tools, timeframe for foundational work, recruitment and on-boarding of staff, access to guidelines and policy, physical infrastructure and partnership formation.

Establishment processes, activities and tools

Extensive foundational work within a compressed timeframe

As a landmark reform representing significant transformational change in service delivery, an extensive array of establishment activities and processes were required to set-up and operationalise The Orange Door. The Orange Door model represents fundamental changes to ways of working, practice, culture and service delivery, as well as new organisational and governance structures, requiring extensive design, coordination, planning and support to enable roll-out. The government’s initial goal to open in five areas at the end of 2017 was ambitious, and created challenges and urgency for FSV – as a newly formed agency[1] – which ultimately led to delays in rollout.

Overall, FSV has been responsible for ‘establishing The Orange Door and associated governance mechanisms, as well as designing the service model and overseeing statewide delivery.’[2] This was to ensure ‘statewide fidelity and quality in the model, and to ensure The Orange Door and local services were supported to come together and work effectively.’[3] The amount of work to set-up and prepare The Orange Door has been unprecedented, with FSV managing a range of establishment activities, including:

• Development of statewide design, policy, frameworks and guidance to support integrated practice (for example Support and Safety Hubs Statewide Specifications, Interim Integrated Practice Framework and Operational Guidelines), as well as pre-requisite check-list for service commencement.
• Collaboration with local governance and stakeholders to understand the range of local stakeholders’ area specific needs, and to test elements from the statewide design in relation to the local context[4].
• Recruitment of Hub Managers to provide strategic management and oversight of their area of The Orange Door and locally-based support staff to facilitate operations, partnerships and connection to government and the broader service system.
• Coordinating infrastructure arrangements for each area including locating, leasing and designing premises, site fit-out, IT requirements and works.
• Workforce induction, training and development program including an induction training program for all workers in The Orange Door[5].
• Development of Partnership Agreements to formalise the partnership between all partner agencies, including government. The agreement outlined the requirements for working in partnership at The Orange Door, and was signed between all parties, except DHHS.[6]
• Formation of local establishment and governance groups: to monitor and plan for the local context.[7] This included: Hub Leadership Groups (HLGs), Hub Operations Leadership Groups (OLGs), Local Hub Establishment Groups and Aboriginal Advisory groups.[8] [9]
• Communications and engagement: to ensure consistent branding, messaging and feedback with a broad range of stakeholders, including the public, sector stakeholders and across government.

Establishment processes that were managed by the partnership agencies as part of The Orange Door included:

• recruitment of practice leaders and practitioners
• operationalising the model
• internal change management in preparation for services and service delivery
• communications and engagement with their workforce and stakeholders

Establishment activities for the launch areas commenced when FSV was formed in July 2017, following development of the Support and Safety Hub Statewide Concept (released in the same month)[10] and considerable joint work between FSV and stakeholders on service model development. While significant work and consultation was undertaken by the Department of Premier and Cabinet (DPC) to design and develop the statewide concept and secure funding allocation across one year,[11] activities to establish the model occurred within a comparatively compressed timeframe of under 18 months while FSV continued to engage with stakeholders across the state during this establishment phase[12]. Evaluation evidence indicates that timeframes for establishing The Orange Door did not allow for the extensive range of operationalisation activities required to set up the foundational model, and had to be implemented rapidly.[13] The requirement to locate suitable premises and recruit an experienced workforce were key challenges that contributed to delays in the planned timeframes.

A significant volume of change for the sector

The amount of change for the sector – and particularly practitioners – has been unprecedented. Transitioning to The Orange Door service model has necessitated a completely new way of working, in conjunction with learning how to navigate new systems and technology (eg the CRM system), adjust to new processes (ie workflows), a new team model (ie colocation), practice change (ie from case management to intake and assessment) and operational changes arising from bringing together some services that had not previously worked together and aligning to a consistent service response. This has placed immense pressure on staff as they adapt to their new roles and new systems while providing client support.[14] The volume of change in almost every aspect of practice needed over a short period of time has complicated initial implementation – despite the workforce bringing skills, expertise and existing practice and frameworks to enable service to be provided to clients from commencement. Evaluation feedback from external stakeholders was that more a sequential or phased rollout would have supported smoother implementation, to allow for the adoption of new systems and ways of working.[15]

The transition has demanded a level of flexibility and agility from the workforce in order to work within uncertainty as the model, systems and processes have been implemented, refined and updated. This level of change has been significant for the sector without a corresponding focus on change management needs. The nature of the model means that this change management focus needs to be coordinated and progressed by both FSV and partner organisations.

'… what was overlooked was change readiness, there needed to be a phased approach to change management… The scale of the change was just too great, we needed time to prepare the staff for winding up their current practice and then merging their practice to the new system.' – HLG evaluation interview

Establishment activities

The amount of change associated with implementing the numerous components of The Orange Door model, timing considerations and change readiness of the sector are overarching complexities that complicated establishment and initial service delivery of The Orange Door. Figure 3 highlights five key establishment activities – noting this is not an exhaustive list – each of which is discussed in the following section.

Recruitment

Recruitment of The Orange Door workforce community service organisation funded positions across practice leadership and practitioners) began in late 2017. The allocated funding for full-time equivalent units (FTE) was based on outputs rather than pre-determined inputs to allow agencies flexibility to recruit to meet their output targets.[16] Despite early recruitment efforts, none of The Orange Door areas had full contingent of staff at service commencement.[17] Estimates provided to the evaluators suggests that in one area, there was less than half of staff expected on day one, 'despite 100 per cent of L17s[18] coming in. It destroyed morale.'[19] Evaluation evidence indicated the workforce and agencies did not have sufficient preparation time for recruitment, which may be due to delays to ‘recruitment products,’ and 'underestimation of the time required for end to end recruitment.'[20] It is also possible that the predominant functions of intake and assessment may not appeal to all people working within the broader family violence and family services workforce. While this may be the view of some practitioners interviewed as part of this evaluation, it is important to remember that The Orange Door undertakes a range of additional functions.

Similarly recruitment for FSV employed positions[21] has been challenging, and continued after service commencement in most areas. [22] Vacant positions in the initial periods tended to be the Service System Navigator and Operational Support officer[23]. The impacts of these vacancies were felt by The Orange Door workforce. In particular, the absence of an Operational Support Officer in some areas meant that operational processes and infrastructure issues had to be dealt with by the leadership team (eg OLG or HLG teams), distracting their focus from strategic leadership[24].

In addition, subsequent rounds of recruitment have been necessary due to staff-turnover. While we are unable to quantitatively verify what the rates of staff attrition have been, interviews during fieldwork indicated that attrition rates have been higher than expected across most The Orange Door areas, and are likely due to a number of factors:

• Position descriptions that did not accurately reflect actual roles. A consistent theme through the evaluation was that the expectations of a role based on the position description did not align to the reality of the role once operations had begun.
• Role ambiguity for some positions at The Orange Door. Some workers did not have a clear understanding about their role and responsibilities even after a number of months working at The Orange Door.
• Working at The Orange Door has demanded a high level of adaptability to work within the uncertainty that the innovation of service delivery in The Orange Door involves. For many workers this was unexpected; some did not have the skills or willingness to work within this degree of uncertainty, others embraced the innovation. This finding needs to be balanced with the evaluation finding that there has been, and remains significant excitement by many in The Orange Door for the new model.
• Some workers felt that they did not have a choice to move to The Orange Door as the employing agency positions were subsumed into The Orange Door.
• Some workers reported that they have struggled to make the transition to a narrower role of intake and assessment from broader case management and others do not believe they have skill on entry to The Orange Door to work immediately within an integrated service model[25].

It is our understanding that recruitment is now being tailored to these needs by FSV and partner organisations.

There was also some suggestion from the workforce during the evaluation that the anticipated Full Time Equivalent (FTE) allocations for each discipline needs refinement. Some areas, for example may require more men’s workers due to an increasing demand.[26] Quantifying the adequacy of FTE across specialties was beyond the scope of the evaluation, nor could it reasonably have been expected to be predicted with accuracy ahead of the model being implemented given that this model is a world first initiative in service delivery.

Induction training

FSV facilitated a four-day induction training program to introduce staff to the service model and practice framework, and to provide foundational knowledge on working with family violence, children and families and perpetrators.[27] It aimed to equip workers to commence service delivery from the first day of operations, and guide integrated and safe practice.[28] Content delivered during the induction training program included:

• The Orange Door service model
• Interim Integrated Practice Framework
• use of the CRM platform
• the 'Client Experience Toolkit'
• family violence practice
• children and family practice
• working with child protection
• perpetrator accountability and visibility
• risk assessment and management for adults and children, Tools for Risk Assessment and Management (TRAM), including contextualising The Family Violence Multi-Agency Risk Assessment and Management Framework (MARAM) The Orange Door
• information sharing within The Orange Door context

Modules were delivered by FSV, Domestic Violence Resource Centre Victoria (DVRCV), No To Violence (NTV), DHHS Child Protection and the Centre for Excellence in Child and Family Welfare (CFECFW). Aboriginal self-determination and people’s agency and intersectionality and diversity were not included in detail as part of the initial training program, noting that these materials are to be developed and delivered post service commencement as they need to reflect and be tailored to the local area and community.

Each area initially received two rounds of training to minimise disruption to service delivery and allow partner agencies to send half their workforce to either of the training dates. Further rounds of induction training have been run post service commencement as new staff have been recruited.

FSV conducted an evaluation of the training program primarily through feedback forms which focused on the content. Overall, the evaluation found that while a positive impact was seen in terms of building 'on existing skills and knowledge of the participants across all training modules,' participants wanted further information on how The Orange Door would operate in practice in order to obtain a better understanding of their role.[29] This finding was supported with our evaluation data: and in particular, further clarity about integrated practice and how to operationalise this at The Orange Door was required. As The Orange Door continues to operate and it becomes clearer how the model is operationalised in practice, FSV are well placed to build on and tailor these early induction programs to meet the emerging needs of the workforce.

Evaluation evidence indicated that for some, the training was pitched at a low skill level for practitioners for their own speciality but did not provide them with sufficient understanding of disciplines outside of their experience. Training content is discussed further in Section 6, although it is important to acknowledge that FSV is refining the induction training program based on findings from its evaluation, as well as developing a Workforce Strategy for The Orange Door which includes 'effective approaches to workforce induction and on-boarding for new workers.'[30]

Guidelines and policy

FSV produced a comprehensive range of frameworks and practice guidance to ensure 'a consistent, statewide approach to operating The Orange Door across the launch areas.' Guidance materials were intended to be read in conjunction with other resources, which are outlined in Table 1.[31]

Table 1: Foundation documents

During evaluation fieldwork, many practitioners admitted that they had not read the guidance materials. While this was identified in the orientation training, when investigated, The Orange Door staff indicated it was due to the amount of documentation that had been produced, the length of the content and time pressures.[32]

Those who had read the documentation did not find the content useful for preparing them to work at The Orange Door as they expected more definitive practice based guidance including examples of how the model would or could work in practice. For example, while a vison of integrated practice was outlined in the Interim Integrated Practice Framework, there were no clear benchmarks or detail about how to operationalise this vision. As a result, people reverted back to how they previously operated.[33] Several Hub Leadership Group (HLG) representatives reported that they were continuing to work through what integrated practice looks like in The Orange Door six months after service commencement. It is important to note that though these foundational documents and guidance materials do acknowledge that the model is intended to be informed by evidence, and it seems reasonable that initial documentation would need to be iterated in response to ‘user’ feedback (in this case through use by The Orange Door workforce).

The Orange Door service model was never intended to be 'static or fixed, but rather informed by emerging needs and evidence,' and evolve and mature in response to 'community needs, co-design, and practice learnings.'[34] Despite this, however this evaluation found that greater parameters and prescription were needed in the service model to provide a sufficient starting-point for establishment (particularly in the absence of sequenced and ongoing change management processes which will necessarily need to be a joint responsibility of FSV and partner organisations), including more definitive guidelines on how to operationalise The Orange Door.

Additionally service model requirements (including those in the service specifications) were considered by many in the workforce to be too high-level to underpin operationalisation. Instead practitioners wanted practical operational guidelines to help mobilise operations. In time this may be developed by the OLG, however at the time of fieldwork, these issues were still being worked through. While practitioners reported that workflow needed to be tailored to the specific area, they requested a baseline to give them confidence that they were progressing in the right direction.[35] It is unclear if they expected FSV or local HLGs and OLGs to provide this direction.

There was confusion about the level of consistency of operational requirements with the guidelines in Support and Safety Hubs Service Model. Some leadership members indicated that a certain level of prescription was required to facilitate understanding of their operational requirements, and for FSV to clearly denote aspects which could be tailored to the local context and which required state wide standardisation. We understand that FSV are aware of these tensions and are currently considering how to address this issue. Tensions between statewide consistency and localised flexibility are likely to remain an ongoing issue in a reform of this scale and complexity. As further areas are established, we could expect to see clarity emerge about which requirements need statewide consistency.

Systems and tools

In addition to practice guidance, new Information Technology systems were developed to support service delivery, including the CRM, CIP and TRAM[36]. The CRM has undergone planned continual development[37] since service commencement to improve its functionality and to support service delivery. Practitioners reported that the initial functionality of the CRM was difficult to operate, time consuming and had provided limited support for their work. This is expected in any first iteration of a new IT system. In addition, there was a lack of supporting guidance or system workflows to help practitioners navigate the system, largely due to the ongoing development of the CRM.[38] The delay between induction training – where practitioners received CRM training – and service commencement further meant that some practitioners reported that they had to re-learn the system. The evaluation found that the workforce has varying levels of competency and confidence using the CRM which was confounded at that time by a range of IT functionality issues in The Orange Door.[39]

Evaluation evidence found that the issues with functionality of the CRM have impacted service delivery. The time practitioners have spent trying to navigate and overcome issues with the CRM compromised their ability to work through cases efficiently in these early stages. These difficulties compounded an already challenging transition to The Orange Door. It is however important to note that during the second round of evaluation fieldwork many practitioners reported that subsequent updates to CRM had already improved functionality.

'…the CRM went live on day one. A new data system on day one was not ideal. There should have been a phased approach to learning the systems.... we’re falling behind in cases' [40] – Practitioner interview

While the time pressures associated with opening The Orange Door created challenges to CRM deployment, and likely impacted on the readiness of the system at service commencement, the workforce did report the benefits of an evolving IT system and iterative FSV upgrades. For example, practitioners have seen tangible changes in the functionality of the CRM as a result of feedback they provided, and are beginning to see a more operational and fit-for-purpose system emerging. Practitioners were also complimentary of the CRM trainers, and appreciated them coming to The Orange Door to provide on-the-ground support.[41]

Similarly, the CIP has taken time to become fully operational, although these issues have not impacted The Orange Door workforce to the same degree as CRM, given the system sits outside The Orange Door. For example, establishment challenges for CIP included:

• on-boarding of agencies took longer than anticipated
• recruitment of suitably qualified data custodians
• early CIP reports taking longer to prepare than in the initial testing (nearly three hours compared to one hour respectively). Since ‘go live’, modifications were made to the platform to improve efficiency, and data custodians have become accustomed to the platform.[42] However, demand remains high in terms of the time it takes to prepare the CIP report. This may remain an operational challenge into the future

Despite the above challenges (many of which are reasonable given the innovation that CIP represents), there is strong evaluation evidence that CIP has been valuable for obtaining information to help practitioners more accurately assess and manage risk. Commencement reports (produced for each area) demonstrate that practitioners were using the CIP in the first months of service delivery, and this was verified during the fieldwork. While there is variation in the amount of reports that were requested across The Orange Door areas, this may be due to the different timeframes in which data was recorded.[43] See Section 5.3 for a timeline of CIP reports by The Orange Door area.

Physical premises

The Victorian Government committed to working with local communities, and local and statewide service providers, to establish the location of each physical site of The Orange Door.[44] FSV conducted extensive work to ensure client-centric design of The Orange Door physical site (see Chapter 5 for further discussion). The physical site of The Orange Door was an important part of establishment as the facilities functioned to bring together different workforces under one space, and colocate practitioners to support multidisciplinary practice. There were significant challenges in finding appropriate premises for the physical site for The Orange Door, as well as completing the fit-out within the timelines that FSV needed to work within.[45] Additionally, it is our understanding that minimum viable requirements for ‘go live’ were still being reviewed with each HLG leading up to service commencement, leaving insufficient time for areas to comply with the new criteria.[46] It is also our understanding that this will not be repeated in subsequent rollouts of The Orange Door and FSV have adjusted sequencing and go live criteria for future sites.

Given these time pressures, a number of issues with the physical premises were encountered at service commencement for several areas. While we understand there have been adjustments to the sites since then (and following our fieldwork) to rectify a number of these issues, it is important to acknowledge the problems and highlight how they impacted establishment and initial service delivery. An overview of these issues and their impacts on the workforce, culture and service delivery at the time of the evaluation are summarised in Table 2.

Table 2: Issues encountered with physical premises at establishment

Notwithstanding the above challenges, the workforce recognised The Orange Door site – as a key enabler for bringing together disparate agencies, and colocation was a foundation for building integrated practice:

'People sitting next to each other means they can learn from one another.' [48] – practitioner interview

It is our understanding that a number of the issues identified have subsequently been addressed by FSV or by The Orange Door area.

Partnership formation

The Orange Door is underpinned by a partnership approach to:

• ensure collective leadership
• harness specialist skills and expertise from a range of specialties to deliver the service model functions and
• offer a holistic response to individuals and families[49]

The Orange Door partnerships comprise of government and non-government agencies across different sectors. To formalise this, Partnership Agreements were put in place to 'lay the foundation for collective leadership of the Hubs.'[50] Partnership Agreements served to create a legal mechanism across the organisations and key government agencies in the Hubs to 'embed collective leadership, set down a shared vision, goals and principals, support integration of services, formalise governance mechanisms and effectively deliver Hub functions.'[51]

Evaluation evidence indicates a perception by many within the HLGs and OLGS that the Partnership Agreements and Service Agreements did not include the level of detail they were expecting to provide a suitable foundation for the partnership at The Orange Door. This is to be expected for such a novel service delivery model, and was predicted through the inclusion of a 12 month review in initial Partnership Agreements. It is also unclear if the Partnership Agreements and Service Agreements would have been sufficient at any point in The Orange Door service model, or if a lack of understanding or specificity in these documents reflects the point in time and need to refine these as the service model began to be operationalised and resourcing/funding allocation refined within the first few years of operations. Key requirements of the Partnership Agreements include: obligations of The Orange Door partners including resource and funding contributions (in the Service Agreements); establishment of the HLG (including its functions and expected terms of reference); and details of how complaints are to be managed as a collective.[52]

While Partnership Agreements provided a solid starting point for establishing and defining governance arrangements and operations, further discussion will be needed with the agencies to ensure that there is sufficient detail within these agreements to embed leadership and governance within the new partnerships. This was particularly relevant in the context of the matrix model which adds a layer of additional complexity to establishing governance arrangements (see Section 3.2). In addition, some leadership groups discussed the need for Hub Leadership Group terms of reference (included in the Partnership Agreement) to be reviewed and updated. The need for revisions, further detail or updates were reported to evaluators as covering areas such as: the role of the Operations Leadership Group (OLG), data collection and communication management at The Orange Door. It is our understanding that these aspects are currently being reviewed and endorsed by FSV.[53]

Collective leadership in these initial stages has been impacted by a perceived imbalance of contributions from partner agencies at The Orange Door. Some leadership groups reported that while there are variances in the FTE contribution and funding that each community service organisations receives attached to their service agreement, the contribution they are expected to make to The Orange Door has not as yet been adjusted to reflect the contribution. For example, some community service organisations with smaller funding contributions who provide staff with specific expertise are finding that early indications of demand mean that they do not have sufficient workers in The Orange Door. It is unclear if this is a point in time issue, or one that will remain once operations stabilise after the initial implementation phase. This needs to be considered in the light of the existing relationships and networks in the areas The Orange Door has been established. It also needs to be considered in the context that collective leadership takes time to be established and embedded.

Besides Partnership Agreements, FSV has been instrumental in forming local establishment and governance groups to plan and implement The Orange Door at the local context. We understand that FSV and some HLGs facilitated a number of workshops (including local establishment group forums and broader sector forums) to bring sectors together and consider ways of working in collaboration, as well as to try to further define some operational aspects prior to service commencement.[54] While this was considered imperative for establishing the group and initiating co-design, a number of stakeholders reported that more lead time for the HLG and OLG would have been valuable prior to service commencement, specifically to test and refine systems and processes.

A further contributing factor to slowing down the operating rhythm of leadership groups was a number of 'false starts'[55] associated with the opening of The Orange Door.

'Open day was a key challenge, having a date which changed many times and then confirmed at short notice… we had not prepared our staff and then had three to four weeks before we moved in.'[56] – OLG member interview

This has been addressed by FSV in considering what they are able to control in terms of start-up responsibilities and in a new sequencing and ‘go live’ criteria for establishing a new The Orange Door area.

A key enabling factor to collaboration – and particularly the HLG – has been various pre-existing alliances such as the Child First and Family Services Alliances, Family Violence Regional Integration Committees and Indigenous Family Violence Regional Action Groups (now Dhelk Dja Regional Action Groups). For a number of Orange Door areas, these existing networks have served as the platform for partnerships to build from, and they strengthened the leadership groups through prior working relationships and arrangements. Indeed these leadership groups reported that it would be very difficult to form partnerships so rapidly in the absence of their past working relationships.[57]

While this evaluation has focused on partnerships between community service organisations that comprise The Orange Door, further investigation is required to understand establishment of The Orange Door within the broader service sector, given the role of The Orange Door in connecting clients to the right services and coordinating these connections. It is important to understand the progress in developing and sustaining these collaborations, both operationally (i.e. establishing referral pathways and practices to and from The Orange Door) but also in terms of perception of The Orange Door and its role within the system. Limited information is available at this point in time for the evaluation and it was not within scope for this evaluation to examine the role of The Orange Door within the broader service system, however a number of indicative findings have emerged which may need to be explored and validated in future evaluations. These findings include:

• the need to understanding how The Orange Door aligns within the family violence, family and child wellbeing and the broader social service sectors
• the need for supporting systems and infrastructure to support referrals, noting that this was a work in progress during our fieldwork (ie workflows)
• recognition from The Orange Door areas that establishing engagement with the sector is an ongoing, long term activity, and a key role of the Service System Navigator (SSN)

Partnerships and networks enable information sharing and referral practices

Collaboration is central for information sharing to connect disparate agencies and sectors, and share critical information about clients. Despite expected challenges with establishing these connections in the early stages of The Orange Door, there is evidence that information sharing is an aspect of The Orange Door at present – particularly in relation to information transfer and collation within The Orange Door and information sharing between The Orange Door and external referral agencies. It is also important to note that there is a lack of consistency across The Orange Door areas in relation to referral processes and time frames.

Information sharing in The Orange Door has been facilitated by Part 5B of the Family Violence Protection Act which streamlines the sharing of confidential information as it treats community service organisations within The Orange Door as one organisation or partnership.[58] The Act allows practitioners to collect, use and disclose confidential information to other practitioners only within The Orange Door so long as the purpose is related to the provision of The Orange Door services. In particular it allows practitioners to share information with a colleague to draw from the expertise of a worker from a different specialisation.[59] Practitioners recognised the relevance and importance of this to their practice at The Orange Door.[60]

As The Orange Door evolves, the various components of ‘information sharing’ will become clearer in terms of the operational environment. Distinctions will be able to be made between:

1. Information sharing to support service delivery within The Orange Door.
2. Information sharing between external parties and The Orange Door (either collecting and/or disclosing information as part of a referral) or for assessment and risk management.
3. Information sharing from The Orange Door to a client to manage risk or provide service.
4. Information sharing to support monitoring and performance of The Orange Door.

The nature of the developmental evaluation early in the establishment of The Orange Door meant the focus on information sharing was limited to sharing within The Orange Door (and The Orange Door partner agencies 1).

In addition to this legislated mechanism[61], there are a number of factors that currently support information sharing, and the provision of services, within The Orange Door:

• Colocation of practitioners from different disciplines is important for information sharing and working collaboratively. Many practitioners acknowledged that information sharing is easier and faster than if they were situated in their employing agencies. However, it was reported that information sharing occurs in an inconsistent manner, is often supported by previous relationships and occurs more easily between certain disciplines (for example specialist victim survivor family violence services and child and family wellbeing services). This is to be expected at this early stage and information sharing is likely to be further embedded as The Orange Door evolves.
• Multidisciplinary teams and meetings such as cluster teams[62] or function (ie triage) meetings comprising of practitioners from each discipline. This has embedded a more structured approach to information sharing than colocation and has promoted more consistent information sharing. This was seen particularly in ‘function teams’ (such as initial screening teams) which were underpinned by an agreed and operational workflow process.[63] Operational workflow processes could be embedded in future areas of The Orange Door from the outset.
• Access to employing agency’s client record systems to obtain information from a range of different sources to better understand the risks and needs for an individual or a family. Practitioners retain access and use of their employing agency system so they are able to search for relevant information about a case. The partnership in this respect facilitates quicker access to a more comprehensive range of information. The benefits of access to partner information were widely regarded by the workforce and stakeholders.
• The CRM is intended to assist with gathering information and record management to facilitate sharing of client information between practitioners at The Orange Door.[64] Teething issues with the CRM functionality at the time of the evaluation currently limits the ease of information sharing at The Orange Door. We understand that a number of updates have subsequently been made and functionality has improved. The system has the potential to be an invaluable repository of information to support The Orange Door workforce, as it consolidates new and existing information from across the partnership.

Further discussion about information sharing protocols is discussed in Section 7.

Further discussion about information sharing challenges beyond The Orange Door is detailed in Section 8, and includes:

• supporting secure data transfer between siloed systems
• formal feedback loops for oversight of client outcomes
• clarity in terms of the amount of information practitioners should provide about and/or to a client

In relation to referral practices, evaluation evidence suggests that existing networks are instrumental for setting up referrals to The Orange Door. Having a foundation of existing referral pathways to work through has been useful, and practitioners have maintained their connections to services they are familiar with, including their employing agency to facilitate both referrals to and from The Orange Door. Going forward, it will be important that the CRM collects granular data on referrals from The Orange Door to allow for a more detailed understanding of the referral network within which The Orange Door exists in each area.

It is however unclear to what extent practitioners are referring to services outside of the partnership at this point in time.[65] This may largely be due to a focus and prioritisation on internal process development and operations at The Orange Door in the early stages of implementation. During evaluation fieldwork we saw the refinement and development (and later endorsement) of a referral form and process at one area that was focused on the partner services. Workers hoped that this template would improve information sharing and consistency and formalise referral pathways within the partnership. At this stage of The Orange Door’s evolution, it is to be expected that attention would be on internal development, and efforts to implement robust systems and processes that can be replicated and rolled out more broadly will follow.

As such, referral practices and processes remain a work in progress. Our evaluation meetings with leadership teams confirmed that further work is required to develop referral pathways and relationships with universal services (such as schools and hospitals).[66] This requires concerted and ongoing effort, particularly as many stakeholders believe that the role and function of The Orange Door is not widely understood at present. There is some limited evidence of inappropriate referral pathways into The Orange Door occurring because of the perceived lack of understanding about The Orange Door. This is already anticipated and will need to be monitored by FSV and partner agencies to ensure that The Orange Door is not seen as a ‘catch all’ social service.

'Education with the broader system needs to be done, as currently we’re seen as a one stop shop.'[67] – Practitioner interview

Leadership teams also reported that local services and referral pathways need to be mapped, and that this gap is particularly challenging for The Orange Door areas which service a large geographic area or where there are a large number of service providers.

For further information on referrals in and out of The Orange Door, refer to Section 4.1.2 and 4.2.1.

Key finding 1:

Extensive foundational work to operationalise The Orange Door was completed within a compressed timeframe. The timeframe negatively impacted on a range of establishment and operational activities. Evidence of the impact of this compressed timeframe was still being seen at The Orange Door at the time of this evaluation, despite each area managing to provide the full suite of foundational services

Key finding 2:

Significant change management to support the implementation of The Orange Door service model is needed by both FSV and partner organisations. Initial focus has been on establishment and operational activities in a compressed timeframe with a need to shift the focus now to change management support

Key finding 3:

Each area of The Orange Door areas was not at full contingency of staffing during the fieldwork. There have been a range of challenges in the recruitment of appropriate staff and in retaining those staff. This will be addressed by FSV through a workforce strategy

Key finding 4:

Roles and responsibilities need increased clarification and standardisation across partner agencies. This is important to facilitate effective matrix management

Key finding 5:

Initial induction training was provided but the content of induction training will need to evolve in response to feedback. It will also need to be tailored to meet specific workforce needs

Key finding 6:

FSV’s foundational documents provide overarching direction for The Orange Door, however they are not well understood or used by The Orange Door practitioners. Further operational ‘translation’ of these documents is needed to help the workforce understand how to operationalise the model

Key finding 7:

A range of physical infrastructure issues impacted operations and colocation of the workforce in some areas at service commencement. These have been or continue to be addressed by FSV

Key finding 8:

Collaboration is central for information sharing within The Orange Door. Despite early stage challenges with establishing strong collaboration, information sharing is already happening within The Orange Door primarily through the CRM, colocation and multidisciplinary working arrangements, and through access to employing agencies’ client record systems.

To what extent are governance arrangements for each Orange Door site operating effectively?

Governance of The Orange Door

Governance of The Orange Door is a combination of specialist service delivery expertise (from discrete service sectors) as well as government management, oversight and systems planning. A ‘hybrid model’ was conceived to support transition of The Orange Door as a new feature in the service system.[68]

The model is operationalised at The Orange Door with FSV providing oversight and leadership facilitation, and the partner agencies leading local implementation, management and service delivery of The Orange Door. Both FSV and partner agencies come together under a partnership agreement to foster collective leadership.

FSV employs Hub (The Orange Door) Managers at The Orange Door area to provide strategic and operational oversight. Hub Managers are supported and report to Assistant Directors in the Support and Safety Hubs Operations branch based at FSV.

Hub leadership groups (HLG) have been formed for each area of The Orange Door, to lead local implementation and management of the model along with an Operations Leadership Group (OLG) to manage operational processes and policies. Both the HLG and OLG comprise of a representative from each partner agency, and others (for example the Department of Health and Human Services and Victoria Police).[69]

Governance arrangements contribute to The Orange Door effectively delivering its functions

The Orange Door governance model for the areas is supported by terms of reference, Partnership Agreements (discussed in Section 3.1) and 'Service Agreements between core agencies and the Department of Health and Human Services (DHHS)' that 'may evolve over time.'[70] These form a formal basis for governance at the outset of establishment of The Orange Door.

A collective approach to governance was designed to give 'government and non-government agencies a stake in the success of The Orange Door, support collaboration, and promote shared responsibility and mutual accountability for establishment and implementation.[71]

Bringing together numerous organisations across a variety of settings has been ambitious, particularly within the challenging timeframes discussed earlier. Establishing expansive governance arrangements at The Orange Door has had its challenges, particularly in terms of coordination, embedding roles and responsibilities and providing a cohesive and united approach to implementation.

'How do you bring a group of different agencies together to conceptualise how they will think about this in a coherent way?'[72] – HLG member interview

It is important to recognise that some leadership groups are further progressed and have clearer roles and responsibilities than others. Evaluation evidence indicates that The Orange Door areas that had previous working relationships or alliances were able to leverage from these, and 'hit the ground running' when it came to service commencement. This is to be expected in a reform of this scale, given the scope of change for the sector and interests involved. Additional lead time for the OLG groups to establish as an operational group prior to commencement of a new area are recommended for future rollouts.

'It is not clear what the local governance arrangements for The Orange Door are. Before we have the risk of people falling between the gaps of services, and now we have the risk of people falling through the gaps in coordination – local coordination.'[73] – Stakeholder interview

In the early establishment phases, the lack of clarity/understanding of the governance arrangements at The Orange Door was raised in relation to the division between strategy, management and operations. While the service model, service specifications, Partnership Agreements (and HLG terms of reference included in the Partnership Agreements) provided information on the intended role of governance groups, interviewees stated that there was a need for these documents to be more prescriptive about the role of the HLG and OLG.[74] This is a challenge in the early phase of the roll out as the balance between statewide consistency and local flexibility will be evolving. It could however be expected that some iterations will be needed to governance structures and arrangements after the initial implementation phase of reforms of this scale.

At some areas, the lack of clarity about governance arrangements during the intense early implementation phase has resulted in the conflation of roles and responsibilities between the HLG and OLG, particularly during the initial months of operations. The focus for both these groups during this time was establishment and operational issues, with no clear boundary between what operational issues needed to be contained to the OLG and their decision-making authority. While greater consideration of operational matters could be expected in the early stages of implementation, HLGs now need to refocus their efforts to move beyond operational matters to more strategic matters. The lack of reporting and accountability of the OLG to the HLG has also been unclear, contributing to the HLG having detailed involvement in operations.[75] It is our understanding that most areas have now conducted workshops with their OLG and HLGs to further define governance roles and decision making and that FSV is working with partner organisations to review the Partnership Agreement.

In those areas where governance roles were experienced as being unclear, we have identified that this resulted in delays to operational decision-making. For example, there have been delays to endorsing operational processes and workflows. At some areas the large number of agencies represented at the HLG and OLG has delayed decision-making with the desire to gain consensus from all partner agencies. In comparison, The Orange Door areas with smaller HLGs were seen to be more efficient in making decisions. Pathways for escalation and identification of risks were also unclear.[76]

Governance arrangements embed collective leadership of The Orange Door

As discussed in Section 3.1, there have been challenges to establishing collective leadership quickly. This was particularly the case when community service organisations perceived that there were inequitable contributions of FTE and funding attached to service agreements. Collective leadership and the establishment of a well-functioning ‘operating rhythm’ was also challenging for community service organisations that had not previously worked together.

Collective leadership of The Orange Door and how this is structured in terms of operational management of services is within a matrix model of management. In a matrix model, the management structure has employees who report to both a ‘functional’ manager and a ‘product’ manager. While such lateral coordination can facilitate joint responsibility – a key aim of Partnership Agreements – it is also acknowledged to be one of the most complex management structures in terms of implementation and sustainability.[77] This complexity appeared to be amplified in larger partnerships of The Orange Door. Leadership teams are still trying to determine how collective leadership can work in some areas and it may require concerted effort to work through these challenges at each area.

'We’re having a leadership meeting this afternoon where [an employing agency] have got an external facilitator to work out how the matrix model works.'[78] – Practitioner interview

As efforts to establish an effective matrix model continue, the complexity of the arrangement has been experienced at The Orange Door in a number of ways:

• The authority of Hub Managers to direct The Orange Door operations. While Hub Managers are tasked with providing strategic and operational leadership, in reality this is challenged in a matrix model as all partners have shared governance of The Orange Door. As a result, no one individual or organisation can steer or guide the operations, and this can delay decision-making. Research on matrix models suggests that a lead or coordinating role is critical to success of these models.[79] Evaluation evidence indicated that the workforce looks to the Hub Manager for direction but the Hub Manager role is structured to influence rather than direct operations.
• Co-ordination in people management. In the matrix model at The Orange Door, practitioners in effect ‘report to’ or receive direction from three ‘managers’: their team leader at The Orange Door, their employing agency (for professional development) and the practice leader (for case related queries), all of which could be from different agencies. This presents logistical challenges for even basic functions, such as scheduling holiday arrangements or attendance at training sessions, complicating backfilling of positions to ensure sufficient staff contingent at The Orange Door.
• Consistency of practice due to employing agencies’ practice background. While The Orange Door has been working to adjust and clarify roles and responsibilities for staff in a formal and informal manner, some practitioners are unclear as to the roles and responsibilities of their colleagues. This has at times led to some staff reverting to pre-Orange Door practices and aligning with the roles and responsibilities of their employing agencies. For example, some practitioners reported they were undertaking more ‘case management’ type activities.

Governance arrangements support integrated practice and integration

While a vision of integrated practice was outlined in FSV’s Interim Integrated Practice Framework, the central challenge in establishing integrated service delivery in The Orange Door will be to articulate and realise a shared understanding of what this looks like on the ground.

'… nobody said, ‘this is what integration means’, so all agencies made up their own definitions'[80] – Practitioner interview

The added complexity of supporting integrated practice and integration results from the different ideological and practice frameworks that family violence, perpetrator services and family service sectors bring to The Orange Door. Services within these sectors have historically operated independently and in different ways. This complexity presents challenges for embedding integrated practice and operating cohesively in a complex matrix arrangement where leadership groups are still trying to determine how integrated practice should be operationalised. This is to be expected to some degree in the early stages of implementation and while there are many case examples of integrated practice, a clearer definition is needed on what this means in terms of staff retaining their specialisation. In particular, there are different practice approaches which have been ‘brought’ over to The Orange Door, which shape how they conduct practice, and potentially undermine integration between disciplines. Having said this, the evaluation also found a strong commitment by the workforce to integrating disciplines within The Orange Door – indicating that working through these challenges was a goal the workforce shares.

'We’ve still had cultural issues with workers saying they’re not doing women, men or children’s work'[81] – Practitioner interview

A number of leadership positions have been identified as key to driving integrated practice, specifically the Integrated Practice Lead. Currently, roles and responsibilities for this position lacks clarity at an operational level, and will need to be determined more specifically in the future.

At the time of the evaluation, no publicly identifiable access points had been formally established. The Statewide Concept outlines that satellite access points, or ‘spokes’ will be established at alternative physical locations 'where people can access support offered by The Orange Door.'[82] Spokes may be tailored for targeting specific access needs, such as for Aboriginal people or people in rural and remote areas, or perpetrators of family violence. As ‘spokes’ are established, they will deliver core functions of The Orange Door, such as 'multi-disciplinary screening and triage, needs assessment and support planning.'[83] [84]

At present, staff work from Colac (associated with The Orange Door in Barwon) and Swan Hill (associated with The Orange Door in Mallee). These have not formally commenced as publicly identifiable ‘access points’, and integration is ongoing. Notwithstanding, evaluation evidence noted differences in how Colac and Swan Hill are currently interacting with The Orange Door in Barwon and The Orange Door in Mallee respectively:

• Staff in Colac are situated within Colac Area Health, which is an integrated health service that provides medical and allied health services as well as community services. Colac Area Health operates independently of The Orange Door in Barwon, and has built on an existing service model which encompasses the colocated services at Colac Area Health including family violence case management services, family counselling, and mental health services.
• The Orange Door staff located in Swan Hill are 2.5 FTE workers and participate in the morning screening meeting held at The Orange Door in Mallee. The physical location is situated within the offices of the Mallee Domestic Violence Service (MDVS). There is some question over how this location influences the intended interdisciplinary nature of The Orange Door. Swan Hill practitioners consult with practice leads (in Mallee) as necessary. [85]

Key finding 9:

Pre-existing working relationships mobilise effective governance arrangements, as well as an agreed sense of vision and direction for The Orange Door. This can be complicated if there are numerous employing agencies involved in the partnership with competing interests and expectations of The Orange Door

Key finding 10:

The matrix management model has been challenging to implement in the context of an innovative service model and new partnership structure. As clarity in roles and responsibilities, accountability and decision-making are defined and operationalised in each area of The Orange Door, the matrix management model may become more effective for the workforce

Key finding 11:

The integration of alternative access options requires implementation in order for these to work effectively. As FSV formally establishes publicly identifiable access points, a focus will be needed on integration with the primary physical site

[1] FSV was established in July 2017.

[2] DHHS, ‘Support and Safety Hubs – flexible funding and brokerage’, 2018.

[3] FSV, Hubs Service Model Slide Deck.

[4] FSV, ‘Co-design and establishment of The Orange Door Factsheet’, November 2018.

[5] FSV, ‘Communique’ July 2018.

[6] FSV, ‘Partnership Agreements for the Support and Safety Hubs Factsheet’, July 2018.

[7] FSV, ‘Support and Safety Hubs: Statewide concept’, July 2017.

[8] FSV, ‘Co-design and establishment of The Orange Door Factsheet’, November 2018.

[9] At the time of fieldwork, Aboriginal Advisory Groups for The Orange Door areas were still being established. There was however representation of Aboriginal services on the HLGs.

[10] The Support and Safety Hubs: Statewide concept was released in July 2017. Hub Leadership Groups were convened in February 2017.

[11] This included: co-design of The Orange Door through a series of Community Conversations in Victoria led by Department of Premier and Cabinet in September 2016. This was followed by consultation workshops held in in Geelong, Pakenham, Horsham, Swan Hill, Shepparton, Sale and Craigieburn, as well as with Aboriginal and Diverse Community working groups to shape thinking on the statewide model, design principals and core elements of the model (see co-design and establishment of The Orange Door Factsheet November 2018). In early 2017, a series of workshops with an expert design group were conducted to consider the functional model for The Orange Door and interface with the broader service system. From July 2017. FSV worked with members of the Victim Survivors’ Advisory Council to obtain insights on the functions and features of The Orange Door from service users perspectives.

[12] Most of The Orange Door areas noted that while some early establishment workshops occurred in early 2017 (such as ways of working together), more formal establishment activities to operationalise the model at the local level occurred from July 2017

[13] Evaluation interview – HLG, November 2018.

[14] FSV, Communique, July 2018.

[15] FSV, ‘The Orange Door Commencement report’, May – June 2018

[16] FSV, ‘The Orange Door – commencement report’, 2018.

[17] Within the first period of operations, Barwon, Bayside, NEMA and Mallee had recruited 78.1%; 56.3%; 71.4% and 87.7% of all funded positions respectively. See The Orange Door – commencement reports. However, it is important to acknowledge that the delay in NEMA opening by six weeks meant that this workforce was at a higher level of recruitment. This then led to different workforce challenges in this site (for example, where community service organisations could place their workers in the absence of an operational physical site) prior to commencement.

[18] L17 reports are referrals from Victoria Police Victoria Police officers who attend family violence incidents make referrals (L17s) to specialist family violence services, family services and child protection as necessary

[19] Evaluation data from fieldwork

[20] FSV, ‘Lessons Learned – Tranche 1 Support and Safety Hubs’, 2018.

[21] FSV staff roles include Hub Manager, Service System Navigator, Operational Support Officer, Strategic Planning and Reporting Officers, Client Support and Administration Officer

[22] 56.5%, 45.2%, 58.8% of FSV positions filled in Barwon, Bayside and Mallee after one month of operations. NEMA had a full complement of FSV staff in August 2018

[23] It is unclear if the vacancies in these position was due to difficulties recruiting appropriately qualified personnel into these positions or the condensed time frame for establishment.

[25] It was beyond the scope of the evaluation to do a full workforce profile, an assessment of workers skills to work in an integrated manner was not able to be made. Evaluation judgements are made based on reporting of concerns about skill levels reported by multiple practitioners across multiple sites.

[26] Qualitative evaluation evidence from fieldwork

[27] FSV, ‘Communique July 2018’.

[28] FSV, The Orange Door Induction Training program. Evaluation report, 2018.

[29] FSV, ‘The Orange Door Induction Training program. Evaluation report’, 2018.

[30] FSV, ‘Workforce Strategy for the Orange Door – project outline’, 2018.

[31] This is in addition to other supporting resources such as Client Relationship Management (CRM) system guidelines (2018) Central Information Point guidelines (2018), Family violence information sharing guidelines (2018), and Family violence risk assessment and risk management framework (September 2018).

[32] Qualitative evaluation data from fieldwork

[33] Qualitative evaluation data from fieldwork

[34] FSV, ‘Support and Safety Hubs: Statewide concept’, July 2017.

[35] Information provided by The Orange Door OLG, November 2018.

[36] Please note, not all workers in The Orange Door use CRM (for example, child protection community based practitioners continue to record data using the DHHS CRIS client case record system). Please note, TRAM was initially called ICRAT.

[37] At the time of writing, CRM Release 5 training session was being rolled out.

[38] FSV, ‘CIP-CRM Lessons Learned Register’, July 2018.

[39] FSV, ‘Communique’, July 2018.

[40] Qualitative evaluation evidence from fieldwork

[41] Qualitative evaluation evidence from fieldwork. See also FSV, Communique July 2018.

[42] FSV, internal document: ‘CIP-CRM – Lessons Learned Register’, 18 July 2018.

[43] For example, NEMA requests were collected over 7.5 weeks and later in the CIP’s development as opposed to 7 weeks for Bayside, 4.5 weeks for Barwon and Mallee respectively earlier in the CIP rollout

[44] FSV, ‘Support and Safety Hubs: Statewide concept’, July 2017.

[45] We understand similar challenges have delayed opening of The Orange Door in the inner Gippsland area.

[46] FSV, ‘CIP-CRM Lessons Learned Register’, 18 July 2018.

[47] Barwon Support and Safety hub risk register May 2018

[48] Qualitative evaluation evidence from fieldwork

[49] FSV, ‘Support and Safety Hubs. Service Specifications.’ April 2018.

[50] FSV, ‘Support and Safety Hubs. Service Specifications.’ April 2018.

[51] FSV, Hub Partnership Agreement Template, October 2018

[52] FSV, Hub Partnership Agreement Template, October 2018

[53] FSV, ‘Table of governance discussions – 30 Day Review’, 2018

[54] Qualitative evaluation evidence from fieldwork

[55] HLG and OLG reported ‘false starts’ of dates for commencement, however commencement dates were always tentative and in constant reviews as physical infrastructure and premises were established.

[56] Qualitative evaluation evidence from fieldwork

[57] Qualitative evaluation evidence from fieldwork

[58] FSV, ‘Support and Safety Hubs. Interim Integrated Practice Framework’, April 2018.

[59] FSV, ‘Support and Safety Hubs. Interim Integrated Practice Framework’, April 2018.

[60] FSV, ‘30 Day review report’, 2018.

[61] The family violence information sharing and child information sharing schemes are being rolled out alongside the MARAM reforms. The family violence information sharing scheme commenced in February 2018 with a select group of workforces, including The Orange Door, permitted to share information with each other relevant to assessing or managing family violence risk. The scheme was expanded to further organizations and services in September 2018 along with the commencement of the MARAM reforms and the child information sharing scheme allowing information to be shared to promote the wellbeing or safety of a child between broadly similar workforces. These reforms will include further workforces in 020 and is expected to improve the consistency and amount of information sharing between services.

[62] For example, a multidisciplinary team supervised by a team leader who manages capacity of workload across practitioners in the team

[63] For example intake and triage at Mallee where practitioners were in an established operating rhythm

[64] FSV, ‘Support and Safety Hubs. Interim Integrated Practice Framework’, April 2018.

[65] Data on referrals was unavailable to corroborate this.

[66] Respondents to the 30 Day Review survey also identified priority service interfaces as: core family violence services and family services to facilitate allocation and referrals, crisis services, adolescent family violence services, housing and homelessness services, alcohol and drug services, mental health services, general health services, courts, police, legal services, specialist community sector organisations supporting Aboriginal, and culturally and linguistically diverse community members.

[67] Qualitative evaluation evidence from fieldwork

[68] Victorian Government, ‘Support and Safety Hubs: Concept paper’, July 2017.

[69] Description of each leadership group, membership and roles will be provided as an appendix in the final report.

[70] FSV, ‘Support and Safety Hubs. Service model’, April 2018.

[71] FSV, ‘Support and Safety Hubs. Service model’, April 2018 p11.

[72] Qualitative evaluation evidence from fieldwork

[73] Information provided during stakeholder interview. October 2018.

[74] Qualitative evaluation evidence from fieldwork

[75] FSV, ‘Lessons Learned Tranche 1 Support and Safety Hubs’, 2018

[76] FSV, ‘Lessons Learned Tranche 1 Support and Safety Hubs’, 2018

[77] Hanover Research 2013; Lawson & Gobeli 1987; Turner, Utley and Westbrook, 1988: Katz & Allen, 1985; Sy, 2005

[78] Qualitative evaluation evidence from fieldwork

[79] Hanleybrown, F. Kania, J. & Kramer, M., ‘Channeling change: Making collective impact work’ in Stanford Social Innovation Review, January 26, 2012. ANROWS ‘Meta-evaluation of existing interagency partnerships, collaboration, coordination and/or integrated interventions and service responses to violence against women’ Landscapes: State of knowledge paper. September 2015, p13.

[80] Qualitative evaluation evidence from fieldwork

[81] Qualitative evaluation evidence from fieldwork

[82] Qualitative evaluation evidence from fieldwork

[83] Qualitative evaluation evidence from fieldwork

[84] It should be acknowledged that the establishment of access points for The Orange Door was not identified as an initial priority in the first roll out of The Orange Door in the foundational documents.

[85] Qualitative evaluation data from fieldwork

To what extent is The Orange Door operating as intended, and making use of systems and processes?

The suite of foundational services being offered

The Orange Door is designed to mobilise the service system for women, children and young people experiencing family violence and families who need support with the care, development and wellbeing of children.[1] The service model documents the suite of services and responses intended to be delivered by The Orange Door in the initial establishment and the role of The Orange Door in connecting people to services. A summary of the service model is shown in Figure 4.

Figure 4: PwC Summary of The Orange Door service model

Source: PwC analysis and interpretation summary based on FSV (2018), ‘Support and Safety Hubs: Service Model’.

Prior to the establishment of The Orange Door, most people experiencing family violence were referred to individual services (predominantly by police) and/or needed to navigate the service ‘system’ to find services to meet their needs. The role of Child FIRST has been to assess the needs of families and refer them to a service that can assist - it provides an entry point to an integrated local network of family services for children and families experiencing difficulties, which The Orange Door has replaced in areas where it has been established.

Figure 5: PwC explanation of client pathways prior to establishment of The Orange Door

Source: PwC summary and explanation adapted from FSV, Support and Safety Hubs: Statewide Concept, July 2017.

The Orange Door receives L17 reports, referrals from professionals (such as child protection practitioners) or the community, as well as provides an access point for self-referrals. As the key access point into the service system, The Orange Door assesses client needs and connects them to services that can provide appropriate interventions.

Importantly, people are able to access services directly if they are appropriate or desired, except for people referred via police L17 reports, who are referred to the relevant area.

Figure 6 provides a visual representation of the critical role The Orange Door performs in assisting clients to access the services they need.

Figure 6: Client pathways in The Orange Door

Source: Adapted from FSV, Support and Safety Hubs: Statewide Concept, July 2017.

These diagrams demonstrate the significant refocus of client pathways that The Orange Door represents in the service system. To this end, we would expect very significant investment in defining, establishing and implementing new processes and systems at almost every stage of the client pathway through the system. This evaluation found evidence that The Orange Door areas involved in the evaluation had spent their early months of establishment defining and developing these new processes.

The Orange Door is largely delivering against the suite of foundation services outlined in the service model, with some elements of the model stronger than others (for instance, CIP and crisis response by The Orange Door). A focus is now needed on other elements of the model including:

• Communications and engagement with the community to promote visibility of The Orange Door: Evidence from fieldwork indicated that practitioners reported that the role of The Orange Door was not clear to many parts of the sector and to clients. This was demonstrated by providers in the broader services system directing clients to The Orange Door whose needs could be better met elsewhere (e.g. by homelessness services). This is to be expected so early in the implementation of such an innovative model as The Orange Door, but focused effort is now needed to improve the communities understanding of the model.
• Assessment and planning for perpetrators: Evaluation evidence indicated that often the default response by The Orange Door for men who perpetrated violence was to direct them straight into behaviour change programs in most cases[2]. As a practitioner stated:

'If we’re not case managing these men, we’re not creating the cycle of change. There needs to be some case management like at [name of Aboriginal service] so that they can build somewhat of a relationship…instead of just offering them one phone call' – Practitioner interview

• Outreach: The service model and specifications outline the requirements for delivery of outreach and home visits. We found that discipline-based differences (stemming from previous experience and guidance from their employing agency as well as their practice philosophy) meant that this was being undertaken inconsistently – with family violence-focused practitioners often more reluctant to undertake home visits in these early stages of implementation. Practitioners reported that they need greater clarity about when outreach was suitable and should occur.[3]
• Review and monitoring of client experience: FSV and The Orange Door have made attempts to understand client experience during or following their service from The Orange Door, including client interviews as part of this evaluation. Uptake from clients to participate in interviews as part of this evaluation however, has not been high and the limited qualitative data collected from interviews represent a small sample size of client experience. As such, experience of the service model in the early stages of implementation by clients is unable to be confirmed without validation with clients which is an area of work that FSV is currently progressing.
• Documented feedback loops with external services: Formal feedback loops do not as yet exist between The Orange Door and external services. Qualitative evidence from fieldwork indicated that there were no consistent methods to seeking feedback from external services, nor to determining whether those services had provided satisfactory service after their service had been allocated to a client or a client was referred to them. In an interview, a client called for greater ‘exit planning’ to ensure that the service responses are appropriate – in this case, the client interpreted ‘exit planning’ as the process of referring them to an external service for supports.

'I think a lot of the time, and for me too, women are afraid to even ask for [help] because, and when you’ve been in an abusive relationship and particularly like me when you’ve been in one for a very long time you lose so much sense of yourself that to actually ask or to say that look I’m not really happy with this worker I need you to refer me somewhere else is perhaps something that you’re not going to do because you’re, you don’t even know how to stand up for yourself. You lose that sense of self … A bit of more follow up I think. If that’s at all able to be done would be good.' – Client Interview

• Practice level interfaces: Recognising the breadth of service interfaces that will contribute to the effective delivery of The Orange Door, FSV is taking a phased approach to developing statewide guidance.[4] For example, the interface specifies that The Orange Door and the Victims Support Agency (VSA) will have a close working relationship, and will develop protocols to support referrals, information sharing and where appropriate, joint-agency risk assessment, needs assessment and planning, particularly in instances where the ‘primary aggressor’ is incorrectly identified[5]. Evidence from fieldwork indicates that this does not appear to have occurred at this point in time. At a local level, The Orange Door practitioners perceive that accountability for ensuring interfaces are developed lies with the System Service Navigator. However, given that the VSA is a statewide agency this may be challenging to coordinate on a local level. Service interfaces will require a focus in the next twelve months to progress.

Referrals to The Orange Door

As specified in the service model, the referral sources/access to The Orange Door include direct contact (on-site, via phone or online or through outreach), L17 reports from Victoria Police, statewide after hours services, and professional referrals (from Child Protection, maternal child health services, legal services etc). While there is data to estimate demand for some parts of The Orange Door service offering, there is not sufficient data for a comprehensive estimate of the overall demand for The Orange Door. Each data source reflects caseload for a subset of potential clients for The Orange Door. This is to be expected given the innovation of The Orange Door bringing together clients who are victim survivors of family violence, family violence perpetrators and families and children who require support.

Demand forecasts and workforce planning was undertaken by DPC and FSV prior to establishment. These forecasts used data relating to the number of L17 referrals and referrals relating to child and family wellbeing in order to help determine resourcing in each area of The Orange Door. Given that The Orange Door is a new service model, this appears to have been a practical way to estimate referrals in the absence of an existing evidence base on consolidated demand.

The evaluation found that referrals to The Orange Door are largely generated from the expected sources. Baseline analysis undertaken by FSV[6] following the commencement of The Orange Door benchmarked referral sources against their equivalent referral types prior to the establishment of The Orange Door. This analysis indicated that L17 numbers stayed relatively stable pre and post-establishment, while referrals from other sources increased compared to the number of Child FIRST cases in the previous year. Overall, most of the demand for The Orange Door came through L17s and other professionals such as teachers and health workers at the time of the evaluation. Direct contact by clients seeking assistance (comprising calls to the 1800 numbers and walk-ins) are around a tenth of all contacts. Third party referrals from members of the public make up less than one per cent.[7]

More detailed data collection and longitudinal tracking will be required to identify any precise changes or trends in relation to referral sources over time.

Demand for The Orange Door

Prior to the formation of FSV, DPC completed work on the potential demand for family violence services following the RCFV. Once established, FSV commissioned Deloitte to undertake a demand-forecasting analysis.[8] Deloitte based demand forecasts on caseload targets (which employing agencies determined using their service use data), FTE's, and data modelling – including using previous reports and predicted growth patterns. The analysis focused on service provision prior to establishment of The Orange Door and the government used this information to inform funding for community service organisations. Community service organisations then individually determined their practitioner workforce needs for The Orang Door.[9]

Box 1: Sources of demand for The Orange Door

Qualitative evaluation data from The Orange Door workforce indicates a perception that the volume of cases The Orange Door has serviced has been greater than anticipated (with the exception of Mallee).[10] We are unable to confirm the accuracy of this perception without access to prior quantitative forecasts. This is further complicated by a number of factors regarding the context of the early roll out of The Orange Door areas:

• Qualitative data from the evaluation suggests that some staff in The Orange Door are working with clients for a greater time than anticipated by FSV and the service model and specification documents,[11] meaning that cases are taking longer to progress through The Orange Door than practitioners believe they should. This appears to be primarily due to a lack of shared understanding amongst practitioners about what ‘intake and assessment’ means in practice and what types of activities they need to do and at what point service should cease from The Orange Door and be provided by an external service. Some practitioners reported that they would ‘hold’ a case if they were not confident that a client could be connected to an appropriate service (for example, if a support worker was absent) – this is however in line with the service model. Further discussion of active engagement and risk monitoring (often termed as ‘active holding’ by practitioners) is in Section 4.2.2.
• Practitioners and employing agencies reported some concerns about the capacity of external agencies to service referrals from The Orange Door. This creates a ‘bottleneck’ where The Orange Door does not allocate services or make referrals due to a perceived lack of capacity in the service system. Stakeholders interviewed, perceived that these capacity constraints are exacerbated by workforce shortages in the sector with the transition of skilled workers from these agencies into The Orange Door. The capacity of the service sector to respond to referrals from The Orange Door was not within scope for assessment as part of this evaluation but should be a key consideration of FSV and partner agencies in future evaluations of The Orange Door.
• Qualitative data from the evaluation suggests that some cases take longer to progress than others at this point in time. A number of practice leaders and practitioners reported that while family violence-focused cases could, in theory, be delivered quickly (that is, ‘screening, identification and triage’, ‘assessment and planning’ and ‘connection to service’ within a few weeks),[12] child wellbeing-focused cases are inherently longer to assess than family violence victim survivor cases and require more detailed and different assessments, including a home visit. The resourcing differences, which lead to these cases remaining in The Orange Door for longer, include the need to create profiles for all family members and the greater number of required checks.[13] Practitioners at some Orange Door areas also suggested that there is a much broader range of services and interventions associated with child wellbeing cases, which can be difficult to find and connect clients to, particularly in areas that service a large population or geographic area.[14] Current default data within CRM is unable to accurately verify the length of time cases are ‘held’ at The Orange Door at this stage; therefore, evaluation findings were not able to be quantitatively verified as to the different length of time different cases remained within The Orange Door.
• The number of workers in The Orange Door in the initial months following commencement was lower than the desired FTE levels, particularly in Barwon and BPA. The staffing vacancies in the early months of operations (particularly leading into the high demand time of Christmas) likely contributed to backlogs in case processing. Combined with issues of accessing and using the CRM, this reinforced the view among practitioners that demand was much higher than anticipated.
• Most practitioners who came into The Orange Door from Child FIRST brought with them an existing caseload, whereas those who came from family violence agencies waited for new cases to come in through the L17 portal. This has created differences in workload between practice disciplines at the outset. However, this cannot be verified by quantitative data at this point in time.

Efficiencies in processing some referrals through the L17 portal were not achieved in the early months of operations. At the same time that L17s went ‘live’ for The Orange Door, practitioners and team leaders reported needing to spend time on developing standard operating processes and procedures before L17 referrals could be responded to appropriately. This necessarily led to an immediate ‘backlog’ of referrals that were unable to be processed with the same efficiencies as experienced prior to The Orange Door. It is probable that this would not be experienced to the same degree when future areas of The Orange Door are established as these areas will be able to leverage existing processes developed by the first four areas to achieve efficiency relatively quickly.

Box 2: Quantitative data on demand

Demand for The Orange Door is primarily represented through the number of referrals and number of cases created for clients. Cases represent the number of incidences; that is one client can have multiple cases in any given period (but not at once). As shown in Figure 7, the number of referrals screened in The Orange Door areas in BPA and NEMA has been considerably larger relative to their larger catchment populations and trending higher than in the Barwon and Mallee areas. This is particularly pronounced in NEMA, where The Orange Door commenced operation in July 2018.

Figure 7: Total screenings (referrals) recorded on the CRM by The Orange Door area, July – December 2018[15] [16]

Source: PwC analysis of CRM data provided by FSV. Note that The Orange Door commenced in Barwon, BPA and Mallee in May 2018 and in NEMA in July 2018.

Screening, identification and triage

Screening, identification and triage in The Orange Door determine whether an immediate emergency response is required and/or if The Orange Door can provide the most appropriate service response. The process considers the following questions:

• Is there an immediate risk to safety for any family member?
• Is there or has there been family violence?
• Are there concerns about a child’s safety, wellbeing or development?
• Who are the people associated with the referral or contact?
• What is the priority of action needed?

In addition, the screening process includes entering referrals into the CRM, conducting history checks, identifying family members and associated people to the case or referral, with particular emphasis on discovering whether there are any children or young people involved (especially for L17 referrals). Where associated people are identified, the screening questions are applied to consider their needs individually.

Triage helps to determine the priority and urgency of The Orange Door’s response to the person, and results in internal assignment of the client(s) to a practitioner within The Orange Door.

It is important to note that each area of The Orange Door undertakes screening, identification and triage slightly differently. This may reflect a need for local flexibility or the absence of clear direction about statewide standardised processes however we have been unable to verify the underlying reason and it may be too early to determine this. Table 3 describes the processes for screening, identification and triage at each Orange Door area.

Table 3: Summary of typical processes in each Orange Door areas as of end 2018

Source: Information provided by The Orange Door workforce in fieldwork. Note that Barwon and Mallee have similar intake structures so have been grouped together, as do BPA and NEMA.

Evidence from fieldwork indicated that assessment of some cases was delayed for months at a time, especially where demand was high or where a backlog existed. This was exacerbated by the prioritisation of self-referrals which led to third-party referrals being continually 'overtaken in the queue' in the very early stages of implementation.[17] This was verified through a client interview (below), however it is important to note that initial interview information from clients would indicate clients have limited (if any) understanding of the different stages of The Orange Door process. This actually reflects a seamless service from the perspective of the client and is a positive early finding of The Orange Door experience. This could be further explored through FSV’s current program of work on client experience and feedback.

'The hardest thing that’s happened is I’m still under the intake team [after around four months], waiting to progress on to the next service that I’m supposed to go to. So, like I’m still dealing with the original person that I dealt with who kept saying: ‘I’m only the temporary worker, I’m only the temporary worker.’ And, which I’m so lucky, she’s fantastic. But I think that when you’re like in this sort of situation and like this is the first time I’ve actually felt safe and settled that sometimes you want a bit, you know a bit more solid thing. Like this is my worker. I’m going to be able to tell her this stuff and I’m not going to have to repeat it again in a month or so when I finally get a worker type thing. But she’s been really good, and she’s been all over everything …. It’s just that next step to more permanent. To help you settle in yourself'. – Client Interview

'Don’t take forever. I mean my situation is not urgent, but it might be different if there’s women out there who need like urgent help. Just don’t f*** around on the poor lady. [Provided specifics about delays in housing and legal processes.] Just like get things done for families in general.' – Client Interview

Despite the time delays experienced by some clients, even in this early stage of implementation there are indications that most clients are effectively screened, triaged and provided with a response to their immediate risks and needs.

Apart from the number of screenings recorded on CRM, quantitative data on the screening, identification and triage stage of the client pathway is not readily available at present but would be expected to become available as the CRM matures in function.

Key finding 12:

The Orange Door offers all functions required to operate and provide service to clients. While to be expected at this very early stage of implementation, some of these functions require refinement and/or standardisation to work effectively and efficiently for all clients. However, early data from clients indicate that some are experiencing ‘seamless’ services while they transition between functions within The Orange Door

Key finding 13:

The ‘review and monitoring’ function articulated in the service model is not established yet and will mature over time as The Orange Door rolls out across the state

Key finding 14:

The ‘connection to services’ and ‘service response’ functions in the service model are not yet fully matured in The Orange Door areas. This is to be expected at this early stage of implementation

Key finding 15:

There are distinct differences in the way ‘walk-in’ and phone referrals enter The Orange Door compared to L17s, emails and Child Protection referrals. Evaluative judgements on the impact of these differences on service quality and client experiences are unable to be made at this point in time

Key finding 16:

A combination of high demand, staffing vacancies and a lack of clarity by the The Orange Door workforce about how to operationalise appropriate screening, identification and triage processes, resulted in some clients experiencing significant wait times for support and referrals in the early stages of implementation. This will need to be monitored to determine if this is a point in time or systemic issue

To what extent are clients connected to the right services at the right time?[18]

Service responses are tailored to need and risk

The Orange Door is intended to work with clients to tailor their safety plans and/or risk management plans, provide targeted interventions and connection to services based on individual needs, as identified through screening, triage, identification and assessment.[19]

The two primary determinants of whether service responses are tailored to need and risk are:

1. Whether the risk and needs assessments have been undertaken effectively (see Section 4.3) to inform tailored responses.

Evaluation evidence indicates that risk and needs assessments appear to be undertaken effectively within The Orange Door. The majority of The Orange Door workforce appear to have sufficient experience to recognise risks to their clients and what they need to manage those risks. Please refer to Section 4.3 for a more in-depth discussion of risk assessments.

Given the limited data on risk and needs assessment recorded on the CRM at this point in time, it is difficult to make more in-depth conclusions about the quality of risk assessments and if responses to these assessments are effectively tailored to meet the needs of clients.

2. The capacity of the service system to provide tailored responses.

The Orange Door workforce has limited awareness of the service system apart from individual professional knowledge from prior work experience. Automated systems are not yet in place for practitioners to determine capacity or waiting times for services to which they may wish to refer – meaning practitioners are often reliant on a range of, often time consuming processes that have been developed with core agencies to support allocations. This limits practitioner’s ability to ensure that the support provided is tailored to an individual client’s risk and needs. As detailed in Section 4.1.1, the review and monitoring aspect of The Orange Door model and the collaboration and connections with the broader sector system is still being established and would not reasonably be expected to be established at this early stage of the statewide The Orange Door roll out.

Clients interviewed for this evaluation raised concerns with the appropriateness of service responses, as the following anecdotes highlight:

• A client reported feeling unsure about whether they were able to provide feedback to The Orange Door if their relationship with their case manager was not providing what they needed.
• One client reported that the service response represented 'tinkering around the edges' rather than addressing fundamental factors in their situation.
• One client did not receive support from the service they were referred to, even after following up with The Orange Door practitioner they were working with.
• One client reported that they believed The Orange Door workers were not necessarily aware of the services available in the broader system.

As with risk and needs assessments, data recorded on the CRM relating to connection to services is limited at this point in time. However, available data recorded on the CRM does indicate that around half the cases in The Orange Door areas are closed after the client had engaged with the service system or had their needs met.[20] This is a positive outcome early in the roll out of The Orange Door.

There are however, a significant proportion of Orange Door clients whose cases are closed without them having accessed services or received support from The Orange Door. For example, around ten per cent of clients refuse services and a large proportion in each area are not contactable (however we do not have any further details about these clients at this point in time). The closure of cases where contact has not been made may or may not be appropriate for any given case. No evaluative judgement is made here relating to the appropriateness of case closures.

Similarly, data from client interviews as part of this evaluation were limited and may be influenced by negative bias (when people are more likely to recall the things they didn’t like in comparison to what they did like). These factors make it difficult to draw firm evaluation conclusions about whether service responses are being tailored to need and risk at this early stage of The Orange Door roll out.

Active management and support

Following assessment, allocations of services to clients or referrals to services can occur immediately. However, if this is not possible or appropriate,[21] active engagement and risk monitoring (often termed ‘active holding’) responses must be put in place.[22]

The line between limited interventions as part of the assessment, active engagement and risk monitoring stages and case management is often blurred in practice. At a broader level, a number of FSV staff and The Orange Door workforce reported uncertainty about whether practitioners were undertaking more active engagement and risk monitoring (active holding) than is intended in The Orange Door model. Practitioners gave a range of reasons why this lack of certainty might be the case:

• Many practitioners reflected that they had been hired from roles where they undertook case management and it is their established practice to support clients through difficult periods. In addition, they reported that their position descriptions and expectations set by employing agencies did not always reflect the intake and assessment role of The Orange Door and they had understood their roles would have more substantive case management responsibilities.
• Confusion about the role of The Orange Door practitioner in the overall service to the client. For example, some practitioners reported that they understood that the service model called for case managers in external services to be the primary conduit with schools for a client’s case, however, they were criticised by these external case managers if The Orange Door had not already contacted the school as part of the assessment phase. The service model does not state that this needs to occur, but clearly there is a misalignment in the expectation of external case managers and the role of The Orange Door in these cases.
• Some practitioners reported that it was difficult not to become emotionally invested in certain cases, particularly where the victim may be at immediate risk of harm, and they were inclined to use any resources available to assist. This lengthened practitioners engagement with the client beyond immediate assessment, planning and rapid referral.
• Some clients develop a personal bond for the first person they engaged with for assistance, which is often The Orange Door practitioner. One practitioner described a client calling her months after she first entered The Orange Door and asking for help or advice. While there is deliberate flexibility built into the model to accommodate client preferences, it was unclear to practitioners at this early stage, how they could make decisions about which aspect is appropriate to vary for particular client needs.

In very few cases did practitioners mention that their active engagement and risk monitoring (‘active holding’) was coordinated in collaboration with core services. In most cases, it appeared that they were supporting clients using their own knowledge and expertise.[23]

Interviews with clients revealed mixed attitudes towards greater involvement by The Orange Door in their cases:

• One client wanted The Orange Door to keep their case open in order to be able to follow-up to ensure the service response provided was appropriate and satisfactory.
• Client interviews revealed a desire by clients for more assistance to both manage and remain alert to the various support options available:

'The coordination….like if Orange Door could sort of oversee bits, to make sure things are being done. Like following up. And letting you know everything that actually is available out there ….'[24]. – Client Interview

One client felt that the scope of The Orange Door was too narrow:

'I like to think of an iceberg. You know, their initial tip is like really good. And you know you’re sort of balancing up on this tip and you’re like working it all out, doing it and it’s supportive and then, but when it gets harder as you go along, which is usually the case when you’ve got like in-depth trauma and all stuff the family law issues and everything else at the bottom of the iceberg is so much bigger. And all you’re doing is just rocking that iceberg. You’re not chiselling at it. And I think that, that’s what if Orange Door is going to help you with that top bit I think it needs to continue down at least sort of to the water line... And like I know there’s so many women and so many families and the men that go through this it’s going to be difficult to spread it around, especially for such a new service, but I think that looking at the mission and thing of Orange Door like I hope going forward that it’s going to be that thing that can make you chip that iceberg down. You know, not just leave you floundering like everybody else does.'[25] – Client Interview

Evaluation data from both practitioners and clients highlight that understanding of The Orange Door and its role is only just emerging within the community. This is to be expected this early in the implementation and roll out of The Orange Door. Clearer messaging at the intake stage may be required to establish client expectations regarding the role of The Orange Door.

Key finding 17:

The capacity of the system to provide tailored service responses is limited by The Orange Door workers’ knowledge of services in the sector but will be enhanced once the ‘review and monitoring’ function is fully developed

Key finding 18:

Data limitations mean that we are unable to make firm evaluative judgements about whether service responses are tailored to need and risk at this point in time

Key finding 19:

Clients who are not able to be immediately allocated or referred to services are provided with active management and support

Key finding 20:

The Orange Door workers would benefit from clearer guidance using examples from practice about where their role in screening, intake and assessment ends, including ‘active holding’, and where case management begins

Key finding 21:

Many of the (limited sample of) clients interviewed as part of the evaluation expressed a desire for greater support from The Orange Door, even after services had been allocated to them, or they had been referred out to services

To what extent is The Orange Door effective in assessment and management of risk and need?

Risk and needs assessments

Risk assessment in The Orange Door aims to understand any child wellbeing concerns, family violence risk to all family members and the risk posed by the perpetrator. The Statewide Concept envisages that The Orange Door workers use consistent processes and tools to assess risk.

In practice and at this point in time in the early operations of The Orange Door, consistent approaches to risk assessments are only emerging. The evaluation found:

• Individual practitioners reported that they used family violence risk assessment in different ways. Some found the risk assessment structure helpful while others considered them mainly as extra paperwork. Evidence from fieldwork indicated that some practitioners do not upload all their risk assessments to the CRM (either as a scanned copy or by inputting information to TRAM) at this point in time.
• Each of The Orange Door areas appears to have produced its own template for child and family wellbeing risk assessments, apparently due to the need to shorten the standard template. Some practitioners also stated that they found the Best Interests Assessment (BIA) used in their area was not well structured.[26] This means that risk assessments of child wellbeing concerns may be undertaken using different considerations and focusing on different factors in each area (which is consistent with what was occurring prior to the establishment of The Orange Door).

There appeared to be inconsistency about whether a multidisciplinary lens was applied to risk assessments. One practitioner stated that:

'…a lot of practitioners don’t understand the risk assessments coming out of the triage team…they often look at it from a family violence or child wellbeing perspective…not from a combined perspective'[27]. – Practitioner interview

There was some confusion about when the RAMP should become involved and where it should sit (that is, within The Orange Door structure or outside it). An example of this was the description of the screening meeting that occurred each morning in Mallee (attended by team leaders and practice leaders) as a 'mini-RAMP' where multidisciplinary risk assessment occurred.

Figure 8 describes the numbers of risk assessments (specifically the Interim Comprehensive Risk Assessment Tool (ICRAT) undertaken in July and August 2018. ICRAT was updated and renamed TRAM so the figure also shows TRAM following its introduction in September 2018). These started relatively high at the start of 2018-19 and fell in subsequent months, with the exception of Barwon, which had the opposite trend. While there is not a clear explanation for this, it may be because practitioners were not recording risk assessments in CRM, were not using the tool, or were finding it challenging to implement the data collection methods required.

Figure 8: Total number of risk assessments by The Orange Door area, July – December 2018

Source: PwC analysis of CRM data provided by FSV. Note that The Orange Door commenced in NEMA in July 2018. Note that ICRAT was replaced by TRAM in late-August 2019. ICRAT included four assessment types: Adult Victim Survivor, Child Victim Survivor, Adult Perpetrator, and Adolescents Who Use Violence. TRAM includes two assessment types: Adult Comprehensive Tools and Child Comprehensive Tools. The total shown in the figure above is the total of all types of ICRAT/TRAM risk assessments.

One way to gauge the level of risk faced by clients of The Orange Door is to use the categories assigned to them in risk assessments reported on the CRM. As shown in Figure 9, in all areas except for Mallee, around a third of risk assessments are classed as ‘elevated risk’, and at least five per cent are classed as ‘requires immediate protection’. In Mallee, these proportions are lower. Given that around a third of risk assessments are kept ‘blank’ in the CRM in order to finalise the assessment, it is difficult to determine whether this is due to practice differences or poor data recording at this early stage of implementation.

Figure 9: Level of risk identified, October – December 2018

Source: PwC analysis of CRM data provided by FSV. Note that as around a third of risk assessments are recorded on CRM as ‘(blank)’, these proportions may not be an accurate reflection of actual risk as assessed by practitioners.

The use of CIP in risk assessments is detailed in Section 5.3.1.

Client choice

Ensuring choice and control is a fundamental part of The Orange Door model – and the family violence reforms more broadly – to put people at the centre and in control of their lives. The concept predicates that The Orange Door will provide accurate information and support to enable people to make informed choices about their own needs and priorities and decide what they need to be safe and to recover. The Orange Door, as outlined in the Statewide Concept, is designed to specifically support the choice and self-determination of Aboriginal people, including referrals to an Aboriginal service or support by an Aboriginal worker.[28]

Within The Orange Door, evidence from fieldwork indicates that there appeared to be a strong commitment to a client-centric approach. However, practitioners typically discussed how they took client preferences and choice into account during the intake and ‘connection to services’ stages rather than the assessment and planning stage. During the assessment and planning stage:

• There was a greater focus on acquiring information to inform risk assessments, for which practitioners were more directed by the TRAM templates than client choice. Nonetheless, client voices were an important part of safety planning, as practitioners needed to determine what was realistic for the victim-survivor or family.[29]
• Aboriginal clients’ voices appeared to take a stronger role during the assessment and planning phase, with practitioners and Aboriginal workers in particular recounting how the support of the community[30] was an important factor to consider in risk assessment and planning.
• Practitioners reported that it is relatively easy to deliver on immediate client requests and choices in crisis situations involving family violence. Responses to requests for lock changes or installation of security equipment appeared to occur relatively fast and practitioners were able to give many examples of immediate support like this. In general, practitioners always followed requests by victim-survivors not to contact perpetrators, however, evidence from fieldwork indicated that some practitioners had been directed by their employing agency not to wait for a victim-survivor to give their permission before contacting the perpetrator.[31] There is some variability across areas in relation to gaining the approval of the victim survivor ahead of contact being made with the perpetrators. This may reflect different practices within the sector and may require further consideration to support the development of a consistent approach. There was insufficient data from the evaluation to determine if this is a widespread practice in The Orange Door.
• One area where practitioners reported that they found it challenging to take clients choices and preferences into account was in relation to non-family violence child wellbeing cases – particularly where there may be the potential for Child Protection involvement. In contrast to family violence cases where an adult (most frequently a woman) is usually a victim-survivor, in child wellbeing cases, no adults may be victim-survivors of family violence. While child wellbeing-focused practitioners are highly skilled at engaging with children, clients under 18 years of age have less legal and financial agency to make their own choices.

Feedback from client interviews indicated that while clients did not distinguish between assessment, planning by The Orange Door and connection to external services, some expected ongoing engagement with the practitioner at The Orange Door:

One client was satisfied with the assistance that they received, but wished The Orange Door practitioner was able to ‘go deeper’ into the fundamental issues causing them difficulty.

A client recounted how The Orange Door consulted with them before reaching out to their violent partner and kept them informed throughout the engagement.

Clients expected The Orange Door workers to be proactive about suggesting potential solutions (such as service responses) and supporting choices about services and solutions.

While data is limited it is a very positive early finding that some clients are experiencing ‘seamless’ services within The Orange Door and have a positive relationship with The Orange Door workers. Further exploration is needed to better understand if and how client choice is incorporated into assessment and planning functions at The Orange Door.

Key finding 22: Qualitative data from fieldwork indicates that risk and needs assessments appear to be undertaken somewhat inconsistently within The Orange Door. Consistency in recording of risk assessments require monitoring by FSV to ensure that quality and consistency of data recording continues to grow as service delivery continues and the service model evolves. Validation through client feedback is required to determine if risk and needs assessments lead to responses that are tailored to the needs of the client

Key finding 23: The Orange Door practitioners have access to a range of tools to assist them to assess risk. Most of the family violence specific tools are standardised and in early stages of embedding in practice. The child wellbeing tools are not yet standardised (nor was there a standardised tool prior to the establishment of The Orange Door)

Key finding 24: The Orange Door practitioners report a strong commitment to a client-centric approach to practice

Key finding 25: There is some evidence that at these early stages, the voices of Aboriginal clients play a strong role in service choice and in determining the support provided to them

Key finding 26: Client preferences and choice appear more likely to be taken into account in relation to referral rather than assessment and planning, and more likely for adult clients of The Orange Door

[1] FSV, ‘Support and Safety Hubs: Statewide concept’, July 2017.

[2] The FSV Perpetrator Practice Guidance provides a range of service responses including crisis support and referral to MBC or men’s case management as options for responding to perpetrator risk.

[3] Qualitative evaluation data from fieldwork

[4] Current interface guidance has been developed for the interface between Magistrates Court of Victoria, Victim Support Agency, Child Protection and Integrated Family Services, Men’s Referral Services, RAMP, Victoria Police and Sexual Assault services.

[5] FSV, Statewide concept: service model, April 2018.

[6] Baseline Analysis conducted by FSV, December 2018.

[7] PwC analysis based on CRM data for July – December 2018 provided by FSV in January 2019.

[8] These forecasts were not provided to PwC to inform the evaluation. As such, no conclusions on the appropriateness of demand forecasts or method of calculation have been made in this report.

[9] Information provided by FSV, January 2019.

[10] Mallee experienced steady demand for its services following commencement, however, due to the higher FTE staffing at this time, practitioners reported that they perceived they were better able to deal with this caseload.

[11] Information provided by FSV, November 2018.

[12] Practitioners and FSV staff refer to these functions as intake, assessment and referral out.

[13] Qualitative evidence from fieldwork

[14] Practitioners noted the array of interventions and services that could be appropriate for child wellbeing cases, such as playgroups, maternal and child wellbeing, etc. which can be difficult to keep a current view of.

[15] CRM data for Barwon, BPA and Mallee in May and June are not shown as data collection and validation in these periods was inconsistent among The Orange Door areas.

[16] CRM data used throughout this report was extracted progressively from October 2018 to January 2019.

[17] Qualitative evaluation data from fieldwork

[18] Aligns with the Family Violence Outcomes Framework system domains. Indicators to inform this domain may be refined based on findings/available data from this developmental evaluation of The Orange Door

[19] FSV, ‘Support and Safety Hubs: Service model’, April 2018.

[20] PwC analysis of CRM data for October – December 2018 provided by FSV, January 2019.

[21] Based on qualitative evaluation evidence from fieldwork, this primarily appears to be where service capacity is not available, or the appropriate response is not yet clear.

[22] FSV, ‘Support and Safety Hubs: Service model’, April 2018.

[23] Qualitative evaluation evidence from fieldwork

[24] Qualitative evaluation evidence from fieldwork

[25] Information provided by a client during fieldwork

[26] Qualitative evaluation data from fieldwork

[27] Qualitative evaluation data from fieldwork

[28] FSV, ‘Support and Safety Hubs: Statewide concept’, July 2017.

[29] Information provided by The Orange Door workers during fieldwork.

[30] Community in this instance refers to community from an Aboriginal service system point of view

[31] Practitioners who work with perpetrators stated that at the commencement of The Orange Door, they had waited for the family violence-focused worker to inform them that the victim-survivor agreed that their partner could be contacted, however, after this policy resulted in a sharp reduction in referrals to men’s behaviour change programs, it was agreed that the respondent on an L17 could be contacted immediately. Refer to the Interim Integrated Practice Framework and Perpetrator Practice Guidance for further discussion on the sharing of information related to risk.

To what extent is The Orange Door visible, accessible and safe (including culturally safe) at each site?

Physical location and physical accessibility

The Support and Safety Hubs: statewide concept envisaged that The Orange Door would be accessible through an area-based access network that includes a primary physical community-based site as well as outreach or mobile Hub practitioners who can engage with people where they feel comfortable.[1] The Orange Door was intended to initially operate from 9am to 5pm Monday to Friday (with the intention of including extended hours – although this is not part of initial establishment at this stage). The physical sites are designed to have a reception area where members of the public can make enquiries. Perpetrators are not to be encouraged to come to The Orange Door as their first contact with the service system, and Men’s Behaviour Change programs are not delivered at The Orange Door (physical) site.[2]

The locations of The Orange Door have been chosen for their visibility and proximity to public transport and other community services/institutions. Table 4 provides a summary of the features of The Orange Door locations.

Table 4: Physical features of The Orange Door physical locations

The Orange Door primary locations appear to be situated in relatively visible and accessible areas, with access to public transport. Evidence from fieldwork indicates concerns that it may be difficult for some clients to access the locations from public transport if they did not live nearby a train line. Despite the limited number of clients interviewed for the evaluation, a number of clients stated that physical accessibility was often challenging, even for those with a car. For example, client feedback indicated that they were not able to find disability parking near one physical site of The Orange Door.

However, it was accepted that the locations chosen were the most acceptable locations within the DHHS area, and that increasing accessibility would require greater outreach or opening of formal access points. The opening of access points is planned and will come online in terms of the planned roll out.

Other community services are located near all of The Orange Door locations, however certain areas (The Orange Door in Mallee, Bayside Peninsula Area and the staff based at Colac) appear to have stronger connections with and between nearby organisations based on their proximity. Moreover, evidence from fieldwork indicates that it was not uncommon for clients to access The Orange Door straight from the nearby courts, and this was particularly the case in NEMA.

The Orange Door buildings themselves have all been fitted out similarly, with a consistent ‘look and feel’ for the reception and client areas. This involves a large open wooden desk at the reception area, lockable rooms for interviews and lounge areas (in all locations, this is situated behind a sliding door that can be electronically locked). Opinions differed on the appropriateness of the reception and client area:

One client interviewed compared it to sitting in their lounge room and commented that their children were given things to do and eat.

Some staff expressed concern that the reception area was not able to be easily secured in case of a disturbance or aggressive person. Some of the CSOs – who sit behind the reception desk – reported they felt exposed by the open desk, and indicated that clients sometimes did not appear to know where to stand and needed to come to their side of the desk watching them input their details on their computer[3]. Furthermore, in Bayside Peninsula, the electronic sliding doors used to separate the reception area from the foyer and the client lounge area from the reception area were not able to be closed quickly to stop someone getting through.

'Families I know would be intimidated coming into a building like this, the fear of them taking their children away, whether they’re going to be treated with respect. The physical space isn’t safe to be able to deal with trauma, people are having to wear sunglasses inside.' – Practitioner interview

There has been a small number of situations where physical security issues have been raised and subsequently improved at The Orange Door. For example, we understand there were security concerns relating to CCTV screens inside interview rooms where perpetrators can be interviewed and that the screens have now been turned off in response to these initial concerns.

A separate office section for practitioners and FSV staff is accessible from the client area through locked door. Most staff view the office spaces as broadly suitable and that it was ‘their home’. It is a positive finding that some staff are reporting a sense of belonging to the space so early in the roll out of The Orange Door. Localised concerns around hot desk arrangements, comfort factors (e.g. blinds) and site specific safety issues were raised during fieldwork and are to be expected in the very early stages of fit-out of a new premise.

Access options

Apart from physical access to The Orange Door, the service model specified a number of options to access its services. The model outlines how the Service System Navigator plays a role in developing access options within the local context of the area.

Telephone and email access

Each of The Orange Door areas has implemented its telephone (toll-free number provided by FSV) and email access slightly differently. These two access options have been established and are operational in each of the four locations. The primary concern regarding these options has been how to manage demand. One example of this is that in BPA, practitioners reported that clients had told them that phone calls that were not answered rung out – that is, there was no voicemail system or other overflow mechanism such as a central call centre to capture them.

Outreach

Practitioners are expected to undertake outreach with clients and families in settings where they feel most comfortable. The extent of outreach undertaken by practitioners at this point in time differed based on their primary specialty of practice:

• Child and family wellbeing practitioners are experienced in undertaking home visits as part of their previous practice in Child FIRST, Child Protection or other family services. They reported that they still preferred to undertake home visits as part of the assessment and planning stage where possible, although time constraints meant that this was not always possible.
• Family violence-focused practitioners reported reluctance about home visits due to risk of violence from the perpetrator.
• Home visits were a regular part of Aboriginal practitioners’ work. They reported that they would often accompany other workers and that ‘showing their face’ often led to increased engagement from the family.
• Community-based Child Protection practitioners also undertake a large number of home visits. Primarily this was to ensure the families understood the possibility of formal engagement by Child Protection were concerns not addressed.
• There was some evidence of emerging practice in one area of police accompanying practitioners on home visits to try and engage families earlier.

Practitioners reported being unsure if the way in which outreach and home visits were occurring at this early stage of implementation were appropriate and in line with the intended The Orange Door model. Further definition (and/or reconfirmation) of the outreach role will be required once The Orange Door is through the initial stages of establishment to ensure the outreach access option is developed.

Alternative physical access options, including out posted workers

The concept envisaged that a limited number of The Orange Door practitioners will operate as out-posted workers from their employing organisations with the capacity to deliver the full Orange Door foundational service offering.[4] While this has not yet been formally established as intended in the model, alternative access points have already emerged.

• Swan Hill: 2.5 FTE The Orange Door practitioners were working in Swan Hill at the time of the evaluation fieldwork. During the morning screening meeting held in Mildura, those cases that enter The Orange Door from Swan Hill or surrounding areas are assigned to the practitioners in Swan Hill. The physical location is situated within the offices of the Mallee Domestic Violence Service (MDVS).
• Colac: The physical location is within Colac Area Health, which is an integrated health service that provides medical and allied health services as well as community services. The existing service model means that there are well-established working relationships between practitioners at The Orange Door and other colocated services such as family violence case management services, family counselling, paediatricians, mental health, drug and alcohol services.
  • As the service is located within a broader integrated service model, The Orange Door has been able to engage with external services in Colac Area Health to promote the services of The Orange Door, and build the capacity of more mainstream services to identify family violence and refer to The Orange Door.
  • The close proximity of these services within the one physical site makes it easier for informal information sharing such as following up on a referral. Practitioners in Colac are more easily able to seek information from other services such as medical health records and homelessness as they are all located on site.
• An alternative pathway for L17 reports involving an Aboriginal Affected Family Member (AFM), where L17s originating from Mildura and Robinvale are sent directly to Meminar (an Aboriginal service), while also being visible to The Orange Door staff. This is based on the Koori police family violence protocol, which was in place prior to The Orange Door.[5]

At the time of the evaluation fieldwork, BPA and NEMA only operate at their primary locations. There is evidence that alternative access points are evolving based on local needs as envisaged in The Orange Door concept.

Service System Navigator

The Service System Navigator (SSN) role (employed by FSV) is currently the key networking element in The Orange Door, designed to:

• establish and maintain practice interface agreements with key services in the area, as well as partnering with the local service sector to identify opportunities to address local service gaps
• resolve system and navigation access issues and work with the governance structure to effect change
• monitor progress of service engagement, connections and service capacity and provide advice to internal and external stakeholders regarding service interfaces

Evidence from the evaluation identified that the role is not currently functioning as intended in these early stages.

• Many practitioners reported having little if any interaction with the SSN (some did not know which person was in the role at their area of The Orange Door) and were not clear what the value of this role was. This is in many ways to be expected during these early months where the focus of The Orange Door is on initiating service delivery, operationalising the model and ensuring basic business processes were in place.
• Other practitioners stated that it was useful for the SSN to be in place before The Orange Door commenced operations, but that until the client pathways had been clearly defined by FSV through the SSN in each area, it was difficult for the practitioners to establish connections.
• Some SSNs appeared to lead the operationalisation of integrated practice in the absence of other practical clarification. This was not a responsibility identified as part of the SSN role and may be specific to the early implementation phase and start–up demands.
• There was concern about whether the SSN role had appropriate authority to create the partnerships on behalf of The Orange Door, particularly because outside stakeholders and community sectors had no reference to what the SSN role was (at that point in time). While there are position descriptions that define these roles, at an operational level at this early stage of implementation, it is unclear to the workforce if the SSN and Hub Manager have the same authority or decision making level within The Orange Door (despite being at the same VPS grade levels).
• In the service model, the SSN forms part of a feedback loop with services and advice to practitioners on the service system. There was no evidence during fieldwork of this feedback loop occurring at this early stage of implementation. Partially, this is as the ‘review and monitoring’ stage of the service model has not been implemented (see Section 4.1.1), however, this is may also be due to the SSN’s inability to have an understanding of demand of at this early stage of operations.

Safe and inclusive environment

Cultural safety and accessibility for clients and workforce

The concept paper specifies that The Orange Door will:

• Support the choice and self-determination of Aboriginal people by recognising their needs and working closely with Aboriginal services, including Aboriginal Controlled Community Organisations (ACCOs) where present.
• Actively tailor their services to the needs of multicultural communities in their local area, including access to interpreter services and employing workers from multicultural communities.
• Have the capacity to recognise and meet the specific needs of people with disabilities, people from LGBTIQ communities, older people experiencing violence and adolescents who use violence in the home.

The Orange Door has made a commitment to inclusive practice for Aboriginal communities. This has included the establishment of an Aboriginal Practice Leader role. The feedback from practitioners has been very positive about the role of the Aboriginal Practice Leaders. There is not data available at this point in time to determine whether the current FTE resourcing for Aboriginal Practice Leads in each area is sufficient to meet demand, or how cultural safety practice is being embedded through their work to The Orange Door workforce. Practitioners are clear when they should be engaging with the Aboriginal Practice Leader for Aboriginal clients and are grateful for the guidance and support:

'We try to identify Aboriginal clients very quickly, but then I will go on to ask what mob they’re from… Indigenous workers will have fantastic engagement skills and that’s important because people are very fearful in that community because of what happened historically.' – Aboriginal Practice Lead interview

'The work around Aboriginal clients is amazing – everyone is consulting [practitioner name] on Aboriginal clients, and now [the Aboriginal service] is saying we can’t cope with our demand, we’ve never had so many.'

'I went on a [home] visit with a non-Indigenous worker and she [the client] didn’t open the door to her but as soon as I showed my face, she opened the door.'[6] – Practitioner interview

There is also evidence that practitioners at The Orange Door are using the Aboriginal Practice Leaders to support their practice.

In terms of Aboriginal inclusive practice within The Orange Door, there are still fundamental differences in Aboriginal practice that have not been resolved and cannot be resolved through cultural awareness programs alone. For example, the language and approach to engaging with perpetrators is very different between mainstream and the varied approaches in Aboriginal practice, which may not even use the term perpetrator or the concepts of ‘holding to account’:

'For non-Aboriginal people, they name the things they do, whereas we have always done practice without naming it.' – Practitioner interview

'Working with the whole family – it’s an Aboriginal approach, an Aboriginal AFM will always ask for help from the partner.' – Practitioner interview

While cultural safety programs can cover content such as the impact of colonisation, disadvantage and trans-generational trauma, there is room for improvement to build the skills and capacity of non-Aboriginal practitioners to engage with Aboriginal clients. One of the key mechanisms to build this capacity and resolve differences in Aboriginal practice with other service’s practice is to embed Aboriginal voices and engagement from the outset of service design and delivery. FSV has the foundations for this in the development of a range of guidance and policy documents including Everybody Matters: inclusion and equity statement and the toolkit and resources being developed through the Intersectionality Capacity building project.

Consideration of cultural safety includes not only working with clients but the safety of the Aboriginal workforce within The Orange Door. The feedback from the Aboriginal workforce in The Orange Door is mixed.

'All the Indigenous people here migrate together, sometimes we don’t feel safe in this space, to be you… The presence of more diverse people would make us feel safer.'

'As an Aboriginal woman, I feel FSV and the Manager here are very respectful to Aboriginal culture, from the heart, not just in a tokenistic way.' – Aboriginal practitioner interview

Box 3: Quantitative data on Aboriginal people accessing The Orange Door

The Orange Door appears to be receiving a relatively consistent number of cases involving Aboriginal people as shown in Figure 10. Across all The Orange Door areas, 7.3 per cent of clients are recorded as Aboriginal.

Figure 10: Number of clients identified as Aboriginal and/or Torres Strait Islander by The Orange Door area, July – December 2018

Source: PwC analysis of CRM data provided by FSV. Note that The Orange Door commenced in NEMA in July 2018. There is a large proportion of clients where Aboriginal status is not recorded (‘blank’) so the actual number may be higher.

Based on the available data and evidence from fieldwork, it appears that Aboriginal and/or Torres Strait Islander clients are using The Orange Door; however they may be under-recorded in the CRM.

Cultural safety and accessibility for clients with diverse needs

Evaluation of the accessibility of The Orange Door by clients from the diverse cohorts identified by the RCFV is limited at this point in time. This is because there is currently inadequate data on clients with diverse needs recorded on the CRM and because of the small number of client interviews conducted for the evaluation. It is important that The Orange Door can meet the specific and intersecting needs of clients with a disability, older people (particularly those experiencing family violence in the form of elder abuse), LGBTIQ clients, children and young people (either as victim survivors of family violence, perpetrators of family violence and/or those subject to child wellbeing concerns) are responded to by The Orange Door. As mentioned above, FSV has the foundations for this in the development of a range of guidance and policy documents including Everybody Matters: Inclusion and Equity Statement and the toolkit and resources being developed through the Intersectionality Capacity building project.

There was limited evaluation data available on the cultural safety and accessibility of The Orange Door for clients with diverse needs at this stage. The data that was available was generated through qualitative interviewing of The Orange Door workforce and through stakeholder interviews with peak organisations. This data indicates that attempts are being made by The Orange Door to tailor responses to clients with diverse needs. Examples include assigning practitioners to clients of a similar cultural background, or referring clients to services that are known to have culturally tailored programs. However further investigation is required to make evaluative judgements on the accessibility and cultural safety of The Orange Door for these clients. It is important to acknowledge that the findings from this evaluation are that we have insufficient data to evaluate how culturally safe and accessible The Orange Door is at this point in time – this does not mean that The Orange Door is not providing accessible and safe services, rather that we have insufficient evidence to make an evaluative judgement.

Box 4: Availability of quantitative data on clients with other diverse needs

At this point in time, quantitative data relating to clients with other diverse needs, such as the elderly and LGBTIQ, are not recorded in the CRM sufficient to enable analysis. It is possible to record date of birth (DOB), gender and sexuality factors in the CRM, but DOB cannot be reported on yet and recording of LGBTIQ status is too low to report on at this stage.

Since October 2018, data fields showing primary spoken language, country of birth and whether an interpreter was required have been reported on; however, virtually all records are blank for these fields. For example, of the 16,714 cases created in the CRM between October and December 2018, 98 per cent of the ‘primary spoken language’ fields, and 99 per cent of the ‘interpreter required’ fields are blank. These statistics, particularly the former, do not correspond to evaluation evidence from fieldwork that indicates The Orange Door is servicing clients from multicultural backgrounds.

The Orange Door has recruited practitioners from diverse backgrounds and language skills, noting that language is only one aspect of inclusion. Inclusive practice is still dependent on the practitioner’s knowledge of the different dynamics of family violence in cultures and practice:

'We have clients from diverse backgrounds who are matched to workers that suit that background where possible. We have Mandarin, Farsi-speaking workers, Aboriginal and Maori workers – that said, the experience of diverse clients depends on the skills of the practitioner.' – Practitioner interview

The Orange Door also regularly uses interpreters to support service delivery for clients who do not speak English. However the use of translators and interpreters can be problematic when translating terms such as ‘family violence’, and in insuring trust in the fidelity of the translation particularly in small communities where confidentiality may also be an important consideration. This is a well known challenge in this space and was reflected by FSV, stakeholders and practitioners during interviews.

'Regularly using a translator, maybe once a month (phone interpreter can be clunky and booking them can be a challenge), in-person ones are often from their [the client’s] community so there may be a conflict. Most clients are quite reluctant because they think it’ll be someone in their community.' – Practitioner interview

Community awareness of the role of The Orange Door

The Orange Door is considered physically visible in terms of street frontage, branding and information materials. However, in terms of the role of The Orange Door, at this early stage of implementation, there was evidence of only emerging awareness of The Orange Door by the broader community.

Information about referrals being made to The Orange Door indicates that there is some awareness of the role of The Orange Door, at least by services who have in the past referred to Child FIRST or family violence services (ie L17s or referrals from professionals). Over time, awareness of the role of The Orange Door would be expected to increase amongst other universal services (eg health professionals, schools).

The role of The Orange Door as an intake and assessment service only is not well understood, even by referring agencies (at this point in time). Practitioners reported referrals being received for a range of issues that were not directly related to child wellbeing or family violence issues. For examples, housing or homelessness problems in the absence of family violence or respite issues for families with a child with disability[7]. It is unclear if this lack of awareness of the role is time sensitive (and only evidenced in this early stage of implementation) or if this lack of clarity over the role will persist well beyond the first year of operations.

In terms of the general community awareness, there is evidence that clients also self-refer for a range of difficulties unrelated to family violence or child wellbeing issues. This would indicate a lack of awareness of the role of The Orange Door. However, it may also indicate a lack of appropriate or accessible social support systems in the area and a desire by people having difficulties to try any service that appears to provide assistance to people. Awareness in the potential client groups for The Orange Door requires validation through direct client feedback before conclusions can be drawn on the awareness of the role of The Orange Door.

There has been some feedback that there is even less awareness of The Orange Door in diverse communities:

'Never heard of an Aboriginal service referring people onto The Orange Door. Aboriginal services aren’t really aware of who we are and what we do.' – Aboriginal practitioner

This would be an area for further validation and exploration in future evaluations. It is reasonable to expect in the early stages of implementation that awareness of the service would not have reached all client sub-groups.

Key finding 27:

The Orange Door areas have been established in locations that are visible and accessible for most clients and in close proximity to community service providers and public transport. There are some challenges for some clients to access some of the physical locations (e.g. not on major transport route for some locations in the catchment)

Key finding 28:

The maturity of The Orange Door has meant that some access options, in particular outreach and publicly identifiable access points, have not been operationalised at the time of this evaluation and will be established in line with the planned roll out

Key finding 29:

The physical space at The Orange Door appears to be broadly functional and fit for purpose and FSV has undertaken significant work on the facilities. However, some safety concerns remain and have been identified to FSV during the course of this evaluation

Key finding 30:

The role of the Service System Navigator has not been sufficiently defined and understood by The Orange Door workforce to enable development of service system pathways and connections at the early stage of implementation. The focus of the role will shift from operations to the service system once operational issues are resolved

Key finding 31:

Access via phone could be improved during business hours with the implementation of a message bank that would operate when all lines are busy or the call is not answered. It is understood that this is part of a broader program of IT and infrastructure upgrades planned by FSV

Key finding 32:

Data limitations prevent us making evaluative judgements on whether The Orange Door is accessible and responsive to the needs of clients with diverse needs. Further investigation is required

Key finding 33:

The physical locations and branding of The Orange Door support community awareness of the role of The Orange Door. There is evidence of emerging understanding of the role as an intake and assessment service (notably by those organisations who had previously referred to family violence or Child First services).

To what extent has The Orange Door ensured clients receive information and options that respond to their needs and enable informed choices?

As discussed in Section 4.3.2, The Orange Door is designed to provide clients with information to enable client agency and choice. This information is intended to help people to understand their risk, prioritise needs and understand options for support.[8]

Evidence from fieldwork indicates that practitioners are not aware of guidance about how much information should be shared with clients in which circumstances. At this early stage, this has caused them to err on the side of providing less information rather than more information. The extent of provision of information to clients throughout their journey through The Orange Door is outlined in Table 5 below.

Table 5: Information provided to clients

Further analysis on client choices and information sharing can be found in Sections 4.3.2, 7.1 and 7.2. Additional data on the experiences and views of clients will be required to make further conclusions.

Key finding 34:

While sharing of information about clients and service within The Orange Door appears to be occurring, there appears to be limited information is shared with clients of The Orange Door at this point in time. However further data is needed to validate the frequency, amount and type of information shared, and if clients perceive that information provided meets their needs

How is The Orange Door working to keep perpetrators in view to keep women and children safe?

Information sharing informs risk assessments and risk planning

The Orange Door collects and analyses information about clients from a range of sources. These include: clients themselves; police; CIP; professionals and other referrers; home agency databases; and CRM. This information is used by practitioners to assess the risk of family violence, assess child safety, wellbeing and development concerns and to keep perpetrators in view. This not only involves collecting information from external sources, but also multidisciplinary collaboration to provide rapid responses to perpetrators’ use of violence.[11]

A key component of keeping perpetrators in view is through access to critical information which allows workers to assess the risk posed by a perpetrator of family violence. A number of reforms help facilitate this, and help to enable an integrated approach, including:

• The Family Violence Information Sharing Scheme (the scheme) was created under Part 5A of the Family Violence Protection Act 2008. The scheme enables information to be shared between selected government agencies and funded organisations (including The Orange Door), prescribed as ‘information sharing entities’ or ‘risk assessment entities’ in order to assess or manage family violence risk. Consent does not need to be given by perpetrators for their information to be shared by information sharing entities, or consent of alleged perpetrators if shared by risk assessment entities.[12]
• Under Part 5B of the Family Violence Protection Act 2008, workers in The Orange Door are able to collect, use and disclose confidential information with other workers where relevant to delivering The Orange Door services, “in a way that gives precedence to safety and wellbeing over privacy.”[13]

The evaluation found a lack of understanding about what ‘keeping perpetrators in view’ actually means in the context of The Orange Door service environment. This is likely to be a point in time issue reflecting the emerging perpetrator service practice, variation in practice across The Orange Door areas at this point in time, a lack of a definition of ‘keeping perpetrators in view’ in foundational documents, and the innovation of the integrated model and how perpetrator services could be integrated in a way to keep women, children and others experiencing family violence, safe. As such, we have examined mechanisms that go to the sharing of information about perpetrators behaviour that can be used to assess risk, stay in contact with perpetrators and engage them in interventions to change their behaviour as part of the evaluation.

Information sharing informs risk assessments and risk planning

The three key ways that information about perpetrators is shared in practice are the use of CIP, history checks on CRM and employing agencies’ databases and multidisciplinary practice.

Central Information Point

The development of the CIP is in response to the recommendations of the RCFV. CIP was established to provide frontline workers with timely access to critical information for the assessment and management of family violence. The CIP allows representatives from Court Services Victoria, Victoria Police, Corrections and DHHS to consolidate critical information about perpetrators of family violence into a single report within hours[14] to enable assessment on the risk posed by a perpetrator of family violence.

To request a CIP, practitioners in The Orange Door typically need to get approval from a team leader or practice lead. There appears to be some inconsistencies in the extent to which CIP requests need to be justified, with some leaders “waving them through” while others “will grill [practitioners]”. Evidence from evaluation fieldwork indicates that some practitioners believe that there is a cap of five CIP requests per day for each area, however, the CIP team is understood to be prepared for a greater number of requests. Practitioners reported a lack of clarity on when and how a decision should/could be made to initiate a CIP request. CIP reports are generally made in the assessment and planning stage, but also in some cases during screening, identification and triage.

'When you have that history [from CIP] it changes your perspective and you can safety plan more vigilantly.' – Practitioner interview

There has been strong evidence from practitioners at each The Orange Door during the evaluation that the CIP has been effective for improving risk assessment and outcomes for clients, particularly those who were previously unknown to The Orange Door. Practitioners reported that having access to information from across the service system has facilitated more accurate assessment and planning for clients, based on having a more fulsome picture of the current situation and history.

Box 5: Use of CIP in The Orange Door

In the first half of 2018-19, the four areas of The Orange Door areas together recorded an average of just under 10 CIP requests per business day.[15] Figure 11 shows the CIP requests created per month for each area in that period.

Figure 11: Number of CIP reports created (as recorded on the CRM) by The Orange Door area, July – December 2018

Source: PwC analysis of CRM data provided by FSV. Note that The Orange Door commenced in NEMA in July 2018.

There appear to be some inconsistencies in how CIP is used in each area. Practitioners acknowledged that the CIP is specifically for family violence cases, but cited child wellbeing cases where it would have been useful to inform risk and needs assessments. Some practitioners stated that they had used CIP when the case only peripherally related to family violence (which is consistent and allowed within the legislation). One other inconsistency that practitioners referred to was whether information gained from a CIP request could be shared with affected parties, eg to inform safety planning. The legislation does allow for sharing of perpetrator information with the victim survivor to assist in managing their safety, however practitioners reported variation in their understanding of if this was allowed, and if and when they shared information. The evaluation found that while CIP is considered a positive tool for The Orange Door and there is FSV guidance material regarding the use of CIP, there is a need for greater operational clarity and consistency over when, how and how often CIP requests could be made to assist practitioners at the operational level.

Case study of CIP use

A CIP request was made when a practitioner was concerned about the risk a perpetrator posed to a client. The client has disclosed that there was a history of family violence for her and also a previous partner of the perpetrator. The CIP report indicated an extensive history of family violence perpetrated by the man with numerous women dating back a decade. Importantly, this information indicated a number of instances when the perpetrator offended at the location a woman had parked her car while away from home. This information immediately changed the safety planning the practitioner did with The Orange Door client with a much stronger focus on safety while travelling to and from work and other known locations (shops, child care, etc.). –real example reported by practitioners

CRM

CRM is expected to be used by practitioners to capture, store and record all case information. It also integrates with a number of systems, including the L17 portal, CIP and the TRAM, and in future, others will be built in.[16]

Over time, CRM will build up a considerable amount of client and case data that can be used to inform risk assessments. While CRM’s recent introduction means that historical information currently held is limited, practitioners provided some examples where searches for clients or related family members across CRM had yielded additional information that would have been otherwise unknown. As it is rolled out across all DHHS areas and through progressive upgrades delivered and planned by FSV, CRM will enable quicker and more automated information sharing. This should speed up searches of perpetrators’ historical cases, meaning that CIP can be deployed only on more critical cases.

Multidisciplinary practice

Multidisciplinary practice is a fundamental aspect of The Orange Door. There is evidence that The Orange Door workforce is collaborating on assessment and planning supported by physical colocation and personal working relationships. The collaboration is inconsistent and not yet embedded within practice, especially when working with perpetrators. In particular:

• For some family violence-focused practitioners (who sometimes call themselves ‘women’s workers’), there is an ideological barrier to sharing information with anyone who is perceived to be ‘helping the male perpetrator’. This appeared to stem from their perceived priority to help the victim survivor above anything else, and viewpoint that providing information to a ‘men’s worker’ may increase risk to their client.[17] Some practitioners working with perpetrators reported that they would prefer not to know about the victim survivor’s circumstances in case they inadvertently disclosed details to the perpetrator when speaking to them.
• Some practitioners do not yet clearly understand or are aware of the FSV Support and Safety Hubs: Perpetrator practice guidance (April, 2018) on when and how information should be shared between victim survivor and perpetrator focused practitioners[18]. This means that practitioners are generally taking a risk-averse approach to sharing information at this point in time.[19]
• Multidisciplinary practice was most evidently displayed by the Aboriginal practitioners at The Orange Door, who provided support to both the victim survivor and perpetrator in most cases.[20] According to Aboriginal practice leads and specialist family violence practice leads, this is because the tradition of providing support to Aboriginal families tends to be community-led, involving those outside the family to assist the victim survivors and perpetrators. As such, the integration of perpetrator services with victim survivor services is aligned with existing practice.
• Perpetrator interventions beyond Men’s Behaviour Change (MBC) programs are not yet matured. During fieldwork, MBC programs were the only intervention allocated to non-Aboriginal clients that practitioners reported. The wrap-around nature of Aboriginal services means that perpetrators are provided support from the same service that are supporting victim-survivors, and that support involves cultural connection and individual counselling rather than just behaviour change groups.

Insights gained from client interviews for this evaluation regarding the role of The Orange Door in keeping perpetrators in view were limited. One participant expressed gratitude that The Orange Door had reached out to their partner as they were keen for their partner to receive support and assistance. This was viewed by the client as a sign that The Orange Door was non-discriminatory.

As engagement with perpetrators and/or data that demonstrates the numbers of perpetrators engaged through The Orange Door is currently limited (also see Sections 4.1.1), further investigation into how information is shared to keep perpetrators in view is required before making firm evaluation judgements on whether The Orange Door is contributing to keeping perpetrators in view.

Use of information to inform service responses for perpetrators in The Orange Door

Keeping perpetrators in view also means providing them with opportunities to change their behaviour by connecting with services that assist in holding them accountable for their actions. This involves assessing perpetrators’ preparedness for change and connecting them to services to stop or reduce their use of violence and improve their parenting.[21]

Service responses for perpetrators is determined by the information collected as part of the assessment and planning stage function of The Orange Door. Thus, the success and limitations of that stage assist and constrain information sharing in the later ‘connection to services’ stage of The Orange Door service delivery. CRM data on the connection to services function is currently limited and appear to be inconsistently recorded. Despite the data limitations, qualitative data collected as part of the evaluation indicate that there are concerns about information sharing specific to that function:

Contact with men is limited. Practitioners working with perpetrators stated that they tried to call three times over three days and if they were not successful in getting in touch, they closed the case. In general, practitioners did not attempt to use contact methods other than phone. If they could not contact the perpetrator, assessing their preparedness for change is difficult, and connecting them with a service mostly impossible.[22]

[We] don’t actually get on to many clients; there’s only a few [over their time at The Orange Door] that I’ve actually had real conversations with” – Practitioner interview

• One perpetrator focused practitioner often collaborates closely with a family violence-focused practitioner and reported that the information collected through CIP and other sources is useful to inform safety planning even if they are unable to get in contact with the perpetrator.[23]
• Some practitioners reported their concerns that there is no senior practitioner with whom to consult specific to their practice of working with perpetrators of family violence. Ironically, this has the potential to improve information sharing through multidisciplinary practice if these practitioners consult with the advanced family violence practice lead, however, this does not always occur as yet. While this is not the intent of the role, there is a perception in the workforce that the advanced family violence practice lead specialises in victim survivor practice.
• CRM does not currently record the outcome or output of the service response beyond The Orange Door. Moreover, data recording of allocations of core services to clients and referrals to non-core services on the CRM is currently poor. This means that it is difficult for practitioners to track the use of service responses by perpetrators and the success of these interventions.[24]

Further research on service responses, involving higher quality CRM data, will be required to provide insights into whether service responses for perpetrators are tailored based on shared information.

Key finding 35:

There is an evolving understanding about what it means at an operational level for The Orange Door to ‘keep perpetrators in view

Key finding 36:

CIP has been effective for improving risk assessment and outcomes for clients, particularly those who were previously unknown to The Orange Door. Operational clarity is needed for the workforce as to when, how and how often CIP request can be made

Key finding 37:

The CRM has the potential to be a source of client and case data that can inform risk assessment, however data recording is not yet adequate to effectively support this function

Key finding 38:

The Orange Door workforce is collaborating on assessment and planning supported by physical colocation and personal working relationships. The collaboration is inconsistent and not yet embedded within practice, especially when working with perpetrators. Collaboration on assessment and planning is most evidently displayed by Aboriginal practitioners and aligns with their community-led approach to service provision

Key finding 38:

CRM data on connection to services are limited. As such, the evaluation was unable to determine if service responses for perpetrators are tailored based on shared information

Key finding 39:

Support to perpetrators by The Orange Door is limited as it is dependent on their voluntary engagement with the service. Many do not respond to attempts by The Orange Door to contact them. This was anticipated and is reflected in the guidance documents for The Orange Door

[1] FSV, ‘Support and Safety Hubs: Statewide Concept’, July 2017.

[2] FSV, ‘Support and Safety Hubs: Frequently Asked Questions – Public Version’, May 2018.

[3] It is possible that this increased transparency between The Orange Door and the client to ensure accuracy and comprehensiveness of information. However, during this evaluation, this was considered by practitioners not to be appropriate.

[4] FSV, Support and Safety Hub Statewide Concept’, July 2017.

[5] Qualitative evaluation data from fieldwork

[6]

[7] Please note, it is recognized that respite for children with disability may also be a child wellbeing concern and as such, fall within the scope of The Orange Door services.

[8] FSV, ‘Support and Safety Hubs: Concept paper’, July 2017.

[9] There are however a range of written information pamphlets etc. that have been prepared, tailored to client groups and available at The Orange Door.

[10] Qualitative evaluation evidence from fieldwork

[11] FSV, ‘Support and Safety Hubs: Service Specifications’, April 2018.

[12] FSV, ‘Support and Safety Hubs: Frequently Asked Questions’, 30 May 2018.

[13] Family Violence Protection Act 2008: Part 5B – Information sharing relating to Support and Safety Hubs: 184

[14] State Government of Victoria, ‘The Central Information Point’, 2018.

[15] PwC analysis using CRM data provided by FSV.

[16] FSV, ‘Support and Safety Hubs: Service Model’, April 2018.

[17] Qualitative evaluation evidence from fieldwork

[18] FSV Support and Safety Hubs: Perpetrator practice guidance, April 2018

[19] Qualitative evaluation evidence from fieldwork

[20] Qualitative evaluation evidence from fieldwork

[21] FSV, Support and Safety Hubs: Service Specifications’, April 2018.

[22] Qualitative evaluation evidence from fieldwork

[23] Qualitative evaluation evidence from fieldwork

[24] Qualitative evaluation evidence from fieldwork

To what extent is the integrated and multi-disciplinary approach to service provision at each The Orange Door site working to support better outcomes?

FSV developed The Support and Safety Hubs: Interim Integrated Practice Framework (‘the Interim Integrated Practice Framework’) to guide consistent practice in the first areas of The Orange Door. The framework is intended to support practitioners to deliver integrated support for clients through providing clear guidance about the requirements of their role as a practitioner in The Orange Door. Within the Interim Integrated Practice Framework, integrated practice is referred to as ‘the integration of theoretical frameworks, philosophical underpinnings and perspectives, and practice skills and approaches that guide a practitioner’s work’. Additionally, the guide notes that practitioners will be:

• required to maintain their specialisation and adopt a multi-agency approach to service delivery
• supported to develop additional skills and capability to address the safety and needs of women, children, families and men

While integrated practice underpins all work of The Orange Door, the Interim Integrated Practice Framework makes it clear that the vision for The Orange Door service delivery is that 'practitioners will be required to maintain their specialisation and to adopt a multidisciplinary approach to service delivery.'

The Interim Integrated Practice Framework refers to integrated service as 'a number of services working together, collaborating and coordinating their support, services and interventions to people seeking, or referred for, help.'[1] However, there is no sector-wide agreed understanding of what ‘integrated service delivery’ looks like at an operational level. As indicted previously, there is a lack of understanding by the workforce in how to operationalise the model in the early stages of The Orange Door.

The Interim Integrated Practice Framework has been used as the benchmark for integration, recognising that the evaluation could not assess the level of integration but could gauge the ‘direction of travel’. The Interim Integrated Practice Framework describes:

• the expected capability of the workforce at a high level – 'practitioners will be required to maintain their specialisation and adopt a multi-agency approach to service delivery'
• the expectation of practice to develop capability – 'practitioners will need to draw on the strengths and specialisation of colleagues in the Hub to gain confidence and capability to work across the range of issues experienced by people accessing the Hub'
• the expected collaboration between practitioners – 'Hub practitioners will be able to: apply their specialist knowledge and expertise into their practice with all people who access the Hub… collaborate with and draw on the strengths of their specialist colleagues as part of a multi-agency approach to practice … collaborate with their colleagues in developing integrated risk and needs assessments and planning as part of a multi-agency approach, ensuring all individual risks and needs have been analysed and considered in decision making'
• the expected integration of practice across all functions of The Orange Door
• the expected coordination of responses to 'deliver coordinated responses with their colleagues that promote meaningful engagement with services that result in sustainable outcomes.'[2]

During evaluation field work, many practitioners admitted that they had not read the guidance materials including the Interim Integrated Practice Framework. While reading core practice guidance could reasonably be expected of staff given it was identified in the orientation training, when investigated, The Orange Door staff indicated it was because of: the amount of documentation that had been produced; the length of the content;[3] and time pressures:

'No one has had the time to read [the policy documents] and embed them. So then you revert to what you know.'[4] – Practitioner interview

Those that had read the documentation did not find the content useful for preparing them to work at The Orange Door, as they expected more definitive practice based guidance including detailed examples as to how the integrated model would or could work in practice.

Generally, across all areas, there is evidence of the following aspects outlined in the Interim Integrated Practice Framework in Table 6 below.

Table 6: Interim Integrated practice framework elements evident at The Orange Door

The Interim Integrated Practice Framework has been used as a theoretical benchmark, with key elements of integrated practice in that document used to inform initial evaluative judgements on whether aspects of integrated practice (including the coordination of support) are occurring in different service delivery points within The Orange Door.

Figure 12 provides a visual overview of the point in time evidence collected for the evaluation relating to integrated practice across the functions in each Orange Door area.

Download Figure 12: Overview of integration at each function of The Orange Door (as of September 2018):

Figure 12: Overview of integration at each function of The Orange Door - as of September 2018

Word 44.23 KB

(opens in a new window)

As The Orange Door is in varied stages of implementing integrated practice and because we have very limited data collected directly from clients in this evaluation, the evaluation was unable to determine if multidisciplinary, collaborative or integrated practice was itself contributing to better outcomes for clients (nor does this early evaluation consider client outcomes).

A key finding of this evaluation however, is that there is a genuine commitment and enthusiasm by practitioners of The Orange Door to work in an integrated manner that builds their capabilities and confidence in working in integrated and/or multidisciplinary teams to improve outcomes for clients. This commitment is a strong foundation upon which the evolution of practice towards a more integrated vision will be achieved.

It is worth noting that the evaluation team conducted site visits and collected data from October – December 2018 and the subsequent pace of the reforms and implementation means that some of the issues raised regarding integrated practice during the evaluation have already been addressed. For example, we understand that there have been steps to introduce reflective practice across disciplines in one of The Orange Door areas since our evaluation fieldwork in order to provide formal and structural supports to encourage integrated practice. This was an opportunity identified to FSV early in the evaluation which has been subsequently actioned.

Key roles in The Orange Door that can assist in promoting integrated practice include the Advanced Family Violence Practice Lead and the Integrated Practice Lead. The formalisation of these roles is a positive foundation upon which to base evolving integrated practice in The Orange Door. Some further refinement is needed for these roles to sharpen their focus to promote integrated practice as The Orange Door model evolves (see Section 6.2).

Key finding 40:

There is a commitment from The Orange Door workforce to develop capability in other specialisations and work in an integrated manner in order to improves outcomes for clients

Key finding 41:

The length of written guidance and documents produced about The Orange Door – concepts, framework, operational guidance, etc – deters practitioners from engaging with and understanding key information to inform how they integrate services

Key finding 42:

A culture of collaboration is evolving at The Orange Door in line with the vision described in the Interim Integrated Practice Framework and tends to happen organically between individuals, and between certain disciplines

Key finding 43:

The ability of the current workforce at The Orange Door to work in an integrated manner is currently impacted by staff shortages and high demand

Key finding 44:

There are limited structural and formal opportunities for practitioners to learn about other disciplines and practice approaches. Time for reflective practice for teams would support integrated practice.

To what extent does The Orange Door workforce have the resources, capacity and specialist expertise to undertake the full range of functions articulated in the foundational offer?

The Orange Door Service Model describes a staffing model where different workforces and practices will combine to create a team and consolidated intake point for each area. This is to be achieved through a partnership of community service organisations who each contribute workers employed by their organisations to work as part of The Orange Door team. Alongside Client Support Officers (CSOs) employed staff are staff employed by FSV and DHHS (Child Protection). Each worker continues to be employed and managed by their employing ‘home’ organisation with their terms and conditions of employment being protected.

Key roles are identified in the Service Model and outlined in Table 7 which also describes evaluation findings related to each role.

Table 7: Roles within The Orange Door, allocation of roles and evaluation findings about roles

The evaluation found that the staffing for The Orange Door has been challenging in the initial set up and implementation phase for all areas including:

• Challenges in recruiting appropriately qualified and experienced staff due to a scarcity of experienced workers more broadly.
• A lack of alignment between workers expectations of their roles (and position descriptions) and the reality of their roles within the service model.
• Workload for practitioners from the combination of under-staffing, set up demands and almost immediate backlog in cases (in some cases due in part to partner organisations bringing across existing backlogs).
• A lack of ‘lead in’ time for practitioners to acclimatise to the environment, service delivery context and new systems prior to service commencement.
• Insufficient work spaces to accommodate the workforce for some areas (BPA and Barwon) during the early months of service commencement.

Despite the challenges, the evaluation found strong evidence that the workforce are committed to the concept of The Orange Door and were persisting with finding solutions to operational challenges to try to ensure services were delivered to clients.

Key finding 45:

There is a lack of operational clarity and understanding for The Orange Door workforce about the role of the Integrated Practice Lead and the Service System Navigator during the early roll out of The Orange Door

Key finding 46:

There is a perception that the Advanced Family Violence Practice Lead role pertains only to practice with victim survivors. Although the Advanced Family Violence Practitioner role was established to provide guidance to both victim survivor and perpetrator workers, workers perceive that there is a gap in leadership for perpetrator services

Key finding 47:

Additional and/or different resourcing is needed to account for establishment demands, peak demand times of the year and to accommodate staff leave arrangements during the Christmas and the New Year period

To what extent does The Orange Door workforce have the skills to recognise and manage signs of family violence, including perpetration, and child vulnerability

Capacity for recognising family violence and perpetration

The evaluation found that many practitioners entered The Orange Door with skills in recognising and responding to family violence. These skills and knowledge were reflected most strongly in practitioners who had previously worked with victim survivors and who were experienced in using risk assessment frameworks similar to those now used in The Orange Door. Similarly, those who had worked with perpetrators entered The Orange Door with a suite of practice skills and interventions aimed at recognising escalating risk with men, and the use of interventions and other resources to try to mitigate risk. For the workforce who had experience in child and family services, many had exposure to identifying and managing risk where there were concerns about family violence as a factor in child wellbeing cases (noting not all child wellbeing cases involve family violence). The inclusion of practitioners with skills in recognising and responding to family violence perpetration at the outset of The Orange Door has meant that many are able to focus on learning new systems and process of The Orange Door, rather than attempting to acquire these practice skills.

The evaluation found that working with perpetrators to reduce risk is an emerging area of practice. Practitioners who work within this speciality acknowledged to evaluators that their area of practice is maturing and lacks the level of evidence to inform practice to the same degree as practice areas of child wellbeing and practice with victim survivors.

The evaluation found mixed evidence of the workforce improving their capacity to work with perpetrators of family violence to reduce risk. There is some evidence of skill and knowledge sharing across teams, learning from practitioners who work in this space, and early induction training provided some overview of this practice area to all staff. However, the evaluation also found that some of the workforce (commonly those who work within a victim-survivor speciality) found the concept of working with perpetrators challenging to reconcile with their own perspectives on risk and response, and expressed to evaluators their apprehension about building their capacity to work with perpetrators (either because they did not currently have the skills to do this or because they found it difficult to understand how they could reconcile this with their practice ideology).

In terms of structural and system supports, the capacity for The Orange Door workforce to recognise and respond to family violence is supported by systems and processes of risk assessment. The Advanced Family Violence Practitioner role is a key support for staff to recognise and manage family violence however the workforce perceives this role as related to victim survivor practice rather than perpetrator practice. In the context of an emerging field of practice of working with perpetrators, and the significant innovation of including perpetrator services within The Orange Door, the absence of clear practice leadership for perpetrator services was noted. There is some evidence that in some areas, the most experienced staff in the specialty of perpetrator services are, in the absence of a formal role, assuming responsibility for practice leadership in perpetrator services for The Orange Door.

Capacity for recognising child vulnerability

At the time of field work, The Orange Door staff working with child wellbeing cases did not yet have a standardised risk assessment tool and as such, had developed localised ‘Best Interests Assessment’ (BIA) tools (based on the Best Interests Case Practice Model utilised by Child FIRST Alliances). It is important to acknowledge that a standardised tool did not exist prior to the establishment of The Orange Door. Practitioners indicated that they perceived the focus of implementation efforts of The Orange Door had been on the family violence processes and procedures, and that a focus on child and wellbeing processes and procedures would occur later.

The evaluation found that most practitioners who had previously worked within Child FIRST had skills, knowledge and experience in recognising and responding to child vulnerability concerns. However, many practitioners who had a specialty in family violence victim-survivor or perpetrator services expressed concerns regarding their ability at that point in time to adequately identify risk and respond to child wellbeing concerns – particularly in the absence of family violence concerns.

In terms of structural and systems supporting capacity for identifying child wellbeing concerns, the Child Protection Team Leader and senior Child Protection practitioners provide a key support for The Orange Door workforce in supporting the identification and response to child wellbeing concerns. However, the absence of a standardised risk identification tool for child wellbeing cases across The Orange Door areas was reported. For these staff, working with an assessment tool for child wellbeing was a new experience and most expressed their view that induction and training provided by The Orange Door at that stage was not sufficient to allow them to feel confident in risk identification for child wellbeing cases (in the absence of family violence).

It is important to note, the evaluation also found early signs of collaboration and multi-disciplinary working that was informally building the skills of the workforce in identifying and responding to child vulnerability concerns. Colocation and ability to partner for home visits for child wellbeing cases with the community based Child Protection practitioner or other staff with expertise in child wellbeing was contributing to upskilling staff who had not previously worked with children. This is a positive finding given the very early stages of The Orange Door establishment.

Training and support

Workers enter The Orange Door with pre-existing skills, knowledge and training. In this context, it is to be expected that the workforce will have variable skills. The focus of the evaluation was on training and support provided to the workforce by FSV or agencies once they had entered The Orange Door (as opposed to profiling all training previously undertaken by individuals).

There was an expectation expressed by FSV staff and practitioners that professional development specific to practice would be provided by the employing home agency. However, the evaluation found that team based reflective practice opportunities (in comparison to professional development) as an Orange Door team were not available in the early months of operations. There was an expectation from practitioners that this would be provided at The Orange Door rather than through employing agencies. It is our understanding that opportunities for reflective practice have now commenced.

Evidence from fieldwork indicated the following:

• The workforce reported that orientation and induction training provided did not adequately meet their needs. While many practitioners indicated that the induction training was a starting point and acknowledged the importance of understanding the role of perpetrator services in The Orange Door (in particular), given this emerging field of practice and the innovation of bringing this service together with victim survivor services, more detailed training was needed. At the point in time, the evaluation found that the initial induction training had not clarified understanding of how integrated practice was intended to be operationalised at The Orange Door.
• Training was pitched at a low level for their own specialty and individual skill level and did not provide the workers with sufficient understanding of disciplines outside of their area. For example, many practitioners commented that the two hour training on perpetrator practice was inadequate to assist them to understand the practice framework and ideology of these services. This was considered particularly important when the practice is fundamentally different to their own. We understand that FSV is refining the Induction Training program based on findings from their evaluation, including more comprehensive training on each of the discipline areas, [5] and undertaking a participant profile ahead of training in order to understand the level of experience in the room. Consideration will need to be given to the challenges in releasing staff to undertake large amounts of training whilst still delivering services. The workforce sought more in-depth and tailored induction and training to meet their initial orientation needs when entering The Orange Door.
• A one-size-fits-all approach to training will not work. Client Support Officers did not think that induction training was appropriate for the work they do, and was instead tailored for practitioners. They felt there was an absence of content that was directly relevant to their role and could equip them with the necessary skills for working at The Orange Door. For example, there was no training on dealing with agitated clients or engagement techniques, which they considered critical for their specific responsibilities.[6] These topic areas were expected by workers to be covered (either in induction or in other FSV initiated training) as they related directly to their roles at The Orange Door.

At the point of fieldwork, there was some evidence that some areas were in the early stages of planning a schedule of training and professional development for the workforce at The Orange Door (to be delivered through a combination of FSV and partner agencies. This will be supported by current work by FSV on a workforce strategy. As such, findings relating to training and induction need to be understood as being part of the very early stages of implementation of The Orange Door. There is a need for service to be embedded further prior to understanding in detail the specific needs for structured training and professional development.

Key finding 48:

As perpetrator services is an emerging field of practice, increased leadership in this practice may be required in The Orange Door to build the capacity of the family violence and child wellbeing workers to identify and respond to risk of violence being perpetrated

Key finding 49:

The TRAM provides a standardised tool to support the workforce to identify and respond to risk of family violence. The Orange Door does not yet have a standardised tool to support the workforce to identify and respond to child wellbeing risks (nor was one in existence prior to the establishment of The Orange Door)

Key finding 50:

The workforce seeks: more detailed and tailored induction tailored to their speciality backgrounds; more knowledge and understanding of the integrated service model; and more knowledge and understanding of how to work with perpetrators from perpetrator services.

To what extent does The Orange Door workforce have the skills to meet the needs of diverse client groups (eg Aboriginal, CALD and LGBTI clients)?[7]

The Orange Door Service Model identifies the need for a system to be able to respond effectively and safely for all Victorians, inclusive of their diverse characteristics and intersectional considerations.

In terms of the workforce’s capacity to meet the diverse needs of Aboriginal and Torres Strait Islander clients, the role of the Aboriginal Practice Lead provides a key resource to build the skills of The Orange Door workforce through collaborative practice. The evaluation does not have evidence to determine if the FTE resourcing for these leadership roles is sufficient to meet the current demand of cases and the need to build cultural safety practice across The Orange Door workforce.

The evaluation found varied skills and abilities in the workforce’s capacity to meet the needs of clients from multicultural backgrounds. Some staff brought a background and previous training on working with clients from multicultural backgrounds to The Orange Door, and there was some evidence of working with partner agencies and other localised multicultural specific services to assist tailoring of service to multicultural clients (for example, referring to a service where there is a staff member with the same language as the client). Training for working with multicultural clients and training on intersectionality had not been provided to The Orange Door workforce at that point in time either by FSV or systematically to the workforce by any partner agency.

The evaluation was unable to determine the workforce’s capacity to meet the diverse needs of clients with a disability, people from LGBTIQ communities and older people in this early stage of The Orange Door operations. Practitioners were unable to identify clients with these diversity characteristics during interview and data reporting is not possible via the CRM at this point in time. It is our understanding that FSV is currently progressing work to support workforce capability in intersectional approaches and this will include for the workforce of The Orange Door.

In terms of the workforce’s capacity to meet the diverse needs of children and young people, including young people who themselves are perpetrators of family violence, there was evidence that those practitioners with a speciality in child wellbeing brought to The Orange Door considerable skills and expertise to work with these clients. An overview of working with children and young people had been provided to The Orange Door workforce through induction training.

Case example from clients interviewed for this evaluation

Feedback from the six client interviews did not provide significant detail on the workforce’s ability to meet the needs of diverse clients. One client interviewed stated that they were satisfied with the cultural programs that The Orange Door had referred them to. In a discussion of cultural programs, the client identified the need for The Orange Door to have in place strategies to protect client confidentiality. In this case some of the people that the family interacted with at the cultural programs were connected to (either through the kinship system or their community of residence) the violent ex-partner/father of the participant and their children. The client stated that they were satisfied with how The Orange Door and other workers sought to protect the family’s privacy.

In looking forward towards increasing the skills of the workforce to meet the needs of diverse client groups, the evaluation found that clear expectations need to be conveyed to the workforce about assigning responsibility to provide training and professional development. Whilst responding to our diverse community should be any employing organisation’s responsibility, and FSV staff indicated a belief that this was indeed the responsibility of employing agencies, many practitioners expressed a view that FSV needs to provide training to all Orange Door workers to ensure a consistent and baseline level of skills across the workforce.

Key finding 51:

The evaluation was unable to determine with certainty if The Orange Door is meeting the needs of diverse client groups at that point in time. Further investigation and validation is required through client feedback

Key finding 52:

Clarity is needed for the workforce on who should provide training and professional development to meet the diverse needs of The Orange Door clients

[1] FSV, ‘Support and Safety Hubs: Interim Integrated Practice Framework’, April 2018, p36.

[2] FSV, ‘Support and Safety Hubs: Interim Integrated Practice Framework’, April 2018, p45.

[3] Qualitative evaluation data from fieldwork

[4] Qualitative evaluation data from fieldwork

[5] The Orange Door Induction Training program evaluation recommended that the family violence, victim survivor and perpetrator training, and working with vulnerable children and families training be extended to one day each

[6] For example, critical incidence training

[7] We have retained the language agreed to in the Evaluation Framework for this evaluation for the evaluation questions – however noting that the FSV nomenclature has now been confirmed to refer to ‘multicultural’ and ‘LGBTIQ’ which is now referred to in the discussion.

To what extent are mechanisms in place to ensure information is shared and used?

PwC interpretation of the Statewide Concept indicates that information sharing procedures between The Orange Door and referring bodies are intended to address the following issues:

1. Clients and their needs aren’t missed.
2. Duplication of effort is minimised.
3. Comprehensive risk assessment and management is based on the best information available across the system.
4. Information is shared, used for service delivery and for referral to keep perpetrators in view. (Section 5.3 includes further detail on how The Orange Door is working to keep perpetrators in view).
5. The Orange Door workers, professionals and community organisations can easily access information about where to connect clients to.
6. Sensitive information that practitioners have access to is managed carefully.
7. A feedback loop about people’s engagement with services in and beyond The Orange Door can be established.

Some information sharing procedures between The Orange Door and referring bodies have been established and these appear to be driven by FSV and the partnerships in each area. Based on evidence from fieldwork, there remains significant work to realise the intended extent of information sharing between The Orange Door and referring bodies. This is to be expected so early in implementation. This evaluation focused primarily on information sharing within The Orange Door at this early stage of implementation.

Table 8: PwC analysis of the extent of information sharing and information management within The Orange Door

The evaluation examined information sharing within The Orange Door, and identified enablers of information sharing and use within The Orange Door including:

• Commitment by FSV and The Orange Door staff to sharing information. Evidence from fieldwork indicates that all practitioners believe information should be shared where practicable and safe and where allowed under the information sharing legislation. Furthermore, information sharing is a key concept in the service model and specifications developed by FSV, highlighting its importance to the reforms.
• Recently enacted legislation enables greater information sharing (see Section 5.3).
• The introduction of the CIP, which not only provides far greater information than previously (through a consolidated report of key public sector agencies information), but also has the potential to make it easier to obtain information and improve information sharing (see Section 5.3).
• The varied backgrounds of The Orange Door practitioners. The Orange Door brings together decades of experience across the family violence, perpetrator and child wellbeing sectors into one place. Colocation and multidisciplinary practice have enabled The Orange Door practitioners to learn from their colleagues and share information about services in the sector.
• The partnership model. The Orange Door partnerships have the potential to provide the foundations for the structure of information sharing within the partnership, which can be leveraged to provide a model for the broader sector. Currently, the way information is shared within the partnerships is through their practitioners collocated in The Orange Door and their connection with their employing agencies, and through HLG and OLG meetings.

The barriers to information sharing and use include:

• Accountability for enabling information sharing. At this early stage in the establishment of The Orange Door, at an operational level, it is not clear which role or group is accountable for developing the protocols and systems required to effectively share and use information within The Orange Door. All practitioners should be undertaking information sharing and using policies and procedures to facilitate this. Organisations also need to be providing leadership, and in time, this ultimately needs to be maintained as a shared responsibility. Although information sharing requires extensive collaboration across organisations, one individual or group ultimately requires the authority to progress this work, make difficult decisions and report on developments during this early stage of implementation of The Orange Door.
• Siloed information collection databases. One of the primary rationales for CIP’s existence is that information about people is held on individual services’ databases. The CRM is a further addition to the number of databases recording data about people, but has replaced the Integrated Reports and Information System (IRIS), HDC, SHIP and other case management databases in The Orange Door areas. Information will not be effectively shared until individual databases are able to ‘speak to each other’. The siloed nature of the databases mean that practitioners and case managers often need to duplicate recording work undertaken by other agencies on their own database.
• The model is not designed as a closed loop system. The Orange Door is intended to primarily undertake an ‘intake and assessment’ function, with the primary goal being to connect clients with services as quickly and effectively as possible, rather than guiding clients through every stage of support. As discussed in Section 4.1.1, formal feedback mechanisms have not yet been established and it is not clear to The Orange Door practitioners how these mechanisms would operate in practice.

Key finding 53:

Information sharing within The Orange Door would be strengthened and duplication of effort reduced if practitioners were easily able to access information about the range of services that clients can connect with, and a feedback loop (review and monitoring system) with the sector was established.

Key finding 54:

At this early stage of implementation of The Orange Door, at an operational level, the responsibility for operationally supporting information sharing appears to be pooled between the Service System Navigator, the HLG, the Hub Manager and FSV, meaning no one person or group has overall accountability at this point in time. This may be appropriate in the longer term, but in initial implementation stages, it would benefit from one role taking initial leadership.

To what extent is information sharing contributing to effective assessment and coordinated response to family violence risk?

The evaluation was only able to examine this issue from one side (The Orange Door sharing of information, and coordination of responses within The Orange Door) through practitioner interviews, data analysis and peak stakeholder interviews. To fully understand the extent to which information sharing is contributing to assessment and coordinated response, further investigation is needed to understand the experience of the service system with which The Orange Door interacts, shares information and refers. Please note Sections 4.3 and 5.3 discuss similar evaluation findings (with a focus on risk assessment). Key mechanisms for sharing of information into The Orange Door are discussed in Section 5.3 and relate to the CIP, CRM, use of partner agencies’ client databases and multidisciplinary practice.

At this early stage of The Orange Door service delivery, the evaluation is unable to determine with certainty the extent to which information sharing within The Orange Door is contributing to a coordinated response to family violence. To do this, investigation is needed into the impact of information sharing from the non-The Orange Door service system. The evaluation is however able to provide evidence of the ‘direction of travel’ towards this goal[1].

The CRM is a database that practitioners in The Orange Door from multiple employing home agencies are able to use (providing they are practitioners of The Orange Door) and has the potential to be a key source of information sharing. Initial functional problems are being addressed through iterative upgrades to the system, and as data recording accuracy and comprehensiveness is improved, this should provide a key mechanism for information sharing via practitioners in The Orange Door.

Utilising the client data bases of employing home agencies is a key way in which The Orange Door is currently sharing information for the purposes of coordinating responses to family violence. When a referral to The Orange Door is made, practitioners check their home agencies databases to determine if previous services have been provided to the client, or if there is pre-existing information that can be shared. Practitioners indicated that where there was a pre-existing relationship, this assisted in making a timely referral to that agency to reinstate supports or to have the agency share information with The Orange Door to inform the risk and needs assessment.

The evaluation found that CIP is a key mechanism for sharing information, and there is good evidence that CIP reports are used to coordinate responses to family violence. There was evidence from practitioners that CIP reports were used to share information within The Orange Door to improve immediate risk assessment but also as a means to facilitate access to family violence services in a timely manner. For example, there is evidence that the information in the CIP reports is used to justify and expedite access to family violence case management in the context of waiting lists for this service. It is unknown from this evaluation if information from CIP reports is shared with clients, or if (and in what ways) this information is shared with agencies the client is referred to.

Case example

In response to a CIP request, a member of the CIP team called The Orange Door practitioner ahead of the provision of the written CIP report to advise that the CIP process had identified that the perpetrator was to be released from prison within the next 24 hours. This information was not known to the client or The Orange Door. In response, the practitioner contacted the client, and mobilised the partner agency and referring agencies to immediately put in place strategies to help keep the woman safe.[2]

Key finding 55: The information sharing that is occurring within The Orange Door is contributing to coordinating responses to family violence at The Orange Door through key mechanisms including the CRM, use of partner agencies client databases and CIP.

To what extent have The Orange Door systems and processes supported system integration and resulted in better collaboration between services

Facilitation of system integration and service access

The Orange Door is formed between government and service providers who have undertaken a formal Partnership Agreement within each area. Partnership Agreements commit the partner agencies to provide staffing to The Orange Door, as well as establishing a formal mechanism for collective leadership, shared vision, goals and actions, support integration of services, formalise governance mechanisms and create a process for dispute resolution.

In addition to the Partnership Agreements, The Orange Door concept envisaged that The Orange Door – specifically the Hub establishment groups, distinct from the HLGs – develop timely and effective referral pathways and protocols for information sharing, as well as agreements with private providers where necessary. These pathways would connect The Orange Door with statewide and local services including housing, mental health, alcohol and drugs and maternal and child health services. The purpose of these pathways and agreements is to:

• ensure information gathered about a person or family (including information about assessment of risk) can be transferred to a service directly, accurately and in a timely fashion to avoid a person having to ‘re-tell their story’ or practitioners having to complete multiple risk or needs assessments
• confirm that the service is available and suitable before the referral is made, that the referral is recorded and take up is checked following referral
• support the person to undertake any eligibility assessment process required[3]

Based on evidence from fieldwork and client interviews, we found that pathways and agreements are not yet matured enough to facilitate system integration and service access. The service model does pose some challenges to integration:

• Practitioners’ knowledge about services available in the broader sector: During fieldwork, practitioners stated that they mainly relied upon their prior experience to inform services available outside the partnership rather than any consolidated source of information about services.
• The access and referral mechanisms are not designed to deeply explore a client’s needs: Practitioners do not feel clear about how deeply they are able to explore their clients’ circumstances and attempt to address their deeper needs. If they attempt to investigate a client’s concerns around her situation and address them (for example, by referring to counselling), they are concerned that they are beginning to provide case management. If they simply refer the client/s to the family violence or child wellbeing agency in the partnership, they are concerned that they have not adequately addressed their client’s needs. This has important implications for a practitioner’s ability to make comprehensive risk assessment if they are uncertain about how deep they are able to explore needs.
• Isolated data systems for transmitting data: Data recording systems used by The Orange Door and partnership agencies do not allow information gathered about a person or family to be transferred to a service securely. Some allocations or referrals are made using a phone call, email, hard copy, or in person. None of these methods provide the depth of information to the case manager so the extent that they can pick up where The Orange Door left off and progress support to the client. The adequacy of information transferred during referral from The Orange Door to referred agencies requires investigation and confirmation by FSV as The Orange Door evolves (noting the need to comply with privacy legislation).
• Partnership organisations’ commitment to the model: Evidence from fieldwork, including from HLG and OLG meetings, indicates that some partner agencies have concerns about The Orange Door model, both in relation to how to operationalises integrated practice, and in understanding the responsibility of the Hub Manager. This inhibits integration and service access as decisions can be delayed at an HLG or OLG level. It is unclear if concerns about aspects of the model expressed by some partner organisations reflect an ongoing ‘lack of commitment’, the immaturity of collective leadership by the HLG (at this point in time of the evaluation) or a response to the early challenges of implementation. This may become clearer within each HLG over time and as the operational model evolves.
• A review and monitoring function: There is no formalised feedback loop with services outside the partnership in this early stage of implementation to determine whether the client has received service and if this service has met their needs. Within the partnership, the practice of review and monitoring is somewhat more integrated as The Orange Door practitioners are able to check in with their employing agencies (or ask a colleague to). Nevertheless, there are no clear responsibilities in The Orange Door to check up on previous allocations or referrals, nor a technology enabled solution to allow someone to do so. The absence of this function was felt by clients too. One client stated:

'Maybe maintain contact for a little bit longer than maybe just one phone call so that as you go through the process there’s still I don’t know even monthly contact. And I know that’s probably a big ask because they’re inundated with people but just to make sure that processes are running smoothly with you and if you need to be referred to somewhere else. I think a lot of the time and for me too women are afraid to even ask for [help] because and when you’ve been in an abusive relationship and particularly like me when you’ve been in one for a very long time you lose so much sense of yourself that to actually ask or to say that look I’m not really happy with this worker I need you to refer me somewhere else is perhaps something that you’re not going to do because you’re you don’t even know how to stand up for yourself. You lose that sense of self … A bit for more follow up I think. If that’s at all able to be done would be good.'– Client interview

• Perceived lack of capacity of the service system: During fieldwork, some practitioners indicated that they did not refer, or delayed referring due to constraints in the service system, (for instance, case management workers reaching their maximum number of clients). According to FSV, in almost all cases, this perception was incorrect, however, it highlights the absence of the capacity for either The Orange Door practitioners or FSV to concretely gauge supply in the service system.

Timing and effectiveness of connections to services

The concept paper states that The Orange Door will connect people quickly and seamlessly to the supports and services they need. Employing agencies can allocate services to clients, or referrals can be made to a broader range of supports and agencies. Where clients are not able to be provided with a service response straight away, their risk and needs will be actively monitored.

Based on evidence from fieldwork, connections to services are not always timely at this early stage of The Orange Door implementation.

A number of FSV staff and The Orange Door workforce expressed uncertainty about whether practitioners were taking longer to connect clients to services than anticipated due to greater effort required to assess some clients and undertake a higher level of intervention:

• Many practitioners stated that their case management backgrounds meant that they were used to supporting clients through difficult periods. They also indicated that it was difficult not to become emotionally invested in certain cases, and that clients developed a personal bond with them.
• Practitioners also stated that the services appropriate for cases involving family violence are often more predictable than those involving child wellbeing. They are more likely to require brokerage and less likely to require home visits. This means that victim-survivors of family violence can be connected more rapidly to services than can families where child wellbeing is the focus.
• The extent of assessment undertaken in child wellbeing cases means that the time between assessment of these cases to being connected with services may be longer. [4]

Clients interviewed also touched on the inability of The Orange Door to monitor connections with services:

• To one client, it appeared The Orange Door workers were not always aware of the services and products available to survivors of family violence.

'And letting you know everything that actually is available out there... And just to give you a bit more especially if like you’re new to being involved with a DV service you know it’s just we’re here as an ear or we’re here to get you out temporarily or you know which is what seems to happen a bit … Because I’ve spoken like when I’ve been speaking like I deal with [worker’s name] when I’m talking to [worker’s name] and that I was talking to another girl [worker’s name] and I’ll bring things up stuff and they’ll go I didn’t even know that existed …. I think at the intake there’s probably more that they can give you more [pause] just arm you with knowledge… If they [The Orange Door] were armed with more information and in simpler terms … I really think that overall they need more training and knowledge of the services that are out there that actually do good and actually do stuff you know. ' – Client interview

Another client stated that they had concerns about the capacity of The Orange Door to ensure connections to services delivered appropriate outcomes for clients:

'The only one thing that I can say that didn’t occur was she said to me that she felt I needed to go to [name of a sexual assault service] … and that never happened. And it didn’t happen with [name of a counselling and housing service] either. That would be the only thing I’d say. She [the worker] said she was going to talk to someone about that at The Orange Door and get back to me and that never happened. Now whether that’s her [the worker’s] fault or that’s the referral agency … I don’t know. But that would be the only negative thing I would say … It’s understandable that things slip through the net and I’m not angry about it at all because I think that the service that they provided was pretty good.' – Client interview

Two factors mean that it is difficult to quantitatively analyse the timeliness and effectiveness of connection to services and critical interfaces at this point in the establishment of The Orange Door.

• While CRM does capture the time the case was opened and the time of case closure, this does not necessarily reflect the sum of time spent on the case.
• As detailed in Section 4.1.1 and 4.3.2, there is limited follow up with external services to determine the outcomes of allocation or referral. The piecemeal nature of funding and service delivery means that tracking client outcomes may be difficult.

Improved recording of allocations of core services, referrals out of The Orange Door, and The Orange Door service response on the CRM will be required in order to conduct further quantitative analysis of allocation and referrals out of The Orange Door.

Key finding 56:

The nature of assessment reported to be required in child wellbeing cases means that the time between initial intake, assessment and then referral to services may be longer than for other types of cases in The Orange Door.

Key finding 57:

Connections to services may be delayed and may not always be effective for a variety of reasons. These include difficulty in obtaining information required for assessment, lack of knowledge and clarity amongst The Orange Door practitioners about the services available and the undeveloped nature of the ‘review and monitoring’ function at this point in time.

Key finding 58:

Some clients interviewed indicated that they believe that The Orange Door does not have sufficient understanding about the service response needed or available within the broader service system. This may be a point in time finding from the evaluation and would require validation with additional clients before firm evaluation judgements could be made about if this will be an ongoing experience of clients.

[1] Please not, that there is a Monash University review of the Family Violence Information Sharing Scheme underway and due to report in 2020. The review will consider any adverse impacts or unintended consequences of the scheme – and the findings from this evaluation may inform this review.

[2] Qualitative evaluation evidence from fieldwork

[3] FSV, ‘Support and Safety Hubs: Concept paper’, July 2017.

[4] Qualitative evaluation evidence from fieldwork

The scale of change that The Orange Door has faced in the early establishment phase cannot be underestimated. The context of the family violence reforms and the early stages of the roll out of The Orange Door are critical when considering the findings of this evaluation. It was not expected that The Orange Door would be operating seamlessly in an integrated manner at the time in which the 2018 evaluation was undertaken. The Orange Door represents a new way of working for FSV and the partner agencies and FSV is a new agency itself. Integrated practice with victim survivors, perpetrators and child and family services is a new model, based in new facilities, with new processes, and new staff.

There has been enormous effort on the part of FSV, all agencies and their staff involved to establish The Orange Door and this was evident to our evaluation team during field work and stakeholder interviews. The findings recognise this effort, and the opportunities for improvement identified in the report are intended to further support and develop the effective delivery of integrated services and integrated practice in future areas.

It is the nature of a developmental evaluation to allow for rapid change to implementation in response to emerging evaluation findings. FSV have already made changes, or are addressing some issues identified in our findings and subsequent recommendations. Examples of this are facilities upgrades, a new ‘go live’ criteria before services commence, the development of a workforce strategy, and further CRM iterations, all undertaken during the period that the evaluation was conducted.

In line with a developmental approach to evaluation of a program in the very early stages of implementation, we frame our ‘recommendations’ in terms of opportunities we have identified. Making ‘recommendations’ implies that FSV are unaware of issues or do not already have a staged plan to address these. It also assumes that The Orange Door will not have evolved between the time of our evaluation field work and the provision of this evaluation report. As such, we frame our advice in relation to the opportunities we see are available at this point in time to strengthen The Orange Door model and initial service offering, and further develop establishment processes for subsequent areas as The Orange Door is rolled out across Victoria.

Overarching themes of our findings and opportunities

The many findings of the evaluation can be grouped into a number of overarching themes that relate to more than one evaluation question or line of inquiry, or relate to only one The Orange Door area.

Overarching themes of the evaluation findings are:

1. The commitment by FSV, partner agencies and the workforce to the concept of integrated practice and The Orange Door model has been sufficient to overcome most of the start-up challenges.
2. Foundational key concepts in The Orange Door model are clearly evident and are defined in a range of practice guidance and foundational documents. These need refining to operationalise them in practice.
3. Consistency of practices and processes within and between The Orange Door areas now needs focus. Clearer operational guidance is needed around the processes and tools that can be standardised across The Orange Door areas as opposed to those that allow for local variation. This may become clearer as more areas are established.
4. The volume of change in practice posed by The Orange Door should not be underestimated and needs to be factored into planning and development in the first 12 months in any new area where The Orange Door is established.
5. The integration of perpetrator services in The Orange Door requires focused effort to include these elements of the service system in a more effective way.
6. The Orange Door is still perceived as primarily a family violence-focused initiative by child wellbeing practitioners.
7. CIP has been positive for information sharing to inform risk assessment and practice.
8. Outcomes and the experience of clients was unable to be assessed with confidence as part of this evaluation. Further exploration of how to validate findings through client feedback should be a focus of future evaluation effort.
9. Data quality issues in this early stage of implementation have had an impact on service delivery and performance management within each The Orange Door area.

Assessment of opportunities

The developmental nature of the evaluation, the scale of the evaluation and the multiple lines of inquiry have understandably generated a significant number of findings and opportunities. It is not possible for FSV and/or partner agencies to implement all of these, and as such, we have used a matrix to assess the value of each opportunity and our view on the effort of implementation. In this assessment we have defined:

Value: The contribution made towards The Orange Door service model and associated reforms.

Implementation: The likely degree of implementation complexity.

Figure 13 shows this matrix and situates findings within different quadrants using a traffic light system.

Figure 13: Matrix to assess opportunities for improvement to The Orange Door

The relationship of the opportunities to the evaluation questions

The opportunities are presented in order to address each of the evaluation questions:

cross-referencing an opportunity for improvement against key findings from the evaluation

identifying our assessment of the value and implementation effort of each opportunity using a traffic light system

identifying if the opportunity relates to specific operational areas, future areas or the broader FSV program of work.

The role of FSV as a ‘system steward’ during early implementation of The Orange Door

At the foundation of The Orange Door concept is a need for collective leadership through the organisations in The Orange Door partnership – where each organisation brings their specific skills and services to the collaborative effort of The Orange Door intake and assessment system. It is to be expected that as the model evolves and service processes and procedures become embedded, that each HLG will move from their early focus on operational issues to a more strategic focus on localising the service to meet the needs of their clients. This is aligned with the concept model of The Orange Door – and in time, this should be achievable.

However, at this early stage of implementation, uncertainty over how to operationalise the model and the pressures of establishment indicate that FSV needs to play a larger role as ‘system steward’ than originally anticipated. The nature and outcomes of The Orange Door will be influenced by many different stakeholders (including partner agencies) working together. In this context, FSV will need to oversee the ways in which The Orange Door is being implemented and attempt to steer the system more directly towards certain outcomes, to reinforce positive attributes of the model, and to identify and correct any process and operational issues that detract from positive client outcomes. This goes to the issue of which operational issues of The Orange Door needs statewide standardisation and what can be localised for each area. In the early days of The Orange Door, it is advised that FSV adopt a more overt system steward role to lead The Orange Door more directly towards the outcomes we all hope The Orange Door can achieve for Victorians. With this in mind, we have identified a range of opportunities for improvements to be made, and have deliberately indicated where we believe FSV should take a leadership role in progressing these. This does not mean that FSV alone are responsible, or even that they can alone make these improvements. Many will require collaborative efforts by the partner agencies but at this stage, FSV have a role to lead the partnerships in addressing these improvement areas.

Download conclusion and opportunity summary:

Conclusions and opportunities - Evaluation questions - The Orange Door Final Evaluation Report

PDF 219.02 KB

(opens in a new window)

Download Appendix A Evaluation framework and Appendix B Evaluation methodology.

Appendix A and B - The Orange Door Final Evaluation Report

PDF 345.87 KB

(opens in a new window)