2022-23 Quarter 1 Fire Services Outcome Framework Progress Report

This Fire Services Outcomes Framework – Progress Report provides the Country Fire Authority (CFA) and Fire Rescue Victoria (FRV) Quarter 1 (Q1) FY 2022-23 fire services performance indicators.

The purpose of this report is to meet FSIM’s legislative requirements under section 141 of the Fire Rescue Victoria Act 1958 (FRV Act) to publish a report that sets out the fire service agencies quarterly outcomes update and relevant, supporting data from CFA and FRV.

This report provides quarterly results provided by CFA and FRV against their respective performance measures and agency commentary on quarterly performance. FSIM has included historical data against measures provided by agencies, where available, dating back to the start of reform (1 July 2020) as a reference for the reader.

Reporting on Outcomes

Section 140(1) of the FRV Act requires CFA and FRV to each prepare an outcomes framework that sets out their respective outcomes-based fire service performance measures. In developing their outcomes frameworks, CFA and FRV each determined their own strategic organisational priorities and the associated indicators and measures of performance to be included in those frameworks.

Publishing agencies’ quarterly outcome indicator data provides a level of transparency of the fire services organisational trends and outputs as the work of the agencies continues. Two years into reporting on outcomes, both CFA and FRV have improved indicator selection and data identification and collection. Explanations of baselines, targets, and supporting data for many of the agencies’ indicators is progressing and some indicators and measures continue to be refined.

Agencies are not yet able to show clear linkages between actions taken or programs implemented and a resulting impact on the baseline/target, service delivery indicator or broader outcome. FSIM has noted this issue previously and recognises that it will take some time and agencies are actively considering how their program measurement can provide a story of progress or impact as their data and analysis further matures.

Tracking the progress of agencies towards their outcomes is a long-term proposition and many trends or patterns are only observable over an extended period of time. For this reason, FSIM intends to provide an annual assessment of agency outcome progress in its Q4 FY 2022-23 report. For quarters one to 3, FSIM notes particular indicators in the summary by exception where a result:

• shows a continuing off-track trend over repeated quarters
• demonstrates a notable achievement
• is a significant change from the previous quarter (positive or negative).

FSIM notes that some performance measures are influenced by factors outside the control of CFA or FRV and in some instances, agencies may not be able to provide substantiated reasons for changes to results within the 3 categories listed above. FSIM will continue to work with agencies to explore clearer linkages and explanations of contributing factors to results as their outcomes reporting continues to develop.

Maturing outcomes frameworks

The Outcomes Frameworks of CFA and FRV signal the strategic priorities of each organisation, and each provides a set of measures to demonstrate progress towards strategic goals. The ongoing review and refinement of both Outcomes Frameworks is important to assist agencies to assess progress towards delivering against their objectives and support informed, evidence-based decision making.

Across Q1 FY 2022-23, CFA has been developing its Year 3 Outcomes Framework, with this work focusing on the continuous improvement of metrics used (rather than a substantive change to structure or approach) and intends to report against the Year 3 framework in Q2 FY 2022-23.

FRV finalised its Year 2 Outcomes Framework in FY 2021-22 but continues to report against a selection of performance outputs in the absence of a full suite of measures against its year 2 framework. FSIM has included the targets FRV set in year one for these output measures as a historical reference for the reader, however, FSIM notes that FRV does not intend to set updated targets for these measures. FRV advises that it intends to report against all Year 2 Outcomes Framework indicators in Q2 FY 2022-23, using output measures as a proxy. This is because data to support meaningful outcome measures is not available for some of the indicators. FRV is therefore reporting on relevant output measures in their place, as these outputs are expected to contribute to achieving the intended outcomes.

Outcomes framework Q1 results commentary overview

Data quality

FSIM notes that in Q1 FY2022-23, FRV identified data issues, which resulted in a revision of previously reported data from FY 2021-22. The data issues affected 14,000 records relating to 8 measures[1] as a result of an IT incident occurring 1-3 September 2022. FRV detected the issue on 12 September 2022 and subsequently updated the relevant data. FRV has implemented new practices, including automated responses, 30-day error log for references and scheduled back-ups of the database to minimise the risk of repeated occurrences of this issue.

Service Level Agreements (SLAs)

FSIM has commented in previous quarters on the need to progress and finalise SLAs (also known by both agencies as Service Level Deeds of Agreements), that completion of clear, agreed, and implemented SLAs between agencies is fundamental to achieving complementary fire service and underpin the broader success of the Fire Services Reform. These SLAs provide a foundation for role clarity and functions for volunteers and staff in both agencies and are required to harmonise procedures. While FRV continues to report on the progress of SLAs (as part of its Outputs Performance report / year one framework and noting that updated targets for this indicator will not be set), the reported 88% relates to agreements agreed “in principle”, not executed agreements.

FSIM has noted in past reports that service level agreements have been challenging and complex and agencies have developed interim work arounds where a documented solution is yet to be agreed. Three years into reform, these foundational agreements – which set out operational and service delivery expectations and requirements – are not finalised. FSIM is aware of ongoing work by both CFA and FRV to negotiate, progress and finalise agreements and note that there is in principle agreement on a proportion of SLAs. However, the ongoing delays, and in some cases inability to finalise agreements, has impacted service delivery efficiency.

Safety and well-being

CFA reported a fourth consecutive quarter of not meeting its baseline for hazard reporting with a result of 34 reports against a 5-year baseline of 106 for the quarter. FSIM notes that as part of reform, CFA staff reduced by approximately one third and it will take time for the 5-year baseline to more accurately reflect CFA’s current staffing numbers and associated hazard reporting. CFA also notes that the key contributing factor to this result was the impact of COVID-19 related restrictions on brigade activities, including requirements to self-isolate. CFA is working to implement a new hazard reporting checklist to assist with targeting key hazards for identification and control and FSIM looks forward to CFA providing further detail about how the new checklist impacts CFA’s ability to provide a safer workplace, once implemented.

CFA did not meet its baseline this quarter for its decrease in emergency response injuries indicator. CFA reported an emergency response lost time injury rate of 79.8 against a baseline of 71.7 for this quarter. CFA intends to review its Occupational Health and Safety Management system including incident reporting and investigation procedures in FY 2022-23 to increase in the reliability of data and underpin more informed insights. CFA also did not meet its baseline for unplanned absences and correlates this result with an increase in COVID-19 related absences during the winter months compared to 2021.

Community preparedness

CFA has significantly increased the number of homes where members have installed a smoke alarm as part of their smoke alarm installation program, compared to Q1 FY 2021-22. CFA is also tracking well in all three related indicators, comprising installation of smoke alarms at homes, replacement of inoperable smoke alarms and increase in the number of vulnerable community members provided with a fire prevention advice.

FRV reported an increase in engagements for structured programs, largely due to Fire Ed sessions being delivered to schools. FRV also reported positive results for its outcome measure of number of people engaged by Fire Equipment Services which it attributes to the resumption and increased take-up (following the easing of COVID-19 related restrictions) of in-person emergency management training.

Capability

FRV continues to report below target results for both its specialist capability and core maintenance skills output indicators. FRV has updated its Skills Maintenance database (SMDB) to now include Division B data and reported a result of 86% against a target of 95%. FRV noted that performance was lower than expected primarily due to staffing issues such as unfilled vacancies and sick leave and changes to the database settings which negatively skewed the data. FRV has identified the issue and aims to rectify it by next quarter.

FRV reported a result of 43% against a target of 100% for its specialist capability indicator. FRV notes that skills maintenance activities for specialist capabilities are not conducted every quarter and, as such, Marine and Urban Search and Rescue Response (USAR), reported 0% for Q1 which reduced the overall percentage to 43%. FRV proposes to adjust the target to reflect the planned delivery schedule, rather than an even distribution across the 4 quarters which may provide a more accurate representation of specialist skills maintenance across the year. FRV also note that operational staffing requirements limit the number of staff that can be released ‘off shift’ for skills maintenance which is also impacting the result. To address this, FRV is conducting a review of all department positions to ensure there are enough operational positions and investigating strategies to encourage greater uptake of training positions. FRV is also working toward development of a learning management system, which should assist in addressing reporting shortfalls.

Footnote

[1] The eight measures are: 1.1.1a The number and rate of preventable residential structure fires attended; 1.1.1b Number of preventable residential structure fires attended; 1.1.1c Number of preventable residential structure fires by common causes and locations; 1.1.3 Percentage of residential structure fires with smoke alarms by type of residence; 1.2.1 Number of preventable fires in higher risk buildings (class 5-9); 1.2.2 Percentage of structure fires in non-residential buildings with smoke alarms or sprinklers; 1.3.3 Number of incidents other than structure fires and other non-fire related incidents; 2.1.1b Proportion of preventable residential structure fires contained to room of origin by time of day

Reference to fire services agencies

References to the fire services in this document relate specifically to CFA and FRV. Where more than one fire services agency is referenced in this document, they are listed in alphabetical order.

Comparison between agencies’ performance against published Outcomes Frameworks

As the 2 outcomes frameworks are based on the agencies’ own strategic and operational priorities and data, comparisons between the agencies should not be made.

Legislative requirements to report on outcomes

The performance measures that CFA and FRV report on are set by the agencies in their outcomes frameworks. Section 140(1) of the FRV Act requires CFA and FRV to each prepare an outcomes framework that sets out their respective outcomes-based fire service performance measures. Section 140(4) of the FRV Act further requires CFA and FRV to prepare and submit to FSIM a quarterly update that sets out their performance against their performance indicators.

Reporting and data limitations

CFA and FRV published their respective Year 2 Outcomes Frameworks in Q1 FY 2021-22. CFA continues to review and refine its Year 2 Outcomes Framework and is undertaking planning to realise improvement opportunities in its Year 3 Outcomes Framework. FRV has finalised its Year 2 Outcomes Framework however, for some indicators, outcome metrics (measures) are yet to be developed because of challenges accessing meaningful data. As a result, FRV continues to report progress against some outcome indicators with year one output measures, while it refines its performance reporting framework.

Report structure

This report comprises FSIM summary (section 1) and the full suite of CFA and FRV performance measure results received for this quarter (sections 4 – 6). CFA and FRV performance results are presented with trend data from 1 July 2020, an explanation of each measure, confirmation of whether the agency met or did not meet the target or baseline, and commentary from CFA or FRV to provide further insight on the result.

Fire Services Implementation Monitor (FSIM)

FSIM was appointed in 2020 in accordance with the Fire Rescue Victoria Act 1958 (FRV Act). FSIM’s functions under the FRV Act are to assess the effectiveness of agencies in delivering against Implementation Plan actions and provide independent assurance to government and the community on the progress made towards modern fire services providing for a safer Victoria. FSIM is required to prepare and publish quarterly reports on CFA and FRV Outcomes Frameworks measures under s141 of the FRV Act.

Publication of reports

FSIM publishes quarterly reports on this website.

CFA’s Outcomes Framework uses an annual baseline total and year to date (YTD) baseline total. This report includes the YTD baseline which is a rolling 5-year average of the cumulative, quarterly performance against indicators.

CFA categorises Year 2 Outcomes Framework indicators into four domains:

1. Strong relationships to keep communities safe
2. A responsive and innovative volunteer service
3. High performing volunteer and paid workforce, and
4. A continuous improvement culture.

Data is available from page 8 in the report document.

2022-23 Fire Services Outcome Framework Progress Report Quarter 1

PDF 1.4 MB

(opens in a new window)

FRV published its Year 2 Outcomes Framework in FY 2021-22. FRV is reporting against most of the performance indicators as outlined in this framework. However, as data for some indicators is unavailable, FRV continues to report against eight indicators to provide a level of data continuity. FRV has not set a FY 2022-23 target for these indicators on the basis that FRV is now formally reporting against its Year 2 Outcomes Framework.

FSIM has included the year one targets set by FRV for these data sets to provide the reader with a reference point. FSIM further notes that some targets are based on a YTD cumulative result and have been identified as such in this report.

FRV categorises its Year One Outcomes Framework / Outputs indicators into 4 domains:

• Community
• Service
• People, and
• Collaboration.

Data is available from page 29 in the report document.

2022-23 Fire Services Outcome Framework Progress Report Quarter 1

PDF 1.4 MB

(opens in a new window)

FRV is progressing work to report against the full suite of measures for its Year 2 Outcomes Framework. FSIM notes that as FRV consolidates outcomes reporting to replace performance reporting, a baseline will be established across suitable outcome measures.

FRV categorises Year 2 Outcomes Framework measures into three domains:

1. Prevention and Preparedness;
2. Response and Recovery, and
3. Organisational Excellence.

Data is available from page 35 in the report document.

2022-23 Fire Services Outcome Framework Progress Report Quarter 1

PDF 1.4 MB

(opens in a new window)