Evaluation of the National Workforce Centre for Child Mental Health (National Workforce Centre) was designed and commenced alongside the Centre’s establishment in 2017. Over time, a Process and Impact Evaluation has evolved, with annual reporting to the Australian Government Department of Health and Aged Care on findings, and regular internal use of formative findings to inform program development. Here we present an overview of the current evaluation model.
On this page
Our evaluation approach
We take a participatory approach to both monitoring and evaluation. Emerging Minds’ operational teams play a central role in identifying and monitoring key indicators, and also work with the Emerging Minds Research and Evaluation team co-designing theory and logic models and conducting evaluation studies.
Emerging Minds’ advisory forums, user advisors and subject matter experts also contribute to the design and delivery of evaluation activities.
Our research partners have contributed specific independent research and evaluation studies, and also provide advice on the overall National Workforce Centre evaluation framework.
The extent of complexity and continuous improvement within the National Workforce Centre makes it necessary to have a responsive evaluation approach that feeds findings back into program development in a timely way. Emerging Minds’ evaluation activities are therefore conducted continuously and planned as part of program development and innovation. The Developmental Evaluation1 approach enables the Research and Evaluation team to better integrate ongoing learning and reflection, and to work alongside program units within their timeframes and strategic priorities.
The following flowchart (Figure 1.) describes the benefits of partnering with external evaluators who bring the authority of a research institute, in addition to subject matter and economic expertise, for elements of the evaluation. Collaborations with external evaluators occur alongside internal evaluation components in data development, collection and analysis as well as embedding evaluative thinking and program theory approaches within the organisation’s operations.
Hybrid research and evaluation model, 2020
See accessible data from Figure 1. Emerging Minds’ 2020 hybrid research and evaluation model
Figure 1. Emerging Minds' 2020 hybrid research and evaluation model
Embedding evaluative thinking
In the following video (5 minutes, 8 seconds) Emerging Minds’ Manager of Research and Evaluation, Dr Melinda Goodyear, talks about embedding developmental evaluative thinking throughout the organisation.
Our program logic
The program logic for the National Workforce Centre has three streams of activities and outcomes which look for engagement and changes at the practitioner level, the organisational level and at the systems and policy level (Figure 2). Under this logic, the National Workforce Centre program works towards an overall service system(s) which is equipped to identify, assess and support children at risk of and experiencing emerging mental health difficulties.
In the short term, our program logic anticipated that the National Workforce Centre would register changes in professionals’ knowledge and awareness regarding prevention and early intervention to support child mental health, while gradually developing their skills, confidence and competence. As the program transitions to mid-term after five years of operation, focus also shifts towards embedding changes in practice, and systems drivers to support workforces and organisations toward this end.
National Workforce Centre program logic
See accessible data from Figure 2. National Workforce Centre program logic flowchart
Figure 2. National Workforce Centre program logic flowchart
Monitoring, evaluation and quality improvement
With many components of in the National Workforce Centre now well established, there has been a shift to monitoring (rather than evaluating) key performance areas including:
- program reach
- engagement of target audiences
- relevance of and satisfaction with resources and products; and
- post-learning impacts.
Monitoring data is used to inform continuous quality improvement of National Workforce Centre program components, contributes to data analysed for evaluation purposes, and is reported six-monthly to the Department of Health and Aged Care.
Emerging Minds’ evaluation resources are targeted at assessing new and developing National Workforce Centre program components – including innovative resources, engagement and implementation support strategies – and investigating evaluation and research questions that remain of interest. Consistent with the National Workforce Centre program logic, evaluation is now increasingly focused on measuring and understanding changes in practitioner behaviour within various workforce groups, and in the mechanisms of change within organisations and service systems, while seeking to identify early- to medium-term impacts.
Key evaluation findings 2017–2022
The ongoing evaluation of the National Workforce Centre is reported annually to the Australia Government Department of Health and Aged Care. Below are some of the key themes from findings of the evaluation over the years.
-
![]()
Practitioners who complete courses improve their knowledge
Across the range of online courses, users demonstrate an average of 18% increase in the knowledge, confidence and skills pre- to post-completion of a course.
The extent to which Emerging Minds online courses are meeting their learning objectives is measured by asking users to complete a few questions about the course material before and after completing the course. Evaluation analysis showed the increases following course completion resulted were statistically significant and of high impact.
Learning from Emerging Minds courses is sustained over time
Around 40% of users agree to be a part of our evaluation by completing a workforce development questionnaire every three months to monitor the impact of their learning over time.
Data from the three-monthly workforce development questionnaire suggests there are statistically significant advances in knowledge, confidence, competence and practice around child mental health in those engaging with the National Workforce Centre.
Workforce questionnaire total scores from the first questionnaire (on day 0) to the ninth questionnaire (at 27 months)
See accessible data from Figure 3. Workforce questionnaire total scores from the first questionnaire (on day 0) to the ninth questionnaire (at 27 months)
Figure 3. Workforce questionnaire total scores from the first questionnaire (on day 0) to the ninth questionnaire (at 27 months)
Figure 3 note: Participants who answered ‘not applicable’ for all or part of the questionnaire were systematically removed to avoid confounding the mean calculated for single items or the total for the six item scale. Results for each time point may not be from the same participants. Items were rated on a seven point scale where 1 = strongly disagree and 7 = strongly agree.
As at 2022, the workforce development questionnaire comprises seven questions around knowledge, confidence, skills and practice (items). Each question showed improvement after engagement, and that improvement was stable and maintained over time, without regression, while a few of the questions showed further increases in the second year. Increases in confidence to identify at-risk children and to talk to parents about their children’s mental health were seen in the first three months and continued to grow two years after first engagement. While measuring practice is more difficult, the workforce development questionnaire measures an increase in practitioners reporting regular practice of talking to parents about the impact of mental health on parenting, and this continues to grow at two years of engagement. Improvements were seen in all professional groups but an early shift in knowledge, confidence and skills was most pronounced for clinical health professionals and tertiary students.
You can join the evaluation when first registering for Emerging Minds Learning, or register your interest in participating in future research projects.
Emerging Minds Learning is being accessed across Australia
Evaluation of Emerging Minds Learning in 2021 showed National Workforce Centre registered users represent a wide geographical reach across Australia and vulnerable communities.
The highest proportion of professionals work in Victoria and NSW, followed by South Australia and Queensland. The greatest representation of professionals from regional or remote areas by state or territory was the Northern Territory (98.2%), followed by Tasmania (21.6%), Queensland (18.4%) and Western Australia (18.0%). For each state and territory there were users from across a wide range of socioeconomic conditions, as described by the Australian Bureau of Statistics’ SEIFA Index for Relative Socioeconomic Disadvantage.
Remoteness classification of registration from postcode, as a proportion of registrations from each state
See accessible data from Figure 4. Remoteness classification of registration from postcode, as a proportion of registrations from each state
Figure 4. Remoteness classification of registration from postcode, as a proportion of registrations from each state
Figure 4 note: Registration data as at 30 June 2021.
-
![]()
Engagement with Emerging Minds continues to grow
Emerging Minds audience continues to grow with increasing numbers of new users registering and new course completions each period.
There was a particular extreme spike in users registering and engaging with Emerging Minds in 2020 during the peak of COVID-19 restrictions.
Emerging Minds users show greater competence in child mental health knowledge and practice than those not engaged with Emerging Minds
Emerging Minds’ inaugural National Workforce Survey for Parent, Family and Child Mental Health showed that workers who had actively engaged with National Workforce Centre resources rated their capabilities significantly higher than those who were aware of Emerging Minds but not yet engaged, and those who had never heard of Emerging Minds before.
The survey asked workers in Australia to rate their capabilities in understanding and responding to children’s mental health. Workers who engaged with Emerging Minds rated their capability around 15–20% higher than those who had never heard of Emerging Minds, across six domains of child mental health competency.
Differences in mean scores on child mental health workforce capabilities by exposure to Emerging Minds resources
See accessible data from Figure 5. Differences in mean scores on child mental health workforce capabilities by exposure to Emerging Minds resources
Figure 5. Differences in mean scores on child mental health workforce capabilities by exposure to Emerging Minds resources
Implementation case studies
Emerging Minds partners with organisations to support them to embed practices that better support the mental health and wellbeing of children. Partnerships can be with agencies, sectors, educational institutions, professional bodies or governments. This ‘Partnerships and Implementation’ approach works with partners on projects of high priority to organisations and with potential to improve services and care for children and their families. To understand whether these implementation partnerships are effective and why, we use a realist (Pawson and Tilly, 1997) case study approach to analyse change processes and outcomes in the varying contexts in which Emerging Minds and its partners work. Read examples of implementation case studies below:
References
Patton, M. Q. (2011). Developmental Evaluation: Applying complexity concepts to enhance innovation and use. New York: The Guildford Press.