Bulletin – December 2021 Australian Economy Do RBA School Talks Improve Student Outcomes?

Introduction

There has been a stark decline in the size and diversity of the economics student population in Australian high schools since the early 1990s (Dwyer 2018; Livermore and Major 2021). To address this decline and promote economic literacy in the wider community, the Reserve Bank of Australia (RBA) established a public education program in 2016 to support teachers and students. This article presents the results of a study designed to evaluate the student outcomes of one component of this education program: educational talks to high school students.

In the talks delivered to Year 11 and 12 students and their teachers, RBA economists discuss and answer questions about monetary policy and current economic conditions (two components of the Economics syllabus across the states). These talks typically run for one hour, which includes 40 minutes for a detailed presentation delivered by an RBA economist and 20 minutes answering questions from students and teachers on monetary policy, economics and careers. The talks aim to improve the economic literacy and confidence of students and teachers, inspire students who may be interested in further study or a career in economics, and increase the diversity of the economics student cohort and profession.

Over the past five years, around 22,000 high school students across all education sectors and states in Australia have attended an RBA school talk about monetary policy and current economic conditions (Graph 1). These students were mostly in Years 11 and 12. RBA talks to high school students more generally have reached more than 30,000 pupils over the same period. The education program has consistently received positive feedback from economics teachers. However, it is important to consider the feedback of students who attend these talks to evaluate the effectiveness of the talks program, specifically: are the school talks improving student confidence, understanding and perceptions of economics? Such feedback can be used to further improve the school talks program. To obtain a robust indication of the program's effectiveness, a controlled experiment using surveys of students' feedback was conducted.

Survey design and methodology

To quantitatively evaluate whether the school talks program leads to improved outcomes for students, we asked student participants to complete two surveys, and assessed changes in their responses over this time. The surveys included questions on: student confidence with economics; their interest in further study or a career in economics; and their understanding of concepts covered in the school talks program, such as the RBA's inflation target and the current rate of inflation (for the full list of survey questions, see Appendix A). The potential sample consisted of Year 11 and 12 students in schools across Australia who participated in the school talks.

As students were likely to be learning additional content between the first and second survey, as well as the fact the survey itself may have invoked some thinking around these issues, we utilised a randomised control trial design to more rigorously consider the impact that school talks have on student outcomes. We randomly allocated some schools to a treatment group and other schools to a passive control group.^[1] For the treatment group, one survey was administered prior to the RBA school talk and one survey afterwards, such that the change in responses across the two surveys would reflect the impact of the school talk as well as other influences. The control group was administered both surveys prior to the RBA school talk. Therefore, if there were changes in the responses across the two surveys in the control group, this would reflect some random variation as well as students learning from sources other than the RBA talk – including from taking the first survey, for example. Typically, control groups do not receive a treatment or intervention.^[2] However, we decided this would not be appropriate in this study, as we aimed to support all students with a school talk. Instead, the control group received a talk following their two survey responses.

Students completed the two surveys around one week apart. This spacing was chosen to strike a balance between measuring medium- to long-term information retention and reducing the interference of teaching or other interventions in the responses.

Figure 1 below illustrates this survey design. The orange boxes represent the first and second survey of the control group administered prior to the students receiving the school talk. The green boxes represent the first and second survey of the treatment group administered either side of the talk.

The surveys were administered by teachers and were designed to take up little teaching time. Participation was voluntary and each survey took students on average nine minutes to complete. Between May and August 2021, a total of 2,900 students across 99 schools submitted at least one survey. In order to identify changes in student responses, both surveys needed to be completed and clearly attributable to the same student, yielding a final sample for comparison of 658 students across 64 schools; 342 across 40 schools in the treatment group and 316 across 24 schools the control group.^[3]

We measured the changes in responses across the two treatment group surveys relative to the change in responses across the two control group surveys to evaluate the effect of the school talk on student outcomes. That is, once we had controlled for the additional learning taking place over a week, as well as each student's pre-existing confidence, perceptions and understanding, how did the change in confidence, perceptions and understanding of students in the treatment group differ to the changes in the control group? For further information, we asked students for their feedback on the school talk itself.

Results

Student-perceived value of the talks

The survey included a number of questions asking for feedback on the talk: was the talk useful, engaging, clear and helpful for understanding key economic concepts? The students were given seven options, ranging from strongly disagree to strongly agree; the average and median student response was ‘agree’ to each of these questions (Graph 2). While this positive feedback may be partly due to response bias, it is consistent with the anecdotal positive feedback received from teachers over the past few years.

Four formats of talks were examined:

• Webinar: large multi-school online events with one RBA presenter.
• At school: an RBA economist provided an in-person talk in a school classroom or auditorium, typically to one class at a time.
• At the Bank: an RBA economist provided an in-person talk at the RBA Head Office in a small room, typically to one or two small classes at a time.
• 1:1 Zoom: an RBA economist provided a talk over Zoom, typically to one class at a time.

All different formats of talks received positive feedback in an absolute sense – that is, scores were greater than four (which represented indifference). However, student responses indicated that the Webinars were not as useful, engaging, clear or helpful as talks provided in the other formats.

There are likely a number of reasons why Webinars received less-positive feedback. First, the Webinar series was designed to be somewhat shorter than the other talks in order to hold the attention of students. Anecdotal feedback from teachers suggested the talks may have been too short to cover the content in great detail. Additionally, Webinars had much larger audiences (up to 10 schools in each Webinar) than other talks (which were delivered to a maximum of two schools at a time). Due to the size of the audiences in the Webinars, interactivity and engagement was limited to students or teachers typing questions into a ‘Q&A’ function. Anecdotal feedback suggested students often watched the Webinar on a large screen in a classroom or hall, unable to be seen by or engage with the speaker. It is likely that this lack of interactivity reduced the level of engagement and perceived usefulness of the talks overall.

In contrast to the Webinars, the 1:1 Zoom talks, which were also conducted online with single classes or single schools were perceived to be just as useful, engaging, clear and helpful as the talks provided in person. The less-positive responses to the Webinars were therefore likely due to their passive nature when delivered to large groups, rather than simply because they were online.

Student-perceived understanding and confidence

Understanding economic concepts is difficult and, given the constantly changing nature of economics, many teachers have reported that staying up to date with current economic conditions is challenging.^[4] Therefore, an important part of evaluating whether the talks program is effective was to consider how students' perceived understanding of, and confidence with, economic concepts and economic conditions changes after receiving a talk. In both surveys, we asked students to rate their understanding of monetary policy, current economic conditions and their overall confidence in completing their economics subject on a scale of one to five (with five being the most understanding and confident).

Student self-reported understanding of monetary policy was found to have improved considerably in the treatment group, while there was no discernible change in the control group (Graph 3). (The treatment effect is estimated to be 0.38 standard deviations – a large effect in education settings, although within the range of effects from studies looking at self-reported outcomes of programs (Durlak et al 2011)). The improvement was evident for both Year 11 and Year 12 students in the sample.

Students' self-reported understanding of current economic conditions also improved in the treatment group when compared to the control group (Graph 4). Students reported a higher baseline understanding for current economic conditions than for monetary policy, which makes sense given the timing of the study during the COVID-19 pandemic when the students were likely to be highly attuned to the volatile economic conditions.

Student confidence in completing and understanding their economics subject was also found to improve across the two surveys in the treatment group, while it remained stable in the control group. These improvements were not as large as those for the specific questions relating to understanding monetary policy and current economic conditions, but still represented a statistically significant change relative to the control group. This relatively smaller treatment effect likely reflects the fact that monetary policy and current economic conditions make up only a portion of the Economics syllabus that students are required to learn and understand.

Across all three of these questions on student-perceived understanding and confidence, there did not appear to be a difference between male and female students. Likewise, both Year 11 and Year 12 students reported improvements in confidence and understanding following the RBA school talk. Furthermore, the socioeconomic status of students and whether the student attended a government or non-government school did not appear to determine their perceived changes in understanding or confidence.

Self-reported interest in further study and a career in economics were broadly unchanged between the first and second survey. This was consistent across the control and treatment group. While the talks were perceived to have improved student understanding of monetary policy and current economic conditions, the talks themselves did not influence their career or study plans immediately after the talk. This is not completely surprising. Students were surveyed around one week after the 40-minute school talk – it is unlikely that such a small intervention would invoke a large change in the aggregate career path or study intentions of students within such a short timeframe. While there was no change in the responses between the first and second surveys, the initial survey responses on these two questions suggested that female students were much less interested in further study or a career in economics than male students. These findings are in line with previous research (Lovicu 2021; Livermore and Major 2021).

Measured student understanding

While perceived understanding is an important element of students' confidence and engagement with economics, actual understanding is also important – securing an increase in actual understanding is one of the more ambitious outcomes of the school talks program.^[5]

The RBA school talks were found to increase the measured learning (or understanding) of information covered, in addition to improving student-perceived understanding. We constructed a high-level indicator of understanding by tallying all the responses of the knowledge-based questions and identifying the number that were correct. We found there was an increase in the total correct responses between the first and second survey in the treatment group, while there was no similar improvement in the control group. Upon accounting for students' self-reported sex, year and their school's socioeconomic status, we found the difference between the treatment group and control group was statistically significant and thereby that the RBA school talks improved students' measurable understanding.^[6] This reinforces the above finding that students believe they understand more about monetary policy and current economic conditions following an RBA school talk.

The improvements in measured understanding and information retention in the treatment group were found across all the formats used to deliver the school talks: Webinars; talks conducted at the RBA; talks conducted at the school; and Zoom talks conducted with one or two schools. Therefore, despite the Webinar talks receiving slightly less positive student feedback (discussed above), they still resulted in information being conveyed to and retained by students. This is likely because the Webinars covered the key concepts, even if they did not include as much time for student engagement and reinforcement.

Male and female students both benefited and did not appear to have a different response to the RBA school talks. Likewise, whether a student goes to a government or non-government school did not appear to determine their improvement in understanding. Socioeconomic status also did not appear to be a determining factor in how students received or retained information from the RBA talk.

Improvements in understanding were evident for both Year 11 and Year 12 students. However, there was a much larger increase in measured understanding among Year 11 students than year 12 students (Graph 7). This was primarily due to the nature of the questions asked; Year 12 students had a much higher measured understanding of the content prior to the talk and as a result we did not see as large an increase in their scores. In contrast, Year 11 students had lower pre-talk measured understanding and so we saw much larger increases following an RBA talk.

Among Year 11 students, we saw large increases in measured understanding of relatively simple monetary policy concepts following a school talk, such as knowledge of the RBA's inflation target and the tools used to conduct monetary policy (i.e. whether it is taxes, government spending or interest rates). In contrast, the survey measured a smaller improvement for Year 12 students, because their pre-talk understanding was already very high (prior to the talk, 92 per cent of Year 12 students correctly identified the Bank's inflation target, compared with 67 per cent of Year 11 students). It is important to note here that Year 12 students may have gained a deeper understanding of these concepts as a result of the school talk – for example, there is a big difference between knowing the RBA inflation target is 2–3 per cent and understanding what an inflation target is used to achieve and why. The surveyed ‘knowledge’ questions were unable to identify these changes in understanding, but the student self-reported increases suggest this might have been the case. More detailed questions may have identified these changes, but would have come at an increased time burden for teachers and students, which may have weighed on participation.

More complex questions that involved understanding the decision rules behind monetary policy or the mechanisms involved also saw increases in measured understanding in the treatment group. For example, students were given an inflation rate and an unemployment rate and asked which direction the cash rate should be moved in order to fulfil the RBA's objectives.^[7] There was a statistically significant increase in the number of students correctly identifying the required or likely direction of interest rates. In contrast, survey questions that separately identified and investigated the transmission mechanisms of monetary policy saw no discernible increase in correct student responses. This likely reflects the complexity of these concepts; because RBA school talks are only 40 minutes long, there is a trade-off between explaining these concepts and mechanisms in more detail and covering other content, such as an update on current economic conditions.

Overall, following the school talks, students were more able to correctly identify current economic conditions, the basic features of monetary policy and the decision rules involved in monetary policy; however, they did not exhibit a measured increase in their understanding of the mechanisms through which monetary policy operates.

Implications of results

There are a number of important implications and learnings from these results:

1. Previous Bank research has found that there are both interest and performance reasons why Year 12 students who study economics do not preference or enrol in economics in university (Lovicu 2021). While our survey analysis found no change in student interest in further study following an RBA school talk, we identified large and significant changes in confidence, as well as in both perceived and actual understanding of key economic concepts. The school talk interventions are therefore among the factors that may assist students to overcome these performance barriers to enrolling in economics at university. While these improvements were found across the diversity spectrum in this survey, Lovicu (2021) found that less socially advantaged students are more likely to face performance barriers. Therefore, to the extent that these school talks improve student performance, they may benefit socially disadvantaged students more, increasing the size and diversity of the economics student cohort.
2. The survey responses suggest that online talks, if targeted correctly, can be just as effective for economic literacy and advocacy as talks conducted in person. In-person talks, especially those where an RBA economist attends a school, can be resource intensive and less practical for schools located outside of capital cities. The survey responses suggest that talks can be delivered to schools in an online format without any considerable loss of quality or impact. As a result, the school talks program could be expanded considerably to areas outside New South Wales, as well as to more remote and regional areas. While the vast majority of schools teaching economics are in metro areas, this assistance to teachers and students who were previously unable to participate is likely to further improve the diversity and size of the economics student cohort.
3. Tailoring talks to the year level of the students participating is likely to be beneficial. The surveys showed that there are very different levels of understanding of the core concepts of monetary policy between Year 11 and Year 12 students. As such, Year 12 students may benefit more from a thorough explanation of the intricate details and complexities of monetary policy and the current economic environment, with less time spent on the more basic concepts. Of course, the level of presumed knowledge may depend upon the school and cohort. Therefore, having a flexible approach to the content and structure of the school talks is likely to lead to further improvements in understanding.

These results point to a number of avenues for further work. In particular, overseas research has found the role-modelling effect to be important to understanding and engagement (D'Acunto, Fuster and Weber 2021). It is possible that the characteristics of the economist presenter in the RBA school talks may lead to a differentiated result. For example, female students may respond more positively to talks conducted by female economists. We intended to test this hypothesis; however, the limited sample of schools and the fact that a majority of the talks were conducted by a male economist, meant this was not possible. Further work could explore this representation issue.

Overall, results from this survey and other feedback from students and teachers enables the RBA Education team to continually adapt the talks to ensure they remain valuable for teachers and students. For example, the feedback on Webinars has resulted in us extending the duration of the Webinar talks going forward.

Conclusion

The Reserve Bank delivers school talks to Year 11 and Year 12 students to improve economic literacy and encourage a larger and more diverse student economics cohort. Students who participated in the school talks and survey program found them useful, helpful for understanding key economic concepts and engaging. Student-perceived understanding of monetary policy and current economic conditions was also found to increase following the RBA school talk, leading to improved student confidence with their economics subject. In addition to these perceived effects, measured understanding of the content covered in the school talk also increased.

These findings corroborate the strong positive feedback from teachers over the past few years. Additionally, these results provide some guidance for the future of the school talks program. These results suggest that students find small online talks similarly engaging, useful and helpful for understanding key economic concepts as talks conducted in person. Therefore, the geographic reach of the school talks program could likely be expanded considerably without sacrificing the quality of the content provided to students. Further, the content and delivery of the school talks could be more targeted for Year 11 and Year 12 students, respectively.

These proposed changes, as well as the existing benefits students derive from the school talks, should enable students to engage more effectively with economics content, lead to further improvements in economic literacy, and continue to contribute positively to the size and diversity of the economics student population.

Appendix A

Footnotes

The author completed this work while with the Public Access and Education team. This work was made possible by the participation of schools, which was facilitated by the NSW Department of Education, the Victorian Department of Education, principals, teachers and students. We would particularly like to thank Alexander Symonds, Christine Dowd, Christine Reid and Bronwyn Hession for providing guidance on the school surveys, and Andrea Fitzpatrick for participating in a pilot. I would like to thank Tanya Livermore and Christina You for their support with both the survey design and organisation of the school talks. This work benefited considerably from the advice and feedback provided by participants at internal presentations, in particular James Bishop and Adam Gorajek. [*]

Randomisation is by schools rather than students because treatment is administered at the school level rather than the individual level. Individual-level randomisation would require teachers to administer the survey to different students at different times, or exclude some students from the talks (though these excluded students may affected by proximity-based knowledge spillovers from students given the treatment). [1]

In some fields, like psychology, the control group often receives a different treatment that has been shown to be effective. [2]

This large drop in participation for the second survey is likely to reflect the competing demands of students and teachers, along with disruptions stemming from the reintroduction of lockdowns. Specifically, some schools did not complete the second survey at all, some students within schools did not complete the second survey, resulting in the inability to match a student's first and second survey. [3]

An RBA survey of 262 high school economics teachers conducted in 2021 identified that staying up to date with economic conditions was by far the most commonly cited challenge. [4]

An improvement in measured understanding is also not subject to a response bias from students reporting better understanding simply because they received a talk. [5]

While our analysis controlled for males and females, students were asked ‘how would you describe your gender?’ with four response options: ‘male’; ‘female’; ‘other’; and ‘prefer not to say’. [6]

To ensure clarity of the question and remove doubt around measurements and exact targets, students were given large deviations from the RBA objectives – for example, an unemployment rate of 8 per cent and inflation rate of 1 per cent. [7]

References

D'Acunto F, A Fuster and M Weber (2021), ‘Diverse Policy Committees Can Reach Underrepresented Groups’, University of Chicago Becker Friedman Institute for Economics Working Paper No 2021-95, LawFin Working Paper No 21, Chicago Booth Research Paper No 21-20, Fama-Miller Working Paper.

Durlak JA, RP Weissberg, AB Dymnicki, RD Taylor and KB Schellinger (2011), ‘The Impact of Enhancing Students’ Social and Emotional Learning: A Meta-analysis of School-based Universal Interventions', Child Development, 82(1), pp 405–432.

Dwyer J (2018), ‘What Happened to the Study of Economics?’, Address to the Business Educators Australasia Annual Meeting, Sydney, 26 May. Available at <https://www.rba.gov.au/speeches/2018/sp-so-2018-05-26.html>.

Livermore T and M Major (2021), ‘What Is Driving Participation and Diversity Trends in Economics? A Survey of High School Students’, RBA Research Discussion Paper No 2021-06.

Lovicu G-P (2021), ‘The Transition from High School to University Economics’, RBA Bulletin, June.