Bulletin – September 2023 Finance Reading through the Lines: Price-setting Indicators from Earnings Calls

Abstract

This article explores how information in earnings call transcripts from Australian firms can contribute to the Reserve Bank’s understanding of their price-setting behaviour, as a complement to information gathered from the Bank’s liaison program. A large language model is used to process and analyse earnings call transcripts and construct new sentiment indicators for input costs, demand, prices and supply shortages from them. These indicators, starting in 2007 and updated to capture the latest August earnings season, provide useful information about economic conditions and price-setting behaviour, including about developments during the recent period of unusually high inflation.

Introduction

Consumer price inflation in Australia, as in other countries, has been unusually high in recent years. While demand conditions have played an important role, supply-side factors have been the biggest driver of the increase in inflation and have been front of mind for company executives and policymakers (Graph 1).[1] These pressures have moderated recently, and inflation has passed its peak; however, it is forecast to remain high for some time yet.

Graph 1
Graph 1: A 2-panel graph showing text-based indicators tracking the level of interest in price pressures over time. It does this by monitoring the usage of words related to price pressures. The top panel shows an index measuring the number of mentions of upstream price pressures (and related terms) in listed Australian firms' earnings calls since 2007. The index peaks in late 2022 at over 3 standard deviations above its mean. It has since fallen to be around 1 standard deviation above its mean value as of September 2023. The bottom panel shows the same thing except this time measuring the number of mentions of upstream price pressures (and related terms) in the RBA’s Statement on Monetary Policy. The index also peaks in late 2022 at over 3 standard deviations above its mean. It has since fallen to be around 1 standard deviation above its mean value.

The impact of these upstream cost increases on consumer prices is not well captured by inflation models, and so other alternative and timely sources of information are important in assessing the inflation outlook. For this reason, the Reserve Bank has been monitoring firms’ price-setting behaviour using insights from the Bank’s liaison program. This article describes an approach involving use of a large language model to process and analyse earnings call transcripts and construct new sentiment indicators based on what firms have said about input costs, demand conditions, final prices and supply shortages. These indicators complement other statistical information, including from the business liaison program. They also allow us to draw inferences about the determinants of firms’ pricing behaviour.

Analysing earnings calls using a large language model

Earnings call transcripts are a rich source of information about firms’ own business conditions and economic and financial conditions more broadly. Earnings call transcripts also offer qualitative information that is not captured by traditional financial statements or other quantitative data. The sentiment and language used by executives during these calls provides additional context, nuance and insights that is valuable for economists trying to understand how firms make choices and how those choices affect prices.

Earnings calls typically take place a few hours after the release of earnings results. Most large listed Australian firms hold earnings calls during the February and August ‘earnings season’, covering results over the period to December and June, respectively, although there are several firms that release their results on a different schedule. During the calls, company executives deliver prepared remarks summarising the overall business position of the company and the operating environment. This is followed by a question-and-answer session where all interested parties – including institutional and individual investors and expert analysts – can ask questions about the outlook or probe into other issues.

For our research, we process and analyse around 5,500 earnings call transcripts, involving over 750,000 paragraphs of text from 2007 to the most recent August earnings season.[2] The challenge in working with such a rich set of textual data is to systematically examine the transcripts in a way that incorporates the nuance and context of the discussions that took place. Techniques in natural language processing – a field that uses computers to process and analyse large amounts of text – have now advanced to the point where such discussions can be processed and analysed in a meaningful way. This allows us to construct informative quantitative metrics, which we can track over time to examine factors affecting firms’ price-setting behaviour.

To examine the text of the earnings call transcripts we use a large language model.[3] Broadly speaking, a large language model is an artificial intelligence algorithm that uses deep learning and was trained on massive amounts of textual data. For each paragraph in a transcript, the model assigns a probability over several pre-defined topic labels independently. To map from a probabilistic measure to an indicator variable, we assign a topic label to a paragraph by giving it a score of +1 if the probability for that label is greater than a pre-defined threshold. In the stylised example shown in Figure 1, the model classifies the paragraph as being about ‘hiring difficulties’ and ‘labour costs increasing’. To construct the ‘labour cost indicator’, we sum the number of paragraphs in the transcript labelled ‘labour costs increasing’, subtract the sum of those labelled ‘labour costs decreasing’ and divide the balance by the total number of paragraphs in the transcript. These scores are then aggregated across all transcripts in each period. In practice, we construct new indicators for a variety of input costs (including labour costs) as well as final prices, demand and references to supply shortages.

Figure 1: A Stylised Example Classification from the Large Language Model
Figure 1: This figure shows a stylised example of how the large language model referred to in this Bulletin article works. The figure flows from top to bottom. The top shows the two model inputs. The first input is a paragraph from an earnings call transcript. The second input is a list of pre-defined topics that the model must classify the paragraph into. The bottom shows the model output. The model outputs a set of probabilities for each pre-defined topic, which indicate the probability the paragraph is about each topic. In the example given, the paragraph is: 'So, yes, we have added to head count in our data and analytics team to try and speed up processing times. And we’ve made our reward package more attractive, though, in the current environment, it’s still challenging to find the right people'. The model assigns a 98 per cent probability that this paragraph is about the topic 'Hiring difficulties' and an 86 per cent probability that it is about 'Labour costs increasing'.

New indicators complement existing soft information

Graph 2 overlays the new aggregated earnings call indicators for total input costs, demand, final prices and labour costs with similar indicators obtained from the following two sources:

  1. The Reserve Bank’s liaison program – this is a formal program of economic intelligence gathering established over 20 years ago, through which Bank staff meet frequently with firms from a pool of around 900 active contacts (Dwyer, McLoughlin and Walker 2022). Details of these discussions are systematically recorded in confidential ‘diary notes’. We use the text of these notes to construct indices for input costs, demand, final prices and labour costs using a similar approach as that applied to firms’ earnings calls.
  2. A monthly survey of around 400 firms from the National Australia Bank (NAB) – this is a survey designed to produce statistical indices related to business conditions. We compare our text-based indices to the NAB survey-based indices for purchase costs, forward orders, selling prices and labour costs.
Graph 2
Graph 2: A 4-panel graph showing how well the new earnings call indicators align with other soft indicators. The top left panel compares the earnings call indicator for input costs to conceptually similar indicators derived from a business conditions survey and from the RBA’s liaison program. All three indicators track each other quite closely. In a similar way, the top right panel compares the earnings call indicator for demand to its counterpart from a business conditions survey and from the RBA's liaison program. Again, all three series track each other closely; however, the indicator derived from business surveys has fallen by more than the other indicators recently. The bottom two panels do the same thing, but this time for prices and labour costs. Like the top panels all three series are closely correlated.

Each source paints a similar picture about recent developments in costs, prices and demand. This is useful for economic analysis, providing confidence in the signal extracted from the liaison program. This is especially true given each source of information has its own strengths and weaknesses.

Earnings calls provide consistent firm-level information from many firms over time and allow for different indices to be constructed and monitored. However, earnings calls are limited to larger listed companies and most of the information is only updated during the earnings season. Information from the Reserve Bank’s business liaisons are timelier, but the composition of firms changes from period to period and responses are influenced by the topics that are covered. Finally, business survey indicators provide consistent information over time, but sample sizes tend to be smaller, firm-level information is not readily available and analysis is limited to a small number of pre-existing indicators.

The more disaggregated measures of input cost indices from earnings calls also align with other relevant indicators. For instance, transportation bottlenecks, as well as other supply disruptions, resulted in the recent sharp rise in goods price inflation. The COVID-19 pandemic limited the capacity of domestic and global firms to produce and deliver products and Russia’s invasion of Ukraine led to sharp increases in the prices of energy and other commodities. Broadly in line with other indicators of shipping costs and delivery times, the earnings call indices confirm that supply disruptions and increased transport costs were prominent issues from 2021 onwards (Graph 3). The more recent decline in these earnings call measures has been less pronounced, which could reflect coverage of past events in earnings calls, that earnings calls are capturing a broader sense of supply constraints than just those two factors, or that supply disruptions can impact firms with a long lag.

Graph 3
Graph 3: A 4-panel graph showing some indicators of supply disruptions and costs of transport. The top left panel shows shipping costs, which increased four-fold over the beginning of the COVID-19 pandemic but have returned to around previous levels over 2022. The top right panel shows a purchasing manager's index of delivery times. Similar to shipping costs, the series indicates delivery times increased over the pandemic but have declined from 2022. The bottom two panels show indicators constructed from earning calls. The left panel shows a transport cost index, which increased in early 2022. The index has declined in 2023 but is still at higher levels than from 2016–2017. The right panel shows an index of supply shortages, which increased substantially over the pandemic, peaking in 2022 and declining (but not fully unwinding) over 2023.

New indicators track official statistical measures

Our new aggregated indicators for input costs, labour costs and prices obtained after the February and August earnings seasons can be compared to official statistics for the growth in producer prices (PPI), compensation of employees (COE) and consumer prices (CPI) over the six months to March and September, respectively. Graph 4 shows the results of such a comparison. The correlations show that over the past 15 years or so, the highest associations between our new indicators and their related statistical counterparts occurs when the series are compared to each other in the same period, with peak correlations of between 0.75 and 0.5.[4] The exception is the earnings call indicator for prices, with the peak correlation occurring when the new indicator leads consumer price inflation by six months. This exercise suggests that information derived from earnings calls obtained immediately after the February and August earnings seasons correspond well to official statistics.[5]

Graph 4
Graph 4: A 3-panel graph comparing the new earnings call indicators to official statistical measures. The top panel compares the earnings call indicator for input costs to producer price inflation. The middle panel compares the earnings call indicator for labour costs to labour cost inflation.  The bottom panel compares the earnings call indicator for final prices to consumer price inflation. The correlations plotted in each panel show that over the past 15 years or so, the highest associations between the new earnings call indicators and their related statistical counterparts occurs when the series are compared to each other in the same period, with peak correlations of between 0.75 and 0.5.  The exception is the earnings call indicator for prices in the bottom panel, with the peak correlation occurring when the new indicator leads consumer price inflation by 6 months.

Developments in the new earnings call indicators can also be compared to developments in the official statistics over the recent period of high inflation. The broader earnings call index for input costs seems better at capturing the role of global factors in recent inflation outcomes, relative to the more disaggregated index for import costs (Graph 5). In particular, the input cost index picked up the recent pressure on firms costs from global goods price inflation a little sooner, and more clearly than the import cost index. The later and smaller increase in the import costs index could reflect a tendency for discussions in earnings calls to focus on specific causes (e.g. the invasion of Ukraine, or the shortage of semiconductors), or perhaps even specific prices, rather than identifying and distinguishing the domestic versus imported origin of price pressures. It could also reflect that higher global goods prices impact Australian inflation through an increase in the prices of domestically produced materials, because of integration between the domestic and foreign markets. For instance, increases in global timber prices contributed to higher prices for Australian timber, which in turn contributed to strong inflation in new dwelling construction costs.

Graph 5
Graph 5: A 3-panel graph showing tradable inflation and import costs. The top panel shows import prices for retail. Import prices picked up over 2022. The middle panel shows earnings call indices for input costs and import costs. The input cost series picks up quite a bit over 2022, while the import cost series increases but not as much. The bottom panel shows tradable inflation, which is higher at the end of 2022 than any other point in the graph, which begins in 2005. Tradeable inflation has declined a bit in 2023 but is still at the top of its recent historical range.

The earnings call indicator for labour costs is consistent with official data showing a substantial increase in labour costs over the past year. Graph 6 shows the sharp rise in average earnings per worker, and the substantial increase in market services inflation, which includes parts of the CPI where labour is the most important cost for businesses.

Graph 6
Graph 6: A 2-panel graph showing labour costs and a measure of labour-sensitive consumer price inflation. The top panel shows two series. The first series shows average earnings per worker. The second series shows market services inflation. Average earnings per worker can be a little volatile. Both series have increased over the past 3 years and are currently very strong. The bottom panel shows an earnings call indicator of labour costs. This series increased from a below average value in 2019, to a very high level over 2022. The series has decreased a little over 2023.

More broadly, supply factors and strong demand have contributed to the pick-up in CPI inflation since 2021, according to several methods for assessing their relative contribution (Beckers, Hambur and Williams 2023). The new earnings call indices support this evidence, confirming that a pick-up in demand contributed to stronger inflation outcomes from 2021, alongside disruptions to supply (Graph 7). More recently, the earnings call demand index has declined by more than some other indicators.

Graph 7
Graph 7: A 2-panel graph. The left panel shows a decomposition of year-ended inflation. It indicates that supply factors have contributed to half of the substantial increase in inflation over the year to 2022. Demand has also contributed, and a smaller portion is ambiguous. Over 2023, inflation has declined, which is attributed to a lower contribution from the supply portion. The right panel shows earnings call indicators. There is a supply shortages index and a demand index. Both the demand and supply series increased over the beginning over the pandemic. The supply series has increased by more, and both have declined recently. However, their current values are still above the range over the past 15 years

Firm-level insights into price-setting behaviour

Because the new earnings call indicators are available at the firm level, we can use them to examine correlations between the sentiment of firms’ discussions about final prices and the sentiment of their discussions about input costs and demand. This allows for inferences to be drawn about the determinants of firms’ pricing behaviour that could be relevant for understanding the dynamics of the inflation process. To estimate these conditional correlations, we use the regression analysis outlined in Appendix A, with our full regression results provided there.

Three findings from this empirical exercise stand out:

  1. Final price sentiment has a stronger association with sentiment about input costs compared to sentiment about demand, after controlling for changes in the operating environment that are common to all firms, including the effect of global supply shocks. This is consistent with survey-based findings that firms’ predominant pricing strategy is to set prices as a mark-up over costs (Park, Rayner and D’Arcy 2010).
  2. Discussions on final prices appear to have become more sensitive to (or at least more correlated with) sentiment about import costs in the post-COVID operating environment, after controlling for changes in the operating environment that are common to all firms. This could indicate that firms have become more attuned to changes in import costs since 2021.
  3. Final price setting sentiment has a stronger association with input costs when the change in sentiment is positive compared to when the change in sentiment is negative. This suggests that rising prices are likely to remain front of mind for company executives even as supply pressures moderate. This asymmetry is consistent with the type of firm-level behaviour reported for US firms in Pitschner (2020) from an analysis of the text of corporate filings.

Finally, we can run a similar regression to that in Appendix A but allow the correlation between demand and input costs with final prices to vary by industry. In this regression we control for changes in the operating environment that are common to all firms in an industry. We find the association between price-setting sentiment and input cost/demand sentiment differs significantly across industries, suggesting there is significant heterogeneity in firms’ price-setting behaviour.

These experimental findings appear to indicate that aggregate price-setting behaviour could depend on the source of the shocks firms face (demand- or cost-driven), the direction of the shock (with firms’ reacting more to cost increases relative to decreases), and which industries are most affected. Taken together, this underscores the importance of continuing to develop rich multisector models of the economy to better understand firms’ reactions to different types of shocks.

Conclusion

This article introduces new sentiment indicators based on earnings call transcripts from Australian firms that are processed and analysed using a large language model. The model can classify text taking account of the subtleties and nuance of natural language. The signal from these indicators about input costs, demand, final prices and supply shortages tracks current economic conditions well. Regression analysis, uncovering conditional correlations between the sentiment of final price discussions and the sentiment of discussions about input costs and demand, allows for inferences to be drawn regarding firms’ price-setting behaviour. The results are consistent with firms using pricing strategies that focus on a mark-up over costs. They are also consistent with firms being more reactive to rising, rather than falling, input costs.

Going forward, the Reserve Bank will use these new sentiment indicators, together with other similar indicators, to monitor developments in current economic and financial conditions. Over time, these indicators will be developed and refined as the capabilities of large language models are further advanced.

Appendix A: Regression analysis

The following panel regressions are run to estimate the association between the sentiment of firms’ discussions about final prices and the sentiment of their discussions about input costs and demand using data from 2007:

pit = αi + θt + γ1demit07-20 + γ2demit21-23 + n=1Nβ1,nicn,it07-20 + n=1Nβ2,nicn,it21-23 + eit.

Here pit denotes the final price sentiment index for firm i in time period t, measured at a quarterly frequency.[6] The term demit07-20 is the demand sentiment index from firms’ earnings calls from 2007 to 2020, while demit21-23 is the index from 2021. Likewise, icn,it is the input cost index for input cost sentiment index n ∈ N. Including separate coefficients for each time period allows us to examine if there is anything different about the period since 2021 – which has been characterised by sizable supply shocks – relative to the historical sample.

All regressions include firm, αi, and time, θt, fixed effects. The firm fixed effects allow us to control for unobservable differences in the language used by each firm during their earnings calls. The time fixed effects control for changes in the operating environment that are common to all firms.[7]

In the second regression below, instead of separate coefficients for each time period, we include separate coefficients for demand and input costs according to whether the change in sentiment was positive or negative. This allows us to examine whether price-setting sentiment changes asymmetrically in response to positive or negative changes in input costs or demand sentiment.

pit = αi + θt + γ1demitΔ+ve + γ2demitΔ-ve + β1icitΔ+ve + β2icitΔ-ve + eit.

The full regression results are set out in Table A.1.

Table A.1: Associations with Final Price Sentiment
Differences in related coefficient estimates in parentheses, 2007–2023 (September quarter)
Table A.1 shows coefficient estimates and their significance from panel regressions. These regressions are run to estimate the association between the sentiment of firms’ discussions about final prices in their earnings calls and the sentiment of their discussions about input costs and demand, using data from 2007. All regressions include firm and time fixed effects. The coefficients in the first column indicate that final price sentiment has a stronger association with sentiment about input costs compared to sentiment about demand, after controlling for changes in the operating environment that are common to all firms, including the effect of global supply shocks. The coefficients in the second column indicate that final price-setting sentiment has a stronger association with input costs when the change in sentiment is positive compared to when the change in sentiment is negative. The coefficients in the third column indicate that discussions on final prices appear to have become more sensitive to (or at least more correlated with) sentiment about import costs and supply constraints in the post-COVID operating environment, after controlling for changes in the operating environment that are common to all firms.
  Aggregate input costs Aggregate input costs Disaggregated input costs
Demand2007–2020 0.085*** 0.092***
Demand2021–2023 0.080*** 0.084***
(–0.048) (–0.009)
Input costs2007–2020 0.199***
Input costs2021–2023 0.162***
(–0.037)
Demand∆-ve 0.096***
Demand∆+ve 0.081***
(–0.015)
Input costs∆-ve 0.152***
Input costs∆+ve 0.193***
(+0.040*)
Import costs2007–2020 0.045
Import costs2021–2023 0.165**
(+0.120*)
Labour costs2007–2020 0.076*
Labour costs2021–2023 −0.003
(–0.079)
Supply shortages2007–2020 0.094
Supply shortages2021–2023 0.117**
(+0.023)
Transport costs2007–2020 0.144*
Transport costs2021–2023 0.138**
(–0.006)
Sample 5145 4599 5145
Within R2 0.174 0.183 0.122

Note: Standard errors are clustered at the industry level; ***, ** and * denote statistical significance at the 1, 5 and 10 per cent levels, respectively.

Sources: RBA; Reuters.

Endnotes

The authors are from Economic Group. They would like to thank Tim Taylor, Yad Haidari and Ewan Rankin for their input. [*]

See Beckers, Hambur and Williams (2023). [1]

For more details on the textual data and model used to analyse the transcripts, see Windsor and Zang (2023). [2]

The large language model we use is a zero-shot text classifier called bart-large-mnli (Lewis et al 2019), which is available on the open-source Hugging Face repository. Zero-shot learning is a machine learning approach where a model can classify text into unseen categories that were not present in the training data. [3]

These correlations are a little weaker if the period since 2021 is excluded. This period has been characterised by sizable movements in the indicators and the official statistics. [4]

For a more in-depth examination of these associations, see Windsor and Zang (2023). [5]

The methodology used here is similar to that used in Young et al (2021). [6]

For more details of the model used, see Windsor and Zang (2023). [7]

References

Beckers B, J Hambur and T Williams (2023), ‘Estimating the Relative Contributions of Supply and Demand Drivers to Inflation in Australia’, RBA Bulletin, June.

Dwyer J, K McLoughlin and A Walker (2022), ‘The Reserve Bank’s Liaison Program Turns 21’, RBA Bulletin, September.

Lewis M, Y Liu, N Goyal, M Ghazvininejad, A Mohamed, O Levy, V Stoyanov and L Zettlemoyer (2019), ‘BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension’, arXiv 1910.13461.

Park A, V Rayner and P D’Arcy (2010), ‘Price-setting Behaviour – Insights from Australian Firms’, RBA Bulletin, June.

Pitschner S (2020), ‘How do Firms Set Prices? Narrative Evidence from Corporate Filings’, European Economic Review, 124.

Windsor C and M Zang (2023), Firms’ Price-setting Behaviour: Insights from Earnings Calls, RBA Research Discussion Paper No 2023-06.

Young H, A Monken, F Haberkorn and E and Van Leemput (2021), ‘Effects of Supply Chain Bottlenecks on Prices using Textual Analysis’, FEDS Notes 20211203.