Reading through the Lines: Price-setting Indicators from Earnings Calls | Bulletin – September 2023

Bulletin – September 2023 Finance Reading through the Lines: Price-setting Indicators from Earnings Calls

21 September 2023

Tomas Cokis, Callan Windsor and Max Zang ^[*]

Download 847KB

machine learning, modelling, technology, business, finance

Abstract

This article explores how information in earnings call transcripts from Australian firms can contribute to the Reserve Bank’s understanding of their price-setting behaviour, as a complement to information gathered from the Bank’s liaison program. A large language model is used to process and analyse earnings call transcripts and construct new sentiment indicators for input costs, demand, prices and supply shortages from them. These indicators, starting in 2007 and updated to capture the latest August earnings season, provide useful information about economic conditions and price-setting behaviour, including about developments during the recent period of unusually high inflation.

Introduction

Consumer price inflation in Australia, as in other countries, has been unusually high in recent years. While demand conditions have played an important role, supply-side factors have been the biggest driver of the increase in inflation and have been front of mind for company executives and policymakers (Graph 1).^[1] These pressures have moderated recently, and inflation has passed its peak; however, it is forecast to remain high for some time yet.

Graph 1

The impact of these upstream cost increases on consumer prices is not well captured by inflation models, and so other alternative and timely sources of information are important in assessing the inflation outlook. For this reason, the Reserve Bank has been monitoring firms’ price-setting behaviour using insights from the Bank’s liaison program. This article describes an approach involving use of a large language model to process and analyse earnings call transcripts and construct new sentiment indicators based on what firms have said about input costs, demand conditions, final prices and supply shortages. These indicators complement other statistical information, including from the business liaison program. They also allow us to draw inferences about the determinants of firms’ pricing behaviour.

Analysing earnings calls using a large language model

Earnings call transcripts are a rich source of information about firms’ own business conditions and economic and financial conditions more broadly. Earnings call transcripts also offer qualitative information that is not captured by traditional financial statements or other quantitative data. The sentiment and language used by executives during these calls provides additional context, nuance and insights that is valuable for economists trying to understand how firms make choices and how those choices affect prices.

Earnings calls typically take place a few hours after the release of earnings results. Most large listed Australian firms hold earnings calls during the February and August ‘earnings season’, covering results over the period to December and June, respectively, although there are several firms that release their results on a different schedule. During the calls, company executives deliver prepared remarks summarising the overall business position of the company and the operating environment. This is followed by a question-and-answer session where all interested parties – including institutional and individual investors and expert analysts – can ask questions about the outlook or probe into other issues.

For our research, we process and analyse around 5,500 earnings call transcripts, involving over 750,000 paragraphs of text from 2007 to the most recent August earnings season.^[2] The challenge in working with such a rich set of textual data is to systematically examine the transcripts in a way that incorporates the nuance and context of the discussions that took place. Techniques in natural language processing – a field that uses computers to process and analyse large amounts of text – have now advanced to the point where such discussions can be processed and analysed in a meaningful way. This allows us to construct informative quantitative metrics, which we can track over time to examine factors affecting firms’ price-setting behaviour.

To examine the text of the earnings call transcripts we use a large language model.^[3] Broadly speaking, a large language model is an artificial intelligence algorithm that uses deep learning and was trained on massive amounts of textual data. For each paragraph in a transcript, the model assigns a probability over several pre-defined topic labels independently. To map from a probabilistic measure to an indicator variable, we assign a topic label to a paragraph by giving it a score of +1 if the probability for that label is greater than a pre-defined threshold. In the stylised example shown in Figure 1, the model classifies the paragraph as being about ‘hiring difficulties’ and ‘labour costs increasing’. To construct the ‘labour cost indicator’, we sum the number of paragraphs in the transcript labelled ‘labour costs increasing’, subtract the sum of those labelled ‘labour costs decreasing’ and divide the balance by the total number of paragraphs in the transcript. These scores are then aggregated across all transcripts in each period. In practice, we construct new indicators for a variety of input costs (including labour costs) as well as final prices, demand and references to supply shortages.

Figure 1: A Stylised Example Classification from the Large Language Model

New indicators complement existing soft information

Graph 2 overlays the new aggregated earnings call indicators for total input costs, demand, final prices and labour costs with similar indicators obtained from the following two sources:

The Reserve Bank’s liaison program – this is a formal program of economic intelligence gathering established over 20 years ago, through which Bank staff meet frequently with firms from a pool of around 900 active contacts (Dwyer, McLoughlin and Walker 2022). Details of these discussions are systematically recorded in confidential ‘diary notes’. We use the text of these notes to construct indices for input costs, demand, final prices and labour costs using a similar approach as that applied to firms’ earnings calls.
A monthly survey of around 400 firms from the National Australia Bank (NAB) – this is a survey designed to produce statistical indices related to business conditions. We compare our text-based indices to the NAB survey-based indices for purchase costs, forward orders, selling prices and labour costs.

Graph 2

Each source paints a similar picture about recent developments in costs, prices and demand. This is useful for economic analysis, providing confidence in the signal extracted from the liaison program. This is especially true given each source of information has its own strengths and weaknesses.

Earnings calls provide consistent firm-level information from many firms over time and allow for different indices to be constructed and monitored. However, earnings calls are limited to larger listed companies and most of the information is only updated during the earnings season. Information from the Reserve Bank’s business liaisons are timelier, but the composition of firms changes from period to period and responses are influenced by the topics that are covered. Finally, business survey indicators provide consistent information over time, but sample sizes tend to be smaller, firm-level information is not readily available and analysis is limited to a small number of pre-existing indicators.

The more disaggregated measures of input cost indices from earnings calls also align with other relevant indicators. For instance, transportation bottlenecks, as well as other supply disruptions, resulted in the recent sharp rise in goods price inflation. The COVID-19 pandemic limited the capacity of domestic and global firms to produce and deliver products and Russia’s invasion of Ukraine led to sharp increases in the prices of energy and other commodities. Broadly in line with other indicators of shipping costs and delivery times, the earnings call indices confirm that supply disruptions and increased transport costs were prominent issues from 2021 onwards (Graph 3). The more recent decline in these earnings call measures has been less pronounced, which could reflect coverage of past events in earnings calls, that earnings calls are capturing a broader sense of supply constraints than just those two factors, or that supply disruptions can impact firms with a long lag.

Graph 3

New indicators track official statistical measures

Our new aggregated indicators for input costs, labour costs and prices obtained after the February and August earnings seasons can be compared to official statistics for the growth in producer prices (PPI), compensation of employees (COE) and consumer prices (CPI) over the six months to March and September, respectively. Graph 4 shows the results of such a comparison. The correlations show that over the past 15 years or so, the highest associations between our new indicators and their related statistical counterparts occurs when the series are compared to each other in the same period, with peak correlations of between 0.75 and 0.5.^[4] The exception is the earnings call indicator for prices, with the peak correlation occurring when the new indicator leads consumer price inflation by six months. This exercise suggests that information derived from earnings calls obtained immediately after the February and August earnings seasons correspond well to official statistics.^[5]

Graph 4

Developments in the new earnings call indicators can also be compared to developments in the official statistics over the recent period of high inflation. The broader earnings call index for input costs seems better at capturing the role of global factors in recent inflation outcomes, relative to the more disaggregated index for import costs (Graph 5). In particular, the input cost index picked up the recent pressure on firms costs from global goods price inflation a little sooner, and more clearly than the import cost index. The later and smaller increase in the import costs index could reflect a tendency for discussions in earnings calls to focus on specific causes (e.g. the invasion of Ukraine, or the shortage of semiconductors), or perhaps even specific prices, rather than identifying and distinguishing the domestic versus imported origin of price pressures. It could also reflect that higher global goods prices impact Australian inflation through an increase in the prices of domestically produced materials, because of integration between the domestic and foreign markets. For instance, increases in global timber prices contributed to higher prices for Australian timber, which in turn contributed to strong inflation in new dwelling construction costs.

Graph 5

The earnings call indicator for labour costs is consistent with official data showing a substantial increase in labour costs over the past year. Graph 6 shows the sharp rise in average earnings per worker, and the substantial increase in market services inflation, which includes parts of the CPI where labour is the most important cost for businesses.

Graph 6

More broadly, supply factors and strong demand have contributed to the pick-up in CPI inflation since 2021, according to several methods for assessing their relative contribution (Beckers, Hambur and Williams 2023). The new earnings call indices support this evidence, confirming that a pick-up in demand contributed to stronger inflation outcomes from 2021, alongside disruptions to supply (Graph 7). More recently, the earnings call demand index has declined by more than some other indicators.

Graph 7

Firm-level insights into price-setting behaviour

Because the new earnings call indicators are available at the firm level, we can use them to examine correlations between the sentiment of firms’ discussions about final prices and the sentiment of their discussions about input costs and demand. This allows for inferences to be drawn about the determinants of firms’ pricing behaviour that could be relevant for understanding the dynamics of the inflation process. To estimate these conditional correlations, we use the regression analysis outlined in Appendix A, with our full regression results provided there.

Three findings from this empirical exercise stand out:

Final price sentiment has a stronger association with sentiment about input costs compared to sentiment about demand, after controlling for changes in the operating environment that are common to all firms, including the effect of global supply shocks. This is consistent with survey-based findings that firms’ predominant pricing strategy is to set prices as a mark-up over costs (Park, Rayner and D’Arcy 2010).
Discussions on final prices appear to have become more sensitive to (or at least more correlated with) sentiment about import costs in the post-COVID operating environment, after controlling for changes in the operating environment that are common to all firms. This could indicate that firms have become more attuned to changes in import costs since 2021.
Final price setting sentiment has a stronger association with input costs when the change in sentiment is positive compared to when the change in sentiment is negative. This suggests that rising prices are likely to remain front of mind for company executives even as supply pressures moderate. This asymmetry is consistent with the type of firm-level behaviour reported for US firms in Pitschner (2020) from an analysis of the text of corporate filings.

Finally, we can run a similar regression to that in Appendix A but allow the correlation between demand and input costs with final prices to vary by industry. In this regression we control for changes in the operating environment that are common to all firms in an industry. We find the association between price-setting sentiment and input cost/demand sentiment differs significantly across industries, suggesting there is significant heterogeneity in firms’ price-setting behaviour.

These experimental findings appear to indicate that aggregate price-setting behaviour could depend on the source of the shocks firms face (demand- or cost-driven), the direction of the shock (with firms’ reacting more to cost increases relative to decreases), and which industries are most affected. Taken together, this underscores the importance of continuing to develop rich multisector models of the economy to better understand firms’ reactions to different types of shocks.

Conclusion

This article introduces new sentiment indicators based on earnings call transcripts from Australian firms that are processed and analysed using a large language model. The model can classify text taking account of the subtleties and nuance of natural language. The signal from these indicators about input costs, demand, final prices and supply shortages tracks current economic conditions well. Regression analysis, uncovering conditional correlations between the sentiment of final price discussions and the sentiment of discussions about input costs and demand, allows for inferences to be drawn regarding firms’ price-setting behaviour. The results are consistent with firms using pricing strategies that focus on a mark-up over costs. They are also consistent with firms being more reactive to rising, rather than falling, input costs.

Going forward, the Reserve Bank will use these new sentiment indicators, together with other similar indicators, to monitor developments in current economic and financial conditions. Over time, these indicators will be developed and refined as the capabilities of large language models are further advanced.

Appendix A: Regression analysis

The following panel regressions are run to estimate the association between the sentiment of firms’ discussions about final prices and the sentiment of their discussions about input costs and demand using data from 2007:

p_{it} = α_{i} + θ_{t} + γ_{1} {dem}_{it}^{07-20} + γ_{2} {dem}_{it}^{21-23} + ∑_{n=1}^{N} β_{1, n} {ic}_{n,it}^{07-20} + ∑_{n=1}^{N} β_{2, n} {ic}_{n,it}^{21-23} + e_{it} .

Here $p_{it}$ denotes the final price sentiment index for firm $i$ in time period $t$ , measured at a quarterly frequency.^[6] The term ${dem}_{it}^{07-20}$ is the demand sentiment index from firms’ earnings calls from 2007 to 2020, while ${dem}_{it}^{21-23}$ is the index from 2021. Likewise, ${ic}_{n,it}$ is the input cost index for input cost sentiment index $n ∈ N$ . Including separate coefficients for each time period allows us to examine if there is anything different about the period since 2021 – which has been characterised by sizable supply shocks – relative to the historical sample.

All regressions include firm, $α_{i}$ , and time, $θ_{t}$ , fixed effects. The firm fixed effects allow us to control for unobservable differences in the language used by each firm during their earnings calls. The time fixed effects control for changes in the operating environment that are common to all firms.^[7]

In the second regression below, instead of separate coefficients for each time period, we include separate coefficients for demand and input costs according to whether the change in sentiment was positive or negative. This allows us to examine whether price-setting sentiment changes asymmetrically in response to positive or negative changes in input costs or demand sentiment.

p_{it} = α_{i} + θ_{t} + γ_{1} {dem}_{it}^{Δ+ve} + γ_{2} {dem}_{it}^{Δ-ve} + β_{1} {ic}_{it}^{Δ+ve} + β_{2} {ic}_{it}^{Δ-ve} + e_{it} .

The full regression results are set out in Table A.1.

Table A.1: Associations with Final Price Sentiment
Differences in related coefficient estimates in parentheses, 2007–2023 (September quarter)

Table A.1 shows coefficient estimates and their significance from panel regressions. These regressions are run to estimate the association between the sentiment of firms’ discussions about final prices in their earnings calls and the sentiment of their discussions about input costs and demand, using data from 2007. All regressions include firm and time fixed effects. The coefficients in the first column indicate that final price sentiment has a stronger association with sentiment about input costs compared to sentiment about demand, after controlling for changes in the operating environment that are common to all firms, including the effect of global supply shocks. The coefficients in the second column indicate that final price-setting sentiment has a stronger association with input costs when the change in sentiment is positive compared to when the change in sentiment is negative. The coefficients in the third column indicate that discussions on final prices appear to have become more sensitive to (or at least more correlated with) sentiment about import costs and supply constraints in the post-COVID operating environment, after controlling for changes in the operating environment that are common to all firms.
	Aggregate input costs	Aggregate input costs	Disaggregated input costs
Demand^2007–2020	0.085***		0.092***
Demand^2021–2023	0.080***		0.084***
	(–0.048)		(–0.009)
Input costs^2007–2020	0.199***
Input costs^2021–2023	0.162***
	(–0.037)
Demand^∆-ve		0.096***
Demand^∆+ve		0.081***
		(–0.015)
Input costs^∆-ve		0.152***
Input costs^∆+ve		0.193***
		(+0.040*)
Import costs^2007–2020			0.045
Import costs^2021–2023			0.165**
			(+0.120*)
Labour costs^2007–2020			0.076*
Labour costs^2021–2023			−0.003
			(–0.079)
Supply shortages^2007–2020			0.094
Supply shortages^2021–2023			0.117**
			(+0.023)
Transport costs^2007–2020			0.144*
Transport costs^2021–2023			0.138**
			(–0.006)
Sample	5145	4599	5145
Within R²	0.174	0.183	0.122
Note: Standard errors are clustered at the industry level; *, and * denote statistical significance at the 1, 5 and 10 per cent levels, respectively. Sources: RBA; Reuters.

Endnotes

The authors are from Economic Group. They would like to thank Tim Taylor, Yad Haidari and Ewan Rankin for their input. [*]

See Beckers, Hambur and Williams (2023). [1]

For more details on the textual data and model used to analyse the transcripts, see Windsor and Zang (2023). [2]

The large language model we use is a zero-shot text classifier called bart-large-mnli (Lewis et al 2019), which is available on the open-source Hugging Face repository. Zero-shot learning is a machine learning approach where a model can classify text into unseen categories that were not present in the training data. [3]

These correlations are a little weaker if the period since 2021 is excluded. This period has been characterised by sizable movements in the indicators and the official statistics. [4]

For a more in-depth examination of these associations, see Windsor and Zang (2023). [5]

The methodology used here is similar to that used in Young et al (2021). [6]

For more details of the model used, see Windsor and Zang (2023). [7]

References

Beckers B, J Hambur and T Williams (2023), ‘Estimating the Relative Contributions of Supply and Demand Drivers to Inflation in Australia’, RBA Bulletin, June.

Dwyer J, K McLoughlin and A Walker (2022), ‘The Reserve Bank’s Liaison Program Turns 21’, RBA Bulletin, September.

Lewis M, Y Liu, N Goyal, M Ghazvininejad, A Mohamed, O Levy, V Stoyanov and L Zettlemoyer (2019), ‘BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension’, arXiv 1910.13461.

Park A, V Rayner and P D’Arcy (2010), ‘Price-setting Behaviour – Insights from Australian Firms’, RBA Bulletin, June.

Pitschner S (2020), ‘How do Firms Set Prices? Narrative Evidence from Corporate Filings’, European Economic Review, 124.

Windsor C and M Zang (2023), Firms’ Price-setting Behaviour: Insights from Earnings Calls, RBA Research Discussion Paper No 2023-06.

Young H, A Monken, F Haberkorn and E and Van Leemput (2021), ‘Effects of Supply Chain Bottlenecks on Prices using Textual Analysis’, FEDS Notes 20211203.