RDP 2025-06: An AI-powered Tool for Central Bank Business Liaisons: Quantitative Indicators and On-demand Insights from Firms Appendix B: Spot-checking the Precision of the LM-based Topic Classifications

In this appendix, we perform a back-of-the-envelope assessment of the accuracy of all 14 topics classified by the transformer-based LM. To do this, we randomly sample 10 paragraphs from each topic, each with an LM-based topic probability of greater than 90 per cent, for a total of 140 paragraphs. We then assess the accuracy of these LM-based classifications allowing us to do a spot check of the LM's precision for each topic. The results are presented in Table B1 and show that topics that are more broadly defined (e.g. ‘costs’) or open to interpretation (e.g. ‘financing conditions’) – and so requiring more context and nuance to properly classify – are less precisely identified by the model.