2 of 27

Paper and authors

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

Ian P. McCarthy

Tim Hannigan

Andre Spicer

3 of 27

Our aim

Understand how to use chatbots for content generation work while mitigating the epistemic risks (i.e., the process of producing knowledge) associated with botshit.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

4 of 27

Our approach

Explain how generative chatbots work and can produce untrue and nonsensical responses.
Recognize that chatbot responses can be thought of as provisional knowledge.
Explain the distinctions and relationship between:

Bullshit
Hallucinations
Botshit

Present a typology of chatbot work modes and related botshit risks.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

5 of 27

Stochastic parrots

The Large Language Model (LLM) technology underlying generative chatbots has the potential to hallucinate untruths.
This is because LLMs are designed to predict responses rather than know the meaning of these responses.
Like a parrot, LLMs excel at regurgitating learned content without comprehending context or significance.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

6 of 27

Thus...

There’s a concern that generative chatbot technology will reduce the cost it takes humans to bullshit to zero while not lowering the cost of producing truthful or accurate knowledge (Klein, 2023).

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

7 of 27

Chatbots unveiled:�knowing versus predicting

Reinforcement Learning from Human Feedback (RHLF) is a seven-step technique for teaching LLMs and other artificial intelligence (AI) systems (Ouyang et al. 2022).
LLMs rely on pattern analysis to predict suitable responses based on their training data but lack inherent knowledge systems to evaluate truthfulness.
This means LLMs can hallucinate untruths.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

8 of 27

Reinforcement Learning from Human Feedback (RLHF): The ChatGPT LLM process	Description	Risk of generating LLM hallucinations
1. Data collection	A large text data set is compiled to capture diverse topics, contexts, and linguistic styles.	If the data is biased, not current, incomplete, or inaccurate, the LLM and human users can learn and perpetuate its responses.
2. Data preprocessing	The data is cleaned to remove irrelevant text and correct errors and then converted for uniform encoding.	Preprocessing inadvertently removes meaningful content or adds errors that alter the context or meaning of some text.
3. Tokenization	The data is split into ‘tokens’, which can be as short as one character or as long as one word.	When language contexts are poorly understood, tokenization results in wrong or reduced meaning, interpretation errors, and false outputs.
4. Unsupervised learning to form a baseline model	The tokenized data trains the LLM transformer to make predictions. The LLM learns from the data’s inherent structure without supervision.	The LLM learns to predict content but does not understand its meaning, leading it to generate outputs that sound plausible but are incorrect or nonsensical.

9 of 27

Reinforcement Learning from Human Feedback (RLHF): The ChatGPT LLM process	Description	Risk of generating LLM hallucinations
5. Reinforcement Learning from Human Feedback: �i) supervised fine-tuning of model (SFT)	A team of human labelers curates a small set of demonstration data. They select a set of prompts and write down expected outputs for each (i.e., desired output behavior). This is used to fine-tune the model with supervised learning.	This process is very costly, and the amount of data used is small (about 12,000 data points). Prompts are sampled from user requests (from old models). This means the SFT only covers a relatively small set of possibilities.
6. Reinforcement Learning from Human Feedback: �ii) training a reward model (RW)	The human labelers repeatedly run these prompts against the SFT model and get multiple outputs per prompt. They rank the prompts for mimicking human preferences. This is used to train a reward model (RM).	Human labelers agree to a set of common guidelines they will follow. There is no accountability for this, which can skew the reward model.
7. Reinforcement Learning from Human Feedback: �iii) fine-tuning SFT model through proximal policy optimization (PPO)	A reinforcement learning process is continually run using the proximal policy optimization (PPO) algorithm on both the SFT and RM. The PPO uses a “value function” to calculate the difference between expected and current outputs.	If faced with a prompt about a fact not covered by the training data (SFT and RM), the LLM will likely generate an incorrect or made-up response.

10 of 27

Provisional knowledge

LLM-generated content is not always incorrect; it just lacks the basis of a truth claim.
Thus, LLM outputs are provisional knowledge (Hannigan et al., 2018) in that they have no utility or impact until the output is applied as part of an organizational routine or task
Until applied, chatbot generated contact lacks knowledge legitimacy (Deephouse et al., 2017) or accountability (Buhmann et al., 2019).

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

11 of 27

Bullshit, hallucinations and botshit

Bullshit: Human-generated content that has no regard for the truth, which a human then applies to communication and decision-making tasks (Frankfurt, 2009, McCarthy et al., 2020, Spicer, 2017).
Hallucination: When an LLM generates seemingly realistic responses that are untrue, nonsensical, or unfaithful to the provided source input.
Botshit: Chatbot-generated content that is not grounded in truth (i.e., hallucinations) and is then uncritically used by a human for communication and decision-making tasks (Hannigan et al. 2024).

12 of 27

	Bullshit	Botshit
Defined	Human-generated content that has no regard for the truth which a human then applies for communication and decision-making tasks (Frankfurt, 2009, McCarthy et al., 2020, Spicer, 2017). For example, a human produces a report using evidence that they have made up and is untrue, and the report is presented to others.	Chatbot generated-content that is not grounded in truth (i.e., hallucinations) and is then uncritically used by a human for communication and decision-making tasks. For example, a human produces a report using chatbot generated content that is untrue, and the report is presented to others.
Types	Pseudo-profound bullshit: statements that seem deep and meaningful (Pennycook et al. 2015) Persuasive bullshit: statements that aim to impress or persuade (Littrell et al. 2021a) Evasive bullshit: statements that strategically circumvent the truth (Littrell et al. 2021a) Social bullshit: statements that tease, exaggerate, joke, or troll (McCarthy et al., 2020; Spicer, 2017)	Intrinsic botshit: the human application of a chatbot response that contradicts the chatbot’s training data (Ji et al., 2023; Sun et al., 2023) Extrinsic botshit: the human application of a chatbot response that cannot be verified as true or false by the chatbot’s training data (Ji et al., 2023; Sun et al., 2023; Maynez et al., 2020)

13 of 27

	Bullshit	Botshit
Insights	Humans are more likely to generate and use bullshit: The more unintelligent, dishonest, and insincere they are (Littrell et al., 2021b). The expectations for them to have an opinion are high, and they expect to get away with it (Petrocelli, 2018). If their bosses frequently spout bullshit (Ferreira et al., 2022). Humans are more likely to believe and spread bullshit: If they have a low capacity for analytical thinking (Pennycook et al., 2015). If they think it is made by a scientist (Hoogeveen et al. 2022). If it is appealing, aligned with existing beliefs, and seems credible (McCarthy et al., 2020).	Chatbots are more likely to generate hallucinations for humans to use and transform into botshit when there are: Data collection, preprocessing and tokenization problems limit factual knowledge alignment between the training data and the desired response (Sun et al., 2023). Ambiguous prompts misdirect the chatbot (White et al., 2023). Problems with the training and modeling choices of the LLM transformer (Raunak et al., 2021). Issues with fine-tuning efforts (Ramponi, 2022) based on uncertainty around ground truth (Lebovitz et al., 2021).

14 of 27

A typology of chatbot work modes

To help reduce and avoid the epistemic risks of using chatbot-generated content for work tasks, users should consider two questions when using chatbots for different work:

How important is chatbot response veracity for the task?
And how easy is it to verify the veracity of the chatbot response?

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

15 of 27

A typology of chatbot work modes

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

16 of 27

Using chatbots with integrity

Users should be aware that each mode of chatbot work comes with a specific epistemic risk:

ignorance for augmented
miscalibration for authenticated
routinization for automated
black boxing for autonomous

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

17 of 27

Ignorance - automated

Ignorance is when chatbot users overlook or are unaware of the technology’s potentially useful and harmful outputs.
Users blindly rely on the technology in a limited way.
Users are relatively closed in their view of the value and hazards that come from this mode of chatbot work.
Organizations should work on ways to ensure that users are prompted to incorporate outputs produced by chatbots into their decision-making.
Chatbot content could be anonymized to avoid blindly rejecting it.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

18 of 27

Miscalibration - authenticated

Users systematically view chatbot responses as having more veracity and value for work than they do and do not authenticate enough.
Users excessively distrust the veracity and value of chatbot responses, resulting in high authentication and response rejection levels.
Users under or over utilize chatbot responses in contrast to other sources of information.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

19 of 27

Routinization-automated

Users lose over-sight of the work being automated and effectively ‘fall asleep at the wheel’.
Ensure users do not lose focus of the need to question trust in the chatbot.
Require chatbot work mode to be periodically accompanied by manual work and engagement.
Ensure users remain alert and engaged and effectively monitor the output of automated chatbot work.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

20 of 27

Black boxing - autonomous

Black boxing is the extent a user knows the internal workings of the chatbot.
Are the business model and algorithms behind the chatbot secret, opaque, inaccessible, and fixed?
When should users be required to learn how their chatbot works?
Black boxing of a chatbot could be beneficial as it inhibits users from gaming or sabotaging the technology for personal gain or perverse agendas.

Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. In Business Horizons.

21 of 27

Guardrails (i.e., rules, guidelines, or limitations for chatbot use) for how the technology, organizations, and users can mitigate botshit risks and enhance the truthiness of chatbot use for work.

22 of 27

Technology-oriented guardrails

As outlined in our account on how chatbots work, these guardrails focus on the technical aspects and capabilities of a chatbot and its LLM.
Ensure the mechanics and scope of an LLM are appropriate for the mode of chatbot work it is being used for.
Different modes require different fact-checking modules to verify the accuracy of information before responding.
Cross-reference routines that train the LLM data using trusted sources, flagging (or correcting false or misleading responses).

23 of 27

Organization-oriented guardrails

A code of conduct that specifies appropriate and acceptable use of chatbots to ensure veracity, integrity, and responsible use of chatbot-generated content.
Outline employee training on the capabilities and limitations of chatbots as per the four chatbot work modes.
Include prompt engineering training to formulate effective prompts for each mode of chatbot work.
Rules to promote transparency and disclosure to properly investigate, learn from, and prevent future botshit.

24 of 27

User-oriented guardrails

Guardrails to develop chatbot user abilities that mitigate the risks of botshit in the workplace.
Critical thinking and fact-checking suited to each of the four modes of chatbot work in our typology.
Chatbot users should have the courage and responsibility to speak up and question the veracity of the chatbot responses.

25 of 27

In sum

To master chatbot-generated provisional knowledge and mitigate the risks of possible botshit, consider our:

Accessible account of how generative chatbots work.
Insights about bullshit, hallucinations, and botshit
Framework of four modes of chatbot work and related advice to avoid blindly using chatbot predictions.

26 of 27

References

Buhmann, A., Paßmann, J., & Fieseler, C. (2019). Managing Algorithmic Accountability: Balancing Reputational Concerns, Engagement Strategies, and the Potential of Rational Discourse. Journal of Business Ethics. https://doi.org/10.1007/s10551-019-04226-4
Deephouse, D.L., Bundy, J., Tost, L.P., Suchman, M.C. (2017). Organizational legitimacy: six key questions. In: Greenwood, R., Oliver, C., Lawrence, T., Meyer, R. (Eds.), The SAGE Handbook of Organizational Institutionalism, 2nd ed. Sage publications, Thousand Oaks, CA
Ferreira, C., Hannah, D., McCarthy, I., Pitt, L. & Lord Ferguson, S., (2022). This place is full of it: Towards an organizational bullshit perception scale. Psychological Reports, 125(1), pp.448-463.
Frankfurt, H. G. (2005). On Bullshit. Princeton University Press.
Hannigan, T. R., Seidel, V. P., & Yakis Douglas, B. (2018). Product innovation rumors as forms of open innovation. Research Policy, 47(5), 953–964
Hannigan, T., McCarthy I.P. and Spicer, A. (2024) Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots. Forthcoming in Business Horizons.
Hoogeveen, S., Haaf, J.M., Bulbulia, J.A., Ross, R.M., McKay, R., Altay, S., Bendixen, T., Berniūnas, R., Cheshin, A., Gentili, C. and Georgescu, R. (2022). The Einstein effect provides global evidence for scientific source credibility effects and the influence of religiosity. Nature Human Behaviour, 6(4), pp.523-535.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), 248:1-248:38
Klein, E. (2023). The Ezra Klein Show. https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-klein-interviews-gary-marcus.html January 6, 2023
Lebovitz, S., Levina, N., & Lifshitz-Assaf, H. (2021). Is AI ground truth really true? The dangers of training and evaluating ai tools based on experts’ know-what. MIS Quarterly, 45(3), 1501–1525.
Littrell, S., Risko, E.F. and Fugelsang, J.A. (2021a). The bullshitting frequency scale: Development and psychometric properties. British Journal of Social Psychology, 60(1), pp.248-270

27 of 27

References

Littrell, S., Risko, E. F., & Fugelsang, J. A. (2021b). ‘You can’t bullshit a bullshitter’(or can you?): Bullshitting frequency predicts receptivity to various types of misleading information. British journal of social psychology 60(4), 1484-1505.
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661.
McCarthy, I. P., Hannah, D., Pitt, L. F., & McCarthy, J. M. (2020). Confronting indifference toward truth: Dealing with workplace bullshit. Business Horizons, 63(3), 253-263..
Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S., Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, & R. Lowe. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730-27744
Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2015). On the reception and detection of pseudo-profound bullshit. Judgment and Decision-making, 10(6), 549-563.
Petrocelli, J. V. (2018). Antecedents of bullshitting. Journal of Experimental Social Psychology, 76, 249-258.
Ramponi, M. (2022). ‘How ChatGPT actually works’, Retrieved from https://www.assemblyai.com/blog/how-chatgpt-actually-works/
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021). The curious case of hallucinations in neural machine translation. arXiv preprint arXiv:2104.06683.
Spicer, A. (2017). Business Bullshit. Routledge.
Spicer, A. (2020). Playing the bullshit game: How empty and misleading communication takes over organizations. Organization Theory, 1(2), p.2631787720929704.
Sun, W., Shi, Z., Gao, S., Ren, P., de Rijke, M., & Ren, Z. (2023). Contrastive learning reduces hallucination in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 11, pp. 13618-13626).
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (arXiv:2302.11382). arXiv. https://doi.org/10.48550/arXiv.2302.11382