Boost Your Skills

Why to Use it? - Relevance of the Subject addressed

The AIBotOrNot? Game addresses the subject of AI-Human interaction, and particularly Conversational AI, a field which has become very popular since OpenAI released ChatGPT on November 30^th, 2022, and which has received extensive attention particularly in the business community.

The relevance of this subject is confirmed by the very high attention that consulting companies, technology providers and academics have recently dedicated to it by: A 2023 Report from Gardner [1] provides an overview of the Hype Cycle surrounding AI these days, IBM [3] published 2024 a report reviewing the deployment of Chatbots with several use cases in entreprises, McKinsey [3] reported 2023 on different value creation dimensions, Accenture [4] presented their perspective on generative AI and its expected impact on work and business in general, focussing also on relevant adoption essentials, Microsoft’s [5] report on the future of work emphasis the émergence and importance of Large Language Models (LLMs), and to review and understand AI trends, the Stanford University’s [6] Index Report remains a valuable source of insights.

1. [Hype Cycle] Gartner Reveals Three Technologies That Will Transform Customer Service and Support By 2028, Gartner, August 30, 2023. (5 min)

2. [Chatbots] Conversational AI use cases for enterprises, IBM, February 23, 2024. (10 min)

3. [Value Creation] The economic potential of generative AI: The next productivity frontier, McKinsey Digital, June 14, 2023.

4. [Adoption Essentials] A new era of generative AI for everyone The technology underpinning ChatGPT will transform work and reinvent business, Accenture, 2023.

5. [LLMs] Microsoft New Future of Work Report 2023, Microsoft, 2023.

6. [AI Trends] The AI INDEX Report : Measuring Trends in AI, Stanford University, 2024.

In more general terms, our organizations and our lives are increasingly populated by AI Bots of different types (you can see a number of interesting videos on how chatbots developed over time), as their development has become much more affordable than in the past, and their value has been demonstrated in many different areas. Interacting with AI Bots will therefore become very normal, as they will be increasingly embedded in websites, platforms, apps, social networks as well as in different types of robots - from flexible and conversational industrial robots, to tools we use in our daily life, to pseudo-pets like Boston Dynamics Spot [7], or pseudo-humans becoming apparently so attractive that many consider making them become their friend, partner or even marry them [8, 9, 10].

1. [Pseudo-Pets] Spot, The agile Mobile Robot, Boston Dynamics, 2024.

2. [Research] Why Not Marry a Robot, International Conference on Love and Sex with Robots, 2017.

3. [Marrying Robots] Getting married to robots « will be considered normal by the end of the century » , Sun, 2021.

4. [Marrying Robots] Chinese Man « marries » robot he built himself, The Guardian, 2021.

At the core of these analyses remains the challenge to understand how people interact with AI technologies in order to improve their AI literacy and adopt AI to support personal and organizational tasks. AI-Literacy has thus emerged as an important domain which allows to get a deeper understanding of the degree to which people understand AI applications and develop the confidence to use them effectively. AI-Human interaction principles and practices are essential for both increasing AI Literacy and supporting AI Adoption.

As many claim that AI will significantly change our private and professional lives, the subject of AI-Human interaction is worth focussing on, even if just for a few minutes and in a playful way, as you can do with the AIBotOrNot? Game using it either as an icebreaker game, or as a way to support further reflections and learning, particularly with a group.

How to Debrief the Game Experience

As mentioned in the "Playing Scenarios" section, this Game has been designed to provide a basis to:

1. gain several valuable Insights

2. understand key Principles

3. develop and enhance practical Capabilities

related to AI-Human interaction and the opportunities and challenges of Conversational AI.

In this section you will find material related to 3 Building Blocks supporting the game debriefing process.

The first 3 sections focus on Insights, Principles and Capabilities, and a fourth section provides details on one specific Capability – Attention to Detail – which is particularly relevant to identify if a statement is human or AI-generated.

In each section the insight, principle, and capabilities presented are illustrated with concrete examples of statements, aimed at helping Participants to improve their understanding of both the theory and the practice of AI-Human interaction.

Key Insights

The first step in the debriefing consists in reflecting on (and ideally discuss together) the rapidly evolving capabilities of AI Bots, focussing first on technology-related insights, such as the three listed below. Within a group, this can be done by engaging the Participants/Teams in a discussion about how impressively far AI has already come when it comes to conversational capabilities (1), about advances in the underlying NLP technology (2), and about the limitations that still need to be addressed (3). The AI Bot sentences provided below (in red) can be injected in the discussion as concrete examples illustrating each insight.

1. Understanding AI's Linguistic Abilities:

Recognize how advanced AI can mimic human language and produce coherent, contextually appropriate statements.
Example: An AI-generated statement like "The sunset over the ocean was breathtaking," shows how AI can create vivid imagery using descriptive language.

2. Advances in Natural Language Processing (NLP):

Gain insight into the progress of NLP technologies and their ability to handle complex language tasks.
Example: An AI-generated response to "Tell me a joke" might be "Why did the scarecrow win an award? Because he was outstanding in his field!" showcasing AI's ability to understand and create humor.

3. Limitations of AI:

Identify areas where AI might fall short, such as producing content that involves personal experiences or emotional nuances.
Example: AI might generate a statement such as "I remember the smell of my grandmother's cookies," which lacks authenticity because AI cannot have personal memories.

The second step consists in recognizing that beyond the technology there is a need to better understand human language and behavior appreciating the nuances of human communication (4) and starting to identify some of the limits of current linguistic patterns (5) AI Bots when it comes to mimic humans. Also here the AI Bot sentences provided below (in red) can be used to stretch the reflection and fuel the discussion.

4. Nuances of Human Communication:

Appreciate the subtle differences in how humans express thoughts, emotions, and experiences compared to AI.
Example: A human might say, "I felt a rush of nostalgia hearing that song," capturing complex emotions that AI struggles to express naturally.

5. Patterns in AI Responses:

Notice patterns or commonalities in AI-generated text, such as repetitive structures or a tendency to stick to facts and avoid personal anecdotes.
Example: AI might frequently use structured sentences like "The capital of France is Paris," sticking as much as possible to factual, straightforward information without personal flair.

Key Principles

After reviewing Key Insights, the debriefing can focus on discussing a number of Principles which are relevant in the area of AI-Human Interaction starting with the Turing Test as a way to study the distinguishability of AI and human communication and the fact that depending on the context it might not be always necessary/desirable to perfectly mimic human speech and conversation.

Principle of Turing Test: Learn about the Turing Test and its relevance in evaluating the indistinguishability of AI and human communication.

Example: Discussing how a statement like "I enjoy reading science fiction novels in my spare time" could pass as human in a Turing Test challenging students to think about AI's indistinguishability.

Human-Centered Design: Recognize the importance of designing AI systems that complement and enhance human capabilities rather than simply mimicking them.

Example: Considering how a conversational AI assistant designed to help with customer service can improve by adding empathetic responses like "I'm really sorry to hear that you had a bad experience."

The reflection can be extended through a comparison between the different ways in which AI Bots and Humans produce and interpret language discussing for instance how data-driven AI Bots can be as biased as humans, and how humans have a natural tendency to operate with contextual cues.

Contextual Understanding: Appreciate the importance of context in communication and how AI models can sometimes struggle to grasp nuanced contextual cues.

Example: Observing how AI might misunderstand context in statements like "He put the turkey in the oven and it was stuffed." where "it" could ambiguously refer to either the turkey or the oven.

AI Bots Data and Bias: Gain awareness of how AI models are trained on large datasets and the potential biases that can emerge from these datasets.

Example: Recognizing bias in AI-generated statements like "Doctors are always men" highlighting the importance of diverse training data to avoid reinforcing stereotypes.

Key Capabilities

The debriefing can then be used to make explicit and reflect on the multiple capabilities required for distinguishing between AI Bot and Human statements as well as their relevance for becoming better AI “players” (and AI-informed Citizens):

Analytical Thinking: Enhance the ability to analyze statements critically and identify subtle clues indicating AI or human origin.

Example: Scrutinizing a statement like "I went to the store to buy groceries" for signs of AI such as a lack of personal touch or specificity.

Collaboration & Persuasion: Improve teamwork and persuasive communication as players discuss and debate their choices.

Example: In a team setting players might debate whether a statement like "My favorite hobby is painting" is AI-generated based on its generic nature versus a human's personal touch.

The reflection and debriefing could be concluded by discussing th etype of details that need to be given particular attention when analysing correctly AI-generated or Human speech:

Attention to Detail: Sharpen your attention to linguistic details, spotting subtle hints that indicate whether a statement is human or AI-generated.

Example: Noticing that AI-generated text often lacks contractions, such as "I cannot wait for the weekend," versus the more natural human version, "I can't wait for the weekend."

As this capability is very important, more details on this subject are provided in the next section.

Attention to Details - Where AI Bots and Human tend to differ

To conclude the debriefing, we suggest to go one step further in helping Participants to develop the capabilities by discussing with them of a number of important “Symptoms”. 10 of them are listed below, together with several examples to help illustrating how paying attention to small details can aid in distinguishing between AI-generated and human-written statements, providing a deeper understanding of the nuances of natural language.

1. Contractions and Informality:

AI: "I do not like the taste of coffee."
Human: "I don't like the taste of coffee."
Detail Noticed: Humans often use contractions in casual conversation, whereas AI might use more formal language.

2. Specificity in Descriptions:

AI: "The movie was good and entertaining."
Human: "The movie was gripping, with an amazing plot twist in the second half."
Detail Noticed: Humans tend to provide more specific and nuanced descriptions.

3. Use of Idioms and Colloquialisms:

AI: "I find it difficult to wake up early in the morning."
Human: "I'm not a morning person; waking up early is a real struggle for me."
Detail Noticed: Humans often use idiomatic expressions that AI might not use correctly.

4. Expressions of Uncertainty or Emotions:

AI: "I believe it is going to rain tomorrow."
Human: "I have a feeling it might rain tomorrow, but I'm not entirely sure."
Detail Noticed: Humans often express uncertainty or emotions more naturally.

5. Contextual Adjustments:

AI: "The weather is good."
Human: "The weather is perfect for a beach day."
Detail Noticed: Humans adjust their statements based on context, adding relevant details.

6. Personal Experiences and Anecdotes:

AI: "Traveling can be enjoyable and educational."
Human: "Last summer, I traveled to Japan and learned so much about their culture."
Detail Noticed: Humans include personal experiences and anecdotes to make statements more engaging.

7. Subtle Humor:

AI: "This is a funny situation."
Human: "This is like one of those days where you spill coffee on your shirt right before a big meeting."
Detail Noticed: Humans often use situational humor and specific examples that AI might not generate.

8. Emotional Nuances:

AI: "I am happy with the results."
Human: "I'm absolutely thrilled with how everything turned out!"
Detail Noticed: Humans use a wider range of emotional expressions and intensifiers.

9. Varied Sentence Structures:

AI: "I went to the store. I bought some apples. I went home."
Human: "I went to the store, picked up some fresh apples, and then headed home."
Detail Noticed: Humans often use varied and complex sentence structures to make speech more natural.

10. Cultural References:

AI: "This is a popular song."
Human: "This song has been stuck in my head ever since I heard it on the radio last week."
Detail Noticed: Humans frequently include cultural references and context that AI might not.

Authors Note

I designed and documented this Game with the editorial and software development support of GPT4o. The Game is supposed to be played with a group, but it is inspired from single-player online social Turing games in which players chat for a short period (typically 2 minutes) and then guess whether their conversational partner is a human or an AI bot. If your students are interested in such online games, you should advise them to try (HumanOrNot) or (Human or Not? // A Social Turing Game) to fostering their critical thinking and analytical skills in identifying AI-generated text. You can see below 2 examples of interaction with these online games.

The idea behind this Game has been also inspired by Hanschmann, Gnewuch and Maedche’s article BotOrNot: A Platform for Conducting Experiments with Undisclosed Chat Agents, MuC '22: Proceedings of Mensch und Computer 2022, pp 618–621.

Future versions of this Game :

I tend to conclude a game session asking the Participants to imagine how this Game might evolve over time. I particularly appreciate the vision that in a not-so-distant future the Game (or something similar) might be used at the beginning of every meeting or group session, mainly to identify who in the room - including the instructor - is a real Human or an AI Bot 😊