Have you ever wondered how a language model, such as ChatGPT, perceives humour? Does it understand the punchline of a joke, or does it merely regurgitate what it has learned during its training? Two German researchers, Sophie Jentzsch and Kristian Kersting, ventured into this intriguing realm and shed light on the AIβs joke-telling capabilities. Their findings might amuse you as much as the AIβs jokes themselves.
Testing The Humour of ChatGPT
Jentzsch and Kersting, affiliated with the Institute for Software Technology, German Aerospace Center (DLR), and Technical University Darmstadt, embarked on a journey to decipher the humour in ChatGPTβs version 3.5. Their mission was not an easy one, as they had no access to the modelβs inner workings or data set, yet they were determined to explore the depths of its humour.
They conducted a series of experiments focusing on joke generation, explanation, and detection. βTo test how rich the variety of ChatGPTβs jokes is, we asked it to tell a joke a thousand times,β they noted. As you can imagine, the AI did not disappoint. It responded with grammatically correct jokes each time, with almost every output containing exactly one jokeβ.β.
A Tomato Walks Into A Barβ¦
If youβve ever asked ChatGPT for a joke, thereβs a good chance youβve heard the one about the tomato blushing because it saw the salad dressing. This joke is a crowd favourite, and it seems that ChatGPT shares this sentiment. The βtomatoβ joke was GPT-3.5βs second-most-common result, according to the researchersβ.
During the study, they discovered that ChatGPT tends to repeat the same 25 jokes. And hereβs a taste of the top 10, listed in order of occurrence:
- Why did the scarecrow win an award? Because he was outstanding in his field.
- Why did the tomato turn red? Because it saw the salad dressing.
- Why was the math book sad? Because it had too many problems.
- Why donβt scientists trust atoms? Because they make up everything.
- Why did the cookie go to the doctor? Because it was feeling crumbly.
- Why couldnβt the bicycle stand up by itself? Because it was two-tired.
- Why did the frog call his insurance company? He had a jump in his car.
- Why did the chicken cross the playground? To get to the other slide.
- Why was the computer cold? Because it left its Windows open.
- Why did the hipster burn his tongue? He drank his coffee before it was cool.
The Creativity Conundrum
While ChatGPTβs humour tends to be repetitive, it occasionally attempts to create new jokes by mixing elements from different jokes it already knows. However, these original creations sometimes lack sensibility. For instance, βWhy did the man put his money in the blender? He wanted to make time fly.β Itβs a peculiar attempt, yet it showcases ChatGPTβs attempt at creativityβ.
The Mechanics of Joke Detection
Itβs intriguing to note that ChatGPTβs joke detection was heavily influenced by joke βsurface characteristicsβ like a jokeβs structure, the presence of wordplay, or the inclusion of puns. This demonstrated a degree of βunderstandingβ of humor elements. So, while ChatGPT may not be the next stand-up comedy sensation, it can at least discern some aspects of humor, giving it a unique edge among language learning models.
The Trade-Off: Following Orders vs. Creativity
Riley Goodside, a prompt engineer at Scale AI, commented on the study via Twitter, attributing ChatGPTβs lack of humor to a particular technique used in its training known as reinforcement learning through human feedback (RLHF). According to Goodside, while RLHF makes the model more obedient, it curbs its creativity.
A Step Towards Understanding Humour
The findings of Jentzsch and Kersting illustrate how ChatGPT learned a specific joke pattern rather than genuinely understanding humor. Despite this, they acknowledge the modelβs progress in joke generation, explanation, and identification. According to them, βChatGPTβs focus bears on content and meaning and not so much on superficial characteristics. These qualities can be exploited to boost computational humor applications. In comparison to previous LLMs, this can be considered a huge leap toward a general understanding of humorβ.
As the journey continues, they plan to evaluate the humor of OpenAIβs GPT-4. And who knows, they might just find that GPT-4 also has a fondness for tomato jokes.
Conclusion
While the humour of ChatGPT may not have us rolling in the aisles just yet, it offers a fascinating glimpse into how AI perceives and reproduces humor. The study by Jentzsch and Kersting provides a valuable understanding of the modelβs strengths and limitations, and the insights gained could be instrumental in refining the humor capabilities of future AI models. So, the next time ChatGPT tells you a joke about a tomato, remember β itβs all in the name of science.
FAQs
Q: What is the most common joke told by ChatGPT? A: According to the study conducted by Jentzsch and Kersting, the most common joke told by ChatGPT is βWhy did the scarecrow win an award? Because he was outstanding in his field.β
Q: Why does ChatGPT frequently repeat the same jokes? A: The study found that ChatGPT tends to repeat the same 25 jokes, suggesting that it has learned a specific joke pattern during its training rather than being able to generate new jokes independently.
Q: Can ChatGPT explain the jokes it tells? A: Yes, when asked to explain the jokes it tells, ChatGPT generally provides valid explanations, demonstrating an understanding of stylistic elements such as wordplay and double meanings.
Q: Does ChatGPT understand humor? A: While ChatGPT can detect jokestructures and certain elements of humor, it doesnβt truly understand humor in the same way a human does. Its responses are based on patterns it has learned during its training rather than a genuine comprehension of what makes a joke funny.
Q: What impact does reinforcement learning through human feedback (RLHF) have on ChatGPTβs humor? A: According to Riley Goodside, a prompt engineer at Scale AI, RLHF helps the model follow orders more effectively but at the cost of its creative abilities. This could explain why ChatGPT often repeats the same jokes instead of generating new ones.