How to control your little bullshit-generator?

Firstly, I really value the technology behind ChatGPT and use it with incredible efficiency in my daily work. However, I'm also aware and regularly analyze how it can hallucinate, invent plausible-sounding stuff, and weave it confidently into its responses. In other words, it can bullshit quite well.

Right now, I'm learning about processing digital images with artificial intelligence techniques. You know, like in NCIS when they enhance a blurry image from a street camera to read what the suspect in the car wrote on a little note. Well, it's not exactly that, but it's interesting and important...

I had a very useful discussion with ChatGPT about this topic. Then, I specifically wanted to get its opinion on a piece of code from the GitHub repository. Instead of pasting the code, I provided the link to the page. The response seemed accurate, but suspicious. When I looked more closely at the code, I noticed that it referred to something that wasn't even there, making the whole response completely irrelevant and misleading. I encountered similar issues when discussing politics or other societal topics, and tried to update the discussion with a linked article. The response seemed relevant, but upon closer examination, it turned out it wasn't reflecting on the actual content.

Last time, I didn’t let it slide, and I pressed the little bullshitter:

"What are you talking about??? There's no such part in the code that you mentioned! You probably can't self-reflect in this sense, but why did you refer to things that weren't in the code I linked?"

To its credit, it gave a clear and enlightening response:

"Your question is completely valid, and thank you for highlighting this! Such errors typically arise from summarizing based on general patterns I know about similar codes, rather than focusing solely on the file and context you provided. This was misleading and did not accurately reflect reality."

TIP 1

Based on this, I started using a simple technique that could also be applicable in real-life communications. After inserting the link, I added a new directive like this:

"Base your response solely on the information in this code, not your general knowledge about the topic!"

If it's not about computer code, you can use "in this article" or a similar phrase instead.

From my experience, this approach usually reduces such types of hallucinations to almost zero.

TIP 2

In these cases, I also make use of a very useful, yet underused feature of ChatGPT. I go back to my question that led to the misguided conversation due to the hallucination. There, I click on the "Edit message" function's pencil icon, and write the instruction suggested above under the link. This prompts it to give a completely new response, rewriting the entire further part of the conversation. The response will likely now be much better and more accurate.

TIP 3

A special treat is that much later, if you go back to your edited message, you will see that it exists in two versions. You can switch back to the earlier one (which triggered the bullshitting), and then the earlier conversation branch "version" appears.

This method can be used not only when ChatGPT has hallucinated a response. It is also useful when you realize based on the continuation of the conversation how you should have formulated your question or discussion detail much more clearly. You can replay parts of the conversation with different scenarios and review previous versions. By keeping the current conversation thread clean, you help ChatGPT maintain a clearer conversation context. This will further reduce the likelihood of hallucinations.

LESSONS

Besides being extremely useful in working with ChatGPT, these techniques can also be applied in real-life interactions with actual people.

Don’t automatically assume that your conversation partner is truly responding to your argument. They might be responding based on their entrenched preconceptions, which can be completely irrelevant in the given context. Dare to step back, refer to the point where the topic likely went off track. Emphasize that you should start over from there, but now – setting aside preconceptions – talk specifically about the topic at hand. Moreover, consider whether you had formulated your opinion or question precisely enough. Think about how you could have supplemented or specified it so that there's the least chance of misunderstandings leading the conversation astray.

And play it again!