Business learnings from a coding experiment with AI

I don’t know about you, but I have plenty of ideas about cool apps. They generally fall into 3 categories: (a) that app already exists (80%), (b) it is actually a terrible idea (19%) or (c) it’s way more complicated than the idea I made up in my mind in about 20 seconds and requires an organization with 100s of people to work on it (1%).

The other challenge, besides having the right idea, is complexity. Although I have been trained as a coder, it’s been … well … a few years! Nowadays tools make it way easier but there is still a learning curve.

On the other hand: AI. Could I build an app with generative AI support? I gave it a try.

I have a subscription to Claude.ai so I decided to use it (instead of paying for a specialized AI). The goal was less of creating a commercial app and more to “play” with generative AI.

What happened

• With a little bit of work, I could get a “functioning” simple app up and running in about 4 hours. No real knowledge necessary.
• I was not very detailed on my specifications, so AI made up a lot of things. A lot made sense (with a few surprisingly good "ideas"); a significant chunk was completely off the mark.
• Many requested functionalities did not work (e.g. there is a login page, but you don’t need to provide any information to log in).
• As I went back and forth with Claude to update the code, it created some bugs that Claude was not really good at solving.

What I learned

• AI changed the definition of insanity 

You can keep doing the exact same thing – have the same prompt – and get a different result. Maybe it is my data analytics brain, but this feels the biggest challenge with using AI – reproducibility (or lack thereof). Go to your favorite image generating AI and try the same prompt again and again… Every time you get different images. It shows a level of “creativity” which might be useful in some contexts. The challenge is if you want to build on something AI is generating. Let’s say you like the image you got but just want to change the color scheme. Asking the same prompt, while adding the color scheme, will generate brand new images, not the same image in a different color.

In a business context where continuous improvement is oftentimes a cornerstone, it is creating some challenges. You cannot really take an answer you are 80% happy with and tweak the prompt to get it to the finish line. Take a business presentation for instance. You can build a deck with a clean and clear story line and talking points. You will, inevitably, as you go through the review process, update the story line, change some of the facts you focus on, etc. A predictable tool would allow you to make a few changes to the prompt and regenerate the full deck with only the changes asked for. AI will regenerate a brand-new deck and potentially create other changes you will not like. Of course, a solution could be to ask the AI to update the deck in the same context (same chat) instead of re-prompting from the start. It is maybe a better option but…

• AI has bad, selective, non-discerning memory

Think back to the image example. Instead of re-prompting, you could just ask for the same image in a different color. Yet, you get a different (although similar) image in the new color scheme. As I was experimenting with generating code, I had to remind the AI a few times the coding environment I was in so that its answers made sense. That’s another challenge of generative AI. Even though within a chat AI is keeping context, it doesn’t always keep track of the entire context. From my experiment, it seems like AI can suggest changes along the way of the back-and-forth chat without adding it to the overall context. It “forgets” its own answers.

In a business context, back to the deck example, it might mean a slide is added based on a request through chat (not a new prompt) that does not fit within the bigger context. Let’s say you want to expand your SWOT analysis with a TOWS matrix on an additional slide. It is very possible your lists of Strengths, Weaknesses, Opportunities, and Threats will be different on each slide. A good way to check if your audience is paying attention, I guess! Which lead to …

• AI is making stuff up

This is well-known but needs to be kept top of mind. AI wants to please more than it wants to be truthful (or, rather, AI will get to an answer, whether it “knows” it or not because… well.. in spite of the name, it is not intelligent). As part of my job search, I am regularly asking AI to suggest updates or adjustments to my resume to align with the job description I am pursuing. It does a phenomenal job at identifying language and expectations from the job description to add them to the resume… even if it has no base in the reality of my experience! Details, details… It also moves bullet points around (so a relevant bullet point from a job 15 years ago suddenly shows up in my last experience). I also realized in an application that the most obvious fit from my background, my industry experience, was not mentioned (besides the company I worked at). AI found a lot of details to improve but completely missed the big picture (and I did too on this specific one... you live, you learn!). With my resume, I (hopefully!) know pretty well what I did and did not do. When asking complex questions to AI, it can be more challenging. AI seems to fall under an outsourcing best practice – never outsource (to AI) a task you are not capable of evaluating fully the quality of the result.

It was a fun experiment and got me into a rabbit hole for a little while (there may or may not be a multi-platform app coming out of this experiment :-)). What has been your experience? Are you a more advanced user who figured out how to work around these shortcomings? Maybe you learned something I did not?

PS: I asked Claude.ai to improve this post, and it did a terrible job. I do feel the quality of the outcome lately has been dropping. I am wondering if it is the novelty effect going away, my expectation raising, or some negative effect of AI usage on itself (e.g. maybe using its own generated content to feed its learning algorithm) ...

#AI #GenerativeAI #AIAndBusiness #Coding #FunWithAI