Coding with ChatGPT - A Self-Experiment

For someone in a leadership position within a software engineering organization like myself, the paradigm shift in software development brought about by emerging AI/LLM services raises many questions:

1. Should every software engineer use this technology?
2. What about intellectual property rights? What are the implications for the company?
3. How effective are LLMs in accelerating software development, and what about the quality of their proposed solutions?

Today, I’d like to shed some light on question #3.

I finally decided to conduct an experiment to explore how much ChatGPT can assist with typical coding tasks. I’m not fond of dry research; I always need practical project to wrap my brain around a topic. So here we are: me planning a small iOS app requiring a cloud backend to function. It’s not a trivial task but also not overly complex (I’ll share details about the app in a later post 😉).

Warning: This post will get quite nerdy, but feel free to skip to the “My Learnings” and “Conclusion” sections if you’re short on time.

Starting Conditions

When it comes to iOS development in Swift, I consider myself an expert with hands-on experience. In contrast, my knowledge of cloud backends is more theoretical; I understand AWS conceptually but have little hands-on experience. For this experiment, I used the paid ChatGPT-4o option.

Getting Started

I began by describing the backend requirements to ChatGPT in plain English. Being somewhat familiar with AWS, I was quite specific about the technical requirements: a REST API, endpoint names, S3 for storage, and Lambda for compute.

The result was astounding: a detailed, step-by-step guide on how to configure a backend architecture using the AWS console. Nice! But I wanted Infrastructure as Code (IaC). No problem—after informing ChatGPT, it provided CloudFormation YAML. However, as a software engineer, I wanted AWS CDK. After another iteration, ChatGPT generated a CDK source file in Python. Getting closer, but I prefer JavaScript—easier to read for someone who grew up with C-like languages. ChatGPT delivered.

At first glance, the code looked fine. I ran cdk deploy—and hit a long error message: “no stack exported.” Thankfully, the built-in AI in my terminal (I use Warp) suggested the issue: the exports weren’t in TypeScript format. After a quick fix, deployment to AWS worked! Not bad for about 30 minutes of effort.

My Learnings

• Talk to ChatGPT as if you’re explaining to a very junior engineer. Mention every detail; don’t assume anything.
• Expect things to go wrong or to be only 90% correct (particularly bad in software engineering).

Does It Work?

Deployment is only the first step—does it actually work? The short answer: No. None of the Lambdas functioned. Testing them in the AWS console revealed errors like “Internal Server Error” and “…unable to locate aws-sdk…”. ChatGPT suggested the likely culprit: the Node.js version. AWS supports versions 14 and 16 for Lambdas, but ChatGPT had chosen 18. After fixing this, I encountered the next issue: “User is not authorized to access this resource with an explicit deny.”

The problem was with the “Custom Authenticator” feature in API Gateway. Debugging revealed that authentication headers from HTTPS requests weren’t being passed to the Lambda. ChatGPT had omitted essential configuration options. After asking for a solution, it provided the missing configurations.

With that resolved, the REST APIs finally worked as expected! The AWS backend was now in an alpha version. Total time: three hours—most of it spent debugging. Still, not bad.

Funny side note: I had specified a useless API endpoint, and ChatGPT happily implemented it without flagging the architectural oversight.

My Learnings (Continued)

• ChatGPT is a language model, not a domain expert. While the output is often correct, it reflects a mix of code snippets from different points in time. Using JavaScript instead of TypeScript hints at older practices, while choosing Node.js v18 points to newer ones. It lacks contextual understanding and mixes elements in ways that don’t always work for source code.
• Working with ChatGPT requires an iterative approach. Keep asking questions about errors or deficiencies, and it often corrects itself (though I did encounter one endless loop).
• Don’t expect ChatGPT to act as a proactive software architect. It will implement poorly designed architectures unless explicitly asked for an assessment.
• Don’t blindly trust the output. (Again, not a new learning, but always worth repeating.)

Conclusion

The experience was surprisingly good, but with caveats.

ChatGPT is an excellent coding assistant if the user has a baseline understanding of software development. LLMs help overcome the “blank page” problem, providing a starting point to iterate on. However, the user must understand the expected output and know how to ask the right follow-up questions.

Can an inexperienced person write an iOS app or design an AWS backend with ChatGPT? Clearly, no.

Implications for Software Development Organizations

• Allow SW developers to use LLMs: Used as assistive tools, they can save time and increase productivity.

• Team benefits: Debugging and iterative problem-solving with LLMs can help make agile teams more cross-functional. Why can’t a frontend engineer assist with backend tasks if a tool like ChatGPT is available?

• Compliance considerations: Before adopting LLM tools, consult your legal and compliance teams. Many current offerings provide enterprise solutions that address security and privacy concerns.

We are still at the beginning of this AI/LLM journey. It will be exciting to see how specialized AI models trained specifically for software engineering will further improve use cases like the one described.

Disclaimer

I used ChatGPT because I had a subscription and appreciated their excellent (native!) macOS app. Results would likely be similar with Claude, MS Copilot, or other tools.

Feedback

What about you? What are your experiences? What’s the recommendation in your company or organization? Are there better services than ChatGPT for these tasks?