Can We Make AI Translation Code Aware?

Ever since Artificial Intelligence became a thing, it's been more or less the Wild West of AI product development (read 90% of products just prompt ChatGPT) over the past year. The European Union has taken the first steps to introduce legislation in hopes that the Yee-Haw age of AI Cowboys will slow down, at the same time, praying it'll not over regulate and scare off investment in the field.

Optimisation or Innovation?

When we look at translation industry, the amount of innovation in terms of AI products is much more tame. We see a lot of process automation, but that's not really new, is it? AI replacing translators? Not for the foreseeable future. We see much more optimisation than we see innovation. My guess is that it is due to the fact that creating bilingual AI models is that much harder. There are a lot (and I mean A LOT) of translation assistants embedded in TMS offerings for a hefty price. Is it a wise investment from the buyer side, you be the judge. Companies love to call them innovations, but are they really?

For AI to work as we've dreamed it would, we need a lot more context points. So when we think about AI innovation in localisation we often think awareness. How can we make it more aware of the content surrounding the piece we're trying to translate? A very interesting LangOps concept is a knowledge graph. It has been around for a while but made bilingual by Coreon . You can read more about it here.

Recent discussions with localisation specialists in a number of software development companies, pointed to a pressing issue. Post-localisation QA and implementation is a huge time and money investment to get it right. Now, imagine that every translator you employ also knows code. Close to impossible right? Instead, how about we explore the abilities of AI translation models to become more code-aware?

How can a custom AI model become code aware?

So our team at Native Localization got to work on concepts of how to make it happen. It is worth mentioning that our approach is to go small in the first place, meaning that we create seperate and secure branches of AI models and individually train them on data provided by the client. That allows us to pinpoint the model to be a lot less general and a lot more client and service specific.

When it comes to teaching code to the AI, we landed on three approaches, all of which supplement the MVLP model we use as our main basis for initial localisation deployment.

Prompting your way through the problem

Our initial idea was to create prompts that direct the AI model towards taking into the account the maximum number of characters per allowed space. We also made sure the model recognises character encoding. This worked well, to an extent. We could prompt ourselves through simpler, reperesentative websites, but that was not what we were looking for. Good enough of a starting point, however!

Feeding Code to the Custom AI Model

Looking at this alarming survey showing "more than two-thirds (67%) of programmers admitting pushing code into a production environment without testing, with more than a quarter (28%) acknowledging they do so on a regular basis", it showed us one thing. AI models are good at coding! So we brought in the heavy artillery. One of our clients, who got behind our dream of AI becoming code-aware coming true, shared the source code of a small app they have created. A small online platform built for a specific case (marketing). We took the code and started training our MVLP model with it. The outcome was better, but not surprising. We saw instances where AI self-corrected, but it still requires context, now between three languages.

Code aware QA check

It finally made sense to separate the code awareness from the language handling. We decided to create a niche model, trained on the same data the client provided, but now serving as a QA tool. To start, we limited the model to UI and content testing. We trained the model to use basic localisation QA scripts and referenced the scripts to the locations in code the content is likely to be found. The results were much better than creating the soup of AI like we did in the concept 2. We also introduced a failsafe, where the model had to report and log things it was unable to adjust. AI was able to go through a checklist and create a rather good output, not yet enough though.

Thoughts

Code awareness in theory is possible. When polished, it adds another layer of awareness for the localisation product and that can be considered a true innovation. That being said, a lot of AI innovation works on the premises of being "good enough". So what level of post-localisation QA relief is considered good enough? We'll let you know, because we're a bit obsessed with it.

From training point of view, we established that even a bilingual AI model is a pain to develop and train. Code is also a language, so creating an interconnection between three languages currently serves as a multiplier of confusion, so it makes sense to keep the code QA model separate.

For now, we're working with the objective of being "good enough to be great" which I think is a great way of formulating it and not get too depressed about the results, so stay tuned!

P.S. Autumn landscape has turned into winter wonderland, because I live in Northern Europe and I get all four seasons here. :)