Vivian – Bouncerock

January 27, 2025March 2, 2025

AI Translation Meets Context: How to Translate Efficiently with AI

What role does Gen-AI play in translation? Why would it even help improve the accuracy of translations? This article addresses both of these questions by unraveling the myth of traditional machine translations and the new buzzy AI translation, their differences, and how to best utilize them by understanding how they work differently. But first, let’s take a look at Raiverb’s appraoch to translating efficiently with AI and how to do that.

Raiverb is a CAT developed from the ground up as a simple-to-use toolbox for translation and localization specialists. The core tool is the Translation Center, this tool was designed around the simple idea to use Gen-AI in order to bring context to translations.

Why Context Matters in AI Translation?

Lack of Context: What Advantage Does Context Bring?

First, let’s try to understand one thing: why is context that important?

Lack of context in AI translation means missing key elements that define the true meaning behind words, such as tone, setting, relationships, and emotions. For example, translating the phrase “I’m sorry” could have vastly different meanings in English depending on the context—it could be an apology, a form of sympathy, or a polite gesture. Without knowing these nuances, the translation could be completely off.

The context tells us which tone, choice of words, or cultural considerations are needed, so the translation is not just linguistically correct but also culturally appropriate. This idea of bringing context is fundamental, as this is how, in our view, using AI in translation makes the most sense. This happens to be what humans have also used for decades, and the very reason why they use CAT software in the first place for display of context while translating.

By bringing in context, the goal is to improve the accuracy and consistency in translations in order to avoid common errors from classic Translation Machines or unsupervised generative AI.

This will be developed in a second part, but first, here is a simple overview of what happens in Raiverb, a CAT designed to bring in as much context as possible for AI to be truly effective.

Maximizing Contextual Data for AI Translation in Raiverb

Now that you understand WHY context matters most in translation, it is time to tell you the HOW.

When you run a typical job on Raiverb, what happens is:
1. You have imported a document. The source text is split into relevant segments (called Translation Units), most often paragraphs, but not always.
2. Depending on the file the user imports, and with the user help, all possible metadata is extracted in order to provide the maximum context to the AI: comments, IDs, translations in other languages, time tags, etc.
3. Each segment is then analyzed and classified upon its relevance and similarity to other segments. The software knows what segments are repeated, similar, or not related at all.
3. Segments and all relevant data are submitted to the AI, translated, and saved separately. Past segments can be referenced at any point if they are relevant to the segment being translated.
4. A glossary is created automatically depending on user instructions and/or relevance to the project.
5. All translations are saved and kept for later reuse.

Accordingly, the steps here are pretty straightforward:

Import your content
Just like in the Translation Center, you can click “Import File” and find the source file you want to translate.
Alternatively, drag-and-drop will work as well.
Setup the content
The Importer will appear. Tell Raiverb which column is the source, and which column is your context (IDs, comments, alternative targets, character limits, etc.).
Import your knowledge base
This will typically includes Glossary and Translation Memory. You can click “Import” and find the corresponding files, or a simple drag-and-drop.
Setup the extra comment options
Type what guidelines you want AI to follow (including industry, tone, style, glossary extraction instruction, etc.) for the global source, or individually for a specific Translation Unit.
Click on “Start”
Here we go, that’s it!

Still want to check out the detailed steps? Please refer to the User Manual here.

Machine Translation vs AI Translation: Key Differences

By maximizing the contextual data and keeping track of past translations, Raiverb has a consistent output, which makes editing easier.

What are the side benefits?

As a byproduct of its operating method, there’re two side benefits derieved from Raiverb’s mechanism of bringing in context.

1. Raiverb lifts the common restrictions in terms of size of the projects that can be run using it.

When there is a cap on file size (which most commonly seen as 5MB-10MB as in most popular automatic translators such as Google or DeepL), it makes it hard to work with large projects if you still want to use AI. This is because a large project, when loaded on browser, can be very slow too. Raiverb is there to undo that limitation.

Being designed as a standalone, downloadable application, and not as a browser SaaS, is a deliberate choice that Raiverb makes. It aims to be different from SaaS which tended to limit the scope of projects that can be worked on. One common complaint by linguists about browser CAT services is that they are slow and frustrating to work with, due to the limitations of networks, and the inevitable UI limitations of input-intensive operations done through browsers.

2. Raiverb is not tied to one particular Gen-AI service provider.

The reason why Raiverb provides users with multiple Gen-AI server choice, is to offer them the flexibility to choose their preferred provider, allowing them to select the one that best fits their project needs. By not being locked into one AI provider, users can experiment with different models and choose the one that delivers the best results for their specific language pair, project type, or tone.

Now that you know HOW Raiverb leverages AI for translation, you must have a question: but why not bringing in context for traditional machine translators also? Why AI only? To answer this, we must understand the major difference between the two technologies.

Underlying Concepts Behind Machine Translation

Translation has been one of the earliest application of natural language models.
The underlying technological concepts behind Machine Translation, and generative conversational AI like ChatGPT are the same, but both are created for different objectives.

Traditional Machine Translation (Google Translate, DeepL) has been around for years, and has seen impressive jumps in quality since their inception. But despite their incredible progress, they have always been hindered by two major obstacles:

1. The inability to fully understand the context of a text;

2. The limitations in creative writing potential.

DeepL can deliver translations with an impressive quality. It can even use custom glossaries, or process whole files. But for large projects, projects sustained over time, or projects with a lot of comments or essential metadata holding the full context of isolated strings of text, it still shows limitations.

This is because Machine Translations deliver a translation of a given text done according to the most likely entry on their trained base of very high quality reference files. These databases include lots of nuances and subtleties in expressions and formulations. However, sadly, if you ask it to translate “How are you?”, it won’t be able to know whether to a formal or informal “you” in French or Chinese: it doesn’t know who is talking, and who is listening.
This may come, but is not quite there yet.

On the other hand, conversational AI such as ChatGPT can do that. AI has the ability to grasp the context of a text. This is why sometimes (but not always, and we come to that later) it simply appears to better for translations that come with detailed instructions.

Computer Assisted Translation, or CAT Tool

There’s nothing wrong with machine translation itself as a productivity tool to help translators boost their speed. However, it often falls short when it comes to understanding context—a critical element in producing accurate and natural translations. This is where Computer-Assisted Translation, or CAT tools come into play.

Using CAT, translators can see the content they need to translate, along with all the metadata they need in order to know more about the specific context of what they are translating.

Say you’re translating a dialogue between two characters. If you’re working on a game project, these dialogues may not be placed in a continuity. They may be one dialogue choice from the player among many. If the translator doesn’t know about the gender, the relationship with the other character(s) being adressed, the location that dialogue takes place in, or any relevant information, it is very easy to get a translation that is not acurate, or looks strange.

There are whole memes on the Internet built around cold, direct translations done without context, because it can be hilarious to see characters in a movie adressing each other like they’re complete strangers after 3 hours of adventures together.

Not to mention some languages use different words, tones, or grammar adjustments depending on these situational facts. And this does not only concern dialogues, but also UI in a software, subtitles in a video, even the translation of PPT files can be highly situational, and may look very wrong if done without context.

On the other hand, conversational AI such as ChatGPT has the ability to understand context. But they also show their limits.

Fundamental Limitations of Automated Translation

This is a statement concerning Translation, but also AI in general.
Many people worry about the progress of AI, mostly for several reasons:

First, it is expected and now well documented that AI, or at least Large Language Models as we know them today, will meet important limitations due to the diminishing returns on training volumes: the more volume you train them on, the less it seems to improve the overall quality of the models.

But more fundamentally, and I dare say philosophically: no matter how complex the neural networks can get, their architecture is solely based around completing a sequence in the most logical order, based on a trained database. This is the congregated experience of human knowledge, but the full experience of what it is to be human is more than the sum of the data created by humans.

Humans can corner themselves into absurd situations, which are often the result of their own contradicting emotions, which they, sometimes and hopefully use to turn into jokes. Machine neural networks do not create conflicts, they do not have contradicting emotions. They simply weight and quantify, then complete. But they will never create jokes about themselves. There is nothing to joke about in what they do.
When they do not function as expected, they hallucinate, or crash. They do not self-correct.

This is the exact reason why they will never be able to do any type of highly creative translation. Some translations are inherently creative, or at least require to undersand what an original author tried to mean. This can be very different from a culture to another. While AI may have references to cultural subtleties if their training contains any, they will never create their own subtlety.

Paring AI with CAT Tool in Translation

Consistency Maintanence

AI translation systems often struggle to maintain consistency in style and tone, which is crucial for preserving a brand’s voice or an author’s unique style. For example, a brand’s messaging might require a formal tone in one context and a conversational tone in another. Without supervision, AI may fail to adapt appropriately, leading to inconsistent translations.

This is where CAT tools, with their translation memories (TMs) and glossaries, play a vital role. By providing AI with predefined terminology and style guidelines, CAT tools help ensure consistency across translations.

Polysemy Disambiguation

AI systems frequently encounter challenges with polysemy—words that have multiple meanings. Without sufficient context, AI may choose the wrong interpretation, resulting in inaccurate translations. For instance, the English word “bank” could refer to a financial institution or the side of a river. Human oversight, combined with CAT tools, is essential to disambiguate such terms and ensure accuracy.

Handling New Terms and Jargon

AI systems often struggle with new words, technical terms, or industry-specific jargon, especially if these terms are absent from their training data. This can lead to awkward or incorrect translations. CAT tools, with their ability to integrate custom glossaries and TMs, provide a solution by equipping AI with the necessary terminology to handle specialized content.

Given these limitations, it is clear that AI cannot operate effectively in isolation. It requires supervision and contextual support to produce high-quality translations. This is where the combination of generative AI and CAT tools becomes invaluable. CAT tools provide the structured frameworks—such as translation memories, glossaries, and style guides—that AI needs to function effectively. Meanwhile, human translators bring the creativity, cultural understanding, and critical thinking necessary to verify and refine AI-generated outputs.

Conclusion

By now, it should be clear why context is everything in AI translation, ensuring better accuracy and relevance in every project—something that’s often lacking in machine-generated translations. The solution? By supervising AI. This is exactly why Raiverb focuses on providing as much context as possible for the AI—ensuring it doesn’t hallucinate, misinterpret, or generate translations that stray from our intent. In fact, one of the most valuable skills for today’s translators is understanding how these technologies work, so we can leverage them to their fullest potential and enhance our own craft.

July 13, 2024February 16, 2025

Behind Language Detection: Who Won the War Against ‘Tofu’

There was a period in computing, the early 2010s, where character encoding support was far from universal. Trying to install and run non-standard ASCII software or open files was a bit of a game of luck. Chracters had to be installed separately to be fully supported. And not just the IME, but actual character encoding support. Even then, nothing was gauranteed. In fact, it remained pretty common to see this on a regular basis: garbled text.

Fast forward to today, the situation has improved dramatically, thanks to advancements like UTF-8. But why was this happening, and why is it rare now? Brace yourselves; we’re diving into character encoding history and its evolution to UTF-8. By the end, you’ll have a clear understanding of why this transformation occurred, why OCR (or what we call image-to-text analyzer) can correctly detect languages.

How Characters Work in Computing: The Basics

To understand how an OCR tool can tell one language from another, let’s first delve into how computers interpret text. And let’s start with some computing 101 reminders. Nothing hard, promise.

Computers function with electric pulses of 0 and 1. That’s a bit. By convention, we put these bits in packs of 8, which is a byte. Each bit is either ON or OFF, which makes 256 different combinations of possibilities. Therefore your computer can count from 0 to 255 using a single byte.

What Is Hexadecimal, and Why Does It Matter?

Here is the hardest part: for certain purposes, we often split this byte in 2, so we get 4 bits, that’s 16 different possibilities, what is called hexadecimal. To represent these numbers we don’t have in our traditional decimal thinking, we use A, B, C, D, E and F. Which means A=10, B=11, etc.

For example:

Link has a tomato color:
<a href="#" style="color: #ff6347;">
The color value, ff6347, is actually 3 hexadecimal numbers: one for red, one for green, and one for blue. “ff” is the highest number possible (16×16=256). We can then know the reds are full in this color, like the name “tomato” would suggest.

Yes, I know, this is confusing and the reason is not you. Rather, we are using familiar numbers and letters to represent a different way of thinking. So don’t sweat it, it’s fine.

ASCII: The Foundation of Character Encoding

Back to our text. As far as your computer is concerned, a text is a list of characters (technically, an array). In your computer, phone, or any kind of intelligent device, each character is put in a grid, and we use the hexadecimals we just mentioned as the rows and colums of this grid.

In the beginning of computing, power was scarse and memory was limited, so in order to be as effective as possible, it was determined the smallest grid possible that would fit as much usable data as possible could be achieved by using a single byte.

Not even a single byte actually, but 7 bits (that leaves one bit to do other things).
This ended with the first convention for language representation: the American Standard Code for Information Interchange, or ASCII, was born.

Fun Fact: Not Every Character is Visible

Note that not every character is designed to be visible. For example, character 13 is End of Line. Therefore, when you press “Enter”, you are actually writing the character number 13 (or D) into your document or chatbox, which your program knows to interpret as a going to the next line.

For a deeper dive into ASCII and binary systems, check out ASCII Overview on W3C.

A Diversity of Encodings

The Early Limitations of ASCII and the Rise of Extended-ASCII

The ASCII standard, which only supported the 26 base letters of the English alphabet, was insufficient for encoding languages other than English. To address this limitation, the original 128-character slot system was quickly abandoned in favor of Extended-ASCII, which expanded the character set to 256 slots. While this was a step forward, it still didn’t provide the capability to support all the world’s languages within a single encoding.

At the time, Extended-ASCII was seen as “good enough,” especially for English-centric systems. However, the expansion was still far from enough to accommodate the complexities of global languages. This challenge, similar to those faced in modern translation technologies, highlights the importance of language detection, where an accurate identification of the language is key to ensuring the right encoding and format are applied.

Language-Specific Encoding Systems and Compatibility Issues

In the absence of a universal standard, each language began to adopt its own character encoding system, complete with unique grids and mappings. This led to the rise of various encoding formats, each designed to meet the specific needs of its language. While this approach addressed immediate issues, it also introduced significant compatibility problems.

For instance, Chinese computing was dominated for years by the GB-2312 (GB standing for 国标) encoding for Simplified Chinese, a system that remains widely used today. In parallel, the Big-5 encoding system was used for Traditional Chinese characters. These encoding systems created one of the most significant sources of incompatibility in the Chinese computing world.

The Role of Encoding Agreements in System Interactions

In computing, different systems must constantly communicate with each other. For example, an operating system (OS) must communicate with software, files must be read by software, and webpages must be rendered by browsers. The first step in this communication is often an agreement on which character encoding to use.

When this agreement is not reached, or when one system doesn’t support the necessary encoding format, the risk arises that the wrong encoding will be applied. As a result, characters may be mapped incorrectly, leading to distorted or unreadable content—often referred to as “garbled characters.”

In older systems, the OS was typically designed to understand only a specific set of character formats. If a system wasn’t built to accommodate other encodings, errors were common. When these systems tried to read data in unsupported formats, characters would often display incorrectly or not at all, creating a frustrating experience for users and developers alike.

This issue becomes especially important in Optical Character Recognition (OCR) systems. OCR technology relies on accurately interpreting and converting images of text into machine-readable text. If the encoding agreement between the OCR system and the software used to process the recognized text is misaligned, the output can be garbled or inconsistent. Ensuring that OCR systems use the correct character encoding is crucial for achieving accurate text recognition, especially when handling documents in multiple languages or specialized formats. Without proper encoding support, OCR-generated content may suffer from errors, making it difficult for users to extract meaningful information from scanned documents.

UTF-8: The Universal Solution

In a legitimate effort to harmonize character systems and solve this issue, the Unicode Consortium pushed for the adoption of a single universal format that would be as widespread as possible. A format that would contain all forms of characters from all languages possible.

After several tries, UTF-8 was the format that stuck, and was widely adopted. This still is the most used format around the world. UTF-8 can accommodate 1,112,064 characters. It supports 1 byte, 2 bytes, 3 bytes and 4 bytes of data. This means that it is not just one grid, but four grids coexisting within a single encoding. This allows it to be compatible with ASCII, because it uses the same single-byte grid.

UTF-8 handles most existing forms of written communication, including emojis, which are, as far as your computer is concerned, regular characters with their reserved space in the grid. UTF-8 is a standard managed by the Unicode Consortium. There is still a lot of free space (meaning empty cells in the grid), that’s why emojis can be regularly added.

Despite its widespread adoption, display issues can still arise. These are often due to unsupported fonts rather than encoding errors. This is particularly relevant when using OCR systems. If the OCR software misinterprets or fails to apply UTF-8 encoding when processing text from images, it can result in incorrectly displayed characters. This can lead to text that appears distorted or unreadable, even when the encoding is theoretically correct. Therefore, ensuring the correct use of UTF-8 encoding in OCR systems is essential for maintaining the integrity and readability of converted text.

The Truth About Fonts

The last key concept to mention is fonts. So first let’s get something out of the way:

Fonts and encoding are two different things

Special characters look ugly, but don’t blame the encoding, it’s the font

A font is a graphical representation of the character matched in the encoding grid. It’s the “picture” your computer will show to the final user.
But it’s up to each font designer to draw what they want wherever they want. Or to not draw anything.

The famous font Wingding, which was Microsoft’s first attempt at showing emojis, has symbols instead of letters. But for all intents and purposes, it’s still a font, which means you will be able to see regular letters whenever you switch to a regular font.

Even though UTF-8 is widely used, not every font supports all characters in the UTF-8 encoding standard. In fact, very few fonts actually do, and the reason is easy to understand: comprehensively supporting the tens of thousands of characters across dozens of different languages is an extremely tedious task.

Font Limitations

Originally, when a computer would stumble upon a character unsupported by the current font, it would display placeholder empty squares:

□□□□□□□□□□

That is why it was so easy to confuse an encoding issue with a font issue. But both issues are very different in nature, as you now understand. Modern software are designed to display a default font if they can’t find the right character. It may not look always good, but it is still better than a tofu placeholder.

The famous Google “Noto” font is the result of Google’s effort at having a font that would never return placeholder square.

This font barely supports the base ASCII characters, but you can use it with a UTF-8 encoding

Language Detection in CAT Tools: A Game Changer for Translation

That’s it for encoding and fonts. That’s not an easy topic to tackle, but why does it matter for understanding character enoding? Mastering its fundamentals can improve troubleshooting for text display issues and ensure smooth handling of multilingual content. And also, understanding the fundamentals of UTF-8 will allow us to do more cool things like detecting languages, for example, in CAT tools.

Language detection is a critical feature in modern translation technologies like CAT tools, automating many processes and eliminating manual input errors. Here are a few ways language detection in CAT tools enhances translation workflows:

Automatic Language Selection

When importing content, CAT tools’ translator automatically identify the source and target languages, streamlining the setup process for translators.

OCR Integration

In a CAT tool where Optical Character Recognition (OCR) is performed, language detection ensures accurate text extraction by adapting to the document’s language.

LQA and Consistency Checks

Language detection enables efficient Language Quality Assurance (LQA), often an essential feature in CAT tool, by identifying inconsistencies in terminology or syntax based on the detected language.

Pro Tip: Explore how our CAT tool leverages advanced language detection to optimize translation workflows and reduce manual effort.

Thanks to innovations like UTF-8 and robust language detection systems, what was once a complex and error-prone process has become seamless and intuitive. Whether handling multilingual content or ensuring precise OCR results, language detection is the backbone of modern translation technology. By automating tasks like language selection and consistency checks in CAT tools, translators now can focus on crafting high-quality translations.