VALHALLA, An ML Model That Learns Language Like Babies

VALHALLA is a machine that senses a source text in one language, hallucinates a visual, and then translates it into a language.

We babble and mimic our way of acquiring languages as newborns. We don't begin by reading raw text, which necessitates a foundational awareness of the universe as well as a sophisticated capacity to analyze and deduce descriptions and relationships. Rather, people begin their language journey by pointing and engaging with their surroundings, basing their words and understanding their meaning on the physical and social world. We will eventually be able to construct whole sentences to explain complicated thoughts. Similarly, employing additional sensory information, such as multimedia, in conjunction with new and unfamiliar words, such as flashcards with images, aids language acquisition, and retention. Humans may therefore properly comprehend new, unseen sentences in context without the use of supporting media with sufficient expertise; nevertheless, visualizing a picture based on the original language can help.

What is VALHALLA?

VALHALLA is a novel machine learning model developed by researchers from MIT, IBM, and the University of California, San Diego, in which a trained neural network perceives a source text in one language, hallucinates a picture of what it looks like, and then uses both to translate into a target language. The researchers observed that their method outperforms text-only translation in terms of machine translation accuracy. It also helped circumstances with extended phrases, languages with limited resources, and situations in which a portion of the original sentence is unavailable to machine translation.

Machine translation is an "extremely practical technology that is being used by millions of people every day," according to study co-author Yoon Kim, an assistant professor in MIT's Department of Electrical Engineering and Computer Science with affiliations in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and the MIT-IBM Watson AI Lab. Because "when humans are performing language processing tasks, we're doing so within a grounded, situated world," Kim says, "there's been an interesting development in how one might use non-text information — for example, images, audio, or other grounding information — to tackle practical tasks involving language." The scientists hypothesized that coupling hallucinated visuals with text during inference mimics that process, giving context for increased performance over existing state-of-the-art systems, which rely only on text.

Learning to have hallucinations based on pictures

Before we venture out on our own to learn new languages and translate, we are frequently given examples and practice. Machine translation systems are similar; however, if images are employed during training, these AI approaches will also require visual assistance for assessment, restricting their application, according to Panda.

"You might not have a picture with relation to the source text in real-world settings." So, rather than using an external picture as input during inference, may we leverage visual hallucination – the capacity to conceive visual sceneries — to improve machine translation systems? Panda declares.

To do this, the researchers employed an encoder-decoder architecture with two transformers, a sort of neural network model that is well-suited for sequence-dependent input, such as language, and can pay attention to important words and sentence semantics. One transformer creates a visual hallucination, while the other uses the first transformer's outputs to accomplish the multimodal translation.

There are two streams of translation during training: a source phrase and a ground-truth picture coupled with it, and the identical source sentence is visually hallucinated to create a text-image pair. Tokenize the ground-truth image and sentence into representations that transformers can handle; in the case of the sentence, each word is a token. The original text is tokenized once more, but this time through the visual hallucination transformer, which produces a hallucination, which is a distinct picture representation of the sentence. The researchers used an autoregression to check for congruency between the ground-truth and hallucinated representations, such as homonyms: a reference to an animal "bat" isn't hallucinated as a baseball bat. The hallucination transformer then optimizes based on the difference between them.

The two sets of tokens, each holding the phrase representation and either the hallucinated or ground-truth picture, are then concurrently processed through the multimodal translation transformer. The tokenized text translation outputs are compared to each other and to the target sentence in another language to be comparable. Any discrepancies are then fed back to the translation transformer for additional tweaking. Because photos are unlikely to be accessible in ordinary circumstances, the ground-truth image stream is turned off during testing.

The desired text can be visualized

The researchers put VALHALLA to the test against other cutting-edge multimodal and text-only translation systems. They employed available benchmark datasets including ground-truth photos and source sentences, as well as a text-only news article translation dataset. The researchers evaluated it on 13 different tasks, including translation in well-resourced languages (such as English, German, and French), under-resourced languages (such as English to Romanian), and non-English languages (like Spanish to French). The team also experimented with different transformer model sizes, how accuracy varies with sentence length, and translation in a constrained textual context, where chunks of the text were hidden from machine translators.

The researchers discovered considerable increases in data economy over text-only translation approaches, as well as the fact that smaller models outperformed the bigger base model. VALHALLA's effectiveness over other approaches improved as sentences became longer, which the researchers ascribed to the presence of more ambiguous terms. VALHALLA was able to retrieve and translate the original text in circumstances when part of the phrase was concealed, which surprised the team.

While VALHALLA is effective, the researchers point out that the method has drawbacks, such as the need that pairs of sentences to be tagged with an image, which might make data collection more expensive. In its ground domain, it also outperforms text-only news pieces. Furthermore, as Kim and Panda point out, a method like VALHALLA is still a black box, with the assumption that hallucinated sights are giving important information, and the team hopes to research what and how the model learns to verify their techniques.