Enhancing Multilingual Models with Byte-Level Language Models: Introducing Bolmo by Ai2

Enterprises seeking tokenizer-free multilingual models are increasingly adopting byte-level language models to enhance performance and reliability in dealing with noisy or low-resource text. To cater to this demand, the Allen Institute for AI (Ai2) has unveiled Bolmo, a new series of models that capitalize on its Olmo 3 models by transforming them into byte-level models while retaining their core capabilities.

Ai2 has introduced two versions of Bolmo: Bolmo 7B and Bolmo 1B, touted as “the first fully open byte-level language model” by Ai2. These models have demonstrated competitive performance, sometimes even surpassing other byte-level and character-based models, according to Ai2.

Byte-level language models work directly on raw UTF-8 bytes, eliminating the need for predefined vocabularies or tokenizers. This feature enables them to handle misspellings, rare languages, and unconventional text more reliably, making them ideal for tasks such as moderation, edge deployments, and multilingual applications.

The Development of Bolmo and its Architecture

Ai2 trained the Bolmo models using a combination of its Dolma 3 data mix, which was instrumental in training its Olmo flagship models, along with open-source code datasets and character-level data.

The primary objective of Ai2 is to offer a reproducible blueprint for converting strong subword language models into byte-level models that can be easily adopted and extended by the community. To support this goal, Ai2 plans to release checkpoints, code, and a comprehensive research paper to assist other organizations in building byte-level models based on the Olmo ecosystem.

Instead of training a byte-level model from scratch, Ai2 opted to convert an existing Olmo 3 7B checkpoint into a byte-level model in two stages. Initially, Ai2 froze the Olmo 3 transformer to train specific components such as the local encoder and decoder, the boundary predictor, and the language modeling head. This initial training phase was designed to be efficient and cost-effective, requiring only 9.8 billion tokens.

In the subsequent stage, the model was unfrozen and further trained with additional tokens. Ai2 highlighted that the byte-level approach adopted by Bolmo circumvents the vocabulary limitations inherent in traditional subword models.

Performance Evaluation and Comparison

While byte-level language models are not as prevalent as standard language models, research in this area is expanding. Meta introduced its BLT architecture last year, aiming to develop a robust model that processes raw data without relying on fixed vocabularies.

Other notable models in this domain include ByT5, Stanford’s MrT5, and Canine. Ai2 conducted a comprehensive evaluation of Bolmo using various metrics such as math, STEM reasoning, question answering, general knowledge, and code proficiency.

Bolmo 7B exhibited strong performance, surpassing character-focused benchmarks like CUTE and EXECUTE, and enhancing accuracy compared to the base LLM Olmo 3. Moreover, Bolmo 7B outperformed models of similar sizes in coding, math, multiple-choice question answering, and character-level comprehension.

Advantages of Byte-Level Models for Enterprises

Enterprises are increasingly adopting a hybrid model structure that combines different models and sizes to meet their requirements. Ai2 emphasizes the benefits of integrating byte-level models, not just for enhanced robustness and multilingual capabilities but also for their seamless integration into existing model ecosystems.

According to Ai2, the dynamic hierarchical setup of byte-level models allows for easy compression adjustments, offering organizations greater flexibility. By retrofitting a robust subword model like Bolmo rather than starting from scratch, Ai2 provides a low-risk pathway for enterprises seeking enhanced performance without disrupting their existing infrastructure.