Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models

arXiv:2608.195292026-08-21

A framework that lets pretrained LLMs generate machine-only codes -- like recommendation item IDs or legal citation markers -- right alongside plain text

Pretrained LLMs can only natively handle natural-language tokens, so systems that represent information as discrete machine codes (recommendation item IDs, graph structures, etc.) have had to either translate everything into text or rely on separate task-specific models. This paper introduces UniLang, which adds these machine codes as new vocabulary tokens to a pretrained LLM, aligns their meaning to the LLM's embedding space via contrastive learning, and then generates text and machine codes together under one autoregressive objective. Tested on two very different problems -- predicting a user's next movie/product and predicting which legal precedent sentence gets quoted -- UniLang beat strong specialized baselines on both.

METAL MEDIA explanatory visual

A framework that lets pretrained LLMs generate machine-only codes -- like recommendation item IDs or legal citation markers -- right alongside plain text

  1. 01The problem: information like recommendation item codes or graph relations is represented as discrete machine-native symbols rather than natural language, so it sits outside a pretrained LLM's normal token vocabulary and is hard for the LLM to generate directly.
  2. 02Building the codes: item descriptions (e.g., a movie's title and genre) are embedded with a text encoder and then compressed by an RQ-VAE (Residual Quantized VAE) into a short sequence of discrete codes, called a Semantic ID (SID), representing each item.
  3. 03Grounding into the LLM: 1,024 new tokens for these SIDs are added to the LLM's vocabulary, and their embeddings are pre-trained with an InfoNCE contrastive loss so a code's embedding lines up with the embedding of its matching text description -- giving the raw codes real meaning inside the LLM's space.
  4. 04Joint fine-tuning: the aligned code tokens and the LLM's original text tokens are merged into one vocabulary, and a Llama-3.2-1B-Instruct model is fine-tuned with LoRA to autoregressively generate mixed text-and-code sequences for each task.
  5. 05Results: on movie/product next-item recommendation, UniLang improved NDCG@5 by up to 151.7% over the strongest baseline (MovieLens-20M); on legal precedent prediction it improved Recall@1 by up to 49.4% over the strongest baseline (10k dataset); ablations showed training fails entirely without the contrastive pre-alignment step.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. The problem: information like recommendation item codes or graph relations is represented as discrete machine-native symbols rather than natural language, so it sits outside a pretrained LLM's normal token vocabulary and is hard for the LLM to generate directly.
  2. Building the codes: item descriptions (e.g., a movie's title and genre) are embedded with a text encoder and then compressed by an RQ-VAE (Residual Quantized VAE) into a short sequence of discrete codes, called a Semantic ID (SID), representing each item.
  3. Grounding into the LLM: 1,024 new tokens for these SIDs are added to the LLM's vocabulary, and their embeddings are pre-trained with an InfoNCE contrastive loss so a code's embedding lines up with the embedding of its matching text description -- giving the raw codes real meaning inside the LLM's space.
  4. Joint fine-tuning: the aligned code tokens and the LLM's original text tokens are merged into one vocabulary, and a Llama-3.2-1B-Instruct model is fine-tuned with LoRA to autoregressively generate mixed text-and-code sequences for each task.
  5. Results: on movie/product next-item recommendation, UniLang improved NDCG@5 by up to 151.7% over the strongest baseline (MovieLens-20M); on legal precedent prediction it improved Recall@1 by up to 49.4% over the strongest baseline (10k dataset); ablations showed training fails entirely without the contrastive pre-alignment step.
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 0
Figure 1: Example of the sequential prediction task.
Figure 1: Example of the sequential prediction task.
Table 2: Run-to-run variability on MovieLens-20M (mean ± SE).
metricmean ± standard error
Recall@50.1908 ± 0.00016
NDCG@50.1378 ± 0.00017
Recall@100.2596 ± 0.00014
NDCG@100.1600 ± 0.00014
Figure 2: Example of the legal precedent prediction task. Given context from the citing opinion (Garcia-Giraldo v. United States), predict the quotation sentence(s) or paragraph(s) from the cited opinion (United States v. Broce), which is unknown at inference time.
Figure 2: Example of the legal precedent prediction task. Given context from the citing opinion (Garcia-Giraldo v. United States), predict the quotation sentence(s) or paragraph(s) from the cited opinion (United States v. Broce), which is unknown at inference time.
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 3
Table 4: Statistics of the sequential recommendation datasets.
Dataset#users#items#actionsAvg. lengthDensity
Beauty40,22654,5420.35m8.80.02%
ML-1m6,0403,4161m163.54.79%
ML-20m138,49326,74420m144.40.54%
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 4
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 5
Table 6: Summary statistics of legal precedent dataset text features
FeatureMeanStdMinMax
Length of cited text (chars)3062252418,342
Length of citing context (chars)562216514,062
Figure 4: Ablation test on MovieLens-20m.
Figure 4: Ablation test on MovieLens-20m.
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 7
Table 7: Validation subset sizes for model selection
DatasetValidation sizeSample size
Beauty40,2265,000
MovieLens-1m6,0401,000
MovieLens-20m138,4931,000
10k103,8121,000
20k134,7371,000
50k190,0511,000
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 8
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 9
Table 8: RQ-VAE hyperparameters.
ML-1MLePaRD
HyperparameterBeauty
ML-20M10k / 20k / 50k
Figure 6: RQ-VAE training progress on different datasets.
Figure 6: RQ-VAE training progress on different datasets.
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models figure 11
Table 9: Machine token alignment hyperparameters.
ML-1MLePaRD
DatasetBeauty
ML-20M10k / 20k / 50k
Table 10: SFT hyperparameters.
ML-1MLePaRD
DatasetBeauty
ML-20M10k / 20k / 50k

Why it matters

It offers a practical path for real systems -- recommenders, medical coding, graph-structured data -- that store information as machine codes to directly tap into a pretrained LLM's language and world knowledge, instead of clumsily converting everything to text or building a separate model per task. One framework handling two structurally unrelated prediction problems suggests broader reuse potential across other symbolic domains.

Terms in this paper

  • RQ-VAE (Residual Quantized VAE) · An autoencoder that compresses continuous vectors into a short sequence of discrete codes through multiple quantization stages
  • Semantic ID (SID) · A machine-native identifier representing one item as a short sequence of discrete codes
  • InfoNCE contrastive loss · A training objective that pulls matching representations together and pushes non-matching ones apart to align two embedding spaces
  • LoRA (Low-Rank Adaptation) · A lightweight fine-tuning method that trains a small set of added parameters instead of the whole model
  • autoregressive generation · Generating a sequence one token at a time, each based on the tokens produced so far

Original abstract (English)

Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language. While these representations are compact and preserve task-relevant structure, they lie outside the linguistic token space of pretrained large language models (LLMs), creating a fundamental divide between language modeling and structured prediction. We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units alongside natural-language tokens. UniLang expands the LLM's vocabulary and embedding space with grounded machine-native representations, enabling textual and symbolic tokens to be jointly modeled and generated under a single autoregressive objective. This unified interface allows pretrained LLMs to directly operate on machine-native representations without requiring them to be verbalized as natural language or relying on task-specific architectures. We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction, spanning different domains and types of structured prediction. Across both tasks, UniLang consistently outperforms strong baselines, demonstrating a path toward extending pretrained LLMs beyond language and using them as a common generative modeling backbone for heterogeneous machine-native representations.

Authors · Su Yan, Rakesh Iyer

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Su Yan et al., arXiv:2608.19529, CC BY 4.0