One Pass, Two Indexes: UEmbed Unifies Sparse and Dense Multimodal Retrieval in a Causal Decoder
UEmbed emits sparse lexical and dense vector embeddings from one decoder-only multimodal pass—closing the dense–sparse gap on MMEB-v2 without bidirectional retrofits or cross-modal bolt-ons.
Search and retrieval-augmented generation share a quiet assumption: you pick either dense vectors or sparse lexical weights, and you pick either a bidirectional encoder or a cross-modal bolt-on. A paper posted to arXiv on August 3, 2026 challenges that split directly.
Researchers from CASIA, Alibaba Group, the University of Chinese Academy of Sciences, and Yale introduce UEmbed, a decoder-only multimodal embedding model that emits both sparse lexical and dense representations in a single causal forward pass—without auxiliary cross-modal modules.
The problem UEmbed targets
Learned sparse retrieval (LSR) methods such as SPLADE improved on BM25 by learning which terms to weight and which related terms to inject. But the authors argue three constraints have persisted:
- Architecture: LSR has stayed tied to bidirectional encoders. Adapting decoder-only LLMs typically means converting them to bidirectional behavior or bespoke training curricula—breaking compatibility with causal serving stacks like vLLM.
- Modality: Multimodal sparse retrieval often adds separate cross-modal modules rather than deriving sparse weights from one backbone.
- Utility: Sparse models are evaluated mainly on classical text benchmarks, leaving agentic and hybrid use cases under-explored.
UEmbed's design goal is blunt: one decoder-only multimodal language model, one pass, two retrieval modes.
How it works: partitioned projection
The core trick addresses what the paper calls the "information bottleneck" of projecting a single token's hidden state into a vocabulary-sized sparse vector.
UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets (derived via k-means clustering). Each special token's causal hidden state predicts sparse weights over its assigned subset; the subsets concatenate into the full sparse vector. Dense embeddings come from the EOS token preceding the special tokens.
Because attention is causal, each special token summarizes the input from its position forward—producing semantically meaningful lexical activations the authors illustrate with cross-modal examples on their project page.
Benchmark numbers (public-data training only)
The team releases UEmbed at 2B, 4B, and 9B scales built on Qwen3.5 backbones, trained on public data:
| Model | MMEB-v2 Dense | MMEB-v2 Sparse |
|---|---|---|
| UEmbed-9B | 71.8 | 71.0 |
On MMEB-v2, the authors report UEmbed-9B leads public-data-trained models in dense retrieval and sets state of the art for sparse retrieval. They note the dense–sparse gap closes to within one point at every scale tested.
On BEIR (nine datasets, nDCG@10), UEmbed remains competitive with strong dense and sparse baselines including SPLADE-v3.
These are benchmark claims on specific suites—not a universal statement that hybrid sparse+dense beats every specialist on every task. But the parity between modes is the headline: the same checkpoint serves both inverted-index search and vector search.
Three practical claims beyond leaderboard scores
The paper emphasizes utility along three axes:
Hybrid scoring: Combining dense and sparse outputs from the same model improves text and visual-document retrieval versus either mode alone.
Serving compatibility: Because sparse weights emerge from a causal decoder without bidirectional retrofitting, the model retains compatibility with high-throughput causal inference stacks and standard inverted indexes.
Agentic search: On BrowseComp-Plus, the authors report the sparse mode reduces tool-call costs while maintaining comparable recall—relevant as agents spend increasing token budgets on retrieval loops.
Why this matters for RAG stacks
Production retrieval pipelines often maintain separate sparse and dense indexes, separate encoders, and fragile fusion rules. UEmbed's pitch is operational simplification: one multimodal encoder, two index types, native hybrid scoring.
For multimodal RAG—documents with charts, screenshots, and mixed layouts—the absence of auxiliary cross-modal modules also matters architecturally. Prior sparse multimodal work bolted modality bridges onto text-centric sparse models. UEmbed argues sparse weights can come directly from the same causal MLLM backbone used for dense embeddings.
Limits and open questions
The paper is a preprint, not peer-reviewed production guidance. MMEB-v2 and BEIR scores do not automatically translate to latency, cost, or freshness in a live search product. The 9B public-data model may still trail proprietary closed models on some tasks—the authors compare against public-data baselines explicitly.
Nor does unified embedding eliminate the hard parts of retrieval engineering: chunking, freshness, access control, and evaluation on domain-specific corpora still dominate deployment risk.
Still, for a field that treated "sparse from decoders" and "multimodal sparse without extra modules" as separate open problems, UEmbed bundles both into one architecture—and ships checkpoints at three scales with a public project page at alibaba-nlp.github.io/UEmbed.
### Sources
- arXiv — UEmbed: Unified Sparse and Dense Multimodal Embeddings (2608.02583) (August 3, 2026)
- Alibaba NLP — UEmbed project page (August 2026)