Can AI Express How Angry It Sounds? Fine-Grained Emotion Intensity Control in TTS

Hi, I’m Keito Inoshita from Affectosphere Group.

“Slightly angry” and “furious” sound completely different when a human voice actor delivers them. The difference isn’t just volume — it’s in the pace, the pitch, the tightness in the throat, the way syllables compress or elongate under pressure.

For AI text-to-speech, getting the presence of emotion into speech has gotten easier. Getting the intensity right — that’s still a hard problem.

A 2026 paper from Yihang Lin, Li Zhou, Congwei Cao, Dongchu Xie, Xiaoxue Gao, Chen Zhang, and Haizhou Li (arXiv:2606.13006) attacks this problem directly. The framework they propose, called Emo-LiPO, treats emotion intensity as a ranking problem and applies Listwise Preference Optimization to fine-tune the alignment between what an emotional text prompt says and what the synthesized audio actually conveys.

---

3 Points for Today

1. The gap: LLM-based TTS can express emotion, but fine-grained control of emotion intensity — especially at high intensities — remains an unsolved challenge.
2. The approach: Emo-LiPO reframes intensity control as a ranking problem and uses Listwise Preference Optimization (LiPO) to directly optimize the relative ordering of intensity levels across candidates.
3. The result: Emo-LiPO outperforms existing baselines across all emotion categories, with the most notable improvements at high intensity levels. A new dataset, ESD-plus, is also released to support this line of research.

---

① Why Emotion Intensity Is Harder Than Emotion Category

Let’s start with why this problem is harder than it might look.

Modern LLM-based TTS learns from paired text-audio data. If enough examples exist for a given emotion — “angry speech,” “happy speech” — the model can learn what those categories sound like and reproduce them.

But emotion intensity is a different dimension. If the training data labels say “anger,” they don’t distinguish between mild irritation and barely-contained rage. Whatever distribution of intensities ended up in the dataset becomes the implicit norm, and the model learns to produce something in that range without any explicit signal about the relative ordering of intensity levels.

This creates a specific blind spot at the extremes. High-intensity emotional speech — deep grief, explosive anger, overwhelming joy — tends to be underrepresented in most training corpora. Rare data means less reliable learning, which means the generated speech either undershoots the intensity target or produces something unnaturally compressed.

Prior work has tried to address this by predicting intensity scores directly via regression, or by classifying intensity into discrete levels. The limitation is that these approaches don’t fully leverage the relative ordering of intensity — the information that “level 4 is more intense than level 3” is not explicitly shaped into the optimization objective.

---

② The Core Idea: Ranking, Not Scoring

Emo-LiPO’s key contribution is reframing emotion intensity control as a listwise ranking problem.

In a standard supervised setting, learning happens from absolute labels: “this input maps to this output.” In the intensity control problem, that means “this emotional prompt should produce audio at this intensity level.”

The listwise framing changes the structure of the learning signal. Instead of asking “what is the correct intensity for this prompt?” it asks “given multiple candidate outputs at different intensity levels, which ordering is most consistent with the prompt?”

For a text prompt specifying high emotional intensity, the model learns that a high-intensity audio rendering should be preferred over a medium-intensity one, which should be preferred over a low-intensity one. The full ordered list is part of the training signal.

Listwise Preference Optimization (LiPO) is the mechanism that operationalizes this. Unlike pairwise comparison methods that compare two outputs at a time, LiPO optimizes over the entire ranked list simultaneously. This means the relationships among all intensity levels — not just adjacent pairs — contribute to the gradient.

The practical consequence is that the model gets richer supervision about the structure of the intensity space, which matters most at the extremes where binary comparisons would provide the least information.

---

③ ESD-plus: A Dataset Built for This Problem

The research also introduces a new dataset, ESD-plus, designed specifically to support fine-grained emotion intensity research.

The data infrastructure problem here is real. To study emotion intensity control properly, you need speech data that covers multiple emotion categories, multiple intensity levels, and multiple speakers — and the interaction across all three. Existing datasets either lack intensity-level annotations, cover too few speakers, or don’t provide the cross-category coverage needed for robust evaluation.

ESD-plus is designed to fill that gap. It covers multiple speakers and emotion categories across multiple intensity levels, providing the training and evaluation substrate that Emo-LiPO requires.

The release of ESD-plus is itself a contribution beyond the framework. Benchmark datasets drive progress in a field. Having a well-annotated, multi-intensity, multi-speaker corpus available to the research community means that other approaches can be evaluated against a common standard, and progress can be tracked more systematically.

---

④ What the Results Show

Emo-LiPO outperforms existing baselines across all emotion categories — and the gains are most pronounced at high intensity levels.

This pattern is interpretable in terms of the framework design. High-intensity emotional speech is exactly where prior methods struggled, and it’s where the ranking approach has the most leverage. The explicit relative ordering in the LiPO objective gives the model direct supervisory signal for the “extreme” end of the intensity spectrum where training data is sparsest and where regression or classification objectives are least informative.

The result suggests the listwise approach captures something important about how emotion intensity is actually structured — as a continuum with meaningful relative relationships, rather than a set of discrete classes or an absolute value to predict.

---

⑤ What This Means for Affective AI

From an affective AI perspective, the implications go beyond audio quality.

Emotion intensity control matters for any system where an AI voice needs to respond appropriately to human emotional state. A customer support agent that responds to mild frustration and intense distress with the same vocal tone has calibrated its emotional output incorrectly, even if it says the right words. The voice layer of empathetic AI needs to match intensity, not just valence.

For interactive companions, educational AI, and mental health support applications, the stakes are higher. If an AI speaks to someone in deep distress with a cheerfully moderate tone, the mismatch itself becomes a communication failure. Fine-grained emotion intensity control is the infrastructure that lets the voice channel carry the right weight.

On the production side, the improvement at high intensity levels is particularly interesting for content applications — narration, audiobooks, dramatic readings — where the expressiveness ceiling of the AI voice directly limits what kinds of content it can produce. An AI voice that can credibly deliver high-intensity emotional speech opens up content categories that flat or mid-range TTS can’t reach.

The question of how well AI can express the degree of emotion — not just the kind — is one of the more underexplored frontiers in affective computing. Emo-LiPO is a meaningful step toward closing that gap.

That’s all for today!

---

References

1. Yihang Lin, Li Zhou, Congwei Cao, Dongchu Xie, Xiaoxue Gao, Chen Zhang, Haizhou Li (2026). Emo-LiPO: Listwise Preference Optimization for Fine-Grained Emotion Intensity Control in LLM-based Text-to-Speech. arXiv preprint arXiv:2606.13006.

* This article was written in part with AI assistance and may contain inaccuracies.