Emotion Detection in 2026: Accuracy, Law and Real Cost

Emotion detection is software that infers an emotional state from something it can measure: a facial movement, the pitch of a voice, a word choice, a heart rate. The engineering is not the hard part any more — you can wire a working prototype together in a weekend. The hard part is that the accuracy numbers in the literature are measured on splits you cannot reproduce in production, the datasets behind the best models forbid you from selling anything, and since February 2025 two of the most commercially attractive use cases are illegal in the European Union.

This is the guide we give clients on the first call. It covers what the four measurable signals actually measure, what accuracy survives an honest evaluation, which datasets and checkpoints you may legally ship, what the EU AI Act and the US biometric statutes require in July 2026, what the vendors charge, and where the buy-versus-build line sits. Every number carries a year and a source. Where the evidence is thin, we say so.

Why Fora Soft wrote this guide

Fora Soft is a software development company founded in 2005. We have delivered 250+ projects with 50 in-house engineers, almost all of them video-first: real-time communication, streaming, and the AI layers that sit on top of them. Emotion detection comes up on discovery calls constantly, and it gets quietly descoped far more often than it gets built, usually once the legal and accuracy reality lands.

We have built it, once, and small. EmoProof is an internal iOS proof of concept that photographs the reader while they scroll a news article, scores the facial expression with a Core ML model on the device, adds a voice-sentiment pass, and keeps a private journal of how different content moves the reader's mood. Nothing leaves the phone. That constraint was not a privacy flourish — it was the only architecture that made the product defensible. It is the only emotion-specific product we have built, it never went to production, and we would rather say that plainly than imply a track record we do not have.

We have also been on the other side of the conversation. BrainCert is a live virtual-classroom platform we build for, running $3M ARR across 100K+ customers and 500M+ classroom minutes. A live classroom is exactly where product teams ask for per-student engagement scoring, and exactly where Article 5(1)(f) of the EU AI Act now says no. Most of the value in this guide is in knowing which of those two stories your product is.

What emotion detection actually is

Emotion detection is an AI system that infers a person's emotional state from measurable signals rather than asking them. In practice that means one of four inputs — a face, a voice, a piece of text, or a physiological trace — passed through a classifier that returns a label such as happy, angry or neutral, or a pair of continuous values for valence and arousal.

The term is used loosely, and the looseness has legal consequences, because the EU AI Act attaches obligations to a specific definition and not to a marketing word. Here is the vocabulary, sorted by what it actually means.

The science problem you inherit

Before the law, the science. In 2019 five researchers led by Lisa Feldman Barrett reviewed more than a thousand studies for Psychological Science in the Public Interest and concluded that facial movements are not reliable evidence of emotional state. The paper is Emotional Expressions Reconsidered (20(1), 1–68, DOI 10.1177/1529100619832930), and its central sentence is aimed squarely at people building products:

“It is not possible to confidently infer happiness from a smile, anger from a scowl, or sadness from a frown, as much of current technology tries to do when applying what are mistakenly believed to be the scientific facts.”

Barrett, Adolphs, Marsella, Martinez & Pollak, 2019

The measured evidence behind that sentence is worth quoting precisely, because it gets mangled online. The review leans on an expanded meta-analysis by Durán and Fernández-Dols (2018) covering 131 effect sizes from 76 studies and 4,487 participants. Two numbers come out of it: an average correlation of r = .31 between the intensity of a facial configuration and a measure of emotion, and an average proportion of .22 — the expected facial configuration was present during the emotional event about 22% of the time. The peer-reviewed version of that meta-analysis (Durán & Fernández-Dols, Emotion, 2021) puts co-occurrence at .13 for full coded expressions, .23 once partial expressions count, and the correlation at .30. Those are not accuracy figures for a model. They are the ceiling on what any model reading a face can recover, because the signal it reads is only weakly tied to the thing you want to know. And the argument is live: a 2023 comment in the same journal re-read the same data as evidence of a moderate, not negligible, association. Either way, nobody in the debate is defending 90%.

What makes 2026 different is that the regulator has written the critique into binding law. Recital 44 of the AI Act states there are serious concerns about the scientific basis of emotion-inference systems and names their shortcomings as limited reliability, the lack of specificity and the limited generalisability. You are no longer arguing with a psychologist. You are arguing with the text your market regulator will read back to you.

The practical response is not to abandon the technology. It is to stop claiming more than you measure. Hume AI calls its product Expression Measurement and tells developers that expressions “should not be treated as direct inferences of emotional experience.” Google's MediaPipe Face Landmarker model card says the primary intended application is AR entertainment and that it is not intended for human life-critical decisions. Both companies are hedging in public. Copy them.

The four signals you can measure

Every emotion detection product is built on one or more of four channels. None of them observes an emotion. Each observes a physical proxy, and each fails in a different way.

Figure 1. Face, voice, text and physiology — what each channel measures, where each one breaks, and one licence-clean option per channel.

Facial emotion detection: geometry, not feeling

The modern face pipeline detects a face, fits a mesh, and reads the geometry. MediaPipe Face Landmarker returns 478 landmarks plus 52 blendshape coefficients — ARKit-style deformation values such as jawOpen and browInnerUp under Apache-2.0, tuned by Google for real-time use on mobile. Note what it does not return: an emotion. Mapping blendshapes to emotion labels is your inference and your liability.

Face breaks on head pose beyond about 30 degrees, on backlight, on masks and heavy glasses, and on cultural display rules — the learned conventions about when it is appropriate to show what you feel. It also breaks on people who cannot move their face in the expected way, which is a larger group than most product teams assume.

Speech emotion recognition: prosody, not words

Speech emotion recognition reads pitch contour, energy, speech rate, jitter, shimmer and spectral tilt. The eGeMAPS feature set is the standard starting point and it is small enough to compute on a phone. Watch the licence on the extractor itself: openSMILE, the reference implementation, ships under the audEERING Research License, which forbids commercial use and explicitly extends that to features extracted with it. Either buy the commercial licence or compute the same descriptors with librosa or torchaudio, which are BSD and BSD-style. Voice generalises across demographics better than face does, and it degrades faster than face does with room noise, aggressive codecs, and any assumption that there is exactly one speaker per channel.

Text: the only channel with clean licences

Text emotion detection classifies word choice, negation, intensifiers and emoji against a taxonomy. Google's GoEmotions set (27 emotions plus neutral, 58,000 Reddit comments) is the practical default, and it is the one corner of this field where the licensing is uncomplicated. It is also the channel the AI Act's prohibition does not reach, because a transcript is not biometric data.

Physiology: arousal, not category

Heart-rate variability, electrodermal activity and remote photoplethysmography tell you how activated someone is, not what they are feeling. Arousal is a genuinely useful signal for stress and fatigue work. rPPG in particular is sensitive to movement, ambient lighting and skin tone, so treat any accuracy figure from a controlled lab as an upper bound you will not see.

The channel splits into two engineering problems that get confused. Contact sensing — a wrist wearable, a chest strap, finger electrodes — gives you clean inter-beat intervals and skin conductance, and the hard part is not the signal but getting the device onto the user and keeping it there. Camera-based rPPG gives you pulse from the same webcam frame you are already reading for face, and the hard part is that it degrades with motion, with low or flickering light, and measurably with darker skin tones, because the green-channel absorption the method depends on is weaker. If your product serves a general population, rPPG needs its own subgroup evaluation before it goes anywhere near a decision.

Watch the taxonomy trap too. The physiological research corpora label along valence and arousal, not along Ekman categories — DEAP (32 participants, EEG plus peripheral signals), SEED (EEG, 15 participants) and WESAD (wrist and chest, 15 participants) all take that shape. So a physiological model does not drop into a face-model pipeline without a mapping layer you have to design and defend. Sample sizes in the teens and thirties are also the norm here, which is a good reason to treat published subject-independent accuracies as directional rather than as targets, and every one of these corpora is released for academic use under terms you have to read before a product touches them.

Where physiology earns its place is as a gate rather than a label: high arousal plus negative text sentiment is a far safer trigger for a human handover than any seven-class emotion prediction, and it is much easier to defend to a regulator, because you are reporting activation rather than claiming to know a feeling.

How accurate is emotion detection?

Plan for 45–60% on a six- to eight-class task with people the model has never seen before, and treat anything above 85% as a claim about the evaluation protocol rather than about the model. The only two figures in the table below that come from audited subject-independent splits are 48.1% and 53.1%; the higher numbers all sit on splits nobody outside the authors can verify. On a binary task — engaged or not, escalating or not — you can reach the high eighties, which is why the products that ship successfully almost always collapse to two or three classes.

Figure 2. The published number and the number to plan around. Every headline figure here is real; each one is measured under a protocol that flatters it.

The benchmarks, honestly

One caution before the table. The last column is not always a subject-independent number, because two of these corpora make one impossible: FER2013 and AffectNet ship no identity labels at all, so nobody can build a subject-disjoint split on them or verify that a published one exists. Read their figures as split-unaudited rather than validated.

Three rows deserve a second look. RAVDESS is the clearest illustration of the protocol problem: a June 2026 paper in Scientific Reports ran one multimodal model both ways and reported 95.83% on a random split against 48.06% ± 9.76% with the test speakers held out entirely. That is a 47.8-point gap produced by nothing but the choice of split. The same paper measured CREMA-D at 73.54% against 53.12% ± 2.65%, there under subject-exclusive speaker-independent 5-fold cross-validation rather than leave-one-speaker-out, so the size of the collapse is corpus-specific, but the direction never is. If a vendor quotes accuracy and will not name the split, the figure carries no information.

IEMOCAP is the second. It ships with no nominal test set, so every paper picks its own. When Antoniou and colleagues re-measured nine published systems for ICASSP 2023 they got weighted accuracies spanning 63.8% to 78.2%; three systems could not be reproduced at all and only two had shared weights. A number above 80% on IEMOCAP is an invitation to ask which fold, not a state of the art.

MELD is the third, and it carries the cheapest sanity check in the field — provided you compare like with like. Its test set is 48% neutral, so answering “neutral” every time scores about 48% accuracy but only about 31 weighted F1. Audio-only systems land near 41.8 weighted F1: ten points above the trivial baseline, and nowhere near the 69 that gets quoted from text-only models. Before you accept any dialogue-emotion number, ask what the trivial baseline scores on the same metric. Half the comparisons in this field fail that question, including some that flatter the sceptics.

What happens between the benchmark and production

A 2026 review in Springer's Discover Robotics, synthesising 42 empirical studies, put figures on the drop from controlled conditions to the real world: speech systems lost 8.6% ± 3.0% on average, facial systems 14.4% ± 7.1%. Face loses more because lighting, pose and camera quality vary more than microphones do. Budget for the larger number if your users are on laptops in their kitchens.

Do large language models help?

On fine-grained labels, not yet. A July 2026 evaluation on a 13-class emotion taxonomy put Gemini 2.5 Flash at 0.363 macro-F1, GPT-5.4 at 0.291 and Claude Sonnet 4.6 at 0.159 zero-shot. Their raw accuracies were statistically indistinguishable (McNemar, p > 0.10); macro-F1 was never significance-tested, and the authors describe that spread as practically rather than statistically significant, attributing the lowest score to a class-imbalance bias. For scale, the fine-tuned BERT baseline shipped with GoEmotions scores 0.46 across 28 classes. Different dataset, so not a head-to-head — but a small supervised classifier is not the thing losing here. Frontier models reason well about emotion across a whole conversation and label single utterances poorly. Use them for the session summary, not the per-turn score.

Emotion detection datasets: the licence trap

The most common reason an emotion detection project dies in our experience is not accuracy and not law. It is that somebody fine-tuned a checkpoint on a corpus whose licence forbids commercial use, and the discovery happens during due diligence rather than during design.

Whether a trained checkpoint is a copyright derivative of its training data is unsettled, and it does not have to be settled to hurt you. Most emotion corpora restrict use by contract, and those terms bind you whichever way the copyright question eventually lands. Read the terms, not the licence badge on the repository: IEMOCAP's clause 3 expressly captures “derivatives thereof”, while the RAVDESS commercial licence expressly releases model weights. Two corpora, opposite answers, same question.

Figure 3. Eight standard emotion corpora against three questions. Only two of them are clean for a commercial product, and neither of them is the one everybody trains on.

Datasets

Pre-trained models

The same trap, one layer up. These are the checkpoints teams actually reach for, with what their licences really permit as of July 2026.

Is emotion detection legal in 2026?

In the European Union, emotion detection is prohibited in workplaces and education institutions and permitted everywhere else with conditions. The prohibition took effect on 2 February 2025 and the 2026 Digital Omnibus did not touch it. Outside the EU there is no equivalent ban, but the biometric statutes described in the next section reach further than most teams expect.

What Article 5(1)(f) actually says

The prohibition covers “the placing on the market, the putting into service for this specific purpose, or the use of AI systems to infer emotions of a natural person in the areas of workplace and education institutions, except where the use of the AI system is intended to be put in place or into the market for medical or safety reasons.” Penalties sit at the top tier under Article 99(3): up to €35,000,000 or 7% of total worldwide annual turnover, whichever is higher. Article 99(6) reverses that for SMEs and start-ups, capping them at whichever is lower — worth knowing before the number ends up in a board deck.

Read the scope carefully, because the ban is narrower than the panic around it. It reaches only inference, only from biometric data, and only in those two settings. Text-based emotion analysis of a support ticket sits outside it. Hiring sits inside it because the Commission's guidelines on prohibited practices extend the workplace limb to candidates during selection and hiring. Admissions sits inside it separately, through the education limb, which reaches candidates during the admissibility process — and which the same guidelines say carries no limitation as to environment: online, in person and blended are all covered.

The carve-out nobody quotes: Recital 18

Recital 18 states that the definition “does not include physical states, such as pain or fatigue, including, for example, systems used in detecting the state of fatigue of professional pilots or drivers for the purpose of preventing accidents.” That single sentence is why driver-drowsiness monitoring is a growing legal market in Europe while classroom engagement scoring is not. If your product measures a physical state and says so, you are in a different regulatory box.

The medical and safety exception is narrower than it reads

Commission guidance points at CE-marked medical devices as the intended shape of the therapeutic exception. General wellbeing, stress or burnout monitoring for employees does not qualify. If you are relying on this exception, you should expect to have to show a device classification, not an intention.

Everything permitted is still high-risk

Emotion recognition that is not prohibited lands in Annex III point 1(c) as high-risk. Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026 and in force from 27 July, moved the Annex III obligations from 2 August 2026 to 2 December 2027, and Annex I embedded products to 2 August 2028. That is breathing room for the conformity assessment, the risk management system and the technical documentation — not a reprieve.

The date that has not moved: 2 August 2026

Article 50(3) requires deployers of an emotion recognition or biometric categorisation system to inform the people exposed to it that the system is operating, and to process their data in line with the GDPR. The only carve-out is for systems permitted by law to detect, prevent or investigate criminal offences. It sits in Chapter IV, which the omnibus left on the original 2 August 2026 date. The penalty tier is Article 99(4)(g): up to €15,000,000 or 3% of turnover. If you operate an emotion feature in the EU, the notice has to be live on 2 August 2026.

There is also enforcement precedent under general data protection law that predates the AI Act, and it cuts against the usual assumption. In February 2022 the Hungarian authority fined a bank HUF 250 million and ordered it to stop inferring emotions from recorded customer calls. It expressly held that the voice analysis was not Article 9 special-category data, because the output could not uniquely identify anyone. Transparency, legal basis and privacy by design under Articles 5, 6, 12, 13 and 25 were enough on their own, and the decision is public. Staying outside the biometric definition does not put you outside enforcement.

The US rules that actually bite

There is no federal emotion-AI rule in the United States. What exists is a set of biometric statutes that reach emotion pipelines indirectly, plus a children's privacy rule that reaches them squarely, plus an advertising-law hook the FTC has built and never used.

Illinois BIPA

BIPA requires a written release before collecting a biometric identifier and gives individuals a private right of action with liquidated damages of $1,000 for negligent violations and $5,000 for reckless or intentional ones. Whether inferring emotion without identifying anyone triggers BIPA is unresolved. The Ninth Circuit in Zellmer v. Meta, 104 F.4th 1117 (2024), applied a capability test: scans of face geometry are covered only if they can identify a person. District courts in the Northern District of Illinois have not settled on it either: Martell v. X Corp. and Castelaz v. Estée Lauder adopted a capability requirement in 2024, Brown v. AS Beauty Group did not, and Konow v. Brink's declined to decide, holding the claim survived either way. No court has ruled on emotion inference specifically, on the merits.

Public Act 103-0769, effective 2 August 2024, capped recovery rather than accrual: repeated collection by the same method from the same person supports a single recovery, as does repeated disclosure to the same recipient. Courts have been explicit that it does not change how claims accrue. The Seventh Circuit held in Clay v. Union Pacific Railroad Co. on 1 April 2026 that the cap applies retroactively to cases that were pending when it was enacted.

Texas CUBI and Washington's My Health My Data Act

Texas CUBI covers “a retina or iris scan, fingerprint, voiceprint, or record of hand or face geometry”. Note the voiceprint limb: that is the one that catches speech emotion, and most summaries drop it. CUBI is enforced only by the Attorney General, who has used it twice at enormous scale — $1.4bn from Meta in July 2024, $1.375bn from Google in May 2025. Since the Texas AI statute took effect on 1 January 2026, training or storing biometric identifiers inside an AI system is exempt unless it is done to uniquely identify a specific individual. The $25,000-per-violation penalty in § 503.001(d) is untouched, and a CUBI breach can now also be charged under Chapter 552 — additively, not instead. Do not count on the Chapter 552 notice-and-cure window: it attaches to Chapter 552 claims, not to an Attorney General action brought under § 503.001(d).

Washington, California, Colorado

Washington's My Health My Data Act sweeps in “mental health status” as consumer health data, which is close enough to an emotion score to plan around, and is enforced through the state consumer protection act rather than through its own damages provision. California's CPPA automated decision-making regulations took effect on 1 January 2026 and are the only US rule that names the thing outright: “vocal intonation, facial expression, and gesture (e.g., to identify or infer emotion).” The ADMT compliance obligation itself bites on 1 January 2027, with the first risk-assessment submission in April 2028. And Colorado's AI Act is the one to watch and the one not to plan against yet: as originally passed it reaches any system that is a substantial factor in an employment or education decision, which would put an emotion score inside a hiring funnel squarely in scope, but it has been delayed, amended and re-enacted more than once and enforcement has been held up in litigation. Check its current text and status before you rely on it in either direction.

COPPA, if any of your users might be children

The amended COPPA rule reached full compliance on 22 April 2026. It never uses the word emotion. It does not need to: the 2025 amendments added a biometric prong at 16 CFR 312.2(10) covering “a biometric identifier that can be used for the automated or semi-automated recognition of an individual, such as … voiceprints; gait patterns; facial templates; or faceprints.” A face mesh or a voice embedding computed from an under-13 user is personal information on its face. The older prong — “a photograph, video, or audio file where such file contains a child's image or voice” — catches the capture step as well.

The FTC: a loaded hook, never fired

We could find no FTC enforcement action about emotion-AI accuracy claims. Not in Operation AI Comply (September 2024), not in IntelliVision (final order January 2025), not anywhere else. Several AI-written sites assert otherwise; the assertion does not survive a search of ftc.gov. What does exist is the Policy Statement on Biometric Information of 18 May 2023 (Matter No. P225402), which says plainly that “as with other types of technologies, false or unsubstantiated marketing claims relating to the validity, reliability, accuracy, performance, fairness, or efficacy of technologies using biometric information constitute deceptive practices in violation of Section 5 of the FTC Act.” The doctrine is built. Assume it gets used the first time an emotion vendor publishes a number it cannot reproduce.

The one place regulation pushes emotion tech forward

EU Regulation 2019/2144 makes driver monitoring mandatory, and it is the rare case where the rules create demand. Driver drowsiness and attention warning became mandatory for new vehicle types on 6 July 2022 and for all new vehicles on 7 July 2024. Advanced driver distraction warning followed for new types on 7 July 2024 and for all new vehicles on 7 July 2026. The delegated regulations behind both require them to work without biometric identification of occupants, but that is not what puts them outside the AI Act. Recital 18 does, by excluding physical states such as fatigue from the emotion-recognition definition altogether.

Emotion detection API pricing: buy or build

Buy below roughly 1,250 session-hours of video a month and build on-device above it. Two general-purpose clouds still sell face-based emotion inference — AWS Rekognition at $0.0010 an image and Google Cloud Vision at $1.50 per 1,000 — alongside a shrinking set of specialists, while Microsoft withdrew the feature and Hume retired its self-serve measurement API in June 2026.

The vendor market told a clear story between 2022 and 2026: Microsoft removed emotion inference, Amazon kept it and added a legal warning to its own API reference, Google kept it and added nothing, and the leading specialist retired its self-serve product. If you are choosing a supplier, that trajectory matters more than any feature comparison.

Two conclusions follow. First, if your architecture depends on a single emotion API, you are one product decision away from a rewrite — three of the seven rows above changed materially in the last four years. Second, the vendor that hedges hardest in its documentation is usually the one that has thought about this most; Amazon's warning is a feature, not a weakness.

What emotion detection costs

At 18 million images a month, a cloud emotion API costs about $12,000 a month on AWS Rekognition list pricing (July 2026), against roughly $36,000 once to build the same thing on-device plus about $500 a month to keep it current. Break-even lands at 3.1 months. Here is where those numbers come from.

Take a live e-learning platform running 10,000 student-session-hours a month, sampling one frame every two seconds. That is 1,800 frames an hour and 18,000,000 images a month. Here is the arithmetic, out loud.

Figure 4. Cumulative cost of a cloud emotion API against an on-device build at 10,000 session-hours a month. The lines cross in the first quarter.

Buying it

AWS Rekognition DetectFaces, us-east-1, July 2026 list price

  1,000,000 images  x  $0.0010  =  $ 1,000
  4,000,000 images  x  $0.0008  =  $ 3,200
 13,000,000 images  x  $0.0006  =  $ 7,800
  -------------------------------------------
 18,000,000 images             =  $12,000 / month  =  $144,000 / year

Google Cloud Vision, same volume
  4,999,000 x $0.0015 + 13,000,000 x $0.0006  =  $15,298.50 / month

Building it

The on-device version replaces the per-frame cost with a one-off engineering cost. Our scope for a single-modality on-device feature is around 480 engineer-hours all-in: model selection and licence clearance, the on-device inference pipeline, a subject-independent evaluation suite, a stratified subgroup test set, the consent screen and the Article 50 notice, and the aggregation backend that receives scores rather than frames. At $75 an hour that is $36,000, plus roughly $500 a month to keep the model and the subgroup evaluation current as devices and populations change. Collecting the test set is inside that number, not on top of it.

Be careful with the comparison, because most published buy-versus-build maths cheats here and ours would too if we left it alone. Six of those seven line items are owed on both paths: a cloud buyer still clears the vendor's terms, still owes the Article 50(3) notice, still needs the subject-independent evaluation and the stratified subgroup set, and still needs an aggregation layer. The genuinely incremental piece is the on-device inference pipeline, call it 120–160 hours. A cloud integration is not free either — budget 60–100 hours for it. So the honest differential is 20 to 100 engineer-hours, roughly $1,500 to $7,500. Compare that against the running cost, and the build looks even better at high volume and clearly worse at low volume than the headline arithmetic below suggests. On the differential alone the crossover arrives much earlier than the 1,250-hour line below. We show the total-cost version because that is the number finance signs off on, and because it is the conservative one.

Break-even against the cloud API

  $36,000  /  ($12,000 - $500)  =  3.1 months

Break-even by volume, amortising the build over 24 months

  monthly cost of building  =  $36,000 / 24  +  $500  =  $2,000
  $2,000 of tiered API      =  1,000,000 x $0.0010  +  1,250,000 x $0.0008
                            =  2,250,000 frames  =  1,250 session-hours / month

Below about 1,250 session-hours a month, buy the API and spend the difference on the product. Above it, every extra hour of video is free on the device and costs between $0.45 and $1.80 in the cloud depending on which AWS tier you have reached — the floor drops at 35M images a month, which is about 19,400 session-hours, so very large platforms should re-run this with the $0.00025 rate before deciding. The privacy story improves on-device at every volume.

One caveat on our own number: 480 hours is what our teams take, and our teams use AI-assisted engineering throughout, so it sits below the figures you will see from agencies that do not. Treat it as our estimate rather than an industry rate, ask any vendor including us to break it down by workstream, and re-run the arithmetic with whatever number you actually get. The shape of the calculation does not change.

The costs that are not in the table

• Your own test set. Academic corpora will not match your users. Collecting and annotating 500–1,000 consented clips across the demographics you actually serve is usually 2–4 weeks and the single highest-value line item in the budget.
• Subgroup evaluation, repeated. Not a launch task. Re-run it every time the model, the camera pipeline or the user base changes.
• A DPIA, and a fundamental rights impact assessment if you are a deployer in scope. Plan for legal review time, not just engineering time.
• The abstain path. Designing what the product does when the model says “I don't know” is real product work and it is always underestimated.
• Model refresh. Face pipelines drift as phone cameras change. Budget one refresh a year.

Demographic accuracy gaps

Emotion models fail unevenly across groups, the published gaps are smaller than the internet claims and larger than any product can ignore, and the most useful finding is that the biggest gap is usually age rather than race. Here is what the peer-reviewed record actually says.

What the studies found

• Bryant & Howard, AIES 2019. Tested commercial emotion APIs on children's faces. Affectiva averaged a 48.9% match rate against a claimed accuracy in the high 90th percentile, with a 15.34% failure-to-compute rate; Face++, Google, Microsoft, Amazon and Sighthound all failed to compute on under 1% of images. Human raters scored above 80% on the same images.
• Rhue, 2018. Scored NBA player photographs through two commercial APIs and found Face++ read Black players as roughly twice as angry as white players at equivalent smiles. That anger result holds across specifications. The contempt result usually quoted next to it is a Microsoft output rather than a Face++ one, is significant in only two of six specifications, and sits on a model the author herself says does not fit the data. Quote the first and not the second. This is an unrefereed working paper — useful, not conclusive.
• Xu et al., ECCV 2020 Workshops. The most careful subgroup breakdown available. On RAF-DB, measuring mean class-wise accuracy on an unaugmented ResNet-18 baseline, the largest disparity was age: 65.9% for the 20–39 bracket against 51.3% for the over-70s, about 15 points. Race spanned about 8 points and, contrary to the usual summary, the weakest group was Asian at 60.0% while African-American faces scored 68.1%, above the Caucasian 65.9%. Gender spanned about 2 points. The authors warn that the over-70 and African-American cells hold only 92 and 204 images, so those two numbers are unstable — which is itself the lesson about cell size.
• Kim et al., AIES 2021. Four commercial systems, measured in 2019 and again in 2020: least accurate on older adults both years, with little to no gender disparity.

The gap in the literature you should worry about

No peer-reviewed audit since 2021 has run a demographically stratified benchmark through named commercial emotion APIs. Microsoft's withdrawal removed the largest target, and vendor terms of service discourage the rest. Amazon still ships emotion detection at scale with no published demographic audit that we could find. You cannot outsource this measurement to the research community. You have to do it yourself.

The protocol we run

1. Build a stratified test set from your own users, with consent. Minimum 30 clips per cell, cells defined by the axes that matter for your product — typically age bracket, Fitzpatrick skin type, gender presentation, and device class.
2. Evaluate subject-independently. No person appears in both training and test. This is the single change that turns a flattering number into a usable one.
3. Report per-group confusion matrices, not a single accuracy. Aggregate accuracy hides exactly the failure you are looking for.
4. Set a ship threshold before you look at the results. We use a maximum 8-point gap in per-class recall between the best and worst cell, and we write it down first so the number cannot move afterwards.
5. Publish a model card. Training sources, evaluation protocol, per-subgroup results, known failure modes. Customers and auditors will ask, and having it written before they do turns a security questionnaire from a project into an attachment.

The architecture we ship

Our default for a video-first product in 2026 is on-device inference with aggregate-only telemetry. It costs less at scale, it removes most of the compliance surface, and it survives a procurement questionnaire without a lawyer in the room.

1. Consent before the first frame

A two-screen onboarding that states what the feature measures, where inference happens, what leaves the device, and how to turn it off. Under Article 50(3) the notice has to reach everyone exposed to the system, not only the account holder — in a group video call that means every participant, not just the host.

2. Inference on the device

MediaPipe Face Landmarker in WebAssembly for the browser or via the Tasks API on mobile, feeding a small classifier head you trained yourself. On the audio side, eGeMAPS-style descriptors computed with librosa or torchaudio feed a lightweight model — use openSMILE itself only under a commercial licence. Both run comfortably in real time on hardware from 2021 onwards.

3. Fusion with an explicit abstain

Score each available channel, weight by the reliability you measured, and refuse to answer when the evidence is thin. The refusal path is the part teams skip and the part that keeps the feature honest.

from typing import NamedTuple

class Verdict(NamedTuple):
    label: str
    confidence: float
    reason: str

def fuse(scores, weights, min_conf=0.55, min_agreement=0.70, min_channels=1):
    """scores: {channel: (label, confidence)}, already mapped to ONE taxonomy.
    weights: measured reliability per channel, from our own subgroup test set."""
    live = {c: v for c, v in scores.items() if v is not None}
    if len(live) < min_channels:
        return Verdict("unknown", 0.0, "too few channels")

    agg, wsum = {}, 0.0
    for channel, (label, conf) in live.items():
        w = weights.get(channel, 0.0)
        agg[label] = agg.get(label, 0.0) + conf * w
        wsum += w
    total = sum(agg.values())
    if wsum == 0.0 or total <= 0.0:
        return Verdict("unknown", 0.0, "no usable signal")

    label, weighted = max(agg.items(), key=lambda kv: kv[1])

    # two independent gates, both must clear
    agreement = weighted / total     # do the live channels point the same way?
    mean_conf = total / wsum         # is any of them actually confident?

    if mean_conf < min_conf:
        return Verdict("unknown", mean_conf, "all channels weak")
    if agreement < min_agreement:
        return Verdict("unknown", mean_conf, "channels disagree")
    return Verdict(label, mean_conf, "ok")

Two gates rather than one, because they catch different failures. Agreement on its own is useless when a single channel is live: with one input the agreement ratio is always exactly 1.0, so a 2% face score comes out as certainty. The mean-confidence gate is what stops that, and single-channel sessions are the common case — most of your users have the camera off. Map every channel into one label space before you call this, too: a face model saying happy and a text model saying joy agree, and string equality will not know it.

4. What actually leaves the device

Derived scores on a fixed cadence — a five-second rolling average is usually enough — and never a frame, a face embedding or an audio buffer. If raw biometric data never reaches your servers, most of the GDPR Article 9 exposure and the whole of the enterprise-procurement argument goes away with it.

5. Aggregation, storage and retention

Aggregate across people before you display anything. “Cohort attention fell 18% over the last ten minutes” is defensible and useful; “this student looks sad” is neither. Keep derived scores for as long as the product genuinely needs them and no longer, and keep raw biometrics for zero seconds by not collecting them.

6. Keep the inference layer swappable

Put a hard interface between the emotion layer and the business logic. You will replace the model — to close a subgroup gap, to move off a corpus whose licence turned out to be wrong, or because a vendor retired the endpoint, which as the table above shows is not hypothetical.

Mini case study: EmoProof

The ask. A founder wanted to know whether the news people read was measurably changing their mood, and wanted to prove it without becoming a privacy story. The obvious build — upload frames, score them in the cloud, show a dashboard — would have collected biometric data from consumers in every jurisdiction the app shipped to.

What we built. An iOS app that takes a snapshot while the user reads, runs a Core ML facial expression model entirely on the device, adds a voice-sentiment pass, and writes the result to a local journal with trend statistics over time. No frame, no audio and no embedding leaves the phone. The output is framed as the user's own record of their own reactions, not as a claim about what they feel.

What we learned. Two things, and both changed how we scope this work. On-device inference cost us accuracy relative to a cloud model, and it bought back more than it cost in what the product was allowed to be. And the interesting signal was never the per-frame label — it was the trend across sessions, which is far more stable than any individual inference. If we rebuilt it today we would drop straight to three classes and spend the saved effort on the trend view. Talk to us about your version of this if it sounds like your product.

Where emotion detection pays off

Four categories work commercially in 2026, one is legally interesting because regulation creates the demand, and one keeps being requested and keeps being prohibited.

Contact centres and customer experience

Real-time prosody scoring that coaches an agent during an escalating call is the most established commercial use of speech emotion. In the EU, note the direction of travel carefully: a hint delivered to the agent about the customer is a different product from a score about the agent delivered to a supervisor, and only one of those is legal.

Market and content research with consented panels

Facial coding against advertising and content, on panels that opted in and are paid. The methodology is mature, the consent is explicit, and the unit of analysis is the panel rather than the person. VocalViews, a market-research video marketplace we built with over a million participants, is the platform shape this work sits inside.

Clinical and wellbeing research

Voice biomarkers for depression and anxiety remain an active research area, and the cautionary tale is worth the paragraph. Kintsugi was the best-known voice-biomarker company in the field; it was never FDA-cleared, its own site described the product as an investigational device, and it shut down commercial operations on 10 February 2026 and open-sourced the models. If you are building here, treat regulatory clearance as the product, not as a later step.

Driver and operator monitoring

The only segment where regulation mandates the technology rather than restricting it, and the only one where the AI Act's own recitals put you outside the definition. Fatigue is a physical state. Build it as fatigue detection, document it as fatigue detection, and stay out of the emotion box entirely.

Accessibility and expression-driven input

Blendshape output used as a control surface — a raised eyebrow as a switch, a held expression as a confirm — is emotion detection's technology stack applied to a problem where the user is deliberately signalling rather than being read. Consent is inherent, and the accuracy bar is far more forgiving.

The one everybody asks for

Per-student engagement scoring in a live classroom is the single most requested emotion feature we see, and in the EU it has been prohibited since February 2025. The workable version is a non-biometric one: attention proxies from application focus, response latency, participation rate, poll answers. It is less impressive in a demo and it ships. We cover the adjacent AI features that do work in our guide to AI video features for remote learning.

When not to build it

The most valuable thing on this page may be the list of situations where the right answer is to stop. We have talked more clients out of this feature than we have built it for.

• When a non-biometric proxy answers the same question. If you want to know whether a lesson landed, a two-click reaction widget outperforms a face model on accuracy, on cost and on consent. Ask what decision the score feeds before you build the score.
• When the output would drive a consequential decision. Hiring, admissions, promotion, clinical triage. The science does not support it, the AI Act reaches into hiring through the workplace limb, and a wrong answer is a lawsuit rather than a bad recommendation.
• When you cannot collect a representative test set. No test set means no subgroup evaluation, which means you are shipping an unmeasured failure mode to a population you have not looked at.
• When the feature has to work on everyone. Face-only pipelines exclude users with facial paralysis, users wearing face coverings, users in the dark, and users who simply keep the camera off. If there is no graceful fallback, the feature is a tax on a subset of your users.
• When your differentiation depends on the accuracy claim. If the pitch is “we detect emotion accurately”, you are one FTC complaint or one independent test away from a very bad week.
• When it is a demo feature. Emotion detection demos beautifully and retains poorly. If nobody can name the workflow it changes, it is a screenshot, not a feature.

Five questions before you commit

Run these in order. Most products stop at the second one, and stopping there is a win, not a failure.

Figure 5. The four gates we walk clients through. Gate two — would a non-biometric signal do — ends most of these conversations, cheaply.

Seven mistakes that kill launches

1. Quoting a benchmark number as a product number. The 95.8% on RAVDESS is real and it is not yours. Publish what your model does on your users, or publish nothing.
2. Fine-tuning on AffectNet and finding out later. The single most common licence failure in this field. Clear the corpus before the first training run, not before the first customer audit.
3. Shipping a seven-class model when three would do. Fewer classes means better accuracy, more consistent annotation, cheaper evaluation and a clearer UI. Almost every product we have seen over-specified the label set.
4. No abstain class. A model that always answers will always be wrong on the hard cases, loudly, in front of the user who cares most.
5. Surfacing an individual's score to another individual. Aggregate first. Per-person emotional readouts shown to a manager or a teacher are the exact pattern the AI Act prohibits and the exact pattern that generates press coverage.
6. Treating consent as a checkbox at the end. Under Art. 50(3) the duty runs to everyone exposed to the system, from first exposure. Retrofitting that into a group-call UI is far more work than designing it in.
7. Building face-only. Cameras are off, faces are occluded, lighting is bad. Text and voice give you a fallback path that costs a fraction of the face pipeline.

KPIs and a pre-launch checklist

Six numbers to watch from the first production session

Pre-launch checklist

1. Legal review completed against the AI Act, GDPR, and the biometric statutes of every state you sell into.
2. Article 50(3) notice live, reaching every exposed person and not only the account holder.
3. Licence chain cleared and documented for every dataset and every checkpoint in the pipeline.
4. Subject-independent evaluation run on a stratified test set collected from your own users.
5. Model card published, with per-subgroup results and a named list of failure modes.
6. Abstain path designed, implemented and visible in the UI.
7. Aggregation layer verified to make per-individual readout impossible, not merely discouraged.
8. Retention policy implemented: derived scores time-boxed, raw biometrics not collected.
9. Accessibility path for users who will not or cannot supply a face or a voice.
10. Incident plan for the day the model misclassifies someone in public, because it will.

FAQ

What is emotion detection?

Emotion detection is an AI system that infers a person's emotional state from a measurable signal — a facial movement, voice prosody, word choice, or a physiological trace such as heart-rate variability — rather than asking them. It returns either a categorical label such as happy or angry, or continuous values for valence and arousal. Under the EU AI Act the regulated form of it is inference from biometric data specifically.

How accurate is emotion detection in 2026?

Plan for 45–60% on a six- to eight-class task with people the model has never seen: the audited subject-independent numbers are 48.1% on RAVDESS and 53.1% on CREMA-D. The best published figures are higher — 79.8% on FER2013, 64.3% on AffectNet-8, and 69.2 weighted F1 on MELD from text alone — but the largest gains in the literature come from evaluation protocols that let test subjects appear in training. One 2026 study measured 95.83% on a random split and 48.06% leave-one-speaker-out with the same model on RAVDESS.

Is emotion detection legal?

In the European Union it has been prohibited in workplaces and education institutions since 2 February 2025 under Article 5(1)(f) of the AI Act, with fines up to €35 million or 7% of global turnover; medical and safety uses are the only exception. Everywhere else it is permitted subject to data protection law. From 2 August 2026 anyone deploying an emotion recognition system in the EU must inform the people exposed to it, under Article 50(3).

Which emotion detection API should you use in 2026?

AWS Rekognition DetectFaces returns eight emotion labels plus UNKNOWN at $0.0010 per image, falling to $0.00025 above 35 million images a month, and is the most straightforward option — though emotions are off by default, so you have to ask for the EMOTIONS attribute. Google Cloud Vision returns four coarse likelihood buckets at $1.50 per 1,000 images. Microsoft retired emotion inference in 2022–2023 and Hume sunset its self-serve Expression Measurement API in June 2026, so the field is narrower than most listicles suggest.

Can I train an emotion model on FER2013 or AffectNet and sell the result?

No, on the plain reading of both licences. FER2013 was never licensed for reuse — the 2013 contest granted rights over the winners' code, not the images — and AffectNet is released for academic research only, by gated academic request. Both restrict use by contract, and the terms are written to follow the weights — IEMOCAP's licence names “derivatives thereof” explicitly. CREMA-D (ODbL) and GoEmotions (Apache-2.0) are the commercially clean options.

What is the difference between emotion detection and sentiment analysis?

Sentiment analysis scores the polarity of an opinion — positive, negative, neutral — and almost always operates on text. Emotion detection assigns a specific emotional state and can operate on faces, voices, text or physiological signals. The distinction is legally load-bearing in the EU: the AI Act's obligations attach to inference from biometric data, so text sentiment sits outside them and webcam emotion inference sits inside.

Do large language models do emotion detection well?

Not on fine-grained labels. A July 2026 evaluation on a 13-class emotion taxonomy put Gemini 2.5 Flash at 0.363 macro-F1, GPT-5.4 at 0.291 and Claude Sonnet 4.6 at 0.159 zero-shot. For scale, the fine-tuned BERT baseline that shipped with the 28-class GoEmotions dataset scores 0.46 — a different benchmark, so not a head-to-head, but a small supervised classifier is not the thing struggling here. Frontier models are strong at reasoning about emotion across a whole conversation and weak at labelling a single utterance, so use them for session summaries rather than per-turn scoring.

How much does it cost to build emotion detection into a product?

Our scope for a single-modality on-device feature is around 480 engineer-hours, which is roughly $36,000 at $75 an hour, plus about $500 a month of upkeep. A cloud API at 18 million images a month costs about $12,000 a month on AWS list pricing, so the build pays for itself in around three months at that volume. Below roughly 1,250 session-hours a month the API is cheaper and you should buy.

Does emotion detection work on children?

Worse than on adults, and the compliance burden is higher. Bryant and Howard's 2019 study found commercial APIs performed markedly worse on children's faces, with one vendor averaging a 48.9% match rate against human raters above 80%. Under the amended COPPA rule, in full effect since 22 April 2026, any pipeline that captures a camera frame or audio from a user under 13 is collecting personal information, regardless of whether the emotion score itself is covered.

Should emotion inference run on the device or in the cloud?

On the device by default for consumer products. You give up some accuracy — we saw that on EmoProof, and there is no published head-to-head that would let us put a number on it — in exchange for a design where no biometric data leaves the user's hardware. That removes most of the GDPR Article 9 exposure, most of the US biometric-statute exposure, and most of an enterprise security questionnaire. Above about 1,250 session-hours a month it is cheaper as well.

Can emotion detection tell if someone is lying or hiding their feelings?

No, and any vendor claiming otherwise is describing a capability the underlying science does not support. The meta-analysis inside Barrett and colleagues' 2019 review found the expected facial configuration present during the corresponding emotional event in about 22% of cases. Deception detection from facial or vocal cues has an even weaker evidence base, and in the EU inferring intentions from biometric data falls inside the AI Act's definition of an emotion recognition system.

What are the limitations of AI emotion detection?

Four, and they compound. The science is weak — the meta-analysis inside Barrett and colleagues' 2019 review found the expected facial configuration present in about 22% of emotional events, and Recital 44 of the EU AI Act names limited reliability, lack of specificity and limited generalisability as the shortcomings. Accuracy collapses on unseen people: one 2026 study measured 95.83% on a random split and 48.06% with the test speakers held out. Subgroups differ by up to about 15 points by age (Xu et al., ECCV 2020 Workshops). And almost every state-of-the-art face checkpoint is trained on research-only data you cannot ship.

What to read next

Ready to scope an emotion feature that ships?

The engineering here is a solved problem. What decides whether an emotion detection feature reaches production is a shorter list than most teams expect: is the use case on the right side of Article 5, can you legally ship the weights you trained, does the accuracy number survive a subject-independent split, and does the volume justify a build over an API. Get those four right and the rest is a normal project.

Fora Soft builds video and AI software — 250+ projects since 2005, 50 in-house engineers, and enough scars from on-device inference to know where this goes wrong. If you are scoping emotion detection and want a second opinion before you commit a sprint to it, that is a conversation we enjoy having. We also build the surrounding product: AI integration into an existing platform, or a custom software build from scratch with a dedicated team.