Real-Time Anomaly Detection in Video Surveillance 2026

Why Fora Soft wrote this real-time anomaly detection playbook

We’ve built production video-surveillance and video-analytics products since 2005 — 250+ projects, 50 in-house engineers, and a computer-vision team that ships this class of system every week. We’ve been the sole development team behind VALT (video surveillance by Intelligent Video Solutions) for 10+ years: 770+ US organisations, 50,000+ users, HIPAA. We also built the web platform for NetcamStudio. So this page is the opinionated 2026 version of what we’d tell any CTO who asked “how do I do real-time anomaly detection without burning 18 months?”

The underlying tech moved fast in 2024–2026: vision-language models started matching weakly-supervised accuracy while explaining their calls, edge NPUs (Hailo-8, Jetson Orin) got cheap enough for fleet deployment, and the EU AI Act put real documentation duties on anything that watches a person in real time — then moved the compliance deadline. This playbook is built around the decisions that actually move a project (model family, deployment topology, dataset strategy, integration protocol, compliance posture) and the traps on each. For adjacent context, read our real-time video analytics and real-time video processing with AI guides, plus our ONVIF profiles in security systems primer and the deeper ONVIF Profile M write-up.

What real-time anomaly detection is worth in 2026

Short version: the money is real and growing, which is exactly why the buying bar is rising. The anomaly-detection market was $2.56 billion in 2023 and is projected to reach $10.5 billion by 2035 (Market Research Future, 2025). The broader video-analytics market sat at $12.33 billion in 2024 and is growing at roughly 22.6% a year (Precedence Research, 2025). Translation for a buyer: budgets exist, vendors are multiplying, and “we have AI cameras” is no longer a differentiator — latency, accuracy on your footage, and a defensible audit trail are.

That is the frame for everything below. We are not going to sell you on the category; we are going to help you right-size it and avoid paying for capability you can’t use.

What counts as an “anomaly” in video surveillance

The word “anomaly” is doing too much work in most RFPs. In practice it covers four different problem classes, and the model you pick depends on which one you mean:

• Behavioural anomalies — a person running in a restricted area, loitering, jumping a fence, climbing onto a platform. Temporal and motion models win here.
• Physical anomalies — an unattended package, a puddle on a factory floor, a fallen sign. Frame-level detection plus persistence tracking.
• Density / crowd anomalies — queue overflow, crowd crush, platform density. Heat-map regression plus threshold logic.
• Temporal / schedule anomalies — movement in an off-hours zone, a vehicle stationary too long. Often solved by rules plus a learned normal model.

Naming the class before picking a model is the single biggest win. We see RFPs every quarter that ask for “Transformer-based video anomaly detection” when the real requirement is persistent-object detection with a schedule overlay. Using the right tool saves 3–6 months and a chunk of the inference budget.

The real-time latency budget

Real-time here means “fast enough to act before the event ends.” That splits into three capability tiers, and the tier you need dictates the topology.

Figure 1. Latency decides capability. Under ~200 ms you can prevent; 200–500 ms you intercept; past that you only investigate.

The table puts concrete numbers on each row:

Table 1. Latency budgets and realistic deployment topologies. Network RTT adds 50–100 ms on WAN paths, so anything under 300 ms needs the inference on-site.

How real-time anomaly detection works in 2026

The pipeline is the same three steps it always was: learn what “normal” looks like for a camera, score each incoming clip against that baseline, and raise an alert when the score crosses a threshold. What changed in 2025–26 is how the scoring model is built — and what it can tell you.

For years the strongest results on the real-world benchmark, UCF-Crime, came from weakly-supervised Multiple-Instance Learning (MIL) on I3D video features: RTFM, MGFN and similar sit around 84–87% video-level AUC. In 2025–26, vision-language models caught up. Fine-grained prompting of Qwen2.5-VL-7B reaches about 89.8% AUC on UCF-Crime (WACV 2026), with Holmes-VAD (~87.7%) and HiProbe-VAD (~88.9%) close behind. Training-free approaches such as VERA and ASK-HINT push interpretability further without touching model weights.

Figure 3. On UCF-Crime, 2025–26 vision-language models match weak-supervision AUC and add a human-readable reason for each flag.

The accuracy delta is modest. The usable delta is not. A classic MIL model gives you a score: “0.87, anomalous.” A vision-language model gives you a score and a sentence: “a person is climbing the perimeter fence.” That rationale is what a tired operator at 3 a.m. needs to triage fast — and, as we’ll see below, it’s close to what an EU AI Act auditor will ask you to log anyway. Our honest position: run a fast, cheap detector at the edge for recall, and reserve the VLM for re-scoring and explaining the alerts that matter. We haven’t seen a real fleet where paying for VLM inference on every frame of every camera makes economic sense yet.

The model families that matter in 2026

Table 2. Model families and where they fit. “Edge feasible” assumes a Jetson Orin-class NPU with INT8 quantisation.

Datasets and benchmarks worth knowing

Benchmarks don’t translate to production — but they anchor conversations. The current 2025–26 reference numbers, with the inflated figures corrected:

• UCF-Crime — 1,900 untrimmed real-world surveillance videos, 13 anomaly classes (abuse, arson, robbery…), 290 test videos. Weakly-supervised SOTA is roughly 86–90% video-level AUC (not the ~97% some vendor decks quote). See our commentary on this dataset.
• XD-Violence — 4,754 untrimmed videos from movies and YouTube, 6 anomaly classes, 800 test videos, with audio. CLIP-based methods reach ~85% average precision.
• ShanghaiTech Campus — 437 videos of campus life. Reconstruction and frame-prediction methods report ~95–98% frame-level AUC (an easier, single-scene setting).
• Avenue — 37 short videos (16 train, 21 test), unusual pedestrian behaviour; ~85–90% AUC, a quick sanity dataset, not a production proxy.
• MSAD / newer 2024–25 sets — multi-scene, domain-shift benchmarks. AUC typically drops 10–15 points versus single-scene numbers — a useful reality check.

The honest read: a model scoring 88% on UCF-Crime will still need fine-tuning on 2–6 weeks of your actual site footage before customer-grade performance. Plan for that data-collection and relabelling cost upfront — it is the line item most RFPs forget.

Edge vs cloud vs hybrid — the deployment call

Figure 2. The hybrid pattern we ship: detect at the edge, re-score and audit in the cloud, fan out events over ONVIF Profile M and MQTT.

1. Edge only. A Jetson Orin, Hailo-8 or Coral module next to each camera. 4–16 concurrent 1080p streams per device, 50–200 ms inference, no WAN dependency, strongest privacy story (frames never leave the site). Hardware $200–$600 per unit. Best for small sites, regulated industries, poor connectivity.

2. Cloud only. Stream to an NVIDIA L4 / A100. 100+ streams per GPU, 20–50 ms inference, central retraining, hot model swaps. Cost $0.50–$3 per stream per hour on managed plans. Best for large centralised fleets with good WAN and a forensic focus. Not for <300 ms responses — the network round-trip eats the budget.

3. Hybrid (our default). Edge detects at high precision and moderate recall; the cloud re-scores borderline events (often with a VLM), owns the model registry, runs drift detection and archival. A typical split is 60–70% of alerts resolved at the edge at 95%+ precision, cloud catches the rest. This is the pattern the mature VMS integrations use in the field.

4. What drives the choice. Latency budget (see Table 1), camera count, site connectivity, compliance regime (edge-only is far easier under the EU AI Act), and dataset locality.

5. The anti-pattern. Pushing raw 1080p to the cloud for a latency-critical use case just to reuse an existing cloud GPU. You’ll pay in bandwidth, latency and regulatory exposure.

Edge hardware for on-camera inference in 2026

The edge tier got genuinely affordable, which is what makes edge-first realistic for a whole fleet rather than a flagship camera. Three tiers cover almost everything we deploy:

• NVIDIA Jetson Orin Nano Super — up to 67 TOPS (INT8), about $249, roughly 4 cameras per unit, sub-25 W. The default when you want CUDA tooling and headroom for a small VLM or re-scorer at the edge.
• Hailo-8 — ~26 TOPS at ~2.5 W (about 10 TOPS per watt), excellent for running several HD detection streams on a tight power and thermal budget. The trade-off is a specialised compiler and a narrower model zoo. Hailo-10 extends this toward generative-at-edge.
• Google Coral Edge TPU — ~4 TOPS at ~2 W. The lightweight tier for a single stream of classification or object detection where a full NPU is overkill.

Rule of thumb we use when sizing: budget one Jetson Orin Nano Super per 4 cameras for behavioural anomaly detection at 1080p, or one Hailo-8 board per 4–6 streams when power draw and enclosure heat are the constraint. Then leave 30% headroom — models grow, and a box running at 95% utilisation drops frames the first hot afternoon.

Integration protocols — ONVIF, RTSP, WebRTC, MQTT

Detection without integration is a demo. Four protocols carry a production system, and our ONVIF profiles in security systems primer goes deeper on the first:

• ONVIF Profile M. The metadata and analytics profile, broadly adopted by Milestone, Genetec, Avigilon, Hikvision and Axis. If your detector can’t emit ONVIF events, it won’t plug into enterprise VMS. See the ONVIF Profile M specification for the conformance details.
• RTSP. Still the default camera stream. 30–50 ms LAN latency, 100–300 ms WAN. Edge boxes pull RTSP, run inference, push events.
• WebRTC. 20–100 ms latency, better firewall traversal, stronger encryption. We reach for it in operator UIs when sub-second delivery matters — the same stack we use on Worldcast Live.
• MQTT. The alerting bus. The detector publishes event/shoplift/zone-5/conf-0.87; SIEM, ticketing (Jira, ServiceNow), access control and staff mobile apps subscribe.

Vendor pricing and build-vs-buy math

First, the honest pricing picture, because most vendor pages hide it. The AI-analytics licence alone runs about $3–$15 / camera / month in 2026; bundled cloud VSaaS (storage + analytics) runs $10–$60 / camera / month, with most business plans landing $10–$30 for 30-day retention. Named points: Spot AI is roughly $99 / camera / month with an NVR from $2,199 per location; Avigilon runs past $200 / camera and sells only through integrators; Ambient.ai and Verint price on request as an analytics overlay on your existing VMS and cameras. We compare the AI-native platforms in our AI-native VMS platforms explainer.

Table 3. 2025–26 indicative vendor pricing. Enterprise contracts vary widely and most vendors quote on request; treat these as order-of-magnitude, not quotes.

Figure 4. For 100 cameras, SaaS cost grows linearly while a custom build front-loads then flattens — crossover lands near 20 months.

The build-vs-buy crossover for a 100-camera deployment sits around 18–24 months. SaaS cost accumulates linearly with cameras; a custom build front-loads the engineering and then runs on amortised hardware plus ops. The chart uses conservative assumptions ($60 / camera / month SaaS versus an $80k build and $2k / month to run) and lands the crossover near month 20. Because we lean on agent-assisted engineering, our MVP cycles on custom anomaly detection tend to come in shorter than they did two years ago — but we still price conservatively and avoid numbers we can’t defend on paper. When you’re ready to own the stack, video surveillance development is where that work lives.

Privacy, GDPR and the EU AI Act in 2026

This is where deployments actually stall, and the rules changed in 2026, so read this before you scope. The short version: the obligations are real, but the standalone high-risk deadline moved to the end of 2027, and two duties already apply.

1. The EU AI Act timeline (as of 2026). The Digital Omnibus, adopted by the Council on 29 June 2026, pushed the high-risk obligations for standalone Annex III systems from 2 August 2026 to 2 December 2027 (AI embedded in regulated products moves to 2 August 2028). What did not move: the Article 5 prohibitions — including real-time remote biometric identification in public spaces — and the Article 4 AI-literacy duty have both applied since 2 February 2025, and the Article 50 transparency duty still starts 2 August 2026. Our EU AI Act glossary tracks the detail.

2. What high-risk obligations mean when they bite. Risk management, data governance, technical documentation, event logging, human oversight, accuracy and resilience targets, conformity assessment and post-market monitoring. We treat every one of those as an engineering artefact — model cards, data-flow diagrams, immutable audit logs — not legal boilerplate bolted on at the end.

3. GDPR Article 9. Biometric data is a “special category.” Face-based anomaly detection almost always triggers Article 9 and needs an explicit legal basis (rarely consent; more often substantial public interest or employment law).

4. GDPR Article 35 (DPIA), UK ICO and US state laws. Any meaningful-scale deployment needs a Data Protection Impact Assessment (budget 2–4 weeks of DPO time). UK live facial recognition is effectively off-limits for private operators; Illinois BIPA, Texas CUBI and California CCPA add consent or disclosure burdens.

5. Practical consequences. Prefer gait, posture or object-centric detection over face-based where you can. Keep frames on-site when latency allows. Log every detection with its confidence, the reviewer ID and the disposition — that audit trail is your legal evidence of human oversight, and it is exactly the record a VLM rationale helps you produce.

Use cases that pay for themselves

1. Retail shrink reduction. Paired with trained floor staff, real-time shoplift alerts reduce shrink meaningfully in the first year; a common pattern we’ve shipped matches point-of-sale data to video and flags mismatches (fewer items scanned than bagged) within 30 seconds. See our retail video analytics deep-dive.

2. Transit safety. Platform-edge and fall detection at <200 ms enables automated train-hold or gate-close. Several metro systems added anomaly-detection overlays to existing CCTV between 2022 and 2025.

3. Perimeter security. Fence-climb and vehicle-intrusion detection at remote sites; a 90–95% true-positive rate at <10 false alerts per 1,000 hours is achievable with current models and honest tuning.

4. Industrial safety (PPE, spills, behaviours). YOLO-class object detection plus anomaly logic; OSHA-grade reporting and real reductions in recordable incidents in the deployments we’ve seen.

5. Elderly fall detection in care facilities. Edge inference on Jetson Orin; high sensitivity with <500 ms to a staff alert. Healthcare privacy regimes (HIPAA in the US) demand on-site storage and tight retention — exactly the constraint VALT was built around.

Mini case: surveillance analytics shipped without launch drama

Situation. An enterprise surveillance vendor needed an anomaly-detection overlay across multi-camera sites ahead of a major platform update. The dev team had green CI, but the proposed model (Transformer-based, cloud-only) would have missed the 300 ms latency budget on the firm’s WAN, and regulator-grade audit trails were a hard requirement.

12-week plan. Swap to a hybrid architecture: quantised I3D + weakly-supervised MIL head at the edge (Jetson Orin), a cloud re-scorer for borderline events, an ONVIF Profile M event bus for VMS integration, and MQTT fan-out to on-site staff and the central SOC. Drift monitoring on rolling 7-day cohorts. A formal DPIA and AI Act documentation ran in parallel from week one. NetcamStudio is the class of surveillance UI we typically ship on top.

Outcome. Rolled to multi-site production in 12 weeks. End-to-end latency landed in the 180–220 ms band, inside the 300 ms SLA. The false-positive rate after four weeks of fine-tuning came in under 3% for the top-priority anomaly classes. The audit trail passed the regulator’s follow-up inspection with zero findings, and the client kept the regression programme on retainer. Want a similar assessment? Book a 30-minute call.

A decision framework — right-sizing in five questions

1. What is the latency budget? Under 200 ms forces edge; 200–500 ms allows hybrid; over a second is cloud and batch.

2. Which anomaly class are you solving? Behavioural, physical, density or temporal — each wants a different model family. Put the class in the RFP, not “Transformer.”

3. How many labelled examples do you have? Fewer than 100 → start with a vision-language / zero-shot detector. 100–1,000 → weakly-supervised MIL. More than 10,000 → fully supervised.

4. What is the compliance regime? EU AI Act, HIPAA, GDPR, state biometric laws — each shapes where frames can live and what metadata you must log.

5. Build, buy or hybrid? Fewer than 20 cameras or a sub-18-month horizon → SaaS. More than 100 cameras, niche anomaly classes, regulator-grade audit → custom, often wrapped around your existing VMS.

Five pitfalls we keep seeing in anomaly-detection programmes

1. Chasing benchmark accuracy instead of site accuracy. An 88% UCF-Crime model rarely survives first contact with your cameras. Budget 2–6 weeks of site data collection and fine-tuning.

2. Ignoring false positives. Even 1% FP at 30 fps on 100 cameras is tens of thousands of alerts a day. Ensembles, confidence thresholds and human-in-the-loop review are not optional.

3. Under-engineering drift detection. Seasonality, time-of-day and site changes move the distribution silently. Monthly retraining and rolling-window drift checks belong in the runbook on day one.

4. Assuming cloud-only works for real-time. WAN round-trip plus GPU inference plus alert fan-out rarely fits under 300 ms in production. Edge-first is usually the right call.

5. Treating the AI Act and DPIA as after-the-fact paperwork. They are engineering requirements. Map them to data flows, logs and model-card artefacts from week one, and the moved 2027 deadline becomes runway, not an excuse to skip them.

KPIs that prove the system is working

1. Quality KPIs. Frame-level AP per class ≥ 0.85 on site data; recall on top-priority classes ≥ 0.9; false-positive rate ≤ 5 per 1,000 hours on reviewed alerts; drift ΔAUC < 2 points week-on-week.

2. Business KPIs. Time-to-intervention for flagged events; documented incidents prevented; shrink or loss reduction versus baseline; occupant NPS on safety perception.

3. Reliability KPIs. p95 end-to-end latency against the target budget; model-registry uptime; edge-box crash-free days; alert-delivery SLO ≥ 99.9%.

When NOT to do real-time anomaly detection

• No one will act on the alerts. Without a response team, alerts pile up and credibility collapses. Fix staffing first.
• Latency budget is minutes, not milliseconds. Offline or batch analytics is far cheaper and often more accurate.
• Ambiguous anomaly class. If you can’t name the behaviour in one sentence, you won’t be able to label it either.
• Camera quality is the bottleneck. 360p dome cameras and fish-eye lenses cap accuracy below what any model will deliver.
• Compliance will veto it. If the use case violates biometric consent or local AI Act rules, invest in non-AI mitigations instead.

FAQ

What latency is realistic for real-time anomaly detection?

End-to-end budgets we hit in production: 150–250 ms on edge (Jetson Orin / Hailo-8) for I3D or MIL-class models; 300–500 ms for hybrid edge+cloud; 500 ms+ for cloud-only Transformers. Anything claimed under 100 ms usually means camera-to-alert on a heavily quantised light model.

Do I need fully labelled data?

Usually no. Weakly-supervised methods (MIL, RTFM, MGFN) learn from video-level labels and land within a few points of fully-supervised AUC on UCF-Crime. Vision-language detectors can bootstrap from a handful of examples. Full frame-level labelling is mostly a regulated-path requirement.

Edge, cloud or hybrid?

Hybrid by default. Edge for latency and privacy; cloud for the model registry, drift, re-scoring borderline alerts and audit storage. Pure cloud is fine for forensic analytics; pure edge is fine for small regulated deployments. Pure anything else is usually a cost or SLA mistake.

How much does it cost to build vs buy?

The AI-analytics licence alone is about $3–$15 per camera per month; bundled VSaaS runs $10–$60. Named vendors: Spot AI ~$99/camera/month, Avigilon $200+/camera. A custom build front-loads engineering and hardware and amortises at roughly 18–24 months for 100 cameras. Above that scale, custom is usually cheaper over three years.

Is real-time anomaly detection compliant with the EU AI Act in 2026?

It can be, and you have more runway than the old headlines suggest. The Digital Omnibus (adopted June 2026) moved high-risk obligations for standalone Annex III systems to 2 December 2027. But Article 5 bans (including real-time remote biometric ID in public spaces) have applied since February 2025, and Article 50 transparency duties start 2 August 2026. Build the documentation, logging and human-oversight artefacts now; certify before the 2027 deadline.

What is ONVIF Profile M and do I need it?

Profile M is the ONVIF analytics and metadata profile — a standard way for detectors to publish events and metadata to VMS such as Milestone, Genetec and Avigilon. If your detector doesn’t speak it, enterprise-VMS integration becomes custom work.

How do I manage false positives at scale?

Three levers: higher confidence thresholds (trade recall for precision on non-critical classes), ensemble models voting, and human-in-the-loop review on the top-N daily alerts with feedback flowing back to retraining. Expect 4–8 weeks of tuning before alert fatigue drops to an acceptable level.

How often should I retrain?

Monthly incremental retraining on newly labelled cohorts; a full retrain each quarter on the last 6–12 months of data. More often in seasonal environments (retail, transit, outdoors). Instrument drift detection (PSI, KS-test on feature distributions) and let it trigger retraining rather than running on a calendar.

What to Read Next

Ready to ship real-time anomaly detection that actually scales?

The 2026 version of this problem is less about “can ML do it” and more about picking the right combination of model family, deployment topology, integration protocol and compliance posture for your cameras, your latency SLA and your regulator. The math favours hybrid edge+cloud stacks; the compliance bar rose, then its deadline moved to 2027; and the model options are genuinely better than they were 24 months ago — vision-language models now explain their own alerts.

If you read the latency-budget row first, picked the anomaly class honestly, wired ONVIF Profile M in early, kept frames on-site where the regulator demanded it, and instrumented drift from day one — you have most of what it takes. The rest is taste and iteration. Our video-surveillance knowledge hub has the deeper reference material when you want it.

If that sounds like the next 12 weeks of your roadmap, we’re happy to help — one review session, a discovery week, or running the build.