AIBridge Streaming failure when streaming over 25-30 Cameras

Summary:

When more than ~25-30 cameras are fetched from an AIBridge instance, the connector becomes unable to relay RTSP streams from the VMS recording server. RTSP DESCRIBE succeeds but SETUP times out. The issue resolves when camera count is reduced below ~25.


Observed Behavior:

  1. Streaming container receives RTSP SETUP requests from clients and forwards them to the connector for VMS stream relay
  2. Connector container logs repeated errors: “Error receiving command from destination” → “stream started, but no one requested it”
  3. Streaming container logs: “RTSP SETUP request using TCP handled with error” → “waiting for response timed out”
  4. RTSP DESCRIBE always succeeds, and following RTSP SETUP call fails with Timeout.

Key observations:

  • Container restarts do not resolve the issue — the overload recurs immediately as clients reconnect
  • Observed reducing camera count from ~32 to ~20 immediately resolves all failures
  • The connector appears to have a concurrency ceiling around 20-25 simultaneous streams

Questions:

  1. Is there a known concurrency limit for AIBridge for simultaneous stream relay operations?
  2. Is there a configuration to increase the AIBridge’s stream relay capacity or parallelize SETUP handling?

Error messages:

aibridge-connector:

30 June 2026, 19:43
{"time":"2026-06-30T14:13:37.598836273Z","level":"ERROR","msg":"Error receiving command from destination (address: '172.17.0.7:50051')","error":"rpc error: code = Unknown desc = stream started, but no one requested it"}


aibridge-streaming:

30 June 2026, 19:43
{"time":"2026-06-30T14:16:17.229630805Z","level":"ERROR","msg":"[ /8cfb2b88-0cb9-4285-a798-850493b60020/28dc44c3-079e-4c94-8ec9-60363451eb40 ] RTSP SETUP request using TCP handled with error","error":"waiting for response timed out"}

Environment:

  • AIBridge version: 3.0.0 (connector image: connector-3.0.0)
  • VMS: XProtect Corporate 2024 R2 (with 7 Recording Servers)

Hi @ihmvisualreasoningvms ,

There’s not a cap on the connector that should affect the RTSP streams.

In the meantime I wanted to ask you if:

  1. Have you seen this issue only on 3.0.0 ?
  2. Can you test on your environment 2.0.5 and 2.0.4 ?
  3. Are you running on a compose or k8s environment ?
  4. Can you check RAM and CPU when you reach ~25-30 rtsp streams ?
  5. How much RAM does the AIB host has ?

We will also review this internally. But your feedback on your specific setup could provide more info.

If possible please attach the log files on a zip (per each version of AIB that you test) so we can also take a look at it.

  1. Yes the same error were seen on the previous versions of AIBridge.
  2. Unfortunately, We wont be able to test back older versions as we were facing more problems with it. And our’s is a production system where we have high numbers of cameras.
  3. We are running aibridge containers on our hardware using AWS ECS.
  4. RAM and CPU both are under the limits.
    4.a) We have total 11GB memory assigned to AIBridge of which only 16-20% is used.
    4.b) CPU usage is also under 16% during this time. There is nothing else running on this hardware.
  5. Hardware has 61.37 GiB RAM.

Hardware details: NVIDIA Jetson AGX Orin 64GB

Hi,

can you describe the system / network architecture; where is XProtect deployed, where is AI Bridge deployed, where is your Analytics App deployed?

AI Bridge is tested to run on Docker Compose and Kubernetes installations. AWS ECS is not tested and we have seen issues with the combination in other installations.

The log files will still give a good input to the review.

XProtect is running on Device1, AIBridge and Analytics app are running on Device2 in the same local network under one roof. They are all connected locally through wired connection.

Our hardware deployments are managed through AWS ECS. Once the AIBridge service is deployed to the hardware through AWS ECS the device has docker pre-installed using which we run the AIBridge containers.

We have successfully run AIBridge and are able to stream cameras through it. We were able to successfully run more than 30 cameras/AIBridge instance for days. Which later after some days of streaming suddenly fails with the same error as attached below.

Attaching the failing logs below:
(Note: You can ignore the initial “Error in routing stream“ errors, those were the streaming calls from clients when AIBridge was getting initialized)

aibridge-connector-logs.csv (51.9 KB)

aibridge-kafka-broker-logs.csv (297.9 KB)

aibridge-kafka-zookeeper-logs.csv (20.1 KB)

aibridge-streaming-logs.csv (45.3 KB)

Had my AI Agent debug the issue. Attaching below its analysis just in case. Do you think this might be the problem?

Let me consider. What could put the connector in a permanent bad state that:

Sometimes happens, sometimes doesn’t (non-deterministic)

Restart sometimes fixes (new roll of the dice)

Restart sometimes doesn’t fix (same bad luck)

Reducing cameras fixes (reduces load below some threshold)

Adding cameras back works after (once in good state, it stays good)

Wait — that last point is key. Once we reduce cameras and it stabilizes, we add them back and it keeps working. That means 31 cameras IS within capacity when the connector is in a good state. But during startup, if too many cameras
try to SETUP simultaneously before any are established, it enters a bad state it can’t recover from.

This sounds like a deadlock or resource starvation during the initial burst:

Task starts, all 31 cameras try SETUP within the jitter window (55s)

If too many hit the connector simultaneously, it fills some internal queue/buffer

The filled queue causes timeouts → VMS pushes back unrequested streams → “no one requested it” flood

The flood keeps the queue full → new requests can never get through

Self-sustaining failure loop — it never recovers because the retry storm maintains the overload

This explains everything:

Reducing cameras breaks the loop (fewer retries = queue drains)

Adding them back works (they trickle in one at a time, not all at once)

Restart sometimes works (different jitter timing = fewer simultaneous SETUPs at startup)

Restart sometimes doesn’t work (same bad luck with timing)

The root fix would be: Have the connector reject/queue excess requests gracefully instead of entering a permanent failure loop.

We retry the streaming failures with random jitter (1-55 seconds).

Team, Could you find anything on this?

Hi @ihmvisualreasoningvms ,

We have been able to reproduce this issue in our lab.

We are currently working on a fix for it. It will be part of the next drop of AI Bridge (3.0.1).

Thanks for reporting it and providing all the log details.

Thank you so much team.
What would be the ETA for this fix?

Quick Question: Do you know why was this sometimes getting fixed on AIBridge restarts and sometimes not? Sometimes reducing number of streamed cameras recovers it and then we can add it back without any issues.

We are pleased to announce that Milestone AI Bridge 3.0.1 is planned for release on Tuesday, July 28, around 12:00 PM CEST.

In regards of the issue, it is due to some race-conditions that were present in the Connector container.

Please check the release notes next week and let us know if it fixed your installation.

BR

The issue seems to have resolved after the 3.0.1 update. Thanks team.

1 Like