AI Bridge Docker 2.0.5 doesn't allow a certain amount of metadata devices to be created

Vince · May 8, 2026, 4:23pm

I’ve decided to switch from Kubernetes to Docker for the AI Bridge 2.0.5. It is much more stable and manageable.

I’m running into an issue though where if there are so many metadata devices created, that the recording server crashes due to some sort of SQL/error/storage issue. I have yet to confirm what is exactly the issue. But I can replicate it by adding a certain amount of meta data devices per camera and at a certain point the recording server just crashes. We have a very fast raid array attached to the server with over 1 TB of storage and 128Gbs of ram. So this isn’t associated directly with actual hardware specifications.

Yesterday (5/7/2026) we called Milestone support on this (case # MSC00088667), so there is history of the situation. The recording server was in cycle reboot until we started deleting metadata devices. We have close to 400 cameras, and each camera has a metadata device subscribed to a topic. I dumped some logs into AI - The best it could configure - “A camera driver attempted to push a video frame into a PipelineQueue<T> where either the queue itself or its target buffer was null.”

Not sure if the processing server was built that way, to handle only a certain amount of metadata devices or it is a bug, or some configuration we have to change.

Any help on this would be greatly appreciated.

ferguzman · May 11, 2026, 2:45pm

Hello @Vince,

The short answer is no, AI Bridge is not designed in anyway to prevent or limit the amount of neither the metadata devices nor the metadata streams.

The logs you mention refer to the operation of the Recording Server an how it deals with devices. Since the lines you are mentioning refer to a camera pushing video but you are pushing metadata, I wonder if these are related to your metadata issue.

Do you see any logged errors from the Processing Server side? Please, share your logs if you find anything that catches your attention.

Regards,

Fer

Vince · May 12, 2026, 4:04pm

Hi @ferguzman ! thanks for your response. Attached is two files. One is logs of the recording database. These logs start around 9:40am - (but was the same starting around 8am when the problem started happening). But you can see in the logs that the recording server database was in a cycle of rebooting and creating new tables. This cycle of creation/deleting in the DB stopped as soon as I started deleting metadata devices where the problem first happened.

The second log is from devicehandling (I think this is what your referring to as processing server logs?) if not, let me know what specific log may help.

The issue started happening around 8am or just a bit before or after on 2026-05-07. I added a metadata device for a series of cameras called Yarbs. I added about 12 or so. Then when I did the next one, a warning popped up and said lost connection to recording server. After that point - the recording server was in a constant state of starting then stopping until those newly added metadata devices were deleted.

The DeviceHandling log shows a snippet from about 7:50am to 8am.

The specific log in question is on line 1960 - I believe this is a breadcrumb of the issue. There were 3 other, but the log is too big to upload here.

System.NullReferenceException: Object reference not set to an instance of an object.

at VideoOS.Recorder.ObjectModel.PipelineQueue`1.PutFrameInQueue(T newFrame)

at VideoOS.Recorder.ObjectModel.Pipeline.ThreadFunctionPart1()

I really can’t subscribe to any more metadata devices without risking our recording server going down. So please let me know what you need or if any other logs can help

DeviceHandling1.log (435.4 KB)

Database_2026_05_07-1.log (9.8 MB)

ferguzman · May 13, 2026, 3:31pm

Hi @Vince,

Thanks for sharing the logs, I can see the errors on the Recording Server side - on the ingestion of metadata to the VMS using the VPS drivers.

Can you please check on the AI Bridge Docker container logs? A good start would be to check at the same timestamps you are seeing the entries on the DeviceHandling.log file. The AI Bridge containers for this investigation are connector, proxy and broker.

Let me know your findings.

Vince · May 13, 2026, 8:09pm

I don’t see anything that stands out to me in regards to this. Here are the files I parsed in that time frame. I did notice for proxy and broker logs there are no yarbs. which makes sense. and connection errors in these logs coincide with the recording server being down.

connector1.log (7.6 KB)

proxy1.log (1.7 MB)

broker1.log (65.0 KB)

ferguzman · May 14, 2026, 10:29am

Thanks for sharing the logs, @vince.
We reviewed both recorder-side and AI Bridge-side logs. The strongest failure signal is still the recorder side: the Recording Server crashes with System.NullReferenceException in VideoOS.Recorder.ObjectModel.PipelineQueue<T>.PutFrameInQueue, and the restart loop stops when metadata devices are removed.

AI Bridge is able to create large numbers of metadata routes and forward streams, so it does not appear to be enforcing a metadata-device limit, as mentioned before.

At this point the evidence points primarily to a Recording Server defect or scale issue in metadata pipeline handling, with AIB possibly acting as a trigger rather than the root cause.

Vince · May 14, 2026, 3:15pm

Thanks for your response @ferguzman . And the most I can make sense of the error “System.NullReferenceException“ is some malformed data the recording server did not like. But I have 29 other versions of that IVA app running and ingesting the same metadata format, if something came in abnormal from the IVA app, it would reject it before trying to format to the correct onvif frame for AIB ingestion, and even at that, it wouldn’t probably break anything, it would just silently fail. So that leaves me to believe it was the metadata device created.

I suppose I can go forward again, and slowly re-create the metadata devices and see what happens.

I did notice you guys have a newer version of AI Bridge 3.0.0. I was wondering if I updated to that version, if that would fix this issue, although the release notes don’t allude to anything like that.

Is it possible to get some insight into how metadata devices react with a new bridge version?

currently I’m able to bring down the AIB container, then bring it back up, restore the IVA containers and the metadata devices on the management client still work and can receive in metadata still.

I was wondering if I was to update to a completely new version, which would require all existing libraries of the bridge to be wiped clean and start all over. Then I would re-create the IVA containers with all the same ID’s, would the metadata devices work still? The VPS address should be the same.

I ask because its a time-consuming process to create metadata devices through the UI management client for each camera when you have hundreds. And currently I can search, enable and disable existing metadata devices through Milestone PS tools, but I can’t programmatically create/subscribe to metadata devices, at least from what I’ve found so far.

So I would love for our AIB to stay up to date with the current version, but also I’m weary of breaking metadata devices on the front end, and having to create them all over.

Your knowledge or insight on this thought would be greatly appreciated.

ferguzman · May 18, 2026, 3:05pm

Hello @Vince,

Indeed, there is a new release of AI Bridge 3.0.0 but this newer version will not solve the problems you are facing with the metadata subscription on the Recording Server.

For a more detailed view of what is included you can check the release notes for the 3.0.0 version. You may find that there is no modificacion on the metadata devices behavior and as long as the identifiers and the enpopoints remain the same, you will not need to re-subscribe to topics on each device after the upgrade.

Even though the upgrade is not going to solve the metadata issues, it is recomended to always run the latest version, just have in mind what is mentioned on the release notes about the EOL of ingress-nginx and the need of configuring a storage provisioner for Kubernetes setups.

Regards,

Fer

Vince · May 18, 2026, 4:22pm

Thanks for your response @ferguzman . I’ll move forward to updating to 3.0.0

have a great day

Vince · May 19, 2026, 3:12pm

@ferguzman I’m not sure if I should have created a new post. but this may be related. I tried updating to 3.0.0 - and I’m getting this now on the front end UI when I click on the bridge. (processing server)

===================================

An error occurred while parsing EntityName. Line 1, position 2387. (System.Xml)

Program location:

at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseEntityName()
at System.Xml.XmlTextReaderImpl.ParseEntityReference()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at VideoOS.Administration.AddIn.PlatformConfigurationManager.GetOptionsConfiguration(Guid toolsOptionsPluginId)
at VideoOS.Administration.AddIn.XPCOConfiguration.GetOptionsConfiguration(Guid optionsDialogId, Boolean userPrivate)
at VideoOS.ProcessingServer.Plugin.Admin.UI.TopicSubscriptions.Load()
at VideoOS.ProcessingServer.Plugin.Admin.UI.ProcessingServerUserControl.FillContent(Item item)
at VideoOS.ProcessingServer.Plugin.Admin.UI.ProcessingServerItemManager.FillUserControl(Item item)
at VideoOS.Administration.AddIn.UserControlPlatformInfo.InitCheckAndFillUserControl(Item item)

I also tried reverting back to 2.0.5 and I’m getting the same thing. Also any logs from the back end looks good. I can access the graphql explorer and GQL API.

This seems to be a processing server / front end issue? I also downloaded the correct processing server version as well.

any suggestions to fix this? I tried looking at device handling log and didn’t see any connection to the issue.

ferguzman · May 20, 2026, 8:55am

@Vince I’d say the topics subscription database entries got corrupted, you will not find any problems on the Recording Server nor AI Bridge backend, now the problem is on the topics subscription data, still is not related to your original post, metadata devices and Recording Server restart loops.

I would suggest to start over again with the AI Bridge topics subscription and the metadata devices creation bu first, let’s try to dig a bit more on this particular problem.

What version of XPCO are you using?
What is the Processing Server plugin version?
Do you have access to the database to run queries?

Vince · May 21, 2026, 2:10pm

@ferguzman - 2025 R3

Processing Server 3.0.0

Yes, I can get access to query database

Update: Originally I had a container that couldn’t register, which when troubleshooting it, is when I found the error on the management client side. So the bridge broke completely. But I did end up re-registering the bridge again with 3.0.0. But the error was still there. So I just stopped and just thought everything was broken.

So I assumed since the issue is with the processing server, the metadata devices were broken. But just out of curiosity I started up all my containers (30) - and they all registered properly and the metadata devices are still working. I’m receiving metadata on all 500 something metadata devices.

But the error is still there. Which doesn’t allow me to see the container names and the endpoint of my app, which shows version and the metadata being received. And also more importantly, doesn’t allow me to register any new cameras to any metadata stream, as when I click on processing server tab in camera options - the same error pops up, but it actually crashes the management client

Just thought you would like to know this odd behavior.

ferguzman · May 26, 2026, 4:34pm

Hi @Vince, this is so hard to reproduce, I’ve been trying but I cannot reach your scenario, even after upgrading from different versions.

This error means your topic subscription database is broken. Data is encrypted and is not easy to rebuild. I would suggest you, unfortunatelly, to recreate the whole database - yes, you will need to re-subscribe to all devices/topics:

Delete all the metadata devices that have been created by the former topics subscription - just to start clean since you will need to re-subscribe to them.
Navigate to Processing Servers, select the node, right click and click “Unregister” on the context menu.
Docker compose down AI Bridge and up again.
Wait for the init container to finish.
Start subscribing the topic/device again.

I’m really sorry you need to go through this process again, but since the database is now corrupted, it cannot be recovered.

Regards,

Fer

Vince · May 26, 2026, 4:54pm

I figured, and actually that’s what I’m in the process of doing now. I can’t even access the processing server to un-register off the topic. I’m having to just delete the metadata devices themselves. I’ve already un-registered the bridge as well.

I started running through all the scenarios, and when I updated from 2.0.5 to 3.0.0, I’m not 100% sure I updated the processing server plugin to 3.0.0, it might have been left on 2.0.5 , the names are identical, and I might have pulled the plugin off the wrong folder.

So I’m just starting completely fresh. And since you have a testing environment, maybe you can confirm that was the issue as well. So purposefully running AI Bridge 3.0.0 but with processing server plugin 2.0.5.

Also maybe you can put in a suggestion for the AI bridge to be able to programmatically subscribe to the topic without having to use the milestone management client GUI, either through the REST API, or Milestone PS tools.

Thanks for your help and research

Vince · May 26, 2026, 8:39pm

@ferguzman - So I’ve deleted everything associated with the processing server plugin, I made sure there are no other clients that are running Management client with the processing server plugin. And I’m still getting the same error. At this point I’ve have removed all metadata devices. I even tried rolling back to 2.0.5, still the same error.

I need to fix this. I can’t imagine even if I used the wrong processing server version, it would not break this hard. Please any help would be appreciated.

===================================

An error occurred while parsing EntityName. Line 1, position 2387. (System.Xml)

Program location:

at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseEntityName()
at System.Xml.XmlTextReaderImpl.ParseEntityReference()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at VideoOS.Administration.AddIn.PlatformConfigurationManager.GetOptionsConfiguration(Guid toolsOptionsPluginId)
at VideoOS.Administration.AddIn.XPCOConfiguration.GetOptionsConfiguration(Guid optionsDialogId, Boolean userPrivate)
at VideoOS.ProcessingServer.Plugin.Admin.UI.TopicSubscriptions.Load()
at VideoOS.ProcessingServer.Plugin.Admin.UI.ProcessingServerUserControl.FillContent(Item item)
at VideoOS.ProcessingServer.Plugin.Admin.UI.ProcessingServerItemManager.FillUserControl(Item item)
at VideoOS.Administration.AddIn.UserControlPlatformInfo.InitCheckAndFillUserControl(Item item)

Vince · May 26, 2026, 10:25pm

I queried the database for the user that is assigned to create the AI bridge. which is user “AIB”
the last time the configuration setting was changed by AIB was on the same date and time of when the first error happened. And found the IDCustomSetting that is associated with the error

Vince · May 27, 2026, 11:08pm

@ferguzman - UPDATE:

I ended up setting up a testing environment, I was unable to re-produce the error as well on my end. I even tried running 3.0.0 in the backend with processing plugin server 2.0.5, (just in case I forgot to update it) and still was working, with no errors or issues.

I was able to dig into the stack trace that led me to this file ID 8c2ce0e4-d755-4171-95e1-1154a67f83d1

I found what I believe is the same database line in preprod as in the production. I was able to remove it in the testing environment, with no negative consequences to the system. It was regenerated automatically when I subscribed to new metadata topic.

Tomorrow its scheduled to remove the corrupted database line in the production system, and will update if it fixed the issue.

I have backed up that corrupted database line, and interested in de-encrypting it to see what was the issue.

Vince · May 28, 2026, 2:26pm

Just completing this post, showing the fix to the issue in case anyone else runs into this problem.

Indeed the error shown earlier in this post can be fixed my removing the corrupted configuration for subscriptions for processor plugin in the database file. After removing the IDCustomSetting 8c2ce0e4-d755-4171-95e1-1154a67f83d1 from the database the configuration issue went away, this was in connection to the error TopicSubscriptions.Load().

cheers