Using the AI Bridge (k8s) - After a VMS restart have to re-apply the topic subscriptions

Vince · April 15, 2026, 8:30pm

When the VMS system restarts, to get the connection going again, I have to un-click the App subscriptions under Processing settings, save, then enable again and save. Doesn’t seem like this is the way it should work, especially when you have 300+ cameras. Every command on the k8s end shows stream isn’t connected. And even after a fresh-deployment of a pod, still doesn’t work until I re-subscribe through the UI. How can I fix this?

HansK · April 16, 2026, 7:32am

Does this happens for all three topic types (Metadata, Video and Analytics Events)?

As we have discussed Metadata before, I wonder what the state of the VPS Drivers are on the VMS. Does the logs reveal anything?

Please share what version of the VMS and AI Bridge you are experiencing this with.

Vince · April 16, 2026, 3:25pm

Right now I’m only doing metadata. I’m also using the latest AI bridge 2.0.5. and VMS version 2025 R3.

for DeviceHandling.log - this is when the system was shutdown - I currently have 4 different (pods/topics) The other 3 are identical to this output - This just showing one metadata topic registered

When system came back online:

At this point, I noticed no bounding boxes coming through, ran some logs on k8s end:

kubectl logs -n aibridge deploy/aibridge-aibridge-connector --tail=200

2026/04/15 18:57:54 Could not observe endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:57:59 Could not observe endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:57:59 Could not get services from endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:57:59 Could not get unregistration requested app ids by endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:57:59 Could not observe endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:06 Could not observe endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:06 Could not get services from endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:06 Could not get unregistration requested app ids by endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:06 Could not observe endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:09 Could not observe endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:09 Could not get services from endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:09 Could not get unregistration requested app ids by endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026/04/15 18:58:09 Could not observe endpoint: Post “https://172.16.0.247/ManagementServer/ServiceRegistrationService.svc”: dial tcp 172.16.0.247:443: connect: no route to host 2026-04-15T18:58:20Z Recording Server joined pool of endpoints to get status from (http://recordings.wifieye.local:7563/recorderstatusservice/recorderstatusservice2.asmx) 2026-04-15T18:59:35Z Status session established with Recording Server (http://recordings.wifieye.local:7563/recorderstatusservice/recorderstatusservice2.asmx)

which is during the time the server was shutdown, and at the end of the log it shows it connected.

kubectl logs -n aibridge deploy/aibridge-aibridge-proxy --tail=50

2026-04-15T18:12:33Z Forwarding metadata in onvif_analytics format from source 7dabe6a4-c0bd-4619-8205-4cb0970dfbf4/28dc44c3-079e-4c94-8ec9-60363451eb40 (gRPC / REST) to VPS connection started 2026-04-15T18:12:48Z Forwarding metadata in onvif_analytics format from source 7dabe6a4-c0bd-4619-8205-4cb0970dfbf4/28dc44c3-079e-4c94-8ec9-60363451eb40 to VPS connection stopped after 2 frames (did not get data for 14.677360588s) 2026-04-15T18:13:49Z Forwarding metadata in onvif_analytics format from source 7dabe6a4-c0bd-4619-8205-4cb0970dfbf4/28dc44c3-079e-4c94-8ec9-60363451eb40 (gRPC / REST) to VPS connection started 2026-04-15T18:14:09Z Forwarding metadata in onvif_analytics format from source 07b12fd9-0f86-42ac-b07c-949f0fc9eba7/28dc44c3-079e-4c94-8ec9-60363451eb40 (gRPC / REST) to VPS connection started 2026-04-15T18:14:22Z Forwarding metadata in onvif_analytics format from source 07b12fd9-0f86-42ac-b07c-949f0fc9eba7/28dc44c3-079e-4c94-8ec9-60363451eb40 to VPS connection stopped after 6 frames (did not get data for 11.377487071s) 2026-04-15T18:15:13Z Forwarding metadata in onvif_analytics format from source 7dabe6a4-c0bd-4619-8205-4cb0970dfbf4/28dc44c3-079e-4c94-8ec9-60363451eb40 to VPS connection stopped after 257 frames (did not get data for 13.001224688s) 2026-04-15T18:15:24Z VPS metadata connection closed (metadata.7f136bab-b628-4ccb-97e6-4bd850a93c5b.superior-soil-objects.onvif_analytics.418237df-b0cd-48f7-98e2-ae89e1f2d148_28dc44c3-079e-4c94-8ec9-60363451eb40) 2026-04-15T18:15:24Z VPS metadata connection closed (metadata.261dab1d-03d5-4bb6-b2fb-00c1d0a0ed24.psc-objects.onvif_analytics.9387bcff-735c-4833-a24b-9ed43719d484_28dc44c3-079e-4c94-8ec9-60363451eb40) 2026-04-15T18:15:24Z VPS metadata connection closed (metadata.e2fcb25e-f2df-4267-baf1-34070ce47ffc.svs-objects.onvif_analytics.07b12fd9-0f86-42ac-b07c-949f0fc9eba7_28dc44c3-079e-4c94-8ec9-60363451eb40) 2026-04-15T18:15:25Z VPS metadata connection closed (metadata.571e2c8e-ca99-41af-9170-988c6f844a8c.pacwest-objects.onvif_analytics.7dabe6a4-c0bd-4619-8205-4cb0970dfbf4_28dc44c3-079e-4c94-8ec9-60363451eb40)

even though it shows the connector re-established connection, the VPS connections never connected.

Checking into the broker - no routes were ever re-established -

kubectl logs -n aibridge deploy/aibridge-aibridge-broker --tail=100
Component: Broker
Built: Mon Mar 30 15:21:22 UTC 2026
GoVersion: go1.24.1
-brokers aibridge-aibridge-kafka-broker:29092
-grpc-port 8383
-log-file-enabled false
-log-max-age 15
-log-max-backups 15
-log-max-size 100
-rest-port 8382
-stream-subscriptions-topic voyager.topics.daim.vmsbridge.stream_subscriptions

2026-04-14T19:58:15Z Server starting …
2026-04-14T19:58:15Z Verifying existence of topics in kafka cluster …
2026-04-14T19:58:16Z Creating kafka cluster admin failed (will retry in 5 seconds): kafka: client has run out of available brokers to talk to: dial tcp 10.106.131.141:29092: connect: connection refused
2026-04-14T19:58:21Z Topic voyager.topics.daim.vmsbridge.stream_subscriptions does not yet exist; will wait 5 seconds and check again
2026-04-14T19:58:26Z Verifying existence of topics in kafka cluster succeeded
2026-04-14T19:58:26Z Creating new kafka producer …
2026-04-14T19:58:26Z Creating new kafka producer succeeded
2026-04-14T19:58:26Z Creating new kafka consumer for group ‘8531cca9-281c-48b3-9146-b8d8bf108e3d’ …
2026-04-14T19:58:26Z Creating new kafka consumer succeeded
2026-04-14T19:58:26Z Server started
2026-04-14T20:32:47Z New active routes (1 added)

metadata.e2fcb25e-f2df-4267-baf1-34070ce47ffc.svs-objects.onvif_analytics.07b12fd9-0f86-42ac-b07c-949f0fc9eba7_28dc44c3-079e-4c94-8ec9-60363451eb40
- 2b58556a-6bd8-4826-bcdb-00cb3d91eca7 → 10.244.0.92:8786
  2026-04-14T21:14:05Z New active routes (1 added)
metadata.e2fcb25e-f2df-4267-baf1-34070ce47ffc.svs-objects.onvif_analytics.07b12fd9-0f86-42ac-b07c-949f0fc9eba7_28dc44c3-079e-4c94-8ec9-60363451eb40
- 2b58556a-6bd8-4826-bcdb-00cb3d91eca7 → 10.244.0.92:8786
metadata.571e2c8e-ca99-41af-9170-988c6f844a8c.pacwest-objects.onvif_analytics.7dabe6a4-c0bd-4619-8205-4cb0970dfbf4_28dc44c3-079e-4c94-8ec9-60363451eb40
- 51295283-4340-444d-b418-b61046473b64 → 10.244.0.92:8786
  2026-04-14T21:14:46Z New active routes (1 added)
metadata.571e2c8e-ca99-41af-9170-988c6f844a8c.pacwest-objects.onvif_analytics.7dabe6a4-c0bd-4619-8205-4cb0970dfbf4_28dc44c3-079e-4c94-8ec9-60363451eb40
- 51295283-4340-444d-b418-b61046473b64 → 10.244.0.92:8786
metadata.261dab1d-03d5-4bb6-b2fb-00c1d0a0ed24.psc-objects.onvif_analytics.9387bcff-735c-4833-a24b-9ed43719d484_28dc44c3-079e-4c94-8ec9-60363451eb40
- a1d9a5b5-bcca-4fd4-b469-5e4e7565531d → 10.244.0.92:8786
metadata.e2fcb25e-f2df-4267-baf1-34070ce47ffc.svs-objects.onvif_analytics.07b12fd9-0f86-42ac-b07c-949f0fc9eba7_28dc44c3-079e-4c94-8ec9-60363451eb40
- 2b58556a-6bd8-4826-bcdb-00cb3d91eca7 → 10.244.0.92:8786
  2026-04-14T21:15:02Z New active routes (1 added)
metadata.e2fcb25e-f2df-4267-baf1-34070ce47ffc.svs-objects.onvif_analytics.07b12fd9-0f86-42ac-b07c-949f0fc9eba7_28dc44c3-079e-4c94-8ec9-60363451eb40
- 2b58556a-6bd8-4826-bcdb-00cb3d91eca7 → 10.244.0.92:8786
metadata.571e2c8e-ca99-41af-9170-988c6f844a8c.pacwest-objects.onvif_analytics.7dabe6a4-c0bd-4619-8205-4cb0970dfbf4_28dc44c3-079e-4c94-8ec9-60363451eb40
- 51295283-4340-444d-b418-b61046473b64 → 10.244.0.92:8786
metadata.261dab1d-03d5-4bb6-b2fb-00c1d0a0ed24.psc-objects.onvif_analytics.9387bcff-735c-4833-a24b-9ed43719d484_28dc44c3-079e-4c94-8ec9-60363451eb40
- a1d9a5b5-bcca-4fd4-b469-5e4e7565531d → 10.244.0.92:8786
metadata.7f136bab-b628-4ccb-97e6-4bd850a93c5b.superior-soil-objects.onvif_analytics.418237df-b0cd-48f7-98e2-ae89e1f2d148_28dc44c3-079e-4c94-8ec9-60363451eb40
- 9c54a3e1-9428-4c26-a801-8a51d031ac53 → 10.244.0.92:8786
  2026-04-15T18:59:36Z New active routes (1 closed)
metadata.261dab1d-03d5-4bb6-b2fb-00c1d0a0ed24.psc-objects.onvif_analytics.9387bcff-735c-4833-a24b-9ed43719d484_28dc44c3-079e-4c94-8ec9-60363451eb40
- a1d9a5b5-bcca-4fd4-b469-5e4e7565531d → 10.244.0.92:8786
metadata.7f136bab-b628-4ccb-97e6-4bd850a93c5b.superior-soil-objects.onvif_analytics.418237df-b0cd-48f7-98e2-ae89e1f2d148_28dc44c3-079e-4c94-8ec9-60363451eb40
- 9c54a3e1-9428-4c26-a801-8a51d031ac53 → 10.244.0.92:8786
metadata.e2fcb25e-f2df-4267-baf1-34070ce47ffc.svs-objects.onvif_analytics.07b12fd9-0f86-42ac-b07c-949f0fc9eba7_28dc44c3-079e-4c94-8ec9-60363451eb40
- 2b58556a-6bd8-4826-bcdb-00cb3d91eca7 → 10.244.0.92:8786
  2026-04-15T19:00:38Z New active routes (1 closed)
metadata.7f136bab-b628-4ccb-97e6-4bd850a93c5b.superior-soil-objects.onvif_analytics.418237df-b0cd-48f7-98e2-ae89e1f2d148_28dc44c3-079e-4c94-8ec9-60363451eb40
- 9c54a3e1-9428-4c26-a801-8a51d031ac53 → 10.244.0.92:8786
metadata.e2fcb25e-f2df-4267-baf1-34070ce47ffc.svs-objects.onvif_analytics.07b12fd9-0f86-42ac-b07c-949f0fc9eba7_28dc44c3-079e-4c94-8ec9-60363451eb40
- 2b58556a-6bd8-4826-bcdb-00cb3d91eca7 → 10.244.0.92:8786
  2026-04-15T19:03:28Z New active routes (1 closed)
metadata.7f136bab-b628-4ccb-97e6-4bd850a93c5b.superior-soil-objects.onvif_analytics.418237df-b0cd-48f7-98e2-ae89e1f2d148_28dc44c3-079e-4c94-8ec9-60363451eb40
- 9c54a3e1-9428-4c26-a801-8a51d031ac53 → 10.244.0.92:8786
  2026-04-15T19:19:46Z New active routes (1 closed)
no active routes

At this point, I tried re-registering the customer pods, and doing some restarts but nothing worked. Until I decided to just un-tick and tick the boxes back in the UI, and connections were made again. Which the logs show:

The VPS Drivers looked normal. I just found something odd from yesterdays recovery. I see metadata flowing under Devices/Metadata/Processing Server Metadata Group

image411×167 6.02 KB

but when you go under cameras metadata it looks like this. And currently receiving metadata on all 4 cameras.

image753×357 10.3 KB

But when I go to Recording servers and the devices in question -

image733×353 11.1 KB

Hope this helps, and let me know if I’m missing anything