I’m doing a POC install at a client’s, and I’ve run into an issue that didn’t come up when doing local development.
(We’re using AI Bridge v1.6.0 and things worked fine locally during development.)
It’s worth mentioning that the client has the following:
- A network solely consisting of static IPs; i.e. no DNS
- XProtect Corporate 2022 R3, whereas we developed against 2021 R2 (or whatever the minimum version of XProtect is for AI Bridge)
After adding in some HostAlias entries into the AI Bridge Helm chart, I can see the bounding box metadata from our IVA making it into the correct Kafka topic (I can play them back using `kafka-console-consumer`).
However, despite verifying the sensor IDs and place IDs are correct in the messages, and that the feeds are subscribed to the correct AI Bridge topic (I can see the appropriate metadata “hardware” created for the feeds and that they’re associated with the feeds themselves after being created), the metadata never seems to make it to the Smart Client (I’ve also verified that the metadata hardware is recognized in the “Bounding Box Provider” list in the Smart Client).
I’m not sure if there’s something failing silently because it’s relying on some DNS name (I’ve used IPs wherever possible), or if there’s an issue with the version of XProtect being used, or something else.
All logs look ok, and, like I said, I can see the messages in the Kafka topic.
Does anyone have any thoughts as to where else I can check or how else I can troubleshoot?
As always, thanks in advance.
I would first check the log files of the “proxy” container. Here you should first of all see a log message similar to the one below.
2023-04-14T06:22:37Z VPS metadata connection established (metadata.3bc3c404-4e48-4e59-951d-a247aa0acf15.objects.deepstream_minimal.d365898e-10ea-4124-a803-08cd00e0aba5_28dc44c3-079e-4c94-8ec9-60363451eb43) with source video resolution [960x540]
This indicates that XProtect successfully connected to AI Bridge and now is ready to receive metadata (make sure the video resolution matches the actual resolution; otherwise AI Bridge will not be able to correctly convert DeepStream minimal to ONVIF.
Also in the proxy container, you should see a log entry similar to the following
2023-04-14T06:26:13Z Forwarding metadata in deepstream_minimal format from source d365898e-10ea-4124-a803-08cd00e0aba5/28dc44c3-079e-4c94-8ec9-60363451eb43 (gRPC / REST) to VPS connection started
This message indicates that AI Bridge received at least one metadata message from the IVA app and that it has now been forwarded it to XProtect. If metadata messages are not received from the IVA app for at least 10 seconds, you will see a log message like this
2023-04-14T06:33:40Z Forwarding metadata in deepstream_minimal format from source d365898e-10ea-4124-a803-08cd00e0aba5/28dc44c3-079e-4c94-8ec9-60363451eb43 to VPS connection stopped after 217 frames (did not get data for 14.866215335s)
To confirm that XProtect is actually receiving the metadata, you can go to the Management Client and click on the metadata device. If the metadata icon (in the preview view) is moving, then data is being received
If bounding boxes are still not showing up in the Smart Client, then it could be an issue with the timestamps that are associated with the metadata. They must match the timestamps of the video frames from which they are generated.
John, thanks for the quick reply!
Here’s what I’m seeing in the proxy logs since it started (it’s been running for awhile, and I’ve edited out any sensitive IP addresses):
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Proxy, build: Fri Mar 17 14:05:56 UTC 2023
-brokers aib-aibridge-kafka-broker:29092
-config-responses-topic voyager.topics.daim.gateway.config_responses
-gateway-id 1b80eaa0-203d-4dc0-ae3b-9bf4b85ec992
-gateway-version 1.0.0
-network-config-file
-sparql-query-endpoint http://aib-aibridge-fuseki:3030/repositories/voyager/query
-sparql-timeout-in-seconds 10
-stream-subscriptions-resend-seconds 300
-stream-subscriptions-topic voyager.topics.daim.vmsbridge.stream_subscriptions
-streaming-port 8786
-vps-authorization true
-vps-port 8787
-vps-tls-certificate-file server.crt
-vps-tls-enabled false
-vps-tls-key-file server.key
2023-04-14T01:31:11Z Server starting ...
2023-04-14T01:31:11Z Verifying gateway graph ...
2023-04-14T01:31:11Z Verifying gateway graph failed (will retry in 5 seconds): Post "http://aib-aibridge-fuseki:3030/repositories/voyager/query": dial tcp 10.110.243.47:3030: connect: connection refused
2023-04-14T01:31:16Z Verifying gateway graph failed (will retry in 5 seconds): gateway instance of expected version could not be found in graph
2023-04-14T01:31:21Z Verifying gateway graph succeeded
2023-04-14T01:31:21Z Verifying existence of topics in kafka cluster ...
2023-04-14T01:31:21Z Verifying existence of topics in kafka cluster succeeded
2023-04-14T01:31:21Z Creating new kafka producer ...
2023-04-14T01:31:21Z Creating new kafka producer succeeded
2023-04-14T01:31:21Z Creating new kafka consumer for group '5bcb9b8a-f9d1-4e5d-9fe1-ce0d69489f8a' ...
2023-04-14T01:31:21Z Creating new kafka consumer succeeded
2023-04-14T01:31:21Z Creating new kafka consumer for group 'daim.vmsbridge.proxy.event' ...
2023-04-14T01:31:21Z Creating new kafka consumer succeeded
2023-04-14T01:31:21Z Server started
2023-04-14T01:31:24Z Event map initialized
- empty
2023-04-14T01:31:24Z Event map empty; waiting for updated event map to arrive
2023-04-14T01:31:39Z Event map updated
- event.12355b21-5a25-4a1d-b6d2-f6e02c9b95b5.<topic>.analytics_event -> [ http://<VMS IP> ]
2023-04-14T02:08:15Z Unauthorized attempt to subscribe for video: invalid token
2023-04-14T02:14:23Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:16:32Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:19:08Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:25:39Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:26:48Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:27:00Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:30:59Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:36:08Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:38:33Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:43:33Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T02:54:30Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T03:24:17Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T03:25:01Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T03:25:22Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T06:09:25Z Got new security token for endpoint 'http://<VMS IP>'.
2023-04-14T10:04:25Z Got new security token for endpoint 'http://<VMS IP>'.
2023-04-14T13:50:24Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
2023-04-14T13:59:25Z Got new security token for endpoint 'http://<VMS IP>'.
2023-04-14T14:32:18Z Sending analytics event to 'http://<VMS IP>:22331/Central/AlarmServiceToken' succeeded.
It would look like none of the log entries you mentioned are appearing in the logs at all…Any idea what might be causing that, or where I should look/check?
It might be worthwhile noting that the Management Server Configurator has encryption turned off, i.e. it isn’t used.
Could it be that the management server can’t resolve the AI Bridge’s hostname? As mentioned earlier, these folks don’t use DNS. I’m not sure how AI Bridge and the Management Server communicate, so this might be something? Although, I did try adding an entry into the Management Server VM’s hosts file (didn’t reboot, though, as that shouldn’t be necessary) but still no joy.
In the meantime, I’ll keep digging for possible answers.
Thank you as always for your help, John. You are an absolute legend.
This line from the log file indicate that XProtect cannot connect to AI Bridge to subscribe for metadata (even though it says video).
2023-04-14T02:08:15Z Unauthorized attempt to subscribe for video: invalid token
More precisely, AI Bridge is complaining because the token provided by XProtect is somehow invalid (probably missing). What version of XProtect are you using? If using 2022 R1 or later, you should be fine.
As a temporary solution, you can try to disable authorization. You do this by changing the command line option “-vps-authorization=true” to false for the proxy container (look in the templates/aibridge-proxy.yml file).
Interesting. I had suspected that since I’d been developing against 2021 R2 (I think) that required that same auth flag to be set to “false”, but the client is running 2022 R3 (in fact it was just upgraded to that the week I was visiting) and that auth flag, I’m pretty sure, is set to “true”.
I’ll definitely give setting that flag to “false” a try when I’m able. Thanks as always for your insight!
Hey John,
I’m finally back on this.
So I decided to check things out in my local dev environment which is running XProtect 2021 R2 and AI Bridge 1.6. The IVA works with everything except the metadata, which of course I find weird.
--vps-authorization is set to false. Looking at the proxy container’s logs, I see very similar entries as the above, although without the “invalid token” bit:
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Proxy, build: Fri Mar 17 14:05:56 UTC 2023
-brokers aib-aibridge-kafka-broker:29092
-config-responses-topic voyager.topics.daim.gateway.config_responses
-gateway-id 1b80eaa0-203d-4dc0-ae3b-9bf4b85ec992
-gateway-version 1.0.0
-network-config-file
-sparql-query-endpoint http://aib-aibridge-fuseki:3030/repositories/voyager/query
-sparql-timeout-in-seconds 10
-stream-subscriptions-resend-seconds 300
-stream-subscriptions-topic voyager.topics.daim.vmsbridge.stream_subscriptions
-streaming-port 8786
-vps-authorization false
-vps-port 8787
-vps-tls-certificate-file server.crt
-vps-tls-enabled false
-vps-tls-key-file server.key
2023-04-25T20:15:23Z Server starting ...
2023-04-25T20:15:23Z Verifying gateway graph ...
2023-04-25T20:15:23Z Verifying gateway graph failed (will retry in 5 seconds): Post "http://aib-aibridge-fuseki:3030/repositories/voyager/query": dial tcp 10.105.116.102:3030: connect: connection refused
2023-04-25T20:15:28Z Verifying gateway graph failed (will retry in 5 seconds): Post "http://aib-aibridge-fuseki:3030/repositories/voyager/query": dial tcp 10.105.116.102:3030: connect: connection refused
2023-04-25T20:15:33Z Verifying gateway graph failed (will retry in 5 seconds): gateway instance of expected version could not be found in graph
2023-04-25T20:15:38Z Verifying gateway graph succeeded
2023-04-25T20:15:38Z Verifying existence of topics in kafka cluster ...
2023-04-25T20:15:38Z Verifying existence of topics in kafka cluster succeeded
2023-04-25T20:15:38Z Creating new kafka producer ...
2023-04-25T20:15:38Z Creating new kafka producer succeeded
2023-04-25T20:15:38Z Creating new kafka consumer for group 'd3681036-9217-4298-9bae-b64212c80dc5' ...
2023-04-25T20:15:38Z Creating new kafka consumer succeeded
2023-04-25T20:15:38Z Creating new kafka consumer for group 'daim.vmsbridge.proxy.event' ...
2023-04-25T20:15:38Z Creating new kafka consumer succeeded
2023-04-25T20:15:38Z Server started
2023-04-25T20:15:41Z Event map initialized
- empty
2023-04-25T20:15:41Z Event map empty; waiting for updated event map to arrive
2023-04-25T20:15:43Z Event map updated
- event.12355b21-5a25-4a1d-b6d2-f6e02c9b95b5.aaa.analytics_event -> [ http://vms-url ]
2023-04-25T20:17:02Z Sending analytics event to 'http://vms-url:22331/Central/AlarmServiceToken' succeeded.
2023-04-25T20:17:02Z Sending analytics event to 'http://vms-url:22331/Central/AlarmServiceToken' succeeded.
2023-04-25T20:17:50Z Sending analytics event to 'http://vms-url:22331/Central/AlarmServiceToken' succeeded.
2023-04-25T20:17:50Z Sending analytics event to 'http://vms-url:22331/Central/AlarmServiceToken' succeeded.
(The rest of the entries are all the same as the last couple.)
At this point I’m convinced I’ve messed something up with the configuration, but I can’t quite see what. It could be that I’m now back to using a single server as the AI Bridge, but it’s using a bridge ID/config from a different server. In other words, this server used to run another AI Bridge, but is now running a different AI Bridge (with an updated GUID). The “older” AI Bridge is not running. I don’t know if this would mess anything up or not.
I haven’t had a chance to look into the original problem from this thread with the client, but I’ve now run into something seemingly similar with my local environment. It’s likely something either obvious or something I’ve accidentally messed up under the hood, but I can’t seem to find what it is.
As always, thank you for your assistance!
In the proxy log, I would expect to find an entry similar to this
2023-04-14T06:22:37Z VPS metadata connection established (metadata.3bc3c404-4e48-4e59-951d-a247aa0acf15.objects.deepstream_minimal.d365898e-10ea-4124-a803-08cd00e0aba5_28dc44c3-079e-4c94-8ec9-60363451eb43) with source video resolution [960x540]
This log entry will appear when XProtect (the Recording Server) is connecting to AI Bridge to fetch metadata. So, since the log entry is not there, I would suspect that the issue is somehow related to a connectivity issue between XProtect and AI Bridge.
I would first check that the correct URL is used from XProtect. You can check this by opening the Management Client and navigate to the metadata device like shown below. The URL shown is the one that the Recording Server will use to connect to AI Bridge. If this is not pointing to the correct machine (or the hostname cannot be resolved) then you will not get the expected log entry for the proxy container.
The URL could maybe be wrong due to you having had multiple AI Bridge’s running; so it might still point to the old one somehow. If it is wrong, then you can just delete the hardware and re-subscribe to re-create the hardware with the correct URL.
Excellent call re the URL. I thought it might have been as much, but I wasn’t sure where to find that info. Updating it fixed it locally for me–thanks!
I’ll keep you posted once I’m able to look at the version deployed at my client’s.
