I am working with developing a profile G compatible device for use with XProtect. The device is often fitted to a vehicle and communicates over a cellular modem interface. Occasionally, the connection between XProtect and the device is unreliable due to poor signal. The automatic remote recording retrieval feature is therefore important for edge recording recovery after connection problems in order to backfill the recording timeline in XProtect’s Recording Server. This mechanism itself should be tolerant if the connection is still poor.
I have instrumented the device side software to simulate such conditions, for example, if I add a delay when responding to FindRecordings / FindEvents, such that both requests fail due to XProtect’s 5 second timeout, then I find that this retrieval is marked as SUCCEDED in XProtect’s EdgeStorage.log file and the retrieval is never tried again.
I would say that for robust operation that XProtect should keep retrying to retrieve the edge recording until either it succeeds, or fails due to a negative response from the device (service not supported, authentication failure etc), or the device indicates that the requested recording is not available. This should not just affect the FindRecordings and FindEvents messages, but also other messages that are needed to retrieve the recording on a multi-channel device, for example, GetServices, GetVideoSources, GetProfiles, GetRecordingSummary, GetMediaAttributes, FindRecordings, GetRecordingSearchResults, EndSearch, GetRecordingJobs, GetReplayUri, and any other related requests.
With XProtect’s current “single shot” 5 second timeout, then I will find it impossible to make this feature reliable enough for customers to rely on.
Making the 5 second timeout configurable may also help. Some mobile connections are so poor that responding to messages may take much more than 5 seconds on occasions. But the more important thing is to add a retry mechanism.
I am using XProtect version 2024 R1 with a Professional+ Test Licence and Device Pack 13.2a. I have tried both the ONVIF driver as well as the ONVIF16 driver.
Hello Cliff Ede,
We do have retry mechanism for recording retrieval. We have to see why the retrieval is marked as SUCCEDED. Can you please send us the above-mentioned log where the retrieval is marked as SUCCEDED along with the driver logs and Wireshark of the communication between the device and XProtect for the same period of time so we can investigate?
Hi,
I did a test for which GetReplayUri timed out after 15 seconds in XProtect due to not receiving a response. The XProtect EdgeStorage log marks the job as SUCCEEDED. It is the last job in the log files. I attach a zip of the Wireshark and the whole of the XProtect Recording Server Logs directory.
Thanks,
Cliff
Hi Cliff Ede,
I have reviewed the logs you sent me. You are right that in this case, when GetReplayUri failed we don’t retry. We can consider in this case to retry because we already found a recording for the period we want to retrieve. The problem with retrying is that we want to avoid endless retries. And we don’t know when the device will be connected. If we retry 3 times in the next several minutes and the device is still not connected you will have the same result. And if there is more than 1 gap in the video, we will never try to fill the other gaps in the video. To solve your problem, you can manually retrieve the video when you have connection or set a Rule in Managment client video to be retrieved automatically in the evenings or at some moment when you know there is stable connection.
Hi Kalin,
When I perform tests with a device that is moving in a vehicle then sometimes the signal is good and sometimes poor - in many cases it would be impossible to establish a pattern in terms of time of day when the connectivity will be known to be good. You mentioned “If we retry 3 times in the next several minutes and the device is still not connected you will have the same result” - that is not actually always true, even in areas of mediocre signal I find that the majority of requests get responses within the timeout and this can be the case even while the vehicle is stationary - hence even in these cases having a retry policy will help even if limited to 3 retries. Having a user configurable limit would be even better in my opinion. You mention “And if there is more than 1 gap in the video, we will never try to fill the other gaps in the video.” - I don’t see any validity in this argument, there is no reason to believe that a timeout for one message will happen the net time. In fact Milestone sends many messages to the device for each retrieve, getReplayUri just happens to be the last one in the sequence.
Ultimately, having a retry policy, even limited to 3 retries, or extended to many more retries (I guess some limit needs to be put in place, user configurable?), would benefit the reliability in my opinion. Falling back to an operator having to make manual retrieves will undoubtedly limit the usefulness of this feature.
Many thanks,
Cliff
Hi Cliff,
The way the automatic retrieval is made should work perfectly in most cases with no additional interaction from the user. If we make everything user configurable we will need huge UI an d it will add complexity for most of the users. I understand you case if very specific because the device is in moving vehicle. We had customer with similar problem. The devices were placed in public buses which were circulating all they and returning to the garage 1 a.m to 5 a.m. The solution was to retrieve the recordings automatically in that period when they had stable connection by making a Rule. That’s why I proposed similar solution for you. Make a rule for certain period of time or when connection is established with the device and no operator interaction will be needed.
Regards,
Kalin
