I had been running an IVA against AI Bridge and an XProtect Corporate install for about 75 minutes when suddenly the RTSP feeds went dead. (This was a cross 3 separate pieces of StableFPS “hardware” with some 6 “cameras” in use, with the IVA consuming each of the 6 feeds 3 times to simulate load.)
Looking in the Management Client, I can see a message saying, “The proxy server serving the device driver ‘StableFPS.dll’ is unavailable. This affects the following hardware [hardware list]. An attempt to restart the driver will be initiated.” followed by some “Communication Error” messages (as I’d expect). It doesn’t look like the driver was able to restart until I rebooted the box.
I’m hoping the issue isn’t with the IVA or AI Bridge (frankly I can’t see how it could be, but I’d like to be sure).
Has anyone come across something like this before? Is there someplace else I can look to troubleshoot to see exactly what caused the issue?
Thanks in advance for any help.
It sounds like your StableFPS driver is crashing. The StableFPS driver will load the video files you specify into memory and then play them in a loop from there. Since the “proxy server” that is loading the driver is a 32-bit process, it can only handle a certain amount of memory. If the video files you provide in total are bigger than 2Gb, you might run into crashes like this.
Thanks, John.
I’m using two video files across three pieces of hardware: The first file is some 35MB and the second is some 60MB, so, at first blush, I don’t think memory is the issue, unless there are other factors that go into how much memory is used (I also only have a handful of cameras “enabled”).
Are there any other logs I can look at to help troubleshoot?
Interestingly enough, looking back through the Server Logs, this has happened several times before. In fact, I left XProtect running all day today but without AI Bridge attached to it. The same crash happened after some 6.5 hours.
Just to rule out the memory issue, you can check the process named “ProxySrv.exe” and see if it is high on memory (or whether it is slowly increasing => leaking).
Otherwise, checkout the log file here:
[C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Recording Server\Logs\DeviceHandling.log
This one might reveal in more detail what goes wrong.
Watching the ProxySrv process (at the start there were 2), one of them slowly grew from 125MB to about 250MB over the course of 6 hours, but it seemed to plateau at that point. A third such process spawned eventually, but it’s consistent around 170MB.
Looking in the log file you mentioned, I don’t see much, except when the StableFPS.dll driver goes down, I only really see a bunch of “Device communication error (NoDataException). Error: GetMediaDataBlock returned no data.” INFO messages. I also see a similar message, “Device communication error (DriverProxyServerUnavailableException). Error: GetMedaiDataBlock : stream context ”. There are other messages around the same time, but none really seem to say much other than reporting on stopping frame group media db consumers and “Deice communication stopped” messages.
Taking a look in RecorderEngine.log, I see some more entries. Here’s a snippet:
2023-02-10 17:46:36.453-05:00 [ 105] WARNING - DriverProxyUnavailable from proxy '5000-0'
2023-02-10 17:46:36.458-05:00 [ 3] INFO - (UpdateHardwareCommandHandler->ReloadDriver) HardwareId: a9c2d342-6f94-4547-a802-a1e64dba9a96
2023-02-10 17:46:36.458-05:00 [ 29] INFO - (UpdateHardwareCommandHandler->ReloadDriver) HardwareId: 2683e2e1-0e5b-4e36-a1eb-b1d1a1aa844e
2023-02-10 17:46:36.458-05:00 [ 120] INFO - (UpdateHardwareCommandHandler->ReloadDriver) HardwareId: 50a55d2d-4415-4a4d-980a-f1e67958a8d9
2023-02-10 17:46:36.462-05:00 [ 105] ERROR - The proxy server for the 'StableFPS.dll' driver has become unavailable, driver will be reloaded.
Affected hardware: StableFPS_T800_Suspect (localhost), StableFPS_T800_Traffic, StableFPS_T800 (localhost)
Exception: VideoOS.IO.Drivers.Interop.DriverProxyServerUnavailableException: GetMediaDataBlock : stream context 2268819372128 : 2288519806976
at VideoOS.IO.Drivers.Interop.InteropDevice.ThrowProxyUnavailableErrorIfNeeded(UInt32 error, String errorMessageFormat, Object[] errorFormatArguments)
at VideoOS.IO.Drivers.Interop.InteropDeviceMediaStream.GetMediaDataBlock()
at VideoOS.IO.Drivers.InteropMediaStream.GetMediaDataBlock()
at VideoOS.IO.Drivers.InteropMediaStream.GetNextFrameInternal_GetFrame(Boolean& stayInLoop, Boolean& mustCloseStream, Boolean& supressError, Exception& eventException)
at VideoOS.IO.Drivers.InteropMediaStream.GetNextFrameInternal()
Driver dll name: F:\Milestone\XProtect Recording Server\Drivers\NativeDrivers\StableFPS.dll
Proxy server name: 5000-0
Server name: localhost:21:103
Native error message: Proxy error code: 164
2023-02-10 17:46:36.577-05:00 [ 29] INFO - Calling Driver.Stop(). HardwareId: 2683e2e1-0e5b-4e36-a1eb-b1d1a1aa844e
2023-02-10 17:46:36.602-05:00 [ 29] INFO - Hardware '2683e2e1-0e5b-4e36-a1eb-b1d1a1aa844e' (32 streams) already exists on proxy instance '5000-0'
2023-02-10 17:46:36.656-05:00 [ 3] INFO - Calling Driver.Stop(). HardwareId: a9c2d342-6f94-4547-a802-a1e64dba9a96
2023-02-10 17:46:36.671-05:00 [ 120] INFO - Calling Driver.Stop(). HardwareId: 50a55d2d-4415-4a4d-980a-f1e67958a8d9
2023-02-10 17:46:37.006-05:00 [ 120] INFO - Hardware '50a55d2d-4415-4a4d-980a-f1e67958a8d9' (32 streams) already exists on proxy instance '5000-0'
2023-02-10 17:46:37.007-05:00 [ 3] INFO - Hardware 'a9c2d342-6f94-4547-a802-a1e64dba9a96' (32 streams) already exists on proxy instance '5000-0'
The entry immediately preceding the first entry in the above snippet is some 6+ hours prior on startup, so this seems to be what’s reported when the issue started.
Anything else I can look at or report back with?
Thanks as always!
Could you please see if you can find a crash dump and send it to us for analysis?
See if you find a file here - [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Recording Server\NativeDrivers\CrashDumps
Please send the file to us. I have made a Support case (MSC1680378) in your name, the most practical would be if you upload the file on the case..
Thanks for this, but no such path exists.
In [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Recording Server, I see the following folders:
- DriverCache
- EventData
- Logs
- Offline
- Secure
Could it be located somewhere else?
Strange. Please check this Registry entry. If you have it check the folder. If you do not have it try to create it.
BTW. What version of XProtect are you using?
The registry key above does exist and has the same values as your screenshot.
I’m running XProtect Corporate 2021 R2.
Now our suspicion is that it does not crash but restarts for some other reason.. Our best chance to find out more is if the logs will give us more. Can you please use the following article to enable some deep logging and then share the logs with us?
https://supportcommunity.milestonesys.com/s/article/running-the-ProxySrv-in-debug-mode?language=en_US
One thing that I believe would also be valuable information: If you change the video file you use as source for the StableFPS device does it make a difference? Basically we would really like you to test with another “source”; does this happen if you have a different source?
You mention two files, do you see the same issue with both? If you have tried a file coming with the StableFPS installer, does it happen also?
How do I tell which video is causing the crash? I’m running both at the same time using different hardware. In fact, it’s actually three pieces of “hardware” (i.e. StableFPS-based), with two of them running the same file, and a third running a second (different) file. I haven’t tried with the files that come with the StableFPS installer (one of the files is from a client; the other is the traffic video that comes with the NVIDIA DeepStream platform).
Anyway, I’ve made the changes according to the link sent and will be monitoring the logs. I use DebugView++ typically instead of DebugView. I’m hoping this will be all right, but please let me know if it isn’t. I’ll let it run until the next crash, which should be in the next 6-8 hours, give or take.
I’ve attached the DebugView++ log cropped to about 5 mins before what I believe is the crash until about 5 mins after the crash. The crash time I believe is around 17:33:49. I hope this proves useful.
If you need additional logging, I have the log in its entirety that lasts some 6-7 hours.
Hello Duncan,
I have looked at the log files and I want to share a few observations.
You seem to have two metadata devices that tries to read data from the following locations:
- [F:\Milestone\XProtect](file:F:/Milestone/XProtect) Recording Server\Drivers\NativeDrivers\StableFPS_DATA\Metadata\GPS\
- [F:\Milestone\XProtect](file:F:/Milestone/XProtect) Recording Server\Drivers\NativeDrivers\StableFPS_DATA\Metadata\BoundingBox\
They try this every 10th second and fails every time.
At 2023/02/22 17:33:49.245 the following is logged:
-– StableFPS ERROR — TID: 6976 Func: CmStableFPSMediaProvider::LoadMediaBlockFromFile | Msg: FILE OPEN ERROR | Val: 409600
After this error, the drivers are reloaded and a new proxy server process is started.
I suspect, that the read problems from the above locations causes some kind of leak (handles or memory).
My suggestion is, that you
- Validate that the above locations exists and contains data as expected
- Check the setting of Stable FPS for the two devices reading from these locations
- Possibly try to disable the two metadata devices to see if the problem goes away - at least to be able to pinpoint whether they are causing the restarts or not.
Regards
Klaus
Hi Klaus,
Thank you for checking this out.
- I have verified that both paths exist; however, the “GPS” directory is empty.
- How do I tell which metadata device corresponds to which of the directories listed above? When I used StableFPS, I don’t recall having set any of those directories.
You should have three metadata devices on each hardware device.
- Metadata 1 (i.e. port 1) is bounding boxes
- Metadata 2 (i.e. port 2) is GPS
- Metadata 3 (i.e. port 3) is Motion data
GPS data (Metadata 2) currently not supported.
The source of data for the other two devices is
…\StableFPS_DATA\Metadata\BoundingBox and …\StableFPS_DATA\Metadata\Motion
located in the driver install folder.
Which data files to use is configured in hardware settings in the management client
“301 MetadataBoundingBoxFiles”
“303 MetadataMotionFiles”
To see if these devices is causing the restarts that you experience, I would try to disable all metadata devices. If the failures stop, I would start by enabling them again, one at a time and then observe which device that may cause it to fail again.
Also - I guess that you don’t need to have the metadata devices running if there is no data for them anyway.
Regards
Klaus
Ok, so it’s looking like if I have more than one Bounding Box metadata device enabled, XProtect eventually crashes. I’m not quite sure what to make of that, unless the metadata devices have some shared attribute that’s causing a contention issue or something else?
