Extended various timers, but the WebRTC video stops partway through

005bH000003cd53QAA · July 7, 2025, 9:47am

After setting the ApiGateway AppPool recycle interval and ClientTokenTimeoutMinutes to sufficiently long values and continuously streaming video via WebRTC, the video stopped after 30 hours, 33 minutes, and 37 seconds.

Please tell me the possible causes and how to resolve this issue.

Specifically, I made the following settings and continued streaming video via WebRTC:

Set the recycle interval of the VideoOS ApiGateway AppPool to a sufficiently long time (10,080 minutes).
Changed in [C:\Program](file:C:/Program) Files\Milestone\XProtect Management Server\IIS\ManagementServer\web.config to a sufficiently long value (71,587 minutes), and restarted the XProtect Management Server service.

However, the WebRTC video stopped after 30 hours, 33 minutes, and 37 seconds.

Note that during this 30 hours, 33 minutes, and 37 seconds, the WebRTC session was not renewed, but since ClientTokenTimeoutMinutes was set to a sufficiently long value, I believe this is unrelated.

Also, since this issue is reproducible, please let me know if there is a way to collect XProtect logs or other information for further investigation.

Best Regards.

Bo_Ellegard_Andersen · July 9, 2025, 9:25am

Maybe debug logs for the API Gateway could have information to explain the stop.

The API Gateway logs are at [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) API Gateway\Logs

Try to change the default log level to Debug and look into API Gateway logs.

In order to set the default log level to Debug, just add the following to the appsettings.production.json:

"Logging": {
    "LogLevel": {
      "Default": "Debug"
    }
  }

“[C:\Program](file:C:/Program) Files\Milestone\XProtect API Gateway\appsettings.production.json”

If the stop is a side effect of the camera stream being disrupted or similar this might be mentioned in the Management Client, Server Logs, System logs as communication error or similar. It is very interesting whether what you observe can also be seen as a disruption of the stream in the Smart Client or the issue is only affecting WebRTC.

You say it is reproducible, does it always fail after approximately the same time period? (around 30 hours?)

005bH000003cd53QAA · July 11, 2025, 12:56am

1. There was no interruption of the camera stream.

2. The issue is reproducible. The longest duration before stopping was 39:57:35. In several runs, the stream also stopped after 4 to 5 hours.

3. API Gateway Debug Log

I changed the API Gateway log level to Debug, and started displaying video via WebRTC from two cameras simultaneously after 17:53 on 2025/7/9.

However, the streams stopped around 22:52 on 7/9.

- cameraId: 2c5ba955-ee18-4dbb-9ece-d10a1dd18b8e

- cameraId: 80a6db53-5d1e-4458-bb16-5cab6371b41e

I have attached the [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) API Gateway\Logs\gateway.log file from that time.

It can be confirmed that there are no log outputs at all after 17:53 on 7/9 (Wed).

It is unclear whether this is normal or abnormal.

Additionally, there is something I am concerned about.

Besides the VideoOS ApiGateway AppPool, there are other AppPools with a recycle interval (default: 29H).

Please let me know if these are related to this issue.

DefaultAppPool

VideoOS Classic AppPool

VideoOS IDP AppPool

VideoOS IM AppPool

VideoOS Management Server AppPool

VideoOS ReportServer AppPool

VideoOS ShareServer AppPool

Incidentally, between 17:53 and the time the streams stopped around 22:52 on 2025/7/9, the following events were recorded:

- There is a record that the VideoOS IDP AppPool was recycled at 20:42:56 on 7/9.

- There is also a record that the VideoOS Management Server AppPool was recycled at 20:43:02 on 7/9.

Best Rehards.

Bo_Ellegard_Andersen · July 11, 2025, 8:19am

Please look into Windows Event Viewer, here you can see the recycling of AppPools. Does the stream stop at a time coinciding with AppPool recycling?

We suspect that nothing is put in the API Gateway logs is normal and expected.

005bH000003cd53QAA · July 11, 2025, 9:01am

I have already checked the Windows Event Viewer.

There were records in the Event Viewer at the same time as the recycle logs for the VideoOS IDP AppPool and VideoOS Management Server AppPool.

However, the timing of the stream stopping does not coincide with the AppPool recycling.

Bo_Ellegard_Andersen · July 17, 2025, 12:23pm

I need a general idea what you want to achieve and what the issue is?

Using the WebRTC javascript sample it should refresh the token and it should reconnect automatically and thus recover from a AppPool recycle.

In your testing

Does the video stream get interrupted and never resume?

Does the video stream get interrupted and the resume is not acceptable for you? (Takes too long?)

If you want absolutely no interruption in the video stream, for how long would you want that?

-

The sample uses peerConnection.onconnectionstatechange to check the state of the WebRTC session.

https://doc.developer.milestonesys.com/html/index.html?base=samples/protocolsamples/webrtc_javascript/readme.html&tree=tree_3.html

(Please make sure you do not use an old version of the sample.)

005bH000003cd53QAA · July 23, 2025, 9:44am

I apologize for the delayed response.

We are developing a browser feature utilizing WebRTC, referring to the sample.

In this context, we are currently testing the behavior of WebRTC session disconnection by setting various values for ClientTokenTimeoutMinutes.

I have three questions regarding this.

1. Procedure for Changing the ClientTokenTimeoutMinutes Value

Please let me know the official procedure to apply changes when modifying the ClientTokenTimeoutMinutes value.

At present, I am restarting all Milestone services, but is this sufficient?

2. How to Confirm Disconnection When ClientTokenTimeoutMinutes Is Exceeded (WebRTC Session Disconnection)

If the access token is not refreshed via the refresh token and the WebRTC session is not updated, and the ClientTokenTimeoutMinutes value is exceeded,

the video display via WebRTC stops. However, sometimes the WebRTC session is explicitly disconnected (as in the example log output below), and sometimes it is not.

[C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) API Gateway\Logs\gateway.log

2025-07-19 08:02:03.755+09:00 [ 79] DEBUG - IceConnectionStateChanged sessionId: 67b3961f-1363-4284-823c-01aeea5b3a15, disconnected

2025-07-19 08:02:03.755+09:00 [ 79] DEBUG - ConnectionStateChange sessionId: 67b3961f-1363-4284-823c-01aeea5b3a15, disconnected

2025-07-19 08:02:03.755+09:00 [ 79] DEBUG - Video source stopped

2025-07-19 08:02:03.755+09:00 [ 79] DEBUG - Closing session: 67b3961f-1363-4284-823c-01aeea5b3a15

2025-07-19 08:02:03.768+09:00 [ 79] DEBUG - ConnectionStateChange sessionId: 67b3961f-1363-4284-823c-01aeea5b3a15, closed

2025-07-19 08:02:03.768+09:00 [ 79] DEBUG - Closing session: 67b3961f-1363-4284-823c-01aeea5b3a15

2025-07-19 08:02:03.768+09:00 [ 79] DEBUG - Removed session: 67b3961f-1363-4284-823c-01aeea5b3a15

Could you explain why there is a difference, and whether this is by design?

3. How to Confirm When ClientTokenTimeoutMinutes Is Exceeded (Other Logs)

Regardless of whether the WebRTC session is explicitly disconnected or not, I have found that the following logs are output.

I understand that these logs are output because the WebRTC session has been disconnected—is this correct?

[C:\ProgramData\Milestone\MIPSDK\MIP20250723.log](file:C:/ProgramData/Milestone/MIPSDK/MIP20250723.log)

2025-07-23 17:43:32.814 Error: BaseLiveSource:PollThread ():Error reading from stream!

[C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Recording Server\Logs\SecurityTokenHandler.log

2025-07-23 17:43:32.812+09:00 [ 34] WARNING - Token verify failed: Expired

2025-07-23 17:43:35.841+09:00 [ 284] WARNING - Token verify failed: Expired

2025-07-23 17:43:38.882+09:00 [ 131] WARNING - Token verify failed: Expired

Best Regards.

0053X00000Cs42WQAR · July 28, 2025, 9:13am

You have hit vacation time, so I will do my best to answer.

Procedure for changing ClientTokenTimeoutMinutes

This is not something that should be changed very often as it will interrupt the entire system. This should be configured rarely and after that all Milestone services should be restarted.

ClientTokenTimeoutMinutes affects the token for the recording server. When using WebRTC the token used is an OAuth token, which have a timeout of 3600 seconds. This can be seen when the the OAuth token is retrived, by reading “expires_in” (reponse of post to /IDP/connect/token).

How to Confirm Disconnection When ClientTokenTimeoutMinutes Is Exceeded

Once a connection is established with WebRTC, we have no way of sending the reason for a disconnect to the client. The communication is now happening through WebRTC and the protocol does not have a way of sending error ids. So the client cannot know why the disconnect happens. To ensure the disconnect does not happen, the client should read expires_in and ensure to refresh the token before this runs out. Note: ClientTokenTimeoutMinutes should never be set to below 60 minutes as this would give issues for Oauth tokens.

If the token expires, the connection should become disconnected every time. I’m not sure why you experience something different. But you may have issues with the long timeout of ClientTokenTimoutMinutes. This could cause issues other places in the system and I would suggest that you lower it.

How to Confirm When ClientTokenTimeoutMinutes Is Exceeded

The recording server log will contain information about the Token expired.

My suggestion

In order to have a system which is as stable as possible, set the ClientTokenTimoutMinutes around 240 minutes. Have the client refresh the OAuth token after the expires_in valule and make the client try to reconnect if disconnected. The reconnect will handle the issue with the IIS reset and also if any network issues happens.

005bH000003cd53QAA · July 31, 2025, 10:35pm

Regarding the response, I understand and acknowledge the following:

1. The Milestone service needs to be restarted after changing ClientTokenTimeoutMinutes

-> I have confirmed that the method we are currently implementing is correct.

ClientTokenTimeoutMinutes should not be set to less than 60 minutes

-> Understood.

There is no official method to confirm when ClientTokenTimeoutMinutes has been exceeded.

-> We will confirm this by checking that the following Recording Server log output is generated, which we are currently aware of:

ex)

[C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Recording Server\Logs\SecurityTokenHandler.log

2025-07-23 22:54:38.008+09:00 [ 194] WARNING - Token verify failed: Expired