We had an issue last night between our management server and recording servers. I am trying to dig into this but have run out of ideas. Long story short is we had to reboot the management server but would really like to know what happened.

MIP
, , ,
1

this is what is event viewer kept throwing on all recording servers out of the blue

Error communicating with: “recordingserver”:7563 with command api error message: Token expiredException details:

System.Web.Services.Protocols.SoapException: Token expired

at VideoOS.Event.Server.Communication.Corporate.RecorderStatusService2Central.CheckTokenRefreshOrRethrowException(Exception e)

at VideoOS.Event.Server.Communication.Corporate.RecorderStatusService2Central.StartStatusSession()

at VideoOS.Event.Server.Communication.Corporate.StatusSessionRecorder.Run()

2

This also stopped any user from connecting and causing the clients to crash as well

3

This sounds like it’s not a software development / MIP SDK question so is best suited for the Support Community or even a technical support case. In short, it sounds like your Management Server stopped functioning either in part or whole. Clients get a token when they login, and that token has an expiry of 4 hours.

<!-- From C:\Program Files\Milestone\XProtect Management Server\IIS\ManagementServer\Web.config -->
<add key="ClientTokenTimoutMinutes" value="240" />

The client will attempt to renew it’s token sometime before it expires, and normally it is able to do so. But if the Management Server is not responding properly, the Management Server may not be able to issue a refreshed token to the client, so the client keeps using the token it has. Eventually it expires, and the Recording Server refuses to respond to requests containing that token.

Another way this could fail is the Recording Server could lose it’s connection with the Management Server, so it’s no longer able to receive notification of new tokens or to ask the management server to validate a token. In that case you’d see a different error message - something like “failed to validate token”.

Since you specifically saw the “token expired” message, we know that one or more clients were using expired tokens. That means they were unable to refresh their tokens.

If a token refresh fails, it’s either because the client was unable to reach the management server (network issue) or the management server was experiencing some kind of an issue. Since restarting the Management Server fixed it (for now) then the only way to understand the issue better will be to look at the log files and Windows Event Logs on the Management Server. The logs are located at “[C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Management Server\Logs”.

I would recommend applying the cumulative patches to begin with and if you need more help to get to the bottom of it, I would lean toward opening a support case (through your system integrator if you don’t currently have a care premium subscription). You can also use the support community but for this issue where extensive log review and possibly memory dump collection/analysis may be needed, the forum would not be very efficient.