Management client/server error when using MessageCommunication.TransmitMessage

MIP
, , ,
1

Hi team,

I have noticed Management client crashes sometimes when running our plug-in. The plug-in requests recording server states (Responding/Not Responding) using MessageCommunication.TransmitMessage(new Message(MessageCommunication.ProvideCurrentStateRequest, { “server-id-1”, “server-id-2” }), null, null, null)). As you see server IDs are passed to filter items state response - only recording servers needed. MessageCommunication.TransmitMessage method is invoked inside timer - each 10 seconds - in order to update recording server states. So sometimes on some machines Management client just crashes with following exception:

************** Exception Text **************

************** Inner Exception **************

************** Inner Exception **************

Exception source: mscorlib

Exception type: System.ServiceModel.FaultException

Exception message: The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework SDK documentation and inspect the server trace logs.

Exception target site: HandleReturnMessage

Server stack trace:

at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at ICommunicationService.TransmitMessage(String token, FQID clientEndPoint, ExternalMessage message)

at VideoOS.Platform.Messaging.Internal.CommunicationClient.Transmit(Message message, FQID destinationEndPoint, FQID destinationObject, FQID sourceObject)

************** Outer Exception **************

Exception source: VideoOS.Platform

Exception type: VideoOS.Platform.MIPException

Exception message: Unable to Transmit

Exception target site: Transmit

at VideoOS.Platform.Messaging.Internal.CommunicationClient.Transmit(Message message, FQID destinationEndPoint, FQID destinationObject, FQID sourceObject)

at VideoOS.Platform.Messaging.MessageCommunication.TransmitMessage(Message message, FQID destinationEndPoint, FQID destinationObject, FQID source)

at BridgeManagementClient.Models.ManagementClient.ManagementPluginRootModel.<ServerModel_OnRequestRecordingServerStates>b__22_0()

at System.Threading.Tasks.Task.Execute()

************** Outer Exception **************

Exception source: A Task’s exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.

Exception type: System.AggregateException

Exception message: A Task’s exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.

Exception target site: A Task’s exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.

The behavior is reproduced sometimes on Windows 10 running 2020 R2 - single machine with Management/Event/Recording servers (all running) and Management client running.

Thanks in advance,

Nikolay.

2

Can you please provide Event Server logs and MIP logs covering the time of the incident for us for further investigation?

Event Server log - [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Event Server\logs

MIP log - [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Event Server\logs\MIPLogs

Please note that ProgramData is a hidden folder.

3

Hi,

Actually I have found the cause of the Management client crash - was unhandled MIPException thrown when MessageCommunication.TransmitMessage method is called and Event server not running. Even MessageCommunication.IsConnected property is checked in my code - if it is false (not connected) do not try to invoke TransmitMessage - sometimes the issue was reproduced. I assume this property is updated with a little delay when Event server is stopping. So, the steps to reproduce the issue:

  • start Management client → Event server is running
  • plug-in periodically poll for items (recording server) states - everything fine
  • stop Event server - if IsConnected property is still true (not updated) the plug-in will call TransmitMessage and error occurs

So the resolution was to catch Exception in order to prevent client crashing.

The other question is why the Event server is stopped (maybe crashed). I’ll ask for these logs (from the client machine) and if provided will send them.

Regards,

Nikolay.

4

Milestone Development has started an investigation and came to the conclusion that this was due to the plugin not handling an exception coming from message communication (can actually be seen from stack trace), which is same as yours. We will fix the issue so please keep eye on the news from Milestone.