Event server MIP logs showing errors and filling up drive with logs when using custom Message Communication.

Corporate 2023R3 with latest hotfixes

In the Smart Client, we are sending a message to a custom service application every second.

We do not have any code running in the Event Server.

The class we are sending through MIP comms looks like this:

[Serializable]
public class XProtectCommand
{                     
    public XProtectCommand() { }       
 
    public EndPointIdentityData Source;
    public string Entry; //Can be any Serializable data
    public string EntryType; //Can be any Serializable data           
}

And we serialize several different classes to JSON and put it in the Entry:

internal static void SendCommandToMapService(Command cmd)
{
    try
    {               
        Debug.WriteLine($"Sending {cmd} {cmd.GetType()}");
        var settings = new JsonSerializerSettings()
        {
            ReferenceLoopHandling = ReferenceLoopHandling.Serialize,
            PreserveReferencesHandling = PreserveReferencesHandling.All,
        };
 
        XProtectCommand data = new XProtectCommand()
        {
            Source = new EndPointIdentityData()
            {
                IdentityName = Command.SourceIdSmartClient,
                EndPointFQID = MessageCommunicationManager.EndPointFQID
            },
            Entry = Command.Serialize(cmd, settings),
            EntryType = cmd.GetType().AssemblyQualifiedName,
 
        };
        Instance._messageCommunication.TransmitMessage(new VideoOS.Platform.Messaging.Message(Command.MessageIdMapsCmdLine, data), null, null, null);
       
        Logger.Log("Sending command: " + cmd.ToString());
    }
    catch (Exception)
    {
        Logger.Log(" (unable to send)");
    }
}

The code works fine, we are able to get our data in the service application, but when the Smart Client is open, the MIP logs keep logging this error multiple times per second and it fills up the disk.

The error complains about a parameter named “type” which is NULL, but I have no idea why it is NULL or why the Event Server is trying to deserialize it.

2024-03-27 11:15:02.928+00:00 [ 137] ERROR - ExtenalMessage.Deserialize Unable to deserialize class:ORBNETMapsPlugin.Comms.XProtectCommand, ORBNETMapsPlugin, Version=1.0.3.27663, Culture=neutral, PublicKeyToken=null, Exception:Value cannot be null.

Parameter name: type

2024-03-27 11:15:02.934+00:00 [ 119] ERROR - ExtenalMessage.Deserialize Unable to deserialize class:ORBNETMapsPlugin.Comms.XProtectCommand, ORBNETMapsPlugin, Version=1.0.3.27663, Culture=neutral, PublicKeyToken=null, Exception:Value cannot be null.

Parameter name: type

2024-03-27 11:15:02.935+00:00 [ 97] ERROR - ExtenalMessage.Deserialize Unable to deserialize class:ORBNETMapsPlugin.Comms.XProtectCommand, ORBNETMapsPlugin, Version=1.0.3.27663, Culture=neutral, PublicKeyToken=null, Exception:Value cannot be null.

Parameter name: type

We have been using this technique for a long time, and I think the error only starting appearing since 2023R3.

Interestingly, it seems the event server is also tracking multiple version of the Smart Client plugin.

Also, the errors messages are being disaplyed much faster than any data we are sending.

Hi,

Are you sure you are not perhaps using several different versions of your plugin in the separate plugin locations? SmartClient plugins may be found both in {Program Files}\Milestone\MIPPlugins and in {Program Files}\Milestone\XProtect Smart Client\MIPPlugins.

The fact that your error messages are referencing several different versions of your dll seems to indicate that something has gone very wrong, at least.

As for why it tries to deserialize your message, and why it fails, my guess would be that it is related. Is it possible your smart client plugin is loading a different version of this dll to the one loaded by the receiving application?

Best regards,

Simon

Hi Simon, unfortunately not in this case,

we have the same issue without the differing version, there is no SC in this case. Only a client service.

Do you know what the “type” parameter could be in this case?

Could it be the endpoint type?

EndPointType endPointType

In which case null should be valid as its a transmit to all.

If the EndPoint is null, all EndPoints are requested to transmit the specific messages.

Hi Justin,

Can you share what the ORBNETMapsPlugin.Comms.XProtectCommand class looks like? It seems to fail trying to deserialize that class, and I’d like to try to reproduce this error to see if I can figure out what goes wrong.

Best Regards

Simon

Hi Simon,

The XProtectCommand class is like this:

  [Serializable]
  public class XProtectCommand
  {
      public XProtectCommand() { }
 
      public EndPointIdentityData Source;
      public string Entry; //Can be any Serializable data
      public string EntryType; //Can be any Serializable data           
  }

And we use Entry for serializing a class to JSON and EntryType so we can deserialize it back using Newtonsoft.

For example, this is how we transmit a message:

internal static void SendCommandToMapService(Command cmd)
{
    try
    {      
        XProtectCommand data = new XProtectCommand()
        {
            Source = new EndPointIdentityData()
            {
                IdentityName = Command.SourceIdSmartClient,
                EndPointFQID = MessageCommunicationManager.EndPointFQID
            },
            Entry = Command.Serialize(cmd),
            EntryType = cmd.GetType().AssemblyQualifiedName,
        };
        
        Instance._messageCommunication.TransmitMessage(new VideoOS.Platform.Messaging.Message(Command.MessageIdMapsCmdLine, data), null, null, null);              
    }
    catch (Exception ex)
    {
        Logger.Log($"Unable to send command: {ex.Message}");
    }
}

Hi, I have done some further investigations, and the “type” parameter is the type of your data class, in this case the one defined in the error message you’re getting, i.e. ORBNETMapsPlugin.Comms.XProtectCommand, ORBNETMapsPlugin, Version=1.0.3.27663, Culture=neutral, PublicKeyToken=null. This is the message signature that is received, but it looks like the SmartClient cannot find the type specified. Please note that the .dll used in your plugin must be identical across all places that wish to communicate using a custom class. If any one of them has a version that is even slightly different, it will fail and cause this exact exception. Is it possible that your message is defined in two different .dll’s? Or perhaps you are using a different build on the server than on the SmartClient?

It took me a while to track down the exception, because at first I overlooked the spelling mistake in the error message leading my searches to come up empty, but it is in fact thrown in our code, specifically by new DataContractSerializer(type), where type is the type you define.

Since that type is null, because reflection is unable to find the class specified, the DataContractSerializer constructor throws a null reference exception. Type.GetType(string) is extremely finicky and needs the class to be very specifically the one defined. Even a build number difference will throw it off.

Hope this helps.

Best regards,

Simon

Hi Simon,

We are definitely always running the same DLL versions on both sides.

What can happen is that during an upgrade, not all the Smart Client will be updated at the same time. But even after everything is updated and the DLLs are the same, the error still shows.

Have you been able to reproduce the issue locally?

“Please note that the .dll used in your plugin must be identical across all places that wish to communicate using a custom class” This was not the case in previous version. If the DLLs versions are not the same, trhe solution still works, i.e. our service can still receive messages from the Smart Client. The only issue seems to be the excess logging of the comms error in the MIP logs.

Also, we don’t want everything to crash when a big site is upgrading all their Smart Clients progressively, it would not make sense to have to stop all the Event Server during the upgrade of all the clients.

“Or perhaps you are using a different build on the server than on the SmartClient?”

=> No, this also happens when building on a single server, i.e. the Smart Client plugin is the same as the service DLL

“Since that type is null, because reflection is unable to find the class specified, the DataContractSerializer constructor throws a null reference exception.”

=> As I mentioned, our messages are still getting through when this happens. Could it be that somehow you are keeping track of old DLL versions?

Hi Eric,

I think we need more information to figure out what’s going on. I haven’t been able to reproduce the issue, except in cases where the builds of the message don’t match.

It shouldn’t be happening, especially not when the functionality works. I’m wondering, what other plugins are running on the machines, if any? I can guarantee that no part of the VMS is keeping track of old versions, assuming you don’t have the files lying around in plugin folders that we scan.

Do you have any sort of folder structure going on inside your plugin folder?

I think the easiest way to track down this problem is for me to send you a .dll that will output the stack trace as part of the same error message, that way I have some idea of what exactly is causing the issue. An alternate method might be for you to try and enable mip message tracing, by placing MIPSettings.xml in your smart client binaries folder, with the values for TraceMessageCommunication and DebugLogging set to true, and the smart client option “Advanced”->“Logging (for technical support)” set to Enabled. This should provide detailed tracing of the messages being sent back and forth, and of who is subscribing to them, which might also help.

I’d also like to see the part of your code where you deserialize your data, if possible.

Let me know how you’d like to proceed,

Best Regards,

Simon

MIPSettings.xml contents, for reference:

<?xml version="1.0" encoding="utf-8" ?>

False

False

False

False

False

False

False

False

Hi Simon, we have been able to reproduce this now that we had some more time to have a look at it calmly.

We have opened support case:

The support request is registered as MSC1963741 / “Unable to deserialize” error in Event Server for MIP communication