Unable to connect to "EventServer.CommunicationService". Object reference not set to an instance of an object.

MIP
, , , ,
1

We are experiencing a problem with a plugin that, upon startup, sometimes generates this error in the Event Server’s log:

2019-08-06 1:19:45 PM UTC-07:00 Error Unobserved task exception Unobserved task exception: System.AggregateException: A Task’s exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. —> VideoOS.Platform.MIPException: Unable to connect to EventServer.CommunicationService —> System.NullReferenceException: Object reference not set to an instance of an object.

at VideoOS.Platform.Messaging.Internal.CommunicationClientOnServer.InitSession()

— End of inner exception stack trace —

at VideoOS.Platform.Messaging.Internal.CommunicationClientOnServer.InitSession()

at VideoOS.Platform.Messaging.Internal.CommunicationClientOnServer.Transmit(Message message, FQID destinationEndPoint, FQID destinationObject, FQID sourceObject)

at VideoOS.Platform.Messaging.MessageCommunication.TransmitMessage(Message message, FQID destinationEndPoint, FQID destinationObject, FQID source)

at BoschPanel.Background.PanelConnectionBackgroundPlugin.d__15.MoveNext()

— End of inner exception stack trace —

- → (Inner Exception #0) VideoOS.Platform.MIPException: Unable to connect to EventServer.CommunicationService —> System.NullReferenceException: Object reference not set to an instance of an object.

at VideoOS.Platform.Messaging.Internal.CommunicationClientOnServer.InitSession()

— End of inner exception stack trace —

at VideoOS.Platform.Messaging.Internal.CommunicationClientOnServer.InitSession()

at VideoOS.Platform.Messaging.Internal.CommunicationClientOnServer.Transmit(Message message, FQID destinationEndPoint, FQID destinationObject, FQID sourceObject)

at VideoOS.Platform.Messaging.MessageCommunication.TransmitMessage(Message message, FQID destinationEndPoint, FQID destinationObject, FQID source)

at BoschPanel.Background.PanelConnectionBackgroundPlugin.d__15.MoveNext()<—

This error occurs at the start up of the Event Server and the very first time the plugin attempts to call VideoOS.Platform.Messaging.MessageCommunication.TransmitMessage() in response to handling a ConfigurationChanged event.

I suspect the problem may be a timing issue where some component has not been fully initialized or something and so a required object has not been instantiated which leads to the “Object reference not set to an instance of an object” message. The problem is, I can’t figure out which object reference is being referred to.

Any suggestions on how I can go about identifying the object in question?

2

Hi. Milestone Development will start investigation regarding this topic. We will be back when we receive a reply from them.

3

We have been investigating this, but so far been unable to find out how this can happen. Looking at the code there is no obvious candidates for a null reference to happen and we also made a small plugin in an attempt to reproduce, but no matter how often we try it does not happen in our test system.

Could you please let us know what version of XProtect you are running and if possible also the lines of code subscribing to the configuration changed event (including where the subscription is done) plus the ones around the message communication initialization and transmission?

4

The version is XProtect Corporate 2019R1.

Here is the full Init method of our class derived from VideoOS.Platform.Background.BackgroundPlugin:

        public override void Init()
        {
            Log.Info("******************************************************************************************************");
            Log.Info("******************************************************************************************************");
            Log.Info("**                                                                                                  **");
            Log.Info("**                                                                                                  **");
            Log.Info("**                                    Server Startup                                                **");
            Log.Info("**                                                                                                  **");
            Log.Info("**                                                                                                  **");
            Log.Info("******************************************************************************************************");
            Log.Info("******************************************************************************************************");
            Log.Info("Init");
            Console.WriteLine("Init PanelConnectionBackgroundPlugin");
            ITransportProviderFactory transportProviderFactory = new TransportProviderFactory();
 
            _sessionManager = new SessionManager(transportProviderFactory, new AesCrypto());
            _messageBus = _sessionManager.MessageBus;
            _configurationTaskCancellationTokenSource = new CancellationTokenSource();
            //Listen for configuration changes in a separate thread per Milestone support.
            Task.Run(async () =>
            {
                _sessionMasterList = new ConcurrentDictionary<FQID, Session>();
 
                while (true)
                {
                    try
                    {
                        _configurationTaskCancellationTokenSource.Token.ThrowIfCancellationRequested();
                        Log.Info("Configuration task - Waiting for configuration");
                        List<PanelConfiguration> newPanelConfigurationList = await _configurationChangeQueue.ReceiveAsync();
                        Log.Info($"{newPanelConfigurationList.Count} configurations pulled from Configuration Change Queue");
                        await ProcessConfigurations(newPanelConfigurationList);
                    }
                    catch (Exception e)
                    {
                        Log.Error("Configuration task - Error Processing configuration changed", e);
                        break;
                    }
                }
            },_configurationTaskCancellationTokenSource.Token );
 
            _configurationChangedReceiver = EnvironmentManager.Instance.RegisterReceiver(ConfigChangedHandler, new MessageIdFilter(MessageId.Server.ConfigurationChangedIndication));
 
            RegisterForMessageBusEvents();
            Log.Info($"Init - Ended");
        }

And the ConfigurationChangedHandler:

        private async Task ConfigChangedHandler(Message message, FQID destination, FQID sender)
        {
            Log.Info($"ConfigurationChanged: {message.MessageId}");
            List<PanelConfiguration> panelConfigurationList = new List<PanelConfiguration>();
            lock (Locker)
            {
                if (_messageCommunication == null)
                {
                    Log.Info("ConfigChangedHandler - Reqistering for messages");
                    MessageCommunicationManager.Start(EnvironmentManager.Instance.MasterSite.ServerId);
                    _messageCommunication =
                        MessageCommunicationManager.Get(EnvironmentManager.Instance.MasterSite.ServerId);
 
                    _messageCommunication.TransmitMessage(new Message(BoschMessageIds.BackgroundPluginStartingMessage),
                        _messageCommunication.ServerEndPointFQID, null, null);
 
                    _requestPanelsMessageRegistration =
                        _messageCommunication.RegisterCommunicationFilter(OnPanelsRequestedMessage,
                            new CommunicationIdFilter(BoschMessageIds.RequestPanelsMessage));
                }
 
                Log.Info("Retrieving panel configuration");
 
                var items = Configuration.Instance.GetItems(ItemHierarchy.UserDefined)
                    .Where(i => i.FQID.Kind == BoschPanelDefinition.BoschPanelPanelKind)
                    .OrderBy(i => i.Name);
                Log.Info($"Received {items.Count()} items");
                foreach (var item in items)
                {
                    Log.Info($"Panel configuration: {item.Name} [{item.FQID.ObjectId}]; Count: {item.Properties.Count}");
                    panelConfigurationList.Add(item.ToPanelConfiguration());
                }
                Log.Info("Queueing item list");
            }
            await _configurationChangeQueue.SendAsync(panelConfigurationList);
        }
5

Thank you for sharing this. This gave me an idea what to look for and I also managed to reproduce.

Would it be possible for you to move the following two lines to your Init method (before you register for the configuration changed indication) instead:

                   MessageCommunicationManager.Start(EnvironmentManager.Instance.MasterSite.ServerId);

                   \_messageCommunication =

                       MessageCommunicationManager.Get(EnvironmentManager.Instance.MasterSite.ServerId);

The problem is that if it has not already been started the MessageCommunicationManager takes a little time to get fully started, so if you are calling TransmitMessage immediately after some things are not entirely in place yet. We can probably ensure that the null reference will not happen, but it would still mean that your TransmitMessage will not go through, so I believe the best solution is if you can start up the message communication already at the time of Init.

6

That’s great news and I will make that change and report back. Can you give me any hint on how you reproduced the problem? Hopefully that can help me understand and explain why it’s happening in production but I couldn’t reproduce it in my lab.

7

I moved my call of Start and Get to the config changed handler in same way as you have it and then I managed to reproduce. But it only happens approximately 1 out of 5 so is definitely a timing issue and not easily reproducible.

I have now also spotted what is the problem on our side, so I will make a fix for that, but that will not be released until 2020 R1, so if you can update your code for now it would be great.