Is new events subscription broken in AlarmEventViewer sample in MIP SDK 2020 R2?

MIP
, , , ,
1

I’m trying to implement an event listener app using AlarmEventViewer sample as a reference.

I’m working with XPCO 2020 R2, MIP SDK 2020 R2 and Visual Studio 2019.

I can compile and run the AlarmEventViewer sample.

I can connect to the XPCO server and collect and display past alarms / events.

However the application never gets the new alarms or events occurring while it is running: the list is never updated.

I can get new alarms / events only by changing ViewMode.

Do you know if the AlarmEventViewer sample is now broken in MIP SDK 2020 R2?

Can anyone test the sample with the same setup, in order to let me know if I’m doing something wrong?

2

I also tried other MIP component samples and MIP plugin samples which are internally using the MessageCommunicationManager class.

Each of them seems not to work.

Now I’m doubting about my compiler setup or the configuration of the project.

I noticed that the new MIP 2020 R2 required .NET 4.7 while its sample .csproj use 4.6.

I tried compiling both .NET versions 4.6 and 4.7 but I had no difference.

Am I missing to add some DLL reference or something else?

I just installed MS Visual Studio 2019 Community, then the MIP SDK 2020 R2 installer and the run opened the sample projects and compiled/run them (as I did in the past).

3

Doing some test it seems that the issue is at this stage of the code:

                MessageCommunicationManager.Start(EnvironmentManager.Instance.MasterSite.ServerId);
_messageCommunication MessageCommunicationManager.Get(EnvironmentManager.Instance.MasterSite.ServerId);

When trying to setup a new message communication it never succeeds.

I also enabled some more tracing with:

EnvironmentManager.Instance.TraceMessageCommunication = true;

What I get fro the Debug output is:

MIP: Debug: MessageCommunicationManager ():Start - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
MIP: Debug: MessageCommunication ():Init - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
MIP: Debug: CommunicationClient ():Init - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
MIP: Debug: CommunicationClient background thread ():Now starting...
MIP: Debug: CommunicationClient ():InitSession - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
Eccezione generata: 'System.IO.FileNotFoundException' in mscorlib.dll
MIP: Debug: CommunicationClient ():Thread - Received Message Try ..... 15ce5491-89f5-4dc4-8684-f54c59a51a4e
Eccezione generata: 'System.ServiceModel.FaultException' in System.ServiceModel.dll
Eccezione generata: 'System.ServiceModel.FaultException' in mscorlib.dll
MIP: Debug: CommunicationClient ():ClientClose start - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
MIP: Debug: CommunicationClient ():ClientClose end - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
MIP: Debug: CommunicationClient ():Il server non è stato in grado di elaborare la richiesta di callback a causa di un errore interno. Per ulteriori informazioni sull'errore, attivare IncludeExceptionDetailInFaults (da ServiceBehaviorAttribute o dal comportamento di configurazione <serviceDebug>) nel server in modo da restituire le informazioni sull'eccezione al client oppure attivare la traccia come descritto nella documentazione Microsoft .NET Framework SDK e analizzare i registri di traccia server.
MIP: Debug: CommunicationClient ():InitSession - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
MIP: Debug: CommunicationClient ():Thread - Received Message Try ..... 15ce5491-89f5-4dc4-8684-f54c59a51a4e
Eccezione generata: 'System.ServiceModel.FaultException' in System.ServiceModel.dll
Eccezione generata: 'System.ServiceModel.FaultException' in mscorlib.dll
MIP: Debug: CommunicationClient ():ClientClose start - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6
MIP: Debug: CommunicationClient ():ClientClose end - 41ce433b-7d5a-44a0-b400-9a7ab06f78f6

It keeps repeating the last lines: trying to Init the session, doing the Received Message Try with (probably the Exception), Closing client and reinitializing session again and again.

What could be?

All dll are in place in the directory.

4

Milestone Development will start an investigation. Are you using standalone?

5

I’m running and compiling the application in a standalone installation with XPCO 2020 R2 with test license.

Even using the default pre-compiled applications like MIP SDK Tray Manager returns exception on some interaction with the server.

E.g. Event Server Information buttons fails to collect any info and return exception debug traces.

Just to add on this same server we run HikVision plugins (HikANPRPlugin and HikServicePlugin) and Vanderbilit SPC plugins (SPCLink and FixComm)

6

Can you please check Event Server logs both the (normal)Event Server logs and the MIP logs?

Please go to Event server icon in the task tray and right click on the Event server icon. You will see the menu “Show Event Server logs” and “Show MIP logs”.

And please use the “Trace” option of the MIP SDK system tray tool to enable more tracing.

Go to the task tray and find “MIP SDK Tray Manager” icon, on the right click you will see “traces..” in the list.

7

Hi Rie,

here attached the Event Server log file

8

Here attached the MIP Logs file

9

Thank you for the logs. Unfortunately, there are any errors in the files. Can I ask you to check your Windows Event Viewer to see if there are any errors in there for further investigation?

10

Hi Rie,

for some unknown reason it seems that at least our test server started to reply to Event Server Information buttons in MIP SDK Tray Manager.

We absolutely did nothing to the server, so I have no clue about what could have fixed the issue.

I will check with our compiled Component app, to verify if even that issue is solved.

I also checked the Windows Event Viewer but I found no evidence of possible error generated by the Event Server in the recent past, while I was testing the Component app.

11

We continued the tests and we keep having issues with the RegisterCommunicationFilter method.

In our test server (XPCO 2020 R2 running on Windows 10 Pro) everything works.

In a couple of production servers with standard installation we are not able to run this event notification registration.

One server is XP Corporate2020 R2 running on Windows Server 2016 and another is XP Expert 2020 R1 running on Windows Server 2019.

Running the following code (where Evento di prova is the Italian EventType string for test events) in a try block

CommunicationIdFilter commfilter = new VideoOS.Platform.Messaging.CommunicationIdFilter(VideoOS.Platform.Messaging.MessageId.Server.NewEventsIndication, "Evento di prova");
                    _obj = _messageCommunication.RegisterCommunicationFilter(NewEventMessageHandler, commfilter, null, EndPointType.Any);
_messageCommunication.WaitForCommunicationFilterRegistration(commfilter, new TimeSpan(0,0,10));

we wait for 10 seconds and we catch an exception with message: Timeout waiting for the communication filter: ServerNewEventsIndication

12

Is there any reason for having these timeouts in CommunicationFilter registration?

Docs and comments state that in some cases you could wait up to 3-4 secs before a successful CommFilter registration.

Here we wait 10 secs and nothing happens.

I even tried to wait for 60 secs and nothing happens.

Is there any other way to check if the server is listening for this kind of CommFilter registrations?

At this point I’m doubting it is listening at all for CommFilter registration because of some unknown reason.

13

I did some investigation on production servers.

I found that the RegisterCommunicationFilter method for getting events as soon as they occur does work in Expert 2019 R2 and Expert 2019 R3 versions.

On the contrary on Expert 2020 R1 and Corporate 2020 R2 the same method never successfully registers on production servers.

These 2020 servers are installed in the same exact way as the 2019 others .

However the method works in a Corporate 2020 R2 test server, with MIP SDK installed.

I’m starting to suspect that the installation of MIP SDK fixes somehow something broken or deprecated in 2020 version.

Is there anyone using this RegisterCommunicationFilter method in a component integration app connected to a 2020 server without MIP SDK installed for collecting Analytics Events?

14

I have a suspicion! My suspicion is that you might be missing some dependent file or you have a mixture of dependent files from different versions of the MIP SDK.

Try to remove all the dependent dlls then make sure you copy new dlls from the bin folder of the MIP SDK you are building your solution with.

See also - https://developer.milestonesys.com/s/article/Dependent-files-in-MIP-SDK

15

We created ex-novo the app from the MIPLibrary template and the issue still exists only on some 2020 servers

The dll files in References are VideoOS.Platform, VideoOS.Platform.SDK, VideoOS.Platform.SDK.

They all point to the files in [C:\Program](file:C:/Program) Files\Milestone\MIPSDK\Bin\ directory

After the compiling the solution, dependent dll files has been copied to the release dir using CopyUIFiles.bat in [C:\Program](file:C:/Program) Files\Milestone\MIPSDK\Bin\

After all these steps I cannot see any way for messing with dlls or having the wrong ones.

Having the same program (and the same happens with the sample AlarmEventViewer) running on a server and not running on another leads me to think that something in the server is different, not in the program.

Is anyone able to run even the sample AlarmEventViewer on a 2020 server without MIP SDK installed?

I suspect the MIP SDK installation in some way customizes the local server…

I tested the same app on another Corporate 2020 R1 (test license) with no MIP SDK installed: the app works.

Now I suspect that the Event Server on not working servers is somehow congested.

Maybe out of memory, maybe with too many incoming events.

Is there a chance that MIP Communication Service does not forward subscribed events to external applications while it does the forwarding for local MIP plugin services?

16

Please troubleshoot if port 22333 is blocked or working fine:

https://developer.milestonesys.com/s/article/Ports-for-alarms-from-Milestone-Event-Server-service

Please try to put this in the AlarmEventViewer sample and examine the logs after reproducing the failing behavior.

EnvironmentManager.Instance.TraceMessageCommunication = true;

If the extended logging does not lead you directly to a solution please share the log for me to see it.

17

Hi Bo,

the port 22333 is open and I can connect.

Enabling the trace I have no final clue about what could be the root cause of the issue.

The log is quite messy because there is a lot of log entries coming from a Vanderbilt SPC Milestone plugin.

It is used to collect and report status and manage three anti-intrusion systems, so I cannot disable it.

It creates a MIP log file of 2.8 GB each day.

We had to move the log directory to a separate drive, in order not to be included in server backups.

It seems that this plugin is badly written at least in its log part, as it creates a lot of unnecessary log entries, but there is no way to prevent it.

We already asked the developer if there is a way to limit the kind of logged infos: they told us to disable completely MIP log if we want to save space…

They won’t change the code, neither they will re-compile it without all these unnecessary log prints (even without trace enabled!)

However looking at the log I found these lines:

2020-07-31 16:06:46.816 UTC+02:00  Error     CommunicationService.RegisterCommunicationFilter  Exception source:      mscorlib
Exception type:        System.Collections.Generic.KeyNotFoundException
Exception message:     La chiave specificata non era presente nel dizionario.
Exception target site: ThrowKeyNotFoundException
   in System.ThrowHelper.ThrowKeyNotFoundException()
   in System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   in VideoOS.Event.Server.MIP.CommunicationService.CommunicationService.VideoOS.Event.Server.MIP.CommunicationService.ICommunicationService.Receive(String token, FQID endPoint, Int32 timeoutMilliSeconds)

They keep repeating now and then in the file.

Another “strange” couple of lines is the following:

2020-07-31 16:06:34.369 UTC+02:00  Info      HandleState           Handle state. StateType=14, State=1, Value = 0, Id=9d577d03-3fce-446d-808f-5a89ec77c1ac, UpdateTime=31/07/2020 14:06:34
2020-07-31 16:06:34.369 UTC+02:00  Info      MoveNext              Removing CUDA message from sending. MessageId=1503 and Source Id = Server:XPCORS:cssd-nvr-01  Id:9d577d03-3fce-446d-808f-5a89ec77c1ac, ObjectId:9d577d03-3fce-446d-808f-5a89ec77c1ac, Type:3b25fe94-7c2f-499a-86df-2fa68aa3e1b5

And then also these lines keep occurring very frequently in the log file:

2020-07-31 16:33:00.544 UTC+02:00  Info      CommunicationService  MessageHandler after lock and end - Server.NewEventsIndication
2020-07-31 16:33:00.544 UTC+02:00  Info      CommunicationService  MessageHandler begin - Server.NewEventIndication
2020-07-31 16:33:00.544 UTC+02:00  Info      CommunicationService  MessageHandler foreach : SPCCONFIG.DO
2020-07-31 16:33:00.544 UTC+02:00  Info      CommunicationService  MessageHandler foreach : SPCCONFIG.SIMUL
2020-07-31 16:33:00.544 UTC+02:00  Info      CommunicationService  MessageHandler after lock and end - Server.NewEventsIndication

However all these logs and errors seems to be generated by the Vanderbilt SPC plugin, as they can be found even while I was not running the external application.

I will try to collect the same log from the other not working server…

Maybe there is less noise due to other plugins.

18

I did the same test with trace enabled in the other not working server and I got no log entry at all.

It seems that the issue does not generate any log.

In this second server I have no VanderBilt SPC plugin running.

Just collecting Analytics Events from an external source, and one of our server plugins which is monitoring some external devices and generates other Analytics Events.

19

What I wanted you to do was not clear..

We would like you to enable this (EnvironmentManager.Instance.TraceMessageCommunication = true;) client side and then see the logs in [C:\ProgramData\Milestone\MIPSDK](file:C:/ProgramData/Milestone/MIPSDK) pertaining to the client. For clarity run this in a client workstation not running plugins..

20

Hi Bo,

I just did the test you requested.

The log is the attached file.

I just zeroed GUID of installed licenses and obfuscated some other plugin name/description in the file.

All the rest of the file is untouched.

I modified and recompiled the source code as requested.

I started the AlarmEventViewer application on a different machine (not the event server / management server itself but a failover server).

I logged in the server using the application.

I changed the application mode from Alarms mode to Analytics Events mode (see logs strarting from line 24).

I generated a couple of new Analytics Events in the main server.

Nothing occurred in the application window.

I closed the application.

I tried the same application on a differente server-client couple (our test server which is working!) and the log is much more clear, with no CommunicationClient ():ClientClose entries.