Whether I can filter the video data before decompressing and rendering video in Smart client

MIP
, , ,
1

Dear Milestone,

I use Milestone smart client to view the video from a H264 Camera, which contains some SEI NAL data and some redundant P NAL. I wonder whether in the Plugin integration I can access these bytestreams to remove the SEI, the redundant P NALs, then I can feed back the bytestream to the Smart client to be decoded / rendered normally as usual?

Thank you in advance for your helps

Best wishes

Son

2

No.

You could with a Smart Client plugin get the raw video data, but you will from there not be able to inject the data again for use in the Camera view items.

3

Dear Bo Andersen,

Thank you very much for your answer. I just profit to ask you two more questions:

-It seems to me that the Recording server does some kind of filtering over the h264 stream ? I nolonger capture the SEI NALUs in the MileStone plugin sample code to extract the raw (compressed) video, which surely contains several SEIs

-Back to my initial question, so to filter the video data before decompressing and rendering video, I should use at least the component integration shouldn,t I ? Can you show me some Milsestone classes / components which do the decoding and rendering video, please?. Because the Camera view can perform these operations so I think I can reuse these components without incorporating third party ones?

Thank you in advance for your answer

Sincerely yours

Son

4

If you see the SEI in the video going in to XProtect but not in the video you read as raw data from XProtect the data is probably filtered out in the camera driver.

ImageViewerControl (which is the core part of the camera view) does decoding and rendering but you cannot do any manipulation to the data.

With classes like RawLiveSource, JpegLiveSource, BitmapLiveSource, RawVideoSource, JpegVideoSource and BitmapVideoSource you can read the data in raw or decoded format. You are with this data welcome to develop you own rendering. Perhaps you will find the samples Media Live Viewer / Media Playback Viewer (or Media RGB Enhancement Live / Media RGB Enhancement Playback) interesting.

5

Dear Bo Andersen,

I can understand from your answer that the camera driver (filtering out the SEIs) is found in the Recording Server. With component / protocol integration, can I modify the behavour of that filtering process?. Otherwise how can I forward user-defined information (currently found in SEI) to SmartClient?

I just have a look on the code of MediaLiveViewer. C# is not my strong point :-). Can you point out briefly which class I should use to decode the video (h264/MJPEG).

Thank you again for your help

6

You cannot modify the drivers, we would have to report to driver development, if they evaluate and find that information is missing and is a bug, they will have to develop a fix for the bug.

About drivers - https://developer.milestonesys.com/s/article/About-hardware-device-driver-development-and-the-Milestone-Integration-Platform-MIP-SDK

In order to report a bug we will need details on driver used and the particulars on the SEI.

I did a list; classes like RawLiveSource, JpegLiveSource, BitmapLiveSource, RawVideoSource, JpegVideoSource and BitmapVideoSource

If developing C++ is more your suit, please look at

http://doc.developer.milestonesys.com/html/index.html?base=reference/architecture/cpp_development.html&tree=tree_2.html

and

http://doc.developer.milestonesys.com/html/index.html?base=mmtkhelp/main.html&tree=tree_2.html

-

http://doc.developer.milestonesys.com/html/index.html?base=samples/medialiveservice_cpp.html&tree=tree_2.html

Note: If this is plugin development you must use .NET, and the C# sample are probably still the best fit..

7

Thank you very much for your prompt answer.

I just wonder why the defaut camera driver (found at the Recording Server of XProtect Essiential+) just filters out the SEI NALs which is also a generic data structure of H264 compression. Maybe in other version of XProtect (Corporate) we can forward unfiltered raw bytestream H264 from the camera without even modifying the driver.

Of course the final solution is we join you in a partner program to have more flexibilty in customizing the XProtect SDK.

Best regards

8

There is hundreds of camera drivers and each might be different, but there is probably not more than one that fits your camera. You find the information what driver is used in the Management Client when you pick the device and look under Hardware properties, Driver.

The drivers are on the other hand all the same in all XProtect and you will find no difference between XProtect Essential+ and XProtect Corporate when it comes to this.

As a solution partner you will have the the MIP SDK which currently does not offer driver development methods.

If you are a camera manufacturer you can join the Camera Partner programme - https://www.milestonesys.com/community/become-a-partner/camera-partner/ where you will have closer contact with the driver development management team.