We have implemented a driver with the Driver Framework for a body-worn camera. The livestream pulls through correctly on the Smart Client only when hardware acceleration is enabled.
We have tested on a few machines which does not support hardware acceleration, where the feed does not show. On a machine which does support hardware acceleration, the moment we turn off hardware acceleration, the feed does not work anymore, neither does the recorded stream.
I have extracted the raw h264 data from the export that the customer did provide, so an MKV export is no longer needed.
I debugged the decoder, and now this is going to be a bit technical:
Any h264 video begins with a data structure that contains a byte (profile_idc) that indicates the decoding standard that the video adheres to, and for the video in question that number is 249:
Now, 249 is not a value that we recognize as a valid profile ID, so the decoder returns UMC_ERR_UNSUPPORTED. I looked in the specs for h264 to see if it is a new value that has been introduced since our decoder was implemented, and as far as I can tell it is not.
That leads us to the question of why the Intel and Nvidia hardware decoders will decode the video, and there we don’t have the luxury to debug right in to the source code, but one could guess that they are simply less strict about validating the profile_idc parameter, and make an assumption about what 249 means, which happens to be correct.
We looked up the Yulong BWC-R3H camera, and it seems to come from a Chinese company that has stopped sharing news on their official web page after the year 2020, which in the first place looks like it is basically a template that hasn’t been fully customized and finalized. I suspect it might be a bit difficult to get in touch with them and ask them what is going on.
Is the format that Milestone expects in the H264 data field documented somewhere perhaps, ideally with an example? I’m de-encapsulating the stream and I suspect I’m doing it incorrectly.
I’m only referring to the official specifications. It’s an extremely dry document that defines the specs with almost mathematical precision, and it doesn’t have any examples (https://www.itu.int/rec/T-REC-H.264-202408-I/en). Details about video decoding standards is not exactly a topic that is well covered with examples and tutorials online. However, paragraph 7.4.2.1.1 on page 76 gives some insight into how the profile_idc parameter is constructed. Notice that the last two bits are reserved zero bits, and the value 249 that we got from the attached file is an odd number, so one of the mandatory zero bits is actually set!
It is interesting that the paragraph describes individual bits, while the rest of the document only mentions specific values that the field can have, like on page 46 it says-
The developer told me that the details about the constraint_set flag is not correct in his reply, but maintains that profile_idc value 249 is not valid
I mean, do I just pop the H264 NAL units in the byte[] arrays into Milestone, starting with 00 00 00 01 or 00 00 01 ? If that’s not the case, I need to revisit my code.
If that’s not the case, a single byte array example of a correct data binary keyframe that XProtect expects would really help! I have the parseable H264 on my end, but I am unsure what headers and other encapsulation and the stuff around it should be when passing it to XProtect as H264 directly with the VideoCodecType.H264 flag.
Apologies for the lots of edits in the previous post.
This is an example of what I get for the first keyframe sent to XProtect. It seems correct on my end, making me thing maybe I should have another encapsulation rather than pure H264 NAL units? I have compared this to an ffmpeg remux to direct H264 as well.
The below is sent as a single entry into the byte array. I have tried many things, but it seems prepending the SPS and PPS to keyframes works better to decode with acceleration, maybe giving a hint as to what I’m doing wrong. I am working in the dark as I have no documentation regarding what XProtect expects with the H264 codec, which is why I’m asking.
Prepending the GenericByteHeader did not work, and I note the PlaybackReadResponse as well as the GetLiveFrameResult methods already have entries for the header structures in the driver development framework.
I am unsure how to proceed here. If I could get a single example that worked what you expect in the data field, pretty sure it would get me through this problem.
Another option that might work is if you send me the extracted datastream, maybe? That might give me a hint as to what I’m incorrectly passing to the system. A working example, however, would be preferable as I then don’t need to guess.
Hi, I am not sure exactly what is going wrong, but maybe the following can assist in finding the root cause of the issue. Here is the code from our H.264 test driver’s GetLiveFrameInternal:
Thanks SO much!! The rawvideo dump allowed me to figure out what’s wrong with the stream and fix it. The level_idc parameter in the sequence parameter set was incorrect in the H264 header. Updating that to the correct value fixed the problem.
