Ffmpeg-recorded hap not compatible with HapPlayer

I’m using @NickHardeman 's ofxFFmpegRecorder and setting

m_Recorder.setVideoCodec(“hap”);

Videos recorded with this do not play with @mantissa 's ofxDSHapVideoPlayer.

Until now I’ve only ever used ofxDSHapVideoPlayer to play videos converted to hap with VirtualDub, which I believe also uses ffmpeg to make the conversion. Just now, playing a VirtualDub-produced file does work. Does anyone know why this is? I need to record and play up to five videos live so hap seems like it’d be necessary for this due to its high speed.

What kind of file do you actually get when you record? Is it really an AVI file (as is needed for ofxDSHapVideoPlayer)? Is it actually using the HAP codec? You can use a tool like mediainfo to check. If it is an AVI file (not just a .mov file with an AVI extension) then take a closer look at the information from media info - if it is using the HAP codec there are also some more settings for using HAP - I remember some differences with HAP, HAP Alpha and HAPQ when using ofxDSHapVideoPlayer. Compare the mediainfo report of your recorded file and the file that does play back. Most likely there are some settings you can add to the FFMPEG command that will let you end up with a compatible file.

OK, so I downloaded MediaInfo and ran it on two files.

  1. a file that I converted with VirtualDub and which runs in ofxDSHapVideoPlayer. I remember that I selected hapAlpha when converting this:

General
Complete name                            : D:\videoSokkyou2\avi\000.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Format profile                           : OpenDML
File size                                : 5.59 GiB
Duration                                 : 7 min 45 s
Overall bit rate                         : 103 Mb/s
Writing library                          : VirtualDub build 35491/release

Video
ID                                       : 0
Format                                   : Hap5
Codec ID                                 : Hap5
Duration                                 : 7 min 45 s
Bit rate                                 : 103 Mb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 30.000 FPS
Bits/(Pixel*Frame)                       : 3.731
Stream size                              : 5.59 GiB (100%)
  1. A file recorded in oF with @NickHardeman 's ofxFFmpegrecorder fork:
General
Complete name                            : C:\Users\selli\openFrameworks\apps\myApps\videoCrossRecordTest\bin\data\0.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 17.6 MiB
Duration                                 : 3 s 800 ms
Overall bit rate                         : 38.8 Mb/s
Writing application                      : Lavf58.76.100

Video
ID                                       : 0
Format                                   : Hap1
Codec ID                                 : Hap1
Duration                                 : 3 s 800 ms
Bit rate                                 : 38.8 Mb/s
Width                                    : 800 pixels
Height                                   : 448 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 30.000 FPS
Bits/(Pixel*Frame)                       : 3.605
Stream size                              : 17.6 MiB (100%)

So one thing is that it’s Hap1 vs Hap5… though I just found an issue on Github where someone reports not being able to play FFmpeg-generated hap files. Not knowledgeable at all about these things so not sure how to proceed except to post on that topic and pester @mantissa to answer…

This is from some info about HAP found here:

Names and Identifiers

Where Hap frames are present in a stream or container and identifiers are required, the following usage is recommended:

Texture Format(s) Human-Readable Name Four-Character Code
RGB DXT1/BC1 Hap Hap1
RGBA DXT5/BC3 Hap Alpha Hap5
Scaled YCoCg DXT5/BC3 Hap Q HapY
Scaled YCoCg DXT5/BC3 + Alpha RGTC1/BC4 Hap Q Alpha HapM
Alpha RGTC1/BC4 Hap Alpha-Only HapA
RGBA BPTC/BC7 UNORM Hap R Hap7
RGB BPTC/BC6U and BC6S Hap HDR HapH

This implies that HAP5 is actually Hap Alpha.

This is the header for the HAP encode for FFMPEG
https://ffmpeg.org/doxygen/3.3/hapenc_8c.html

It seems to support HAP5

Detailed Description

Hap encoder.

Fourcc: Hap1, Hap5, HapY

Definition in file hapenc.c.

Maybe you can try use HAP Alpha as your codec instead of hap.

Here is a snippet of how that would look in FFMPEG:

ffmpeg -i yourSourceFile.mov -c:v hap -format hap_alpha outputName.mov

This is taken from here:

This is worth a try to solve your issue.

Hey @fresla Thanks a lot for getting this info for me. In @NickHardeman 's ofxFFmpegRecorder::startCustomRecord() I added this if-statement as a test:

args.push_back("-pix_fmt " + mInputPixFmt);
    args.push_back("-vcodec rawvideo");
    args.push_back("-i -");  

    args.push_back("-vcodec " + m_VideCodec); 
    
    if (m_VideCodec == "hap") //added the code from here
    {
        std::cout << "set format " << std::endl;
        args.push_back("-format hap_alpha");
    } //...til here
    args.push_back("-b:v " + std::to_string(m_BitRate) + "k");
    args.push_back("-r " + std::to_string(m_Fps));
    args.push_back("-framerate " + std::to_string(m_Fps));

which seems to work, as MediaInfo now shows recorded files as having the codec Hap5:

General
Complete name                            : C:\Users\selli\openFrameworks\apps\myApps\videoCrossRecordTest\bin\data\0.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 142 MiB
Duration                                 : 24 s 867 ms
Overall bit rate                         : 47.8 Mb/s
Writing application                      : Lavf58.76.100

Video
ID                                       : 0
Format                                   : Hap5
Codec ID                                 : Hap5
Duration                                 : 24 s 867 ms
Bit rate                                 : 47.8 Mb/s
Width                                    : 800 pixels
Height                                   : 448 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 30.000 FPS
Bits/(Pixel*Frame)                       : 4.450
Stream size                              : 142 MiB (100%)

but it still doesn’t play. Here again is the VirtualDub-converted file that plays in ofxDSHapVideoPlayer:

General
Complete name                            : D:\videoSokkyou2\avi\000.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Format profile                           : OpenDML
File size                                : 5.59 GiB
Duration                                 : 7 min 45 s
Overall bit rate                         : 103 Mb/s
Writing library                          : VirtualDub build 35491/release

Video
ID                                       : 0
Format                                   : Hap5
Codec ID                                 : Hap5
Duration                                 : 7 min 45 s
Bit rate                                 : 103 Mb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 30.000 FPS
Bits/(Pixel*Frame)                       : 3.731
Stream size                              : 5.59 GiB (100%)

As I posted here, I’ve become increasingly frustrated and am probably going to use TouchDesigner for this project as HapQ reading & writing works out of the box there. Maybe if I hear from @mantissa I’ll give it another try in the future.

@fresla TD isn’t the magical solution I’d hoped, so I’m still interested in figuring this out. Do you have any ideas, @pierre_tardif00 ?

Just curious what the reason is for using HAP?

Maybe it was mentioned somewhere else? Is it to have as minimal impact as possible on fps when writing the file?

Have you also tried using ofxFFmpegRecorder to write an mp4?
That you would be able to play back easily with ofVideoPlayer.

A while back we were trying to write 4K video as quickly as possible to disk and found that the fastest performance ( while still producing an mp4 ) was using ffmpeg but with a pipe that let you write the frames from memory.

It might take a day or so but I’ve been meaning to clean this code up and share it as it can be quite handy.

Another test that might be useful is to try playing back the videos saved in HAP with the latest VLC client. If it plays fine there it is not an issue with saving but an issue with the playback.

In which case you might have better luck with a GStreamer based player.

1 Like

Just a quick note. This task is kind of hard to do on any platform I have tried it on. There are always issues to overcome. I saw your post about of add ons etc and this is sometimes tough to manage but it gets easier and its also part of the (joy/terror) work of making things with computers. These issues get easier to solve. I like code over node for many reasons.

As for the recording as Theo says hap is not the only way to go. It seems you are on windows, you can also install some lab decoders that will allow you to play back prores (and this is a bit more compatible than hap). Using hardware encoding to record h264 or h265 might also be a good pathway. Ffmpef supports a number of hardware encoders (Nvidia etc). You will need to specify. You can also use a virtual sound card to do some internal routing that will let you do audio processing and record the output using ffmpeg.

You could also hand off the recording to another app like spout recorder and use the command line (maybe via a keystroke utility) to record files and then just play them with OF.

In short the task you want to do is not so simple, there are many ways to get it done and each has some downfalls.

@theo Thanks for your reply. I’ve focused on hap because I need to play back multiple videos simultaneously and hap runs very quickly. If other formats of those that you and @fresla mentioned can also be played back with little slowdown I’m happy to explore them. I used @mantissa 's ofxDSHapVideoPlayer in my last project to play back up to 7 pre-converted hap-avi files simultaneously (and there was only occasional slowdown during quick quick consecutive loading of files) so I wanted to try using it here… If there’s anything as fast as hapthat fits in a mp4 container I’d love to use ofVideoPlayer!

The files recorded with ofxFFmpegRecorder do indeed play in VLC. But ofxDSHapVideoPlayer apparently has some requirement they’re not meeting.

and thanks, @fresla As I said, I need fast playback of five videos for this project, would those codects work for that? I think for sound, I’ll just stick with Pure Data and OSC for this project (getting video working is hard enough) The audio recording works great already and I’ll also be able to add effects. I can use OSC to set the video frames in oF… once I get video working.

@s_e_p, I don’t have much insight to offer you here. To be honest, you’re asking me about a project I worked on for work 7 years ago (with a handful others). Nor am I an export on HAP encoding.

I haven’t tried recording a HAP file in realtime but why is this a requirement? Why not record something that works (.png sequences or .mp4) and then transcode it to HAP using your preferred tool?

If you really wanted to figure this out, try digging into (i.e. debugging) the ofxDSHapVideoPlayer code and see what the direct show player thinks it is playing. If will likely complain or fail if the codec settings are not recognized.

Good luck!

@mantissa Hey, thanks for your reply. I want to play back the videos immediately after recording (so there’s no time for conversion) and play back multiple videos simultaneously. Would this work with png sequences or mp4? (if png sequence just means writing a single png per frame I imagine the size would end up being much larger than hap…)

just jumping in quickly to say that I’ve never seen HAP (or HAP style codecs) work in realtime. they always take a long time and seem very intensive to compress (but are very fast on playback).

my guess is that png sequences and mp4s with certain settings would work well for saving quickly and also replaying quickly, but in all cases you probably need to experiment to find the thing that impacts frame rates the least. For example, it’s not png related, but this add-on is for faster jpeg loading and saving

and I’d experiment with timing. maybe also look at threaded image saving, if saving images is having an impact on frame rate.

Also, is there a reason to write to disk? For example, I’ve done things with arrays to textures before to record video in memory. That can be tremendously faster than round tripping to disk.

Hey, yes, I’m trying to have video delay of a variety of lengths.

a circle buffer in memory works fine for up to around 5 minutes, but I think I need to write to disk if I want to have longer delays. Is saving an hour’s worth of jpegs/pngs the answer…? (Png I guess since I want transparency) My 500gb ssd might not suffice.

I do get amazing playback results with HAP so I understand the desire to use it. H254/5 are definitely worth testing too though. Another option could be to record an easy format using this addon and then use virtual dub via the command line to reencode the files to HAP. This is a little clunky but you can package the virtual dub exe with your app (likely not for distribution). This way it will still be portable. OF can mange files and versions. You can even use it to join separate audio and video files into one.

@mantissa I thought the textureFormat variable and the enum for hap type might be important. Printed the variable for both files (virtualdub generated hap and ofxFFmpegRecorder generated hap) and both have the same value corresponding to Hap Alpha.

@NickHardeman @pierre_tardif00 Does either of you know what the correct color format is for various codecs, or how to find out? If I set it in setup to OF_IMAGE_COLOR_ALPHA before recorsing, the console says something like

‘rgb24’ not compatible with ‘hap’ codec. Switching to ‘rgba’

(I think this happens for h264 also)Is there a way to find out what oF keyword corresponds to the correct ‘rgba’ color format? This might be the key to getting ofxDSHapVideoPlayer to play the file properly.

I recently ( about a week ago ) pushed some changes to account for rgba video pixels. Check to make sure these changes are in your ofxFFmpegRecorder

and you can also set the output type here:

Thinking more about what you are trying to do ( which is definitely non-trivial :slight_smile: ) .

I would maybe skip HAP all together ( for one thing a 60 minute HAP file will be absolutely huge ) and instead try doing doing 10 minute mp4 videos where you save the to a folder like so:

recorded-video-100001.mp4
recorded-video-100002.mp4
recorded-video-100003.mp4
recorded-video-100004.mp4

As soon as you have a video saved you play it back ( with ofVideoPlayer ) and then when the video reaches the end you close it, delete the file ( with ofFile::removeFile ) and play the next one in the list.

If you can get this working then you might find you need to make it a little more complex where you have multiple ofVideoPlayer’s and you load the next one close to one the first one is done to reduce any time between the end of video 1 and the start of video 2.

Adding audio to an existing file with a raw ffmpeg command is not that hard. So you could potentially handle that separately.

Anyway just in terms of the general approach that would be my take.

Or just saving out raw jpgs ( faster disk write ) / pngs ( slower disk write ) and playing back the folder of pngs from oldest to newest.

Yes @NickHardeman solution v elegant. You don’t find v often rgba pixels alignement in cameras in the wild, so not seen it myself much as a pixel alignement format.

It is all dependant on your stride (pixels alignement) and the FFmpeg codec you use to encode your data.

In the case of mpeg4, you can check which flags you can add by doing :
./ffmpeg -h encoder=mpeg4 -hide_banner

My custom FFmpeg isn’t compiled with Hap codec, can’t help with it sorry, but if you assign the video codec using setVideoCodec, you could do it.
I would honnestly drop it and record rgba as @NickHardeman showed it as flags will need tuning,a dn i would stir away from OF to test only in command line but it’s not super simple to do…
Then if yo are set on HAP, you can transcode it easy, there are examples online.

EDIT: @zach ofxTurboJpeg seems to be working very well- I can draw four delays + the camera feed @ 32-35fps. We’ll see how it goes when I start adding effects, but it’s a start!

My computer died and I had to buy a new one but got back on this project today. I’m trying three approaches based on your suggestions. Here’s how they’re going.

(N.b: I want to display up to five different videos at once with five different delays. I might want to change these, but for now it’d be: 0 seconds, 5 seconds, 30 seconds, 5 minutes and 1 hour. For 0 I can just draw the camera feed. For 5 and 30 seconds I could use a shared circular buffer. For 5 minutes and more are what I need a solution for, though of course it’d be more elegant to use the same solution for everything)

Solution 1:@theo I’ve tried your suggestion of writing raw jpgs. It seems to work well enough for just one image stream @ 30fps, but loading and drawing a second image every frame drops the framerate to the 21-23 range. My code is as follows:

void ofApp::update(){
	m_camera.update();
	ofImage f = m_camera.getPixels();
	f.save(std::to_string(frameIndex++) + ".jpg");
}

//--------------------------------------------------------------
void ofApp::draw(){
	int readIndexA = frameIndex - 30;
	if (readIndexA < 0)
	{
		return;
	}
	ofImage a,b,c,d;
	a.load(std::to_string(readIndexA) + ".jpg");
	
	int readIndexB = frameIndex - 25;
	b.load(std::to_string(readIndexB) + ".jpg");
	//int readIndexC = frameIndex - 20;
	//c.load(std::to_string(readIndexC) + ".jpg");
	//int readIndexD = frameIndex - 10;
	//d.load(std::to_string(readIndexD) + ".jpg");
	ofEnableBlendMode(OF_BLENDMODE_ADD);
	
	ofSetColor(255, 255, 255, 170);
	a.draw(0, 0, 1920, 1080);
	b.draw(0, 0, 1920, 1080);
	//c.draw(0, 0, 1920, 1080);
	//d.draw(0, 0, 1920, 1080);
	ofDisableAlphaBlending();

	ofSetColor(0, 255, 0);
	ofDrawBitmapStringHighlight("FPS: " + std::to_string(ofGetFrameRate()), 10, 16);
}

While writing this I just re-noticed @zach 's message about /ofxTurboJpeg. Will give that a shot next!

Solution 2: Continue trying to get ofxDSHapVideoPlayer working with ofxFFmpegRecord-ed videos: @NickHardeman Using setOutputPixelFormat gets rid of the warning/auto-format-setting message but setting it to the OF_IMAGE_COLOR_ALPHA doesn’t help, even though this is the same pixel format used by ofxDSHapVideoPlayer for hapAlpha:

//from ofxDSHapVideoPlayer.cpp 
 else if (textureFormat == HapTextureFormat_RGBA_DXT5){
				 texData.glInternalFormat = GL_COMPRESSED_RGBA_S3TC_DXT5_EXT;
				 pix.allocate(width, height, OF_IMAGE_COLOR_ALPHA);

@pierre_tardif00 I tried ffmpeg -h encoder=hap -hide_banner like you said and got this:

  -format            <int>        E..V.... (from 11 to 15) (default hap)
     hap                          E..V.... Hap 1 (DXT1 textures)
     hap_alpha                    E..V.... Hap Alpha (DXT5 textures)
     hap_q                        E..V.... Hap Q (DXT5-YCoCg textures)
  -chunks            <int>        E..V.... chunk count (from 1 to 64) (default 1)
  -compressor        <int>        E..V.... second-stage compressor (from 160 to 176) (default snappy)
     none                         E..V.... None
     snappy                       E..V.... Snappy

Besides format, which I know is being set correctly, could chunks or compressor have something to do with the problem…? I also ran ffmpeg -i on both the compatible virtualDub video and the incompatible ofxFFmpegRecorded video files and got this:

VirtualDub:

[avi @ 00000000006c7380] mov tag found in avi (fourcc Hap5)
[avi @ 00000000006c7380] non-interleaved AVI
Input #0, avi, from '120.avi':
  Duration: 00:01:18.63, start: 0.000000, bitrate: 427040 kb/s
    Stream #0:0: Video: hap (Hap5 / 0x35706148), rgba, 1920x1080, 427118 kb/s, 59.94 fps, 59.94 tbr, 59.94 tbn, 59.94 tbc

ofxFFmpegRecorder:

[avi @ 0000000000e97380] mov tag found in avi (fourcc Hap5)
Input #0, avi, from '0.avi':
  Metadata:
    encoder         : Lavf57.67.100
  Duration: 00:00:02.70, start: 0.000000, bitrate: 120392 kb/s
    Stream #0:0: Video: hap (Hap5 / 0x35706148), rgba, 1280x720, 121874 kb/s, 30 fps, 30 tbr, 30 tbn, 30 tbc

Do you think the lack of the “non-interleaved AVI” data point in the second video is concerning? I guess the next step is to follow the ofxDSHapVideoPlayer’s load() and play() through each line of code and spot differences between what happens with each of the two videos, though that’s so tedious I think I’ll try to advance the other two solutions first.

Solution 3: Trying to record in other codecs: @fresla Did you mean H264/265 ? I didn’t see 254/265 on the codec list for the ffmpeg version included with ofxFFmpegRecorder. 5 video playback with ofVideoPlayer seems to run smoothly enough with h264 avis… Trying to switch between reading a file and writing it on the same frame doesn’t work, so maybe I need three files per delay: one for writing, one for loading and one for recording. This seems inefficient, but I’ll give it a try if I don’t make progress with the jpeg solution. Haven’t tried .mp4 yet. @theo Do you recommend a fast codec for mp4?

@theo @zach Any ideas for how to sync audio with the jpeg stream? I’m recording and playing both in two different ways and, while it works for a delay of a few seconds, a 30-minute delay produces a disparity of a couple of seconds:

Video. I set ofSetFrameRate(30) in setup() and…

void ofApp::update()
{
	ofImage f = m_camera.getPixels();
	m_turbo.save(&f, std::to_string(m_frameIndex++) + ".jpg", 75);
	if (m_frameIndex >= m_maxDelayFr)
	{
		m_frameIndex = 0;
	}

    int delayedFrame = m_frameIndex - delay;
	int f = delayedFrame >= 0 ? delayedFrame : delayedFrame + maxDelay;
	currentFrame = f;
}
void ofApp::draw()
{
    ofImage img; 
	string file = std::to_string(currentFrame) + ".jpg";
	turbo.load(img, file);
	if (img.isAllocated())
	{
		img.draw(rect);
	}
}

Audio: In Pure Data, I record one wav-file and, as soon as its done, I start reading it and writing a second file and continue alternating between the two. I could send an OSC value from Pure Data corresponding to the reading of the wav-files, but I’m not sure how to integrate it into the oF code… I guess I could send a single OSC message whenever I start to read a wav-file to get the current m_frameIndex value, then increment from there based on a stream of OSC messages. I’ll try it out. EDIT: This works! Woo!