YCbCr to RGB conversion (IOS and ARKit related) via GLSL

iOS
#1

Hello!

I’m currently messing around with ARKit and one issue I’m having is learning about how to deal with color spaces. The camera feed from the ARKit api supplies the data in YCbCr format and I’m trying to convert it to RGB instead. I’ve tried to adapt the Metal example but it doesn’t appear to work correctly(or I’m just doing something wrong :p)

This is what I believe is doing the conversion in the Metal shader

This is my poor attempt at conversion

If it helps, textures I’m bringing into GLSL

  • have linear filtering
  • wrap is set to CLAMP_TO_EDGE
  • they’re being read as GL_LUMINANCE textures from the CVPixelBufferRef that ARKit supplies.

Any ideas? going on google hasn’t been too helpful either.
Thanks!

#2

I was just about to dig into arkit – if you want to post any progress on github happy to take a look.

I wonder if this would help?

#3

Hey Zach - appreciate you being willing to take a look, uploaded everything here

I don’t know how you normally work but I was just working outta the app/myApps folder from the main oF folder. Basically follows the iosNative example, minus extra view controllers and apps I didn’t need. ARKit stuff is getting handled in ARProcessor.h / ARProcessor.mm, hopefully the rest is pretty straightforward.

Small sidenote - I know I put shaders directly in the c++ files, but code is still referencing shaders in the bin/data directory; gonna finish moving that over later

I’ll take a stab trying out the code in that stackoverflow post a bit later, thanks for that! Not sure how I didn’t come across it already.

#4

I figured it out! … kind of haha. The overall tone is a bit warmer than it ought to be I think, but for the most part the colors are now correct and good enough, but I might take a stab at tweaking it later. Turns out I didn’t quite have the right texture setup going and I had to tweak the texture creation function a bit.

Figured it out after perusing through the code for this project

and looked at how Brad was setting things up in this file

as well as looking at his fragment shader which is here

I updated the Github repo too

#5

cool ! your code is really helpful for getting started. I mucked around a bit and threw up and example project which draws images in 3d when you tap

on dropbox / and linked to it from an issue on github. Once I understand more I’ll do a proper PR but there might be bits and pieces from what I did which might be useful like converting from simd matrix to ofMatrix, etc.

#6

Sweet! Appreciate the kind words, I’m glad it was helpful for getting started with things!

Sadly I’ve been roped into a more important, but much less interesting project at my day job so my free time is now pretty much shot :disappointed: but I’ll definitely take a look at some point and see if there’s anything I can roll in quickly(assuming you don’t get to a PR before I get to check things out).

Really still just getting started myself in all of this so anything else you see that could use improvement by all means, please feel free to let me know.

I saw your github issue, I’ll respond to that on there :tada:

#7

yes, session.currentFrame.camera.transform did the trick for me, too. also, when drawing anchors, i did the following with the projection & transformation matrix:

void ARProcessor::drawAnchors(){
    cam.begin();
    ofSetMatrixMode(ofMatrixMode::OF_MATRIX_PROJECTION);
    ofLoadMatrix(cameraProjection);
    ofSetMatrixMode(ofMatrixMode::OF_MATRIX_MODELVIEW);
    cam.setTransformMatrix(cameraTransform);

    for(int i=0; i < [session.currentFrame.anchors count]; i++ ) {
        ofMatrix4x4 anchor_transform = convert<matrix_float4x4, ofMatrix4x4>(session.currentFrame.anchors[i].transform);
        ofPushMatrix();
        ofMultMatrix(anchor_transform);
          // draw here ...
        ofPopMatrix();
    }
    cam.end();
}

dunno if it’s the most effective way, but it works as you can see on this video: https://twitter.com/_stc/status/895251580567552004

#8

Started to play around with AnchorPlanes and it seems that my camera image is distorted. I checked, the incoming camera pixel resolution is 1280x720 while the ipad display resolution is 2732x2048 so they can’t fit. Is there a common way to undistort/recover the image?

IMG_0283.jpg2732×2048 654 KB

#9

I don’t have my mac or a device in front of me at the moment - but off the top of my head maybe it’s the project settings? I seem to recall them being set to default as a Universal App but specifically for iPhones.

Another thing you could try - I don’t remember why I switched it but when I’m grabbing the textures, I’m using CVPixelBufferGetWidth and there is a CVPixelBufferGetWidthOfPlane (similar functions for height), that might be something I need to switch out

I’ll check it out when I get a moment

#10

made a quick comparison: upper screenshot is openframeworks, bottom is an empty metal / ar view. seems that apple is cropping the camera image and showing only a part of it

IMG_0284.PNG942×1640 1.06 MB

so i made an ofFbo and remap the camera image to the background plane:
in setup:

// our actual capture resolution
int captureWidth = 1280;
int captureHeight = 720;

cFbo.allocate(captureWidth,captureHeight,GL_RGBA);
float ratio = captureWidth / float(captureHeight);
cFboWidth = ofGetHeight() * ratio;
cFboHeight = ofGetHeight();

and in draw:

cFbo.begin();
cameraShader.begin();
cameraPlane.draw();
cameraShader.end();
cFbo.end();
cFbo.draw((ofGetWidth()-cFboWidth)/2,0,cFboWidth,cFboHeight);

IMG_0285.PNG926×1569 1.07 MB

so the camera image is cropped a bit and aligned in the middle of the scene, as with the raw apple-metal view as you see on the second image. hope this helps a bit…

#11

Weird - that’s really interesting, I totally would have never spotted that difference.

Thanks for figuring that out! Kinda timely too since I was gonna eventually get the camera image rendering into an FBO anyways. I should have a little bit more time today so I’ll get that rolled in as soon as I can.

#12

I also saw the video distortion issue, it’s nice to see a side by side and it’s pretty easy to crop in :slight_smile:

do you want to change the repo to be more of an addon, ie “ofxArKit” – that way we can better post issues / PRs ? I’ve been looking at how to detect horizontal planes and it would be great to have a place to discuss what issues we are running into / have branches on gh, etc.

#13

I decided what the heck - it’s not a huge change, figured I might as well at least start doing this now before I have to go to work haha.

I’ve pushed your suggested changes but I don’t have a device at the moment in front of me to test with so it may break if you decide to try it :smile:

But I will definitely test it later on.

#14

yup! That’s basically what I was thinking about the past couple of days and started cleaning things up and trying to flush things out into more of a addon format, with the intention of having it be both oF and Cinder compatible.

I was going to wait till I had things working but I suppose there’s no harm in starting it now too. I’ll get to that right now

1 Like
#15

@stc @zach ok!

addon setup is done and I believe working. I sent collaborator invites to the both of you to make things a little easier to contribute any changes you might want to help with.

The way i generally prefer to work is that changes ought to be made to a develop branch, then merged into master once things are confirmed working

to everyone else, PRs are more than welcome!

Now I gotta finish getting ready to go to my day job gahhhh. haha