I’m currently messing around with ARKit and one issue I’m having is learning about how to deal with color spaces. The camera feed from the ARKit api supplies the data in YCbCr format and I’m trying to convert it to RGB instead. I’ve tried to adapt the Metal example but it doesn’t appear to work correctly(or I’m just doing something wrong :p)
This is what I believe is doing the conversion in the Metal shader
This is my poor attempt at conversion
If it helps, textures I’m bringing into GLSL
have linear filtering
wrap is set to CLAMP_TO_EDGE
they’re being read as GL_LUMINANCE textures from the CVPixelBufferRef that ARKit supplies.
Any ideas? going on google hasn’t been too helpful either.
Hey Zach - appreciate you being willing to take a look, uploaded everything here
I don’t know how you normally work but I was just working outta the app/myApps folder from the main oF folder. Basically follows the iosNative example, minus extra view controllers and apps I didn’t need. ARKit stuff is getting handled in ARProcessor.h / ARProcessor.mm, hopefully the rest is pretty straightforward.
Small sidenote - I know I put shaders directly in the c++ files, but code is still referencing shaders in the bin/data directory; gonna finish moving that over later
I’ll take a stab trying out the code in that stackoverflow post a bit later, thanks for that! Not sure how I didn’t come across it already.
I figured it out! … kind of haha. The overall tone is a bit warmer than it ought to be I think, but for the most part the colors are now correct and good enough, but I might take a stab at tweaking it later. Turns out I didn’t quite have the right texture setup going and I had to tweak the texture creation function a bit.
Figured it out after perusing through the code for this project
and looked at how Brad was setting things up in this file
as well as looking at his fragment shader which is here
cool ! your code is really helpful for getting started. I mucked around a bit and threw up and example project which draws images in 3d when you tap
on dropbox / and linked to it from an issue on github. Once I understand more I’ll do a proper PR but there might be bits and pieces from what I did which might be useful like converting from simd matrix to ofMatrix, etc.
Sweet! Appreciate the kind words, I’m glad it was helpful for getting started with things!
Sadly I’ve been roped into a more important, but much less interesting project at my day job so my free time is now pretty much shot but I’ll definitely take a look at some point and see if there’s anything I can roll in quickly(assuming you don’t get to a PR before I get to check things out).
Really still just getting started myself in all of this so anything else you see that could use improvement by all means, please feel free to let me know.
I saw your github issue, I’ll respond to that on there
Started to play around with AnchorPlanes and it seems that my camera image is distorted. I checked, the incoming camera pixel resolution is 1280x720 while the ipad display resolution is 2732x2048 so they can’t fit. Is there a common way to undistort/recover the image?
I don’t have my mac or a device in front of me at the moment - but off the top of my head maybe it’s the project settings? I seem to recall them being set to default as a Universal App but specifically for iPhones.
Another thing you could try - I don’t remember why I switched it but when I’m grabbing the textures, I’m using CVPixelBufferGetWidth and there is a CVPixelBufferGetWidthOfPlane (similar functions for height), that might be something I need to switch out
Weird - that’s really interesting, I totally would have never spotted that difference.
Thanks for figuring that out! Kinda timely too since I was gonna eventually get the camera image rendering into an FBO anyways. I should have a little bit more time today so I’ll get that rolled in as soon as I can.
I also saw the video distortion issue, it’s nice to see a side by side and it’s pretty easy to crop in
do you want to change the repo to be more of an addon, ie “ofxArKit” – that way we can better post issues / PRs ? I’ve been looking at how to detect horizontal planes and it would be great to have a place to discuss what issues we are running into / have branches on gh, etc.
yup! That’s basically what I was thinking about the past couple of days and started cleaning things up and trying to flush things out into more of a addon format, with the intention of having it be both oF and Cinder compatible.
I was going to wait till I had things working but I suppose there’s no harm in starting it now too. I’ll get to that right now