openFrameworks and openCv image processing issue with doing analysing video and rendering manipulated images back to the user with color palette

I tried posting this on StackOverflow but didn’t receive any useful help so am hoping the community here will help a beginner out. So to help explain my problem I will explain a bit what my end goal is. I am a OF beginner so the context of if this is a beginner topic or an advanced topic I hope I have right :smile:

Anyways… I am working on a project with OpenFrameworks using ofxCV, ofxOpencv and ofxColorQuantizer. Technically, the project is analyzing live video captured via webcam and analysis’s the image in real time to gather and output the most prominent color in the current frame. When generating the most prominent color I am using the pixel difference between the current frame and the previous frame to generate the what colors have updated and use the updated or moving areas of the video frame to figure out the most prominent colors.

The reason for using the pixel difference’s to generate the color pallet is because I want to solve for the case of a user walks into the video frame, I want try and gather the color pallet of the person, for instance what they are wearing. For example red shirt, blue pants will be in the pallet and the white background will be excluded.

I have a strong background in Javascript and canvas but am fairly new to OpenFrameworks and C++ which is why I think I am running into a roadblock with this problem I described above.

Along with OpenFrameworks I am using ofxCV, ofxOpencv and ofxColorQuantizer as tools for this installation. I am taking a webcam image than making it a cv:Mat than using pyrdown on the webcam image twice followed by a absdiff of the mat which I am than trying to pass the mat into the ofxColorQuantizer. This is where I think I am running into problems — I don’t think the ofxColorQuantizer likes the mat format of the image I am trying to use. I’ve tried looking for the different image format to try and convert the image to to solve this issue but I haven’t been able to come to solution.

For efficiencies I am hoping to to the color difference and color prominence calculations on the smaller image (after I pyrdown’ the image) and display the full image on the screen and the generated color palette is displayed at the bottom left like in the ofxColorQuantizer example.

I think there may be other ways to speed up the code but at the moment I am trying to get this portion of the app working first.

I have my main.cpp set up as follows:

#include "ofMain.h"
#include "ofApp.h"
#include "ofAppGlutWindow.h"

//========================================================================
int main( ){
    ofAppGlutWindow window;
    ofSetupOpenGL(&window, 1024,768, OF_WINDOW);            // <-------- setup the GL context

//  ofSetupOpenGL(1024,768,OF_WINDOW);          // <-------- setup the GL context

    // this kicks off the running of my app
    // can be OF_WINDOW or OF_FULLSCREEN
    // pass in width and height too:
    ofRunApp(new ofApp());

}

My ofApp.h file is as follows:

 #pragma once

#include "ofMain.h"

#include "ofxOpenCv.h"
#include "ofxCv.h"
#include "ofxColorQuantizer.h"

class ofApp : public ofBaseApp{

    public:
        void setup();
        void update();
        void draw();

        ofVideoGrabber cam;
        ofPixels previous;
        ofImage diff;

        void kMeansTest();

        ofImage image;
        ofImage img;
        cv::Mat matA, matB;

        ofImage diffCopy;
        ofImage outputImage;

        ofxCv::RunningBackground background;

        ofxColorQuantizer colorQuantizer;

        // a scalar is like an ofVec4f but normally used for storing color information
        cv::Scalar diffMean;

};

And finally my ofApp.cpp is below:

#include "ofApp.h"

using namespace ofxCv;
using namespace cv;

//--------------------------------------------------------------
void ofApp::setup(){

    ofSetVerticalSync(true);
    cam.initGrabber(320, 240);

    // get our colors
    colorQuantizer.setNumColors(3);

    // resize the window to match the image
    //  ofSetWindowShape(image.getWidth(), image.getHeight());
    ofSetWindowShape(800, 600);

    // imitate() will set up previous and diff
    // so they have the same size and type as cam
    imitate(previous, cam);
    imitate(diff, cam);

    imitate(previous, outputImage);
    imitate(diff, outputImage);

}

//--------------------------------------------------------------
void ofApp::update(){

    cam.update();
    if(cam.isFrameNew()) {

        matA = ofxCv::toCv(cam.getPixelsRef());
        ofxCv::pyrDown(matA, matB);
        ofxCv::pyrDown(matB, matA);
        ofxCv::medianBlur(matA, 3);

        ofxCv::toOf(matA, outputImage);

        // take the absolute difference of prev and cam and save it inside diff
        absdiff(previous, outputImage, diff);
    }

}

//--------------------------------------------------------------
void ofApp::draw(){

    // If the image is ready to draw, then draw it
    if(outputImage.isAllocated()) {
        outputImage.update();
        outputImage.draw(0, 0, ofGetWidth(), ofGetHeight());
    }

    ofBackground(100,100,100);

    ofSetColor(255);

    ofImage diffCopy;
    diffCopy = diff;
    diffCopy.resize(diffCopy.getWidth()/2, diffCopy.getHeight()/2);


    // there is some sort of bug / issue going on here...
    // prevent the app from compiling
    // comment out to run and see blank page
    colorQuantizer.quantize(diffCopy.getPixelsRef());

    ofLog() << "the number is " << outputImage.getHeight();
    ofLog() << "the number is " << diffCopy.getHeight();

    ofSetColor(255);
    img.update();
    //  cam.draw(0, 0, 800, 600);
    outputImage.draw(0, 0, 800, 600);

    //  colorQuantizer.draw(ofPoint(0, cam.getHeight()-20));
    colorQuantizer.draw(ofPoint(0, 600-20));

    // use the [] operator to get elements from a Scalar
    float diffRed = diffMean[0];
    float diffGreen = diffMean[1];
    float diffBlue = diffMean[2];

    ofSetColor(255, 0, 0);
    ofRect(0, 0, diffRed, 10);
    ofSetColor(0, 255, 0);
    ofRect(0, 15, diffGreen, 10);
    ofSetColor(0, 0, 255);
    ofRect(0, 30, diffBlue, 10);

}

//--------------------------------------------------------------
void ofApp::kMeansTest(){

    cv::Mat samples = (cv::Mat_<float>(8, 1) << 31 , 2 , 10 , 11 , 25 , 27, 2, 1);
    cv::Mat labels;

    // double kmeans(const Mat& samples, int clusterCount, Mat& labels,
    cv::TermCriteria termcrit;
    int attempts, flags;
    cv::Mat centers;
    double compactness = cv::kmeans(samples, 3, labels, cv::TermCriteria(), 2, cv::KMEANS_PP_CENTERS, centers);

    cout<<"labels:"<<endl;
    for(int i = 0; i < labels.rows; ++i)
    {
        cout<<labels.at<int>(0, i)<<endl;
    }

    cout<<"\ncenters:"<<endl;
    for(int i = 0; i < centers.rows; ++i)
    {
        cout<<centers.at<float>(0, i)<<endl;
    }

    cout<<"\ncompactness: "<<compactness<<endl;
}

Apologies in advance for the state of my code — I’m still getting into C++ and openFrameworks.

My question is what is the image format openFrameworks is using for grabbing the webcam image, what is the image format that openCV expects and what should I use to switch back from a mat image to an ofImage and is there a way to getPixelsRef from a mat image?

The area of code that I think I have something wrong is the following logic.

I have this line of code which gets the video frame from the webcam matA = ofxCv::toCv(cam.getPixelsRef());
Than do a couple ofxCv procedures on the frame such as ofxCv::pyrDown(matA, matB); which I think changes the image format or pixel format of the frame
Than I convert the frame back to OF with ofxCv::toOf(matA, outputImage);,
Next I get the difference in the pixels between the current frame and the last frame, create a copy of the difference between the two frames. Potentially the issue lies here with the diff output image format
Which I pass the diff copy to colorQuantizer.quantize(diffCopy.getPixelsRef()); to try and generate the color palette in for the change in pixels.
It is the colorQuantizer class and function call that is giving me an error which reads thread error [ error ] ofTexture: allocate(): ofTextureData has 0 width and/or height: 0x0
with an EXC_BAD_ACCESS

And lastly, could there be an alternative cause for the exc_bad_access thread error rather than image formatting? Being new to c++ I’m just guessing and going off instinct of what I think the rood cause of my problem is.

I know there’s a lot here so any help or direction would be greatly appreciated. OR if you think I am attacking this problem from the wrong angle I would love to hear an alternative path to go down :smile:

Many thanks in advance.