Strange EXC_BAD_ACCESS crash / mem leak in ofxJSON with ofxHttpUtils?

dennyroberts · May 15, 2015, 5:07pm

So part of my program tries to upload an image to an API using ofxJSON and ofxHttpUtils. It’s in a separate thread from the main part of the program. Here’s the code and it all works fine when there is internet. It loops over every file in an upload directory, and then individually tries to push them to the API.

Basically how I understand it is that when you addForm() to the httpUtils object (in this case, upload_httpUtils), it’s linked to a listener that fills a JSON object with whatever it hears (upload_test). When we hear that upload_test["image_format"] is “gif” instead of null (part of the response from the API), then it knows it’s uploaded successfully .

    // go through and try to upload everything in that directory
    for(int i = 0; i < items; i++){            
        ofLogNotice() << timestamp() << "Uploading gif " + dir.getName(i);
        cout << "Uploading " + dir.getName(i) << endl;
        string code = dir.getName(i).substr(dir.getName(i).find(".")-6,6);
        stringstream timeSS;
        timeSS << ofGetUnixTime();
        string time = timeSS.str();
        string format = dir.getName(i).substr(dir.getName(i).find(".")+1,3);  // get the filetype
        
        // communicate with API
        ofxHttpForm form;
        form.action = upload_action_url;
        form.method = OFX_HTTP_POST;
        form.addFormField("booth_number", boothKey);
        form.addFormField("pw", "oO*r$U$kgU%Xg$cQu41&");
        form.addFormField("image_code",code);
        form.addFormField("event_id", eventID);
        form.addFormField("event_slug", eventSlug);
        form.addFormField("orientation","vertical");
        form.addFormField("time_stamp",time);
        form.addFormField("image_format",format);
        form.addFile("my_photo",dir.getPath(i));
        
        upload_httpUtils.addForm(form);
        
        // look for a response
        timeOut=0;
        cout << "Attempting upload... " << endl;
        while (timeOut < 30) { // try to upload for 30 seconds
            Json::Value formatJSON = upload_test["image_format"]; // *** this is the line that crashes
            format = ofToString(formatJSON);
            format.erase(std::remove(format.begin(), format.end(), '\n'), format.end());
            format.erase(std::remove(format.begin(), format.end(), '\"'), format.end());
            
            if (format == "gif"){
                // if successfully uploaded, then move the file
                cout << "Successfully uploaded gif " << dir.getName(i) << endl;
                ofLog() << timestamp() << "Successfully uploaded gif " << dir.getName(i);
                
                // reset the response listener
                string empty = "{}";
                upload_test = empty;

                // move the gif file out of the upload directory
                string moveString = "mv -f " + dir.getPath(i) + " " + eventPath + "/gifs";
                const char * command = moveString.c_str();
                system(command);
                break;
            }
            
            ofSleepMillis(1000);
            timeOut+=1;
            if (timeOut	== 30) {
                cout << "Couldn't upload " << dir.getName(i) << endl;
                ofLogError() << timestamp() << "Couldn't upload " << dir.getName(i);
                
                ofLogError() << timestamp() << "booth_number: " << boothKey;
                ofLogError() << timestamp() << "pw: " << "oO*r$U$kgU%Xg$cQu41&";
                ofLogError() << timestamp() << "image_code: " << code;
                ofLogError() << timestamp() << "event_id: " << eventID;
                ofLogError() << timestamp()  << "event_slug: " << eventSlug;
                ofLogError() << timestamp() << "orientation: " << "vertical";
                ofLogError() << timestamp() << "time_stamp: " << time;
                ofLogError() << timestamp() << "image_format: " << format;
                ofLogError() << timestamp() << "my_photo: " << dir.getPath(i);
                
                string empty = "{}";
                upload_test = empty;
                upload_httpUtils.clearQueue();
            }
        }
        if (timeOut > 30) {
            break;
        }

So this all works fine when I’m online. When I go offline, I get a ton of non-critical errors, like 100 a second, basically httpUtils saying that the host can’t be reached (which is fine, because we’re offline) and then one from ofxJSON talking about parsing a string:

[ error ] ofxHttpUtils: ofxHttpUtils error doPostForm--
[ error ] ofxHttpUtils: Host not found: www.thebos.co
[ error ] ofxJSONElement::parse: Unable to parse string: * Line 1, Column 1
  Syntax error: value, object or array expected.

So now the weird part. If I let it run like that, spitting all those errors out super fast, the app crashes after about three minutes (just sending the errors to the console, not a log file, because the log file quickly gets to be a few GB with all those errors). If I quiet the errors by commenting out those lines, then sometimes it crashes at about 10 minutes, sometimes it can go longer than 30 minutes with no crashes.

Watching the memory under the different scenarios is also interesting, it seems that if I quiet those error messages, then the app starts to slowly leak memory.

Here’s a piece of the crash log:

Thread 22 Crashed:
0   libsystem_platform.dylib      	0x9c46baa4 _platform_strcmp + 84
1   cc.openFrameworks.ofapp       	0x0000838c Json::Value::resolveReference(char const*, bool) + 556 (stl_map.h:427)
2   cc.openFrameworks.ofapp       	0x00008150 Json::Value::operator[](char const*) + 32 (jsoncpp.cpp:2464)
3   cc.openFrameworks.ofapp       	0x00010aba threadedGifUpload::threadedFunction() + 6762 (thread.h:126)
4   cc.openFrameworks.ofapp       	0x003fda92 ofThread::run() + 386 (ofThread.cpp:207)
5   cc.openFrameworks.ofapp       	0x0004084b Poco::ThreadImpl::runnableEntry(void*) + 121
6   libsystem_pthread.dylib       	0x99169e13 _pthread_body + 138
7   libsystem_pthread.dylib       	0x99169d89 _pthread_start + 162
8   libsystem_pthread.dylib       	0x99167e52 thread_start + 34

So my questions are:

Am I using ofxHttpUtils incorrectly?
Why does the act of logging or not logging an error affect the memory usage of the app?
Why is my app crashing and how can I avoid it? (both proper-form and hacky answers welcome)

andrewlandry · May 15, 2015, 5:41pm

It’s hard for me to say what the real problem is that’s making it crash.

My first thought is that httpUtils probably shouldn’t be trying to run 100 times per second and that trying every few seconds is probably a good place to start… or waiting a certain amount of time after receiving the first Host not found error.

As for removing the errors, I think the Host not found error is useful and I would just try less often than 100 times per second after that error is returned once instead of hammering a dead connection.

I would try not to get to the [ error ] ofxJSONElement::parse: Unable to parse string: error, since you could check for the presence of a non-null value before trying to parse it.

There might be other things to consider, but those are the things I would start with.

dennyroberts · July 1, 2015, 9:09pm

Little update, I fixed the crash on that line. The problem was indeed trying to parse it when it was null, so adding this fixed the problem:

if (upload_test["image_formate"] != ofxJSONElement::null) { etc }

However, I’m still having problems related to HttpUtils.

Huge memory leak when running offline. When I turn off wifi and HttpUtils is trying to post, I can just watch the memory climb and climb.
Occasionally I get crashes due to BAD_ACCESS when HttpUtils tries to do this: if(response.status!=-1) forms.pop(); so basically when it tries to clear its queue of forms that haven’t gone through. I also get SIGPIPE crashes… related?

The crashes seem to happen most often right after turning on or off the Wifi.

Anyway I’m stuck for the moment with this app that works great online but is buggy as all hell when the internet goes out. Any ideas?

ryanww · July 4, 2015, 3:07pm

I had similar issues back when I was trying to do something similar with uploading videos. shortly there after I decided to just use curl.

I would save the files to the drive, push new uploads into a vector with all the form data and such. In another thread, I made an uploaded that would grab the data for a file and upload it using ofSystem and forming the correct curl command. Generally I found it to never have any issues when a host was unreachable and I would just parse the reply from the command in real time and handle logic based on that.

The other thing I’d suggest looking into is ofxHTTP. Seems pretty solid although I haven’t used it myself yet.