Thread related error on ofxLibwebsockets

For the last months I’ve been struggling to fix this error.

I’m using @robotconscience’s [ofxLibwebsockets][1] addon to communicate with a web user interface.

handleSocketJsonBroadcast(eventX &e) receives events from different parts of the app and merge their content into a Json object. An string of this Json will be sent on the next update call (30fps).
I believe the code its based on the [example_server_echo][2].

#include "socketController.h"

void socketController::setup() {
    ofAddListener(eventX::SOCKET_JSON_BROADCAST, this, &socketController::handleSocketJsonBroadcast);
    ofxLibwebsockets::ServerOptions options = ofxLibwebsockets::defaultServerOptions();
    options.port = 9092;
    options.bUseSSL = false; // you'll have to manually accept this self-signed cert if 'true'!
    rootPath = ofToDataPath("ui", true);  
    options.documentRoot = rootPath;
    bSetup = server.setup( options );
    server.addListener(this);
    scope["scope"]="";
    messagesBuffer.resize(0);
    tempMessagesBuffer.resize(0);
}

string socketController::getLocalhostUrl()
{
    string base = "http://localhost:";
    return base.append(ofToString(server.getPort()));
}

void socketController::update()
{
    isLock = true;
        tempMessagesBuffer = messagesBuffer;
        messagesBuffer.clear();
    isLock = false;
    
    if(tempMessagesBuffer.size() <=0)
        return;
    
    for(int i = 0; i < tempMessagesBuffer.size(); i++)
        merge(scope,tempMessagesBuffer[i]);
  
    server.send(scope.toStyledString());
    scope["scope"] = "";
    tempMessagesBuffer.clear();
}

void socketController::onMessage( ofxLibwebsockets::Event& args ){

    if ( !args.json.isNull() )
    {
        static eventX event;
        event.json = args.json;
        ofNotifyEvent(eventX::UI_INPUT, event);
    }
}

void socketController::handleSocketJsonBroadcast(eventX &e)
{
    if(isLock) //the merge() function can be quite intensive, so we make sure  we don't modify messagesBuffer while its executing.
    {
        ofLogError("socketController::handleSocketJsonBroadcast IS LOCKED!");
        return;
    }
    
    messagesBuffer.push_back(e.json);
}

void socketController::merge(Json::Value& target, Json::Value obj)
{
    if (!target.isObject())
    {
        ofLogError("socketController::merge - target is not an object");
        return;
    }
    if(!obj.isObject())
    {
        ofLogError("socketController::merge - obj is not an object");
        return;
    }
    
    for (const auto& key : obj.getMemberNames())
    {
        if (target[key].isNull())
            target[key] = obj[key];
        
        else if (target[key].isObject())
            merge(target[key], obj[key]);
        
        else
            target[key] = obj[key];
    }
}

void socketController::onConnect( ofxLibwebsockets::Event& args )
{
    ofLogNotice("socketController:: Connected");
}

void socketController::onOpen( ofxLibwebsockets::Event& args )
{
    ofLogWarning("New connection from " + args.conn.getClientIP() + ", " + args.conn.getClientName());
}

void socketController::onClose( ofxLibwebsockets::Event& args )
{
    ofLogWarning("Connection closed");
}

void socketController::onIdle( ofxLibwebsockets::Event& args )
{
}

void socketController::onBroadcast( ofxLibwebsockets::Event& args )
{
}

It seems to be thread related issue, but I’m unable to figure out what.
I’m currently on Master branch but I’ve had this error on 0.84 already.
I’m getting it randomly , sometimes after 1 h , sometimes after 2 minutes.

The error varies slightly sometimes. Like this one:

Sometimes I also get this output on the console:

[2014:9835] WARN: zero length libwebsocket_write attempt
[2014:9836] WARN: zero length libwebsocket_write attempt
[2014:9836] WARN: zero length libwebsocket_write attempt
[2014:9836] WARN: zero length libwebsocket_write attempt
[2014:9837] WARN: zero length libwebsocket_write attempt
[2014:9837] WARN: zero length libwebsocket_write attempt
[2014:9837] WARN: zero length libwebsocket_write attempt
[2014:9837] WARN: zero length libwebsocket_write attempt
[2014:9838] WARN: zero length libwebsocket_write attempt
[2014:9838] WARN: zero length libwebsocket_write attempt
[2014:9838] WARN: zero length libwebsocket_write attempt
[2014:9839] WARN: zero length libwebsocket_write attempt
[2014:9839] WARN: zero length libwebsocket_write attempt
[2014:9839] WARN: zero length libwebsocket_write attempt
[2014:9840] WARN: zero length libwebsocket_write attempt
LudovicoDebug(23063,0xb031d000) malloc: *** error for object 0x288d000: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Any help will be appreciated! Thanks ^^
[1]: https://github.com/labatrockwell/ofxLibwebsockets
[2]: https://github.com/labatrockwell/ofxLibwebsockets/blob/master/example_server_echo/src/ofApp.cpp

Hm! I do think it’s a threading issue. Have you tried not broadcasting that event in the websockets “onMessage”?

Try queuing up incoming data in a vector, then ripping through it in update() instead of immediately dispatching the event. It looks like it’s hanging on to some reference to a Message and crashing. You might even want to use an ofMutex and lock when you access this vector in update.

Give that a shot and let me know how that looks. I have heard of people seeing weirdness in nested Poco events (which is what you’re doing when you dispatch from onMessage), but that could be more feeling than fact.

Ah! That makes total sense. Thanks for taking the time to check it out!

I’ve tried using ofMutex as you suggested, but I think I might be missunderstanding something because I still getting some crashes (definetly way less than before), or maybe there is something else. Although the crashes are the same.

To my understanding, in my case, I should create two ofMutex objects because I’m dealing with three threads. The main thread, the ofxlibwebsockets thread and the one that sends the events I’m handling at handleSocketJsonBroadcast(eventX &e) function.

Everytime a shared data have to be accessed I should lock the ofMutex object and unlock it after. If another thread finds a lock function it will wait until the first thread unlocks it.

This is my current implementation (just relevant code):

#include "socketController.h"

void socketController::setup() {
    ofAddListener(eventX::SOCKET_JSON_BROADCAST, this, &socketController::handleSocketJsonBroadcast);
    ofxLibwebsockets::ServerOptions options = ofxLibwebsockets::defaultServerOptions();
    options.port = 9092;
    options.bUseSSL = false; // you'll have to manually accept this self-signed cert if 'true'!
    rootPath = ofToDataPath("ui", true); //If release, we use the copied version in the app bundle 
    options.documentRoot = rootPath;
    bSetup = server.setup( options );
    server.addListener(this);
    scope["scope"]="";
    sendIsLock = false;
    receiveIsLock = false;
    receivedMessages.resize(0);
    toSendMessages.resize(0);
}

void socketController::update()
{
    //IN
    mutexIn.lock();
        for(int i = 0; i < receivedMessages.size(); i++)
        {
            static eventX event;
            event.json = receivedMessages[i];
            ofNotifyEvent(eventX::UI_INPUT, event);
        }
        receivedMessages.clear();
    mutexIn.unlock();

    //OUT
    mutexOut.lock();
        vector <Json::Value> tempMessages= toSendMessages;
        toSendMessages.clear();
    mutexOut.unlock();
    
    if(tempMessages.size() <=0)
        return;
  
    for(int i = 0; i < tempMessages.size(); i++)
        merge(scope,tempMessages[i]);
  
    server.send(scope.toStyledString());
    scope["scope"] = "";
    tempMessages.clear();
}

void socketController::onMessage( ofxLibwebsockets::Event& args ){
    mutexIn.lock();
    if ( !args.json.isNull() )
        receivedMessages.push_back(args.json);
    mutexIn.unlock();
}

void socketController::handleSocketJsonBroadcast(eventX &e)
{
    mutexOut.lock();
        toSendMessages.push_back(e.json);
    mutexOut.unlock();
}

I’ve also tried to lock() and unlock() only on the update() function as you suggested, although I don’t understand what will happend if the thread is locked and onMessage() tries to access the data for example.

In both cases I’m getting this error.

Thanks.

Sorry for the delay! Didn’t get an email you replied for some reason.

That’s a weird one. Looks like you’re sending pretty big strings (totally fine), but otherwise there’s some strange thread stuff going on. If the function got that far, messages.size() is not 0, so it shouldn’t bail there… And that’s a private var so little chance of you updating it.

For now, try to move all sending into your main thread? That could be the issue; I’m not sure how thread-safe calling send() is (and I should work on that / am open to suggestions). Does that make sense?

Hi @robotconscience, I’m not sure I understand what you are suggesting. Isn’t the sending stuff already on the main thread? The stuff on the update() function.

Oof! Yep, you’re correct. Ok, I’ll have to do some digging in the addon to see if I can replicate the bug. Sorry this is persisting so long! About how often do you think you’re sending?

To replicate the error faster I’m sending messages at 60fps (it itakes between few seconds and 5 minutes to crash) but it normally runs at 30fps (it may take few hours to crash)
I’m recieving messages just on user input.
Also, I may have more than one client connected but doesn’t seems to effect it. It crashes anyway with just one.

Thanks!
(I also didn’t get any notification email, maybe it doesn’t send them if the forum is open on the browser?)

Hm! I’m working from the echo_server example, sending ~1.5 mb of JSON every frame at 60 fps. Websocket can handle it until I try to send back from the socket. It immediately closes! I’m assuming your messages go back and forth as well?

Seems like part of a bigger mystery, will keep it it. Have to switch onto some other stuff but I’ll look at this more later! In the mean time I did push some code to the dev branch with one new failsafe error check from libwebsockets. Could be helpful, go ahead and try both versions!

OK! Try the newest code in master. I need to a better comb-through, but essentially sending was not threadsafe. I was letting people immediately write the the socket if under a certain size, which could create a race condition where one is trying to write and read simultaneously… bad news.

Now all writing is handled internally by a single ofxLibwebsockets thread. Give it a shot (master and dev both have it, but master has the release version of libwebsockets so better for now) and let me know how it goes!

It doesn’t go back an forth. If the user doesn’t interact it just sends a message when the connection is stablished. Not sure if normal but I get some disconnections sometimes (then the client tries to reconnect back every 500ms).

I’ve checked out your changes on master. I doesn’t seem it fixed it. I still get the exact same error.

The application is running a ton of other intensive stuff at the same time. I tried with a more powerful machine and it has run for +10h without problems.

Thanks

Hi!
With your latest changes it seems that it loses connection on every second messages sent after connecting. From client to server or from server to client doesn’t matter.

I tried in my app and in the server_echo example.

Output after sending two messages from client to server:

new connection open
on idle
got message This is a message from client
on idle
[712:9387] WARN: zero length libwebsocket_write attempt
on idle
got message This is a second message from client
on idle
[733:7390] WARN: zero length libwebsocket_write attempt
on idle
on close

And in the JavaScript console

WebSocket connection to 'ws://localhost:3456/' failed: Received start of new message but previous message is unfinished.

I tried diferent ports to see it some process on my system was blocking or something but got no sucess.

Thanks.

Hm, I can’t replicate that behavior. So weird! Try switching back to the master branch, it’s a bit more stable at the moment (guessing from that log that you’re in dev).

I am still getting some weird behavior when I send huge data really often, though. Will have to keep poking away at this, sorry it’s causing you so much trouble!

Hey @robotconscience! Thanks a lot for taking the time to look at this.
I’m now under heavy refactoring and during this time I’ve been unable to reproduce the error.
I will guess there are more things other than whatever bug ofxLibwebsocket may have, if any.
I’ll do some stress testing later on and let you know how it goes.
Thanks again! :wink:

Hi @robotconscience I’ve been battling a bit more with it. Nothing changed after refactoring. Losing connection as told before.

I’ve been trying this today for a project and was using the simple echo example. After about two messages, it closes the connection. I do see in the output of the OF app “zero length libwebsocket_write attempt
on idle”. I haven’t modified anything from that example.

1 Like

Hey all, sorry been traveling on a project.

Going to look at this a bit later! Let me know if you have feedback, will get stuff going out as quick as I can.

I looked at it but don’t have much time at all. I did roll back to the previous commit and the problem isn’t there. So it is just something in the most recent commit. It defiantly is odd that after two messages, it will drop the connection.

@ryanww and @xavivives Try pulling the master now. I made a mistake, wasn’t sending the “done” frame over websockets if the message wasn’t bigger than the default message size. Tested with the basic examples and looks good.

Give it a shot, and let me know how it looks! This is much more threadsafe, should fix both of your issues.

It is working for me @robotconscience. Thanks!!

@robotconscience I’ve been testing and running at 30fps and haven’t got a single crash.
When I stress it to +60fps it crashes after few minutes.
So far it has failed in two different places. I’ll keep testing.

This on (notice dataSize its negative)

And this one:

I’m pretty sure in both cases I was just sending data from server to client, with no feedback.

Thanks.