Random segmentation fault on threaded HTTP requests with Raspberry Pi 4

Hello!
The scope of this section of the app is to test if there’s an internet connection available by sending an HTTP_HEAD request to a public DNS. After some time (minutes, hours even days) a SIGSEGV appears and the app crashes. When doing gdb debugging, backtrace always displays the following output:

Thread 1 "project-counter_" received signal SIGSEGV, Segmentation fault.
0xb5e1db48 in ?? () from /usr/lib/arm-linux-gnueabihf/libopencv_core.so.3.2
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x42ff3ffc: 
#0  0xb5e1db48 in ?? () from /usr/lib/arm-linux-gnueabihf/libopencv_core.so.3.2
Cannot access memory at address 0x42ff3ffc

The class inherits ofThread and it’s set to sleep for 20 seconds.

Class header

class ConnectionHandler : public ofThread
{
public:
	ConnectionHandler();

	void setup(ConfigurationManager *config);
	void threadedFunction();
	bool pingServer(string scheme, string ip);
	bool isConnected();

private:
	ConfigurationManager	*config;
	std::map<std::string, std::string> headers;
	ofxHTTP::ClientSessionSettings sessionSettings;
	bool connected;
	long sleepTime;
};

ConnectionHandler.cpp

#include "ConnectionHandler.h"

ConnectionHandler::ConnectionHandler()
{

}
//--------------------------------------------------------------------------
void ConnectionHandler::setup(ConfigurationManager *config) 
{
	startThread(false);
	setThreadName("ConnectionHandlerThread");
	this->config = config;
	sleepTime = config->systemConfig.getCheckConnectionTime() * 1000;
	sessionSettings.setKeepAliveTimeout(Poco::Timespan::SECONDS * 10);
}
//--------------------------------------------------------------------------
bool ConnectionHandler::isConnected() 
{
	//Poco::ScopedLock<ofMutex> lock(mutex);

	bool _connected = false;
	if (isThreadRunning()) lock();
	_connected = connected;
	if (isThreadRunning()) unlock();

	return _connected;
}
//--------------------------------------------------------------------------
void ConnectionHandler::threadedFunction()
{
	while (isThreadRunning())
	{
		ofLogNotice("ConnectionHandler::threadedFunction")
			<< "I will try to have a response from the outside world.";

		// Try ping and check response
		pingServer("https://", config->serverConfig.pbDNSServer);

		ofLogNotice("ConnectionHandler::threadedFunction")
			<< "Ping " << config->serverConfig.pbDNSServer
			<< " result " << std::boolalpha << connected;

		sleep(sleepTime);
	}
}
//--------------------------------------------------------------------------
bool ConnectionHandler::pingServer(string scheme, string ip)
{
	auto startT = chrono::steady_clock::now();
	// Create the URI with the url
	string url = scheme; // Using a secure scheme
	url.append(ip); // Append server address
	Poco::URI uri(url);

	std::string path(uri.getPathAndQuery());
	if (path.empty()) path = "/";

	HTTPRequest request(HTTPRequest::HTTP_HEAD, path, HTTPMessage::HTTP_1_1);
	
	bool success = false;
	try // Do the ping!
	{
		//HTTPResponse response;
		//HTTPSClientSession *httpsSession = new HTTPSClientSession(uri.getHost(),
		//														  uri.getPort());
		//httpsSession->setTimeout(Poco::Timespan(20, 0));
		//httpsSession->sendRequest(request);
		//std::istream *stream = &httpsSession->receiveResponse(response);
		
		HTTPSClientSession session(uri.getHost(), uri.getPort());
		session.setTimeout(Poco::Timespan(10, 0));
		session.sendRequest(request);
		HTTPResponse response;
		std::istream &is = session.receiveResponse(response);

		// For any status
		ofLogNotice("ConnectionHandler::pingServer")
			<< "Response from server: " << response.getStatus()
			<< " " << response.getReason();
		
		success = true;
	}
	catch (const Poco::Exception& exc) {
		ofLogError("pingServer::sendRequest")
			<< "Poco Network Exception: " << exc.displayText() << endl
			<< "\tHost URL = " << uri.getScheme() << uri.getHost() << ":" << uri.getPort();
	}
	catch (const std::exception& exc) {
		ofLogError("pingServer::sendRequest") << exc.what();
	}

	auto endT = chrono::steady_clock::now();
	long t = chrono::duration_cast<chrono::milliseconds>(endT - startT).count();
	ofLogNotice("ConnectionHandler::pingServer")
		<< "Time elapsed " << t << "ms";

	lock();
	connected = success;
	unlock();

	return success;
}
//--------------------------------------------------------------------------

Another approach was to use ofxTaskQueue with Poco::Task and sending notifications to the queue. After hours of testing, the same segmentation fault appeared.

Here is the task code:

void CheckConnectionTask::runTask()
{
	if (isCancelled())
	{
		return;
	}

	// Try ping and check response
	pingServer("https://", config->serverConfig.pbDNSServer);

	postNotification(new Poco::TaskCustomNotification<HTTPResponseInfo>(this, checkInfo));

	ofLogNotice("CheckConnectionTask::runTask")
		<< "Ping " << config->serverConfig.pbDNSServer
		<< " result " << std::boolalpha << checkInfo.success;
}

Tested both codes on a Raspberry Pi 4 Model B (2GB RAM) and Raspberry Pi 3 Model B (1GB RAM) for the first implementation.
Using the latest version of_v0.11.0_linuxarmv6l_release

Any help would be most appreciated. Thanks!

Hi, I think that it is because you are trying to access an https site.
I think I´ve seen this problem because and it was because of https. Checkout ofxHTTP which can handle it properly

Hello Roy, thanks for the tip. I’ve already tried the ofxHTTP::GetRequest and got the same result: segmentation fault thrown after 15 minutes. I’ve also had once the application running 3 days in a row until it finally crashed with the same SIGSEGV.

bummer. maybe you can wrap the section with a try/catch block.

That could come handy, I’m going to wrap it with try/catch as you suggest. Thanks!

All sections related to the thread were blocked with try but none of them catched the error. The segmentation fault appeared just after the first ofLogNotice in the threadedFunction.

Hi
Maybe i’m missing the obvious (i’m a totaly newbe in debuggin), but it seems that the SIGSEGV it’s raised by openCV lib.
Maybe you can create a simple app with just the code to check the connection and test if it still crash

HTH

Hi @angelo , I did not looked thoroughly at you initial post.
But as @Bencilari suggests it might be triggered by opencv. On my experience SIGSEGVs are quite hard to find and often triggered by some “other” code, as it is essentially one piece of code trying to access some memory that it should not be accessing. Try narrowing down by removing parts of your app that do not “bother”, or start the other way round, make test apps with only the minimal parts and interactions and check how these behave.

Thanks for the answers @roymacdonald and @Bencilari. I made a test with the threaded ping method in a standalone project and was stable for hours. As you have said, it may be other parts of the code triggering the SIGSEGV. Right now I’m trying to go step-by-step in order to identify the cause. So I think the segmentation fault wasn’t caused by this particular implementation ofThread + HTTP requests.

1 Like

Hei!
Thanks @roymacdonald and @Bencilari, it appears the segmentation fault was indeed in other part of the app.
So in a nutshell, there’s a script called CameraManager that opens the camera devices (both RGB USB type or Depth Camera, like Orbbec Astra).
Then in the update, one should ALWAYS check the integrity of the cv::Mat frame (passed from the camera):

if (!frame.empty()){//do your magic here}

This way, and for many hours and days of trying to figure it out, it’s finally stable.

2 Likes