ofxTwitter archive and live search

I’m working on this project which I am trying to get both live tweets and “archived/old” tweets using the ofxTwitter API. I can search for live tweets easily using client.search(string) and I can also search old tweets using:

ofxTwitter::SearchQuery query(string)
query.setCount(50);
client.search(query)

My problem is that after doing a search query for older tweets I can’t search for live feed again, it gets stuck on the client.search(string) part.
I’ll now show some of the code that would help solve the problem.

This is my searchTweet function.

void ofApp::searchTweet(bool archive, string text)
{
	cout << "Searching..." << endl;
	if (archive)
	{
		ofxTwitter::SearchQuery query(text);
		query.setCount(50);
		client.search(query);
	}
	else
	{
		cout << "done";
		client.search(text);
		cout << " dOnE ";
	}
	cout << "DONE" << endl;
}

This is where I call for my live tweets

if (ofGetElapsedTimeMillis() >= updateTweetWordSearch)
		{
			updateTweetWordSearch += 6000;
			cout << "get tweets live" << endl;
			searchTweet(false, myTweetWords[updateTweetWordSearchCount]);
			if (updateTweetWordSearchCount < myTweetWords.size() - 1)
			{
				updateTweetWordSearchCount++;
			}
			else
			{
				updateTweetWordSearchCount = 0;
			}
		}

This is where I call for my “archive/old” tweets

if (searchBar.inside(xMouseClick, yMouseClick))
	{
		changeTextBoxColour = true;
		userCanType = true;
	}
	else if (searchButton.inside(xMouseClick, yMouseClick))
	{
		changeTextBoxColour = false;
		userCanType = false;
		if (userInput != "")
		{
			count = 0;
			cout << "get tweets archive" << endl;
			searchTweet(true, userInput);
			myTweets.clear();
			userInput = "";
		}
	}

Any help would be appreciated. @bakercp

Hey @RICARDO_GRACA, currently the ofxTwitter::SearchClient is set up as a polling client, which means that internally it has a polling thread that searches for the given ofxTwitter::SearchQuery every X number of seconds. It keeps track of what is a “new” tweet by monitoring the reported since_id returned with the Twitter query response. Then it delivers only new messages. When you call .search(...) on a given client, it stops the polling thread and restarts the polling thread with the new ofxTwitter::SearchQuery.

Thus, stopping a thread by calling client.search(...) can take a little while to timeout so it will block (stall your program), so it’s not recommended to do it often.

Currently there is no algorithmic way in ofxTwitter to sequentially page through archived Tweets (page size is limited), though this is certainly possible (I just haven’t needed it for my projects).

You have control over all aspects of the REST search by configuring ofxTwitter::SearchQuery for archived searches, so you might try that.

Take a look at the built-in properties here:

Additional properties can be added manually as the ofxTwitter::SearchQuery search query extends Poco::Net::NameValueCollection (link) to hold its query parameters.

You can reference the Standard Twitter Search API docs to see if there are any useful ones missing in my implementation: https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets

To start, I’d recommend doing something like:

ofxTwitter::SearchQuery query(string);
query.setUntil(2019, 1, 28); // Get only tweets posted until two days ago.
client.search(query);

… though this will only get the last X number before the date (which is limited to 100 by the API and the count parameter).

Note that according to the documentation, the standard (non-enterprise) REST search API only archives the last 7 days of tweets, so any setUntil(...) parameters must be within the last 7 days.

A much better solution would be to modify the REST client to page through the last 7 days of tweets. Basically, the logic here would need to be changed to not just look for the newest tweets, but attempt to gather previous tweets by dynamically modifying the since_id, max_id, count, etc until enough archived tweets were collected. Here is the current latest tweets only logic:

1 Like

Great information! Thanks :wink:

1 Like