ofxTensorFlow2 - TensorFlow 2 addon

danomatika · February 10, 2021, 2:25pm

ofxaddons_thumbnail

Howdy all,

We are happy to announce the release of ofxTensorFlow2:

https://github.com/zkmkarlsruhe/ofxTensorFlow2

ofxTensorFlow2 is an openFrameworks addon for loading and running ML models trained with the TensorFlow 2 ML (Machine Learning) library using libtensorflow, the cppflow TF2 wrapper library, and openFrameworks wrappers and conversion functions. Development was influenced in part by Memo Akten’s TF1 ofxMSTensorFlow.

So far, we have tested on macOS and Linux. Windows should work, but we haven’t tried it directly yet. The macOS libtensorflow builds provided by the TF website are cpu-only, so they run slowly but work. Hopefully support for the Apple M1’s Neural Cores will be added at some point…

This addon is developed by the ZKM | Hertz-Lab as part of the project »The Intelligent Museum«.

pandereto · February 11, 2021, 9:55am

Thats super cool!! thanks for sharing

fresla · February 12, 2021, 12:55am

I am really keen to give this a go, but I wanted to check if I am doing something wrong before I go with github.

I am working on OSX with Xcode and the master branch of OF.

As per the instructions:
I cloned the addon
git clone git@hertz-gitlab.zkm.de:Hertz-Lab/Research/intelligent-museum/ofxTensorFlow2.git

Did the submodule update:
git submodule update --init --recursive

Ran the scripts to download the tensorflow C library
./scripts/download_tensorflow.sh

Add c++ 14 support in:
OF_ROOT/libs/openFrameworksCompiled/project/osx/CoreOF.xcconfig

By replacing this line:
CLANG_CXX_LANGUAGE_STANDARD = c++11
With this:
CLANG_CXX_LANGUAGE_STANDARD[arch=x86_64] = c++14

I then used the project generator to generate an example, and added:
$OF_PATH/addons/ofxTensorFlow2/scripts/macos_install_libs.sh "$TARGET_BUILD_DIR/$PRODUCT_NAME.app";

To the second run script under build phases.

I get many errors in ofConstants.h and ofEvents.h
A lot of them are:
Use of undeclared identifier 'self'

I went through the instructions several times but I am not sure if I missed something or if there is an error.

danomatika · February 17, 2021, 4:05pm

I get many errors in ofConstants.h and ofEvents.h
A lot of them are:
Use of undeclared identifier 'self'

Which example?

What files show this (ie. which file/.line is show in the Xcode) and can you paste the full error output?

UPDATE: There isn’t a usage of “self” in any of the source files and the add-on doesn’t have any Obj-C code which would use it.

fresla · February 19, 2021, 8:00am

Hi, so far this happens with “example_style_transfer” and “example_basics” the log output from where the error messages start is as follows:

In file included from ../../../../libs/openFrameworks/3d/of3dUtils.h:3:
/Users/fredrodrigues/My Code/Fred Rodrigues/Code/OF_GITT/openFrameworks/libs/openFrameworks/utils/ofConstants.h:430:50: error: expected ')'
        return std::unique_ptr<T>(new U[sizeof...(Args)]{std::forward<Args>(args)...});
                                                        ^
/Users/fredrodrigues/My Code/Fred Rodrigues/Code/OF_GITT/openFrameworks/libs/openFrameworks/utils/ofConstants.h:430:27: note: to match this '('
        return std::unique_ptr<T>(new U[sizeof...(Args)]{std::forward<Args>(args)...});
                                 ^
In file included from /Users/fredrodrigues/My Code/Fred Rodrigues/Code/OF_GITT/openFrameworks/libs/openFrameworks/3d/of3dUtils.cpp:2:
In file included from /Users/fredrodrigues/My Code/Fred Rodrigues/Code/OF_GITT/openFrameworks/libs/openFrameworks/app/ofAppRunner.h:5:
In file included from ../../../../libs/openFrameworks/app/ofMainLoop.h:4:
In file included from /Users/fredrodrigues/My Code/Fred Rodrigues/Code/OF_GITT/openFrameworks/libs/openFrameworks/events/ofEvents.h:4:
In file included from ../../../../libs/openFrameworks/events/ofEventUtils.h:4:
../../../../libs/openFrameworks/events/ofEvent.h:204:40: error: a space is required between consecutive right angle brackets (use '> >')
                        std::vector<std::shared_ptr<Function>> functions;
                                                            ^~
                                                            > >
../../../../libs/openFrameworks/events/ofEvent.h:220:25: error: function definition does not declare parameters
                std::shared_ptr<Data> self{new Data};
                                      ^
../../../../libs/openFrameworks/events/ofEvent.h:153:4: error: use of undeclared identifier 'self'
                        self->functions = mom.self->functions;
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:161:33: error: use of undeclared identifier 'self'
                        std::unique_lock<Mutex> lck2(self->mtx);
                                                     ^
../../../../libs/openFrameworks/events/ofEvent.h:162:4: error: use of undeclared identifier 'self'
                        self->functions = mom.self->functions;
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:163:4: error: use of undeclared identifier 'self'
                        self->enabled = mom.self->enabled;
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:169:4: error: use of undeclared identifier 'self'
                        self->functions = std::move(mom.self->functions);
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:170:4: error: use of undeclared identifier 'self'
                        self->enabled = std::move(mom.self->enabled);
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:178:33: error: use of undeclared identifier 'self'
                        std::unique_lock<Mutex> lck2(self->mtx);
                                                     ^
../../../../libs/openFrameworks/events/ofEvent.h:179:4: error: use of undeclared identifier 'self'
                        self->functions = mom.self->functions;
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:180:4: error: use of undeclared identifier 'self'
                        self->enabled = mom.self->enabled;
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:185:4: error: use of undeclared identifier 'self'
                        self->enabled = true;
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:189:4: error: use of undeclared identifier 'self'
                        self->enabled = false;
                        ^
../../../../libs/openFrameworks/events/ofEvent.h:193:11: error: use of undeclared identifier 'self'
                        return self->enabled;
                               ^
../../../../libs/openFrameworks/events/ofEvent.h:197:11: error: use of undeclared identifier 'self'
                        return self->functions.size();
                               ^
../../../../libs/openFrameworks/events/ofEvent.h:245:40: error: use of undeclared identifier 'self'
                        return std::make_unique<EventToken>(self,*f.id);
                                                            ^
../../../../libs/openFrameworks/events/ofEvent.h:250:32: error: use of undeclared identifier 'self'
                        std::unique_lock<Mutex> lck(self->mtx);
                                                    ^
../../../../libs/openFrameworks/events/ofEvent.h:251:14: error: use of undeclared identifier 'self'
                        auto it = self->functions.begin();
                                  ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.

All my other OF projects are working fine, and when I revert the change for the c++14 my projects work.

I do have spaces in my folder names that enclose my OF folder - this is something from an enforced structure that I don’t have control over - I am wondering if this is part of the issue.

danomatika · February 19, 2021, 9:04am

Which version of OF are you using? Latest 0.11 stable release or development checked out from git?

UPDATE: I see you already noted this:

I am working on OSX with Xcode and the master branch of OF.

We haven’t tested with bleeding edge OF, only the current release version.

danomatika · February 19, 2021, 9:22am

Looking further, the first error line indicates where the problem likely is:

github.com

openframeworks/openFrameworks/blob/patch-release/libs/openFrameworks/utils/ofConstants.h#L410


#ifndef HAS_TLS
	#if defined(__clang__) && __clang__
		#if __has_feature(cxx_thread_local) && !defined(__MINGW64__) && !defined(__MINGW32__) && !defined(__ANDROID__) && !defined(TARGET_OF_IOS)
			#define HAS_TLS 1
		#endif
    #elif !defined(TARGET_WIN32) || _MSC_VER
		#define HAS_TLS 1
	#endif
#endif


//------------------------------------------------ make_unique
// This is a helper method for make unique on platforms that support C++11, but not C++14.
#if !defined(NO_OF_MAKE_UNIQUE) && (defined(_MSC_VER) && _MSC_VER < 1800) || (!defined(_MSC_VER) && __cplusplus <= 201103L)


// Implementation for C++11 platforms that do not yet have std::make_unique.
// Implementation from http://stackoverflow.com/a/13512344/1518329
namespace std {




template <typename T, typename... Args>
std::unique_ptr<T> make_unique_helper(std::false_type, Args&&... args) {

It seems like the implementation of make_unique support for C++11 is on and clashing with C++14. Try turning it off using the define which is checked for there by adding NO_OF_MAKE_UNIQUE to the build, maybe in libs/openFrameworksCompiled/project/osx/Shared.xcconfig:

OTHER_CFLAGS = $(OF_CORE_CFLAGS) -DNO_OF_MAKE_UNIQUE

danomatika · February 19, 2021, 10:45am

Also, if you are on Windows, the Windows libtensorflow builds from the TF website are currently not up to date, so may run into further issues right now:

fresla · February 20, 2021, 5:08am

I just downloaded the release version and everything works as expected. However the fix you suggested did not work for the master branch.

danomatika · February 22, 2021, 2:53pm

Good to hear it works with the current release.

Feel free to open an issue on Github we can use to track support with the development version of OF, but our general focus is mainly on supporting release versions only as it’s too much work otherwise.

grimus · February 23, 2021, 6:48am

Nice work Dan! Looking forward to trying this out…

prismspecs · April 13, 2021, 1:27pm

Thank you for this! Great to see such good things continue to come out of ZKM.

I’m starting to experiment with this for an upcoming installation. I got the basic example to run, but when I try to insert my own model I get:

what():  No operation named "serving_default_input_1" exists

I’m guessing this is because my model has a different input shape than what is created with:

auto input = cppflow::fill({1, 2, 2, 3}, 1.0f);

My model needs to input scalars and outputs one single scalar. (Just testing a simple addition NN). I’m not quite sure what to do with this line above, is there a way to print out diagnostic info on whichever model is loaded? That way I could see perhaps what it expects as input. Or, if you can help me think through what the fill function does I am a bit lost.

For reference, my model in Keras:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(2,)),
    keras.layers.Dense(10, activation=tf.nn.relu),
    keras.layers.Dense(10, activation=tf.nn.relu),
    keras.layers.Dense(1)
])

In any case, thank you!

danomatika · April 13, 2021, 1:38pm

@prismspecs You can define the input and output names based on your model by passing vectors to setup(). This is used in the example_basics_multi_IO example:

github.com

zkmkarlsruhe/ofxTensorFlow2/blob/main/example_basics_multi_IO/src/ofApp.cpp#L30


	ofSetFrameRate(60);
	ofSetVerticalSync(true);
	ofSetWindowTitle("example_basics_multi_IO");

	// load model, bail out on error
	if(!model.load("model")) {
		std::exit(EXIT_FAILURE);
	}
	model.printOperations();

	// setup: define the input and output names
	std::vector<std::string> inputNames = {
		"serving_default_body",
		"serving_default_tags",
		"serving_default_title"
	};
	std::vector<std::string> outputNames = {
		"StatefulPartitionedCall:0",
		"StatefulPartitionedCall:1"
	};
	model.setup(inputNames, outputNames);

prismspecs · April 13, 2021, 1:48pm

@danomatika thank you. Just to clarify before I try it out, do I also then need to give names to my input and output layers on the Keras side? Or can I just do this for the OF app?

bytosaur · April 13, 2021, 1:57pm

@prismspecs by default the model uses the standard names for input and output tensors. If your names differ you can set the up as explain by @danomatika.

From the multi IO example:
Could you please run the following line to see your models signature?

saved_model_cli show --dir path/to/model/ --tag_set serve --signature_def serving_default

prismspecs · April 13, 2021, 2:03pm

@bytosaur thank you, output is as follows:

The given SavedModel SignatureDef contains the following input(s):
  inputs['flatten_input'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 2)
      name: serving_default_flatten_input:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['dense_2'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict

bytosaur · April 13, 2021, 2:09pm

during initialization try:

std::vector<std::string> inputNames = {
 "serving_default_flatten_input",
 };

std::vector<std::string> outputNames = {
"StatefulPartitionedCall",
};

model.setup(inputNames, outputNames);

prismspecs · April 13, 2021, 3:42pm

This works, thank you so much for your help!

bytosaur · May 6, 2021, 1:23pm

Hello everyone!

I am happy to announce release 1.1.0 of ofxTensorFlow2. Most importantly, the former submodule cppflow2 is now the main branch of cppflow (before that the main branch was using TF1). Please use the instructions from the changelog to update your local repository (or simply reinstall this addon).

From CHANGES.txt:

1.1.0: 2021 May 06

added threaded model background loading via loadAsync() function
added ofxTF2::setMaxGPUMemory()
added ofxTF2::setContext()
fixed ofxTF2::getTensorShape()
updated cppflow as cppflow2 is now the main branch, make sure to update the
local submodule with:
git submodule deinit libs/cppflow/
git submodule update --init --recursive
updated README.md information on training with Python and TF2
updated download scripts

Stay tuned for some new examples!

Happy computing!
bytosaur

prismspecs · May 14, 2021, 11:37am

Thanks for the update. Just curious, are there plans to make using a pkl file possible? I’m thinking of the possibility to use StyleGAN2 models.