I want to run Libtorch(Pytorch C++ Frontend) in OpenFrameworks

HI, All.

I want to run Libtorch (Pytorch C++ Frontend) in OpenFraneworks.

LibTorch Site: https://pytorch.org/tutorials/advanced/cpp_frontend.html

If LibTorch is run in OpenFrameworks, It will be very powerful Deep Solution IDE(deep learning + GUI Production for OSX, Win10, & Linux).

The setup() + update() structure of OF is very suitable for trained model’s fast inference.

Like this:
setup() {
// trained deep model loading
// etc init
}
update() {
// fast inference repeatedly of inputed.
}

But I can’t include <torch/torch.h> in XCode.

At CMake method, CMakeLists.txt is like this:

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(dcgan)

find_package(Torch REQUIRED)

add_executable(dcgan dcgan.cpp)
target_link_libraries(dcgan “${TORCH_LIBRARIES}”)
set_property(TARGET dcgan PROPERTY CXX_STANDARD 11)

and install libtorch for mac: https://download.pytorch.org/libtorch/cpu/libtorch-macos-latest.zip

How can I include libtorch header in XCode ??

Thanks in advance.

Simple Libtorch Code:

#include <torch/torch.h>
#include <iostream.h>

int main() {
torch::Tensor tensor = torch::eye(3);
std::cout << tensor << std::endl;
}

Hi @bemoregt, I just spent hours myself to get this simple example working.
My libtorch folder is next to the src folder of the OF project.
In the build settings of the targets (not the project) I had to add 2 paths to the header search path:

  • libtorch/include
  • libtorch/include/torch/csrc/api/include

I had to add libraries to link:

  • libtorch/lib/libtorch.dylib
  • libtorch/lib/libc10.dylib

After that the project builds, but will crash on execution because it can’t find the libraries.
I had to go to the build phases tab and copy the 2 libraries above and libiomp5.dylib during the build process. The destination is set to products directory.

I saw that there are more dylib files in the libtorch/lib folder and maybe the OF app will crash when you actually make something more complex than printing an identity matrix, but I think by then you know how to fix it.

I hope that helped. Let me know if you manage to make it run :slight_smile:

Cheers,
Felix

[Update:]
I just noticed that this only works when starting the app from within Xcode.
I’m trying to fix it so it can be started from the app bundle, but libiomp5 gives me a hard time with weird paths when executing…
I’ll post an update when I get it working.

1 Like

AFAIK, linked dynamic libraries (.dylib) will only work when executing through Xcode. For standalone apps, it needs static (.a) libraries.

1 Like

That’s good to know :sweat_smile:
Then I guess I have to build libtorch from source,
because the release only contains a dynamic library.
I’ll update after the build finishes.

1 Like

what error messages were you getting, something rpath related?
I usually add a build phases in xcode, similar to the following code:

cp $OF_PATH/addons/ofxNDI/libs/libndi/lib/osx/libndi.dylib $TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/

cd $TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/

install_name_tool -change @rpath/libndi.3.dylib @executable_path/libndi.3.dylib $PRODUCT_NAME
1 Like

Yes, I have something like that.

# Copy libtorch, libc10 and libiomp5
rsync -aved "./libtorch.dylib" "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/Frameworks/";
rsync -aved "./libc10.dylib" "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/Frameworks/";
rsync -aved "./libiomp5.dylib" "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/Frameworks/";
# change path for execution
install_name_tool -change @rpath/libtorch.dylib @executable_path/../Frameworks/libtorch.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME";
install_name_tool -change @rpath/libc10.dylib @executable_path/../Frameworks/libc10.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME";
install_name_tool -change @rpath/libiomp5.dylib @executable_path/../Frameworks/libiomp5.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME";

But I still get an error for libiomp5:
dyld: Library not loaded: @rpath/libiomp5.dylib

1 Like

maybe the otool helps to see whats going on.

2 Likes

The output of otool shows the expected path after running the build phase script from Xcode, but the error remains the same.

    @executable_path/../Frameworks/libfmodex.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1671.60.107)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 50.1.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 492.0.0)
	/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1575.17.0)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 946.0.0)
	/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
	/System/Library/Frameworks/QTKit.framework/Versions/A/QTKit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.255.3)
	/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
	@executable_path/../Frameworks/libtorch.dylib (compatibility version 0.0.0, current version 0.0.0)
	@executable_path/../Frameworks/libc10.dylib (compatibility version 0.0.0, current version 0.0.0)
	@executable_path/../Frameworks/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1265.9.0)
	/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1575.17.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
1 Like

Hi, @felixd.

What’s this?

Is this for MacOSX also?

And… this

I wonder.
Thanks.
@bemoregt.

Hi, @bemoregt.
Those two are new to me.
I’ll try to test both of them later (today or tomorrow) and let you know if I managed to make them run on MacOS (10.14/Mojave).

Best,
Felix

1 Like

Thank you, @felixd.
I’ll also try that…

“The highland is right there.”

Best,
@bemoregt.

Hi,

just a quick update.
I managed to compile and run the dcgan example from:

I opened the included Xcode project inside the Dugan folder and changed the search paths for frameworks, header and lib

Header Search Path = …/libtorch/include/**
Library Search Path = …/libtorch/lib

Best,
@felixd

1 Like

Me again with another update.

I am trying to compile the image classification example from:

One dependency that is not mentioned in the readme (not even as optional for the examples) is OpenCV !


Edit 2: I followed the instructions here to build OpenCV from source.
Problems I had were with protobuf because I had other versions installed in homebrew and anaconda.
Also the tesseract baseapi header couldn’t be found, so I exported the path to the installation from my homebrew directory.

export CPATH="/usr/local/Cellar/tesseract/4.1.0/include"

After compiling and installing OpenCV I added the following to the Xcode header search path:

/usr/local/include/opencv4
I used anaconda to quickly install the library.
conda install -c menpo opencv3

Then I downloaded libTorch for macOS from

Quicklink:
https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.3.1.zip

and copied the unzipped folder as is into addons/ofxLibTorch/libs.

After an attempt to compile in Xcode I get errors for 3 lines in BaseModel.cpp.


Edit 1: I just found out that make_unique is part of the C++14 standard and my Xcode is still set to C++11. So instead of changing the code (which is wrong) I changed the used language dialect in the project settings.

Changing
    mDevice = std::make_unique<torch::Device>(torch::kCUDA);
else
    mDevice = std::make_unique<torch::Device>(torch::kCPU);
to
    mDevice = torch::make_unique<torch::Device>(torch::kCUDA);
else
    mDevice = torch::make_unique<torch::Device>(torch::kCPU);
makes 2 of them go away. The last error is:

No viable overloaded “=” BaseModel.cpp

Now I am stuck at:

No matching member function for call to “forward”. Candidate template ignored couldn’t infer template argument ‘T’

Maybe someone else knows how to fix that?

Best,
@felixd

1 Like

Hi, @felixd

In my OpenCV case(OSX Mojave),

pkg-config --cflags --libs /usr/local/Cellar/opencv/4.1.2/lib/pkgconfig/opencv4.pc
-I/usr/local/Cellar/opencv/4.1.2/include/opencv4/opencv -I/usr/local/Cellar/opencv/4.1.2/include/opencv4 -L/usr/local/Cellar/opencv/4.1.2/lib -lopencv_gapi -lopencv_stitching -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_dnn_objdetect -lopencv_dnn_superres -lopencv_dpm -lopencv_highgui -lopencv_face -lopencv_freetype -lopencv_fuzzy -lopencv_hfs -lopencv_img_hash -lopencv_line_descriptor -lopencv_quality -lopencv_reg -lopencv_rgbd -lopencv_saliency -lopencv_sfm -lopencv_stereo -lopencv_structured_light -lopencv_phase_unwrapping -lopencv_superres -lopencv_optflow -lopencv_surface_matching -lopencv_tracking -lopencv_datasets -lopencv_text -lopencv_dnn -lopencv_plot -lopencv_videostab -lopencv_video -lopencv_videoio -lopencv_xfeatures2d -lopencv_shape -lopencv_ml -lopencv_ximgproc -lopencv_xobjdetect -lopencv_objdetect -lopencv_calib3d -lopencv_imgcodecs -lopencv_features2d -lopencv_flann -lopencv_xphoto -lopencv_photo -lopencv_imgproc -lopencv_core

And add header & lib search path like this,

I hope it helpful.

Thanks.
@bemoregt.

1 Like

:tada:

I managed :laughing:
I have a running ofxApp (v.0.10.1) that trains and tests on the MNIST data using the ofxLibTorch add-on.
It is still a bit of a hack and it blocks the window until it finishes, but it runs. I even can save and load the model (although not with the built in mechanisms. I wonder why :thinking:)

To make it work, I first compiled openCV from source following the tutorial from Adrian Rosenbrock.
I changed the working directory to be inside the ofxLibTorch/libs folder.

As described in my previous post, I had some trouble with Google’s protobuf, maybe because I skipped the part where Adrian uses a virtual environment for building. Also the tesseract thing mentioned before was a bit weird, but exporting the path is a quick fix.

Next, I downloaded libTorch for macOS and put it in ofxLibTorch/libs.


Edit: It turns out libTorch v1.3.1 is not working. In the end I used v1.2.0 which can be downloaded here:
https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.2.0.zip


After that I created a new ofxProject with the projectGenerator and selected the add-ons ofxLibTorch and ofxImGui.

At this point the build will fail with a missing torch/torch.h

To overcome this problem we have to add the following paths to the header search path in BuildSettings:

../../../addons/ofxLibTorch/libs/libtorch/include
../../../addons/ofxLibTorch/libs/libtorch/include/torch/csrc/api/include

Now when compiling we are missing opencv2/opencv.hpp.
So we add the path as well. If you followed Adrian’s tutorial and installed the libraries, the path should be

/usr/local/include/opencv4

Now we (or at least I was) are greeted with the error “No template named make_unique in namespace std”.
This is when Xcode is using the C++11 compiler settings.
In the build settings at Apple Clang - Language - C++ we can set C++14 (see picture in previous post).

Then I end up with 2 errors in the ofxLibTorch add-on.
One is a missing conversion before printing in the forward step of the VGG16 model.
I submitted a pull request to fix this.

The other one is in the loadModel function of BaseModel.cpp. The from file created object can not be assigned to the shared pointer. I opened a new issue and for now simply ignore the problem by commenting line 18 in BaseModel.cpp.

Now the project will compile, but the linker can’t find the required libraries.
So we add the following to the library search path:
../../../addons/ofxLibTorch/libs/libtorch/lib

and in BuildPhases -> Link Binary With Libraries
we add from others and then all files in
../../../addons/ofxLibTorch/libs/libtorch/lib

except the libtorch_python.dylib.

Now there is only references to the OpenCV libraries missing.
And I think here it get’s a bit ugly :disappointed_relieved:

Like with libTorch, we add the following to the library search path:
../../../addons/ofxLibTorch/libs/opencv/build/lib

and in BuildPhases -> Link Binary With Libraries
we add from others and then all files in ../../../addons/ofxLibTorch/libs/opencv/build/lib

Now the project compiles successfully.
But we need to do two more things before we can run it :expressionless:

First, we will add all the libraries we included in the linking in the copy section of BuildPhase.
All the libraries from the opencv path and everything except the libtorch_python.dylib from the lib torch path. This time it is easier, because they will show up in the dialog under Frameworks.

And finally, In BuildPhases -> Run Script (if there are multiple, then the one after the linking step)

there is this line like this somewhere:
install_name_tool -change @executable_path/libfmodex.dylib @executable_path/../Frameworks/libfmodex.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME";

we have to copy and modify it for some of the linked libraries

install_name_tool -change @rpath/libiomp5.dylib @executable_path/../Frameworks/libiomp5.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME"; install_name_tool -change @rpath/libc10.dylib @executable_path/../Frameworks/libc10.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME"; install_name_tool -change @rpath/libcaffe2_observers.dylib @executable_path/../Frameworks/libcaffe2_observers.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME"; install_name_tool -change @rpath/libcaffe2_detectron_ops.dylib @executable_path/../Frameworks/libcaffe2_detectron_ops.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME"; install_name_tool -change @rpath/libshm.dylib @executable_path/../Frameworks/libshm.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME"; install_name_tool -change @rpath/libtorch.dylib @executable_path/../Frameworks/libtorch.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME"; install_name_tool -change @rpath/libmklml.dylib @executable_path/../Frameworks/libmklml.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME"; install_name_tool -change @rpath/libcaffe2_module_test_dynamic.dylib @executable_path/../Frameworks/libcaffe2_module_test_dynamic.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.app/Contents/MacOS/$PRODUCT_NAME";

Now it will finally run :sweat_smile:

In a bit I will upload a small test project here.
@bemoregt, let me know if it works on your machine too.

1 Like

Hi, @felixd @ayruos @thomasgeissl

I met this error when run xcode(ofxLibtorch+image classification demo).

What’s wrong to me ?

Thanks
@bemoregt.

Hey @bemoregt,

It looks like you included the CarbonCore Framework.
Do you need that to run the example? Can you try removing it?

1 Like

Hi, @felixd

How can I remove it?

@bemoregt.

Ah, I made a very uneducated guess :grimacing:
It seems this framework is part of what is needed on macOS for the basic functionality of open frameworks. I don’t really know what is causing the error and I also can’t reproduce it.
I tried compiling the ImageClassification example myself and get different errors.
Starting from Xcode’s inability to infer the type of the forward function to the fact that the vgg16 model it is trying to load is not part of the example and has to be downloaded from somewhere :slightly_frowning_face:

Best,
@felixd

1 Like