Machine Learning collection for OF

i’ve started aggregating example applications for machine learning in OF here: https://github.com/ml4a/ml4a-ofx

it focuses on use cases which are interactive or real-time in nature, so not things like deepdream/stylenet. the workflow for most of them is to do the analysis/learning with an offline script, save the results to JSON, and load it into an OF app for interaction/browsing. right now there are 4 apps with corresponding scripts:

  • image t-SNE viewer: arranges a collection of images onto a 2d layout
  • audio t-SNE viewer: same thing but for samples of audio (works best on short samples)
  • convnetOSC: continuously extracts the fc7 encoding of the webcam image in a convolutional neural network (the second-to-last layer activations) and sends all 4096 of them over OSC. this needs some refinement, but basically works, and nicely with wekinator (see transfer learning section here for video).
  • reverse image search: this extracts the encodings from a collection of images, then saves k nearest neighbors to each one, which you can browse. it’s more or less a simplified version of the technique used by golan, kyle, et al in terrapattern which leverages a pre-trained network for the encodings instead of training your own.

in the future, i’d like to add some more categories of examples, like training your own simple network, and some things for gesture recognition (with ofxLearn and/or ofxGrt), porting Kinect2Gesture, and some modules that interface with Wekinator.

i’m working on it alongside other ML stuff that doesn’t use OF, kind of on a topic by topic basis, so it’ll grow slowly. if anyone has ideas, this repo is welcoming for contributions, suggestions, or ideas.

9 Likes