Sharing 300mbit/s of images&data between c# apps and OF on local computer

Hey all,

I’m using the C# Kinect app from the Microsoft Kinect Develpment toolkit as I’ve found it to be very stable with 3+ kinects.

Now I want to get the resulting data in OF (depth & color images, skeleton info, etc). Currently for one kinect I use an tcpip connection to localhost, piping raw video data from one to the other at ±100mbit/s, in addition to OSC for sending skeleton info. An advantage of this approach is that we can split OF and kinect code in our setups, taking out the kinect overhead from our apps.

Now, I imagine that for 3 kinects this approach would be unfeasable due to bandwith & processing limitations, especially on the receiving end, (300Mbit/s…), but didn’t try it though.

Any suggestions? Current idea’s:

  • I thought of sending 2 jpgs (RGB, 16bit-DEPTH+8bit-playerinfo) per received kinect frame and decompressing and decomposing them in OF.
  • a syphon type of app would be awesome on windows …
  • is there something like on windows, similar to the pipe in unix? this would remove the TCPIP overhead
  • or can I increase the packet size on localhost connections from 15kb to eg 1mb? This would reduce the TCPIP overhead.

Thanks in advance!

Cheers,
kj

Hi,
I would go the syphon type way. Don’t know if there is some sort of implementation already available for windows, but you want some “inter process communication” right? So you have the concept of “named shared memory” I guess on the C# side it is the concept of “memory mapped files”. As long as you stay on the same windows-machine it should be no problem to write with your C# programm into the shared memory and read it with OF.
Here some great articles with more explanations:
http://msdn.microsoft.com/en-us/library/dd997372.aspx
http://msdn.microsoft.com/de-de/library/ms810613.aspx
http://stackoverflow.com/questions/2262815/posix-shared-memory-vs-mapped-files
Perhaps you have do deal with some data conversion, but this is the fastest way to communicate and share data on the same machine.

cheers

That is exactly what I was looking for, thanks alot!

Hi !

This sounds really neat. Can you provide any advice as to how you were able to stream image data from 1 kinect ? I’m trying to solve this now by splitting the video feed into several fragments over UDP but I’m having some trouble.

Hi Ben,

See here: http://www.hangaar.net/#past-forward-technical-info

I’m planning to clean the code, add color frames (jpeg compression) & release it on github, both the sender in c# and the receiver for OF.

I’ve found that the cpu gain and system stability went up (even with the crappy code below :D) on the OF side by not using the kinect locally. And a windows pc running an almost default example was very stable (and you have features like face tracking, seated mode, etc available).

If youre intrested, this is the code used in the project described (OF receiver):

  
  
void Input::threadedFunction() {  
    networkStatus = "Connecting to " + ofToString(kinectServerIP) + " at port " + ofToString(kinectServerPort);  
	cout << "Connecting to " << kinectServerIP << " @ " << kinectServerPort << endl;  
		while(isThreadRunning()) {  
		//try to connect  
		tcpClientConnected = tcpClient.setup(kinectServerIP, kinectServerPort,true);  
            if (!tcpClientConnected) {  
                networkStatus = "Connection failed to " + ofToString(kinectServerIP) + " at port " + ofToString(kinectServerPort) + ". Retrying... ";  
			//cout << "Connection failed. Waiting to retry..." << endl;  
			ofSleepMillis(5000);  
			continue;  
		}  
		  
		int framecntr = 0;  
		int seconds = 0;  
		//We are tcpClientConnected  
		while (tcpClient.isConnected()) {  
		  
  
			int width = 320;  
			int size = 320*240;  
			char* receivePos = pixelsRcv;  
			int length = 1280;  
			  
			int totalReceivedBytes = 0;  
			int receivedBytes = 0;  
            int time1 = ofGetElapsedTimeMillis();  
            int time2 = 0;  
			while (totalReceivedBytes < size  && tcpClient.isConnected() && receivedBytes != -1) {  
				receivedBytes = tcpClient.receiveRawBytes(receivePos, length); //returns received bytes  
                if (time2 == 0) time2 = ofGetElapsedTimeMillis();  
				totalReceivedBytes += receivedBytes;  
				receivePos += receivedBytes;  
			}  
           // cout << "Wait: " << time2-time1<< ", rcv: " << ofGetElapsedTimeMillis()-time2 << endl;  
			if (!tcpClient.isConnected() || receivedBytes == -1) {  
				cout << "Connection lost!" << endl;  
                networkStatus = "Connection lost";  
				tcpClientConnected = false;  
				continue;  
			}  
			framecntr++;  
			int time = (int)(ofGetElapsedTimeMillis()/1000);  
			if (seconds != time) {  
				seconds = time;  
                networkStatus = "Connected to " + ofToString(kinectServerIP) + " at port " + ofToString(kinectServerPort) +". Receiving frames at " + ofToString(framecntr);  
				//cout << "\t Kinect server FPS: " << framecntr << endl;  
				framecntr= 0;  
			}  
			//lock();   
			{	  
				//cout << "writepixels start" << endl;  
				for (int j = 0; j < 320*240;j++) {						  
					unsigned char valx = (pixelsRcv[j]< 0? 256+pixelsRcv[j] : pixelsRcv[j]);  
  
					if (valx >= (128+64)) {  
						//belong to unknown player (3,4,5,6,7,8)  
						char val = valx - 128 - 64;					  
						tcpClientLabelPixelsReceived[j] = 255;						  
						tcpClientDepthPixelsReceived[j] = val * 4 ;  
					} else if (valx >= (128)) {  
						char val = valx - 128;  
						//belong to player 2  
						  
						tcpClientLabelPixelsReceived[j] = 255;  
						tcpClientDepthPixelsReceived[j] = val * 4 ;  
					} else if (valx >= 64){  
						char val = valx - 64;  
						//belong to player 1  
						tcpClientDepthPixelsReceived[j] = val * 4 ;						  
						tcpClientLabelPixelsReceived[j] = 255;  
						  
					} else {  
						//belong to none  
						tcpClientDepthPixelsReceived[j] = valx * 4 ;						  
						tcpClientLabelPixelsReceived[j] = 0;  
					}  
				}  
				//cout << "writepixels stop" << endl;  
				depthLowRes.setFromPixels(tcpClientDepthPixelsReceived,320,240);  
				labelLowRes.setFromPixels(tcpClientLabelPixelsReceived,320,240);  
				//scale & warp  
				depthHiResCalib.warpIntoMe(depthLowRes, srcCalibLoRes, dstCalib);  
				labelHiResCalib.warpIntoMe(labelLowRes, srcCalibLoRes, dstCalib);  
  
				lock();  
				labelHiResCalibDoubleBuffer = labelHiResCalib;  
				depthHiResCalibDoubleBuffer = depthHiResCalib;  
				unlock();  
  
				  
				threadRcvAndReseizeTime = ofGetElapsedTimeMillis() - time2;  
				newFrame = true;  
			}  
			//unlock();  
		}   
	}  
}  
  
  

And this is the C# code on windows for sending the frames. Its an adaptation of the SkeletonBasics-WPF example.

  
  
  
  
    using System.Threading;  
    using System.Net;  
  
...  
  
        TcpClient client;  
...  
  
      private void WindowLoaded(object sender, RoutedEventArgs e)  
        {  
....      
 if (null != this.sensor)  
            {  
                //open tcp connection  
                client = new TcpClient();  
                client.Connect(IPAddress.Loopback, 1001);  
           .......  
           }  
         }  
       private void SensorDepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)  
        {  
            using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())  
            {  
                if (depthFrame != null)  
                {  
                    // Copy the pixel data from the image to a temporary array  
                    depthFrame.CopyDepthImagePixelDataTo(this.depthPixels);  
  
                    //generate new byte array  
                    byte[] outputPlayerAndDepth = new Byte[320 * 240];   
  
  
                    // Get the min and max reliable depth for the current frame  
                    int minDepth = MinMinDepth;  
                    int maxDepth = MaxMaxDepth;  
  
                    int currentFrame = 0;  
                    for (int y = 0; y < 480; y++)  
                    {  
                        for (int x = 0; x < 640; x++)  
                        {  
                            int i = y * 640 + x;  
                            //goto 320x240  
                            if (x % 2 == 0 && y % 2 == 0)  
                            {  
                                short d = depthPixels[i].Depth;  
                                if (d >= maxDepth - 1) d = (short)(maxDepth - 2);  
                                if (d <= minDepth ) d =0;  
  
                                byte depth = intensityTable[d];  
                                int player = depthPixels[i].PlayerIndex;  
  
                                //get the most significant bits of depth  
                                byte value = (byte)depth;  
                                //push them into 6 bytes  
                                value /= 4; //from 256 values to 64 => /4  
                                if (value >= 64)  
                                    value = 63;  
                                //value = unchecked((byte)((value << 2) & 0xFF)); //of * 4...  
                                //change the 2 player bits  
                                if (player == 0) {  }  
                                else if (player == 1) {  value += 64; }  
                                else if (player == 2) { value += 128; }  
                                else if (player >= 3) { value += 64+128; };  
                                  
                                //add it to our output  
                                outputPlayerAndDepth[currentFrame] = value;  
  
                                currentFrame++;  
                            }  
                        }  
                    }  
  
                    //here we should send the whole bunch  
                    // Get a client stream for reading and writing.  
                    /*  
                    using (NetworkStream stream = client.GetStream())  
                    {  
                        // server is ready   
                        stream.Write(outputPlayerAndDepth, 0, outputPlayerAndDepth.Length);  
                    }  
                    */  
                    for (int i = 0; i < 320 * 240; i += 1280)  
                    {  
                        client.GetStream().Write(outputPlayerAndDepth, i, 1280);  
                    }  
                }  
            }  
        }  
  
  
  

just a quick update about this

I managed to “memory map” a kinect 640x480 stream from c# (MS SDK drivers 1.7) to C++ OF.
The stream includes:

  • the calibrated color (RGB mapped to depth), 3 bytes/pixel
  • depth (11 bit depth + 7 bit user), 2 bytes/pixel
  • the world image (point cloud, real world posiitions from the kinect in xyz 32 bit floating point), 12 bytes/pixel
  • skeleton info (1 byte/pixel)
    Thats ‘streaming’ about 165mb (or 1 gbit) per second!
    The receiving end takes about 1ms to process the stream and throw it into char arrays, 1 additional ms to make images of it.

and this works for up to three kinects simultaneously! (didn’t test more)

So what this means is:

  • you can effectivly decouple the kinect(s) update code from OF, and get it directly from the c# code (more roboust than any kinect SDK plugin, especially for multiple kinects!).
  • The update in OF takes virtually no time, so lots of cpu/gpu free for the cool graphics stuff!
  • starting/stopping an OF app that uses the kinect does not take more time then the empty example, no more waiting for all kinects to have initialised (the streaming c# app just keeps running)
  • virtually no added delay (it feels even a bit more responsive imo when comparing to using ofxOpenNi)
  • all this uses “only” a 165 MB footprint in the memory per kinect.

bottom line: I have 3 kinects running & updating at full speed, using about ± 50% cpu (i7) from the c# app. On the same computer, the OF app is running at full speed with virtually no delay! I’m sure even more optimizations are possible.

Going to clean the code and put it onto github (c# app and OF addon) in a few days.

sloopi, thanks again for the extremely useful pointer!

Kj

This sounds really promising. I’d never heard of ‘memory mapping’ till i read this post. I’m currently working on something similar- hacked away at the Kinect for windows c++ FaceTracking example to send the face and skeleton data via osc (thanks to some help from Ben). But being able to send the rgb and depth image as well would be even better. Looking forward to seeing how this comes along- well done Kj.

thanks. did a commit some time back: https://github.com/Kj1/ofxKinectMemoryMapped
you need to compile the c# project also using MS SDK 1.7 & VS2010.
(relevant code in kinectsensoritem.cs)

Good luck, didn’t test it thoroughly but it works for my purposes, hope it does for yours too!

Lots of improvements still possible, but I lack the time to finish it :frowning: