Kinect v2 | mouse-control | WinRT

beginners
#1

Hi all!

Currently I´m working on a mouse-control with the Microsoft Kinect, oF and WinRT. I´ve implemented a color-stream include hand-tracking, that works perfect.

Now I want to control the mouse with my hand, and here is my problem. I´m using WinRT, so i can´t use the SetCursorPos()-function.

After searching in the inet, I found out, that I must deactivate the original mousecursor, and implement a relative one, but I have no idea how that works, because I have no experience in WinRT-programming… :frowning:

void ofApp::UpdateBodyFrameColor()
{
	BodyFrame^ bdyFrame = bodyReader->AcquireLatestFrame();
	if (nullptr != bdyFrame)
	{
		bdyFrame->GetAndRefreshBodyData(bdyArray);
		for each (Body^ body in bdyArray)
		{
			if (body->IsTracked)
			{
				WFC::IMapView<JointType, Joint>^ joints = body->Joints;
				for each(auto joint in joints)
				{
					if (joint->Key == JointType::HandLeft)
					{
						CameraSpacePoint position = joint->Value.Position;
						ColorSpacePoint colorSpacePoint = coordMapper->MapCameraPointToColorSpace(position);
						lhx = colorSpacePoint.X;
						lhy = colorSpacePoint.Y;
					}'
					if (body->HandLeftConfidence == TrackingConfidence::High)
					{
						switch (body->HandLeftState)
						{

						case HandState::Open:
							colorLeftHand = ofColor::green;
							// !!!! here should be the mouse control !!!!
							break;
							
						case HandState::Closed:
							colorLeftHand = ofColor::red;
							// !!!!! here should be done a left click !!!!
							break;

						default:
							colorLeftHand = ofColor::white;
						}
					}
				}
			}
		}
	}
}

Thx and BG Epi

#2

So, I´ve solved a part of the problem :slightly_smiling:

Now, I´m able to track the “new” mousecursor on the position of my hands, when my hand is open or closed, and do a mouseclick while it´s closed. (function to success ^^ : ofNotifiyMouseMove() , ofNotifyMousePressed(), ofNotifyMouseReleased()
If both hands are close, you´ll enter a Standby-Mode where all inputs are disabled.

But the problem is the mousecursor is only in the “App-Window” but I´ll need them on the “real” desktop :slight_smile:

Have someone any ideas?

Attached you can find my new code:

`void ofApp::UpdateBodyFrameColor()
{

// Frames aktualisieren
BodyFrame^ bodyFrame = bodyFrameReader->AcquireLatestFrame();

// StandBy-Zeiten errechnen
timeTicksCurrent = GetTickCount64();
timeTicksElapsed = timeTicksCurrent - timeTicksOld;

// Falls ein Frame erkannt wurde
if (nullptr != bodyFrame)
{
    // Überprüfung ob genügend Zeit vergangen ist
    if (timeTicksElapsed > 3000)
        standByModeActive = true;

    // Body-Daten aktualisieren und jeden Body durchlaufen
    bodyFrame->GetAndRefreshBodyData(bodyData);
    for each (Body^ body in bodyData)
    {
        // Wenn der Body erkannt wurde, Skeleton-Tracken
        if (body->IsTracked)
        {
            // Körperteile sortieren und durchlaufen
            WFC::IMapView<JointType, Joint>^ joints = body->Joints;
            for each (auto joint in joints)
            {
                // Linke Hand
                if (joint->Key == JointType::HandLeft)
                {
                    // Koordinaten deklarieren...
                    CameraSpacePoint cameraPos = joint->Value.Position; // 3D
                    ColorSpacePoint colorPos = coordinateMapper->MapCameraPointToColorSpace(cameraPos); // 2D

                    // ... und initialisieren
                    lhx = colorPos.X;
                    lhy = colorPos.Y;
                }

                // Linke Hand...
                if (body->HandLeftConfidence == TrackingConfidence::High && mode == 0)
                {
                    // ... offen -> Maussteuerung
                    if (body->HandLeftState == HandState::Open)
                    {
                        //colorLeftHand = ofColor::green;
                        ofNotifyMouseMoved(lhx, lhy);
                        ofNotifyMouseReleased(lhx, lhy, 0x01);
                    }

                    // ... geschlossen -> linke Maustaste gedrückt
                    else if (body->HandLeftState == HandState::Closed)
                    {
                        colorLeftHand = ofColor::red;
                        ofNotifyMouseMoved(lhx, lhy);
                        ofNotifyMousePressed(lhx, lhy, 0x01);
                    }
                    
                }
                // linke und rechte Hand... 
                if (body->HandLeftConfidence == TrackingConfidence::High && body->HandRightConfidence == TrackingConfidence::High && standByModeActive == true)
                {
                    // ... geschlossen -> StandBy-Modus aktivieren
                    if (body->HandLeftState == HandState::Closed && body->HandRightState == HandState::Closed)
                    {
                        colorLeftHand = ofColor::white;
                        keyPressed(' ');
                        standByModeActive = false;
                        timeTicksOld = timeTicksCurrent;
                    }
                }
                // ... ansonsten normales Bild anzeigen
                else
                {
                    colorLeftHand = ofColor::white;
                    ofNotifyMouseReleased(lhx, lhy, 0x01);
                }
            }
        }
    }
}

}`