Chantal Paz - 03695811

LA 124 - OL1: Physics for Artist

Instructor - Samuel Tsitrin

21 July 2014

        An Experimentation on How to Convert the Xbox Kinect Away From Its Commercial Use and Into a Motion Capture Camera and Aspects of the Kinect’s Specifications and Its Relation to Physics

Introduction of the Kinect:

A first impression of the Kinect will leave one questioning, “What is it?”

For those unfamiliar with the Kinect, it is a piece of hardware developed by the Microsoft company and was first introduced to the public back in the 2009 E3 convention. The Kinect was to be paired with the Xbox 360 gaming consoles or sold separately for those who already owned the console. Currently, there is a newer model of the Kinect for Xbox One, but that is a different story.

The purpose of the Kinect was to evolve gaming from a sitting leasure to a more interactive experience. This meant that gaming was now to be a full body exercise, which can get gamers more involved with whatever game they choose to play. When the Kinect is plugged into the console, the gamer could get the camera’s attention by waving one’s hand back and forth, or one could choose to go with voice control. This could also be used as an alternative to the controllers.

Sadly, the expectation of the Kinect was left to a low level. Currently, it seems as if the only games compatible for the Kinect include a small variety of sport, dance, and exercise games. This does not cause too much excitement from gamers, leaving the Kinect almost forgotten.

Specifications:

The Kinect is less of a camera and more of a trio of major subsystems sensors which include the optical subsystem being the camera and depth sensors, the audio subsystem being the microphone, and the motor subsystem being the tilt motor.

The Optical Subsystem

        Starting with the middle camera from the image below, the first to be introduced is the color sensor. The color sensor is actually an RGB camera that can sense which colors are presented and display very accurate images with a resolution of 1280 x 960.

Next we have the IR Emitter and IR Depth Sensor. The IR that is in the name stands for infrared. What the IR Emitter does is it projects an infrared laser or light beam and in return is read by the IR Depth Sensor. How this works is that the light beam reflects itself back to the Depth Sensor and the Depth Sensor will be reading distance as a pattern. For instance if one holds his or her hand in front of his or her face, the pattern on the hand and on the face will be different because of the depth between the hand and the sensor and the face and the sensor are different. This gives the sensor the information it needs to understand what the environment is like.

The Audio Subsystem

If you notice closely on the image below, you will notice that the Kinect doesn’t just have one microphone but four. This might give you a clue that these microphones are not just for audio recordings. The reason for the multiple microphones is to get the best sense of the location. These microphones can find the location of the sound but even better the source of the sound wave. This is excellent for the voice control feature to be used while there is background noise present. Another thing that is not visible is the audio processor, just to add to the audio subsystem.

The Motor Subsystem

Also to touch upon the Tilt Motor, the Tilt Motor is to be thought of as a joint or the neck of the Kinect. This joint is not to be moved manually, that could lead to malfunctions. The Tilt Motor will actually move automatically. Reasons for the adjustment include the setup process to give the best sense of the location of the room. Also, the Kinect will adjust itself depending on the height of the players. If a taller player wants to join the game, the Kinect will correct itself to ensure the taller player has enough room to play.

image from msdn.microsoft.com (2014)

Specifications for the Kinect

chart from msdn.microsoft.com (2014)

Accelerometer:

The Accelerometer deserves to have its own section. An accelerometer measures acceleration forces. A reminder that acceleration is the change of velocity rate:

Acceleration = change of velocity / time

The accelerometer is made up of  three axis, of X, Y and Z, just as the accelerometer in a smartphone whereas cars only have two. What the accelerometer is used for is to detect if its orientation is balanced and it can also be used to make visual maps of the floor plane for a more accurate reading of the environment.

The way that the accelerometer reads the data is by converting the information to a 3-D vector, which is returned as Vector4 with its components being X, Y, Z, and W. The W component will always be set to (0.0).  A single vector’s acceleration reading will be the direction and magnitude.

Now the units in the vector are actually in gravity units. So the g will be unit and since we are on Earth, the g will equal 9.8m/s2

Kinect Accelerometer coordinate system

 from msdn. microsoft.com (2014)

Example: Let’s use free fall to explain. Say we are standing in front of the Kinect sensor with a ball held and dropped from rest. Knowing that gravity on Earth is 9.8m/s2 , if the ball fell for 0.5 seconds, how fast is the ball going?

For this free fall example we will use v = gt

The velocity will be the gravitational acceleration, which is 9.8m/s2

The time will be 0.5

The ball will be falling at 4.9 m/s

The Kinect Sensor can read the movements in real time, but perhaps there is a small unnoticeable lag which is left unmentioned, the sensor will also read the ball falling 4.9 m/s

For an acceleration example: A player who is originally standing  takes off  at a speed of  4.2 m/s in 2 seconds. What is the acceleration of the player?

First we use this equation:  a =  vf – vi   =   D v

                                          t              t

a = 4.2m/s – 0m/s = 4.2m/s = 2.1 m/s2

                                                  2s              2s

So the acceleration read by the accelerometer will be reading that same thing since it is capturing everything in real time.

Infrared of Kinect:

As mentioned before, the way the Kinect can read movement is with its IR emitter and IR sensor. But what really is going on can only be seen with a device that can view the infrared light. Humans can not see the infrared because we can only see a range from 400 nm of light wavelengths to 750 nm of light wavelengths or in terms of light  frequency 430 THz to 750 THz. Infrared on the other hand has a frequency that is “below red”. Infrared has a range of 1 mm of light wavelengths to 750 nm of light wavelengths. Infrared is out of range of human vision so the one way to see infrared is with the assistance of a camera. If we use a camera that can see infrared light, then we can see what is actually going on with the Kinect. With that, one will see that the IR emitter is actually projecting thousands of small dots that is being scattered all across the room. The IR depth sensor will then read the movement of the dots that are being emitted and therefore get the information it needs to read the environment and the player.

chart edited on Google Docs by myself but originally from  CRC Handbook of Chemistry and Physics (92nd ed. ed.)

image provided by http://www.photobiology.info/Visser-Rolinski_files/Fig2.png 

Distance of Kinect:

Distance on the Kinect can be a big issue if one does not have enough space to move around in. Quite a lot of space is needed, about a 6ft x 8ft of empty space. It is likely the sensor can not read the front corners of the rectangular space; I walked to the corner and it lost sight of me.

Photo taken by me from Xbox Kinect settings (2014)

The sensor reads the floor is a graph with each square about 2 ft x 2ft each, but that is just an estimate. The difference between the player and the sensor is crucial for accurate readings. If one positions the Kinect manually or just does not have enough room, one will find that their legs are not being read, which can be an issue depending on the game one is playing. Such as in the image below, half my legs are visible but that is not good enough for the sensor to read this. To solve this issue, all one has to do back up just enough and / or manually lower the tilt of the Kinect. I purposely tilted the Kinect up so that I could get this image.

Image taken by me from Xbox Kinect Settings (2014)

Just to show how Kinect reads objects that are right in front of the sensor is really interesting. If one is too close to the sensor, the sensor will start to show doubles. Just see for yourself.

As you can see in the image below, some features of the arm is not being correctly read as it seems that there is a gap near my armpit area.

Image taken by me from Xbox Kinect settings (2014)

Setting up the Kinect for motion capture:

Here where the steps I tried to follow -

1) First you have to have the Xbox Kinect with you.

Picture taken by me of the Kinect (2014)

2) You will want to buy an adaptor since the Kinect won’t be able to connect straight to your laptop and it will need a power supply. This will cost around $6 or more.

Picture taken by me of the adaptor that I used (2014)

3) Here is where I was having issues with  finding the driver, Sensor Kinect,  and the middlewares, two pieces of OpenNi.

The first to have downloaded was a middleware, OpenNi from Primesense. Primesense is the company where Microsoft was able to obtain and license the technology for the Kinect. I looked everywhere for a link that would not leave me to a dead end. It seemed that none of the links were working. My questions were not answered until I stumbled upon this article that was written in February 2014. The article from createdigitalmotion.com states that the OpenNi websites closed back in April 2014 mainly because it was bought by Apple. With this closed, I do not know what else to do.

4) Assuming it worked, then at this point you can state creating what you wanted like animations, play games, or create applications. There is intensive coding that can be done with Kinect to bring out other cool features that it is capable of.

How Does Motion Capture Compare to other forms of animations?:

According to the Article “Motion Capture” from MIT Communications Forum, Maureen Furniss wrote, “Right from the beginning, one finds that talking about motion capture can be difficult. Deciding on which term to employ is one of the challenges, since so many exist:

• the term itself, 'motion capture', typically is abbreviated as 'mocap'
• another popular term is 'performance animation,' taking the emphasis off the capturing of data, which is seen as just one part of the entire performance animation process
• performance capture, a variation on both of the above terms
• also heard are 'virtual theater', 'digital puppetry', [1] and 'real-time animation' [2]
• additionally, the devil's rotoscope' seems to be a favorite term among traditional animation people [3]”

I haven’t had any experience with motion capture but I’ve seen a bit of motion capture done in movies, animations, and even video games. When I think of motion capture, I imagine it to be like a sci- fi movie where one can just jump on a platform and control a large robot. Motion capture is similar, well almost. In the traditional form of motion capture, the actor usually has to wear a spandex suit plotted dots all along the suit and multiple dots are also stuck onto the actors face as well.

I’d imagine motion capture to be the quicker way to get the action one is trying to create. I am not sure if motion capture is considered the “faster and cheaper” alternative to animating since not too many movies are done this way, perhaps because the cost of the software is too expensive or too limited. The use of motion capture, though, is limited when it comes to animations such. Only the laws of physics can be recorded.  with motion capture, some principles of animation like exaggeration will have to be animated later using a different technique. Motion capture has its pros and cons but it depends on what one is looking for or trying to achieve, then motion capture be the right way to go.

Still from Naughty Dog’s “The Last of Us” provided by killstreakmedia.com

How Others Are Using Kinect?:

Many have noticed the potential the Kinect has available and what other uses it could be helpful with. Before this project, I had not put too much thought into the Kinect as it was just sitting on the shelf gathering dust. After doing further research on this piece of equipment, there is a handful of tasks that could help design the future in areas outside of video games. Animation is one of the many uses, and one I originally tried to achieve myself. Another great use that is already being used is actually in my hospital and perhaps other Kaiser Permanente's, the use of the hands free control on the directories and other digital interactive bulletin boards. With this, patients can find out more information from the  bulletins without the risk of spreading illness. There are digital interactive bulletins in the waiting room where kids are encouraged to play games while using their whole body. Kids will have fun and will forget they are actually doing exercises.

Doctors are using the sensor in the operating rooms to get a 3D model of the patient’s X- Ray and now have the ability to rotate the X-Ray around without having to touch a physical remote, making the rotation hands free.

Kinect is also being used in retail, specifically a large mirror like monitor that will allow one to try on clothes virtually. This can be really helpful for the store employees so that the clothes don’t get ruined from wear or having to restock the clothes that was left in the fitting room. Most importantly, the customer will be able to try on as much clothes and experiment with colors, styles, and sizes as mush as they like resulting in a happier customer which is always good for business.

There are likely many more uses that we have not thought up yet. The possibilities on how this device can change our life is exciting. It is funny how this sensor was intended for video games yet was left out and forgotten in the gaming industry. Now it has found a better purpose that will be so much more helpful than games. The future is full of surprises and the Kinect is definitely part of it.  

Conclusion:

After many, many, many trials I was unsuccessful in making the sensor into the motion capture camera I had planned. Though I failed, that does not mean that I do not know how to make this project succeed. To proceed, though, I would need more time and more funding. I did not spend much to find my discoveries in this project, just $6 for an adaptor. What I would need to make this project work is the other Kinect sensor specifically for Windows. This alone would be $199 which is not too bad of a price, but just not something I would pay for at the moment. Another thing would be the iPi MoCap software that would range from $295 to $1,495. I currently do not have the money to be experimenting with so this is out of reach for a while. I am interested in continuing this project because I would like to see it work, this would be great for my animations. Since the Kinect I was using will no longer be of value for motion capture to me, perhaps I find another use for it by continuing the experiment.

Bibliography:  

"Calculating Acceleration." Calculating Acceleration. N.p., n.d. Web. 01 Aug. 2014. <http://www2.franciscan.edu/academic/MathSci/MathScienceIntegation/MathScienceIntegation-836.htm>.

Castle, Alex. "How to Hack Your Kinect." Maximum PC. N.p., 05 Mar. 2011. Web. 01 Aug. 2014. <http://www.maximumpc.com/article/how-tos/how_hack_your_kinect>

"Channel 9: Videos about the People Building Microsoft Products and Services."Main. N.p., n.d. Web. 01 Aug. 2014. http://channel9.msdn.com/ 

Electromagnetic Spectrum with with Visible Light Highlighted. Digital image. Wikipedia, The Free Encyclopedia., n.d. Web. 01 Aug. 2014. <http://www.photobiology.info/Visser-Rolinski_files/Fig2.png>.

Furniss, Maureen. "Motion Capture." Motion Capture. MIT Communications Forum, n.d. Web. 01 Aug. 2014. http://web.mit.edu/comm-forum/papers/furniss.html 

1. Brad deGraf and Emre Yilmaz, "Puppetology: Science or Cult?" Animation World Magazine3:11 (Feb 1999).

2. Leslie Bishko, letter to the author, 11 June 1999; Richard Cray, letter to the author, 17 June 1999.

3. Shane Cooper, interview with Andrea Cooper at his Protozoa office, November 1998.

Haynes, William M., ed. (2011). CRC Handbook of Chemistry and Physics (92nd ed. ed.). CRC Press. p. 10.233. ISBN 1-4398-5511-0.

"How-to: Successfully Install Kinect on Windows (OpenNI and NITE)." - CodeProject. N.p., n.d. Web. 01 Aug. 2014. http://www.codeproject.com/Articles/148251/How-to-Successfully-Install-Kinect-on-Windows-Open

The Kinect Accelerometer Coordinate System. Digital image. Accelerometer. Microsoft; Developer Network, n.d. Web. 01 Aug. 2014. <http://msdn.microsoft.com/en-us/library/jj663790.aspx>.

"Kinect for Windows Gallery." Kinect for Windows Gallery. N.p., n.d. Web. 01 Aug. 2014.

<http://www.microsoft.com/en-us/kinectforwindows/meetkinect/gallery.aspx>.

"Kinect for Windows Sensor Components and Specifications." Kinect for Windows Sensor Components and Specifications. N.p., n.d. Web. 01 Aug. 2014. <http://msdn.microsoft.com/en-us/library/jj131033.aspx>.

Kinect for Windows Sensor. Digital image. Kinect for Windows Sensor Components and Specifications. Microsoft; Developer Network, n.d. Web. 01 Aug. 2014. <http://msdn.microsoft.com/en-us/library/jj131033.aspx>.

"Kinect Setup." Kinect Setup. Xbox.com, n.d. Web. 01 Aug. 2014 http://support.xbox.com/en-GB/xbox-360/kinect/kinect-sensor-setup 

Kirn, Peter. ""Open" Kinect Tools Go Closed and Dead, Limiting Artist and Hacker Options; Call for Help." Create Digital Motion RSS. Create Digital Motion, n.d. Web. 01 Aug. 2014. <http://createdigitalmotion.com/2014/02/open-kinect-tools-go-closed-and-dead-limiting-artist-and-hacker-options-call-for-help/>.

 Specifications for the Kinect. N.p., n.d. Web. 01 Aug. 2014. <http://msdn.microsoft.com/en-us/library/jj131033.aspx>.

The Kinect Accelerometer Coordinate System. Digital image. Accelerometer. Microsoft; Developer Network, n.d. Web. 01 Aug. 2014. <http://msdn.microsoft.com/en-us/library/jj663790.aspx>.

The Last Of Us; Motion Capture. Digital image. Kill Streak Media. Naughty Dog, n.d. Web. 1 Aug. 2014. <http://killstreakmedia.com/wp-content/uploads/2013/12/vlcsnap-2013-12-22-21h02m38s161.png>.

Wikipedia contributors. "Infrared." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 29 Jul. 2014. Web. 1 Aug. 2014. <http://en.wikipedia.org/w/index.php?title=Special:Cite&page=Infrared&id=618984182>

Wikipedia contributors. "Kinect." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 18 Jul. 2014. Web. 24 Jul. 2014. http://en.wikipedia.org/w/index.php?title=Kinect&oldid=617424927

Wikipedia contributors. "Motion capture." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 31 Jul. 2014. Web. 1 Aug. 2014. http://en.wikipedia.org/wiki/Motion_captur