Wednesday, October 28, 2009

How to select 3D object from cursor/touch in OpenGL ES

When writing 3D games there is a requirement quite common (forget about OpenGL ES at the moment):

How to select an 3D object in the scene, by mouse/touch?

This problem is not so trivial as it seems. Actually the more you think about it, the more you will find it is not so easy to solve.

Basically, we are trying to map the windows coordinate back to the 3D coordinate. If you understand how an object in OpenGL coordinate is projected onto your 2D display:

screen_vector = ((3D_vector_4x1) * model_view_matrix_4x4) * projection_matrix_4x4;

All we need to do is do the reverse, multiply the screen_vector with the reverse of model view/projection matrices:

3D_vector_4x1 = (screen_vector * (projection_matrix_4x4_inv)) * model_view_matrix_4x4_inv;

Actually, in desktop version of OpenGL, you don't have to do the calculation yourself. There is the mighty gluUnproject that handles it for you.

But there is still a problem. Since your LCD screen is 2D, you only get the x,y component of screen_vector. Where is z? z is actually the depth component and you can find it in the depth buffer. In desktop version of OpenGL, you can call glReadPixels with GL_DEPTH_COMPONENT flag. Of course it is not the full story: the depth buffer only stores pixel depth values that are minimal (unless you change the OpenGL depth function), when transformed back to 3D_vector_4x1 you only get the nearest object. Usually it is desired, but if your wanted object is painted behind other transparent polygons, there is a little bit of problem - but hey it's not a big deal.

But when comes to OpenGL ES, there is a big problem implementing all these:
1. You don't get GLU package. So there is no gluUnproject.
2. You don't have access to the depth buffer. (No GL_DEPTH_COMPONENT for you!)

So the previous solution won't work!
Face it, in OpenGL ES it is a one-way road: you can get windows coordinate from a 3D coordinate, but there is no other way around. Then how to solve the problem?

Well... brutal force always works. First, you define what object is selectable in the game world. Second, you calculate the window coordinate of that object during OpenGL paint, keeping it updated. Last, when a mouse click/touch happens, compare the event windows coordinate to all existing window coordinates that updated during OpenGL paint, and pick one that is nearest. Of course, it is not universal and not all objects in the world can be selected; but hey, do you really need that?

Or you can use ray-tracing.

Thursday, October 22, 2009

The Matrix Phone Improved - been Objective C, now go back to C++

The first version of The Matrix Phone is submitted to AppStore and is pending for review; however, this is far from the end. I'm now planning to develop a better game scheme for The Matrix Phone.

(Now the geek part)
As my first application that is written in Objective C, it is far from optimal: I were not familar with Objective C at that time - now I realize that the code is a mess, and is almost impossible to refactory to support new gaming schemes. After a period of struggling, I decide to re-write it in C++ (plus some Objective C code for iphone); I just have to do this anyway, maybe The Matrix Phone can be on Microsoft Zune.

Been Objective C, now go back to C++:
I know the STL containers are not the only way, but without them and use NS containers of Objective C just don't feel right! Basically, the NS containers is generic typeless "store-by-pointer" containers like those in Java - but without garbage collection. And you can't store regular C structure by value; and there is no auto boxing/unboxing so even if it is int you have to pack it/cast it back manually with NSNumber. So I avoid using it and end up with stinky malloc.
And string operation using NSMutableString is so painful. Even putting two strings together is a hard job. No simple "+" like Java; no container-like operations like std::string. So I just use C string but that's no better...

Thursday, October 15, 2009

The Matrix Phone demo, and technical thoughts

The Matrix Phone is a simple application that simulates the digital rain effect seem in my favorite movie trilogy The Matrix.

The first demo version of The Matrix Phone is complete. Please check out the video attached (I hope this one has a higher resolution than YouTube!).

I will submit this app to the App Store soon... Please wait patiently.

Now... the technical discussion. (Ignore the following if you are not a true geek)
The Matrix Phone supports only iPhone 3GS and third generation iPod Touch since it uses OpenGL 2.0, and OpenGL 2.0 is not backwards compatible. Since the resolution of the video is low, it is hard to see the glow and the interlaced scan line effect that intentionally simulates the old CRT (please see the screenshot); the effect is done through a pixel shader... (maybe it is an over kill, maybe it can also be implemented using blending of the old-school Open GL 1)