3D scenes are represented in computer memory by making use virtual 3D models with vertices and polygons. These 3D models are then rendered to a 2D output device such as a computer screen. So how do we pick objects in a 3D scene using a 2D input device such as a mouse?
We can achieve 3D object picking by making use of colors. The image below shows two cube models (one cube in front of the other), with a (x,y) mouse click that occurred roughly at the center of the image. So how can we determine which model the user intended to select?
Firstly, we will assume that when the user makes a (x,y) selection in the 3D scene, that the object closest to him/her is the model that needs to be picked. Let's render the scene again, but this time we use an unique single color for each model in its entirety, with the OpenGL lighting feature disabled. The result might look something similar to the following image:
All we need to do now is to read the rendered OpenGL buffer and get the color of the pixel at the (x,y) coordinate of the mouse. In the example above we'll get an orange color. We can then determine from the orange color, that Model B was picked by the user.
Although this picking technique requires two renders of our scene, once for display and once for picking, it provides a very easy and effective way to give users the ability to pick 3D models using a 2D input device.
' assign the new value to the position object Position.x = newPos.x Position.y = newPos.y Position.z = newPos.z12. Add a second method named "SetColor" to the R3DTModel class with a parameter list of "newCol As R3DTColor".
Dim i as Integer ' loop variable Dim poly As R3DTPolygon ' temporary polygon object ' loop through all the polygons and update their colors with the new value for i = 0 to Polygon.Ubound poly = Polygon(i) ' get next polygon ' assign the color values poly.SurfaceColor.Red = newCol.Red poly.SurfaceColor.Green = newCol.Green poly.SurfaceColor.Blue = newCol.Blue next i14. Add a class named "R3DTScene" to the project.
' enable depth buffer to ensure objects in distance is not drawn over objects closer to us OpenGL.glEnable OpenGL.GL_DEPTH_TEST OpenGL.glDepthMask OpenGL.GL_TRUE ' make sure only back faces are culled and enable culling OpenGL.glCullFace OpenGL.GL_BACK OpenGL.glEnable OpenGL.GL_CULL_FACE ' enable lighing OpenGL.glEnable OpenGL.GL_LIGHTING18. Add a method named "R3DT_IntegerToColor" to the R3DT_OpenGL module with a parameter list of "intVal As Integer".
Dim colMB As new MemoryBlock(4) ' memory block used to store color value Dim col As R3DTColor ' load the integer map name value into the memory block colMB.UInt32Value(0) = intVal ' use the red, green, and blue values in the memory block to instantiate a color object col = new R3DTColor(colMB.UInt8Value(0), colMB.UInt8Value(1), colMB.UInt8Value(2)) return col ' return the color to the caller21. Open the R3DT_RenderModel method in the R3DT_OpenGL module.
Dim i As Integer // loop variable Dim j As Integer // loop variable Dim poly As R3DTPolygon ' temporay polygon used for drawing Dim vertex As R3DTVector ' temporary vertex used for drawing Dim col As R3DTColor ' temporary object to store color OpenGL.glPushMatrix ' save the current matrix ' set the position of the model OpenGL.glTranslatef model.Position.x, model.Position.y, model.Position.z ' if picking model, set the color once using the model's map name ' all the models of this polygon will be drawn with the same color when drawn for picking if PickModel then col = R3DT_IntegerToColor(model.MapName) ' convert the map name to a color OpenGL.glColor3ub(col.Red, col.Green, col.Blue) ' set the color of the model end if ' loop through all the polygons of the model and draw each one for i = 0 to model.Polygon.Ubound ' get the next polygon from our model poly = model.Polygon(i) ' set the color of the polygon only if we are not picking models if not PickModel then OpenGL.glColor3d poly.SurfaceColor.Red, poly.SurfaceColor.Green, poly.SurfaceColor.Blue end if ' set the normal of the polygon OpenGL.glNormal3d poly.Normal.x, poly.Normal.y, poly.Normal.z ' draw the polygon OpenGL.glBegin OpenGL.GL_POLYGON ' start drawing of polygon ' loop through all the vertices of the current polgon for j = 0 to poly.VertexIndex.Ubound ' get the next vertex of the polygon vertex = model.Vertex(poly.VertexIndex(j)) ' add the vertex to the OpenGL vertex list for the current polygon OpenGL.glVertex3d vertex.x, vertex.y, vertex.z next j OpenGL.glEnd ' end drawing of polygon next i OpenGL.glPopMatrix ' restore the matrix we saved before rendering the model24. Add a method named "R3DT_RenderScene" to the R3DT_OpenGL module with a parameter list of "scene As R3DTScene, Optional PickModel As Boolean = False".
Dim i As Integer ' loop variable Dim model As R3DTModel ' tempory model used for rendering OpenGL.glPushMatrix ' save matrix OpenGL.glLoadIdentity() ' load identity matrix ' lighting needs to be disabled when drawing a model map for picking if PickModel then OpenGL.glDisable OpenGL.GL_LIGHTING ' disable lighting end if ' clear the background and depth buffer OpenGL.glClearColor(0, 0, 0, 1) OpenGL.glClear(OpenGL.GL_COLOR_BUFFER_BIT + OpenGL.GL_DEPTH_BUFFER_BIT) ' move back a bit so that we can see the object OpenGL.glTranslatef 0.0, 0.0, -20.0 ' loop through all the models in the scene for i = 0 to Scene.Model.Ubound ' get the next model model = Scene.Model(i) ' render the model R3DT_RenderModel model, PickModel next i ' enable lighting again if we disabled it for model picking if PickModel then OpenGL.glEnable OpenGL.GL_LIGHTING ' enable lighting end if OpenGL.glPopMatrix ' restore matrix26. Add a method named "R3DT_PickModel" to the R3DT_OpenGL module with a parameter list of "scene As R3DTScene, x As Integer, y As Integer".
Dim model As R3DTModel ' this variable will hold the model that is picked Dim i As Integer ' loop variable Dim pixelData As New MemoryBlock(4) ' holds pixel data read from OpenGL buffer Dim pickMapName As Integer ' first we need to set up map names for all the models in the scene for i = 0 to scene.Model.Ubound scene.Model(i).MapName = i + 1 next i ' clear the background and depth buffer OpenGL.glClearColor(0, 0, 0, 1) OpenGL.glClear(OpenGL.GL_COLOR_BUFFER_BIT + OpenGL.GL_DEPTH_BUFFER_BIT) ' render the model map of the scene R3DT_RenderScene scene, True ' get the pixel value at the given (x, y) position OpenGL.glReadPixels(x, y, 1, 1, OpenGL.GL_RGB, OpenGL.GL_UNSIGNED_BYTE, pixelData) ' store the color of the pixel as the map name pickMapName = pixelData.UInt32Value(0) ' did we get a pixel at (x, y) other than black? if pickMapName > 0 then ' yes, so get the model with this mapname i = 0 while (i <= scene.Model.Ubound) and (model = nil) ' did we find the model with the picked map name? if scene.Model(i).MapName = pickMapName then ' yes, so assign it to our return model object model = scene.Model(i) end if i = i + 1 ' move to next model wend end if return model ' return the model object to the caller29. Add a property named "Scene" of type R3DTScene to Window1.
Dim i As Integer ' loop variable Dim model As R3DTModel ' tempory model object Dim pos As new R3DTVector(0, 0, 0) ' vector to store random position Dim col As new R3DTColor(1, 1, 1) ' object to store random color Dim rnd as New Random ' object to generate random numbers with ' instantiate the Scene object Scene = new R3DTScene ' generate random models in our 3D scene for i = 1 to 100 ' generate random position pos.x = rnd.InRange(0, 30) - 15 pos.y = rnd.InRange(0, 30) - 15 pos.z = rnd.InRange(0, 10) - 5 ' generate random color col.Red = rnd.InRange(1, 100) / 255 col.Green = rnd.InRange(100, 255) / 255 col.Blue = rnd.InRange(1, 255) / 255 ' instantiate a new cube model model = R3DT_GetModel_Cube model.SetPosition pos ' set the position of the cube model.SetColor col ' set teh color of the cube Scene.Model.Append model ' add the model to our scene next i31. Add the following code to the Paint event of Window1:
' refresh the OpenGL surface OpenGLSurface1.Render32. Add the following code to the Open event of OpenGLSurface1:
R3DT_InitOpenGL ' initalize OpenGL environment ' instantiate memory blocks used to configure our light with Dim light_position As new MemoryBlock(16) Dim light_ambience As new MemoryBlock(16) Dim light_diffANDspec As new MemoryBlock(16) ' define position of the light light_position.SingleValue(0) = 0.0 light_position.SingleValue(4) = 0.0 light_position.SingleValue(8) = 1.0 light_position.SingleValue(12) = 0.0 ' define ambience of the light light_ambience.SingleValue(0) = 0.0 light_ambience.SingleValue(4) = 0.0 light_ambience.SingleValue(8) = 0.0 light_ambience.SingleValue(12) = 1.0 ' define diffuse and specular of the light light_diffANDspec.SingleValue(0) = 1.0 light_diffANDspec.SingleValue(4) = 1.0 light_diffANDspec.SingleValue(8) = 1.0 light_diffANDspec.SingleValue(12) = 1.0 ' apply light settings OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_POSITION, light_position ' set position OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_AMBIENT, light_ambience ' set ambience OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_DIFFUSE, light_diffANDspec ' set diffuse OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_SPECULAR, light_diffANDspec ' set specular OpenGL.glEnable OpenGL.GL_LIGHT0 ' enable our light source ' enable overwriting of material properties with vertex colors OpenGL.glEnable OpenGL.GL_COLOR_MATERIAL33. Add the following code to the Resized event of OpenGLSurface1:
' set the viewport rectangle OpenGL.glViewport 0, 0, OpenGLSurface1.Width, OpenGLSurface1.Height ' set up the perspective projection settings OpenGL.glMatrixMode OpenGL.GL_PROJECTION OpenGL.glLoadIdentity OpenGL.gluPerspective 60.0, OpenGLSurface1.Width / OpenGLSurface1.Height, 1, 100.0 ' select and reset the modelview matrix OpenGL.glMatrixMode OpenGL.GL_MODELVIEW OpenGL.glLoadIdentity34. Add the following code to the Render event of OpenGLSurface1:
' render the Scene with OpenGL R3DT_RenderScene Scene35. Add the following code to the MouseDown event of OpenGLSurface1:
Dim model As R3DTModel ' model that was picked Dim rnd As Random ' object to generate random numbers with ' pick the model located at the (x, y) position of the mouse ' the OpenGL y axis increase from bottom to top, so we need to ajust our y value to accomondate this Me.MakeCurrent model = R3DT_PickModel(Scene, x, OpenGLSurface1.Height - y) ' was there a model located at this position? if model <> nil then rnd = new Random ' yes, so lets give the model a new random color model.SetColor (rnd.InRange(100, 255) / 255, rnd.InRange(1, 200) / 255, rnd.InRange(1, 200) / 255) OpenGLSurface1.Render ' refresh the OpenGL view end if36. Save and run the project.
Let's first have a look at the new methods and properties added to the classes we created in Tutorial 9 - Order in chaos.
The MapName property that we added to the R3DTModel class will be used to store the Integer representation of the "RGB-map-color" used to identify the model during our picking routine.
We added a second SetPosition method to the R3DTModel class that takes a single R3DTVector parameter, instead of separate x, y, and z values. Strictly speaking this method does not expand the capabilities of our R3DTModel class, but simply makes it easier for us to set new model positions using existing vector objects. The addition of a second SetPosition method with a different parameter signature, is better known as method overloading or polymorphism in the world of computer science.
Similarly, we added a second SetColor method to R3DTModel, that simply makes things easier when working with models and existing R3DTColor color objects.
A new class, named R3DTScene is added to our project. This class has an array (Model()) that holds all stores all the model objects of your scene. The image below provides a basic illustration of the purpose of the R3DTScene class.
With the R3DT_InitOpenGL method added to the R3DT_OpenGL module, we started to create a initialization function for your 3D engine. In future tutorials and projects we can now simply call the R3DT_InitOpenGL method in the OpenGLSurface.Open event, instead of retyping the same code each time.
The new R3DT_IntegerToColor method is a helper function we use in our picking routines. It simply converts an Integer to a corresponding RGB color value.
A new optional parameter named PickModel (with its default value set to False), is added to our R3DT_RenderModel method. Instead of writing a new rendering routine to pick models, we tweak the existing render routine so that we can use it for both display and pick rendering. Using one method, for both display and picking, makes maintainability easier in the future when we improve and optimize the rendering routine.
The R3DT_RenderScene is similar to the R3DT_RenderModel method, with the difference being that it renders all the models found in a scene to the screen, instead of just a single model. Take note that the first parameter passed to it is of the type R3DTScene. It also has a PickModel argument (with its default value set to False), so that we can use it for both display and pick rendering.
R3DT_PickModel is the method that does the magic of picking a model in a 3D scene using 2-dimensional x and y values. Note that we only pass it a R3DTScene object, together with the required x and y values, and it will automatically return the R3DTModel object that was picked. If the given (x,y) coordinate does not contain a model (e.g. a black area on the screen), then R3DT_PickModel will return a Nil value.
Finally we put our R3DT_PickModel to good use on line 7 in the OpenGLSurface1.MouseDown event. There is one last concept we need to understand though. Notice how we give the value (OpenGLSurface1.Height-y) instead of y, when calling R3DT_PickModel. The coordinate system of OpenGL is implemented a little bit different than the coordinate system used for computer screens. A computer screen's y-axis increases from top to bottom, whereas the y-axis in OpenGL increases from bottom to top (see image below). We therefore need to match the screen (x,y) coordinate to the OpenGL (x,y) coordinate, by subtracting y from the total height of the OpenGL surface.
That concludes this tutorial. See if you can replace the code in the Window1.Open with your own code, and build your own scene made up from 3D cube models.